Chartbeat’s suggestion for how to measure content quality

From Chartbeat CEO Tony Haile:

Pageviews do not measure the quality of a piece of content or its ability to hold and engage an audience; it’s a measure of the provocativeness of link copy. That’s it. It’s highly gameable, and by separating the metric of success (clicking on the link) from any relation to the content itself, that means the cheapest, most provocative link creator will always have the advantage.

Here’s the thing though. When you actually sit down with publishers, across the board there is an enormous degree of consistency around what their goals are and what really matters to them — and it’s not pageviews. What matters is building an audience who knows who you are, likes what you do and most importantly comes back to you again and again…

When you eschew chasing pageviews and make building a loyal audience your goal, it not only aligns commercial and editorial goals, but also dramatically increases the pace of cultural change within newsrooms. It turns out that what loyal audiences care about is what good editorial teams care about too: great articles that capture time and attention.

Tony is right that pageviews is a poor measure of article quality. Sites that optimize for pageviews inevitably publish provocative headlines and populist content. But time on page (which Chartbeat provides) isn’t a solution, because shorter articles aren’t necessarily lower quality or less valuable. This is particularly true in finance, where investors want information which has maximum signal-to-noise ratio, is fast to read, and is delivered instantaneously. Like this.

So what’s the right way to measure content quality for us?

9 thoughts on “Chartbeat’s suggestion for how to measure content quality

  1. “Read to End” would seem to be the obvious possibility (though that might incentivize shorter articles). “Sharing” would seem to be a powerful endorsement as well. What we really need is a way of measuring an investors demonstrated thought “that was valuable, I’ll take action on it”.

  2. Engagement in comments on the article? The challenge of course is the noise created when a discussion veers off topic, but it seems a reasonable theory that a quality article can and will generate quality conversation.

  3. Quick and easy solution for all articles/news updates would be adding a “like” button.
    you can also use your already existing “read later” bookmark as an internal (yet limited) metric.
    Last but not least – if you add an “event” (for example “read more” / “Like”) on google analytics, you will get a more accurate time on site, which is also a nice metric.
    further reading here

    • Gadi, thanks for your comment. When we tried that, we discovered an interesting problem. Because Seeking Alpha articles are often buy or sell opinions about stocks, people seemed to “like” articles that they agreed with rather than which they thought were good articles. In aggregate, the “likes” told us little about article quality.

      Perhaps the solution is to design actions which don’t confuse rating quality with some other motivation, such as social sharing or expressing your own opinion about the topic. As you pointed out, “bookmark” might do that, because it’s your own personal tool with no social motivation.

      • I agree.
        How about if at the end of the article you were to place:
        “How would you rate the professionalism of the article writer?” and put a scale of 1-10.
        It might not be full-proof, but it would definitely focus the reader to rate the article quality.

  4. I like read-to-end as it at least demonstrates that the article held a user’s attention. However, it is also highly correlated with length of article and it might be that you can only judge quality once you have finished reading the thesis so users might reach the end but decide they didn’t like it. Equally many of the other measures can also be unclear:
    – Read later might just mean the article is too long to read now
    – How do you know if a ‘like’ means “I agree” or “I like it a lot and will be back for more”
    – I might like an article a lot but think it is interesting to me and not my friends so I won’t share it

    We offer Seeking Alpha users the ability to add Seeking Alpha content to their workflow through downloading our app (and getting notifications) or signing up for email alerts. Maybe the % of users who sign up for these after reading a piece of content is a good measure. It demonstrates how much long term commitment has been generated by a piece of content.

  5. Thinking about this again today, I’d like to suggest a combination of an old metric and a new one:

    Back before page views, freelance writers used to be paid by the word. Maybe moving back to such a model is exactly what we need to make a read to end payment system make the most sense. We could pay based on (word count) * (number of read to ends). That would mean we’d pay more for longer content that people actually find worth reading and less for click bait with a clever title.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s