comScore widget matrix numbers are innacurate

Techcrunch recently published a post about “The Widget Kings” which promoted the comScore widget matrix as a symbol of rank among widget manufacturers. We did a little research on the accuracy of these numbers – to make a long story short; we found the numbers entirely inaccurate and incomplete as ranking of widget vendors.

This is not a new perspective, both GigaOm and Jeff Jarvis posted about this back in June 2007 when the comScore list was first released.

It’s difficult enough to track traffic accurately on the internet, much less widgets, so we weren't surprised to see some inconsistencies; it is to be expected when reports like these are first generated. But when the numbers are deceptive and wrong, the report loses all credibility as an independent ranking of widget vendors.

Let’s compare the list in April from the report just released in November. For our analysis, we looked into the changes in the standings and tried to validate their statistics with Compete and Alexa. While we appreciate that comScore, Compete and Alexa don't all track the same way, we were hoping these sites could at least get a sense of whether these other sites might show traffic increasing or decreasing over that time period.

April 2007 comScore Widget Matrix

November 2007 comScore Widget Matrix

Here are the things that jump out immediately.

1) Brightcove is off the list. They went from 16.9 million unique to less that 14.9 million? Let’s try to corroborate that. Here are charts from Compete.com and Alexa.com

Again, traffic is difficult to measure, but at the very least, both Compete and Alexa point to flat growth, not an 11% loss in audience.

2) Slide.com dropped from 117.1 million uniques to 39 million. Sounds like they are in trouble? Not according to Alexa and Compete.

3) Musicplaylist.us at 15 million uniques in 4/07 and 11/07…

How does this work? Traffic to musicplaylist looks to be in a freefall.

I could go on – none of the numbers seem to make sense. Is comScore playing a shell game for their paying clients? Or is this a true third party representation of widget traffic?

Let us know your thoughts in the comments!

Comments

[Erick Schonfeld]

While I agree the comScore widget numbers are deeply flawed (TC was not "promoting" the numbers, we were reporting them, and we updated the original post with numbers provided from Newsgator and others to reflect the inconsistencies and omissions in the comscore data), the Compete and Alexa data you present here as a counterweight is meaningless.

Compete and Alexa measure traffic to the corresponding Websites, not impressions on widgets scattered throughout the Web. You are comparing apples to oranges.

comScore's Widget Metrix is an attempt to pull out just the widget numbers. It obviously needs a lot of work. But a third-party auditor of widget traffic is sorely needed.

So why don't you help them improve their measurements instead of complaining about it?

[Jeff Nolan]

Erick,
We're all in violent agreement here and certainly didn't intend to portray TC as complicit in any presumed deception.

We are simply pushing the agenda of a need for a transparent and reliable third party auditing service. This is not to suggest the Comscore data is incorrect either, just incomplete. If the auditing service is pay-to-play, then they should disclose that as well.

If there is one thing that sites like TC could do that would be positive, it would be to cover these numbers with a healthy dose of skepticism, not to devalue Comscore or anyone else, but to remind readers that there is no agreed upon reliable and consistent way to report traffic numbers. At best there are a collection of services that serve to triangulate on reasonable number ranges.

[Walker Fenton]

Erick,

We’re in agreement – the numbers are tough to follow – and thank you for updating your story to include our numbers. The point of this post was not to discredit TC, but to analyze the comScore data.

The point of providing domain specific traffic was meant to show a relative comparison of the traffic… (from the post)

“While we appreciate that comScore, Compete and Alexa don't all track the same way, we were hoping these sites could at least get a sense of whether these other sites might show traffic increasing or decreasing over that time period.”

While we didn’t measure the statistical significance of the traffic to the domain against widget traffic – we don’t believe it’s a leap of faith that the two are related.

We would love to help comScore improve their measurements, I would just like to make sure that their motivations as an independent auditor are not financial.

[Thedigi Talhobo]

I'd guess that the Brightcove drop has a whole lot to do with their dropping the personal platform. When you take the tool out of the hands of the people, its tough to keep an audience.

Just a theory.