comScore widget matrix numbers are innacurate

Techcrunch recently published a post about “The Widget Kings” which promoted the comScore widget matrix as a symbol of rank among widget manufacturers. We did a little research on the accuracy of these numbers – to make a long story short; we found the numbers entirely inaccurate and incomplete as ranking of widget vendors.

This is not a new perspective, both GigaOm and Jeff Jarvis posted about this back in June 2007 when the comScore list was first released.

It’s difficult enough to track traffic accurately on the internet, much less widgets, so we weren't surprised to see some inconsistencies; it is to be expected when reports like these are first generated. But when the numbers are deceptive and wrong, the report loses all credibility as an independent ranking of widget vendors.

Let’s compare the list in April from the report just released in November. For our analysis, we looked into the changes in the standings and tried to validate their statistics with Compete and Alexa. While we appreciate that comScore, Compete and Alexa don't all track the same way, we were hoping these sites could at least get a sense of whether these other sites might show traffic increasing or decreasing over that time period.

April 2007 comScore Widget Matrix

November 2007 comScore Widget Matrix

Here are the things that jump out immediately.

1) Brightcove is off the list. They went from 16.9 million unique to less that 14.9 million? Let’s try to corroborate that. Here are charts from Compete.com and Alexa.com

Again, traffic is difficult to measure, but at the very least, both Compete and Alexa point to flat growth, not an 11% loss in audience.

2) Slide.com dropped from 117.1 million uniques to 39 million. Sounds like they are in trouble? Not according to Alexa and Compete.

3) Musicplaylist.us at 15 million uniques in 4/07 and 11/07…

How does this work? Traffic to musicplaylist looks to be in a freefall.

I could go on – none of the numbers seem to make sense. Is comScore playing a shell game for their paying clients? Or is this a true third party representation of widget traffic?

Let us know your thoughts in the comments!

Comments

[Erick Schonfeld]

While I agree the comScore widget numbers are deeply flawed (TC was not "promoting" the numbers, we were reporting them, and we updated the original post with numbers provided from Newsgator and others to reflect the inconsistencies and omissions in the comscore data), the Compete and Alexa data you present here as a counterweight is meaningless.

Compete and Alexa measure traffic to the corresponding Websites, not impressions on widgets scattered throughout the Web. You are comparing apples to oranges.

comScore's Widget Metrix is an attempt to pull out just the widget numbers. It obviously needs a lot of work. But a third-party auditor of widget traffic is sorely needed.

So why don't you help them improve their measurements instead of complaining about it?

[Jeff Nolan]

Erick,
We're all in violent agreement here and certainly didn't intend to portray TC as complicit in any presumed deception.

We are simply pushing the agenda of a need for a transparent and reliable third party auditing service. This is not to suggest the Comscore data is incorrect either, just incomplete. If the auditing service is pay-to-play, then they should disclose that as well.

If there is one thing that sites like TC could do that would be positive, it would be to cover these numbers with a healthy dose of skepticism, not to devalue Comscore or anyone else, but to remind readers that there is no agreed upon reliable and consistent way to report traffic numbers. At best there are a collection of services that serve to triangulate on reasonable number ranges.

[Walker Fenton]

Erick,

We’re in agreement – the numbers are tough to follow – and thank you for updating your story to include our numbers. The point of this post was not to discredit TC, but to analyze the comScore data.

The point of providing domain specific traffic was meant to show a relative comparison of the traffic… (from the post)

“While we appreciate that comScore, Compete and Alexa don't all track the same way, we were hoping these sites could at least get a sense of whether these other sites might show traffic increasing or decreasing over that time period.”

While we didn’t measure the statistical significance of the traffic to the domain against widget traffic – we don’t believe it’s a leap of faith that the two are related.

We would love to help comScore improve their measurements, I would just like to make sure that their motivations as an independent auditor are not financial.

[Thedigi Talhobo]

I'd guess that the Brightcove drop has a whole lot to do with their dropping the personal platform. When you take the tool out of the hands of the people, its tough to keep an audience.

Just a theory.

[Mike Shaw]

The first graph looks at worldwide data and the second graph is US-audience only.

[buy ambien without a prescription]

Where I can find 2008 year statistic?

[Alopecia]

google!!!!
n.n
Or some place else jeje
lol
Cheryl

[microgaming]

Nice statistics and charts, I love charts because they depict the reality much easier than words. Thanks a lot for your research job and posting it here, much appreciated. I came to this blog by googling and I think I will drop by more often. I am doing a research job and this post helped me a lot.

[Treatment For Hair Loss]

Interesting graphics, thanks for the information!

[buy viagra]

He vanished for a bit, but, god, what a great post. I'm so glad he didn't stay away.

[Cialis en Espana]

You have a very good blog that the main thing a lot of interesting and useful!

[Ecig Reviews]

Widgets arent created equal.

[Vapor King]

does anybody use widgets anymore?

[carr]

hola

[adenoid surgery]

That's absolute right these numbers are innacurate

[acomplia]

All charts going down. Bad sign.

[generic cialis]

I am currently conducting an investigation regarding this issue. The information presented in your blog is really interesting so I want to thank and also congratulate the great work. thanks again.

[generic propecia]

I also meant to tell you that you can comment to your heart's
content in the comment section and it doesn't turn on any lights nor
do you receive any GA notification. You're only aware of it if you
check your question activity regularly. Feel free to chat with us
there as much as you like.

[cialis online]

your blog is very interesting. the information it contains is very comprehensive and entertaining. I thank you for your help.

[zenegra 100mg]

The recent explosion of user-generated content has helped create a worldwide marketplace for widgets,” said Linda Boland Abraham, executive vice president at comScore. “comScore is excited to be providing measurement for this developing content medium

[generic viagra]

comScore, a leader in measuring the digital world, today announced the launch of comScore Widget Metrix, a new service to track the usage of widgets across the Web. comScore currently tracks Web widgets, which are data files that can be embedded into a site’s HTML code and are typically displayed in a small viewing pane on the site.

[lisa]

You can also try http://www.estimix.com – a free tool that provides a nice summary of the website performance. The estimation provided by estimix is the result of a complex analysis based on factors like: the age of the website, the demographic structure of the traffic, the countries where the website is popular and sources of the traffic.

[bangbros]

This is not a new perspective, both GigaOm and Jeff Jarvis posted about this back in June 2007 when the comScore list was first released.

[cheap caverta]

Hey this is really very interesting post found here... Good job done by both GigaOm and Jeff Jarvis...

Thanks very much!

[Ways to Lose Weight]

Of course they're inaccurate.