The day Gemius lost credibility

This longish piece is about a Polish internet audience measurement company, Gemius S.A (a Nielsen//Netratings for Poland you can think), and how it lost credibility in my eyes.

Gemius is conducting two kinds of Internet audience measurement: site-centric and panel.

The site-centric measurement consists in having a piece of Javascript pasted on pages of sites that want to have that kind of measurement (they get reports of page views, unique users etc.). All major Polish portal sites are using that measurement and tens of thousands of smallers sites (through stat.pl which offers them for free). It's hugely popular with the monthly number of tracked page impressions counted in billions.

Gemius clearly knows a lot about the internet usage in Poland. It's been publishing some summary statistics from this site-centric measurement at ranking.pl on a weekly basis. You can find there hard data on the usage of popular browsers, operating systems, screen resolutions, cookies acceptance etc. You can also find stats on the popularity of search engines and directories as measured by the number of redirects they generate to the sites that are monitored.

The panel measurement by Gemius known under the name of PBI/Gemius Megapanel consists of about 20,000 strong group of internet users who agree to have a special tracking software installed on their systems. This gives Gemius additional data on the internet usage and if you combine that with site-centric measurement you get a really good picture of which sites are popular.

The results from the panel measurement, however, are not freely available. You have to pay hard money for them and only a snippet of top 20 most popular sites is free and published monthly at audyt.gemius.pl.

The panel measurement conducted by Gemius has been quite controversial since day one. The first publication of the results was postponed by about a year, there were controversies around the member companies of PBI which decides on many aspects of the panel, and even I raised some privacy issues related to it back in February this year.

One of the controversies is also the fact that only users of Internet Explorer can take part in the panel but Gemius has been promising for a long time that support for alternative browsers will be added soon. This exclusion of about 20% internet users of Firefox, Opera, etc. is often used as an argument against accuracy of the panel results in general. I would say the panel results in general are accurate despite that exclusion except for one site that Gemius doesn't have any site-centric hard data to compare with panel results. The site is Google, and here is what makes me think so:

1. Google search is builtin into Firefox/Mozilla. It's very natural to use it especially that Google results are of decent quality.
2. People who use Firefox/Mozilla or other alternative browsers can be said to be those "in the know" when it comes to technology, software etc. I would say they are very likely to use a top-notch site (Google) instead of Polish ones which return very irrelevant results, serve lay-over Flash ads on the results pages and put their own sites in the top positions. Those in the know are also more likely to do much more searches than average users (I do about 100-200 searches a day).
3. People who are the panel must agree to install a spyware-like software on their systems for no benefit and do this in the times of spyware, viruses and malware being one of the biggest nightmares for internet users. Many opponents of the accuracy of Gemius panel results say these panelists must be "naive internet users". Naive is not the right word here in my opinion. I would call them just clueless noobs who think that internet = Onet + WP + Gadu-Gadu (a popular Polish messenger). These kind of users are not, in my view, very likely to use Google but rather the searches offered by the popular portal sites, i.e. Onet and WP.

So, considering the above three points I would say that Gemius panel results for Google usage may not be very accurate especially when it comes to the number of searches being made (page views).

The other thing about the panel results is that the sites audiences are measures in real users (unlike the so called unique users which are based on cookies that often get deleted) but a site user is defined as a person who makes at least one page impression on a given site in a month (that one impression is enough to be considered a visit in that measurement and it's actually that one visit that is used to define a site user).

This kind of metric is not very useful in my opinion because it favors search engine optimized sites which get a lot of one-time visitors who get counted as users of the site. It leads to sites like idg.pl or gery.pl which barely have 10 million page impressions per month being reported in the top 20 most popular sites while they fail miserably if you look at the numbers of their returning users. On the other hand, sites that are heavily used but not search engine friendly and thus without random visitors (like grono.net, an invitation-only Polish social networking site that is becoming increasingly popular) are nowhere to be found in the top of popularity rankings.

Site rankings with that metric are also easy to be manipulated. For example, if onet.pl, which gets millions of visitors, puts up on its homepage a link or an ad to one of its services and keeps it for some time, it will get many people clicking on it who will be counted as users of that site. If you consider that Gemius is occasionally publishing reports on popularity of specialized sites (like the recent finance and money-related just before Onet's stock sale) then it gives you something to think about. But that's off-topic. What I want to say here is that site rankings with metric that defines real users as those who make at least one page impression don't make much sense.

I'm getting to the point, read carefully now.

In site-centric comparison of search engines as measured by the number of redirects that are made to sites monitored by Gemius, Google has nearly a 70% market share and keeps growing according to ranking.pl. Search engines of the two most popular Polish portal sites, Onet.pl and WP, have a relatively small 11-14% share and that number is getting smaller and smaller each month. This data from ranking.pl is often used by the press and it clearly doesn't make Onet.pl and WP look good in the search engine market.

In the panel results of search engines usage as measured by the number of page impressions being made (which is proportional to the number of searches being made and thus the actual usage), Google has a whopping 80% share while Onet and WP a mere 7%. However, the panel results for searches engines as measured by the number of real users with metric that considers a user someone who makes at least one page impression on the site, Google has a share of 66% (this is a percentage of all users who use search engines only, not of all internet users) while Onet almost half of it (31%) and WP 25% (the percentages don't add up to 100% because people can be users of many sites).

Onet and WP, the two largest sponsors of companies financing Gemius panel, clearly look much better in the panel results if you take the real users metric which, as I said earlier, can be quite inaccurate for Google, and are not very useful with a site user being someone who makes one page impression on it.

Now, considering all this things, Gemius is doing something truly inexplicable:

1. It's taking these rought estimate panel results for search engines and publishing it along site-centric hard data on ranking.pl;
2. It's taking these panel results that you normally have to pay for and publishing them on ranking.pl for free;
3. It's taking panel results of only search engines to publish for free on ranking.pl, not directory or other;
4. It's taking these good for Onet and WP but least useful and accurate metric for search engines usage and makes it the main metric (it's listed as the first in the ranking menu and, what's most important, presented on the ranking.pl home page);

Moreover, Gemius is doing that at the times when there is not a single week without an article in the press talking about how huge spending on advertising in search engines is going to be in the near future.

It is doing that dumb thing clearly as a service for its two largest sponsors major customers and it's definitely not something that will win Gemius trust, at least not mine. If it's motives are to give a fairer picture of the search engine usage in Poland, it shouldn't make the least relevant metric the main one and shouldn't start fairness improvements here but rather with its IM usage rankings that for some reason include only IMs with site-centric measurement in panel results effectively excluding Skype from it which in turn leads to Skype competitors downplaying its popularity in Poland despite Poland being the second most popular country among its millions of users.

UPDATE: Read the update here