Last month in the Wall Street Journal, I read an article by Carl Bialik, aka “The Numbers Guy,” in which he mentioned the use of retailers’ UPC sales scanner data in the consumer packaged goods industry for measuring manufacturers’ market share trends. According to The Numbers Guy “Industrywide U.S. sales data have been hamstrung by the omission of the country’s biggest retailer since 2001, when Wal-Mart stopped sharing its figures.” As you can imagine, because of the importance of Wal-Mart, the omission of their data in the syndicated tracking services raises major issues of accuracy and reliability of the data for gauging market trends.

The article reminded me that the need to build a representative sample was foremost in our minds back in 1999 when we first began designing the comScore service. We considered the option of building a database by trying to obtain clickstream data directly from the ISPs, but abandoned it because we decided that the likelihood was too high that we wouldn’t be able to obtain cooperation from all of the major providers. Because of this, we concluded that an Internet user sample obtained directly from ISPs could face valid criticism if consumers’ online behavior turned out to be different across the various ISPs -- and if the major ISPs weren’t represented in the sample. Think of the need for clickstream data from each ISP in an online audience ratings service as being analogous to requiring sales data from each retailer in a CPG sales tracking database. It was for this reason that we concluded it was necessary for comScore to directly recruit a panel of Internet users so that we could be sure we had representation of all ISPs in our database. Now, eight years later, I thought it would be interesting to examine the wisdom of that decision by examining consumer behavior across the major ISPs -- as measured through the comScore database.

I decided that examining consumers’ search behavior would provide a rather interesting case in point. I started by looking at Google’s share of the search market within the major ISPs:

As the above chart reveals, Google’s market share varies by more than 20 percentage points, from a high of almost 60 percent within Comcast ISP subscribers to a low of slightly more than 30 percent among AOL users. That’s evidence enough to conclude that without representation from the major ISPs one would have a skewed sample when it comes to measuring the performance of the search market leader.

But, wait, there’s more.

Next, I looked at the share that each of the other major search engines -- Yahoo, MSN and AOL – held within their “partner” ISPs. That is, SBC in the case of Yahoo, Verizon in the case of MSN and AOL in the case of AOL:

I was initially taken aback by the degree to which consumers’ online search behavior varies dramatically by ISP. Specifically, each search engine’s market share within subscribers to its partner ISP is far, far higher than the engine’s overall market share. Take MSN. Its overall share of searches is 10.3 percent. Yet, within subscribers to Verizon’s ISP service MSN’s share climbs to 39.4 percent! But, as I thought about it, I realized that these results simply reflect the preferred relationship between the ISP provider and the particular search engine. As an example, the terms and conditions of the SBC high speed service specify that: “AT&T Yahoo! High Speed Internet is provided by AT&T Internet Services with customized content, services, and applications from Yahoo!” It’s easy to see how a user of SBC’s ISP could become more oriented to the use of Yahoo’s search engine than other engines – especially if they’ve installed the Yahoo! toolbar. A similar dynamic exists with the other search engines and their partner ISPs.

My second reaction was to breathe a sigh of relief that back in 1999 Magid and I had the foresight to not go the route of trying to acquire user data directly from the ISPs. Without clickstream data from the important ISP providers, my analysis clearly shows that an online audience measurement service can be decidedly biased.

comScore's cookie deletion white paper is now available to the public.

This white paper goes into much greater detail than previously released, and is a must-read for those serious about understanding Web measurement. Please visit www.comscore.com/request/cookie_deletion.asp to download the study.

Hi, my name is James Lamberti, SVP Search and Media at comScore. I’m heavily involved in our search work and often partner with clients and the industry to dive deeper into the role and value of search. On June 27th, I am delivering a keynote at Searchnomics 2007, titled “Beyond Direct Response: Search's Role in the Media Mix for Branding and Offline Sales.” This keynote is a great opportunity to show the industry, using a variety of real data points amassed over the past year, that search is much more than the direct-response vehicle that launched the industry.

One key data point at the center of this discussion is the offline sales impact of search. We have conducted research across dozens of retail and non-retail categories and find consistently that 65% to 90% of conversion occurs offline. These figures alone have a profound impact on the strategy and economics of search. For example, pure-plays like eBay and Amazon rely extensively on search to drive ecommerce. How do the economics change when multi-channel nationals like Wal-Mart and Target achieve 5x or 10x the ROI as pure-plays? Issues like the value of search in branding, its link to other media, and the opportunity to re-engage agency creatives are all a part of the next level of search marketing. Be sure to check out the Searchnomics 2007 event and check back here over the next few days to hear more thoughts about the keynote theme.

As we head toward Searchnomics 2007 on June 27, I’d like to share with you some more thoughts on the future of search marketing.

Branding is a big issue for the search industry and could potentially change the economics of search. Brand-building ad dollars are a huge part of media spending and an attractive market for us as an industry. However, the direct response ecosystem we’ve built makes it hard to adapt to the measurement needs of brand marketers. So what can we do to demonstrate search’s effectiveness to brand marketers and capture these dollars? (Let’s leave aside for a moment the debate over the ability to brand via search – I will come back to that in a later post.)

The most classic and rigorous approach to measuring ROI is to test the effectiveness of a search campaign in driving classic branding measures like awareness, purchase intent, and likelihood to recommend. Using a classic test/control approach, this is the best way to convince a major national advertiser of the value of search in branding. Be warned – not all campaigns will succeed on these measures! That said, in the past year, 75% of search brand tests conducted by comScore showed a significant lift in at least one of the branding metrics. Message association was most often impacted (75%), but awareness and purchase intent were improved 50% of the time. There are a few common threads running through the successful campaigns: integration with other forms of media both online and offline, the “creativity” behind the keyword selection, including integration with the brand communication or creative elements in the offline media, and the size and reach of the campaign.

While major marketers will expect this classic type of testing design, there are alternatives for measuring smaller campaigns. Traditionally, we in the industry shy away from the use of reach and frequency metrics to evaluate search campaigns. For example, I frequently hear the phrase: “search is not a reach vehicle.”

My answer to that is “who are you trying to reach?” If you want to reach the total U.S. Internet population or some major demographic within it, then yes – search will not achieve reach in the same way as TV, Print, Radio or Digital Image campaigns. In fact you will be lucky to reach 1% or 2% of those broad targets using search. But if you are trying to reach, for example, the 15 million consumers who buy a PC in a quarter., search offers one of the most effective, if not the most effective, means of reaching your target audience.

Here’s a rough example for all you numbers people:

~ 15 Million PCs Bought in a Quarter
~ 7 Million People Searched on “Computers” or “Laptops”
= 47% Reach of Search Among Likely “In-Market” Computer Buyers

Can you think of a TV campaign that has that type of reach within a target segment of prospective buyers? Imagine if the industry began reporting search campaigns in this manner? You would get the attention of the CMO of a branding team in a hurry. You’re speaking her/his language and presenting search marketing efforts using data comparable to the offline world. A discussion using reach, frequency, gross rating points (GRP), and impact on awareness / purchase intent is far more likely to engage the brand marketers than keyword lists, natural and paid-click rates, or direct response ROI calculations. We have the data to speak their language – let’s use it!