I joined comScore as Chief Research officer on May 30, but this is my first post to the comScore blog. I’ve spent a lot of time these past three months meeting with clients and working with industry groups (e.g., MRC, IAB, and ARF), hearing first hand about your perspective on the hot issues in Internet audience measurement and online metrics. As I wrote in my September 7 column for Mediapost’s Online Metrics Insider, I’m thrilled to be working in this space at this time.

Let me take this opportunity to share with you some headlines from the front here and in upcoming posts.

comScore Speaks on Metrics Panels at OMMA, MIXX

Last week, comScore President/CEO Magid Abraham appeared on a panel at the IAB’s MIXX conference moderated by Bob Liodice, the President/CEO of the Association of National Advertisers. In his opening statement, Magid made a brief and concise but forceful case that server-centric counts of Unique Visitors are inflated. Often the panel-versus-server debate dominates this kind of forum; but here Magid managed to pre-empt that discussion.

Increasingly, as comScore and others work to educate the industry, it is becoming clear that site-centric web analytics are simply not measuring the same thing that audience measurement services do. There is absolutely an indispensable place in the publisher business model for such data - but to conflate the measurement of cumulative persons-based audiences (which is what comScore does) with the tracking of the behavior of machines (which is what site-centric data does) is simply not appropriate.

With respect to the counting of Unique Visitors, site-centric data typically can not account for the differences between bots or spiders and people; the inflation that results from cookie deletion; duplication across the work and home audiences; or in-country versus international traffic. Increasingly though, it looks like the online metrics community understands and has assimilated these issues; often I speak with clients who tell me that they don’t expect our UV estimates to match their internal data, for all the reasons above.

And of course, site-centric data cannot tell you about cross-site duplication, or about demographic composition, or about reach/frequency. But then, I’m preaching to the choir, aren’t I?

The Media Rating Council (MRC) was founded in 1964 as an outgrowth of congressional hearings, with the mandate of auditing and accrediting audience measurement companies in the US. They accredit Nielsen TV ratings, Arbitron radio ratings, MRI’s magazine ratings, and Scarborough’s newspaper ratings, among others. comScore has been working with the MRC and their auditor for over a year, first going through the formal pre-audit process, then scoping and defining the audit phase. Last week, we announced that we have contracted with the MRC to commence the formal audit of the Media Metrix service.

An MRC audit and subsequent accreditation is generally seen as assuring users that the measurement company provides full transparency and disclosure. It does that, but it does more: MRC accreditation assures users that the measurement company maintains a commitment to striving for the highest quality methods and procedures in producing its information. comScore’s entry into the MRC audit process signals an important stage in the evolution of the company—and of the Internet itself as an advertising medium.

Much has been written in recent weeks about client concerns over the quality of market research work conducted using online “access panels.” (That’s the term popular in Europe, describing an online panel from which survey respondents may be drawn.) Concern seems centered on two key issues: (1) the problem of so-called “professional respondents;” and (2) the question of repeatability: will the same survey conducted against the same panel on different occasions lead to the same business decision?

The Advertising Research Foundation, under the stewardship of President/CEO Bob Barocci and research guru Bill Cook, has formed an Online Research Quality Council to address these concerns. Unlike some other associations whose membership is comprised exclusively of research companies, the ARF constituencies include advertisers, ad agencies, academia, and media companies in addition to research suppliers, providing the broadest base yet for such an initiative. I’m on the ORQC Steering Committee, along with colleagues from Ipsos, TNS, NPD, Harris Interactive and Synnovate.

One of the challenges in assuring high levels of research quality in online work is that the field of providers is crowded. Marketers have an expectation that the research supplier should be accountable for quality, and should refuse to take on projects where the specifications and cost pressures would lead to compromises in research quality. The problem is, though, that when the best-practices companies draw these lines in the sand, there will always be someone else happy to do the work and send an invoice. So an unfortunate outcome of a commitment to research quality on the part of the leaders in the space will inevitably be a shift in volume of work from high-quality to lesser-quality providers. That is a problem, and one which an industry-wide commitment to a standard set of quality metrics can help solve.

For the part of comScore, this focus on research quality in the online panel space has only upside. We are able to identify the survey-taking behavior of our panelists in other panels (Gian Fulgoni’s work in this area has been well-received and oft-quoted), and we have been able to demonstrate empirically the behavioral differences of heavy survey responders as compared to the general online population. Indeed, we are able to eliminate these heavy responders from our surveys, or to control for their impact. As for repeatability of results, since we can empirically measure the online behavior of our panelists, much of our research is wholly immune from the vagaries associated with self-reported behavior as collected from respondents through questionnaires. Frankly, I think the industry focus on quality will only increase the demand for comScore’s services.

The first meeting of the ORQC was held at the Time Warner Center on September 10 with almost 200 persons in attendance (and more on the phone). Four committees have been formed: the “Define Quality” committee; the “Quality Matters” committee; the “Identify and Adopt Metrics” committee; and the “Funding and Outreach” committee. If your company is an ARF member, I would urge you to join the council and volunteer for one of more of the committees.

In addition, the ORQC has formed a Client Advisory Committee, which thus far includes Dr. Tom Evans (ESPN); Leroy Leiker (Bank of America); Amy Raihill (Shell); Michele Salazar (McDonald’s); James Mendelsohn (Capital One); Bob Sanders (Allstate); Keith Stevens (Kraft); and John Willard (Bayer).

The IAB and the MRC are working on a document with the working title, IAB Audience Reach Measurement Guidelines. The arduous work of crafting and wordsmithing is going on under the auspices of the IAB Audience Measurement Work Group, and the leadership of MRC Executive Director George Ivie (he’s the guy putting pen to paper).

This document, once ratified, will help to sort out some of the confusion in the online metrics space over panel-centric versus site-centric “Unique” estimates. The document essentially provides three sets of reach guidelines: one for user-centric measurement (such as is provided by comScore); one for site-centric server data; and one for third-party ad-server networks. The Reach Definition Guidelines will clarify that all Uniques are not created equal. Specifically, the guidelines make clear that “Unique Cookies” (the Uniques counted by site-centric web analytics providers) are not the same thing as “Unique Visitors” (the Uniques counted by comScore.)

I don’t want to go into too much detail, because the document is in draft form and subject to change; because it is not yet official; and because quite frankly it is not my place to “scoop” the IAB. But this passage from the most current draft I’ve seen proves instructive:

“Unique Cookies (unduplicated Cookied Browsers) do not generally represent users or people accessing Internet content or advertising due to several complexities surrounding use of browsers.”

This is Alistair Sutcliffe’s first post to comScore Voices so an introduction is in order. Alistair spearheads Advertising Solutions at comScore, providing analytical consulting for comScore clients in the area of advertising measurement and effectiveness. He is a seasoned industry analyst with a background in marketing research, statistical modeling and media consulting. Prior to joining comScore, he worked in the analytic and consulting divisions of VNU and for IRI. He holds a B.S in Mathematics from Lancaster University, England and an M.S in Statistics from The Ohio State University. –G.F.

Much has been written about how new Internet technologies are making page view measurement less relevant when comparing consumer interaction with different Web sites. Many parties suggest replacing the page view with a measure of engagement.

It is often overlooked that different metrics have different utility to different parties when ranking Web sites. Media planners, publishers, networks, and the press all have different perspectives on what type of engagement metric is best for their needs.

The truth is that different situations need different metrics. A metric useful for a media planner in an agency looking at display ad inventory may not be useful for a marketer who has just designed an interactive Web site. Let’s look at some metrics, both traditional and new Web 2.0 advertising metrics, and see where each may be most useful

Unique visitors is a measure of how many unique (i.e. different) individuals visit a site over some period of time. This is an appropriate metric to quantify the size of the unduplicated audience a site reaches. Combining the unique visitor metric with the frequency of visits, you have a measure of both reach and loyalty.

Time spent comes into play when you want to compare the level of engagement between sites. This is especially true when looking at categories like social networking sites. The number of visits alone won’t help you measure the success of these sites in serving their customers. One site may be able to create environments where the level of duration for each visit is 10x the time spent per visit at another site. For other sites like search engines, the brevity of the visit is a virtue, so time spent is not a good measure of engagement; repeat usage might be better.

Page views, which are much less relevant in a Web 2.0 environment for sites using technologies such as AJAX, are still a helpful metric for many sites. Page views will likely be used as a measure of involvement for sites that maintain a traditional page definition.

For media planners and publishers, comScore has introduced a new metric that evaluates sites based on their ability to serve advertising media -- the number of ad views. Ad views are the number of advertising impressions that will be seen by a visitor to that publisher Web site. Each site has a different ability to deliver ads to its visitors. Media planners can use ad views by publisher site to derive actual reach and frequency levels against specific target groups of visitors.

When you know how many ad views are being displayed to different demographic consumer segments, media buying decisions can be improved. Planners can easily compare sites based on their ability to deliver the number of ad impressions they would like to buy. Page views are insufficient because sites display a different number of ads per page, and in a Web 2.0 environment, it is possible that dozens of ads can be served to consumers without a new page being displayed.

Ad views can be combined with other metrics, such as visits, to understand whether a consumer visiting a site will be bombarded by dozens or just a few ads in a visit. A very large number of ad views per visit may mean unacceptable publisher clutter, where a message could be lost among those from dozens of other advertisers.

Having a wide array of different metrics available is a good thing -- one size does not fit all. The focus on just one metric may create a convenient ranking of Web sites, but advertising views and duration metrics provide valuable insight across publishers and can help drive significant improvements in media planning and buying.

Advertising, it turns out, is a lot like physics; they’re both all about time and space. Newspaper, magazine, billboard, and place-based advertisers buy space; TV and radio advertisers buy time. The online business originally aligned around a spatial construct (banner ads defined by area with two dimensions, and existing on a page); but increasingly, online advertisers are migrating toward a temporal construct, especially with the advent of online video.

Historically, Media Math has been pretty simple: “how many,” and “how much?” “How many” is the reach, or the cumulative unduplicated audience (cume), of a media vehicle or ad campaign. Online, our measure of cume is the Unique Visitor. “How much” is tonnage of consumption. In the mid-90s, the online advertising business aligned around the construct of the Page View as the atom for counting Internet consumption (Unique Visitors consume Page Views). In network TV, by contrast, the atom of tonnage is the average minute.

True to the tautology of Media Math, Unique Visitors multiplied by Page Views yields Gross Impressions.

If you followed all that, congratulations! You too can be a media researcher—or just look like one.

Lately, of course, there has been some controversy around the efficacy of the Page View as the atom of Internet consumption—largely triggered by the penetration of AJAX technology, which enables content to refresh on screen without serving a new Page View. In early July, another online metrics company announced that it would stop providing Page View-based rankings, leading to a spate of articles in the business press like this one from Forbes, asking “Is the Page View Dead?”

Meanwhile, everyone even peripherally interested in advertising knows about the industry’s love affair with engagement: two years strong and starting to look serious. With apologies to the ARF, I’m not sure anyone has developed a consensus definition of just what engagement is; we can all agree, though, that whatever it is, we want it. And we’re darned sure we want to measure it.

Engagement is a powerful idea because it represents an attempt to get at the quality of exposure. Not all exposures are engaged exposures, and we want our measurement of ad media to account for the difference. It goes beyond “How many?” and “How much?” to ask, “How good?”

These two developments—the decline in the efficacy of the Page View and the increased demand for engagement—dovetail nicely in Internet metrics, because both argue for a reconsideration of time and space. Specifically, I contend that they argue for a shift in emphasis from Page Views to duration-based audience metrics. Which, in turn, requires some thinking about how consumers, fundamentally, pay attention.

As anyone with a teenager in the house knows, today’s consumers live in an era of multi-tasking. We’re watching Lost on TV with the phone tucked between shoulder and ear while reading email and conducting three chats on IM. Developing a full picture of consumer online behavior now requires that we capture different kinds of media interactions that occur at the same time—and the best way to measure time turns out to be, by measuring time.

Maybe we need to think about online media consumption in two flavors: Time Spent, and Engaged Time Spent. What if we could track the time consumers spend with each web property—whether comprised of pages, audio, video, IM or widget—in a way that allows for capturing multi-tasking behavior? Say I’ve got Facebook and CNN.com open on my screen, along with two IM windows and a ballgame. My engagement to any one of these things ebbs and flows, but I’m spending time with all of them simultaneously. I hop to an IM window and trade messages with a friend, and for two minutes, I’m engaged with the IM client. Then I click onto Facebook to see if my friend likes the same movies as I do, and for three minutes I’m engaged with Facebook. All the while, I’m accruing time with each property (remember: a TV people meter doesn’t stop counting viewing if the panelist answers the phone or picks up a magazine.)

I’m laying all this out because I think this is where Internet audience measurement needs to go—tracking both total Time Spent (maybe we call it “Multi-tasking Time Spent”) and Engaged Time Spent. With these two metrics, advertisers and publishers could start thinking about things like Share of Time Spent; what percent of a website’s Time Spent is Engaged Time Spent, and how can they drive that figure higher? Do the same properties that accrue the highest Total Time Spent also accrue the highest Engaged Time Spent? How do consumers interact with multiple entities when they share screen space?

Some of you might be rolling your eyes right about now; “The last thing we need is MORE metrics!” But I disagree. Consumers use online media in ways that are varied, nuanced and complex; we need to make sure our metrics keep up. And we need to make sure that we keep developing metrics that are driven by consumer experience—that are customer-focused—in addition to the machine-based metrics that are the bailiwick of the Web Analytics side of the equation.

So no, I don’t think the Page View is dead. But I do think that Duration-based metrics will continue to rise in prominence, and that we’ll probably see more of them. Me, I think it’s about time.

What do you think?

Louise Story’s October 22nd article in the New York Times, entitled “How Many Site Hits? Depends Who’s Counting” highlights the challenges of getting up to speed on the continuing debate about the differences between the online unique visitor counts provided by publishers’ web server logs and the independently measured unique visitor counts from the panel-based audience measurement services of comScore and NetRatings. With a wide ranging set of interviews, the article reflects many of the myths that plague our industry today.

Ms. Story begins her article by underlining the contrasting figures between comScore and NetRatings panel-based data versus Style.com’s internal server log data. However, the comparison turns out to be apples to oranges. In this case, the apples are U.S. data while the oranges are worldwide data. The article later acknowledges that Style.com’s figures are worldwide while the comScore and NetRatings data were U.S. In fact, if you were to compare comScore’s worldwide figures for September (which we do publish, contrary to the statement that “Conde Nast counts international readers and ComScore and Neilsen (sic)/Netratings do not”), you would see that we reported 1.28 million visitors to Style.com, certainly much closer to their internally reported 1.8 million, a number which is undoubtedly inflated due to the impact of cookie deletion (which I will discuss further below).

One of the other key factors causing differences between server logs and panel data is people visiting the same sites using both home and work computers. Since site servers are counting computers and not people, servers double count such people while the panel services do not. As the article explains: “But online publishers say that their [comScore’s and NetRatings’] systems drastically undercount people who use the Web during work hours, particularly in offices where corporate software makes the wanderings invisible to the tracking systems.” While it is true that comScore’s work panel cannot be sourced from every company, it is important to point out that comScore applies different statistical weights and projection factors to different population segments. For every company that prohibits the download of software on employees’ machines, there are others that allow it. Employees from the latter companies are used to represent the others and the entire work population. This is similar to opinion polls that use people who agree to answer a survey to represent the attitudes and opinions of people who hang up on the caller. Further, one of the advantages of the comScore panel is that we have a segment of panelists that allow us to monitor both their home and work computers, thereby enabling us to understand overlapping usage between these locations and adjust for it. The ability to accurately filter out overlapping home and work usage is just one example of the advantages of panel data over server logs.

The article also gives scant attention given to the single most prominent source of discrepancy between panel-based data and server-based data: cookie deletion. This issue is mentioned at end of the piece, which is summarized thusly: “To make matters more complicated, consumers who delete cookies — small bits of computer code that track their online wanderings — are also over-counted by publishers’ servers, by most accounts.” comScore published a white paper in June that studied the degree to which cookie deletion inflated server-based data. The results showed that 30% of U.S. Internet users delete their cookies at least once per month and that this group deletes their cookies, on average, 5 times in the month, thereby leading to an overstatement in server-based counts of unique visitors of as much as 2.5x.

Now, this isn’t the first time an article on web metrics has failed to point to this key issue more prominently. An August 2007 article in Fortune, entitled “The Online Numbers Game” drummed up similar controversy, using Digg.com as an example:

“A number of web entrepreneurs believe the two companies shortchange them. Consider Digg.com, a site that lets users submit and rank news stories. Its own server logs recorded 10.8 million unique U.S. visitors in July. ComScore reported 4.6 million, and Nielsen//NetRatings 4.7 million.”

So let’s examine this example for a moment. What we have is both comScore and NetRatings in general agreement over the audience size for Digg, but both differing dramatically from Digg’s own server logs. If you divide Digg’s reported 10.8 million by comScore’s 4.6 million, you get an overstatement factor of 2.3x. If you read our cookie deletion white paper, the obvious conclusion is that this is completely consistent with our findings that the overstatement factor caused by cookie deletion can be as high as 2.5x. Unfortunately, the article never even mentions cookie deletion as a likely source of discrepancy.

There is no shortage of publishers with a vested interest in claiming our data are inaccurate, because as one publisher noted in the Times article, “Everyone likes bigger numbers.” comScore’s only vested interest, on the other hand, is in being as accurate as possible. True media accountability cannot exist without a high degree of accuracy, and accurate media measurement is at the core of comScore’s business.

Ultimately, the disparities often cited between panels and server logs cannot be explained by simple methodological flaws. When numbers diverge by a factor of 2 or 3, there are larger forces at play, with cookie deletion and cookie rejection clearly being the most prominent. If the industry wants to put this issue to rest, there needs to be an acknowledgement on the part of the media world that cookie deletion is one of the root causes of this controversy and that its impact is significant.

Obviously this controversy will live on for awhile. However, we are making progress. The IAB is sponsoring an educational conference on Audience Measurement on November 29, where there will be an attempt to explain what the numbers mean and why they could be different. One of these days, as part of this education, hopefully one of these vocal publishers will agree to share their detailed data publicly for closer scrutiny. It would be interesting to see what happens under the spotlight!