Causal Attribution - Proposing a better industry standard for measuring digital advertising effectiveness
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Causal Attribution - Proposing a better industry standard for measuring digital advertising effectiveness

on

  • 512 views

Research conducted at Collective with Jeremy Stanley, CTO, Justin Evans, Strategy Officer, and Peter Weingard, CMO, discusses the pitfalls of current digital display measurement methods, and proposes ...

Research conducted at Collective with Jeremy Stanley, CTO, Justin Evans, Strategy Officer, and Peter Weingard, CMO, discusses the pitfalls of current digital display measurement methods, and proposes alternative measures.

Statistics

Views

Total Views
512
Views on SlideShare
512
Embed Views
0

Actions

Likes
0
Downloads
9
Comments
0

0 Embeds 0

No embeds

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Causal Attribution - Proposing a better industry standard for measuring digital advertising effectiveness Document Transcript

  • 1. Proposing A Better Industry Standard Measure of Digital Advertising Effectiveness New research provides a foundation for a better system of measuring the effect of digital advertising and provides a wake-up call to marketers using existing, often badly misleading, attribution solutions.
  • 2. 2 causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness executive summary// Billions of advertising dollars are being wasted. That’s because industry standard measures like click through rate (CTR), post click and post impression attribution are not only inherently flawed, but are being widely manipulated by Identical pools of users is that one pool (the test) ROI measurement would lead to increased was potentially exposed to advertising, and the digital advertising spending, higher returns for other pool (the control) was not – ensuring that advertisers and an incentive for publishers to any difference in performance observed on the create quality, engaging content. Finally, con- two pools must have been caused by the decision sumers would benefit from fewer pay-walls, and to advertise to the test group. more relevant advertisements that could truly intermediaries, and often trick marketers into // In multiple live campaign tests, the Causal At- optimizing away from their best prospects. tribution approach provided an unbiased method In this study, Collective presents an alternative method for measuring digital advertising, of linking true ROI to advertising spend, including exposing campaigns that aren’t working. which uses rigorous experiments to measure the // Because the experiment is cookie based, ROI increase in desired outcomes caused by display can be cascaded down to individual Audience advertising, verses a correlation effect measured Segments, providing rich insights into the types of by existing attribution solutions. users who are being influenced by the advertising. The Causal Attribution measurement approach outlined in this research will allow advertisers, for the first time, to accurately evaluate how their online advertising affects the behavior of specific audiences and measure the value generated from their advertising spend. Key findings: // Existing attribution solutions are either too subjective, misleading, or complex to provide a meaningful industry-standard for advertising // Beyond measuring online conversions or their proxies, Causal Attribution can be used to measure offline conversions and brand lift. Further uses of Causal Attribution include: // Directly comparing the performance of different sources of media. // Testing multiple, potentially radically different, creatives to identify which audiences are most influenced by each. measurement. // Quantifying the effect of frequency (how many // Existing attribution solutions often steal credit times each cookie is shown an advertisement) from other advertising sources and misdirect mar- on ROI. keters into making poor media decisions. // Causal Attribution measures the real impact of a campaign by creating an experiment where the only difference between two otherwise We believe that widespread adoption of Causal Attribution could have a tremendous positive impact on the digital advertising industry. Proper help their purchase decisions.
  • 3. causal attribution transparent Positioning A Better Industry Standard Measure of Display Advertising Effectiveness CAUSAL ATTRIBUTION // CLICK THROUGH RATE // LAST CLICK ATTRIBUTION // LAST IMPRESSION ATTRIBUTION // WEIGHTED ATTRIBUTION // black box 3 ALGORITHMIC ATTRIBUTION // misleading accurate
  • 4. 4 causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness The problem with current online attribution models // misleading, easy to game, too subjective or too complex The maturation of the Web as an advertising medium has spawned several attribution techniques. These techniques attempt to leverage the Web’s unique ability to record a recipient’s response to ad exposure in near real time to quantify the effects of a campaign on brand value and, ultimately, sales. Advertising practitioners routinely rely on the most commonly deployed methodologies to make significant marketing investment decisions. Existing attribution models, however, have severe biases, and when optimized to (or gamed by intermediaries to “compete” on performance), these methodologies can erode brand value. The research that follows demonstrates, for the first time, how a Causal Attribution solution can provide marketers with a true accounting of what advertisements and audiences drive ROI. The research will also show how the current slate of attribution methodologies fail to provide marketers with a reliable and transparent method for understanding the true value of the media they buy, and in some cases, produces effects counter to the marketer’s objectives. But first, a brief overview of the most commonly used attribution methods: “ Half the money I spend on advertising is wasted; the trouble is I don’t know which half. {As true in digital in 2012 as it was in print in 1900} –John Wanamaker ”
  • 5. 5 causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness Algorithmic Attribution // (sometimes called ‘Interactive’ or, ‘Media Mix’) DEFINITION In general, attribution algorithms can be divided In fact, it is often some other unmeasured factor Computer algorithm assigns credit into two categories - weighting schemes and that was responsible for the observed desired for outcomes by analyzing data. predictive models - both of which have significant outcomes. theoretical and practical weaknesses. Weighting pros scheme attribution algorithms rely on human Can overcome some limitations judgment to assign weights to different types in simpler metrics. of advertising events that occurred prior to the desired outcome. The choice of which events to weight and how much weight to assign to each For example, suppose that a campaign is running on two very similar sites (A and B), and the computer is trying to determine how much credit to assign to each site. Suppose that site A was bought with retargeting, while site B was bought without retargeting. The computer will see signifi- cons is subjective, even when informed by data, and Often confuses correlation may not be any more accurate than using a single for causation. flawed measure alone. Requires measuring all user In response to these shortcomings, other compa- are advertised to. This will lead the computer to interactions in all channels. nies have developed attribution algorithms that give site A far more credit than it deserves. No industry standard likely to use computers to assign credit rather than human be developed. derived weights. These solutions work by assem- Only programmers really know how they work. right media right creative events that occurred prior to those outcomes. Then, using a multivariate predictive model where the dependent variable is binary (desired outcome what you get right audience bling a data set of all outcomes and all advertising ?? ??? ?? ??? ?? ??? or not) and the independent variables are all advertising events, these algorithms attempt to identify which types of advertising events were more likely to precede the desired outcome than the undesired outcome. bottom line These computer based attribution algorithms “Pay no attention to the man behind (often referred to as machine learning algorithms the curtain.” or predictive models) suffer from significant limitations when used for attribution. First and foremost, they often confuse correlation with causality. This occurs when the computer finds that certain advertisements tend to precede the occurrence of the desired outcome in historical data (the two cantly higher conversions for users reached on site A, driven by the retargeted audience being more likely to convert regardless of whether or not they Compound this simple example with the plethora of types of audience targeting, frequency capping, distribution channels (display, video, social, mobile, television, etc.), and other compounding effects (convenience, social and repeat purchases) and it becomes clear that these computer based attribution algorithms are just more complex, with no real hope of discovering the ‘truth’. While some attribution models may improve over traditional measures, like last-click or last-view based attribution, their “black box” nature makes gauging success across vendors and campaigns challenging. Only their creators truly know how the algorithm works, what it measures and how it evaluates ROI. Hence, it is unlikely that a single form of Algorithmic Attribution could ever become an industry standard. are correlated). The computer concludes, incorrectly, that the advertisements caused the desired outcomes, and assigns them credit. CONFUSES CORRELATION FOR CAUSATION.
  • 6. 6 causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness Click Through Rate // DEFINITION The de facto standard of web metrics, CTR campaigns showed no correlation between CTR Clicks per impression. reports the cumulative number of clicks recorded and brand lift nor purchase intent as measured on a display ad campaign, divided by the by independent post-impression surveys. Hence, number of impressions served. The oldest and optimization of campaigns to achieve higher CTR pros easiest measure to capture remains one of the may in fact be reducing brand ROI. Always on. most commonly used techniques to assess No conversions required. campaign performance (Chief Marketer 2011 Interactive Marketing Survey, 2011; Digital Proxy for engagement, as the ad Display Advertising 2010, Collective/ Advertiser must be visible to be clicked. Perceptions, 2010), despite ample evidence of cons Very few people click. Clickers rarely buy. Most clicks are accidental or fraudulent. its irrelevance as a meaningful measure (Natural Born Clickers, comScore with Starcom USA and Tacoda, 2009/2010; CTR: Brand Marketing’s Most Misleading Measure, Collective 2010). CTR is also an easy metric for third parties to manipulate by running ads in high-click environments, such as gaming and mobile sites, where users will be more likely to accidentally click on the advertisement. These accidental clicks often frustrate the user by taking them away from the content they were viewing and placing them on a landing page for the brand. In these cases, the frustrating experience may create negative associations with the brand, causing users to be The case against CTR is overwhelming. The less likely to convert than if no advertising had Easily gamed by placing ads near comScore/Starcom study showed that only 16% been done at all. high click content (e.g., games). of all Internet users in 2010 clicked on a display ad in a month, down from 32% a year earlier. what you get Collective’s own examination of one billion advertising impressions served in the first months right audience of 2011 revealed that 99% of stable user cookies right media examined never clicked on an ad, and that those right creative who did were more than two times as likely to click again in the future. The study also estimated bottom line Digital advertising’s oldest metric is also its most misleading. that as many as 20% of the clicks were accidental, while Click Forensics estimated in October of 2010 as many as 23% of clicks were fraudulent. Further, a study conducted by Collective of 100 CLICKERS DON’T BUY.
  • 7. 7 causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness last click Attribution // DEFINITION Last click is a derivation of the Click through rate Last ad clicked prior to outcome. born of “lower funnel” search advertising, which The study went on to draw the following metaphor: assigns credit only to the last click that occurred within a designated period of time prior to the To illustrate the faulty logic of the model, pros desired outcome (the look-back window). This imagine that you’re standing in the grocery store Measures only productive clicks. method disregards all clicks where no conversion knowing precisely what you want to buy. You’ve event followed, thus removing all accidental clicks. seen the product ads on TV, the full-page ad in However, by assigning 100% of the credit to the a magazine, and a full color mailer that actually last click this method fails to account for all the made you hungry just looking at the pictures. previous advertising messages the user consumed You’ve even clipped a coupon and brought it prior to making the last click, and thus tends to with you to the supermarket. When you ask the undervalue display and video advertising. Further, grocery clerk where to find the specific item, he Ads that build awareness or intent many users click on search advertisements as a smiles, points and says, “Aisle five.” Off you go are given no credit. means of navigating to the product home page, to aisle five, find the item, pay, and leave the and would have clicked on an organic link had the store. If you applied the “last ad” model to this search advertising not appeared. In this fashion, scenario, the grocery clerk would get 100% of last click attribution often significantly over-values the credit for your purchase (no wonder he’s search advertising. smiling). As a result, marketers would invest Difficult for intermediaries to game. Can be linked to ROI. cons Some users would have converted anyway. heavily in grocery clerks, and they’d pull their The 2008 Atlas study referenced earlier cast what you get advertising dollars from the marketing channels further doubt on Last click methodologies, point- that actually piqued your interest or moved you ing out that, “between 93% and 95% of audience through the funnel toward the purchase. right audience engagements with online advertising receive no right media credit at all when advertisers review the ROI on right creative their campaigns,” because of misplaced emphasis on Last click attribution. Quotes the study, “The bottom line Useful for ‘infomercial’ products. “last ad” model forces marketers to place greater importance on the aspects of their advertising that support the model, rather than the aspects that support their advertising success.” DISPLAY IS OFTEN UNDERVALUED.
  • 8. 8 causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness last impression // DEFINITION Sometimes called the view-through conversion Last impression attribution is also very easy to Last ad viewed prior to outcome. rate (VTR), last impression attribution assigns manipulate (even more so than click through rate). 100% of the credit to the last advertising Intermediaries are incentivized to target users impression served (or viewed) within a given who are likely to convert, regardless of whether pros period of time prior to the desired outcome. In they will be influenced by the advertisement, with Gives credit to views not clicked on. some cases, conversions where the user clicked the cheapest media buy possible. In the worst are not attributed to viewed impressions, in case scenario, when an advertiser works with other cases they are. In contrast to the last click only one intermediary, that provider can simply attribution method, the look-back window is often hit every stable cookie with a single poor quality much longer (7, 14, 30 or even 60 or 90 days). No advertisement once per N days (where N is the clear standard exists within the industry. Google, look-back window) and they will receive credit for for example, defines VTR in its Help Center as “a close to 100% of conversions. cons Ads do not have to influence consumer to receive 100% credit. Quality of media, placement and measure of the number of online conversions that creative have little impact. happened within 30 days after a user saw, but did Easily gamed through ‘spray & pray’ not click, a display ad.” media buying. Last impression attribution was introduced to Overvalues retargeting (would have overcome shortcomings in last click attribution by bought anyway). ensuring that viewed impressions also received credit for conversions. In practice, however, it what you get gives far too much credit to digital advertising. When the standard is an impression delivered, right audience the user may not even have been able to see right media the advertisement that receives credit. When right creative the standard is a ‘viewed’ impression then while the advertisement may be visible, there is no bottom line Encourages advertising to consumers who would have converted anyway. guarantee that the consumer was influenced by the advertisement to complete the desired outcome. This leads to retargeting receiving far more credit under last impression attribution schemes than it should, due to the fact that the users would have completed the desired outcome regardless of whether or not they saw an advertisement, yet the advertisement gets 100% of the credit anyway. 1 Determining if an advertisement is visible remains a challenge for many vendors. The vendor must run code in the browser to determine if, and for how long, an advertisement is viewable to the user. Such code is often confounded by iFrames, a publisher technology that quarantines advertisements. See www.adexchanger.com/data-driven-thinking/viewable-impression for more details.
  • 9. 9 causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness Causal Attribution // DEFINITION Research from Collective’s Data Sciences team Experiment measures outcomes explores the viability of a Causal Attribution caused by advertising. measurement scheme that, through a carefully In the sections that follow, we will: // Illustrate how Causal Attribution works through constructed experiment, directly observes a simple six step process. pros the change in desired outcomes caused by // Outline the types of outcomes that can be Directly quantifies ROI. incremental digital advertising spend. Because measured using Causal Attribution. this method uses an A/B test experimental // Examine in detail the results of two real Unbiased by all other advertising. design established prior to the incremental Transparent and easy to administer. advertising, the results are unbiased by organic Provides rich audience analytics. conversions and by other advertising spend, including spend offline. cons Further, because the design is based on random Requires large audience database. audience groups, rather than randomly selected Does not always produce results that are statistically significant. impressions, Causal Attribution can measure the cumulative effect of multiple advertising impressions over a period of time on individual users. This is accomplished without having to waste any impressions on public service announcements (PSAs), a common requirement campaigns where Causal Attribution was used to quantify ROI and identify optimal audiences. // Introduce an alternate methodology which uses PSA advertisements. // Outline additional applications that can optimize creative, media and frequency. We will then conclude with a more thorough discussion of the benefits (and limitations) of Causal Attribution for advertisers, and a discussion of how widespread adoption of this best practice could impact the industry as a whole. of other experiment based digital measurement what you get approaches. When combined with a measure right audience of desired outcome value (e.g., profit from a right media conversion), Causal Attribution can measure right creative the true return on investment (ROI) driven by incremental advertising spend. bottom line Measures causality, not correlation, providing a true measure of ROI. SIMPLE, SCIENTIFIC, UNBIASED, TRUE ROI.
  • 10. 10 causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness how causal attribution works // The great French physicist and mathematician, Henri Poincaré, once said that “experiment is the sole source of truth; it alone can teach us something new; it alone can give us certainty.” Without experimenting, no amount of conjecture or analysis of data can prove a scientific theory. Similarly, without experimenting, we cannot prove that advertising has caused a desired outcome, nor can we truly optimize our ROI. Causal Attribution works by creating an experiment where the only difference between two otherwise identical pools of users is that one pool (the test) was potentially exposed to advertising, and the other pool (the control) was not. This design ensures that it was our decision to advertise to the test group that caused any statistically significant difference measured in the desired outcome rates of the test and control pools. In practice, there are some subtle and important decisions to be made in conducting a Causal Attribution experiment. The following six steps outline the process and illustrate how Collective conducts these experiments on behalf of advertisers. Step 1 // Step 2 // Step 3 // Define the Audience Cloud Divide the users into test and control groups Deliver advertisements only to the test group We first limit the experiment to cookies that Before the campaign begins, we randomly divide We then begin the advertising campaign, and are likely to persist for the duration of the the entire Audience Cloud into test and control ensure that cookies in the control group are experiment. This is done to limit the impact of groups using a robust random number generator. never exposed to the advertising. In practice, cookie deletion. At Collective, we define a stable This split is repeated with a different initial seed this is done by negatively targeting an audience cookie to be one seen at least once within the for every advertising campaign, ensuring that segment in our ad-serving engine that includes all last 28 days and on at least two separate days the randomization is unique for each advertiser of the cookies in the control group. Whenever an over the life of the cookie. This ensures that the and there is no cross-experiment pollution. In impression arrives for a user in the control group, cookie was recently seen, and was persistent for practice, we typically choose anywhere from 5% the campaign is excluded from the potential set of at least a period of two days (and so unlikely to to 50% of the cookies to be in the control group, campaigns to be shown. have been automatically deleted by the browser depending upon the length of the campaign and after the session). the frequency of the desired outcome absent Note that for the purposes of measuring the advertising. impact of the campaign we do not need to separate days of the prior 28 are 22 times more Given an Audience Cloud of 200m users, this targeting, media selection, creative optimization likely to appear the following day than cookies means our control group will consist of anywhere or frequency capping was used in the campaign. seen on only one day. On average at Collective, from 10m to 100m users. These sample sizes This is possible because we are comparing the this provides a stable universe of approximately ensure that the test and control groups will be entire control group with the entire test group, 200 million users, which we refer to as our balanced across all other influencing factors, such regardless of who the ad was delivered to. Audience Cloud. as audience demographics and exposure to other Being able to ignore these other factors drastically advertising (either digital or traditional). simplifies the process of analyzing the experiment, We have found that cookies seen on at least two adjust the experiment to control for any audience and also ensures that we can run the experiment on any campaign without disruption, allowing for ongoing monitoring and optimization. 2 Negatively targeting the control group also allows the campaign to deliver outside of the universe of stable cookies, which is often desirable for meeting delivery goals and maximizing reach. One complication of this approach is that when computing ROI we must remove spend on impressions outside of the test and control groups (see step 6).
  • 11. (audience cloud 200 million cookies) TEST TEST ST EMAINING 90%) (REMAINING 90%) 90%) (REMAINING # CONTROL OUTCOM # CONTROL OUTCOMES CONTROL # CONTROL OUTCOMES CONTROL CONTROL = = = OUTCOME # CONTROL COOKIESCOOKIE OUTCOME RATE OUTCOME RATE RATE CONTROL COOKIES # CONTROL # CONTROL NTROL CONTROL ANDOM 10%) (RANDOM 10%) 10%) (RANDOM split test & control Step 4 // # TEST OUTCOMES # TEST OUTCOM # TEST OUTCOMES TEST TEST TEST = = = OUTCOME RATE OUTCOME # TEST COOKIESCOOKIE OUTCOME RATE RATE TEST COOKIES # TEST # ADVERTISE TOTO TEST TEST ADVERTISE TEST TO ADVERTISE OBSERVE OUTCOMES OBSERVE OUTCOMES OBSERVE OUTCOMES campaign runs Step 5 // measure outcomes Step 6 // Observe desired outcomes on all users Measure the desired outcomes rates Compute the causal lift During the course of the campaign, and for a After the campaign has concluded and the We define the causal lift of the campaign as the period of time after the campaign concludes we follow-up period has elapsed we can compute lift observed in the desired outcome rates in the observe all desired outcomes on all cookies in the rate of desired outcomes, defined as the test group over the control group. For example, the Audience Cloud. total number of outcomes divided by the total suppose we find that 0.12% of users in the test number of cookies, in both the test and the group purchased the product, but only 0.10% It is critically important that we observe all control groups. Note that, depending upon the of users in the control group purchased the desired outcomes, as the outcome rate in type of outcome being measured, you can either product. Given that these two user pools were the control group serves as our baseline for allow cookies to complete the desired outcome randomly chosen from the same population, and evaluating lift caused by the advertisement. more than once or only once. For example, if the only difference was our decision to advertise For example, applying a last view or last click the outcome is a purchase, then you should to the control group, we can conclude that the attribution scheme would discard those desired count all purchases from each cookie. If instead, advertising caused users to be 20% more likely to outcomes that did not have a view or click event the outcome is a survey response or a site convert over the time period in question. within a selected look-back window. Thus, registration, you may wish to count the outcome the control group conversion rate would be only once per cookie. artificially reduced to 0% and the experiment would be invalidated. Similarly, all desired outcomes should be counted in the experiment, regardless of the path that the user took to complete the desired outcome. For example, if we are measuring online product purchases, then we should track all purchases online regardless of how the user clicked to arrive at the checkout page – be it directly through the URL, through an organic or sponsored search, or by clicking on a display or video advertisement. This ensures that we capture the full impact of the advertising impressions by including all purchases that the advertising could have influenced.
  • 12. 12 causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness desired outcomes that causal attribution can measure // Causal Attribution can measure a cross section of advertising objectives: Online conversions or actions (proxies): Off-line conversions: Brand measurement: Though a less immediate measure, off-line sales can An important and often overlooked metric in Consumer online purchasing or other desired online be applied to Causal Attribution logic through the digital advertising today is evaluating the impact activities, such as visiting a website, completing a import of anonymized point of sale and/or credit of advertising spend online on brand awareness, registration, viewing a video, etc., can be measured card transaction data, available through third party message recall and future purchase intent. using outcome pixels, such as DART Spotlight vendors. Similarly, an advertiser’s internal CRM Custom surveys executed through rich media ad activities or AMP audience pixels. In general, these purchase log data can be anonymized and moved units can record the responses directly to a user pixels should be closely linked to potential revenue online to a cookie through a data management cookie so that the (self-reported) results of a brand generation, and should be widely accessible to platform. This approach allows the advertiser to measurement survey can be observed across consumers, regardless of their path to conversion. quantify the impact their online advertising is having the test and control groups . At the conclusion This ensures the full impact of the advertising spend on offline sales without having to use any personally of the experiment, any lift observed in desired is captured, and also helps to ensure statistical identifiable information (thus protecting consumer survey response rates in the test group over the significance of results. privacy) in the design or analysis of the experiment. control group can be credited to the incremental advertising spend. 3 Note that for brand measurement, it can be significantly more cost effective to conduct the Causal Attribution study using a creative based approach with public service announcements (PSAs). This limits the audience where surveys should be collected to just those users who received an advertisement in the test group or a PSA in the control group. See the PSA Methodology section for more information.
  • 13. exAMPLES of Causal Attribution Tests on Live Campaigns // The Collective Data Sciences team analyzed the behaviors of 200 million stable audience cookies across 14 live advertising campaigns during a period from October 2011 through January 2012. In all, 380 million advertising impressions were examined. The audience profiles examined contained an average of 70 data attributes, such as location, demographics and enthusiast behaviors. Two Selected examples follow.
  • 14. 14 causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness case study Travel brand // In this test, advertisements for a national hotel The campaign cost $25,000, and so the ROI for In the chart to the right, we show how each enthu- chain were analyzed for their ability to drive online the advertiser was 32%. Meaning that, for every siast group compares to the average by plotting bookings. The test analyzed a test group of 180 $100 the advertiser invested in this campaign, they the percentage lift (on the horizontal axis) for a million cookies and a control group of 20 million made $32 of incremental profit. particular metric for each group. cookies. At the end of two months, the exposed group converted at a 0.0050% rate, and the control group converted at a 0.0044% rate. This means that the advertising spend caused users to be 14% more likely to convert than they would have otherwise. Using a difference in proportions test , we find that the p-value for this lift is 0.00005, indicating that it is highly statistically significant, with less than a 1 in 20,000 chance of observing a lift this high randomly. What is more, we can convert this lift directly into a return on investment (ROI). The test group converted at 0.006 percentage points higher than the control group. Multiplying this rate by the number of cookies in the test group we can conclude that there were 1,099 additional conversions caused by this campaign. The client valued online The experiment becomes even more interesting The metrics shown are the control conversion rate as the nature of the converting customers is exam- in gray (how often the control group converts ab- ined and we compare Causal Attribution results sent advertising), the click through rate in pink and for audience segments to performance measured the causal lift in blue (how much more likely users using other attribution methods. are to convert when exposed to advertisements). For example, we can segment the Audience Cloud by the context of the pages each user most often frequents. We call these segments enthusiast behaviors, as they indicate that a cookie is enthu- The enthusiast behaviors where we observed at least 500,000 users in the control groups have been sorted on the vertical axis by the control conversion rate lift in descending order. siastic about a category of content (e.g., music, We find that the users most likely to book hotel sports, food). Because the test and control groups rooms in the control group are those reading were randomly selected from the entire Audience about real estate and, not surprisingly, travel. Cloud, each of these enthusiast groups will be ran- These groups are also among the highest in click domly split between the test and control popula- through rate. Yet their causal lift is actually nega- tions, and we can analyze the causal lift generated tive, indicating that advertising to these groups within each segment. will not make them any more likely to convert than they already were. conversions at $30 each, therefore, this additional advertising spend generated $32,970 worth of incremental value for the hotel chain. criteria Control Test 20 180 882 9,037 0.0044% 0.0050% # cookies # conversions conversion rate TEST RESULTS: TRAVEL BRAND // While a 32% ROI is a powerful result (every $100 spent drove $32 of incremental profit), audience targeting could have HAVE GENERATED AN ROI AS HIGH AS 150%. conversion lift statistical significance conversions caused value of a conversion 14% 0.00005 1,099 CONVERSIONS / COOKIES (TEST RATE - CONTROL RATE ) / CONTROL RATE P - VALUE FOR CONVERSION LIFE EXCEEDING 0 % (TEST RATE - CONTROL RATE) X TEST COOKIES $30 value generated $32,970 spend $25,000 roi 32% CONVERSIONS CAUSED X VALUE OF CONVERSION (VALUE GENERATED - SPEND) / SPEND
  • 15. 15 causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness clickers control converters causal lift MEASURE REAL real estate CLIC TRAVEL travel CON sports SPORTS CAS weather WEATHER food FOOD shopping SHOPPING FAMILY family POLITICAL political AUTOMOautomotive SCIENTIFIC scientific health HEALTH technology TECHNOL- MUSIC music finance FINANCE FASHION fashion games GAMES -100 -100 -50 -50 0 50 MEASURE LIFT (%) 0 50 100 100 measure lift (%) While real estate and travel enthusiasts book many hotel rooms, they were not influenced by the advertising. People reading about fashion and shopping were. 150 150
  • 16. 16 causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness case study : travel brand (continued) In contrast, near the bottom of the chart we find What then is the optimal audience profile for this By selectively targeting those segments exhibiting that in the control group those users reading hotel chain? To answer that question, we can higher ROIs, an advertiser can drive materially about fashion are less likely to book hotel rooms analyze the return on investment for every higher performance. Further, sophisticated look than those reading about travel, and are less likely audience segment. alike modeling techniques can be employed to to click on the hotel chains ads. However, they score every user in the audience cloud from lowest While the brand achieved a 37% ROI on the are more than 50% more likely to convert when expected ROI to highest expected ROI, using campaign as a whole, there are large audience they were exposed to the campaign. Thus, we a multitude of factors including demographics, groups where the ROI is significantly higher. Users can conclude that the advertising is significantly geography and behavior online. Then, a custom who are older and users with higher incomes influencing these users despite their low response audience group can be created of just those users generate ROIs ranging from 50% to 125%. rates using traditional attribution methods. who we expect to deliver the highest ROI for the Similarly, there are large pockets of users by advertiser. geography, browser and enthusiast behavior who generate much higher ROIs. 150 -100 -150 AUDIENCE SEGMENT JOBS FASHION GADGETS PETS SHOPPING TECHNOLOGY HEALTH HOUSEHOLD FINANCE SCIENTIFIC MUSIC POLITICAL GAMES FOOD WEATHER SPORTS AUTOMOTIVE TRAVEL RELIGION REAL ESTATE FAMILY FIREFOX FITNESS CHROME IE WN CENTRAL NEW ENGLAND PACIFIC MOUNTAIN S ATLANTIC MIDDLE ATLANTIC WS CENTRAL MALE FEMALE 250+ 150-250 75-100 100-150 25-50 50-75 64+ 00-25 35-44 55-64 EN CENTRAL -50 45-54 18-24 0 25-34 50 ES CENTRAL 100
  • 17. 17 causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness case study : travel brand (continued) To illustrate the impact on performance that op- This is illustrated in the chart to the right, in which This finding also holds true for click through rate. timizing to ROI can have, suppose the advertiser each audience segment is a circle arrayed on a Were the advertiser to select the top 20% of audi- selected the top 20% of audience segments by plot where the x-axis is that audience’s conversion ence segments by their CTR alone, they would their conversion rate alone. The advertiser would rate in the control group, and the y-axis is that achieve a CTR of 0.048% and a causal ROI of 45%. achieve a conversion rate of 0.093% and a causal audience’s ROI. Last view conversion attribution If instead they were to select the top 20% of audi- ROI of 33%. If instead they were to select the top analysis includes the 6 pink audience groups in ence segments by ROI, their CTR would drop to 20% of audience segments by ROI, their conver- the top 20%, whereas an ROI analysis correctly 0.026% (a 46% decrease), yet their ROI would rise sion rate would drop to 0.059% (a 35% decrease), identifies the 6 blue audience groups instead. (The to 104% (a 129% increase). significantly hurting their last view impression 4 split color circles in the upper right were selected conversion rate. However, their ROI would rise to by both strategies.) 104% (a 216% increase), generating more than 3 times as much value per dollar invested. key optimize to conversion rate optimize to roi optimize to BOTH RETURN ON INVESTMENT (%) 100 50 0 -50 0.002 0.004 0.006 0.008 0.010 0.012 CONVERSION RATE (%) optimizing on causal attribution drives 216% higher rois than last view.
  • 18. 18 causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness case study retail brand // In this test, advertisements for a fashion retail This experiment exposed for the client that This audience analysis highlights how misleading brand were analyzed for their ability to drive their current advertising campaign was not existing attribution solutions can be. By targeting registrations for an online website. The test driving enough registrations to justify their high-income young audiences, a partner could analyzed 180 million exposed cookies and a media spend. It is critical to observe that in this drive a very high last impression conversion rate. control group of 20 million. The exposed group case the standard attribution models painted By targeting middle-income older audiences, a converted at a 0.0185% rate, and the control a very different picture. Whereas the standard partner could drive a very high click through rate. group at a 0.0184% rate, meaning that the performance measures being used by the client But neither strategy would materially impact causal test group that included users exposed to the led them to believe that this campaign was one lift – that is, convince consumers to register online advertising was just slightly more likely to register of their best performing, our Causal Attribution who would not have registered anyway. for the website as those who had not been study revealed that it was actually having almost exposed to the ads. The p-value for this lift is 0.37, no impact on online registrations. This ability to indicating that it is not statistically significant, with definitively identify campaigns and optimization more than a 1 in 3 chance of observing lift at this strategies that are not working is one of the level by chance alone. greatest advantages of using Causal Attribution. The campaign caused 188 conversions, valued at In general, we find that audience groups that are naturally predisposed to be likely to convert or click are rarely the groups that will be most influenced to convert by online advertising. Only through the analysis of Causal Attribution results can an advertiser truly discover who the optimal $10 each, for a total of $1,880 of value generated For example, the control converters, those who for the advertiser. The campaign cost $15,000, converted without being advertised to, were demonstrating a negative 87% return young and generally mid-to-high income, whereas on investment. the ‘clickers’ were older and lower income. In contrast, the blue causal lift lines better represent the actual responsiveness of a group to online ads. In this case, causal lift was fairly consistent audience is for a given advertising campaign. The results of this analysis led the advertiser to conclude that they should experiment with new creative strategies, and use Causal ROI to test which audience groups are most receptive to each creative. regardless of age and household income. criteria Control Test 20 180 3,689 33,389 0.0044% 0.0185% # cookies TEST RESULTS: retail BRAND // Traditional metrics (last click, last view) positioned this campaign as one of the top 2 on # conversions conversion rate conversion lift statistical significance 1% 0.37213 conversions caused 188 Attribution proved that it wasn’t value of a conversion delivering value. The advertiser (TEST RATE - CONTROL RATE ) / CONTROL RATE P - VALUE FOR CONVERSION LIFE EXCEEDING 0 % (TEST RATE - CONTROL RATE) X TEST COOKIES $10 value generated CONVERSIONS / COOKIES this advertiser’s buy, yet Causal is using the rich reporting from the Causal Attribution analysis to revisit their creative and audience targeting strategies. spend roi $1880 $15,000 -87% CONVERSIONS CAUSED X VALUE OF CONVERSION (VALUE GENERATED - SPEND) / SPEND
  • 19. 19 causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness measurement clickers control converters causal lift AGE HHI Mea 150 100 50 0 -50 -100 18-24 25-34 35-44 45-54 55-64 65+ inverses : converters were young but clickers were older. 0-25 25-50 50-75 75-100 100-150 150-250 250+ converters were wealthier; clickers, less so.
  • 20. 20 causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness psa Methodology// An alternative to the methodology outlined in One common misconception is that in conducting this paper is to execute a Causal Attribution a creative-based Causal Attribution experiment experiment through the use of public service one can analyze the impact that frequency has on announcements (PSAs) in the creative serving the desired outcome rate. This can be attempted engine. Steps 1 & 2 are the same, but in step 3 by evaluating the lift in test over control instead of negatively targeting the control group, conversions by cookie stratified by the frequency we serve a public service announcement (PSA) with which the cookie was reached (by either to the control group, and serve the campaign the PSA or real advertisement). Thus, one might advertisement to the test group. Note that the observe that the lift in conversions for users with test and control groups must still be cookie based, a frequency of 5 was 20% higher than the lift in rather than impression based, or else cookies conversions for users with a frequency of 1. will move back and forth between the test and control groups polluting any cookie based This type of analysis, however, is seriously flawed. conversion analyses. The primary driving factor behind this measured The main advantage of serving a PSA is that it advertisement, but is instead the user’s frequency narrows the scope of the experiment from all of use of the Internet, and the percentage of 200m cookies in the Audience Cloud to just those those impressions purchased. Thus, any difference served an advertisement (either PSA or from the in conversion rates by frequency could be caused campaign) in the experiment. In some cases, this by these underlying biases rather than by having can improve the statistical significance of the been exposed to the advertisement results. It can also dramatically reduce the number additional times // of desired outcomes that must be gathered when measuring brand lift impact. However, the requirement to serve a PSA is cost prohibitive, especially when conducting Causal Attribution on a routine basis with a large control group. An additional advantage to using a PSA is that control over the audience targeting and ad decisioning is no longer needed. Instead, the experiment is conducted purely in the creative serving, ensuring that the control group only receives PSA advertisements. This allows Causal Attribution to be extended across media buys spanning multiple partners and channels. frequency is not the decision to serve the
  • 21. 21 causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness further Applications // In addition to analyzing ROI at the campaign and audience profile level, there are three significant additional applications of Causal Attribution. While we will only briefly cover them in this white paper, they will be the basis of additional research in the future. Note that in the following sub-sections we describe each experiment in isolation, but they can in fact be conducted simultaneously using multivariate testing strategies. Creative Analysis Media Analysis Frequency Analysis The choice and execution of creative, be it display The choice of where an advertisement is run is Causal Attribution can also be used to quantify or video, is undoubtedly one of the most influen- also a critical decision in any digital campaign. the effect of frequency (how many times each tial decisions an advertiser can make in executing Causal Attribution experiments can be designed cookie is shown an advertisement) on ROI. This a digital advertising campaign. Rather than relying to test the impact that advertising in a premium is accomplished by dividing the Audience Cloud upon intuition alone in designing creative, Causal environment has over advertising elsewhere on the into several smaller pools (e.g., 2 million cookies Attribution allows an advertiser to test multiple, Internet, in either longer tail inventory or with user each). One is kept as a control group, and the oth- potentially radically different, creatives to identify created content. Again, this is accomplished by ers are targeted in the same manner with varying which audiences are most influenced by each. dividing the test group into multiple pools, each of frequency caps. For example, we might create four Thus, an advertiser can increase the effective reach which is targeted only in certain ad environments. small pools: of their campaign and truly serve the right audi- Similarly, one can compare inventory purchased ence with the right creative. This is accomplished directly from publishers with inventory purchased by dividing the test audience into random groups, indirectly through exchanges. // A control pool that will never be shown an advertisement. each of which is only ever exposed to one type of // A pool with a 1 per 1 day frequency cap. creative. Combined with dynamic creative optimi- // A pool with no frequency cap. zation, a given creative can be further optimized to achieve maximal ROI. Keeping each pool relatively small increases our chances of hitting cookies in the pools at close to the desired frequency cap rate.
  • 22. 22 causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness the benefits and limitations for causal attribution for advertisers // THE BENEFITS THE LIMITATIONS The present study establishes that Causal Attribu- Simply put, no other attribution system can pro- tion provides a compelling system for evaluating vide all of these advantages. However, there are digital advertising ROI that avoids many of the some limitations of Causal Attribution that should At Collective, we firmly believe that the benefits to pitfalls present in other attribution solutions. The be noted: advertisers of Causal Attribution far outweigh the following are some of the principle benefits of Causal Attribution to advertisers: // Directly quantifies return on investment (value generated from spend) // Depends on browser cookies, which are often deleted (biases ROI low) // Limits the reach of a given campaign by the size THE CONCLUSION: A PROPOSAL FOR A NEW INDUSTRY MEASUREMENT STANDARD limitations. Combined with our findings on how current methodologies can produce false success signals that misguide marketers into making poor media planning and optimization decisions, we of the control group believe that Causal Attribution should be adopted // Unbiased by all other advertising (occurs in // Does not always produce results that are statis- as an industry standard. both test and control groups) tically significant // Transparent and easy to administer (no proprietary algorithms) // Provides rich audience analytics (identify consumers who will be influenced) // Effective across all channels where cookies are used (display, video, mobile web) 4 Statistical significance is governed by three factors: i) the size of the control group, ii) the amount of lift generated by the campaign, iii) the proportion of users completing the desired outcome absent advertising. In practice, we have found that the third factor is most often to blame for inconclusive results, and caution against running causal attribution studies based on very rare desired outcome events.
  • 23. 23 causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness How Causal Attribution Could Change the Industry // “ Causal attribution measurement will allow advertisers, for the first time, to accurately evaluate how their online advertising affects the behavior of specific audiences and measure the ROI generated from their advertising spend. ” – Jeremy Stanley, SVP Product and Data Sciences, Collective The digital advertising industry has become 20% of that spend was wasted due to misleading Ultimately, more advertising dollars would flow enamored with data, technology and algorithms. measurement systems. Then this implies that $1.8 into digital channels, Advertisers would get higher Billions of data points are analyzed, inventory is billion of spend was wasted in 2011 in the returns on their investment and publishers would dynamically selected through real time bidding, US alone. be rewarded for generating higher quality engag- and sophisticated audience, media and creative optimization strategies are deployed. And yet, all of this investment is frequently evaluated using misleading attribution methodologies. When coupled with highly competitive and sophisticated media partners using questionable optimization tactics, these metrics are severely hindering Widespread adoption of Causal Attribution would have an enormous impact on the entire advertising ecosystem. Digital advertisers would rapidly cut the fraction of their spend that is not effective, and be willing to pay significantly more for the right audiences and ad environments that deliver the industry. high ROIs. CMOs would be willing to spend more To quantify the economic impact of the problem can concretely measure the value the advertis- today, consider that Forrester estimates in their ing is generating. Publishers would respond by “US Interactive Marketing Forecast, 2011 to 2016” limiting the supply of lower quality and less visible report that the market for display advertising was advertising inventory, and instead focus on provid- close to $12 billion in the US in 2011. Assume ing advertisers with impressions that have a higher that 75% of this spend was for direct response probability of influencing consumers. campaigns. Further, conservatively assume that of their advertising budgets in digital where they ing content. Finally, consumers would win by enjoying higher quality content with fewer pay-walls, and by receiving more relevant advertisements that could truly help their purchase decisions.
  • 24. 24 causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness measurement strategies // CLICK THROUGH RATE // LAST CLICK ATTRIBUTION // LAST IMPRESSION ATTRIBUTION // DEFINITION DEFINITION DEFINITION Clicks per impression. Last ad clicked prior to outcome. Last ad viewed prior to outcome. pros pros pros Always on. Measures only productive clicks. Gives credit to views not clicked on. No conversions required. Proxy for engagement, as the ad must be visible to be clicked. Difficult for intermediaries to game. Can be linked to ROI. cons cons cons Very few people click. Ads that build awareness or intent Ads do not have to influence consumer are given no credit. to receive 100% credit. Some users would have converted anyway. Quality of media, placement and creative Clickers rarely buy. Most clicks are accidental or fraudulent. have little impact. Easily gamed by placing ads near high Easily gamed through ‘spray & pray’ click content (e.g., games). media buying. Overvalues retargeting (would have bought anyway). what you get what you get what you get right audience right audience right audience right media right media right media right creative right creative right creative bottom line bottom line bottom line Digital advertising’s oldest metric is Useful for ‘infomercial’ products. Encourages advertising to consumers also its most misleading. who would have converted anyway.
  • 25. 25 causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness WEIGHTED ATTRIBUTION // ALGORITHMIC ATTRIBUTION // CAUSAL ATTRIBUTION // DEFINITION DEFINITION DEFINITION User determines a weighting scheme Computer algorithm assigns credit Experiment measures outcomes for mixing CTR, last click and last view. for outcomes by analyzing data. caused by advertising. pros pros pros Averages out some of the misleading Can overcome some limitations Directly quantifies ROI. facets of simpler metrics. in simpler metrics. Unbiased by all other advertising. Gives the advertiser or agency “knobs Transparent and easy to administer. and dials“to control. Provides rich audience analytics. cons cons cons Poor metrics cannot be mixed together Often confuses correlation for causation. Requires large audience database. Requires measuring all user interactions Does not always produce results in all channels. that are statistically significant. into a good metric. Choice of weights, even when informed by data, is highly subjective. Still provides no direct link to ROI. No industry standard likely to be developed. Only programmers really know how they work. what you get what you get what you get right audience right audience right media right media right creative right creative bottom line bottom line Garbage in, garbage out. “Pay no attention to the man behind Measures causality, not correlation, the curtain.” providing a true measure of ROI. ?? ??? ?? ??? ?? ??? right audience right media right creative bottom line
  • 26. about collective // Collective intelligently connects brands to audiences with high-impact experiences across display, video and mobile. Collective’s AMP(R) Data and Media Management platform powers the ad businesses of over 50 leading media brands, including our flagship media products, Collective Display and Collective Video.(R) Collective’s complete buy-side solution, Ensemble,(TM) provides brand advertisers with audience buying combined with rich media and DCO. Collective is headquartered in New York with offices in Atlanta, Boston, Chicago, Dallas, Detroit, Los Angeles, San Francisco, London and Bangalore. Collective’s investors include Accel Partners(R), Greycroft Partners and iNovia Capital. For more information, please visit www.collective.com. Press contact // Laura Colona Director of Communications lcolona@collective.com sales // contactus@collective.com 99 Park Avenue, 5th Floor New York, NY 10017 888-460-9513