At the Advertising Research Foundation’s 2011 Annual re:think convention, Duncan Watts & David Reilly, Principal Research Scientists at Yahoo! Labs, gave a presentation on Randomized Experiments in Advertising, entitled Breaking Wanamaker’s Curse. The presentation shows how to use randomized experiments to measure advertising effectiveness.
WordWrite Senior Account Executive Deanna Ferrari presented this to a group of senior citizens interested in learning more about Twitter and Facebook (for client Redstone Highlands).
WordWrite Senior Account Executive Deanna Ferrari presented this to the Clarion University of Pennsylvania PRSSA chapter, where she serves as professional advisor.
At the Advertising Research Foundation’s 2011 Annual re:think convention, Grant McCracken, Ph.D.-MIT, Author of Chief Culture Officer presented Great Creative: getting back to basics. Great Creative is a presentation that presents the idea that social media is not sufficient media and challenges advertisers to get back to the basics with traditional advertising to make brands matter.
At the Advertising Research Foundation’s (ARF) 2011 annual re:think convention, a key issues forum presentation was held entitled AD IMPACT. The panelists discussed researching ROI to build better campaigns. Five learning points were discussed. Emphasis was placed on the importance of reviewing previous research. Panelists included Barbara Zack-Chief Analytics Officer at Nielsen IAG Research & Kimberly Stanford-Sr. Campaign Insight Lead at Microsoft Corporation. The panel was moderated by George Terhanian-President, Global Solutions at Toluna.
At the Advertising Research Foundation’s (ARF) 2011 annual re:think convention, a key issues forum presentation was held entitled Blend, Balance, and Stabilize Respondent Sources: Building Representative Samples. Through the use of empirical examples and side by side tests, methods and solutions for building representative samples are presented. The presentation was given by Mitchell Eggers Ph.D, Chief Scientist at GMI inc. & Eli Drake Statistician at GMI inc.
At the Advertising Research Foundation’s (ARF) 2011 annual re:think convention, a key issues forum presentation was held entitled Engagement as a Behavioral and Sales-Correlated ROI Proxy. Presenters included Robert Passikoff, President of Brand Keys, Inc. & Charles Kennedy
SVP of Sales and Strategy Research for ABC Television Network. Questions such as “What drives media engagement” were addressed. The Disney Media and Advertising Lab and the effects of cross platform consumption were discussed.
Mobile Lunch Trucks penetrate deeply into the Hispanic and General Market population with product advertising. Also advertising on lunch trucks provide companies with an opportunity to target a segment, that can’t be reached by regular media.
WordWrite Senior Account Executive Deanna Ferrari presented this to a group of senior citizens interested in learning more about Twitter and Facebook (for client Redstone Highlands).
WordWrite Senior Account Executive Deanna Ferrari presented this to the Clarion University of Pennsylvania PRSSA chapter, where she serves as professional advisor.
At the Advertising Research Foundation’s 2011 Annual re:think convention, Grant McCracken, Ph.D.-MIT, Author of Chief Culture Officer presented Great Creative: getting back to basics. Great Creative is a presentation that presents the idea that social media is not sufficient media and challenges advertisers to get back to the basics with traditional advertising to make brands matter.
At the Advertising Research Foundation’s (ARF) 2011 annual re:think convention, a key issues forum presentation was held entitled AD IMPACT. The panelists discussed researching ROI to build better campaigns. Five learning points were discussed. Emphasis was placed on the importance of reviewing previous research. Panelists included Barbara Zack-Chief Analytics Officer at Nielsen IAG Research & Kimberly Stanford-Sr. Campaign Insight Lead at Microsoft Corporation. The panel was moderated by George Terhanian-President, Global Solutions at Toluna.
At the Advertising Research Foundation’s (ARF) 2011 annual re:think convention, a key issues forum presentation was held entitled Blend, Balance, and Stabilize Respondent Sources: Building Representative Samples. Through the use of empirical examples and side by side tests, methods and solutions for building representative samples are presented. The presentation was given by Mitchell Eggers Ph.D, Chief Scientist at GMI inc. & Eli Drake Statistician at GMI inc.
At the Advertising Research Foundation’s (ARF) 2011 annual re:think convention, a key issues forum presentation was held entitled Engagement as a Behavioral and Sales-Correlated ROI Proxy. Presenters included Robert Passikoff, President of Brand Keys, Inc. & Charles Kennedy
SVP of Sales and Strategy Research for ABC Television Network. Questions such as “What drives media engagement” were addressed. The Disney Media and Advertising Lab and the effects of cross platform consumption were discussed.
Mobile Lunch Trucks penetrate deeply into the Hispanic and General Market population with product advertising. Also advertising on lunch trucks provide companies with an opportunity to target a segment, that can’t be reached by regular media.
Discriminant analysis Analisis Diskriminan adalah salah satu tehnik analisa Statistika dependensi yang memiliki kegunaan untuk mengklasifikasikan objek beberapa kelompok. Pengelompokan dengan analisis diskriminan ini terjadi karena ada pengaruh satu atau lebih variabel lain yang merupakan variabel independen.
Delivered by Peter York, Founder and CEO of Algorhythm, at the 2016 Annual Community Meeting & Nonprofit Expo.
Attend our next event:
http://www.unitedwaynca.org/events/members
Webinar: Getting the Most Out of True Impact 2.0Localytics
In this webinar, we discuss best practices for measuring the success of your mobile marketing campaigns and walk through True Impact 2.0, which is Localytics' completely redesigned campaign performance tool. You will learn how True Impact 2.0 can prove that your campaigns are delivering meaningful results. We'll also show you how True Impact 2.0 can help you optimize every message you send.
Project TitlePROJECT TITLE Deployment of complete Open Sou.docxbriancrawford30935
Project Title:
PROJECT TITLE: Deployment of complete Open Source network infrastructure with equivalent provision for every necessary service provided in a typical Windows environment such as Active Directory, File and Printer Sharing, Firewalls, DNS, DHCP, Email Service, Web Service, FTP service, Chat Service, Certificate Services etc.
Project Fundamentals:
Please describe your IT project by answering these questions. Your answers need to address why this project is worth doing.
PROBLEM DEFINITION (Your IT project should solve a well-defined problem):
· What is the problem that you are addressing?
· Who is the end user, and what is the end user profile?
· What is the target market?
· Who is the organization?
PROPOSED IT SOLUTION (You should have a top-level idea of the solution or how you will solve the problem):
· What is the significance of this project?
· What is the proposed solution or approach?
· How do you propose to complete your project (It is important to explain how you propose to complete your project)?
· What tools and/or methodology (e.g. Network Diagram, IP Addressing, Security Technologies, Virtualization, Operating Systems, etc.) will be used to design, implement, and deliver the proposed solution?
· What type of resources (e.g., software, hardware, virtualization techniques, etc.) will you need to complete your project?
· Briefly describe the schedule of activities you will engage in to complete your project.
· As appropriate, include a budget with projected expenses and their importance to the project.
IMPACT ANALYSIS (Describe how the following issues impact your problem and its solution):
· Organizational
· Networking standards
· Security
· Ethical
· Social
· Legal
· Economic
· Target market/end user
CAUTION: Students often consider the impact analysis as an afterthought and give little serious thought to this section. However, considering these issues is an integral part of designing an IT solution or system in the broader context. Designs and their implementations have failed for lack of consideration of such issues.
REFLECTION ANALYSIS:
· Why does this proposal qualify as a capstone project?
· What technologies and methodologies does your capstone proposal incorporate that demonstrate your learning experience at Herzing?
· Are there any new technologies that will be utilized?
· How will your project further knowledge, understanding, or increase your skills in your discipline?
Running head: MANAGED CARE MYTHS 1
MANAGED CARE MYTHS 6
Managed Care Myths
Jessica Seifert
Rasmussen College
Feburary 25, 2018
Annotated Bibliography
Cordina, J., Kumar, R., & Moss, C. (2015). Debunking common myths about healthcare consumerism. McKinsey & Company.
Although true health consumerism is as yet developing graduall.
In 2011 targeting will replace reach as the key factor in any marketing campaign. As media channels grow exponentially advertising messages are becoming noise. How do you cut through to reach the people that matter?
Insights from Coalition for Innovative Media Measurement (CIMM) were presented at Advertising Research Foundation’s (ARF) AM 6.0 conference, held in 2011. Through the use of eDiaries and the example of moms, a presentation was given on the value of USA touchpoints to increase ROI. Presenters included Jane Clarke-Managing Director of Coalition for Innovative Media Measurement & Jim Spaeth-Partner at Media Behavior Institute.
As part of the emerging new topics series, a presentation entitled UX: The User Experience Cross-Screen, was given at the Advertising Research Foundation’s (ARF) AM 6.0 conference held in 2011. A research study including questions, methods, and findings on cross-screen users was presented. The presentation was given by, Richard Zackon-Facilitator for Council for Research Excellence & Rick Ducey- Chief Strategy Officer at BIA/Kelsey.
A presentation on new ad models, was given at the Advertising Research Foundation’s (ARF) AM 6.0 conference held in 2011. Dr. Duane Varan-Executive Director at Audience Research Labs Murdoch University gave the presentation. Varan presented studies done by the Beyond 30 research program.
At the Advertising Research Foundation’s (ARF) AM 6.0 conference held in 2011, a Keynote presentation was given entitled Meeting the Challenges of Cross Media Research. Google presented three initiatives and how they are trying to help meet the challenges of cross media research. Keynote presenters from Google were Owen Charlebois-Global Manager, Advertising, Marketing and Media Research & ElissaLee, Ph.D.-Group Manager,Quantitative Measurement.
A presentation entitled Impact of Online and Television Advertising on Consumer Behavior, was given at the Advertising Research Foundation’s (ARF) AM 6.0 conference held in 2011. The presentation includes, the challenges to measuring cross-platform media impacts, comscore methods, case studies and results. Presenters included Joan FitzGerald-Vice President of comScore & Alan Vaughn- Statistical Analyst at comScore.
A presentation entitled Following the Sports Fan with Arbitron’s PPM, was given at the Advertising Research Foundation’s (ARF) AM 6.0 conference held in 2011. A study using Arbitron Portable People Meter™ (PPM™) service was presented. The presentation was given by Glenn Enoch- VP, Integrated Media Research at ESPN Inc., Kelly Johnson- Director, Media and Promotion Research at ESPN Inc., & Lung Huang- VP, National Account Services at Arbitron Inc.
A presentation entitled BrandLife – Showing the Client What to Do Visually and What to Avoid was given at the Advertising Research Foundation’s (ARF) AM 6.0 conference held in 2011. The Interpretation of visual Semiotic “Answers” is presented. The presentation is given by Steve Wolf- Commercial Director, Global Qualitative & SVP/Division Director at Synovate Qualitative.
At the Advertising Research Foundation’s (ARF) AM 6.0 conference held in 2011, a presentation was given on Facebook’s information platform. The job of research team at Facebook along with the topic of making research interesting to users is discussed. The presentation is given by Brad Smallwood-Head of Measurement and Insights at Facebook
A presentation entitled The Television Audience Redefined: An Update, was given at the Advertising Research Foundation’s (ARF) AM 6.0 conference held in 2011. Insights into the profiles of various consumers across a variety of media platforms, was presented. The presentation was given by David Poltrack-Chief Research Officer of CBS Corporation & President of CBS Vision.
A presentation entitled Measuring the Moving Screen: The Future of Tablet Measurement, was given at the Advertising Research Foundation’s (ARF) AM 6.0 conference held in 2011. A presentation was given on the profile tablet users as well as the findings regarding ad effectiveness on iPads vs iPhones and other devices. The presentation was given by Matt O’Grady- EVP, Media Audience Measurement at The Nielsen Company.
A presentation entitled MAXIMIZING MOMENTUM: Insights from Verizon, SMG & Cisco, was given at the Advertising Research Foundation’s (ARF) AM 6.0 conference. Insights on Brand building were presented. Presenters included Patrick McLean-Executive Director at Verizon Interactive, Helen Katz-SVP/Research Director at Starcom MediaVest Group, & Charlie Treadwell-Marketing Strategy Manager Corporate Affairs Marketing at Cisco.
A presentation entitled Integrated Social Media: Business Impact Study 2011, was given at the Advertising Research Foundation’s (ARF) AM 6.0 conference. The presentation explores the impact that social media has on sales. Presenters included Irfan Kamal- SVP, Digital/Social at Ogilvy & Dr. Walter Carl- Founder and Chief Research Officer at ChatThreads.
A presentation entitled Global Perspectives on Measurement Issues, was given at the Advertising Research Foundation’s (ARF) AM 6.0 conference held in 2011. Presenters included Horst Stipp-EVP, Global Business Strategy at the Advertising Research Foundation & LuizG. Duarte, Ph.D.-Corporate Research at DIRECTV Latin America.
A presentation entitled Insights for the Future of Online Video Commercialization given at the Advertising Research Foundation’s (ARF) AM 6.0 conference held in 2011. Test on online commercial viewing done by TBS was presented. The presentation included the research design and methods used as well as the findings. The presentation was given by Stacey Lynn Schulman-SVP, Ad Sales & Sports Research at Turner Broadcasting Sales, Inc.
At the Advertising Research Foundation’s (ARF) 2011 annual re:think convention, a key issues forum presentation was held entitled New Shopper Journeys: The Influence of Digital Touchpoints. The presentation explores how digital has an effect on shopper’s behaviors. Presentation is given by Alex Charlton- Director at essential research, Mike Hess- EVP of Research at Carat N.A., & Beth Uyenco- Global Research Director at Microsoft.
At the Advertising Research Foundation’s (ARF) 2011 annual re:think convention, a key issues forum presentation was held entitled Supercharging the Path to Purchase: Using Word-of-Mouth to Drive More Consumers to Buy. The presentation looks at the effects of Word of Mouth advertising as it relates to female consumers. Presenters include Ed Keller-CEO of Keller Fay Group & Tony Cardinale-EVP, Brand Planning & Strategic Insights at NBC Universal.
At the Advertising Research Foundation’s (ARF) 2011 annual re:think convention, a key issues forum presentation was held entitled SHOPPER INSIGHTS. Ashmeed Ali-Senior Manager at Yahoo! & David Gill-Vice President at The Nielsen Company discussed when Mobile Goes Shopping. This presentation explored how consumers use their mobiles to research products and advertising on mobile devices. Manila Austin-Director of Research at Communispace Corporation moderated the discussion.
At the Advertising Research Foundation’s (ARF) 2011 annual re:think convention, a key issues forum presentation was held entitled The Power of Relevancy The Biometric Impact of Online Advertising. The presentation includes topics such as measuring the biological impact, and the contextual and personal relevancy of ads. The presenters include Tony Marlow-Strategic Insights at YAHOO! & Brian Levine-President of Innerscope Research.
At the Advertising Research Foundation’s (ARF) 2011 annual re:think convention, a key issues forum presentation was held entitled Understanding the Relationship Between Social Media and TV Viewership. The presentation used a case study on the World cup to present findings. Presenters included, Jon Gibs-SVP at The Nielsen Company & David Coletti-VP at ESPN.
More from The Advertising Research Foundation (20)
Company Valuation webinar series - Tuesday, 4 June 2024FelixPerez547899
This session provided an update as to the latest valuation data in the UK and then delved into a discussion on the upcoming election and the impacts on valuation. We finished, as always with a Q&A
VAT Registration Outlined In UAE: Benefits and Requirementsuae taxgpt
Vat Registration is a legal obligation for businesses meeting the threshold requirement, helping companies avoid fines and ramifications. Contact now!
https://viralsocialtrends.com/vat-registration-outlined-in-uae/
Recruiting in the Digital Age: A Social Media MasterclassLuanWise
In this masterclass, presented at the Global HR Summit on 5th June 2024, Luan Wise explored the essential features of social media platforms that support talent acquisition, including LinkedIn, Facebook, Instagram, X (formerly Twitter) and TikTok.
In the Adani-Hindenburg case, what is SEBI investigating.pptxAdani case
Adani SEBI investigation revealed that the latter had sought information from five foreign jurisdictions concerning the holdings of the firm’s foreign portfolio investors (FPIs) in relation to the alleged violations of the MPS Regulations. Nevertheless, the economic interest of the twelve FPIs based in tax haven jurisdictions still needs to be determined. The Adani Group firms classed these FPIs as public shareholders. According to Hindenburg, FPIs were used to get around regulatory standards.
Buy Verified PayPal Account | Buy Google 5 Star Reviewsusawebmarket
Buy Verified PayPal Account
Looking to buy verified PayPal accounts? Discover 7 expert tips for safely purchasing a verified PayPal account in 2024. Ensure security and reliability for your transactions.
PayPal Services Features-
🟢 Email Access
🟢 Bank Added
🟢 Card Verified
🟢 Full SSN Provided
🟢 Phone Number Access
🟢 Driving License Copy
🟢 Fasted Delivery
Client Satisfaction is Our First priority. Our services is very appropriate to buy. We assume that the first-rate way to purchase our offerings is to order on the website. If you have any worry in our cooperation usually You can order us on Skype or Telegram.
24/7 Hours Reply/Please Contact
usawebmarketEmail: support@usawebmarket.com
Skype: usawebmarket
Telegram: @usawebmarket
WhatsApp: +1(218) 203-5951
USA WEB MARKET is the Best Verified PayPal, Payoneer, Cash App, Skrill, Neteller, Stripe Account and SEO, SMM Service provider.100%Satisfection granted.100% replacement Granted.
Building Your Employer Brand with Social MediaLuanWise
Presented at The Global HR Summit, 6th June 2024
In this keynote, Luan Wise will provide invaluable insights to elevate your employer brand on social media platforms including LinkedIn, Facebook, Instagram, X (formerly Twitter) and TikTok. You'll learn how compelling content can authentically showcase your company culture, values, and employee experiences to support your talent acquisition and retention objectives. Additionally, you'll understand the power of employee advocacy to amplify reach and engagement – helping to position your organization as an employer of choice in today's competitive talent landscape.
Understanding User Needs and Satisfying ThemAggregage
https://www.productmanagementtoday.com/frs/26903918/understanding-user-needs-and-satisfying-them
We know we want to create products which our customers find to be valuable. Whether we label it as customer-centric or product-led depends on how long we've been doing product management. There are three challenges we face when doing this. The obvious challenge is figuring out what our users need; the non-obvious challenges are in creating a shared understanding of those needs and in sensing if what we're doing is meeting those needs.
In this webinar, we won't focus on the research methods for discovering user-needs. We will focus on synthesis of the needs we discover, communication and alignment tools, and how we operationalize addressing those needs.
Industry expert Scott Sehlhorst will:
• Introduce a taxonomy for user goals with real world examples
• Present the Onion Diagram, a tool for contextualizing task-level goals
• Illustrate how customer journey maps capture activity-level and task-level goals
• Demonstrate the best approach to selection and prioritization of user-goals to address
• Highlight the crucial benchmarks, observable changes, in ensuring fulfillment of customer needs
LA HUG - Video Testimonials with Chynna Morgan - June 2024Lital Barkan
Have you ever heard that user-generated content or video testimonials can take your brand to the next level? We will explore how you can effectively use video testimonials to leverage and boost your sales, content strategy, and increase your CRM data.🤯
We will dig deeper into:
1. How to capture video testimonials that convert from your audience 🎥
2. How to leverage your testimonials to boost your sales 💲
3. How you can capture more CRM data to understand your audience better through video testimonials. 📊
An introduction to the cryptocurrency investment platform Binance Savings.Any kyc Account
Learn how to use Binance Savings to expand your bitcoin holdings. Discover how to maximize your earnings on one of the most reliable cryptocurrency exchange platforms, as well as how to earn interest on your cryptocurrency holdings and the various savings choices available.
B2B payments are rapidly changing. Find out the 5 key questions you need to be asking yourself to be sure you are mastering B2B payments today. Learn more at www.BlueSnap.com.
Helen Lubchak: Тренди в управлінні проєктами та miltech (UA)
Breaking wanamaker's curse randomized experiments in advertising
1.
2. Breaking Wanamaker’s Curse
Randomized Experiments in
Advertising
Duncan Watts David Reiley
Principal Research Scientist Principal Research Scientist
Yahoo! Labs Yahoo! Labs
3. Half the money I spend
on advertising is wasted.
The trouble is,
I don’t know
which half.
John Wanamaker (1838-1922)
4. That Was 100 Years Ago
• John Wanamaker died in 1922
– Before quantum mechanics, penicillin, radar, lasers,
jet engines, space travel, computers, the Internet…
• How is that we have made such dramatic
progress in physical and engineering
sciences, yet still don’t know how to measure
Ad Effectiveness?
5. Two Responses
1. “Measuring advertising effectiveness is
impossible!”
– We know that advertising works, but its effects are too
long-term, intangible, etc. to measure
2. “We do know how to measure ad effectiveness,
and do it all the time!”
– Metrics have improved vastly, as have statistical
(marketing mix) models
6. The “It’s Impossible” Response
• How is possible both to “know” that advertising works and
also that its effects can’t be measured?
– Seem like logically contradictory statements
• Resolution seems to be that although effects of individuals
campaigns are “impossible” to measure, seems obvious that
advertising as a whole “works”
• But this is still problematic
– We also know that at some level medical science “works” but this doesn’t
mean we don’t try to measure the effectiveness of particular procedures,
drugs etc.
– Also not clear that effectiveness of campaigns can’t be measured
• Why not apply the same standards of evidence to advertising
that we do to medical (and other) science?
7. The “Already Done” Response
• True that Advertising is increasingly metrics-driven
– Neilsen, Comscore, survey and panel data, market research, server
data, sales data
• Also true that statistical modeling has become more
sophisticated
• But observational data alone only reveals
correlations between sales and advertising,
not whether advertising causes sales
– Statistical models try to “identify” causal effects, but this is
notoriously hard to do
8. Correlation vs. Causation
• “Everyone knows that correlation is not causation”
• But it is remarkably easy to get them mixed up in
practice
– If Mary went on a diet and also lost 20lbs, the diet caused the weight
loss, right?
• Well maybe, but she was probably doing other things as well
(watching what she eats in general, exercising, etc.)
• Nevertheless, we focus on the diet and tell a plausible story
• True of human behavior in particular
– We have so much intuition, we can always come up with a plausible
story about why someone did what they did
– Tempting to infer causality, when in reality all we know is that “X
happened and then Y happened”
9. Problems with Causal Inference
from Observational Data
• MMM exploits observational variation in advertising
levels to estimate a model
– Advertising levels must vary so that we can estimate slope coefficients
• Key question: what causes the variation in
advertising?
• Reverse causality?
– Advertisers setting their budget based on previous year’s sales
• Correlations will be strong even if ads have no effect
– Big product launches attract large ad campaigns
– Advertisers spend more during holidays
10. Selection Problems
• HBR article by the founder and president of ComScore
(Abraham, 2008) illustrates the state of the art:
– Compares those who saw an online ad with those who didn’t.
Measures huge effects for search ads, smaller for display ads.
– Potential problem: the two samples do not come from the same population
– Example: Who sees an ad for eTrade on Google?
• Those who search for “online brokerage” and similar keywords
• Does the ad actually cause the difference in sales?
• In general, people who see a given ad not the same as people
who don’t see it
– Because they read certain magazines, browse certain website, search for
certain terms, or visit certain places
– In focusing on most likely consumers, targeting also makes it more difficult to
identify the “marginal” consumer
11. What’s the Solution?
• Randomized, controlled experiments
– Randomly assign everyone in target population to “treatment”
and “control” groups
– Only treatment group is exposed to campaign
– Measure the difference in outcomes
• Randomization is the key
– Only by randomizing treatment can the various confounding factors
be eliminated
• In medical science, this is uncontroversial
– Observational studies also suffer from selection problem
• People who select into a particular treatment or behavior are different
from people who don’t
– Randomized trials therefore the gold standard
12. Can it Be Done?
• Experiments are hard to design properly, and harder to
implement
– Often hard to control who sees an ad
– Also hard to measure outcomes, which may be distant in space and time from
treatment
• However, it can be done
– Split cable TV Experiments in 1980’s/90’s
– Direct mail marketing
– Increasingly possible in Online Advertising
• David Reiley, Randall Lewis and colleagues at Yahoo! Labs
have conducted series of groundbreaking field experiments
– Over to David…
13. At Yahoo! Labs, We are Putting the
Experimental Method to Good Use
• Several different clients have agreed to careful
experiments using a control group:
– Three retailers. (Offline sales data!)
– Several online service providers
– Internal Yahoo! properties
• A variety of outcomes can be measured:
– Online sales or other conversions
– Offline sales, in special cases
– Survey questions (brand affinity, etc.)
– Online searches
14. Case Study: A Retailer Found Large Effects of
Yahoo! Display Ads on Existing Customers
• This retailer keeps careful records, attributing >90% of in-store purchases
to the correct individual customer
• We found 1.6 million customers who matched (name and address, either
email or snail-mail) between the databases of the retailer and Yahoo!
• 80% of matched customers assigned to the treatment group
– Targeted with retail-image ad campaigns from the retailer
• 20% assigned to the control group
– Do not see these retailer ads
• Ad campaigns are “Run of Network” on Yahoo!
• Following the online ad campaigns, we received both online and in-store
sales data: for each week, for each person
– Third party de-identifies observations to protect customer identities
– Retailer disguises all sales amounts (R$) with a scalar multiple of USD
15. Descriptive Statistics Indicate a Valid
Treatment-Control Randomization
Control Treatment
% Female 59.5% 59.7%
% Retailer Ad Views > 0 0% 63.7%
% Y! Page Views > 0 76.4% 76.4%
Mean Y! Page Views Per Person 358 363
% Ad Impressions Clicked (CTR) - 0.28%
% Viewers Clicking At Least Once - 7.21%
16. Experimental Differences Show a
Positive Increase in Sales Due to the Ads
During
Mean Sales/Person Campaign
(2 wks)
R$1.84
Control Group
(0.03)
1.89
Treatment Group
(0.02)
95% C.I. for treatment is R$0.05±0.07.
(For treatment effect on the treated: R$0.08±0.11.)
(Standard errors in parentheses.)
17. Suppose We Had No Experiment, and Just Compared
Spending by Those Who Did or Did Not See Ads
During
Mean Sales/Person Campaign
(2 wks)
R$1.84
Control Group
(0.03)
1.89
Treatment Group
(0.02)
1.81
Exposed (64% of TG)
(0.02)
2.04
Unexposed (36% of TG)
(0.03)
We would conclude that ads decrease sales by R$0.23 per person!
Not comparing apples to apples here.
(Standard errors in parentheses.)
18. Pre-Campaign Data Emphasize that the
Non-Experimental Sales Differences Have
No Causal Relationship to the Ad Exposures
Before During
Mean Sales/Person Campaign Campaign
(2 wks) (2 wks)
R$1.84
Control Group
(0.03)
1.89
Treatment Group
(0.02)
1.81 1.81
Exposed (64% of TG)
(0.02) (0.02)
2.15 2.04
Unexposed (36% of TG)
(0.03) (0.03)
People who browse enough to see the ads also have a lower
baseline propensity to purchase from the retailer!
(Standard errors in parentheses.)
19. Ad Exposures Appear to Have Prevented a Normal
Decline in Sales During this Time Period
Before During
Mean Sales/Person Difference
Campaign Campaign (During-Before)
(2 wks) (2 wks)
R$1.95 R$1.84 -R$0.10
Control Group
(0.04) (0.03) (0.05)
1.93 1.89
Treatment Group
(0.02) (0.02)
1.81 1.81 0.00
Exposed (64% of TG)
(0.02) (0.02) (0.03)
2.15 2.04 -0.10
Unexposed (36% of TG)
(0.03) (0.03) (0.04)
Control group falls. Untreated group falls.
Treated group holds constant.
(Standard errors in parentheses.)
20. Our Difference-in-Difference Estimate Yields a
Statistically and Economically Significant Effect
• Estimated effect per customer of viewing ads
– Mean = R$ .102, SE = R$ .043
• Estimated sales impact for the retailer
– R$83,000 ± 70,000
• 95% confidence interval
• Based on 814,052 treated individuals
• Compare with cost of about R$25,000
• 325% increase in revenue relative to cost
• Note the wide confidence interval. But it’s actually
much narrower than that for a “successful” IRI
BehaviorScan test
– More like R$83,000±190,000
21. What Happens After the Two-Week
Campaign is Over?
• Positive effects during the campaign could be
followed by
– Negative effects (intertemporal substitution)
– Equal sales (short-lived effect of advertising)
– Higher sales (persistence beyond the campaign)
• We can distinguish between these hypotheses by
looking at the week following the two weeks of
the campaign
22. We See a Positive Impact on Sales in the
Week After the Campaign Ends
• Previous two-week estimate
– R$0.102 (0.043) per person
• Estimate for third week
– R$0.61 (0.024) per person
– As large as the effect per week during the campaign
• Including the third week, the total impact of the ads becomes
– R$135,000±85,000
– Compared with cost of R$25,000
• Extending out five weeks, the total looks as high as
R$250,000±190,000 (compared with cost of R$33,000)
(Standard errors in parentheses.)
23. We Break Down the Experimental
Lift by Sales Channel
3-Week Sales Lift/Person: Total Offline Online
R$0.166 R$0.155 R$0.011
Viewed Ads
(0.05) (0.05) (0.02)
93% of the effect occurs in stores!
Three-week difference-in-difference estimator.
24. Do We Capture the Effects of Ads by
Measuring Only Clicks? No.
3-Week Sales Lift/Person: Total Offline Online
R$0.166 R$0.155 R$0.011
Viewed Ads
(0.05) (0.05) (0.02)
0.508 0.215 0.292
Clicked [7%]
(0.02) (0.02) (0.03)
0.139 0.150 -0.010
Didn’t Click [93%]
(0.05) (0.05) (0.02)
78% of the total lift comes from viewers who never clicked!
With an experiment, no attribution model is required
Three-week difference-in-difference estimator.
25. The Increase Appears to Consist of About ¼ Increase
in Transactions, ¾ Increase in Basket Size
• Prob(transaction) increases by 0.1% (0.05%)
– Baseline amount = 6.5%
– Percentage increase = 1.5%
• Mean basket size increases by R$1.75 (0.74)
– Baseline amount = R$41.
– Percentage increase = 4.2%
• Both effects are statistically significant at the
5% level
26. Wasted Half? Older Customers’ Purchases
Responded More than Younger Ones’
Kernel smoothing of treatment-control difference, Epanechnikov kernel, bandwidth ~2 years.
27. We Designed a Second Study to Measure
the Impact of Frequency
• This time, we have 3 million matched users
• Two campaigns in two weeks
• Three equal-sized treatment groups
– Control (no ads)
– Half frequency (17 impressions/person on average)
– Full frequency (34 impressions/person on average)
• This time, we deliver Y! house ads as control impressions
– Mark hypothetical views of the Control group
– Also mark views the Half group would have seen in the Full group
• Again we see in-store and online transaction data for each
customer during the experiment
• Real US dollars this time
29. Frequency Has Surprisingly High Marginal Impact
When Going from 17 to 34 Ads Per Person
Purchases During 2 Weeks Control Half Full
$17.62 $17.98 $18.22
Mean purchase amount
(0.17) (0.17) (0.17)
0.36 0.60
Difference from Control
(0.24) (0.24)
Doubling the frequency increases sales by 50% more.
Diminishing returns not as high as we might have thought!
We include 60% of users,, all those who viewed either a treatment or a control ad.
This aggregates effects of both campaigns. Second week showed much larger effects than the first.
30. The Impact of the Ads is Greatest for Those
Who Live Within Two Miles of a Retail Store
$0.50 avg. effect for all customers: 7X ROAS.
$3.00 avg. effect for those within 2 miles of a store: 36X ROAS!
31. At Yahoo!, We Can Also Measure the
Increase in Searches Due to Display Ads
In the treatment group, 1300 searches take place
within 5 minutes of an ad impression in our retail ad campaign
32. Without an Experiment, We Would
Have Overestimated The Effects
True incremental impact: 560 searches, not 1314
33. Correlated Online Behaviors can Lead to
Overestimates of the Effects of Advertising
• Activity bias: People who are doing an activity online are
much more likely to be doing other online activities at around
the same time
• Lewis, Rao, and Reiley (2011), “Here, There, and Everywhere,”
documents several examples of activity bias
– Impact of display ads on search queries for an advertiser’s brand
– Impact of display ads on online conversions (new account applications)
– Impact of Y! video ads shown on Amazon Mechanical Turk for Y! page views
• In each case, failing to use an experiment gives us an
overestimate of true causal effects
34. We Intend to Do Lots More
Ad-Effectiveness Experiments at Yahoo!
• Currently building an experimentation platform to
make the process easy and scalable
• We’re pleased to introduce Ken Mallon and AdLabs,
an entire unit devoted to measuring effectiveness
• We’re looking for clients interested in using
experiments to measure online ad effectiveness
– Willing to share online or offline conversion data
– Willing to give up reach in order to invest in information about what
works best
• Targeting
• Creative
• Frequency
• Etc.
35. Lots of Yahoos to Thank, Including
• Randall Lewis • Garrett Johnson
• Taylor Schreiner • Justin Rao
• Valter Sciarillo • Ken Mallon
• Meredith Gordon • Erin Carlson
• Christine Turner • Rick Grimes
• Iwan Sakran • Melissa Chickering
• Sergiy Matusevych • Jim Zepp
36. The Problem with Obvious
• Paradoxically, our intuition for human
behavior may actually impede our
understanding of it
• We can always imagine how advertising
will affect people
– Not the same as knowing how it affects
people
• Similar problems arise in business,
government and science as well
– “It’s not rocket science”
– Ironic, because we’ve made more progress in
rocket science since Wanamaker’s day than in
dealing with human affairs
• Read more at everythingisobvious.com