SlideShare a Scribd company logo
1 of 17
Download to read offline
Morphing Theory and Applications
to Online Experiments:
Learning While Earning
DDTT 2015
Gui Liberali
Erasmus University
Laboratory for
Optimization of Digital
Experiments @ Erasmus
Morphing
One size does not fit all
(a) General content, large-load, graphical morph (b) Focused, small-load, verbal morph
Visual
Technical
Content
More
Content
Audio
Less
Content
General
Content
Website designs as Morphs
Appeals to Verbal
Cognitive Style
Appeals to Visual
Cognitive Style
Appeals to Holistic, impulsive cognitive
style
Appeals to Analytical, deliberative
cognitive style
One size does not fit all
Banners as Morphs
•  From learning-then-earning to the world of
learning-while-earning
•  Higher CTR, conversions
•  Better user experience
Where’s your A/B
Testing ?
•  From aggregate to disaggregate data
•  From demographics to cognitive styles
1. Learning about Styles vs Morphs
Firm	
  Side:	
  Banner	
  Ads	
  
	
  	
   Buy!	
  
versus	
  
Learn	
  
More	
  
Inform
a4ve	
  
Emo4o
nal	
  	
  
Time-­‐
pressu
re	
  	
  
	
  	
  	
  	
  	
  
	
  Consumer	
  
Side:	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  
4	
  Cogni4ve	
  
Styles	
  
Verbal-­‐Impulsive	
   0.1	
   0.11	
   0.2	
   0.13	
  
Visual-­‐Impulsive	
   0.11	
   0.2	
   0.13	
   0.1	
  
Verbal-­‐Delibera4ve	
   0.2	
   0.13	
   0.1	
   0.11	
  
Visual-­‐Delibera4ve	
   0.11	
   0.2	
   0.13	
   0.1	
  
Optimization: from Earn to Learn
• Maximize conversion by picking the best among ads for current user
• If we knew the conversion probabilities perfectly we would simply serve the
optimal morph
• If we knew nothing about the conversion probabilities we would randomize
and learn
Need to balance learning
the true cognitive style
(exploitation) and providing
the optimal morph given
what we know now
(exploration).
For each person!
Morphing: real-time adaptation of the
firm’s actions to each consumer
Cognitive-style, segment, position in
the purchase funnel, ..
Website design, banner, advertising
campaign, media copy per channel,
promotion, price, product bundle, product
recommendation …
Consumer
latent
variable
Marketing
instrument
to be adapted
(morphed)
x
Morphing - Evidences from the Field
Morphing Banner Advertising (2014)
•  Morphing almost doubled click-through rates for context-
matched banners (83%, p<0.01 and 97%, p=0.028) lifts,
respectively, for banners and for consumers
•  Was live on CNET.com (8.4M visitors/day) for a month
•  Extends behavioral targeting to avoid local max solutions
Website Morphing (2009)
•  Online sales of BT group’s broadband plans can be
increased up to 20% by adapting the website to
match cognitive styles using our adaptive learning
algorithm
Website Morphing 2.0 (2014)
•  NPV reward improvement of 69% over the NPV of
the original algorithm; corresponds to $17.5 million
for BT group if implemented
•  Proof-of-Feasibility on Suruga Bank
•  Generalized version of morphing, including when to
morph, switching costs, multiple morphs
Morphing Algorithm
Challenges Addressed
1.  What is the style of the current user?
–  Update our beliefs as new clicks become available
2.  What is the optimal morph for each style?
–  Update as we observe successes and failures
Morphing	
  
Choice	
  descriptor	
  
Priors	
  on	
  distribu5on	
  of	
  
styles	
  
Updated	
  user	
  
style	
  descriptor	
   On-­‐line	
  
Op4miza4on	
  
Pgm:	
  probability	
  of	
  
purchase	
  for	
  individual	
  in	
  
style	
  g	
  given	
  morph	
  m	
  	
  
Banner	
  Ads	
  -­‐>	
   1	
   2	
   3	
   4	
  
	
  	
  	
  	
  	
  
Cogni4ve	
  
Style	
  
Impulsive	
  
Delibera4ve	
   0.1	
   0.11	
   0.2	
   0.13	
  
Illustra5ve	
  example	
  
User	
  
Op5mal	
  morph	
  for	
  current	
  	
  user	
  
Website	
  
	
  
User	
  makes	
  a	
  
purchase	
  or	
  
quit	
  
Bayesian	
  
update	
  
Optimal solution with Gittins’ indices
(assume we know cognitive style)
Gittins’ indices for the
eight morphs.
Morph that
was chosen.
0
1
2
3
4
5
6
7
8
0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000
Visitor
ChosenMorph
0.35
0.45
0.55
0.65
0.75
0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000
Visitor
Gittins'Indices
Morph 0 Morph 1 Morph 2 Morph 3 Morph 4 Morph 5 Morph 6 Morph 7
System experiments with Morph 3
for a while before settling back to
Morph 2.
•  What is the cognitive style of the current user?
–  Update our beliefs as new clicks become available
•  What is the optimal morph for each style?
–  Update as we observe successes and failures
Website
Morphing
(2009)
•  How to handle repeat visits and cumulative learning ?
–  Field experiment on cnet.com w/ AT&T banner ads
–  Roll-back mechanism
Morphing
Banner
Advertising
(2014)
•  Generalized Morphing Algorithm: w/ When to Morph
–  DP solves trade-off between gains from morphing early (longer
exposure) and gains from morphing later (better knowledge)
–  Exposure to multiple morphs
•  Attribution problem: recency, exposure, both
•  Switching costs: changing morphs only if expected rewards exceed
penalty for switching
Website
Morphing 2.0
(2014)
Morphing Trilogy
Morphing in the Field
•  Can be used normatively (e.g., field experiments) or descriptively (secondary
data)
•  Two tuning parameters makes it flexible to various applications
–  Switching costs
–  Different attribution theories/regimes
•  Morphing is appropriate to contexts with
–  High volume of observations
–  Clear success/failure outcome
–  Latent variable
Laboratory on Optimization of Digital
Experiments (LODE) @ Erasmus University
•  Executive education course on Digital Experimentation and Big Data
•  Research Fridays: Bi-monthly meetings of academics and founding companies at
Erasmus University
•  Bi-yearly publication of research priorities for research on A/B testing
•  Erasmus students running A/B testing & morphing with real-world company traffic
–  Module 1: Text-analysis methods
–  Module 2: A/B experiments
–  Module 3: Morphing
References
•  Hauser JR, Urban GL, Liberali G, Braun M (2009) Website morphing. Marketing
Sci. 28(2):202–224.
•  Urban G., Liberali G., MacDonald E., Bordley R., Hauser J. (2014) Morphing
Banner Advertising, Marketing Science, 33(1): 27-46.
•  Hauser J., Liberali G., Urban G. (Summer 2014) Website Morphing 2.0: Switching
Costs, Partial Exposure, Random Exit, and When to Morph. Management Science.
Thank you!
Contact: liberali@mit.edu and liberali@rsm.nl

More Related Content

Similar to Gui Liberali, Morphing Websites - Digital Data Tips Tuesday #4 - Growth through Data - Amsterdam - May 19th 2015 - #DDTT

Lean startups en el mundo real ejemplos y metricas
Lean startups en el mundo real  ejemplos y metricasLean startups en el mundo real  ejemplos y metricas
Lean startups en el mundo real ejemplos y metricasSoftware Guru
 
Big Data LDN 2017: Advanced Analytics Applied to Marketing Attribution
Big Data LDN 2017: Advanced Analytics Applied to Marketing AttributionBig Data LDN 2017: Advanced Analytics Applied to Marketing Attribution
Big Data LDN 2017: Advanced Analytics Applied to Marketing AttributionMatt Stubbs
 
WEBINAR: How To Bring Native And Programmatic Advertising Together
WEBINAR: How To Bring Native And Programmatic Advertising TogetherWEBINAR: How To Bring Native And Programmatic Advertising Together
WEBINAR: How To Bring Native And Programmatic Advertising TogetherInMobi
 
UMCD 2011 Conference - Is Your Elephant Shy?
UMCD 2011 Conference - Is Your Elephant Shy?UMCD 2011 Conference - Is Your Elephant Shy?
UMCD 2011 Conference - Is Your Elephant Shy?Tim Elleston
 
Auditing Agency Performance 110109x
Auditing Agency Performance 110109xAuditing Agency Performance 110109x
Auditing Agency Performance 110109xShane Crombie
 
New challenges for scalable machine learning in online advertising
New challenges for scalable machine learning in online advertisingNew challenges for scalable machine learning in online advertising
New challenges for scalable machine learning in online advertisingOlivier Koch
 
Ad Yield Optimization @ Spotify - DataGotham 2013
Ad Yield Optimization @ Spotify - DataGotham 2013Ad Yield Optimization @ Spotify - DataGotham 2013
Ad Yield Optimization @ Spotify - DataGotham 2013Kinshuk Mishra
 
Measurement module 100704
Measurement module 100704Measurement module 100704
Measurement module 100704GroupM
 
SalesFUSION Webinars - Taming the flood of digital data
SalesFUSION Webinars - Taming the flood of digital dataSalesFUSION Webinars - Taming the flood of digital data
SalesFUSION Webinars - Taming the flood of digital dataSalesfusion
 
Self Service Online Research - online communities for research and insights
Self Service Online Research - online communities for research and insightsSelf Service Online Research - online communities for research and insights
Self Service Online Research - online communities for research and insightsStephen Thompson
 
Narrative Mind Week 8 H4D Stanford 2016
Narrative Mind Week 8 H4D Stanford 2016Narrative Mind Week 8 H4D Stanford 2016
Narrative Mind Week 8 H4D Stanford 2016Stanford University
 
Estudio34 Presents- Dara Fitzgerald Brighton SEO-Next Gen Measurement With Go...
Estudio34 Presents- Dara Fitzgerald Brighton SEO-Next Gen Measurement With Go...Estudio34 Presents- Dara Fitzgerald Brighton SEO-Next Gen Measurement With Go...
Estudio34 Presents- Dara Fitzgerald Brighton SEO-Next Gen Measurement With Go...William Renedo
 
JBP-Cobalt all services
JBP-Cobalt all servicesJBP-Cobalt all services
JBP-Cobalt all servicesDuy Nguyen
 
Collaborative visualization supporting complex data driven service build - Bi...
Collaborative visualization supporting complex data driven service build - Bi...Collaborative visualization supporting complex data driven service build - Bi...
Collaborative visualization supporting complex data driven service build - Bi...webwinkelvakdag
 
Programmatic Advertising Solutions for Retailers and Brands
Programmatic Advertising Solutions for Retailers and BrandsProgrammatic Advertising Solutions for Retailers and Brands
Programmatic Advertising Solutions for Retailers and BrandsVeronika Sonsev
 
Lean Startup Metrics & Analytics
Lean Startup Metrics & AnalyticsLean Startup Metrics & Analytics
Lean Startup Metrics & AnalyticsNicola Junior Vitto
 
Harnessing the content beast – Content marketing in the multiscreen world
Harnessing the content beast – Content marketing in the multiscreen worldHarnessing the content beast – Content marketing in the multiscreen world
Harnessing the content beast – Content marketing in the multiscreen worldThomas Robbins
 
Brent Chudoba Presentation
Brent Chudoba PresentationBrent Chudoba Presentation
Brent Chudoba PresentationMediabistro
 
Ria Sankar - How to Build Winning Products - Product School Bellevue - 83018
Ria Sankar - How to Build Winning Products - Product School Bellevue - 83018 Ria Sankar - How to Build Winning Products - Product School Bellevue - 83018
Ria Sankar - How to Build Winning Products - Product School Bellevue - 83018 Ria Sankar
 

Similar to Gui Liberali, Morphing Websites - Digital Data Tips Tuesday #4 - Growth through Data - Amsterdam - May 19th 2015 - #DDTT (20)

Lean startups en el mundo real ejemplos y metricas
Lean startups en el mundo real  ejemplos y metricasLean startups en el mundo real  ejemplos y metricas
Lean startups en el mundo real ejemplos y metricas
 
Big Data LDN 2017: Advanced Analytics Applied to Marketing Attribution
Big Data LDN 2017: Advanced Analytics Applied to Marketing AttributionBig Data LDN 2017: Advanced Analytics Applied to Marketing Attribution
Big Data LDN 2017: Advanced Analytics Applied to Marketing Attribution
 
WEBINAR: How To Bring Native And Programmatic Advertising Together
WEBINAR: How To Bring Native And Programmatic Advertising TogetherWEBINAR: How To Bring Native And Programmatic Advertising Together
WEBINAR: How To Bring Native And Programmatic Advertising Together
 
UMCD 2011 Conference - Is Your Elephant Shy?
UMCD 2011 Conference - Is Your Elephant Shy?UMCD 2011 Conference - Is Your Elephant Shy?
UMCD 2011 Conference - Is Your Elephant Shy?
 
Auditing Agency Performance 110109x
Auditing Agency Performance 110109xAuditing Agency Performance 110109x
Auditing Agency Performance 110109x
 
New challenges for scalable machine learning in online advertising
New challenges for scalable machine learning in online advertisingNew challenges for scalable machine learning in online advertising
New challenges for scalable machine learning in online advertising
 
Ad Yield Optimization @ Spotify - DataGotham 2013
Ad Yield Optimization @ Spotify - DataGotham 2013Ad Yield Optimization @ Spotify - DataGotham 2013
Ad Yield Optimization @ Spotify - DataGotham 2013
 
Measurement module 100704
Measurement module 100704Measurement module 100704
Measurement module 100704
 
SalesFUSION Webinars - Taming the flood of digital data
SalesFUSION Webinars - Taming the flood of digital dataSalesFUSION Webinars - Taming the flood of digital data
SalesFUSION Webinars - Taming the flood of digital data
 
Self Service Online Research - online communities for research and insights
Self Service Online Research - online communities for research and insightsSelf Service Online Research - online communities for research and insights
Self Service Online Research - online communities for research and insights
 
Narrative Mind Week 8 H4D Stanford 2016
Narrative Mind Week 8 H4D Stanford 2016Narrative Mind Week 8 H4D Stanford 2016
Narrative Mind Week 8 H4D Stanford 2016
 
Estudio34 Presents- Dara Fitzgerald Brighton SEO-Next Gen Measurement With Go...
Estudio34 Presents- Dara Fitzgerald Brighton SEO-Next Gen Measurement With Go...Estudio34 Presents- Dara Fitzgerald Brighton SEO-Next Gen Measurement With Go...
Estudio34 Presents- Dara Fitzgerald Brighton SEO-Next Gen Measurement With Go...
 
JBP-Cobalt all services
JBP-Cobalt all servicesJBP-Cobalt all services
JBP-Cobalt all services
 
Collaborative visualization supporting complex data driven service build - Bi...
Collaborative visualization supporting complex data driven service build - Bi...Collaborative visualization supporting complex data driven service build - Bi...
Collaborative visualization supporting complex data driven service build - Bi...
 
Programmatic Advertising Solutions for Retailers and Brands
Programmatic Advertising Solutions for Retailers and BrandsProgrammatic Advertising Solutions for Retailers and Brands
Programmatic Advertising Solutions for Retailers and Brands
 
Lean Startup Metrics & Analytics
Lean Startup Metrics & AnalyticsLean Startup Metrics & Analytics
Lean Startup Metrics & Analytics
 
Harnessing the content beast – Content marketing in the multiscreen world
Harnessing the content beast – Content marketing in the multiscreen worldHarnessing the content beast – Content marketing in the multiscreen world
Harnessing the content beast – Content marketing in the multiscreen world
 
Archaic to Advanced in Akron
Archaic to Advanced in AkronArchaic to Advanced in Akron
Archaic to Advanced in Akron
 
Brent Chudoba Presentation
Brent Chudoba PresentationBrent Chudoba Presentation
Brent Chudoba Presentation
 
Ria Sankar - How to Build Winning Products - Product School Bellevue - 83018
Ria Sankar - How to Build Winning Products - Product School Bellevue - 83018 Ria Sankar - How to Build Winning Products - Product School Bellevue - 83018
Ria Sankar - How to Build Winning Products - Product School Bellevue - 83018
 

More from Webanalisten .nl

DDTT11: Hans Willemse - 21-01-20
DDTT11: Hans Willemse - 21-01-20DDTT11: Hans Willemse - 21-01-20
DDTT11: Hans Willemse - 21-01-20Webanalisten .nl
 
DDTT11: Ton Wesseling - 21-01-20
DDTT11: Ton Wesseling - 21-01-20DDTT11: Ton Wesseling - 21-01-20
DDTT11: Ton Wesseling - 21-01-20Webanalisten .nl
 
CH2019 keynote: Guy Yalif - AI and personalization demystified
CH2019 keynote: Guy Yalif - AI and personalization demystifiedCH2019 keynote: Guy Yalif - AI and personalization demystified
CH2019 keynote: Guy Yalif - AI and personalization demystifiedWebanalisten .nl
 
CH2019 keynote: Aurélie Pols - Data governance
CH2019 keynote: Aurélie Pols - Data governanceCH2019 keynote: Aurélie Pols - Data governance
CH2019 keynote: Aurélie Pols - Data governanceWebanalisten .nl
 
CH2019 keynote: Lucia van den Brink - Keynote pitch winner
CH2019 keynote: Lucia van den Brink - Keynote pitch winnerCH2019 keynote: Lucia van den Brink - Keynote pitch winner
CH2019 keynote: Lucia van den Brink - Keynote pitch winnerWebanalisten .nl
 
CH2019 keynote: Elke Geraerts - Authenthic intelligence
CH2019 keynote: Elke Geraerts - Authenthic intelligenceCH2019 keynote: Elke Geraerts - Authenthic intelligence
CH2019 keynote: Elke Geraerts - Authenthic intelligenceWebanalisten .nl
 
CH2019 keynote: Lukas Vermeer - One neat trick to run better experiments
CH2019 keynote: Lukas Vermeer - One neat trick to run better experimentsCH2019 keynote: Lukas Vermeer - One neat trick to run better experiments
CH2019 keynote: Lukas Vermeer - One neat trick to run better experimentsWebanalisten .nl
 
CH2019 keynote: Roger Dooley - Friction
CH2019 keynote: Roger Dooley - FrictionCH2019 keynote: Roger Dooley - Friction
CH2019 keynote: Roger Dooley - FrictionWebanalisten .nl
 
Conversion Hotel 2018 Keynote: Ton Wesseling
Conversion Hotel 2018 Keynote: Ton WesselingConversion Hotel 2018 Keynote: Ton Wesseling
Conversion Hotel 2018 Keynote: Ton WesselingWebanalisten .nl
 
Conversion Hotel 2018 Keynote: Nir Eyal
Conversion Hotel 2018 Keynote: Nir EyalConversion Hotel 2018 Keynote: Nir Eyal
Conversion Hotel 2018 Keynote: Nir EyalWebanalisten .nl
 
Conversion Hotel 2018 Keynote: Erin Weigel
Conversion Hotel 2018 Keynote: Erin WeigelConversion Hotel 2018 Keynote: Erin Weigel
Conversion Hotel 2018 Keynote: Erin WeigelWebanalisten .nl
 
Conversion Hotel 2018 Keynote: Lea Pica
Conversion Hotel 2018 Keynote: Lea PicaConversion Hotel 2018 Keynote: Lea Pica
Conversion Hotel 2018 Keynote: Lea PicaWebanalisten .nl
 
Conversion Hotel 2018 Keynote: Greg Shapiro
Conversion Hotel 2018 Keynote: Greg ShapiroConversion Hotel 2018 Keynote: Greg Shapiro
Conversion Hotel 2018 Keynote: Greg ShapiroWebanalisten .nl
 
Conversion Hotel 2018 Keynote: TU Delft Solar Team
Conversion Hotel 2018 Keynote: TU Delft Solar TeamConversion Hotel 2018 Keynote: TU Delft Solar Team
Conversion Hotel 2018 Keynote: TU Delft Solar TeamWebanalisten .nl
 
Conversion Hotel 2018 Keynote: Karl Gilis
Conversion Hotel 2018 Keynote: Karl GilisConversion Hotel 2018 Keynote: Karl Gilis
Conversion Hotel 2018 Keynote: Karl GilisWebanalisten .nl
 
Conversion Hotel 2018 Keynote: Lisa Pierson
Conversion Hotel 2018 Keynote: Lisa PiersonConversion Hotel 2018 Keynote: Lisa Pierson
Conversion Hotel 2018 Keynote: Lisa PiersonWebanalisten .nl
 
Conversion Hotel 2018 Keynote: Chad Sanderson
Conversion Hotel 2018 Keynote: Chad SandersonConversion Hotel 2018 Keynote: Chad Sanderson
Conversion Hotel 2018 Keynote: Chad SandersonWebanalisten .nl
 
Conversion Hotel 2018 Keynote: Aleksander Fabijan
Conversion Hotel 2018 Keynote: Aleksander FabijanConversion Hotel 2018 Keynote: Aleksander Fabijan
Conversion Hotel 2018 Keynote: Aleksander FabijanWebanalisten .nl
 
Conversion Hotel 2018 Keynote: Lizzie Eardley
Conversion Hotel 2018 Keynote: Lizzie EardleyConversion Hotel 2018 Keynote: Lizzie Eardley
Conversion Hotel 2018 Keynote: Lizzie EardleyWebanalisten .nl
 
Conversion Hotel 2018 Keynote: Michael Aagaard
Conversion Hotel 2018 Keynote: Michael AagaardConversion Hotel 2018 Keynote: Michael Aagaard
Conversion Hotel 2018 Keynote: Michael AagaardWebanalisten .nl
 

More from Webanalisten .nl (20)

DDTT11: Hans Willemse - 21-01-20
DDTT11: Hans Willemse - 21-01-20DDTT11: Hans Willemse - 21-01-20
DDTT11: Hans Willemse - 21-01-20
 
DDTT11: Ton Wesseling - 21-01-20
DDTT11: Ton Wesseling - 21-01-20DDTT11: Ton Wesseling - 21-01-20
DDTT11: Ton Wesseling - 21-01-20
 
CH2019 keynote: Guy Yalif - AI and personalization demystified
CH2019 keynote: Guy Yalif - AI and personalization demystifiedCH2019 keynote: Guy Yalif - AI and personalization demystified
CH2019 keynote: Guy Yalif - AI and personalization demystified
 
CH2019 keynote: Aurélie Pols - Data governance
CH2019 keynote: Aurélie Pols - Data governanceCH2019 keynote: Aurélie Pols - Data governance
CH2019 keynote: Aurélie Pols - Data governance
 
CH2019 keynote: Lucia van den Brink - Keynote pitch winner
CH2019 keynote: Lucia van den Brink - Keynote pitch winnerCH2019 keynote: Lucia van den Brink - Keynote pitch winner
CH2019 keynote: Lucia van den Brink - Keynote pitch winner
 
CH2019 keynote: Elke Geraerts - Authenthic intelligence
CH2019 keynote: Elke Geraerts - Authenthic intelligenceCH2019 keynote: Elke Geraerts - Authenthic intelligence
CH2019 keynote: Elke Geraerts - Authenthic intelligence
 
CH2019 keynote: Lukas Vermeer - One neat trick to run better experiments
CH2019 keynote: Lukas Vermeer - One neat trick to run better experimentsCH2019 keynote: Lukas Vermeer - One neat trick to run better experiments
CH2019 keynote: Lukas Vermeer - One neat trick to run better experiments
 
CH2019 keynote: Roger Dooley - Friction
CH2019 keynote: Roger Dooley - FrictionCH2019 keynote: Roger Dooley - Friction
CH2019 keynote: Roger Dooley - Friction
 
Conversion Hotel 2018 Keynote: Ton Wesseling
Conversion Hotel 2018 Keynote: Ton WesselingConversion Hotel 2018 Keynote: Ton Wesseling
Conversion Hotel 2018 Keynote: Ton Wesseling
 
Conversion Hotel 2018 Keynote: Nir Eyal
Conversion Hotel 2018 Keynote: Nir EyalConversion Hotel 2018 Keynote: Nir Eyal
Conversion Hotel 2018 Keynote: Nir Eyal
 
Conversion Hotel 2018 Keynote: Erin Weigel
Conversion Hotel 2018 Keynote: Erin WeigelConversion Hotel 2018 Keynote: Erin Weigel
Conversion Hotel 2018 Keynote: Erin Weigel
 
Conversion Hotel 2018 Keynote: Lea Pica
Conversion Hotel 2018 Keynote: Lea PicaConversion Hotel 2018 Keynote: Lea Pica
Conversion Hotel 2018 Keynote: Lea Pica
 
Conversion Hotel 2018 Keynote: Greg Shapiro
Conversion Hotel 2018 Keynote: Greg ShapiroConversion Hotel 2018 Keynote: Greg Shapiro
Conversion Hotel 2018 Keynote: Greg Shapiro
 
Conversion Hotel 2018 Keynote: TU Delft Solar Team
Conversion Hotel 2018 Keynote: TU Delft Solar TeamConversion Hotel 2018 Keynote: TU Delft Solar Team
Conversion Hotel 2018 Keynote: TU Delft Solar Team
 
Conversion Hotel 2018 Keynote: Karl Gilis
Conversion Hotel 2018 Keynote: Karl GilisConversion Hotel 2018 Keynote: Karl Gilis
Conversion Hotel 2018 Keynote: Karl Gilis
 
Conversion Hotel 2018 Keynote: Lisa Pierson
Conversion Hotel 2018 Keynote: Lisa PiersonConversion Hotel 2018 Keynote: Lisa Pierson
Conversion Hotel 2018 Keynote: Lisa Pierson
 
Conversion Hotel 2018 Keynote: Chad Sanderson
Conversion Hotel 2018 Keynote: Chad SandersonConversion Hotel 2018 Keynote: Chad Sanderson
Conversion Hotel 2018 Keynote: Chad Sanderson
 
Conversion Hotel 2018 Keynote: Aleksander Fabijan
Conversion Hotel 2018 Keynote: Aleksander FabijanConversion Hotel 2018 Keynote: Aleksander Fabijan
Conversion Hotel 2018 Keynote: Aleksander Fabijan
 
Conversion Hotel 2018 Keynote: Lizzie Eardley
Conversion Hotel 2018 Keynote: Lizzie EardleyConversion Hotel 2018 Keynote: Lizzie Eardley
Conversion Hotel 2018 Keynote: Lizzie Eardley
 
Conversion Hotel 2018 Keynote: Michael Aagaard
Conversion Hotel 2018 Keynote: Michael AagaardConversion Hotel 2018 Keynote: Michael Aagaard
Conversion Hotel 2018 Keynote: Michael Aagaard
 

Recently uploaded

The Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptx
The Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptxThe Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptx
The Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptxTasha Penwell
 
SWOT Analysis Slides Powerpoint Template.pptx
SWOT Analysis Slides Powerpoint Template.pptxSWOT Analysis Slides Powerpoint Template.pptx
SWOT Analysis Slides Powerpoint Template.pptxviniciusperissetr
 
办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree
办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree
办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degreeyuu sss
 
Decoding Patterns: Customer Churn Prediction Data Analysis Project
Decoding Patterns: Customer Churn Prediction Data Analysis ProjectDecoding Patterns: Customer Churn Prediction Data Analysis Project
Decoding Patterns: Customer Churn Prediction Data Analysis ProjectBoston Institute of Analytics
 
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Boston Institute of Analytics
 
Statistics, Data Analysis, and Decision Modeling, 5th edition by James R. Eva...
Statistics, Data Analysis, and Decision Modeling, 5th edition by James R. Eva...Statistics, Data Analysis, and Decision Modeling, 5th edition by James R. Eva...
Statistics, Data Analysis, and Decision Modeling, 5th edition by James R. Eva...ssuserf63bd7
 
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBoston Institute of Analytics
 
Predicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdfPredicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdfBoston Institute of Analytics
 
办美国加州大学伯克利分校毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
办美国加州大学伯克利分校毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree办美国加州大学伯克利分校毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
办美国加州大学伯克利分校毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degreeyuu sss
 
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming PipelinesConf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming PipelinesTimothy Spann
 
Identifying Appropriate Test Statistics Involving Population Mean
Identifying Appropriate Test Statistics Involving Population MeanIdentifying Appropriate Test Statistics Involving Population Mean
Identifying Appropriate Test Statistics Involving Population MeanMYRABACSAFRA2
 
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...Boston Institute of Analytics
 
Advanced Machine Learning for Business Professionals
Advanced Machine Learning for Business ProfessionalsAdvanced Machine Learning for Business Professionals
Advanced Machine Learning for Business ProfessionalsVICTOR MAESTRE RAMIREZ
 
Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...
Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...
Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...Thomas Poetter
 
NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...
NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...
NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...Amil Baba Dawood bangali
 
RABBIT: A CLI tool for identifying bots based on their GitHub events.
RABBIT: A CLI tool for identifying bots based on their GitHub events.RABBIT: A CLI tool for identifying bots based on their GitHub events.
RABBIT: A CLI tool for identifying bots based on their GitHub events.natarajan8993
 
Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Cathrine Wilhelmsen
 
Defining Constituents, Data Vizzes and Telling a Data Story
Defining Constituents, Data Vizzes and Telling a Data StoryDefining Constituents, Data Vizzes and Telling a Data Story
Defining Constituents, Data Vizzes and Telling a Data StoryJeremy Anderson
 
Biometric Authentication: The Evolution, Applications, Benefits and Challenge...
Biometric Authentication: The Evolution, Applications, Benefits and Challenge...Biometric Authentication: The Evolution, Applications, Benefits and Challenge...
Biometric Authentication: The Evolution, Applications, Benefits and Challenge...GQ Research
 

Recently uploaded (20)

The Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptx
The Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptxThe Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptx
The Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptx
 
SWOT Analysis Slides Powerpoint Template.pptx
SWOT Analysis Slides Powerpoint Template.pptxSWOT Analysis Slides Powerpoint Template.pptx
SWOT Analysis Slides Powerpoint Template.pptx
 
办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree
办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree
办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree
 
Decoding Patterns: Customer Churn Prediction Data Analysis Project
Decoding Patterns: Customer Churn Prediction Data Analysis ProjectDecoding Patterns: Customer Churn Prediction Data Analysis Project
Decoding Patterns: Customer Churn Prediction Data Analysis Project
 
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
 
Insurance Churn Prediction Data Analysis Project
Insurance Churn Prediction Data Analysis ProjectInsurance Churn Prediction Data Analysis Project
Insurance Churn Prediction Data Analysis Project
 
Statistics, Data Analysis, and Decision Modeling, 5th edition by James R. Eva...
Statistics, Data Analysis, and Decision Modeling, 5th edition by James R. Eva...Statistics, Data Analysis, and Decision Modeling, 5th edition by James R. Eva...
Statistics, Data Analysis, and Decision Modeling, 5th edition by James R. Eva...
 
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
 
Predicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdfPredicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdf
 
办美国加州大学伯克利分校毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
办美国加州大学伯克利分校毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree办美国加州大学伯克利分校毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
办美国加州大学伯克利分校毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
 
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming PipelinesConf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
 
Identifying Appropriate Test Statistics Involving Population Mean
Identifying Appropriate Test Statistics Involving Population MeanIdentifying Appropriate Test Statistics Involving Population Mean
Identifying Appropriate Test Statistics Involving Population Mean
 
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
 
Advanced Machine Learning for Business Professionals
Advanced Machine Learning for Business ProfessionalsAdvanced Machine Learning for Business Professionals
Advanced Machine Learning for Business Professionals
 
Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...
Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...
Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...
 
NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...
NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...
NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...
 
RABBIT: A CLI tool for identifying bots based on their GitHub events.
RABBIT: A CLI tool for identifying bots based on their GitHub events.RABBIT: A CLI tool for identifying bots based on their GitHub events.
RABBIT: A CLI tool for identifying bots based on their GitHub events.
 
Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)
 
Defining Constituents, Data Vizzes and Telling a Data Story
Defining Constituents, Data Vizzes and Telling a Data StoryDefining Constituents, Data Vizzes and Telling a Data Story
Defining Constituents, Data Vizzes and Telling a Data Story
 
Biometric Authentication: The Evolution, Applications, Benefits and Challenge...
Biometric Authentication: The Evolution, Applications, Benefits and Challenge...Biometric Authentication: The Evolution, Applications, Benefits and Challenge...
Biometric Authentication: The Evolution, Applications, Benefits and Challenge...
 

Gui Liberali, Morphing Websites - Digital Data Tips Tuesday #4 - Growth through Data - Amsterdam - May 19th 2015 - #DDTT

  • 1. Morphing Theory and Applications to Online Experiments: Learning While Earning DDTT 2015 Gui Liberali Erasmus University Laboratory for Optimization of Digital Experiments @ Erasmus
  • 3. One size does not fit all (a) General content, large-load, graphical morph (b) Focused, small-load, verbal morph Visual Technical Content More Content Audio Less Content General Content Website designs as Morphs Appeals to Verbal Cognitive Style Appeals to Visual Cognitive Style
  • 4. Appeals to Holistic, impulsive cognitive style Appeals to Analytical, deliberative cognitive style One size does not fit all Banners as Morphs
  • 5. •  From learning-then-earning to the world of learning-while-earning •  Higher CTR, conversions •  Better user experience Where’s your A/B Testing ? •  From aggregate to disaggregate data •  From demographics to cognitive styles
  • 6. 1. Learning about Styles vs Morphs Firm  Side:  Banner  Ads       Buy!   versus   Learn   More   Inform a4ve   Emo4o nal     Time-­‐ pressu re                Consumer   Side:                             4  Cogni4ve   Styles   Verbal-­‐Impulsive   0.1   0.11   0.2   0.13   Visual-­‐Impulsive   0.11   0.2   0.13   0.1   Verbal-­‐Delibera4ve   0.2   0.13   0.1   0.11   Visual-­‐Delibera4ve   0.11   0.2   0.13   0.1  
  • 7. Optimization: from Earn to Learn • Maximize conversion by picking the best among ads for current user • If we knew the conversion probabilities perfectly we would simply serve the optimal morph • If we knew nothing about the conversion probabilities we would randomize and learn Need to balance learning the true cognitive style (exploitation) and providing the optimal morph given what we know now (exploration). For each person!
  • 8. Morphing: real-time adaptation of the firm’s actions to each consumer Cognitive-style, segment, position in the purchase funnel, .. Website design, banner, advertising campaign, media copy per channel, promotion, price, product bundle, product recommendation … Consumer latent variable Marketing instrument to be adapted (morphed) x
  • 9. Morphing - Evidences from the Field Morphing Banner Advertising (2014) •  Morphing almost doubled click-through rates for context- matched banners (83%, p<0.01 and 97%, p=0.028) lifts, respectively, for banners and for consumers •  Was live on CNET.com (8.4M visitors/day) for a month •  Extends behavioral targeting to avoid local max solutions Website Morphing (2009) •  Online sales of BT group’s broadband plans can be increased up to 20% by adapting the website to match cognitive styles using our adaptive learning algorithm Website Morphing 2.0 (2014) •  NPV reward improvement of 69% over the NPV of the original algorithm; corresponds to $17.5 million for BT group if implemented •  Proof-of-Feasibility on Suruga Bank •  Generalized version of morphing, including when to morph, switching costs, multiple morphs
  • 10. Morphing Algorithm Challenges Addressed 1.  What is the style of the current user? –  Update our beliefs as new clicks become available 2.  What is the optimal morph for each style? –  Update as we observe successes and failures
  • 11. Morphing   Choice  descriptor   Priors  on  distribu5on  of   styles   Updated  user   style  descriptor   On-­‐line   Op4miza4on   Pgm:  probability  of   purchase  for  individual  in   style  g  given  morph  m     Banner  Ads  -­‐>   1   2   3   4             Cogni4ve   Style   Impulsive   Delibera4ve   0.1   0.11   0.2   0.13   Illustra5ve  example   User   Op5mal  morph  for  current    user   Website     User  makes  a   purchase  or   quit   Bayesian   update  
  • 12. Optimal solution with Gittins’ indices (assume we know cognitive style) Gittins’ indices for the eight morphs. Morph that was chosen. 0 1 2 3 4 5 6 7 8 0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000 Visitor ChosenMorph 0.35 0.45 0.55 0.65 0.75 0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000 Visitor Gittins'Indices Morph 0 Morph 1 Morph 2 Morph 3 Morph 4 Morph 5 Morph 6 Morph 7 System experiments with Morph 3 for a while before settling back to Morph 2.
  • 13. •  What is the cognitive style of the current user? –  Update our beliefs as new clicks become available •  What is the optimal morph for each style? –  Update as we observe successes and failures Website Morphing (2009) •  How to handle repeat visits and cumulative learning ? –  Field experiment on cnet.com w/ AT&T banner ads –  Roll-back mechanism Morphing Banner Advertising (2014) •  Generalized Morphing Algorithm: w/ When to Morph –  DP solves trade-off between gains from morphing early (longer exposure) and gains from morphing later (better knowledge) –  Exposure to multiple morphs •  Attribution problem: recency, exposure, both •  Switching costs: changing morphs only if expected rewards exceed penalty for switching Website Morphing 2.0 (2014) Morphing Trilogy
  • 14. Morphing in the Field •  Can be used normatively (e.g., field experiments) or descriptively (secondary data) •  Two tuning parameters makes it flexible to various applications –  Switching costs –  Different attribution theories/regimes •  Morphing is appropriate to contexts with –  High volume of observations –  Clear success/failure outcome –  Latent variable
  • 15. Laboratory on Optimization of Digital Experiments (LODE) @ Erasmus University •  Executive education course on Digital Experimentation and Big Data •  Research Fridays: Bi-monthly meetings of academics and founding companies at Erasmus University •  Bi-yearly publication of research priorities for research on A/B testing •  Erasmus students running A/B testing & morphing with real-world company traffic –  Module 1: Text-analysis methods –  Module 2: A/B experiments –  Module 3: Morphing
  • 16. References •  Hauser JR, Urban GL, Liberali G, Braun M (2009) Website morphing. Marketing Sci. 28(2):202–224. •  Urban G., Liberali G., MacDonald E., Bordley R., Hauser J. (2014) Morphing Banner Advertising, Marketing Science, 33(1): 27-46. •  Hauser J., Liberali G., Urban G. (Summer 2014) Website Morphing 2.0: Switching Costs, Partial Exposure, Random Exit, and When to Morph. Management Science.