SlideShare a Scribd company logo
1 of 10
Optimizely Meetup
November 4, 2013
Introduction
Control
Variation #1
Unrecognizable Test
Unrecognizable A/B Test Results
Unrecognizable A/B Test Results
4% Reduction in Overall Creative Production
Spend Annually!
Why Does This Matter?
Savings Call Out Test
Lessons Learned
1. Test Significant Differences
2. Document and Socialize Results
3. Establish and Meet Regularly with a Core Group

More Related Content

More from Gilt Tech Talks

iOS Testing With Appium at Gilt
iOS Testing With Appium at GiltiOS Testing With Appium at Gilt
iOS Testing With Appium at GiltGilt Tech Talks
 
Handling Changes to Your Server-Side Data Model
Handling Changes to Your Server-Side Data ModelHandling Changes to Your Server-Side Data Model
Handling Changes to Your Server-Side Data ModelGilt Tech Talks
 
Beyond the Crystal Ball: The Agile PMO
Beyond the Crystal Ball: The Agile PMOBeyond the Crystal Ball: The Agile PMO
Beyond the Crystal Ball: The Agile PMOGilt Tech Talks
 
Exploring Docker at Gilt
Exploring Docker at GiltExploring Docker at Gilt
Exploring Docker at GiltGilt Tech Talks
 
Scaling Gilt: from monolith ruby app to micro service scala service architecture
Scaling Gilt: from monolith ruby app to micro service scala service architectureScaling Gilt: from monolith ruby app to micro service scala service architecture
Scaling Gilt: from monolith ruby app to micro service scala service architectureGilt Tech Talks
 
PostgreSQL Setup Using Docker
PostgreSQL Setup Using DockerPostgreSQL Setup Using Docker
PostgreSQL Setup Using DockerGilt Tech Talks
 
Roadmap for the Trillion Sensor Universe -- a Gilt-hosted, Internet of Things...
Roadmap for the Trillion Sensor Universe -- a Gilt-hosted, Internet of Things...Roadmap for the Trillion Sensor Universe -- a Gilt-hosted, Internet of Things...
Roadmap for the Trillion Sensor Universe -- a Gilt-hosted, Internet of Things...Gilt Tech Talks
 
Scala Self Types by Gregor Heine, Principal Software Engineer at Gilt
Scala Self Types by Gregor Heine, Principal Software Engineer at GiltScala Self Types by Gregor Heine, Principal Software Engineer at Gilt
Scala Self Types by Gregor Heine, Principal Software Engineer at GiltGilt Tech Talks
 

More from Gilt Tech Talks (11)

An Intro to Swift
An Intro to SwiftAn Intro to Swift
An Intro to Swift
 
iOS Testing With Appium at Gilt
iOS Testing With Appium at GiltiOS Testing With Appium at Gilt
iOS Testing With Appium at Gilt
 
Handling Changes to Your Server-Side Data Model
Handling Changes to Your Server-Side Data ModelHandling Changes to Your Server-Side Data Model
Handling Changes to Your Server-Side Data Model
 
Mobile Testing at Gilt
Mobile Testing at GiltMobile Testing at Gilt
Mobile Testing at Gilt
 
Beyond the Crystal Ball: The Agile PMO
Beyond the Crystal Ball: The Agile PMOBeyond the Crystal Ball: The Agile PMO
Beyond the Crystal Ball: The Agile PMO
 
Exploring Docker at Gilt
Exploring Docker at GiltExploring Docker at Gilt
Exploring Docker at Gilt
 
Scaling Gilt: from monolith ruby app to micro service scala service architecture
Scaling Gilt: from monolith ruby app to micro service scala service architectureScaling Gilt: from monolith ruby app to micro service scala service architecture
Scaling Gilt: from monolith ruby app to micro service scala service architecture
 
PostgreSQL Setup Using Docker
PostgreSQL Setup Using DockerPostgreSQL Setup Using Docker
PostgreSQL Setup Using Docker
 
Roadmap for the Trillion Sensor Universe -- a Gilt-hosted, Internet of Things...
Roadmap for the Trillion Sensor Universe -- a Gilt-hosted, Internet of Things...Roadmap for the Trillion Sensor Universe -- a Gilt-hosted, Internet of Things...
Roadmap for the Trillion Sensor Universe -- a Gilt-hosted, Internet of Things...
 
Scala Self Types by Gregor Heine, Principal Software Engineer at Gilt
Scala Self Types by Gregor Heine, Principal Software Engineer at GiltScala Self Types by Gregor Heine, Principal Software Engineer at Gilt
Scala Self Types by Gregor Heine, Principal Software Engineer at Gilt
 
Virtualization at Gilt
Virtualization at GiltVirtualization at Gilt
Virtualization at Gilt
 

Recently uploaded

Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Mattias Andersson
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University
 
Streamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupStreamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupFlorian Wilhelm
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):comworks
 
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 3652toLead Limited
 
Pigging Solutions Piggable Sweeping Elbows
Pigging Solutions Piggable Sweeping ElbowsPigging Solutions Piggable Sweeping Elbows
Pigging Solutions Piggable Sweeping ElbowsPigging Solutions
 
Science&tech:THE INFORMATION AGE STS.pdf
Science&tech:THE INFORMATION AGE STS.pdfScience&tech:THE INFORMATION AGE STS.pdf
Science&tech:THE INFORMATION AGE STS.pdfjimielynbastida
 
Transcript: New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024BookNet Canada
 
Bluetooth Controlled Car with Arduino.pdf
Bluetooth Controlled Car with Arduino.pdfBluetooth Controlled Car with Arduino.pdf
Bluetooth Controlled Car with Arduino.pdfngoud9212
 
Build your next Gen AI Breakthrough - April 2024
Build your next Gen AI Breakthrough - April 2024Build your next Gen AI Breakthrough - April 2024
Build your next Gen AI Breakthrough - April 2024Neo4j
 
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...shyamraj55
 
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationRidwan Fadjar
 
Unblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesUnblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesSinan KOZAK
 
Benefits Of Flutter Compared To Other Frameworks
Benefits Of Flutter Compared To Other FrameworksBenefits Of Flutter Compared To Other Frameworks
Benefits Of Flutter Compared To Other FrameworksSoftradix Technologies
 
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Wonjun Hwang
 
My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024The Digital Insurer
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst
 

Recently uploaded (20)

Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
 
Streamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupStreamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project Setup
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):
 
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
 
Pigging Solutions Piggable Sweeping Elbows
Pigging Solutions Piggable Sweeping ElbowsPigging Solutions Piggable Sweeping Elbows
Pigging Solutions Piggable Sweeping Elbows
 
Science&tech:THE INFORMATION AGE STS.pdf
Science&tech:THE INFORMATION AGE STS.pdfScience&tech:THE INFORMATION AGE STS.pdf
Science&tech:THE INFORMATION AGE STS.pdf
 
Transcript: New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
 
Bluetooth Controlled Car with Arduino.pdf
Bluetooth Controlled Car with Arduino.pdfBluetooth Controlled Car with Arduino.pdf
Bluetooth Controlled Car with Arduino.pdf
 
Build your next Gen AI Breakthrough - April 2024
Build your next Gen AI Breakthrough - April 2024Build your next Gen AI Breakthrough - April 2024
Build your next Gen AI Breakthrough - April 2024
 
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
 
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 Presentation
 
Hot Sexy call girls in Panjabi Bagh 🔝 9953056974 🔝 Delhi escort Service
Hot Sexy call girls in Panjabi Bagh 🔝 9953056974 🔝 Delhi escort ServiceHot Sexy call girls in Panjabi Bagh 🔝 9953056974 🔝 Delhi escort Service
Hot Sexy call girls in Panjabi Bagh 🔝 9953056974 🔝 Delhi escort Service
 
Unblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesUnblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen Frames
 
Benefits Of Flutter Compared To Other Frameworks
Benefits Of Flutter Compared To Other FrameworksBenefits Of Flutter Compared To Other Frameworks
Benefits Of Flutter Compared To Other Frameworks
 
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
 
My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR Systems
 

Optimizely at Gilt--November 2013 presentation