Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
opticon2017
Decisions at Scale:
Empower your Team to do
More with Experiment Data
Giannis Psaroudakis
Product Manager, Opt...
Bing had 8 years
of consecutive
market share
growth
Source: Comscore
“The growth of experimentation is the major reason Bing is
profitable and its share of U.S desktop searches nearly tripled...
Old Reality Culture of
Top-Down Innovation
Embrace Success
Make Decisions
Follow Orders
Bottom-Up Innovation
Embrace Failu...
Experimentation Hero
10’s experiments / year
Experimentation Program
100’s experiments / year
Culture of Experimentation
1...
Optimizely Data Platform
Mission
Make Customer Data Useful Through Experimentation
ResultsEvents Stats EngineAudiences Metrics
Process signals
across customer
experiences
Target user groups
with common
att...
Meet Kristen
Director of
Experimentation
Empower every other team in her organization
to make validated decisions with high velocity.
Center of Excellence
Support
Sales Marketing
E-commerce
Engineering
Center of Excellence
Support
Sales Marketing
E-commerce
?
? ?
?
?
Engineering
Can't leverage the results Experimentation is not useful
Can’t measure what matters to them Reduced adoption of the progra...
“I want to support the measurement
needs of all teams across the
organization”
1. Can’t measure what matters
3. No confide...
E-commerce
Product Manager
“I want to measure the impact of
changing our e-commerce experience
on our revenue metrics”
Sales Leads
Manager
“I want to measure the impact of our
online customer outreach campaigns on
offline conversions, such a...
App Performance
Engineer
“I want to measure the impact of our
code changes on the application’s
page load time”
Social Media
Marketing Manager
“I want to increase the impact of our
social media campaigns by reducing
the bounce rate on...
Sources
E-Commerce
Sales Leads
Engineering
Social Media
Metrics
Web/Mobile
Internal System
Server
Web/Mobile
Revenue
Offli...
Sources
X Event API
Bring your own signals into
experimentation no matter where
they live.
Web, Mobile
& Full-Stack
Offlin...
Optimizely
Business
Intelligence
System
RPV: $145
LTV: $2145
ticket_count: +1
Call
Center
Sources
Metrics Builder
Measure the metrics that matter most to your
business with greater precision and flexibility.
Metrics
! In...
Bounce & Exit Rage
Metrics
Conversions
Revenue
Custom Value
Measure distinct user actions
Measure any numerical goal
check...
Teams can measure what matters to them
• Identify the signals and metrics teams need
• Incorporate them into experimentati...
“I want teams to have flexibility in
the way they analyze the results”
1. Can’t measure what matters
3. No confidence in t...
Analysis Method
E-Commerce
Sales Leads
Engineering
Social Media
Results Page
Analytics Software
Raw Data
Spreadsheet
Analysis Methods
Results Export
Expand your analysis workflow with a
variety of new and improved methods
to export your re...
Analysis Methods
GET /v2/experiments/{experiment_id}/results
...
"metrics": [
{
"name":"items in cart"
"results": {
"12345...
Custom StatsDashboards Debugging
Analysis Methods
Data Export
Custom ETL Pipeline
Develop your own custom integration to bring
experimentation results into your analytics tool.
Analysis Methods
+
Custom
I...
1. Can’t measure what matters
3. No confidence in the results
2. Can't leverage the results Teams can analyze the results
...
“I want tools our teams can trust
and have faith that the results will
manifest”
1. Can’t measure what matters
3. No confi...
1. False Discoveries
2. Skewed Results (Outliers / Bots)
3. Data Discrepancies
Top reasons teams lose
confidence on experi...
1.False Discoveries
2. Skewed Results (Outliers / Bots)
3. Data Discrepancies
Top reasons teams lose
confidence on experim...
Results Confidence
Stats Engine
Validate decisions with statistical confidence
! Error control for multiple hypothesis tes...
1. False Discoveries
2.Skewed Results (Outliers / Bots)
3. Data Discrepancies
Top reasons teams lose
confidence on experim...
Income per Person
$50-70K Billions!
Average Order
$500
Revenue
per Order ($)
Results Confidence
Revenue
per Order ($)
Results Confidence
Extreme Orders!
Average Order
$500
Revenue
per Order ($)
Results Confidence
Outlier Threshold
Average Order
$500
Extreme Orders!
Outlier Filtering
Improve the integrity of your decisions by
filtering outliers from the results.*
! Enabled on demand
! C...
Bot Filtering
Enhanced Bot Filtering to improve the
fidelity of the results
! Enhanced bot filtering that complies to indu...
1. False Discoveries
2. Skewed Results (Bots / Outliers)
3.Data Discrepancies
Top reasons teams lose
confidence on experim...
Numbers don't match!
1. Product Differences
2. Implementation Bugs
3.Event Timing Issues
Leading factors
of analytics
disc...
visitor counter: +1
First Optimizely event
First 3rd-party analytics event
Optimizely
visitor counter: +1
Other Analytics
...
visitor counter: +1
First Optimizely event
First 3rd-party analytics event
Optimizely
visitor counter: +1
Other Analytics
...
Hold/SendEvents API
Mitigate discrepancies between
Optimizely and your web analytics.
window.optimizely.push({type: "holdE...
visitor counter: +1
First 3rd-party analytics event Optimizely &
Other AnalyticsFirst Optimizely event
Results Confidence
visitor counter: +1
First 3rd-party analytics event Optimizely &
Other AnalyticsFirst Optimizely event
reduction in discre...
1. Can’t measure what matters
3. No confidence in the results
2. Need help to analyze experiments
Teams have confidence in...
Decisions at Scale
!Grow adoption by enabling teams to measure what matters to them
!Grow adoption by enabling teams to an...
Thank you!
Opticon 2017 Decisions at Scale
Opticon 2017 Decisions at Scale
Opticon 2017 Decisions at Scale
Opticon 2017 Decisions at Scale
Opticon 2017 Decisions at Scale
Opticon 2017 Decisions at Scale
Upcoming SlideShare
Loading in …5
×

Opticon 2017 Decisions at Scale

409 views

Published on

Decisions at Scale: Empower Your Team to do More with Experiment Data

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Opticon 2017 Decisions at Scale

  1. 1. opticon2017 Decisions at Scale: Empower your Team to do More with Experiment Data Giannis Psaroudakis Product Manager, Optimizely
  2. 2. Bing had 8 years of consecutive market share growth Source: Comscore
  3. 3. “The growth of experimentation is the major reason Bing is profitable and its share of U.S desktop searches nearly tripled.” Ronny Kohavi, GM of Analysis and Experimentation, Microsoft Source: Comscore
  4. 4. Old Reality Culture of Top-Down Innovation Embrace Success Make Decisions Follow Orders Bottom-Up Innovation Embrace Failure Validate Decisions Follow Data
  5. 5. Experimentation Hero 10’s experiments / year Experimentation Program 100’s experiments / year Culture of Experimentation 1000’s experiments / year Scaling the Experimentation Program
  6. 6. Optimizely Data Platform
  7. 7. Mission Make Customer Data Useful Through Experimentation
  8. 8. ResultsEvents Stats EngineAudiences Metrics Process signals across customer experiences Target user groups with common attributes Measure the impact of hypotheses Understand the impact of hypotheses Validate hypotheses with statistical confidence 6.5+Billion Events Daily 1.2 + Billion Experiences Daily 50+Million Unique Users Daily
  9. 9. Meet Kristen
  10. 10. Director of Experimentation
  11. 11. Empower every other team in her organization to make validated decisions with high velocity.
  12. 12. Center of Excellence Support Sales Marketing E-commerce Engineering
  13. 13. Center of Excellence Support Sales Marketing E-commerce ? ? ? ? ? Engineering
  14. 14. Can't leverage the results Experimentation is not useful Can’t measure what matters to them Reduced adoption of the program Don't have confidence in the results Losing trust in experimentation
  15. 15. “I want to support the measurement needs of all teams across the organization” 1. Can’t measure what matters 3. No confidence in the results 2. Can't leverage the results
  16. 16. E-commerce Product Manager “I want to measure the impact of changing our e-commerce experience on our revenue metrics”
  17. 17. Sales Leads Manager “I want to measure the impact of our online customer outreach campaigns on offline conversions, such as phone calls”
  18. 18. App Performance Engineer “I want to measure the impact of our code changes on the application’s page load time”
  19. 19. Social Media Marketing Manager “I want to increase the impact of our social media campaigns by reducing the bounce rate on landing pages”
  20. 20. Sources E-Commerce Sales Leads Engineering Social Media Metrics Web/Mobile Internal System Server Web/Mobile Revenue Offline Conversions Page Load Time Bounce Rate
  21. 21. Sources X Event API Bring your own signals into experimentation no matter where they live. Web, Mobile & Full-Stack Offline Conversions Business Intelligence Secure Environments Event APISnippet / SDKs Optimizely X
  22. 22. Optimizely Business Intelligence System RPV: $145 LTV: $2145 ticket_count: +1 Call Center Sources
  23. 23. Metrics Builder Measure the metrics that matter most to your business with greater precision and flexibility. Metrics ! Intuitive interface ! New metric types ! Metrics calculated in real-time
  24. 24. Bounce & Exit Rage Metrics Conversions Revenue Custom Value Measure distinct user actions Measure any numerical goal checkouts per visitor, page views per visitor Measure transaction amounts revenue per visitor (RPV), average order value (AOV) Session Duration Measure engagement duration avg. time spent on news article pages, avg. time spent on site Measure user abandonment page load time per visitor, purchased items per order bounce rate on homepage, exit rate on checkout page Metric Description Examples (2018) (in beta November 2017)
  25. 25. Teams can measure what matters to them • Identify the signals and metrics teams need • Incorporate them into experimentation • Scale by enabling everyone to measure what matters to them 1. Can’t measure what matters 3. No confidence in the results 2. Can't leverage the results
  26. 26. “I want teams to have flexibility in the way they analyze the results” 1. Can’t measure what matters 3. No confidence in the results 2. Can't leverage the results
  27. 27. Analysis Method E-Commerce Sales Leads Engineering Social Media Results Page Analytics Software Raw Data Spreadsheet
  28. 28. Analysis Methods Results Export Expand your analysis workflow with a variety of new and improved methods to export your results. Results API CSV Export Raw Data Export Analytics Integration
  29. 29. Analysis Methods GET /v2/experiments/{experiment_id}/results ... "metrics": [ { "name":"items in cart" "results": { "123456": { "is_baseline":false, "lift": { "confidence_interval": [ 1.152505, 2.612566 ], "is_significant":true, "significance":0.9885, "value":0.44 "visitors_remaining":2601 } } ... } Your Application Custo m Alerts Automated Actions Aggregated Reporting Results API
  30. 30. Custom StatsDashboards Debugging Analysis Methods Data Export Custom ETL Pipeline
  31. 31. Develop your own custom integration to bring experimentation results into your analytics tool. Analysis Methods + Custom Integration Framework
  32. 32. 1. Can’t measure what matters 3. No confidence in the results 2. Can't leverage the results Teams can analyze the results • Understand how teams are doing analysis • Connect experiment data with their workflows • Scale by enabling everyone to analyze
  33. 33. “I want tools our teams can trust and have faith that the results will manifest” 1. Can’t measure what matters 3. No confidence in the results 2. Need help to analyze experiments
  34. 34. 1. False Discoveries 2. Skewed Results (Outliers / Bots) 3. Data Discrepancies Top reasons teams lose confidence on experimentation?!
  35. 35. 1.False Discoveries 2. Skewed Results (Outliers / Bots) 3. Data Discrepancies Top reasons teams lose confidence on experimentation?!
  36. 36. Results Confidence Stats Engine Validate decisions with statistical confidence ! Error control for multiple hypothesis testing ! Prevents bad decisions due to ‘peeking’ ! Configurable risk tolerance
  37. 37. 1. False Discoveries 2.Skewed Results (Outliers / Bots) 3. Data Discrepancies Top reasons teams lose confidence on experimentation
  38. 38. Income per Person $50-70K Billions!
  39. 39. Average Order $500 Revenue per Order ($) Results Confidence
  40. 40. Revenue per Order ($) Results Confidence Extreme Orders! Average Order $500
  41. 41. Revenue per Order ($) Results Confidence Outlier Threshold Average Order $500 Extreme Orders!
  42. 42. Outlier Filtering Improve the integrity of your decisions by filtering outliers from the results.* ! Enabled on demand ! Calculated in real-time ! Configurable thresholds Results Confidence *Availability: Coming in beta November 2017. Compatible with revenue and custom value metrics.
  43. 43. Bot Filtering Enhanced Bot Filtering to improve the fidelity of the results ! Enhanced bot filtering that complies to industry standard (IAB/ABC List) for web analytics ! Bots automatically filtered from the results OPTIMIZELY Results Confidence
  44. 44. 1. False Discoveries 2. Skewed Results (Bots / Outliers) 3.Data Discrepancies Top reasons teams lose confidence on experimentation
  45. 45. Numbers don't match! 1. Product Differences 2. Implementation Bugs 3.Event Timing Issues Leading factors of analytics discrepancies
  46. 46. visitor counter: +1 First Optimizely event First 3rd-party analytics event Optimizely visitor counter: +1 Other Analytics Results Confidence
  47. 47. visitor counter: +1 First Optimizely event First 3rd-party analytics event Optimizely visitor counter: +1 Other Analytics +2 +1 Results Confidence
  48. 48. Hold/SendEvents API Mitigate discrepancies between Optimizely and your web analytics. window.optimizely.push({type: "holdEvents"}); window.optimizely.push({type: "sendEvents"}); HoldEvents: Instruct Optimizely to hold the events in a browser queue. SendEvents: Instruct Optimizely to release the events from the browser queue. Results Confidence
  49. 49. visitor counter: +1 First 3rd-party analytics event Optimizely & Other AnalyticsFirst Optimizely event Results Confidence
  50. 50. visitor counter: +1 First 3rd-party analytics event Optimizely & Other AnalyticsFirst Optimizely event reduction in discrepancies related to event timing 90% up to Results Confidence
  51. 51. 1. Can’t measure what matters 3. No confidence in the results 2. Need help to analyze experiments Teams have confidence in the results • Identify sources of bias or discrepancies • Filter them out from the results • Scale by enabling everyone to analyze the results with confidence
  52. 52. Decisions at Scale !Grow adoption by enabling teams to measure what matters to them !Grow adoption by enabling teams to analyze using their own workflows !Maintain adoption by giving teams confidence on the results
  53. 53. Thank you!

×