Under the Hood: Experiment-Driven Product Design
•
3 likes•483 views
1. The webinar covered optimizing product design through experiment-driven approaches. It discussed AB testing, the experiment design canvas tool, and tips for integrating experimentation into the design process. 2. The experiment design canvas is used to map business objectives and key results to user goals, problems, solutions, and results from a micro and macro level. 3. An example was provided of an experiment to address a page load error by testing different troubleshooting option displays, which resulted in a 63% winner that reduced errors.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Under the Hood: Experiment-Driven Product Design
Similar to Under the Hood: Experiment-Driven Product Design (20)
More from Optimizely
More from Optimizely (20)
Recently uploaded
Recently uploaded (20)
Under the Hood: Experiment-Driven Product Design
- 1. 1 WEBINAR SERIES Optimizely Under the Hood Experiment-Driven Product Design
- 2. 2 Meet the designers Zach Leach Staff Product Designer Optimizely Shane Fontane Staff Product Designer Optimizely
- 3. Housekeeping ● This session is being recorded ● You will receive a copy of the recording and slides ● We’ll answer questions at the end
- 4. Today... ● What’s an AB Test? ● The Experiment Design Canvas ● Getting Things Done! ● FAQ ● QA
- 6. 6 Experimentation: A Simple Example
- 7. 7 What’s an AB Test?
- 8. 8 Click Me
- 9. 9 Click Me Click Me
- 10. 10 Click Me Click Me 50% of Traffic 50% of Traffic
- 11. 11 Click Me Click Me 50% of Traffic 50% of Traffic 1.2% of Visitors Clicked 3.0% of Visitors Clicked
- 12. 12 Experimentation is a powerful tool.
- 13. 13 Experimentation is a powerful tool. It’s even more powerful when it’s built into your process.
- 15. 15 Experiment Design Canvas Business Objective Business Outcome 1 2 User User Goal Problem Solution Result 3 4 5 6 7 MICROMACRO
- 16. 16 Experiment Design Canvas Business Objective Business Outcome 1 2 User User Goal Problem Solution Result 3 4 5 6 7 MICROMACRO
- 17. 17 Experiment Design Canvas Business Objective Business Outcome 1 2 User User Goal Problem Solution Result 3 4 5 6 7 MICROMACRO
- 18. 18 Experiment Design Canvas Business Objective Business Outcome 1 2 User User Goal Problem Solution Result 3 4 5 6 7 MICROMACRO
- 19. 19 Experiment Design Canvas Business Objective Business Outcome 1 2 User User Goal Problem Solution Result 3 4 5 6 7 MICROMACRO
- 20. 20 Experiment Design Canvas Business Objective Business Outcome 1 2 User User Goal Problem Solution Result 3 4 5 6 7 MICROMACRO
- 21. 21 Experiment Design Canvas Business Objective Business Outcome 1 2 User User Goal Problem Solution Result 3 4 5 6 7 MICROMACRO
- 22. 22 Experiment Design Canvas Business Objective Business Outcome 1 2 User User Goal Problem Solution Result 3 4 5 6 7 MICROMACRO
- 23. 23 Experiment Design Canvas Business Objective Business Outcome 1 2 User User Goal Problem Solution Result 3 4 5 6 7 MICROMACRO
- 24. 24 Experiment Design Canvas Business Objective Business Outcome 1 2 User User Goal Problem Solution Result 3 4 5 6 7 MICROMACRO
- 25. 25 Experiment Design Canvas Business Objective Business Outcome 1 2 User User Goal Problem Solution Result 3 4 5 6 7 MICROMACRO
- 27. 27 Experiment Design Canvas Business Objective Business Outcome 1 2 User User Goal Problem Solution Result 3 4 5 6 7 MICROMACRO Drive More Experimentation
- 28. 28 Experiment Design Canvas Business Objective Key Result 1 2 User User Goal Problem Solution Result 3 4 5 6 7 Drive More Experimentation Experiments started per month MICROMACRO
- 29. 29 Experiment Design Canvas Business Objective Key Result 1 2 User User Goal Problem Solution Result 3 4 5 6 7 Drive More Experimentation Experiments started per month Web customers MICROMACRO
- 30. 30 Experiment Design Canvas Business Objective Key Result 1 2 User User Goal Problem Solution Result 3 4 5 6 7 Drive More Experimentation Experiments started per month Make changes to visitors’ experience using the visual editor MICROMACRO Web customers
- 31. 31 Experiment Design Canvas Business Objective Key Result 1 2 User User Goal Problem Solution Result 3 4 5 6 7 Drive More Experimentation Experiments started per month Make changes to visitors’ experience using the visual editor Browser blocks mixed content MICROMACRO Web customers
- 32. 32 Experiment Design Canvas Business Objective Key Result 1 2 User User Goal Problem Solution Result 3 4 5 6 7 Drive More Experimentation Experiments started per month Make changes to visitors’ experience using the visual editor Browser blocks mixed content Display troubleshooting options MICROMACRO Web customers
- 33. 33 Experiment Design Canvas Business Objective Key Result 1 2 User User Goal Problem Solution Result 3 4 5 6 7 Drive More Experimentation Experiments started per month Make changes to visitors’ experience using the visual editor Browser blocks mixed content Reduce page load errors MICROMACRO Web customers Display troubleshooting options
- 34. 34 Our Hypothesis Web customers want to make changes to their visitors’ experience using the visual editor BUT the browser blocks mixed content. IF we display troubleshooting options THEN we will reduce page load errors.
- 35. 35 Design Our Variations A Original B Desktop C All Options
- 36. 36 And the winner is... A Original B Desktop C All Options 59%
- 37. 37 And the winner is... A Original B Desktop C All Options 59% 60%
- 38. 38 And the winner is... A Original B Desktop C All Options 59% 60% 63%
- 39. 39 Roll out! A Original B Desktop C All Options 59% 60% 63%
- 40. 40 Roll out! B Desktop C All Options 60% 63%
- 41. 41 Roll out! C All Options 63%
- 42. 42 100% of our users now see...
- 45. 45 Business Objective Grow customer maturity Business Outcome Monthly Active Reporting Users Background
- 46. 46 Not all web and full stack customers know we have a new Program Management product. If we announce the launch of the new product, then people will start using it. Our Hypothesis
- 47. 47
- 48. 48
- 49. 49
- 50. 50
- 51. 51 Clicked The Sidebar Item 10.89% And then clicked the call to action 0.58% Resulting in ~200 new leads How did it do?
- 52. 52 We’ve proved it works. Now to Optimize it.
- 53. 53 Hotfixes
- 54. 54 Prioritization challenges and how they can be overcome using Optimizely
- 55. 55
- 56. 56
- 57. 57 “How do I see developer documentation?”
- 58. 58
- 59. 59
- 60. 60 Visited Dev Docs 5.77% Dev Docs Views 467 How did it do?
- 61. A few things to remember... 1. A better hypothesis inspires better designs. 2. Start with a design, measure then optimize! 3. Overcome prioritization challenges using hotfixes.
- 62. 62 FAQ
- 63. What’s the hardest part about integrating experimentation into your design process? “ ”
- 64. How can you use experimentation to drive alignment between PMs and designers? “ ”
- 65. How do you make sure that you are choosing the right metrics to measure your designs? “ ”
- 66. 66 QA
- 67. 67 Thanks