Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Martijn Beijk & Charles Goodall
Similar to Martijn Beijk & Charles Goodall (20)
More from World_Forum_Disrupt
More from World_Forum_Disrupt (20)
Recently uploaded
Recently uploaded (20)
Martijn Beijk & Charles Goodall
- 1. 1 Who Optimizes the Optimizers Experiment-driven Product Development Martijn Beijk Director, Solutions Engineering, EMEA martijn.beijk@optimizely.com
- 3. I work at the World’s Largest Digital Laboratory 1.000.000+ experiments and counting
- 4. 4 Optimizely is constantly researching the characteristics and impact of high performing experimentation programs ● Analysis of over 1,000 companies and more than 100,000 experiments ● Identification of best practices for experimentation Global Optimization Benchmark ● Analysis of 14,000 experiments to identify the best practices that make companies more successful in experimentation Experiment Design and Performance ● Analysis of 1,000’s of experiments to understand how risk-taking and innovation evolve over time ● Analysis of how risk-taking affects experimentation performance Experimentation and Innovation Experimentation and Firm Performance ● Analysis of how the scale of experimentation affects the financial performance of organizations
- 5. “Our success is a function of how many experiments we do per year, per month, per week, per day.” Jeff Bezos “Our aim is to create the best product for our customers, and we do that through constant innovation and testing.” Gillan Tans, CEO “Our company culture encourages experimentation and free flow of ideas.” Larry Page “We use experimentation and testing to inform as much of the business as we possibly can” - Gregory Peters, CPO Today’s Digital Leaders Win By Using Experimentation At-Scale
- 6. 6
- 7. 7 A 2019 Alpha Product Management Report uncovered that the ability to innovate and experiment is being hindered by costly internal politics and distractions
- 8. 8 “63% of organizations today have to constantly adjust their market position“ Source: “Accelerating Europe to Customer Speed (July ‘19)”, Survey of 300 digital experts
- 9. 9 Deploying product is the most expensive way to validate an idea Silicon Valley Product Group, Marty Cagan
- 10. 10 Discovery vs Delivery Build the right product Fake it Move Fast Build things that don’t scale Discovery Delivery Build the product right Till you make it Don’t Break Things and then build things that do scale
- 11. BUILD LEARN MEASURE Product Experimentation Think hypothetically ● Painted doors ● Validating MVPEs ● Qualitative research ● Exploratory A/B testing Quantify product impact ● Feature validation ● Multivariate tests ● Iterative adjustments ● Measuring business impact Release confidently ● Frequent, small releases ● Feature flagging ● Staged rollouts ● Safe rollbacks
- 12. 12 Building a Culture of Experimentation Today’s Reality Unable to evaluate the impact of many product changes Opinion-led decision making process What The Future Holds Every change to the product is robustly evaluated Hypothesis-led product roadmap (and business decisions) Bottom-up ideation
- 13. 13 The Science Common product release approaches
- 14. 14 Prove a concept works with the minimum viable experiment Build it Don’t build it Wins Loses
- 15. 15 Build a new feature Run an experiment to see what impact it has Implement it Implement it anyway Wins Loses Confidence Bargaining This is a great idea we definitely need to make this change. Should we maybe test this? I’m sure it’ll win but just in case! But we did see a 1.3% uplift in clicks between 8-9pm, right? We put too much work into this to not implement it. Doubt Acceptance / resignation
- 16. 16 Old: Traditional Product Development Requirements Specifications Design Build QA / Test Maintenance Problem: known Solution: known Unit of Progress: Advancement to Next Stage Waterfall
- 17. 17 Common: Agile Product Development Solution: in development Problem: known Unit of Progress: A line of working code “Product Owner” or in-house customer
- 18. 18 What we’ve seen work best
- 19. 19 Now & Future: Experiment Driven Product Development Solution: unvalidated Problem: sometimes unvalidated Unit of Progress: Validated learning Experiment results, Confirmed hypothesis Data, Feedback, Insights
- 20. 20 Best-in-class companies: Release all the time. Measure everything. Iterate rapidly.
- 21. 21 It is no longer about product releases. Today is all about continuous delivery AND continuous experimentation. Reference: Death of the Release: The Move to Continuous Delivery and Experimentation:
- 22. 22 RELEASE ≠ DEPLOY Separate code deployment from feature enablement to drive rapid iteration on digital products Limit the blast radius by gradually rolling out features in controlled experiments
- 23. 23 Experiment Driven Development | Building “stuff” quickly is not enough, product investments must deliver business value ▪ Remove guesswork and validate business impact of development investments ▪ Learn throughout development to iterate towards a final deliverable ▪ Multiple validation points provide opportunity to redirect resources from non-impact projects Agile Experiment Driven Development Code ReleaseIdea Validate & Prototype ReleaseIdea Refine A/B TestA/B Test Continue? Continue? Continue? Rework?
- 24. 24 Decision Tree for Experiment Driven Dev Teams Have users validated our hypotheses, demonstrating that our proposed solution is the best way to achieve the desired results? Discover Sprint Cycle MVP Build Sprint Cycle Incremental Feature Build Sprint Cycle Experiment Review MVP Go-Live Decision Experiment Definition Do we have proven product-market fit and technical ability to deliver?No Yes No Yes MVP Feature Definition Incrementa l Feature Definition
- 25. 25Experiment Driven Development for Agile Teams Discovery Sprint Experiment Launched Experiment Definition Team develops initial features and spends 3 weeks testing some ideas and learning from mock-ups This is Discovery because risk is high. Goal is achieve new learnings and we don’t have a product-market fit yet. Product Backlog Requirements Enhancements or Fixes Features
- 26. 26Experiment Driven Development for Agile Teams Discovery Sprint MVP Build Sprint Experiment Launched MVP Go-Live Decision MVP Feature Definition Experiment Definition Team develops initial features and spends 3 weeks testing some ideas and learning from mock-ups Team builds an initial version and deploys it to pilot customers This is Discovery because risk is high. Goal is achieve new learnings and we don’t have a product-market fit yet. This is MVP or Prototype build because we have good indication of a market need and the solution is feasible. We want to build something small to uncover additional learnings Product Backlog Requirements Enhancements or Fixes Features
- 27. 27Experiment Driven Development for Agile Teams Discovery Sprint MVP Build Sprint Feature Refinement Sprint Experiment Launched MVP Go-Live Decision MVP Feature Definition Experiment Definition Incrementa l Feature Definition Team develops initial features and spends 3 weeks testing some ideas and learning from mock-ups Team builds an initial version and deploys it to pilot customers Team improves the feature based on direct feedback (usability tests, customer responses) This is Discovery because risk is high. Goal is achieve new learnings and we don’t have a product-market fit yet. This is MVP or Prototype build because we have good indication of a market need and the solution is feasible. We want to build something small to uncover additional learnings This is Feature Build. The solution is in production and we are incrementally adding value to it or optimizing the feature. Product Backlog Requirements Enhancements or Fixes Features
- 28. 28 “Painted door” experiments to validate customer demand User studies and rapid prototyping to validate feature design Design Discover Experiment Driven Development for Agile Teams MVP Build Sprint Feature Refinement Sprint Discovery Sprint Experiment Launched Experiment Definition
- 29. 29Experiment Driven Development for Agile Teams Feature Flags for trunk-based development and adopting Continuous Delivery practices Build MVP Build Sprint Feature Refinement Discovery Sprint MVP Go-Live Decision MVP Feature Definition
- 30. 30Experiment Driven Development for Agile Teams Feature Flags for trunk-based development and adopting Continuous Delivery practices Build MVP Build Sprint Feature Refinement Feature Tests to prove business impact of the MVP Discovery Sprint Experiment MVP Feature Definition
- 31. 31Experiment Driven Development for Agile Teams Feature Flags for trunk-based development and adopting Continuous Delivery practices Build Feature Tests to prove business impact of the MVP Experiment MVP Build Sprint Feature Refinement Discovery Sprint Feature Rollouts to monitor quality and performance Roll Out MVP Feature Definition MVP Go-Live Decision
- 32. 32 Feature Configuration and Feature Tests to determine the optimal combination of variable values OR (if larger work effort) Experiment Driven Development for Agile Teams MVP Build Sprint Feature Refinement Sprint Discovery Sprint Refine Experiment Build Roll Out Feature Flags Feature Tests Feature Rollouts
- 33. 33 Feature flags to limit blast radius “Painted door” experiments to validate customer demand (thru marketing squad) Feature tests to prove business impact Staged rollouts to monitor quality and performance Growth experiments and CRO to improve usage, adoption, and conversion rate User studies and rapid prototyping to validate feature design Discover Design Build Experiment Roll Out Iterate
- 34. Audiences Areas KPIs Goal Levers Experiments What problem are we trying to solve? How will we know if we’ve solved it? Where is the problem? Who does it impact and how? What solutions could exist? Which is the best solution? Experimentation is a Problem-Solving Framework
- 35. 35 “Understanding how to solve your user’s biggest problems become your experiment learning goals.“