Successfully reported this slideshow.

Using Experimentation to Deliver Value at Supersonic Speeds

3

Share

1 of 50
1 of 50

Using Experimentation to Deliver Value at Supersonic Speeds

3

Share

Download to read offline

As far back as 1922, aeronautic experts knew how to travel at supersonic speeds, but not without the plane breaking apart and killing the pilot. Have rapid advancements in continuous integration over the past quarter-century placed software delivery in a similar dilemma? In this workshop, Dean Peters will speak to the question “How do we rapidly deliver valuable software without breaking the product and killing the company?”

In showing how companies such as Netflix and Microsoft are breaking the sound barrier using feature flags to iterate safely and rapidly validate features of value, Dean will demonstrate how to power your decisions and delight your customers by aligning your product offerings on a feature-flag driven culture of experimentation.

As far back as 1922, aeronautic experts knew how to travel at supersonic speeds, but not without the plane breaking apart and killing the pilot. Have rapid advancements in continuous integration over the past quarter-century placed software delivery in a similar dilemma? In this workshop, Dean Peters will speak to the question “How do we rapidly deliver valuable software without breaking the product and killing the company?”

In showing how companies such as Netflix and Microsoft are breaking the sound barrier using feature flags to iterate safely and rapidly validate features of value, Dean will demonstrate how to power your decisions and delight your customers by aligning your product offerings on a feature-flag driven culture of experimentation.

More Related Content

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Using Experimentation to Deliver Value at Supersonic Speeds

  1. 1. using experimentation to deliver value at supersonic speeds by dean peters (@deanpeters)
  2. 2. The Vision
  3. 3. Imagine working in a product culture where conversations are fueled by experimentation and validation.
  4. 4. Imagine a product culture not overrun by HiPPOs & RHiNOs
  5. 5. But Dean, how do we get from 0 to ProdOps in no time flat? Are there any historic examples of big visions that can guide us?
  6. 6. There was never a problem with a lack of vision when it came to breaking the sound barrier
  7. 7. The problem was traveling at supersonic speeds without the plane breaking apart and killing the pilot.
  8. 8. Is the our vision of rapid feature delivery at a similar historic turning point?
  9. 9. First, a little about me ... 1995 Senior Developer ● 2005 Director of Product ● ● ● 2010 Product Team Lead ● ● ● ● McClatchy 2015 Product Manager ● ● ● ● ● 2000 Principal Developer ● ● 2019 Principal Product Manager ●●●●●●● ●
  10. 10. Second, a little bit about what I’ve seen ... 1995 Every Couple Years ● 2005 Every Couple Weeks ● ● ● 2010 Twice a Week ● ● ● ● 2015 Several Times per Week ● ● ● ● ● 2000 Twice a Year ● ● 2019 Hundreds of Time Per Day ●●●●●●● ●
  11. 11. If we’re delivering features faster than ever before, then why isn’t product management celebrating? Why aren’t we having ProdOps parties like our DevOps peeps?
  12. 12. British Airways bug this past May grounds all flights from Heathrow Is it because we’re failing fast, but fatally? Kindle 3.6.1 for update iOS deletes user’s books Hulu Super Bowl LII Outage South Carolina Lottery everyone’s a winner bug SunCorp Bank disappearing money upgrade Facebook & Instagram Jan & Feb’18 Outages Uber outage leaves drivers stranded Fiat’s deadly airbag bug Dow drops due to bug impacting Fidelity, T. Rowe Price, Bank of America, & TD Ameritrade Twitter downed by special characters Kraken Cryptocurrency 4 day outage Epic Games’ Fortnite vulnerable since 1995 U.S. Customs New Year’s Day surprise outage Knight Capital’s Feature Flag flub Bugs cause Lockheed Martin delivers late on the F-35b India Air 787’s grounded Six Thousand 911 callers unable to connect last April Apple forced to pull iOS 8 update Bad software implementation causes U.S. National Grid Gas Co. loses $945 million U.K. Airspace closed for a day Amazon’s 1p pricing snafu Starbucks cash register bug closes 60% of its stores RBS integration w/3rd party causes 600,000 payments to fail Bloomberg U.K. trading terminals go down HBSC finds a bug causing 275,000 payments to fail Prison Software approves ‘good behavior’ releases for 3,200 criminals too many Nest Bug leaves users in the cold Bitcoin Unlimited memory leaks Welsh National Healthcare System unable to access patient records TBS Bank locks out millions of users
  13. 13. The 2018 Tricentis ‘Fail Watch Report’ details 606 recent software failures that affected over 3.6 billion people resulting in $1.7 trillion in lost revenue. In examining the collapse of 101 start-ups between 2017 & 2018, CBInsights cited the top 20 contributing causes, half indicating failures in value delivery.
  14. 14. Are we simply delivering crap faster?
  15. 15. How do we align on the supersonic delivery of valuable features without the product breaking apart and killing the company?
  16. 16. The Strategy
  17. 17. In breaking the Sound Barrier ... hypotheses were stated tests were run causality was analysed
  18. 18. State a Hypothesis Source: xkcd.com - Creative Commons Attribution-NonCommercial 2.5 License. A narrowly focused if … then … feature statement. Whose value is measured by impact on key behavioral metrics.
  19. 19. Deploy Rapidly in Tiny Test Increments Source: xkcd.com - Creative Commons Attribution-NonCommercial 2.5 License. Trunk-based branching that supports very small and very automated rapid deployment. Telemetry to capture both technical and product metrics.
  20. 20. Learn via Causation Source: xkcd.com - Creative Commons Attribution-NonCommercial 2.5 License. Tests include scientific elements of control. Random assignment of treatments. Statistical significance.
  21. 21. Learn Build Measure Lean Build, Measure, Learn Model Evolve from simply talking about Build Measure Learn
  22. 22. HypothesizeAnalyze on Causation Better BuildsMeasure DevOps Scientific Test Treatments Experimentation as a Service To Driving Value via a Culture of Experimentation
  23. 23. Ronny Kohavi’s observations based on experiments at Microsoft ● 1/3 of ideas were positive ideas and statistically significant ● 1/3 of ideas were flat: no statistically significant difference ● 1/3 of ideas were negative and statistically significant Avinash Kaushik, author of ‘Web Analytics: An Hour a Day,’ wrote in his Experimentation and Testing primer that “80% of the time you/we are wrong about what a customer wants.”
  24. 24. The Tactics
  25. 25. Feature Featur e Feature Feature Featur e Featur e Featur e Featur e Featur e Sprint 1 Sprint 2 Sprint 3 Sprint 5Sprint 4 ‘Lean’ into smaller feedback cycles Waterfall Agile Lean w/ Feature Flags DeployBuildBUFD
  26. 26. How do we align on a culture of experimentation? By measuring behaviors of value Through the tactical implementation of feature flags.
  27. 27. A Feature Flag is Code Source: How to implement feature flags and A|B testing; msdn blog Feature Flag Segment switch (myFeatureCondition) { case 1: result = myApp.NewTreatment(); break; case 2: result = myApp.OldTreatment(); break; default: result = myApp.OldTreatment(); }
  28. 28. Discussions with DevOps ● Encourage trunk-based development in tiny branches ● Allow incomplete & in-progress features to be deployed as dark code ● Protect against poorly performing features with a kill-switch ● Facilitate progressive rollouts.
  29. 29. Persuading Product & Marketing ● Supports early adopter and preview programs ● Allows visibility management via subscriptions, user rights, or sales packages ● Empowers QA to quickly toggle a wide variety of test environments ● Supports A/B testing and generates statistically significant results
  30. 30. Some Conversational Caveats ● Feature Flags introduce Technical Debt ● Feature Flags requires a cultural shift in thinking ● Feature Flags must be short lived ● Feature Flags will get messy really fast if not well managed ● Feature Flags wile give some of your B2B customers pause
  31. 31. It still beats the alternative of ZEbRA & WoLF prioritization! Source: Behold, the Product Management Prioritization Menagerie - DeanOnDelivery.com
  32. 32. The Example
  33. 33. A ProductCraft Use Case
  34. 34. ● We believe that we can increase reader engagement per session by offering ‘Related Stories’ instead of ‘Trending Stories’ to users arriving at a story via a search action. (nice hypothesis there Deano, but how do we measure it?) Value Hypothesis
  35. 35. Pick an OOB Growth, Value, or Ops metric based on context Sources: How to Use the Google HEART Framework to Measure and Improve Your App’s UX DevOps Implementation Services - Veritis & AARRR vs. RARRA Sequences - AppGrowth Summit Happiness Engagement Adoption Retention Task Success
  36. 36. Defining our example test with Google’s value HEART metric Baseline Test Happiness NPS NPS & Interviews Engagement Clicks on Trending Stories Clicks on Related Stories Adoption A/A Test A/B Test Retention Active Visitors who Actively Search Active Visitors who Actively Search Task Success Bounce on Trending Stories Clicked Bounce on Related Stories Clicked
  37. 37. ● We believe that we can increase reader engagement per session by offering ‘Related Stories’ instead of ‘Trending Stories’ to users arriving at a story via a search action. ● We will know we have succeeded if we see a lift in click-through behaviors on related stories offered in the sidebar. Value Hypothesis with Success Metric
  38. 38. Run tiny experiments rapidly and incrementally Test 1 Baseline A/A Results on Trending Test 2 Replace Trending with Related Test 3 Measure Relevance of Selected Stories Test 4 Measure Number of Relevant Stories
  39. 39. Visualization of our Hypothesis Sources: How to Choose the Right Metrics for Your Experiments; Spli.io & How to Use the Stickiness Ratio to Measure Product Health; Pendo
  40. 40. Feature Flag Powered Progressive Rollouts Source: How to implement feature flags and A|B testing; msdn blog Feature Flag Segment 5% 50% 95%
  41. 41. The Execution
  42. 42. How do we rapidly deliver valuable features without the breaking product and killing the company?
  43. 43. How do we move from a culture of seagull management & ticket-slinging? Source: Behold, the Product Management Prioritization Menagerie - DeanOnDelivery.com
  44. 44. We pursue a vision of rapid value delivery Through strategic experimentation Tactically facilitated by feature flags Focusing daily release activities that measure causality Supersonic Software Delivery through Experimentation
  45. 45. Where to Start? ● Dogooding ● Proof of Concepts ● Painted Doors ● Canary Tests ● Trial Users vs. Paid ● SMB v. Enterprise ● Early Adopter Programs You may already have some experiments in play
  46. 46. An example of some experimentation success ... ● 4 years Citrix + Pendo luv ● 10 Pendo subscriptions ● Enjoyed by both Marketing & Product ● Used for Onboarding, User Adoption, Retention & Engagement, Upsell ● 22 Citrix products and services ● 500+ live guides ● 2,000+ feature tags ● Dogfooding the WorkSpace offering
  47. 47. And you’ve got friends
  48. 48. Questions? Comments? Conversations? Dean Peters (@deanpeters)
  49. 49. Useful URLs Books I referenced ● Unlearn ● Accelerate ● Lean Analytics Valuable Vendor e-Books ● Understanding Experimentation Platforms | Split.io ● Effective Feature Management eBook | LaunchDarkly ● The Ultimate Guide For Feature Toggles | Rollout.io Podcasts I Mentioned ● DeliverIT - Episode 91 ● ProductCraft - Cast #4 ● Product to Product - #NoEstimates Shameless self-promotion ● DeanOnDelivery blog (& bookcovers) ● LinkedIn ● Twitter ● Experimental Podcast Page (With 3 test casts posted by 28Sep19 you can enjoy & brutally critique!-) Articles by Authors I Admire ● Ronny Kohavi (pdf) ● Avinash Kaushik (blog) ● Martin Fowler (blog)
  50. 50. www.pendo.io And thanks to Pendo and session attendees for giving me this awesome opportunity to share my vision on transforming product culture through experimentation!

Editor's Notes

  • [30 seconds]
  • [60 seconds]
  • [60 seconds]
  • [30 seconds]
  • [30 seconds]
  • [120 seconds]
  • [120 seconds]
  • [15 seconds]
  • [60 seconds]
  • [60 seconds]
  • [15 seconds]
  • [45 seconds]
  • [120 seconds]
  • [30 seconds]
  • [15 seconds]
  • [15 seconds]
  • [30 seconds]
  • [60 seconds]
  • [90 seconds]
  • [60 seconds]
  • [15 seconds]
  • [120 seconds]
  • [180 seconds]
  • [15 seconds]
  • [90 seconds]
  • [15 seconds]
  • [30 seconds]
  • [120 seconds]
  • [180 seconds]
  • [240 seconds]
  • [30 seconds]
  • [15 seconds]
  • [60 seconds]

  • [30 seconds]
  • [240 seconds]
  • [360 seconds]
  • [30 seconds]
  • [180 seconds]
  • [30 seconds]
  • [30 seconds]
  • [15 seconds]
  • [15 seconds]
  • [30 seconds]
  • [90 seconds]
  • [480 seconds]
  • [180 seconds]
  • [30 seconds]
  • ×