Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Program Management 101: Best Practices from Optimizely-on-Optimizely

187 views

Published on

What does it take to run a best-in-class experimentation program? What sets these programs apart isn’t some kind of magic formula; instead, they invested in making experimentation a core practice across the business with a dedicated team in place, best-in-class technology and a practice of evangelizing their impact on the business.

Learn from Optimizely’s own Experimentation Program Manager, Becca Bruggman, and the Product Manager of Optimizely Program Management, Kris McKee, on exactly how they took our own internal Product experimentation program from ad-hoc to a robust Center of Excellence. They will take you through the lifecycle of an idea: from concept, to experiment, results, and then ultimately iterating to the next experiment — all in just 30 minutes!

Published in: Business
  • Be the first to comment

  • Be the first to like this

Program Management 101: Best Practices from Optimizely-on-Optimizely

  1. 1. Congratulations to our Outperform Award Winners! Most Dramatic Business Impact Most Customer- Obsessed Company Culture Most Transformative Innovation Most Inspiring Social Impact
  2. 2. 2  Welcome to Day 2 of Opticon!  Sessions today will be recorded and will be available after Opticon  Share your feedback and rate sessions on the mobile app
  3. 3. Kris McKee Senior Product Manager Rebecca Bruggman Staff Technical Program Manager Program Management 101: Best Practices from Optimizely-on-Optimizely
  4. 4. 4 Experimentation is the next great business transformation The Surprising Power of Online Experiments “Our success is a function of how many experiments we do per year, per month, per week, per day.” Jeff Bezos “Our company culture encourages experimentation and free flow of ideas.” Larry Page “One of the things I’m most proud of, and I think what is the key to our success, is the testing framework we’ve built.” Mark Zuckerberg “Instead of saying ‘I have an idea,’ what if you said ‘I have a new hypothesis, let’s go test it.’ ” Satya Nadella
  5. 5. 5 Experimentation velocity is an indicator of unlocked potential What HBR says: “Growth takes off once the experimentation platform allows a user to take part in multiple experiments at the same time, supporting virtually unlimited concurrent tests.” What Forrester says: “Insights-driven businesses are growing by more than 30% annually and are on track to earn $1.8 trillion by 2021.” What MIT Sloan says: “Digital experimentation should not be a ‘once and done’ effort. Rather, it is about developing a culture of experimentation in which new business ideas are continually tried and tested. Build and iterate on the successes, and learn from the failures.”
  6. 6. 6 Implementing a complete program may seem like this ...
  7. 7. 7 … but, challenges are manageable with right strategy & tools Coordination As you scale, coordinating across additional teams and stakeholders becomes more and more difficult Resources Time, people, and effort is wasted duplicating and reinventing different processes Prioritization Ideation can be siloed leading to inefficient prioritization and execution Oversight & knowledge sharing Lack of executive oversight can lead to serious problems in a global organization
  8. 8. 8 Promote an experimentation mindset Set program goals and measure performance Collaborate to generate great ideas Refine experimentation ideas Build an experimentation roadmap Share and evangelize learnings Combine a foundation of mindset and goal setting for experimentation program success
  9. 9. 9 Promote an experimentation mindset Set program goals and measure performance Collaborate to generate great ideas Refine experimentation ideas Build an experimentation roadmap Share and evangelize learnings Framework for experimentation program success
  10. 10. 10 Promote an experimentation mindset Set program goals and measure performance Collaborate to generate great ideas Refine experimentation ideas Build an experimentation roadmap Share and evangelize learnings Framework for experimentation program success
  11. 11. 11 ● Implement ‘test and learn’ decision making ● Base all decisions on data ● Understand the “why” behind your data by asking the right questions ● Avoid complacency through continuous iteration Build a foundation of culture and mindset
  12. 12. 12 Promote an experimentation mindset Set program goals and measure performance Collaborate to generate great ideas Refine experimentation ideas Build an experimentation roadmap Share and evangelize learnings Framework for experimentation program success
  13. 13. 13 Velocity & Win Rate Testing velocity lets you measure your team’s operational performance Measure Program Win rate helps teams measure the quality of their hypothesis and impact to the business
  14. 14. 14 Using Program Reporting Right ● Set monthly, quarterly, and annual goals for performance ● Measure and report on a monthly basis ● Re-assess on an annual basis ● Integrate the learnings into daily operations of your work
  15. 15. 15 Some things you may learn: ● Some teams are more effective than others in moving an idea to an experiment ● Win rates may be significantly lower on certain pages of your site ● Testing velocity may slow around holidays ● Certain experiment strategies work more effectively Insights on your program
  16. 16. 16 As you begin to understand your program: ● Track how win rate and velocity impact your overall program value ● Think about ROI and where you can maximize return ● Take a portfolio approach to your experimentation program Assessing Opportunity
  17. 17. 17 Promote an experimentation mindset Set program goals and measure performance Collaborate to generate great ideas Refine experimentation ideas Build an experimentation roadmap Share and evangelize learnings Framework for experimentation program success
  18. 18. 18 Promote an experimentation mindset Set program goals and measure performance Collaborate to generate great ideas Refine experimentation ideas Build an experimentation roadmap Share and evangelize learnings Framework for experimentation program success
  19. 19. 19 ● Train teams on how to write a hypothesis ● The best hypothesis are rooted in data and deep insights into your business ● By systematic: create a process for submission, scoring, and acceptance ● Give constructive feedback Democratize (good) ideation
  20. 20. 20 Capture Ideas Give structure to testing backlog by categorizing ideas by site / page Collaborate widely Always start with a hypothesis - give teams examples of what a good one looks like Naming conventions can help keep things organized and easy to report
  21. 21. 21 Capture Ideas with Slack Integrate your Slack instance to make idea capturing easy Collaborate Widely Allow your collaborators to work where they spend the most time - Slack
  22. 22. 22 Promote an experimentation mindset Set program goals and measure performance Collaborate to generate great ideas Refine experimentation ideas Build an experimentation roadmap Share and evangelize learnings Framework for experimentation program success
  23. 23. 23 ● Set expectations by answering why, when, where, who and how of each idea ● Define primary, secondary, and monitoring goals ● Calculate your min detectable effect (MDE) to determine if results will be valuable enough to justify cost Define your experiment
  24. 24. 24 Commenting Give structure to testing backlog by categorizing ideas by site / page Prioritize backlog Always start with a hypothesis - give teams examples of what a good one looks like Naming conventions can help keep things organized and easy to report
  25. 25. 25 Promote an experimentation mindset Set program goals and measure performance Collaborate to generate great ideas Refine experimentation ideas Build an experimentation roadmap Share and evangelize learnings Framework for experimentation program success
  26. 26. 26 – Selecting & Accepting Ideas Backlog status lets your team know it’s still under review by the central group Prioritize backlog Active status lets everyone know you’ve selected this for experimentation and are beginning work
  27. 27. 27 Scoring Potential of the experiment to positively impact your goal Prioritize backlog Impact to the business if the experiment wins Level of Effort to implement the experiment in Optimizely LOVE strategic importance, executive support, biased opinions
  28. 28. 28 ● Create a prioritization framework, starting with product and business goals ● Evaluate testing ideas based on potential, impact, effort, and LOVE (personal preference) ● Consider doing an objective comparison based on a key metric ● Prioritization is critical to the quality and velocity of experimentation Prioritize with scale in mind
  29. 29. 29 Promote an experimentation mindset Set program goals and measure performance Collaborate to generate great ideas Refine experimentation ideas Build an experimentation roadmap Share and evangelize learnings Framework for experimentation program success
  30. 30. 30 ● Encourage company-wide participation ● Let teammates share in successes ● Routinely communicate across departments Celebrate widely
  31. 31. 3131 Data-based Idea Generation A strong understanding of business goals and key metrics should guide idea generation and experiment prioritization Democratize Ideation Cast a wide net and encourage ideation throughout the firm to drive innovative ideas and a culture of experimentation Celebrate Widely Share wins across entire firm and celebrate collaboration and encourage continued experimentation mindset Ongoing collaboration and communication is core to building a strong experimentation program Set Goals and Publish results Set win rate and experimentation velocity goals and publish results widely
  32. 32. What is an Experimentation Program?
  33. 33. 33 ADEPT Experimentation Program Center of Excellence Center of Excellence Enablement Best Practices Sharing Learning Program Measurement Ideation Prioritization Design Execution Results Analysis Team 1 Team 2 Team 3 Team 4
  34. 34. ADEPT COE Core Team Claire Vo Becca Bruggman
  35. 35. 35 Marketing Experimentation Program Individual Team SDRs PMM Ideation Ideation Prioritization Design Build QA & Launch Analyze Sharing Learnings Execution Assists Individual Team Development Design Content Analytics Product Managers CSMs
  36. 36. Michael O’Connell Marketing Experimentation Team Bryce WellingtonTakeshi Young Becca Bruggman
  37. 37. 37 Promote an experimentation mindset Set program goals and measure performance Collaborate to generate great ideas Refine experimentation ideas Build an experimentation roadmap Share and evangelize learnings Focus on Ideation Loop
  38. 38. Experiment Ideation on ADEPT
  39. 39. 39 Heavily commented Slack Conversions
  40. 40. 40 Painted Door for User Research
  41. 41. 41 Quantitative Evidence Adoption of Segmentation correlated with ~50% increase in results page retention
  42. 42. 42 Quantitative Evidence
  43. 43. 44 Variation Design
  44. 44. 45 Variation Design
  45. 45. 46 Results
  46. 46. Experiment Ideation on Marketing
  47. 47. Ideation Sessions
  48. 48. Original Variation 1 Variation 2
  49. 49. Process
  50. 50. 54 Process: What is our Program?
  51. 51. 55 Process: How to Run an Experiment?
  52. 52. 56 Process: How to Run a Full Stack Experiment?
  53. 53. 57 Process: How to make new Audiences, Pages, etc?
  54. 54. Visibility
  55. 55. Visibility: All Company Emails, Physical Walls
  56. 56. 60 Visibility: Experiment Review
  57. 57. 61 Visibility: Program Oversight via Slack integration
  58. 58. 62 End-to-End Process Log Idea Score Idea Log JIRA & Link Add Idea to Card Experiment Review Visible Changes Results Review
  59. 59. Thank you!
  60. 60. Q&A

×