Program Management 101: Best Practices from Optimizely-on-Optimizely
•Download as PPTX, PDF•
0 likes•469 views
What does it take to run a best-in-class experimentation program? What sets these programs apart isn’t some kind of magic formula; instead, they invested in making experimentation a core practice across the business with a dedicated team in place, best-in-class technology and a practice of evangelizing their impact on the business. Learn from Optimizely’s own Experimentation Program Manager, Becca Bruggman, and the Product Manager of Optimizely Program Management, Kris McKee, on exactly how they took our own internal Product experimentation program from ad-hoc to a robust Center of Excellence. They will take you through the lifecycle of an idea: from concept, to experiment, results, and then ultimately iterating to the next experiment — all in just 30 minutes!
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Program Management 101: Best Practices from Optimizely-on-Optimizely
Similar to Program Management 101: Best Practices from Optimizely-on-Optimizely (20)
More from Optimizely
More from Optimizely (20)
Recently uploaded
Recently uploaded (20)
Program Management 101: Best Practices from Optimizely-on-Optimizely
- 1. Congratulations to our Outperform Award Winners! Most Dramatic Business Impact Most Customer- Obsessed Company Culture Most Transformative Innovation Most Inspiring Social Impact
- 2. 2 Welcome to Day 2 of Opticon! Sessions today will be recorded and will be available after Opticon Share your feedback and rate sessions on the mobile app
- 3. Kris McKee Senior Product Manager Rebecca Bruggman Staff Technical Program Manager Program Management 101: Best Practices from Optimizely-on-Optimizely
- 4. 4 Experimentation is the next great business transformation The Surprising Power of Online Experiments “Our success is a function of how many experiments we do per year, per month, per week, per day.” Jeff Bezos “Our company culture encourages experimentation and free flow of ideas.” Larry Page “One of the things I’m most proud of, and I think what is the key to our success, is the testing framework we’ve built.” Mark Zuckerberg “Instead of saying ‘I have an idea,’ what if you said ‘I have a new hypothesis, let’s go test it.’ ” Satya Nadella
- 5. 5 Experimentation velocity is an indicator of unlocked potential What HBR says: “Growth takes off once the experimentation platform allows a user to take part in multiple experiments at the same time, supporting virtually unlimited concurrent tests.” What Forrester says: “Insights-driven businesses are growing by more than 30% annually and are on track to earn $1.8 trillion by 2021.” What MIT Sloan says: “Digital experimentation should not be a ‘once and done’ effort. Rather, it is about developing a culture of experimentation in which new business ideas are continually tried and tested. Build and iterate on the successes, and learn from the failures.”
- 6. 6 Implementing a complete program may seem like this ...
- 7. 7 … but, challenges are manageable with right strategy & tools Coordination As you scale, coordinating across additional teams and stakeholders becomes more and more difficult Resources Time, people, and effort is wasted duplicating and reinventing different processes Prioritization Ideation can be siloed leading to inefficient prioritization and execution Oversight & knowledge sharing Lack of executive oversight can lead to serious problems in a global organization
- 8. 8 Promote an experimentation mindset Set program goals and measure performance Collaborate to generate great ideas Refine experimentation ideas Build an experimentation roadmap Share and evangelize learnings Combine a foundation of mindset and goal setting for experimentation program success
- 9. 9 Promote an experimentation mindset Set program goals and measure performance Collaborate to generate great ideas Refine experimentation ideas Build an experimentation roadmap Share and evangelize learnings Framework for experimentation program success
- 10. 10 Promote an experimentation mindset Set program goals and measure performance Collaborate to generate great ideas Refine experimentation ideas Build an experimentation roadmap Share and evangelize learnings Framework for experimentation program success
- 11. 11 ● Implement ‘test and learn’ decision making ● Base all decisions on data ● Understand the “why” behind your data by asking the right questions ● Avoid complacency through continuous iteration Build a foundation of culture and mindset
- 12. 12 Promote an experimentation mindset Set program goals and measure performance Collaborate to generate great ideas Refine experimentation ideas Build an experimentation roadmap Share and evangelize learnings Framework for experimentation program success
- 13. 13 Velocity & Win Rate Testing velocity lets you measure your team’s operational performance Measure Program Win rate helps teams measure the quality of their hypothesis and impact to the business
- 14. 14 Using Program Reporting Right ● Set monthly, quarterly, and annual goals for performance ● Measure and report on a monthly basis ● Re-assess on an annual basis ● Integrate the learnings into daily operations of your work
- 15. 15 Some things you may learn: ● Some teams are more effective than others in moving an idea to an experiment ● Win rates may be significantly lower on certain pages of your site ● Testing velocity may slow around holidays ● Certain experiment strategies work more effectively Insights on your program
- 16. 16 As you begin to understand your program: ● Track how win rate and velocity impact your overall program value ● Think about ROI and where you can maximize return ● Take a portfolio approach to your experimentation program Assessing Opportunity
- 17. 17 Promote an experimentation mindset Set program goals and measure performance Collaborate to generate great ideas Refine experimentation ideas Build an experimentation roadmap Share and evangelize learnings Framework for experimentation program success
- 18. 18 Promote an experimentation mindset Set program goals and measure performance Collaborate to generate great ideas Refine experimentation ideas Build an experimentation roadmap Share and evangelize learnings Framework for experimentation program success
- 19. 19 ● Train teams on how to write a hypothesis ● The best hypothesis are rooted in data and deep insights into your business ● By systematic: create a process for submission, scoring, and acceptance ● Give constructive feedback Democratize (good) ideation
- 20. 20 Capture Ideas Give structure to testing backlog by categorizing ideas by site / page Collaborate widely Always start with a hypothesis - give teams examples of what a good one looks like Naming conventions can help keep things organized and easy to report
- 21. 21 Capture Ideas with Slack Integrate your Slack instance to make idea capturing easy Collaborate Widely Allow your collaborators to work where they spend the most time - Slack
- 22. 22 Promote an experimentation mindset Set program goals and measure performance Collaborate to generate great ideas Refine experimentation ideas Build an experimentation roadmap Share and evangelize learnings Framework for experimentation program success
- 23. 23 ● Set expectations by answering why, when, where, who and how of each idea ● Define primary, secondary, and monitoring goals ● Calculate your min detectable effect (MDE) to determine if results will be valuable enough to justify cost Define your experiment
- 24. 24 Commenting Give structure to testing backlog by categorizing ideas by site / page Prioritize backlog Always start with a hypothesis - give teams examples of what a good one looks like Naming conventions can help keep things organized and easy to report
- 25. 25 Promote an experimentation mindset Set program goals and measure performance Collaborate to generate great ideas Refine experimentation ideas Build an experimentation roadmap Share and evangelize learnings Framework for experimentation program success
- 26. 26 – Selecting & Accepting Ideas Backlog status lets your team know it’s still under review by the central group Prioritize backlog Active status lets everyone know you’ve selected this for experimentation and are beginning work
- 27. 27 Scoring Potential of the experiment to positively impact your goal Prioritize backlog Impact to the business if the experiment wins Level of Effort to implement the experiment in Optimizely LOVE strategic importance, executive support, biased opinions
- 28. 28 ● Create a prioritization framework, starting with product and business goals ● Evaluate testing ideas based on potential, impact, effort, and LOVE (personal preference) ● Consider doing an objective comparison based on a key metric ● Prioritization is critical to the quality and velocity of experimentation Prioritize with scale in mind
- 29. 29 Promote an experimentation mindset Set program goals and measure performance Collaborate to generate great ideas Refine experimentation ideas Build an experimentation roadmap Share and evangelize learnings Framework for experimentation program success
- 30. 30 ● Encourage company-wide participation ● Let teammates share in successes ● Routinely communicate across departments Celebrate widely
- 31. 3131 Data-based Idea Generation A strong understanding of business goals and key metrics should guide idea generation and experiment prioritization Democratize Ideation Cast a wide net and encourage ideation throughout the firm to drive innovative ideas and a culture of experimentation Celebrate Widely Share wins across entire firm and celebrate collaboration and encourage continued experimentation mindset Ongoing collaboration and communication is core to building a strong experimentation program Set Goals and Publish results Set win rate and experimentation velocity goals and publish results widely
- 33. 33 ADEPT Experimentation Program Center of Excellence Center of Excellence Enablement Best Practices Sharing Learning Program Measurement Ideation Prioritization Design Execution Results Analysis Team 1 Team 2 Team 3 Team 4
- 34. ADEPT COE Core Team Claire Vo Becca Bruggman
- 35. 35 Marketing Experimentation Program Individual Team SDRs PMM Ideation Ideation Prioritization Design Build QA & Launch Analyze Sharing Learnings Execution Assists Individual Team Development Design Content Analytics Product Managers CSMs
- 36. Michael O’Connell Marketing Experimentation Team Bryce WellingtonTakeshi Young Becca Bruggman
- 37. 37 Promote an experimentation mindset Set program goals and measure performance Collaborate to generate great ideas Refine experimentation ideas Build an experimentation roadmap Share and evangelize learnings Focus on Ideation Loop
- 39. 39 Heavily commented Slack Conversions
- 40. 40 Painted Door for User Research
- 41. 41 Quantitative Evidence Adoption of Segmentation correlated with ~50% increase in results page retention
- 46. 46 Results
- 53. Process
- 54. 54 Process: What is our Program?
- 55. 55 Process: How to Run an Experiment?
- 56. 56 Process: How to Run a Full Stack Experiment?
- 57. 57 Process: How to make new Audiences, Pages, etc?
- 58. Visibility
- 59. Visibility: All Company Emails, Physical Walls
- 61. 61 Visibility: Program Oversight via Slack integration
- 62. 62 End-to-End Process Log Idea Score Idea Log JIRA & Link Add Idea to Card Experiment Review Visible Changes Results Review
- 63. Thank you!
- 64. Q&A