Download as PDF, PPTX
More Related Content
![[Webinar] How Vivid Seats Ensures Experimentation Success](https://cdn.slidesharecdn.com/ss_thumbnails/optimizelywebinarslideshare-180426184156-thumbnail.jpg?width=640&height=640&fit=bounds)
![[Webinar] Innovate Faster by Adopting The Modern Growth Stack](https://cdn.slidesharecdn.com/ss_thumbnails/webinarinnovatefasterwiththemoderngrowthstack-180124222436-thumbnail.jpg?width=640&height=640&fit=bounds)
![[Webinar] Visa's Journey to a Culture of Experimentation](https://cdn.slidesharecdn.com/ss_thumbnails/optimizelywebinar-visasjourneytoacultureofexperimentation-180227210737-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Better Decisions with Experimentation
Better Decisions with Experimentation
- 1. Better Decisions with Experimentation TomTsao Atlasssian LESSONS FROM THE GROWTH OF ATLASSIAN’S EXPERIMENTATION TEAM Wes McCabe Optimizely
- 2.
- 3. Housekeeping ● We arerecording this webinar ● You’ll receive the recording and slides ● We’ll answer all questions at the end
- 4. Agenda ● Introduction ● Whatis Optimizely? ● Atlassian’s Experimentation Program ● Q&A
- 5. Better Decisions with Experimentation TomTsao Atlassian LESSONS FROM THE GROWTH OF ATLASSIAN’S EXPERIMENTATION TEAM Wes McCabe Optimizely
- 6. Digital Experience Optimization: DigitalProducts, Commerce & Campaigns Up to 5X Increase in Yield: Revenue, Share of wallet, Funnel conversion, Risk mitigation, Ops efficiency Partner of Choice: Work with leading global enterprises & “digital disruptors” including 26 of F100 OUR COMPANY Digital Experimentation Platform: Next gen “Test and Learn” System Replaces Digital Guesswork: with Evidence-based optimization Speeds Innovation & Optimization: Single platform for marketing & product teams Best in class stats & machine learning Consumer grade usability Enterprise program management & prof. services 2 OUR SOLUTION
- 7. 5 D I GI TA L E X P E R I M E N TAT I O N EMBRACE OR PERISH 10X MORE SCALE THE GREATEST ROI IN SaaS THE NEXT BIG THING Our Point of View
- 8. “Our success isa function of how many experiments we do per year, per month, per week, per day.” “Instead of saying ‘I have an idea,’ what if you said ‘I have a new hypothesis, let’s go test it.’ “Our company culture encourages experimentation and free flow of ideas.” 3 “One of the things I’m most proud of, and I think what is the key to our success, is this testing framework we’ve built.” Experimentation is the Next Great Business Transformation Jeff Bezos Larry Page Mark Zuckerberg Satya Nadella The Surprising Power of Online Experiments
- 9. 10x more experiments Consumer-gradeusability Open data integration Maximum yield of business value UX and feature-level experiments and personalization at every digital touch point Enterprise-wide management & governance Captures, governs and shares ideation, analyses & results Optimizely X Unlocks the Experimentation Best Practices of the World’s Greatest Digital Companies World’s most trusted outcomes BestinclassStatsEngine FasttimetoresultsviaML Accelerates digital innovation Speeds dev ops & deployment De-risks continuous feature delivery Ensures success of new features Unifying flagging & experiments enables controlled testing of new features while maintaining high performance Ideate Manage StoreGovern Analyze Share Open Data Integration Security & compliance Stats Engine Stats Accelerator Consumer- grade usability APIs & Developer Tools Feature Flags Open source SDKs X-Channel Full Stack Experimentation Personali- zation Recommen- dations Web Experimentation PRODUCTS COMMERCE CAMPAIGNS
- 10. B U SI N E S S V A L U E VELOCITY/VOLUME LEVEL 1 Executional Start LEVEL 2 Foundational Growth LEVEL 3 Cross-functional Advancement LEVEL 4 Operational Excellence LEVEL 5 Culture of Experimentation 10 Our Products and Services Take You on Your Experimentation Journey
- 11. TOM TSAO |HEAD OF ECOMMERCE | ATLASSIAN Better Decisions with Experimentation Lessons from the growth of Atlassian’s experimentation team
- 12. We believe behindevery great human achievement, there is a team. Our mission is to unleash the potential of every team. OUR MISSION
- 13.
- 14. Agenda History of experimentation Casestudies What’s next
- 15.
- 16. B U SI N E S S V A L U E VELOCITY/VOLUME LEVEL 1 Executional Start LEVEL 2 Foundational Growth LEVEL 3 Cross-functional Advancement LEVEL 4 Operational Excellence LEVEL 5 Culture of Experimentation 10 Optimizely’s Products and Services Take You on Your Experimentation Journey
- 17. B U SI N E S S V A L U E VELOCITY/VOLUME LEVEL 1 Executional Start LEVEL 2 Foundational Growth LEVEL 3 Cross-functional Advancement LEVEL 4 Operational Excellence LEVEL 5 Culture of Experimentation In Product Experiments Identified users Controlled environment Easier to prove value Evaluation Experiments Remove credit card for trial Web Experiments Anonymous visitors Multiple channels Campaign Experiments Optimizing spend throughout the funnel ATLASSIAN’S EXPERIMENTATION JOURNEY 2014 2015 2016 2017
- 18. Experiments help us makedecisions WHAT WE BELIEVE
- 19. Idea A vsB vs C Not HIPPO Roadmap decisions Feature “spoofing” Learnings Listening to our customers Learnings = Success + Failures Biz Decisions Remove credit card for trials Experiments Decisions
- 20.
- 21. PERSONALIZED HEADLINES DRIVECONVERSIONS Hypothesis Headlines personalized to visitors’ company can drive conversions Result +20% increase in Enterprise contact form submissions
- 22. INDUSTRY PERSONALIZATION DRIVESVISITORS TO RELEVANT CONTENT BB is marketed to “professional teams” control
- 23. INDUSTRY PERSONALIZATION DRIVESVISITORS TO RELEVANT CONTENT test Hypothesis Promoting BB UNLIMITED free repositories to EDU audience, would increase signups
- 24. INDUSTRY PERSONALIZATION DRIVESVISITORS TO RELEVANT CONTENT test Result INCONCLUSIVE In this use case, personalization did not move the metric Hypothesis Promoting BB UNLIMITED free repositories to EDU audience, would increase signups
- 25. An advanced casestudy Product Bundling
- 26. PRODUCT “BUNDLE” CHOICESFOR JIRA control Do visitors really know what they want? Confluence
- 27. PRODUCT “BUNDLE” CHOICESFOR JIRA Hypothesis We believed that the 3rd product bundle was chosen by most visitors, because they got “3” products for FREE Confluence test
- 28. PRODUCT “BUNDLE” CHOICESFOR JIRA test Hypothesis “Uninformed choice” bundling
- 29. +167 -46 JSW JSD PRODUCT“BUNDLE” CHOICES FOR JIRA -266 Confluence Signup CONVERSION
- 30. +167 -46 JSW JSD PRODUCT“BUNDLE” CHOICES FOR JIRA -266 Confluence Signup CONVERSION First Day ACTIVATION Week1 (Trial) +110 +50 -145 +66 +8 -71
- 31. +167 -46 JSW JSD PRODUCT“BUNDLE” CHOICES FOR JIRA -266 Confluence Signup CONVERSION First Day ACTIVATION Week1 (Trial) Week2 (after trial) RETENTION +110 +50 -145 +66 +8 -71 +151 +11 +4
- 32. PRODUCT “BUNDLE” CHOICESFOR JIRA Result Bundling more products yields higher initial sign- ups, but lower retention Hypothesis “Uninformed choice” bundling
- 33.
- 34.
- 35. Optimize for conversion LEARNINGS retention (theright metric for us)
- 36.
- 37. After Integrated Buyer ExperienceTeam with a culture of experimentation Before Fragmented, one-off experimentation “heroes” Exp team Web team Web Exp team
- 38. The right culture depends on havingthe right platform Optimizely
- 39. Why Optimizely? Key reasonswe chose Optimizely Better Performance Robust APIs Auto-Traffic Allocation X-Domain instrumentation DCP (dynamic customer profile) 1 2 3 4 5
- 40. B U SI N E S S V A L U E VELOCITY/VOLUME LEVEL 1 Executional Start LEVEL 2 Foundational Growth LEVEL 3 Cross-functional Advancement LEVEL 4 Operational Excellence LEVEL 5 Culture of Experimentation ATLASSIAN’S EXPERIMENTATION JOURNEY Culture of Experimentation Integrated Buyer Experience Team Optimizely X 2018 In Product Experiments Identified users Controlled environment Easier to prove value Evaluation Experiments Remove credit card for trial Web Experiments Anonymous visitors Multiple channels Campaign Experiments Optimizing spend throughout the funnel 2014 2015 2016 2017