Optimism Webinar 3: Lessons from Digital Leaders - How to go beyond A/B testing
•Download as PPTX, PDF•
3 likes•446 views
Learn how leading brands have taken their digital to new heights. Join this webinar for insight into the digital journeys of leading brands and uncover the steps they have taken to grow and evolve their digital experimentation programs. We’ll be diving into how these digital leaders got to where they are now, looking at their approaches to change in terms of team, culture, strategy and technology. And what kind of results that change has delivered for their businesses. In short, Webinar 3 provides a unique opportunity to learn from real life examples of digital experimentation in action. By discovering those key ‘wish I'd known’ moments, you can turbocharge your own progress. Save time, money and headaches along the way. And begin your 2019 with a renewed sense of optimism. Join Jason G'Sell, our Senior Training Consultant, in this session to: - Hear the digital transformation journeys of leading brands. - Learn from real life examples of challenges they've had to overcome in teams, culture, strategy and technology. - Discover actionable insight including key takeaways and "wish they'd have known" moments. Webinar link: https://www.optimizely.com/resources/how-to-go-beyond-ab-testing/ Optimism Series webinar link: https://www.optimizely.com/optimism-webinar-series/
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Optimism Webinar 3: Lessons from Digital Leaders - How to go beyond A/B testing
Similar to Optimism Webinar 3: Lessons from Digital Leaders - How to go beyond A/B testing (20)
More from Optimizely
More from Optimizely (20)
Recently uploaded
Recently uploaded (20)
Optimism Webinar 3: Lessons from Digital Leaders - How to go beyond A/B testing
- 1. Lessons from Digital Leaders - How to Go Beyond A/B Testing Jason G’Sell Senior Training Consultant
- 2. Jason has been a member of Optimizely’s Customer Success, Education, and Services teams for over three years, enabling hundreds of businesses to build their experimentation programs, increase their testing and personalization skills, and derive meaningful insights from their experiment data. OPTIMIZELYSERVICES2018 Jason G’Sell Sr. Training Consultant, EMEA Opticon Training Day Managed five training workshops tailored to all skill levels delivered at the annual Opticon conference. Launch Customer Education in Europe Expanded Optimizely Customer Education training and support from US to EMEA region Selected experience Customer s Customized Training Programs Build and deliver bespoke virtual and in- person training programs for large enterprises to effectively onboard hundreds of teams globally Education & Certification University of Notre Dame, USA Bachelor of Arts in Sociology, Minor in Gender Studies Association of Talent Development ATD Training Certificate Sr. Training Consultant, Optimizely Training customers on high-value skills across Web Experimentation, Personalization, and Full Stack Launch Manager, Optimizely Onboarding Web Experimentation customers at- scale, helping them achieve value as quickly as possible Customer Success Manager – Implementation, ClearSlide Developed train-the-trainer program to enable onboarding for thousands of sales reps nationally Areas of expertise & industry
- 3. 3 Webinar will be recorded and shared along with the slides Q&A – Use your control panel to ask questions Housekeeping
- 4. Where you are Where you need to go How to get there
- 5. 5 Digital experimentation is the reliable process of delivering winning digital experiences without guesswork or risk.
- 6. 6 Improve ROI on digital innovations and investments Test everything and remove guesswork Gain insight into what your customers really want Move fast and scale up your development process Why Digital Experimentation?
- 7. 7 What impact can Digital Experimentation make? “Which new user experiences will maximize our transition to a digital acquisition model?” Increase engagement with video content on iPlayer 50% increase in ‘next episode’ views Improve customer experience by creating a Digital Center of Excellence Seven figure saving on call center costs Increase Instant Ink subscription service signs ups 37% increase in enrollment
- 8. 8 Where are you on your journey?
- 10. 10 BUSINESS VALUE VELOCITY/VOLUME LEVEL 1 Executional Start LEVEL 2 Foundational Growth LEVEL 3 Cross-Functional Advancement LEVEL 4 Operational Excellence LEVEL 5 Culture of Experimentation
- 12. 12
- 13. 13 BUSINESS VALUE VELOCITY/VOLUME LEVEL 1 Executional Start LEVEL 2 Foundational Growth LEVEL 3 Cross-Functional Advancement LEVEL 4 Operational Excellence LEVEL 5 Culture of Experimentation
- 14. 14 Bring on Testing Platform Implement Process Program KPI’s
- 15. 15 Velocity Variation Count Duration Stat Sig Rate Win Rate Goal CountPerformance Benchmarks Complexity Benchmarks OPERATIONAL BENCHMARKS
- 16. 16 Testing Product Development Program Vision Educate Stakeholders
- 17. 17 Success was due to the right foundation of team, ownership, and success criteria. Benefit from experimentation is best realized when it’s anchored to strategic goals Growing along the maturity curve gets more difficult because of increasing needs. Remember it’s a long haul!Ramkumar Ravichandran Director of Analytics, Experimentation & Data Science Visa
- 18. 18
- 19. 19 BUSINESS VALUE VELOCITY/VOLUME LEVEL 1 Executional Start LEVEL 2 Foundational Growth LEVEL 3 Cross-Functional Advancement LEVEL 4 Operational Excellence LEVEL 5 Culture of Experimentation
- 20. 20 PHASE 1: One Man Show
- 21. 21 “Let’s try Trustpilot on the homepage”
- 23. 23 “User’s are overwhelmed by choice. Let’s highlight for them the best combination of price and convenience.”
- 24. 24 PHASE 3: Center of Excellence
- 25. 25 Velocity Variation Count Duration Stat Sig Rate Win Rate Goal CountPerformance Benchmarks Complexity Benchmarks OPERATIONAL BENCHMARKS
- 26. 26 Bridge the gaps between CRO and Product Management Move from owning products to owning problems. Prioritize using a model Increase velocityBenjamin Groenewoud Product Owner & CRO Manager Travix
- 27. 27
- 28. 28 BUSINESS VALUE VELOCITY/VOLUME LEVEL 1 Executional Start LEVEL 2 Foundational Growth LEVEL 3 Cross-Functional Advancement LEVEL 4 Operational Excellence LEVEL 5 Culture of Experimentation
- 29. 29
- 30. 30 “AFTER addressing basic site improvements, A/B Testing becomes less of a broad stroke tactic and more focused on user level and behavioral conditions.” David Kornfield Marketing Manager Brooks Running
- 31. 31
- 32. 32
- 33. 33 Takeaways Optimize for end-to-end experiences, not just clicks. Identify a meaningful segment, understand their behavior by asking “what are they trying to do? What makes them different? Where is the experience falling short? Measure the impact!
- 34. 34
- 35. 35
- 36. 36 BUSINESS VALUE VELOCITY/VOLUME LEVEL 1 Executional Start LEVEL 2 Foundational Growth LEVEL 3 Cross-Functional Advancement LEVEL 4 Operational Excellence LEVEL 5 Culture of Experimentation
- 37. Democratize ideation and empower your entire team to participate Make the experimentation process more accessible and transparent Improve your program workflow with new tools for collaboration Optimizely Program Management
- 38. 38 “PREVIOUSLY we had no central location to track and analyse the results of experiments. Thanks for Program Management, it’s not much easier for teams to learn from each other.” Erin McLaine, Director of Experimentation & Global Product Delivery Hero
- 40. 40 Webinar Takeaways Use the Maturity Model to evaluate where your program has the biggest opportunity for improvement – Team, Culture, Strategy, or Technology Decide which metrics you’ll use as goals for your program and regularly review them. Focus on velocity! If you can increase the number of experiments you run, everything else will follow. Communicate, communicate, communicate. Tell stakeholders where you are in your maturity and where you can go on the journey.
- 41. 41 BUSINESS VALUE VELOCITY/VOLUME LEVEL 1 Executional Start LEVEL 2 Foundational Growth LEVEL 3 Cross-Functional Advancement LEVEL 4 Operational Excellence LEVEL 5 Culture of Experimentation ?
- 43. 0 1 0 0 2 0 0 3 0 0 4 0 0 5 0 0 +56%5 0 1 5 0 2 0 1 2 2 0 1 3 2 0 1 4 2 0 1 5 2 0 1 6 2 0 1 7 Fortune 1000
- 44. 1 5 0 0 1 0 0 2 0 0 3 0 0 4 0 0 5 0 0 6 0 0 7 0 0 Fortune 1000 Culture of Experimentation Index 2 0 1 2 2 0 1 3 2 0 1 4 2 0 1 5 2 0 1 6 2 0 1 7 +756% 5 0 +56%
- 45. “Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day…” J E F F B E Z O S
- 47. 47 Start tomorrow with… Complete the Maturity Model evaluation at Optimizely.com/maturity-model Establish your program benchmark measures Listen to the first two installments of the Optimism series Set roadmap variables that create unique measures Review regularity your progress
- 48. Q&A
- 49. Thank you!
Editor's Notes
- I work with European companies such as Yoox, Inditex, and JustEat
- In our first episode, we took you through the steps on your journey to implementing a high-performance program that unleashes the full power of experimentation. Today we’re going to show examples from four leading companies that have managed to climb the maturity ladder, share with you the challenges they fought in getting there, and the strategies they used to push ever higher. But first – a quick explanation of what digital experimentation is and how it can bring value to your company…
- Digital experimentation allows you to gain the reliable insight you need to continually fine tune every aspect of the customer journey. It’s becoming the way successful companies do business: not only mega brands like Amazon, HP, VISA or Sky, but also the numerous startups for whom experimentation is an intrinsic part of their DNA. By replacing the guesswork with hard data, you consistently deploy compelling experiences, maintain a competitive advantage and maximize ROI from your digital investments
- Improve ROI: You will only invest in innovations you know will work, rather than waste budget on those that don’t Test everything, remove guesswork: Make decisions based on hard facts. It allows you to test big ideas but also drive progress through continuous incremental Gain insight in what your customers really want: Not only know what customers do online, but also improve their digital experiences which will go on to impact other things like customer experience and loyalty Move fast and scale up: Digital experimentation allows for a faster development process from idea to roll out, not only in your web experiences but across your entire technology stack!
- BBC The BBC have a catch-up service for television programs and radio shows called called iPlayer that allows people to get up-to-date on television and radio programs across devices They had a goal to increase the amount of time spend and engagement with video content on iPlayer Using Digital Experimentation, they tested autoplaying the next episode on iPlayer and were able to increase engagement by 50% Sky.com Sky begun investing heavily in digital innovation for Sky.com in 2015 and their focus in this area has seen them build a Digital Center of Excellence Their focus on providing an excellent customer experience saw them run an experiment that sought to improve the experience with the call center As a result, they were able to reduce annual call center figures by seven figures HP Looking to increase the number of sign-ups to their Instant Ink subscription service They tested different enrolement offers, including offereubf a free trial and positioning the service as a printer feature They were able to increase enrollment to their subscription service by 37% using Digital Experimentation In fact, HP too have built a Center of Excellence that saw them run almost 500 experiment campaigns and has driven an incremental $21 million in revenue with Optimizely You can find more details on these and other stories on the results of Digital Experimentation on the customer and resources pages of our website
- To help you understand where you are in your experimentation journey, we created an online maturity model evaluation that we launched at Opticon last year - this is a brief online evaluation that measures your team across 4 categories that I spoke about before: Team – what types of roles do you have working with your experimentation program and how much time do they dedicate to experimentation Culture – how widely is experimentation embraced within your organization Strategy – what types of experiments do you run and how complex are those experiments Technology – what supporting technologies do you use with your experimentation program
- And based on your responses, it places you along our maturity curve that as you saw yesterday has 5 levels: Level 1 for executional start – this is the first stage when you are just getting started and harnessing initial learnings of experimentation Level 2 foundational growth – is about taking those initial learnings and building a solid foundation with a team and clear strategy Level 3 cross-functional advancement – is about taking those learnings from one team and applying them to others across marketing and product Level 4 operational excellence – is about building on your program so that it is operationally strong and finely tuned in each category Level 5 culture of experimentation – which we’ve talked about before You may be wondering, where do our customers fall on this curve. We found that most companies are level 1 or 2. That's a good thing, because you're still way ahead of the curve - and there's a ton of room for growth. You can also be at different levels for each of the four categories – team, culture, strategy, and technology
- I will take you through Visa’s journey and show you how their growth is indicative of a company in Level 1 moving to Level 3 by working on team, culture, strategy, and technology!
- Launched the product in 2014 – first started just by collecting data – what are the strengths and pain points? They then had levers they could pull to improve the UX and conversion rate. They then brought on Optimizely in 2015. First pivotal decision. How? Crafting the right story, approaching the right stakeholders with clear success criteria, gathering executive support Second: They then started implementing process: A/B Kanban, Iterating into a team framework and process set up for selecting right experiments, setting them up correctly/quickly,
- Third: Implementing Program KPI’s (# of tests/month, % successful tests, % learning tests, % workaround / bug fix tests Program ROI time from ideation to deployment time from test outcome to product implementation
- Fourth: Integrate experimentation into product development and release cycle Fifth: Education stakeholders on full potential and long term program vision - Sharing with stakeholders where they are along the maturity curve. Continuously engage stakeholders to show that a lot more is possible and should be done Sixth: regular communication up to help with mindshare This entire journey took 3 years! In 2017 they educated on full potential and vision for experimentation. Evangelised! Made experimentation a critical part of each product manager’s toolbox
- Started as a one-man show - One person doing everything - No structured process - Basic tests - A/B testing is a small marketing project - Focus on fast money, but not much value - No understanding of customer behavior Then on to a true CRO ”A-team” – Multidisciplinary team of 7-8 people - Using defined hypothesis frameworks - Continuous communication about results - Incolvement at the c-level - Not really integrated in overall product strategy - Products still developed and shipped then experimented Then to making their product organization true insights-driven Program driven by three KPI’s: Velocity, Quality, Value ($ per test) Optimization “lighthouse team” aka Center of Excellence Insights from experiments drive their roadmap priorities Testing everything: UX, API versions, algorithms
- Travix moved from Level 1 to Level 3 by focusing on team, strategy, technology, and culture! And based on your responses, it places you along our maturity curve that as you saw yesterday has 5 levels: Level 1 for executional start – this is the first stage when you are just getting started and harnessing initial learnings of experimentation Level 2 foundational growth – is about taking those initial learnings and building a solid foundation with a team and clear strategy Level 3 cross-functional advancement – is about taking those learnings from one team and applying them to others across marketing and product Level 4 operational excellence – is about building on your program so that it is operationally strong and finely tuned in each category Level 5 culture of experimentation – which we’ve talked about before You may be wondering, where do our customers fall on this curve
- Started as a one-man show - One person doing everything - No structured process - Basic tests – shifting USP’s and changing button colors - A/B testing is a small marketing project, considered “icing on the cake” (non-essential) - Focus on fast money, but not much value - No understanding of customer behavior
- Started as a one-man show - One person doing everything - No structured process - Basic tests – shifting USP’s and changing button colors - A/B testing is a small marketing project, considered “icing on the cake” (non-essential) - Focus on fast money, but not much value - No understanding of customer behavior
- Needed More skills to make an effective CRO team - Psychologist - UX Designer - Developer - Product Owner They proved a business case and proved that increasing their experiment velocity would really pay off. One year later, on to a true CRO ”A-team” – Multidisciplinary team of 7-8 people - Started with a big pre-study using all of the data they had: GA, NPS, support call center, Hot Jar, Market research - Using defined hypothesis frameworks - Continuous communication about results - Incolvement at the c-level Then to making their product organization true insights-driven Program driven by three KPI’s: Velocity, Quality, Value ($ per test) Optimization “lighthouse team” aka Center of Excellence Insights from experiments drive their roadmap priorities Testing everything: UX, API versions, algorithms
- Starting to learn the drivers of user behavior Landing page: - Goal directed Search: - Complexity Vaoidance - Automony Checkout - Price perception - Motivation - Uncertainty BUT - Not really integrated in overall product strategy - Products still developed and shipped then experimented CRO was at the end of the product development process and they wanted to move it further up (similar to Visa)
- CRO Driving product strategy Customer Intelligence driving product strategy Then to making their product organization true insights-driven Program driven by three KPI’s: Velocity, Quality, Value ($ per test) Insights from experiments drive their roadmap priorities Dedicated analysts per product domain Focus on making implementing tests and winners easier Testing everything: UX, API versions, algorithms
- Then to making their product organization true insights-driven Program driven by three KPI’s: Velocity Quality Value ($ per test) Optimization “lighthouse team” aka Center of Excellence Insights from experiments drive their roadmap priorities Testing everything: UX, API versions, algorithms
- Output: product roadmap Outcomes: experiment velocity experiment quality experiment value Experimentation is not front-end/web/marketing only. You need to take it to product. This also applies to PM’s. Your customer journey is more than just a website.
- Moving beyond optimizing conversion rate
- Brooks running moved from Level 1 to Level 2 by focusing on strategy and technology And based on your responses, it places you along our maturity curve that as you saw yesterday has 5 levels: Level 1 for executional start – this is the first stage when you are just getting started and harnessing initial learnings of experimentation Level 2 foundational growth – is about taking those initial learnings and building a solid foundation with a team and clear strategy Level 3 cross-functional advancement – is about taking those learnings from one team and applying them to others across marketing and product Level 4 operational excellence – is about building on your program so that it is operationally strong and finely tuned in each category Level 5 culture of experimentation – which we’ve talked about before You may be wondering, where do our customers fall on this curve
- After picking the low hanging fruit, what do you do next?
- Moving beyond optimizing conversion rate: “You need to dig into your performance data and address specific needs of customer cohorts.” You’ve picked the low hanging fruit. Now what?
- For the test segment, return rate declined nearly 80%. Not only did Brooks have more product in stock, but customers appreciated the personal touch, with 88% saying they appreciated the customer service offer to help. Conversion rate increased slightly by 2% which the team believes is due to less webrooming—when shoppers look up products online, then choose to buy them in a physical store—since the customer service reps were able to address size concerns upfront. In line with their expectations, AOV decreased, yet add-on rate of ancillary products increased, perhaps due to purchasers not having to wait on return credit to complete their running gear collection. Brooks also measured the cost of customer service time — is having humans answer shoppers’ questions more expensive than the cost of returns? They found the answer to be no. Customer service engagements averaged five minutes, which was less expensive than the hard costs of fulfillment and return shipping on every returned order.
- Nearly 30 brands globally!
- Delivery Hero moved from Level 2 to Level 3 by focusing on team, technology, and culture Challenge: How do we scale experimentation within our organization? We need a mechanism And based on your responses, it places you along our maturity curve that as you saw yesterday has 5 levels: Level 1 for executional start – this is the first stage when you are just getting started and harnessing initial learnings of experimentation Level 2 foundational growth – is about taking those initial learnings and building a solid foundation with a team and clear strategy Level 3 cross-functional advancement – is about taking those learnings from one team and applying them to others across marketing and product Level 4 operational excellence – is about building on your program so that it is operationally strong and finely tuned in each category Level 5 culture of experimentation – which we’ve talked about before You may be wondering, where do our customers fall on this curve
- To promote the exchange of knowledge and experimentation with its employees worldwide, the teams use Program Management: This centralised management interface enables all teams – both worldwide and at the headquarters in Berlin – to gain an overview of the experimentation programme. All employees can see which experiments the others are currently working on, what worked for them in the past and what did not. The idea behind it is, "if a test works well in South America, it might also work in Kuwait.”
- In the summer of 2017, Delivery Hero rolled out Optimizely as its experimentation platform across all of the more than 40 markets. Next, the team wants to work on driving their "culture of experimentation". It is about promoting data-driven optimization and ensuring that teams share results and learn from each other’s experiences. Optimizely "was not designed especially for developers, product managers or marketing, but actually for everyone within the enterprise."
- So, we’ve seen strategies that different companies have used to increase their maturity. But what does level 5 look like? Which companies are there?
- [DAN] In fact, today’s tech giants have figured out how to weave Bill Bowerman’s process into their DNA. They use experimentation to guide their decision-making at every level. The results have been astounding.
- [DAN] The Fortune 1000 have collectively grown their market cap by 56% over the last five years. That’s a modest return for five years.
- [DAN] Companies that embody a Culture of Experimentation have grown their market cap by 756% over the last five years. We call this the Culture of Experimentation Index. Why have these companies done so well? Because their leadership understands the connection between experimentation and growth.
- [DAN] So celebrating wins can help spread the word about the power of experimentation. But celebrating wins is kind of obvious. What’s less obvious is that it’s just as important to celebrate the losses. Jeff Bezos, founder and CEO of Amazon, personally embodies this habit. [CLICK]