Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

From iOS to TiVo: In-app Digital Experience Testing

44 views

Published on

Experimentation doesn’t end with the desktop experience. You can experiment everywhere — across all of your digital touchpoints, to drive acquisition, engagement and retention on every channel. In this session you’ll hear how two iconic brands, Fox Networks and Nike, have leveraged multiple channels to build their experimentation programs.

At Fox, experimentation involves testing every new feature on their iOS, TiVo and FireTV applications. For Nike, they want to democratize testing across the entire company. Today, they are empowering PMs, engineers, and marketers across the SNKRS and Training Apps, as well as the core Nike.com experience. Join this session to learn how these retail and media leaders have focused their efforts at pleasing their customers wherever they are.

Published in: Business
  • Be the first to comment

  • Be the first to like this

From iOS to TiVo: In-app Digital Experience Testing

  1. 1. Congratulations to our Outperform Award Winners! Most Dramatic Business Impact Most Customer- Obsessed Company Culture Most Transformative Innovation Most Inspiring Social Impact
  2. 2. 2  Welcome to Day 2 of Opticon!  Sessions today will be recorded and will be available after Opticon  Share your feedback and rate sessions on the mobile app
  3. 3. Will Close Director of Digital Experiments, Nike Alex Alwan Sr. Product Manager Optimization, Fox Networks From iOS to TiVo In-App Digital experience testing
  4. 4. Will Close Director of Digital Experiments, Nike Running Digital Experiments at Nike
  5. 5. 5 At Nike: Making Sport a Daily Habit  Iconic, inspirational shoe & sportswear brand.  Experimentation and athlete feedback has always been critical to Nike’s product development process (and still is today with LeBron, KD...) Founder Bill Bowerman was the Men’s Track Coach of the University of Oregon. He experimented with the design of footwear for athletes – building shoes to the exact specifications of his runners.
  6. 6. 6 2015: Will joined Nike, to start a “Test & Learn” culture  Sports brand, with a culture of winning.  One of the world’s strongest brands.  Strong intuition (art) has led decision-making at Nike for years.  With unprecedented success, globally.  Applying consumer data (science) was a new concept to digital product owners.  Analytics systems in particular were still immature.  Custom in-house, JavaScript testing solution on Nike.com.  4 person team was dedicated to running Nike.com tests only. With a technology background (Microsoft, Xbox, Dell) and specifically in product management, web development, Analytics and A/B Testing.
  7. 7. 7 Mobile Cart: Primary button (Checkout) is out of view Move Checkout button to top & make it “sticky”
  8. 8. 8 Mobile Cart: Primary button (Checkout) is out of view Move Checkout button to top & make it “sticky” % to Checkout = +5.5% lift Multi-item Conversion = +5.67% lift
  9. 9. 9 In Checkout: Using a simple consumer-value proposition Membership value proposition was lengthy marketing-speak. In Checkout, most consumers want to quickly Place their Order, not read. Selection of Membership Checkbox = +172% lift New NIKE Members = +30.3% lift
  10. 10. 10 Mobile PDP: How to best show multiple related products? Static 2x2 layout vs. carousel vs. 2x2 layout with pagination. Related Usage = +14.4% lift Revenues from Related = +97% lift
  11. 11. 11 Small, centralized team executing tests  4 team members.  Ran about 80-90 tests a year.  Technically constrained, front-end only.  Lots of low-hanging fruit.  Examples showed big impact. Using a home-grown Nike.com testing tool, we showed huge impacts from very small UX changes and leadership recognized the value of A/B testing.
  12. 12. “ ” macklemore The greats weren’t GREAT because at birth they could paint. The greats were great because they PAINT A LOT.
  13. 13. 13 2018: Nike deploys Optimizely X Full Stack to scale To enable experimentation from any connected, digital experience – Nike adopts the Full Stack capability across the enterprise.
  14. 14. 14  Enable Testing by All Product Teams  Critical to Product Planning  Informs key roadmap priorities  Launch + product KPIs increase  Features & experiments  Create 3-4 options, monitor results  All tests run by 1 small, central team  Behavior tagging = one of many goals  Engagement data = nice-to-have  Success = the product launches  We launch = Features only  We prototype & design = 1 version Before Today
  15. 15. 1515 No technical limits  Support for Client-side & Server-side experiments  Built for performance Easy to integrate  Lots of example code to leverage.  Developer documentation was easy to follow.  Integration was quick. Increase Test volume  16 Digital experience squads are integrated & running experiments.  Concurrent Test volume has grown 7x – from approx. 5 to 35. Changing norms  Growing organic interest in “Test & Learn” capability.  Leadership is requiring all teams to experiment.  North-star goal: All new UX changes in Nike Digital experiences are rolled out via Feature Tests. Test from anywhere Benefits to-date from federating A/B Test execution to all Nike digital experience squads, via Optimizely X Full Stack.
  16. 16. 16 Recent Test Results (1 of 3) Subhead
  17. 17. 17 Recent Test Results (1 of 3) Subhead
  18. 18. 18
  19. 19. 19
  20. 20. 20
  21. 21. 21
  22. 22. 22 Visibility is key  Creating visibility across the entire Nike program is helpful for leadership.  Many different teams have different ways to work. Standardizing where appropriate creates efficiencies.  Communicate in multiple formats & cadences (Email, Meetings, Slack..) Bold ideas matter  Test features that are going to be released anyway.  Good or Bad, Bold ideas help illustrate a learning – and get the audience’s attention (story).  Engage your Design & UX teams early and often – to take more risk. Build a community  Enable a broad group of “Test & Learn” teammates.  Collaborate frequently with this group & make sure they are supported.  Use the Voting system for Social feedback on tests.  Give them the Stage & Forum to Share. Growing up – culture insights During this maturation process, we recognized a few insights to “growing up”.
  23. 23. 23 Optimizely Teams + Slack  Slack integration for Optimizely Teams & Program Management is an easy method to make intake of ideas easy across the company.  /Optimizely.idea [Idea Name] Feature Tests vs. A/B  There are a lot of benefits to planning your Mobile Apps in a modular way – using Optimizely Features.  All Mobile App teams are embracing Feature Tests. Tell a Story  Don’t hide behind statistics and data points.  Testing is about validating a hypothesis & learning.  Share a valuable, insightful story. Simplify.  Use the thumb. Growing up – culture tips & tricks During this maturation process, we recognized a few insights to “growing up”.
  24. 24. Alex Alwan Sr. Product Manager, Optimization Running Digital Experiments at Fox
  25. 25. Alex Alwan Sr. Product Manager, Product Strategy Product Management is an art Experimentation is not a choice Experimentation is a tactic not a goal
  26. 26. Optimization Team From 1 to 2 to 3 to 6 to 4 to 2 people From “Let’s give AB testing a try” to “Why the heck we didn’t AB test that?” From “dunno man, that looks complicated” to “Just follow these steps”
  27. 27. Experimentation Purposes Improvement Testing Discovery Testing Feature Testing Great to achieve OKRs ”Splitting sign-up and payment screens will improve conversion rate” Great to understand what’s going on, then improve “What would happen if we remove Shows Favoriting from onboarding?” Great to validate/measure features’ impact “What’s the impact of live streaming on VOD viewership?”
  28. 28. Scaling Is Challenging It’s hard to change people New Skills = New Responsibilities http://millhoppervet.com Someone: “Why AB test? Just launch and see what happens” Me: Use success stories Convince the leaders first
  29. 29. Product Owner vs. Experiment Owner Responsible Accountable Consulted Informed Scaling Is Challenging
  30. 30. Scaling Is Challenging “But.. it’s nice and warm inside the box” Optimization is a tactic not a goal Focus on producing learnings not only results pbs.org
  31. 31. Scaling Is Challenging When there’s no big goal to inspire ipwatchdog.com You see the light at the end of the tunnel? Optimization should serve the company’s OKRs
  32. 32. Scaling Tips Start small but ask big questions Quantity first, quality later Integrate Data Tracking and A/B Testing systems Center Of Excellence is NOT a gatekeeper
  33. 33. From Web to Mobile/tvOS Web: Extensions Full Stack: Variables High Flexibility Specific Functionality
  34. 34. From Web to Mobile/tvOS More Content Starts From Redesigned Collection +31%
  35. 35. From Web to Mobile/tvOS Add-To-Favorite Rate +15% Push Opt-in Rate +60%
  36. 36. Tips For Mobile/tvOS Experimentation Create a useless variable for testing purposes Focus on generic capabilities to re-use Don’t break the app, build feature flags
  37. 37. Feature Flags Takes one out of three possible values: “ON”: Display feature/design to all users “OFF: Don’t display feature/design at all “OPTIMIZELY”: Wait for Optimizely's decision If Invalid, fall-back to “OFF”
  38. 38. What’s Next? Optimization As Part Of Product Strategy
  39. 39. Company OKR Product OKRProduct OKR Kye Result Kye Result Kye Result Kye Result Initiative Initiative Initiative Initiative Initiative Hypotheses Hypotheses Where do we need to go?
  40. 40. Company OKR Product OKRProduct OKR Key Result Key Result Key Result Key Result Initiative Initiative Initiative Initiative Initiative Hypotheses Hypotheses How do we know we’re getting there?
  41. 41. Company OKR Product OKRProduct OKR Key Result Key Result Key Result Key Result Initiative Initiative Initiative Initiative Initiative Experiments Experiments What will we do to get there?
  42. 42. Company OKR Product OKRProduct OKR Key Result Key Result Key Result Key Result Initiative Initiative Initiative Initiative Initiative Experiments ExperimentsExperiments Experiments How can we move fast?
  43. 43. From OKRs to Initiatives, and beyond!
  44. 44. How to define and plan initiatives? Find Areas of Improvement Design Variations Create KPIs Reports Define Hypotheses Plan Features as Experiments Track Key Results
  45. 45. Create KPIs Reports 2% Visit -> Subscribe 3 Months 90% Tipping Point 10% Trial Cancelation Visitor to Subscriber Funnel By Day/Hour Visitor to Subscriber Funnel By User Traits Key Result Quality Metrics Deeper Analysis
  46. 46. Find Areas of Improvement Traffic source User Traits User Behavior Drops and Opportunities Referral traffic from politics news on foxnews.com are 60% more likely to convert Safari Mobile users drop by 85% on funnel’s 2nd step Users who watch on >1 platform are 90% less likely to cancel subscription
  47. 47. Define Hypotheses User Behavior Users who watch on >1 platform are 90% less likely to cancel subscription Highlight available platforms on the marketing landing page Educate users on other available platforms through email/push messages Display banner ads Fix iOS deep links
  48. 48. Find Areas of Improvement Design Variations Create KPIs Reports Define Hypotheses Plan Features as Experiments Track Key Results Achieve Growth Hit OKRs Cultivate a data-driven culture Empower product managers to own their OKRs Improve efficiency Transform features into experiments Goals
  49. 49. Q&A
  50. 50. Thank you!
  51. 51.  Head to the Expo Hall for a quick break and say hello to our sponsors  Up next...learn how to measure experiment impact with Optimizely's Co-founder Pete Koomen!

×