Opticon 2015-Hypothesize, Analyze, Optimize: Using Hidden Insights to Perfect your Experiment Process
•
1 like•1,054 views
Featuring speakers from Rue La La, Princess Cruises Abby Lokshin, Product Manager, Mobile, Rue La La Abel Ramos, Product Owner, Shopping and Booking Web Scrum Team, Princess Cruises Testing is an iterative process, oftentimes with confusing or inconclusive results. However, every experiment has something to teach you, and all too often, critical information is left unnoticed! Learn how to look for insights hidden behind the numbers and maximize the utility of every experiment you run.
Recommended
Recommended
More Related Content
What's hot
What's hot (18)
Similar to Opticon 2015-Hypothesize, Analyze, Optimize: Using Hidden Insights to Perfect your Experiment Process
Similar to Opticon 2015-Hypothesize, Analyze, Optimize: Using Hidden Insights to Perfect your Experiment Process (20)
More from Optimizely
More from Optimizely (20)
Recently uploaded
Recently uploaded (20)
Opticon 2015-Hypothesize, Analyze, Optimize: Using Hidden Insights to Perfect your Experiment Process
- 1. #opticon2015 Hypothesize, Analyze, Optimize: UsingHiddenInsightstoPerfectyour ExperimentProcess Abby Lokshin Mobile Product Manager, Rue La La alokshin@ruelala.com Abel Ramos Product Manager, Ecommerce Princess Cruises @abelramos58 abelramos@princesscruises.com
- 2. #opticon2015 Iteration – The Key to your Testing Success Abel Ramos Product Manager, Ecommerce Princess Cruises @abelramos58 abelramos@princesscruises.com
- 3. “Every strike brings me closer to the next homerun.” – Babe Ruth
- 4. “We Failed”
- 6. The next greatest search UI ……or so we thought.
- 7. Oops, What Happened? Let the fun begin. Where do we start with these types of results? Time to regroup.
- 8. Introducing the Next Greatest Search UI…… no, really
- 9. OK, Someone Want to Interpret These Results? Could we be getting closer?
- 10. “Hey, what about our mobile traffic?” – Joseph Espana, Web Designer Princess Cruises
- 11. Understanding your Business Segments How is your Mobile Traffic affecting your results ?
- 12. “Life is a series of experiences, each one of which makes us bigger, even though sometimes it is hard to realize this.” – Henry Ford
- 13. Know your Audience Segment your website traffic for your tests.
- 14. What are the Obstacles Preventing Testing Success? How are your promotional efforts affecting your testing?
- 15. Review all of your Data Points Constantly Use all available data resources to help with your testing.
- 16. What Does the Future Hold for Testing at Princess Cruises? Are your testinggoals inline with larger company initiatives?
- 17. “We Won” Testing is an iterative process and it takes time.
- 18. #opticon2015 Thank you! Abel Ramos Product Manager, Ecommerce Princess Cruises @abelramos58 abelramos@princesscruises.com
- 19. #opticon2015 What should we test (again)? Thetestinglifecycle Abby Lokshin Mobile Product Manager, Rue La La alokshin@ruelala.com #opticon2015
- 20. RUE LA LA LEADING E-COMMERCE LIFESTYLE DESTINATION 15M+ MEMBERS NEW BOUTIQUES DAILY EXPIRE IN 2-3 DAYS AN OVERVIEW
- 21. RUE LA LA USER FLOW
- 22. Kicking off the testing process
- 23. TEST IDEATION WHAT HAPPENS MULTIPLE STAKEHOLDERS UNIQUE KPIS DISORGANIZED BACKLOG
- 24. Determine common themes to define primary goals
- 25. TESTING BUCKETS GROUPED BY PRIMARY GOALS
- 26. Prioritization
- 27. TEST PRIORITIZATION 60 SECOND BUSINESS CASE Coordination Effort Creative Effort Initial Development Effort Rollout Effort Strategic Alignment Revenue Impact Customer Experience Value Lowers Cost PM Priority RANK: HIGH, MEDIUM, LOW SCORE: 100 PTS. TOTAL + EFFORT BUSINESS VALUE
- 28. TEST PRIORITIZATION 60 SECOND BUSINESS CASE
- 29. An organized backlog leads to a stronger sprint
- 30. EXPIRING FREE SHIPPING TEST TESTING GOALS: • USER SEES QUICK TIP • USER ADDS TO CART • USER CHECKS OUT • PRIMARY GOAL: REVENUE RESULTS: 4.2%+ CONVERSION NEXT STEPS: ROLL OUT! QUICK TIP ON PRODUCT DETAIL LEVEL OF EFFORT: LOW BUSINESS VALUE: HIGH
- 31. What if the test is inconclusive?
- 32. MEN’S SUIT SHOP MIX & MATCH BOUTIQUE TESTING GOALS: • USER BUILDS A SUIT • USER ADDS TO CART • USER CHECKS OUT • PRIMARY GOAL: REVENUE RESULTS: NO CLEAR WINNER
- 33. What impacted the primary goal?
- 34. MEN’S SUIT SHOP SEVERAL ISSUES TOO MANY STEPS CONFUSING UI AUDIENCE TOO SMALL STATISTICALLY INSIGNIFICANT
- 35. MEN’S SUIT SHOP NEXT STEPS RE-RUN, WITH MODIFICATIONS: • FEMALE AUDIENCE • FEWER STEPS • LONGER TIME FRAME • MORE BOUTIQUES
- 36. What if converting isn’t the problem?
- 37. CATEGORY SORT TEST SHOPPING PREFERENCES TESTING GOALS: • USER INTERACTS WITH SORTING • USER CHECKS OUT • PRIMARY GOAL: REVENUE RESULTS: NO CLEAR WINNER VAR. A VAR. B
- 38. CATEGORY SORT TEST WHAT HAPPENED NO IMPACT ON THE WHOLE MANY CATEGORY COMBINATIONS CAN WE FIND ANY TRENDS? VAR. A VAR. B
- 39. CATEGORY SORT TEST INTERESTING TRENDS TREND IN VARIATION A: FEMALES RANK WOMEN’S FIRST • 4%+ IN $/VISIT TREND IN VARIATION B: MALES RANK MEN’S FIRST • 31%+ IN $/VISIT • 32%+ IN PURCHASES VAR. A VAR. B
- 40. CATEGORY SORT TEST INTERESTING TRENDS NEW TESTS: • DEFAULT MEN TO MEN’S SECTION • RE-OPEN TO LAST VIEWED CATEGORY VAR. A VAR. B
- 41. REMEMBER… TESTING IS NEVER ONE AND DONE CONSIDER THE DATA COLLECT FEEDBACK REMEMBER THE PRIMARY GOAL KEEP ITERATING
- 42. #opticon2015 Abby Lokshin Mobile Product Manager, Rue La La Thank you! alokshin@ruelala.com