[Elite Camp 2016] Craig Sullivan - Elite Camp Summary Session
•
1 like•5,133 views
Summing up what we just learned at this conference.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (6)
Similar to [Elite Camp 2016] Craig Sullivan - Elite Camp Summary Session
Similar to [Elite Camp 2016] Craig Sullivan - Elite Camp Summary Session (20)
More from CXL
More from CXL (20)
Recently uploaded
Recently uploaded (20)
[Elite Camp 2016] Craig Sullivan - Elite Camp Summary Session
- 1. The AB Testing Hype Cycle Budapest - Netbooster – 11 Nov 2015 @OptimiseOrDie It’s a Wrap! Elite Camp 2016
- 2. How was your trip to Estonia, my dear? @OptimiseOrDie
- 4. Peep Laja @OptimiseOrDie • Peeeeep - A car sound • Piip - Estonian Grocery Store • Pap - Common blog problem • Peahp - Texan pipe • Peppie - Food for small dogs • Payp - Micro payment system
- 5. World CRO Dude Dickens Character PEP PIP
- 6. The solution? Less Vowels! @OptimiseOrDie • Uusaastaöövastuvõtuhommikuidüll • Sünnipäevanädalalõpupeopärastlõunaväsimus • Peep needs to lose a fucking vowel! Pe¨p
- 7. • I love you Estonia • Funny opener x 2 • Summary Slides • Tool Slides • Wrap • See you next year! Roundup Outline @OptimiseOrDie
- 8. • I love you Estonia • Funny opener x 2 • Summary Slides • Tool Slides • Wrap • See you next year! Roundup Outline @OptimiseOrDie • I love you Estonia • Funny opener x 2 • Summary Slides • Tool Slides • Wrap • See you next year!
- 9. • I love you Estonia • Funny opener x 2 • Summary Slides • Tool Slides • Wrap • See you next year! 1. This is F****ing Hard! @OptimiseOrDie • After 15 years, I see no easy shortcuts • The tools, knowledge and techniques are better • The foundations are still the same • It takes dedication, perspiration, determination and skill! “The boundary layer between the mind of the customer and your product is where all the best stuff is always found."
- 10. • I love you Estonia • Funny opener x 2 • Summary Slides • Tool Slides • Wrap • See you next year! 2. Humility and Ignorance @OptimiseOrDie • Paul Rouke talked about this • The best optimisers show humility and empathy • This is one of the most important traits • Everyone thinks they know the answer • It’s good to admit you don’t know! • Acquire a better quality of ignorance…
- 11. • Your customers are not the same • Your site is not the same • Your advertising and traffic aren’t the same • Your UX is not the same • Your X-Device Mix is not the same • Use them to inform or suggest approaches • They’re like the picture on meal packets • “Serving Suggestion Only” • There are obvious BEST PRACTICES but these are usually in the category of ‘bugs’ or ‘UX problems’ or ‘D’oh’ Just fix that now! We have no clue Reading Best Practice Bulls**t @OptimiseOrDie
- 12. • Further reading: bit.ly/BestPUnicorns bit.ly/BPSuccess bit.ly/BestpFAIL Just fix that now! We have no clue Reading Best Practice Bulls**t @OptimiseOrDie
- 13. • I love you Estonia • Funny opener x 2 • Summary Slides • Tool Slides • Wrap • See you next year! Better Quality Ignorance @OptimiseOrDie • Let me explain!
- 14. • I love you Estonia • Funny opener x 2 • Summary Slides • Tool Slides • Wrap • See you next year! 3. Foundations of Data @OptimiseOrDie • Building layers of bullshit starts with collection! • If you get the fundamentals wrong, all the metrics, reports and dashboards will be broken • 3 years, 127 configurations = 122 were broken • Even Google’s GA config was completely biased • Get a health check! • Speak to me about this service • Get the basics right, then worry about fripperies!
- 15. • I love you Estonia • Funny opener x 2 • Summary Slides • Tool Slides • Wrap • See you next year! 4. Organisational Design @OptimiseOrDie • Recently working on Organisational Design • Working with several large organisations • Designing scalable growth models and teams • Why does it nearly always fail? • Because the entire method of building, improving and growing customer use of products HAS to change • Without this, it’s another layer of bullshit • Fundamental reworking is required • Andre, the FT and Rockboost!
- 16. • I love you Estonia • Funny opener x 2 • Summary Slides • Tool Slides • Wrap • See you next year! 4. Organisational Design @OptimiseOrDie • Finnair slide
- 17. • I love you Estonia • Funny opener x 2 • Summary Slides • Tool Slides • Wrap • See you next year! 5. Failure is Great! @OptimiseOrDie • Embrace failure! • Velocity (cycle time) and Volume (complete per month) are far more important than success rate • 20% success rate on 1000 tests far better than 80% on 100! • If you can’t handle being proved wrong or you can’t deal with counter intuitive results, change careers! • Failure is an integral part of this work • Everything around you & Scientific Experiments
- 18. • I love you Estonia • Funny opener x 2 • Summary Slides • Tool Slides • Wrap • See you next year! 6. Mature your metrics! @OptimiseOrDie • This came up repeatedly… • Moving along the maturity curve from short term metrics towards longer term value • For example, not just measuring conversion but repeat business, customer satisfaction scores, lifetime value • If you are not continually challenging, maturing and improving your core metrics, you are standing still • The impact of product change or testing may be deeper and longer term than you realise!
- 19. • I love you Estonia • Funny opener x 2 • Summary Slides • Tool Slides • Wrap • See you next year! 7. Prioritisation Scoring @OptimiseOrDie • If you aren’t prioritising, you’re wasting time • If you are doing discovery for CRO properly, you will expose hundreds of issues, bugs, places to test and things to try! • You have to prioritise around the effort vs. the potential reward • There are many scoring systems (PIE, ICE, TIR) but I find that the best design is client specific
- 20. • I love you Estonia • Funny opener x 2 • Summary Slides • Tool Slides • Wrap • See you next year! 7. Prioritisation Scoring @OptimiseOrDie All my clients have core dimensions of: • Effort / Cost / Difficulty • Potential Uplift / Reward / ROI Secondary dimensions can include: • Time to market • Risk of change • Complexity or Dependency
- 21. • I love you Estonia • Funny opener x 2 • Summary Slides • Tool Slides • Wrap • See you next year! 8. Quant & Qual balance @OptimiseOrDie • You need Qualitative and Quantitative methods • Data is great but it tells you what’s happening, where it’s happening and how big the issue is. • It will not tell you the why • You need qualitative tools (polls, surveys, feedback, UX research, interviews, session replay etc.) • Arrogance comes from thinking you know the solution before you understand the problem
- 22. • I love you Estonia • Funny opener x 2 • Summary Slides • Tool Slides • Wrap • See you next year! 9. Motivation vs. Usability @OptimiseOrDie • Usability vs. Motivation? • Seems fairly clear – work the motivation layer! • There are 3 areas you can work on though: • Motivation • Usability • Stuff that’s completely broken?
- 23. @OptimiseOrDie • Clothing Brand 1M per month, 2 bugs • European Travel site 2M per month, 13 bugs • Telecoms operator 1.4M per month, 3 bugs • Travel Aggregator 0.6M per month, 5 bugs • Event and Ticket Site 2.5M per month, 11 bugs • Total developer time to fix ALL of these = < 5 days! • At a cost of £8000 to find and fix, the ROI was 91,150% • In many cases, savings larger than the ENTIRE IT budget How Much do these bugs cost?
- 24. • I love you Estonia • Funny opener x 2 • Summary Slides • Tool Slides • Wrap • See you next year! 10. You Need a F***ING PROCESS @OptimiseOrDie • Andre and others talked about this • You need a process, methodology and org structure • Tools and Enthusiasm are not enough • You need to design your own • Use an existing methodology and then customise • Find your own way of working – beware of being a slave! • Methodologies are good slaves but bad masters!
- 25. • I love you Estonia • Funny opener x 2 • Summary Slides • Tool Slides • Wrap • See you next year! 11. Psychology, Emotion, Copywriting @OptimiseOrDie • Thank you Amy! • This is one of the secrets of CRO (not to Amy) • You MUST have copywriting, persuasion and psychology skills embedded in or available to CRO teams • Over 65% of my successes involve copy changes • Words are the lens through which your brand, product, form, interaction or message is read. • Your brand IS this copy and interactive element
- 26. • I love you Estonia • Funny opener x 2 • Summary Slides • Tool Slides • Wrap • See you next year! 12. Cross Device Optimisation (XDO) @OptimiseOrDie • One of my secrets that hardly anyone else uses
- 27. 12. Cross Device Experience Map MOBILE iOS Resolution (Model) Android Mobile Device Info Windows Mobile Device Info TABLET iOSNOTHING Android Mobile Device Info Windows Resolution (Model) DESKTOP Windows Mac Linux Browser Version Browser version Browser version
- 28. 12. CRO Grids • An automated tool to pull a CRO grid from any site • Identify key segments that are over and under performing • Exploit Device and Browser knowledge to stem leakage • Create dynamic ‘on the fly’ funnels using events, regex, segments • A multi-dimensional automated funnel generator • Uses 10 years of knowledge optimising sites • Tuned and improved by all my optimisation buddies! • Alpha is live – www.profitgrid.io • Don’t complain about UX – tell me about the data! • Any questions, feedback – email me sullivac@gmail.com
- 29. 5. We’ve built a tool to do this!
- 31. • Growth Optimisation is hard work Get used to it! • Humility and Ignorance Be humble and accept your ignorance • Data Foundations Regular health checks & collection checks • Organisational Design If it isn’t changing you, it isn’t working • Failure is good! Embrace and love failure • Mature your metrics Signals for long term value and profit Summary @OptimiseOrDie
- 32. • Always be Prioritising Without scoring, you are burning rubber • Qual and Quant balance A balance of where, how big and why! • Motivation vs. Usability Functionality and Device comes FIRST! • You need a PROCESS Borrow, Steal, Adapt & Optimise this • Psychology, Emotion, Copywriting Copywriting is vital for CRO • Cross Device Optimisation Read my article Try out our free tool for pulling CRO summaries from any site: www.profitgrid.io Summary @OptimiseOrDie
- 33. @OptimiseOrDie
- 34. Hacking efficiency/time/ delight/productivity/fun Tell your friends good Lovable Empathetic Delightful Self actualisation Life changing Rewiring FUNCTIONAL ACCESSIBLE USABLE USEFUL PERSUASIVE COMPELLING TRANSFORMATIONAL Device Browser Mobile Audience Performance Accessibility Platforms Comprehension Affordance Simplicity Relevance Clarity Context Utility Tasks Goals Steps Completion Progression Resonant Authentic Compelling Consistent Benefits Value Trust Social Integration and Proof Habits Nudges Addictions Lifecycle Promotions Personalisation Brand Belonging Trusted Emotive @OptimiseOrDie
- 35. 35
- 37. WE’RE ALL WINGING IT
- 39. Thank You! Email me sullivac@gmail.com Slides Will be sent to you! LinkedIn linkd.in/pvrg14
- 42. The AB Testing Hype Cycle Budapest - Netbooster – 11 Nov 2015 @OptimiseOrDie It’s a Wrap! #CH2015
- 43. What About the Organisation? @OptimiseOrDie
- 44. @OptimiseOrDie Burn Down the Silos • Non agile, non iterative design • Silos work on product separately • No ‘One Team’ per product/theme • Large teams, unwieldy coordination • Pass the product around • More PMs and BAs than a conference • Endless sucking signoff • AB testing done the same way!
- 45. @OptimiseOrDie FT, Booking, Skyscanner, ABN, Expedia + more • Small teams (6-15) with direct access to publish • Ability to set and get metrics data directly • Tools, Autonomy, Lack of interference • No Project Managers or Business Analysts • Business defines ‘outcomes’ – teams deliver • No long signoff chain • No pesky meddling fools • 18 Month projects over budget?
- 46. @OptimiseOrDie 10. FT Example • 100s of releases a day! • Iterative MVP approach • Launch as alpha, beta, pilot, phased rollout • Like getting in a shower • Read more at labs.ft.com
- 47. CRO 47
- 48. @OptimiseOrDie 10. Positive Attributes • Rapid, Iterative, User Centred & Agile Design. No Silos. • Small empowered autonomous teams • Polymaths and Overlap • Toolkit & Analytics investment • Persuasive copywriting & Psychology • Great Testing & Optimisation Tools
- 49. @OptimiseOrDie • Agile, Lean, Iterative x silo teams • Ability to get and set metrics • Autonomy, Control, Velocity • Iterative MVP approach • Work on outcomes, not features ! Burn Down the Silos!
- 50. 50 Time ROI
- 51. 51
- 52. Summary Grid @OptimiseOrDie Do Your Research Prioritise Hypothesise • Heuristic analysis • Usability testing • Heatmap tools • Session replay • Web Analytics • Qualitative surveys • Polls • Chat & Text analysis • Interviews • Device compatibility • Performance • Use TIR or PIE • Prioritise around opportunity vs. cost • Estimate political difficulty or friction as well as resource costs • There is no best practice on prioritisation • Continually re-prioritise against new discoveries • Use the collective brain • Assemble and circulate your research • Set clear goals and have a process for sketching • Sketch the Hypothesis • Build W/F or mockup • Set ONE primary (possibly one secondary) success KPI
- 53. Summary Grid @OptimiseOrDie Pre Flight Checks Run Analyse & Report • QA testing • Device & Browser compatibility • Analytics tracking • Ecommerce tracking • Launch • Set your test length with a calculator • Run it, stop it, analyse it • Don’t run over time unless your sample is smaller than expected • Learn about the statistics • Watch tests closely! • Wash up analysis • Share findings widely • Don’t share useless stuff • What is useful across the organisation? • Store in knowledgebase • Intellectual Property • Use as training!
- 54. DISCOVERY CYCLE @OptimiseOrDie Discover Ideas Prioritise Build & QA Test or Live Measure Data Learn
- 55. TEST CYCLE @OptimiseOrDie Hypothesis Design Sketch Wire Frame Mockup / Prototype Signoff (multiple) Build QA Soft and Hard Launch Analyse & Publish Learn
- 56. What did I learn personally? @OptimiseOrDie • That best practice is emerging • It’s about how you can do this properly! • Methodology & Frameworks – not tactics • Shared mindset across CRO practitioners • Stories beat statistics, Data beats opinion • I always have lots to learn • “Hold strong opinions, but loosely”
- 58. Go Forth and Optimise! Email me sullivac@gmail.com Slides Will be mailed to you! Linkedin linkd.in/pvrg14