Download as PDF, PPTX
![The 3 pillars:
• “Make it about the customer”
• “Show it, don’t just say it”
• “Test concepts – not elements or tactics”]
Cycle: Research, Strategy, Hypothesis,
Prioritisation, Design, Align team, Launch,
QA, Run, Analyse
Talia Wolf – Points to note
@OptimiseOrDie](https://image.slidesharecdn.com/ch201511craigsullivan-151203141053-lva1-app6892/75/Craig-Sullivan-Keynote-speaker-summary-final-thoughts-Conversion-Hotel-2015-40-2048.jpg)
Uploaded byWebanalisten .nl
Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015
The document discusses key insights and strategies for A/B testing and optimization from various speakers at a conference. It emphasizes the importance of using a structured process, harnessing qualitative and quantitative data, and focusing on customer-centric concepts. Additionally, it calls for breaking down organizational silos and adopting agile methodologies for improved outcomes in digital marketing and conversion rate optimization.
More Related Content
Similar to Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015
![[Elite Camp 2016] Craig Sullivan - Elite Camp Summary Session](https://cdn.slidesharecdn.com/ss_thumbnails/18elitecamp-craigsullivan-roundup-160708121125-thumbnail.jpg?width=640&height=640&fit=bounds)
Craig Sullivan - Keynote speaker summary & final thoughts - Conversion Hotel 2015
- 1. The AB TestingHype Cycle Budapest - Netbooster – 11 Nov 2015 @OptimiseOrDie It’s a Wrap! #CH2015
- 2. How was yourtrip to Texel, dear? @OptimiseOrDie
- 3. • Genres – FacebookPoll • 3-4 Sets • Spotify Playlist • Outline set • Feedback • Mix live! Music Optimisation @OptimiseOrDie
- 5. Peep Laja @OptimiseOrDie • Peeeeep -A car sound • Pap - Common blog problem • Peahp - Texan pipe • Peppie - Food for small dogs • Payp - Micro payment system
- 6. World CRO DudeDickens Character PE PPIP
- 7. Peep Laja –Points to note @OptimiseOrDie 6 key pillars • Heuristic analysis & walkthrough • Session Replay / Mouse Tracking / Scroll / Click • Technical (compatibility & performance) • Web Analytics • User Testing • Qualitative Surveys
- 8. “Stop copying yourcompetitors. They may not know what the f*** they are doing either.” “We need better data, not more data” “Use a process – not 100 tactics. Discover what actually matters and lift heavy ass weights.” Peep Laja – Quotes @OptimiseOrDie
- 10.
- 11.
- 12. Chris Stucchio –Quotes @OptimiseOrDie “Misinterpretations of Significance.” “Bayesian statisticians can never agree to disagree.” “CRO is more like trading – the goal is to get more conversions.”
- 13. • The unconferencesession feedback! • This stuff isn’t easy • It’s about decision making though • Do we keep control, deploy challenger or test further? • We can’t educate the world easily but we can build tools and process that avoids or minimises poor decision making. Chris Stucchio – Points to note @OptimiseOrDie
- 15. @OptimiseOrDie 4. If youhave 4 hours PLUS • Snap interviews (Sales, Customer Services, Tech Support) • Run a quick poll or survey (See my tools slides) Less Bullshit!
- 16. Oli Gardner –Quotes @OptimiseOrDie “Shitting marketing pisses me off…” “Delight should be the default, not the exception.” “Every page has one primary goal.” “Personalisation or targeting is still a hypothesis until it’s tested”
- 17. • NSAMCWADLP –Never Start a Marketing Campaign Without a Dedicated Landing Page • 4 corners – Copy, Design, Interaction, Psychology • A multi-faceted approach is required Oli Gardner – Points to note @OptimiseOrDie
- 18. “ Affordance isthe ability of a user to perceive that some action is possible” – like a door handle or push plate! Affordance isn’t about stuff that can’t be inferred (for example, hover states, feedback messages, hidden interactions) http://www.jnd.org/dn.mss/affordances_and.html Affordance @OptimiseOrDie
- 20. “Best Practices don’twork” “The best optimisation champions integrate both yin and yang into their work.” “You need a qual and quant mindset” “Don’t start with tactics. Process is critical but framework thinking is the best way.” “Optimise the Optimisation” “It’s about understanding buyers through testing” Chris Goward – Quotes @OptimiseOrDie
- 21. • Marketing + Scientificprocess – CRO • UX, Analytics, Persuasion Principles, Business Context and your Prior test results – are all part of your testing framework • NPS is vital (Lovefilm example) • Measure NPS service metrics over time! • Continuous Improvement – not SLEDD* Chris Goward – Points to note @OptimiseOrDie
- 23.
- 24.
- 25.
- 26. • “Stories beat Statistics” • Storytellingis vital to our work “We saw this, so checked that and found this, which taught us that” • We use persuasion on customers but not internally within our company? • “Show their digital journey” Brent Dykes – Points to note @OptimiseOrDie
- 28.
- 29.
- 30. “Use the collectivebrain” “The Mixing of ideas and unexpected encounters through unplanned meetings” “The most important thing is the insights that you learn” Jorden Lentze – Quotes @OptimiseOrDie
- 31. • Note thecore team – CRO, Dev, Psychologist, Product, UX, Analyst, Copywriter • Explore the problem domain, focus on the problem at hand, incubate, get insights, follow through! • LISTEN to customers, MAKE incremental changes AND big ideas, AB Test and Measure impact • Sliders can work but most implementations suck! Jorden Lentze – Points to note @OptimiseOrDie
- 32.
- 35.
- 36. • Ethics are avital and ongoing debate • It’s all about intent for me • Read these articles – very good www.measuringu.com/blog/ethical-treatment.php www.twitter.com/search?q=optimiseordie%20ethics&src=typd Harry Brignull – Points to note @OptimiseOrDie
- 38.
- 39. “We like tothink of ourselves as rational people” “We’re basically buying better versions of ourselves” “Don’t talk about yourself – make it about them” “It’s about testing concepts, not elements or tactics. You can’t learn anything from these kind of tests” “It’s about the strategy and knowledge you acquire” Talia Wolf – Quotes @OptimiseOrDie
- 40. The 3 pillars: • “Makeit about the customer” • “Show it, don’t just say it” • “Test concepts – not elements or tactics”] Cycle: Research, Strategy, Hypothesis, Prioritisation, Design, Align team, Launch, QA, Run, Analyse Talia Wolf – Points to note @OptimiseOrDie
- 41. The AB TestingHype Cycle Budapest - Netbooster – 11 Nov 2015 @OptimiseOrDie It’s a Wrap! #CH2015
- 42. What About theOrganisation? @OptimiseOrDie
- 43. @OptimiseOrDie Burn Down theSilos • Non agile, non iterative design • Silos work on product separately • No ‘One Team’ per product/theme • Large teams, unwieldy coordination • Pass the product around • More PMs and BAs than a conference • Endless sucking signoff • AB testing done the same way!
- 44. @OptimiseOrDie FT, Booking, Skyscanner,ABN, Expedia + more • Small teams (6-15) with direct access to publish • Ability to set and get metrics data directly • Tools, Autonomy, Lack of interference • No Project Managers or Business Analysts • Business defines ‘outcomes’ – teams deliver • No long signoff chain • No pesky meddling fools • 18 Month projects over budget?
- 45. @OptimiseOrDie FT Example • 100sof releases a day! • Iterative MVP approach • Launch as alpha, beta, pilot, phased rollout • Like getting in a shower • Read more at labs.ft.com
- 46.
- 47. @OptimiseOrDie 10. Positive Attributes • Rapid, Iterative, User Centred & Agile Design. No Silos. • Small empowered autonomous teams • Polymaths and Overlap • Toolkit & Analytics investment • Persuasive copywriting & Psychology • Great Testing & Optimisation Tools
- 48. @OptimiseOrDie • Agile, Lean,Iterative x silo teams • Ability to get and set metrics • Autonomy, Control, Velocity • Iterative MVP approach • Work on outcomes, not features ! Burn Down the Silos!
- 49.
- 50.
- 51. Summary Grid @OptimiseOrDie Do YourResearch Prioritise Hypothesise • Heuristic analysis • Usability testing • Heatmap tools • Session replay • Web Analytics • Qualitative surveys • Polls • Chat & Text analysis • Interviews • Device compatibility • Performance • Use TIR or PIE • Prioritise around opportunity vs. cost • Estimate political difficulty or friction as well as resource costs • There is no best practice on prioritisation • Continually re-prioritise against new discoveries • Use the collective brain • Assemble and circulate your research • Set clear goals and have a process for sketching • Sketch the Hypothesis • Build W/F or mockup • Set ONE primary (possibly one secondary) success KPI
- 52. Summary Grid @OptimiseOrDie Pre FlightChecks Run Analyse & Report • QA testing • Device & Browser compatibility • Analytics tracking • Ecommerce tracking • Launch • Set your test length with a calculator • Run it, stop it, analyse it • Don’t run over time unless your sample is smaller than expected • Learn about the statistics • Watch tests closely! • Wash up analysis • Share findings widely • Don’t share useless stuff • What is useful across the organisation? • Store in knowledgebase • Intellectual Property • Use as training!
- 53. DISCOVERY CYCLE @OptimiseOrDie Discover Ideas Priori/se Build & QA Test or Live Measure Data Learn
- 54. TEST CYCLE @OptimiseOrDie Hypothesis Design Sketch Wire Frame Mockup / Prototype Signoff (mul/ple) Build QA SoH and Hard Launch Analyse & Publish Learn
- 55. What did Ilearn personally? @OptimiseOrDie • That best practice is emerging • It’s about how you can do this properly! • Methodology & Frameworks – not tactics • Shared mindset across CRO practitioners • Stories beat statistics, Data beats opinion • I always have lots to learn • “Hold strong opinions, but loosely”
- 57. Go Forth andOptimise! Email me sullivac@gmail.com Slides Will be mailed to you! Linkedin linkd.in/pvrg14