SlideShare a Scribd company logo
My year in testing...Cool stuff, hard lessons, new insights Craig Sullivan, Belron @OptimiseOrDie
Introduction ,[object Object]
Websites, processes, traffic, revenue, profit, costs
A/B and Multi-variate testing
PPC, Landing page and funnel optimisation
User experience design, prototyping, lab testing
Online self-servicing and contact deflection
Customer Research & Insight
Web Analytics
Site search analytics
Browser and email compatibility
Mobile UX and optimisation
Prove it!  I do UX with numbers
CX Manager for Belron®  (Carglass®)
35 websites, 19 languages, 8 figure revenue p.a.
Endured 18+ years of meetings with time wasters
I get to do stuff with crowds,[object Object]
 What is optimisation? We’re a customer experience team, but focused on how we can influence customer behaviour in nice ways!   So, our definition of Optimization: “A set of techniques, implemented in order to influence customer behaviour towards these outcomes:” Increased revenue or profit Removing cost from the business or contact center Increasing productivity or labor flexibility Increasing NPS (Customer Sat) scores      Simply delighting customers All work is challenged on this basis!
Insight - Inputs
Insight - Inputs
Small team, big challenge Team of 3 – 2 x optimisers, 1 x part time designer Higher ROI without spending more (or lower costs) Stuff that doesn’t hinder SEO or PPC Measurable ROI on changes, not ego or opinion Tools with low or zero IT footprint, easy to deploy Low cost or fast return techniques Specialised  tools for specific optimisation work Optimisation is a culture, not a tool though We don’t let IT run our website! MVT and A/B testing has the fastest ROI of *any* technique
What’s our testing mix? A/B testing – Homepage, Landing pages, Funnels MVT testing – high traffic landing or home pages Mini (less than 8 variables) Midi (8-32) Maxi (>32 variables) And 5 types of specialised testing: Funnel testing – bedding in new funnels Landing page optimisation Cross channel testing  (e.g. Chat, Contact deflection) New Feature testing Telephony MVT
Where do we get ideas from? Analytics data Customer insight – contact, customer feedback, research Usability testing Diary studies Clicktale recordings Business owners – market data, customer mix, local knowledge Customer contact centre immersion (listening) Books, Websites, Tips Other optimisers Test data – variables plotted on axis Making lots of mistakes – Fail faster
Killer app list Analytics Talent				 Session recording			:	Clicktale/Tealeaf Performance optimisation		:	Google/Strangeloop Feedback 				:	4Q/Kampyle Usability testing			:	See resources Remote testing			:	See resources Forms  and search analytics		:	Clicktale/Other Testing tools				:	Browsercam Deviceanywhere Google speed test			:	Cool stuff! Survey tools				:	See resources Multi-variate and A/B testing		:	Free, Paid or Both
Case StudyInternational Multi-Variate testing Tools & Techniques: ClicktaleGoogle AnalyticsAutonomy Optimost
The challenge Use MVT  to increase conversion internationally  35 websites, 19 languages, several bi or tri-lingual websites Core sites and pages are similar but there are: ,[object Object]
 Language differences
 Different market conditions
 Huge variations in traffic size
Very different local marketing approaches (traffic mix),[object Object]
What about us? Easy wins?  In priority order: Homepage Core funnel (common to all websites) High traffic landing pages (hero SEO and PPC terms) Local landing pages Influencing pages Lower traffic landing pages
How do you find your test areas? Analytics talent, tools and instrumentation Clicktale or Tealeaf – for session recording Identify key loss or influence pages Develop a leak model (see resources) Usability testing & UCD Contact centre feedback vital Traffic * opportunity is the key Grade targets by impact, time, resource, difficulty Free evaluation – ask me!
Homepage testing Run tests in key international hubs Cascade using ‘hub and spoke’ model Run A/B best hypothesis for smallest traffic Maxi (32+ variables), Midi (8-32), Mini (8), A/B (2+) UK, DE, FR = Maxi AUS, IT, ES = Midi NZ, AT, IE = Mini or A/B
International Traffic Volumes Hub and Spoke model – homepage traffic
Typical test lifecycle
Key learnings – on page elements Less is more – simplicity, comprehension, clarity in design Copy is king – not content, but key persuasive copy Large, prominent, visible, button like CTA CTA above fold, especially on landing page Do not split attention on CTA  (Save, Cancel, Print, Back, Help) Use goal oriented keywords (Fix my glass!) Where is the core value proposition? Maintain a scent – PPC advert, Landing page, Funnel, Reflect the user CTA wording, headlines, copy, framing = 50%+ of uplift Think about emotional states and make use of them Use badging, testimonials, social proofing and trust symbols We run a betting pool – even the best people are no better than 50/50 Guidelines are good but every page and site will vary
Key learnings – test approach Stop testing subtleties (e.g. Button colour) Cultural factors are minimal Market and traffic composition are much larger drivers Techniques work well, regardless of market or language Huge similarities in testing, but not identical results Hub and spoke works, but only at a traffic level Faster cycles of smaller tests = better (fail faster) A fast test that fails is better than a slow test You must not test inside a bubble – keep challenging  Use A/B to test radical approaches/disruptive change
Summary Don’t overcook tests with too many variables Don’t test things very similar (blue vs. green button?) Don’t be timid – fail faster, be bolder International tests show little cultural differences Market conditions & traffic mix are more important Biggest drivers on page =  Copy, CTA and CTA copy, Images, Badging, Nav/Layout Biggest external factors = Page and site performance, Browser compatibility We are now doing up to 7M page tests per month
UK Wave 2 – 21/9 to 26/10
+12.5%
+8.3%
+13.9%
+5.9%
What is the ROI then? +11.2%
Another example +1-2.5%
People images that suck…
Photo Guidelines Groups, vans, stores, corporate guff = NO Single person – not groups or even person + customer Open, Friendly, Natural, Smiling, Engaged = YES Uniform – has huge effect, even wearing branded ball cap improves conversion Look straight at viewer or slightly towards CTA Plain backgrounds only No folded arms, arms on hips, pointing, scowling, cheesy smiles, hands behind back, hanging useless by the side, clasped by groin, fiddling with hands Arms with prop instead – hold a clipboard, rag, tool – I’m working for *you* Female images work best in almost every country – they slay the guys Now we get slightly better images with the guidelines! We still got a genital framing one last week Photoshoots are now used to drive advertising strategy Video tests of potential ‘actors’ as a trial
Case Study #2New funnel implementation Tools & Techniques: ClicktaleGoogle AnalyticsSpeed Trap (analytics)OptimostBrowsercamSurvey toolsMarket researchDiary studies
The challenge Without operational change, improve conversion No systems or process changes Reskinning rather than repurpose Minimal investment
The process
What happened? €200k cost – user centered redesign (web/app/mobile) Mobile and App – superb results New funnel – converts 0.5% better Ow!  What happened? Works worse than old funnel in key areas
The result V 1.1 – Early funnel is a win, mid funnel great but end funnel sucks V 1.2 – Early funnel – d’oh - introduced page fold issue / layout / copy V 1.2 – Mid funnel - Bugs, copy issues, alternative approach fails V 1.2 – Split Customer details page into 2 pages helps end funnel V 1.3 – Reverted most early/mid funnel changes, kept end funnel V 1.4 – Optimisation on copy/buttons/messages (WIP)
Takeaways Usable in lab?  Doesn’t mean conversion gain User testing doesn’t uncover bugs, compatibility, forms, many persuasion and contextual issues Use clicktale to bed new process in No need to go back to lab or scrap the project You can see fixes working in real-time! Optimize and fix as you go Rapid iterative changes, instead of SLED You must watch channel shifts – phone, email etc.
What’s the ROI like on this work? Reduction in call time, questions, hassle Faster booking time (< 10%) Greater customer delight – feedback.  Less questions and worries 12.4% from funnel work, 12.5% from MVT 3.5% increase in downstream conversion rate 5.5% increase in NPS (Customer sat) score  4% increase from mobile optimised site Total increase in conversion (against control) = 32% Actual increase in conversion = 41% NO additional marketing spend needed On one site = > several hundred thousand € per month Yearly budget for usability, staff, tools – paid within weeks
Cross channel optimisation Chat : we removed 17 phone calls per week for 325 x 8m chat Chat increased ringback volumes - didn’t increase conversion Use dynamic phone numbering (PPC, SEO, Funnels, Tests) Measure all contact channel volumes (contact forms, other) Instrument A/B funnels with multiple phone numbers Key funnels and channel influencing tests – instrument them! Not applicable to all tests but critical to measure If you make a mistake here, you’re doomed. Anyway – who cares if it’s not your KPI?
Cross channel optimisation #2 Call centre optimisation Emergency phone switchoff Telephony MVT Chat testing Usability testing of internal apps Simple no training overspill systems Contact deflection systems and self servicing Customer reminders  Safelite – Technician profile and ‘on their way’ notice
What’s next Segmentation, Segmentation, Segmentation E.g. - screen size work on Mobile site, iPads Better testing tools, self administered A/B More funnel optimisation, increased volumes of page tests Move from 8 tests -> 20+ tests per month International MVT – now onto landing pages #1, homepage #3 Telephony and Email MVT Landing page generation – human and rules based Hero term PPC, SEO + Local landing pages a priority The perfect clothes store journey
My advice Invest in optimisation tools and talent Tools may be low cost – expertise is worth paying for Biggest ROI is high traffic landing page optimisation Funnel is good but page is quicker (weeks vs 6 mth) Work on images, copy, call-to-action, layout, hurry up messages, scarcity calls, badging, value proposition Always measure these tests with other channels… Chat and funnel test show cross channel is critical Optimise low hanging fruit – then segments Start a gambling club at work – have fun!  Try crazy stuff! More questions?  Grab me after for a beer!
Further reading… Email	:   Sullivac@gmail.com Twitter	:   @OptimiseOrDie www.linkedin.com/in/craigsullivan
Resources…
EFFORT--> (Size of bubble = ROI                                       COST -->

More Related Content

What's hot

EPiServer Marketing Arena - Turning web traffic into revenue
EPiServer Marketing Arena - Turning web traffic into revenueEPiServer Marketing Arena - Turning web traffic into revenue
EPiServer Marketing Arena - Turning web traffic into revenue
Episerver
 
How to Startup your SEO for SaaS - Fabrizio Ballarini - Duda Jan 2022
How to Startup your SEO for SaaS - Fabrizio Ballarini - Duda Jan 2022How to Startup your SEO for SaaS - Fabrizio Ballarini - Duda Jan 2022
How to Startup your SEO for SaaS - Fabrizio Ballarini - Duda Jan 2022
Fabrizio Ballarini
 
Optimizing Wordpress for Conversions (WPSF April 2, 2014)
Optimizing Wordpress for Conversions (WPSF April 2, 2014)Optimizing Wordpress for Conversions (WPSF April 2, 2014)
Optimizing Wordpress for Conversions (WPSF April 2, 2014)
Arun Sivashankaran
 
Stephen kenwright conversion elite
Stephen kenwright conversion eliteStephen kenwright conversion elite
Stephen kenwright conversion elite
Andy Brown
 
Lean Product Management for ISVs (October 14, 2014)
Lean Product Management for ISVs (October 14, 2014)Lean Product Management for ISVs (October 14, 2014)
Lean Product Management for ISVs (October 14, 2014)
Salesforce Partners
 
2016 #WCFAY Anatomy of a Website
2016 #WCFAY Anatomy of a Website2016 #WCFAY Anatomy of a Website
2016 #WCFAY Anatomy of a Website
Jamie's Notebook
 
People Power - How Moss Bros built their optimisation culture - Paul Rouke at...
People Power - How Moss Bros built their optimisation culture - Paul Rouke at...People Power - How Moss Bros built their optimisation culture - Paul Rouke at...
People Power - How Moss Bros built their optimisation culture - Paul Rouke at...
PRWD
 
Stan Gaidar: How far can you go improving user experience with AI?
Stan Gaidar: How far can you go improving user experience with AI? Stan Gaidar: How far can you go improving user experience with AI?
Stan Gaidar: How far can you go improving user experience with AI?
Lviv Startup Club
 
element^n - Picking the Perfect Partner
element^n - Picking the Perfect Partnerelement^n - Picking the Perfect Partner
element^n - Picking the Perfect Partner
Jennifer Carey
 
Jw conversion elite talk final black
Jw conversion elite talk final blackJw conversion elite talk final black
Jw conversion elite talk final black
Andy Brown
 
Step by step guide to revenue growth
Step by step guide to revenue growth  Step by step guide to revenue growth
Step by step guide to revenue growth
markroberge
 
The Website Redesign Checklist
The Website Redesign ChecklistThe Website Redesign Checklist
The Website Redesign Checklist
HTMLcut.com
 
Chicago Nonprofit + WIT Dreamforce Global Gathering
Chicago Nonprofit + WIT Dreamforce Global GatheringChicago Nonprofit + WIT Dreamforce Global Gathering
Chicago Nonprofit + WIT Dreamforce Global Gathering
csupilowski
 
Are you ready for Lightning to strike?
Are you ready for Lightning to strike? Are you ready for Lightning to strike?
Are you ready for Lightning to strike?
Desynit
 
BrightonSEO 2019 - Crawl Budget is dead, please welcome Rendering Budget
BrightonSEO 2019 - Crawl Budget is dead, please welcome Rendering BudgetBrightonSEO 2019 - Crawl Budget is dead, please welcome Rendering Budget
BrightonSEO 2019 - Crawl Budget is dead, please welcome Rendering Budget
Botify
 
Why the Way You Collect the Voice of the Customer Matters
Why the Way You Collect the Voice of the Customer MattersWhy the Way You Collect the Voice of the Customer Matters
Why the Way You Collect the Voice of the Customer Matters
AIPMM Administration
 
Comment utiliser le SEO en B2B pour générer des leads ? - Nahid Ouhelli - SEO...
Comment utiliser le SEO en B2B pour générer des leads ? - Nahid Ouhelli - SEO...Comment utiliser le SEO en B2B pour générer des leads ? - Nahid Ouhelli - SEO...
Comment utiliser le SEO en B2B pour générer des leads ? - Nahid Ouhelli - SEO...
SEO CAMP
 
UX Analytics Cemal Büyükgökçesu
UX Analytics Cemal BüyükgökçesuUX Analytics Cemal Büyükgökçesu
UX Analytics Cemal Büyükgökçesu
Userspots
 

What's hot (18)

EPiServer Marketing Arena - Turning web traffic into revenue
EPiServer Marketing Arena - Turning web traffic into revenueEPiServer Marketing Arena - Turning web traffic into revenue
EPiServer Marketing Arena - Turning web traffic into revenue
 
How to Startup your SEO for SaaS - Fabrizio Ballarini - Duda Jan 2022
How to Startup your SEO for SaaS - Fabrizio Ballarini - Duda Jan 2022How to Startup your SEO for SaaS - Fabrizio Ballarini - Duda Jan 2022
How to Startup your SEO for SaaS - Fabrizio Ballarini - Duda Jan 2022
 
Optimizing Wordpress for Conversions (WPSF April 2, 2014)
Optimizing Wordpress for Conversions (WPSF April 2, 2014)Optimizing Wordpress for Conversions (WPSF April 2, 2014)
Optimizing Wordpress for Conversions (WPSF April 2, 2014)
 
Stephen kenwright conversion elite
Stephen kenwright conversion eliteStephen kenwright conversion elite
Stephen kenwright conversion elite
 
Lean Product Management for ISVs (October 14, 2014)
Lean Product Management for ISVs (October 14, 2014)Lean Product Management for ISVs (October 14, 2014)
Lean Product Management for ISVs (October 14, 2014)
 
2016 #WCFAY Anatomy of a Website
2016 #WCFAY Anatomy of a Website2016 #WCFAY Anatomy of a Website
2016 #WCFAY Anatomy of a Website
 
People Power - How Moss Bros built their optimisation culture - Paul Rouke at...
People Power - How Moss Bros built their optimisation culture - Paul Rouke at...People Power - How Moss Bros built their optimisation culture - Paul Rouke at...
People Power - How Moss Bros built their optimisation culture - Paul Rouke at...
 
Stan Gaidar: How far can you go improving user experience with AI?
Stan Gaidar: How far can you go improving user experience with AI? Stan Gaidar: How far can you go improving user experience with AI?
Stan Gaidar: How far can you go improving user experience with AI?
 
element^n - Picking the Perfect Partner
element^n - Picking the Perfect Partnerelement^n - Picking the Perfect Partner
element^n - Picking the Perfect Partner
 
Jw conversion elite talk final black
Jw conversion elite talk final blackJw conversion elite talk final black
Jw conversion elite talk final black
 
Step by step guide to revenue growth
Step by step guide to revenue growth  Step by step guide to revenue growth
Step by step guide to revenue growth
 
The Website Redesign Checklist
The Website Redesign ChecklistThe Website Redesign Checklist
The Website Redesign Checklist
 
Chicago Nonprofit + WIT Dreamforce Global Gathering
Chicago Nonprofit + WIT Dreamforce Global GatheringChicago Nonprofit + WIT Dreamforce Global Gathering
Chicago Nonprofit + WIT Dreamforce Global Gathering
 
Are you ready for Lightning to strike?
Are you ready for Lightning to strike? Are you ready for Lightning to strike?
Are you ready for Lightning to strike?
 
BrightonSEO 2019 - Crawl Budget is dead, please welcome Rendering Budget
BrightonSEO 2019 - Crawl Budget is dead, please welcome Rendering BudgetBrightonSEO 2019 - Crawl Budget is dead, please welcome Rendering Budget
BrightonSEO 2019 - Crawl Budget is dead, please welcome Rendering Budget
 
Why the Way You Collect the Voice of the Customer Matters
Why the Way You Collect the Voice of the Customer MattersWhy the Way You Collect the Voice of the Customer Matters
Why the Way You Collect the Voice of the Customer Matters
 
Comment utiliser le SEO en B2B pour générer des leads ? - Nahid Ouhelli - SEO...
Comment utiliser le SEO en B2B pour générer des leads ? - Nahid Ouhelli - SEO...Comment utiliser le SEO en B2B pour générer des leads ? - Nahid Ouhelli - SEO...
Comment utiliser le SEO en B2B pour générer des leads ? - Nahid Ouhelli - SEO...
 
UX Analytics Cemal Büyükgökçesu
UX Analytics Cemal BüyükgökçesuUX Analytics Cemal Büyükgökçesu
UX Analytics Cemal Büyükgökçesu
 

Similar to Conversion Summit Slides - 1 Sep 2011 - Craig Sullivan

Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...
Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...
Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...
Craig Sullivan
 
Cross channel testing insights and recommendations - jump - oct 12 2011 - w...
Cross channel testing   insights and recommendations - jump - oct 12 2011 - w...Cross channel testing   insights and recommendations - jump - oct 12 2011 - w...
Cross channel testing insights and recommendations - jump - oct 12 2011 - w...
Craig Sullivan
 
Radically Improve Conversion Rates - eMSF 2009
Radically Improve Conversion Rates - eMSF 2009Radically Improve Conversion Rates - eMSF 2009
Radically Improve Conversion Rates - eMSF 2009
Kayden Kelly
 
Startup Metrics for Pirates (Startonomics Beijing, June 2009)
Startup Metrics for Pirates (Startonomics Beijing, June 2009)Startup Metrics for Pirates (Startonomics Beijing, June 2009)
Startup Metrics for Pirates (Startonomics Beijing, June 2009)
Dave McClure
 
Conversion Optimization Framework to Build Sustainable and Repeat Growth
Conversion Optimization Framework to Build Sustainable and Repeat GrowthConversion Optimization Framework to Build Sustainable and Repeat Growth
Conversion Optimization Framework to Build Sustainable and Repeat Growth
Tushar Purohit
 
Numbers not Napkins: Simple Startup Metrics
Numbers not Napkins: Simple Startup MetricsNumbers not Napkins: Simple Startup Metrics
Numbers not Napkins: Simple Startup Metrics
Dave McClure
 
Web Analytics: A Measurement System of Website
Web Analytics: A Measurement System of WebsiteWeb Analytics: A Measurement System of Website
Web Analytics: A Measurement System of Website
ITDogadjaji.com
 
How to be a Web 2.0 Metrics Jedi (Web 2.0 Expo, April 2009)
How to be a Web 2.0 Metrics Jedi  (Web 2.0 Expo, April 2009)How to be a Web 2.0 Metrics Jedi  (Web 2.0 Expo, April 2009)
How to be a Web 2.0 Metrics Jedi (Web 2.0 Expo, April 2009)
Dan Olsen
 
Startup Metrics for Pirates (March 2009)
Startup Metrics for Pirates (March 2009)Startup Metrics for Pirates (March 2009)
Startup Metrics for Pirates (March 2009)
Dave McClure
 
How To Convert More Web Traffic Into Sales And Leads
How To  Convert  More  Web  Traffic Into  Sales And  LeadsHow To  Convert  More  Web  Traffic Into  Sales And  Leads
How To Convert More Web Traffic Into Sales And Leads
Adam de Jong
 
How to Convert More Traffic Into Sales and Leads
How to Convert More Traffic Into Sales and LeadsHow to Convert More Traffic Into Sales and Leads
How to Convert More Traffic Into Sales and Leads
National Positions
 
Conversion Rate Optmization
Conversion Rate OptmizationConversion Rate Optmization
Conversion Rate Optmization
Edureka!
 
Startup Metrics For Pirates: AARRR! (Startonomics SF Oct 2008)
Startup Metrics For Pirates: AARRR! (Startonomics SF Oct 2008)Startup Metrics For Pirates: AARRR! (Startonomics SF Oct 2008)
Startup Metrics For Pirates: AARRR! (Startonomics SF Oct 2008)
Dave McClure
 
Passionate People: Why and How to measure statistics on your website
Passionate People: Why and How to measure statistics on your websitePassionate People: Why and How to measure statistics on your website
Passionate People: Why and How to measure statistics on your website
Patrice Roulive
 
Post Click Marketing: Optimizing Conversions
Post Click Marketing: Optimizing ConversionsPost Click Marketing: Optimizing Conversions
Post Click Marketing: Optimizing Conversions
Adam Proehl
 
Widemile and Microsoft Multivariate Testing Case Study
Widemile and Microsoft Multivariate Testing Case StudyWidemile and Microsoft Multivariate Testing Case Study
Widemile and Microsoft Multivariate Testing Case Study
Widemile
 
Startup Metrics 4 Pirates (July 2011)
Startup Metrics 4 Pirates (July 2011)Startup Metrics 4 Pirates (July 2011)
Startup Metrics 4 Pirates (July 2011)
Dave McClure
 
Startup Metrics for Pirates (SeedCamp 2008)
Startup Metrics for Pirates (SeedCamp 2008)Startup Metrics for Pirates (SeedCamp 2008)
Startup Metrics for Pirates (SeedCamp 2008)
Dave McClure
 
Startup Metrics for Pirates (FOWA/Miami, Feb 2009)
Startup Metrics for Pirates (FOWA/Miami, Feb 2009)Startup Metrics for Pirates (FOWA/Miami, Feb 2009)
Startup Metrics for Pirates (FOWA/Miami, Feb 2009)
Dave McClure
 
Stamats: Analytics Webinar
Stamats: Analytics WebinarStamats: Analytics Webinar
Stamats: Analytics Webinar
cjcunniff
 

Similar to Conversion Summit Slides - 1 Sep 2011 - Craig Sullivan (20)

Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...
Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...
Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...
 
Cross channel testing insights and recommendations - jump - oct 12 2011 - w...
Cross channel testing   insights and recommendations - jump - oct 12 2011 - w...Cross channel testing   insights and recommendations - jump - oct 12 2011 - w...
Cross channel testing insights and recommendations - jump - oct 12 2011 - w...
 
Radically Improve Conversion Rates - eMSF 2009
Radically Improve Conversion Rates - eMSF 2009Radically Improve Conversion Rates - eMSF 2009
Radically Improve Conversion Rates - eMSF 2009
 
Startup Metrics for Pirates (Startonomics Beijing, June 2009)
Startup Metrics for Pirates (Startonomics Beijing, June 2009)Startup Metrics for Pirates (Startonomics Beijing, June 2009)
Startup Metrics for Pirates (Startonomics Beijing, June 2009)
 
Conversion Optimization Framework to Build Sustainable and Repeat Growth
Conversion Optimization Framework to Build Sustainable and Repeat GrowthConversion Optimization Framework to Build Sustainable and Repeat Growth
Conversion Optimization Framework to Build Sustainable and Repeat Growth
 
Numbers not Napkins: Simple Startup Metrics
Numbers not Napkins: Simple Startup MetricsNumbers not Napkins: Simple Startup Metrics
Numbers not Napkins: Simple Startup Metrics
 
Web Analytics: A Measurement System of Website
Web Analytics: A Measurement System of WebsiteWeb Analytics: A Measurement System of Website
Web Analytics: A Measurement System of Website
 
How to be a Web 2.0 Metrics Jedi (Web 2.0 Expo, April 2009)
How to be a Web 2.0 Metrics Jedi  (Web 2.0 Expo, April 2009)How to be a Web 2.0 Metrics Jedi  (Web 2.0 Expo, April 2009)
How to be a Web 2.0 Metrics Jedi (Web 2.0 Expo, April 2009)
 
Startup Metrics for Pirates (March 2009)
Startup Metrics for Pirates (March 2009)Startup Metrics for Pirates (March 2009)
Startup Metrics for Pirates (March 2009)
 
How To Convert More Web Traffic Into Sales And Leads
How To  Convert  More  Web  Traffic Into  Sales And  LeadsHow To  Convert  More  Web  Traffic Into  Sales And  Leads
How To Convert More Web Traffic Into Sales And Leads
 
How to Convert More Traffic Into Sales and Leads
How to Convert More Traffic Into Sales and LeadsHow to Convert More Traffic Into Sales and Leads
How to Convert More Traffic Into Sales and Leads
 
Conversion Rate Optmization
Conversion Rate OptmizationConversion Rate Optmization
Conversion Rate Optmization
 
Startup Metrics For Pirates: AARRR! (Startonomics SF Oct 2008)
Startup Metrics For Pirates: AARRR! (Startonomics SF Oct 2008)Startup Metrics For Pirates: AARRR! (Startonomics SF Oct 2008)
Startup Metrics For Pirates: AARRR! (Startonomics SF Oct 2008)
 
Passionate People: Why and How to measure statistics on your website
Passionate People: Why and How to measure statistics on your websitePassionate People: Why and How to measure statistics on your website
Passionate People: Why and How to measure statistics on your website
 
Post Click Marketing: Optimizing Conversions
Post Click Marketing: Optimizing ConversionsPost Click Marketing: Optimizing Conversions
Post Click Marketing: Optimizing Conversions
 
Widemile and Microsoft Multivariate Testing Case Study
Widemile and Microsoft Multivariate Testing Case StudyWidemile and Microsoft Multivariate Testing Case Study
Widemile and Microsoft Multivariate Testing Case Study
 
Startup Metrics 4 Pirates (July 2011)
Startup Metrics 4 Pirates (July 2011)Startup Metrics 4 Pirates (July 2011)
Startup Metrics 4 Pirates (July 2011)
 
Startup Metrics for Pirates (SeedCamp 2008)
Startup Metrics for Pirates (SeedCamp 2008)Startup Metrics for Pirates (SeedCamp 2008)
Startup Metrics for Pirates (SeedCamp 2008)
 
Startup Metrics for Pirates (FOWA/Miami, Feb 2009)
Startup Metrics for Pirates (FOWA/Miami, Feb 2009)Startup Metrics for Pirates (FOWA/Miami, Feb 2009)
Startup Metrics for Pirates (FOWA/Miami, Feb 2009)
 
Stamats: Analytics Webinar
Stamats: Analytics WebinarStamats: Analytics Webinar
Stamats: Analytics Webinar
 

More from Craig Sullivan

Cross Device Optimisation - Google Analytics Shortcuts
Cross Device Optimisation - Google Analytics ShortcutsCross Device Optimisation - Google Analytics Shortcuts
Cross Device Optimisation - Google Analytics Shortcuts
Craig Sullivan
 
Product Design is Poo - And we're all going to die
Product Design is Poo - And we're all going to dieProduct Design is Poo - And we're all going to die
Product Design is Poo - And we're all going to die
Craig Sullivan
 
Product design is Poo - And how to fix it!
Product design is Poo - And how to fix it!Product design is Poo - And how to fix it!
Product design is Poo - And how to fix it!
Craig Sullivan
 
Conversion Research in One Hour
Conversion Research in One HourConversion Research in One Hour
Conversion Research in One Hour
Craig Sullivan
 
Web Analytics Wednesday - Session Replay Tools are Vital
Web Analytics Wednesday - Session Replay Tools are VitalWeb Analytics Wednesday - Session Replay Tools are Vital
Web Analytics Wednesday - Session Replay Tools are Vital
Craig Sullivan
 
eMetrics London - The AB Testing Hype Cycle
eMetrics London - The AB Testing Hype CycleeMetrics London - The AB Testing Hype Cycle
eMetrics London - The AB Testing Hype Cycle
Craig Sullivan
 
Interact London - 21 Oct 2015 - Scaling Stupidity
Interact London - 21 Oct 2015 - Scaling StupidityInteract London - 21 Oct 2015 - Scaling Stupidity
Interact London - 21 Oct 2015 - Scaling Stupidity
Craig Sullivan
 
Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015
Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015
Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015
Craig Sullivan
 
Surviving the hype cycle Shortcuts to split testing success
Surviving the hype cycle   Shortcuts to split testing successSurviving the hype cycle   Shortcuts to split testing success
Surviving the hype cycle Shortcuts to split testing success
Craig Sullivan
 
Myths and Illusions of Cross Device Testing - Elite Camp June 2015
Myths and Illusions of Cross Device Testing - Elite Camp June 2015Myths and Illusions of Cross Device Testing - Elite Camp June 2015
Myths and Illusions of Cross Device Testing - Elite Camp June 2015
Craig Sullivan
 
Myths, Lies and Illusions of AB and Split Testing
Myths, Lies and Illusions of AB and Split TestingMyths, Lies and Illusions of AB and Split Testing
Myths, Lies and Illusions of AB and Split Testing
Craig Sullivan
 
20 Ways to Shaft your Split Tesring : Conversion Conference
20 Ways to Shaft your Split Tesring : Conversion Conference20 Ways to Shaft your Split Tesring : Conversion Conference
20 Ways to Shaft your Split Tesring : Conversion Conference
Craig Sullivan
 
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to MeBrighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
Craig Sullivan
 
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to meDigital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
Craig Sullivan
 
20 top AB testing mistakes and how to avoid them
20 top AB testing mistakes and how to avoid them20 top AB testing mistakes and how to avoid them
20 top AB testing mistakes and how to avoid them
Craig Sullivan
 
#Measurefest : 20 Simple Ways to Fuck Up your AB tests
#Measurefest : 20 Simple Ways to Fuck Up your AB tests#Measurefest : 20 Simple Ways to Fuck Up your AB tests
#Measurefest : 20 Simple Ways to Fuck Up your AB tests
Craig Sullivan
 
#Measurecamp : 18 Simple Ways to F*** up Your AB Testing
#Measurecamp : 18 Simple Ways to F*** up Your AB Testing#Measurecamp : 18 Simple Ways to F*** up Your AB Testing
#Measurecamp : 18 Simple Ways to F*** up Your AB Testing
Craig Sullivan
 
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CROUXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
Craig Sullivan
 
The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014
The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014
The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014
Craig Sullivan
 
12 Things to do Before Your Company Dies : Conversion Conference London - Oct...
12 Things to do Before Your Company Dies : Conversion Conference London - Oct...12 Things to do Before Your Company Dies : Conversion Conference London - Oct...
12 Things to do Before Your Company Dies : Conversion Conference London - Oct...
Craig Sullivan
 

More from Craig Sullivan (20)

Cross Device Optimisation - Google Analytics Shortcuts
Cross Device Optimisation - Google Analytics ShortcutsCross Device Optimisation - Google Analytics Shortcuts
Cross Device Optimisation - Google Analytics Shortcuts
 
Product Design is Poo - And we're all going to die
Product Design is Poo - And we're all going to dieProduct Design is Poo - And we're all going to die
Product Design is Poo - And we're all going to die
 
Product design is Poo - And how to fix it!
Product design is Poo - And how to fix it!Product design is Poo - And how to fix it!
Product design is Poo - And how to fix it!
 
Conversion Research in One Hour
Conversion Research in One HourConversion Research in One Hour
Conversion Research in One Hour
 
Web Analytics Wednesday - Session Replay Tools are Vital
Web Analytics Wednesday - Session Replay Tools are VitalWeb Analytics Wednesday - Session Replay Tools are Vital
Web Analytics Wednesday - Session Replay Tools are Vital
 
eMetrics London - The AB Testing Hype Cycle
eMetrics London - The AB Testing Hype CycleeMetrics London - The AB Testing Hype Cycle
eMetrics London - The AB Testing Hype Cycle
 
Interact London - 21 Oct 2015 - Scaling Stupidity
Interact London - 21 Oct 2015 - Scaling StupidityInteract London - 21 Oct 2015 - Scaling Stupidity
Interact London - 21 Oct 2015 - Scaling Stupidity
 
Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015
Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015
Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015
 
Surviving the hype cycle Shortcuts to split testing success
Surviving the hype cycle   Shortcuts to split testing successSurviving the hype cycle   Shortcuts to split testing success
Surviving the hype cycle Shortcuts to split testing success
 
Myths and Illusions of Cross Device Testing - Elite Camp June 2015
Myths and Illusions of Cross Device Testing - Elite Camp June 2015Myths and Illusions of Cross Device Testing - Elite Camp June 2015
Myths and Illusions of Cross Device Testing - Elite Camp June 2015
 
Myths, Lies and Illusions of AB and Split Testing
Myths, Lies and Illusions of AB and Split TestingMyths, Lies and Illusions of AB and Split Testing
Myths, Lies and Illusions of AB and Split Testing
 
20 Ways to Shaft your Split Tesring : Conversion Conference
20 Ways to Shaft your Split Tesring : Conversion Conference20 Ways to Shaft your Split Tesring : Conversion Conference
20 Ways to Shaft your Split Tesring : Conversion Conference
 
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to MeBrighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
 
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to meDigital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
 
20 top AB testing mistakes and how to avoid them
20 top AB testing mistakes and how to avoid them20 top AB testing mistakes and how to avoid them
20 top AB testing mistakes and how to avoid them
 
#Measurefest : 20 Simple Ways to Fuck Up your AB tests
#Measurefest : 20 Simple Ways to Fuck Up your AB tests#Measurefest : 20 Simple Ways to Fuck Up your AB tests
#Measurefest : 20 Simple Ways to Fuck Up your AB tests
 
#Measurecamp : 18 Simple Ways to F*** up Your AB Testing
#Measurecamp : 18 Simple Ways to F*** up Your AB Testing#Measurecamp : 18 Simple Ways to F*** up Your AB Testing
#Measurecamp : 18 Simple Ways to F*** up Your AB Testing
 
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CROUXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
 
The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014
The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014
The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014
 
12 Things to do Before Your Company Dies : Conversion Conference London - Oct...
12 Things to do Before Your Company Dies : Conversion Conference London - Oct...12 Things to do Before Your Company Dies : Conversion Conference London - Oct...
12 Things to do Before Your Company Dies : Conversion Conference London - Oct...
 

Recently uploaded

Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
c5vrf27qcz
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
MichaelKnudsen27
 
A Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's ArchitectureA Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's Architecture
ScyllaDB
 
"What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w..."What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w...
Fwdays
 
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSFAppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
Ajin Abraham
 
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving
 
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsConnector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
DianaGray10
 
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin..."$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
Fwdays
 
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving
 
Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
Ivo Velitchkov
 
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptxPRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
christinelarrosa
 
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
Jason Yip
 
Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through StorytellingDemystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
Enterprise Knowledge
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
Fwdays
 
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
QA or the Highway - Component Testing: Bridging the gap between frontend appl...QA or the Highway - Component Testing: Bridging the gap between frontend appl...
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
zjhamm304
 
Principle of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptxPrinciple of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptx
BibashShahi
 
Mutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented ChatbotsMutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented Chatbots
Pablo Gómez Abajo
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
DianaGray10
 
From Natural Language to Structured Solr Queries using LLMs
From Natural Language to Structured Solr Queries using LLMsFrom Natural Language to Structured Solr Queries using LLMs
From Natural Language to Structured Solr Queries using LLMs
Sease
 
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid ResearchHarnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Neo4j
 

Recently uploaded (20)

Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
 
A Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's ArchitectureA Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's Architecture
 
"What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w..."What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w...
 
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSFAppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
 
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
 
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsConnector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
 
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin..."$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
 
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024
 
Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
 
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptxPRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
 
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
 
Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through StorytellingDemystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
 
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
QA or the Highway - Component Testing: Bridging the gap between frontend appl...QA or the Highway - Component Testing: Bridging the gap between frontend appl...
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
 
Principle of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptxPrinciple of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptx
 
Mutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented ChatbotsMutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented Chatbots
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
 
From Natural Language to Structured Solr Queries using LLMs
From Natural Language to Structured Solr Queries using LLMsFrom Natural Language to Structured Solr Queries using LLMs
From Natural Language to Structured Solr Queries using LLMs
 
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid ResearchHarnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
 

Conversion Summit Slides - 1 Sep 2011 - Craig Sullivan

  • 1. My year in testing...Cool stuff, hard lessons, new insights Craig Sullivan, Belron @OptimiseOrDie
  • 2.
  • 3. Websites, processes, traffic, revenue, profit, costs
  • 5. PPC, Landing page and funnel optimisation
  • 6. User experience design, prototyping, lab testing
  • 7. Online self-servicing and contact deflection
  • 11. Browser and email compatibility
  • 12. Mobile UX and optimisation
  • 13. Prove it! I do UX with numbers
  • 14. CX Manager for Belron® (Carglass®)
  • 15. 35 websites, 19 languages, 8 figure revenue p.a.
  • 16. Endured 18+ years of meetings with time wasters
  • 17.
  • 18. What is optimisation? We’re a customer experience team, but focused on how we can influence customer behaviour in nice ways! So, our definition of Optimization: “A set of techniques, implemented in order to influence customer behaviour towards these outcomes:” Increased revenue or profit Removing cost from the business or contact center Increasing productivity or labor flexibility Increasing NPS (Customer Sat) scores Simply delighting customers All work is challenged on this basis!
  • 21. Small team, big challenge Team of 3 – 2 x optimisers, 1 x part time designer Higher ROI without spending more (or lower costs) Stuff that doesn’t hinder SEO or PPC Measurable ROI on changes, not ego or opinion Tools with low or zero IT footprint, easy to deploy Low cost or fast return techniques Specialised tools for specific optimisation work Optimisation is a culture, not a tool though We don’t let IT run our website! MVT and A/B testing has the fastest ROI of *any* technique
  • 22. What’s our testing mix? A/B testing – Homepage, Landing pages, Funnels MVT testing – high traffic landing or home pages Mini (less than 8 variables) Midi (8-32) Maxi (>32 variables) And 5 types of specialised testing: Funnel testing – bedding in new funnels Landing page optimisation Cross channel testing (e.g. Chat, Contact deflection) New Feature testing Telephony MVT
  • 23. Where do we get ideas from? Analytics data Customer insight – contact, customer feedback, research Usability testing Diary studies Clicktale recordings Business owners – market data, customer mix, local knowledge Customer contact centre immersion (listening) Books, Websites, Tips Other optimisers Test data – variables plotted on axis Making lots of mistakes – Fail faster
  • 24. Killer app list Analytics Talent Session recording : Clicktale/Tealeaf Performance optimisation : Google/Strangeloop Feedback : 4Q/Kampyle Usability testing : See resources Remote testing : See resources Forms and search analytics : Clicktale/Other Testing tools : Browsercam Deviceanywhere Google speed test : Cool stuff! Survey tools : See resources Multi-variate and A/B testing : Free, Paid or Both
  • 25. Case StudyInternational Multi-Variate testing Tools & Techniques: ClicktaleGoogle AnalyticsAutonomy Optimost
  • 26.
  • 28. Different market conditions
  • 29. Huge variations in traffic size
  • 30.
  • 31. What about us? Easy wins? In priority order: Homepage Core funnel (common to all websites) High traffic landing pages (hero SEO and PPC terms) Local landing pages Influencing pages Lower traffic landing pages
  • 32. How do you find your test areas? Analytics talent, tools and instrumentation Clicktale or Tealeaf – for session recording Identify key loss or influence pages Develop a leak model (see resources) Usability testing & UCD Contact centre feedback vital Traffic * opportunity is the key Grade targets by impact, time, resource, difficulty Free evaluation – ask me!
  • 33. Homepage testing Run tests in key international hubs Cascade using ‘hub and spoke’ model Run A/B best hypothesis for smallest traffic Maxi (32+ variables), Midi (8-32), Mini (8), A/B (2+) UK, DE, FR = Maxi AUS, IT, ES = Midi NZ, AT, IE = Mini or A/B
  • 34. International Traffic Volumes Hub and Spoke model – homepage traffic
  • 36.
  • 37. Key learnings – on page elements Less is more – simplicity, comprehension, clarity in design Copy is king – not content, but key persuasive copy Large, prominent, visible, button like CTA CTA above fold, especially on landing page Do not split attention on CTA (Save, Cancel, Print, Back, Help) Use goal oriented keywords (Fix my glass!) Where is the core value proposition? Maintain a scent – PPC advert, Landing page, Funnel, Reflect the user CTA wording, headlines, copy, framing = 50%+ of uplift Think about emotional states and make use of them Use badging, testimonials, social proofing and trust symbols We run a betting pool – even the best people are no better than 50/50 Guidelines are good but every page and site will vary
  • 38. Key learnings – test approach Stop testing subtleties (e.g. Button colour) Cultural factors are minimal Market and traffic composition are much larger drivers Techniques work well, regardless of market or language Huge similarities in testing, but not identical results Hub and spoke works, but only at a traffic level Faster cycles of smaller tests = better (fail faster) A fast test that fails is better than a slow test You must not test inside a bubble – keep challenging Use A/B to test radical approaches/disruptive change
  • 39. Summary Don’t overcook tests with too many variables Don’t test things very similar (blue vs. green button?) Don’t be timid – fail faster, be bolder International tests show little cultural differences Market conditions & traffic mix are more important Biggest drivers on page = Copy, CTA and CTA copy, Images, Badging, Nav/Layout Biggest external factors = Page and site performance, Browser compatibility We are now doing up to 7M page tests per month
  • 40. UK Wave 2 – 21/9 to 26/10
  • 42. +8.3%
  • 44. +5.9%
  • 45. What is the ROI then? +11.2%
  • 48. Photo Guidelines Groups, vans, stores, corporate guff = NO Single person – not groups or even person + customer Open, Friendly, Natural, Smiling, Engaged = YES Uniform – has huge effect, even wearing branded ball cap improves conversion Look straight at viewer or slightly towards CTA Plain backgrounds only No folded arms, arms on hips, pointing, scowling, cheesy smiles, hands behind back, hanging useless by the side, clasped by groin, fiddling with hands Arms with prop instead – hold a clipboard, rag, tool – I’m working for *you* Female images work best in almost every country – they slay the guys Now we get slightly better images with the guidelines! We still got a genital framing one last week Photoshoots are now used to drive advertising strategy Video tests of potential ‘actors’ as a trial
  • 49. Case Study #2New funnel implementation Tools & Techniques: ClicktaleGoogle AnalyticsSpeed Trap (analytics)OptimostBrowsercamSurvey toolsMarket researchDiary studies
  • 50. The challenge Without operational change, improve conversion No systems or process changes Reskinning rather than repurpose Minimal investment
  • 52. What happened? €200k cost – user centered redesign (web/app/mobile) Mobile and App – superb results New funnel – converts 0.5% better Ow! What happened? Works worse than old funnel in key areas
  • 53. The result V 1.1 – Early funnel is a win, mid funnel great but end funnel sucks V 1.2 – Early funnel – d’oh - introduced page fold issue / layout / copy V 1.2 – Mid funnel - Bugs, copy issues, alternative approach fails V 1.2 – Split Customer details page into 2 pages helps end funnel V 1.3 – Reverted most early/mid funnel changes, kept end funnel V 1.4 – Optimisation on copy/buttons/messages (WIP)
  • 54. Takeaways Usable in lab? Doesn’t mean conversion gain User testing doesn’t uncover bugs, compatibility, forms, many persuasion and contextual issues Use clicktale to bed new process in No need to go back to lab or scrap the project You can see fixes working in real-time! Optimize and fix as you go Rapid iterative changes, instead of SLED You must watch channel shifts – phone, email etc.
  • 55. What’s the ROI like on this work? Reduction in call time, questions, hassle Faster booking time (< 10%) Greater customer delight – feedback. Less questions and worries 12.4% from funnel work, 12.5% from MVT 3.5% increase in downstream conversion rate 5.5% increase in NPS (Customer sat) score 4% increase from mobile optimised site Total increase in conversion (against control) = 32% Actual increase in conversion = 41% NO additional marketing spend needed On one site = > several hundred thousand € per month Yearly budget for usability, staff, tools – paid within weeks
  • 56. Cross channel optimisation Chat : we removed 17 phone calls per week for 325 x 8m chat Chat increased ringback volumes - didn’t increase conversion Use dynamic phone numbering (PPC, SEO, Funnels, Tests) Measure all contact channel volumes (contact forms, other) Instrument A/B funnels with multiple phone numbers Key funnels and channel influencing tests – instrument them! Not applicable to all tests but critical to measure If you make a mistake here, you’re doomed. Anyway – who cares if it’s not your KPI?
  • 57. Cross channel optimisation #2 Call centre optimisation Emergency phone switchoff Telephony MVT Chat testing Usability testing of internal apps Simple no training overspill systems Contact deflection systems and self servicing Customer reminders Safelite – Technician profile and ‘on their way’ notice
  • 58. What’s next Segmentation, Segmentation, Segmentation E.g. - screen size work on Mobile site, iPads Better testing tools, self administered A/B More funnel optimisation, increased volumes of page tests Move from 8 tests -> 20+ tests per month International MVT – now onto landing pages #1, homepage #3 Telephony and Email MVT Landing page generation – human and rules based Hero term PPC, SEO + Local landing pages a priority The perfect clothes store journey
  • 59. My advice Invest in optimisation tools and talent Tools may be low cost – expertise is worth paying for Biggest ROI is high traffic landing page optimisation Funnel is good but page is quicker (weeks vs 6 mth) Work on images, copy, call-to-action, layout, hurry up messages, scarcity calls, badging, value proposition Always measure these tests with other channels… Chat and funnel test show cross channel is critical Optimise low hanging fruit – then segments Start a gambling club at work – have fun! Try crazy stuff! More questions? Grab me after for a beer!
  • 60. Further reading… Email : Sullivac@gmail.com Twitter : @OptimiseOrDie www.linkedin.com/in/craigsullivan
  • 62. EFFORT--> (Size of bubble = ROI COST -->
  • 63. CRO and Testing resources 101 Landing page tips : slidesha.re/8OnBRh 544 Optimisation tips : bit.ly/8mkWOB 108 Optimisation tips : bit.ly/3Z6GrP 32 CRO tips : bit.ly/4BZjcW 57 CRO books : bit.ly/dDjDRJ CRO article list :bit.ly/nEUgui
  • 64. Analytics – where to optimise? It depends on your site... High traffic entry pages High dropout or bail pages Key value pages (credit card) High cost pages (PPC) Conversion tipping points Persuasion points Anywhere there is traffic and CTA! Use a leak model?
  • 65. How a leak model helps How many people continue? How many leave? How many channel switch? Tip! Dynamic site numbering! Where do they go? Are there influence pages? Any error clusters? What are my pain points?
  • 67. Value of leak model? Confirmation page loses 8% visitors One small change reduced this by 75% to 2% 1 page drives 60% traffic flow to phone Quote page has low switch – big test opportunity Bugs in address lookup, branch finder Forms errors contribute hugely to ‘leavers’ Losses aren’t always big – they’re just switching In most cases, over 50% of funnel ‘losses’ are switch Must use dynamic phone numbering!
  • 68. Session recording Vital for optimizers Fills in a ‘missing link’ for insight Rich source of data Segment by attribute/source/behavior Can be used to optimise in real time! Tealeaf www.tealeaf.com Clicktale www.clicktale.com Clixpy www.clixpy.com
  • 69. Performance and Availability Operate internationally? Across the USA? Customers attention span very short ½ a second delay = 15% loss in conversion rate On mobile, this is even more acute Improves your Google quality score Improves your conversion rate International visitors? Get a consumer perspective Measure third party tags Blame someone
  • 70. Performance resources? Read anything by Steve Souders Read Joshua Bixby (ace!) : bit.ly/8YB3ET Use Keynote, Gomez for site monitoring Enable google speed tags – get customer KPIs Consider CDN acceleration (Akamai etc.) Think of the customer context Optimise key traffic/conversion pages first Do A/B splits to confirm ROI Optimise for ‘return control to user’ Image, video, browser, mobile -> squeeze! Optimising batch images? Who owns customer facing performance?
  • 71. Feedback Simplest example – 4q – bail from site Deeper sampling: Light touch – micro conversion (social, newsletter) Starts key process – registration/purchase Gets to tipping point – payment page Get feedback at different funnel points Review the user session if captured Like 4Q but think of an onion
  • 72. Onsite feedback Onsite feedback, always available, immediate, contextual Feedback tools: Kampyle Opinionlab Foresee Polldaddy Crowdsourcing: Uservoice Ideascale Google moderator Kampyle has excellent international support – trial in Q4 2011.
  • 73. Usability Testing and UCD Lab based testing Remote testing tools Eye tracking Prototype testing Low budget testing Not what they say, watch what they do Hurdles to conversion, comprehension, action Excellent for ‘black box’ forms/funnels Fills a serious gap in your knowledge
  • 74. Email Optimisation Delivery rates How the user sees it Webmail & Clients Spam/Junk filters MVT testing Experiment and learn Returnpath Lyris
  • 75. Remote testing, usability, feedback Userzoom www.userzoom.com Usabilla www.usabilla.com Chalkmark www.chalkmark.com Userfly www.userfly.com Silverback www.silverbackapp.com Usertesting www.usertesting.com Loop11 www.loop11.com Pidoco www.pidoco.com Uservue www.techsmith.com Ethnio www.ethnio.com Feedback army www.feedbackarmy.com Five second test www.fivesecondtest.com Mechanical turk www.mechanicalturk.com Open Hallway www.openhallway.com
  • 76. Forms and Search Analytics Ignore at your peril Forms are often seen as a ‘technical’ area What are the common errors? What errors precede abandonment? Validation improvements Form design – Luke Wroblewski Good example – Postcode Lookup CR0 1XA CRO lXA 2.5% loss due to strict validation Search analytics – Hurol Inan
  • 77. Browser Compatibility “Anyone who slaps a ‘this page is best viewed with Browser X’ label on a Web page appears to be yearning for the bad old days, before the Web, when you had very little chance of reading a document written on another computer, another word processor, or another network.” Tim Berners-Lee Money on the table John Lewis test Who doorsteps you? Measure entry page stats only Learn about the segments – they might be valuable Create a browser matrix Use a testing service, not simulators Test mobile devices too Try the ‘Checkout browser’ test
  • 78. Testing tools Browser testing www.browsercam.com Mobile devices www.perfectomobile.com www.deviceanywhere.com Performance testing www.keynote.com www.gomez.com New on the block strangeloopnetworks.com End browser and device discrimination!
  • 79. Survey tools Dust off those survey tools and ask stuff: My personal rating: Surveymonkey www.surveymonkey.com Zoomerang www.zoomerang.com Surveygizmo www.surveygizmo.com