Testing & Optimization - A Deeper Look

334 views
214 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
334
On SlideShare
0
From Embeds
0
Number of Embeds
14
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Testing & Optimization - A Deeper Look

  1. 1. Testing & Optimization: A Deeper Look January 16, 2013 Brendan Regan Marketing Optimization Specialist Analytics Pros
  2. 2. BestPractices – Seattle, WA – 12.6.12Late last year, I presented “Building a SuccessfulOptimization Program.”
  3. 3. TO GET THE SLIDES:http://www.slideshare.net/analyticspros/best-practices-optimizationpresentationbrendanregan
  4. 4. TODAY, we’re focusing in for a deeper look… People | Process | Tools Process | Tools
  5. 5. Why is Optimization“process” worth a webinar? I’m glad you asked.
  6. 6. “Process is boring.”
  7. 7. Process is the stuff of superheroes!
  8. 8. Process lets you…1. Achieve valid, effective tests2. Take full advantage of business opportunities
  9. 9. Without Process With ProcessUnclear, misleading results  Statistically valid results Results that don’t add $  Results that add to the bottom line High effort, low return  Reasonable effort, huge return Opinion-based testing  Data-driven testing “Inconclusive” results (a costly tie)  Clear winners and losers 
  10. 10. Take advantage of opportunities1. Time to market – How fast can you get improvements in place?2. Test the right stuff, in the right order – Low Investment, high Return
  11. 11. Maximize Return Over Time
  12. 12. Today, You Will…1. Learn the Scientific Method…for Marketers2. Learn a proven Optimization process3. Discover 4 hidden BestPractices nuggets 
  13. 13. The Scientific Method…for Marketers1. Define the problem2. Research/observe the problem3. Form a hypothesis4. Conduct experiment5. Analyze experiment results Look familiar?6. Form a conclusion7. Socialize results
  14. 14. Optimization Process Optimization Science Process Marketing
  15. 15. 1) Business Goals Image credit: Square2Marketing.com
  16. 16. 1) Business Goals• In science-speak, this is “define the problem”• Resist temptation to: – skip it – assume• Multiple goals OK – prioritize
  17. 17. 1) Business Goals• Example: – Conversion Rate = primary business goal – Forward to a friend engagement = secondary – Test for Conversion Rate – Analyze/monitor F2F engagement
  18. 18. Hidden BestPractice Nugget #1:Test as “close to” revenue as you can.Testing further upstream can bemisleading.
  19. 19. 2) Understand Target Audience Image credit: Closed Loop Marketing
  20. 20. 2) Understand Target Audience• In science-speak, this is “research the problem”• Generally qualitative data (“market research”) – personas – feedback forms – focus groups – social media listening – chat transcripts – call center monitoring – demographics/psychographics
  21. 21. 3) Web Data Analysis
  22. 22. 3) Data Analysis (Quantitative)• In science-speak, this is “research the problem”• Web Analytics – High traffic points – Decision/conversion points – Persuasion points – High abandonment points• Survey Data
  23. 23. 4) Heuristic Analysis Image credit: Printed Circuit University
  24. 24. 4) Heuristic Analysis• In science-speak, this is “research the problem”• Are “best practices” (rules of thumb) applied?• Are you at parity with competitors and industry leaders?
  25. 25. 5) Capture Challenges (roadblocks) No, not this kind of Roadblock. Image credit: YoJoe.com
  26. 26. 5) Capture Challenges (roadblocks) Image credit: ArtsJournal.com
  27. 27. 5) Capture Challenges (roadblocks)• Challenges presented by your site/marketing to your prospects• Ex: “Landing page lacks ‘social proof’ content.”
  28. 28. 6) Challenges » Hypotheses
  29. 29. 6) Challenges » Hypotheses• Every challenge/roadblock has potential solution(s)• Best guess, best solution = hypothesis• Ex: “Adding ‘social proof’ content will increase conversion rate.”
  30. 30. Hidden BestPractice Nugget #2:• Hypothesis: “Adding ‘social proof’ content will increase conversion rate.”• Research Question: “Which version of social proof content will perform best?”
  31. 31. 7) Design Experiment
  32. 32. 7) Design Experiment• Formal documentation• Capture: – Goal(s) – Problem/challenge – Quantitative/Qualitative proof – Hypothesis – Specifications
  33. 33. Hidden BestPractice Nugget #3:“Confidence level” - the statistical probabilitythat your results are not due to chance.Own your confidence level. Most tools are 95%.Get buy-in on tool’s default or other confidencelevel.
  34. 34. 8) Design Variations
  35. 35. 8) Design Variations• Sketch/wireframe• Copy deck• Designs/mockups• Approval process –Per experiment design –Marketing/Legal blessing
  36. 36. 9) Build Variations
  37. 37. 9) Build Variations• You may have to write code • You may “throw away” code• Page load speed needs to be consistent!
  38. 38. 10) Implement Experiment
  39. 39. 10) Implement Experiment• Integrate w/ testing tool• Test before you test – X-browser – Acceptance testing?• Launch & Monitor – 7 days minimum to mitigate “day-of-week” – Patience, please 
  40. 40. 11) Analyze Results No, not this kind of analysis.
  41. 41. 11) Analyze Results
  42. 42. 11) Analyze Results• Google Analytics integrations – Optimizely, Visual Website Optimizer, etc.
  43. 43. 11) Analyze Results• What does the result mean about the target audience?• Are the learnings applicable elsewhere? – Other sites? – Other campaigns? – Other channels?
  44. 44. 12) Report Findings
  45. 45. 12) Report Findings• If +, tie it to revenue – “estimated”
  46. 46. 12) Report Findings• If +, tie it to revenue – “estimated”• If -, what did you learn?
  47. 47. 13) Next Steps• Install winning version – You don’t make $ until you install the winner! – Production-ready code, SEO, etc.• Follow-up test ideas• Add to Marketing knowledge base
  48. 48. Image credit: http://rahuleink.wordpress.com/2010/04/23/what-is-six-sigma/HiddenBestPracticeNugget #4:Optimize your Optimization. Stay lean,learn as you go, smooth out the processusing Six Sigma or other methodologies.
  49. 49. Key Takeaways1. Optimization Process is your friend2. Optimization Process maps to science… – …but bows to Marketing3. Optimize your own Optimization efforts – “…the best is the enemy of the good.” Voltaire (1694–1778)
  50. 50. Questions?

×