Process lets you…1. Achieve valid, effective tests2. Take full advantage of business opportunities
Without Process With ProcessUnclear, misleading results Statistically valid results Results that don’t add $ Results that add to the bottom line High effort, low return Reasonable effort, huge return Opinion-based testing Data-driven testing “Inconclusive” results (a costly tie) Clear winners and losers
Take advantage of opportunities1. Time to market – How fast can you get improvements in place?2. Test the right stuff, in the right order – Low Investment, high Return
Today, You Will…1. Learn the Scientific Method…for Marketers2. Learn a proven Optimization process3. Discover 4 hidden BestPractices nuggets
The Scientific Method…for Marketers1. Define the problem2. Research/observe the problem3. Form a hypothesis4. Conduct experiment5. Analyze experiment results Look familiar?6. Form a conclusion7. Socialize results
Optimization Process Optimization Science Process Marketing
1) Business Goals Image credit: Square2Marketing.com
1) Business Goals• In science-speak, this is “define the problem”• Resist temptation to: – skip it – assume• Multiple goals OK – prioritize
1) Business Goals• Example: – Conversion Rate = primary business goal – Forward to a friend engagement = secondary – Test for Conversion Rate – Analyze/monitor F2F engagement
Hidden BestPractice Nugget #1:Test as “close to” revenue as you can.Testing further upstream can bemisleading.
2) Understand Target Audience• In science-speak, this is “research the problem”• Generally qualitative data (“market research”) – personas – feedback forms – focus groups – social media listening – chat transcripts – call center monitoring – demographics/psychographics
3) Data Analysis (Quantitative)• In science-speak, this is “research the problem”• Web Analytics – High traffic points – Decision/conversion points – Persuasion points – High abandonment points• Survey Data
4) Heuristic Analysis Image credit: Printed Circuit University
4) Heuristic Analysis• In science-speak, this is “research the problem”• Are “best practices” (rules of thumb) applied?• Are you at parity with competitors and industry leaders?
5) Capture Challenges (roadblocks) No, not this kind of Roadblock. Image credit: YoJoe.com
Hidden BestPractice Nugget #3:“Confidence level” - the statistical probabilitythat your results are not due to chance.Own your confidence level. Most tools are 95%.Get buy-in on tool’s default or other confidencelevel.
12) Report Findings• If +, tie it to revenue – “estimated”
12) Report Findings• If +, tie it to revenue – “estimated”• If -, what did you learn?
13) Next Steps• Install winning version – You don’t make $ until you install the winner! – Production-ready code, SEO, etc.• Follow-up test ideas• Add to Marketing knowledge base
Image credit: http://rahuleink.wordpress.com/2010/04/23/what-is-six-sigma/HiddenBestPracticeNugget #4:Optimize your Optimization. Stay lean,learn as you go, smooth out the processusing Six Sigma or other methodologies.
Key Takeaways1. Optimization Process is your friend2. Optimization Process maps to science… – …but bows to Marketing3. Optimize your own Optimization efforts – “…the best is the enemy of the good.” Voltaire (1694–1778)