Web Optimization Process Management

3,423 views

Published on

Tim Wilson's presentation at eMetrics San Francisco in April 2013 covering an approach for analysts to establish and manage a well-defined process for hypothesis discovery, prioritization, and testing to help pivot companies to be data-driven -- to orient towards a culture of validated learning.

Published in: Technology, Business

Web Optimization Process Management

  1. 1. Web OptimizationProcess ManagementTim WilsonVP of Measurement & Analytics, Clearhead /emetrics eMetrics, April 2013 @tgwilson
  2. 2. /emetrics @tgwilson
  3. 3. Theanalystas anagent ofchange /emetrics @tgwilson
  4. 4. Our job: reorientthe businesstowards validatedlearning. /emetrics @tgwilson
  5. 5. Everyone wants tobe data-driventhese days. /emetrics @tgwilson
  6. 6. Being Data Driven People Process Technology /emetrics @tgwilson
  7. 7. Being Data DrivenThe Business’ perspective People Process Technology /emetrics @tgwilson
  8. 8. Being Data DrivenThe Analyst’s perspective People Process Technology /emetrics @tgwilson
  9. 9. Being Data DrivenMy perspective People Process Technology /emetrics @tgwilson
  10. 10. Let’s talk about aprocess revolution. /emetrics @tgwilson
  11. 11. Every revolutionneeds a mobilizingcatch-phrase. /emetrics @tgwilson
  12. 12. A.D.A.P.T.toAct & Learn /emetrics @tgwilson
  13. 13. Align on goals and KPIsDiscover hypotheses across the companyAssess the hypotheses and your approachPrioritize the hypothesesTest the highest priority hypothesesAct on the resultsLearn for the future /emetrics @tgwilson
  14. 14. Align on goals and KPIsDiscover hypotheses across the companyAssess the hypotheses and your approachPrioritize the hypothesesTest the highest priority hypothesesAct on the resultsLearn for the future /emetrics @tgwilson
  15. 15. Alignment meansclearly establishedgoals and KPIs. /emetrics @tgwilson
  16. 16. /emetrics @tgwilson
  17. 17. /emetrics @tgwilson
  18. 18. Magic questions.1.  What are we trying to achieve?2.  How do we know if we’ve done that? /emetrics @tgwilson
  19. 19. A Process: Goals and KPIs Two Magic −  What are we trying to achieve? Questions −  How will we know if we’ve done that? Establish −  Open-ended questions Appropriate −  “What?” and “How?” (not “Why?”) Measures −  The power of restatement −  Historical data Establish −  Back-of-the-napkin creativity Targets −  Bracketing /emetrics @tgwilson
  20. 20. Online Revenue Engagement ReachDrive online revenue through the presence of xxxx Increase overall site engagement Increase the reach of [site].comConversion Rate Lift +14.3% New Profiles Created 2,707 SEO Visits to xxxx 116 Visits that interacted with xxxx 7.08% Total Profiles created since launch 3,391 20 250 Visits that did not interact with xxxx 6.19% 200 1515.0% 150 1010.0% 100 5.0% 50 5 0.0% 0 3/10 3/26 3/10 3/26 0 3/10 3/26 Overall conversion Projected rate Sitewide rate for the site without xxxx % Lift % Interacting w/ xxxx 13.8% +3.19% +3.14% +1.6% Visits that interacted 36,605 Social Shares of xxxx 251 Visits that did not interact 229,125 50Avg. Order Value Lift +2.7% 20.0% 40 Visits that interacted with xxxx $23.37 15.0% 30 20 Visits that did not interact with xxxx $22.76 10.0% 10$40 5.0% 0 0.0% 3/10 3/26$20 3/10 3/26 $0 3/10 3/26 Time on Site Lift +77.6% Social Referral Visits 46 Overall AOV Projected AOV Sitewide for the site without xxxx % Lift Visits that interacted with xxxx 24:49 8 $23.04 $22.96 +0.3% Visits that did not interact with xxxx 13:59 6 30:00 4Total Revenue Lift $8,976 25:00 20:00 2 Conversion rate lift only $7,389 15:00 0 AOV lift only $1,389 10:00 3/10 3/26 05:00 AOV lift on incremental orders $199 00:00 3/10 3/26 Refer to the glossary for a detailed definition for each metric.Annualized Lift (ignoring seasonality) $192,727 /emetrics @tgwilson
  21. 21. Align on goals and KPIsDiscover hypotheses across the companyAssess the hypotheses and your approachPrioritize the hypothesesTest the highest priority hypothesesAct on the resultsLearn for the future /emetrics @tgwilson
  22. 22. An analysis isnothing more thantesting a hypothesis. /emetrics @tgwilson
  23. 23. Where dohypotheses comefrom? /emetrics @tgwilson
  24. 24. Everyone Else at the Company Our Stakeholders Analysts/emetrics @tgwilson
  25. 25. Everyone in thebusiness hashypotheses! /emetrics @tgwilson
  26. 26. The trick is gettingthem articulatedclearly. /emetrics @tgwilson
  27. 27. What we say to marketers “What is your hypothesis?” What they hear “I’m dragging you back to the 8th grade science class you hated.”/emetrics @tgwilson
  28. 28. I believe that…[some idea]If I am right, then…[some action] /emetrics @tgwilson
  29. 29. Each idea can beadded to yourhypothesis library.­  Google Spreadsheet­  Excel­  uservoice.com­  Sharepoint­  Lotus Notes­  Jira /emetrics @tgwilson
  30. 30. Align on goals and KPIsDiscover hypotheses across the companyAssess the hypotheses and your approachPrioritize the hypothesesTest the highest priority hypothesesAct on the resultsLearn for the future /emetrics @tgwilson
  31. 31. To assess the effortrequired, you have todetermine theapproach. /emetrics @tgwilson
  32. 32. Flickr / justinbaeder/emetrics @tgwilson
  33. 33. The analyst mustbe aware of the fulltool belt. /emetrics @tgwilson
  34. 34. I believe that…[some idea]If I am right, then…[some action]To test this, I will…[tool/technology/approach] /emetrics @tgwilson
  35. 35. What’s in yourcompany’s toolbelt? Web Analytics Email SurveysVoice of the Customer Social Media AnalyticsA/B (and MV) Testing Focus Groups Usability Testing Secondary Research /emetrics @tgwilson
  36. 36. Align on goals and KPIsDiscover hypotheses across the companyAssess the hypotheses and your approachPrioritize the hypothesesTest the highest priority hypothesesAct on the resultsLearn for the future /emetrics @tgwilson
  37. 37. It’s not a simple formula. /emetrics @tgwilson
  38. 38. But a formula canguide thediscussion. /emetrics @tgwilson
  39. 39. Likelihood belief is right H M LLikelihood change would occur H M L Likely scale of impact H M LAlignment w/ business goals H M L Effort to test (approach) H M L /emetrics @tgwilson
  40. 40. Align on goals and KPIsDiscover hypotheses across the companyAssess the hypotheses and your approachPrioritize the hypothesesTest the highest priority hypothesesAct on the resultsLearn for the future /emetrics @tgwilson
  41. 41. /emetrics @tgwilson
  42. 42. Align on goals and KPIsDiscover hypotheses across the companyAssess the hypotheses and your approachPrioritize the hypothesesTest the highest priority hypothesesAct on the resultsLearn for the future /emetrics @tgwilson
  43. 43. Driving actionrequires effectivecommunication ofthe results. /emetrics @tgwilson
  44. 44. The datadoes notspeak for itself. /emetrics @tgwilson
  45. 45. /emetrics @tgwilson
  46. 46. Who needs to dowhat? /emetrics @tgwilson
  47. 47. Lead with thebusiness problemstatement. /emetrics @tgwilson
  48. 48. Follow with theanswer and theaction. /emetrics @tgwilson
  49. 49. Do not show yourwork. /emetrics @tgwilson
  50. 50. /emetrics @tgwilson
  51. 51. And then track toaction. /emetrics @tgwilson
  52. 52. We tested…[the idea]As a result, we…[took the action]Resulting in…[quantified result] /emetrics @tgwilson
  53. 53. Align on goals and KPIsDiscover hypotheses across the companyAssess the hypotheses and your approachPrioritize the hypothesesTest the highest priority hypothesesAct on the resultsLearn for the future /emetrics @tgwilson
  54. 54. Learning meanslooking beyond thenear-term action. /emetrics @tgwilson
  55. 55. Hypothesis tested Action taken Deeper learningHypothesis tested No action takenHypothesis tested Action takenHypothesis tested Action taken Deeper learningHypothesis tested No action taken Deeper learningHypothesis tested Action takenHypothesis tested Action takenHypothesis tested Action taken /emetrics @tgwilson
  56. 56. Reorienting thebusiness towardsvalidated learning /emetrics @tgwilson
  57. 57. Align on goals and KPIsDiscover hypotheses across the companyAssess the hypotheses and your approachPrioritize the hypothesesTest the highest priority hypothesesAct on the resultsLearn for the future /emetrics @tgwilson
  58. 58. clearhead.me/emetrics­  This presentation­  Guide: getting to great KPIs­  Hypothesis library template­  Approach options self-audit template­  Recommended reading /emetrics @tgwilson
  59. 59. nthanksN/emetrics @tgwilson

×