Analytics and Optimization 2013

307 views
230 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
307
On SlideShare
0
From Embeds
0
Number of Embeds
11
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Analytics and Optimization 2013

  1. 1. 10 — Analytics & Optimization From Code to Product gidgreen.com/course
  2. 2. Lecture 10 •  Introduction •  Data collection •  Website metrics •  Optimization •  Competitive intelligence •  Surveys •  Tools and books From Code to Product Lecture 10 — Analytics— Slide 2 gidgreen.com/course
  3. 3. Why analytics? •  Quantify success/failure – For yourselves – For investors – Against competition •  Scientific decisions – No blind faith – Fewer arguments – Avoid HiPPO = highest paid person’s opinion From Code to Product Lecture 10 — Analytics— Slide 3 gidgreen.com/course
  4. 4. Good analytics •  Simple •  Few in number •  Relevant •  Unambiguous •  Actionable •  Instant (or nearly) •  Repeatable From Code to Product Lecture 10 — Analytics— Slide 4 gidgreen.com/course
  5. 5. AARRR — Metrics for pirates From Code to Product Lecture 10 — Analytics— Slide 5 gidgreen.com/course Acquisition Site visit or app download Activation Registration or usage Retention Repeat usage Referral Brings other people Revenue Generate cash DaveMcClure,500Startups
  6. 6. Some quotes “What gets measured, gets managed.” — Peter Drucker “The only metrics that entrepreneurs should invest energy in collecting are those that help them make decisions.” — Eric Ries, The Lean Startup From Code to Product Lecture 10 — Analytics— Slide 6 gidgreen.com/course
  7. 7. Lecture 10 •  Introduction •  Data collection •  Website metrics •  Optimization •  Competitive intelligence •  Surveys •  Tools and books From Code to Product Lecture 10 — Analytics— Slide 7 gidgreen.com/course
  8. 8. In-app analytics •  Home rolled or third party •  Store usage information locally – ‘Call home’ when online •  Privacy concerns – Confirmation dialog? •  Complete access to device – But you will be caught! •  Problem: slow iteration From Code to Product Lecture 10 — Analytics— Slide 8 gidgreen.com/course
  9. 9. In-app integration From Code to Product Lecture 10 — Analytics— Slide 9 gidgreen.com/course
  10. 10. Reporting app events From Code to Product Lecture 10 — Analytics— Slide 10 gidgreen.com/course
  11. 11. Web analytics •  All activity visible to site – Users don’t expect privacy •  Web servers log requests – Also: Javascript solutions •  Page view centric – Other events require integration – Coffee break? – Events not sessions From Code to Product Lecture 10 — Analytics— Slide 11 gidgreen.com/course
  12. 12. A web server log line www.websudoku.com 24.186.55.113 [06/May/2012:08:13:02 -0400] "GET / HTTP/1.1” 200 1045 "http://www.google.com/search?q=sudoku” "Mozilla/5.0 (iPhone; CPU iPhone OS 5_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Mobile/9B179 Safari/7534.48.3" From Code to Product Lecture 10 — Analytics— Slide 12 gidgreen.com/course
  13. 13. Javascript tracking code <script type="text/javascript”> var _gaq = _gaq || []; _gaq.push(['_setAccount', 'UA-1165533-3']); _gaq.push(['_trackPageview']); (function() { var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); })(); </script> From Code to Product Lecture 10 — Analytics— Slide 13 gidgreen.com/course
  14. 14. Web metrics alternatives From Code to Product Lecture 4 — UI Design— Slide 14 gidgreen.com/course Server logs Javascript Home-made Integration None Via HTML Server code Convenience Download + analyze Web-based access Up to you Delay None Up to 24 hours Up to you Reporting Varies Advanced Up to you Other events Hard Via API Easy Data leakage None Total! None
  15. 15. Track web users by… •  IP address – Given for every web request – Good for geography – But: proxies, classrooms, router resets •  Cookies – Track user browser over long term – But: clearing, multi-browsing, first request – Customization of web server From Code to Product Lecture 10 — Analytics— Slide 15 gidgreen.com/course
  16. 16. Track web users by… •  Log in – Reliable for registered users – But: anonymous users, multiple accounts – Requires custom logging tools •  Solution: combine! – Intelligently tie IPs, cookies and accounts – Example: user registration •  Data always incomplete From Code to Product Lecture 10 — Analytics— Slide 16 gidgreen.com/course
  17. 17. Lecture 10 •  Introduction •  Data collection •  Website metrics •  Optimization •  Competitive intelligence •  Surveys •  Tools and books From Code to Product Lecture 10 — Analytics— Slide 17 gidgreen.com/course
  18. 18. Basic website metrics From Code to Product Lecture 10 — Analytics— Slide 18 gidgreen.com/course
  19. 19. Immediate questions •  When does one visit end? – GA: 30 minutes without activity •  What makes a visitor unique? – GA: Tracking cookie •  How is duration calculated? – GA: Time between first and last pages •  What makes a visitor new? – GA: Never visited your site before From Code to Product Lecture 10 — Analytics— Slide 19 gidgreen.com/course
  20. 20. Geography From Code to Product Lecture 6 — BM — Advertising— Slide 20 gidgreen.com/course
  21. 21. Demographics From Code to Product Lecture 6 — BM — Advertising— Slide 21 gidgreen.com/course
  22. 22. Frequency report From Code to Product Lecture 10 — Analytics— Slide 22 gidgreen.com/course
  23. 23. Sources of traffic •  Type-in (no referrer) – Includes browser bookmarks •  Search engines – Navigational search = type-in •  Referrals – Website links or social media •  Paid advertising •  Email campaigns From Code to Product Lecture 10 — Analytics— Slide 23 gidgreen.com/course
  24. 24. The multitouch problem •  There’s history before the referrer – Who deserves the credit, e.g. affiliates •  So who gets the credit? – Last click (standard) – First click (unrealistic) – Even split – Split weighted to last From Code to Product Lecture 10 — Analytics— Slide 24 gidgreen.com/course
  25. 25. Search engine queries From Code to Product Lecture 10 — Analytics— Slide 25 gidgreen.com/course Also: internal site search
  26. 26. Popular pages From Code to Product Lecture 10 — Analytics— Slide 26 gidgreen.com/course
  27. 27. Landing/entry pages From Code to Product Lecture 10 — Analytics— Slide 27 gidgreen.com/course “You can’t choose your home page” — A. Kaushik
  28. 28. Clickmaps and heatmaps From Code to Product Lecture 10 — Analytics— Slide 28 gidgreen.com/course
  29. 29. Conversion funnel From Code to Product Lecture 10 — Analytics— Slide 29 gidgreen.com/course Source:www.searchenginejournal.com
  30. 30. Sampling methods •  Popular site => lots of data – Burden to collect, slow to analyze •  Don’t record all events – Choose important pages – Random subset of visitors – Random subset of pageviews •  Sub-sample when analyzing – By page or visitor From Code to Product Lecture 10 — Analytics— Slide 30 gidgreen.com/course
  31. 31. Staleness due to changes in… •  Content •  User familiarity – Early adopters vs ... •  Search engine rankings •  Market needs •  Devices •  Cookies From Code to Product Lecture 10 — Analytics— Slide 31 gidgreen.com/course
  32. 32. Lecture 10 •  Introduction •  Data collection •  Website metrics •  Optimization •  Competitive intelligence •  Surveys •  Tools and books From Code to Product Lecture 10 — Analytics— Slide 32 gidgreen.com/course
  33. 33. Optimization •  You don’t know how users behave – Example: show price early on? •  Small changes => big results – But which small changes? •  Use a scientific methodology – Easy to set up – Easy to get report – Statistical significance From Code to Product Lecture 10 — Analytics— Slide 33 gidgreen.com/course
  34. 34. Wording example From Code to Product Lecture 10 — Analytics— Slide 34 gidgreen.com/course Source:http://www.dustincurtis.com/ you_should_follow_me_on_twitter.html
  35. 35. A/B testing •  Two parallel variations – Current vs challenger •  Assign randomly and evenly – What about previous visitors? – Repeat requests within a session? •  Set test length in advance – Length of time or number of visits •  Chi-squared (or similar) test From Code to Product Lecture 10 — Analytics— Slide 35 gidgreen.com/course
  36. 36. Contingency table Product purchased Not purchased 9 575 13 563 From Code to Product Lecture 10 — Analytics— Slide 36 gidgreen.com/course
  37. 37. Multivariate testing From Code to Product Lecture 10 — Analytics— Slide 37 gidgreen.com/course Source:http://www.getelastic.com/testing- part-1/
  38. 38. Multivariate testing •  Best to use third-party tool •  Full factorial vs partial factorial – Certainty vs efficiency From Code to Product Lecture 10 — Analytics— Slide 38 gidgreen.com/course
  39. 39. Optimization pitfalls •  Preconception driven – Too many similar tests – Checking before it’s done •  Wrong goal – e.g. started vs completed purchases •  Unfair test – Different time periods – New vs returning users From Code to Product Lecture 10 — Analytics— Slide 39 gidgreen.com/course
  40. 40. More complex tests •  Non-binary outcomes – Size of purchase, length of stay •  Cohort / longitudinal tests •  Whole-site multivariate testing •  Pricing – How to prevent a riot? •  Spot diminishing returns – Focus on registration, payment, etc… From Code to Product Lecture 10 — Analytics— Slide 40 gidgreen.com/course
  41. 41. Lecture 10 •  Introduction •  Data collection •  Website metrics •  Optimization •  Competitive intelligence •  Surveys •  Tools and books From Code to Product Lecture 10 — Analytics— Slide 41 gidgreen.com/course
  42. 42. Finding competitors From Code to Product Lecture 10 — Analytics— Slide 42 gidgreen.com/course
  43. 43. Searches for product From Code to Product Lecture 10 — Analytics— Slide 43 gidgreen.com/course
  44. 44. But… From Code to Product Lecture 10 — Analytics— Slide 44 gidgreen.com/course
  45. 45. Ranking for general searches From Code to Product Lecture 10 — Analytics— Slide 45 gidgreen.com/course
  46. 46. App Store searches From Code to Product Lecture 10 — Analytics— Slide 46 gidgreen.com/course
  47. 47. Online mentions From Code to Product Lecture 10 — Analytics— Slide 47 gidgreen.com/course
  48. 48. Website traffic From Code to Product Lecture 10 — Analytics— Slide 48 gidgreen.com/course
  49. 49. Website traffic From Code to Product Lecture 10 — Analytics— Slide 49 gidgreen.com/course
  50. 50. Downloads/installs From Code to Product Lecture 10 — Analytics— Slide 50 gidgreen.com/course
  51. 51. Registrations From Code to Product Lecture 10 — Analytics— Slide 51 gidgreen.com/course
  52. 52. Revenue From Code to Product Lecture 10 — Analytics— Slide 52 gidgreen.com/course Also: UK private companies
  53. 53. Revenue From Code to Product Lecture 10 — Analytics— Slide 53 gidgreen.com/course $200k
  54. 54. Lecture 10 •  Introduction •  Data collection •  Website metrics •  Optimization •  Competitive intelligence •  Surveys •  Tools and books From Code to Product Lecture 10 — Analytics— Slide 54 gidgreen.com/course
  55. 55. Why surveys? •  Customer feedback en masse – Initiated by you (email/web) – Avoid vocal minority •  Understand market – Job descriptions – Size of company – Use of product •  How did you find me? From Code to Product Lecture 10 — Analytics— Slide 55 gidgreen.com/course
  56. 56. Why surveys? •  Help with strategic decisions – Premium offerings – Major new versions •  Customer satisfaction – Quantify word of mouth •  Understand abandonment – But hard to motivate response •  Open-ended feedback From Code to Product Lecture 10 — Analytics— Slide 56 gidgreen.com/course
  57. 57. Sources of bias •  Non-response bias – Busy customer ≠ bad customer •  Response bias – Word questions objectively •  Predictions vs facts – Would you pay? How much? •  Snapshot in time – Lots of data vs ongoing data From Code to Product Lecture 10 — Analytics— Slide 57 gidgreen.com/course
  58. 58. Getting users to survey •  Prominent link in product •  Prize giveaway •  Response to support email •  Mass mailing •  Cold calling •  Bias bias bias ... From Code to Product Lecture 10 — Analytics— Slide 58 gidgreen.com/course
  59. 59. Good survey design •  Keep it short! – Focus on objectives •  Minimize burden on user – Easy questions, especially at start – Multiple choice •  Make it feel anonymous – Social desirability bias •  Free text at end From Code to Product Lecture 10 — Analytics— Slide 59 gidgreen.com/course
  60. 60. Bad questions When did you last go online and buy something? Would you buy our superior product? Are you willing to pay for things online? If we created a reliable and bug-free product which had all of the features that you requested in response to the questions in this survey, would you be willing to pay us $10 per month for it? What are you looking for? From Code to Product Lecture 10 — Analytics— Slide 60 gidgreen.com/course
  61. 61. Analyzing survey data •  Manual review – At least for free text field •  Histograms •  Pairwise correlations – Especially against price •  Clustering – Identify price points – Decide who is worth serving From Code to Product Lecture 10 — Analytics— Slide 61 gidgreen.com/course
  62. 62. Pairwise correlation From Code to Product Lecture 10 — Analytics— Slide 62 gidgreen.com/course R² = 0.04028 0 1 $0 $20 $40 $60 $80 MultipleRecipients?
  63. 63. Mini surveys From Code to Product Lecture 10 — Analytics— Slide 63 gidgreen.com/course
  64. 64. Lecture 10 •  Introduction •  Data collection •  Website metrics •  Optimization •  Competitive intelligence •  Surveys •  Tools and books From Code to Product Lecture 10 — Analytics— Slide 64 gidgreen.com/course
  65. 65. Analytics tools From Code to Product Lecture 10 — Analytics— Slide 65 gidgreen.com/course
  66. 66. Other tools From Code to Product Lecture 10 — Analytics— Slide 66 gidgreen.com/course
  67. 67. Books From Code to Product Lecture 10 — Analytics— Slide 67 gidgreen.com/course
  68. 68. We didn’t cover… •  Social media analytics – Popularity – Sentiment analysis •  Video analytics – Attention – Embeds •  Content reuse From Code to Product Lecture 10 — Analytics— Slide 68 gidgreen.com/course

×