Metrics-Driven Design                            by Joshua Porter                            Twitter: bokardo             ...
Saturday, March 12, 2011
“Unfortunately for me, there was one                           small problem I didn’t see back then.”Saturday, March 12, 2...
Doug Bowman on Design at Google      “                    Yes, it’s true that a team at Google couldn’t decide between two...
41 Shades of Blue Test                                                        link color        Gmail                     ...
Saturday, March 12, 2011
#2200C1          #0044CC                           Google             BingSaturday, March 12, 2011
“                    That blue was worth at                           least $80 million.                           Paul Ra...
Doug Bowman on Design at Google      “                    When a company is filled with engineers, it turns to engineering ...
Spectrum of Design   Intuition-Driven                                                                               Data-D...
Imagine that your design is a mountain...      Metrics-Driven Design               The grass is always greener on other mo...
Imagine that your design is a mountain...                    EXISTING DESIGN      Metrics-Driven Design               The ...
Imagine that your design is a mountain...                               Now                    EXISTING DESIGN      Metric...
Imagine that your design is a mountain...                 Optimized (local maxima)                               Now      ...
Imagine that your design is a mountain...           Potential           Improvement                 Optimized (local maxim...
Imagine that your design is a mountain...                                        Goal                                     ...
Imagine that your design is a mountain...                                      Goal           Potential           Improvem...
Optimization asks:                           What works best in the current model?                                    Desi...
What are Metrics?Saturday, March 12, 2011
DEFINITION                  Metrics are simply numbers                that measure the effectiveness                       ...
5 Reasons why Metrics are a                          Designer’s Best FriendSaturday, March 12, 2011
1. Metrics reduce arguments                          based on opinion.Saturday, March 12, 2011
2. Metrics give you answers                            about what really works.Saturday, March 12, 2011
3. Metrics show you where                           you’re strong as a designer.Saturday, March 12, 2011
4. Metrics allow you to test                               anything you want.Saturday, March 12, 2011
5. Clients love metrics.Saturday, March 12, 2011
PRINCIPLE                  Your metrics will be as unique                        as your business.Saturday, March 12, 2011
Saturday, March 12, 2011
Vanity Metrics      Metrics-Driven Design   Ego   SXSWSaturday, March 12, 2011
The Usage LifecycleSaturday, March 12, 2011
Interested   Trial/beta User   Customer   Passionate                                                             CustomerS...
Acquisition          Conversion      Engagement        Satisfaction           Interested               Trial/beta User    ...
With metrics you are essentially just measuring                how well you move people along the lifecycle!Saturday, Marc...
Acquisition MetricsSaturday, March 12, 2011
CPA - Cost per Acquisition                           If your CPA is higher than your LTV                                  ...
Acquisition vs. Referral: Dropbox                                            Ran Google Adwords campaigns                 ...
Dropbox Lifecycle                              30 days prior to April 2010, Dropbox users                              sen...
Performable Acquisition Metrics                      Comparative Metrics                           Revenue by Channel     ...
Conversion MetricsSaturday, March 12, 2011
Trials / Conversions = Conversion RateSaturday, March 12, 2011
Conversion Funnel Analysis                    A                A         100%                     B               B     60...
Sign-up Conversion Funnel       Original Flow                     1                    2                    3             ...
Sign-up Conversion Funnel          New Flow                                                  2                          4 ...
Engagement Matters: Twitter                                                     http://www.lukew.com/ff/entry.asp?1039    ...
Engagement MetricsSaturday, March 12, 2011
•   Hits                           •   Page views                           •   Visits                           •   Uniqu...
Cohort Analysis                                                                Engagement over time                      M...
Prevention: Facebook Deactivation                                                      Design changes to                  ...
Satisfaction MetricsSaturday, March 12, 2011
Referral: Net Promoter Score            How likely is it that you would recommend              our company to a friend or ...
Net Promoter Score            How likely is it that you would recommend              our company to a friend or colleague?...
Mint.com & Net Promoter Score      “                    Maybe we didn’t have a high viral coefficient                       ...
Emergent MetricsSaturday, March 12, 2011
Emergent Metrics: 5 Friends                                      The magic number is 5.                                   ...
Is there one metric that drives others?      “                    At Blogger, we determined that our most critical        ...
Principles of Metrics-Driven Design               1. Optimize in small steps; innovate with daring leaps.               2....
Saturday, March 12, 2011
This Talk.       http://bokardo.com/talks/metrics-driven-design/Saturday, March 12, 2011
My Life.                    http://performable.com              http://oneflightbooks.comSaturday, March 12, 2011
Upcoming SlideShare
Loading in...5
×

Metrics Driven Design by Joshua Porter

44,475
-1

Published on

Published in: Technology, Business
5 Comments
195 Likes
Statistics
Notes
No Downloads
Views
Total Views
44,475
On Slideshare
0
From Embeds
0
Number of Embeds
20
Actions
Shares
0
Downloads
963
Comments
5
Likes
195
Embeds 0
No embeds

No notes for slide

Metrics Driven Design by Joshua Porter

  1. Metrics-Driven Design by Joshua Porter Twitter: bokardo bokardo 52weeksofUX performable microcopy designing for the social web make them care! by Joshua PorterSaturday, March 12, 2011
  2. Saturday, March 12, 2011
  3. “Unfortunately for me, there was one small problem I didn’t see back then.”Saturday, March 12, 2011
  4. Doug Bowman on Design at Google “ Yes, it’s true that a team at Google couldn’t decide between two blues, so they’re testing 41 shades between each blue to see which one performs better. I had a recent debate over whether a border should be 3, 4 or 5 pixels wide, and was asked to prove my case. I can’t operate in an environment like that. Metrics-Driven Design Daring. Decisive. Conviction. SXSWSaturday, March 12, 2011
  5. 41 Shades of Blue Test link color Gmail Google Search 41 Bucket Split Test: ~2.5% of users each got a shade for 2 weeks.Saturday, March 12, 2011
  6. Saturday, March 12, 2011
  7. #2200C1 #0044CC Google BingSaturday, March 12, 2011
  8. “ That blue was worth at least $80 million. Paul Ray, UX Manager, Bing Metrics-Driven Design Microcopy SXSWSaturday, March 12, 2011
  9. Doug Bowman on Design at Google “ When a company is filled with engineers, it turns to engineering to solve problems. Reduce each decision to a simple logic problem. Remove all subjectivity and just look at the data. Data in your favor? Ok, launch it. Data shows negative effects? Back to the drawing board. And that data eventually becomes a crutch for every decision, paralyzing the company and preventing it from making any daring design decisions. Metrics-Driven Design Daring. Decisive. Conviction. SXSWSaturday, March 12, 2011
  10. Spectrum of Design Intuition-Driven Data-Driven Make best-guesses Every design choice is tested Rely on previous experience Takes others experience with a grain of salt Study what others are doing Design is a logic problem Use best practices, principles & patterns Rely on data for decision-making Aesthetics are integral Aesthetics are secondary Rely on our gut Never trust your gut Creative, visionary Cold, calculating Inherently risky Risk-averse Doug’s words: Assumed: instinctive, subjective, daring deliberate, objective, safeSaturday, March 12, 2011
  11. Imagine that your design is a mountain... Metrics-Driven Design The grass is always greener on other mountainsides. SXSWSaturday, March 12, 2011
  12. Imagine that your design is a mountain... EXISTING DESIGN Metrics-Driven Design The grass is always greener on other mountainsides. SXSWSaturday, March 12, 2011
  13. Imagine that your design is a mountain... Now EXISTING DESIGN Metrics-Driven Design The grass is always greener on other mountainsides. SXSWSaturday, March 12, 2011
  14. Imagine that your design is a mountain... Optimized (local maxima) Now EXISTING DESIGN Metrics-Driven Design The grass is always greener on other mountainsides. SXSWSaturday, March 12, 2011
  15. Imagine that your design is a mountain... Potential Improvement Optimized (local maxima) Now Metrics-Driven Design The grass is always greener on other mountainsides. SXSWSaturday, March 12, 2011
  16. Imagine that your design is a mountain... Goal BETTER DESIGN Current Design EXISTING DESIGN Metrics-Driven Design The grass is always greener on other mountainsides. SXSWSaturday, March 12, 2011
  17. Imagine that your design is a mountain... Goal Potential Improvement BETTER DESIGN Now EXISTING DESIGN Metrics-Driven Design The grass is always greener on other mountainsides. SXSWSaturday, March 12, 2011
  18. Optimization asks: What works best in the current model? Design innovation asks: What is the best possible model?Saturday, March 12, 2011
  19. What are Metrics?Saturday, March 12, 2011
  20. DEFINITION Metrics are simply numbers that measure the effectiveness of your business.Saturday, March 12, 2011
  21. 5 Reasons why Metrics are a Designer’s Best FriendSaturday, March 12, 2011
  22. 1. Metrics reduce arguments based on opinion.Saturday, March 12, 2011
  23. 2. Metrics give you answers about what really works.Saturday, March 12, 2011
  24. 3. Metrics show you where you’re strong as a designer.Saturday, March 12, 2011
  25. 4. Metrics allow you to test anything you want.Saturday, March 12, 2011
  26. 5. Clients love metrics.Saturday, March 12, 2011
  27. PRINCIPLE Your metrics will be as unique as your business.Saturday, March 12, 2011
  28. Saturday, March 12, 2011
  29. Vanity Metrics Metrics-Driven Design Ego SXSWSaturday, March 12, 2011
  30. The Usage LifecycleSaturday, March 12, 2011
  31. Interested Trial/beta User Customer Passionate CustomerSaturday, March 12, 2011
  32. Acquisition Conversion Engagement Satisfaction Interested Trial/beta User Customer Passionate CustomerSaturday, March 12, 2011
  33. With metrics you are essentially just measuring how well you move people along the lifecycle!Saturday, March 12, 2011
  34. Acquisition MetricsSaturday, March 12, 2011
  35. CPA - Cost per Acquisition If your CPA is higher than your LTV then you’re in trouble.Saturday, March 12, 2011
  36. Acquisition vs. Referral: Dropbox Ran Google Adwords campaigns to drive traffic to dropbox.com The traffic that completed the lifecycle: 1) Searched on a keyword 2) Visited their site 3) Signed-up for service 4) Became a customer cost them $233-$388 per person! (for a $99 product) Metrics-Driven Design LTV = Lifetime value SXSWSaturday, March 12, 2011
  37. Dropbox Lifecycle 30 days prior to April 2010, Dropbox users sent 2.8 million direct referral invites. Dropbox’s referral program with 2-sided incentive increased sign-ups by 60% permanently. Metrics-Driven Design Referral = Word of Mouth SXSWSaturday, March 12, 2011
  38. Performable Acquisition Metrics Comparative Metrics Revenue by Channel Revenue by Keyword Metrics-Driven Design Microcopy SXSWSaturday, March 12, 2011
  39. Conversion MetricsSaturday, March 12, 2011
  40. Trials / Conversions = Conversion RateSaturday, March 12, 2011
  41. Conversion Funnel Analysis A A 100% B B 60% C C 20% Metrics-Driven Design SXSWSaturday, March 12, 2011
  42. Sign-up Conversion Funnel Original Flow 1 2 3 4 confirm personal info add your friends invite others getting started Original Conversion Funnel 1 100% of the 100% of people who started the sign-up process, only 2 63% 14% made it to the getting started screen. 3 26% 4 14% Metrics-Driven Design SXSWSaturday, March 12, 2011
  43. Sign-up Conversion Funnel New Flow 2 4 add your friends getting started New Conversion Funnel 2 100% of the 100% of people who started the sign-up process, 86% made it to the getting started screen. 4 86% Metrics-Driven Design Removing stuff is a quick way to improve your funnel SXSWSaturday, March 12, 2011
  44. Engagement Matters: Twitter http://www.lukew.com/ff/entry.asp?1039 Metrics-Driven Design Speed without context is wasted. SXSWSaturday, March 12, 2011
  45. Engagement MetricsSaturday, March 12, 2011
  46. • Hits • Page views • Visits • Unique Visitors • Returning Visitors • Registered Users • Customers • Frequency • Time on Site • Daily Active UsersSaturday, March 12, 2011
  47. Cohort Analysis Engagement over time Month 1 Month 2 Month 3 Month 4 Month 5 Month 6 Month 7 Month 8 Month 9 Month 10 Month 11 Month 12 (Joined in) 100% 20% 19% 13% 13% 10% 12% 11% 7% 7% 7% ? January 100% 21% 16% 13% 11% 9% 9% 7% 7% 7% ? February 100% 24% 20% 17% 15% 13% 11% 10% 10% ? March 100% 31% 27% 24% 19% 15% 12% 12% ? April 100% 31% 27% 25% 21% 18% 16% ? May 100% 39% 28% 24% 20% 19% ? June 100% 40% 33% 27% 23% ? July 100% 47% 41% 32% ? August 100% 52% 43% ? September In this case engagement is improving nicely. 100% 53% ? Of the January cohort, only 20% were engaged in month 2. October Of the October cohort, 53% were engaged in month 2. 100% ? November ? December http://redeye.firstround.com/2008/01/after-the-techc.htmlSaturday, March 12, 2011
  48. Prevention: Facebook Deactivation Design changes to the deactivation page accounted for 1 million members not leaving the service. Julie Zhou, Facebook http://www.readwriteweb.com/archives/what_happens_when_you_deactivate_your_facebook_acc.php Metrics-Driven Design I will miss you after my talk today. SXSWSaturday, March 12, 2011
  49. Satisfaction MetricsSaturday, March 12, 2011
  50. Referral: Net Promoter Score How likely is it that you would recommend our company to a friend or colleague? 0 1 2 3 4 5 6 7 8 9 10 Metrics-Driven Design Do you know what your net promoter score is? SXSWSaturday, March 12, 2011
  51. Net Promoter Score How likely is it that you would recommend our company to a friend or colleague? 0 1 2 3 4 5 6 7 8 9 10 Detractors Passives Promoters Score = % Promoters - % Detractors Metrics-Driven Design Do you know what your net promoter score is? SXSWSaturday, March 12, 2011
  52. Mint.com & Net Promoter Score “ Maybe we didn’t have a high viral coefficient but we had a great net promoter score. Jason Putorti, Lead Designer, Mint.com Metrics-Driven Design SXSWSaturday, March 12, 2011
  53. Emergent MetricsSaturday, March 12, 2011
  54. Emergent Metrics: 5 Friends The magic number is 5. Once a FriendFeed user found five friends, they became active users. Bret Taylor, Friendfeed Metrics-Driven Design SXSWSaturday, March 12, 2011
  55. Is there one metric that drives others? “ At Blogger, we determined that our most critical metric was number of posts. An increase in posts meant that people were not just creating blogs, but updating them, and more posts would drive more readership, which would drive more users, which would drive more posts. Ev Williams founder of Blogger & Twitter Metrics-Driven Design Once you’re lucky, twice you’re good. SXSWSaturday, March 12, 2011
  56. Principles of Metrics-Driven Design 1. Optimize in small steps; innovate with daring leaps. 2. No design survives contact with the user. 3. Small improvements, taken together, yield amazing results.  4. Testing is empowering, reversion is cleansing. 5. Metrics are not creative: human beings are. 6. All team members are responsible for the user experience. 7. If metrics aren’t actionable, they aren’t useful. 8. Design is never done. Metrics-Driven Design Reminder: Principal = Person & Principle = Thing SXSWSaturday, March 12, 2011
  57. Saturday, March 12, 2011
  58. This Talk. http://bokardo.com/talks/metrics-driven-design/Saturday, March 12, 2011
  59. My Life. http://performable.com http://oneflightbooks.comSaturday, March 12, 2011
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×