Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1
Josh Aberant
CMO
t.co/Josh
@jaberant
Being Data Driven
@jaberant 2
Bio - Josh Aberant
SparkPost CMO & Growth
Twitter Postmaster
RestEngine CoFounder
Marketo Director of Privacy
@jaberant 3
Postmaster Role @Twitter
• Growth team
Early member of formalized Growth team built around acquihires
• Mixer ...
4
#GrowthHacking
Data in Action
Being Data Driven
@jaberant 5
@jaberant 6
#GrowthHacking - Data in Action
Growth
User State
Models
Data Experiments
7
@jaberant 8
#GrowthHacking for CX
#GrowthHacking is a capable CX framework (Customer Experience)
• CX is marketing teams e...
9
You are all #GrowthHackers
here, even if you don’t
know it yet
@jaberant 10
The Growth Role
Blend data & deep understanding of user
• Needs, habits, perception, tastes, preferences
Answ...
@jaberant 11
The Growth Role
Data & Experiments Infrastructure Services
• Analytics & experiments
Roll your own (work with...
@jaberant 12
Funnels Have Become Tornados
• Funnels have been blown to bits
• User’s don’t take linear paths to and
throug...
@jaberant 13
Managing Tornadoes
• Breakdown all user mechanics
• Measure & instrument
• Experiment to improve
The classic ...
@jaberant 14
Experimental Results = User Insights
Through experiments we use real world data that to
generate user insight...
15
There’s no magic code
“There is no code. Ultimately growth is a
function of the right mix of a product that
meet import...
#GrowthHacking is
getting users hooked on
your apps & products
More
scientifically...
18
#GrowthHacking = Science
Growth Hacking is a framework to study how users
engage so as to give them a more meaningful a...
@jaberant 19
Systematic & Repeatable
There’s really 3 main things #GrowthHacking does
• See how users activate & engage
• ...
@jaberant 20
To Grow
Instrument & measure
• Observe & understand
Innovate through experimentation
• Try things
• Optimize ...
21
The1%experiments
@jaberant 22
#BeingDataDriven
Life inside a data driven organization
• Anyone can do a 1% experiment
• Indeed, don’t show ...
23
Experiments
24
Experiments are what
make #GrowthHacking
scientific
@jaberant 25
Experiments Made Easy
@jaberant 26
Experiments Made Easy
Test Group
@jaberant 27
Experiments Made Easy
Test Group
Run the change on the test group
Leave the rest alone (they are the control)...
@jaberant 28
Working With Experiments
#BeingDataDriven means a regular cadence of test &
reviews
• Daily standups, weekly ...
@jaberant 29
Focus on what works
Double down on what already works
• With limited resources you don’t have time to work on...
Instrumentation &
User State Models
31
Instrument the metrics
that tell you if your users
are healthy
“Based on experience of which companies stuck with us and
which didn't, we decided that any team that has exchanged
2,000 ...
@jaberant 33
Example Healthy User Metrics
Facebook
• Getting a user to 7 friends in 10 days
Dropbox
• Getting a new user t...
@jaberant 34
Some Common Engagement Metrics
• Logins past 30 days
• Days since last login
• Session length
• Session inter...
@jaberant 35
Metrics That Matter
Ask
• Do metrics indicate that a user is healthy?
• Which phase of the tornado (funnel)?
...
@jaberant 36
User State Models
User state models simplify working with a lot of metrics by
aggregating metrics them into m...
37
User State Models
Growth is all about the quality
and quantity of the user base.
User state models provide a
way of mea...
@jaberant 38
New
Casual Core
At
Risk
User State Models Examples
@jaberant 39
Measuring User State Transitions Example
Example from Pinterest: Measuring flows through the Tornado
• New si...
@jaberant 40
Days since last login
Logins
in past
30 days
Healthy Users
At risk:
focus
engagement
experiments
here
Transit...
@jaberant 41
RUX while you can
• User transitioning out are about to be churned users. Apply
resurrection messaging & expe...
@jaberant 42
User State Models Example
Pyramid example (for Twitter):
• Heavy tweeter
• Heavy non-tweeter
• Medium tweeter...
@jaberant 43
User State Models Example
Xd28s example (from Pinterest):
# of users Pinterest in past 28 days
• 4d28s+ = # o...
@jaberant 44
User State Models Consideration
There’s a lot of debate about whether “middle” user states
are meaningful
• M...
@jaberant 45
User State Models Consideration
There’s no one model that meets all experimental cases
• All models have limi...
@jaberant 46
Counter Metrics
• Don’t lose sight of the forest in the trees. There’s a risk
your user state model doesn’t a...
@jaberant 47
More Meaningful Cohorts w/ User States
Often cohorts are looked at by individual metrics
• In any given exper...
@jaberant 48
Standard Cohort Example
A cohort is a group of people that share something in common
Default cohort analysis ...
@jaberant 49
Cohorts with Users States
If the experiment makes more users healthy it’s a good
experiment
Cohort
Health
Ses...
@jaberant 50
Interactive Dashboard Best Practices
Use dashboards!
• Relevant metrics
• Visually engaging
You’ve to to get ...
@jaberant 51
Interactive Dashboard Best Practices
Keep it current!! - up-to-date data
• Min, hour, week, month, quarter - ...
@jaberant 52
Interactive Dashboard Best Practices
Alerts
• Altering on growth systems
• E.g. email volume & push volume (o...
@jaberant 53
Keep Dashboards Relevant
Key metrics can shift as growth happens and challenges
change
• Maintain your dashbo...
@jaberant 54
Growth Surfaces
NUX
RUX
• The best parts of NUX
Messaging
• Email, Push, app updates
Logged out/ unauthed use...
@jaberant 55
New User Experience (NUX)
Often the Growth team owns the NUX and it is a focus of
intense experimentation
• A...
@jaberant 56
Activation in the NUX
What is the “aha” moment or “must have” experience?
• How do you get users to this poin...
57
Growth comes from
delivering the core product
value to users as early and
often as possible.
@jaberant 58
Resurrected User Experience (RUX)
Often the Growth team own the RUX and it is a focus of
another focus intens...
59
Resurrection is hard
It’s very hard to resurrect people without being
pushy. But if done right emails, notifications an...
@jaberant 60
Messaging
The Growth team often provides messaging capabilities
across the enterprise
• Infrastructure, Servi...
61
App Indexing
via
Deeplinking
@jaberant 62
SEO is back
With App Indexing Google will deeplink from search results
into content in your app.
• 30+ billio...
@jaberant 63
• Why would users want to share your app or product’s
atomic unit?
• How can users share the atomic unit?
• W...
@jaberant 64
Vitality - Your app’s atomic unit
What is the fundamental object that people are consuming
and/or creating?
S...
65
@jaberant 66
Social Proof - Virality Best Practice
Example from Nuzzel App - nuzzel.com
@jaberant 67
Social Proof Best Practices
Pictures
People
• Connections
• Experts
• Celebrities
• Similar people
Don’t incl...
68
“We bet heavily on Twitter. Even if
someone is incredibly enthusiastic about a
product, literal word of mouth will only...
69
Listen
70
@jaberant 71
Listen To Your Users
App & product users will communicate
• They want the stuff they are using to get better
...
72
“If you put that all together, we probably
get 8,000 Zendesk help tickets and 10,000
tweets per month, and we respond t...
73
Finally a word of warning
74
Don’t Spam
@jaberant 75
Don’t Spam
Spam ruins the internet
Users don’t like it
Can easily waste so much good work
Virality doesn’t ma...
76
Now go forth & #GrowthHack
77
Josh Aberant
CMO
t.co/Josh
@jaberant
Being Data Driven
Upcoming SlideShare
Loading in …5
×

Josh Aberant - Data-Driven Digital Growth

475 views

Published on

What does it mean to be in a truly data-driven organization? Josh Aberant dives into the data-driven culture that was the foundation of all decisions within the Twitter Growth team. Hear how #growthhacking can turn data nerds into superstars. In this session, learn methods for making data insights impactful on the business, as well as the benefits of enacting 1% experiments that anyone can do. Dive into some users state models and see how they can help scale data-driven decision making. He’ll cover the best practices that help make a data-driven organization successful and ultra-competitive in an environment where many are still struggling to just get by.

Published in: Technology
  • Be the first to comment

Josh Aberant - Data-Driven Digital Growth

  1. 1. 1 Josh Aberant CMO t.co/Josh @jaberant Being Data Driven
  2. 2. @jaberant 2 Bio - Josh Aberant SparkPost CMO & Growth Twitter Postmaster RestEngine CoFounder Marketo Director of Privacy
  3. 3. @jaberant 3 Postmaster Role @Twitter • Growth team Early member of formalized Growth team built around acquihires • Mixer Labs, Summify, Fluther, RestEngine & more • Build & optimize growth surfaces • Work on Growth surfaces • NUX, RUX, messaging, logged out, distribution/viral • Provide growth services across enterprise Data as a service Experiments as a service • Lots of teams needed/consumed growth services Led to a lot of security work Security discussion is for another day. We are hear to talk #Growth
  4. 4. 4 #GrowthHacking Data in Action Being Data Driven
  5. 5. @jaberant 5
  6. 6. @jaberant 6 #GrowthHacking - Data in Action Growth User State Models Data Experiments
  7. 7. 7
  8. 8. @jaberant 8 #GrowthHacking for CX #GrowthHacking is a capable CX framework (Customer Experience) • CX is marketing teams embracing both data & empathy for the user • The whole breadth of user interactions • Acquisition, Engagement, Loyalty & Advocacy • Listening to and co-designing with the users • #GrowthHacking is methodology for scaling empathy • #GrowthHacking measurable & repeatable way to get at the essence of what the user wants, enjoys & needs
  9. 9. 9 You are all #GrowthHackers here, even if you don’t know it yet
  10. 10. @jaberant 10 The Growth Role Blend data & deep understanding of user • Needs, habits, perception, tastes, preferences Answer questions on user experience. e.g.: • Why are users dropping out of the sign up experience? • Why don’t users come back to the application after the initial download? • Why aren’t users responding to special offers? Experiment to collect real meaningful feedback • Innovate through experimentation
  11. 11. @jaberant 11 The Growth Role Data & Experiments Infrastructure Services • Analytics & experiments Roll your own (work with the Data scientist & devs) or Commercially available solutions - Google Analytics to Optimizely Dashboards • State Models Growth Plan • Define the growth objectives • Decide where in the tornado (funnel) to focus e.g. User acquisition or fighting churn
  12. 12. @jaberant 12 Funnels Have Become Tornados • Funnels have been blown to bits • User’s don’t take linear paths to and through apps or products • They get to your app or product how they want to, when they want to and do whatever they want to do • Users own the relationship
  13. 13. @jaberant 13 Managing Tornadoes • Breakdown all user mechanics • Measure & instrument • Experiment to improve The classic funnel stages still matter - users just don’t go through them linearly • Acquisition, activation, engagement, retention, resurrection
  14. 14. @jaberant 14 Experimental Results = User Insights Through experiments we use real world data that to generate user insights that are fed into the enterprise • Empathy • Thinking like the users think • Understanding essence of users environment, tasks & goals • Living in the world of users & enhancing their experience This isn’t just polling & surveys - experiments enable to see what users truly do when changes are made.
  15. 15. 15 There’s no magic code “There is no code. Ultimately growth is a function of the right mix of a product that meet important user needs and tenacious creative data driven execution that leverage principles of growth and often exploits psychological triggers.” -Sean Ellis
  16. 16. #GrowthHacking is getting users hooked on your apps & products
  17. 17. More scientifically...
  18. 18. 18 #GrowthHacking = Science Growth Hacking is a framework to study how users engage so as to give them a more meaningful app or product that they'll want to use & share. If you want to build apps or products that people are hooked to then make growth hacking core to your development.
  19. 19. @jaberant 19 Systematic & Repeatable There’s really 3 main things #GrowthHacking does • See how users activate & engage • Get them to activate & engage more • Get them to share
  20. 20. @jaberant 20 To Grow Instrument & measure • Observe & understand Innovate through experimentation • Try things • Optimize things • Throw out what doesn’t work, do what does Be a champion of the user
  21. 21. 21 The1%experiments
  22. 22. @jaberant 22 #BeingDataDriven Life inside a data driven organization • Anyone can do a 1% experiment • Indeed, don’t show up to present to execs without experimental data • Innovate through experimentation!
  23. 23. 23 Experiments
  24. 24. 24 Experiments are what make #GrowthHacking scientific
  25. 25. @jaberant 25 Experiments Made Easy
  26. 26. @jaberant 26 Experiments Made Easy Test Group
  27. 27. @jaberant 27 Experiments Made Easy Test Group Run the change on the test group Leave the rest alone (they are the control) Measure the change in the metrics you care about
  28. 28. @jaberant 28 Working With Experiments #BeingDataDriven means a regular cadence of test & reviews • Daily standups, weekly team review, set schedule of reviews around initiatives, backlog mgmt Learn to have instincts point to experimental success • Often an experiment need more time to show results • Or it needs to be expanded beyond 1% Make decisions on how to proceed • Expand, reformulate, place on hold, kill
  29. 29. @jaberant 29 Focus on what works Double down on what already works • With limited resources you don’t have time to work on everything • It’s easier to build on a strength than to improve a weakness • It’s often easier to improve a conversion rate that’s already doing well • Easier to increase engagement of active users compared to people who are inactive
  30. 30. Instrumentation & User State Models
  31. 31. 31 Instrument the metrics that tell you if your users are healthy
  32. 32. “Based on experience of which companies stuck with us and which didn't, we decided that any team that has exchanged 2,000 messages in its history has tried Slack — really tried it. For a team around 50 people that means about 10 hours’ worth of messages. For a typical team of 10 people, that’s maybe a week’s worth of messages. But it hit us that, regardless of any other factor, after 2,000 messages, 93% of those customers are still using Slack today.” - Stewart Butterfield, Slack Founder 32 Healthy User Example
  33. 33. @jaberant 33 Example Healthy User Metrics Facebook • Getting a user to 7 friends in 10 days Dropbox • Getting a new user to link a computer and add a file
  34. 34. @jaberant 34 Some Common Engagement Metrics • Logins past 30 days • Days since last login • Session length • Session interval frequency (how often user launches app) • App screens per session • Interactions • Post • Shares • Views
  35. 35. @jaberant 35 Metrics That Matter Ask • Do metrics indicate that a user is healthy? • Which phase of the tornado (funnel)? Activation, activation, engagement, vitality, resurrection Build • User state models Specific to your user paths (tornado)
  36. 36. @jaberant 36 User State Models User state models simplify working with a lot of metrics by aggregating metrics them into meaningful models that can be used to quickly make decisions on the health of an experimental result • Examples Healthy vs Unhealthy Casual or Core
  37. 37. 37 User State Models Growth is all about the quality and quantity of the user base. User state models provide a way of measuring the quality.
  38. 38. @jaberant 38 New Casual Core At Risk User State Models Examples
  39. 39. @jaberant 39 Measuring User State Transitions Example Example from Pinterest: Measuring flows through the Tornado • New signup: User joins • New -> Dormant: New users doesn’t engage for 28 days • MAU -> Dormant: User was an MAU, but doesn’t engage for 28 days • Dormant -> MAU: Users engages after not using app for 28 days
  40. 40. @jaberant 40 Days since last login Logins in past 30 days Healthy Users At risk: focus engagement experiments here Transitioning out: focus Resurrected User Experience (RUX) experiments here User State Models Examples
  41. 41. @jaberant 41 RUX while you can • User transitioning out are about to be churned users. Apply resurrection messaging & experiments to them before they leave while it is still much easier to reach them. • RUX is generally a subset - the best parts of - the NUX (New User Experience)
  42. 42. @jaberant 42 User State Models Example Pyramid example (for Twitter): • Heavy tweeter • Heavy non-tweeter • Medium tweeters • Medium non-tweeters • Light users • Very light • Nearly light (new users)
  43. 43. @jaberant 43 User State Models Example Xd28s example (from Pinterest): # of users Pinterest in past 28 days • 4d28s+ = # of users that used Pinterest 4 or more days in last 28 Major categories • Core = 14d28s+ • Casual = 4d28s+ • Marginal = <4d28 Example provided by Pinterest Eng Blog
  44. 44. @jaberant 44 User State Models Consideration There’s a lot of debate about whether “middle” user states are meaningful • Medium, casual Or wether is best to always think of these states as transitioning out • “I’m about out of here” group
  45. 45. @jaberant 45 User State Models Consideration There’s no one model that meets all experimental cases • All models have limits • Different models work better in different cases Always consider whether a given model is right for the experiments
  46. 46. @jaberant 46 Counter Metrics • Don’t lose sight of the forest in the trees. There’s a risk your user state model doesn’t apply well to the experiment • Best practice: Pair every metric with appropriate counter metric Signups with activations Activations with churn New paid customers with total revenue This encourages a holistic approach to growth & will help keep you on the right track on when your user state models or serving you well or when you should be using different user state models.
  47. 47. @jaberant 47 More Meaningful Cohorts w/ User States Often cohorts are looked at by individual metrics • In any given experiment the metrics might move different metrics in different directions • Difficult to evaluate an experiment’s results Use user state models in your cohort analysis • Scale decision making • Incorporate multiple metrics • Tells you what to do next If the experiment makes more users healthy its a good experiment
  48. 48. @jaberant 48 Standard Cohort Example A cohort is a group of people that share something in common Default cohort analysis often means clustering users by when they started & seeing how behavior changes over time • E.g. 1 week old cohorts in Feb vs March when a big app change was made at beginning of March Lots of great commercial solutions • Google Analytics, Kissmetrics, Keen.io, Segment, Mixpanel
  49. 49. @jaberant 49 Cohorts with Users States If the experiment makes more users healthy it’s a good experiment Cohort Health Session length Session interval App Screes per session Interactions Views Post + + + + + - + - - - + + + + - + - - - - - + + + - - - -
  50. 50. @jaberant 50 Interactive Dashboard Best Practices Use dashboards! • Relevant metrics • Visually engaging You’ve to to get everyone on the team looking at & using these dashboards • Intuitive default selections • Super easy basics analytical tasks Filtering view, adjusting parameters, drill to examine underlying data Allow team members to understand data and drill down in 1-2 clicks
  51. 51. @jaberant 51 Interactive Dashboard Best Practices Keep it current!! - up-to-date data • Min, hour, week, month, quarter - whatever the right timeline is for best managing this growth function Label metrics meaningfully • Meaningful to others • Make sure metrics support the objective & the audience) Stay away from chart junk • Low value graphics or unintelligible widgets
  52. 52. @jaberant 52 Interactive Dashboard Best Practices Alerts • Altering on growth systems • E.g. email volume & push volume (on per type basis.) • In addition to core product alerting • Need good dashboards & alerting so when alerts go off you can figure out what went wrong
  53. 53. @jaberant 53 Keep Dashboards Relevant Key metrics can shift as growth happens and challenges change • Maintain your dashboards • Are they meeting the current needs of the team?
  54. 54. @jaberant 54 Growth Surfaces NUX RUX • The best parts of NUX Messaging • Email, Push, app updates Logged out/ unauthed user state • SEO • Signups Deeplinking (distribution) • App indexing Viral mechanism
  55. 55. @jaberant 55 New User Experience (NUX) Often the Growth team owns the NUX and it is a focus of intense experimentation • Activate Users Quickly orient & teach users to use your app our service • Move users to a healthy state E.g. Follow people, add friends, link a file
  56. 56. @jaberant 56 Activation in the NUX What is the “aha” moment or “must have” experience? • How do you get users to this point as fast as possible (i.e. in seconds)? • What actions should users take to put them in a healthy state?
  57. 57. 57 Growth comes from delivering the core product value to users as early and often as possible.
  58. 58. @jaberant 58 Resurrected User Experience (RUX) Often the Growth team own the RUX and it is a focus of another focus intense experimentation • How do you bring users back and reactivate them? • What’s changed since the users was last there that they’ll want to stick around longer? • Do you have a way of reaching churned users? Email Notifications App store updates Custom channels
  59. 59. 59 Resurrection is hard It’s very hard to resurrect people without being pushy. But if done right emails, notifications and app store updates can do much for bringing users back #KeepItClassy
  60. 60. @jaberant 60 Messaging The Growth team often provides messaging capabilities across the enterprise • Infrastructure, Services & best practices • Email, push, sms, in-app, app store updates, custom channels • Policy & controls (based on fatigue models) to prevent different teams from over-mailing or over-pushing Prevent user fatigue
  61. 61. 61 App Indexing via Deeplinking
  62. 62. @jaberant 62 SEO is back With App Indexing Google will deeplink from search results into content in your app. • 30+ billion deeplinks & counting • Powerful distribution • Think about the landing experience • The years sent on developing best practices for landing pages on the web... App Indexing + Adwords = Search download ads
  63. 63. @jaberant 63 • Why would users want to share your app or product’s atomic unit? • How can users share the atomic unit? • What are the mechanics of sharing? • How do you reduce sharing friction? • How do you get users to recommend your app? Virality
  64. 64. @jaberant 64 Vitality - Your app’s atomic unit What is the fundamental object that people are consuming and/or creating? Some examples: • Twitter – tweet • Soundcloud - audio track • Instagram – photo • Tumblr – post • Dropbox – file • Codecademy – lesson
  65. 65. 65
  66. 66. @jaberant 66 Social Proof - Virality Best Practice Example from Nuzzel App - nuzzel.com
  67. 67. @jaberant 67 Social Proof Best Practices Pictures People • Connections • Experts • Celebrities • Similar people Don’t include if poor quality
  68. 68. 68 “We bet heavily on Twitter. Even if someone is incredibly enthusiastic about a product, literal word of mouth will only get to a handful of people — but if someone tweets about us, it can be seen by hundreds, even thousands.” - Stewart Butterfield, Slack Founder Share via Twitter
  69. 69. 69 Listen
  70. 70. 70
  71. 71. @jaberant 71 Listen To Your Users App & product users will communicate • They want the stuff they are using to get better Give them ways to talk back to you • Button to talk to you, in-app prompt, surveys Find guidance on what they want • Then experiment to confirm Tag, ticket & study messages from users • Be fastidious
  72. 72. 72 “If you put that all together, we probably get 8,000 Zendesk help tickets and 10,000 tweets per month, and we respond to all of them.” ” - Stewart Butterfield, Slack Founder Listen with Twitter
  73. 73. 73 Finally a word of warning
  74. 74. 74 Don’t Spam
  75. 75. @jaberant 75 Don’t Spam Spam ruins the internet Users don’t like it Can easily waste so much good work Virality doesn’t matter until you can deliver a compelling benefit to users • Nothing worth sharing means just that Avoid gimmicks that drive sign-ups in the short-term but alienate users in the long-run
  76. 76. 76 Now go forth & #GrowthHack
  77. 77. 77 Josh Aberant CMO t.co/Josh @jaberant Being Data Driven

×