What does it mean to be in a truly data-driven organization? Josh Aberant dives into the data-driven culture that was the foundation of all decisions within the Twitter Growth team. Hear how #growthhacking can turn data nerds into superstars. In this session, learn methods for making data insights impactful on the business, as well as the benefits of enacting 1% experiments that anyone can do. Dive into some users state models and see how they can help scale data-driven decision making. He’ll cover the best practices that help make a data-driven organization successful and ultra-competitive in an environment where many are still struggling to just get by.
2. @jaberant 2
Bio - Josh Aberant
SparkPost CMO & Growth
Twitter Postmaster
RestEngine CoFounder
Marketo Director of Privacy
3. @jaberant 3
Postmaster Role @Twitter
• Growth team
Early member of formalized Growth team built around acquihires
• Mixer Labs, Summify, Fluther, RestEngine & more
• Build & optimize growth surfaces
• Work on Growth surfaces
• NUX, RUX, messaging, logged out, distribution/viral
• Provide growth services across enterprise
Data as a service
Experiments as a service
• Lots of teams needed/consumed growth services
Led to a lot of security work
Security discussion is for another day. We are hear to talk #Growth
8. @jaberant 8
#GrowthHacking for CX
#GrowthHacking is a capable CX framework (Customer Experience)
• CX is marketing teams embracing both data & empathy for the user
• The whole breadth of user interactions
• Acquisition, Engagement, Loyalty & Advocacy
• Listening to and co-designing with the users
• #GrowthHacking is methodology for scaling empathy
• #GrowthHacking measurable & repeatable way to get at the
essence of what the user wants, enjoys & needs
9. 9
You are all #GrowthHackers
here, even if you don’t
know it yet
10. @jaberant 10
The Growth Role
Blend data & deep understanding of user
• Needs, habits, perception, tastes, preferences
Answer questions on user experience. e.g.:
• Why are users dropping out of the sign up
experience?
• Why don’t users come back to the application after
the initial download?
• Why aren’t users responding to special offers?
Experiment to collect real meaningful feedback
• Innovate through experimentation
11. @jaberant 11
The Growth Role
Data & Experiments Infrastructure Services
• Analytics & experiments
Roll your own (work with the Data scientist & devs) or
Commercially available solutions - Google Analytics to
Optimizely
Dashboards
• State Models
Growth Plan
• Define the growth objectives
• Decide where in the tornado (funnel) to focus
e.g. User acquisition or fighting churn
12. @jaberant 12
Funnels Have Become Tornados
• Funnels have been blown to bits
• User’s don’t take linear paths to and
through apps or products
• They get to your app or product how they
want to, when they want to and do
whatever they want to do
• Users own the relationship
13. @jaberant 13
Managing Tornadoes
• Breakdown all user mechanics
• Measure & instrument
• Experiment to improve
The classic funnel stages still matter - users just don’t go
through them linearly
• Acquisition, activation, engagement, retention,
resurrection
14. @jaberant 14
Experimental Results = User Insights
Through experiments we use real world data that to
generate user insights that are fed into the enterprise
• Empathy
• Thinking like the users think
• Understanding essence of users environment, tasks &
goals
• Living in the world of users & enhancing their experience
This isn’t just polling & surveys - experiments enable to see
what users truly do when changes are made.
15. 15
There’s no magic code
“There is no code. Ultimately growth is a
function of the right mix of a product that
meet important user needs and tenacious
creative data driven execution that
leverage principles of growth and often
exploits psychological triggers.”
-Sean Ellis
18. 18
#GrowthHacking = Science
Growth Hacking is a framework to study how users
engage so as to give them a more meaningful app or
product that they'll want to use & share.
If you want to build apps or products that people are
hooked to then make growth hacking core to your
development.
19. @jaberant 19
Systematic & Repeatable
There’s really 3 main things #GrowthHacking does
• See how users activate & engage
• Get them to activate & engage more
• Get them to share
20. @jaberant 20
To Grow
Instrument & measure
• Observe & understand
Innovate through experimentation
• Try things
• Optimize things
• Throw out what doesn’t work, do what does
Be a champion of the user
22. @jaberant 22
#BeingDataDriven
Life inside a data driven organization
• Anyone can do a 1% experiment
• Indeed, don’t show up to present to execs without
experimental data
• Innovate through experimentation!
27. @jaberant 27
Experiments Made Easy
Test Group
Run the change on the test group
Leave the rest alone (they are the control)
Measure the change in the metrics you care about
28. @jaberant 28
Working With Experiments
#BeingDataDriven means a regular cadence of test &
reviews
• Daily standups, weekly team review, set schedule of
reviews around initiatives, backlog mgmt
Learn to have instincts point to experimental success
• Often an experiment need more time to show results
• Or it needs to be expanded beyond 1%
Make decisions on how to proceed
• Expand, reformulate, place on hold, kill
29. @jaberant 29
Focus on what works
Double down on what already works
• With limited resources you don’t have time to work on
everything
• It’s easier to build on a strength than to improve a
weakness
• It’s often easier to improve a conversion rate that’s
already doing well
• Easier to increase engagement of active users compared
to people who are inactive
32. “Based on experience of which companies stuck with us and
which didn't, we decided that any team that has exchanged
2,000 messages in its history has tried Slack — really tried
it. For a team around 50 people that means about 10 hours’
worth of messages. For a typical team of 10 people, that’s
maybe a week’s worth of messages. But it hit us that,
regardless of any other factor, after 2,000 messages, 93% of
those customers are still using Slack today.”
- Stewart Butterfield, Slack Founder
32
Healthy User Example
33. @jaberant 33
Example Healthy User Metrics
Facebook
• Getting a user to 7 friends in 10 days
Dropbox
• Getting a new user to link a computer and add a file
34. @jaberant 34
Some Common Engagement Metrics
• Logins past 30 days
• Days since last login
• Session length
• Session interval frequency (how often user launches app)
• App screens per session
• Interactions
• Post
• Shares
• Views
35. @jaberant 35
Metrics That Matter
Ask
• Do metrics indicate that a user is healthy?
• Which phase of the tornado (funnel)?
Activation, activation, engagement, vitality,
resurrection
Build
• User state models
Specific to your user paths (tornado)
36. @jaberant 36
User State Models
User state models simplify working with a lot of metrics by
aggregating metrics them into meaningful models that can
be used to quickly make decisions on the health of an
experimental result
• Examples
Healthy vs Unhealthy
Casual or Core
37. 37
User State Models
Growth is all about the quality
and quantity of the user base.
User state models provide a
way of measuring the quality.
39. @jaberant 39
Measuring User State Transitions Example
Example from Pinterest: Measuring flows through the Tornado
• New signup: User joins
• New -> Dormant: New users doesn’t engage for 28 days
• MAU -> Dormant: User was an MAU, but doesn’t engage for 28
days
• Dormant -> MAU: Users engages after not using app for 28 days
40. @jaberant 40
Days since last login
Logins
in past
30 days
Healthy Users
At risk:
focus
engagement
experiments
here
Transitioning out:
focus Resurrected User
Experience (RUX)
experiments here
User State Models Examples
41. @jaberant 41
RUX while you can
• User transitioning out are about to be churned users. Apply
resurrection messaging & experiments to them before they
leave while it is still much easier to reach them.
• RUX is generally a subset - the best parts of - the NUX
(New User Experience)
42. @jaberant 42
User State Models Example
Pyramid example (for Twitter):
• Heavy tweeter
• Heavy non-tweeter
• Medium tweeters
• Medium non-tweeters
• Light users
• Very light
• Nearly light (new users)
43. @jaberant 43
User State Models Example
Xd28s example (from Pinterest):
# of users Pinterest in past 28 days
• 4d28s+ = # of users that used Pinterest 4 or more days in
last 28
Major categories
• Core = 14d28s+
• Casual = 4d28s+
• Marginal = <4d28
Example provided by Pinterest Eng Blog
44. @jaberant 44
User State Models Consideration
There’s a lot of debate about whether “middle” user states
are meaningful
• Medium, casual
Or wether is best to always think of these states as
transitioning out
• “I’m about out of here” group
45. @jaberant 45
User State Models Consideration
There’s no one model that meets all experimental cases
• All models have limits
• Different models work better in different cases
Always consider whether a given model is right for the
experiments
46. @jaberant 46
Counter Metrics
• Don’t lose sight of the forest in the trees. There’s a risk
your user state model doesn’t apply well to the
experiment
• Best practice: Pair every metric with appropriate counter
metric
Signups with activations
Activations with churn
New paid customers with total revenue
This encourages a holistic approach to growth & will help
keep you on the right track on when your user state models
or serving you well or when you should be using different
user state models.
47. @jaberant 47
More Meaningful Cohorts w/ User States
Often cohorts are looked at by individual metrics
• In any given experiment the metrics might move different
metrics in different directions
• Difficult to evaluate an experiment’s results
Use user state models in your cohort analysis
• Scale decision making
• Incorporate multiple metrics
• Tells you what to do next
If the experiment makes more users healthy its a good
experiment
48. @jaberant 48
Standard Cohort Example
A cohort is a group of people that share something in common
Default cohort analysis often means clustering users by when they
started & seeing how behavior changes over time
• E.g. 1 week old cohorts in Feb vs March when a big app change was
made at beginning of March
Lots of great commercial solutions
• Google Analytics, Kissmetrics, Keen.io, Segment, Mixpanel
49. @jaberant 49
Cohorts with Users States
If the experiment makes more users healthy it’s a good
experiment
Cohort
Health
Session
length
Session
interval
App
Screes
per
session
Interactions Views Post
+ + + + + - +
- - - + + + +
- + - - - - -
+ + + - - - -
50. @jaberant 50
Interactive Dashboard Best Practices
Use dashboards!
• Relevant metrics
• Visually engaging
You’ve to to get everyone on the team looking at & using
these dashboards
• Intuitive default selections
• Super easy basics analytical tasks
Filtering view, adjusting parameters, drill to examine
underlying data
Allow team members to understand data and drill
down in 1-2 clicks
51. @jaberant 51
Interactive Dashboard Best Practices
Keep it current!! - up-to-date data
• Min, hour, week, month, quarter - whatever the right
timeline is for best managing this growth function
Label metrics meaningfully
• Meaningful to others
• Make sure metrics support the objective & the audience)
Stay away from chart junk
• Low value graphics or unintelligible widgets
52. @jaberant 52
Interactive Dashboard Best Practices
Alerts
• Altering on growth systems
• E.g. email volume & push volume (on per type basis.)
• In addition to core product alerting
• Need good dashboards & alerting so when alerts go off
you can figure out what went wrong
53. @jaberant 53
Keep Dashboards Relevant
Key metrics can shift as growth happens and challenges
change
• Maintain your dashboards
• Are they meeting the current needs of the team?
54. @jaberant 54
Growth Surfaces
NUX
RUX
• The best parts of NUX
Messaging
• Email, Push, app updates
Logged out/ unauthed user state
• SEO
• Signups
Deeplinking (distribution)
• App indexing
Viral mechanism
55. @jaberant 55
New User Experience (NUX)
Often the Growth team owns the NUX and it is a focus of
intense experimentation
• Activate Users
Quickly orient & teach users to use your app our
service
• Move users to a healthy state
E.g. Follow people, add friends, link a file
56. @jaberant 56
Activation in the NUX
What is the “aha” moment or “must have” experience?
• How do you get users to this point as fast as possible (i.e.
in seconds)?
• What actions should users take to put them in a healthy
state?
58. @jaberant 58
Resurrected User Experience (RUX)
Often the Growth team own the RUX and it is a focus of
another focus intense experimentation
• How do you bring users back and reactivate them?
• What’s changed since the users was last there that they’ll
want to stick around longer?
• Do you have a way of reaching churned users?
Email
Notifications
App store updates
Custom channels
59. 59
Resurrection is hard
It’s very hard to resurrect people without being
pushy. But if done right emails, notifications and app
store updates can do much for bringing users back
#KeepItClassy
60. @jaberant 60
Messaging
The Growth team often provides messaging capabilities
across the enterprise
• Infrastructure, Services & best practices
• Email, push, sms, in-app, app store updates, custom
channels
• Policy & controls (based on fatigue models) to prevent
different teams from over-mailing or over-pushing
Prevent user fatigue
62. @jaberant 62
SEO is back
With App Indexing Google will deeplink from search results
into content in your app.
• 30+ billion deeplinks & counting
• Powerful distribution
• Think about the landing experience
• The years sent on developing best practices for landing
pages on the web...
App Indexing + Adwords = Search download ads
63. @jaberant 63
• Why would users want to share your app or product’s
atomic unit?
• How can users share the atomic unit?
• What are the mechanics of sharing?
• How do you reduce sharing friction?
• How do you get users to recommend your app?
Virality
64. @jaberant 64
Vitality - Your app’s atomic unit
What is the fundamental object that people are consuming
and/or creating?
Some examples:
• Twitter – tweet
• Soundcloud - audio track
• Instagram – photo
• Tumblr – post
• Dropbox – file
• Codecademy – lesson
67. @jaberant 67
Social Proof Best Practices
Pictures
People
• Connections
• Experts
• Celebrities
• Similar people
Don’t include if poor quality
68. 68
“We bet heavily on Twitter. Even if
someone is incredibly enthusiastic about a
product, literal word of mouth will only
get to a handful of people — but if
someone tweets about us, it can be seen
by hundreds, even thousands.”
- Stewart Butterfield, Slack Founder
Share via Twitter
71. @jaberant 71
Listen To Your Users
App & product users will communicate
• They want the stuff they are using to get better
Give them ways to talk back to you
• Button to talk to you, in-app prompt, surveys
Find guidance on what they want
• Then experiment to confirm
Tag, ticket & study messages from users
• Be fastidious
72. 72
“If you put that all together, we probably
get 8,000 Zendesk help tickets and 10,000
tweets per month, and we respond to all
of them.”
” - Stewart Butterfield, Slack Founder
Listen with Twitter
75. @jaberant 75
Don’t Spam
Spam ruins the internet
Users don’t like it
Can easily waste so much good work
Virality doesn’t matter until you can deliver a compelling
benefit to users
• Nothing worth sharing means just that
Avoid gimmicks that drive sign-ups in the short-term but
alienate users in the long-run