6. If you’re competitor-focused,
you have to
wait until there’s a competitor
doing something.
Being customer-focused
allows you to be more
pioneering.
Jeff Bezos
Amazon@ksavenkov
14. Mastery
Business Metamorphosis
Data Aware
Data Monetisation
Data-driven
Business Optimisation
Analytics
Business Insights
Data-Driven Maturity Index
Collect Data
Business
Monitoring
@ksavenkov
23. IT’S UNCLEAR
who’s responsible?
how to imrove?
how does that relate with
resources spent?
(besides CEO)
(besides “work better”)
(when most of the business
processes are automatic)
@ksavenkov
30. For planning, groups tasks in
Epics that matter
Track results of launched Epics
A culture of data
success
learning
@ksavenkov
31. Case study
Marketing project were tracked until launched
Study of a half-year long cross-promo campaign
discovered a significant loss via COGS.
We’e identified key mechanics that led to the loss.
We’e adjusted the mechanics, re-negotiated with
partners on the ongoing campaigns.
PROFIT
@ksavenkov
33. It doesn’t work this way:
marketing campaign
@ksavenkov
impact!
34. product feature launched
billing failure
pupils back from holidays
AppStore featuring
people back from vacations
@ksavenkov
It doesn’t work this way:
marketing campaign
impact!
35. product feature launched
billing failure
pupils back from holidays
AppStore featuring
people back from vacations
@ksavenkov
It doesn’t work this way:
marketing campaign
impact!
DIFFERENT CHANNELS
CONVERT WITH A
DIFFERENCE IN ORDER OF
MAGNITUDE
36. ALSO:
A lot of Epics that shouldn’t have
been launched at all
A culture of praising a
random success and
explaining failures by
external causes
Impossibility to learn by
mistakes
@ksavenkov
37. formulate measurable hypotheses
carefully plan experiments
define a condition of success/failure
prior to implementation
data collection and attribution, split-
testing, controlled variables,
statistical significance
demonstrate explicit risks
use models built on past data
prioritisation aid
and how the failure affects the roadmap
ability to prove multiple hypotheses
simultaneously
@ksavenkov
38. Case Studies
Estimating mechanics of marketing projects before they launched
Product improvement that results in 2.5x conversion, 2x lifetime
Split testing of targeting and creative materials for ad
campaigns, resulting in conversion 2-3 times higher than organic
Immediate increase of “conversion” from all initiatives to
successful ones
This approach is behind all conclusions in these slides
lots of failed experiments
success despite of all external factors
NO OTHER WAY@ksavenkov
39. Case Study: Recommender system for
Conversion
• Hypotheses to prove:
1. There’re enough users who will use RS output
2. Their conversion will be above average
• A/B testing is the only way:
– different channels convert with up to 20x difference
– current traffic mix is unpredictable and hard to control
in case of app installs
• Do pilots:
– Run with limited resources, then extrapolate and decide
if run full-scale
41. Let’s look at the economics
• In case of using a third-party RS on a CPO basis, in this
case the CPO is limited by $0.14 (actually, much less)
• In case of a flat fee of $1000**/month, this is feasible
starting from 7143 new subscribers/month, or $35K of
marketing budget.
* CAC and marketing budget are model data
** some arbitrary number
44. Look ahead or at least
watch your step, not
backwards
Make the data
work for you
@ksavenkov
45. Daily indicators
Incremental indicators
Leading indicators
Predictive models
spot problems and anomalies just in time
example: baremetrics.io
a perfect input for inbound marketing
accurate goals and perfect financial planning
@ksavenkov
46. Case Study
Some loyal subscribers churned away for 2-3
months, to come back later. It was historically
attributed to holidays and other external factors.
The daily indicators have shown that all churned users
subscribed on weekdays. What a riddle!
It turned out that on weekdays the code base is
frequently deployed to the production server, flushing
the message queue of subscription renewals.
The fix increased a lifetime of paying users by 20%
@ksavenkov
47. Case Study
Predicting a lifetime for users
that registered right now (10%
accuracy)
Accurate unit economics for
contracts with B2B2C partners,
content providers, pricing
@ksavenkov
48. Case Study
A probabilistic model for
segmenting users
Input data for chained
communication
(inbound marketing)
@ksavenkov
53. Affects all business processes that
scale linearly with a headcount:
customer support
editorial office
content management
marketing
@ksavenkov
54. Case Studies
User base grows from 1M to 2M, doubling the
headcount in customer support?
Implemented auto-reply using our knowledge
base and smart templates for support engineers
A number of markets increased, adding more editors?
Created an algorithm to provide a short-lists
based on a user behavior
An amount of UGC explodes, more content-managers?
Improving reduplication and computer aid based
on the collected data
@ksavenkov
58. ITERATION
Case Study
Improving the Agile process
the reality:
@ksavenkov
INFLATING EFFORT
ROADMAPS DESIGN DEVELOPMENT QA SHIPPED
BACKLOG
NEW STUFF
TECH. DEBT
PLANNING
BUGS
UX
SOFTWARE OPS
URGENT
STUFF
UNCLEAR DESIGN
UNCLEAR TECH
ITERATING
60. Case Stuies
• Compare B2B2C deals through unit economics
• Estimate traffic quality for partner ad networks
• Data partnerships
• Targeted user communication
• Personalisation and recommender systems (the next
slide)
• Bonus track: Investigating a large number of purchase
returns for an internet retailer
@ksavenkov
61. Using Recommender Systems
and personalisation
CAC
LTV
Content
Costs
Marketing
Expenses
New
Customers
ARPU
Lifetime
Consumed
Content Mix
Conversion
Retention
Reactivation
Exposed
Content Mix
÷
×
* the recommendation fairy
*
63. Data, models built over the data
and experimental results is the
main asset
created and exploited
by the innovative business
@ksavenkov
“THE UNFAIR
ADVANTAGE”