Conference talk on how to estimate economic effect from using Recommender Systems for B2C content services. Yet Another Conference on Marketing 2015.

2. Driving business goals with
Recommender Systems
Konstantin Savenkov, COO Bookmate
ks@bookmate.com, http://bookmate.com

3. Target audience
4
B2C Services
B2B Recom-
mender
Platforms
run a pilot
estimate costs
and benefits
determine fair
price or scale to
start with
PROFIT
determine value
for potential
clients
run a pilot
set fair pricing
model

4. 5

5. Agenda
• Recommender Systems:Academy,Technology, Business
• Recommender Systems for content discovery
• B2C Content Services: overview and business model
• Driving business goals with Recommender Systems
• customer acquisition cost
• lifetime value
• catalogue exploitation
• Bookmate – E-Contenta case
6
RS
$
$
$
$

6. Agenda
• Recommender Systems:Academy,Technology, Business
• Recommender Systems for content discovery
• B2C Content Services: overview and business model
• Driving business goals with Recommender Systems
• customer acquisition cost
• lifetime value
• catalogue exploitation
• Bookmate – E-Contenta case
7
RS
$
$
$
$

7. Academy vs.Tech vs. Business
8
How to
improve
performance
by X%
How
hard is to
implement
that?
A:
T:
B:
When gains
match costs?

8. Evaluation of Recommender Systems
9
academy business
offline
evaluation
online
evaluation
economic
evaluation
• user
behavior
history
• RMSE
• MAP
• NDCG
• etc.
• live users
• actual UX
• actual
inventory
• NDCG
• CTR
• funnels
• response
time
• live users
• actual UX
• actual
inventory
• business
model
• CAC
• LTV
• COGS
• …PROFIT!

9. 10
“It’s tempting, if the only tool
you have is a hammer, to
treat everything as a nail.”
* Recommender systems are cool, but they don’t substitute old
good traffic quality, UX and pricing.
Abraham Maslow,The Psychology of Science, 1966

10. Agenda
• Recommender Systems:Academy,Technology, Business
• Recommender Systems for content discovery
• B2C Content Services: overview and business model
• Driving business goals with Recommender Systems
• customer acquisition cost
• lifetime value
• catalogue exploitation
• Bookmate – E-Contenta case
11
RS
$
$
$
$

11. Recommender Systems for Content Discovery
• preference elicitation
• hard to describe preferences in a textual form
• weak textual relevance
• limited catalogue
12
“IWANTTO READ SOMETHING…”
EVEN FOR BOOKS!
LOOKING FOR UNKNOWN UNKNOWNS
REGIONAL SEGMENTATION

12. User with a book problem
13
Search case
Recommendation case

13. Recommender Systems in the interface
• Any place in the interface, when number of objects to
show exceeds available space
• Most of the interfaces are list-based
• Hence, order and size of the list can be defined by
either personalized or non-personalized algorithm
• Explaining recommendations is a different topic
14
There is no “no recommender system” setting.
If there’s “just something” or “popularity sorted”, that’s your RS.
!

14. Bookmate Example
15
faceted
filter
search
results
book
page
user
library
notifi-
cations
social
feed

15. Agenda
• Recommender Systems:Academy,Technology, Business
• Recommender Systems for content discovery
• B2C Content Services: overview and business model
• Driving business goals with Recommender Systems
• customer acquisition cost
• lifetime value
• catalogue exploitation
• Bookmate – E-Contenta case
16
RS
$
$
$
$

16. B2C Content Services
17
subscription,
PPD or hybrid
limited attention
and time
content may have
different cost

17. Unit Economics
18
Business at scale
(marginal revenue and
expenses per user)
LTV
Cost of content
CAC
userlifetime
ARPU
ARPU
…
PROFIT!

18. How the product works
• Each connection here is driven and
improved by business activities
• The content itself fits into a sort of a BCG
matrix:
19
GROWTH
COSTS
CAC

19. ×
÷
Driving Business Goals
20
CAC
LTV
Content
Costs
Marketing
Expenses
New
Customers
ARPU
Lifetime
Consumed
Content Mix
Conversion
Retention
Reactivation
Exposed
Content Mix

20. ×
÷
Driving Business Goals
21
CAC
LTV
Content
Costs
Marketing
Expenses
New
Customers
ARPU
Lifetime
Consumed
Content Mix
Conversion
Retention
Reactivation
Exposed
Content Mix
*
* the recommendation fairy

21. Agenda
• Recommender Systems:Academy,Technology, Business
• Recommender Systems for content discovery
• B2C Content Services: overview and business model
• Driving business goals with Recommender Systems
• customer acquisition cost
• lifetime value
• catalogue exploitation
• Bookmate – E-Contenta case
22
RS
$
$
$
$

22. Option1: Improving conversion / САС
23
paywall

23. Option I: Improving conversion / CAC
Hypotheses to prove:
1. There’re enough users who will use RS output
2. Their conversion will be above average
A/B testing is the only way:
§ different channels convert with up to 20x difference
§ current traffic mix is unpredictable and hard to control in case of app installs
Do pilots:
§ Run with limited resources, then extrapolate and decide if run full-scale
24

24. Option I: Improving conversion / CAC
Two approaches to estimate:
1. increase of revenue
2. decrease of CAC
Suits for estimating various models:
§ upfront costs (when the investments return)
§ flat fee (monthly license or added headcount)
§ variable costs (CPO or PaaS model)
25
UNIT ECONOMICS!
subscribers
marketing budget

25. Case Study: Bookmate + E-Contenta
Sounds promising!
Did 40% more users
become converted?
Not really, as there was
just 7% who didn’t
know the book to start
with.
26
Group A
Group B
Decided to use
this channel
Converted
3 starter books
from editors
3 starter books
from a cold-
start RS
2.17*X%
X%
Y%
0.65*Y%
Overall
conversion
Z%
1.4*Z%
three-sigma

26. Let’s check the economics *
• In case of using a third-party RS on a CPO basis, in this case the CPO is limited
by $0.14 (actually, much less)
• In case of a flat fee of $1000**/month, this is feasible starting from 7143 new
subscribers/month, or $35K of marketing budget.
27
* CAC and marketing budget are model data
** some arbitrary number
1000
CAC = $5
Group A
Group B
Blended conversion C%
Blended conversion 1.028*C%
Increased conversion 1.4x
for 7% of users
CAC = $4.86
+28
Blended conversion across all
channels is C%
$5000 of traffic

27. Option 2: Improving retention / LTV
Hypotheses to prove:
1. User pays as long as he finds what to read
2. There’re enough users who will use RS output
3. This channel has a discoverability above average
Ideal experiment:
§ A/B, then count actual lifetime
§ with lifetime close to year, it’s too long to wait
Solution:
§ do separate A/B for different user cohorts (new, 1 month old, 2 months old etc.)
§ estimate significant change in month-to-month retention for each cohort
28

28. Model case: estimating LTV improvement
Let’s assume Recommender System led to 0.5%-3%* increase of month-to-month
retention (old cohorts / new cohorts), Group A estimated lifetime is 9 months*.
29
* model data provided for illustration
That’s an equivalent for:
• increase of the lifetime by 2.6
months for Group B
• increase of LTV by 29% for
Group B
The area between the curves is
equal to # of additional ARPUs

29. Option 3: Better catalogue exploitation
30
Just to give an idea:
• long-tail content in general is cheaper (niche, back-catalogue and public domain)
• driving user out of search already improves margins
• adding a recommender system really changes a balance
• once you have the data from the pilot, estimation is quite straightforward

30. Conclusions for B2C services
• The simplest recommender system would likely give you 80% of all
possible upside. If it doesn’t, the problem is most likely not in the
algorithm.
• If you want to go beyond, run a pilot to assess costs and benefits, then
estimate if you have enough scale to afford the solution.
• If you deal with a third-party Recommender System convince them to
fair pricing (e.g. free period until you have enough scale).
• And, again
31

31. Conclusions for B2B Recommender Platforms
• Based on amount of traffic, price of marketing budget you can estimate value of
your solution for potential customers.
• Based on pilot integrations, you may either define a fair price point for a particular
customer or develop PaaS-style tiered pricing model.
• Doing just a UX-applicable Recommender Systems leaves you a quite tight margin
between LTV and CAC+COGS. Better take on the full user acquisition vertical.
• TEASER: Bookmate + E-Contenta 2.0: E-Contenta integrates with remarketing
solution and provides traffic, not just recommendations.
32

32. 33
Wanna 1 month of Bookmate
for free?
Go
http://bookmate.com/code
Use “yacm2015”
(valid thru June 2015)