How do you build an architecture and solution to consistently drive double-digit increases in revenue and other key metrics for online businesses? The first step is deeply understanding consumer behavior - both declared and inferred - and encoding these as features. The next step is building accurate Machine Learning models that can generalize to new consumer experiences even with the vast variance between and in online businesses. In this talk, Sushant will discuss feature selection, validation, and multi-variate testing strategies that tame this complexity. Combining these approaches are changing this the way consumers engage with a web site as well how businesses understand their consumers and make decisions. Corresponding Youtube video: https://www.youtube.com/watch?v=Jf8HjCq43Dk&feature=youtu.be Presented at Silicon Valley Machine Learning (SVML) group on 1/30/15. Corresponding Meetup link: http://www.meetup.com/Silicon-Valley-Machine-Learning/events/219733855/.

1. PROPRIETARY AND CONFIDENTIAL
Overview of Machine
Learning Opportunities
in Retail
Sushant Shankar | Chief Data Scientist | 01/30/2015
Silicon Valley Machine Learning
1

2. PROPRIETARY AND CONFIDENTIAL
Agenda
1. Product Overview
2. ML Algorithms for Personalization
3. ML Algorithms for Planning
01-30-15 2

3. PROPRIETARY AND CONFIDENTIAL
Product
Overview
01-30-15 3

4. PROPRIETARY AND CONFIDENTIAL 4
A typical visit to an e-commerce site is not straight-forward and not conducive to rules
01-30-15
t
Google: ‘Converse shoes’
Purchase!

5. PROPRIETARY AND CONFIDENTIAL
• Segmentation
• Campaigns
• A/B tests
Current Tools for E-commerce are highly driven by Rules
5
Rules are manually
specifying conditional
probabilities!
01-30-15
Rules drive:

6. PROPRIETARY AND CONFIDENTIAL
The Reflektion Platform leverages Machine Learning to learn the ‘optimal policies’
601-30-15
Implement 1 to 1
experiences
across devices
Measure performance,
identify opportunities
and generate insights
Drive lifetime
value and
incremental
traffic

7. PROPRIETARY AND CONFIDENTIAL
ML Algorithms
for
Personalization
701-30-15

8. PROPRIETARY AND CONFIDENTIAL
Want to learn the Response of a User interacting with a Context
801-30-15
Response
User
Context

9. PROPRIETARY AND CONFIDENTIAL
Typical optimization is maximizing the average responses
01-30-15 9

10. PROPRIETARY AND CONFIDENTIAL
, w
A better approach is to maximize each user’s responses
01-30-15 10

11. PROPRIETARY AND CONFIDENTIAL 1101-30-15
Ideally, we would have the users draw us this curve. Realistically, we need to infer this curve.

12. PROPRIETARY AND CONFIDENTIAL
We can infer this curve through supervised and un-supervised models
1201-30-15
User events Context
Get new
experience
New (user,
context)
Features (slide 13)
Train models (slide 14,15)
...
...
(slide 16)

13. PROPRIETARY AND CONFIDENTIAL
1. Merchandise
2. Brand
3. Site
4. User demographic
5. Core Business Goal
Features need to incorporate domain knowledge
1301-30-15
vs.
User Context
(U, C)
Features
Train
Experience
...
...

14. PROPRIETARY AND CONFIDENTIAL
Variety of Machine Learning models can be used
1401-30-15
Source: Data Mining Methods for
Recommender Systems (2011)
Features
User Context
(U, C)
Features
Train
Experience
...
...

15. PROPRIETARY AND CONFIDENTIAL
Prior
Model Selection is itself a multi-level State Space Search
1501-30-15
Internal Model Evaluation (t)
Data
Properties of
Data
Best Models
⊂ Models
Optimal
Models
User Context
(U, C)
Features
Train
Experience
...
...
Model
Evaluation(s)
Model(s)
Experiments
External Model Evaluation (t)

16. PROPRIETARY AND CONFIDENTIAL
Need to have over-rides that reflect business considerations
1601-30-15
User Context
(U, C)
Features
Train
Experience
...
...

17. PROPRIETARY AND CONFIDENTIAL
ML
Algorithms
for Planning
01-30-15 17

18. PROPRIETARY AND CONFIDENTIAL
How did you drive results? What insights can you provide?
1801-30-15
1. Businesses need to understand how results were driven.
a. Can expose the Machine-learned weights in a digestible way.
2. Can surface these insights into tools to allow businesses to make
decisions about/through:
a. Merchandise
i. Assortment Planning
ii. Inventory Forecasting
b. Marketing
i. Channel Management
ii. User Segmentation
iii. Campaign Management

19. PROPRIETARY AND CONFIDENTIAL
Auto-segmentation of users and contexts
1901-30-15
(Users, Context)
1. Take interesting Users, Contexts, (users,
contexts)
2. Cluster (un)successful behaviors
together to:
a. ‘Personas’ of consumers based on
what are driving KPIs
b. Best contexts
c. Sort out interesting business
opportunities
d. Anomalies from expected behavior

20. PROPRIETARY AND CONFIDENTIAL
5
Predictive models can be used to simulate business decisions
2001-30-15
1
2
3
4
f12
(price, user location,...)
f13(price, user location,...)
f 34
...
f35 ...
...
...
...
∆
3
4
5
1
2

21. PROPRIETARY AND CONFIDENTIAL
We are a growing company and always looking for great talent.
jobs@reflektion.com
Questions?
21

22. PROPRIETARY AND CONFIDENTIAL
Backup Slides
22

23. PROPRIETARY AND CONFIDENTIAL
Marketing funnel in reality is complex
2301-30-15
Source: http://adamhcohen.
com/the-new-marketing-funnel/

24. PROPRIETARY AND CONFIDENTIAL
At any point in the interaction, there is a (User, Context) state
2401-30-15

25. PROPRIETARY AND CONFIDENTIAL 25
(Users,
Context) Metric
01-30-15
Understanding consumer behavior needs to understand user, context, and response attributes