17. “Don’t show
me items that
can’t be
fulfilled”
“Call/chat during
picking regarding
replacements”
“Warn me about
stock level and
let me specify the
replacement”
“Trust the
shopper to
replace”
“Do not
replace item”
More permissive
Convenience Focus
More Proactive
Precision Focus
18. ● Item Found Rate
○ % of ordered items that are fulfilled
● Replacement Satisfaction Rate
○ % of replacements implicitly/explicitly approved by customer
21. In-store activity assumed as ground truth of item being in the store
Model Probability(Found) as a supervised binary classification problem
22. Aggregate Found Rate Statistics at higher levels (Critical for Tail Items)
● Product
● Retailer + Product
● Retailer + Product + Region
XGBoost Model with ~100 features
Example Item Level Features (Effective for Popular Items)
● Time since last not-found/found
● Found Rate in last 6m, 3m, 1 week, 1 day, 6 hours
23. Availability scores low across US
Even in stores where it hasn’t been bought (due to
product level features)
A store in Mission District:
Availability score: bottom 2 percentile
Past found rate: extremely low
From the Vice News report: “The milk
substitute has become so popular in places
like Brooklyn that shortages have broken out,
and a grey market for the most sought-after
brand of the stuff — Oatly — has popped up
online. Sometimes liters of Oatly can sell for
as much as $18 each.”
24. What comes next?
You’ve been freed
Do you know
how hard it is to lead?
-
Source: https://smcl.org/blogs/post/wait-for-it/
26. ● Retailer Location Selection
● Replacement Suggestions
during fulfillment/picking
● Picking Item List Ranking
● Search and Merchandising Ranking
● Low Stock Inform and Mitigate
● Post-Checkout Experience
● Chat with shopper during
fulfillment/picking
Customer Experience Shopper Experience
27. ● Eroding trust in Catalog
● Customer Churn
Show all relevant
items including
low stock
Remove all low
stock items
● Eroding trust in Fulfillment
● Lower shopper productivity
● Customer Churn
How can we create the optimal search experience?
28. Good improvement in Found Rate
Unacceptable drop in Search Conversion Rate
Hidden Items
Query: “Milk chocolate”
We were often removing items customers really wanted!
29. Demoted Items
No drop in Search Conversion Rate
Low improvement in Found Rate
Customers were ordering demoted items anyway!
Query: “Milk chocolate”
30.
31. ● Use data collected from demotion experiment
● Learn a simple model/heuristic to predict whether a
demoted item would be ordered given search
Key Insight:
Highly relevant
Query: “Milk chocolate”
32. Top 2 features describing search context
● # of results
● # of demoted results from top 5
# of items demoted in top 5
results
Don’t operate here
Ensure that the search results are relevant, mostly in stock and retains good selection of products
33.
34. “Don’t show
me items that
can’t be
fulfilled”
“Call/chat during
picking regarding
replacements”
“Warn me about
stock level and
let me specify the
replacement”
“Trust the
shopper to
replace”
“Do not
replace item”
More permissive
Convenience Focus
More Proactive
Precision Focus
37. ● Retailer Location Selection
● Replacement Suggestions
during fulfillment
● Picking Item List Ranking
● Search and Merchandising Ranking
● Low Stock Inform and Mitigate
● Post-Checkout Experience
● Chat with shopper during
fulfillment/picking
40% relative reduction
in low quality deliveries
Customer Experience Shopper Experience
38. ● Pick problems where ML can make unique contributions
○ Hiring/training product minded ML Engineers is challenging
● Pick problems that are fundamental to the business with broad applications
○ Instacart examples: Item Availability, Replacement recommendation, Product Targeting
● New use cases of existing models are often higher ROI than improving the model*
● Design and PM are key partners in getting the most value out of ML efforts