Etsy professionals teach you how to rank with deep visual semantic features. There are different approaches of learning how to rank: pointwise, pairwise, and listwise. Etsy professionals demonstrate their series of transferring deep visual semantic features to large-scale multimodal learning to rank, called Images Don't Lie. Kamelia Aryafar, Senior Data Scientist, Search Sciences, Etsy

1. Learning to Rank with Deep
Visual Semantic Features
@ Etsy
AI with the Best
Kamelia Aryafar, Senior Data Scientist, @karyafar
Search Sciences, Etsy
September 2016
1

2. Etsy

3. Etsy is a global marketplace where people around the world connect,
both online and offline, to make, sell and buy unique goods.
3

4. By the Numbers
1.6M
active sellers
AS OF DECEMBER 31, 2015
24M
active buyers
AS OF DECEMBER 31, 2015
$2.39B
annual GMS
IN 2015
35+M
items for sale
AS OF DECEMBER 31, 2015
Photo by Kirsty-Lyn Jameson
DISCLAIMER
The statistics included
on the following slides
are updated quarterly.

5. 819
employees around
the world
AS OF DECEMBER 31, 2015
9
offices in
7 countries
AS OF DECEMBER 31, 2015
Photo by Emily Andrews
Work and Culture
DISCLAIMER
The statistics included
on the following slides
are updated quarterly.

6. Large and Unique Seller Base
1.6M
active sellers
AS OF SEPTEMBER 30, 2015
95%
of sellers run their
Etsy shop from home
2014 ETSY SELLER SURVEY
76%
consider their
shop a business
2014 ETSY SELLER SURVEY
Photo by Moira K. Lime
DISCLAIMER
The statistics included
on the following slides
are updated quarterly.

7. Etsy Made in Canada
Photo by Jean-Michael Seminaro
24M
active buyers
AS OF DECEMBER 31, 2015
92%
of buyers agree
Etsy
offers products
they can't
find elsewhere
2014 ETSY BUYER SURVEY
DISCLAIMER
The statistics included
on the following slides
are updated quarterly.

8. 8

11. Learning To Rank
11

12. Approaches to Learning to Rank
• Pointwise
- For an item, predict it’s grade (implicit ordering)
- Labels come from interactions with items
- Possible class imbalance
• Pairwise
- Ranking transformed to pairwise classification or regression
- Labels depend on ordering of item pair
- Ability to create balanced classes
• Listwise
- Input is entire set of documents associated with query
- Output is their ranked list
- Eg. Loss is a measure of the distance of ranking generated by the model to the perfect ranking for the
set of documents

13. Pairwise Learning
Each training instance represents a pair of items from same set of search
results in your logs.
<item1, item2>
Learner must learn to order item1 and item2 correctly, with respect to user
preference decisions found in your logs.

14. Features

15. Label Creation for Pairwise Features
{housewarming, gift, photo} - {housewarming, gift, ceramic, tile} → +1
{housewarming, gift, ceramic, tile } - {housewarming, gift, photo} → -1

16. Train Classifier (SVM)

17. Learning to Rank Pipeline

18. Multimodal Learning to Rank

20. Image vs. Text Features
20
Texture
Shape
Color
Title
Tags

21. Extracting Image Features
21

22. title Feature Engineerings Deep Learning
22

23. ImageNet
23Photo from : http://www.image-net.org/

24. Convolutional Neural Nets (CNNs)
24
Photo from: http://cs231n.stanford.edu/

25. VERY DEEP CONVOLUTIONAL NETWORKS FOR LARGE-SCALE IMAGE RECOGNITION
Karen Simonyan & Andrew Zisserman
Images Don’t Lie: Transferring Deep Visual Semantic Features to Large-Scale Multimodal Learning to Rank
Corey Lynch, Kamelia Aryafar & Josh Attenberg, KDD ‘16
Model Specs: VGGnet

26. Transfer Learning
26
Images Don’t Lie: Transferring Deep Visual Semantic Features to Large-Scale Multimodal Learning to Rank
Corey Lynch, Kamelia Aryafar & Josh Attenberg, KDD ‘16

30. Images Don’t Lie: Transferring Deep Visual Semantic Features to Large-Scale Multimodal Learning to Rank
Corey Lynch, Kamelia Aryafar & Josh Attenberg, KDD ‘16

31. Images Don’t Lie: Transferring Deep Visual Semantic Features to Large-Scale Multimodal Learning to Rank
Corey Lynch, Kamelia Aryafar & Josh Attenberg, KDD ‘16

32. 32
Images Don’t Lie: Transferring Deep Visual Semantic Features to Large-Scale Multimodal Learning to Rank
Corey Lynch, Kamelia Aryafar & Josh Attenberg, KDD ‘16

33. Images Don’t Lie: Transferring Deep Visual Semantic Features to Large-Scale Multimodal Learning to Rank
Corey Lynch, Kamelia Aryafar & Josh Attenberg, KDD ‘16

34. Images Don’t Lie: Transferring Deep Visual Semantic Features to Large-Scale Multimodal Learning to Rank
Corey Lynch, Kamelia Aryafar & Josh Attenberg, KDD ‘16

35. Production Pipeline
35

36. Collecting Item Pairs with Labels
36

37. Strategies for Model Application
Real Time Offline
Pros Can handle unseen items No cost to feature
complexity
Cons Latency cost ∝ complexity
Query time feature
computations
Computations compound as
considerations increase

38. Real time model evaluation
Fetch Model
(cache, key-value store)
Apply Ranking
(ranking or reranking pass)
User Query Top-k results
Top-k results
RerankedIndex Ranking Model

39. Gaining Confidence

40. Gaining Confidence

41. Performance Replays

42. Performance Replays

43. Ranking Replays

44. Offline Evaluation Metrics

45. Model Understanding: Side by Side

46. Custom Queries & Explain Logs

47. The future…

48. Further Reading

49. etsy.com/careers

50. Thanks!