This document discusses Zalando's use of AI to improve size and fit recommendations for customers. It outlines several challenges including varying size conventions, limited fit data for new items, and sparse customer purchase histories. It then describes Zalando's approaches to address these, including algorithms that use item images to predict sizes for new items lacking data (SizeNet) and models that learn from customers' past purchases and feedback to provide personalized size recommendations. The goal is to help customers find the right fit on their first purchase to reduce returns and improve the shopping experience.

1. April, 2021 AI in Fashion Size & Fit
AI in Fashion Size &
Fit
Nour Karessli - Applied Scientist
DTC - 14.12.2021
S
M
L
XS
Image source

2. April, 2021 AI in Fashion Size & Fit
Leader in European Fashion
This is Zalando. The starting point for fashion.
2
700
>38
Active Customers
m
450 m
Visits / month
>3,500
Brands
Articles
k

3. April, 2021 AI in Fashion Size & Fit
Image source
Familiar?

4. April, 2021 AI in Fashion Size & Fit
Opportunity
Size
Issues
Image source

5. April, 2021 AI in Fashion Size & Fit
Problem Space
Mass production
Far away from end customers using outdated sizing statistics
5
Image source Image source

6. April, 2021 AI in Fashion Size & Fit
6
Mass production
Far away from end customers using outdated sizing statistics
No uniform sizing conventions
Vary by brand and country (EU, FR, IT,...)
Problem Space
Image source

7. April, 2021 AI in Fashion Size & Fit
7
Mass production
Far away from end customers using outdated sizing statistics
No uniform sizing conventions
Vary by brand and country (EU, FR, IT,...)
Vanity Sizing
Brands intentionally assign smaller sizes to articles to encourage
sales depending on target customers
Problem Space
Image source

8. April, 2021 AI in Fashion Size & Fit
8
Mass production
Far away from end customers using outdated sizing statistics
No uniform sizing conventions
Vary by brand and country (EU, FR, IT,...)
Vanity Sizing
Brands intentionally assign smaller sizes to articles to encourage
sales depending on target customers
Multi-dimensional
For trousers: length, waist, fit, …
Problem Space

9. April, 2021 AI in Fashion Size & Fit
Ensuring Zalando Customers
Get The Right Fit The First
Time
Image source

10. April, 2021 AI in Fashion Size & Fit
Team Skills
10
Size & Fit
Business Analysis &
Development
R&D Engineering
Software Engineering
Fitting Lab
Applied Science
Product & Design

11. April, 2021 AI in Fashion Size & Fit
Article side
● Different size systems
across categories, brands, countries
● New articles
with no sales or returns yet
● Ambiguity / low quality of fit data
e.g. small fit, tight fit
● Delayed feedback due to return process
at least a 3 weeks window to allow returns
Data Science Challenges
11

12. April, 2021 AI in Fashion Size & Fit
Data Science Challenges
12
Customer side
● Multiple customers
many people behind one account
● New customers & data sparsity
no purchases yet or only a few purchases
● Customer behavior / willingness
differs widely across groups
● High expectations & explainability
especially from customers who provide size feedback

13. April, 2021 AI in Fashion Size & Fit
Article Data

14. April, 2021 AI in Fashion Size & Fit
14
Return reasons provided by our customers (online or offline)
● Subjective & noisy
● Delayed
Purchased articles by customers and selected sizes
● Sparsity in particular per Category
Article Data - Purchase and Return Data
Size=S

15. April, 2021 AI in Fashion Size & Fit
15
Article Data - Images
Packshot image
Other images

16. April, 2021 AI in Fashion Size & Fit
16
Article Data - Size and Fit Fitting Lab
Inhouse station to collect expert
feedback and garments technical data
● Size and fit feedback
Too big, too tight, short sleeves,.., etc
● Article measurements

17. April, 2021 AI in Fashion Size & Fit
17
Established fit standards with experts
Working with Fashion experts, we defined consistent fit and shape
terms
Fit is the width of a garment in relation to wearers body
Shape is the silhouette of a garment in relation to wearer’s body
Article Data - Unified Fit Taxonomy

18. April, 2021 AI in Fashion Size & Fit
Customer Data

19. April, 2021 AI in Fashion Size & Fit
19
A destination for customers to interact with their
styles including Size and Fit preferences
Communicate with customers with more transparency to increase
customer trust
Collect feedback on past purchases to improve algorithms
● Fit me well 👍
● Didn’t fit me 👎
● For someone else 👭👫👬
Engage and onboard new customers to
Algorithmic Size Advice
Ask for a Reference Item (brand+size)
Customer Data - Size and Fit Prefences

20. April, 2021 AI in Fashion Size & Fit
20
Get more personal customer data from heavily
engaged customers
● Overall
weight, height, age, gender
● Upper body
top size, shirt collar size, shirt fit, prop. belly, .., etc
● Lower body
pants size, jeans length, jeans width, .., etc
Customer Data - Zalon Customer
Questionnaire from Zalon

21. April, 2021 AI in Fashion Size & Fit
21
Body measurements from 2D images
Using images from customers in tight sport clothes
to get body measurements
Customer Data - Body Data

22. April, 2021 AI in Fashion Size & Fit
Algorithms

23. April, 2021 AI in Fashion Size & Fit
23
Size Advice use-cases
1. Article centric size advice
○ Size Flags
○ SizeNet
2. Customer centric size advice
○ Existing customers: Size Recommendation
○ New Zalando customers: Cold-start Recommendation
■ who shop at Zalon: Zalon Cold-start Reco
■ brand new customers: Reference Item Cold-Start Reco

24. April, 2021 AI in Fashion Size & Fit
24
Different use cases
1. Article centric size advice
○ Size Flags
○ SizeNet
2. Customer centric size advice
○ Existing customers: Size Reco
○ New Zalando customers: Cold-start Reco
■ who shop at Zalon: Zalon Cold-start Reco
■ brand new customers: Reference Item Cold-Start Reco

25. April, 2021 AI in Fashion Size & Fit
25
Article centric size advice - Size Flags

26. April, 2021 AI in Fashion Size & Fit
26
Cold start problem
Zero or few article sales and returns
● Thousands of new articles everyday
● Return process takes a few days to few weeks
● Article may have short lifetime
Contributions
● Demonstrate the rich value of size visual cues in inferring size characteristics of fashion apparel
● Effectively tackle the challenging cold start problem of providing size advice for new articles using images
● Generate large scale confidence-weighted weak annotations from crowd’s subjective feedback → control weak annotations
influence on the final model
Article centric size advice - SizeNet

27. April, 2021 AI in Fashion Size & Fit
27
A student-teacher transfer learning approach
Use weak annotations to learn article size issues from images
Article centric size advice - SizeNet
SizeNet: Weakly Supervised Learning of Visual Size and Fit in Fashion Images
CVPR Workshops 2019
Nour Karessli, Romain Guigourès, Reza Shirvany

28. April, 2021 AI in Fashion Size & Fit
28
Teacher
Binomial Likelihood
- p: expected category returns
- k: size returns of the article
- n: sales of the article
Confidence score
Article centric size advice - SizeNet

29. April, 2021 AI in Fashion Size & Fit
29
Teacher
Binomial Likelihood
- p: expected category returns
- k: size returns of the article
- n: sales of the article
Confidence score
Student
● CNN Backbone Feature Extractor Fashion DNA [Bracher et al. KDD16]
○ Resnet like architecture [He et al. CVPR16]
○ Predicts article metadata such as target gender, category, brand, main color
○ 128 dim Bottleneck features
● Multi-Layer Perceptron
○ 4 fully connected layers with nonlinear activation
○ Sample weighted binary cross entropy loss using sample estimator confidence score
Article centric size advice - SizeNet

30. April, 2021 AI in Fashion Size & Fit
30
Tackling the purchase-feedback lag in return data using a smart prior
Human feedback and computer vision
● Takes advantage of article images to tackle the cold-start problem in new articles
● For articles with no data from Fitting Station, SizeNet helps Size Flags to raise flags earlier
Article centric size advice - Size Flags - Prior
SizeFlags: Reducing Size and Fit Related Returns in Fashion E-Commerce
KDD 2021
Andrea Nestler, Nour Karessli, Karl Hajjar, Rodrigo Weffer, Reza Shirvany

31. April, 2021 AI in Fashion Size & Fit
31
Different use cases
1. Article centric size advice
○ Size Flags
○ SizeNet
2. Customer centric size advice
○ Existing customers: Size Reco
○ New Zalando customers: Cold-start Reco
■ who shop at Zalon: Zalon Cold-start Reco
■ brand new customers: Reference Item Cold-Start Reco

32. April, 2021 AI in Fashion Size & Fit
32
Customer centric size advice - Size Reco

33. April, 2021 AI in Fashion Size & Fit
33
Customer centric size advice - Size Reco
Baseline recommender
Based on purchase history learn a customer size distribution from the sizes kept by the customer and learn an article offset distribution
from the sizes kept and returned by all customers
● Inputs: purchase history of the customer, category of the query article
● Output: recommended size
Deep learning recommender
Trained using as ground truth the sizes kept by the customer
● Inputs: purchase history of the customer, features of the query article
● Output: recommended size

34. April, 2021 AI in Fashion Size & Fit
34
Gradient Boosted Trees
Trained using as ground truth the size purchased by the customer
● Inputs: questionnaire + brand of the query article
● Output: size
The output size is adjusted before recommendation using brand
offsets calculated from kept and return purchases in that brand
Takes about 15-20 purchases for Baseline to catch up
Customer centric size advice - Customer in the loop
Zalon Customer
Personalized Size Recommendations with Human in the Loop
ICML Workshops 2020
Leonidas Lefakis, Evgenii Koriagin, Julia Lasserre, Reza Shirvany
Baseline Cold-start

35. April, 2021 AI in Fashion Size & Fit
35
Customer centric size advice - Customer in the loop
Reference Item
● Input: reference item (brand+size)
● Output: recommended size
The size given by the customer, adjusted by our knowledge of the brand’s behaviour

36. April, 2021 AI in Fashion Size & Fit
36
Deep learning recommender: A meta-learning approach
● Each customer is a new task
● Article embeddings + size embeddings + Embedded Linear
Regression
● At test time, ELR trained on previous purchases
● Size is decoded from the output of ELR
Customer centric size advice - MetalSF
Meta-Learning for Size and Fit Recommendation
SDM 2020
Julia Lasserre, Abdul-Saboor Sheikh, Evgenii Koriagin, Urs Bergmann, Roland Vollgraf, Reza Shirvany

37. April, 2021 AI in Fashion Size & Fit
37
● MetalSF (deep learning model) can ingest customer or
article data easily
● Included zalon questionnaire data
● Improvement noticeable up to 15 purchases
Customer centric size advice - MetalSF
Towards User-in-the-Loop Online Fashion Size Recommendation with Low Cognitive Load
RecSys Workshops 2020
Leonidas Lefakis, Evgenii Koriagin, Julia Lasserre, Reza Shirvany

38. April, 2021 AI in Fashion Size & Fit
38
Thank you!