Dataset Download www.larry-lai.com/fashion.html Fashion compatibility learning is important to many fashion markets such as outfit composition and online fashion recommendation. Unlike previous work, we argue that fashion compatibility is not only a visual appearance compatible problem but also a theme-matters problem. An outfit, which consists of a set of fashion items (e.g., shirt, suit, shoes, etc.), is considered to be compatible for a “dating” event, yet maybe not for a “business” occasion. In this paper, we aim at solving the fashion compatibility problem given specific themes. To this end, we built the first real-world theme-aware fashion dataset comprising 14K around outfits labeled with 32 themes. In this dataset, there are more than 40K fashion items labeled with 152 fine-grained categories. We also propose an attention model learning fashion compatibility given a specific theme. It starts with a category-specific subspace learning, which projects compatible outfit items in certain categories to be close in the subspace. Thanks to strong connections between fashion themes and categories, we then build a theme-attention model over the category-specific embedding space. This model associates themes with the pairwise compatibility with attention, and thus compute the outfit-wise compatibility. To the best of our knowledge, this is the first attempt to estimate outfit compatibility conditional on a theme. We conduct extensive qualitative and quantitative experiments on our new dataset. Our method outperforms the state-of-the-art approaches.

1. 1
Theme-Matters: Learning Theme-Graph for
Fashion Compatibility
Jui-Hsin(Larry) La
i
Dataset Download www.larry-lai.com/fashion.html

2. 2
Outline
01
02
03
Introduction of Research Fashion Compatibility
Theme-aware fashion compatibility is proposed
Experiments
Part 1: Theme-aware outfit score. Part 2: Outfit recommendation
Summary
Summary and Conclusions

3. 3
Theme-Matters
Outfit
Hosiery
Purse
Skirt
Pointed
Travel?
X
Outfit B’
Flats
Hosiery
Purse
Skirt
O
• Visual compatibility • Theme-aware compatibility

4. 4
The Dataset with Fashion Themes
Occasion: Dating
Fit: Young
Style: Girls
Gender: Female
Occasion: Business
Fit: Tall
Style: Office
Gender: Male
Occasion: Travel
Fit: Thin
Style: Nature
Gender: Female
Dataset Download www.larry-lai.com/fashion.html

5. 5
4 Theme Categories and 32 Theme Tags
Occasion Fit
Style Gender
Outfits Outfits
Outfits Outfits

6. 6
Coarse-Grained to Fine-Grained Fashion Categories
The coarse-grained categories are not enough for theme-awar
e
Previous works only addressed a few categories, like 5-category: top, bottom,
shoe, bag, and accessory
However, T-shirt, Polo-shirt, and shirt are all belong to the top
T-shirt is casual, shirt is formal, and Polo-shirt is in between
The fine-grained categorie
s
To model the theme-aware compatibility, fine-grained fashion categories is
necessary
We are the first to use 149 fine-grained categories in Polyvore Dataset
Use the 152 fine-grained categories in FashionJD Dataset

7. 7
Framework of the Theme-aware Model
Theme p
(t,v)
wP
(u,t)
wP
(u,v)
wP
v
u
t
s
r
Theme-Graph
ov
i
ou
i
ot
i
Theme q
(s,r)
wq
(s,v)
wq
(v,r)
wq
Outfit j
ov
j
os
j
or
j
ov
j
Outfit i
ov
i
ou
i
ot
i
Outfit Pool
(c) Theme-Aware Attention Learning
(a) Category-Specific Embedding Network
(b) Embedding Mask
Triplet Loss
m(u,v)
m (v,t)
Compatible
Compatible
Incompatible
d(ou
i , ov
i , m(u,v)
)
m(u,v)
Compatible
Compatible
Compatible
Compatible
m(u,t)
Cross-entropy
Loss
Theme-aware
Attention
f(ov
j ; θ)
f(ou
i ; θ)
f(ov
i ; θ)
f(ot
i; θ)
d(ou
i , ov
j , m(u,v)
)
Loss(ou
i , ov
i , ov
j )

8. 8
Category-Specific Embedding Network
ov
i
ou
i
ov
j
Triplet Loss
m(u,v)
Compatible
Compatible
Incompatible
d(ou
i , ov
i , m(u,v)
)
m(u,v)
f(ov
j ; θ)
f(ou
i ; θ)
f(ov
i ; θ)
d(ou
i , ov
j , m(u,v)
)
Loss(ou
i , ov
i , ov
j )
(a) Category-Specific Embedding Network
Outfit j
ov
j
os
j
or
j
Outfit i
ov
i
ou
i
ot
i
Outfit Pool
A category-specific embedding(triplet) network

9. 9
Embedding Mask
ov
i
ou
i
ot
i
ov
j
(b) Embedding Mask
Triplet Loss
m(u,v)
m (v,t)
Compatible
Compatible
Incompatible
d(ou
i , ov
i , m(u,v)
)
m(u,v)
Compatible
Compatible
Compatible
Compatible
m(u,t)
f(ov
j ; θ)
f(ou
i ; θ)
f(ov
i ; θ)
f(ot
i; θ)
d(ou
i , ov
j , m(u,v)
)
Loss(ou
i , ov
i , ov
j )
The cost is high if training triplet networks for all category-category combinations
Share the embedding , and train the embedding masks
One embedding mask is a 1x1000 vector in the experiments.
f(; θ)

10. 10
Theme-aware Attention Learning
The Theme-Graph is trained with positive and negative outfits
Train one Theme-Graph for one fashion theme
Theme p
(t,v)
wP
(u,t)
wP
(u,v)
wP
v
u
t
s
r
Theme-Graph
ov
i
ou
i
ot
i
Theme q
(s,r)
wq
(s,v)
wq
(v,r)
wq
(c) Theme-Aware Attention Learning
m(u,v)
m (v,t)
Compatible
Compatible
d(ou
i , ov
i , m(u,v)
)
Compatible
Compatible
Compatible
Compatible
m(u,t)
Cross-entropy
Loss
Theme-aware
Attention
f(ou
i ; θ)
f(ov
i ; θ)
f(ot
i; θ)
Outfit i
ov
i
ou
i
ot
i

11. 11
Outline
Introduction of Research Fashion Compatibility
01
02
03
Theme-aware fashion compatibility is proposed
Experiments
Part 1: Theme-aware Outfit Score. Part 2: Outfit recommendation
Summary
Summary and Conclusions

12. 12
Experiments (1): Theme-aware Outfit Scores
Input a new outfit and output its scores of theme-aware fashion compatibility
Input a new outfit
m(u,v)
m (v,t)
d(ou
i , ov
i , m(u,v)
)
m(u,t)
f(ou
i ; θ)
f(ov
i ; θ)
f(ot
i; θ)
Theme p
(t,v)
wP
(u,t)
wP
(u,v)
wP
v
u
t
s
r
Theme-Graph
Theme q
(s,r)
wq
(s,v)
wq
(v,r)
wq
Theme-aware
Scores

13. 13
Experiments (1): Theme-aware Outfit Scores
Compatibility Scores
Baseline: 0.86
Travel: 0.34
Sports: 0.02
Business: 0.86
Baseline: 0.82
Travel: 0.82
Sports: 0.02
Business: 0.78
Baseline: 0.83
Travel: 0.81
Sports: 0.83
Business: 0.02
Outfits
(a)
(c)
(b)
Baseline: 0.92
Travel: 0.27
Sports: 0.18
Business: 0.05
Baseline: 0.49
Travel: 0.23
Sports: 0.02
Business: 0.27
Baseline: 0.78
Travel: 0.13
Sports: 0.02
Business: 0.39
(e)
(d)
(f)

14. 14
Experiments (1): Theme-aware Outfit Scores
The average AUC score increases 3.73% in comparing to the baseline
The average FITB score increases 1.58% in comparing to the baseline

15. 15
Experiments (2): Theme-aware Outfit Recommendation
Input a SKU, and output outfits with theme-aware compatibility
+ + +
+ + + …
v
u
s
r
Theme-Graph
f
(
o
v
j
;
θ
)
f
(
o
v
j
;
θ
)
Theme p
(t,v)
wP
(u,t)
wP
(u,v)
wP
Theme q
(s,r)
wq
(s,v)
wq
(v,r)
wq
Theme-aware
Scores
Given a SKU

16. 16
Experiments (2): Theme-aware Outfit Recommendation
Input a SKU and theme(s),
output an outfit with theme-
aware compatibility
Business
Input a trouser &
business theme
Output the compatible items
Sports
Input a jacket &
sports theme Output the compatible items
Dating
Input a Polo-shirt
& dating theme Output the compatible items

17. 17
Outline
Introduction of Research Fashion Compatibility
01
02
03
Theme-aware fashion compatibility is proposed
Experiments
Part 1: Theme-aware Outfit Score. Part 2: Outfit recommendation
Summary
Summary and Conclusions

18. 18
Theme-aware Fashion Compatibility
The first work to address the theme-aware fashion compatibilit
y
Compatibility needs a theme, a purpose, or a reason
The multiple outfit choices can fit to the 千⼈千⾯ in fashion recommendation
Outfit A’
Long Shirt
Purse High-Heeled
Blouse
Outfit B’
Flats
Hosiery
Purse
Skirt
Business Travel

19. Theme-aware Fashion Compatibility
Function 1: Theme-aware outfit scores
Upload an outfit, and output the theme-aware scores
Integrating the function into 搭配评测服务 with 陳家瑋 & 左佳偉
Expect the Version 1 will be ready by April 30
Function 2: Theme-aware outfit recommendation
Upload a SKU, and output outfits with theme-aware compatibility
The number of returned items in an outfit could be variant, like 2, 3, 4, 5…, which
covers the practical cases in the application
There needs a support of searching pool
Expect the Version 1 will be ready around Middle May

20. 20
THANKS

21. 21
Item-Graph to Category-Graph
Reduce the number of CSN
s
The amount of CSNs is huge
e.g., 87 fine-grained categories => 3,741 edges => 3,741 CSNs
Furthermore the training data is imbalance distribution
Cluster the fine-grained categories and share the CSNs
e.g., 87 fine-grained categories => 5 coarse-grained categories
=> 25 CSNs
Jeans
T-Shirt
Shirt
Hoody
Jacket
Down Coat
Suit
For JD Fashion Dataset:
(5 Coarse-grained categories)
Inner-Top: shirt, t-shirt, hoody, …
Outer-Top: jacket, suit, down coat, …
Bottom: jeans, pants, …
Shoe: …
Bag: …

22. 22
Polyvore Dataset Cleanup
The number of items and outfits in the Polyvore dataset before and after data
cleaning. The original Polyvore dataset does not have type labeling, so its type
statistics are missing
.
While training, the positive samples are outfits from the ground truth dataset and
negative samples are generated by substituting each item in an outfit with the same
type.

23. 23
Fine-grained Categories of Polyvore Dataset:
The crawling data from the Polyvore website contains “category_id” for each fashion
item, which can be recognized as the fine-grained categor
y
Retrieve the 149 fine-grained fashion categories
Here is the mapping table for fine-grained categories to coarse-grained categories
Coarse-grained
Category
Fine-grained Category, the ID references to the “category_id” from the Polyvore dataset
Top
11, 15, 17, 18, 19, 21, 23, 24, 25, 26, 33, 104, 236, 244, 247, 250, 252, 256, 257, 271, 272, 273, 275, 276,
277, 281, 309, 315, 341, 342, 343, 1605, 1606, 4454, 4455, 4456, 4457, 4495, 4496, 4497, 4498, 4516, 4517
Bottom
3, 4, 5, 6, 7, 8, 9, 10, 27, 28, 29, 237, 238, 239, 240, 241, 243, 249, 253, 255, 278, 279, 280, 282, 283, 285,
287, 310, 332, 4452, 4458, 4459, 4518
Shoe
41, 42, 43, 46, 47, 48, 49, 50, 261, 262, 263, 264, 265, 266, 267, 268, 291, 292, 293, 294, 295, 296, 298,
4464, 4465, 4522
Bag 35, 36, 37, 38, 39, 40, 231, 258, 259, 260, 290, 318, 4461, 4462, 4463, 4520, 4521
Accessory
51, 52, 56, 57, 58, 60, 61, 62, 64, 65, 67, 99, 105, 106, 107, 270, 299, 300, 302, 304, 305, 306, 4428, 4447,
4466, 4467, 4468, 4523, 4524, 4525

24. 24
Experiments of Polyvore Dataset:
The proposed method has 2.4% improved in AUC and 3.3% improved in FIT
B
Compared to the baseline work CSN[31]

25. 25
Evaluate A New Combination

26. 26
Subjective Evaluation of Fashion Compatibility
Subjective evaluation between rule-based and Graph+CN
N
5 subjects, 200 evaluation outfits
Score the right one or the left one is better with considering the “Condition”
Ours method is better than the rule-based 8.6% in terms of click rates