Computer Vision meets Fashion (第12回ステアラボ人工知能セミナー)
•Download as PPTX, PDF•
15 likes•4,652 views
講演者: 山口光太 先生 (株式会社サイバーエージェント AI Lab) 概要: 機械学習が様々な産業分野に応用されるにつれ，ファッション分野でもコンピュータビジョンを使った分析が広がりを見せている．EC活用などを目指し数多くのスタートアップ企業がファッション画像の認識をコアにしたデータ分析や検索サービス開発を進めるようになっている．本講演ではコンピュータビジョンをファッション分野の問題に適用したこれまでの研究を俯瞰する．衣服や属性の画像認識に始まる基本的な問題から，商品検索やスタイルの推薦，ファッションを構成する意味要素の理解，Webデータからの学習についての研究事例を，著者がこれまでに取り組んできた研究を含めて解説する．
Recommended
More Related Content
Viewers also liked
Viewers also liked (12)
Similar to Computer Vision meets Fashion (第12回ステアラボ人工知能セミナー)
Similar to Computer Vision meets Fashion (第12回ステアラボ人工知能セミナー) (20)
More from STAIR Lab, Chiba Institute of Technology
More from STAIR Lab, Chiba Institute of Technology (7)
Recently uploaded
Recently uploaded (20)
Computer Vision meets Fashion (第12回ステアラボ人工知能セミナー)
- 1. Computer Vision meets Fashion Kota Yamaguchi CyberAgent, Inc. STAIR Lab AI seminar, Aug 17, 2017
- 2. Who am I Kota Yamaguchi Research scientist CyberAgent, Inc. vision.is.tohoku.ac.jp/~kyamagu Computer vision and machine learning 2017 Assistant professor, Tohoku University 2014 PhD, CS, Stony Brook University 2008 MS, 2006 BE, University of Tokyo twitter.com/kotymg github.com/kyamagu
- 3. Research agenda 1. Learning visual perception on the Web [ACCV16] [ECCV16] 2. Clothing and body recognition [BMVC 15] [TPAMI14] [ICCV13] [CVPR12] 3. Understanding fashion and behavior [WACV15] [ACMMM14] [ECCV14] 4. Language and vision [PACLIC16] [IJCV15] [CVPR12] [NAACL12] [EACL12]
- 5. style.com
- 6. Computer vision provides machine perception to fashion
- 7. For e-commerce Garment search ShortsBlazerT-shirt query results Clothing search [Liu 12] [Kalantidis 13] [Cushen 13] [Kiapour 15] [Liu 16] Recommendation [Liu 12] [McAuley 15] [Veit 15] [Yang 17] Categorization [Borras 03] [Berg 10] [Bourdev 11] [Chen 12] [Bossard 12] [Di 13]
- 8. For re-identification Person identification [Anguelov 07] [Gallagher 08] [Vaquero 09] [Wang 11] [Gallagher 08] [Vaquero 2009]
- 9. For social science Social groups [Murillo 12] [Kwak 13] [Kwak 13] nput: (Z1, Y1), . . . , (ZN , YN ), C, ϵ Output: w, ξ nitialization: H = ∅ epeat (w, ξ) ← solveproblem (8) based on current H; for n = 1 to N do Y∗ n ← argmaxY ∗ n ∈Y { △ (Yn , Y∗ )+ wT Ψ(Zn , Y∗ )} ; end H ← H ∪ { (Y∗ 1 , . . . , Y∗ N )} ; until 1 N N n △ (Yn , Y∗ n ) − 1 N wT N n [Ψ(Zn , Yn ) − Ψ(Zn , Y∗ n )] ≤ ξ + ϵ; gorithm 2: 1-slack formulation for structure SVM. all that wehave6 spatial relations, |A| dimensional fea- e, and C categories of occupations. Then thedimension- y of wa and wb is 6C2 and C × |A|, respectively. Anal- Soccer Player Mara- thoner Chef Lawyer Doctor Firefighter Policeman WaiterSoldier Student Clergy Mailman Construc- tion Labor Teacher Figure 3. Illustrations of the collected occupation database. There are14 occupations and over 7K images in total. hypothesis should be valid. However, the runtime for this n-slack formulation in problem (6) is still polynomial with Occupation [Song 11] [Shao 13] [Shao 13]
- 10. Fashion Industry “The global apparel market is valued at 3 trillion dollars, 3,000 billion, and accounts for 2 percent of the world‘s Gross Domestic Product (GDP).” – FashionUnited.com
- 11. Amazon Echo Look: Hands-Free Camera and Style Assistant https://www.amazon.com/Echo-Hands-Free-Camera-Style-Assistant/dp/B0186JAEWK
- 12. Start-ups • Cutting-edge computer vision in business Fashwell Wide Eyes Technologies ShopagonVasilyMarkable
- 13. Start-ups Original Stitch • Cutting-edge computer vision in business http://jp.techcrunch.com/2017/03/14/original-stitch-open-platform/ Makip
- 14. Computer vision meets fashion • Recent topics 1. Clothing recognition 2. Retrieval for e-commerce 3. Style / trend analysis 4. Learning from big data 5. Image generation • Clothing parsing • Attribute discovery • Style recognition • Trend analysis • Popularity analysis
- 15. Recent topics
- 16. 1. Clothing recognition • Classification • Segmentation Semantic segmentationPose estimation Contexts, joint models, dependency [Shotton 06] [Gould 09] [Liu 09] [Eigen 12] [Singh 13] [Tighe 10, 13, 14] [Dong 13] [Liang 15] [Ferrari 08] [Bourdev 09] [Yang 11] [Ukita 12] [Dantone 13] [Ladicky 13]
- 17. Mix and Match: Joint Model for Clothing and Attribute Recognition matched unmatched [Yamaguchi, BMVC 2015]
- 18. Human Parsing with Contextualized Convolutional Neural Network [Liang, ICCV 2015]
- 19. 2. Retrieval for e-commerce • Street2shop • Attribute-based search • VQA shopping assistant Exact retrieval, interaction, attributes
- 20. Where to Buy It: Matching Street Clothing Photos in Online Shops Liu et al., Street-to-shop: Cross-scenario clothing retrieval via parts alignment and auxiliary set. CVPR 2012 Huang et al., Cross-Domain Image Retrieval With a Dual Attribute-Aware Ranking Network, ICCV 2015 [Kiapour, ICCV 2015]
- 21. Whittlesearch: Image search with relative attribute feedback [Kovashka, CVPR 2012]
- 22. Memory-Augmented Attribute Manipulation Networks for Interactive Fashion Search[Zhao, CVPR 2017]
- 23. Visual Search at eBay • VQA in shopping scenario [Yang, KDD 2017]
- 24. DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations http://mmlab.ie.cuhk.edu.hk/projects/DeepFashion.html [Liu, CVPR 2016] 800K images 50 categories 1K attributes, bbox, landmarks Attribute prediction Street-to-shop In-shop retrieval Landmark detection Retrieval demo: http://fashion.sensetime.com/
- 25. 3. Style / trend analysis • Learning style/outfit as a whole • Item compatibility • Geographical trend • Temporal trend Unsupervised learning, weakly supervised learning
- 26. Fashion Style in 128 Floats: Joint Ranking and Classification Using Weak Data for Feature Extraction [Simo-Serra, CVPR 2016]
- 27. Learning the Latent "Look": Unsupervised Discovery of a Style-Coherent Embedding from Fashion Images [Hsiao, arXiv 2017] Polylingual LDA
- 28. Learning Visual Clothing Style with Heterogeneous Dyadic Co-occurrences [Veit, ICCV 2015] IncompatibleCompatible
- 29. Learning Fashion Compatibility with Bidirectional LSTMs [Han, ACM MM 2017]
- 30. StreetStyle: Exploring world-wide clothing styles from millions of photos [Matzen, arXiv 2017]
- 31. Changing Fashion Cultures [Abe, MIRU 2017]
- 32. Fashion Forward: Forecasting Visual Style in Fashion [Al-Halah, ICCV 2017] Popularity prediction by styles Keyword popularity prediction
- 33. 4. Learning from fashion big data • Social signal + visual signal • Textual signal
- 34. Fashion Conversation Data on Instagram Successful categorization Unsuccessful categorization [Ha, ICWSM 2017]
- 35. When Fashion Meets Big Data: Discriminative Mining of Best Selling Clothing Features[Chen, WWW 2017]
- 36. The Elements of Fashion Style [Vaccaro, UIST 2016] Learning semantic relationship between high-level and low-level fashion concepts in text
- 37. 5. Image generation • Domain translation • Creative inspiration • Virtual fitting
- 38. Pixel-Level Domain Transfer • Generating a product photo given a street snap [Yoo, ECCV 2016] https://github.com/fxia22/PixelDTGAN
- 39. Pose Guided Person Image Generation[Ma, arXiv 2017]
- 40. A Generative Model of People in Clothing • Generating people from pose map and styling pipeline [Lassner, ICCV 2017]
- 41. Recent topics: discussion • Fashion tech is strongly application-oriented • UX in e-commerce and social media • Computer vision as a building block • Deep learning almost solves recognition problems • Contextual modeling is still under investigation • Data issues: research towards unsupervised / weakly-annotated data • Machine learning for creativity?
- 42. Clothing parsing CVPR 2012, ICCV 2013, TPAMI 2014, arXiv
- 44. Fully-convolutional Neural Networks (FCN) • CNN for semantic segmentation • All-convolution architecture Fully Convolutional Networks for Semantic Segmentation Jonathan Long, Evan Shelhamer, Trevor Darrell CVPR 2015
- 45. Looking at outfit to parse clothing FC FC Sigmoid Deconv Sum Crop Deconv Deconv Crop Sum Conv 3xConv Pool5 3xConv Pool4 3xConv 3xConv 3xConv Pool3 Pool2 Pool1 Conv Conv Product Deconv Crop Softmax CRF Loss Outfit encoder Conv Conv 1. Outfit prediction (clothing combination) 2. Filter out inappropriate garments 3. Smooth out boundary FCN Input [Pongsate, arXiv 2017] Input Output
- 46. Parsing results input truth prediction input truth prediction
- 48. Performance [%] Dataset Method Accuracy IoU Fashionista v0.2 Paper doll [ICCV 2013] 84.68 - Clothlets [ACCV 2014] 84.88 - FCN-8s [CVPR 2015] 87.51 33.97 Our model 88.34 37.23 Refined Fashionista FCN-8s [CVPR 2015] 90.09 44.72 Our model 91.74 51.78 CFPD CFPD [TMM 2015] - 42.10 FCN-8s [CVPR 2015] 91.58 51.28 Our model 92.35 54.65
- 49. Refined Fashionista dataset • High-quality, manually- annotated 685 pictures • Major improvement from v0.2 • CVPR 2012 • Used to learn FCN by fine- tuning from pre-trained model [Long 2015]
- 50. Pixel-based annotation using superpixels https://github.com/kyamagu/js-segment-annotator
- 51. Coarse-to-fine superpixels on the fly • SLIC superpixels computed on the client-side • Takes just a second in modern browsers • Efficient annotation from large to small segments
- 52. Limitations • CRF tends to trim small items • sunglasses • watch/bracelet • Dress vs. top+skirt distinction is still hard truth prediction input truth prediction
- 53. Pose estimation using FCN • Human pose as heatmap of parts • Predict heatmap by FCN • Can pose help segmentation, or vice versa?
- 54. Pose estimation results truth prediction prediction predictiontruth truth Good Small mistake Failure
- 55. Discussion • Lack of data size • 685 pictures are not sufficient for deep-learning approach • Global information in segmentation • Local appearance cannot solve the confusion • Need a global prediction of clothing combination to avoid confusion between items
- 57. Visual attribute perception • How does a _______ t-shirt look like? • yellow • large • surfer • comfy • original • popular ... onehourtees.com www.justclick.ie www.matsongraphics.compolyvore.com
- 58. Another question • How many words can we use to describe visual attributes of a t- shirt? • My t-shirt looks __________.
- 59. Automatic attribute discovery • Finding vocabulary of attributes • Open-world recognition challenge • Using pre-trained deep neural networks to identify visual words in the Web data
- 60. Our approach Pre-trained deep CNN beautiful soft blush handmade leather ballet flats. ***please, note, our new blush ballet flats are without the beige trim line (around the edges), still just as beautiful and perhaps even more*** SIZING ✍ how to take measurements ✍ there are a number of ways to measure your feet, however we find the quickest and most reliable practice is by tracing your feet. Here is how to do it: stand on a piece of paper that's bigger than your feet, circle your feet around with a straight standing pencil (without pressing the pencil too hard to the edges of your feet). Once you have the tracing, measure distance between longest and widest points. Compare the measurements to the list below. Image Text white red striped wooden sliky ... Attributes 1. Get Web data 2. Analyze DNN's internal activity
- 61. Web data: unlimited vocabulary with images Textual description Feel So Good ... Purple Halter Maxi Cotton dress 2 Sizes Available Tags used, american casual, summer, shorts, t-shirt, surfer, printed, duffer Etsy dataset: e-commerce Wear dataset: fashion-blog
- 62. Discovery: intuition • Contrast positive and negative sets to identify difference of semantics pink not pink
- 63. Identifying difference at neurons conv1 conv2 conv3 conv4 conv5 fc6 fc7 positive negative Deep neural network Activation histograms unit #1 unit #2 ... KL divergence Images neurons
- 64. Why neural activation? • Discriminability • If the attribute is visual, positive set should activate different set of neurons • Semantic depth • Depth of activating layer should be encoding semantic information ... conv1 conv2 conv3 conv4 conv5 fc6 fc7 Activation histograms Deep neural network
- 65. Kullback-Leibler divergence • Measure of difference between P+ and P- • Used to identify highly- activating units DKL (P+ || P- ) º P+ x å (x)log P+ (x) P- (x)
- 66. KL visualization: shorts Positive Negative pool5 KL average image norm2 KL
- 67. KL visualization: red Positive Negative pool5 KL average image norm2 KL
- 68. Is the attribute visual? • Which attribute is visually perceptible? • Measure the classification performance, and compare against human yellow comfy large original surfer popular
- 69. Visualness • Visualness of word u given a classifier f and dataset D+, D-: V(u| f )º accuracy( f,Du + ,Du - ) positive negative D+ D-
- 70. Discovered attributes lovelybrightorange acrylic NOT lovelyNOT brightNOT orange NOT acrylic elegant NOT elegant
- 71. Discovery in noisy data annotatedfloralNOTannotatedfloral predicted MOST floral predicted LEAST floral False positives False negatives
- 72. Most/least visual attributes Method Most Least Human flip pink red floral blue sleeve purple little black yellow url due last right additional sure free old possible cold Pre-trained + Resample flip pink red yellow green purple floral blue sexy elegant big great due much own favorite new free different good Attribute-tuned flip sexy green floral yellow pink red purple lace loose right same own light happy best small different favorite free Language prior top sleeve front matching waist bottom lace dry own right organic lightweight classic gentle adjustable floral adorable url elastic super
- 73. Perceptual depth • Which layer affects attribute recognition? conv1 conv2 conv3 conv4 conv5 fc6 fc7 orange bright elegant lovely acrylic
- 74. Most salient words (Etsy) norm1 norm2 conv3 conv4 pool5 fc6 fc7 orange green bright flattering lovely many sleeve colorful red pink lovely elegant soft sole vibrant yellow red vintage natural new acrylic bright purple purple romantic beautiful upper cold blue colorful green deep delicate sole flip welcome blue lace waist recycled genuine newborn exact vibrant yellow front chic friendly large yellow ruffle sweet gentle formal sexy floral red orange French formal decorative stretchy waist specific only black delicate romantic great American
- 75. Most salient words (Wear) norm1 norm2 conv3 conv4 pool5 fc6 fc7 blue denim-jacket border- striped-tops kids shorts white-skirt long-skirt green pink stripes bucket-hat half-length flared-skirt suit-style red-black red dark-style hat-n-glasses pants spring midi-skirt red red-socks stripes black denim upper gaucho-pants denim-on- denim red-black backpack sleeveless dotted beret handmade denim-skirt champion red American- casual border-stripes shirt-dress straw-hat pink blue dark-n-dark long-cardigan white-pants overalls white-n-white denim white denim-shirt white-n-white border-striped- tops hair-band white yellow shirt navy stole gingham-check loincloth-style white- coordinate leopard i-am-clumsy outdoor-style mom-style sandals matched-pair white-pants
- 76. Saliency detection • Can we identify salient region of the discovered attribute? tulle-skirt
- 77. Our approach: cumulative field • Accumulating receptive field of highly-activating neurons by KL conv1 conv2 conv3 conv4 conv5 fc6 fc7 masked input ... unit#1 unit#2 unit#3 saliency map ∑
- 78. image sunglasses shorts sneakers gingham check white style yellow human K=64 image human K=64
- 79. Attribute discovery • Web data + deep network • Highly-activating neurons to identify visual stimuli associated to the given word • Neural activations can further identify salient regions
- 80. Studying fashion styles ECCV 2014
- 81. Q: What makes the boy on the right look Harajuku-style? Tie? Shoes? tokyofashion.com
- 82. Goal • Finding what constitutes a fashion style • Approach • Game-based annotation • Attribute factorization Goth
- 85. hipsterwars.com Game-based relative ``style- ness’’ collection Asking our online friends for participation NO MONETARY REWARDS! Initial keyword-search on Google or Fashion SNS
- 86. Participation statistics Most played the game only a few clicks Some motivated users clicked A LOT
- 87. TrueSkill game algorithm • Algorithm to select which pair to play • Idea: • Represent each image by Gaussian over rating • Update Gaussian parameters after each click • Chooses expected-to-tie images for play [R Herbrich, 2007]
- 88. Score distribution after game Most Hipster Least Hipster
- 89. Annotation examples 5 6 7 8 9 0 1 2 3 4 5 6 7 34 10 ECCV-14 submission ID 1534 Most (Predicted) Least (Predicted)PinupGothipster 405 406 407 408 409 410 411 412 413 414 415 416 417 # 1 10 ECCV-14 submission ID 1534 Most (Predicted) Least (Predicted) PinupGothipster MOST LEAST High-quality dataset without Amazon MTurk
- 90. Relative vs. absolute • Gamification resulted in higher quality annotation • Asked MTurk workers 1-10 ratings • Much noisier results from MTurk
- 91. Analyzing what makes her look preppy Factorization results
- 92. Fashion style analysis • Game-based annotation collected high-quality data without monetary rewards • How can we collect seed images?
- 93. Fashion trend analysis WACV 2015
- 94. Fashion trend: Runway to realway Fashion show Street style.com chictopia.com
- 95. Runway dataset ~35k images in 9k fashion shows over 15 years, from 2000 to 2014
- 96. What does it mean by similar? The real challenge is the definition of similarity.
- 97. The query image is given in the left column, while five candidate images are shown in the right columns. 1. Select an image with the most similar outfit to the query. 2. If there is NO similar image, please select NONE. Query image NONE Collecting human judgments to learn similarity Select an image with the most similar outfit to the query image
- 98. Visual processing Pose estimation Foreground segmentation Boundary map
- 99. Runway-to-runway retrieval Retrieving similar styles from other fashion shows
- 100. Runway-to-realway retrieval Retrieving similar styles from street snaps
- 101. Visually analyzing floral trend Runway image of floral Retrieved images in street with timestamp Peaks in spring! % retrieved images
- 102. Runway to realway analysis • What is considered similar in fashion? • Our approach: Learn human judgment • Tracking similarity over time = trend analysis of a specific style
- 103. Visual Popularity Analysis ACM Multimedia 2014
- 105. Like button in Chictopia Long tail Promotion effect? ~300K posts
- 106. Why do some pictures get popular? Content factors Social factors • Active posting • Lots of friends • Good fashion items • Photo quality How much do they matter?
- 107. Regression analysis in 300K posts • Tag TF-IDF • Image composition • Color entropy • Style descriptor • Parse descriptor Popularity • User identity • Previous posts • Node degrees Input Output Social factors Content factors • Votes
- 108. Findings • Your outfit doesn’t matter (!!!) • Popularity is mostly the outcome of the network – social bias • #votes ∝ #followers • People just click on friends’ photos • c.f., Rich-get-richer phenomenon
- 109. Regression performance Factors R2 Spearman Accuracy top 25% Accuracy top 75% Social 0.491 0.682 0.847 0.779 Content 0.248 0.488 0.778 0.737 Social + Content 0.493 0.685 0.845 0.775 Social factors significantly boosts the performance
- 110. What if there is no social network? • Popularity = f ( content factors )?
- 111. Ask crowds! • Collecting popularity votes in Amazon MTurk • No network! 3000 pictures 25 assignments
- 112. Out-of-network popularity #posts #votes No social factors in the voting process
- 113. Task • Predict crowd popularity using Content factors and/or Social factors in Chictopia Social factors Chictopia Content factors MTurk Voting data ?
- 114. Predicting crowd votes Factors R2 Spearman Accuracy top 25% Accuracy top 75% Social 0.423 0.634 0.845 0.787 Content 0.428 0.647 0.888 0.862 Social + Content 0.473 0.686 0.884 0.858 • Content factors matter • Social factors from Chictopia predict crowd votes well • User-content correlation: Top-bloggers consistently post good pictures
- 115. Predicted most popular Predicted least popular Popularity prediction
- 116. The data told us... • Popularity is mostly the outcome of the social network • People click on friends’ photos • Content affects popularity, but we conjecture the existence of user-content correlation
- 117. Computer Vision meets Fashion • Computer vision = machine perception to quantify visual • Tool to analyze semantics of fashion • Research topics • Recognition, street2shop, style understanding, social influence, fashion trend, creativity
Editor's Notes
- https://pixabay.com/en/fashion-glasses-go-pro-female-1478810/
- https://pixabay.com/en/sunglasses-for-women-fashion-summer-178154/
- Analyzing the fashion show and street snaps tells us that the trend comes from the runway to the street.
- Dataset analysis tells us that trends are seasonal.
- People click on a ‘like button’ to show that they are interested in a picture
- The experiment is not clear