Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Duplicated Cooking Recipe Determination Using Multimodal Information

The slides were presented at NLP2020 in collaborative research with the student from NAIST. There are many online platforms that allow people to upload and share cooking recipes with each other. However, within those platforms, many recipes contain the same contents, which can cause many problems and waste a lot of resources. In this work, we tackle the problem of duplicated cooking recipe determination by considering it as a binary classification task, in which we need to classify a pair of recipes into either duplicate or non-duplicate. We proposed a method based on multi-layer perceptron (MLP). Our model takes input features from multiple modalities, which are extracted from the input recipes. We conducted experiments on the Rakuten Recipe database and observed that the proposed method outperformed the baseline and achieved very high performance for the duplicate recipe determination task.

  • Login to see the comments

  • Be the first to like this

Duplicated Cooking Recipe Determination Using Multimodal Information

  1. 1. Duplicated Cooking Recipe Determination Using Multimodal Information March 19, 2020 ●Nguyen The Tung1, Yuki Nakayama2 1Nara Institute of Science and Technology 2Rakuten Institute of Technology, Rakuten, Inc.
  2. 2. 2 Duplicate Recipe Detection nTask description: ØGiven a new recipe, decide whether it is duplicated with a recipe in the database or not. nMethod: ØStep 1: From the database 𝑅"# and the new recipe 𝑟, generate candidates of duplication 𝑅% ⊂ 𝑅"# , • we use the work from previous intern [Oguni+ 2018] ØStep 2: Decide whether a pair of 𝑟, 𝑟′ , 𝑟′ ∈ 𝑅% is duplicate or not. • Our work in this paper – duplicated recipe determination [Oguni+ 2018] Masaki Oguni, Lasguido Nio, Yu Hirate, and Yohei Seki. Method for Detecting Near-duplicate Recipes Based on Nearest Neighbor Search for Features of Cooking Instructions and Food Images (in Japanese)
  3. 3. 3 Related work and Proposal n Deriving a Recipe Similarity Measure for Recommending Healthful Meals [van Pinxteren +2011] Ø Features: cooking instruction + ingredients Ø Human makes decision based on features n Clustering for Closely Similar Recipes to Extract Spam Recipes in User-generated Recipe Sites [Hanai+ 2015] Ø Features: ingredients Ø Cluster recipes into groups n Our proposal: Treat as classification Ø detecting duplicate recipes based on Multi-Layer Perceptron Ø The classifier uses similarity scores of cooking instruction (text), ingredients (text), user ID, and the result photo (image)
  4. 4. 4 Previous work’s pipeline (Step 1) Ingredients Food ImageCooking Instruction Extract Text Vector Extract Image Vector Database Nearest Neighbor Search (NGT) Database Nearest Neighbor Search (NGT) Extract candidates of original recipes Extract ingredients of original recipes Duplicate candidates Calculate ingredient similarity New Recipe [Oguni+ 2018]NGT: Neighborhood Graph and Tree [Iwasaki 2010]
  5. 5. 5 How to extract text vector: SCDV [Mekala+ 2017] n Sparse Composite Document Vectors (SCDV) n Output vector (dimension=10,000) represents the cooking instruction of each recipe. n Vector dimension is too high à use Principal Component Analysis (PCA) to reduce dimension to 2000.
  6. 6. 6 How to extract image vector: Inception-V3 [Szegedy+ 2016] n Convolutional Neural Network model to recognize generic object n We extract 2,048 dimensional vectors from images n We use pre-trained inception-v3 model for ImageNet competition input output 2,048 dimensional vector
  7. 7. 7 How to calculate ingredients similarity 𝐼𝑛𝑔_𝐴 ∗ = {にんにく, ⽶, ねぎ, いくら, 砂糖} 𝐼𝑛𝑔_𝐵 ∗ = {ニンニク, ライス, ネギ, いくら, 砂糖} *A is a near-duplicate recipe candidate B is a original recipe candidate 1. Find common ingredients between Ing_A and Ing_B (Intersection set) 𝐼𝑛𝑡𝑒𝑟𝑠𝑒𝑐𝑡𝑖𝑜𝑛 = 𝐼𝑛𝑔_𝐴 ∩ 𝐼𝑛𝑔_𝐵 = {いくら, 砂糖} 2. Create new sets by removing common ingredients 𝐼𝑛𝑔_𝐴’ = {にんにく, ⽶, ねぎ}, 𝐼𝑛𝑔_𝐵’ = {ニンニク, ライス, ネギ} 3. Convert to Katakana each ingredient 𝐼𝑛𝑔_𝐴’_𝑘 = {ニンニク, コメ, ネギ}, 𝐼𝑛𝑔_𝐵’_𝑘 = {ニンニク, ライス, ネギ} 4. Find common ingredients between Ing_A’_k and Ing_B’_k, add to Intersection. 𝐼𝑛𝑔_𝐴’_𝑘 ∩ 𝐼𝑛𝑔_𝐵’_𝑘 = {ニンニク, ネギ} 5. Create new sets by removing common ingredients 𝐼𝑛𝑔_𝐴’’ = {⽶}, 𝐼𝑛𝑔_𝐵’’ = {ライス} 𝐼𝑛𝑡𝑒𝑟𝑠𝑒𝑐𝑡𝑖𝑜𝑛 = {いくら, 砂糖, ニンニク, ネギ}
  8. 8. 8 How to calculate ingredients similarity 6. Search similar ingredients of each ingredient of Ing_A’’ using word2vec model trained by 1.16 million recipe data (training data). If the system finds ingredients of Ing_B’’ in top 3 search result, we consider as the same ingredient; add it to Intersection. ・Similar word of “⽶” 𝑆𝑖𝑚𝑖𝑙𝑎𝑟 𝑖𝑛𝑔𝑟𝑒𝑑𝑖𝑒𝑛𝑡𝑠 = {こめ, コメ, ライス, レンジ, 五穀⽶} 𝐼𝑛𝑔_𝐴’’ ∩ 𝐼𝑛𝑔_𝐵’’ = {⽶ (ライス)} 𝐼𝑛𝑡𝑒𝑟𝑠𝑒𝑐𝑡𝑖𝑜𝑛 = {いくら, 砂糖, ニンニク, ネギ, ⽶} 7. Create new sets by removing common ingredients. 𝐼𝑛𝑔_𝐴’’’ = 𝜑 , 𝐼𝑛𝑔_𝐵’’’ = 𝜑 , 𝑖𝑛𝑔𝑟𝑒𝑑𝑖𝑒𝑛𝑡 𝑑𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑐𝑒 = 𝐼𝑛𝑔_𝐴’’’ + 𝐼𝑛𝑔_𝐵’’’ 8. Make Union set and calculate the Jaccard similarity. 𝑈𝑛𝑖𝑜𝑛 = 𝐼𝑛𝑔_𝐴’’’ ∪ 𝐼𝑛𝑔_𝐵’’’ ∪ 𝐼𝑛𝑡𝑒𝑟𝑠𝑒𝑐𝑡𝑖𝑜𝑛 𝑖𝑛𝑔𝑟𝑒𝑑𝑖𝑒𝑛𝑡 𝑠𝑖𝑚𝑖𝑙𝑎𝑟𝑖𝑡𝑦 = 𝐼𝑛𝑡𝑒𝑟𝑠𝑒𝑐𝑡𝑖𝑜𝑛 𝑈𝑛𝑖𝑜𝑛
  9. 9. 9 Detection Model using Multi-Layer Perceptron (MLP) n MLP is a simple yet powerful classification model used in various tasks. n The task of duplication detection can be viewed as classification, with two class: duplicate, non-duplicate. Input features duplicate non-duplicate
  10. 10. 10 Experiment: Dataset n 1847 pairs of recipe extracted from database Ø Annotate the pairs with labels of duplication in terms of instruction, ingredients, and image. Ø If either one of those labels is duplicate à the pair is regarded as duplicate. n Training dataset Ø 1547 (1255 duplicate/292 non-duplicate) recipe pairs n Development dataset Ø 100 (50 duplicate/50 non-duplicate) recipe pairs n Test dataset Ø 200 (100 duplicate/100 non-duplicate) recipe pairs
  11. 11. 11 Features extraction n Instruction vector: SCDV à using PCA to reduce dimension to 2000. n Image vector: dimension 2048 (Inception v3) . n The features above are used as input of MLP, we trained a model that classifies input into duplicate or non-duplicate. Features type Description Instruction Euclidean distance between two vectors Image Euclidean distance between two vectors Ingredient 1.0 − 𝑖𝑛𝑔𝑟𝑒𝑑𝑖𝑒𝑛𝑡 𝑠𝑖𝑚𝑖𝑙𝑎𝑟𝑖𝑡𝑦 (p.8) User Identity function: 𝑠𝑐𝑜𝑟𝑒 = N 0, 𝑢𝑖𝑑1 = 𝑢𝑖𝑑2 1, 𝑢𝑖𝑑1 ≠ 𝑢𝑖𝑑2
  12. 12. 12 Results: Our method vs. Previous method n Our method n Previous method [Oguni+ 2018 ] Ø pick top k pairs with highest similarity score and assign them as “duplicate”, the remaining are “non-duplicate” Ø With all values of k, the proposal outperforms the previous method Positive label Negative label Recall Precision F1 Recall Precision F1 0.99 0.92 0.95 0.91 0.99 0.95 Duplicate Non- duplicate Duplicate 99 9 Non- duplicate 1 91 Positive label Negative label k Recall@k Precision@k F1@k Recall@k Precision@k F1@k 20 0.15 0.75 0.25 0.53 0.95 0.68 40 0.34 0.85 0.49 0.59 0.94 0.72 60 0.47 0.78 0.59 0.62 0.87 0.73 80 0.54 0.68 0.60 0.60 0.74 0.67 100 0.64 0.64 0.64 0.64 0.64 0.64 120 0.67 0.56 0.61 0.59 0.47 0.52 140 0.73 0.52 0.61 0.55 0.33 0.41 160 0.82 0.51 0.63 0.55 0.22 0.31 Ground truth Prediction
  13. 13. 13 Results: Effectiveness of features n Both User and Image information help improve performance compared to Instruction + Ingredient set. n Jaccard coefficient is better at representing ingredient similarity than ingredient difference. n We achieved highest result when using all the features. Features Positive label Negative label Recall Precision F1 Recall Precision F1 Instruction + Ingredient (ingredient difference at p.8) 0.89 0.60 0.71 0.40 0.78 0.53 Instruction + Ingredient 0.85 0.66 0.74 0.56 0.79 0.66 Instruction + Ingredient + User 0.94 0.90 0.92 0.90 0.94 0.92 Instruction + Ingredient + Image 0.97 0.91 0.94 0.90 0.97 0.93 Instruction + Ingredient + User + Image 0.99 0.92 0.95 0.91 0.99 0.95
  14. 14. 14 Correct prediction examples Ingredients:酒粕、砂糖、珈琲焼酎, ⽜乳. Sake lees, sugar, shochu, milk. Instruction: 鍋にすべての材料と⽔100g(分量外)を⼊れ、 あたためながら10分ほどとかしまぜてできあがり。 Put all ingredients and 100g of water (outside the amount) into the pan and mix for about 10 minutes with warming. Ingredients:酒粕, タイム, 砂糖. Sake lees, thyme, sugar. Instruction: 鍋にすべての材料と⽔200g(分量外)を⼊れ、 あたためながら10分ほどとかしまぜてできあがり. Put all ingredients and 200g of water (outside the amount) into the pan and stir for about 10 minutes. Prediction: Duplicate Ground truth: Duplicate
  15. 15. 15 Correct prediction examples Ingredients: ブロッコリー, ☆マヨネーズ, ☆プレーンヨーグルト, ☆塩コショウ Broccoli, ☆ mayonnaise, ☆ Plain yogurt, ☆ Salt pepper) Instruction:ブロッコリーは洗い、(⽔分は拭き取らずに)軸の太い部分は⼗字 に切り込みを⼊れラップでふんわり包みます. レンジに約3分かけて取り出し少 し冷めてから切り分けます(冷凍保存も可)☆を混ぜ低カロリーマヨネーズを作 り、付けていただきます。 Ingredients:ブロッコリー, 塩, *だし汁, *醤油, *⾟⼦. Broccoli, salt, * dashi soup, *soy sauce, * pepper. Instruction:ブロッコリーは⼩房に切って、塩を加えた熱湯で茹でてザルにあげ る。*は合わせておく。1のブロッコリーと*を混ぜ合わせ、器に盛る。完成︕. Put all ingredients and 200g of water (outside the amount) into the pan and stir for about 10 minutes. Prediction: Non-duplicate Ground truth: Non-duplicate Wash broccoli (without wiping away moisture) and cut the thick part of the shaft into a cross and wrap it softly with a wrap. Take it out to the range for about 3 minutes, cool it a little, and cut it out (Frozen storage is also possible) ☆ Mix and make low calorie mayonnaise.
  16. 16. 16 Wrong prediction example Ingredients:コーラス, バナナ⼩, プレーンヨーグルト, ⽔⽺羹, きな粉. (Chorus, banana, plain yogurt, water sheep, kinako.) Instruction:バナナは⼩さくちぎり、上記材料と⼀緒に全てミ キサーにかけ、ジュース状になったら出来上がりです. Bananas are chopped into small pieces and put into a mixer with the above ingredients. Ingredients:バナナ, プレーンヨーグルト, みかん, オリゴ糖, りんごジュース. Banana, plain yogurt, mandarin orange, oligosaccharidea, apple juice. Instruction:バナナは⼩さくちぎり、上記材料と⼀緒に全てミ キサーにかけ、ジュース状になったら出来上がりです. Bananas are chopped into small pieces and put into a mixer with the above ingredients Prediction: Duplicate Ground truth: Non-duplicate
  17. 17. 17 Summary n Conclusion Ø Implement duplicate recipe detection system based on MLP. Ø Our proposal outperforms the previous work significantly, reaching 95% accuracy. Ø Image and user information contributes to the task of predicting duplicate recipe pairs. n Future Work Ø Investigating other kind of features. Ø Expand our dataset.
  18. 18. 19 Architecture with Threshold Ingredients Food Image Cooking Instruction Extract Text Vector Extract Image Vector Database Nearest Neighbor Search (NGT) Database Nearest Neighbor Search (NGT) Text similarity > threshold Yes Extract candidates of original recipes Extract ingredients of original recipes Image similarity > threshold We set the threshold to β β = 0.94 We set the threshold to α α = 0.9 System judges posted recipe as near-duplicate recipe Calculate ingredient similarity Ingredient similarity > threshold Posted Recipe (Near-duplicate recipe candidate) Appendix
  19. 19. 20 Experiment on Tsukuba dataset n The paper is not yet published à the results are not confirmed. n Tsukuba dataset contains lots of conflicting samples (example below). Appendix Method Features Recall Precision F1 Tsukuba team (Random Forest) Instruction (n-gram mover distance) + ingredients (ingredient difference) 0.9 0.77 0.83 Our method Instruction + Ingredient 0.64 0.31 0.42 Ingredients: Clam, water, miso. Instruction: Remove the salt in the pan, add well-washed clams and water, and set it on medium heat. When the clam opens, it will come out and throw it away. When you put out the fire and melt the miso, it ’s done https://recipe.rakuten.co.jp/recipe/1190013605 Ingredients: Clam, water, miso. Instruction: The clams are sanded out, rub the shells and wash well. Put the clam and water in the pan and bring it to a boil. When the clam shell opens, the scoop will come out. Stop the fire, melt the miso and let it stand for a while. https://recipe.rakuten.co.jp/recipe/1280001778 Duplicate Ingredients: Mackerel, salt. Instruction: Shake the mackerel and let it sit for 10 minutes Wipe off moisture Bake for 10 minutes on the grill https://recipe.rakuten.co.jp/recipe/1090029 263 Ingredients: Mackerel, salt. Instruction: Lower 2 persimmons and finish 2 Sprinkle salt in the bowl and leave in the refrigerator for 30 minutes Wipe the water from the mackerel and bake for about 9 minutes on the grilled fish. https://recipe.rakuten.co.jp/recipe/1240026533 Non-duplicate
  20. 20. 21 Using raw subtraction vector as input features n Instruction (raw): Absolute of subtraction between text vectors of recipes in the recipe pair. n Image (raw): Same as above, but use image vector. n The resultant vectors were used as input features. n We observe no significant difference compared to the features set using Euclidean distance. On the other hand, computation time and storage requirement increased by a large amount. Appendix Features Positive label Negative label Accuracy Recall Precisio n F1 Recall Precisio n F1 Instruction + Ingredient + User + Image 95% 0.99 0.92 0.95 0.91 0.99 0.95 Instruction (raw) + Ingredient + User + Image (raw) 95% 1 0.91 0.95 0.9 1 0.95

×