1. A study was conducted on recommending alternative dish recipes according to ingredient shortages using deep learning. 2. A joint embedding model was developed that uses a bi-directional LSTM and skip-instruction to jointly embed recipes and food images into a shared space. 3. The model was able to recommend alternative recipes with similar ingredients when ingredients were removed from the original recipe with over 90% accuracy based on the embedded recipe-image representations.

1. Study on Alternative Dish Recommendation
according to Shortage of Ingredients using Deep Learning
2
1

2. k 2
a
a Cd C
→ d o
d
C
C
d C
d
C
d C
d
Cd P
a a

3. 3
……
…
……

4. •
– s u
• bwbn x q ea eh bwbm q l q [1]
– v by s u
• do h t do ”q e q [2]
•
– t y b
• do f m x q [3]
• r
– m q j [4]
4
D(E FSI FSI )''.
s uq ci m q gp x m
“ k )''."
D)E :TSLVN BFSL BNS 2ZN 5FS CMFSL 8TMS T FP J LJ 1J TSLNJ FSI 3J T FM 4 Y NS )'(, FYJ2 NHP/
1TTY Y F NSL 5TTI JKJ JSHJ M TZLM FS 0IF YN[J AN ZF 7SYJ KFHJ 27 (,
D E F [FIT 0RFNF FSI SJ NHMT F FSI 0 YF JY F )'(-"
:JF SNSL 2 T RTIF 4R JIINSL KT 2TTPNSL JHN J FSI 5TTI 7RFLJ 2A )'(-
D E RJ/SJ MYY / RJSJ NSIJ] MYR
D E

5. 5
……
…

6. Recipe1M[1]( 1) ( 2)5
1 80 100 ( )
1 24
1
1 “a cup of milk” → “milk”
2
6
Simple
Cabbage
Rolls
[“cabbage”, “garlic”,
…, “water”, “salt”]
[“Remove the cabbage leaves one at a
time and microwave or parboil.”,
… ,“The insides should look like this.”,
“Coat in ketchup to taste.”]
[1] Marin, Javier and Biswas, et al., (2018), Recipe1M: A Dataset for Learning Cross-Modal Embeddings for
Cooking Recipes and Food Images, arXiv preprint

7. ( )
3
7

8. rt s Z
hb]”“d ”“de Y
6 e”“ ai
”“e ai
• Joint Embedding Model[1]
• Food-CNN[2]
→ ”“yZuel pZvd cJoint Embedding Model
”“ d r g o aibfY ]
8
m ktn Z r Z
( ) 0.9 -0.4 … 0.7
[[
:AK ADN LAIA AMD 1VMER 7ICHNKAR AMD VSA ES AK ' )
5EA MIM - NRR LNDAK LBEDDIM R FN -NNJIM ECI ER AMD 0NND 2LA ER - ' )
' 5NM PI AM IM -TI 0AM HAM NHM NKKAJ :E E ,EKNM IE AMD .EBN AH RS IM ' ( KASE-KICJ ,NNSRS A IM 0NND
EFE EMCER H NT H AM DA SI E IRTAK 2MSE FACE -2 (
pqk
]d

9. Joint Embedding Model 9
bf tk y[1]V
2 i aVS oS R[
7 C FH *D A E LE I 2A FC I E *L H C (
1 HEAE HFII DF C -D AE I FH FF AE A I E .FF 0D IN (
…
…
bi-directional LSTM
skip-instruction
pn r
LSTM
ResNet-50
h de]cg
v e]cg
ℎ"
ℎ#
pnl r
ℎ$
ℎ#
h d r
%#
%#
v r
ℎ$&
pnl e]cg
FC
FC
ms
'()$(%#, %,, -)
ms
P,22

10. 10
… …
-
… …
… …
… …

11. 14 11
.
.
3 2 .
.

12. •
–
•
–
– 3 5
• 1 1
– 4 ( 1) 10 1
•
–
– 1 1
12
3
3
3
1

13. • 20 9
–
13
ID ( )
1 25
2 25
3 27
4 26
5 25
6 25
7 24
8 23
9 25

14. 1.
( / )
2. (1. )
( 1 / 1 1 / 1 )
14
8 1
0
1
2
3
4

15. 15
•
•

16. 3.
( / )
4. (3. ) ( )
5. ( )
→
16

17. IM
• : J
C
• Joint Embedding Model S
:b i :
t J
•
9e Jo iE
•
17
d
• g ) l n J
m
• g ) 8 79 0211 7 812 J
d I ( , (