1) The document describes an augmented reality application called AREasyCooking that allows users to identify ingredients using computer vision, find recipes based on available ingredients, and view instructional cooking videos that can be controlled using eye or voice commands. 2) The application was tested by users who were able to correctly identify ingredients using augmented reality or barcode scanning over 90% of the time on average. 3) When selecting recipes, users found the proper one in the first page of results about 80% of the time.

1. Eye and Voice Control for an
Augmented Reality Cooking Experience
Adrian Iftene, Diana Trandabăț, Vlad Rădulescu
adiftene@info.uaic.ro
16 September 2020
Alexandru Ioan Cuza University of Iasi
Faculty of Computer Science

2. Content
 Augmented reality
 What is augmented reality?
 Domains of augmented reality
 Using Augmented Reality in Gastronomy
 Role of AR in Food Industry
 AR in Diet plan
 Existing applications
 AREasyCooking
 Architecture and Main components
 Evaluation
 Conclusions

3. What is “augmented reality”?
 Augmented Reality (AR) content can be accessed by scanning or viewing a trigger image with a
mobile device that creates a subsequent action [1]

4. 4

5. Role of AR in Food Industry
A Customer Can Know More About A Dish
Can combine real images with images of the drinks
Can see the ingredients
5

6. AR in Diet Plans
 Diet plan – information is presented fast in a simple way, easy to
understand
6

7. Existing Applications
 EatAR: Mobile Augmented Reality Nutrition Assessment [3]
 ServAR: Food Serving Guide [4]
7

8. AREasyCooking
8

9. User preferences
1. Preferences - checkboxes for user preferences (red meat,
pork, poultry, fish, shellfish, and vegetarian)
2. Allergies and diet - the user can add manually any
ingredients that should be avoided in the returned recipes
3. Universal – checkboxes for common ingredients (salt, sugar,
pepper, flour, oil, spices, mustard, mayonnaise, ketchup,
water, etc. ) The user can add new elements like eggs, milk,
rice, potatoes, onions, garlic, etc. – usually represent 50% of
ingredients of a recipe
4. Preferred source – the user can change the order of
preferred sources: Martha Steward, Jamie Oliver, traditional
food, and other

10. Recognize the ingredients
 Augmented Reality – Clarifai API https://www.clarifai.com/developer
 Barcode Reader - Raspberry Pi + Edamam https://developer.edamam.com/

11. Recipe Collections
 Edamam Recipe API
https://developer.edamam.com/ ~ 1.5 million
English recipes + YouTube Search
 Traditional East-European food (from North East
of Romania) - eCULTFOOD Atlas Project [5]

12. Preparing Food – Playing and Controlling a Video
Voice control
(1) “Play” - starts to play a movie
(2) “Stop” - pauses the playing
(3) “Right” - skips a number of seconds from
the movie
(4) “Left” - goes back a number of seconds
Eye control
(1) “Play” - both eyes open
(2) “Stop” - both eyes closed, or a situation
when both eyes are not detected by the
camera
(3) “Right” - user closing the right eye
(4) “Left” – user closing the left eye

13. Evaluation - Identifying Ingredients
 Light - users 1 and 4 tested the application in full light, while users 2 and 3 had dim-light
 Placement - users 1 and 2 had the ingredients crowded one over the other, while for users 3 and 4
there was generous space between the ingredients
User#1 User#2 User#3 User#4 Average
AR fridge 0.58 0.43 0.89 0.82 0.68
AR table 0.83 0.71 0.63 0.91 0.77
Bar Code 1.00 0.93 1.00 0.91 0.96
# of ingredients 12 14 9 11 11.50

14. Evaluation - Selecting the Recipe
 The testers identified a proper recipe on the
first page of the results (with 10 items) in
~80% of cases, and on the second page in ~5%
of the cases
 In ~ 15% of cases
 (1) the recipe was not in our database
 (2) it was not selected in the results
showed to the user - the recipe considered
by the user contained ingredients missing
from the list of identified ingredients

15. Evaluation – Controlling the Movie
 Voice Scenario: (1) Select the movie; (2) Start the video; (3) Control
it with different voice commands (Stop, Play, Left, and Right)
 Problems with noisy environment (when a blender or the water
tap is used)
 Eye Scenario: (1) Select the movie; (2) Place the phone in front of
user; (3) Start the video; (4) Try controlling it with the eyes (pause,
resume, and go forward or backward)
 Problems reported are related to light (in dim-light scenarios, the
phone camera is unable to properly detect the face of the user)
 Problems when Internet connection is discontinued
 Very fast and repeated actions are not helpful
15

16. Conclusions
 AREasyCooking help users to:
(1) quickly select a recipe fitted for the available ingredients
(2) guide them in preparing it
 The feedback of those who participated in the experiments is
encouraging
16

17. Bibliography
[1] https://www.edsurge.com/news/2015-11-02-how-to-transform-your-classroom-with-augmented-reality
[2] http://www.augment.com/blog/5-reasons-use-augmented-reality-education/
[3] https://pdfs.semanticscholar.org/95f6/5cc4b34f822c66db1d3b38fdda572f21aefc.pdf?_ga=2.
237192775.672348225.1598248259-1826830207.1558896666
[4] https://ijbnpa.biomedcentral.com/articles/10.1186/s12966-017-0516-9
[5] http://brain.edusoft.ro/index.php/brain/article/view/754

18. Thank you!
18