The presentation is for accepted paper from KES 2020 Conference.

1. Eye and Voice Control for an
Augmented Reality Cooking Experience
Adrian Iftene, Diana Trandabăț, Vlad Rădulescu
adiftene@info.uaic.ro
16 September 2020
Alexandru Ioan Cuza University of Iasi
Faculty of Computer Science

2. Content
 Augmented reality
 What is augmented reality?
 Domains of augmented reality
 Using Augmented Reality in Gastronomy
 Role of AR in Food Industry
 AR in Diet plan
 Existing applications
 AREasyCooking
 Architecture and Main components
 Evaluation
 Conclusions

3. What is “augmented reality”?
 Augmented Reality (AR) content can be accessed by scanning or viewing a trigger image with a
mobile device that creates a subsequent action [1]

4. 4

5. Role of AR in Food Industry
A Customer Can Know More About A Dish
Can combine real images with images of the drinks
Can see the ingredients
5

6. AR in Diet Plans
 Diet plan – information is presented fast in a simple way, easy to
understand
6

7. Existing Applications
 EatAR: Mobile Augmented Reality Nutrition Assessment [3]
 ServAR: Food Serving Guide [4]
7

8. AREasyCooking
8

9. User preferences
1. Preferences - checkboxes for user preferences (red meat,
pork, poultry, fish, shellfish, and vegetarian)
2. Allergies and diet - the user can add manually any
ingredients that should be avoided in the returned recipes
3. Universal – checkboxes for common ingredients (salt, sugar,
pepper, flour, oil, spices, mustard, mayonnaise, ketchup,
water, etc. ) The user can add new elements like eggs, milk,
rice, potatoes, onions, garlic, etc. – usually represent 50% of
ingredients of a recipe
4. Preferred source – the user can change the order of
preferred sources: Martha Steward, Jamie Oliver, traditional
food, and other

10. Recognize the ingredients
 Augmented Reality – Clarifai API https://www.clarifai.com/developer
 Barcode Reader - Raspberry Pi + Edamam https://developer.edamam.com/

11. Recipe Collections
 Edamam Recipe API
https://developer.edamam.com/ ~ 1.5 million
English recipes + YouTube Search
 Traditional East-European food (from North East
of Romania) - eCULTFOOD Atlas Project [5]

12. Preparing Food – Playing and Controlling a Video
Voice control
(1) “Play” - starts to play a movie
(2) “Stop” - pauses the playing
(3) “Right” - skips a number of seconds from
the movie
(4) “Left” - goes back a number of seconds
Eye control
(1) “Play” - both eyes open
(2) “Stop” - both eyes closed, or a situation
when both eyes are not detected by the
camera
(3) “Right” - user closing the right eye
(4) “Left” – user closing the left eye

13. Evaluation - Identifying Ingredients
 Light - users 1 and 4 tested the application in full light, while users 2 and 3 had dim-light
 Placement - users 1 and 2 had the ingredients crowded one over the other, while for users 3 and 4
there was generous space between the ingredients
User#1 User#2 User#3 User#4 Average
AR fridge 0.58 0.43 0.89 0.82 0.68
AR table 0.83 0.71 0.63 0.91 0.77
Bar Code 1.00 0.93 1.00 0.91 0.96
# of ingredients 12 14 9 11 11.50

14. Evaluation - Selecting the Recipe
 The testers identified a proper recipe on the
first page of the results (with 10 items) in
~80% of cases, and on the second page in ~5%
of the cases
 In ~ 15% of cases
 (1) the recipe was not in our database
 (2) it was not selected in the results
showed to the user - the recipe considered
by the user contained ingredients missing
from the list of identified ingredients

15. Evaluation – Controlling the Movie
 Voice Scenario: (1) Select the movie; (2) Start the video; (3) Control
it with different voice commands (Stop, Play, Left, and Right)
 Problems with noisy environment (when a blender or the water
tap is used)
 Eye Scenario: (1) Select the movie; (2) Place the phone in front of
user; (3) Start the video; (4) Try controlling it with the eyes (pause,
resume, and go forward or backward)
 Problems reported are related to light (in dim-light scenarios, the
phone camera is unable to properly detect the face of the user)
 Problems when Internet connection is discontinued
 Very fast and repeated actions are not helpful
15

16. Conclusions
 AREasyCooking help users to:
(1) quickly select a recipe fitted for the available ingredients
(2) guide them in preparing it
 The feedback of those who participated in the experiments is
encouraging
16

17. Bibliography
[1] https://www.edsurge.com/news/2015-11-02-how-to-transform-your-classroom-with-augmented-reality
[2] http://www.augment.com/blog/5-reasons-use-augmented-reality-education/
[3] https://pdfs.semanticscholar.org/95f6/5cc4b34f822c66db1d3b38fdda572f21aefc.pdf?_ga=2.
237192775.672348225.1598248259-1826830207.1558896666
[4] https://ijbnpa.biomedcentral.com/articles/10.1186/s12966-017-0516-9
[5] http://brain.edusoft.ro/index.php/brain/article/view/754

18. Thank you!
18