Playing VKards to support cognitively impaired people on meal preparation. The document discusses a system called Archipel that uses virtual kitchen cards (VKards) on a touchscreen to help cognitively impaired individuals prepare meals independently. VKards represent recipes, ingredients and utensils. Users select recipes and search for needed items by moving the cards between different areas on the screen. The system detects errors and ensures the proper steps are followed. This allows users to complete meal preparation through intuitive interaction like playing cards rather than relying on memory.

1. Playing VKards to support
cognitively impaired people
on meal preparation
Andrea Piras
Workshop on Interaction Techniques and Metaphors in Assistive Smart Environments, Salzburg – Austria, 18/11/2009

2. Playing VKards to support
cognitively impaired people
on meal preparation
Andrea Piras
piras@crs4.it
CRS4, Italy
Sylvain Giroux
sylvain.giroux@usherbrooke.ca
University of Sherbrooke, Canada
Jeremy Bauchet
jeremy.bauchet@it-sudparis.eu
TELECOM & Management SudParis, France
Workshop on Interaction Techniques and Metaphors in Assistive Smart Environments, Salzburg – Austria, 18/11/2009

3. Cognitive Deficiencies
 They effect the abstract human faculty of information
process and knowledge applying
 reasoning, inference, decision-making, comprehension,
planning, ...
 diseases examples: traumatic brain injury, dementia,
Alzheimer's disease, mental retardation, schizophrenia,
autism
 Millions of people lose temporarily or permanently the
ability to rightly use objects and successful complete
activities of daily living (ADLs)
 People become dependent by families and Health
Institutes
Salzburg, November 18th, 2009 IntTech09 3

4. A solution
 Augment the people's house introducing a set of not-
intrusive technologic solutions to support in ADLs and
promote independent living
 A net of sensors monitors user actions
 A net of effectors helps user to find objects
 A task assistant
 Change the traditional GUI-based interaction
 Teaching and supporting at the same time
 Making some actions “automatic”
 Providing help when required
Salzburg, November 18th, 2009 IntTech09 4

5. Archipel
 It is an autonomous cognitive assistant for ADLs designed
for smart homes and usable by user with lack of initiative
who is alone in the kitchen
 ADL is defined like a decision tree and includes the actions
to address errors
 It knows the position of sensors and objects
Salzburg, November 18th, 2009 IntTech09 5

6. Archipel
 The input received by sensors and by explicit user
actions lets to determine if the user performs wrong
actions and provide the right support
 Output is exposed on appliances, furniture and screen
 It is installed in the DOMUS Laboratory at
the University of Sherbrooke
Salzburg, November 18th, 2009 IntTech09 6

7. Focus our attention on ...
 Kitchen
 It is the room where people spend a lot of time
performing actions and interacting with others
 Meal preparation
 It is a privileged moment for promoting self-esteem
and enhancing autonomy
 Recipe selection and search of ingredients and
utensils
 The preliminary phases of meal preparation
Salzburg, November 18th, 2009 IntTech09 7

8. Change the interaction
 Mouse, keyboard and screen have drawbacks to any
category of users but they are harder for cognitive impaired
people
 We focus on TUI and banish traditional GUI to a secondary
rule
 On preliminary steps of meal preparation,
Archipel mainly collects input from one
touch screen
 Enhance its tangible features
 Substitute the screen with cards
Salzburg, November 18th, 2009 IntTech09 8

9. Virtual Kitchen card – VKard
 We borrow something from four different fields:
 Picture cards and card games
 Related to fun time of people ranging from child to aged
 To reuse such experiences and the pleasure to play to
help and train
 Visual supports used in Picture Exchange
Communication System (PECS) used in AAC
 For texts and images on the cards
 Token+Constraint system
 To take decision when user moves cards
 Visual tags (Amoeba)
 To recognize cards
Salzburg, November 18th, 2009 IntTech09 9

10. VKard examples
 From visual supports of PECS: one image, the name of its subject
(eventually the quantity), Amoeba visual tag on the top right corner,
white background
Salzburg, November 18th, 2009 IntTech09 10

11. Searching an object ...
 Token: VKard
 Constraint: Virtual Kitchen Table (VKTable) and
its four areas
Salzburg, November 18th, 2009 IntTech09 11

12. The procedure
1. Go to VKTable
2. Select the recipe putting its VKard in the recipe area (recipe recognized by
its visual tag)
3. For each ingredient/utensil:
a) Place VKard on the search area (item recognized thanks to the visual tag,
related draw’s light on)
b) Move in the kitchen
c) Open draw
d) Check if she has it
e) Close draw (related draw’s light off)
f) Return on VKTable
g) Move VKard onto missing items area or on found items area (add item to the list
of missing or found objects)
in italics, operations automatically performed by Archipel
Salzburg, November 18th, 2009 IntTech09 12

13. Search completed:
the system ...
 ... has helped to address user errors:
 user does not open/close related ingredient container
 user does not move VKard from search area to missing or found area
 user does not place VKard of next ingredient to search in search area
 user places two recipe VKards on the recipe area
 user places two ingredients on search area
 user places an ingredient VKard on recipe area
 ... implicitly knows:
 the chosen recipe
 if required ingredients are available and it is possible to prepare that
recipe
 ingredients/utensils are missing
Salzburg, November 18th, 2009 IntTech09 13

14. Search completed:
the user ...
 ... has successfully completed the ADL without
striving to remember:
 the ingredients and utensils used for a recipe
 the names of the objects
 next object to search
 their position in the kitchen
 the quantity
 the list of missing objects
 the list of found objects
 She just plays with VKards
Salzburg, November 18th, 2009 IntTech09 14

15. Pros & Cons
 Pros:
 Cheapness (just a colour printer and one
camcorder)
 Ingredients and utensils can be searched any
time and with in any order also without to be
connected to a recipe
 Cons:
 Need of constraint areas
 Occlusions could not let the camcorder to detect
the VKard
Salzburg, November 18th, 2009 IntTech09 15

16. Future works
 No constraint area
 Feedback direct on the VKard
 Integrating electronics inside it
 Use VKards in other ADLs
 to lie table, to shop ingredients, to place
bought items on the right position, to
wash dishes, ...
Salzburg, November 18th, 2009 IntTech09 16

17. Conclusions
 Augmented kitchens can help cognitive impaired people to
autonomously and successfully complete meal preparation
 Sensor networks and cognitive assistants must be coupled with a
user interaction extremely simple and intuitive
 VKards can be used for recipe selection and search of ingredients
and utensils in the kitchen
 For cognitive impaired people having lack of initiative and/or memory
 To reduce their rejection we mainly borrow by card games and by
PECS (well-known practice in autism)
 This is an on-going work
 Tests with users must be performed
 Archipel, VKard and VKTable are parts of the Virtual Collaborative
Kitchen (VCK) vision
Salzburg, November 18th, 2009 IntTech09 17

18. Grazie!
Gracias! Obrigado!
Thank you! Merci!
Danke! Arigato!
Xiè_Xiè! Efharisto!
Aitäh! Kiitos!
Spassiba! 感謝
Salzburg, November 18th, 2009 IntTech09 18