Download to read offline
![CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Arts
Introduction and Motivation
Cooking advanced recipes can be complicated, and
requires following detailed instructions to achieve the
desired result. Textual instructions like paper recipes
are often difficult to comprehend; they use technical
terms or instructions that require prior experience to be
understood (“How much is a handful of chopped basil?”
or “How brown are caramelized onions?”). These types of
information can be transferred more easily using images
or videos, just as cooking shows do. However, cooking
shows are not designed to be followed in real-time.
Although the increasing audience of cooking shows in
European television [4] indicates that there is a
growing interest in this format, no participant of our
preliminary studies followed a recipe while watching a
cooking show. They rather liked the inspiring aspect of
the visual aesthetics. Typically, they would follow
textual instructions when cooking, or not use any kind
of instructions at all.
The final outcome of a recipe depends on many factors
and can fail at any point during the entire process. This
can be demanding and stressful for beginners, perhaps
discouraging them. We are developing PersonalChef, an
interactive kitchen counter (see Fig.1), to avoid
disappointing hobby chefs when preparing unfamiliar
recipes and to increase their confidence.
Related Work
Several systems have been developed to bring technical
innovations into the domestic kitchen environment.
Cooking Navi [7] considers cooking as a time
optimization problem when preparing a menu
consisting of multiple recipes, and tries to optimize
single cooking processes in order to have all dishes
finished at the right time. Semantic Cookbook [10] and
Kitchen of the Future [11] installed various electronic
devices into the kitchen to record and share cooking
sessions. Living Cookbook [12] offers these possibilities
as well, but it further focuses on the social experience
of cooking and collaboration. eyeCOOK [3] and Smart
Kitchen [8] focus on the actual user input. They try to
facilitate active user input or reduce it by tracking
user’s actions in the kitchen using eye-gaze, speech
commands, and foot switches. CounterActive [9]
integrates step-by-step cooking instructions using
multimedia invisibly into the kitchen counter. Besides
research prototypes, there are also commercial
systems. Nintendo’s personal cooking instructor
(nintendo.com) on the small-screen NintendoDS device
is an interactive cookbook and provides live cooking
demonstrations.
While other work often focused on efficiency [1,5], our
system aims to support the entertaining and social
gress (Spotlight on Posters Days 1 & 2) April 12–13, 2010, Atlanta, GA, USA
3404
Figure 1. HeatSink illuminates the stream
of water according to its temperature,
becoming red when hot and blue when cold.
SeeSink
One impediment to automatic faucets in bathroom and
kitchen sinks is the lack of control over temperature and
flow. Nevertheless, their simple application of automation
Figure 2. SeeSink can interpret a variety of tasks being
performed by the user to provide useful hands-free control of
water temperature and flow.
Unfortunately these systems do not directly prevent non-
compliance (which is estimated at 50% in hospitals, for
example [30]). Dirty hands are the primary cause for
infection, and certainly very easy to prevent [3].
CleanSink seeks to motivate critical behavior change by
augmenting the role of the sink as part of the larger context
CHI 2005 PAPERS: Technology in the Home April 2–7 Portland, Oregon, USA
research through design
This paper follows “Research through Design.”
How to evaluates our work
could contribute to the HCI community.
inventionprocess
relevance extensibility](https://image.slidesharecdn.com/panavi-chi2012-presentation-141223081620-conversion-gate02/85/Panavi-CHI2012-Presentation-10-320.jpg)
![CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Arts“situated” cooking support
Cooking support systems corresponding
to users’ situated actions
have been already considered.
Bradbury et al.
an interactive cookbook
supporting situated actions
using eye-tracking and
speech recognition
NATURAL INPUTS
eyeCOOK is designed specifically to use natural input
modalities: those that humans use in human to human, non
mediated communication [5]. To reduce the need for users
to provide explicit input, or change their behavior to
accommodate interface constraints, implicitly provided
attentional cues are observed and interpreted. We believe
that this approach improves the learnability and
intuitiveness of interfaces designed for novice users.
Voice Commands
eyeCOOK uses context-sensitive, localized grammars. This
allows more synonyms for a given speech recognition
command, reducing the chance of misinterpreting a word.
Eye Gaze Commands
When the user is in range of the eye tracker, eyeCOOK
substitutes the object of the user’s gaze for the word ‘this’
in a speech command. For example, ‘Define this’ will
trigger the define operation on the current eye gaze target.
Since current eye trackers are spatially fixed and offer
limited mobility to users, the user will not always be in a
location where eye tracker input is available. Our speech
grammar is designed so that system functionality is not
her/his task. To achieve this, we must
increase sensing capability [3], improve
coordination among appliances [5], and give
appliances the ability to affect the
environment [3,5].
Environmental Sensors
Temperature sensors used to keep track of the
status of the oven and the elements of the
stove can increase the system’s ability to
guide the user’s cooking experience and
could be synchronized with electronic timers.
Appliance Information Integration
Integrating knowledge of the environment
can result in improved functionality, taking
up less of the user’s time and effort. For
example, user recipe preferences, timing
constraints, as determined by the user’s
electronic schedule, and currently available
ingredients, communicated by food storage
areas, can be combined to suggest recipes.
As well, selecting a recipe can result in the
addition of necessary ingredients to an
electronic shopping list stored on the user’s
Personal Data Assistant (PDA).
Active Environmental Actions
The kitchen should not only be aware of its environment,
but it should also be able to affect it. Thus, it should be able
to take actions which increase efficiency, and reduce the
user’s action load, like automatically preheating an oven.
CONCLUSIONS
We have presented eyeCOOK, a gaze and speech enabled
multimodal Attentive User Interface. We have also
presented our vision of an Attentive Kitchen in which
appliances, informed by sensors, coordinate their behavior,
and have the capability to affect the environment. This can
reduce the user’s workload, and permit rationalizing
requests for user attention.
REFERENCES
1. Ju, W. et al. (2001). CounterActive: An Interactive
Cookbook for the Kitchen Counter. Extended Abstracts
of CHI 2001 (Seattle, April 2001) pp. 269-270
2. Norman, D. A. The Invisible Computer, MIT press,
1999
3. Schmidt, A. et al. How to Build Smart Appliances.
IEEE Personal Communications 8(4), August 2001.
pp. 66-71.
Figure 1. eyeCOOK in Page Display Mode
Interactive & Student Posters: Computers Everywhere CHI 2003: NEW HORIZONSPosters: Computers Everywhere CHI 2003: NEW HORIZONS
Olivier et al.
situated coaching system
by integrating projection
systems, RFID,
accelerometers, and
under-floor pressure
sensing technologies.
explore different configurations of sensor-dependent display
behavior significantly helps in the exploration and crafting of
design ideas. Figure 4 shows a prototype under development in
which the ambient displays (on the wall above the main worktop)
respond to the turning of the pages of a specially designed
cookbook. The cookbook has an RFID tag embedded in each page
allowing the kitchen to detect the page that the book is currently
open at. Responding to this, the ambient displays present relevant
food information and even personal media related to the recipe
and past times in the cook’s life when the corresponding meal was
prepared.
4.2 A design tool for users
Another significant challenge in designing pervasive computing
applications is the involvement with users in the design process.
We have conducted a number participatory design exercises
involving older users and found that a significant barrier to
exploring design concepts is adequately explaining the scope of
the technologies involved. By demonstrating simple mappings
between sensors and display in demonstration applications in the
Ambient Kitchen we have found that we can greatly improve lay
users' understanding of the potential functionality. For example,
figure 5 shows one such commonly used application in which
sample recipes are projected in response to the ingredients placed
on the bench. Traffic light indicators on the display also show
which of the recipes ingredients are in the kitchen’s cupboards.
Though a simple demonstration of how sensor data can be
mapped to information sources (and then to information displays)
our experience is that such illustrations can help users think about
both more mundane and adventurous (and useful) applications of
such “invisible” technologies.
Figure 5. The Ambient Kitchen has been used to facilitate
your
different people. For an
discussions and focus groups on the topic of pervasive
computing as part of a wider participatory design process
looking at ICT and nutrition for older people.
4.3 An observatory to collect sensor data
Realizing situated services that are responsive to both
actions and intentions requires significant development of activity
recognition algorithms themselves. As such, multi-sensor
benchmarks of everyday activities are not widely available, and as
part of our own research, and to support the research of
collaborators and the wider research community, we are
developing a number of such benchmarks by capturing data for
multiple subjects and activities, and hand annotating these
datasets. Activities range from gaze data for head pose tracking
algorithm development, primarily using video streams alone (see
figure 6), to naturalistic data sets relating to multi-step food
preparation for which RFID, accelerometer, pressure and video
data is collected and hand annotated.
4.4 An evaluation test bed
Evaluation means different things to
engineer the question is "does it work?" That is, are the functional
requirements met, does it complete certain tests accurately and
without failing. For the human factors engineer the question is
"does it perform a useful function in the context that it is intended
to be used". The latter question can only really be answered by
installing the technology in a range of real contexts. In the case of
kitchen technologies, this means real lived-in homes. Constraints
such as household routines and the different uses different
members of a family use different rooms for at different times can
be critical in the success or failure of home technologies. These
constraints are only really apparent in the context of real home
use. Laboratory-based facilities such as the ambient kitchen thus
are of limited use in this respect. However it is possible to use
them to do more limited evaluations of functional requirements
that still have face validity.
Figure 6. Simultaneously captured data from the embedde
y evaluations of this kind but we are
taken. Thus the teabags might be in a container on work surface,
d
cameras allows us to develop a benchmark for attention
detection algorithms based on tracking head pose and position
from multiple viewpoints.
We yet have to complete an
planning to do so. For example, we will use actors trained using
video recordings of people with dementia carrying out simple
kitchen tasks such as making a hot drink. These recordings were
made of people doing these tasks, which were of their own
choosing, in their own kitchens. The intention is to make initial
tests of algorithms to detect when these people need prompting
because they have made an error or stopped in the middle of the
task. It would be disorienting and therefore unrealistic to bring
people with dementia into the lab but these existing videos can be
used to configure the Ambient Kitchen to match the kitchen in the
video and then to get an actor to perform the sequence of actions
Figure 1. The Ambient Kitchen is a lab-based high fidelity
pervasive computing prototyping environment. The kitchen is
situated in the main research space in Culture Lab, Newcastle
University and uses modified IKEA units and standard
laminate flooring installed within a wooden structure (see top
figures). Significant care was taken that the underlying
technology is not apparent to the people using the kitchen –
even the wall projection is achieved using “up-lighter” style
projection onto mirrors below the overhead cabinets.
Figure 2. Sample off-the-shelf and custom technologies
integrated in the Ambient Kitchen: (a) 4 DLP projectors for
situated displays; (b) 4 Micaz zigbee motes and sensor
boards for object motion sensing; (c) 200 x custom capacitive
sensors for floor pressure measurement; (d) 8 Feig long
range RFID readers (and sample tag).
Figure 3. Using an RFID tagged control object the state of all
the Ambient Kitchen sensors can be examined on the main
display: floor pressure map (left); accelerometers (center-
top); RFID (center-bottom); and video feeds (right).
4. CASE STUDIES
The Ambient Kitchen has been developed to support a range of
research activities around the problem of providing situated
support for people with dementia, and situated services associated
with food planning, preparation and cooking. As a high fidelity
prototyping environment it allows us to support these activities in
a number of different ways, as an experimental space for
designers, for explaining new technologies to users, for collecting
sensor data in benchmark development for activity recognition
algorithms, and for the evaluation of complete solutions in a
naturalistic setting.
Figure 4. A current design scenario in which the pages of a
cookbook have integrated RFID tags. The workbench can
detect the current page and adapts the ambient display
according the page’s contents.
4.1 A design tool for designers
Developing design ideas for pervasive computing applications
usually requires a significant effort on the part of designers to
imagine what interacting in a fully instrumented environment
might be like. In our kitchen scenario, there are no keyboards,
mice or conventional input devices, and the ability to physically
Figure 1. The Ambient Kitchen is a lab-based high fidelity
pervasive computing prototyping environment. The kitchen is
situated in the main research space in Culture Lab, Newcastle
University and uses modified IKEA units and standard
laminate flooring installed within a wooden structure (see top
figures). Significant care was taken that the underlying
technology is not apparent to the people using the kitchen –
even the wall projection is achieved using “up-lighter” style
projection onto mirrors below the overhead cabinets.
Figure 2. Sample off-the-shelf and custom technologies
integrated in the Ambient Kitchen: (a) 4 DLP projectors for
situated displays; (b) 4 Micaz zigbee motes and sensor
boards for object motion sensing; (c) 200 x custom capacitive
sensors for floor pressure measurement; (d) 8 Feig long
range RFID readers (and sample tag).
Figure 3. Using an RFID tagged control object the state of all
the Ambient Kitchen sensors can be examined on the main
display: floor pressure map (left); accelerometers (center-
top); RFID (center-bottom); and video feeds (right).
4. CASE STUDIES
The Ambient Kitchen has been developed to support a range of
research activities around the problem of providing situated
support for people with dementia, and situated services associated
with food planning, preparation and cooking. As a high fidelity
prototyping environment it allows us to support these activities in
a number of different ways, as an experimental space for
designers, for explaining new technologies to users, for collecting
sensor data in benchmark development for activity recognition
algorithms, and for the evaluation of complete solutions in a
naturalistic setting.
Figure 4. A current design scenario in which the pages of a
cookbook have integrated RFID tags. The workbench can
detect the current page and adapts the ambient display
according the page’s contents.
4.1 A design tool for designers
Developing design ideas for pervasive computing applications
usually requires a significant effort on the part of designers to
imagine what interacting in a fully instrumented environment
might be like. In our kitchen scenario, there are no keyboards,
mice or conventional input devices, and the ability to physically
explore different configurations of sensor-dependent display
behavior significantly helps in the exploration and crafting of
design ideas. Figure 4 shows a prototype under development in
which the ambient displays (on the wall above the main worktop)
respond to the turning of the pages of a specially designed
cookbook. The cookbook has an RFID tag embedded in each page
allowing the kitchen to detect the page that the book is currently
open at. Responding to this, the ambient displays present relevant
food information and even personal media related to the recipe
and past times in the cook’s life when the corresponding meal was
prepared.
4.2 A design tool for users
Another significant challenge in designing pervasive computing
applications is the involvement with users in the design process.
We have conducted a number participatory design exercises
involving older users and found that a significant barrier to
exploring design concepts is adequately explaining the scope of
the technologies involved. By demonstrating simple mappings
between sensors and display in demonstration applications in the
Ambient Kitchen we have found that we can greatly improve lay
users' understanding of the potential functionality. For example,
figure 5 shows one such commonly used application in which
sample recipes are projected in response to the ingredients placed
on the bench. Traffic light indicators on the display also show
which of the recipes ingredients are in the kitchen’s cupboards.
Though a simple demonstration of how sensor data can be
mapped to information sources (and then to information displays)
our experience is that such illustrations can help users think about
both more mundane and adventurous (and useful) applications of
such “invisible” technologies.
Figure 5. The Ambient Kitchen has been used to facilitate
your
different people. For an
discussions and focus groups on the topic of pervasive
computing as part of a wider participatory design process
looking at ICT and nutrition for older people.
4.3 An observatory to collect sensor data
Realizing situated services that are responsive to both
actions and intentions requires significant development of activity
recognition algorithms themselves. As such, multi-sensor
benchmarks of everyday activities are not widely available, and as
part of our own research, and to support the research of
collaborators and the wider research community, we are
developing a number of such benchmarks by capturing data for
multiple subjects and activities, and hand annotating these
datasets. Activities range from gaze data for head pose tracking
algorithm development, primarily using video streams alone (see
figure 6), to naturalistic data sets relating to multi-step food
preparation for which RFID, accelerometer, pressure and video
data is collected and hand annotated.
4.4 An evaluation test bed
Evaluation means different things to
engineer the question is "does it work?" That is, are the functional
requirements met, does it complete certain tests accurately and
without failing. For the human factors engineer the question is
"does it perform a useful function in the context that it is intended
to be used". The latter question can only really be answered by
installing the technology in a range of real contexts. In the case of
kitchen technologies, this means real lived-in homes. Constraints
such as household routines and the different uses different
members of a family use different rooms for at different times can
be critical in the success or failure of home technologies. These
constraints are only really apparent in the context of real home
use. Laboratory-based facilities such as the ambient kitchen thus
are of limited use in this respect. However it is possible to use
them to do more limited evaluations of functional requirements
that still have face validity.
Figure 6. Simultaneously captured data from the embedde
y evaluations of this kind but we are
taken. Thus the teabags might be in a container on work surface,
d
cameras allows us to develop a benchmark for attention
detection algorithms based on tracking head pose and position
from multiple viewpoints.
We yet have to complete an
planning to do so. For example, we will use actors trained using
video recordings of people with dementia carrying out simple
kitchen tasks such as making a hot drink. These recordings were
made of people doing these tasks, which were of their own
choosing, in their own kitchens. The intention is to make initial
tests of algorithms to detect when these people need prompting
because they have made an error or stopped in the middle of the
task. It would be disorienting and therefore unrealistic to bring
people with dementia into the lab but these existing videos can be
used to configure the Ambient Kitchen to match the kitchen in the
video and then to get an actor to perform the sequence of actions
Chi et al.
a kitchen providing real-
time feedback, tracking
the number of calories in
food ingredients.
Enabling
Smart Kit
Copyright is held by the author/owner(s).
CHI 2007, April 28–May 3, 2007, San Jose, California, US
ACM 978-1-59593-642-4/07/0004.
Pei-yu (Peggy) Chi
National Taiwan University
1, Sec. 4, Roosevelt Rd.,
Taipei 106, Taiwan
peggychi@csie.org
Jen-hao Chen
National Taiwan University
1, Sec. 4, Roosevelt Rd.,
Taipei 106, Taiwan
r95922023@ntu.edu.tw
Hao-hua Chu
National Taiwan University
1, Sec. 4, Roosevelt Rd.,
Taipei 106, Taiwan
hchu@csie.ntu.edu.tw
Bing-Yu Chen
National Taiwan University
1, Sec. 4, Roosevelt Rd.,
Taipei 106, Taiwan
robin@ntu.edu.tw
LCD
display
smart
cabinet
figure 1. Smart Kitchen promotes
healthy cooking awareness and
cooking interaction to the cook.
smart
stove
smart
counter
CHI 2006 · Work-in-ProgressCHI 2007 • Work-in-Progress
To simu
on a hu
the food
time. W
the coun
an over
observe
testing o
food ing
RFID tag
After th
element
rule eng
activitie
Food ing
This inv
from a s
placed o
holding
up of a
figure 5
nutrition
overall in
For every container in the system,
the interface shows weight
information about food ingredients
the container has.
figure 4. Dialog window for asking
input the name of new food item in
the system.
CHI 2006 · Work-in-ProgressCHI 2007 • Work-in-Progress](https://image.slidesharecdn.com/panavi-chi2012-presentation-141223081620-conversion-gate02/85/Panavi-CHI2012-Presentation-14-320.jpg)
![CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Arts“situated” cooking support
panavi system provides a new user
experience design by situated suggestions;
real-time sensing and giving feedback
the combination of the
pan and the interactive
recipe application
enables users to
challenge professional
culinary arts by
enjoying daily cooking
in domestic kitchens
NATURAL INPUTS
eyeCOOK is designed specifically to use natural input
modalities: those that humans use in human to human, non
mediated communication [5]. To reduce the need for users
to provide explicit input, or change their behavior to
accommodate interface constraints, implicitly provided
attentional cues are observed and interpreted. We believe
that this approach improves the learnability and
intuitiveness of interfaces designed for novice users.
Voice Commands
eyeCOOK uses context-sensitive, localized grammars. This
allows more synonyms for a given speech recognition
command, reducing the chance of misinterpreting a word.
Eye Gaze Commands
her/his task. To achieve this, we must
increase sensing capability [3], improve
coordination among appliances [5], and give
appliances the ability to affect the
environment [3,5].
Environmental Sensors
Temperature sensors used to keep track of the
status of the oven and the elements of the
stove can increase the system’s ability to
guide the user’s cooking experience and
could be synchronized with electronic timers.
Appliance Information Integration
Integrating knowledge of the environment
can result in improved functionality, taking
up less of the user’s time and effort. For
example, user recipe preferences, timing
constraints, as determined by the user’s
electronic schedule, and currently available
ingredients, communicated by food storage
areas, can be combined to suggest recipes.
As well, selecting a recipe can result in the
addition of necessary ingredients to an
electronic shopping list stored on the user’s
Personal Data Assistant (PDA).
Active Environmental Actions
The kitchen should not only be aware of its environment,
but it should also be able to affect it. Thus, it should be able
to take actions which increase efficiency, and reduce the
user’s action load, like automatically preheating an oven.
CONCLUSIONS
We have presented eyeCOOK, a gaze and speech enabled
multimodal Attentive User Interface. We have also
presented our vision of an Attentive Kitchen in which
appliances, informed by sensors, coordinate their behavior,
and have the capability to affect the environment. This can
reduce the user’s workload, and permit rationalizing
requests for user attention.
Figure 1. eyeCOOK in Page Display Mode
Interactive & Student Posters: Computers Everywhere CHI 2003: NEW HORIZONSPosters: Computers Everywhere CHI 2003: NEW HORIZONS
explore different configurations of sensor-dependent display
behavior significantly helps in the exploration and crafting of
design ideas. Figure 4 shows a prototype under development in
which the ambient displays (on the wall above the main worktop)
respond to the turning of the pages of a specially designed
cookbook. The cookbook has an RFID tag embedded in each page
allowing the kitchen to detect the page that the book is currently
open at. Responding to this, the ambient displays present relevant
food information and even personal media related to the recipe
and past times in the cook’s life when the corresponding meal was
prepared.
4.2 A design tool for users
Another significant challenge in designing pervasive computing
applications is the involvement with users in the design process.
We have conducted a number participatory design exercises
involving older users and found that a significant barrier to
exploring design concepts is adequately explaining the scope of
the technologies involved. By demonstrating simple mappings
between sensors and display in demonstration applications in the
Ambient Kitchen we have found that we can greatly improve lay
users' understanding of the potential functionality. For example,
figure 5 shows one such commonly used application in which
sample recipes are projected in response to the ingredients placed
on the bench. Traffic light indicators on the display also show
which of the recipes ingredients are in the kitchen’s cupboards.
Though a simple demonstration of how sensor data can be
mapped to information sources (and then to information displays)
our experience is that such illustrations can help users think about
both more mundane and adventurous (and useful) applications of
such “invisible” technologies.
Figure 5. The Ambient Kitchen has been used to facilitate
your
different people. For an
discussions and focus groups on the topic of pervasive
computing as part of a wider participatory design process
looking at ICT and nutrition for older people.
4.3 An observatory to collect sensor data
Realizing situated services that are responsive to both
actions and intentions requires significant development of activity
recognition algorithms themselves. As such, multi-sensor
benchmarks of everyday activities are not widely available, and as
part of our own research, and to support the research of
collaborators and the wider research community, we are
developing a number of such benchmarks by capturing data for
multiple subjects and activities, and hand annotating these
datasets. Activities range from gaze data for head pose tracking
algorithm development, primarily using video streams alone (see
figure 6), to naturalistic data sets relating to multi-step food
preparation for which RFID, accelerometer, pressure and video
data is collected and hand annotated.
4.4 An evaluation test bed
Evaluation means different things to
engineer the question is "does it work?" That is, are the functional
requirements met, does it complete certain tests accurately and
without failing. For the human factors engineer the question is
"does it perform a useful function in the context that it is intended
to be used". The latter question can only really be answered by
installing the technology in a range of real contexts. In the case of
kitchen technologies, this means real lived-in homes. Constraints
such as household routines and the different uses different
members of a family use different rooms for at different times can
be critical in the success or failure of home technologies. These
constraints are only really apparent in the context of real home
use. Laboratory-based facilities such as the ambient kitchen thus
are of limited use in this respect. However it is possible to use
them to do more limited evaluations of functional requirements
that still have face validity.
Figure 6. Simultaneously captured data from the embedde
y evaluations of this kind but we are
taken. Thus the teabags might be in a container on work surface,
d
cameras allows us to develop a benchmark for attention
detection algorithms based on tracking head pose and position
from multiple viewpoints.
We yet have to complete an
planning to do so. For example, we will use actors trained using
video recordings of people with dementia carrying out simple
kitchen tasks such as making a hot drink. These recordings were
made of people doing these tasks, which were of their own
choosing, in their own kitchens. The intention is to make initial
tests of algorithms to detect when these people need prompting
because they have made an error or stopped in the middle of the
task. It would be disorienting and therefore unrealistic to bring
people with dementia into the lab but these existing videos can be
used to configure the Ambient Kitchen to match the kitchen in the
video and then to get an actor to perform the sequence of actions
Figure 1. The Ambient Kitchen is a lab-based high fidelity
pervasive computing prototyping environment. The kitchen is
situated in the main research space in Culture Lab, Newcastle
University and uses modified IKEA units and standard
laminate flooring installed within a wooden structure (see top
figures). Significant care was taken that the underlying
technology is not apparent to the people using the kitchen –
even the wall projection is achieved using “up-lighter” style
projection onto mirrors below the overhead cabinets.
Figure 2. Sample off-the-shelf and custom technologies
integrated in the Ambient Kitchen: (a) 4 DLP projectors for
situated displays; (b) 4 Micaz zigbee motes and sensor
boards for object motion sensing; (c) 200 x custom capacitive
sensors for floor pressure measurement; (d) 8 Feig long
range RFID readers (and sample tag).
Figure 3. Using an RFID tagged control object the state of all
the Ambient Kitchen sensors can be examined on the main
display: floor pressure map (left); accelerometers (center-
top); RFID (center-bottom); and video feeds (right).
4. CASE STUDIES
The Ambient Kitchen has been developed to support a range of
research activities around the problem of providing situated
support for people with dementia, and situated services associated
with food planning, preparation and cooking. As a high fidelity
prototyping environment it allows us to support these activities in
a number of different ways, as an experimental space for
designers, for explaining new technologies to users, for collecting
sensor data in benchmark development for activity recognition
algorithms, and for the evaluation of complete solutions in a
naturalistic setting.
Figure 4. A current design scenario in which the pages of a
cookbook have integrated RFID tags. The workbench can
detect the current page and adapts the ambient display
according the page’s contents.
4.1 A design tool for designers
Developing design ideas for pervasive computing applications
usually requires a significant effort on the part of designers to
imagine what interacting in a fully instrumented environment
might be like. In our kitchen scenario, there are no keyboards,
mice or conventional input devices, and the ability to physically
Figure 1. The Ambient Kitchen is a lab-based high fidelity
pervasive computing prototyping environment. The kitchen is
situated in the main research space in Culture Lab, Newcastle
University and uses modified IKEA units and standard
laminate flooring installed within a wooden structure (see top
figures). Significant care was taken that the underlying
technology is not apparent to the people using the kitchen –
even the wall projection is achieved using “up-lighter” style
projection onto mirrors below the overhead cabinets.
Figure 2. Sample off-the-shelf and custom technologies
integrated in the Ambient Kitchen: (a) 4 DLP projectors for
situated displays; (b) 4 Micaz zigbee motes and sensor
boards for object motion sensing; (c) 200 x custom capacitive
sensors for floor pressure measurement; (d) 8 Feig long
range RFID readers (and sample tag).
Figure 3. Using an RFID tagged control object the state of all
the Ambient Kitchen sensors can be examined on the main
display: floor pressure map (left); accelerometers (center-
top); RFID (center-bottom); and video feeds (right).
4. CASE STUDIES
The Ambient Kitchen has been developed to support a range of
research activities around the problem of providing situated
support for people with dementia, and situated services associated
with food planning, preparation and cooking. As a high fidelity
prototyping environment it allows us to support these activities in
a number of different ways, as an experimental space for
designers, for explaining new technologies to users, for collecting
sensor data in benchmark development for activity recognition
algorithms, and for the evaluation of complete solutions in a
naturalistic setting.
Figure 4. A current design scenario in which the pages of a
cookbook have integrated RFID tags. The workbench can
detect the current page and adapts the ambient display
according the page’s contents.
4.1 A design tool for designers
Developing design ideas for pervasive computing applications
usually requires a significant effort on the part of designers to
imagine what interacting in a fully instrumented environment
might be like. In our kitchen scenario, there are no keyboards,
mice or conventional input devices, and the ability to physically
explore different configurations of sensor-dependent display
behavior significantly helps in the exploration and crafting of
design ideas. Figure 4 shows a prototype under development in
which the ambient displays (on the wall above the main worktop)
respond to the turning of the pages of a specially designed
cookbook. The cookbook has an RFID tag embedded in each page
allowing the kitchen to detect the page that the book is currently
open at. Responding to this, the ambient displays present relevant
food information and even personal media related to the recipe
and past times in the cook’s life when the corresponding meal was
prepared.
4.2 A design tool for users
Another significant challenge in designing pervasive computing
applications is the involvement with users in the design process.
We have conducted a number participatory design exercises
involving older users and found that a significant barrier to
exploring design concepts is adequately explaining the scope of
the technologies involved. By demonstrating simple mappings
between sensors and display in demonstration applications in the
Ambient Kitchen we have found that we can greatly improve lay
users' understanding of the potential functionality. For example,
figure 5 shows one such commonly used application in which
sample recipes are projected in response to the ingredients placed
on the bench. Traffic light indicators on the display also show
which of the recipes ingredients are in the kitchen’s cupboards.
Though a simple demonstration of how sensor data can be
mapped to information sources (and then to information displays)
our experience is that such illustrations can help users think about
both more mundane and adventurous (and useful) applications of
such “invisible” technologies.
Figure 5. The Ambient Kitchen has been used to facilitate
your
different people. For an
discussions and focus groups on the topic of pervasive
computing as part of a wider participatory design process
looking at ICT and nutrition for older people.
4.3 An observatory to collect sensor data
Realizing situated services that are responsive to both
actions and intentions requires significant development of activity
recognition algorithms themselves. As such, multi-sensor
benchmarks of everyday activities are not widely available, and as
part of our own research, and to support the research of
collaborators and the wider research community, we are
developing a number of such benchmarks by capturing data for
multiple subjects and activities, and hand annotating these
datasets. Activities range from gaze data for head pose tracking
algorithm development, primarily using video streams alone (see
figure 6), to naturalistic data sets relating to multi-step food
preparation for which RFID, accelerometer, pressure and video
data is collected and hand annotated.
4.4 An evaluation test bed
Evaluation means different things to
engineer the question is "does it work?" That is, are the functional
requirements met, does it complete certain tests accurately and
without failing. For the human factors engineer the question is
"does it perform a useful function in the context that it is intended
to be used". The latter question can only really be answered by
installing the technology in a range of real contexts. In the case of
kitchen technologies, this means real lived-in homes. Constraints
such as household routines and the different uses different
members of a family use different rooms for at different times can
be critical in the success or failure of home technologies. These
constraints are only really apparent in the context of real home
use. Laboratory-based facilities such as the ambient kitchen thus
are of limited use in this respect. However it is possible to use
them to do more limited evaluations of functional requirements
that still have face validity.
Figure 6. Simultaneously captured data from the embedde
y evaluations of this kind but we are
taken. Thus the teabags might be in a container on work surface,
d
cameras allows us to develop a benchmark for attention
detection algorithms based on tracking head pose and position
from multiple viewpoints.
We yet have to complete an
planning to do so. For example, we will use actors trained using
video recordings of people with dementia carrying out simple
kitchen tasks such as making a hot drink. These recordings were
made of people doing these tasks, which were of their own
choosing, in their own kitchens. The intention is to make initial
tests of algorithms to detect when these people need prompting
because they have made an error or stopped in the middle of the
task. It would be disorienting and therefore unrealistic to bring
people with dementia into the lab but these existing videos can be
used to configure the Ambient Kitchen to match the kitchen in the
video and then to get an actor to perform the sequence of actions
Enabling N
Smart Kitch
Copyright is held by the author/owner(s).
CHI 2007, April 28–May 3, 2007, San Jose, California, USA.
ACM 978-1-59593-642-4/07/0004.
Pei-yu (Peggy) Chi
National Taiwan University
1, Sec. 4, Roosevelt Rd.,
Taipei 106, Taiwan
peggychi@csie.org
Jen-hao Chen
National Taiwan University
1, Sec. 4, Roosevelt Rd.,
Taipei 106, Taiwan
r95922023@ntu.edu.tw
Hao-hua Chu
National Taiwan University
1, Sec. 4, Roosevelt Rd.,
Taipei 106, Taiwan
hchu@csie.ntu.edu.tw
Bing-Yu Chen
National Taiwan University
1, Sec. 4, Roosevelt Rd.,
Taipei 106, Taiwan
robin@ntu.edu.tw
LCD
display
smart
cabinet
figure 1. Smart Kitchen promotes
healthy cooking awareness and
cooking interaction to the cook.
smart
stove
smart
counter
CHI 2006 · Work-in-ProgressCHI 2007 • Work-in-Progress
To simula
on a hum
the food
time. Wh
the count
an overhe
observer
testing ot
food ingr
RFID tags
After the
elements
rule engin
activities
Food ingr
This invo
from a st
placed on
holding th
up of a se
figure 4. Dialog window for asking
input the name of new food item in
the system.
CHI 2006 · Work-in-ProgressCHI 2007 • Work-in-Progress
interactive cookbook
situated coaching
system
kitchen providing the
number of calories
previous “situated” cooking support systems](https://image.slidesharecdn.com/panavi-chi2012-presentation-141223081620-conversion-gate02/85/Panavi-CHI2012-Presentation-15-320.jpg)
![CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Artsdisplay
Figure 5. Three Kinds of Setting Displayed in the Monitor; (a) Summary, (b) Detail, (c) Condition / Name of each Area [See, (b)]; (b-1) Main Panel,
(b-2) Current Step Panel and Checkbox, (b-3) Comment Panel, (b-4) Temperature Panel, (b-5) Timer Panel
elapsed time, and the timer of boiling the pasta, which is also
a supplemental function to Condition mode.
IMPLEMENTATION & ARCHITECTURE
panavi system mainly consists of three parts; the special
the projector. Actuators, LEDs and vibration motors are
embedded in the handle, controlled by the signals from the
computer system via MOXA-B.
Display and Computer
There are three kinds of setting
displayed in the monitor
(1) Main Panel
(2) Current Step Panel and Checkbox
(3) Comment Panel
(4) Temperature Panel
(5) Timer Panel
Summary Detail Condition
Users can use any setting anytime.](https://image.slidesharecdn.com/panavi-chi2012-presentation-141223081620-conversion-gate02/85/Panavi-CHI2012-Presentation-54-320.jpg)
![CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Artsoriginal cooking sequence
Step 1. Pasta (1) - Boil Water
Step 2. Making Egg Sauce
Step 3. Pasta (2) - Start Cooking Pasta
Step 4. Pasta (3) - Start Timer
Step 5. Pancetta (1) - Fry Pancetta [A]
Step 6. Pancetta (2) Add Wine
Step 7. Pancetta (3) - Seasoning
Step 8. Pancetta (4) - Cooling [B]
Step 9. Pasta (4) - Drain Water
Step 10. Finish (1) - Add Pasta to Pan
Step 11. Finish (2) - Dress with Egg Sauce
Step 12. Finish (3) - Heat Egg Sauce [C]
Step 13. Serve
The original cooking sequence
models the recipe of Carbonara.
Each step is programmed with
settings; temperature, sounds,
and vibrations etc.
LED lamps indicate temperature conditions and a vibration
motor moves when the users’ temperature control is good,
which are embedded within the handle of the pan. The
instructional graphics by the projection can also be checked
on the display monitor to support the recognition of the
projection when the pan is removed from the stove or when
the projection cannot be recognized since there is much
ingredient on the pan. If part of the system fails, other parts
could still help the user (i.e. if the projector fails, the display
and the LED indication work). See, Table 1. The temperature
color changes from white to blue, green, yellow, and red,
depending on the value, which also synchronizes with the
LEDs. White is the default color or indicates when the
condition temperature is too low, blue indicates 5-10 degrees
lower than the proper temperature setting, green indicates
when it is at the proper temperature within 5 degrees, yellow
indicates 5-10 degrees higher, and red indicates over 15
- 0 54 - 64 65 - 74 75 - 84 85 -
ON—
the second
hand of a clock
ON
warning
tone
—
—
Color
TEMP
（°C）
Vibration
Sound
- 69 70 - 79 80 - 89 90 - 99 100 -
B
- 149 150 - 159 160 - 169 170 - 179 180 -A
C
Blue Green Yellow RedWhite
ON
the second
hand of a clock
the second
hand of a clock
Table 1. Parameters to Cook the Carvonara Programed in the System.
Its procedure is divided into 13 steps.](https://image.slidesharecdn.com/panavi-chi2012-presentation-141223081620-conversion-gate02/85/Panavi-CHI2012-Presentation-55-320.jpg)
![CHI 2012 “EATING + COOKING” Monday, May 7th, 2012
panavi: Recipe Medium with a Sensors-Embedded Pan
for Domestic Users to Master Professional Culinary Artssmart daily commodities
This research provided a model for designing
daily commodities (e.g. kitchen utensils)
as smart objects enriching our everyday life.
Figure 5. Three Kinds of Setting Displayed in the Monitor; (a) Summary, (b) Detail, (c) Condition / Name of each Area [See, (b)]; (b-1) Main Panel,
(b-2) Current Step Panel and Checkbox, (b-3) Comment Panel, (b-4) Temperature Panel, (b-5) Timer Panel
elapsed time, and the timer of boiling the pasta, which is also
a supplemental function to Condition mode.
IMPLEMENTATION & ARCHITECTURE
panavi system mainly consists of three parts; the special
kitchen utensil (frying-pan) with embedded sensors and
actuators, the display system connecting the computer with
the touch monitor and the projection system, and the software
system showing cooking sequence information (Figure 6).
panavi
display
Computer
(panavi OS + Original
Cooking Sequence)
Electronic
Circuit
(MOXA-B)
Electronic
Circuit
(MOXA-A)
Actuators
Xbee Wireless
Communication
Sensors
Thermocouple Sensor
Acceleration Sensor
Vibration Motor
LEDs
USB Cable
(Serial Communication)VGA
Cable
Mirror
Projector
Speaker
Projection
Touch
Monitor
Special Pan
Figure 6. System Architecture
the projector. Actuators, LEDs and vibration motors are
embedded in the handle, controlled by the signals from the
computer system via MOXA-B.
Display and Computer
The computer is connected with ‘panavi display’ packaging
touch panel monitor and projector, and ‘panavi OS’ with
the ‘Original Cooking Sequence’ works as an Adobe Flash
application on the computer (as shown in Figure 6). The
panavi OS displays the instructions by analyzing the sensors’
degrees against parameters programed in the system.
Original Cooking Sequence
The original cooking sequence models the recipe of
Carbonara, consisting of videos and photos in addition to
the general text recipes. For the panavi OS, this cooking
sequence was reconstructed by the development team and its
procedure is divided into 13 steps (as shown in Table 2). Each
step is programmed with settings; temperature, sounds, and
vibrations settings etc. (as shown in Table 1).
Preparing Pasta (Table 2, Step 1-4) When the checkbox
of the Step 1 is touched, the elapse time counter from the
beginning (See, Figure 5-5) starts. The normal cooking time
is set to 20 minutes. When Step 4 is checked, the pasta timer
technical framework methodology
this framework senses and analyze users’
state, and generating feedback suitable for
each user’s situations.
this integrated technical elements
should be designed through
the prototyping process
in real domestic environments.
×](https://image.slidesharecdn.com/panavi-chi2012-presentation-141223081620-conversion-gate02/85/Panavi-CHI2012-Presentation-71-320.jpg)
Uploaded byDaisuke Uriu
Panavi CHI2012 Presentation
The document discusses the development of 'panavi,' an innovative cooking medium that incorporates a sensor-embedded frying pan designed to assist domestic users in mastering professional culinary techniques. It highlights the transition from traditional text-based recipes to more interactive and visually-oriented formats, with the aim of increasing user confidence and reducing the complexity of cooking advanced recipes. Various technological approaches within the kitchen environment are examined, including interactive cookbooks and systems that utilize environmental sensors to enhance the cooking experience.
![HCI Research as Problem-Solving [CHI'16, presentation slides]](https://cdn.slidesharecdn.com/ss_thumbnails/hci-research-as-problem-solving-chi16-160512002517-thumbnail.jpg?width=640&height=640&fit=bounds)
Panavi CHI2012 Presentation
- 1. Recipe Medium witha Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Arts Daisuke Uriu Mizuki Namai Satoru Tokuhisa Ryo Kashiwagi Masahiko Inami Naohito Okude
- 2. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artscooking/recipe medium Julia Child “The French Chef” (1963-1973) - the transition of recipe media from “text” to “video” - Her cooking show, translating traditional print or verbal recipe, was a great hit with domestic audiences in the US.
- 3. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artscooking/recipe medium Cooking or recipe medium has entered a transitional period with regards to HCI. Silver Spoon text based recipe book The French Chef TV show Vita Craft IHIQ automatically cooking system product Digital Thermometer Frying Pan product Personal Trainer: Cooking by NINTENDO
- 4. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artsconcept: title of this paper panavi: Recipe Medium with a Sensors- Embedded Pan for Domestic Users to Master Professional Culinary Arts
- 5. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artsconcept: design overview panavi: Recipe Medium with a Sensors- Embedded Pan for Domestic Users to Master Professional Culinary Arts display: digital recipe application sensors- embedded frying pan wirelessly connected with the display programmed the ways of professional chef’s cooking
- 6.
- 8.
- 9. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artsuncertain environment This research attempts to obtain ways of designing computer mediated artifacts that support uncertain domestic environments: kitchens. firewater uncertain domestic situated actions; all activities in kitchens based on activities in kitchens cannot be planned as a static model. users must manage uncertain troubles
- 10. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Arts Introduction and Motivation Cooking advanced recipes can be complicated, and requires following detailed instructions to achieve the desired result. Textual instructions like paper recipes are often difficult to comprehend; they use technical terms or instructions that require prior experience to be understood (“How much is a handful of chopped basil?” or “How brown are caramelized onions?”). These types of information can be transferred more easily using images or videos, just as cooking shows do. However, cooking shows are not designed to be followed in real-time. Although the increasing audience of cooking shows in European television [4] indicates that there is a growing interest in this format, no participant of our preliminary studies followed a recipe while watching a cooking show. They rather liked the inspiring aspect of the visual aesthetics. Typically, they would follow textual instructions when cooking, or not use any kind of instructions at all. The final outcome of a recipe depends on many factors and can fail at any point during the entire process. This can be demanding and stressful for beginners, perhaps discouraging them. We are developing PersonalChef, an interactive kitchen counter (see Fig.1), to avoid disappointing hobby chefs when preparing unfamiliar recipes and to increase their confidence. Related Work Several systems have been developed to bring technical innovations into the domestic kitchen environment. Cooking Navi [7] considers cooking as a time optimization problem when preparing a menu consisting of multiple recipes, and tries to optimize single cooking processes in order to have all dishes finished at the right time. Semantic Cookbook [10] and Kitchen of the Future [11] installed various electronic devices into the kitchen to record and share cooking sessions. Living Cookbook [12] offers these possibilities as well, but it further focuses on the social experience of cooking and collaboration. eyeCOOK [3] and Smart Kitchen [8] focus on the actual user input. They try to facilitate active user input or reduce it by tracking user’s actions in the kitchen using eye-gaze, speech commands, and foot switches. CounterActive [9] integrates step-by-step cooking instructions using multimedia invisibly into the kitchen counter. Besides research prototypes, there are also commercial systems. Nintendo’s personal cooking instructor (nintendo.com) on the small-screen NintendoDS device is an interactive cookbook and provides live cooking demonstrations. While other work often focused on efficiency [1,5], our system aims to support the entertaining and social gress (Spotlight on Posters Days 1 & 2) April 12–13, 2010, Atlanta, GA, USA 3404 Figure 1. HeatSink illuminates the stream of water according to its temperature, becoming red when hot and blue when cold. SeeSink One impediment to automatic faucets in bathroom and kitchen sinks is the lack of control over temperature and flow. Nevertheless, their simple application of automation Figure 2. SeeSink can interpret a variety of tasks being performed by the user to provide useful hands-free control of water temperature and flow. Unfortunately these systems do not directly prevent non- compliance (which is estimated at 50% in hospitals, for example [30]). Dirty hands are the primary cause for infection, and certainly very easy to prevent [3]. CleanSink seeks to motivate critical behavior change by augmenting the role of the sink as part of the larger context CHI 2005 PAPERS: Technology in the Home April 2–7 Portland, Oregon, USA research through design This paper follows “Research through Design.” How to evaluates our work could contribute to the HCI community. inventionprocess relevance extensibility
- 11.
- 12. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artstraditional text based recipes Traditional media for recipes describe steps toward a perfect completion of cooking. Silver Spoon very famous recipe book SPAGHETTI ALLA CARBONARA SPAGHETTI WITH CARBONARA SAUCE Ingredient for 1 person Preparation time: 15' (preparation: 10' cooking: 5') 14 oz (100 g) spaghetti 1.5 oz (45 g) pancetta 1 egg 1 egg yolk Method Prepare egg sauce, beating the egg and the egg yolk with Pecorino cheese and a little pepper in a bowl. Cut the pancetta into small strips. Place a large skillet onto a medium heat and slowly saute the pancetta. Cook the spaghetti in a pot of lightly salted boiling water and drain when al dente. Turn the pasta into the skillet with the pancetta, toss and turn off the heat. Add the egg sauce and a little of the cooking water. Stir for 30 seconds or so with a little heat. Turn off the heat, stir again, and serve immediately. 1 oz (100 g) Pecorino cheese 1 tsp (5 ml) extra virgin olive oil Salt Pepper But it is usually difficult for non-professional users to reproduce the taste of meals.
- 13. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Arts“planned” recipe medium The problem of “planned” traditional recipe is close to Lucy Suchman’s “situated actions.” SPAGHETTI ALLA CARBONARA SPAGHETTI WITH CARBONARA SAUCE Ingredient for 1 person Preparation time: 15' (preparation: 10' cooking: 5') 14 oz (100 g) spaghetti 1.5 oz (45 g) pancetta 1 egg 1 egg yolk Method Prepare egg sauce, beating the egg and the egg yolk with Pecorino cheese and a little pepper in a bowl. Cut the pancetta into small strips. Place a large skillet onto a medium heat and slowly saute the pancetta. Cook the spaghetti in a pot of lightly salted boiling water and drain when al dente. Turn the pasta into the skillet with the pancetta, toss and turn off the heat. Add the egg sauce and a little of the cooking water. Stir for 30 seconds or so with a little heat. Turn off the heat, stir again, and serve immediately. 1 oz (100 g) Pecorino cheese 1 tsp (5 ml) extra virgin olive oil Salt Pepper She revealed the programmed instructions do not support situated actions: the errors and problems users actually face are not expected by the system. the “big green button” came from her research. ethnography research about copy machine “planned” text recipe
- 14. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Arts“situated” cooking support Cooking support systems corresponding to users’ situated actions have been already considered. Bradbury et al. an interactive cookbook supporting situated actions using eye-tracking and speech recognition NATURAL INPUTS eyeCOOK is designed specifically to use natural input modalities: those that humans use in human to human, non mediated communication [5]. To reduce the need for users to provide explicit input, or change their behavior to accommodate interface constraints, implicitly provided attentional cues are observed and interpreted. We believe that this approach improves the learnability and intuitiveness of interfaces designed for novice users. Voice Commands eyeCOOK uses context-sensitive, localized grammars. This allows more synonyms for a given speech recognition command, reducing the chance of misinterpreting a word. Eye Gaze Commands When the user is in range of the eye tracker, eyeCOOK substitutes the object of the user’s gaze for the word ‘this’ in a speech command. For example, ‘Define this’ will trigger the define operation on the current eye gaze target. Since current eye trackers are spatially fixed and offer limited mobility to users, the user will not always be in a location where eye tracker input is available. Our speech grammar is designed so that system functionality is not her/his task. To achieve this, we must increase sensing capability [3], improve coordination among appliances [5], and give appliances the ability to affect the environment [3,5]. Environmental Sensors Temperature sensors used to keep track of the status of the oven and the elements of the stove can increase the system’s ability to guide the user’s cooking experience and could be synchronized with electronic timers. Appliance Information Integration Integrating knowledge of the environment can result in improved functionality, taking up less of the user’s time and effort. For example, user recipe preferences, timing constraints, as determined by the user’s electronic schedule, and currently available ingredients, communicated by food storage areas, can be combined to suggest recipes. As well, selecting a recipe can result in the addition of necessary ingredients to an electronic shopping list stored on the user’s Personal Data Assistant (PDA). Active Environmental Actions The kitchen should not only be aware of its environment, but it should also be able to affect it. Thus, it should be able to take actions which increase efficiency, and reduce the user’s action load, like automatically preheating an oven. CONCLUSIONS We have presented eyeCOOK, a gaze and speech enabled multimodal Attentive User Interface. We have also presented our vision of an Attentive Kitchen in which appliances, informed by sensors, coordinate their behavior, and have the capability to affect the environment. This can reduce the user’s workload, and permit rationalizing requests for user attention. REFERENCES 1. Ju, W. et al. (2001). CounterActive: An Interactive Cookbook for the Kitchen Counter. Extended Abstracts of CHI 2001 (Seattle, April 2001) pp. 269-270 2. Norman, D. A. The Invisible Computer, MIT press, 1999 3. Schmidt, A. et al. How to Build Smart Appliances. IEEE Personal Communications 8(4), August 2001. pp. 66-71. Figure 1. eyeCOOK in Page Display Mode Interactive & Student Posters: Computers Everywhere CHI 2003: NEW HORIZONSPosters: Computers Everywhere CHI 2003: NEW HORIZONS Olivier et al. situated coaching system by integrating projection systems, RFID, accelerometers, and under-floor pressure sensing technologies. explore different configurations of sensor-dependent display behavior significantly helps in the exploration and crafting of design ideas. Figure 4 shows a prototype under development in which the ambient displays (on the wall above the main worktop) respond to the turning of the pages of a specially designed cookbook. The cookbook has an RFID tag embedded in each page allowing the kitchen to detect the page that the book is currently open at. Responding to this, the ambient displays present relevant food information and even personal media related to the recipe and past times in the cook’s life when the corresponding meal was prepared. 4.2 A design tool for users Another significant challenge in designing pervasive computing applications is the involvement with users in the design process. We have conducted a number participatory design exercises involving older users and found that a significant barrier to exploring design concepts is adequately explaining the scope of the technologies involved. By demonstrating simple mappings between sensors and display in demonstration applications in the Ambient Kitchen we have found that we can greatly improve lay users' understanding of the potential functionality. For example, figure 5 shows one such commonly used application in which sample recipes are projected in response to the ingredients placed on the bench. Traffic light indicators on the display also show which of the recipes ingredients are in the kitchen’s cupboards. Though a simple demonstration of how sensor data can be mapped to information sources (and then to information displays) our experience is that such illustrations can help users think about both more mundane and adventurous (and useful) applications of such “invisible” technologies. Figure 5. The Ambient Kitchen has been used to facilitate your different people. For an discussions and focus groups on the topic of pervasive computing as part of a wider participatory design process looking at ICT and nutrition for older people. 4.3 An observatory to collect sensor data Realizing situated services that are responsive to both actions and intentions requires significant development of activity recognition algorithms themselves. As such, multi-sensor benchmarks of everyday activities are not widely available, and as part of our own research, and to support the research of collaborators and the wider research community, we are developing a number of such benchmarks by capturing data for multiple subjects and activities, and hand annotating these datasets. Activities range from gaze data for head pose tracking algorithm development, primarily using video streams alone (see figure 6), to naturalistic data sets relating to multi-step food preparation for which RFID, accelerometer, pressure and video data is collected and hand annotated. 4.4 An evaluation test bed Evaluation means different things to engineer the question is "does it work?" That is, are the functional requirements met, does it complete certain tests accurately and without failing. For the human factors engineer the question is "does it perform a useful function in the context that it is intended to be used". The latter question can only really be answered by installing the technology in a range of real contexts. In the case of kitchen technologies, this means real lived-in homes. Constraints such as household routines and the different uses different members of a family use different rooms for at different times can be critical in the success or failure of home technologies. These constraints are only really apparent in the context of real home use. Laboratory-based facilities such as the ambient kitchen thus are of limited use in this respect. However it is possible to use them to do more limited evaluations of functional requirements that still have face validity. Figure 6. Simultaneously captured data from the embedde y evaluations of this kind but we are taken. Thus the teabags might be in a container on work surface, d cameras allows us to develop a benchmark for attention detection algorithms based on tracking head pose and position from multiple viewpoints. We yet have to complete an planning to do so. For example, we will use actors trained using video recordings of people with dementia carrying out simple kitchen tasks such as making a hot drink. These recordings were made of people doing these tasks, which were of their own choosing, in their own kitchens. The intention is to make initial tests of algorithms to detect when these people need prompting because they have made an error or stopped in the middle of the task. It would be disorienting and therefore unrealistic to bring people with dementia into the lab but these existing videos can be used to configure the Ambient Kitchen to match the kitchen in the video and then to get an actor to perform the sequence of actions Figure 1. The Ambient Kitchen is a lab-based high fidelity pervasive computing prototyping environment. The kitchen is situated in the main research space in Culture Lab, Newcastle University and uses modified IKEA units and standard laminate flooring installed within a wooden structure (see top figures). Significant care was taken that the underlying technology is not apparent to the people using the kitchen – even the wall projection is achieved using “up-lighter” style projection onto mirrors below the overhead cabinets. Figure 2. Sample off-the-shelf and custom technologies integrated in the Ambient Kitchen: (a) 4 DLP projectors for situated displays; (b) 4 Micaz zigbee motes and sensor boards for object motion sensing; (c) 200 x custom capacitive sensors for floor pressure measurement; (d) 8 Feig long range RFID readers (and sample tag). Figure 3. Using an RFID tagged control object the state of all the Ambient Kitchen sensors can be examined on the main display: floor pressure map (left); accelerometers (center- top); RFID (center-bottom); and video feeds (right). 4. CASE STUDIES The Ambient Kitchen has been developed to support a range of research activities around the problem of providing situated support for people with dementia, and situated services associated with food planning, preparation and cooking. As a high fidelity prototyping environment it allows us to support these activities in a number of different ways, as an experimental space for designers, for explaining new technologies to users, for collecting sensor data in benchmark development for activity recognition algorithms, and for the evaluation of complete solutions in a naturalistic setting. Figure 4. A current design scenario in which the pages of a cookbook have integrated RFID tags. The workbench can detect the current page and adapts the ambient display according the page’s contents. 4.1 A design tool for designers Developing design ideas for pervasive computing applications usually requires a significant effort on the part of designers to imagine what interacting in a fully instrumented environment might be like. In our kitchen scenario, there are no keyboards, mice or conventional input devices, and the ability to physically Figure 1. The Ambient Kitchen is a lab-based high fidelity pervasive computing prototyping environment. The kitchen is situated in the main research space in Culture Lab, Newcastle University and uses modified IKEA units and standard laminate flooring installed within a wooden structure (see top figures). Significant care was taken that the underlying technology is not apparent to the people using the kitchen – even the wall projection is achieved using “up-lighter” style projection onto mirrors below the overhead cabinets. Figure 2. Sample off-the-shelf and custom technologies integrated in the Ambient Kitchen: (a) 4 DLP projectors for situated displays; (b) 4 Micaz zigbee motes and sensor boards for object motion sensing; (c) 200 x custom capacitive sensors for floor pressure measurement; (d) 8 Feig long range RFID readers (and sample tag). Figure 3. Using an RFID tagged control object the state of all the Ambient Kitchen sensors can be examined on the main display: floor pressure map (left); accelerometers (center- top); RFID (center-bottom); and video feeds (right). 4. CASE STUDIES The Ambient Kitchen has been developed to support a range of research activities around the problem of providing situated support for people with dementia, and situated services associated with food planning, preparation and cooking. As a high fidelity prototyping environment it allows us to support these activities in a number of different ways, as an experimental space for designers, for explaining new technologies to users, for collecting sensor data in benchmark development for activity recognition algorithms, and for the evaluation of complete solutions in a naturalistic setting. Figure 4. A current design scenario in which the pages of a cookbook have integrated RFID tags. The workbench can detect the current page and adapts the ambient display according the page’s contents. 4.1 A design tool for designers Developing design ideas for pervasive computing applications usually requires a significant effort on the part of designers to imagine what interacting in a fully instrumented environment might be like. In our kitchen scenario, there are no keyboards, mice or conventional input devices, and the ability to physically explore different configurations of sensor-dependent display behavior significantly helps in the exploration and crafting of design ideas. Figure 4 shows a prototype under development in which the ambient displays (on the wall above the main worktop) respond to the turning of the pages of a specially designed cookbook. The cookbook has an RFID tag embedded in each page allowing the kitchen to detect the page that the book is currently open at. Responding to this, the ambient displays present relevant food information and even personal media related to the recipe and past times in the cook’s life when the corresponding meal was prepared. 4.2 A design tool for users Another significant challenge in designing pervasive computing applications is the involvement with users in the design process. We have conducted a number participatory design exercises involving older users and found that a significant barrier to exploring design concepts is adequately explaining the scope of the technologies involved. By demonstrating simple mappings between sensors and display in demonstration applications in the Ambient Kitchen we have found that we can greatly improve lay users' understanding of the potential functionality. For example, figure 5 shows one such commonly used application in which sample recipes are projected in response to the ingredients placed on the bench. Traffic light indicators on the display also show which of the recipes ingredients are in the kitchen’s cupboards. Though a simple demonstration of how sensor data can be mapped to information sources (and then to information displays) our experience is that such illustrations can help users think about both more mundane and adventurous (and useful) applications of such “invisible” technologies. Figure 5. The Ambient Kitchen has been used to facilitate your different people. For an discussions and focus groups on the topic of pervasive computing as part of a wider participatory design process looking at ICT and nutrition for older people. 4.3 An observatory to collect sensor data Realizing situated services that are responsive to both actions and intentions requires significant development of activity recognition algorithms themselves. As such, multi-sensor benchmarks of everyday activities are not widely available, and as part of our own research, and to support the research of collaborators and the wider research community, we are developing a number of such benchmarks by capturing data for multiple subjects and activities, and hand annotating these datasets. Activities range from gaze data for head pose tracking algorithm development, primarily using video streams alone (see figure 6), to naturalistic data sets relating to multi-step food preparation for which RFID, accelerometer, pressure and video data is collected and hand annotated. 4.4 An evaluation test bed Evaluation means different things to engineer the question is "does it work?" That is, are the functional requirements met, does it complete certain tests accurately and without failing. For the human factors engineer the question is "does it perform a useful function in the context that it is intended to be used". The latter question can only really be answered by installing the technology in a range of real contexts. In the case of kitchen technologies, this means real lived-in homes. Constraints such as household routines and the different uses different members of a family use different rooms for at different times can be critical in the success or failure of home technologies. These constraints are only really apparent in the context of real home use. Laboratory-based facilities such as the ambient kitchen thus are of limited use in this respect. However it is possible to use them to do more limited evaluations of functional requirements that still have face validity. Figure 6. Simultaneously captured data from the embedde y evaluations of this kind but we are taken. Thus the teabags might be in a container on work surface, d cameras allows us to develop a benchmark for attention detection algorithms based on tracking head pose and position from multiple viewpoints. We yet have to complete an planning to do so. For example, we will use actors trained using video recordings of people with dementia carrying out simple kitchen tasks such as making a hot drink. These recordings were made of people doing these tasks, which were of their own choosing, in their own kitchens. The intention is to make initial tests of algorithms to detect when these people need prompting because they have made an error or stopped in the middle of the task. It would be disorienting and therefore unrealistic to bring people with dementia into the lab but these existing videos can be used to configure the Ambient Kitchen to match the kitchen in the video and then to get an actor to perform the sequence of actions Chi et al. a kitchen providing real- time feedback, tracking the number of calories in food ingredients. Enabling Smart Kit Copyright is held by the author/owner(s). CHI 2007, April 28–May 3, 2007, San Jose, California, US ACM 978-1-59593-642-4/07/0004. Pei-yu (Peggy) Chi National Taiwan University 1, Sec. 4, Roosevelt Rd., Taipei 106, Taiwan peggychi@csie.org Jen-hao Chen National Taiwan University 1, Sec. 4, Roosevelt Rd., Taipei 106, Taiwan r95922023@ntu.edu.tw Hao-hua Chu National Taiwan University 1, Sec. 4, Roosevelt Rd., Taipei 106, Taiwan hchu@csie.ntu.edu.tw Bing-Yu Chen National Taiwan University 1, Sec. 4, Roosevelt Rd., Taipei 106, Taiwan robin@ntu.edu.tw LCD display smart cabinet figure 1. Smart Kitchen promotes healthy cooking awareness and cooking interaction to the cook. smart stove smart counter CHI 2006 · Work-in-ProgressCHI 2007 • Work-in-Progress To simu on a hu the food time. W the coun an over observe testing o food ing RFID tag After th element rule eng activitie Food ing This inv from a s placed o holding up of a figure 5 nutrition overall in For every container in the system, the interface shows weight information about food ingredients the container has. figure 4. Dialog window for asking input the name of new food item in the system. CHI 2006 · Work-in-ProgressCHI 2007 • Work-in-Progress
- 15. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Arts“situated” cooking support panavi system provides a new user experience design by situated suggestions; real-time sensing and giving feedback the combination of the pan and the interactive recipe application enables users to challenge professional culinary arts by enjoying daily cooking in domestic kitchens NATURAL INPUTS eyeCOOK is designed specifically to use natural input modalities: those that humans use in human to human, non mediated communication [5]. To reduce the need for users to provide explicit input, or change their behavior to accommodate interface constraints, implicitly provided attentional cues are observed and interpreted. We believe that this approach improves the learnability and intuitiveness of interfaces designed for novice users. Voice Commands eyeCOOK uses context-sensitive, localized grammars. This allows more synonyms for a given speech recognition command, reducing the chance of misinterpreting a word. Eye Gaze Commands her/his task. To achieve this, we must increase sensing capability [3], improve coordination among appliances [5], and give appliances the ability to affect the environment [3,5]. Environmental Sensors Temperature sensors used to keep track of the status of the oven and the elements of the stove can increase the system’s ability to guide the user’s cooking experience and could be synchronized with electronic timers. Appliance Information Integration Integrating knowledge of the environment can result in improved functionality, taking up less of the user’s time and effort. For example, user recipe preferences, timing constraints, as determined by the user’s electronic schedule, and currently available ingredients, communicated by food storage areas, can be combined to suggest recipes. As well, selecting a recipe can result in the addition of necessary ingredients to an electronic shopping list stored on the user’s Personal Data Assistant (PDA). Active Environmental Actions The kitchen should not only be aware of its environment, but it should also be able to affect it. Thus, it should be able to take actions which increase efficiency, and reduce the user’s action load, like automatically preheating an oven. CONCLUSIONS We have presented eyeCOOK, a gaze and speech enabled multimodal Attentive User Interface. We have also presented our vision of an Attentive Kitchen in which appliances, informed by sensors, coordinate their behavior, and have the capability to affect the environment. This can reduce the user’s workload, and permit rationalizing requests for user attention. Figure 1. eyeCOOK in Page Display Mode Interactive & Student Posters: Computers Everywhere CHI 2003: NEW HORIZONSPosters: Computers Everywhere CHI 2003: NEW HORIZONS explore different configurations of sensor-dependent display behavior significantly helps in the exploration and crafting of design ideas. Figure 4 shows a prototype under development in which the ambient displays (on the wall above the main worktop) respond to the turning of the pages of a specially designed cookbook. The cookbook has an RFID tag embedded in each page allowing the kitchen to detect the page that the book is currently open at. Responding to this, the ambient displays present relevant food information and even personal media related to the recipe and past times in the cook’s life when the corresponding meal was prepared. 4.2 A design tool for users Another significant challenge in designing pervasive computing applications is the involvement with users in the design process. We have conducted a number participatory design exercises involving older users and found that a significant barrier to exploring design concepts is adequately explaining the scope of the technologies involved. By demonstrating simple mappings between sensors and display in demonstration applications in the Ambient Kitchen we have found that we can greatly improve lay users' understanding of the potential functionality. For example, figure 5 shows one such commonly used application in which sample recipes are projected in response to the ingredients placed on the bench. Traffic light indicators on the display also show which of the recipes ingredients are in the kitchen’s cupboards. Though a simple demonstration of how sensor data can be mapped to information sources (and then to information displays) our experience is that such illustrations can help users think about both more mundane and adventurous (and useful) applications of such “invisible” technologies. Figure 5. The Ambient Kitchen has been used to facilitate your different people. For an discussions and focus groups on the topic of pervasive computing as part of a wider participatory design process looking at ICT and nutrition for older people. 4.3 An observatory to collect sensor data Realizing situated services that are responsive to both actions and intentions requires significant development of activity recognition algorithms themselves. As such, multi-sensor benchmarks of everyday activities are not widely available, and as part of our own research, and to support the research of collaborators and the wider research community, we are developing a number of such benchmarks by capturing data for multiple subjects and activities, and hand annotating these datasets. Activities range from gaze data for head pose tracking algorithm development, primarily using video streams alone (see figure 6), to naturalistic data sets relating to multi-step food preparation for which RFID, accelerometer, pressure and video data is collected and hand annotated. 4.4 An evaluation test bed Evaluation means different things to engineer the question is "does it work?" That is, are the functional requirements met, does it complete certain tests accurately and without failing. For the human factors engineer the question is "does it perform a useful function in the context that it is intended to be used". The latter question can only really be answered by installing the technology in a range of real contexts. In the case of kitchen technologies, this means real lived-in homes. Constraints such as household routines and the different uses different members of a family use different rooms for at different times can be critical in the success or failure of home technologies. These constraints are only really apparent in the context of real home use. Laboratory-based facilities such as the ambient kitchen thus are of limited use in this respect. However it is possible to use them to do more limited evaluations of functional requirements that still have face validity. Figure 6. Simultaneously captured data from the embedde y evaluations of this kind but we are taken. Thus the teabags might be in a container on work surface, d cameras allows us to develop a benchmark for attention detection algorithms based on tracking head pose and position from multiple viewpoints. We yet have to complete an planning to do so. For example, we will use actors trained using video recordings of people with dementia carrying out simple kitchen tasks such as making a hot drink. These recordings were made of people doing these tasks, which were of their own choosing, in their own kitchens. The intention is to make initial tests of algorithms to detect when these people need prompting because they have made an error or stopped in the middle of the task. It would be disorienting and therefore unrealistic to bring people with dementia into the lab but these existing videos can be used to configure the Ambient Kitchen to match the kitchen in the video and then to get an actor to perform the sequence of actions Figure 1. The Ambient Kitchen is a lab-based high fidelity pervasive computing prototyping environment. The kitchen is situated in the main research space in Culture Lab, Newcastle University and uses modified IKEA units and standard laminate flooring installed within a wooden structure (see top figures). Significant care was taken that the underlying technology is not apparent to the people using the kitchen – even the wall projection is achieved using “up-lighter” style projection onto mirrors below the overhead cabinets. Figure 2. Sample off-the-shelf and custom technologies integrated in the Ambient Kitchen: (a) 4 DLP projectors for situated displays; (b) 4 Micaz zigbee motes and sensor boards for object motion sensing; (c) 200 x custom capacitive sensors for floor pressure measurement; (d) 8 Feig long range RFID readers (and sample tag). Figure 3. Using an RFID tagged control object the state of all the Ambient Kitchen sensors can be examined on the main display: floor pressure map (left); accelerometers (center- top); RFID (center-bottom); and video feeds (right). 4. CASE STUDIES The Ambient Kitchen has been developed to support a range of research activities around the problem of providing situated support for people with dementia, and situated services associated with food planning, preparation and cooking. As a high fidelity prototyping environment it allows us to support these activities in a number of different ways, as an experimental space for designers, for explaining new technologies to users, for collecting sensor data in benchmark development for activity recognition algorithms, and for the evaluation of complete solutions in a naturalistic setting. Figure 4. A current design scenario in which the pages of a cookbook have integrated RFID tags. The workbench can detect the current page and adapts the ambient display according the page’s contents. 4.1 A design tool for designers Developing design ideas for pervasive computing applications usually requires a significant effort on the part of designers to imagine what interacting in a fully instrumented environment might be like. In our kitchen scenario, there are no keyboards, mice or conventional input devices, and the ability to physically Figure 1. The Ambient Kitchen is a lab-based high fidelity pervasive computing prototyping environment. The kitchen is situated in the main research space in Culture Lab, Newcastle University and uses modified IKEA units and standard laminate flooring installed within a wooden structure (see top figures). Significant care was taken that the underlying technology is not apparent to the people using the kitchen – even the wall projection is achieved using “up-lighter” style projection onto mirrors below the overhead cabinets. Figure 2. Sample off-the-shelf and custom technologies integrated in the Ambient Kitchen: (a) 4 DLP projectors for situated displays; (b) 4 Micaz zigbee motes and sensor boards for object motion sensing; (c) 200 x custom capacitive sensors for floor pressure measurement; (d) 8 Feig long range RFID readers (and sample tag). Figure 3. Using an RFID tagged control object the state of all the Ambient Kitchen sensors can be examined on the main display: floor pressure map (left); accelerometers (center- top); RFID (center-bottom); and video feeds (right). 4. CASE STUDIES The Ambient Kitchen has been developed to support a range of research activities around the problem of providing situated support for people with dementia, and situated services associated with food planning, preparation and cooking. As a high fidelity prototyping environment it allows us to support these activities in a number of different ways, as an experimental space for designers, for explaining new technologies to users, for collecting sensor data in benchmark development for activity recognition algorithms, and for the evaluation of complete solutions in a naturalistic setting. Figure 4. A current design scenario in which the pages of a cookbook have integrated RFID tags. The workbench can detect the current page and adapts the ambient display according the page’s contents. 4.1 A design tool for designers Developing design ideas for pervasive computing applications usually requires a significant effort on the part of designers to imagine what interacting in a fully instrumented environment might be like. In our kitchen scenario, there are no keyboards, mice or conventional input devices, and the ability to physically explore different configurations of sensor-dependent display behavior significantly helps in the exploration and crafting of design ideas. Figure 4 shows a prototype under development in which the ambient displays (on the wall above the main worktop) respond to the turning of the pages of a specially designed cookbook. The cookbook has an RFID tag embedded in each page allowing the kitchen to detect the page that the book is currently open at. Responding to this, the ambient displays present relevant food information and even personal media related to the recipe and past times in the cook’s life when the corresponding meal was prepared. 4.2 A design tool for users Another significant challenge in designing pervasive computing applications is the involvement with users in the design process. We have conducted a number participatory design exercises involving older users and found that a significant barrier to exploring design concepts is adequately explaining the scope of the technologies involved. By demonstrating simple mappings between sensors and display in demonstration applications in the Ambient Kitchen we have found that we can greatly improve lay users' understanding of the potential functionality. For example, figure 5 shows one such commonly used application in which sample recipes are projected in response to the ingredients placed on the bench. Traffic light indicators on the display also show which of the recipes ingredients are in the kitchen’s cupboards. Though a simple demonstration of how sensor data can be mapped to information sources (and then to information displays) our experience is that such illustrations can help users think about both more mundane and adventurous (and useful) applications of such “invisible” technologies. Figure 5. The Ambient Kitchen has been used to facilitate your different people. For an discussions and focus groups on the topic of pervasive computing as part of a wider participatory design process looking at ICT and nutrition for older people. 4.3 An observatory to collect sensor data Realizing situated services that are responsive to both actions and intentions requires significant development of activity recognition algorithms themselves. As such, multi-sensor benchmarks of everyday activities are not widely available, and as part of our own research, and to support the research of collaborators and the wider research community, we are developing a number of such benchmarks by capturing data for multiple subjects and activities, and hand annotating these datasets. Activities range from gaze data for head pose tracking algorithm development, primarily using video streams alone (see figure 6), to naturalistic data sets relating to multi-step food preparation for which RFID, accelerometer, pressure and video data is collected and hand annotated. 4.4 An evaluation test bed Evaluation means different things to engineer the question is "does it work?" That is, are the functional requirements met, does it complete certain tests accurately and without failing. For the human factors engineer the question is "does it perform a useful function in the context that it is intended to be used". The latter question can only really be answered by installing the technology in a range of real contexts. In the case of kitchen technologies, this means real lived-in homes. Constraints such as household routines and the different uses different members of a family use different rooms for at different times can be critical in the success or failure of home technologies. These constraints are only really apparent in the context of real home use. Laboratory-based facilities such as the ambient kitchen thus are of limited use in this respect. However it is possible to use them to do more limited evaluations of functional requirements that still have face validity. Figure 6. Simultaneously captured data from the embedde y evaluations of this kind but we are taken. Thus the teabags might be in a container on work surface, d cameras allows us to develop a benchmark for attention detection algorithms based on tracking head pose and position from multiple viewpoints. We yet have to complete an planning to do so. For example, we will use actors trained using video recordings of people with dementia carrying out simple kitchen tasks such as making a hot drink. These recordings were made of people doing these tasks, which were of their own choosing, in their own kitchens. The intention is to make initial tests of algorithms to detect when these people need prompting because they have made an error or stopped in the middle of the task. It would be disorienting and therefore unrealistic to bring people with dementia into the lab but these existing videos can be used to configure the Ambient Kitchen to match the kitchen in the video and then to get an actor to perform the sequence of actions Enabling N Smart Kitch Copyright is held by the author/owner(s). CHI 2007, April 28–May 3, 2007, San Jose, California, USA. ACM 978-1-59593-642-4/07/0004. Pei-yu (Peggy) Chi National Taiwan University 1, Sec. 4, Roosevelt Rd., Taipei 106, Taiwan peggychi@csie.org Jen-hao Chen National Taiwan University 1, Sec. 4, Roosevelt Rd., Taipei 106, Taiwan r95922023@ntu.edu.tw Hao-hua Chu National Taiwan University 1, Sec. 4, Roosevelt Rd., Taipei 106, Taiwan hchu@csie.ntu.edu.tw Bing-Yu Chen National Taiwan University 1, Sec. 4, Roosevelt Rd., Taipei 106, Taiwan robin@ntu.edu.tw LCD display smart cabinet figure 1. Smart Kitchen promotes healthy cooking awareness and cooking interaction to the cook. smart stove smart counter CHI 2006 · Work-in-ProgressCHI 2007 • Work-in-Progress To simula on a hum the food time. Wh the count an overhe observer testing ot food ingr RFID tags After the elements rule engin activities Food ingr This invo from a st placed on holding th up of a se figure 4. Dialog window for asking input the name of new food item in the system. CHI 2006 · Work-in-ProgressCHI 2007 • Work-in-Progress interactive cookbook situated coaching system kitchen providing the number of calories previous “situated” cooking support systems
- 16.
- 17. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary ArtsCarbonara: initial menu We chose a recipe of Italian pasta Roman- styled Carbonara as an initial menu. this recipe requires several ways of cooking including sensitive temperature control, which is difficult to master. We quoted a recipe of Tsutomu Ochiai a famous Japanese chef of Italian cuisines who sometimes appears in TV.
- 18. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artsiterative design process sketching experiment: cooking sketching preliminary user study prototyping user study
- 19. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artsiterative design process
- 20. Tinkering: Sketching acircuit sensing temperature
- 21. Tinkering: attaching acircuit embedded with functions of wireless communication and temperature sensing, to an iron made fry-ing pan
- 22. Tinkering: attaching acircuit embedded with functions of wireless communication and temperature sensing, to an iron made fry-ing pan
- 23. Tinkering: A thermocouplesensor attached with the pan using Aluminum tape
- 24. Tinkering: Cooking withmonitoring the temperature
- 25. Tinkering: Cooking withmonitoring the temperature
- 26. Tinkering: Cooking withmonitoring the temperature
- 27. Tinkering: Trials anderrors where to attach the sensor （But it burns toride TOXI）
- 28. Tinkering: Repeating acycle of making a prototype and cooking with it, once a week iteratively during 90days.
- 29. Tinkering: testing designsof user interface, while hardware sketching
- 30. Tinkering: Repeating acycle of making a prototype and cooking with it, once a week iteratively during 90days.
- 31. Tinkering: Repeating acycle of making a prototype and cooking with it, once a week iteratively during 90days.
- 32. Tinkering: Sketching acase for a circuit using the laser cutter after the board design is fixed
- 33. Tinkering: Sketching acase for a circuit using the laser cutter after the board design is fixed
- 34. Tinkering: Considering asystem of projecting the temperature to the fry-ing pan for vision problems
- 35. Tinkering: Considering asystem of projecting the temperature to the fry-ing pan for vision problems
- 36. User Study: Actuallyhaving 2 persons use the prototype
- 37. User Study: Actuallyhaving 2 persons use the prototype
- 38. User Study: Understandingthe need of navigation
- 39. User Study: Needto understand the skill and cognition difference between individuals
- 40. Prototyping: Preparing tocreate Firm Prototype
- 41. Prototyping: Using AluminumMolding
- 42. Prototyping: Using AluminumMolding
- 43. Prototyping: Casting amold to seal sensor
- 44. Prototyping: Creating Circuitusing Manufacturer
- 45. Prototyping: Creating Circuitusing Manufacturer
- 46. Prototyping: Cutting Fry-ingpan Handle using MODELA
- 47. Prototyping: Complete Prototype (whenseparated into pieces)
- 48.
- 49. Exhibition: JST/CREST UbiquitousContents Showcase
- 50. Exhibition: JST/CREST UbiquitousContents Showcase
- 51. Exhibition: JST/CREST UbiquitousContents Showcase
- 52.
- 53. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artsarchitecture overview the touch monitor and the projection system, and the software system showing cooking sequence information (Figure 6). panavi display Computer (panavi OS + Original Cooking Sequence) Electronic Circuit (MOXA-B) Electronic Circuit (MOXA-A) Actuators Xbee Wireless Communication Sensors Thermocouple Sensor Acceleration Sensor Vibration Motor LEDs USB Cable (Serial Communication)VGA Cable Mirror Projector Speaker Projection Touch Monitor Special Pan Figure 6. System Architecture
- 54. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artsdisplay Figure 5. Three Kinds of Setting Displayed in the Monitor; (a) Summary, (b) Detail, (c) Condition / Name of each Area [See, (b)]; (b-1) Main Panel, (b-2) Current Step Panel and Checkbox, (b-3) Comment Panel, (b-4) Temperature Panel, (b-5) Timer Panel elapsed time, and the timer of boiling the pasta, which is also a supplemental function to Condition mode. IMPLEMENTATION & ARCHITECTURE panavi system mainly consists of three parts; the special the projector. Actuators, LEDs and vibration motors are embedded in the handle, controlled by the signals from the computer system via MOXA-B. Display and Computer There are three kinds of setting displayed in the monitor (1) Main Panel (2) Current Step Panel and Checkbox (3) Comment Panel (4) Temperature Panel (5) Timer Panel Summary Detail Condition Users can use any setting anytime.
- 55. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artsoriginal cooking sequence Step 1. Pasta (1) - Boil Water Step 2. Making Egg Sauce Step 3. Pasta (2) - Start Cooking Pasta Step 4. Pasta (3) - Start Timer Step 5. Pancetta (1) - Fry Pancetta [A] Step 6. Pancetta (2) Add Wine Step 7. Pancetta (3) - Seasoning Step 8. Pancetta (4) - Cooling [B] Step 9. Pasta (4) - Drain Water Step 10. Finish (1) - Add Pasta to Pan Step 11. Finish (2) - Dress with Egg Sauce Step 12. Finish (3) - Heat Egg Sauce [C] Step 13. Serve The original cooking sequence models the recipe of Carbonara. Each step is programmed with settings; temperature, sounds, and vibrations etc. LED lamps indicate temperature conditions and a vibration motor moves when the users’ temperature control is good, which are embedded within the handle of the pan. The instructional graphics by the projection can also be checked on the display monitor to support the recognition of the projection when the pan is removed from the stove or when the projection cannot be recognized since there is much ingredient on the pan. If part of the system fails, other parts could still help the user (i.e. if the projector fails, the display and the LED indication work). See, Table 1. The temperature color changes from white to blue, green, yellow, and red, depending on the value, which also synchronizes with the LEDs. White is the default color or indicates when the condition temperature is too low, blue indicates 5-10 degrees lower than the proper temperature setting, green indicates when it is at the proper temperature within 5 degrees, yellow indicates 5-10 degrees higher, and red indicates over 15 - 0 54 - 64 65 - 74 75 - 84 85 - ON— the second hand of a clock ON warning tone — — Color TEMP （°C） Vibration Sound - 69 70 - 79 80 - 89 90 - 99 100 - B - 149 150 - 159 160 - 169 170 - 179 180 -A C Blue Green Yellow RedWhite ON the second hand of a clock the second hand of a clock Table 1. Parameters to Cook the Carvonara Programed in the System. Its procedure is divided into 13 steps.
- 56.
- 57. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artsuser study focused on not only general or common findings but also each user’s originality; their respective backgrounds, and experiences about cooking. We observed how the system effects the users experience while cooking. User A User B User C & D
- 58. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artsmethod 1. prior interview 2. cooking with panavi 3. posterior interview The authors interviewed each user about his/her cooking experience. Firstly, we introduced how to use this system. Next, we showed what is the perfect Carbonara by the photo. During the cooking, we did not help the users except troubles (e.g. system errors). We interviewed the users with watching a video of his/her cooking. They were required to explain intentions and impressions about each action, activity, and process.
- 59. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artssetting selected, because experts or professional chefs are able to cook without the system. Utensils rubber spatula, ladle, tongs, tablespoon, folk Ingredients olive oil, white wine, pancetta, eggs, pasta, salt, black pepper Wet washcloth Wet washcloth Bowls DisplayDisplay PotPot PanPan Table Camera 1 Camera 2 User Camera 3 Plate Figure 7. Layout of the User Study T (F e e is q n h r o c th M T fi e c s
- 60. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Arts User A User A (Study 1) UserA: 23 years old male He has an experience of living on his own for a brief time, but lives with his parents now. Although he has tried to do basic cookings, he could not make well. As a result, he does not do any cooking recently. not enough experience in cooking does not do any cooking recently
- 61. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Arts User B User B (Study 2) User B: 24 years old female She lives with her parents and gives some help to their cooking. They have interests in food and cooking, and hold their own home garden. Raised in such a family environment, she has basic knowledge of cooking, but has not had a chance to challenge authentic menus. has basic knowledge of cooking does simple cooking once or twice a week.
- 62. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary ArtsUser C&D (Study 3) User C: 24 years old male User C lives with his parents and brothers.He has various cooking experiences; as a Boy Scout member and in part-time jobs at a restaurant. He does cooking (mainly breakfast) 3 or 4 times a week. User D: 22 years old female User D lives on her own for 5 years and has a habit of cooking. She mastered the recipe book for beginners her mother gave her, and see web sites to expand her cooking repertoire. basic knowledge of cooking User D UserC various knowledge of cooking cooking 3-4 times a week. prefers to expand her cooking repertoire
- 63. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary ArtsResult (Study 1: User A) He sometimes could not understand texts of instructions. As a result, he spent a lot of time to finish cooking. His pasta was too boiled and soft... At the final stage, he could not stop heating even after the sauce was thick and creamy. His sauce was slightly baked compared to the perfect example. “I could not image the completion, because I had never eaten this menu.” Even he faced many difficulties, finally got a fine dish!
- 64. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary ArtsResult (Study 2: User B) She enjoyed the process like playing a game, and completed a delicious dish except it was slightly scorched. “I am very satisfied to make a delicious dish. I have felt the process was easier than I had imagined.” She easily finished the first 5 steps, just following the instructions without any difficulties. After that, she precisely checked instruction texts and carefully processed the further steps. “At first, I could not understand the meaning of the task on this step. It was difficult so I tried carefully, different from the previous steps.” - Step 6 She enjoyed cooking with panavi. “What an easy cooking!”
- 65. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary ArtsResult (Study 3: User C&D) User D mainly cooked with reading the recipe texts many times. While, User C frequently checked the proper & current temperature and supported her activities. They did not check recipe texts or videos so much, since they relied on their experience. But, in fact, they said “We’d forgotten about watching the video.” Therefore, their cooking was very quickly finished, but their Carbonara was slightly sloppy (not heated well). Both User B&C were disappointed with the result, and said “I want to cook it again using the system.” They quickly finished cooking on their collaborative works.
- 66. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary ArtsReflections This study revealed the difference between the understandings of instructions for each. User A User B User C & D finished in 32min. finished in 18min. finished in 15.5min. The users had distinctive prior knowledge and skills about cooking. - normal cooking time: 20 min.-
- 67. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary ArtsReflections This system could take care each user’s situated actions, even for the beginner user: User A. User A sometimes could not understand texts of instructions. He could keep the proper temperature and did not failed at the last stage. He did not know how to prepare pasta...
- 68. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary ArtsReflections But, at some points, the system could not respond to the users’ situated actions, because the instructions by texts and vide tutorials are fixed and not changeable. User C&D sometimes skipped instructions. User A: “I could not judge between the important steps and the omissible ones, because it was too much information for me to understand and follow.”
- 69. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary ArtsReflections The current system cannot navigate the timing of judging when the heating should be stopped. All users could keep good temperature, but nobody made the perfect one. One of User A was slightly baked. One of User B was was slightly scorched One of User C&D was was slightly sloppy
- 70.
- 71. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artssmart daily commodities This research provided a model for designing daily commodities (e.g. kitchen utensils) as smart objects enriching our everyday life. Figure 5. Three Kinds of Setting Displayed in the Monitor; (a) Summary, (b) Detail, (c) Condition / Name of each Area [See, (b)]; (b-1) Main Panel, (b-2) Current Step Panel and Checkbox, (b-3) Comment Panel, (b-4) Temperature Panel, (b-5) Timer Panel elapsed time, and the timer of boiling the pasta, which is also a supplemental function to Condition mode. IMPLEMENTATION & ARCHITECTURE panavi system mainly consists of three parts; the special kitchen utensil (frying-pan) with embedded sensors and actuators, the display system connecting the computer with the touch monitor and the projection system, and the software system showing cooking sequence information (Figure 6). panavi display Computer (panavi OS + Original Cooking Sequence) Electronic Circuit (MOXA-B) Electronic Circuit (MOXA-A) Actuators Xbee Wireless Communication Sensors Thermocouple Sensor Acceleration Sensor Vibration Motor LEDs USB Cable (Serial Communication)VGA Cable Mirror Projector Speaker Projection Touch Monitor Special Pan Figure 6. System Architecture the projector. Actuators, LEDs and vibration motors are embedded in the handle, controlled by the signals from the computer system via MOXA-B. Display and Computer The computer is connected with ‘panavi display’ packaging touch panel monitor and projector, and ‘panavi OS’ with the ‘Original Cooking Sequence’ works as an Adobe Flash application on the computer (as shown in Figure 6). The panavi OS displays the instructions by analyzing the sensors’ degrees against parameters programed in the system. Original Cooking Sequence The original cooking sequence models the recipe of Carbonara, consisting of videos and photos in addition to the general text recipes. For the panavi OS, this cooking sequence was reconstructed by the development team and its procedure is divided into 13 steps (as shown in Table 2). Each step is programmed with settings; temperature, sounds, and vibrations settings etc. (as shown in Table 1). Preparing Pasta (Table 2, Step 1-4) When the checkbox of the Step 1 is touched, the elapse time counter from the beginning (See, Figure 5-5) starts. The normal cooking time is set to 20 minutes. When Step 4 is checked, the pasta timer technical framework methodology this framework senses and analyze users’ state, and generating feedback suitable for each user’s situations. this integrated technical elements should be designed through the prototyping process in real domestic environments. ×
- 72. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artssmart daily commodities The design of panavi enables the users to concentrate in cooking without noticing if it is a computing device. It seems the same as normal consumer products. This design model is needless of any massive equipment to construct the system.
- 73. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artsfuture visions We would like to measure and model chefs’ cooking ways and apply these data to navigation, and produce various kinds of menu working on our future prototype.
- 74. CHI 2012 “EATING+ COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artsfuture visions We will proceed to develop & improve this system to be more suitable for real domestic contexts, which encourage people’s daily cooking to become more enjoyable.
- 75. “Interactivity” Please come toour demo booth! We will introduce panavi by real cooking show. MON 18:00-20:00 TUE 16:00-19:00 WED 13:00-14:30 Today’s Reception Highlight on Interactivity Interactivity Encore Maybe and so on...
- 76.
- 77.