Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Panavi CHI2012 Presentation

1,501 views

Published on

This slide was used at CHI2012 Conference (http://dl.acm.org/citation.cfm?id=2207695). Paper "panavi: recipe medium with a sensors-embedded pan for domestic users to master professional culinary arts"is here http://panavi.jp/panavi_CHI2012.pdf. http://panavi.jp

Published in: Design
  • Be the first to comment

  • Be the first to like this

Panavi CHI2012 Presentation

  1. 1. Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Arts Daisuke Uriu Mizuki Namai Satoru Tokuhisa Ryo Kashiwagi Masahiko Inami Naohito Okude
  2. 2. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artscooking/recipe medium Julia Child “The French Chef” (1963-1973) - the transition of recipe media from “text” to “video” - Her cooking show, translating traditional print or verbal recipe, was a great hit with domestic audiences in the US.
  3. 3. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artscooking/recipe medium Cooking or recipe medium has entered a transitional period with regards to HCI. Silver Spoon text based recipe book The French Chef TV show Vita Craft IHIQ automatically cooking system product Digital Thermometer Frying Pan product Personal Trainer: Cooking by NINTENDO
  4. 4. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artsconcept: title of this paper panavi: Recipe Medium with a Sensors- Embedded Pan for Domestic Users to Master Professional Culinary Arts
  5. 5. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artsconcept: design overview panavi: Recipe Medium with a Sensors- Embedded Pan for Domestic Users to Master Professional Culinary Arts display: digital recipe application sensors- embedded frying pan wirelessly connected with the display programmed the ways of professional chef’s cooking
  6. 6. video
  7. 7. research contribution
  8. 8. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artsuncertain environment This research attempts to obtain ways of designing computer mediated artifacts that support uncertain domestic environments: kitchens. firewater uncertain domestic situated actions; all activities in kitchens based on activities in kitchens cannot be planned as a static model. users must manage uncertain troubles
  9. 9. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Arts Introduction and Motivation Cooking advanced recipes can be complicated, and requires following detailed instructions to achieve the desired result. Textual instructions like paper recipes are often difficult to comprehend; they use technical terms or instructions that require prior experience to be understood (“How much is a handful of chopped basil?” or “How brown are caramelized onions?”). These types of information can be transferred more easily using images or videos, just as cooking shows do. However, cooking shows are not designed to be followed in real-time. Although the increasing audience of cooking shows in European television [4] indicates that there is a growing interest in this format, no participant of our preliminary studies followed a recipe while watching a cooking show. They rather liked the inspiring aspect of the visual aesthetics. Typically, they would follow textual instructions when cooking, or not use any kind of instructions at all. The final outcome of a recipe depends on many factors and can fail at any point during the entire process. This can be demanding and stressful for beginners, perhaps discouraging them. We are developing PersonalChef, an interactive kitchen counter (see Fig.1), to avoid disappointing hobby chefs when preparing unfamiliar recipes and to increase their confidence. Related Work Several systems have been developed to bring technical innovations into the domestic kitchen environment. Cooking Navi [7] considers cooking as a time optimization problem when preparing a menu consisting of multiple recipes, and tries to optimize single cooking processes in order to have all dishes finished at the right time. Semantic Cookbook [10] and Kitchen of the Future [11] installed various electronic devices into the kitchen to record and share cooking sessions. Living Cookbook [12] offers these possibilities as well, but it further focuses on the social experience of cooking and collaboration. eyeCOOK [3] and Smart Kitchen [8] focus on the actual user input. They try to facilitate active user input or reduce it by tracking user’s actions in the kitchen using eye-gaze, speech commands, and foot switches. CounterActive [9] integrates step-by-step cooking instructions using multimedia invisibly into the kitchen counter. Besides research prototypes, there are also commercial systems. Nintendo’s personal cooking instructor (nintendo.com) on the small-screen NintendoDS device is an interactive cookbook and provides live cooking demonstrations. While other work often focused on efficiency [1,5], our system aims to support the entertaining and social gress (Spotlight on Posters Days 1 & 2) April 12–13, 2010, Atlanta, GA, USA 3404 Figure 1. HeatSink illuminates the stream of water according to its temperature, becoming red when hot and blue when cold. SeeSink One impediment to automatic faucets in bathroom and kitchen sinks is the lack of control over temperature and flow. Nevertheless, their simple application of automation Figure 2. SeeSink can interpret a variety of tasks being performed by the user to provide useful hands-free control of water temperature and flow. Unfortunately these systems do not directly prevent non- compliance (which is estimated at 50% in hospitals, for example [30]). Dirty hands are the primary cause for infection, and certainly very easy to prevent [3]. CleanSink seeks to motivate critical behavior change by augmenting the role of the sink as part of the larger context CHI 2005 PAPERS: Technology in the Home April 2–7 Portland, Oregon, USA research through design This paper follows “Research through Design.” How to evaluates our work could contribute to the HCI community. inventionprocess relevance extensibility
  10. 10. related studies
  11. 11. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artstraditional text based recipes Traditional media for recipes describe steps toward a perfect completion of cooking. Silver Spoon very famous recipe book SPAGHETTI ALLA CARBONARA SPAGHETTI WITH CARBONARA SAUCE Ingredient for 1 person Preparation time: 15' (preparation: 10' cooking: 5') 14 oz (100 g) spaghetti 1.5 oz (45 g) pancetta 1 egg 1 egg yolk Method Prepare egg sauce, beating the egg and the egg yolk with Pecorino cheese and a little pepper in a bowl. Cut the pancetta into small strips. Place a large skillet onto a medium heat and slowly saute the pancetta. Cook the spaghetti in a pot of lightly salted boiling water and drain when al dente. Turn the pasta into the skillet with the pancetta, toss and turn off the heat. Add the egg sauce and a little of the cooking water. Stir for 30 seconds or so with a little heat. Turn off the heat, stir again, and serve immediately. 1 oz (100 g) Pecorino cheese 1 tsp (5 ml) extra virgin olive oil Salt Pepper But it is usually difficult for non-professional users to reproduce the taste of meals.
  12. 12. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Arts“planned” recipe medium The problem of “planned” traditional recipe is close to Lucy Suchman’s “situated actions.” SPAGHETTI ALLA CARBONARA SPAGHETTI WITH CARBONARA SAUCE Ingredient for 1 person Preparation time: 15' (preparation: 10' cooking: 5') 14 oz (100 g) spaghetti 1.5 oz (45 g) pancetta 1 egg 1 egg yolk Method Prepare egg sauce, beating the egg and the egg yolk with Pecorino cheese and a little pepper in a bowl. Cut the pancetta into small strips. Place a large skillet onto a medium heat and slowly saute the pancetta. Cook the spaghetti in a pot of lightly salted boiling water and drain when al dente. Turn the pasta into the skillet with the pancetta, toss and turn off the heat. Add the egg sauce and a little of the cooking water. Stir for 30 seconds or so with a little heat. Turn off the heat, stir again, and serve immediately. 1 oz (100 g) Pecorino cheese 1 tsp (5 ml) extra virgin olive oil Salt Pepper She revealed the programmed instructions do not support situated actions: the errors and problems users actually face are not expected by the system. the “big green button” came from her research. ethnography research about copy machine “planned” text recipe
  13. 13. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Arts“situated” cooking support Cooking support systems corresponding to users’ situated actions have been already considered. Bradbury et al. an interactive cookbook supporting situated actions using eye-tracking and speech recognition NATURAL INPUTS eyeCOOK is designed specifically to use natural input modalities: those that humans use in human to human, non mediated communication [5]. To reduce the need for users to provide explicit input, or change their behavior to accommodate interface constraints, implicitly provided attentional cues are observed and interpreted. We believe that this approach improves the learnability and intuitiveness of interfaces designed for novice users. Voice Commands eyeCOOK uses context-sensitive, localized grammars. This allows more synonyms for a given speech recognition command, reducing the chance of misinterpreting a word. Eye Gaze Commands When the user is in range of the eye tracker, eyeCOOK substitutes the object of the user’s gaze for the word ‘this’ in a speech command. For example, ‘Define this’ will trigger the define operation on the current eye gaze target. Since current eye trackers are spatially fixed and offer limited mobility to users, the user will not always be in a location where eye tracker input is available. Our speech grammar is designed so that system functionality is not her/his task. To achieve this, we must increase sensing capability [3], improve coordination among appliances [5], and give appliances the ability to affect the environment [3,5]. Environmental Sensors Temperature sensors used to keep track of the status of the oven and the elements of the stove can increase the system’s ability to guide the user’s cooking experience and could be synchronized with electronic timers. Appliance Information Integration Integrating knowledge of the environment can result in improved functionality, taking up less of the user’s time and effort. For example, user recipe preferences, timing constraints, as determined by the user’s electronic schedule, and currently available ingredients, communicated by food storage areas, can be combined to suggest recipes. As well, selecting a recipe can result in the addition of necessary ingredients to an electronic shopping list stored on the user’s Personal Data Assistant (PDA). Active Environmental Actions The kitchen should not only be aware of its environment, but it should also be able to affect it. Thus, it should be able to take actions which increase efficiency, and reduce the user’s action load, like automatically preheating an oven. CONCLUSIONS We have presented eyeCOOK, a gaze and speech enabled multimodal Attentive User Interface. We have also presented our vision of an Attentive Kitchen in which appliances, informed by sensors, coordinate their behavior, and have the capability to affect the environment. This can reduce the user’s workload, and permit rationalizing requests for user attention. REFERENCES 1. Ju, W. et al. (2001). CounterActive: An Interactive Cookbook for the Kitchen Counter. Extended Abstracts of CHI 2001 (Seattle, April 2001) pp. 269-270 2. Norman, D. A. The Invisible Computer, MIT press, 1999 3. Schmidt, A. et al. How to Build Smart Appliances. IEEE Personal Communications 8(4), August 2001. pp. 66-71. Figure 1. eyeCOOK in Page Display Mode Interactive & Student Posters: Computers Everywhere CHI 2003: NEW HORIZONSPosters: Computers Everywhere CHI 2003: NEW HORIZONS Olivier et al. situated coaching system by integrating projection systems, RFID, accelerometers, and under-floor pressure sensing technologies. explore different configurations of sensor-dependent display behavior significantly helps in the exploration and crafting of design ideas. Figure 4 shows a prototype under development in which the ambient displays (on the wall above the main worktop) respond to the turning of the pages of a specially designed cookbook. The cookbook has an RFID tag embedded in each page allowing the kitchen to detect the page that the book is currently open at. Responding to this, the ambient displays present relevant food information and even personal media related to the recipe and past times in the cook’s life when the corresponding meal was prepared. 4.2 A design tool for users Another significant challenge in designing pervasive computing applications is the involvement with users in the design process. We have conducted a number participatory design exercises involving older users and found that a significant barrier to exploring design concepts is adequately explaining the scope of the technologies involved. By demonstrating simple mappings between sensors and display in demonstration applications in the Ambient Kitchen we have found that we can greatly improve lay users' understanding of the potential functionality. For example, figure 5 shows one such commonly used application in which sample recipes are projected in response to the ingredients placed on the bench. Traffic light indicators on the display also show which of the recipes ingredients are in the kitchen’s cupboards. Though a simple demonstration of how sensor data can be mapped to information sources (and then to information displays) our experience is that such illustrations can help users think about both more mundane and adventurous (and useful) applications of such “invisible” technologies. Figure 5. The Ambient Kitchen has been used to facilitate your different people. For an discussions and focus groups on the topic of pervasive computing as part of a wider participatory design process looking at ICT and nutrition for older people. 4.3 An observatory to collect sensor data Realizing situated services that are responsive to both actions and intentions requires significant development of activity recognition algorithms themselves. As such, multi-sensor benchmarks of everyday activities are not widely available, and as part of our own research, and to support the research of collaborators and the wider research community, we are developing a number of such benchmarks by capturing data for multiple subjects and activities, and hand annotating these datasets. Activities range from gaze data for head pose tracking algorithm development, primarily using video streams alone (see figure 6), to naturalistic data sets relating to multi-step food preparation for which RFID, accelerometer, pressure and video data is collected and hand annotated. 4.4 An evaluation test bed Evaluation means different things to engineer the question is "does it work?" That is, are the functional requirements met, does it complete certain tests accurately and without failing. For the human factors engineer the question is "does it perform a useful function in the context that it is intended to be used". The latter question can only really be answered by installing the technology in a range of real contexts. In the case of kitchen technologies, this means real lived-in homes. Constraints such as household routines and the different uses different members of a family use different rooms for at different times can be critical in the success or failure of home technologies. These constraints are only really apparent in the context of real home use. Laboratory-based facilities such as the ambient kitchen thus are of limited use in this respect. However it is possible to use them to do more limited evaluations of functional requirements that still have face validity. Figure 6. Simultaneously captured data from the embedde y evaluations of this kind but we are taken. Thus the teabags might be in a container on work surface, d cameras allows us to develop a benchmark for attention detection algorithms based on tracking head pose and position from multiple viewpoints. We yet have to complete an planning to do so. For example, we will use actors trained using video recordings of people with dementia carrying out simple kitchen tasks such as making a hot drink. These recordings were made of people doing these tasks, which were of their own choosing, in their own kitchens. The intention is to make initial tests of algorithms to detect when these people need prompting because they have made an error or stopped in the middle of the task. It would be disorienting and therefore unrealistic to bring people with dementia into the lab but these existing videos can be used to configure the Ambient Kitchen to match the kitchen in the video and then to get an actor to perform the sequence of actions Figure 1. The Ambient Kitchen is a lab-based high fidelity pervasive computing prototyping environment. The kitchen is situated in the main research space in Culture Lab, Newcastle University and uses modified IKEA units and standard laminate flooring installed within a wooden structure (see top figures). Significant care was taken that the underlying technology is not apparent to the people using the kitchen – even the wall projection is achieved using “up-lighter” style projection onto mirrors below the overhead cabinets. Figure 2. Sample off-the-shelf and custom technologies integrated in the Ambient Kitchen: (a) 4 DLP projectors for situated displays; (b) 4 Micaz zigbee motes and sensor boards for object motion sensing; (c) 200 x custom capacitive sensors for floor pressure measurement; (d) 8 Feig long range RFID readers (and sample tag). Figure 3. Using an RFID tagged control object the state of all the Ambient Kitchen sensors can be examined on the main display: floor pressure map (left); accelerometers (center- top); RFID (center-bottom); and video feeds (right). 4. CASE STUDIES The Ambient Kitchen has been developed to support a range of research activities around the problem of providing situated support for people with dementia, and situated services associated with food planning, preparation and cooking. As a high fidelity prototyping environment it allows us to support these activities in a number of different ways, as an experimental space for designers, for explaining new technologies to users, for collecting sensor data in benchmark development for activity recognition algorithms, and for the evaluation of complete solutions in a naturalistic setting. Figure 4. A current design scenario in which the pages of a cookbook have integrated RFID tags. The workbench can detect the current page and adapts the ambient display according the page’s contents. 4.1 A design tool for designers Developing design ideas for pervasive computing applications usually requires a significant effort on the part of designers to imagine what interacting in a fully instrumented environment might be like. In our kitchen scenario, there are no keyboards, mice or conventional input devices, and the ability to physically Figure 1. The Ambient Kitchen is a lab-based high fidelity pervasive computing prototyping environment. The kitchen is situated in the main research space in Culture Lab, Newcastle University and uses modified IKEA units and standard laminate flooring installed within a wooden structure (see top figures). Significant care was taken that the underlying technology is not apparent to the people using the kitchen – even the wall projection is achieved using “up-lighter” style projection onto mirrors below the overhead cabinets. Figure 2. Sample off-the-shelf and custom technologies integrated in the Ambient Kitchen: (a) 4 DLP projectors for situated displays; (b) 4 Micaz zigbee motes and sensor boards for object motion sensing; (c) 200 x custom capacitive sensors for floor pressure measurement; (d) 8 Feig long range RFID readers (and sample tag). Figure 3. Using an RFID tagged control object the state of all the Ambient Kitchen sensors can be examined on the main display: floor pressure map (left); accelerometers (center- top); RFID (center-bottom); and video feeds (right). 4. CASE STUDIES The Ambient Kitchen has been developed to support a range of research activities around the problem of providing situated support for people with dementia, and situated services associated with food planning, preparation and cooking. As a high fidelity prototyping environment it allows us to support these activities in a number of different ways, as an experimental space for designers, for explaining new technologies to users, for collecting sensor data in benchmark development for activity recognition algorithms, and for the evaluation of complete solutions in a naturalistic setting. Figure 4. A current design scenario in which the pages of a cookbook have integrated RFID tags. The workbench can detect the current page and adapts the ambient display according the page’s contents. 4.1 A design tool for designers Developing design ideas for pervasive computing applications usually requires a significant effort on the part of designers to imagine what interacting in a fully instrumented environment might be like. In our kitchen scenario, there are no keyboards, mice or conventional input devices, and the ability to physically explore different configurations of sensor-dependent display behavior significantly helps in the exploration and crafting of design ideas. Figure 4 shows a prototype under development in which the ambient displays (on the wall above the main worktop) respond to the turning of the pages of a specially designed cookbook. The cookbook has an RFID tag embedded in each page allowing the kitchen to detect the page that the book is currently open at. Responding to this, the ambient displays present relevant food information and even personal media related to the recipe and past times in the cook’s life when the corresponding meal was prepared. 4.2 A design tool for users Another significant challenge in designing pervasive computing applications is the involvement with users in the design process. We have conducted a number participatory design exercises involving older users and found that a significant barrier to exploring design concepts is adequately explaining the scope of the technologies involved. By demonstrating simple mappings between sensors and display in demonstration applications in the Ambient Kitchen we have found that we can greatly improve lay users' understanding of the potential functionality. For example, figure 5 shows one such commonly used application in which sample recipes are projected in response to the ingredients placed on the bench. Traffic light indicators on the display also show which of the recipes ingredients are in the kitchen’s cupboards. Though a simple demonstration of how sensor data can be mapped to information sources (and then to information displays) our experience is that such illustrations can help users think about both more mundane and adventurous (and useful) applications of such “invisible” technologies. Figure 5. The Ambient Kitchen has been used to facilitate your different people. For an discussions and focus groups on the topic of pervasive computing as part of a wider participatory design process looking at ICT and nutrition for older people. 4.3 An observatory to collect sensor data Realizing situated services that are responsive to both actions and intentions requires significant development of activity recognition algorithms themselves. As such, multi-sensor benchmarks of everyday activities are not widely available, and as part of our own research, and to support the research of collaborators and the wider research community, we are developing a number of such benchmarks by capturing data for multiple subjects and activities, and hand annotating these datasets. Activities range from gaze data for head pose tracking algorithm development, primarily using video streams alone (see figure 6), to naturalistic data sets relating to multi-step food preparation for which RFID, accelerometer, pressure and video data is collected and hand annotated. 4.4 An evaluation test bed Evaluation means different things to engineer the question is "does it work?" That is, are the functional requirements met, does it complete certain tests accurately and without failing. For the human factors engineer the question is "does it perform a useful function in the context that it is intended to be used". The latter question can only really be answered by installing the technology in a range of real contexts. In the case of kitchen technologies, this means real lived-in homes. Constraints such as household routines and the different uses different members of a family use different rooms for at different times can be critical in the success or failure of home technologies. These constraints are only really apparent in the context of real home use. Laboratory-based facilities such as the ambient kitchen thus are of limited use in this respect. However it is possible to use them to do more limited evaluations of functional requirements that still have face validity. Figure 6. Simultaneously captured data from the embedde y evaluations of this kind but we are taken. Thus the teabags might be in a container on work surface, d cameras allows us to develop a benchmark for attention detection algorithms based on tracking head pose and position from multiple viewpoints. We yet have to complete an planning to do so. For example, we will use actors trained using video recordings of people with dementia carrying out simple kitchen tasks such as making a hot drink. These recordings were made of people doing these tasks, which were of their own choosing, in their own kitchens. The intention is to make initial tests of algorithms to detect when these people need prompting because they have made an error or stopped in the middle of the task. It would be disorienting and therefore unrealistic to bring people with dementia into the lab but these existing videos can be used to configure the Ambient Kitchen to match the kitchen in the video and then to get an actor to perform the sequence of actions Chi et al. a kitchen providing real- time feedback, tracking the number of calories in food ingredients. Enabling Smart Kit Copyright is held by the author/owner(s). CHI 2007, April 28–May 3, 2007, San Jose, California, US ACM 978-1-59593-642-4/07/0004. Pei-yu (Peggy) Chi National Taiwan University 1, Sec. 4, Roosevelt Rd., Taipei 106, Taiwan peggychi@csie.org Jen-hao Chen National Taiwan University 1, Sec. 4, Roosevelt Rd., Taipei 106, Taiwan r95922023@ntu.edu.tw Hao-hua Chu National Taiwan University 1, Sec. 4, Roosevelt Rd., Taipei 106, Taiwan hchu@csie.ntu.edu.tw Bing-Yu Chen National Taiwan University 1, Sec. 4, Roosevelt Rd., Taipei 106, Taiwan robin@ntu.edu.tw LCD display smart cabinet figure 1. Smart Kitchen promotes healthy cooking awareness and cooking interaction to the cook. smart stove smart counter CHI 2006 · Work-in-ProgressCHI 2007 • Work-in-Progress To simu on a hu the food time. W the coun an over observe testing o food ing RFID tag After th element rule eng activitie Food ing This inv from a s placed o holding up of a figure 5 nutrition overall in For every container in the system, the interface shows weight information about food ingredients the container has. figure 4. Dialog window for asking input the name of new food item in the system. CHI 2006 · Work-in-ProgressCHI 2007 • Work-in-Progress
  14. 14. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Arts“situated” cooking support panavi system provides a new user experience design by situated suggestions; real-time sensing and giving feedback the combination of the pan and the interactive recipe application enables users to challenge professional culinary arts by enjoying daily cooking in domestic kitchens NATURAL INPUTS eyeCOOK is designed specifically to use natural input modalities: those that humans use in human to human, non mediated communication [5]. To reduce the need for users to provide explicit input, or change their behavior to accommodate interface constraints, implicitly provided attentional cues are observed and interpreted. We believe that this approach improves the learnability and intuitiveness of interfaces designed for novice users. Voice Commands eyeCOOK uses context-sensitive, localized grammars. This allows more synonyms for a given speech recognition command, reducing the chance of misinterpreting a word. Eye Gaze Commands her/his task. To achieve this, we must increase sensing capability [3], improve coordination among appliances [5], and give appliances the ability to affect the environment [3,5]. Environmental Sensors Temperature sensors used to keep track of the status of the oven and the elements of the stove can increase the system’s ability to guide the user’s cooking experience and could be synchronized with electronic timers. Appliance Information Integration Integrating knowledge of the environment can result in improved functionality, taking up less of the user’s time and effort. For example, user recipe preferences, timing constraints, as determined by the user’s electronic schedule, and currently available ingredients, communicated by food storage areas, can be combined to suggest recipes. As well, selecting a recipe can result in the addition of necessary ingredients to an electronic shopping list stored on the user’s Personal Data Assistant (PDA). Active Environmental Actions The kitchen should not only be aware of its environment, but it should also be able to affect it. Thus, it should be able to take actions which increase efficiency, and reduce the user’s action load, like automatically preheating an oven. CONCLUSIONS We have presented eyeCOOK, a gaze and speech enabled multimodal Attentive User Interface. We have also presented our vision of an Attentive Kitchen in which appliances, informed by sensors, coordinate their behavior, and have the capability to affect the environment. This can reduce the user’s workload, and permit rationalizing requests for user attention. Figure 1. eyeCOOK in Page Display Mode Interactive & Student Posters: Computers Everywhere CHI 2003: NEW HORIZONSPosters: Computers Everywhere CHI 2003: NEW HORIZONS explore different configurations of sensor-dependent display behavior significantly helps in the exploration and crafting of design ideas. Figure 4 shows a prototype under development in which the ambient displays (on the wall above the main worktop) respond to the turning of the pages of a specially designed cookbook. The cookbook has an RFID tag embedded in each page allowing the kitchen to detect the page that the book is currently open at. Responding to this, the ambient displays present relevant food information and even personal media related to the recipe and past times in the cook’s life when the corresponding meal was prepared. 4.2 A design tool for users Another significant challenge in designing pervasive computing applications is the involvement with users in the design process. We have conducted a number participatory design exercises involving older users and found that a significant barrier to exploring design concepts is adequately explaining the scope of the technologies involved. By demonstrating simple mappings between sensors and display in demonstration applications in the Ambient Kitchen we have found that we can greatly improve lay users' understanding of the potential functionality. For example, figure 5 shows one such commonly used application in which sample recipes are projected in response to the ingredients placed on the bench. Traffic light indicators on the display also show which of the recipes ingredients are in the kitchen’s cupboards. Though a simple demonstration of how sensor data can be mapped to information sources (and then to information displays) our experience is that such illustrations can help users think about both more mundane and adventurous (and useful) applications of such “invisible” technologies. Figure 5. The Ambient Kitchen has been used to facilitate your different people. For an discussions and focus groups on the topic of pervasive computing as part of a wider participatory design process looking at ICT and nutrition for older people. 4.3 An observatory to collect sensor data Realizing situated services that are responsive to both actions and intentions requires significant development of activity recognition algorithms themselves. As such, multi-sensor benchmarks of everyday activities are not widely available, and as part of our own research, and to support the research of collaborators and the wider research community, we are developing a number of such benchmarks by capturing data for multiple subjects and activities, and hand annotating these datasets. Activities range from gaze data for head pose tracking algorithm development, primarily using video streams alone (see figure 6), to naturalistic data sets relating to multi-step food preparation for which RFID, accelerometer, pressure and video data is collected and hand annotated. 4.4 An evaluation test bed Evaluation means different things to engineer the question is "does it work?" That is, are the functional requirements met, does it complete certain tests accurately and without failing. For the human factors engineer the question is "does it perform a useful function in the context that it is intended to be used". The latter question can only really be answered by installing the technology in a range of real contexts. In the case of kitchen technologies, this means real lived-in homes. Constraints such as household routines and the different uses different members of a family use different rooms for at different times can be critical in the success or failure of home technologies. These constraints are only really apparent in the context of real home use. Laboratory-based facilities such as the ambient kitchen thus are of limited use in this respect. However it is possible to use them to do more limited evaluations of functional requirements that still have face validity. Figure 6. Simultaneously captured data from the embedde y evaluations of this kind but we are taken. Thus the teabags might be in a container on work surface, d cameras allows us to develop a benchmark for attention detection algorithms based on tracking head pose and position from multiple viewpoints. We yet have to complete an planning to do so. For example, we will use actors trained using video recordings of people with dementia carrying out simple kitchen tasks such as making a hot drink. These recordings were made of people doing these tasks, which were of their own choosing, in their own kitchens. The intention is to make initial tests of algorithms to detect when these people need prompting because they have made an error or stopped in the middle of the task. It would be disorienting and therefore unrealistic to bring people with dementia into the lab but these existing videos can be used to configure the Ambient Kitchen to match the kitchen in the video and then to get an actor to perform the sequence of actions Figure 1. The Ambient Kitchen is a lab-based high fidelity pervasive computing prototyping environment. The kitchen is situated in the main research space in Culture Lab, Newcastle University and uses modified IKEA units and standard laminate flooring installed within a wooden structure (see top figures). Significant care was taken that the underlying technology is not apparent to the people using the kitchen – even the wall projection is achieved using “up-lighter” style projection onto mirrors below the overhead cabinets. Figure 2. Sample off-the-shelf and custom technologies integrated in the Ambient Kitchen: (a) 4 DLP projectors for situated displays; (b) 4 Micaz zigbee motes and sensor boards for object motion sensing; (c) 200 x custom capacitive sensors for floor pressure measurement; (d) 8 Feig long range RFID readers (and sample tag). Figure 3. Using an RFID tagged control object the state of all the Ambient Kitchen sensors can be examined on the main display: floor pressure map (left); accelerometers (center- top); RFID (center-bottom); and video feeds (right). 4. CASE STUDIES The Ambient Kitchen has been developed to support a range of research activities around the problem of providing situated support for people with dementia, and situated services associated with food planning, preparation and cooking. As a high fidelity prototyping environment it allows us to support these activities in a number of different ways, as an experimental space for designers, for explaining new technologies to users, for collecting sensor data in benchmark development for activity recognition algorithms, and for the evaluation of complete solutions in a naturalistic setting. Figure 4. A current design scenario in which the pages of a cookbook have integrated RFID tags. The workbench can detect the current page and adapts the ambient display according the page’s contents. 4.1 A design tool for designers Developing design ideas for pervasive computing applications usually requires a significant effort on the part of designers to imagine what interacting in a fully instrumented environment might be like. In our kitchen scenario, there are no keyboards, mice or conventional input devices, and the ability to physically Figure 1. The Ambient Kitchen is a lab-based high fidelity pervasive computing prototyping environment. The kitchen is situated in the main research space in Culture Lab, Newcastle University and uses modified IKEA units and standard laminate flooring installed within a wooden structure (see top figures). Significant care was taken that the underlying technology is not apparent to the people using the kitchen – even the wall projection is achieved using “up-lighter” style projection onto mirrors below the overhead cabinets. Figure 2. Sample off-the-shelf and custom technologies integrated in the Ambient Kitchen: (a) 4 DLP projectors for situated displays; (b) 4 Micaz zigbee motes and sensor boards for object motion sensing; (c) 200 x custom capacitive sensors for floor pressure measurement; (d) 8 Feig long range RFID readers (and sample tag). Figure 3. Using an RFID tagged control object the state of all the Ambient Kitchen sensors can be examined on the main display: floor pressure map (left); accelerometers (center- top); RFID (center-bottom); and video feeds (right). 4. CASE STUDIES The Ambient Kitchen has been developed to support a range of research activities around the problem of providing situated support for people with dementia, and situated services associated with food planning, preparation and cooking. As a high fidelity prototyping environment it allows us to support these activities in a number of different ways, as an experimental space for designers, for explaining new technologies to users, for collecting sensor data in benchmark development for activity recognition algorithms, and for the evaluation of complete solutions in a naturalistic setting. Figure 4. A current design scenario in which the pages of a cookbook have integrated RFID tags. The workbench can detect the current page and adapts the ambient display according the page’s contents. 4.1 A design tool for designers Developing design ideas for pervasive computing applications usually requires a significant effort on the part of designers to imagine what interacting in a fully instrumented environment might be like. In our kitchen scenario, there are no keyboards, mice or conventional input devices, and the ability to physically explore different configurations of sensor-dependent display behavior significantly helps in the exploration and crafting of design ideas. Figure 4 shows a prototype under development in which the ambient displays (on the wall above the main worktop) respond to the turning of the pages of a specially designed cookbook. The cookbook has an RFID tag embedded in each page allowing the kitchen to detect the page that the book is currently open at. Responding to this, the ambient displays present relevant food information and even personal media related to the recipe and past times in the cook’s life when the corresponding meal was prepared. 4.2 A design tool for users Another significant challenge in designing pervasive computing applications is the involvement with users in the design process. We have conducted a number participatory design exercises involving older users and found that a significant barrier to exploring design concepts is adequately explaining the scope of the technologies involved. By demonstrating simple mappings between sensors and display in demonstration applications in the Ambient Kitchen we have found that we can greatly improve lay users' understanding of the potential functionality. For example, figure 5 shows one such commonly used application in which sample recipes are projected in response to the ingredients placed on the bench. Traffic light indicators on the display also show which of the recipes ingredients are in the kitchen’s cupboards. Though a simple demonstration of how sensor data can be mapped to information sources (and then to information displays) our experience is that such illustrations can help users think about both more mundane and adventurous (and useful) applications of such “invisible” technologies. Figure 5. The Ambient Kitchen has been used to facilitate your different people. For an discussions and focus groups on the topic of pervasive computing as part of a wider participatory design process looking at ICT and nutrition for older people. 4.3 An observatory to collect sensor data Realizing situated services that are responsive to both actions and intentions requires significant development of activity recognition algorithms themselves. As such, multi-sensor benchmarks of everyday activities are not widely available, and as part of our own research, and to support the research of collaborators and the wider research community, we are developing a number of such benchmarks by capturing data for multiple subjects and activities, and hand annotating these datasets. Activities range from gaze data for head pose tracking algorithm development, primarily using video streams alone (see figure 6), to naturalistic data sets relating to multi-step food preparation for which RFID, accelerometer, pressure and video data is collected and hand annotated. 4.4 An evaluation test bed Evaluation means different things to engineer the question is "does it work?" That is, are the functional requirements met, does it complete certain tests accurately and without failing. For the human factors engineer the question is "does it perform a useful function in the context that it is intended to be used". The latter question can only really be answered by installing the technology in a range of real contexts. In the case of kitchen technologies, this means real lived-in homes. Constraints such as household routines and the different uses different members of a family use different rooms for at different times can be critical in the success or failure of home technologies. These constraints are only really apparent in the context of real home use. Laboratory-based facilities such as the ambient kitchen thus are of limited use in this respect. However it is possible to use them to do more limited evaluations of functional requirements that still have face validity. Figure 6. Simultaneously captured data from the embedde y evaluations of this kind but we are taken. Thus the teabags might be in a container on work surface, d cameras allows us to develop a benchmark for attention detection algorithms based on tracking head pose and position from multiple viewpoints. We yet have to complete an planning to do so. For example, we will use actors trained using video recordings of people with dementia carrying out simple kitchen tasks such as making a hot drink. These recordings were made of people doing these tasks, which were of their own choosing, in their own kitchens. The intention is to make initial tests of algorithms to detect when these people need prompting because they have made an error or stopped in the middle of the task. It would be disorienting and therefore unrealistic to bring people with dementia into the lab but these existing videos can be used to configure the Ambient Kitchen to match the kitchen in the video and then to get an actor to perform the sequence of actions Enabling N Smart Kitch Copyright is held by the author/owner(s). CHI 2007, April 28–May 3, 2007, San Jose, California, USA. ACM 978-1-59593-642-4/07/0004. Pei-yu (Peggy) Chi National Taiwan University 1, Sec. 4, Roosevelt Rd., Taipei 106, Taiwan peggychi@csie.org Jen-hao Chen National Taiwan University 1, Sec. 4, Roosevelt Rd., Taipei 106, Taiwan r95922023@ntu.edu.tw Hao-hua Chu National Taiwan University 1, Sec. 4, Roosevelt Rd., Taipei 106, Taiwan hchu@csie.ntu.edu.tw Bing-Yu Chen National Taiwan University 1, Sec. 4, Roosevelt Rd., Taipei 106, Taiwan robin@ntu.edu.tw LCD display smart cabinet figure 1. Smart Kitchen promotes healthy cooking awareness and cooking interaction to the cook. smart stove smart counter CHI 2006 · Work-in-ProgressCHI 2007 • Work-in-Progress To simula on a hum the food time. Wh the count an overhe observer testing ot food ingr RFID tags After the elements rule engin activities Food ingr This invo from a st placed on holding th up of a se figure 4. Dialog window for asking input the name of new food item in the system. CHI 2006 · Work-in-ProgressCHI 2007 • Work-in-Progress interactive cookbook situated coaching system kitchen providing the number of calories previous “situated” cooking support systems
  15. 15. design process
  16. 16. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary ArtsCarbonara: initial menu We chose a recipe of Italian pasta Roman- styled Carbonara as an initial menu. this recipe requires several ways of cooking including sensitive temperature control, which is difficult to master. We quoted a recipe of Tsutomu Ochiai a famous Japanese chef of Italian cuisines who sometimes appears in TV.
  17. 17. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artsiterative design process sketching experiment: cooking sketching preliminary user study prototyping user study
  18. 18. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artsiterative design process
  19. 19. Tinkering: Sketching a circuit sensing temperature
  20. 20. Tinkering: attaching a circuit embedded with functions of wireless communication and temperature sensing, to an iron made fry-ing pan
  21. 21. Tinkering: attaching a circuit embedded with functions of wireless communication and temperature sensing, to an iron made fry-ing pan
  22. 22. Tinkering: A thermocouple sensor attached with the pan using Aluminum tape
  23. 23. Tinkering: Cooking with monitoring the temperature
  24. 24. Tinkering: Cooking with monitoring the temperature
  25. 25. Tinkering: Cooking with monitoring the temperature
  26. 26. Tinkering: Trials and errors where to attach the sensor (But it burns toride TOXI)
  27. 27. Tinkering: Repeating a cycle of making a prototype and cooking with it, once a week iteratively during 90days.
  28. 28. Tinkering: testing designs of user interface, while hardware sketching
  29. 29. Tinkering: Repeating a cycle of making a prototype and cooking with it, once a week iteratively during 90days.
  30. 30. Tinkering: Repeating a cycle of making a prototype and cooking with it, once a week iteratively during 90days.
  31. 31. Tinkering: Sketching a case for a circuit using the laser cutter after the board design is fixed
  32. 32. Tinkering: Sketching a case for a circuit using the laser cutter after the board design is fixed
  33. 33. Tinkering: Considering a system of projecting the temperature to the fry-ing pan for vision problems
  34. 34. Tinkering: Considering a system of projecting the temperature to the fry-ing pan for vision problems
  35. 35. User Study: Actually having 2 persons use the prototype
  36. 36. User Study: Actually having 2 persons use the prototype
  37. 37. User Study: Understanding the need of navigation
  38. 38. User Study: Need to understand the skill and cognition difference between individuals
  39. 39. Prototyping: Preparing to create Firm Prototype
  40. 40. Prototyping: Using Aluminum Molding
  41. 41. Prototyping: Using Aluminum Molding
  42. 42. Prototyping: Casting a mold to seal sensor
  43. 43. Prototyping: Creating Circuit using Manufacturer
  44. 44. Prototyping: Creating Circuit using Manufacturer
  45. 45. Prototyping: Cutting Fry-ing pan Handle using MODELA
  46. 46. Prototyping: Complete Prototype (when separated into pieces)
  47. 47. Prototyping: Completed Prototype
  48. 48. Exhibition: JST/CREST Ubiquitous Contents Showcase
  49. 49. Exhibition: JST/CREST Ubiquitous Contents Showcase
  50. 50. Exhibition: JST/CREST Ubiquitous Contents Showcase
  51. 51. design
  52. 52. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artsarchitecture overview the touch monitor and the projection system, and the software system showing cooking sequence information (Figure 6). panavi display Computer (panavi OS + Original Cooking Sequence) Electronic Circuit (MOXA-B) Electronic Circuit (MOXA-A) Actuators Xbee Wireless Communication Sensors Thermocouple Sensor Acceleration Sensor Vibration Motor LEDs USB Cable (Serial Communication)VGA Cable Mirror Projector Speaker Projection Touch Monitor Special Pan Figure 6. System Architecture
  53. 53. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artsdisplay Figure 5. Three Kinds of Setting Displayed in the Monitor; (a) Summary, (b) Detail, (c) Condition / Name of each Area [See, (b)]; (b-1) Main Panel, (b-2) Current Step Panel and Checkbox, (b-3) Comment Panel, (b-4) Temperature Panel, (b-5) Timer Panel elapsed time, and the timer of boiling the pasta, which is also a supplemental function to Condition mode. IMPLEMENTATION & ARCHITECTURE panavi system mainly consists of three parts; the special the projector. Actuators, LEDs and vibration motors are embedded in the handle, controlled by the signals from the computer system via MOXA-B. Display and Computer There are three kinds of setting displayed in the monitor (1) Main Panel (2) Current Step Panel and Checkbox (3) Comment Panel (4) Temperature Panel (5) Timer Panel Summary Detail Condition Users can use any setting anytime.
  54. 54. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artsoriginal cooking sequence Step 1. Pasta (1) - Boil Water Step 2. Making Egg Sauce Step 3. Pasta (2) - Start Cooking Pasta Step 4. Pasta (3) - Start Timer Step 5. Pancetta (1) - Fry Pancetta [A] Step 6. Pancetta (2) Add Wine Step 7. Pancetta (3) - Seasoning Step 8. Pancetta (4) - Cooling [B] Step 9. Pasta (4) - Drain Water Step 10. Finish (1) - Add Pasta to Pan Step 11. Finish (2) - Dress with Egg Sauce Step 12. Finish (3) - Heat Egg Sauce [C] Step 13. Serve The original cooking sequence models the recipe of Carbonara. Each step is programmed with settings; temperature, sounds, and vibrations etc. LED lamps indicate temperature conditions and a vibration motor moves when the users’ temperature control is good, which are embedded within the handle of the pan. The instructional graphics by the projection can also be checked on the display monitor to support the recognition of the projection when the pan is removed from the stove or when the projection cannot be recognized since there is much ingredient on the pan. If part of the system fails, other parts could still help the user (i.e. if the projector fails, the display and the LED indication work). See, Table 1. The temperature color changes from white to blue, green, yellow, and red, depending on the value, which also synchronizes with the LEDs. White is the default color or indicates when the condition temperature is too low, blue indicates 5-10 degrees lower than the proper temperature setting, green indicates when it is at the proper temperature within 5 degrees, yellow indicates 5-10 degrees higher, and red indicates over 15 - 0 54 - 64 65 - 74 75 - 84 85 - ON— the second hand of a clock ON warning tone — — Color TEMP (°C) Vibration Sound - 69 70 - 79 80 - 89 90 - 99 100 - B - 149 150 - 159 160 - 169 170 - 179 180 -A C Blue Green Yellow RedWhite ON the second hand of a clock the second hand of a clock Table 1. Parameters to Cook the Carvonara Programed in the System. Its procedure is divided into 13 steps.
  55. 55. user study
  56. 56. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artsuser study focused on not only general or common findings but also each user’s originality; their respective backgrounds, and experiences about cooking. We observed how the system effects the users experience while cooking. User A User B User C & D
  57. 57. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artsmethod 1. prior interview 2. cooking with panavi 3. posterior interview The authors interviewed each user about his/her cooking experience. Firstly, we introduced how to use this system. Next, we showed what is the perfect Carbonara by the photo. During the cooking, we did not help the users except troubles (e.g. system errors). We interviewed the users with watching a video of his/her cooking. They were required to explain intentions and impressions about each action, activity, and process.
  58. 58. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artssetting selected, because experts or professional chefs are able to cook without the system. Utensils rubber spatula, ladle, tongs, tablespoon, folk Ingredients olive oil, white wine, pancetta, eggs, pasta, salt, black pepper Wet washcloth Wet washcloth Bowls DisplayDisplay PotPot PanPan Table Camera 1 Camera 2 User Camera 3 Plate Figure 7. Layout of the User Study T (F e e is q n h r o c th M T fi e c s
  59. 59. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Arts User A User A (Study 1) UserA: 23 years old male He has an experience of living on his own for a brief time, but lives with his parents now. Although he has tried to do basic cookings, he could not make well. As a result, he does not do any cooking recently. not enough experience in cooking does not do any cooking recently
  60. 60. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Arts User B User B (Study 2) User B: 24 years old female She lives with her parents and gives some help to their cooking. They have interests in food and cooking, and hold their own home garden. Raised in such a family environment, she has basic knowledge of cooking, but has not had a chance to challenge authentic menus. has basic knowledge of cooking does simple cooking once or twice a week.
  61. 61. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary ArtsUser C&D (Study 3) User C: 24 years old male User C lives with his parents and brothers.He has various cooking experiences; as a Boy Scout member and in part-time jobs at a restaurant. He does cooking (mainly breakfast) 3 or 4 times a week. User D: 22 years old female User D lives on her own for 5 years and has a habit of cooking. She mastered the recipe book for beginners her mother gave her, and see web sites to expand her cooking repertoire. basic knowledge of cooking User D UserC various knowledge of cooking cooking 3-4 times a week. prefers to expand her cooking repertoire
  62. 62. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary ArtsResult (Study 1: User A) He sometimes could not understand texts of instructions. As a result, he spent a lot of time to finish cooking. His pasta was too boiled and soft... At the final stage, he could not stop heating even after the sauce was thick and creamy. His sauce was slightly baked compared to the perfect example. “I could not image the completion, because I had never eaten this menu.” Even he faced many difficulties, finally got a fine dish!
  63. 63. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary ArtsResult (Study 2: User B) She enjoyed the process like playing a game, and completed a delicious dish except it was slightly scorched. “I am very satisfied to make a delicious dish. I have felt the process was easier than I had imagined.” She easily finished the first 5 steps, just following the instructions without any difficulties. After that, she precisely checked instruction texts and carefully processed the further steps. “At first, I could not understand the meaning of the task on this step. It was difficult so I tried carefully, different from the previous steps.” - Step 6 She enjoyed cooking with panavi. “What an easy cooking!”
  64. 64. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary ArtsResult (Study 3: User C&D) User D mainly cooked with reading the recipe texts many times. While, User C frequently checked the proper & current temperature and supported her activities. They did not check recipe texts or videos so much, since they relied on their experience. But, in fact, they said “We’d forgotten about watching the video.” Therefore, their cooking was very quickly finished, but their Carbonara was slightly sloppy (not heated well). Both User B&C were disappointed with the result, and said “I want to cook it again using the system.” They quickly finished cooking on their collaborative works.
  65. 65. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary ArtsReflections This study revealed the difference between the understandings of instructions for each. User A User B User C & D finished in 32min. finished in 18min. finished in 15.5min. The users had distinctive prior knowledge and skills about cooking. - normal cooking time: 20 min.-
  66. 66. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary ArtsReflections This system could take care each user’s situated actions, even for the beginner user: User A. User A sometimes could not understand texts of instructions. He could keep the proper temperature and did not failed at the last stage. He did not know how to prepare pasta...
  67. 67. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary ArtsReflections But, at some points, the system could not respond to the users’ situated actions, because the instructions by texts and vide tutorials are fixed and not changeable. User C&D sometimes skipped instructions. User A: “I could not judge between the important steps and the omissible ones, because it was too much information for me to understand and follow.”
  68. 68. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary ArtsReflections The current system cannot navigate the timing of judging when the heating should be stopped. All users could keep good temperature, but nobody made the perfect one. One of User A was slightly baked. One of User B was was slightly scorched One of User C&D was was slightly sloppy
  69. 69. conclusion
  70. 70. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artssmart daily commodities This research provided a model for designing daily commodities (e.g. kitchen utensils) as smart objects enriching our everyday life. Figure 5. Three Kinds of Setting Displayed in the Monitor; (a) Summary, (b) Detail, (c) Condition / Name of each Area [See, (b)]; (b-1) Main Panel, (b-2) Current Step Panel and Checkbox, (b-3) Comment Panel, (b-4) Temperature Panel, (b-5) Timer Panel elapsed time, and the timer of boiling the pasta, which is also a supplemental function to Condition mode. IMPLEMENTATION & ARCHITECTURE panavi system mainly consists of three parts; the special kitchen utensil (frying-pan) with embedded sensors and actuators, the display system connecting the computer with the touch monitor and the projection system, and the software system showing cooking sequence information (Figure 6). panavi display Computer (panavi OS + Original Cooking Sequence) Electronic Circuit (MOXA-B) Electronic Circuit (MOXA-A) Actuators Xbee Wireless Communication Sensors Thermocouple Sensor Acceleration Sensor Vibration Motor LEDs USB Cable (Serial Communication)VGA Cable Mirror Projector Speaker Projection Touch Monitor Special Pan Figure 6. System Architecture the projector. Actuators, LEDs and vibration motors are embedded in the handle, controlled by the signals from the computer system via MOXA-B. Display and Computer The computer is connected with ‘panavi display’ packaging touch panel monitor and projector, and ‘panavi OS’ with the ‘Original Cooking Sequence’ works as an Adobe Flash application on the computer (as shown in Figure 6). The panavi OS displays the instructions by analyzing the sensors’ degrees against parameters programed in the system. Original Cooking Sequence The original cooking sequence models the recipe of Carbonara, consisting of videos and photos in addition to the general text recipes. For the panavi OS, this cooking sequence was reconstructed by the development team and its procedure is divided into 13 steps (as shown in Table 2). Each step is programmed with settings; temperature, sounds, and vibrations settings etc. (as shown in Table 1). Preparing Pasta (Table 2, Step 1-4) When the checkbox of the Step 1 is touched, the elapse time counter from the beginning (See, Figure 5-5) starts. The normal cooking time is set to 20 minutes. When Step 4 is checked, the pasta timer technical framework methodology this framework senses and analyze users’ state, and generating feedback suitable for each user’s situations. this integrated technical elements should be designed through the prototyping process in real domestic environments. ×
  71. 71. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artssmart daily commodities The design of panavi enables the users to concentrate in cooking without noticing if it is a computing device. It seems the same as normal consumer products. This design model is needless of any massive equipment to construct the system.
  72. 72. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artsfuture visions We would like to measure and model chefs’ cooking ways and apply these data to navigation, and produce various kinds of menu working on our future prototype.
  73. 73. CHI 2012 “EATING + COOKING” Monday, May 7th, 2012 panavi: Recipe Medium with a Sensors-Embedded Pan for Domestic Users to Master Professional Culinary Artsfuture visions We will proceed to develop & improve this system to be more suitable for real domestic contexts, which encourage people’s daily cooking to become more enjoyable.
  74. 74. “Interactivity” Please come to our demo booth! We will introduce panavi by real cooking show. MON 18:00-20:00 TUE 16:00-19:00 WED 13:00-14:30 Today’s Reception Highlight on Interactivity Interactivity Encore Maybe and so on...
  75. 75. thanks!
  76. 76. http://panavi.jp panavi@ok.kmd.keio.ac.jp

×