Towards a Smart Control Room for Crisis Response
Using Visual Perception of Users
Joris Ijsselmuiden, Florian van de Camp, Alexander Schick, Michael Voit, Rainer Stiefelhagen
{iss, ca, sci, vt, stiefe}@iitb.fraunhofer.de
Fraunhofer IITB, Karlsruhe
INTRODUCTION
Due to ever increasing challenges and complexity, there is a high demand for
new human-machine interaction approaches in crisis response scenarios. We
aim at building a smart crisis response control room, in which vision-based
perception of users will be used to facilitate innovative user interfaces and to
support teamwork. Our control room is equipped with several cameras and has a
videowall as the main interaction device. Using real-time computer vision, we can
track and identify the users in the room, and estimate their head orientations and
pointing gestures. In the near future, the room will also be equiped with speech
recognition. In order to build a useful smart control room for crisis response, we
are currently focusing on situation modeling for such rooms and we are
investigating the target crisis response scenarios.
Person tracking, head pose estimation, and gesture recognition
SYSTEM ARCHITECTURE
• All components run in parallel and in real time
• We use several computers, with multithreading and GPU programming to obtain
sufficient computational power
• Our custom-built middleware takes care of network communication
Our smart control room laboratory, containing videowall and cameras • A centralized situation model (blackboard) is kept, describing the situation in the
room and the objects and users in it
GOALS
• All perceptual components can read and write in this situation model and a logic
• Develop new ways of interacting with computers and support interaction between
engine uses it to deduce higher level facts about the situation [6]
humans using: tracking, identification, head pose, gestures, speech, and
situation/user modeling [1,2] • In the near future, our control room laboratory will be extended using some of the
following: speech recognition, standard workstations, a digital situation table [7],
• Conduct user studies to find multimodal system setups that improve computer
tablet PCs, sound, and synthesized speech
supported cooperative work in a crisis response control room
• Improve expressive power, ease of use, intuitiveness, speed, reliability,
adaptability, and cooperation while reducing physical and mental workload
• Create intelligent, context dependant user interfaces through situation modeling
and user modeling
• Challenges in crisis response control rooms include: team-based operation, limits
to mental workload, high cost of failure, time pressure, dense/complex
information, and the user acceptance problem
PERCEPTION
Example interaction with the videowall and a digital situation table in operation
• Tracking and identification [3]
• Head pose and visual focus of attention [4] This work is supported by the FhG Internal Programs under Grant No. 692 026
• Gestures and bodypose [5] (Fraunhofer Attract). It is a collaboration between the Fraunhofer Institute for
• Speech recognition (future work) Information and Data Processing; Business Unit Interactive Analysis and Diagnosis
and the University of Karlsruhe (TH); Faculty of Computer Science, in the
framework of the five-year Fraunhofer internal project “Computer Vision for Human-
Computer Interaction – Interaction in and with attentive rooms”.
REFERENCES
1. Project Webpage (2009) www.iitb.fraunhofer.de/?20718
2. Stiefelhagen, Bernardin, Ekenel, Voit (2008) Tracking Identities and Attention in
Smart Environments - Contributions and Progress in the CHIL Project IEEE
A camera image and its corresponding segmentation and 3D voxel representation International Conference on Face and Gesture Recognition
3. Bernardin, Van de Camp, Stiefelhagen (2007) Automatic Person Detection and
INTERACTION Tracking using Fuzzy Controlled Active Cameras IEEE International Conference
1. Identities are obtained through face recognition (in operation) on Computer Vision and Pattern Recognition
2. User models are used to generate personal user interfaces, obeying the user’s 4. Voit, Stiefelhagen (2008) Deducing the Visual Focus of Attention from Head Pose
preferences, current tasks, and specialized knowledge (future work) Estimation in Dynamic Multi-view Meeting Scenarios 10th International
3. Using person tracking, interfaces are displayed close to the user (in operation) Conference on Multimodal Interfaces
4. Objects on the videowall are manipulated using pointing gestures and directing 5. Nickel, Stiefelhagen (2007) Visual Recognition of Pointing Gestures for Human-
ones visual attention (in operation) Robot Interaction Image and Vision Computing
5. This can be combined with speech recognition and a range of different hand 6. Brdiczka, Crowley, Curín, Kleindienst (2009) Chapter: Situation Modeling, in
gestures (future work) Waibel, Stiefelhagen (Eds.) Computers in the Human Interaction Loop
6. Head pose is employed to analyze the interaction of the team, for example to 7. Bader, Meissner, Tschnerney (2008) Digital Map Table with Fovea-Tablett®:
determine who has been talking to whom (in operation) Smart Furniture for Emergency Operation Centers 5th International Conference
on Information Systems for Crisis Response and Management
7. User-specific information can be displayed on the videowall, at the user’s current
focus of attention and we can make people aware of what they haven’t seen yet
(future work)