Pervasive 2011 Talk on Situated Glyphs

21,446 views

Published on

Pervasive 2011 Talk, discussing the design space of situated glyphs for demanding work environment and a corresponding micro display network system.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
21,446
On SlideShare
0
From Embeds
0
Number of Embeds
17,919
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Pervasive 2011 Talk on Situated Glyphs

  1. 1. Exploring the Design Space for Situated Glyphs to Support Dynamic Work EnvironmentsFahim Kawsar1,2, Jo Vermeulen3, Kevin Smith2, Kris Luyten3, Gerd Kortuem2 1Bell Labs, 2Lancaster University, 3Hasselt University
  2. 2. Motivation
  3. 3. #3$*MN!6*D&6*5$E$)(8$5*&6*#3$*-"&)*6#"/* "6#%,:$"#6* D$%$* ,6$5* D3)$* >(G$6* D$%$* 5$2%$&6$5* "* $&23* -(%:C* .$#&)$5* #$:* 6,23* &6* #3$* 8%"#$5* -(%:6C* 13$* 5$62%8#("* &"5* 6,//$6#("6* -(%* $%%(%* 3&"5)"/* D$%$* $%$*‣5$6/"$5* -(%* >(#3* /&#3$%"/* Medical Professionals deal with large amounts of dynamic data. 6,88)$5*-(%*#3$*,6$%6*-(%*%$5,2"/*#3$*8(#$"#&)*$%%(%6C*3462&"6* &"5* ",%6$6* D#3(,#* &"4* 7"*$&64*#(*,6$*IJ$)8K*:$",*D&6*8%(E5$5*#(*#3$*,6$%6* !"#$"%&(%)*+,"%-. /0)12/31*/4#456"$/+12(4#+"7*0 . 89:;9<= >< 2#("6* &"5* :(#E&#("6* (-* #3$* ,6$%6* D#3*&*6:8)$*&22$66C*@/,%$*U*63(D6*#3$*6$)$2#$5*MN!6* ‣ Encoding patient information visually?*)$"#*0*,"#54%@"$0@122*"1@045;AB$"0/C#"%0*,1*C1*,"%"$12245*,"/#$/D/$)12/+4#0"65$%&#("*("*#3$*646#$:C** for quick understandability. (-* #3$* :$52&)* "-(%:&#("* 646#$:* #3&#* D$* 5$E$)(8$5* *4C"*,"%*,"#&101*"1@&+4#+"/D"$1#$$"0/C#"$*,"/#$/D/$)12/+4#0/#*410E0*"@FE+4#+"/D/#C $* D&6* $E&),&#$5* &"5* &"&)4F$5* >4* &6*-"&)*8%(#(#48$C*13$*646#$:*)&"/,&/$*D&6*(%/"&))4* 1#$$"0/C#/#C1C2E7,0E0*"@*44%C1#/3"1#$+1*"C4%/3"*,"/#$/D/$)12/+4#0/#*41+4,"%"#*0E0*"@ 1,%H63*>,#*D&6*#%&"6)&#$5*#(*O"/)63C* $"0+%/F"$1F4D"GH#"0)+,0E0*"@&0,4I#F"24IJK/C)%"BL&I10$"0/C#"$FE0*)$"#*0?"1# ‣ Combining $G8$%#6C* NO6* 8%$8&%$5* &* )6#* (-* icons to represent complex medical concepts with quick readability M%"0"#0&N1D/$O%4#"%&P421#?*4D"%1#$Q)R"S44$0G$%*8$%-(%:"/*D#3*#3$*&88)2&#("*"* ! K/C)%"B Medical Information System with Iconic User Interfaces ! The Visualisation of Medical Concepts Visualization System (Salman et al., 2010) ! (Zender et al., 2007) ?*)$"#*$"0/C#"%0T?"1#M%"0"#0&N1D/$O%4#"%&P421#?*4D"%1#$Q)R"S44$0 ! U ! $%012*#3"*IQ&#$"#*J6#(%4K*MN!*(-*#3$*-"&)*8%(5,2#* U * V1#C/F2"HFW"+*0 X0,10F""##4*"$7%"D/4)02E&/+4#0,1D"F""#7%4D"#"55"+*/D"1*%"7%"0"#*/#C*1#C/F2"4FW"+*0&*," 21#C)1C""Y)/D12"#*45#4)#0J!"#$"%&:<<ZLG[#*,"$"D"247@"#*45*,"%"Y)/0/*"/424C/+124% Visual Language for Medical Concepts /4+,"@/+12HFW"+*/+4#0&$"0/C#"%02"D"%1C"$*,"0E0*"@1*/+7%47"#0/*/"0450+/"#*/0*0FE$"0/C#/#C 51@/2/"045*1#C/F2"4FW"+*/+4#0G[+4#51@/2/"0@/@/+*,"71%"#*9+,/2$0*%)+*)%"45*,"]6Q?1#$FE $4/#C04+%"1*"$/+4#0I/*,1^7%4_/@1*"+4#*"_*`*,1*"#1F2"04#"/+4#`0@"1#/#C&*,"71%"#*&*4 143
  4. 4. 4. Verbal concepts are linked with the VLsys’s visualexample of the VLsys representation. The verbal concept identifier appearsDisplay with text on roll-over, like a software ‘tool tip.’explanation. An Iconic Language for the Graphical 5. The VLsys presents all concepts in an interactive Visual Language System for representing Medical display (see Cell 3-4). Representation of Medical Concepts (Lamy et Concepts (Zender et al., 2010) !"#$"%&()*$+,-./0)1(2$),&$3%(2.,$")4,5!"##$%!!&( )**+&,,---./01234536*789.512,:;"<(=:;,$,( al., 2008) Display example from a preliminary VLsys Icons (in circles) represent concepts in neuronal plasticity / Alzheimer’s Disease, organized by ontological structure with the most frequent concepts larger, the less frequent smaller. Connecting the icons are line glyphs, which animate on roll- over, representing semantic relations: regulate, modulate, produce, etc.. Each icon pair may represent oneVisual Language System for Representing Medical Concepts 6/11/10 4 or scores of papers or data sets. Clicking an icon pair brings up a A disorder characterized by recurrent episodes of paroxysmal brain dysfunction due to a sudden, disorderly, and window (not shown here) with all excessive neuronal discharge. the related papers (or other data MeSH tree number C10.228.140.490 sources), with links to sources.In this definition we identified the three key terms as: !"#$"%&()*$+,-./0)1(2$),&$3%(2.,$")4,5!"##$%!!&( Using a fisheye views approach, successive levels of interaction repeated neuronal reveal more concepts related to visualized ones. (electrical) discharges [seizures]. In a pilot study, this display wasFrom this breakdown we created individual icons for head, nervous system, electrical discharge and repetition (see Figure 4). understood both faster and moreWe can machine assembled these icons into an epilepsy icon. Modular machine assembly is important because it enables a accurately than a word-basedlimited number of icons and glyphs to be combined in software to represent a large number of concepts (see Figure 4). display (Telemakus). For identification tasks requiring Figure 1 reasoning or associationExamples of VCM icons created by combining central pictograms, external shapes, colors and top-right pictogramsFigure 4. Modular icon construction. Reading left to right, a human head icon is combined with a circle glyph, which is combined with a brain + brain stem the VCM icon Examples of VCM icons created by combining central pictograms, external shapes, colors and top-right picto-icon to make nervous system, which is combined with an electric glyph to make seizure, which is combined with a repetition glyph (arrows) to construct aggregate mean time for thisEpilepsy (repeating seizures). grams. Simple physiological or pathological states are represented by combining a color that indicates the moment at which display was 18% faster than the Drug mo the patient state occurs, a shape that distinguishes pathological (square) and physiological (circle) states, and a central picto- word-based display, andgram. These icons can be reused for building drug and test icons. for identification of related concepts, ties of th the mean time for this display was twice as fast as the word-based Both methods can be combined in the same icon, e.g. the The external shape is brown for antecedents, red for cur- ning. Th display. square with the downward arrow combined with the heart with an ECG signal indicates bradycardia. rent states and orange for risks. Risks play an important role in the expression of adverse effects of drugs (see icons square a difficult to represent: the disease epilepsy for example. Moreover, in a language as rich, complex and numerous as medicine it (a), (b) and (d) on Figure 1). indicate VCM icons for drugs is not practical to develop a custom-designed icon for every term. Our strategy to overcome this is to develop a modular icon VCM represents the therapeutic classes of drugs using the dose). TMoreover, we foundiconaprimitives that epilepsy is often broken software to represent complex concepts. Through evaluation we found that system of that concept such as can be combined in down into several classifications along well-defined lines. icon of the disease they are indicated for (as given by theFor example, the MeSH description for epilepsy indicates that: words that, when analyzed, can be broken into individual representational medical definitions often consist of several key ATC Anatomical Therapeutical Chemical drug classifica- ifiers to tion [43]). A green top-right pictogram is added to the dis- icons. For example, systems are generally based MeSH as: Epilepsy classification epilepsy is described in upon: (1) clinical features of the seizure episodes (e.g., motor seizure), ease icon to obtain the drug icon (see Figure 4). The top- right pictogram is a cross if the route of administration is property (2) etiology (e.g., post-traumatic), not specified, or a tablet, a syringe or a pomade tube to means d ‣ Used effectively for visualising a patient’s situation, providing instructions, (3) anatomic site of seizure origin (e.g., frontal lobe seizure), (4) tendency to spread to other structures in the brain, and (5) temporal patterns (e.g., nocturnal epilepsy). Figure 2 Examples of VCM disease or sign icons Examples of VCM disease or sign icons. Icon (a) simply indicate drugs with an oral, injectable or topical adminis- tration route, respectively. offering task support. indicates the anatomical and functional location of the dis- (MeSH tree C10.228.140.490 continued) ease. Icon (b) provides additional information about the gen- VCM icons for tests In drug monographs, tests are mostly mentioned for the Building sThis structure offers five categories along which icon representations can be developed. Figure 5 shows an example of a eral pathological mechanism, using a shape modifier, and iconpartial iconic representation of various types of epilepsy based on these categories. (c) provides additional information about the function involved, by modifying the central pictogram. Icon (d) com- follow-up of potential adverse effects of drugs. Thus, VCM represents a test using the icon of the risk it can follow up. One or mFigure 5. Pilot epilepsy vocabulary. The ‘root’ epilepsy icon is modified by icon primitives to represent five forms of epilepsy, from left to right: bines both the shape modifier and the pictogram modifica- tion. A blue top-right pictogram is added to the risk icon to obtain the test icon. The top-right pictogram is a stetho- ical elem ‣generalized, temporal lobe, absence, tonic-clonic, and myoclonic, plus a related concept: infantile spasms. All are used in Desktop Environment, primarily used as a communication scope, a test tube, a ray or a signal, for clinical tests, bio- ing cont or adver medium for informed discussion across medical experts. >8?3!(!1@!" 67)5%$,809%/$,.1$-./$(1)1.,$78/7.2%2: to right ture. The (e.g. the Visual Language for Medical Concepts to threeUsing this approach, iconic ‘words’ form basic vocabularies from which numerous specific icons can be assembled. Theserepeated icons form icon families, which forms a context on which many specific vocabularies can be constructed: a disease Figure 3 Examples of VCM disease icons using various shape modifiers in the sevocabulary, a protein/gene vocabulary, a study or trial vocabulary, etc.. Images clustered together in a shared space or context, such as a circle or square containing shape, are interpreted Examples of VCM disease icons using various shape avoid (etogether. (Arnheim, 1969, p. 55 ff.) By controlling image hierarchy and/or reading sequence the VLsys can cause some imagesto be read as primary and others as secondary. By altering the visual form of the images in the reading sequence or hierarchy, modifiers. In icon (a) a pathological agent entering in the is emptythe reader can be stimulated to understand one icon as the primary concept and another image as a supporting or modifying square from the left represents the etiology, in (b) the cell in the drug
  5. 5. ‣ Focuses primarily on co-ordination, and supporting collaborative discussion, planning, (and supporting nomadic activities to some extent) across medical professionals and patients. ‣ does not address information presentation modality - visualisation aspects. ‣ Mostly confined to one space. Ecological Validity and Pervasiveness 421 Downloaded By: [Favela, Jesus] At: 03:53 20 April 2010 (Bardram et al. 2006, 2009, 2010) (Favela et al., 2010) FIGURE 3 The CHIS system (a): The user uses the handheld to learn of the availability and location of colleagues and services (b) and to send context-aware messages (c). information and services according to user context. For instance, when a physi- cian carrying a PDA approaches a patient, the system can automatically display that patient’s clinical record. A second version of CHIS integrates public displays that personalize the infor- mation shown to the user and offer opportunistic access to clinical information (Favela et al., 2004). For instance, if a physician approaches a public display, it detects her presence and personalizes the system, highlighting in the floor map recent additions to clinical records of her patients, messages and services most relevant to her current work activities (Figure 3a). Information from the mobile device can also be seamlessly moved to the public display and vice versa. For instance, a physician can drag a window in the display over her picture to transfer the information contained in it to her PDA. A third and final version of CHIS incorporates mechanisms that let users manage their privacy based on contextual information (Tentori et al., 2006). For instance, a physician can configure the floor map to show only the area where he is located and his role instead of his precise location and name. There is a significant complexity in evaluating such a system in situ. On one (Scupelli et al., 2010) (Wilcox et al., 2010) hand, the system needs to be integrated with the Hospital Information System and an Electronic Patient Record. In addition, the reliability of the location estima- tion component must be assured (Castro & Favela, 2008). Thus, we opted for an approach in which some of these components can be simulated, in order to reduce the cost and the complexity of the evaluation. Situated Displays in Health Care
  6. 6. Nurse can provide saline to patient one or three. Saline Nurse can support patient Machine UD two with regular morning routine. Patient One PO PO Patient Two PO Patient Three UD: Use Device PO : Perform On Discover Understand Select(Zender et al., 2006, Bardram et al., 2006, 2008. Favela et al., 2010) Act
  7. 7. Research Goal
  8. 8. Activity-aware tiny displays distributed across the environment providing task information Discover Understand Select Act
  9. 9. Situated Glyphs
  10. 10. ‣ Adaptive and multivariate graphical units that provide in­situ task information.‣ Visual representations of human activities that are integrated in the environment and linked to physical entities - like objects and people - and are adaptive and mobile.‣ Situated Glyphs can be realised by micro display networks. Situated Glyphs
  11. 11. WhatInformation Affinity How Representation Fidelity Situated Glyphs Where When Spatial Distribution Temporal Distribution Design Space Situated Glyphs
  12. 12. ‣ Followed Nurses to understand their daily routine, information needs, and current work practices (semi structured Interview) ‣ 20-22 beds distributed over 10 rooms in 4 areas. ‣ 1 Head Nurse, 8-10 Nurses, 4-5 at duty all the time ‣ Typical Stay Time for Patients : 3 weeksFormative Study at the Department of Geriatric Psychiatry of the District HospitalMainkofen, Germany supporting elderly patients suffering from dementia. Formative Study
  13. 13. Activity PatternA: do x x : actionA: do x with y y : device / object z : patientA: do x to zA: do x with y to z Insight -1 ActionActivity Objects (Optional) Activity Pattern
  14. 14. Activity PatternA: do x x : actionA: do x with y y : device / object z : patientA: do x to zA: do x with y to z ActionActivity Objects (Optional) Activity Pattern
  15. 15. What Information Affinity How Representation Fidelity Situated Glyphs Where When Spatial Distribution Temporal Distribution Information AffinityWhat to Present in Glyphs (and to Whom)
  16. 16. Identity Confirmation Relationship Information Instructions Explanation Affinity Status Trends‣ Identity and Relationship - Which equipment to use with Whom. Insight - 2‣ Status - Current operation status (ok, faulty) for equipment, last checkup time for patients, etc.‣ Instructions - Guideline to perform a complex operation in sensitive areas, e.g. Emergency Room, Intensive Care Unit etc.‣ Confirmation - Feedback of task completion, and recorded.‣ Explanation - Why the equipment is not working‣ Trends : Equipment status history, patient medication record, etc. Information Types
  17. 17. Identity Confirmation Relationship Information Instructions Explanation Affinity Status Trends‣ Identity and Relationship - Which equipment to use with Whom.‣ Status - Current operation status (ok, faulty) for equipment, last checkup time for patients, etc.‣ Instructions - Guideline to perform a complex operation in sensitive areas, e.g. Emergency Room, Intensive Care Unit etc.‣ Confirmation - Feedback of task completion, and recorded.‣ Explanation - Why the equipment is not working‣ Trends : Equipment status history, patient medication record, etc. Information Types
  18. 18. What Information Affinity How Representation Fidelity Situated Glyphs Where When Spatial Distribution Temporal DistributionRepresentation Fidelity How to Present Glyphs
  19. 19. ‣ Symbolic | Iconic | Indexical (Semiotics Signs) Symbolic Signs Iconic Signs (Static | Animated) Symbolic Colour Coded Signs Iconic Signs (Static | Animated) (a) Symbolic (b) Iconic Direct Images Direct Images with Text Colour Coded Text and Number (c) Indexical Learning from Semiotics
  20. 20. Representation Fidelity Symbolic Iconic Indexical (abstract) (metaphoric) (direct) Identity & Yes Yes Yes Relationship (With Colour, Text (With Colour) (With Multiple Icons)Information Affinity and Number) Yes Yes Yes Status (With Colour) Instruction Insight - 3 No Yes Yes Yes Confirmation Yes Yes (With Colour Code) Explanation No No Yes Trends No Yes Yes (With Animation) Information Vs Fidelity [with Semiotics]
  21. 21. Representation Fidelity Symbolic Iconic Indexical (abstract) (metaphoric) (direct) Identity & Yes Yes Yes Relationship (With Colour, Text (With Colour) (With Multiple Icons)Information Affinity and Number) Yes Yes Yes Status (With Colour) Instruction No Yes Yes Yes Confirmation Yes Yes (With Colour Code) Explanation No No Yes Trends No Yes Yes (With Animation) Information Vs Fidelity [with Semiotics]
  22. 22. Colour Representing ‣ Situated Glyphs are represented (b) with colour, text, number and measure Entity and Relationship blood 3 arrows. Instruction | Explanation pressure Confirmation | Trends ‣ Identity and relationship are managed by a simple colour code Entity A Identity system (e.g. Nurse A is only allowed to interact with patients and 19 3 Entity B Identity equipments marked with a given Status colour code). ‣ Glyphs are initially abstract, on approaching an entity more detail is (a) (c) revealed. Missing / Faulty StatusRepresents “red” coded nurse’s activity of measuring blood pressure with a “red” coded patient numbered“3”, using a “red” coded blood monitoring device numbered “19” which is working fine. Colour + Number + Text + Symbol Example Glyphs
  23. 23. Alternative Glyph Design
  24. 24. Text Snippet + Symbolic + Temporal Stream Design Alternatives
  25. 25. Iconic + Abstract + Temporal Stream Design Alternatives
  26. 26. WhatInformation Affinity How Representation Fidelity Situated Glyphs Where When Spatial Distribution Temporal DistributionSpatial Distribution Where to Present Glyphs
  27. 27. Eppler et al. 2004, Muller et al. 2009, Pousman et al. 2006 Demanding Fragmentation of Attention Context Switch Fine Grained Situated Information CapacityFragmentation of Attention vs Information Capacity
  28. 28. Entity CentricGlyph is placed at each entity, e.g., patient, equipment Activity Centric Glyph is placed at the location of the Insight - 4 activity, e.g.,patient’s bed. Co-Location Centric Glyph is placed at the glance-able space, e.g., nearby wall Design Alternatives
  29. 29. Entity CentricGlyph is placed at each entity, e.g., patient, equipment Activity Centric Glyph is placed at the location of the activity, e.g.,patient’s bed. Co-Location Centric Glyph is placed at the glance-able space, e.g., nearby wall Design Alternatives
  30. 30. What Information Affinity How Representation Fidelity Situated Glyphs Where When Spatial Distribution Temporal DistributionTemporal Distribution When to Present Glyphs
  31. 31. Temporal Distribution Before Before During After Relationship Yes Yes YesInformation Affinity Status Yes - - During Insight - 5 Instruction Yes Yes - Confirmation - - Yes Explanation - Yes - After Trends Yes - - Information Vs Timing
  32. 32. Temporal Distribution Before Before During After Relationship Yes Yes YesInformation Affinity Status Yes - - During Instruction Yes Yes - Confirmation - - Yes Explanation - Yes - After Trends Yes - - Information Vs Timing
  33. 33. System
  34. 34. iPod Touch with Custom Shield 51 mm x 30 mm Jennic JN5139 Micro Controllers OELD-160-G1 Displays (160x128 Pixels) Contiki Operating System TCP/IP Suite on top of ZigBeewith TeCOVisit us at the Demo Session Hardware Prototype
  35. 35. Kawsar & Kortuem, "Supporting Interaction with the Internet of Things across Objects, Time and Space" Internet of Things 2010 Conference (IoT-2010)RESTful Software Infrastructure
  36. 36. Follow-up Studies
  37. 37. Temporal Distribution Control MediationStudy 1 - @ Mainkofen Hospital with Nurses
  38. 38. Spatial DistributionStudy 2 - @ Controlled Lab Settings
  39. 39. ‣ Situated Glyphs as visual representations of human activities to provide in-situ task support information.‣ Activity patterns expressed in action-object pairs for a hospital ward dedicated to dementia patients.‣ A design space analysis for situated glyphs ‣ Information Affinity - Type of Information Need ‣ Representation Fidelity - Semiotics Signs Mapping ‣ Spatial Distribution - 3 Design Alternatives ‣ Temporal Distribution‣ A prototype wearable display for situated glyphs - as building blocks for miniature display network. Take Away Points
  40. 40. Fahim  Kawsar @raswakh-p://www.fahim-­‐kawsar.net Jo  Vermeulen @jozilla h-p://jozilla.net/ Gerd  Kortuem @kortuem h-p://www.kortuem.com

×