Your SlideShare is downloading. ×
Wayne Giang- the 10th annual Human Factors IUW Workshop
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Wayne Giang- the 10th annual Human Factors IUW Workshop


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide
  • Sonification Start
  • Sonification Continue
  • Alarms Auditory
  • Cellphone/Blackberry watch -> Boss
  • Boom!
  • Transcript

    • 1. Use of Multi-Sensory Temporal Synchrony as a Method for Showing Data Relationships Across Modalities Wayne Giang Nov. 13th , 2010
    • 2.
    • 3.
    • 4.
    • 5.
    • 6.
    • 7.
    • 8. Presentation Outline • Thought Experiment • Complex Systems and EID • Multimodal Interfaces • Layouts in Multimodal Interfaces • Temporal Synchrony
    • 9. Controlling Complex Systems • Large, socio-technical, real-time, dynamic systems • Complex systems require special interfaces because (Vicente & Rasmussen, 1992): – Complex systems require complex controllers – Physical systems are governed by constraints – Good controllers must possess a model of the system
    • 10. Ecological Interface Design • Helps with understanding underlying system constraints • Recognize and respond to abnormal events • Map system constraints and relationships onto perceptual objects • Allow operators to use skill-based behaviour
    • 11. Perceptual Relationships • “Visual ways of displaying information that can reduce the need for memory or mental calculation” (Burns & Hajdukiewicz, 2004) • Visual comparisons of orientation, size, shape, location – Visual characteristics that are perceptually easy to evaluate – Also provides grouping and layout information (configural displays) Burns (2000)
    • 12. Multi-modal Interfaces
    • 13. Multi-modal Interfaces • Input and output beyond traditional keyboard/mouse + monitor interactions • Multi-modal presentation  audition, touch, vision • Multiple ResourceTheory (Wickens & Hollands, 2004) (Wickens & Hollands, 2004)
    • 14. Layouts in Multimodal Interfaces • Different models (Sarter, 2006): – Redundancy – Supplementary – Sensory modalities as separate channels of information • Lack of research about how to layout information in different modalities, especially for abstract data • Hypothesis: Layouts in multimodal interfaces will depend on cross-modal relationships
    • 15. Showing Cross-modal Relationships • Different degrees of cross-modal relationships: – Completely automatic– Multisensory Integration – Based on a judgement – Cross-modal Matching • Perceptual relationships  EID • Can we group multi-modal interface information into perceptual objects ? • Are there processing advantages for grouping display information into perceptual events?
    • 16. Three principles of multi-sensory integration (Meredith & Stein, 1993) Spatial Rule – Integration is more likely when the individual sensory stimuli come from roughly the same location Temporal Rule – Integration is more likely when the individual sensory stimuli start from roughly the same time . Principle of Inverse Effectiveness – Integration is more likely when the individual sensory stimuli are vague or weak.
    • 17. Display Integration in Ecological Displays (Burns, 2000) • Impact of spatial proximity (things presented close together) and temporal proximity (things presented at nearly the same time) on understanding ecological displays • Recommendation of using higher spatial proximity and temporal proximity in adaptive problem solving tasks
    • 18. • Spatial location? • Temporal occurrence? Possible Methods for Grouping Crossmodal Data
    • 19. Temporal Synchrony • A type of crossmodal matching – Are things happening at the same time/ same duration? • Ability to detect temporal synchrony developed at a very young age (Lewkowicz, 2000) • “Synchrony window” exists in which things are perceived to be synchronous
    • 20. WhyTemporal Synchrony? • Not as constrained by physical location of displays. – Higher mobility for operators – Lower space footprint • Can be more dynamic • Different methods for invoking temporal synchrony: – Onset – Duration – Rate – Rhythm
    • 21. Audition andTouch • Most commonly used non-visual modalities for information presentation • Share many “amodal” characteristics • Benefits: – Ability to be perceived even when attention is directed elsewhere – More accurate representation of temporal information (double flash illusion)
    • 22. • Onset Possible Groupings usingTemporal Synchrony
    • 23. • Duration Possible Groupings usingTemporal Synchrony
    • 24. • Rate Possible Groupings usingTemporal Synchrony
    • 25. Proposed Methodology- Research Questions • To what degree can we group or compare auditory (earcon/sonification) and tactile (tactor/tactification) information using temporal synchrony and how is this impacted by operator workload? • Are there processing advantages for grouping display information using these perceptual relationships? + Temporal Synchrony? = Perceptual Objects?
    • 26. Take Home Message • Layouts in multi-modal interfaces may be dependent on crossmodal perceptual relationships. • EID can leverage these relationships in complex systems • Temporal synchrony is one method that may be used to group information in multimodal interfaces
    • 27. Questions?