The document discusses using cognitive science theories and approaches to study information visualization and interaction. It proposes taking a human-centered, theoretically grounded approach to decompose tasks based on cognitive architectures and identify challenges to cognitive processing. This would involve using "toy world" studies and computer models to test systems and gain quantitative predictions that can inform interaction design. It provides examples of previous work on multiple object tracking and air traffic control research.

1. Cognitive Approaches for Studying Information Visualization and Interaction Fisher and Dill

2. Cognitive “Interaction Science” of Visual Analytics High need for Cogsci theory base for Visual Analytics Cogsci theories evocative for Visual Analytics Enactive Cog: Perception-Cognition-Action loops Distributed Cog: Problem-solving in joint activities Level of description is too high-- how to use? Interaction science should ground theory in use Cognitive Architecture task decomposition Toy-world studies, CogArch tests e.g.: Fishtank air traffic control & FINSTS Combine observation & computer models How to decompose multiple screen collaborative VA?

3. Background Molecular Biology, Medical Biophysics, Hiram/CWRU Cognitive Neuroscience, Experimental Psychology, UCSC Human-Computer Interaction Inst. for Robotics & Intelligent Systems, UWO Cognitive Science, Rutgers SFU School Interactive Arts and Technology UBC Media And Graphics Interdisciplinary Centre

4. Design Approach Human-centred Theoretically motivated Grounded in practice of design Emphasize design process Evolve human sciences to interaction science Framework for performance analysis Taxonomy of interaction Integrate with (software) engineering models

5. Pasteur’s Quadrant (Stokes) MAGIC

6. Research approach Decompose task based on cognitive architecture Identify interaction challenges to cognitive processing modules Devise “toy world” test of that module’s robustness to challenge Use-inspired basic research Research interaction, not the mind Model data for quantitative prediction

7. Air traffic control research Free Flight ATC “fishtank” projection Change camera position for better view How will global motion affect tracking?

8. Multiple object tracking (Pylyshyn)

9. 3-D Projected display

10. Test performance at different speeds...

11. Fit human tracking function (Lui)

12. ... Then add display motion

13. Tracking vs object speed

14. Tracking vs 2 object speeds

15. Tracking in warped space

16. Tracking in warped space

17. Recap Pasteur: Basic research on fermentation will inform applied work on disease (and beer) We think: Basic research on tracking will give us design guidelines for fishtank ATC displays Should generalize to other systems Modeling leads to quantitative prediction and customization for “personal equations” This is not psychology

18. Design space explosion Buxton/Kasik

19. As interactive technology becomes immersive, multimodal, and ubiquitous The emphasis in interaction design shifts to facilitating perceptual cognition and embodied interaction The emphasis in interaction design shifts to facilitating perceptual cognition and embodied interaction The emphasis in interaction design shifts to facilitating perceptual cognition and embodied interaction

20. Interaction science & design Paul Ehrlich Knew chemistry, but not biochem Used science to reduce design space to 900 Had animal model for testing Number 606 = Salvarsan Advance science and treatment Today Medical science is distributed cognition Clinicians, physiologists biochem... Many will never be first author

21. Interaction design future? Design for key sensory & motor systems Assess specific aspects of interaction Walkthrough or experiment Implement prototype Interaction Science

22. Implementations Perception & attention in dynamic workspaces Air traffic control (HRL, NSERC), Multimodal driving interfaces (Nissan) Cognitive & collaborative aspects of tasks CZWeb/CZTalk Laser pointers, cell phones + situated displays (NTT, Panasonic) CSCL video, attention management/privacy(CANARIE, EU Kaleidoscope) Space constancy in VR Wayfinding in CAD worlds (Boeing) Display customization for perceptual abilities (GM)

23. Acknowledgements Profs Jim Enns (UBC) Sid Fels (UBC) Ron Rensink (UBC) Grads Colin Swindels Ritchie Argue Erin Austin Geniva Lui Alexander Stevenson • Postdocs/Staff – Jason Harrison (UBC) – Barry Po • Grants & partners – Nissan – Boeing – GM – HRL – NSERC – IRIS – MAGIC Endowment Fund

24. Spatial Cognition in complex environments Multiple events at different locations Multimodal events (sight, sound, touch) Action in space Tools FINSTs and indexical cognition Space Constancy Functional space constancy: Ability to interact directly Apparent space constancy: Ability to make judgments Sensory integration (FLMP, calibration by pairing)

25. Communicative Pragmatics & Groupware Psycholinguistic pragmatics theories and methods extended to interactive environments Support for familiar F2F metachannels Embodied communication (gesture, facial expression, body language Prosody Interaction methods to support underlying processes Acknowledgement and repair mechanisms Advancement Layering

26. Personalization Personal Equation: Individual differences in perceptual, indexical attentive, and cognitive processes Test users Model data Describe individual differences in model parameters Customize display for PEI Attentive systems adapt PEI Within a session: fatigue, attentiveness Between sessions: aging changes

28. Other work Combine qualitative and mathematical techniques Highly skilled interaction Study control and communication

29. Grounded theory Open coding Themes arise from the data, phenomena are observed Axial coding Relations are drawn between themes and observations in the data Triangulation Mixed methods to test theories

30. Add HCI aspects/analyses Replace flutes with analog synth Allows us to capture Control: Breath and fingering Communication: Gestural motion Can add HCI variables such as lag, bias, noise etc. to understand how skilled performers compensate May feedback to music controller design

31. Mapping to Visual analytics Build toolset: “Thick Description” + Math models Understand “gold standard” for interaction Technology to support expertise - “virtuosity” How information from multiple senses is integrated “ Tight loop” Perception/action patterns, sequences, rhythms Development of coordination between users