1. Visual Analytics
Cognitive Science
Brian Fisher
SFU Interactive Arts & Technology / Cognitive Science
UBC Media & Graphics Interdisciplinary Centre / Computer science
and
2. My double life
(It’s mostly Zenon’s fault)
• 1991 UCSC Ph.D in Psychology, modeling sensory integration
• 3 Job offers!!!
•Center for Complex Systems, Boca Raton FL (math, sun, ocean)
•NRC Postdoc NASA Ames, San Jose CA (tech, sun, ocean)
•UWO Postdoc w Zenon (hockey, snow, Lake Huron)
• UCSC Prof “I disagree with everything he stands for, but you will learn
more from Pylyshyn than you would at the other places”
3. My double life 2
(Zenon’s fault)
•1991-1993 UWO Postdoc w Zenon (IRIS1 Robotics & AI NCE)
• That fateful party at Zenon’s house— “ a cognitive basis for
intelligent interfaces to complex systems… hmmm”
•1994 RuCCS Visiting Faculty
•1994 - 1998 A Cognitive Basis for Intelligent Interfaces to
Complex Systems (IRIS2)
• Shuttle between UWO & SFU/UBC
• Mission: to bring cognitive science to the ignorant masses
(computer science), work on Telelearning NCE, many
industry projects
Timeline:
4. Human-Computer Interaction
• Traditionally, HCI applies methods & theories
from psychological science to usability of
systems that achieve some external goal— e.g.
formatting document, databases, Web
navigation etc.
• Any theories that are generated are local to
some (small) set of applications
• HCI is a good career prospect for students, but
has a minimal impact of HCI research on
psychological science
5. What if the goal of the
system is human cognition?
• Air traffic control - foresee & avoid conflicts
• Educational technology - learning
• Social computing - personal and societal
• Decision-support system - good decisions
• Collaboration systems - joint activity
Application-driven basic research can advance
psychological science
6. My double life 3
(This part is my fault)
• 1998 - Assoc. Director, UBC Media & Graphics
Interdisciplinary Centre
•Adjunct/Associate prof in Psych, Business, Computer Science
•Work with industry: Boeing, GM, Nissan, NTT, locals
•Enable non-tech units to do (and fund) tech projects
•Incubate tech companies: Thoughtshare, Xing Xing
•Incubate not-for profits: New Ventures BC, VARDEC
•Educational initiatives: HCI grad cert, Vancouver Institute for
Visual Analytics VA training, Grad Certificate in VA
• Business and technology instruction as adjunct professor in
Sauder School of Commerce and in Computer Science
Timeline:
7. •2004 “Why don’t you get a real job?”
• New SFU School of Interactive Arts and Technology
• Design (creative, human-centred, engineering)
• Technology
• Media Arts
• Teach cognitive science to artists, designers and engineers—
“cyberpsychology”
• Keep Associate Directing UBC MAGIC as a “hobby”, doing VA
research
My double life 5
(Still my fault)
8. How do we build a bridge?
IEEE can offer:
• Coding
• Funding
• Awesome
Research
Questions Tampa to Sarasota
Cogsci can offer
• Empirical
Methods
• Theories
• Phenomena
• People who
like running
studies
Challenges: defining boundary
objects, culture clash, publication
venues, academic jobs for
Cognitive science grads
10. Example: Visual Analytics
“This science must be built on integrated perceptual
and cognitive theories that embrace the dynamic
interaction between cognition, perception, and action. It
must provide insight on fundamental cognitive
concepts such as attention and memory. It must build
basic knowledge about the psychological foundations
of concepts such as ‘meaning,’ ‘flow,’ ‘confidence,’ and
‘abstraction.’ “
“Illuminating the Path” (IEEE Press)
“The science of analytical reasoning
facilitated by interactive visual interfaces”
11. What I see at VIS
• Design based on theories (but no effective eval
of those theories)
• Adaptation of methods from cogsci (but original
methods not well understood)
• Rarely, ongoing collaboration with cognitive
scientists
12. My VA work
• Explore how people & groups see and think
with interactive visualization
• Cognitive models of human perception & cognition
• Fitted with individual’s “personal equation”
• Focus on analytical reasoning techniques and
presentation / argumentation
• Cognitive ethnography, lab and field experiments
Build predictive cognitive models
to guide design, evaluation and
customization of visualization
environments
14. SCIENCElab research
• Emergency operations
(DHS VACCINE, NSERC)
• Air Traffic Control (HRL,
NSERC)
• Aircraft Safety (Boeing/
MITACS)
• Computer-aided design
(GM research, BCASI)
• Personalized Medicine,
public Health (CIHR,
NSERC, MITACS)
• Cognitive Architecture
• FINSTS & Multiple
object tracking
• Social cognition
• Psycholinguistic
Discourse analysis
• Ethnography
• Grounded Theory
• Psycholinguistics
• JAT
15. Controller/display systems in air
traffic control
• “Close reading”
NextGen ATC
fishtank mockup
• Change camera
position for better
view
• How will global
motion affect
tracking?
Liu, G.Austen, E. L., Booth, K.S. Fisher, B., Argue, R. Rempel, M.I., & Enns, J. (2005) Multiple
Object Tracking Is Based On Scene, Not Retinal, Coordinates. Journal of Experimental
Psychology: Human Perception and Performance. 31(2),Apr 2005, 235-247.
http://www.youtube.com/watch?v=tKJVB4id_TY
19. Social cog in safety analysis
• “Pair analytics” sessions
•Student visual analyst &
trained domain expert
collaborate on analytic task
•Student “drives”, expert
“navigates”
•Video session & capture
screen
Kaastra, L.T., Arias-Hernandez, R., & Fisher, B. (2014) Tracking Joint Activities in
Visual Analyses . Psychonomics Society Meeting. Long Beach CA.
Kaastra, L.T., & Fisher, B. (2014) Field Experiment Methodology for Pair Analytics.
BELIV 2014: Beyond Time and Errors - Novel Evaluation Methods for Visualization.
Visweek 2014, Paris France.
20. Wade Internship
• Video recorded and screen captured over 10
Paired Analysis sessions using both Tableau
and IN-SPIRE
• Influenced design decisions on:
• 777
• P8-A
• 787
• 747-8
• Changes to pilot training manual
21. Wade Internship
• Presented work to:
• 787 Engineers
• Aviation Safety Community of Practice
• Aerodynamics, Performance, Stability and Control
flight data recorder analysis group
• Advanced Analytics group
• UW Aeronautics and Astronautics students
• Boeing Educational Network webcast (400+)
• 500+ people exposed to Visual Analytics,
Paired Analysis for Aviation Safety
24. What we need
• Institutes/centres/programs that combine:
• Application experts
• VIS people
• Cognitive scientists
• Designers
• Publication venues for translational research
• Two-part conference approach
• Outreach to application venues
• Return to VIS community
26. Conclusion: We track in allocentric
space
• Retinal speed of targets does not determine
performance
• Motion of targets relative to each other does
• But only if motion preserves good metric
characteristics of space
• Explanation is at the level of a human - display
cognitive system
28. On the Death of Visualization
(Lorensen 2004)
• Visualization, alone, is not a solution.
• Visualization is a critical part of many
applications.
• Visualization, the Community, lacks application
domain knowledge.
• Visualization has become a commodity.
• Visualization is not having an impact in
applications.
30. Analytics & Analysis
• Where will computation, logic & math fail?
•Relevance, validity, reliability of data uncertain
•Assumptions of the model may not hold in given situation
•Multiple models to chose from
•Previously unknown pattern in data (data discovery)
* Meyer J., Thomas, J., Diehl, S., Fisher, B., Keim, D., Laidlaw, D. Miksch S.,
Mueller, K. Ribarsky, W., Preim, B., & Ynnerman, A. (2010) From Visualization to
Visually Enabled Reasoning. In “Scientific Visualization: Advanced Concepts”. vol. 1
pp. 227-245. Schloss Dagstuhl - Leibniz-Zentrum fuer Informatik, Germany.
I978-3-939897-19-4
31. Interaction of analysis with
operational planning/management
• Capacity of organization to act
•Human elements: morale, conflicts, politics
•Pragmatics learned through experience
•Estimation of likely response of other actors
• Reasoning ala Toulmin
•Temporal, modal & hybrid situation logics
•Deontic logic in policy, casuistry
•Doxastic logic for actions of others
• Feedback to model choice, parameterization etc.
“From data to knowledge to action”
32. Applications and partners
• Emergency Management w Richmond EOC
• Aircraft Safety, Reliability w Boeing
• Health Management
•Personalized health w LSI, PMI, Molecular You
•Child injury w BC Child & Family Research
Institute
•Environmental effects (Artemus Foundation;
Canadian Environmental Health Atlas)
• e-Government: (Nadya Calderón)
•Brazil twitter analysis (World Bank)
•"Data science for social good" City & U. of
Chicago
33. Human-Computer Interaction
• Traditionally, HCI applies methods & theories
from psychological science to usability of
systems that achieve some external goal— e.g.
formatting document, databases, Web
navigation etc.
• Any theories that are generated are local to
some (small) set of applications
• HCI is a good career prospect for students, but
has a minimal impact of HCI research on
psychological science
34. What if the goal of the system is
itself a psychological one?
• Air traffic control - foresee & avoid conflicts
• Educational technology - learning
• Social computing - personal and societal
• Decision-support system - good decisions
• Collaboration systems - joint activity
Application-driven basic research
can advance psychological science
35. In other cases, psych goals are
part of a larger cognitive system
• Computer-aided design
• Search and rescue
• Emergency management
Application-driven basic research
can advance psychological science
36. Our goals
• Build an ongoing dialog between technology-
aware psychological scientists
• Make others aware of research opportunities
and funding that contribute to basic research in
psychological science
• Ask if APS should play an active role in building
this new kind of psychological science
37. Integrated science approach
• Goals are to advance basic science in areas
that have impact on applications
• Field work: Ethnography,“close reading”, field
experiments
• Translation: Identify research questions for lab
study
• Lab Science: Experiment to test hypothesis
• Translation: Work with developers to insure that
practice is consistent with science
38. Extended-distributed cognition in air
traffic control
• Cognitive architecture from psychology
• Extend to expert human performance
• Cognitive expertise
• Visual expertise
• Visuomotor expertise
• Multimodality & modularity
• Test human capabilities in dynamic display
environments
39. Controller/display systems in air
traffic control
• NextGen ATC “fishtank” projection
• Change camera position for better view
• How will global motion affect tracking?
Liu, G.Austen, E. L., Booth, K.S. Fisher, B., Argue, R. Rempel, M.I., & Enns, J. (2005)
Multiple Object Tracking Is Based On Scene, Not Retinal, Coordinates. Journal of
Experimental Psychology: Human Perception and Performance. 31(2),Apr 2005,
235-247.
http://www.youtube.com/watch?v=tKJVB4id_TY
44. Cogs/IAT 885 Visually Enabled
Reasoning
• Cognitive theory
•Psychology of human reasoning
•Modelling causality
•Alternative (modal/hybrid) logics
• Analytic practice
•Problems and datasets from SEMVAST, Boeing
•Teams w paper, IN-SPIRE, Tableau Jigsaw
• Outcomes
•Learn reflective analytic practice
•Prepare for internship as analyst
45. Textbooks
• How We Reason. Philip Johnson-Laird, Oxford
Press
• Causal Models: How People Think about the
World and Its Alternatives. S. Sloman, Oxford
Press R
• Human Reasoning and Cognitive Science.
Keith Stenning & Michiel van Lambalgen, MIT
Press
46. Students
• Dr. Richard Arias-
Hernández
• Dr. Linda Kaastra
• Dr. Nathalie Prevost
• Samar Al-Hajj
• Nadya Calderón
• Tera Marie Green
• Ali Khalili
• Hon Cheong Lam
• Amanda Pype
• Aaron Smith
• Numerous SFU &
UBC student analysts
47. "Il n'existe pas une catégorie de sciences auxquelles on puisse donner le nom
de sciences appliquées. Il y a la science et les applications de la science, liées
entre elles comme le fruit à l'arbre qui l'a porté"
Louis Pasteur
Pure Basic
Research
(Bohr)
Use-inspired
Basic
Research
(Pasteur)
Sampling,
Description,
Taxonomy
(Audubon)
Pure Applied
Research
(Edison)
Quest for
Fundamental
Understanding?
No
Yes
Consideration of Use ?
No
Yes
(1822–95)
Pasteur’s Quadrant
(Stokes)
