This document discusses the distribution of cognition across individuals, tools, language, and methodology (H-LAM/T systems). It suggests that improving individual effectiveness in society should be approached as a system engineering problem by studying the interacting whole using a synthesis-oriented approach. The document also discusses using control theory and information theory to explain cognitive and social phenomena (cybernetics). Finally, it discusses using paired analytics sessions with a visual analyst and domain expert to collaborate on analytic tasks and influence design decisions in aviation safety.

1. Visually-enabled Leave
Blank
Distribution of
Cognition
Brian Fisher
SFU Interactive Arts & Technology / Cognitive Science
UBC Media & Graphics Interdisciplinary Centre
Calit2 Visiting Scholar, University of California 2012
1

2. 1965 Visual Leave
Blank
Analytics
Individuals who operate effectively in our culture
have already been considerably "augmented." Basic
human capabilities for sensing stimuli, performing
numerous mental operations, and for
communicating with the outside world, are put to
work in our society within a system--an H-LAM/T
system--the individual augmented by the language,
artifacts, and methodology in which he is trained.
Furthermore, we suspect that improving the
effectiveness of the individual as he operates in our
society should be approached as a system-
engineering problem--that is, the H-LAM/T system
should be studied as an interacting whole from a
synthesis-oriented approach.
2Doug Engelbart
2

3. On the Death of
Visualization (2004) Leave
Blank
Can It Survive Without Customers?
Bill Lorensen
• Visualization, alone, is not a solution.
• Visualization is a critical part of many applications.
Text
• Visualization, the Community, lacks application
domain knowledge.
• Visualization has become a commodity.
• Visualization is not having an impact in applications.
3

4. Visual Analytics
Leave
Blank
“The science of analytical reasoning
facilitated by interactive visual interfaces”
“This science must be built on integrated
perceptual and cognitive theories that embrace
the dynamic interaction between cognition,
perception, and action. It must provide insight
on fundamental cognitive concepts such as
attention and memory. It must build basic
knowledge about the psychological foundations
of concepts such as ‘meaning,’ ‘flow,’
‘confidence,’ and ‘abstraction.’ “
“Illuminating the Path” (IEEE Press)
4

5. Cybernetics
Leave
Blank
• 1948 Hixon Symposium, Macy
Conferences
• Use control theory, information
theory to explain cognitive and
social phenomena
• Gurus: Weiner, McColloch &
Pitts, Ashby, Shannon, von
Neumann, Turing
• Application people: Bateson,
Mead
On the Origins of Cognitive Science: The
Mechanization of the Mind by Jean-Pierre
Dupuy
5

6. Cognitive Science Leave
Blank
The study of thought, learning, and mental
organization, which draws on aspects of psychology,
linguistics, philosophy, and computer modelling.
(OED)
Cognitive Science Society founded in 1979
6

7. Founders Leave
Blank
• Daniel Bobrow - AI
• Eugene Charniak - AI
• Allan Collins - Psychology
• Edward Feigenbaum - AI
• Charles Fillmore - Linguistics
• Jerry Fodor - Philosophy
• Walter Kintsch - Psychology
• Donald Norman - Psychology
• Zenon Pylyshyn - Psychology
• Raj Reddy - AI
• Eleanor Rosch - Psychology
• Roger Schank - AI
7

8. Key figures not at Leave
Blank
Dallas
AI Linguistics Neurosci Philosophy Psychology
Chomsky ✓ ✓
Miller ✓ ✓ ✓ ✓
Minsky ✓ ✓ ✓
Newell ✓ ✓
Simon ✓ ✓ ✓
8

9. Leave
Blank
9

10. What Happens in Leave
Blank
the Maze?
• A division of labour?
• Simon’s“nearly decomposable
problems”
• Conceptual & methodological
“trading zone”?
• Galison: Image and Logic: a Material
Culture of Microphysics
• Thagard: Trading Zones in Cognitive
Science
10

11. Cognitive Leave
Blank
Architecture
• Unified Theories of Cognition
(Newell)
• Models = theories
• Models should explain a range of
behaviours
• “You can’t play 20 questions with
nature and expect to win”
• Fodor “Modularity of Mind” 1980
• Egon Brunswick “Conceptual
Framework of Psychology” 1952
11

12. Time scales
(Newell) Leave
Blank
Time scale Psychological
(sec) domain
107
106
• SOCIAL
105
104
103 • Task • RATIONAL
102 (Adaptive)
101
• Unit task • COGNITIVE
100 • Operations (Proximal
10-1 • Attention Mechanisms)
12

13. New experiences,old Leave
Blank
brains
• Transistor density
doubles every 24
months
• Disk density doubles
every 12 months
• Brain volume doubles
every 3 x107 months
Intuitions about architectural processing are
inaccurate = “metacognitive gap”
13

14. Controller/display
systems in air traffic Leave
Blank
control
• NextGen ATC
“fishtank” projection
• Change camera
position for better
view
• How will global
motion affect
tracking? http://www.youtube.com/watch?v=tKJVB4id_TY
Liu, G. Austen, E. L., Booth, K.S. Fisher, B., Argue, R. Rempel, M.I., & Enns, J. (2005)
Multiple Object Tracking Is Based On Scene, Not Retinal, Coordinates. Journal of
Experimental Psychology: Human Perception and Performance. 31(2), Apr 2005,
235-247.
14

15. FINST theory of Leave
Blank
spatial indexing
15

16. Multiple object Leave
Blank
tracking (Pylyshyn)
16

17. 3-D Projected Leave
Blank
display
17

18. Test Performance at
Different Speeds Leave
Blank
18

19. Fit human tracking
Leave
Blank
function (Lui)
19

20. ... Then add display Leave
Blank
motion
20

21. Tracking vs object Leave
Blank
speed
21

22. Tracking in warped
Leave
Blank
space
22

23. Tracking in warped Leave
Blank
space
23

24. Conclusion: We track Leave
Blank
in allocentric space
• Retinal speed of targets does not determine
performance
• Motion of targets relative to each other
does
• But only if motion preserves good metric
characteristics of space
• Explanation is at the level of a human -
display cognitive system
24

25. Optimizing analytic
Leave
Blank
systems
• Personal Equation of Interaction
• Test users
• Model data
• Describe individual differences in model parameters
• Customize display for PEI
• Attentive systems adapt PEI
• Within a session: fatigue, attentiveness
• Between sessions: aging changes
• Personal Equation of Interaction can support
training and selection as well as tuning interaction
25

26. Limits of Leave
Blank
Cyberpsychology
• We show how we can lower the bar for
adoption
• Reduce attentional demands
• Customization for the individual
• Adaptation to conditions of use
• Support perceptual, cognitive, interactive
expertise
• ... but what if the users are bears?
26

27. D-Cog origins Leave
Blank
• Cognition in the Wild 1995
• Navy ships and canoes
• Cognition distributed among personnel
on the ship
•Cognition in the head in canoe navigation
• System of measurement,
communication, confirmation,
protocol navigates ship
• Challenge to cogsci: Mind-as-
computer misses cognition
27

28. Analytics in the Wild Leave
Blank
• Emergency Management (NSERC, DHS)
• Mobile analytics / sensor analytics
• “Virtual EOC” visual analytic environment
• Aircraft Safety, Reliability (Boeing/MITACS)
• “Pair analytics” of complex quant and text data
• Economics and finance (MITACS, NSF)
• Behavioural economics (portfolios)
• Systemic risk analysis
• Healthcare Monitoring & Management (DHS)
• Complex data in health research (CFRI)
• Public health management (BC Injury Research & Prevention)
28

29. Pasteur’s
Quadrant Leave
Blank
(Stokes)
Use-inspired
Pure Basic
Basic
Yes Research
Research
Quest for (Bohr)
(Pasteur)
Fundamental
Understanding? Sampling, Pure Applied
Description,
No Taxonomy Research
(Audubon) (Edison) (1822–95)
No Yes
Consideration of Use ?
"Il n'existe pas une catégorie de sciences auxquelles on puisse donner le nom
de sciences appliquées. Il y a la science et les applications de la science, liées
entre elles comme le fruit à l'arbre qui l'a porté"
Louis Pasteur
29

30. How can D-Cog
Leave
Blank
help design?
• Perspective for designers
• Interpret theories from cognitive sciences for use
in design
• Activity Theory-> Michael Cole, Yrjö Engeström
• D-Cog -> Yvonne Rogers, Jim Hollan
• Develop new, more targeted theories &
methods
• Translation from “clinic” to lab and back again
30

31. D-Cog at Cogsci Leave
Blank
• Pluralistic nature of
Cogsci makes it difficult
to exclude new
approaches
• Also makes it difficult to
come to a conclusion
• CogSci 2013 at
Humboldt University in
Berlin Jul 31 - Aug 3.
31

32. What Happens in Leave
Blank
the Maze?
• A division of
labour?
• Conceptual &
methodological
“trading zone”?
• Galison & Thagard
• A “Mangle of
Practice”?
• Pickering (Exeter)
32

33. Pickering’s Mangle
Leave
Blank
of Practice
• Mangle 2 |ˈmøŋgəәl| |ˈmaŋg(əә)l|
noun
• A large machine for ironing
sheets or other fabrics, usually
when they are damp, using
heated rollers.
• chiefly Brit. a machine having
two or more cylinders turned by
a handle, between which wet
laundry is squeezed (to remove
excess moisture) and pressed
33

34. Mangle
of
Visual
Analy0cs
Research? Leave
Blank
Cognition
Social
Perceptual
Science
Science
Methods
Methods
Cognition
in the wild
Graphic & Computation,
Interaction Visualization
Design Methods
Methods
34
34

35. Leave
Blank
“For five decades I have been driven by an
intuitive certainty that computer supported
argumentation could increase humankind's
collective problem-solving capabilities to a
degree that was (is) greatly unappreciated, and
that its explicit pursuit should become one of
society's high-priority, "grand challenges".
Douglas Engelbart, 2003
35

36. Leave
Blank
Putting theory into practice
(and practice into theory)
36
36

37. Mangle
of
Visual
Analy0cs
Research? Leave
Blank
Cognition
Social
Perceptual
Science
Science
Methods
Methods
Cognition
in the wild
Graphic & Computation,
Interaction Visualization
Design Methods
Methods
37
37

38. D-Cog VA lab Leave
Blank
• Dr. Richard Arias- • Amanda Pype
Hernández • Aaron Smith
• Dr. Linda Kaastra • Numerous SFU &
• Dr. Nathalie Prevost UBC student
• Samar Al-Hajj analysts
• Nadya Calderón
• Tera Marie Green
• Ali Khalili
• Hon Cheong Lam
38

39. Lab Mascot Leave
Blank
39

40. Student analysts Leave
Blank
• Best debrief VAST
2007
• Discovery Exhibition
Best paper 2010
(Andrew Wade) &
2011 (Samar Al-Hajj)
• Analytics are
extracted and
communicated to
tech developers as
methods &
prototypes
40

41. Educational Leave
Blank
Programs
• Applied Visual Analyst (2013)
• Perceptually rich human-information discourse
• Tech-mediated social cognition & collaboration
• Learning model, personal equation
• Analytic Designer
• Develop VA technologies
• Customize display & interaction for user & task
• Consult on organizational roles, communication
• Analytic Researcher
• Advance the field
41

42. Andrew Wade VA Leave
Blank
Challenge
• Aircraft Birdstrike Incident Data
• Aircraft Pitot Tube Icing Data
• Immunology - Flow Cytometry Data
• Healthcare Records Analysis Data
• Domestic Violence Data
• Risk Assessment – Financial Data
• Aircraft Hazardous Bills of Materials Data
• Corporate PC Performance Data
• UBC On-line Course Performance Data
• Justice System Prisoner Management
42

43. Setting the Stage Leave
Blank
• “Pair analytics” sessions
• Student visual analyst &
trained domain expert
collaborate on analytic task
• Student “drives”, expert
“navigates”
• Video session & capture
screen
Arias-Hernandez, R, Kaastra, L.T., and Fisher, B. (2011) Joint Action Theory and Pair
Analytics: In-vivo Studies of Cognition and Social Interaction in Collaborative Visual
Analytics. In L. Carlson, C. Hoelscher, and T. Shipley (Eds.), Proceedings of the 33rd
Annual Conference of the Cognitive Science Society (pp. 3244-3249). Austin TX:
Cognitive Science Society.
43

44. Bird Strikes Leave
Blank
44

45. Leave
Blank
45

46. Leave
Blank
46

47. Leave
Blank
47

48. Tableau Bird Strike
Pair Analysis Leave
Blank
48

49. IN-SPIRE Bird
Strike Pair Analysis Leave
Blank
49

50. Wade Internship Leave
Blank
• Video recorded and screen captured over
10 Paired Analysis sessions using both
Tableau and IN-SPIRE
• Influenced design decisions on:
• 777
• P8-A
• 787
• 747-8
• Changes to pilot training manual
50

51. Wade Internship Leave
Blank
• Presented work to:
• 787 Engineers
• Aviation Safety Community of Practice
• Aerodynamics, Performance, Stability and Control
flight data recorder analysis group
•Advanced Analytics group
•UW Aeronautics and Astronautics students
•Boeing Educational Network webcast (400+)
• 500+ people exposed to Visual Analytics,
Paired Analysis for Aviation Safety
51

52. Understand
technological Leave
Blank
distribution of cognition
• Build on social science (GT, JAT)
approaches to understand organizations
Social
and cognitive work practices.
Sciences
• The innovation here is in the extension
of social science to bridge to the
perceptual and cognitive science
theories that apply to the use of
Cognitive Science in analytical tasks.
52

53. Joint Activity Theory
(Clark) Leave
Blank
• Language is an essentially collaborative
activity, like playing duet or paddling canoe
• We work to build common ground so as to
communicate effectively and efficiently
• Clark’s theory:
• Defines kinds of common ground
• Formalizes the notion of activity as a “joint action”
• Describes the processes by which common
ground is developed through joint action
53

54. Grounding Leave
Blank
• Common ground: set of things mutually believed
by both speaker and hearer
• Hearer must ground or acknowledge
understanding of speakers utterance.
• Clark (1996):
• Principle of closure. Agents performing an action require
evidence, sufficient for current purposes, that they have
succeeded in performing it
• Need to know whether an action succeeded or
failed
54
54

55. Metacognition in
communication
Leave
Blank
• Grounding mechanisms to establish
common referents
• Inflection, gesture, expression
• Hesitation, pausing phrases (e.g. “um”)
• Misc. visual and auditory cues
• Explicit signals
• Speakers change their methods based
upon unconscious assessments of these
cues
55

56. Generalizing Clark Leave
Blank
• Clark’s theory draws upon
joint activity metaphors
(paddling, music) but
focuses on spoken
language
• Kaastra applied that
framework to musical
performance analysis
• Will it work for analytic
duet?
56

57. JAT Analysis of Leave
Blank
Pair Session
• Structuring and navigation markers:
• “Vertical markers” are verbal gestures, such
as “okay,” and “all right,” that signal transitions
between different analytical tasks.
• “Horizontal markers,” such as “yeah” and
“mhmm,” are used to signal continuation
within a singular analytical task
57

58. JAT Analysis of
Pair Session Leave
Blank
• Management of joint attention:
• Gaze, finger-pointing and mouse-point are used by a
speaker to direct joint attention, and by a listener to
confirm that joint attention is in place.
• Use of “self-talk” with on-screen gesturing to inform
about progress on the execution of a cognitively
demanding task: Pauses in analysis were
accompanied by “self-talk” of participants on-task:
• We are currently studying and categorizing “self-talk”
occurrences in PA as indicators of cognitive
workload.
58

59. Emergency
Management Leave
Blank
“Your users are bears”
• Coordinated technological, methodological,
organizational & training support
• Many technologies w/o rich visualization-- small
form factor devices, sensors, data input.
• Example: VA for Emergency Management NSERC
SPP (+ 2 SPP companion proposals)
• Population: cell phones
• First responders: blackberries
• Data fusion centres: geotagged sensor networks, big data
processing
• Command centres: interactive tabletops, walls, etc.
59

60. Visual analytics in
Leave
Blank
the Wild
• Emergency management for mixed-bear systems
• RAH to JIBC, train at Richmond EOC
• CREATE-ERE cross-border earthquake
“experiment”
• Richmond, Vancouver, Provincial EOC
• Field reports, interviews
• Build Advanced Interaction EOC
• Virtual EOC to come
60

61. VA is more than Leave
Blank
technology!
• VA proposes not just new technologies but
new analytic methods
• These new methods will change structures
of communication, selection, & training
• New analytic systems need new analysts (Boeing)
• Analysts take on new roles, organizations change
• VA must address human, organization, and
technology (this includes universities!)
61

62. Translational Science of
Analytical Reasoning Leave
Blank
• VA must develop scientific methods that
help us to augment analytical reasoning
• VA includes human, organization, & tech
• Human: perceptual/cog/motor “cyberpsychology”
• Community: social science of information flow
• Technology: software design & engineering 1&2
• Methods to bridge these levels of analysis
Fisher, B., Green, T.M., Arias-Hernández, R. (2011) "Visual analytics as a translational cognitive
science," Topics in Cognitive Science 3,3 609–625.
62

63. Injury Prevention &
Leave
Blank
Response Group
Analysis
• BC Injury Research & Prevention Unit (Dr Ian Pike)
• Goals:
• Enhance stakeholders’ understanding of injury
indicators.
• Support multi-party analysis.
• Support multi-party decision-making
• Visual Analysis Expert (Samar Al-Hajj) will:
• Manage and guide the group analysis
• Offer stakeholders different approaches to
solving analytical problem.
63

64. Child Injury Indicators
Leave
Blank
Indicators Spanning 1. Mortality Rate
2. Potential Years of Life Lost
Across All Domains 3. Hospital Separations Rate
Overall Health Service 4. Diagnosis-Specific Hospital Separations
5. Hospital Admission - Injury Severity 1
Implications Indicators 6. Hospital Admission - Injury Severity 2
7. Length of Stay in Hospital
8. Cost of Motor Vehicle Injuries
9. Crash Rate
10. Intersection Crash Rate
11. Rural Roadways
Motor Vehicle Injury 12. Drunk Driving
13. Speed
Indicators 14. Young Drivers
15. Graduated Driver Licensing
16. Child Restraints
17. Unrestrained Injuries
18. Child Restraint Laws
19. Bicycle Helmet Laws
Sport, Recreation and 20. Cost of Sport, Recreation and Leisure Injuries
21. Percentage of Sport Specific Injuries (Participation Rates)
Leisure Injury Indicators 22. Requirements that Playgrounds Meet CSA Standards
23. Legislations Requiring Pool Fencing
Other Policy Indicators 24. Window Guard By-law
25. Provincial Standards for Hot Water Tap Temperature
26. Violent Crime Rate
27. Abusive Head Trauma Rate
Violence Indicators 28. Suicide Prevention
29. Anti-Violence/Anti-Bullying Policies
Trauma Care, Quality 30. Access to Pediatric Trauma Centre (PTC)
31. Appropriate Use of Pediatric Trauma Centre (PTC)
and Outcomes 32. Quality of Trauma System
33. Pre-hospital Transport Time
Indicators 34. Presence of a Coordinated Pediatric Trauma System
64

65. Leave
Blank
65

66. Leave
Blank
66

67. Visual Rhetoric Leave
Blank
• Child injury prevention users are creating
and interpreting policies
• Can we build a visual rhetoric for them?
• Concept mapping & related methods
• Diagrammatic reasoning, visual communication
• Visualizing argumentation
• Modal logics (deontic, temporal,doxastic)
• Can we bridge the analyses?
67

68. “How Business
Schools Lost Their Leave
Blank
Way”
Bennis & O’Toole, HBR, 2005
• We certainly do not advocate that business
schools, in revising MBA curricula, abandon
Text
science. Rather, they should encourage and
reward research that illuminates the mysteries
and ambiguities of today's business practices.
Oddly, despite B schools’ scientific emphasis, they
do little in the areas of contemporary science that
probably hold the greatest promise for business
education: cognitive science and neuroscience.
68

69. NSF Science of Leave
Blank
Interaction
• Mixed-initiative data manipulation
and discovery (thinking about data)
• Collaborative analytics
• Multi-modal sensemaking
• Fluid interaction
• Visual discourse (rhetoric)
69

70. Decision Support &
Operational Leave
Blank
Management Analytics
mini-track
70