1. A cognitive architecture-based modelling approach to
understanding biases in visualisation behaviour
David Peebles
University of Hudders

2. eld
November 8, 2014
David Peebles (University of Hudders

3. eld) Cognitive architectures and visualisations November 8, 2014 1 / 16

4. Outline of the talk
Two aims of the research:
To understand at a very detailed level of analysis the processes by
which people interact with external representations of information
(cognitive science).
To develop arti

5. cial agents that can provide expert interpretations of
data presented in external representations (AI).
Structure of the talk:
Human performance modelling.
Cognitive architectures.
ACT-R cognitive architecture.
A recent example: An ACT-R model of graph comprehension.
David Peebles (University of Hudders

6. eld) Cognitive architectures and visualisations November 8, 2014 2 / 16

7. The embodied cognition-task-artefact triad
Embodied
Cognition
Task
Artefact
Interactive
Behaviour
Interactive behaviour:
Emerges from dynamic interaction of goal-directed, task-driven
cognition/perception/action with designed task environment.
A complex combination of bottom-up stimulus driven and top-down
goal and knowledge driven processes.
Makes little sense to consider and investigate cognition in isolation.
David Peebles (University of Hudders

8. eld) Cognitive architectures and visualisations November 8, 2014 3 / 16

9. Understanding behaviour with visualisations
Requires detailed understanding of the wide range of factors that
aect human performance.
Internal to the agent:
Domain and graphical representation knowledge (expertise)
Working memory capacity
Strategies
Learning/adaptation to task environment
External to the agent:
Computational aordances
Emergent/salient features
Gestalt principles of perceptual organisation
Task and environment complexity
David Peebles (University of Hudders

10. eld) Cognitive architectures and visualisations November 8, 2014 4 / 16

11. Human performance modelling
Requires the development of two connected models:
Functional task environment:
Speci

12. es elements of the physical environment relevant to the agent's
goals and particular cognitive, perceptual and motor characteristics
(Gray, Schoelles Myers, 2006).
Human performance model:
Incorporate current theories of the cognitive, perceptual and motor
processes.
Specify the modular architecture of the mind, the functions and
limitations of the various modules, and the connections between them.
Specify the task-general and task-speci

13. c knowledge that the agent
must possess.
Specify the control mechanisms (i.e., interactive routines, methods and
strategies) that determine performance.
David Peebles (University of Hudders

14. eld) Cognitive architectures and visualisations November 8, 2014 5 / 16

15. Cognitive architectures
Theories of how the permanent computational structures and
processes that underlie intelligent behaviour are organised.
Uni

16. ed theories of cognition (Newell, 1990)
Three main symbolic cognitive architectures: Soar (Newell, 1990;
Laird, 2012), Epic (Kieras Meyer, 1997), ACT-R (Anderson, 2007)
All three based on production system architecture.
All can interact with simulated task environments to model
interaction with external representations.
David Peebles (University of Hudders

17. eld) Cognitive architectures and visualisations November 8, 2014 6 / 16

18. The ACT-R cognitive architecture
Visual
Module
Environment
ACT−R Buffers
Pattern
Matching
Production
Execution
Motor
Module
Problem
State
Declarative
Memory
Procedural
Memory
Control
State
Hybrid architecture with symbolic and subsymbolic components
Production system model of procedural memory cognitive control
Semantic network model of declarative memory
Activation-based learning, memory retrieval forgetting mechanisms
Simulated eyes hands for interacting with computer-based tasks
David Peebles (University of Hudders

19. eld) Cognitive architectures and visualisations November 8, 2014 7 / 16

20. Graph comprehension
Initial familiarisation stage prior
to other tasks involving:
Identi

21. cation classi

22. cation
of variables into IV(s) and DV
Association of variables with
axes and representational
features (e.g., colours, shapes,
line styles)
Identi

23. cation of
relationship(s) depicted
May be an end in itself or a
prerequisite for other tasks
David Peebles (University of Hudders

24. eld) Cognitive architectures and visualisations November 8, 2014 8 / 16

25. Interaction graphs
Low High
100
90
80
70
60
50
40
30
20
10
Percent Error as a function of Experience and Time of Day
Percent Error
Experience
Time of Day
Day
Night
Students more likely to
misinterpret (Zacks Tversky,
1999) or inadequately interpret
line graphs (Peebles Ali,
2009, Ali Peebles, 2013)
Percent Error as a function of Experience and Time of Day
Low High
100
90
80
70
60
50
40
30
20
10
●
●
Percent Error
Experience
Time of Day
Day
Night
Line graphs better at depicting
common relationships for
experts (Kosslyn, 2006)
Interpretation facilitated by
recognition of familiar patterns
David Peebles (University of Hudders

26. eld) Cognitive architectures and visualisations November 8, 2014 9 / 16

27. An example expert verbal protocol
Glucose Uptake as a function of Fasting and Relaxation Training
High Low
250
225
200
175
150
125
100
75
50
25
0
l
l
l
l
Glucose Uptake
Fasting
Relaxation Training
Yes
No
1 (Reads) Glucose uptake as a function
of fasting and relaxation training
2 Alright, so we have. . . you're either
fasting or you're not. . .
3 You have relaxation training or you
don't. . .
4 And so. . . not fasting. . . er. . .
5 So there's a big eect of fasting. . .
6 Very little glucose uptake when you're
not fasting. . .
7 And lots of glucose uptake when you
are fasting. . .
8 And a comparatively small eect of
relaxation training. . .
9 That actually interacts with fasting.
David Peebles (University of Hudders

28. eld) Cognitive architectures and visualisations November 8, 2014 10 / 16

29. Stages of comprehension
Glucose Uptake as a function of Fasting and Relaxation Training
High Low
250
225
200
175
150
125
100
75
50
25
0
l
l
l
l
Glucose Uptake
Fasting
Relaxation Training
Yes
No
Comprehension proceeds in the
following order:
1 Read title. Identify variable names
and create declarative chunks.
2 Seek variable labels, identify what
they are by their location and if
required, associate with label levels
3 Associate variable levels with
indicators (position or colour)
4 Look at plot region and attempt to
interpret distances. If a highly salient
pattern exists (e.g., cross, large gap)
process that

30. rst
5 Continue until no more patterns are
recognised
David Peebles (University of Hudders

31. eld) Cognitive architectures and visualisations November 8, 2014 11 / 16

32. Videos of the model
David Peebles (University of Hudders

33. eld) Cognitive architectures and visualisations November 8, 2014 12 / 16

34. An example model protocol
Chick Weight as a function of Diet and Hormone Supplement
Artificial Natural
50
45
40
35
30
25
20
15
10
5
0
l
l
l
l
Chick Weight
Diet
Hormone Supplement
MPE
GCE
text at top of display. . .
[chickweight] [= variable]
[as] [a] [function] [of] [diet] [= variable]
[and] [hormonesupplement] [= variable]
text at bottom of display. . .
[diet] at [bottom] [= IV]
look to nearest text. . .
[natural] is a level of [diet]
[natural] is [right]
[arti

35. cial] is a level of [diet]
[arti

36. cial] is [left]
text at far right of display. . .
[hormonesupplement] at [far-right] [= IV]
look to nearest text. . .
[mpe] is a level of [hormonesupplement]
[gce] is a level of [hormonesupplement]
David Peebles (University of Hudders

37. eld) Cognitive architectures and visualisations November 8, 2014 13 / 16

38. An example model protocol
Chick Weight as a function of Diet and Hormone Supplement
Artificial Natural
50
45
40
35
30
25
20
15
10
5
0
l
l
l
l
Chick Weight
Diet
Hormone Supplement
MPE
GCE
objects in plot region. . .
a [green] [line]
no memory for [green] look to legend. . .
[green] [rectangle]. look for nearest text. . .
[green] represents [gce]
[blue] [rectangle]. look for nearest text. . .
[blue] represents [mpe]
text at far left of display. . .
[chickweight] at [far-left] [= DV]
look to pattern. . .
substantial dierence between legend levels. . .
[0.2] di [blue] = [small] eect [mpe]
[0.2] di [green] = [small] eect [gce]
compare [blue] and [green] levels. . .
[moderate] di: [gce] greater than [mpe]
[moderate] [main] eect [hormonesupplement]
David Peebles (University of Hudders

39. eld) Cognitive architectures and visualisations November 8, 2014 14 / 16

40. An example model protocol
Chick Weight as a function of Diet and Hormone Supplement
Artificial Natural
50
45
40
35
30
25
20
15
10
5
0
l
l
l
l
Chick Weight
Diet
Hormone Supplement
MPE
GCE
identify x-axis levels. . .
[0.4] di [left] = [moderate] eect [arti

41. cial]
[0.4] di [right] = [moderate] eect [natural]
compare [left] and [right] levels. . .
[small] di [natural] [arti

42. cial]
[small] [main] eect [diet]
compare left and right patterns. . .
[0.0] di between points. [neither] bigger
[no] di [same] order = [no-interaction]
for [arti

43. cial], [gce] [mpe]
for [natural], [gce] [mpe]
David Peebles (University of Hudders

44. eld) Cognitive architectures and visualisations November 8, 2014 15 / 16

45. Summary
ACT-R is a powerful, empirically veri

46. ed, tool for creating models of
interactive behaviour with complex task environments.
Model output (errors, task completion times, eye movements, verbal
protocols) can be compared with human data.
Contains learning mechanisms to model adaptation to task
environment and development of expertise.
Can be used as a platform for creating human-level arti

47. cial agents to
interpret graphical representations and visualisations.
David Peebles (University of Hudders

48. eld) Cognitive architectures and visualisations November 8, 2014 16 / 16