1. A computational model of graph
comprehension
David Peebles
February 15, 2012

2. Outline of the talk
1. Introduction
• Empirical studies
• Previous models of
graph use
2. The ACT-R cognitive
architecture
3. An ACT-R model of graph
comprehension

3. Graphical displays of information
• Increasingly used in an ever-widening range of activities
• A current data explosion with new representations and software tools
being developed all the time
• “Open data” and greater public access to information visualisations:
• http://www.informationisbeautiful.net
• http://www.guardian.co.uk/news/datablog
• http://infosthetics.com/
• http://flowingdata.com/
• The emergence of visual analytics (Thomas and Cook, 2005)
• A growing need to teach graphicacy and understand the cognitive
and visual processes that underlie it

4. Costs and benefits of using graphical representations
Benefits
• Information can be grouped
by location, reducing search
• Allows perceptual reasoning
and rapid identification of
salient aspects of data
(Larkin and Simon, 1987)
Costs
• People interpret graphical
elements differently
(Zacks and Tversky, 1999)
• Interpretation can be
distorted by visual features
(Peebles, 2008)

5. Bar and line graphs
Bar graphs
• Interpreted as representing
categories
• People typically encode
height and separate values
• Better for comparing single
quantities
Line graphs
• Interpreted as representing
ordered/scaled data
• People typically encode
slope and continuous change
• Better for identifying trends

6. Line graphs
0
1
2
3
4
5
6
7
8
9
10
JanFebMar AprMayJun Jul AugSepOctNovDec
AmountProduced
Month
Amount of Silver and Gold Produced (in tonnes)
by ‘Precious Metals Plc.’ in 1996
Silver
Gold
Similar, informationally equivalent,
graphs not computationally
equivalent (Larkin and Simon,
1987)
0
1
2
3
4
5
6
7
8
9
10
0 1 2 3 4 5 6 7 8 9 10
Gold
Silver
Amount of Silver and Gold Produced (in tonnes)
by ‘Precious Metals Plc.’ in 1996
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
“How much silver was produced
when 4 tonnes of gold were
produced? (Peebles and Cheng,
2003)

7. Police performance monitors (Peebles, 2008)

8. The case for simulation modelling
• There has been a great deal of graph comprehension research (e.g.,
Carswell and Wickens, 1990; Kosslyn, 1994; Pinker, 1981;
Shah and Carpenter, 1995)
• Still impossible to produce precise advice about optimal display for
particular communicative goal (Meyer, 2000; Kosslyn, 2006)
• Too many variables interacting in complex ways
• Definitive experiments difficult as too many variations in stimulus
conditions possible.
• Computer simulation models of graph reading and comprehension
• allow testing of graphical designs
• allow exploration of ideas and hypotheses about cognitive and
visual mechanisms (McClelland, 2009)

9. Previous process models of graph use
• GOMS-based (task-analytic) models of graph question answering
• UCIE (Lohse, 1993) MA-P (Gillan, 1994)
1. Identify perceptual and cognitive operators from task or
verbal protocol analyses
2. Construct sequences of operators (with execution times)
3. Predict scan paths and task completion times
• Computational models
• BOZ graph generation tool (Casner, 1991)
• CaMerA model of economics expert with multiple
representations (Tabachneck-Schijf et al., 1997)
• ACT-R graph reading model (Peebles and Cheng, 2003)
• None of these models are of graph comprehension

10. Cognitive architectures
• “Unified theories of cognition” (Newell, 1990)
• Theories of core cognitive components, representational formats,
and processes (e.g., memory, cognitive control)
• Main examples of symbolic cognitive architectures:
• Soar (Newell, 1990; Laird, 2012)
• Epic (Kieras and Meyer, 1997)
• ACT-R (Anderson, 2007)
• All three based on production system architecture
• All can interact with simulated task environments
• This allows models of interaction with external representations to be
developed

11. The ACT-R cognitive architecture
• Hybrid architecture with symbolic and subsymbolic components
• Production system model of procedural memory & cognitive control
• Semantic network model of declarative memory
• Activation-based learning, memory retrieval & forgetting mechanisms
• Simulated eyes & hands for interacting with computer-based tasks
Visual
Module
ACT−R Buffers
Environment
Pattern
Matching
Execution
Production
Module
Motor
Control
State
Problem
State
Memory
Procedural
Memory
Declarative

12. Interaction graphs
Low High
10
20
30
40
50
60
70
80
90
100
●
●
Percent Error as a function of Experience and Time of Day
PercentError
Experience
Time of Day
Day
Night
• Display data from 2 × 2
factorial research designs
• Novices often find
interpretation difficult as they
differ from more familiar bar
and line graphs
Low High
10
20
30
40
50
60
70
80
90
100
Percent Error as a function of Experience and Time of Day
PercentError
Experience
Time of Day
Day
Night
• Come in bar and line forms
(both equally likely)
• Primary task is to
comprehend relationships
between variables, not
identify specific values

13. How people use interaction graphs
Low High
10
20
30
40
50
60
70
80
90
100
●
●
Percent Error as a function of Experience and Time of Day
PercentError
Experience
Time of Day
Day
Night
• Novices’ interpretations of
line graphs significantly
worse than bar graphs
(Peebles and Ali, 2009)
• Drawn by salience of lines to
identify legend variable &
ignore x axis variable
Low High
10
20
30
40
50
60
70
80
90
100
Percent Error as a function of Experience and Time of Day
PercentError
Experience
Time of Day
Day
Night
• Expert users’ interpretations
also affected by graph type
• New line graph design
improved novice performance
to bar graph level (Ali and
Peebles, submitted)

14. Alternative designs
Low High
10
20
30
40
50
60
70
80
90
100
●
●
Percent Error as a function of Experience and Time of Day
PercentError
Experience
Time of Day
Day
Night
• “Combined” graph – no
significant improvement
10
20
30
40
50
60
70
80
90
100
●
●
Percent Error as a function of Experience and Time of Day
PercentError
Experience
Low High
Time of Day
Day
Night
• “Colour match” graph –
significant improvement to
bar graph level

15. An ACT-R model of interaction graph comprehension
Aims
• Simulate verbal protocols and eye movement scan paths of experts
and novices interpreting interaction graphs
• Provide a detailed characterisation of:
• the prior knowledge required to interpret interaction graphs
• the information extracted from the diagram
• the knowledge generated during the comprehension process
• the cognitive and perceptual operations involved in interpreting
interaction graphs
• the strategic processes that control comprehension
• what underlies the differences between expert and novice
performance

16. Stages of comprehension
• Verbal protocols reveal that experts produce qualitative descriptions
of the differences between conditions, not individual values.
• Comprehension is typically carried out in two main phases:
1. Variable identification. Labels categorised as dependent or
independent according to location, and associated with levels
and identifiers: left or right (x axis), or colour (legend)
2. Pattern recognition and description. Distances between plot
points compared and used to probe long-term declarative
memory for interpretive knowledge
• Success – retrieved knowledge is used to provide an
interpretation
• Failure – simply describe the identification or comparison
process being carried out

17. Prior declarative graph knowledge (1)
• Three types of knowledge required to interpret interaction graphs:
1. The typical allocation of DV and IVs to graph axes and legend
Low High
10
20
30
40
50
60
70
80
90
100
●
●
Percent Error as a function of Experience and Time of Day
PercentError
Experience
Time of Day
Day
Night

18. Prior declarative graph knowledge (2)
2. The principle that distance between graphical elements indicates
magnitude of relationship between conceptual entities the elements
represent
Low High
10
20
30
40
50
60
70
80
90
100
●
●
Percent Error as a function of Experience and Time of DayPercentError
Experience
Time of Day
Day
Night

19. Prior declarative graph knowledge (3)
3. Graphical/spatial indicators of three interpretive facts:
• Simple effects. Distance between two plot points
• Main effects. Differences in the y-axis location of the midpoints
between two pairs of plot points.
• Interactions. Differences in the inter-point distances between
levels, combined with information about their point ordering.
Low High
10
20
30
40
50
60
70
80
90
100
●
●
Percent Error as a function of Experience and Time of Day
PercentError
Experience
Time of Day
Day
Night

20. Representing and encoding information in the graph
• Model encodes four x-y
coordinate points and spatial
distances between them
• Encoded numerically then
translated to symbolic
descriptions (e.g., “small”,
“very large”). “Elementary
perceptual tasks”
(Cleveland and McGill, 1984)
• Encode spatial distance
between plot points
• Compare magnitude of
two distances and
produce a symbolic
description of difference.
Low High
10
20
30
40
50
60
70
80
90
100
●
●
Percent Error as a function of Experience and Time of Day
PercentError
Experience
Time of Day
Day
Night

21. Knowledge generated during comprehension
• Knowledge chunks
representing the variables
• Three knowledge structures
that accumulate and
associate items of graph and
interpretive information when
identifying:
• Simple effects
• Main effects
• Interactions

22. Stages of comprehension
• Comprehension proceeds in the following order:
1. Read title. Identify variable names and create memory chunks.
2. Seek variable labels, identify what they are by their location and
if required, associate with label levels
3. Associate variable levels with indicators (position or colour)
4. Look at plot region and attempt to interpret distances. If a highly
salient pattern exists (e.g., cross, large gap) process that first
5. Continue until no more patterns are recognised
• Individual production rule for each pattern
• No production rule then pattern not processed

23. Example model output
1 seek text at top of display. . .
2 [m-yield] = [var]
3 [as] [a] [function] [of] [p-density] = [var]
4 [and] [n-level] = [var]
5 seek text at far right of display. . .
6 [n-level] at [far right] = [independent] var
7 look to nearest text. . .
8 [low] = level of [n-level]
9 [high] = level of [n-level]
10 seek objects in plot region. . .
11 [blue] [line]
12 no memory for [blue] look to legend
13 [blue] [rect]. seek near text. . . [blue] = [low]
14 [green] [rect]. seek near text. . . [green] = [high]
15 seek text at far left of display. . .
16 [m-yield] at [far left] = [dependent] var
17 seek text at bottom of display. . .
18 [p-density] at [bottom] = [independent] var
19 look to nearest text. . .
20 [compact] = level of [p-density]. [compact] = [right]
21 [sparse] = level of [p-density]. [sparse] = [left]

24. Example model output
22 identify legend levels. . .
23 [0.0] diff [blue] = [no] [simple] effect [low] [n-level]
24 [0.5] diff [green] [modest] [simple] effect [high] [n-level]
25 compare [blue] & [green] levels. . .
26 [small] diff. [high] [n-level] > [low] [n-level]
27 [small] [main] effect [n-level]
28 identify x axis levels. . .
29 [0.0] diff [left] = [no] [simple] effect [sparse] [p-density]
30 [0.5] diff [right] = [modest] [simple] effect [compact] [p-density]
31 compare [left] & [right] levels. . .
32 [small] diff. [compact] [p-density] > [sparse] [p-density]
33 [small] [main] effect [p-density]
34 compare left and right patterns. . .
35 [0.5] diff in distance between points. [right] bigger
36 [modest] diff & [same] point order = [modest] [interaction]
37 for [sparse] [p-density] [low] [n-level] = [high] [n-level]
38 for [compact] [p-density] [high] [n-level] > [low] [n-level]

25. Limitations of the model
• Model currently focuses on graph knowledge and not the effects of
domain knowledge. Experts do display such domain-general graph
knowledge.
• Early visual processes are not specified. ACT-R models of visual and
spatial processing are being developed.
• Although based on human data, the model has not been rigorously
compared to eye movement and verbal protocol data yet.
• Model of novice users has not been developed yet
• Selectively remove production rules and interpretive declarative
knowledge.

26. Conclusions
• Currently the only computational model of graph comprehension
• Given a data set from a 2 × 2 factorial research design it will produce
an expert level description of the effects
• The model also describes the patterns used to produce the
interpretation
• The model associates variables and their levels, and levels with their
colour identifiers using mechanisms that are functionally equivalent
to the Gestalt laws of proximity and similarity respectively.
• The current model can be considered a first approximation to a more
detailed model that incorporates additional factors to broaden the
scope of behaviour accounted for.
• The model will form the basis of a more general graph
comprehension model that can interpret bar interaction graphs and
more general bar and line graphs.

27. References (1)
• Ali, N. and Peebles, D. (submitted). The effect of gestalt laws of perceptual
organisation on the comprehension of three-variable bar and line graphs
• Anderson, J. R. (2007). How can the human mind occur in the physical
universe? Oxford University Press, New York, NY
• Carswell, C. M. and Wickens, C. D. (1990). The perceptual interaction of
graphical attributes: Configurality, stimulus homogeneity, and object
interaction. Perception & Psychophysics, 47:157–168
• Casner, S. M. (1991). A task-analytic approach to the automated design of
graphic presentations. ACM Transactions on Graphics, 10:111–151
• Gillan, D. J. (1994). A componential model of human interaction with
graphs: 1. linear regression modelling. Human Factors, 36(3):419–440

28. References (2)
• Kieras, D. E. and Meyer, D. E. (1997). An overview of the EPIC architecture
for cognition and performance with application to human-computer
interaction. Human-Computer Interaction, 12:391–438
• Kosslyn, S. M. (1994). Elements of graph design. W. H. Freeman & Co.,
New York
• Kosslyn, S. M. (2006). Graph design for the eye and mind. Oxford
University Press, New York
• Laird, J. E. (2012). The Soar cognitive architecture. MIT Press, Cambridge,
Mass
• Lohse, G. L. (1993). A cognitive model for understanding graphical
perception. Human-Computer Interaction, 8:353–388
• Meyer, J. (2000). Performance with tables and graphs: effects of training
and a visual search model. Ergonomics, 43(11):1840–1865

29. References (3)
• McClelland, J. L. (2009). The place of modeling in cognitive science.
Topics in Cognitive Science, 1:11–38
• Newell, A. (1990). Unified theories of cognition. The William James
lectures. Harvard University Press
• Peebles, D. (2008). The effect of emergent features on judgments of
quantity in configural and seperable displays. Journal of Experimental
Psychology: Applied, 14(2):85–100
• Peebles, D. and Ali, N. (2009). Differences in comprehensibility between
three-variable bar and line graphs. In Proceedings of the Thirty-first Annual
Conference of the Cognitive Science Society, pages 2938–2943, Mahwah,
NJ. Lawrence Erlbaum Associates
• Peebles, D. and Cheng, P. C.-H. (2003). Modeling the effect of task and
graphical representation on response latency in a graph reading task.
Human Factors, 45:28–45

30. References (4)
• Pinker, S. (1981). A theory of graph comprehension. Occasional Paper 10,
Center for Cognitive Science, Massachusetts Institute of Technology
• Shah, P. and Carpenter, P. A. (1995). Conceptual limitations in
comprehending line graphs. Journal of Experimental Psychology: General,
124:43–62
• Tabachneck-Schijf, H. J. M., Leonardo, A. M., and Simon, H. A. (1997).
CaMeRa: A computational model of multiple representations. Cognitive
Science, 21:305–350
• Thomas, J. J. and Cook, K. A. (2005). Illuminating the path: The research
and development agenda for visual analytics. IEEE Press
• Zacks, J. and Tversky, B. (1999). Bars and lines: A study of graphic
communication. Memory and Cognition, 27(6):1073–1079