1. Multiple representations and visual mental
imagery in artificial cognitive systems
David Peebles
Reader in Cognitive Science
Department of Psychology
March 21, 2018

2. Outline of the talk
Part 1
Multi-representation cognition
Visual mental imagery
Part 2
Cognitive architectures & the Common Model of Cognition
Multiple representations in cognitive architectures

3. A bit of context and a caveat. . .
PhD, Uni. Birmingham
Experimental psychology,
connectionist modelling
Postdoc, Uni. Nottingham
Diagrammatic Reasoning
(Peter Cheng & Nigel
Shadbolt), ACT-R
University of Huddersfield
Cognitive modelling
ACT-R cognitive
architecture
Reasoning with external
representations
This talk relates to ongoing
‘multi-representation
cognition’ project with Peter
Cheng (Sussex)
Paper at recent AAAI
workshop ‘A Standard
Model of the Mind’ (Peebles
& Cheng, 2017)
Still a work in progress and
my thinking is not fully
developed

4. Multi-representation cognition
Modern human cognition is multi-representational
External (task environment) representations:
Languages (natural and formal)
Diagrams
Maps
Tables
Menus and tool bars in computer applications
Control panels
Specialised abstract notation systems in academic and
technical domains

5. Internal mental representations
Abstract, ‘amodal’, descriptive propositional representations
Depictive representations grounded in perception
Preserve explicitly information about topological and geometric
relations among problem components.
Information format, operators, information indexing methods,
heuristics and goal structures can differ considerably with
alternative representations (Larkin & Simon, 1987).
An example
Imagine a square with sides of one unit. At opposite corners of
the square add circles with radii 2
3 of a unit centred on the
corners. Do the two circles:
1. overlap?
2. just touch?
3. not touch?

6. Mathematics-based solution
1. “According to Pythagoras’ theorem, the length of the diagonal
between the two opposite corners is the square root of 2”.
2. “That’s about 1.4 units so if we divide that by 2, the centre of
the square is about 0.7 units from each corner”.
3. “The radius of each circle is about 0.66 units, so neither
circle’s perimeter will reach the centre of the square”.
4. “Therefore the circles do not touch”.

7. Imagery-based solution
1. “I’m imagining a square and I can
see that circles with unit radius will
definitely overlap; in fact they
intersect each other at the other
corners of the square”.
2. “Now I’m imagining circles of 1
2 unit
and I can see that they clearly
don’t meet. In fact the circles cross
the mid-point of each side of the
square and curve away from the
centre”.
3. “Now I’m thinking of circles with
radii of 2
3 . It’s hard to be certain
how big they should be, but they
seem to just touch each other”.

8. Visual mental imagery and visual working memory
Two solutions rely upon different mental representations
Declarative and mathematical
Visuo-spatial and exploiting imagery in the mind’s eye.
Visual Mental Imagery (VMI). “Representations that produce
the experience of seeing in the absence of visual input”
“Imagery debate”
Pylyshyn All thoughts, including VMI, are propositional.
Kosslyn VMI is an internal, non-perceptual visual experience
caused either by recollecting or conceptualising something.
VMI are structurally analogous to visual representations, and
are caused, at least in part, by psychological processes
shared with the visual system.
VMI has a functional role in planning, (e.g., simulating actions,
particularly when potential costs of error are high).

9. Processes involved in using visual mental imagery
Generation (from knowledge in LTM)
Maintenance (attention)
Inspection, scanning (attention)
Transformation and manipulation
Translation
Rotation
Scaling, zooming
Restructuring and reinterpretation
Synthesis
Composition (e.g., intersection, union, subtraction)
Key question
What form of internal representation allows these
computational processes to be carried out efficiently?
Symbolic/numerical or array-based?

10. More general questions
How can information from different senses, at different levels
of abstraction, be fluidly used in decision making?
What functional role does specialised spatial and visual
processing play in cognition?
How are spatial, visual and abstract symbolic representations
and processes integrated?
What forms of representation are required (necessary and
sufficient) to support human-level capabilities and
performance?
Do visual and spatial cognition (and visual imagery) demand
non-symbolic, depictive representational formats and
operators?

11. Cognitive architectures
Originated in 1950s but active research programme in 1980s.
Cognitive science – differs from mainstream “narrow” AI and
traditional “divide and conquer” approach of experimental
cognitive psychology
Theories of the core, immutable structures and processes of
the human cognitive system.
Aim: general, human level intelligence modelling human
cognition and performance – broad applicability to wide range
of tasks.
Addresses fundamental question of how cognitive, perceptual,
and motor processes interact and integrate to produce
complex, real-world behaviour.
Not simply theoretical constructs but actual running software
systems, often with vision and motor control.

12. An emerging standard model
Several cognitive architectures in existence (20–30)
Two dominant: ACT-R and Soar (both approx. 30 years old).
Much consolidation and convergence over last decade.
‘Common Model of Cognition’ (Laird, Lebiere & Rosenbloom,
2017)
Perception Action
Task
Environment
Learning
Procedural
Perceptual
Learning
Learning
Declarative
Selection
Action
Short−term memory
Long−term memory
Procedural
Long−term memory
Declarative

13. Symbolic approaches to spatial reasoning
Architectures come from traditional symbolic AI tradition
Often ad hoc mechanisms not intrinsic to the architecture
Most use only descriptive representations
CogSketch (Forbus, Usher, Lovett, Lockwood & Wetzel, 2011)
Diagram Representation System (DRS) (Chandrasekaran,
Kurup, Banerjee, Josephson & Winkler, 2004)
ACT-R (Peebles, 2013; Peebles & Cheng, 2003)
Quebec Mississippi
0
10
20
30
40
50
60
70
80
90
100
q q
q
q
Plant CO2 Uptake as a function of Plant Type and Treatment
PlantCO2Uptake
Plant Type
Treatment
Chilled
Non−chilled

14. Attempts at array-based representations
Some architectures have explored non-symbolic, array-based
representations.
Computation with Multiple Representations (CaMeRa) model
(Tabachneck-Schijf, Leonardo & Simon, 1997)

15. Retinotopic Reasoning (R2) architecture
Aims to model the computational properties of mental imagery
(Kunda, McGreggor & Goel, 2013).
Based, in large part, on array based (non-symbolic)
representations and operators.
Successfully applied to: (a) Raven’s Progressive Matrices
test, (b) Embedded Figures test, (c) Block Design test, and (d)
Paper Folding test.

16. Retinotopic Reasoning (R2) architecture
Similar to CaMeRa (Tabachneck-Schijf et al., 1997)

17. Soar/SVS
Soar/SVS (Spatial Visual System) (Lathrop, Wintermute &
Laird, 2011; Wintermute, 2012)

18. Soar/SVS
Soar/SVS (Spatial Visual System) (Lathrop et al., 2011;
Wintermute, 2012)

19. Similarities in these non-symbolic approaches
Employ representations consisting of two-dimensional arrays
of pixels.
Operators to manipulate array objects.
Forward and backward connections to higher-level (numerical,
symbolic) representations (CaMeRa, Soar/SVS)
Important in that they allow cognitive modelling of processes
akin to those used in visual mental imagery.
None explicitly address the issue of multiple representation
cognition

20. Processes involved in using multiple representations
Initial selection of representations
Coordination of simultaneous representations
Switching asynchronously between representations
Distribution of task information between representations,
across task sub-goals and time
Understanding computational costs of each representation
Potential for cognitive off-loading
User’s familiarity with each representation
Compatibility of different representations

21. Metacognitive knowledge and processes
Monitoring and control processes to handle the selection and
monitoring of, transitions between, and integration of different
representations.
Meta-level information about the characteristics of different
representational formats (e.g., level of precision afforded,
ease of computation, suitability for a given problem etc.).
Use–and be able to choose between–alternative
representations within the same modality (e.g., different types
of diagram).

22. ACT-R
ACT-R (Anderson, 2007) purely symbolic
Visual/Spatial information represented as numbers and
symbols.
Insufficent to model visual mental imagery
Array module currently under development
Visual
Module
ACT−R Buffers
Environment
Pattern
Matching
Execution
Production
Module
Motor
Problem
State
Declarative
Memory
Procedural
Memory
Control
State

23. Implications for cognitive models and artificial
human-like agents
1. Models must incorporate alternative problem representations.
2. Must incorporate some form of meta-cognitive monitoring and
control processes to handle the selection and monitoring of,
transitions between, and integration of different
representations.
3. Must be able to incorporate meta-level information about the
characteristics of different representational formats (e.g., level
of precision afforded, ease of computation, suitability for a
given problem etc.).
4. Must also be able to incorporate–and be able to choose
betweenalternative representations within the same modality.

24. References I
Anderson, J. R. (2007). How can the human mind occur in the
physical universe? New York, NY: Oxford University Press.
Chandrasekaran, B., Kurup, U., Banerjee, B., Josephson, J. R. &
Winkler, R. (2004). An architecture for problem solving with
diagrams [Lecture notes in artificial intelligence 2980]. In A.
Blackwell, K. Marriott & A. Shimojima (Eds.), Diagrammatic
representation and inference (pp. 235–256). Berlin:
Springer-Verlag.
Forbus, K., Usher, J., Lovett, A., Lockwood, K. & Wetzel, J. (2011).
CogSketch: Sketch understanding for cognitive science
research and for education. Topics in Cognitive Science,
3(4), 648–666.
Kunda, M., McGreggor, K. & Goel, A. K. (2013). A computational
model for solving problems from the Raven’s Progressive
Matrices intelligence test using iconic visual representations.
Cognitive Systems Research, 22, 47–66.

25. References II
Laird, J. E., Lebiere, C. & Rosenbloom, P. S. (2017). A standard
model of the mind: Toward a common computational
framework across artificial intelligence, cognitive science,
neuroscience, and robotics. AI Magazine. 38(4).
Larkin, J. H. & Simon, H. A. (1987). Why a diagram is (sometimes)
worth ten thousand words. Cognitive Science, 11, 65–100.
Lathrop, S. D., Wintermute, S. & Laird, J. E. (2011). Exploring the
functional advantages of spatial and visual cognition from an
architectural perspective. Topics in Cognitive Science, 3(4),
796–818.
Peebles, D. (2013). Strategy and pattern recognition in expert
comprehension of 2×2 interaction graphs. Cognitive
Systems Research, 24, 43–51.
Peebles, D. & Cheng, P. C.-H. (2003). Modeling the effect of task
and graphical representation on response latency in a graph
reading task. Human Factors, 45, 28–45.

26. References III
Peebles, D. & Cheng, P. C.-H. (2017, September 11). Multiple
representations in cognitive architectures. In AAAI Fall
Symposium 2017: ‘‘A Standard Model of the Mind”.
FS-17-01–FS-17-05. American Association for the
Advancement of Artificial Intelligence. Washington, VA.
Tabachneck-Schijf, H. J. M., Leonardo, A. M. & Simon, H. A.
(1997). CaMeRa: A computational model of multiple
representations. Cognitive Science, 21, 305–350.
Wintermute, S. (2012). Imagery in cognitive architecture:
Representation and control at multiple levels of abstraction.
Cognitive Systems Research, 19, 1–29.