1. Multiple representations and visual mental
imagery in cognitive architectures
David Peebles
Reader in Cognitive Science
Department of Psychology
June 7, 2018

2. Outline of the talk
Cognitive architectures & the Common Model of Cognition
Attempts to model visual mental imagery
An update on the very long history of representation as it now
applies to current work in cognitive architectures
Multiple representation cognition (Peebles & Cheng, 2017)
Apologies if some of this is familiar to you

3. Cognitive architectures
Theories of the core, immutable structures and processes of
the human cognitive system.
Computer architecture
Perception Action
Task
Environment
Learning
Procedural
Perceptual
Learning
Learning
Declarative
Selection
Action
Short−term memory
Long−term memory
Procedural
Long−term memory
Declarative
Human cognitive architecture
Combine AI methods and cognitive psychology theory.
How cognition, perception, and motor processes interact and
integrate to produce complex, real-world behaviour.
Running software systems.

4. An emerging standard model
49 architectures currently under active development
(Kotseruba & Tsotsos, 2018).
Two dominant: ACT-R and Soar (both approx. 30 years old).
Much consolidation and convergence over past decade.
‘Common Model of Cognition’ (Laird, Lebiere, & Rosenbloom,
2017)

5. Representations in cognitive architectures
Cognitive architectures
come from symbolic,
propositional AI tradition (cf.
Pylyshyn).
colour(square, dark-cyan)
colour(circle, violet-red)
left-of(square, circle)
Descriptive
representations. Discrete,
amodal, symbolic or
numerical entities.
Human visual mental
imagery uses depictive
array-like representations
and visual areas in occipital
lobe (Kosslyn, Thompson, &
Ganis, 2006).
Impossible to determine
which representations are
being used (Anderson,
1978)

6. Symbolic approaches to spatial reasoning
Most use only symbolic descriptive representations
CogSketch (Forbus, Usher, Lovett, Lockwood, & Wetzel, 2011)
Diagram Representation System (DRS) (Chandrasekaran,
Kurup, Banerjee, Josephson, & Winkler, 2004)
ACT-R (Peebles, 2013; Peebles & Cheng, 2003)
Much spatial reasoning is qualitative though
Often ad hoc mechanisms not intrinsic to the architecture
Quebec Mississippi
0
10
20
30
40
50
60
70
80
90
100
q q
q
q
Plant CO2 Uptake as a function of Plant Type and Treatment
PlantCO2Uptake
Plant Type
Treatment
Chilled
Non−chilled

7. Early attempts at array-based representations
Computational Imagery (Glasgow & Papadias, 1992)
Computation with Multiple Representations (CaMeRa) model
(Tabachneck-Schijf, Leonardo, & Simon, 1997)

8. Retinotopic Reasoning (R2) architecture
Aims to model the computational properties of mental imagery
(Kunda, McGreggor, & Goel, 2013).
Based, in large part, on array based (non-symbolic)
representations and operators.
Successfully applied to: (a) Raven’s Progressive Matrices
test, (b) Embedded Figures test, (c) Block Design test, and (d)
Paper Folding test.

9. Retinotopic Reasoning (R2) architecture
Similar to CaMeRa (Tabachneck-Schijf et al., 1997)

10. Soar/SVS
Soar/SVS (Spatial/Visual System) (Lathrop, Wintermute, &
Laird, 2011; Wintermute, 2012)
“[M]ental imagery if functionally essential to spatial and visual
cognition”.

11. Soar/SVS
Soar/SVS (Spatial/Visual System) (Lathrop et al., 2011;
Wintermute, 2012)

12. Similarities in these non-symbolic approaches
Employ representations consisting of two-dimensional arrays
of pixels.
Operators to manipulate array objects (affine transformations,
composition etc.).
Forward and backward connections to higher-level (numerical,
symbolic) representations (CaMeRa, Soar/SVS)
Important in that they allow cognitive modelling of processes
akin to those used in visual mental imagery.
None explicitly address the issue of multiple representation
cognition

13. Multi-representation cognition
An example
Imagine a square with sides of one unit. At opposite corners
of the square add circles with radii 2
3 of a unit centred on the
corners. Do the two circles:
1. overlap?
2. just touch?
3. not touch?

14. Mathematics-based solution
1. “According to Pythagoras’ theorem, the length of the diagonal
between the two opposite corners is the square root of 2”.
2. “That’s about 1.4 units so if we divide that by 2, the centre of
the square is about 0.7 units from each corner”.
3. “The radius of each circle is about 0.66 units, so neither
circle’s perimeter will reach the centre of the square”.
4. “Therefore the circles do not touch”.

15. Imagery-based solution
1. “I’m imagining a square and I can
see that circles with unit radius will
definitely overlap; in fact they
intersect each other at the other
corners of the square”.
2. “Now I’m imagining circles of 1
2 unit
and I can see that they clearly
don’t meet. In fact the circles cross
the mid-point of each side of the
square and curve away from the
centre”.
3. “Now I’m thinking of circles with
radii of 2
3 . It’s hard to be certain
how big they should be, but they
seem to just touch each other”.

16. Multi-representation cognition
Two solutions rely upon different mental representations
Declarative and mathematical
Visuo-spatial and exploiting imagery in the mind’s eye.
Understanding computational costs of each representation
Potential for cognitive off-loading
User’s familiarity with each representation
Compatibility of different representations
Metacognitive knowledge and processes
Monitoring and control processes to handle the selection and
monitoring of, transitions between, and integration of different
representations.
Meta-level information about the characteristics of different
representational formats (e.g., level of precision afforded, ease
of computation, suitability for a given problem etc.).
Initial selection of representations

17. Conclusions
1. The age-old debate about representations underlying imagery
are still going—and has a new impetus.
2. Most cognitive architectures remain with just symbolic
representations (probably for pragmatic reasons rather than
deep theoretical commitments).
3. New attempts to integrate non-symbolic, ‘depictive’,
array-based representations into architectures.
4. The use (or not) of mental imagery requires meta-cognitive
knowledge and processes which need to be incorporated into
cognitive architectures (Peebles & Cheng, 2017).

18. References I
Anderson, J. R. (1978). Arguments concerning representations for
mental imagery.. Psychological Review, 85(4), 249.
Chandrasekaran, B., Kurup, U., Banerjee, B., Josephson, J. R., &
Winkler, R. (2004). An architecture for problem solving with
diagrams [Lecture notes in artificial intelligence 2980]. In A.
Blackwell, K. Marriott, & A. Shimojima (Eds.), Diagrammatic
representation and inference (pp. 235–256). Berlin:
Springer-Verlag.
Forbus, K., Usher, J., Lovett, A., Lockwood, K., & Wetzel, J. (2011).
CogSketch: Sketch understanding for cognitive science
research and for education. Topics in Cognitive Science,
3(4), 648–666.
Glasgow, J. & Papadias, D. (1992). Computational imagery.
Cognitive Science, 16(3), 355–394.
Kosslyn, S. M., Thompson, W. L., & Ganis, G. (2006). The case for
mental imagery. Oxford University Press.

19. References II
Kotseruba, I. & Tsotsos, J. K. (2018). A review of 40 years of
cognitive architecture research: Core cognitive abilities and
practical applications. arXiv preprint, abs/1610.08602.
Retrieved from http://arxiv.org/abs/1610.08602
Kunda, M., McGreggor, K., & Goel, A. K. (2013). A computational
model for solving problems from the raven’s progressive
matrices intelligence test using iconic visual representations.
Cognitive Systems Research, 22, 47–66.
Laird, J. E., Lebiere, C., & Rosenbloom, P. S. (2017). A standard
model of the mind: Toward a common computational
framework across artificial intelligence, cognitive science,
neuroscience, and robotics. AI Magazine. 38(4).
Lathrop, S. D., Wintermute, S., & Laird, J. E. (2011). Exploring the
functional advantages of spatial and visual cognition from an
architectural perspective. Topics in Cognitive Science, 3(4),
796–818.

20. References III
Peebles, D. (2013). Strategy and pattern recognition in expert
comprehension of 2×2 interaction graphs. Cognitive
Systems Research, 24, 43–51.
Peebles, D. & Cheng, P. C.-H. (2003). Modeling the effect of task
and graphical representation on response latency in a graph
reading task. Human Factors, 45, 28–45.
Peebles, D. & Cheng, P. C.-H. (2017, September 11). Multiple
representations in cognitive architectures. In AAAI fall
symposium 2017: ‘‘a standard model of the mind”.
FS-17-01–FS-17-05. American Association for the
Advancement of Artificial Intelligence. Washington, VA.
Tabachneck-Schijf, H. J. M., Leonardo, A. M., & Simon, H. A.
(1997). CaMeRa: A computational model of multiple
representations. Cognitive Science, 21, 305–350.
Wintermute, S. (2012). Imagery in cognitive architecture:
Representation and control at multiple levels of abstraction.
Cognitive Systems Research, 19, 1–29.