Multiple representations and visual mental
imagery in cognitive architectures
Reader in Cognitive Science
Department of Psychology
June 7, 2018
Outline of the talk
Cognitive architectures & the Common Model of Cognition
Attempts to model visual mental imagery
An update on the very long history of representation as it now
applies to current work in cognitive architectures
Multiple representation cognition (Peebles & Cheng, 2017)
Apologies if some of this is familiar to you
Theories of the core, immutable structures and processes of
the human cognitive system.
Human cognitive architecture
Combine AI methods and cognitive psychology theory.
How cognition, perception, and motor processes interact and
integrate to produce complex, real-world behaviour.
Running software systems.
An emerging standard model
49 architectures currently under active development
(Kotseruba & Tsotsos, 2018).
Two dominant: ACT-R and Soar (both approx. 30 years old).
Much consolidation and convergence over past decade.
‘Common Model of Cognition’ (Laird, Lebiere, & Rosenbloom,
Representations in cognitive architectures
come from symbolic,
propositional AI tradition (cf.
amodal, symbolic or
Human visual mental
imagery uses depictive
and visual areas in occipital
lobe (Kosslyn, Thompson, &
Impossible to determine
which representations are
being used (Anderson,
Symbolic approaches to spatial reasoning
Most use only symbolic descriptive representations
CogSketch (Forbus, Usher, Lovett, Lockwood, & Wetzel, 2011)
Diagram Representation System (DRS) (Chandrasekaran,
Kurup, Banerjee, Josephson, & Winkler, 2004)
ACT-R (Peebles, 2013; Peebles & Cheng, 2003)
Much spatial reasoning is qualitative though
Often ad hoc mechanisms not intrinsic to the architecture
Plant CO2 Uptake as a function of Plant Type and Treatment
Early attempts at array-based representations
Computational Imagery (Glasgow & Papadias, 1992)
Computation with Multiple Representations (CaMeRa) model
(Tabachneck-Schijf, Leonardo, & Simon, 1997)
Retinotopic Reasoning (R2) architecture
Aims to model the computational properties of mental imagery
(Kunda, McGreggor, & Goel, 2013).
Based, in large part, on array based (non-symbolic)
representations and operators.
Successfully applied to: (a) Raven’s Progressive Matrices
test, (b) Embedded Figures test, (c) Block Design test, and (d)
Paper Folding test.
Retinotopic Reasoning (R2) architecture
Similar to CaMeRa (Tabachneck-Schijf et al., 1997)
Soar/SVS (Spatial/Visual System) (Lathrop, Wintermute, &
Laird, 2011; Wintermute, 2012)
“[M]ental imagery if functionally essential to spatial and visual
Similarities in these non-symbolic approaches
Employ representations consisting of two-dimensional arrays
Operators to manipulate array objects (afﬁne transformations,
Forward and backward connections to higher-level (numerical,
symbolic) representations (CaMeRa, Soar/SVS)
Important in that they allow cognitive modelling of processes
akin to those used in visual mental imagery.
None explicitly address the issue of multiple representation
Imagine a square with sides of one unit. At opposite corners
of the square add circles with radii 2
3 of a unit centred on the
corners. Do the two circles:
2. just touch?
3. not touch?
1. “According to Pythagoras’ theorem, the length of the diagonal
between the two opposite corners is the square root of 2”.
2. “That’s about 1.4 units so if we divide that by 2, the centre of
the square is about 0.7 units from each corner”.
3. “The radius of each circle is about 0.66 units, so neither
circle’s perimeter will reach the centre of the square”.
4. “Therefore the circles do not touch”.
1. “I’m imagining a square and I can
see that circles with unit radius will
deﬁnitely overlap; in fact they
intersect each other at the other
corners of the square”.
2. “Now I’m imagining circles of 1
and I can see that they clearly
don’t meet. In fact the circles cross
the mid-point of each side of the
square and curve away from the
3. “Now I’m thinking of circles with
radii of 2
3 . It’s hard to be certain
how big they should be, but they
seem to just touch each other”.
Two solutions rely upon different mental representations
Declarative and mathematical
Visuo-spatial and exploiting imagery in the mind’s eye.
Understanding computational costs of each representation
Potential for cognitive off-loading
User’s familiarity with each representation
Compatibility of different representations
Metacognitive knowledge and processes
Monitoring and control processes to handle the selection and
monitoring of, transitions between, and integration of different
Meta-level information about the characteristics of different
representational formats (e.g., level of precision afforded, ease
of computation, suitability for a given problem etc.).
Initial selection of representations
1. The age-old debate about representations underlying imagery
are still going—and has a new impetus.
2. Most cognitive architectures remain with just symbolic
representations (probably for pragmatic reasons rather than
deep theoretical commitments).
3. New attempts to integrate non-symbolic, ‘depictive’,
array-based representations into architectures.
4. The use (or not) of mental imagery requires meta-cognitive
knowledge and processes which need to be incorporated into
cognitive architectures (Peebles & Cheng, 2017).
Anderson, J. R. (1978). Arguments concerning representations for
mental imagery.. Psychological Review, 85(4), 249.
Chandrasekaran, B., Kurup, U., Banerjee, B., Josephson, J. R., &
Winkler, R. (2004). An architecture for problem solving with
diagrams [Lecture notes in artiﬁcial intelligence 2980]. In A.
Blackwell, K. Marriott, & A. Shimojima (Eds.), Diagrammatic
representation and inference (pp. 235–256). Berlin:
Forbus, K., Usher, J., Lovett, A., Lockwood, K., & Wetzel, J. (2011).
CogSketch: Sketch understanding for cognitive science
research and for education. Topics in Cognitive Science,
Glasgow, J. & Papadias, D. (1992). Computational imagery.
Cognitive Science, 16(3), 355–394.
Kosslyn, S. M., Thompson, W. L., & Ganis, G. (2006). The case for
mental imagery. Oxford University Press.
Kotseruba, I. & Tsotsos, J. K. (2018). A review of 40 years of
cognitive architecture research: Core cognitive abilities and
practical applications. arXiv preprint, abs/1610.08602.
Retrieved from http://arxiv.org/abs/1610.08602
Kunda, M., McGreggor, K., & Goel, A. K. (2013). A computational
model for solving problems from the raven’s progressive
matrices intelligence test using iconic visual representations.
Cognitive Systems Research, 22, 47–66.
Laird, J. E., Lebiere, C., & Rosenbloom, P. S. (2017). A standard
model of the mind: Toward a common computational
framework across artiﬁcial intelligence, cognitive science,
neuroscience, and robotics. AI Magazine. 38(4).
Lathrop, S. D., Wintermute, S., & Laird, J. E. (2011). Exploring the
functional advantages of spatial and visual cognition from an
architectural perspective. Topics in Cognitive Science, 3(4),
Peebles, D. (2013). Strategy and pattern recognition in expert
comprehension of 2×2 interaction graphs. Cognitive
Systems Research, 24, 43–51.
Peebles, D. & Cheng, P. C.-H. (2003). Modeling the effect of task
and graphical representation on response latency in a graph
reading task. Human Factors, 45, 28–45.
Peebles, D. & Cheng, P. C.-H. (2017, September 11). Multiple
representations in cognitive architectures. In AAAI fall
symposium 2017: ‘‘a standard model of the mind”.
FS-17-01–FS-17-05. American Association for the
Advancement of Artiﬁcial Intelligence. Washington, VA.
Tabachneck-Schijf, H. J. M., Leonardo, A. M., & Simon, H. A.
(1997). CaMeRa: A computational model of multiple
representations. Cognitive Science, 21, 305–350.
Wintermute, S. (2012). Imagery in cognitive architecture:
Representation and control at multiple levels of abstraction.
Cognitive Systems Research, 19, 1–29.