Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Robert Gordon talk, 21 March 2018


Published on

Invited talk given to the Socio-Technical Systems Research Group. School of Computing Science and Digital Media, Robert Gordon University.

Published in: Science
  • Be the first to comment

  • Be the first to like this

Robert Gordon talk, 21 March 2018

  1. 1. Multiple representations and visual mental imagery in artificial cognitive systems David Peebles Reader in Cognitive Science Department of Psychology March 21, 2018
  2. 2. Outline of the talk Part 1 Multi-representation cognition Visual mental imagery Part 2 Cognitive architectures & the Common Model of Cognition Multiple representations in cognitive architectures
  3. 3. A bit of context and a caveat. . . PhD, Uni. Birmingham Experimental psychology, connectionist modelling Postdoc, Uni. Nottingham Diagrammatic Reasoning (Peter Cheng & Nigel Shadbolt), ACT-R University of Huddersfield Cognitive modelling ACT-R cognitive architecture Reasoning with external representations This talk relates to ongoing ‘multi-representation cognition’ project with Peter Cheng (Sussex) Paper at recent AAAI workshop ‘A Standard Model of the Mind’ (Peebles & Cheng, 2017) Still a work in progress and my thinking is not fully developed
  4. 4. Multi-representation cognition Modern human cognition is multi-representational External (task environment) representations: Languages (natural and formal) Diagrams Maps Tables Menus and tool bars in computer applications Control panels Specialised abstract notation systems in academic and technical domains
  5. 5. Internal mental representations Abstract, ‘amodal’, descriptive propositional representations Depictive representations grounded in perception Preserve explicitly information about topological and geometric relations among problem components. Information format, operators, information indexing methods, heuristics and goal structures can differ considerably with alternative representations (Larkin & Simon, 1987). An example Imagine a square with sides of one unit. At opposite corners of the square add circles with radii 2 3 of a unit centred on the corners. Do the two circles: 1. overlap? 2. just touch? 3. not touch?
  6. 6. Mathematics-based solution 1. “According to Pythagoras’ theorem, the length of the diagonal between the two opposite corners is the square root of 2”. 2. “That’s about 1.4 units so if we divide that by 2, the centre of the square is about 0.7 units from each corner”. 3. “The radius of each circle is about 0.66 units, so neither circle’s perimeter will reach the centre of the square”. 4. “Therefore the circles do not touch”.
  7. 7. Imagery-based solution 1. “I’m imagining a square and I can see that circles with unit radius will definitely overlap; in fact they intersect each other at the other corners of the square”. 2. “Now I’m imagining circles of 1 2 unit and I can see that they clearly don’t meet. In fact the circles cross the mid-point of each side of the square and curve away from the centre”. 3. “Now I’m thinking of circles with radii of 2 3 . It’s hard to be certain how big they should be, but they seem to just touch each other”.
  8. 8. Visual mental imagery and visual working memory Two solutions rely upon different mental representations Declarative and mathematical Visuo-spatial and exploiting imagery in the mind’s eye. Visual Mental Imagery (VMI). “Representations that produce the experience of seeing in the absence of visual input” “Imagery debate” Pylyshyn All thoughts, including VMI, are propositional. Kosslyn VMI is an internal, non-perceptual visual experience caused either by recollecting or conceptualising something. VMI are structurally analogous to visual representations, and are caused, at least in part, by psychological processes shared with the visual system. VMI has a functional role in planning, (e.g., simulating actions, particularly when potential costs of error are high).
  9. 9. Processes involved in using visual mental imagery Generation (from knowledge in LTM) Maintenance (attention) Inspection, scanning (attention) Transformation and manipulation Translation Rotation Scaling, zooming Restructuring and reinterpretation Synthesis Composition (e.g., intersection, union, subtraction) Key question What form of internal representation allows these computational processes to be carried out efficiently? Symbolic/numerical or array-based?
  10. 10. More general questions How can information from different senses, at different levels of abstraction, be fluidly used in decision making? What functional role does specialised spatial and visual processing play in cognition? How are spatial, visual and abstract symbolic representations and processes integrated? What forms of representation are required (necessary and sufficient) to support human-level capabilities and performance? Do visual and spatial cognition (and visual imagery) demand non-symbolic, depictive representational formats and operators?
  11. 11. Cognitive architectures Originated in 1950s but active research programme in 1980s. Cognitive science – differs from mainstream “narrow” AI and traditional “divide and conquer” approach of experimental cognitive psychology Theories of the core, immutable structures and processes of the human cognitive system. Aim: general, human level intelligence modelling human cognition and performance – broad applicability to wide range of tasks. Addresses fundamental question of how cognitive, perceptual, and motor processes interact and integrate to produce complex, real-world behaviour. Not simply theoretical constructs but actual running software systems, often with vision and motor control.
  12. 12. An emerging standard model Several cognitive architectures in existence (20–30) Two dominant: ACT-R and Soar (both approx. 30 years old). Much consolidation and convergence over last decade. ‘Common Model of Cognition’ (Laird, Lebiere & Rosenbloom, 2017) Perception Action Task Environment Learning Procedural Perceptual Learning Learning Declarative Selection Action Short−term memory Long−term memory Procedural Long−term memory Declarative
  13. 13. Symbolic approaches to spatial reasoning Architectures come from traditional symbolic AI tradition Often ad hoc mechanisms not intrinsic to the architecture Most use only descriptive representations CogSketch (Forbus, Usher, Lovett, Lockwood & Wetzel, 2011) Diagram Representation System (DRS) (Chandrasekaran, Kurup, Banerjee, Josephson & Winkler, 2004) ACT-R (Peebles, 2013; Peebles & Cheng, 2003) Quebec Mississippi 0 10 20 30 40 50 60 70 80 90 100 q q q q Plant CO2 Uptake as a function of Plant Type and Treatment PlantCO2Uptake Plant Type Treatment Chilled Non−chilled
  14. 14. Attempts at array-based representations Some architectures have explored non-symbolic, array-based representations. Computation with Multiple Representations (CaMeRa) model (Tabachneck-Schijf, Leonardo & Simon, 1997)
  15. 15. Retinotopic Reasoning (R2) architecture Aims to model the computational properties of mental imagery (Kunda, McGreggor & Goel, 2013). Based, in large part, on array based (non-symbolic) representations and operators. Successfully applied to: (a) Raven’s Progressive Matrices test, (b) Embedded Figures test, (c) Block Design test, and (d) Paper Folding test.
  16. 16. Retinotopic Reasoning (R2) architecture Similar to CaMeRa (Tabachneck-Schijf et al., 1997)
  17. 17. Soar/SVS Soar/SVS (Spatial Visual System) (Lathrop, Wintermute & Laird, 2011; Wintermute, 2012)
  18. 18. Soar/SVS Soar/SVS (Spatial Visual System) (Lathrop et al., 2011; Wintermute, 2012)
  19. 19. Similarities in these non-symbolic approaches Employ representations consisting of two-dimensional arrays of pixels. Operators to manipulate array objects. Forward and backward connections to higher-level (numerical, symbolic) representations (CaMeRa, Soar/SVS) Important in that they allow cognitive modelling of processes akin to those used in visual mental imagery. None explicitly address the issue of multiple representation cognition
  20. 20. Processes involved in using multiple representations Initial selection of representations Coordination of simultaneous representations Switching asynchronously between representations Distribution of task information between representations, across task sub-goals and time Understanding computational costs of each representation Potential for cognitive off-loading User’s familiarity with each representation Compatibility of different representations
  21. 21. Metacognitive knowledge and processes Monitoring and control processes to handle the selection and monitoring of, transitions between, and integration of different representations. Meta-level information about the characteristics of different representational formats (e.g., level of precision afforded, ease of computation, suitability for a given problem etc.). Use–and be able to choose between–alternative representations within the same modality (e.g., different types of diagram).
  22. 22. ACT-R ACT-R (Anderson, 2007) purely symbolic Visual/Spatial information represented as numbers and symbols. Insufficent to model visual mental imagery Array module currently under development Visual Module ACT−R Buffers Environment Pattern Matching Execution Production Module Motor Problem State Declarative Memory Procedural Memory Control State
  23. 23. Implications for cognitive models and artificial human-like agents 1. Models must incorporate alternative problem representations. 2. Must incorporate some form of meta-cognitive monitoring and control processes to handle the selection and monitoring of, transitions between, and integration of different representations. 3. Must be able to incorporate meta-level information about the characteristics of different representational formats (e.g., level of precision afforded, ease of computation, suitability for a given problem etc.). 4. Must also be able to incorporate–and be able to choose betweenalternative representations within the same modality.
  24. 24. References I Anderson, J. R. (2007). How can the human mind occur in the physical universe? New York, NY: Oxford University Press. Chandrasekaran, B., Kurup, U., Banerjee, B., Josephson, J. R. & Winkler, R. (2004). An architecture for problem solving with diagrams [Lecture notes in artificial intelligence 2980]. In A. Blackwell, K. Marriott & A. Shimojima (Eds.), Diagrammatic representation and inference (pp. 235–256). Berlin: Springer-Verlag. Forbus, K., Usher, J., Lovett, A., Lockwood, K. & Wetzel, J. (2011). CogSketch: Sketch understanding for cognitive science research and for education. Topics in Cognitive Science, 3(4), 648–666. Kunda, M., McGreggor, K. & Goel, A. K. (2013). A computational model for solving problems from the Raven’s Progressive Matrices intelligence test using iconic visual representations. Cognitive Systems Research, 22, 47–66.
  25. 25. References II Laird, J. E., Lebiere, C. & Rosenbloom, P. S. (2017). A standard model of the mind: Toward a common computational framework across artificial intelligence, cognitive science, neuroscience, and robotics. AI Magazine. 38(4). Larkin, J. H. & Simon, H. A. (1987). Why a diagram is (sometimes) worth ten thousand words. Cognitive Science, 11, 65–100. Lathrop, S. D., Wintermute, S. & Laird, J. E. (2011). Exploring the functional advantages of spatial and visual cognition from an architectural perspective. Topics in Cognitive Science, 3(4), 796–818. Peebles, D. (2013). Strategy and pattern recognition in expert comprehension of 2×2 interaction graphs. Cognitive Systems Research, 24, 43–51. Peebles, D. & Cheng, P. C.-H. (2003). Modeling the effect of task and graphical representation on response latency in a graph reading task. Human Factors, 45, 28–45.
  26. 26. References III Peebles, D. & Cheng, P. C.-H. (2017, September 11). Multiple representations in cognitive architectures. In AAAI Fall Symposium 2017: ‘‘A Standard Model of the Mind”. FS-17-01–FS-17-05. American Association for the Advancement of Artificial Intelligence. Washington, VA. Tabachneck-Schijf, H. J. M., Leonardo, A. M. & Simon, H. A. (1997). CaMeRa: A computational model of multiple representations. Cognitive Science, 21, 305–350. Wintermute, S. (2012). Imagery in cognitive architecture: Representation and control at multiple levels of abstraction. Cognitive Systems Research, 19, 1–29.