Embodied Cognition - Booch HICSS50

IBM Confidential | Do Not Distribute © 2016 IBM Corporation 1
Embodied Cognition
Grady Booch

Imagine unleashing Watson in the physical world. Give it
eyes, ears, and touch, then let it act in that world with hands
and feet and a face, not just as an action of force but also as
an action of influence. This is embodied cognition: by placing
the cognitive power of Watson in a robot, in an avatar, an
object in your hand, or even in the walls of an operating room,
conference room, or spacecraft, we take Watson's ability to
understand and reason and draw it closer to the natural ways
in which humans live and work. In so doing, we augment
individual human senses and abilities, giving Watson the
ability see a patient's complete medical condition, feel the
flow of a supply chain, or orchestrate the tasks in a day in the
life of an individual.

Embodied Cognition Defined
§ An embodied cognition must
– Be in and of the world
– Reason
– Learn
– Have identity
§ An embodied cognition is not
– Simply an STT -> NLC -> action pipeline

Robotics
Raw curiosity
More
computational
power
Advances in AI
algorithms and
architectures
Better/smaller
sensors and
actuators
Better batteries
3D printing
Advances in neural implants
New military
doctrine
Demographic
changes
Economic need
Advances in
neural implants
Technology push
Demand pull
Raw curiosity

An Agent May Be Embodied In A Robot…

Dependent Autonomous

Fixed place Mobile

Thing-facing
Human-
facing

Predictable Responsive Social

Architecturally, robots may be classified along one of several dimensions

Biologically-
organized
Symbolically-
organized

Thin Client Thick Client

Thin Client Thick Client
By way of context, Watson is largely a symbolically-
organized thin client cognitive system
Biologically-
organized
Symbolically-
organized

An Avatar…

A Space…

Or An Object

Embodied Cognition
17
Use Cases
Concierge
(Robot, Avatar,
Space)
Where is the elevator?
Retail
(Robot, Avatar,
Space)
Do you wanna build a snowman?
Elder Care
(Robot, Avatar,
Space)
I’ve fallen and I can’t get up!
Cobot
(Robot)
Get me a screwdriver.
Manufacturing
(Robot)
Watch me do this.
Transportation
(Robot, Avatar,
Space)
Open the pod bay doors, Watson.
Boardroom
(Avatar, Space)
Help me decide. Companion
(Avatar, Device)
Let’s play a game.

Self
18
Sensors &
Percep,on
Actuators
Agents
Goals & Planning
Pla5orm
Management Infrastructure
Meta
Management
Models
Voluntary Behavior & Skills
Involuntary Behavior &
Reﬂexes
Others
World
Self
Architecture

Points of Technical Confluence
19
Collaboration
Who said that?
Sensory Fusion
I feel sad.
Theory of Mind
I need the Phillips head screwdriver.
Context
Give me that one; no, I mean that one!
Embodied
Conversation
It’s good to see you again, Sandia!
Learning
I’ve just picked up a fault in the AE35 unit.
Devices
In what room was Alyssa working yesterday?
Knowledge
Representation

Embodied Cognition
§ Team
‒ Leader: Grady Booch; Co-Leader: Jeff Kephart
‒ Key Global Team Members: Sam Adams, Rachel Bellamy, Ray Chancey, Jon Connell, Stefan Hamer,
Kohichi Kajitani, Sky Matthews, Nat Mills, Robert Moore, Anton Riabov, Francesca Rossi, Martin Rufli,
Ryuki Tachibana, Hironobu Takagi, John Vergo
§ Theme
‒ Deliver a platform that makes the cognitive power of Watson manifest in the real world with sensors
and actuators, serving as an embodied persona that orchestrates the agency of that system to
augment and collaborate with humans using visual, vocal, and tactile forms of communication
‒ Deploy this platform in the form of robots, avatars, spaces, and objects
‒ Focus on higher levels of reasoning and learning
‒ Enable pluggable agents, goals, plans, skills, reflexes, classifiers, and models so as to facilitate a
vibrant developer and solution ecosystem
§ Short Term Goal (one year)
‒ Go to market with the Watson Group and Watson IoT, delivering the Self kernel, services, and
examplar solutions.
§ Long Term Goal (three years)
‒ Become the de facto industry standard for higher order embodied cognitive solutions
§ Gaps & Issues
‒ There’s more work to be done than IBM has skills or resources to do alone
‒ Maniacal focus is essential: there are many moving parts that cut across organizations boundaries
20
Executive Summary
Sensors &
Percep,on
Actuators
Agents
Goals & Planning
Pla5orm
Management Infrastructure
Meta
Management
Models
Involuntary Behavior &
Reﬂexes
Others
World
Self
Self

Embodied Cognition
Core Technology Topic Project Name Contact/Lab ~ Team Size
Platform Self Booch/Almaden & Chancey/Austin 1 + 4
Goals & Planning
Planning Riabov/Yorktown 2
Embodied Action Moore/Almaden 1
Multi-modal Contextual Interaction Bellamy/Yorktown 5
Embodied Decision Support Systems Rossi/Yorktown 5
Models of the World, Others, and
Self
Spatial Intelligence Rufli/Zurich 4
People Intelligence Hoory/Haifa 5
Spatial Awareness Kephart/Yorktown 4
Theory of Mind TBD
Learning Learning by Example Connell/Yorktown 2
Sensors & Perception/Actuators
Physical Interaction Tachibana/Tokyo 5
Edge Intelligence Hamer/Australia 2
Platform & Meta Management Tooling TBD
21
Solution Area Project Contact
Spaces Watson for Buildings/Watson for Retail TBD/cross-laboratory
Robots, Avatars, & Spaces Watson Labs Chancey/Austin
Robots & Spaces Watson IoT Matthews/Durham
Robots Industrial Robots Kajitani/Tokyo
Spaces Elder Care Takagi/Tokyo & Adams/Raleigh
Spaces M&A Vergo/Yorktown
Spaces Boson Mills/Cambridge
Objects Cognitive Objects Bellamy/Yorktown
Project Portfolio

Embodied Cognition
22
Collaborators
Solution Area Project Contact
Robots Cooperative Robots Carnegie Mellon University
Spaces Multimodal Interaction Rensselaer Polytechnic Institute
Avatars Baby/X University of Auckland
Robots TBD University of California, San Diego
Robots TBD University of California, Berkeley
Robots TBD Massachusetts Institute of Technology
Robots TBD Stanford Research Institute

Background
§ In November 2015, Rob High, Ray Chancey, and I were brainstorming about a suitable grand challenge for the 2016
Embodied Cognition GTO.
§ We settled on a very grand one: build a system to support mission control, flight operations, and the scientific agenda
of NASA’s Mission to Mars; the requirements of this mission lead to the concept of Watson in the walls (an embodied
cognition for the Orion spacecraft and the Mars habitat) together with robots and other devices powered by Watson
(namely, Robonaut and its peers).
§ Recognizing this was a 20+ year research effort, I set out to devise an architecture that would not only support this
Mission to Mars but the journey of which would yield a middleware that would attend to a variety of use cases, from
concierge to retail to elder care and many others.
§ Ray named this architecture “Self”.

Self Basic Principles
§ Augment human capabilities
§ Learn, don’t program
§ Theory of mind and social intelligence
§ Self understanding
§ Platform agnostic
§ Embodiment as a robot, an avatar, a device, or a space
§ Deployment as middleware with microservices in the cloud

Self Significant Design Decisions
§ Self is a hybrid architecture, encompassing explicit symbolic computation in the center together with neural networks at
the edges.
§ Inspired by Minsky’s Society of Mind, behavior takes place in the context of multiple concurrent agents that
communicate opportunistically via blackboards and deterministically via peer to peer connections.
§ Inspired by Brooks’s subsumption architecture, behavior takes place in a hierarchy of cognition, from involuntary
reflexes to voluntary skills to goals and planning.
§ We maintain a clear separation of concerns among perception, actuating, models, and behavior.
§ As much as possible, behavior is either taught or is learned, not programmed.
§ As much as possible – driven by these separation of concerns, the needs of packaging, and performance – all
components are made manifest as RESTful microservices.
§ As much as possible, plans, skills, and reflexes are extensible.
§ Self is intentionally full of strange loops: components of Self are also parts of the models of itself.
§ Self is intentionally fractal: an instance of Self may have models of others, which themselves are other instances of
Self.

Self Markitecture
IBM Confidential 26
Sensors &
Perception
Actuators
Agents
Goals & Planning
Platform Management Infrastructure
Meta Management
Models
Involuntary Behavior & Reflexes
Others
World
Self

Embodied cognition represents the next generation of Watson

Embodied cognition represents the extension of IT into the physical world, bringing
cognition to the edge of the Internet of Things

At this touch point we encounter the intersection of big data and human presence,
mediated by cognitive systems that reason

Text
Watson can be the back end to a variety of embodied
cognitive systems.

Text + Audio

Text + Audio
This is the current state of an embodied Watson
(the low hanging fruit, wherein platforms such as Pepper serve as animated speakers)

Text + Audio
There is much, much more that is possible
(if we extent the embodiment to encompass greater perception and action in the world)

Text + Audio + Vision

Text + Audio + Vision + Emotion

Text + Audio + Vision + Emotion + Motion

+ Physical
interaction

+ Physical
interaction
Socially-intelligent
cooperative robot

Embodied Cognition - Booch HICSS50

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (9)

Similar to Embodied Cognition - Booch HICSS50

Similar to Embodied Cognition - Booch HICSS50 (20)

More from diannepatricia

More from diannepatricia (20)

Recently uploaded

Recently uploaded (20)

Embodied Cognition - Booch HICSS50