IBM is developing Project Intu to enable embodied cognition by placing Watson's cognitive abilities into robots, avatars, objects, and spaces. This would allow Watson to perceive the physical world using senses like vision, hearing, and touch. It would also allow Watson to act in the physical world using effectors like limbs and facial expressions. The goal is for Watson to understand and reason about the world in more natural, human-like ways in order to augment human capabilities.

1. © 2016 IBM Corporation 1
Embodied Cognition with Project Intu
Grady Booch
Chief Architect, Watson/M
gbooch@us.ibm.com
@grady_booch

2. © 2016 IBM Corporation 2
Imagine unleashing Watson in the physical world. Give it
eyes, ears, and touch, then let it act in that world with hands
and feet and a face, not just as an action of force but also as
an action of influence. This is embodied cognition: by placing
the cognitive power of Watson in a robot, in an avatar, an
object in your hand, or even in the walls of an operating room,
conference room, or spacecraft, we take Watson's ability to
understand and reason and draw it closer to the natural ways
in which humans live and work. In so doing, we augment
individual human senses and abilities, giving Watson the
ability see a patient's complete medical condition, feel the
flow of a supply chain, or orchestrate the tasks in a day in the
life of an individual.

3. © 2016 IBM Corporation 3
Embodied Cognition Defined
§ An embodied cognition must
– Be in and of the world
– Reason
– Learn
– Have identity
§ An embodied cognition is not
– Simply an STT -> NLC -> action pipeline

4. © 2016 IBM Corporation 4
An Agent May Be Embodied In A Robot…

5. © 2016 IBM Corporation 5
An Avatar…

6. © 2016 IBM Corporation 6
A Space…

7. © 2016 IBM Corporation 7
Or An Object

8. © 2016 IBM Corporation 8
Embodied Cognition
8
Use Cases
Concierge
(Robot, Avatar,
Space)
Where is the elevator?
Retail
(Robot, Avatar,
Space)
Do you wanna build a snowman?
Elder Care
(Robot, Avatar,
Space)
I’ve fallen and I can’t get up!
Cobot
(Robot)
Get me a screwdriver.
Manufacturing
(Robot)
Watch me do this.
Transportation
(Robot, Avatar,
Space)
Open the pod bay doors, Watson.
Boardroom
(Avatar, Space)
Help me decide. Companion
(Avatar, Device)
Let’s play a game.

9. © 2016 IBM Corporation 9
Points of Technical Confluence
Who said that?
Sensory Fusion
I feel sad.
Theory of Mind
I need the Phillips head screwdriver.
Context
Give me that one; no, I mean that one!
Embodied
Conversation
It’s good to see you again, Sandia!
Learning
I’ve just picked up a fault in the AE35 unit.
Devices
In what room was Alyssa working yesterday?
Knowledge
Representation

10. © 2016 IBM Corporation 10
Self Basic Principles
§ Augment human capabilities
§ Learn, don’t program
§ Theory of mind and social intelligence
§ Self understanding
§ Platform agnostic
§ Embodiment as a robot, an avatar, a device, or a space
§ Deployment as middleware with microservices in the cloud

11. © 2016 IBM Corporation 11
Self Significant Design Decisions
§ Self is a hybrid architecture, encompassing explicit symbolic computation in the center together with neural networks at
the edges.
§ Inspired by Minsky’s Society of Mind, behavior takes place in the context of multiple concurrent agents that
communicate opportunistically via blackboards and deterministically via peer to peer connections.
§ Inspired by Brooks’s subsumption architecture, behavior takes place in a hierarchy of cognition, from involuntary
reflexes to voluntary skills to goals and planning.
§ We maintain a clear separation of concerns among perception, actuating, models, and behavior.
§ As much as possible, behavior is either taught or is learned, not programmed.
§ As much as possible – driven by these separation of concerns, the needs of packaging, and performance – all
components are made manifest as RESTful microservices.
§ As much as possible, plans, skills, and reflexes are extensible.
§ Self is intentionally full of strange loops: components of Self are also parts of the models of itself.
§ Self is intentionally fractal: an instance of Self may have models of others, which themselves are other instances of
Self.

12. © 2016 IBM Corporation 12
Self
12
Sensors &
Percep,on
Actuators
Agents
Goals & Planning
Pla5orm
Management Infrastructure
Meta
Management
Models
Voluntary Behavior & Skills
Involuntary Behavior &
Reflexes
Others
World
Self
Architecture

13. © 2016 IBM Corporation 1313
Project Intu
An Experimental platform for Embodied Cognition
Intu is about behavior -- about creating presence; about
giving the cognitive mind a body through which it can
express itself with intelligent relevance
Coordinates cognitive services seamlessly
Framework for tying multiple services together with different external sensors,
actuators, and services
(i.e. Speaker/Microphone with STT/Conversation Services)
Connects & manages various devices for scale
Ability to extend cognitive services to new devices bringing reasoning and learning
in the physical world with minimum coding
(i.e. Avatars and Devices)
Expands access
Capability to connect disparate and trained knowledge sources together and
integrate 3rd-party services for robust experience
(i.e. Weather API or Nexmo API, etc.)
Simplifies and enhances cognitive services
Pre-trained capabilities or “Behaviors” that work across various devices and
operating systems enrich and create an immersive experience
(i.e. Emotion and Gestures)
Ibm.biz/ProjectIntu
avatars In-the-walls
devices
spaces wearables
robotics

14. © 2016 IBM Corporation 14
Project Intu
An Experimental platform for Embodied Cognition
Ibm.biz/ProjectIntu

15. © 2016 IBM Corporation 15
Project Intu
An Experimental platform for Embodied Cognition
Ibm.biz/ProjectIntu

16. © 2016 IBM Corporation 16
Embodied Cognition
Core Technology Topic Project Name Lab
Goals & Planning
Planning Yorktown
Embodied Action Almaden
Multi-modal Contextual Interaction Yorktown
Ethical Embodied Decision Support Systems Yorktown
Models of the World, Others, and Self
Spatial Intelligence Zurich, Yorktown
People Intelligence Haifa
Theory of Mind Yorktown
Learning Teachable Agents Yorktown
Sensors & Perception/Actuators
Physical Interaction Tokyo
Edge Intelligence Australia
Platform & Meta Management Tooling Yorktown
Solution Area Project
Robots, Avatars, & Spaces Watson Labs
Robots & Spaces Watson IoT
Robots Industrial Robots
Spaces Elder Care
Spaces M&A
Objects Cognitive Objects

17. © 2016 IBM Corporation 17
Imagine unleashing Watson in the physical world. Give it
eyes, ears, and touch, then let it act in that world with hands
and feet and a face, not just as an action of force but also as
an action of influence. This is embodied cognition: by placing
the cognitive power of Watson in a robot, in an avatar, an
object in your hand, or even in the walls of an operating room,
conference room, or spacecraft, we take Watson's ability to
understand and reason and draw it closer to the natural ways
in which humans live and work. In so doing, we augment
individual human senses and abilities, giving Watson the
ability see a patient's complete medical condition, feel the
flow of a supply chain, or orchestrate the tasks in a day in the
life of an individual.