Cognitive assistance at work

© 2015 IBM Corporation
Hamid R. Motahari-Nezhad
IBM Almaden Research Center
San Jose, CA
Cognitive Assistance at Work
Cognitive Assistant for Employees and Citizens
AAAI 2015 Fall Symposium

2
Workers and Work Management
https://www.centrodeinnovacionbbva.com/en/innovation-edge/social-business/social-trend
Onboarding, Orientation and Growth
Communicationand
interactions
WorkandProject
Management
Knowledge Worker

The Work Practices of Human Administrative Assistants
 Human assistant activities
–Calendaring
• Scheduling, information formatting and preparation
–Task Management
–Email Management
• Filtering emails,
• Email classification
 Interruption management
–Mediating interruption
–Prioritizing interruptions
 Taking care of routine tasks
–Tracking
–Following up
–Travel arrangement, and preparation
–Reminding, and organizing
–Managing work of human
• Pre-processing
• Filtering
• Prioritizing
• Compiling information and reports
3
An assistant “will remove much of the burden of administrative chores from its human user and
provide guidance, advice, and assistance in problem solving and decision making.” Gutierrze and Hilfdalgo,
1988

© 2013 IBM Corporation4
Credit: Rob Koplowitz, IBM Insight 2015

COGNITIVE ASSISTANCE
5

Cognitive Assistant
 A software agent (cog) that
– “augments human intelligence” (Engelbart’s definition1 in 1962)
– Performs tasks and offer services (assists human in decision making and taking actions)
– Complements human by offering capabilities that is beyond the ordinary power and reach of human (intelligence
amplification)
 A more technical definition
– Cognitive Assistant offers computational capabilities typically based on Natural Language Processing (NLP),
Machine Learning (ML), and reasoning chains, on large amount of data, which provides cognition powers that
augment and scale human intelligence
 Getting us closer to the vision painted for human-machine partnership in 1960:
– “The hope is that, in not too many years, human brains and computing machines will be coupled together very
tightly, and that the resulting partnership will think as no human brain has ever thought and process data in a way
not approached by the information handling machines we know today”
“Man-Computer Symbiosis , J. C. R. Licklider IRE Transactions on Human Factors in Electronics, volume HFE-1,
pages 4-11, March 1960
6 1 Augmenting Human Intellect: A Conceptual Framework, by Douglas C. Engelbart, October 1962

History of Cognitive Assistants from the lens of AI
7
1945
Memex (Bush)
1962
NLS/Augment
(Engelbart)
1955/6
Logic Theorist
(Newwell, Simon, 1955)
Checker Player
(Samuel, 1956)
Touring Test,
1950
Thinking machines
1966
Eliza
(Weizenbaum)
1965-1987 DENDRAL
1974-1984 MYCIN
1987 Cognitive Tutors
(Anderson)
Apple’s Knowledge
Navigator System
Expert Systems
1965-1987 1992-1998
Virtual Telephone
Assistant
Portico, Wildfire,
Webley;
Speech Recognition
Voice Controlled
2002-08
DARPA PAL
Program
CALO
IRIS

Modern Cognitive Assistants: State of the art (2008-present)
Commercial
 Personal Assistants and Bots
– Siri, Google Now, Microsoft
Cortana, Amazon Echo, FB M
– Braina, Samsung's S Voice, LG's
Voice Mate, SILVIA, HTC's Hidi,
Nuance’ Vlingo
– AIVC, Skyvi, IRIS, Everfriend,
Evi (Q&A), Alme (patient
assistant)
– Viv (Global Brain as a Service)
– x.ai, Telegram bots
 Cognitive and intelligent
platforms
– IBM Watson
– Wolfram Alpha
– Saffron 10
– Vicarious (Captcha)
Open Source/Research
 OAQA
 DeepDive
 OpenCog
 YodaQA
 OpenSherlock
 OpenIRIS
 iCub EU projects
 Cougaar
 Inquire* (intelligent textbook)
8
* Curated knowledge base

A Society of Interacting Cognitive Agents (Bots) and Humans
9
Cognitive
Agent to
Agent
Human to
Human
Cognitive
Agent to
HumanHuman-Cog interaction
Cog-Cog interaction
Cog-mediated Human Interaction
Natural Language
Natural Language, or ACL?
ACL: Agent Communication Language, KQML, etc.
Natural Language-ACL-Natural Language
Weather
Cog
Health
Agent
Personality
Insight Cog.
Provider
Cogs
Travel Cog 1
Travel Cog 2
Planning a Vacation
Trip
Considering preferences,
experience, conditions, cost,
Availability, etc.
Mediated and facilitated by Cogs

A major challenge towards offering cognitive assistance: Building knowledge
models, and knowledge acquisition
 “For an artifact, a computational intelligence, to be able to behave with high levels of performance on complex intellectual
tasks, perhaps surpassing human level, it must have extensive knowledge of the domain”
 The challenge of AI in making progress toward building human-like artifacts:
– Knowledge representation, and (especially) knowledge acquisition
 Approaches
– Build a large knowledge base by reading text
– Distilling from the WWW a huge knowledge base
 Semantic Web and Linked Data methods over the last decade extensively has explored building models, ontologies and
rule-set that contributes to WWW knowledge representation
– Manual and semi-automated, focused on curated ontologies
– Community participation in building ontologies have resulted in creation of large knowledge bases: DBPedia, Yago,
Wikidata, Freebase, MediaWiki, etc.
– Ontologies are expensive to build and scale, and are generic in nature
10
EDWARD A. FEIGENBAUM, Some Challenges and Grand Challenges for
Computational Intelligence, Journal of the ACM, Vol. 50, No. 1, January 2003, pp. 32–40

Lesson Learned from Watson in Jeopardy
 “The Watson program is already a breakthrough technology in AI. For many years it had been largely assumed that for a
computer to go beyond search and really be able to perform complex human language tasks it needed to do one of two
things: either it would “understand” the texts using some kind of deep “knowledge representation,” or it would have a complex
statistical model based on millions of texts.”
– James Hendler, Watson goes to college: How the world’s smartest PC will revolutionize AI, GigaOm, 3/2/2013
 Breakthrough:
– Developing a systematic approach for scalable knowledge model building from large, less reliable data sources, and
deploying a large array of individually imperfect techniques to find right answers
• Building and curating a robust, and comprehensive knowledge base and ruleset is laborious, time consuming and
slow
• Watson approach for building on massive, mixed curated and not-curated and less reliable information sources with
uncertainty has proved effective
11
Source:
Inquire Intelligent
Book

Towards Mass Computing as the Shared Characteristic of Recent Computing
Advances
12
Scalable Computing over
Massive Commodity Hardware
Building Stronger
Super Computers
Cloud Computing
Crowd Computing
Advanced individual
algorithms
Mass computing applied to AI Complex array of algorithms applied to make
sense of data, and offer cognitive assistance
Big
Data
Complex
Analytics

COGNITIVE ASSISTANT FOR WORKERS
13

eAssistant: a cognitive assistant for the enterprise
 A mobile intelligent assistant for the enterprise that assist a user (worker) to be
more productive by supporting following a methodology of monitor, process,
recommend and do actions with the following capabilities
– Understands human language
– Monitors input channels including email, calendar chat and enterprise
information sources
– Builds a model of the user and the world, and is situational aware (context)
– Offer assistance by
• Pre-processing information, and presenting information in desired format
• Categorizing and filtering information
• Gathering and organizing information
• Scheduling meetings
• Identifying requests, and organizing to-dos of its human subject
• Assists in performing tasks such as organizing events, travel assistant, and
learns new tasks
• And, suggest taking certain actions to its human subject that supports
increasing productivity, and growth
14

eAssistant: Cognitive Assistant Types in Work Environment
 Personal (employee) eAssistant
– Personal eAssistants have access to the data space (and applications) that the principal has access to with the same
level of visibility
– While eAssistant is proactive in making suggestions, it takes action under the control and direction of the principal
 Assistant’s eAssistant
– An assistant to Human Assistants helping them to become more productive, and focus on work that require human
judgment
 Expert/Process eAssistants
– Assistants that are experts in a specific domain such as travel policy, human resources, etc.
15
Cognitive Assistant Platform
Individual cognitive
agents
Assistant’s Cognitive
Agents
Expert Cognitive
Agents
Systems of cognitive agents that
collaborate effectively with one
another to support human activities.
Interactions types need to be supported:
• cog-to-cog interactions,
• human-cog interactions, and
• cog-backed human-to-human interactions

Actionable Statement Identification Over Unstructured Conversations
16
Email, Chat, and Calendaring apps are
the most used channels for doing work
in the enterprise
Addressing the work organization and
management for Knowledge workers:
monitoring communication channels (email,
chat), and:
- capturing, prioritizing and organizing work
of a worker
- Identifying actionable statements
(requests, commitments, questions) and
track them over the course of
conversations

Inbox - Verse Highlighting actionable statements Recommending fulfilment actions
IBM Insight 2015 – The session on “Given your collaboration tools a brain”

Send File Action Archetype Send File Action Archetype Send File Action Archetype

Invite/Calendar Action Archetype Automated Invite Parameters Extraction Calendar Entry Creation

Integration with Watson Health Integration with Watson Health Integration with Watson Health

eAssistant App and APIs
21
Watson (& BigInsight NLP) Apps and Services on BlueMix
CollaborationTools
Enterprise Repositories, Applications and Data Sources
Feeds
Repositories
Document
collections
…
eAssistant Apps
Personal
Knowledge
Graph Builder
Conversation Analytics,
Auto-Response,
Prioritization
Calendar and
Scheduling
Assistant
Context-aware
Information
Finder
To-do, Task
and Process
Assistant
Cognitive Work Assistant APIs
Semantic Role
Labeling
POS tagging
Dependency
Analysis
Co-reference
resolution
Named Entity
Recognition
Knowledge
Graph
Builder

COGNITIVE BPM
Cognitive Assistance for Case Management
22

Cognitive assistance in case management for knowledge workers
 Knowledge workers are involved in handling cases in the work context (in domains such as social care, legal,
government services, citizen services, etc.).
 Cognitive case management is about providing cognitive support to knowledge workers in handling customer cases.
 Cognitive assistance for employees: Handling and managing cases involves understanding policies, laws, rules,
regulations, processes, plans, as well as customers, surrounding world, news, social networks, etc.
 Cognitive assistance for customers/agents: Assists citizens by empowering them by knowing their rights and
responsibilities, and helping them to expedite the progress of the case
23
Citizens
Assistant
Business
Employees/
agents
Plansworkflows
Rules
Policies
Regulations
Templates
Instructions/
Procedures
ApplicationsSchedules
Communications such as
email, chat, social media,
etc.
Organization
Cog. Agent
Unstructured Linked Information

Spectrum of work, and Cognitive Assistance
24
Ref: Motahar-Nezhad, Swanson, 2013, and Sandy Kemsley, Column2
Spectrum of Work
Cognitive
BPM

Cognitive BPM Systems
 A Cognitive BPM system offers the computational capability of a cognitive system to provide
support the whole lifecycle of processes over structured and unstructured information sources,
and continuously discovers, learns and acts to achieve a desired process outcome
– Meets two pressing needs: supporting complex process decisions, and processing large
amount of data
– Intelligent and integrated process (model) definition, reasoning and adaptation
• Process is not assumed apriori defined; discovered, learned and customized based on
accumulated knowledge and experience
– Intelligence supporting the execution of process
• When, What action (how) and whom to contact
– The need for revisiting some basic abstractions of BPM
• Real-world, and real-time course of actions
• New information availability changes course of actions in a plan
• Fluid actions/tasks, notion of task completion, and process/plan adaptation
25

Process Definition,
Discovery, Learning
Process
Enactment
Process/
Environment
Sensing
Process
Analytics
Proactive/ Reactive
Process
Response/Adaptation
Cognitive BPM Lifecycle
26
Environment Sensing
Data
sources
Data Processing/
Analytics
Process
Composition /
Enactment Update
Process
Monitoring/Analytics
IoT

Use Case: Cognitive Case Assistance
 Assume an executive admin is managing an event organization process for their department
– Step 1: sending invite to an event to employees in their department, through email and requests for RSVP
• Cognitive BPM (1): Q&A ability for the admin: How many have confirmed, how many pending, how
many not answered
• Cognitive BPM (2): Predictive analytics: how many will eventually RSVP?
• Cognitive BPM (3): Diagnostic analytics: why some not accepted (customers in case of marketing
case)?
– Step 2: Ordering place, food, transportation, etc
• Cognitive BPM (1): tracking of the process steps, which vendor have replied, which ones pending,
have questions, etc.
• Cognitive BPM (2): keeping track of synchronization and consistency (dates, amounts, numbers, etc.)
among different steps
– Step 3: Pre-event steps
• Reminding people who have RSVPed
• Compiling and sending logistic information (from different steps)
27

Research in Support of Cognitive BPM in Work Assistant Space
Task, commitment and task lifecycle extraction from workers interactions over email and chat
28
Anup K. Kalia, Hamid R. Motahari Nezhad, Claudio Bartolini, Munindar P. Singh: Monitoring Commitments in People-Driven Service Engagements. IEEE SCC
2013: 160-167

Research Directions
 Abstractions and models for Cognitive Work Assistants and Cognitive Processes
 Knowledge representation models, and scalable knowledge acquisition methods from
unstructured information (text, image, etc.) and building actionable knowledge graphs
 Cognitive Work Assistants
–Cognitive augmentation of workers in work environments, and in process management
 Cognitive Process Management System
–Analytics on unstructured information to support process understanding
–Analytics to support process adaptation, customization and configuration
–Proactive process adaptation
 Learning and teaching tasks and processes to cognitive agents
–Interactive learning where cognitive agents ask process questions
–Gradual learning through experience, and process improvement
29

QUESTIONS?
Thank You!
30

Cognitive assistance at work

More Related Content

What's hot

Viewers also liked

Similar to Cognitive assistance at work

Recently uploaded

Cognitive assistance at work