1.
Engage and be Engaging:
Building Compassionate and
Personalized Conversational
Systems
Rama Akkiraju
IBM Distinguished Engineer

2.
Please note IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice and
at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general productdirection and it should
not be relied on in making a purchasing decision.
The information mentionedregarding potential future products is not a commitment, promise, or legal obligation to
deliver any material, code or functionality. Information about potential futureproducts may not be incorporated into
any contract.
The development, release, and timing of any future features or functionality described for our products remains at
our sole discretion.
Performance is based on measurements and projections using standard IBM benchmarks in a controlled
environment. The actual throughput or performance that any user will experience will vary depending upon many
factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O
configuration, the storage configuration, and theworkload processed. Therefore, no assurance can be given that
an individual user will achieve results similar to those stated here.
11/21/16World of Watson 2016 2

3.
Consider this
conversation
Presentation Title
8/11/16
Watson
Developer
Conference
2016 3
What kind of cuisine do
you prefer?
What’s a good place to
eat in San Francisco?
Are you a vegetarian?
Italian, or Mexican, or
Indian, or Chinese
Do you like a fine dining
experience or fast food?
Yes
Are you driving, taking a
taxi or using public
transportation?
Fine dining
Do you prefer a
restaurant that has a
parking lot of its own, or
are you okay to find
street parking?
Driving
I can recommend three
choices: Greens on
Marina, Gracious Madre
on Mission, and Enjoy
Chinese on Kearney…
Parking lot would be
good
Works! But takes too
many turns to arrive at
a solution and solution
is not personalized

4.
Consider this
revised
conversation
Presentation Title
8/11/16
Watson
Developer
Conference
2016 4
What’s a good place to
eat in San Francisco?
Better! Takes far fewer
turns and offers much
more personalized
experience
What kind of cuisine do
you prefer?
Joan, because you are
a vegetarian, open to
experiences, prefer fine
dining, and are likely to
drive given you are
currently renting a car,
I’d recommend Greens,
a fusion vegetarian
Restaurant on 2 Marina
Blvd. It is a 4.1/5-­rated
vegetarian restaurant
with ample parking and
is within 1 mile from
your current location,
so you could even walk
to it.
Italian, or Mexican, or
Indian, or Chinese
Cognitive assistant is
demonstrating
understanding of
Joan’s preferences,
personality, life events
and personalizing
answers accordingly.

5.
Consider this
conversation
Presentation Title
8/11/16
Watson
Developer
Conference
2016 5
There are proven,
effective ways to deal
with bullying and bullies.
Would you like me to
help you?
I’m getting bullied in
school. I’m so upset.
Are you facing verbal,
physical or social
bullying?
Yes. Please.
Where has this been
happening?
I guess, social and
verbal bullying. I can’t
focus on my studies. My
situation is so hopeless.
Why do you think they
are bullying you?
On the school bus,
during recess, after
school, during projects –
everywhere
Have you tried avoiding
those who are bullying
you?
I think it’s because I’m
overweight. They call
me names. It’s so
demeaning.
Here is a short video on
how to deal with bullying.
This can help you.
I can’t. they are my
teammates. I have to do
a project with them.
Works! But lacks
compassion

6.
Consider this
revised
conversation
Presentation Title
8/11/16
Watson
Developer
Conference
2016 6
Kathy, I’m so sorry to
hear that you are facing
bullying in school.
Don’t feel distressed.
You are not alone in
this fight. There are
proven, effective ways
to deal with bullying
and bullies. I’m here to
listen and to help you in
any way I can. Would
you like me to help
you?
I’m getting bullied in
school. I’m so upset.
Are you facing verbal,
physical or social
bullying?
Yes. Please.
Oh! That is indeed
unfortunate! Don’t lose
hope, Kathy. You can
manage the situation.
I’m here to help you.
First, let me ask you
this. Where has the
bullying been
happening?
I guess, social and
verbal bullying. I can’t
focus on my studies. My
situation is so hopeless.
Better! Compassionate
companion

7.
Consider this
revised
conversation
Presentation Title
8/11/16
Watson
Developer
Conference
2016 7
I see. Why do you think
they are bullying you?
On the school bus,
during recess, after
school, during projects –
everywhere
I’m sorry to hear that.
That is indeed very
insensitive of them to
say that. Have you tried
avoiding those who are
bullying you?
I think it’s because I’m
overweight. They call
me names. It’s so
demeaning.
I see. Kathy, I can help
you in the following
ways. I have access to
some videos that may
help you. After you had
a chance to watch them
and grasp the
techniques suggested,
if you wish, we can also
do some role play. That
way, you can be
prepared to deal with
them next time you face
them.
I can’t. They are my
teammates. I have to do
a project with them.
Cognitive assistant is
demonstrating
understanding of
Kathy’s emotional state
and is working with her
in a compassionate
manner.
Better! Compassionate
companion

8.
To build compassionate and
personalized conversational systems,
three core models are needed:
Presentation Title
8/11/16
Watson
Developer
Conference
2016 8
Naturally3
Interact2
People1
Understand people at a deeper level
Understand and respond in
various mediums in which
interactions can occur
Interact2 naturally3 with people1
Input types: Text, speech, gestures
Mediums: Computers, mobile
devices, robots, avatars
Understand styles of human interaction
and optimize human-­computer interaction

9.
Building block 1:
People modeling/
user modeling
Naturally3
Interact2
People1

10.
People are
complex beings
with many facets
“Who are these
people?”
“How do they
make
decisions?”
“What do they
value?”
“How do they
think?”
“What is their
context?”
“What are
their
preferences?”
“What are
their
emotions?”
“How do they
act?”
“How are they
feeling?”
“What are their
skills?”
“How do they
express
themselves?”

11.
User modeling:
Our framework
An individual takes action based on the combination of his/her unique
being and environment
Act
Be
Feel
Context
Think Options
Explore
&
Decide
Inner State Environment Outer State

12.
User modeling: Our framework
Act
Search
Preferences
Communications Decisions
Commitments
Purchases
Context
Life Style, Events Sociological
Economic
Political Technological
Options
Price Promotions
Products/Services Place
Feel
Perceptions
Emotions
Sensations
Attitudes
Influences
Sentiments
Be
Personality
Needs, Values
Beliefs
Motives
Identity
Goals, Ambitions
Interests
Think
Knowledge
Skills
Opinions
Cognitive Style
Explore
&
Decide
Choices Consequences
Session
Intent
Time

13.
11/21/16
Watson
Developer
Conference
201613
Ready to use Still in R&D

14.
Use Personality
Insights to engage
with individuals at
personalized level
11/21/16
Watson
Developer
Conference
201614
Watson-­pi-­demo.mybluemix.net
Source: https://www.army.mil/article/78562/Leaving_the_battlefield__Soldier_shares_story_of_PTSD

15.
Emotional
Analysis helps
build empathetic
systems
11/21/16
Watson
Developer
Conference
201615
sentiment-­and-­emotion.mybluemix.net

16.
Use Tone
Analyzer to
understand and
fine tune your
message
11/21/16
Watson
Developer
Conference
201616
tone-­analyzer-­demo.mybluemix.net

17.
Building block 2:
Human interaction
patterns
Naturally3
Interact2
People1

18.
Natural
interactions
among people
11/21/16
Watson
Developer
Conference
201618
Verbal (expressive, aggressive, passive) ,
Non-­verbal (gestures, facial expressions, postures)

19.
Dialog strategies Start
Giving an
extra
Acknowledging
Need
Description of
Need
Anger
Acknowledging
w/o encouraging
Refocus
statements
Active
Listening
Possibility of
mistake Admitting
mistake
Allowing
venting
Apology
Smiles
Arranging
Follow-­up
Need cannot
be fulfilled on
the spot
Assurance of
effort
Assurance of
result
Mistake has
been made Bonus buyoff Broken record
Uncooperative
customer
Closing positively
Common
Courtesy
Completing
Follow-­up
Contact
Security
Aggressiveness Disengaging
Distraction
Frustration Empathy
statement
Expediting
Expert
Recommend
ation
Explain
Reasoning or
action
Embarrassment
Face-­Saving
Out
Conflict
Finding
Agreement Points
Following up
Helpless
Offering
Choice
Empowering
Preventive
strike
Privacy
insurance
Privacy
concern
Probing
question
Pros and
Cons
Providing
Alternatives
Providing
Takeaway
Confusion
Providing
Explanation
Questioning
instead of
stating
Referral to
supervisor
Referral to 3rd
party
Lost focus Refocus
Inappropriate
behavior
Setting Limits
Critical
Neutral mode
Summarize the
conversation
Silence
Thank-­you
Timeout
Use customer
name
Verbal
Softeners
When questionYou’re right
Action
Negative
Emotion
Monologue
End External Giving
Emotions
General
States
Gratitude
Statement
Happiness
Work by Michal Shmueli-­Scheuer,
Jonathan Herzig, Guy Feigenblat, David
Konopnicki

20.
Tone-­aware
Expressions in
Conversations
11/21/16
Watson
Developer
Conference
201620
food-­coach.
mybluemix.net

21.
11/21/16World of Watson 2016 Page 21

22.
Auto-­generated
response in
customer care
domain
11/21/16
Watson
Developer
Conference
201622
Agent
responses
are auto-­
generated.

23.
Bot personas
and expressive
paraphrasing
11/21/16
Watson
Developer
Conference
201623

24.
Personalized
conversational
systems
11/21/16
Watson
Developer
Conference
201624
Based on customer’s emotion and personality the cognitive bot
personalizes responses
Detect user emotion
Bot understands user’s
personality and
preferences and
provides
recommendations
Change bot expression
based on user emotion

25.
Building block 3:
Understand various mediums
in which human-­computer
interaction can occur
Naturally3
Interact2
People1

26.
Tone-­aware
Expressions in
Conversations
11/21/16
Watson
Developer
Conference
201626
Speech-­to-­Text Text-­to-­Speech
speech-­to-­text-­
demo.mybluemix.net
text-­to-­speech-­
demo.mybluemix.net

27.
Deriving human
expressions from
speech, images
and video
Speech
– Non-­verbal clues: pauses,
volume, intonation, pitch
11/21/16
Watson
Developer
Conference
201627
Video, images
– Gestures
– Facial expressions
– Eye contact
– Posture
– Tone of voice
– Distance

28.
Bringing it all together:
Building compassionate and
personalized conversational
systems
Naturally3
Interact2
People1

29.
Deep-­dive:
Watson Food-­Coach App
Demonstrates how to
incorporate tone and emotion
detection into Conversations
and to design compassionate
dialog flow
29

30.
11/21/16World of Watson 2016 Page 30
Intents
Entities

31.
Food Coach Dialog Flow
31

32.
Building Compassionate and
Personalized Conversational Agents
11/21/16
Watson
Developer
Conference
201632
Naturally3
Interact2
People1
• We are developing the platform and the
building blocks for implementing
compassionate and personalized
conversational agents.
• Checkout our services, Build your
conversational apps and give us your
feedback!

33.
Notices and
disclaimers
continued
Information concerning non-­IBM products was obtained from the suppliers of those products, their published announcements or other
publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of
performance, compatibility or any other claims related to non-­IBM products. Questions on the capabilities of non-­IBM products should be
addressed to the suppliers of those products. IBM does not warrant the quality of any third-­party products, or the ability of any such third-­
party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED,
INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE.
The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents,
copyrights, trademarks or other intellectual property right.
IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document
Management System™, FASP®, FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM
SmartCloud®, IBM Social Business®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON,
OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®,
pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ,
Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-­Force® and System z® Z/OS, are trademarks of
International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be
trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at:
www.ibm.com/legal/copytrade.shtml.
33 11/21/16World of Watson 2016

34.
Thank you
11/21/16
Watson
Developer
Conference
201634