Have you watched the movie Her? Have you ever wondered or wished to have your own AI companion just like Samantha, who could understand you better than you know about yourself, and could tell you what you really are, whom your best partner may be, and which career path would be best for you? In this talk, I will present a computational framework for building responsible and empathetic Artificial Intelligent (AI) agents who can deeply understand their users as unique individuals and responsibly guide their behavior in both virtual and real world. Starting with a live demo of showing how an AI interviewer chats with a user to automatically derive his/her personality characteristics and provide personalized recommendations, I will highlight the technical advances of the framework in two aspects. First, I will present a computational, evidence-based approach to Big 5 personality inference, which enables an AI agent to deeply understand a user’s unique characteristics by analyzing the user’s chat text on the fly. Second, I will describe a model-based conversation engine that couples deep learning with rules to support a natural conversation and rapid customization of a conversational agent. I will describe the initial applications of our AI agents in the real world, from talent selection to student teaming to user experience research. Finally, I will discuss the wider implications of our work on building hyper-personalized systems and their impact on our lives. Here is the Youtube video of the talk given as a keynote at ACM IUI 2019: https://www.youtube.com/watch?v=EKhs9e0yx_g

1. Ge#ng Virtually
Personal
Making Responsible and
Empathe4c “Her” for Everyone
1
Michelle Zhou
Juji, Inc.

2. Outline
• Mo=va=on
• Live Demo
• Core Technologies
• Applica=ons
• Summary
2

3. 3

4. Live Demo

5. 5
Responsible & Empathe=c Personas
(REPs)

6. empathe'c
the ability to understand and
share the feelings of another
“E” in REPs
– Ac=ve listening
– Reading between
the lines

7. “E” in REPs: An Example
The presentation I did
really flopped. And I
worked so hard on it.
That sounds really disappointing. You
seem like a very innovative person,
perhaps there is a way to improve it

8. responsible
the obliga=on to care for
someone
“R” in REPs
– Objec=ve
– Genuine

9. “R” in REPs: An Example
9
Ambitious &
eager to
impress
Humble &
Cautious
“Let’s focus on the
specifics. Facts always
speak for themselves.”
“Everyone has a gift. I’d
love to hear more of
your thoughts.”

10. Outline
• Mo=va=on
• Live Demo
• Core Technologies
• Applica=ons
• Summary
10

11. 11
Text Chat Interface
Personality
Engine
Conversa=on
Engine
GUI
IDE
REP
Generator
REP
Compiler
REP
Run=me
Creator
End User

12. 12
Text Chat Interface
Personality
Engine
Conversa=on
Engine
GUI
IDE
REP
Generator
REP
Compiler
REP
Run=me
Creator
End User

13. Reading between
the Lines
13
personality
needs
interest
mo=va=on
PERSONALITY INFERENCE

14. 14
Individual Differences
What you like to do
What you are
good at
How you handle
life’s challenges
Dimensions defining people as
unique individuals

15. 15
Power of Individual
Differences
Predict important life
outcomes
Can be developed
[Roberts et al. 2007; Edmonds et al. 2008]

16. – context insensi=ve
– scale subjec=vity
– faking due to social
desirability
16
Challenges of Tradi=onal
Measurements
I love children
I insult people
Item-based
Survey
[Fan et al. 2012; Ziegler et al. 2010]

17. Psycholinguis=cs
17
“ I love to hangout with
friends … We often go to
the beach together with ….
afraid … But I must …”
Extroversion
[Tausczik & Pennebaker 2010; Pennebaker 2011]

18. Computa=onal Psychometrics 1.0
18
LIWC 1st. Person Pl. 2nd Person . . . Numbers
β 0.11 0.16 -0.12
Extroversion
Free text
T
Regression
S ≈ f T,( β)
[Yarkoni 2010; Yang & Li 2013; Chen et al. 2014; Gou et al. 2014; Preo=uc-Pietro et al. 2016]
Survey scores

19. 19
Compu=ng Trait
Scores from Text
S = F( f (wi )× βi )
i
∑
“ … together... with ….”
“afraid .., but I must ....”
β1
β3 β5 β4
β2
[Mahmud et al. 2013, Lee et al. 2014]
Limita'on: Rely on surveys

20. Computa=onal Psychometrics 2.0
[de Ayala 2009] Item Response Theory (IRT)
B ≈ f (T)

21. T
Computa=onal Psychometrics 2.0
Item Response Theory (IRT) Generalized
B ≈ f (T)

22. Evidence-based Inference
X = uj + λjϑi +ξ
22 [Zhou et al. 2019]
occurrence rate of evidence j
Discrimina=ve power of j
Evidence (item)
Latent trait

23. Openness
Conscien'ousness
Extroversion Agreeableness
Neuro'cism
Big 5 Personality
[Digman 1990; DeYoung et al. 2007]
Competence
Orderliness
Du=fulness
Achievement-striving
Self-discipline
Cau=ousness
23

24. – Reliability
– Validity
24
Evalua=ng Quality
of Inference
[Stone 1992; de Ayala 1995; Woods 2015;
Zhou et al. 2019, Xiao et al. 2019)

25. Reading between
the Lines
Generalized IRT by text items
– Authen=c
– Portable
25
personality
needs
interest
mo=va=on
SUMMARY

26. 26
Text Chat Interface
Personality
Engine
Conversa=on
Engine
GUI
IDE
REP
Generator
REP
Compiler
REP
Run=me
Creator
End User

27. Ac=ve Listening
27
Paraphrasing
Summarizing
Guiding
[Bauer et al. 2010, Decker 1989, Gordon 1977,
Rogers & Farson 2015]
CONVERSATION UI DESIGN

28. 28
In my current job, I experience [a] lack of value
systems associated with work, i.e., sense of
ownership, taking ini4a4ve, having a sense of
urgency, being a team player, etc. I handle this
by learning …
The biggest challenge I've faced is finding a
since [sense] of purpose. Being around like
minded individuals who are constantly wan4ng
more out of life through countless jobs I've
never found …
Most challenges are met as an opportunity to
grow. Hardest part is losing friends
“
“
“
– Highly diverse
user expressions
– Ambiguous
– Uncoopera=ve
Challenges
What’s the top challenge
you are facing?

29. Exis=ng Approaches
29
Conversa=on
Management
Natural Lang
Genera=on
Natural Lang
Understanding
[Traum 2017; Williams et al. 2017; Lowe et al. 2017; Serban et al. 2016]
Challenges
– Hard to bootstrap
– Hard to port

30. 30 [Card et al. 1983; Colby 1972; Houghton 1986; Lewin 2000]
The Psychology of
Human-Computer
Interac=on
User
Machine
Time
Conversa=on
Topic
Conversa=on
Logic
Conversa=on
Acts

31. Model-based
Conversa=on UI
– handling uncertain=es
– larger space
– the “humane” factor
31 hqps://www.w3.org/TR/mbui-intro/
Abstract
Dialog
Concrete
Dialog
Personalized
Dialog
User
AI
Persona
Plasorm
Task
Final
Dialog

32. Τ = (ρ, α, εΑ, SΤ)
Conversa=on Topics
(Goal + Operator)
Conversa=on Logic
(Method)
Model-based Conversa=on UI Design

33. Conversa=on Topics
User-agent interac=ons to
achieve a goal
Τ = (ρ, α, εΑ, SΤ)
QUD + Task + State
[Grosz&Sidner 1986; Ginzburg 1995; Traum 2017]

34. Conversa=on Topics
Τ = (ρ, α, εΑ, SΤ)
“What’s your hobby”
Precondi=on
∄(hobby)
An Example

35. Conversa=on Topics
Τ = (ρ, α, εΑ, SΤ)
“What’s your hobby”
An Example
ac=ons
store()

36. Conversa=on Topics
Τ = (ρ, α, εΑ, SΤ)
“What’s your hobby”
An Example conversa=on acts
IF (= (classified-cat) PHY)
THEN (phy-response)

37. Conversa=on Topics
Τ = (ρ, α, εΑ, SΤ)
“What’s your hobby”
An Example conversa=on acts
IF
THEN
. . .

38. Conversa=on Topics
Τ = (ρ, α, εΑ, SΤ)
“What’s your hobby”
An Example
subtopics
“Why do you enjoy it”

39. Conversa=on Topics
dialog library for reuse and
auto-genera=on
Examples: hobby, challenges,
achievements, opinions
Τ = (ρ, α, εΑ, SΤ)

40. 40
Methods that thread topics
and guide a dialog process
Conversa=on Logic

41. Conversa=on Flow
41
An Example
REP as AI Interviewer
T1 T2 T3
DT2
DT1 DTN
. . .
ST1 ST2

42. Live Demo

43. Ac=ve Listening
Model-based conversa=on UI
design
– Agile
– Flexible
– Reusable (auto-genera=on)
43
Paraphrasing
Summarizing
Guiding
SUMMARY

44. Outline
• Mo=va=on
• Basic Concepts + Live Demo
• Core Technologies
• Applica'ons
• Summary
44

45. 45
Talent
Interview
Website
Concierge
Career
Counseling
User
Studies
Team
Forma=on
Personal
Guide
Customer
onboarding

46. User Studies
– 78% beqer data quality than
Qualtrics
– ease survey fa=gue
– predicted gamer DNA
46
AN EXAMPLE

47. Career Counseling
– More authen=c data
– “cheat sheet” for
human coaches
47
AN EXAMPLE

48. Outline
• Mo=va=on
• Basic Concepts + Live Demo
• Core Technologies
• Applica=ons
• Summary
48

49. 49
– Listening ac=vely
– Reading between the lines
Personalizing guidance
REPs ≈ “Her” v0.01
SUMMARY

50. Acknowledgements
• Huahai Yang
• Wenxi Chen
• Ziang Xiao
• Ransom Williams
• Amar Mehta
• Sara Birns
• Taejun Song
• Steven Landau
• Vu Anh Phung
• Yihan Zhao
• Jingyi Li
• Francis Thai
• Connie Truong
50
AFOSR FA9550-15-C-0032 | DoD I Corps Program W911NF-17-1-0599 | DigitX partners
• Tracy Chi
• Gloria Mark
• Jinyan Fan
• Wai-Tat Fu
• Vera Liao

51. References
• Gou, L., Zhou, M.X., & Yang, H. KnowMe and ShareMe: Understanding automa=cally discovered
personality traits from social media and user sharing preferences. CHI ’14, 955–964.
• Digman, J. M. Personality structure: Emergence of the five-factor model. Annual Review of Psychology
1990, 41(1): 417–440.
• B. Grosz and C. Sidner. 1986. Aqen=on, inten=on, and the structure of discourse. Comput. Ling. 12, 3
(1986), 175–204.
• R. J. de Ayala. 2009. The Theory and Prac=ce of Item Response Theory. Guilford Press.
• C. DeYoung, L. Quilty, and J. Peterson. 2007. Between facets and domains: 10 aspects of the Big Five. J.
Personality 1184 Social Psychol. 93 (2007), 880–896.
• Lucas, G., Gratch, J., King, A., & Morency, L. It’s only a computer: virtual humans increase willingness to
disclose. Comp. in Human Behavior , 2014, vol. 37: 94-100.
• Li, J., Zhou, M.X., Yang, H., and Mark, G. Confiding in and listening to virtual agents: The effect of
personality. Proc. ACM IUI 2017 , 275-286.
• Zhou, M.X., Mark, G., Li, J., and Yang, H. Trus=ng Virtual Agents: The Effect of Personality. In the special
issue IUI 2017 Highlights in ACM TiiS 9:2-3, 2019.
• Ziang Xiao, Michelle X. Zhou, Wat-Tat Fu: Who should be my teammates: using a conversa=onal agent to
understand individuals and help teaming. IUI 2019: 437-447.
• Yang Wang, Liang Gou, Anbang Xu, Michelle X. Zhou, Huahai Yang, Hernan Badenes. VeilMe: An
Interac=ve Visualiza=on Tool for Privacy Configura=on of Using Personality Traits. CHI 2015: 817-826.
51

52. References
• Y. Tausczik and J. Pennebaker. 2010. The psychological meaning of words: LIWC and computerized text
analysis methods. J. Lang. Social Psychol. 29, 1 (2010), 24–54.
• Pennebaker, J. 2011 The Secret Life of Pronouns.
• Card, S. K., Newell, A., & Moran, T. P. (1983). The Psychology of Human-Computer Interac=on.
• Roberts, B. W., Kuncel, N., Shiner, R., N., Caspi, A., & Goldberg, L. R. (2007). The power of personality: The
compara=ve validity of personality traits, socio-economic status, and cogni=ve ability for predic=ng
important life outcomes. Perspec=ves in Psychological Science, 2, 313-345.
• Edmonds, G., Jackson, J. J., Fayard, J. V., & Roberts, B.W. (2008). Is character fate, or is there hope to
change my personality yet? Social and Personality Psychology Compass, 2, 399-413.
• K. Lee, J.Mahmud, J. Chen, andM. X. Zhou, and J. Nichols. 2014.Who will retweet this? In Automa=cally
Iden=fying and Engaging Strangers on Twiqer to Spread Informa=on (IUI’14). 247–256.
• J. Mahmud, M. X. Zhou, N. Megiddo, J. Nichols, and C. Drews. 2013. Recommending targeted strangers
from whom to solicit informa=on on social media. In IUI’13. 37–48.
• D. Traum. 2017. Computa=onal approaches to dialogue. In The Routledge Handbook of Language and
Dialogue, E. Weigand (Ed.), New York.
• I. Serban, A. Sordoni, Y. Bengio, A. Courville, and J. Pineau. 2016. Building end-to-end dialogue systems
using genera=ve hierarchical neural network models. In AAAI’16. 3776–3783.
• T. Yarkoni. 2010. Personality in 100,000 words: A large-scale analysis of personality and word usage
among bloggers. J. Res. Personality 44, 3 (2010), 363–373.
52