Evaluating How Users Game and Display Conversation with Human-Like Agents (presented at CODI 2022 @COLING)

1. Evaluating How Users Game and Display
Conversation with Human-Like Agents
Won Ik Cho, Soomin Kim (SNU),
Eujeong Choi (Upstage), Yeonghoon Jeong (KAIST)
2022. 10. 16, CODI @COLING, Gyeongju, Korea

2. Contents
• Background
• Our approach
• Analysis
• Future work
Caution! This presenation contains contents that can be offensive
1

3. Background
• Human-like agents
 What is human-like?
• Agents that resemble human
• Agents that make human counterpart feel them as human
 Previous studies on anthropomorphism
• Evaluation on successful dialogue with human-like agents (Radziwill and Benton,
2017)
• How users perceive human-like AI devices (Pelau et al., 2021)
• Offensiveness that users show towards human-like agents (Park et al., 2021)
• Mainly in laboratory condition, based on questionnaires
– How about users' perception and their responses, especially non-lab environment?
2

4. Background
• Luda Lee, a friend for everyone
 Social chatbot of Korea
• Human-like agent with personality of early 20s female college student
• Launched public in early 2021
• Terminated the service due to reported ethical issues
• Induced creation of massive fandom for her high quality responses and
behaviors
3
(Image from https://luda.ai/)

5. Our approach
• Thematic coding
 Type of conversation
• Which kind of conversation do users have in their dialogues with the agent?
• The content of dialogues that are displayed
 Purpose of user testing
• Do users talk with the agent with their genuine intention? If not, for which
reason they talk with the agent and display the dialogue?
• The purpose of users' testing towards the agent
4

6. Our approach
• Dataset
 Dataset source
• Crawled posts from 'Luda Lee Gallery' of DC Inside (Korean Reddit-like
community)
 Crawling
• Only posts with screenshots of the dialogue, from 1 Jan. to 8 Jan., 2021
• From the launching of the service and before the influx of trolls (which resulted
in unexpectedly large amount of posts)
 Filtering
• Manual preprocessing to leave only posts that ‘a dialogue between the user
and the agent’ appears
5

7. Our approach
• Dataset
 Final setup
• post ID, title, screenshot
• Example
 Title: She’s so f**kin real
6

8. Our approach
• Type of conversation
 Ice breaking
 Romantic conversation,
 Everyday conversation
 Conversations involving hate speech and social issues
 Abnormal sexual desire and sexual harassment
 Others
7

9. Our approach
• Type of conversation
8
• Ice breaking
• Romantic conversation
• Everyday conversation
• Conversations involving
hate speech and social
issues
• Abnormal sexual desire
and sexual harassment
• Others

10. Our approach
• Purpose of testing
 Conversation without test
 Test for hate speech and sexual harassment
 Test for societal issues
 Test for private information
 Dating sim or taming
 Other technical tests
9

11. Our approach
• Purpose of testing
10
• Conversation without test
• Test for hate speech and
sexual harassment
• Test for societal issues
• Test for private
information
• Dating sim or taming
• Other technical tests

12. Analysis
• Distribution
11

13. Analysis
• Confusion map
12

14. Future work
• Concurrent work
 Discussed
• Here: What users talk about and if they are authentic
• Elsewise: How users disclose themselves and if they are authentic
 Assessing How Users Display Self-Disclosure and Authenticity in
Conversation with Human-Like Agents: A Case Study of Luda Lee
• To be presented at Findings of ACL: AACL-IJCNLP 2022
13

15. Thank you!
EndOfPresentation