Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Believe it or not: Designing a Human-AI
Partnership for Mixed-Initiative Fact-Checking
Joint work with
An Thanh Nguyen (UT...
“The place where people & technology meet”
~ Wobbrock et al., 2009
“iSchools” now exist at over 65 universities around the...
Information Literacy
National Information Literacy Awareness Month,
US Presidential Proclamation, October 1, 2009.
“Though...
“Truthiness”
“Truthiness is tearing apart our country... It used to
be everyone was entitled to their own opinion, but
not...
Journalist Fact-Checking
5Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiativ...
Danny Sullivan,
April 7, 2017
6Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Init...
Crowdsourced Fact Checking
7
Automated Fact-Checking
8Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative...
Fake News Challenge
9Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fac...
10Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
11Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
Challenges...
MemeBrowser (Ryu et al., ACM HyperText’12)
http://odyssey.ischool.utexas.edu/mb
13Matt Lease (UT Austin) • Believe it or n...
• Crowdsourced stance labels
– Hybrid AI + Human (near real-time) Prediction
• Joint model of stance, veracity, & annotato...
Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
This Work
...
Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
This Work
...
Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
Primary In...
Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
Source Rep...
Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
System Arc...
Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
Data: Trai...
Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
User Study...
Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
User Study...
Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
User Study...
Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
User Study...
Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
User Study...
Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
User Study...
Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
User Study...
Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
Before see...
Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
After seei...
Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
Study 1: S...
Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
Study 1: S...
Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
User Study...
Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
User Study...
Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
• “Slider”...
Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
Discussion...
Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
Conclusion...
Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
Thank You!...
Upcoming SlideShare
Loading in …5
×

of

Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 1 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 2 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 3 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 4 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 5 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 6 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 7 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 8 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 9 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 10 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 11 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 12 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 13 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 14 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 15 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 16 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 17 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 18 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 19 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 20 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 21 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 22 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 23 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 24 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 25 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 26 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 27 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 28 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 29 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 30 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 31 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 32 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 33 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 34 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 35 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 36 Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Slide 37
Upcoming SlideShare
What to Upload to SlideShare
Next

1 Like

Share

Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking

Presented at the 31st ACM User Interface Software and Technology Symposium (UIST), 2018. Paper: https://www.ischool.utexas.edu/~ml/papers/nguyen-uist18.pdf

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking

  1. 1. Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Joint work with An Thanh Nguyen (UT), Byron Wallace (Northeastern), & more! Matt Lease School of Information @mattlease University of Texas at Austin ml@utexas.edu Slides: slideshare.net/mattlease
  2. 2. “The place where people & technology meet” ~ Wobbrock et al., 2009 “iSchools” now exist at over 65 universities around the world www.ischools.org What’s an Information School? 2Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
  3. 3. Information Literacy National Information Literacy Awareness Month, US Presidential Proclamation, October 1, 2009. “Though we may know how to find the information we need, we must also know how to evaluate it. Over the past decade, we have seen a crisis of authenticity emerge. We now live in a world where anyone can publish an opinion or perspective, whether true or not, and have that opinion amplified…” 3Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
  4. 4. “Truthiness” “Truthiness is tearing apart our country... It used to be everyone was entitled to their own opinion, but not their own facts. But that’s not the case anymore.” – Stephen Colbert (Jan. 25, 2006) 4Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
  5. 5. Journalist Fact-Checking 5Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
  6. 6. Danny Sullivan, April 7, 2017 6Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
  7. 7. Crowdsourced Fact Checking 7
  8. 8. Automated Fact-Checking 8Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
  9. 9. Fake News Challenge 9Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
  10. 10. 10Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
  11. 11. 11Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
  12. 12. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Challenges • Fair, Accountable, & Transparent (AI) – Why trust “black box” classifier? – How do we reason about potential bias? – Do people really only want to know true vs. false? – How to integrate human knowledge/experience? • Joint AI + Human Reasoning, Correct Errors, Personalization • How to design strong Human + AI Partnerships? – Horvitz, CHI’99: mixed-initiative design – Dove et al., CHI’17 “Machine Learning As a Design Material” 12
  13. 13. MemeBrowser (Ryu et al., ACM HyperText’12) http://odyssey.ischool.utexas.edu/mb 13Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
  14. 14. • Crowdsourced stance labels – Hybrid AI + Human (near real-time) Prediction • Joint model of stance, veracity, & annotators – Interaction between variables – Interpretable • Source on github Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Nguyen et al., AAAI’18 14
  15. 15. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking This Work 15 Demo!
  16. 16. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking This Work 16 http://fcweb.pythonanywhere.com
  17. 17. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Primary Interface 17
  18. 18. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Source Reputation 18
  19. 19. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking System Architecture • Google Search API • Two logistic regression models – Stance (Ferreira & Vlachos ’16) w/ same features • average accuracy > 70% but variable over claims – Veracity (Popat et al. ‘17) – Scikit-learn, L1 regularization, Liblinear solver, & default parameters 19
  20. 20. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Data: Train & Test Emergent (Ferreira & Vlachos ’16) Accuracy of prediction models 20
  21. 21. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking User Study 1: Setup • 2 Groups: Control vs. System 21
  22. 22. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking User Study 1: Setup • 2 Groups: Control vs. System 22
  23. 23. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking User Study 1: Setup • 2 Groups: Control vs. System 23
  24. 24. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking User Study 1: Setup • 2 Groups: Control vs. System – 113 participants (58 control, 55 system) – MTurk • 1. Asked to predict claim veracity – Likert: Def. False, Prob. F., neutral, Prob. True, Def. T. – Error: distance • For a (definitely) false claim, PF -> error=1, DT -> error=4 • 2. Shown model’s claim prediction 24
  25. 25. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking User Study 1: Setup • 2 Groups: Control vs. System 25
  26. 26. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking User Study 1: Setup • 2 Groups: Control vs. System 26
  27. 27. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking User Study 1: Setup • 2 Groups: Control vs. System – 113 participants (58 control, 55 system) – MTurk • 1. Asked to predict claim veracity – Likert: Def. False, Prob. F., neutral, Prob. True, Def. T. – Error: distance • For a (definitely) false claim, PF -> error=1, DT -> error=4 • 2. Shown model’s claim prediction – Asked if they want to change their answer? 27
  28. 28. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Before seeing model’s claim prediction • “System” group: – claims 1-2: > avg. prediction error – claim 4: < error – claims 3, 5: only small differences • Human accuracy in claim prediction roughly follows model’s accuracy in stance prediction – i.e., helped when model correct, hurt when not 28
  29. 29. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking After seeing model’s claim prediction • “System” group: – Smaller changes for errors – change answers > “control” group • Human accuracy in claim prediction roughly follows model’s accuracy in veracity prediction – i.e., helped when model correct, hurt when not 29
  30. 30. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Study 1: Statistical Significance (1 of 2) • Mixed-effects Generalized Linear Model (GLM) • Before seeing model’s claim predictions – CSP: # correct stance predictions seen – WSP: # wrong stance predictions seen – 2-tail test includes unlikely possibility that seeing correct stance predictions increases human error 30
  31. 31. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Study 1: Statistical Significance (2 of 2) • Mixed-effects Generalized Linear Model (GLM) • After seeing model’s claim predictions – CSP: # correct stance predictions seen – WSP: # wrong stance predictions seen – 2-tail test includes unlikely possibility that seeing correct stance predictions increases human error 31
  32. 32. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking User Study 2: Setup • 2 Groups: Control vs. Slider 32
  33. 33. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking User Study 2: Setup • 2 Groups: Control vs. Slider – 109 participants (51 control, 58 slider) – MTurk • 1. Asked to predict claim veracity – Likert: Def. False, Prob. F., neutral, Prob. True, Def. T. and indicate confidence in prediction – (-20,+20) score range: accuracy x confidence • eg, a correct answer with 75% confidence -> 20x75%=15 33
  34. 34. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking • “Slider” group: – Claims 1-3: > score on average – Claims 4-5: < first quartiles, but medians same • Some participants negatively impacted by slider – No statistically significant difference on average 34 User Study 2: Results
  35. 35. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Discussion & Future Work • Fact Checking & IR (Lease, DESIRES’18) – How to diversify search results for controversial topics? – Information evaluation (eg, vaccination & autism) • Potential harm as well as good – Potential added confusion, data / algorithmic bias – Potential for personal “echo chamber” – Adversarial applications • Future Work – Making personalization more visible – Collaborative use, small and big • 1st author Nguyen looking for a postdoc! 35
  36. 36. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Conclusion • Fact-checking more than black-box prediction – Interaction, exploration, trust • We proposed a mixed-initiative human + AI partnership for fact-checking – Back-end AI + front-end interface/interaction – Support AI + human collaboration – Fair, Accountable, & Transparent (FAT) AI 36
  37. 37. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Thank You! Slides: slideshare.net/mattlease Lab: ir.ischool.utexas.edu 37
  • AnubrataDas

    Oct. 18, 2018

Presented at the 31st ACM User Interface Software and Technology Symposium (UIST), 2018. Paper: https://www.ischool.utexas.edu/~ml/papers/nguyen-uist18.pdf

Views

Total views

1,186

On Slideshare

0

From embeds

0

Number of embeds

16

Actions

Downloads

0

Shares

0

Comments

0

Likes

1

×