Talk presented September 6, 2019 at the University of Texas at Austin, USA

1. Designing Human-AI Partnerships
to Combat Misinfomation
Matt Lease
School of Information
University of Texas at Austin
ml@utexas.edu
@mattlease
Slides: slideshare.net/mattlease

2. • Background: UT iSchool and Good Systems
• Prototype: A Human + AI Partnership for Misinformation
• User Bias: Detecting & Mitigating User Bias
• Wrap-up Discussion
2
Roadmap
Matt Lease (University of Texas at Austin)

3. “The place where people & technology meet”
~ Wobbrock et al., 2009
“iSchools” now exist at over 100 universities around the world
What’s an Information School?
3

4. Matt Lease (University of Texas at Austin) 4
Good Systems: 8-Year UT “Moonshot” Project
Goal: design a future of Artificial Intelligence (AI)
technologies to meet society’s needs and values.
http://goodsystems.utexas.edu

5. Misinformation: a long history
“Truthiness is tearing apart our country... It used to be,
everyone was entitled to their own opinion, but not
their own facts. But that’s not the case anymore.”
– Stephen Colbert (Jan. 25, 2006)
“You furnish the pictures and I’ll furnish the war.”
– William Randolph Hearst (Jan. 25, 1898)
5

6. Matt Lease (University of Texas at Austin)
Fact Checking by Journalists
6

7. Automatic Fact Checking (by AI)
7Matt Lease (University of Texas at Austin)

8. Matt Lease (University of Texas at Austin)
Design Challenges for AI Fact Checking
Fair, Accountable, & Transparent (“FAT”) AI
8

9. Matt Lease (University of Texas at Austin)
Design Challenges for AI Fact Checking
Fair, Accountable, & Transparent (“FAT”) AI
• Is “fact” vs. “fake” all people want to know?
9

10. Matt Lease (University of Texas at Austin)
Design Challenges for AI Fact Checking
Fair, Accountable, & Transparent (“FAT”) AI
• Is “fact” vs. “fake” all people want to know?
• Why trust a “black box” classifier?
10

11. Matt Lease (University of Texas at Austin)
Design Challenges for AI Fact Checking
Fair, Accountable, & Transparent (“FAT”) AI
• Is “fact” vs. “fake” all people want to know?
• Why trust a “black box” classifier?
• How to detect & mitigate bias (user or AI)?
11

12. Matt Lease (University of Texas at Austin)
Design Challenges for AI Fact Checking
Fair, Accountable, & Transparent (“FAT”) AI
• Is “fact” vs. “fake” all people want to know?
• Why trust a “black box” classifier?
• How to detect & mitigate bias (user or AI)?
• How can we combine human knowledge & experience
with AI speed & scalability?
– Goals: human insights, reduced AI error, & personalization
12

13. Matt Lease (University of Texas at Austin)
Design Challenge: Human Interaction with AI?
13

14. Believe it or not: Designing a Human-AI
Partnership for Mixed-Initiative Fact-Checking
Joint work with
An Thanh Nguyen (UT), Byron Wallace (Northeastern), & more!

15. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking 15
Demo!
Nguyen et al., UIST’18

16. Matt Lease (University of Texas at Austin)
Empowering Users: Vision
16

17. Matt Lease (University of Texas at Austin)
Empowering Users: Challenge
17

18. Matt Lease (University of Texas at Austin)
Empowering Users: Challenge
18

19. Matt Lease (University of Texas at Austin)
A new form of echo chamber?
User control promotes transparency & trust, but what about bias?
19

20. 20
Anubrata Das, Kunjan Mehta and Matthew Lease
SIGIR 2019 Workshop on Fair, Accountable, Confidential, Transparent, and
Safe Information Retrieval (FACTS-IR). July 25, 2019
CobWeb: A Research Prototype for Exploring
User Bias in Political Fact-Checking

21. Matt Lease (University of Texas at Austin) 21
We introduce
“political leaning” (bias)
as a function of the
adjusted reputation of
the news sources

22. 22
We introduce
“political leaning” (bias)
as a function of the
adjusted reputation of
the news sources
CobWeb; Anubrata Das, Kunjan Mehta and Matthew Lease
Stance
Sources

23. 23
We introduce
“political leaning” (bias)
as a function of the
adjusted reputation of
the news sources
CobWeb; Anubrata Das, Kunjan Mehta and Matthew Lease
Imaginary
Source Bias
Stance
Sources

24. 24
We introduce
“political leaning” (bias)
as a function of the
adjusted reputation of
the news sources
CobWeb; Anubrata Das, Kunjan Mehta and Matthew Lease
A user can alter the
source reputation
Imaginary
Source Bias
Stance
Sources

25. 25
We introduce
“political leaning” (bias)
as a function of the
adjusted reputation of
the news sources
CobWeb; Anubrata Das, Kunjan Mehta and Matthew Lease
A user can alter the
source reputation
Changing reputation
scores changes the
predicted correctness
Imaginary
Source Bias
Stance
Sources

26. 26
We introduce
“political leaning” (bias)
as a function of the
adjusted reputation of
the news sources
CobWeb; Anubrata Das, Kunjan Mehta and Matthew Lease
A user can alter the
source reputation
Changing reputation
scores changes the
predicted correctness
Changing reputation
scores changes the
overall political leaning
Imaginary
Source Bias
Stance
Sources

27. 27
We introduce
“political leaning” (bias)
as a function of the
adjusted reputation of
the news sources
CobWeb; Anubrata Das, Kunjan Mehta and Matthew Lease
Imaginary
Source Bias
Stance
Sources

28. 28
We introduce
“political leaning” (bias)
as a function of the
adjusted reputation of
the news sources
CobWeb; Anubrata Das, Kunjan Mehta and Matthew Lease
A user can alter the
overall political leaning
Imaginary
Source Bias
Stance
Sources

29. 29
We introduce
“political leaning” (bias)
as a function of the
adjusted reputation of
the news sources
CobWeb; Anubrata Das, Kunjan Mehta and Matthew Lease
A user can alter the
overall political leaning
Imaginary
Source Bias
Stance
Sources
Changing the overall
political leaning changes
the source reputations

30. 30
We introduce
“political leaning” (bias)
as a function of the
adjusted reputation of
the news sources
CobWeb; Anubrata Das, Kunjan Mehta and Matthew Lease
Changing the overall
political leaning
changes the predicted
correctness
A user can alter the
overall political leaning
Imaginary
Source Bias
Stance
Sources
Changing the overall
political leaning changes
the source reputations

31. Matt Lease (University of Texas at Austin)
Wrap-up Discussion
• Black-box AI is not enough! Want interaction, exploration, & trust.
– Useful problem to ground work on Fair, Accountable, & Transparent (FAT) AI
• Design challenge: human + AI partnership
• Search engines need more than fact-checking
– How to diversify search results for controversial topics?
– How to assess potential harm? (eg, vaccination & autism)
• AI: new risks of harm as well as potential for good
– Potential added confusion, data / algorithmic bias
– Potential for personal “echo chamber”
– Adversarial applications
31

32. Matt Lease (University of Texas at Austin) – ml@utexas.edu
Thank You!
32
Lab: ir.ischool.utexas.edu
@mattlease
Slides: slideshare.net/mattlease