We compare models of credibility from multiple populations across several online social networks to construct a more thorough understanding of true and false stories’ propagations. Targeting these multiple platforms, we develop an automated classification model over an existing set of crowdsourced, Twitter credibility assessments, verified against journalistically assessed fake news stories that were highly shared on Facebook. We then apply this model to information sharing behaviors in Reddit to determine whether predictors of credibility are consistent across popular social media platforms. Conversations threads about these stories are then aggregated across these platforms to determine whether these multiple views improve our ability to identify stories of questionable veracity.

1. Features of Accuracy in Social
Media Conversations
Cody Buntain (@codybuntain)
cbuntain@cs.umd.edu
NASN 2017, 27 July 2017 1

2. Society is increasingly
getting news from social
media
2

3. 3

4. No guarantee that content on social media is
accurate or true
4

5. 5

6. 6

7. Prior Work
7

8. Castillo, Mendoza, and Poblete. 2013. Predicting
information credibility in time-sensitive social media.
Internet Research.
8

9. Starbird, Maddock, Orand, Achterman, and Mason,
“Rumors, False Flags, and Digital Vigilantes:
Misinformation on Twitter after the 2013 Boston
Marathon Bombing,” iConference 2014.
9

10. ≠
?
10

11. ≠
?
11

12. ≠
?
12

13. ≠
13

14. What’s the point of this talk?
≠
14

15. What’s the point of this talk?
≠
15

16. Research Questions
What features correlate with (perceptions of)
accuracy in social media?
Do accuracy judgements differ across
populations?
How well do accuracy judgement models
identify “false news” stories?
16

17. CREDBANK Event-Related
Topics Shared
on Twitter
Stories from
9 Major Media
Organizations
Rumors Shared
on Twitter
Building a Data Set
17

18. 18

19. What features correlate with (perceptions of)
accuracy in social media?
19

20. What features correlate with (perceptions of)
accuracy in social media?
3rd-Person Pronouns
1st-Person Pronouns
Disagreement
Media Frequency
Friends Over Time
Hashtag Proportion
Media Proportion
Feature Weight
-3 -2.25 -1.5 -0.75 0 0.75 1.5
20

21. What features correlate with (perceptions of)
accuracy in social media?
Hashtag Proportions
Multiple "?!"
"?" Count
Verified Author
Author Follower
Media Proportion
Author Account Age
Emoticons
Tweet Length
Feature Weight
-0.75 0 0.75 1.5 2.25 3
CREDBANK
21

22. What features correlate with (perceptions of)
accuracy in social media?
Verified Author
Author Follower
Media Proportion
Author Account Age
Emoticons
Tweet Length
Hashtag Proportions
Multiple "?!"
"?" Count
Mention Frequency
3rd-Person Pronouns
1st-Person Pronouns
Disagreement
Media Frequency
Friends Over Time
C
R
ED
B
A
N
K
22

23. What features correlate with (perceptions of)
accuracy in social media?
C
R
ED
B
A
N
K
23

24. What features correlate with (perceptions of)
accuracy in social media?
C
R
ED
B
A
N
K
24

25. Do accuracy judgements differ across
populations?
25

26. 0.0 0.2 0.4 0.6 0.8 1.0
)35
0.0
0.2
0.4
0.6
0.8
1.0
735
5andRP (A8C 50.00%)
3-->C (A8C 57.80%)
C-->3 (A8C 55.31%)
Do accuracy judgements differ across
populations?
26

27. Do accuracy judgements differ across
populations?
CREDBANK
CREDBANK
χ2(1, N = 210) =
5.637, p = 0.01758
χ2(1, N = 330) =
0.2143, p = 0.6434
27

28. Do accuracy judgements differ across
populations?
CREDBANK
https://xkcd.com/240/
28

29. Do accuracy judgements differ across
populations?
CREDBANK
https://xkcd.com/240/
29

30. How well do accuracy judgement models
identify “false news” stories?
CREDBANK
30

31. How well do accuracy judgement models
identify “false news” stories?
31

32. How well do accuracy judgement models
identify “false news” stories?
0.0 0.2 0.4 0.6 0.8 1.0
)35
0.0
0.2
0.4
0.6
0.8
1.0
735
5DndRP (A8C 50.00%)
C5(D%A1. (A8C 74.56%)
3H(0( (A8C 36.98%)
32

33. 0.0 0.2 0.4 0.6 0.8 1.0
)35
0.0
0.2
0.4
0.6
0.8
1.0
735
5DndRP (A8C 50.00%)
C5(D%A1. (A8C 68.17%)
3H(0( (A8C 54.83%)
How well do accuracy judgement models
identify “false news” stories?
33

34. How well do accuracy judgement models
identify “false news” stories?
34

35. Limitations Galore

36. ≠CREDBANK

37. ≠

38. What’s the point of this talk?
≠
38

39. What’s the point of this talk?
≠
39

40. What’s the point of this talk?
≠
40

41. What’s the point of this talk?
≠
41

42. Cody Buntain (@codybuntain)
cbuntain@cs.umd.edu
NASN 2017, 27 July 2017
Thank you!
42

43. Questions?
Cody Buntain (@codybuntain)
cbuntain@cs.umd.edu
NASN 2017, 27 July 2017 43