The document proposes a method called CURB to optimize the scheduling of fact-checking fake news stories based on crowdsourced flags and exposures. It models the spread of stories as stochastic processes and formulates fact-checking as a stochastic optimal control problem. The CURB algorithm dynamically determines the optimal time to fact-check stories based on their current misinformation rates. Evaluation on real Twitter and Weibo datasets shows CURB achieves better precision and reduction of misinformation spread compared to baseline methods.

1. Leveraging the Crowd to Detect and Reduce
the Spread of Fake News and Misinformation
Jooyeon Kim, Behzad Tabibian, Alice Oh, Bernhard
Schölkopf, Manuel Gomez Rodriguez / MPI + KAIST
img credit: http://www.pewinternet.org/2017/10/19/the-future-of-truth-and-misinformation-online/

2. 1. Post-Truth Era and Fake News
2. Crowd-Powered Fact-Checking
3. Fact-Checking Scheduling Problem
4. Algorithm: CURB
5. Detect and Reduce the Spread of Fake News

3. 1. Post-Truth Era and Fake News
2. Crowd-Powered Fact-Checking
3. Fact-Checking Scheduling Problem
4. Algorithm: CURB
5. Detect and Reduce the Spread of Fake News
img credit: http://static.adweek.com/adweek.com-prod/wp-content/uploads/2017/07/FakeNewsOnLaptop-840x460.jpg

4. Post-Truth
Relating to or denoting
circumstances in which
objective facts are less
influential in shaping public
opinion than appeals to
emotion and personal
belief.
- Oxford Dictionaries
img credit: http://www.creativityworks.net/
1

5. Opinionated, Inaccurate Fake News
2

6. 1. Post-Truth Era and Fake News
2. Crowd-Powered Fact-Checking
3. Fact-Checking Scheduling Problem
4. Algorithm: CURB
5. Detect and Reduce the Spread of Fake News
img credit: https://crowdsourcedtesting.com/resources/wp-content/uploads/2014/05/canstockphoto12241921.jpg

7. Crowd-powered Fact-checking
Shares &
Re-Shares
Exposures
Flags
Fact-Check
3

8. Crowd-powered Fact-checking
Shares &
Re-Shares
Exposures
Flags
Fact-Check
Christine
3

9. Crowd-powered Fact-checking
Shares &
Re-Shares
Exposures
Flags
Fact-Check
Christine
Beth
3

10. Crowd-powered Fact-checking
Shares &
Re-Shares
Exposures
Flags
Fact-Check
Christine
Beth Bob
3

11. Crowd-powered Fact-checking
Shares &
Re-Shares
Exposures
Flags
Fact-Check
Christine
Beth Bob
3

12. Crowd-powered Fact-checking
Shares &
Re-Shares
Exposures
Flags
Fact-Check
Christine
Beth Bob David
3

13. Crowd-powered Fact-checking
Shares &
Re-Shares
Exposures
Flags
Fact-Check
Christine
Beth Bob David
3

14. Crowd-powered Fact-checking
Shares &
Re-Shares
Exposures
Flags
Fact-Check
Christine
Beth Bob David
Joe
3

15. Crowd-powered Fact-checking
Shares &
Re-Shares
Exposures
Flags
Fact-Check
Christine
Beth Bob David
Joe
Chris
3

16. Crowd-powered Fact-checking
Shares &
Re-Shares
Exposures
Flags
Fact-Check
Christine
Beth Bob David
Joe
Chris
3

17. Crowd-powered Fact-checking
Shares &
Re-Shares
Exposures
Flags
Fact-Check
Christine
Beth Bob David
Joe
Chris
3

18. Challenges & Solutions
Fact-checking is costly
4
Number of Exposures is
uncertain
Trade-off between
flags and exposures

19. Challenges & Solutions
Fact-checking is costly
Probabilistic
exposure models
4
Number of Exposures is
uncertain
Trade-off between
flags and exposures

20. Challenges & Solutions
Fact-checking is costly
Probabilistic
exposure models
4
Number of Exposures is
uncertain
Trade-off between
flags and exposures

21. Exposure vs. Flag Trade-off
Time
#User
# Exposure
# Flag
4

22. Time
#User
# Exposure
# Flag
Optimal
Fact-checking
time
Exposure vs. Flag Trade-off
4

23. Time
#User
# Exposure
# Flag
Optimal
Fact-checking
time
Report Late:
More users exposed to
misinformation if the story is
actually fake
Exposure vs. Flag Trade-off
4

24. #User
Time
# Exposure
# Flag
Optimal
Fact-checking
time
Report Early:
Less confidence whether the story
is fake or not;
More stories to fact-check: costly
Exposure vs. Flag Trade-off
4

25. Challenges & Solutions
Fact-checking is costly
Probabilistic
exposure models
4
Number of Exposures is
uncertain
Trade-off between
flags and exposures

26. Challenges & Solutions
Fact-checking is costly
Probabilistic
exposure models
Optimal Fact-checking
4
Number of Exposures is
uncertain
Trade-off between
flags and exposures

27. Reduce the spread of misinformation
Research Overview
5

28. Reduce the spread of misinformation
Research Overview
Flags, exposures
Quantify
Shares, (re) shares
Happens when
5

29. Reduce the spread of misinformation
Research Overview
Flags, exposures
Quantify
Shares, (re) shares
Happens when
Solve
Stochastic optimal control theory
5

30. Reduce the spread of misinformation
Research Overview
Flags, exposures
Quantify
Shares, (re) shares
Happens when
Solve
Stochastic optimal control theory
5
Optimal fact-checking
time

31. 1. Post-Truth Era and Fake News
2. Crowd-Powered Fact-Checking
3. Fact-Checking Scheduling Problem
4. Algorithm: CURB
5. Detect and Redue the Spread of Fake News
img credit: http://www.musicteachingadventures.com/wp-content/uploads/2013/08/clock-time.jpg

32. Modeling
6

33. Modeling
Shares &
Re-Shares
One Counting Process / Story
6

34. Modeling
Shares &
Re-Shares
One Counting Process / Story
Exposures
One Counting Process / Story
6

35. Modeling
Shares &
Re-Shares
One Counting Process / Story
Exposures
One Counting Process / Story
Flags
Estimate from
Historical Data
6

36. Modeling
Shares &
Re-Shares
One Counting Process / Story
Exposures
One Counting Process / Story
Flags
Estimate from
Historical Data
Fact-Check
One Survival Process / Story
6

37. Probability of (a story) being
misinformation given a user flag
Probability of (a story) being
misinformation not given a user flag
Estimated Number of Users
Exposed to Misinformation
7

38. Estimated Number of Users
Exposed to Misinformation
7

39. Misinformation Rate
8

40. Misinformation Rate
8

41. Historical data (# flags, #exposures)
+ Posterior estimation
Assume Beta Prior:
Posterior:
Misinformation Rate
8

42. Historical data (# flags, #exposures)
+ Posterior estimation
Assume Beta Prior:
Posterior:
Misinformation Rate
8

43. The Fact Checking Scheduling Problem
9

44. The Fact Checking Scheduling Problem
Fact Checking Rate:
we need to optimize this!
9

45. The Fact Checking Scheduling Problem
9

46. Nondecreasing quadratic convex loss function
The Fact Checking Scheduling Problem
9

47. Nondecreasing quadratic convex loss function
The Fact Checking Scheduling Problem
9Misinformation Rate Fact-checking Rate

48. Nondecreasing quadratic convex loss function
The Fact Checking Scheduling Problem
9Misinformation Rate Fact-checking Rate
Tunable parameter for
the exposure-flag trade-off

49. Misinformation Intensity Fact-checking Intensity:
We need to optimize this!
Tunable parameter for
the exposure-flag trade-off
Nondecreasing quadratic convex loss function
Represent tuples of the counting processes and their
corresponding rates as stochastic differential equations
(SDEs) with jumps
The Fact Checking Scheduling Problem
9

50. Misinformation Intensity Fact-checking Intensity:
We need to optimize this!
Tunable parameter for
the exposure-flag trade-off
Nondecreasing quadratic convex loss function
Solve the stochastic optimal control
problem for jump SDEs
Represent tuples of the counting processes and their
corresponding rates as stochastic differential equations
(SDEs) with jumps
The Fact Checking Scheduling Problem
9

51. 1. Post-Truth Era and Fake News
2. Crowd-Powered Fact-Checking
3. Fact-Checking Scheduling Problem
4. Algorithm: CURB
5. Detect and Reduce the Spread of Fake News
img credit: https://cdn-images-1.medium.com/max/1280/1*Jx3X4pD0KdiP8QDhi3d4Og.png

52. The Optimal Fact-checking Rate
It only depends on the current rate of
misinformation!
10

53. The CURB Algorithm
Exposures
Re-Shares
Flags
Fact-Check
11

54. The CURB Algorithm
Exposures
Re-Shares
Flags
Fact-Check
11

55. t0
The CURB Algorithm
Exposures
Re-Shares
Flags
Fact-Check
11

56. t0
The CURB Algorithm
Exposures
Re-Shares
Flags
Fact-Check
11

57. t0
The CURB Algorithm
Exposures
Re-Shares
Flags
Fact-Check
11

58. t0
The CURB Algorithm
Exposures
Re-Shares
Flags
Fact-Check
11

59. t0
The CURB Algorithm
Exposures
Re-Shares
Flags
Fact-Check
t1
t1 ~ f(t0, )
11

60. t0
The CURB Algorithm
Exposures
Re-Shares
Flags
Fact-Check
t1
t1 ~ f(t0, )
11

61. t0
The CURB Algorithm
Exposures
Re-Shares
Flags
Fact-Check
t1
t1 ~ f(t0, )
11

62. t0
The CURB Algorithm
Exposures
Re-Shares
Flags
Fact-Check
t1
t1 ~ f(t0, )
11

63. t0
The CURB Algorithm
Exposures
Re-Shares
Flags
Fact-Check
t1 t2
t1 ~ f(t0, )
t2 ~ f(t1, )
11

64. t0
The CURB Algorithm
Exposures
Re-Shares
Flags
Fact-Check
t1 t2
t1 ~ f(t0, )
t2 ~ f(t1, )
11

65. t0
The CURB Algorithm
Exposures
Re-Shares
Flags
Fact-Check
t1 t2
t1 ~ f(t0, )
t2 ~ f(t1, )
11

66. t0
The CURB Algorithm
Exposures
Re-Shares
Flags
Fact-Check
t1 t2
t1 ~ f(t0, )
t2 ~ f(t1, )
11

67. t0
The CURB Algorithm
Exposures
Re-Shares
Flags
Fact-Check
t1 t2t3
t1 ~ f(t0, )
t2 ~ f(t1, )
t3 ~ f(t2, )
11

68. t0
The CURB Algorithm
Exposures
Re-Shares
Flags
Fact-Check
t1 t2t3
t1 ~ f(t0, )
t2 ~ f(t1, )
t3 ~ f(t2, )
11

69. t0
The CURB Algorithm
Exposures
Re-Shares
Flags
Fact-Check
t1 t2t3
t1 ~ f(t0, )
t2 ~ f(t1, )
t3 ~ f(t2, )
11

70. t0
The CURB Algorithm
Exposures
Re-Shares
Flags
Fact-Check
t1 t2t3
t1 ~ f(t0, )
t2 ~ f(t1, )
t3 ~ f(t2, )
11

71. t0
The CURB Algorithm
Exposures
Re-Shares
Flags
Fact-Check
t1 t2t3
t1 ~ f(t0, )
t2 ~ f(t1, )
t3 ~ f(t2, )
t4 ~ f(t3, )
11
t4

72. t0
The CURB Algorithm
Exposures
Re-Shares
Flags
Fact-Check
t1 t2t3
t1 ~ f(t0, )
t2 ~ f(t1, )
t3 ~ f(t2, )
t4 ~ f(t3, )
11
t4

73. t0
The CURB Algorithm
Exposures
Re-Shares
Flags
Fact-Check
t1 t2t3
t1 ~ f(t0, )
t2 ~ f(t1, )
t3 ~ f(t2, )
t4 ~ f(t3, )
Sample O(Ne(tf)) times
11
t4

74. t0
The CURB Algorithm
Exposures
Re-Shares
Flags
Fact-Check
t1 t2t3
t1 ~ f(t0, )
t2 ~ f(t1, )
t3 ~ f(t2, )
t4 ~ f(t3, )
The function f(..) uses
standard thinning
Superposition principle
Sample O(Ne(tf)) times
11
t4

75. 1. Post-Truth Era and Fake News
2. Crowd-Powered Fact-Checking
3. Fact-Checking Scheduling Problem
4. Algorithm: CURB
5. Detect and Reduce the Spread of Fake News
img credit: https://www.socialpinpoint.com/2016/wp-content/uploads/managing-misinformation-1.jpg

76. Datasets
Publicly available manually annotated data from Twitter, Weibo
NO information about timings of exposures nor flags Simulate via sampling
Model’s efficacy is robust throughout different settings
• 28k shares & re-shares from 19k users
• 7 fake stories & 39 genuine stories
• 93k shares & re-shares from 89k users
• 23 fake stories & 133 genuine stories
12

77. Datasets
Publicly available manually annotated data from Twitter, Weibo
NO information about timings of exposures nor flags Simulate via sampling
Model’s efficacy is robust throughout multiple parameter settings
• 28k shares & re-shares from 19k users
• 7 fake stories & 39 genuine stories
• 93k shares & re-shares from 89k users
• 23 fake stories & 133 genuine stories
12

78. Evaluation Metrics
Two Metrics: precision & misinformation reduction
• Fraction of fact checked stories
that are fake
• Fraction of exposures to misinformation
that fact checking prevented
Each accounts for the different ends of the trade-off
Precision
Misinfo.
Reduction
13

79. Performance vs. # Fact-checksPrecision
.1
.16
.23
.29
.35
# Fact-checks
0 10 20 30 40
Misinfo.Reduction
.0
.25
.5
.75
1.0
# Fact-checks
0 10 20 30 40
15
<Twitter>

80. Comparison Methods
Oracle
• A variant of CURB
• Access to the true flag probability
• Upper bound of CURB
Flag Ratio
• Rate: ratio between the number of flags
and the total number of exposures
Flag Sum
• Fact-check when a certain number of
flags are accumulated
Exposure • Rate: proportional to the exposure rate
14

81. Performance vs. # Fact-checksPrecision
.1
.16
.23
.29
.35
# Fact-checks
0 10 20 30 40
CURB
Oracle
Flag Ratio
Flag Sum
Exposure
Misinfo.Reduction
.0
.25
.5
.75
1.0
# Fact-checks
0 10 20 30 40
CURB
Oracle
Flag Ratio
Flag Sum
Exposure
15
<Twitter>

82. Performance vs. Time#Users
0
10000
20000
30000
40000
Time (days)
300 400 500 600 700 800 900
CURB
Oracle
Flag Ratio
Flag Sum
Exposure
Total
16
<Twitter>

83. Conclusion
• Aim to reduce the spread of misinformation
• Use user flag, time information for modeling
• Yield better results (precision, misinfo. reduction)
compared to the baselines
17
Jooyeon Kim jooyeon.kim@kaist.ac.kr
more at learning.mpi-sws.org/curb/

84. Leveraging the Crowd to Detect and Reduce
the Spread of Fake News and Misinformation
Jooyeon Kim jooyeon.kim@kaist.ac.kr
more at learning.mpi-sws.org/curb/
img credit: http://www.pewinternet.org/2017/10/19/the-future-of-truth-and-misinformation-online/