CAA is a tool that aims to facilitate the verification of user-generated videos. It aggregates context from various sources like video metadata, comments, related tweets and past cases to generate a verification report. A user study found it helped users debunk around 70% of fake videos and verify around 80% of real videos. The tool utilizes reverse image search, text analysis and crowdsourced information. Future work includes expanding platform coverage, improving performance and usability.

1. Context Aggregation and Analysis: A Tool for User-
Generated Video Verification
Olga Papadopoulou, Dimitrios Giomelakis, Lazaros Apostolidis,
Symeon Papadopoulos, Yiannis Kompatsiaris
ROME 2019
Workshop on Reducing Online Misinformation Exposure
25 July 2019
1SIGIR'19 Workshop: ROME 2019

2. User generated videos
2
Syrian hero boy rescue girl in shootout
4,647,792 views
Brussels airport bombing
203,783 views
Staged Reuse
Left: https://https://www.youtube.com/watch?v=mgwO6oni-wY
Right video: https://www.youtube.com/watch?v=PhSKWlJk9ew
SIGIR'19 Workshop: ROME 2019

3. Context Aggregation and Analysis
3
A tool that aims to facilitate the verification of user-generated videos.
Platform APIs
CAA
COMPONENTS
Verification
report
https://caa.iti.gr
SIGIR'19 Workshop: ROME 2019

4. Web Interface: Context Aggregation and Analysis
4
https://caa.iti.gr
Provide URL Start Verification
SIGIR'19 Workshop: ROME 2019

5. General
• Video
metadata
• Source
metadata
• Thumbnails
Wisdom of
the crowd
• Comments
• Tweets
sharing the
video
Wisdom of
the past
• Cross-
checking
with known
cases
Wisdom of
the machine
• Machine
learning
system
Video and account metadata
5
CAA: A tool that aims to facilitate the verification of user-generated videos.
Platform APIs
CAA
COMPONENTS
Verification
report
SIGIR'19 Workshop: ROME 2019

6. Video Thumbnails
6
User – defined video thumbnails are provided by the Platform APIs.
 Apply Reverse Image Search to Google and Yandex.
 Label the video as possibly ‘clickbait’:
The preferred thumbnail, which is shown when the video appears in lists, is checked whether
it exists in the video or not.
Used as thumbnail in a Fake video
A real image of
the explosion in
Brussels
SIGIR'19 Workshop: ROME 2019

7. General
• Video
metadata
• Source
metadata
• Thumbnails
Wisdom of
the crowd
• Comments
• Tweets
sharing the
video
Wisdom of
the past
• Cross-
checking
with known
cases
Wisdom of
the machine
• Machine
learning
system
Wisdom of the crowd
7
CAA: A tool that aims to facilitate the verification of user-generated videos.
Platform APIs
CAA
COMPONENTS
Verification
report
SIGIR'19 Workshop: ROME 2019

8. Wisdom of the crowd
8
Video
comments
Verification-related
Available in 7 languages Links
User- defined keywords
English Keywords: lies, fake,
wrong, lie, confirm, where, location,
lying, false, incorrect, misleading,
propaganda, liar
SIGIR'19 Workshop: ROME 2019

9. Wisdom of the crowd
9
Twitter timeline:
 The tweets sharing the submitted video URL for YouTube and Facebook videos.
 The retweets of a submitted Twitter video.
A tweet is posted couple of hours
after the Video was shared on
YouTube (redline) and explains that
the claim of ISIS being the
target of the bombing is false.
Claim: Bombing over ISIS area
SIGIR'19 Workshop: ROME 2019

10. SIGIR'19 Workshop: ROME 2019
General
• Video
metadata
• Source
metadata
• Thumbnails
Wisdom of
the crowd
• Comments
• Tweets
sharing the
video
Wisdom of
the past
• Cross-
checking
with known
cases
Wisdom of
the machine
• Machine
learning
system
Wisdom of the past
10
CAA: A tool that aims to facilitate the verification of user-generated videos.
Platform APIs
CAA
COMPONENTS
Verification
report

11. Wisdom of the past
11
 Use a background collection of debunked and verified user-generated videos.
 Cross check whether the submitted video exists in the dataset (Near duplicate detection*)
Submitted video
Matched video
Background collection
*Kordopatis-Zilos, Giorgos, et al. "Near-duplicate video retrieval with deep metric learning.
"Proceedings of the IEEE International Conference on Computer Vision. 2017.
SIGIR'19 Workshop: ROME 2019

12. Unique cases: 200 fake and 180 real.
Videos (including near duplicates from YouTube, Facebook and Twitter): 2920 fake and 2090 real
Cascade: the initial video of a case and all its near duplicates in chronological order*.
Fake Video Corpus
12
Staged Tampered Reused
*Papadopoulou, Olga, et al. "A corpus of debunked and verified user-generated videos.“
Online Information Review 43.1 (2019): 72-88.
SIGIR'19 Workshop: ROME 2019

13. SIGIR'19 Workshop: ROME 2019
General
• Video
metadata
• Source
metadata
• Thumbnails
Wisdom of
the crowd
• Comments
• Tweets
sharing the
video
Wisdom of
the past
• Cross-
checking
with known
cases
Wisdom of
the machine
• Machine
learning
system
Wisdom of the machine
13
CAA: A tool that aims to facilitate the verification of user-generated videos.
Platform APIs
CAA
COMPONENTS
Verification
report

14. Wisdom of the machine
14
Precision Recall F-score
0.66 0.81 0.72
Text features are extracted from the video title
Aggregating the prediction scores across all
cascades the automatic verification approach
is proven valuable cue.
1. Text length
2. # words
3. Contains ‘?’ marks
4. Contains ‘!’ marks
5..Contains 1st pronoun
6. Contains 2nd pronoun
7. Contains 3rd pronoun
8. # uppercases
9. # positive sentiment words
10. # negative sentiment words
11. # slang words
12. Has ‘:’ symbol
13. # ‘?’ marks
14. # ‘!’ marks
SIGIR'19 Workshop: ROME 2019

15. User Study
15
Tasks:
 Debunking the 200 fake videos of the FVC
 Verifying the 180 real videos of the FVC
Users:
 A male with journalistic background
 A female with computer engineering background
Procedure:
1. Submit a video URL to the tool
2. Check and analyse the produced verification report
3. Decide about the video veracity
4. Record the results and the time spent on the task
Labels:
 True: If a fake video is debunked or if a real video is verified
 False: if the debunking or veryfying of a fake/real video fails
 Uncertain: there are indicators that create doubts about the
video credibility but there is no concrete evidence proving that the video is fake or real.
Is Debunked # videos Time (sec)
True 132 208
False 46 272
Uncertain 22 270
~70% of the fake videos were succesfully debunked
Is Verified # videos
True 140
False 29
Uncertain 11
~80% of the real videos
were succesfully verified
SIGIR'19 Workshop: ROME 2019

16. User Study
16
Verification cues # videos
Google reverse image
search
70
Verification related
comments
64
Links comments 38
Video comments 18
Yandex reverse image
search
15
Video metadata 14
Twitter timeline 5
Free text comments 1
Video thumbnail 1
1. Google reverse image search is the most helpful
feature. The evaluated videos are already discussed
through the web.
For breaking news videos such online information
might be considerably more limited for some time
(minutes to hours) after a video is posted.
2. Comments: Overall the features related to comments
(verification, links etc.) contribute to the debunking of
more than half of the videos.
SIGIR'19 Workshop: ROME 2019

17. Limitations
17
Lack of APIs for implementing more platforms
e.g. WhatsApp, Instagram.
Walled Garden issue: Limitations on the information that is accessible
programmatically even when this information is publicly available.
Long response times.
SIGIR'19 Workshop: ROME 2019

18. Conclusions and Future Work
18
CAA is available as:
 UI: https://caa.iti.gr
 Component of the InVID – WeWerify verification plugin
The tool has proven valuable for journalists and citizens:
 12,000 unique users, from all over the world (United States, France, India, Saudi Arabia and
other countries) were recorded using the tool from late 2017 until May 2019.
 ~70% of the fake and ~80% of the real video of the FVC were successfully debunked and verified
consulting the CAA tool.
 F-score of 0.72 is achieved.
For future work:
 Conduct a larger user study with more users having different knowledge background
 Improve the performance of the tool, making it easier to interpret by non-trained users.
SIGIR'19 Workshop: ROME 2019

19. Thank you for your attention!
19
https://caa.iti.gr
FVC: https://mklab.iti.gr/results/fake-video-corpus/
SIGIR'19 Workshop: ROME 2019
Useful links:
Olga Papadopoulou
Information Technologies Institute of CERTH
olgapapa@iti.gr