Presentation of the InVID project, with a focus on the developed verification components, at the annual IPTC Meeting that took place in Athens.

1. InVID: Verification of Social Media
Video Content for the News Industry
Symeon (Akis) Papadopoulos
Researcher @ Information Technologies Institute, Centre for Research & Technology Hellas
IPTC Spring Meeting 2018
Apr 23-25, 2018 @ Athens, Greece

2. Illustration: Chad Crowe

3. https://medium.com/1st-draft/fake-news-its-complicated-d0f773766c79

4. Media-based disinformation

5. Real or fake?

6. Real or fake?

7. Real or fake?

8. Real or fake?

9. Real or fake?

10. Real or fake?

11. Real or fake?

12. Real or fake?

13. A bit of “historical” background
2011-2014 2013-2016 2016-2018
Trend detection
Social media search
Quality & veracity
of social media
Social media video
verification

14. Towards computational verification

15. Towards computational verification

16. Towards computational verification

17. Credibility cues (aka features)

18. Credibility cues (aka features)

19. Credibility cues (aka features)

20. Building the classification model

21. The “Verifying Multimedia Use” Task
•VMU: Organized in 2015 and 2016 as
part of the MediaEval benchmarking
initiative
•Goal: compare automated approaches
for fake tweet detection
•Outcomes: several methods from
different research groups across the
globe were tested and compared

22. Tweet Verification Corpus
• 53 events or hoaxes involving false and/or real
imagery and videos
• 257 cases of “fake” content, 261 of “real”
• 10,634 tweets sharing “fake” content, 7,223 tweets
sharing “real” content
• Examples events:
• Hurricane Sandy
• Boston Marathon bombing
• Sochi Olympics
• MA Flight 370
• Nepal Earthquake…
https://github.com/MKLab-ITI/image-verification-corpus

23. Experimental validation
92.5% accuracy in identifying misleading posts
88-98% accuracy depending on language
(major languages tested: en, fr, es, nl)
New features, bagging and agreement-based
retraining led to significant improvements!
One of the top performing methods in the
VMU 2015 & 2016 tasks!

24. InVID in a nutshell
InVID builds a platform providing services to
detect, authenticate and check the reliability
and accuracy of newsworthy video files and
video content spread via social media

25. Project consortium and funding agency
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 687786
InVID: In Video Veritas – Verification of Social Media Video
Content for the News Industry
Start date: 1 January 2016
Duration: 36 months

26. Overall approach

27. Verification components & tools
Components
• Video forensics
• Near-duplicate detection
• Context aggregation and analysis
Applications
• Multimodal analytics dashboard
• Verification application
• Verification plugin

28. Video forensics
• Luminance analysis
• Focus: depth of field analysis
• Chrome: noise luminance analysis
• Color analysis
• Spectrum locus: frames in the color chromaticity diagram
• Fluor: replay in spectral relative color
• Spectral analysis
• Q4: video inspection into the DCT domain
• Time Frequency (audio): spectral analysis of audio comp.
• Encoding compliance
• Q2: detection of multiple quantization
• Motion Vectors: macro-blocks displacement
• Cobalt: encoding or blocking artifacts coherence

29. Motion vectors

30. Focus

31. Cobalt

32. Video forensics: limitations
• Signals are often not strong enough
• Often necessary to inspect multiple filters
• Need experience and expert knowledge

33. Towards automated video forensics
• Binary classification
• Apply Cobalt filter to videos
• Fine-tune a pre-trained DCNN
• GoogLeNet model
• Extended by a 128-unit FC layer
• Replaced 1000-class output with a 2-class output
• Trained using tampered and untampered video
frames
• When a new video arrives
• Apply Cobalt filter
• Use the model to classify it as tampered or not

34. NIST MFC2018 dataset
23 tampered videos and their corresponding untampered sources

35. Preliminary results
• 46 videos total; one frame extracted every two
seconds; ground-truth at video-level; accuracy (%)
• Promising initial results, should be further improved
• Test more filters
• Better annotate training/test data
• More training examples
Fold Accuracy (%)
1 71.43
2 100.00
3 75.00
4 71.43
5 85.71
Fold Accuracy (%)
1 45.95
2 97.84
3 72.96
4 78.49
5 90.50
Frame-level Video-level (majority voting)

36. Near-duplicate video detection
• Variety of definitions and understandings regarding the
near-duplicate videos
• Adopt definition by Wu et al. (2007)
• photometric variations: gamma, contrast, brightness, etc.
• editing operations: resize, shift, crop, flip
• insertion of patterns: caption, logo, subtitles, sliding captions, etc.
• re-encoding: video format, compression
• video modifications: frame rate, frame insertion, deletion, swap
X. Wu, A. G. Hauptmann, and C. W. Ngo. Practical elimination of near-duplicates from web video search. In
Proceedings of the 15th ACM international conference on Multimedia, pp. 218-227, 2007

37. Proposed approach
• Extract frame descriptors from the intermediate
CNN layers
• Train a Deep Neural Network (DNN)
• Deep Metric Learning (DML)
• Improved approximation of video distances
Metric Learning architecture DNN architecture

38. Results
• Outperforms six state-of-the-art approaches
Evaluation dataset No/Other dataset
Method ACC PPT SMVH CNN-L DMLcc CH LS DMLvcdb
mAP 0.944 0.958 0.971 0.974 0.981 0.892 0.954 0.969
mean Average Precision

39. Context analysis and
aggregation
• Available at: http://caa.iti.gr
• YouTube, Facebook and Twitter videos
• metadata from APIs
• mentioned locations
• “verification”-related comments
• thumbnails for near-duplicate search
• weather at time and location of video
• video sharing on Twitter

40. Tip in comment led to debunking
A comment points to second 23 of the video
where suddenly the snake appears out of nowhere

41. # verification comments too high
1550 verification-related comments
out of 4219 total number of comments

42. Tweets sharing video are flagged
37 out of 43 tweets sharing
the video are classified as fake

43. Video verification experiments
• 117 fake videos and 110 real videos
• The dataset covers different types of manipulation:
• staged videos,
• videos misrepresenting the depicted event,
• videos of past events claimed to be captured now,
• digitally manipulated videos.
• A supervised learning approach using credibility
features extracted from video comments and video
metadata managed to achieve promising accuracy:
P=72%, R=86%, F=79%

44. Limitations
• Models are still based on aged training data
(could be affected by concept drift…)
• Results not always easy to justify or explain
to end users
• A well-informed adversary can easily fool the
model by emulating “credible-looking” posts
• Journalists are still expected to make the
final decision!

45. Multimodal analytics dashboard
Newsworthy video discovery

46. Verification application
Verification in the Newsroom

47. InVID verification plugin
Verification on the Browser
Get it for free: https://goo.gl/Fo8i73

48. InVID plugin in practice
https://factuel.afp.com/crash-dun-avion-en-algerie-une-vieille-video-dun-
crash-en-alaska-resurgit-tort

49. The future of disinformation

50. Thank you!
http://reveal-mklab.iti.gr/reveal/fake/
http://caa.iti.gr
Get in touch!
Akis Papadopoulos papadop@iti.gr / @sympap

51. References
• Boididou, C., Papadopoulos, S., Kompatsiaris, Y., Schifferes, S., & Newman, N. (2014,
April). Challenges of computational verification in social multimedia. In Proceedings
of the 23rd International Conference on World Wide Web (pp. 743-748). ACM
• Boididou, C., Middleton, S. E., Jin, Z., Papadopoulos, S., Dang-Nguyen, D. T., Boato, G.,
& Kompatsiaris, Y. (2017). Verifying information with multimedia content on twitter.
Multimedia Tools and Applications, 1-27
• Boididou, C., Papadopoulos, S., Apostolidis, L., & Kompatsiaris, Y. (2017, June).
Learning to Detect Misleading Content on Twitter. In Proceedings of the 2017 ACM
on International Conference on Multimedia Retrieval (pp. 278-286). ACM
• Castillo, C., Mendoza, M., & Poblete, B. (2011, March). Information credibility on
twitter. In Proceedings of the 20th international conference on World Wide Web (pp.
675-684). ACM
• Liu, M. Y., Breuel, T., & Kautz, J. (2017). Unsupervised Image-to-Image Translation
Networks. arXiv preprint arXiv:1703.00848
• Papadopoulou, O., Zampoglou, M., Papadopoulos, S., & Kompatsiaris, Y. (2017, June).
Web Video Verification using Contextual Cues. In Proceedings of the 2nd
International Workshop on Multimedia Forensics and Security (pp. 6-10). ACM
• Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to-image translation
using cycle-consistent adversarial networks. arXiv preprint arXiv:1703.10593.