Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Video & AI: capabilities and limitations of AI in detecting video manipulations

83 views

Published on

Invited presentation given by Dr. Vasileios Mezaris during the Greek Media Literacy Week 2019; specifically, presented in the international conference on "Disinformation in Cyberspace: Media literacy meets Artificial Intelligence" that was organized as part of the Media Literacy Week 2019 in Athens, Greece, on November 15, 2019.

Published in: Science
  • Be the first to comment

Video & AI: capabilities and limitations of AI in detecting video manipulations

  1. 1. Video & AI: capabilities and limitations of AI in detecting video manipulations Athens, Nov. 2019 Research Director, CERTH-ITI Dr. Vasileios Mezaris
  2. 2. 2 • Deep fakes are new… • …but fakes are not a new phenomenon! AI as facilitator of disinformation Haiti earthquake video hoax (Jan. 2010): Video from a previous earthquake in California published by major media, slightly modified (cropped) and re-circulated in social media as a Haiti video; picked- up and aired by mainstream media for hours Presidential fake picture (Jan. 2013): Video frame of a surgery operation in Mexico in 2008, was sold to and was mistakenly published by a top newspaper, supposedly depicting the Venezuelan president Hugo Chavez under surgery in a Cuban hospital. https://www.youtube.com/watch?v=JbzVhzNaTdI
  3. 3. 3 • What makes deep fakes different?… • Easier to create high-quality(?) fakes -> more people can do it • Easier to create multiple fakes -> more media items can be created in support of a disinformation campaign; can give rise to Distributed Denial of Truth (DDoT) attacks • …and what remains the same? • A deep fake is usually an altered version of an original video; only the means of introducing the alteration differs • The techniques typically applied for detecting video-borne disinformation are still useful (but there is also room for developing techniques specifically tailored to detecting deep fakes) AI as facilitator of disinformation www.invid-project.eu
  4. 4. 4 • How can AI help? • Understand the structure of the video; select characteristic frames for visual inspection or reverse search on the web • Find near-duplicate videos, or semantically-related videos (of the same or similar incidents) • Video forensics AI for combating disinformation
  5. 5. 5 • Tool for verification of selected newsworthy videos InVID Verification Plugin • Free! Get it from: https://www.invid-project.eu/verify Continues to be developed; now with new look in v0.71 (released June 7, 2019) • • Check prior video use: reverse video search on the Web • Check contextual information: Social- media-based contextual analysis • View location, time and other video metadata A browser plugin to debunk fake news and to verify videos and images >15.000 users • Keyframe/image inspection by magnifying glass • Check image (keyframe) forensics
  6. 6. 6 • Tool for advanced video verification • Check prior video use at the video-segment level: InVID Verification Application • • • Reverse video search also in the InVID repository; inspection by parallel playback of query and duplicate video • Check video origin and rights: • Video logo detection • Video rights management • Check video forensics: • Video forensic filters • Frame-level video inspection • Check additional contextual information • Store the automatic analysis results and manually add evaluation comments and notes • Free demo version accessible at: http://invid.condat.de
  7. 7. 7 InVID Verification Plugin – Reverse search https://www.youtube.com/watch?v=OVAxQA3gMEo “Video, the crowd in panic flees from Notre Dame in Paris after the attack of an armed man”
  8. 8. 8 InVID Verification Plugin – Reverse search https://www.youtube.com/watch?v=OVAxQA3gMEo “Video, the crowd in panic flees from Notre Dame in Paris after the attack of an armed man”
  9. 9. 9 InVID Verification Plugin – Reverse search https://www.youtube.com/watch?v=OVAxQA3gMEo “Video, the crowd in panic flees from Notre Dame in Paris after the attack of an armed man”
  10. 10. 10 InVID Verification Plugin – Reverse search https://www.youtube.com/watch?v=OVAxQA3gMEo “Video, the crowd in panic flees from Notre Dame in Paris after the attack of an armed man”
  11. 11. 11 InVID Verification Plugin – Reverse search https://www.youtube.com/watch?v=OVAxQA3gMEo “Video, the crowd in panic flees from Notre Dame in Paris after the attack of an armed man”
  12. 12. 12 • Video forensics – a quick AI-based approach • Filters generating heat maps (e.g. Cobalt: compares the original video with a modified version of it, re-quantized using MPEG-4 at a different quality level, and generates the error video) • Use deep learning architectures (e.g. GoogLeNet, ResNet) to learn from both the video itself, and the outputs of these filters • No knowledge of the generator’s characteristics is assumed • Results are promising, but not a complete, deployable solution yet [1][2] AI-based video forensics [1] M. Zampoglou, F. Markatopoulou, G. Mercier, D. Touska, E. Apostolidis, S. Papadopoulos, R. Cozien, I. Patras, V. Mezaris, I. Kompatsiaris, "Detecting tampered videos with multimedia forensics and deep learning", Proc. 25th Int. Conf. on Multimedia Modeling (MMM2019), Thessaloniki, Greece, Springer LNCS vol. 11295, pp. 374-386, Jan. 2019. [2] "Video verification in the fake news era", V. Mezaris, L. Nixon, S. Papadopoulos, D. Teyssou (Editors), Springer, 2019.
  13. 13. 13 AI-based video forensics Detected Fakes Fakes not detected (lower resolution) Videos from the InVID Fake Video Corpus 1280x720p480x360p 320x240p 320x240p
  14. 14. 14 • Video forensics – a steganalysis- & AI-based approach • Steganalysis features are extracted from each frame [3] • Sliding windows of 128x128 pixels are considered • High pass filtering of pixel values, then calculate and quantize residuals; co- occurrence matrices are calculated • A recurrent autoencoder is trained on pristine frames of the video • The trained autoencoder assesses the remaining frames of the video • Forged spatial regions in manipulated frames are singled out as anomalous • Good results plus localization of the forgery; but sensitive to encoding AI-based video forensics [3] D. D’Avino, D. Cozzolino, G. Poggi, L. Verdoliva, “Autoencoder with recurrent neural networks for video forgery detection”. In: Electronic Imaging 2017.7 (2017), pp. 92–99.
  15. 15. 15 • Video forensics – a steganalysis- & AI-based approach AI-based video forensics Video from the UNINA dataset http://www.grip.unina.it/
  16. 16. 16 • Looking at the greater picture – directions & thoughts on deep fakes and AI-based video forensics • Manipulated videos and deep fakes still usually come from an original: if you find the original out there, you have documented the faking • Reverse video search is still a powerful tool • AI has great potential; is already used in different techniques for combatting disinformation, and will be used even more for video forensics and detecting deep fakes • But, no single tech seems to be the solution; we need a portfolio of tools [4] • Complete automation is a challenge! AI-based video forensics [4] "Video verification in the fake news era", V. Mezaris, L. Nixon, S. Papadopoulos, D. Teyssou (Editors), Springer, 2019.
  17. 17. 17 Dr. Vasileios Mezaris Information Technologies Institute Centre for Research and Technology Hellas Thermi-Thessaloniki, Greece Tel: +30 2311 257770 Email: bmezaris@iti.gr, web: http://www.iti.gr/~bmezaris/ For more information on the InVID project, visit www.invid-project.eu Follow InVID on Twitter @InVID_EU Contact
  18. 18. 18 This work was supported in part by the EU’s Horizon 2020 research and innovation programme under grant agreement H2020-687786 InVID. Acknowledgement

×