Talk on the AI4Media Workshop on GANs for Media Content Generation, October 1st 2020, https://ai4media.eu/events/gan-media-generation-workshop-oct-2020/
Presentation of web-based service developed within REVEAL and InVID on Experts’ Meeting on Digital Image Authentication and Classification, December 6, 2017.
DEEPFAKE DETECTION TECHNIQUES: A REVIEWvivatechijri
Noteworthy advancements in the field of deep learning have led to the rise of highly realistic AI generated fake videos, these videos are commonly known as Deepfakes. They refer to manipulated videos, that are generated by sophisticated AI, that yield formed videos and tones that seem to be original. Although this technology has numerous beneficial applications, there are also significant concerns about the disadvantages of the same. So there is a need to develop a system that would detect and mitigate the negative impact of these AI generated videos on society. The videos that get transferred through social media are of low quality, so the detection of such videos becomes difficult. Many researchers in the past have done analysis on Deepfake detection which were based on Machine Learning, Support Vector Machine and Deep Learning based techniques such as Convolution Neural Network with or without LSTM .This paper analyses various techniques that are used by several researchers to detect Deepfake videos.
Presentation of the InVID tools for image forensics analysisInVID Project
Presentation of the InVID tools for image forensics analysis, at the Media Informatics Lab meeting on detection and verification of socially shared videos.
Graduation Project - Face Login : A Robust Face Identification System for Sec...Ahmed Gad
Face login is my 2015 graduation project started in 2014 and lasted 1.5 years of work.
Generally, it is an identification system using face images. It is a multi-use system but it was mainly created to authorize users to login into their system.
There is an IEEE paper published by the project algorithm used in ICCES 2014 http://ieeexplore.ieee.org/abstract/document/7030929/.
Here is its citation Semary, Noura A., and Ahmed Fawzi Gad. "A proposed framework for robust face identification system." Computer Engineering & Systems (ICCES), 2014 9th International Conference on. IEEE, 2014.
A YouTube video describing the project generally.
https://www.youtube.com/watch?v=OUvaPW70Eko
Find me on:
AFCIT
http://www.afcit.xyz
YouTube
https://www.youtube.com/channel/UCuewOYbBXH5gwhfOrQOZOdw
Google Plus
https://plus.google.com/u/0/+AhmedGadIT
SlideShare
https://www.slideshare.net/AhmedGadFCIT
LinkedIn
https://www.linkedin.com/in/ahmedfgad/
ResearchGate
https://www.researchgate.net/profile/Ahmed_Gad13
Academia
https://www.academia.edu/
Google Scholar
https://scholar.google.com.eg/citations?user=r07tjocAAAAJ&hl=en
Mendelay
https://www.mendeley.com/profiles/ahmed-gad12/
ORCID
https://orcid.org/0000-0003-1978-8574
StackOverFlow
http://stackoverflow.com/users/5426539/ahmed-gad
Twitter
https://twitter.com/ahmedfgad
Facebook
https://www.facebook.com/ahmed.f.gadd
Pinterest
https://www.pinterest.com/ahmedfgad/
Presentation of web-based service developed within REVEAL and InVID on Experts’ Meeting on Digital Image Authentication and Classification, December 6, 2017.
DEEPFAKE DETECTION TECHNIQUES: A REVIEWvivatechijri
Noteworthy advancements in the field of deep learning have led to the rise of highly realistic AI generated fake videos, these videos are commonly known as Deepfakes. They refer to manipulated videos, that are generated by sophisticated AI, that yield formed videos and tones that seem to be original. Although this technology has numerous beneficial applications, there are also significant concerns about the disadvantages of the same. So there is a need to develop a system that would detect and mitigate the negative impact of these AI generated videos on society. The videos that get transferred through social media are of low quality, so the detection of such videos becomes difficult. Many researchers in the past have done analysis on Deepfake detection which were based on Machine Learning, Support Vector Machine and Deep Learning based techniques such as Convolution Neural Network with or without LSTM .This paper analyses various techniques that are used by several researchers to detect Deepfake videos.
Presentation of the InVID tools for image forensics analysisInVID Project
Presentation of the InVID tools for image forensics analysis, at the Media Informatics Lab meeting on detection and verification of socially shared videos.
Graduation Project - Face Login : A Robust Face Identification System for Sec...Ahmed Gad
Face login is my 2015 graduation project started in 2014 and lasted 1.5 years of work.
Generally, it is an identification system using face images. It is a multi-use system but it was mainly created to authorize users to login into their system.
There is an IEEE paper published by the project algorithm used in ICCES 2014 http://ieeexplore.ieee.org/abstract/document/7030929/.
Here is its citation Semary, Noura A., and Ahmed Fawzi Gad. "A proposed framework for robust face identification system." Computer Engineering & Systems (ICCES), 2014 9th International Conference on. IEEE, 2014.
A YouTube video describing the project generally.
https://www.youtube.com/watch?v=OUvaPW70Eko
Find me on:
AFCIT
http://www.afcit.xyz
YouTube
https://www.youtube.com/channel/UCuewOYbBXH5gwhfOrQOZOdw
Google Plus
https://plus.google.com/u/0/+AhmedGadIT
SlideShare
https://www.slideshare.net/AhmedGadFCIT
LinkedIn
https://www.linkedin.com/in/ahmedfgad/
ResearchGate
https://www.researchgate.net/profile/Ahmed_Gad13
Academia
https://www.academia.edu/
Google Scholar
https://scholar.google.com.eg/citations?user=r07tjocAAAAJ&hl=en
Mendelay
https://www.mendeley.com/profiles/ahmed-gad12/
ORCID
https://orcid.org/0000-0003-1978-8574
StackOverFlow
http://stackoverflow.com/users/5426539/ahmed-gad
Twitter
https://twitter.com/ahmedfgad
Facebook
https://www.facebook.com/ahmed.f.gadd
Pinterest
https://www.pinterest.com/ahmedfgad/
A Proposed Framework for Robust Face Identification SystemAhmed Gad
Presentation of IEEE paper "A Proposed Framework for Robust Face Identification System"
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?tp=&arnumber=7030929&url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel7%2F7022265%2F7030901%2F07030929.pdf%3Farnumber%3D7030929
Find me on:
AFCIT
http://www.afcit.xyz
YouTube
https://www.youtube.com/channel/UCuewOYbBXH5gwhfOrQOZOdw
Google Plus
https://plus.google.com/u/0/+AhmedGadIT
SlideShare
https://www.slideshare.net/AhmedGadFCIT
LinkedIn
https://www.linkedin.com/in/ahmedfgad/
ResearchGate
https://www.researchgate.net/profile/Ahmed_Gad13
Academia
https://www.academia.edu/
Google Scholar
https://scholar.google.com.eg/citations?user=r07tjocAAAAJ&hl=en
Mendelay
https://www.mendeley.com/profiles/ahmed-gad12/
ORCID
https://orcid.org/0000-0003-1978-8574
StackOverFlow
http://stackoverflow.com/users/5426539/ahmed-gad
Twitter
https://twitter.com/ahmedfgad
Facebook
https://www.facebook.com/ahmed.f.gadd
Pinterest
https://www.pinterest.com/ahmedfgad/
MixTaiwan 20170222 清大電機 孫民 AI The Next Big ThingMix Taiwan
講師簡介:
孫民助理教授│清華大學電機系
孫民博士目前任教於國立清華大學電機系,他畢業於國立交通大學電子工程學系後,取得史坦福電機碩士、密西根安雅堡電機系統組博士、以及西雅圖華盛頓大學計算機工程博士後的經歷。他的研究興趣在電腦視覺、機器學習、以及人機互動領域,近年來基於深度學習在電腦視覺的突破,他致力於開發橫跨人工智慧不同子領域的系統,如自動影片文字描述(視覺x自然語言)、以及與人類行為互動的智慧機器(視覺 x 控制)。
https://telecombcn-dl.github.io/2018-dlcv/
Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of large-scale annotated datasets and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which were previously addressed with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or image captioning.
Summary of problems and research results on the problem of verifying multimedia content on the Internet. Includes results from the REVEAL and InVID research projects. Presented at the Technology Forum, Thessaloniki, May 16, 2018.
Mobile Interactive Hologram VerificationJens Grubert
Our presentation on mobile interactive hologram verification at ISMAR 2013 in Adelaide, Australia
ABSTRACT:
Verification of paper documents is an important part of checking
a person’s identity, authorization for access or simply establishing
a trusted currency. Many documents such as passports or paper
bills include holograms or other view-dependent elements that are
difficult to forge and therefore are used to verify the genuineness
of that document. View-dependent elements change their appearance
based both on the viewing direction as well as dominant light
sources, thus it requires special knowledge and training to accurately
distinguish original elements from forgeries. We present an
interactive application for mobile devices that integrates the recognition
of the documents with the interactive verification of viewdependent
elements. The system recognizes and tracks the paper
document, provides user guidance for view alignment and presents
a stored image of the element’s appearance depending on the current
view of the document also recording user decisions. We describe
how to model and capture the underlying spatially varying
BRDF representation of view-dependent elements. Furthermore,
we evaluate this approach within a user study and demonstrate that
such a setup captures images that are recognizable and that can be
correctly verified.
DeepFake Detection: Challenges, Progress and Hands-on Demonstration of Techno...Symeon Papadopoulos
Slides accompanying an online webinar on DeepFake Detection and a hands-on demonstration of the MeVer DeepFake Detection service. The webinar is supported by the US-Paris Tech Challenge award for our work on the InVID-WeVerify plugin.
Face Recognition System for Door UnlockingHassan Tariq
This is age of Modern Technology and it's becoming necessity
for everyone. Our project is on one of the most basic
daily life security system. As there was a time, when you
had to open the door by yourself or u needed a key of
some sort or a person for guarding some room.
our project changes that view, as we have automated
that old method. It's user friendly and no human interaction
is needed.Door unlocking to provide essential security to our homes, bank lockers , server rooms , private chambers and offices etc.
Presentation on Face detection and recognition - Credits goes to Mr Shriram, "https://www.hackster.io/sriram17ei/facial-recognition-opencv-python-9bc724"
For the full video of this presentation, please visit:
https://www.embedded-vision.com/platinum-members/luxoft/embedded-vision-training/videos/pages/may-2017-embedded-vision-summit
For more information about embedded vision, please visit:
http://www.embedded-vision.com
Alexey Rybakov, Senior Director for Embedded Systems at Luxoft, presents the "Deep Learning Beyond Cats and Cars: Developing a Real-life DNN-based Embedded Vision Product for Agriculture, Construction, Medical, or Retail" tutorial at the May 2017 Embedded Vision Summit.
By now we know very well how to design and train a neural network to recognize cats, dogs and cars. But what about real projects — for example, in agriculture, construction, medical, and retail? This how-to talk provides an overview of what it takes to design, train, and fine-tune a real-life DNN-based embedded vision solution. Rybakov explores algorithmic, data set, training, and optimization decisions that take you from proofs-of-concepts to solid, reliable, and highly optimized systems. This material is based on Luxoft's own successes, failures, and lessons learned while implementing embedded vision solutions.
A Proposed Framework for Robust Face Identification SystemAhmed Gad
Presentation of IEEE paper "A Proposed Framework for Robust Face Identification System"
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?tp=&arnumber=7030929&url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel7%2F7022265%2F7030901%2F07030929.pdf%3Farnumber%3D7030929
Find me on:
AFCIT
http://www.afcit.xyz
YouTube
https://www.youtube.com/channel/UCuewOYbBXH5gwhfOrQOZOdw
Google Plus
https://plus.google.com/u/0/+AhmedGadIT
SlideShare
https://www.slideshare.net/AhmedGadFCIT
LinkedIn
https://www.linkedin.com/in/ahmedfgad/
ResearchGate
https://www.researchgate.net/profile/Ahmed_Gad13
Academia
https://www.academia.edu/
Google Scholar
https://scholar.google.com.eg/citations?user=r07tjocAAAAJ&hl=en
Mendelay
https://www.mendeley.com/profiles/ahmed-gad12/
ORCID
https://orcid.org/0000-0003-1978-8574
StackOverFlow
http://stackoverflow.com/users/5426539/ahmed-gad
Twitter
https://twitter.com/ahmedfgad
Facebook
https://www.facebook.com/ahmed.f.gadd
Pinterest
https://www.pinterest.com/ahmedfgad/
MixTaiwan 20170222 清大電機 孫民 AI The Next Big ThingMix Taiwan
講師簡介:
孫民助理教授│清華大學電機系
孫民博士目前任教於國立清華大學電機系,他畢業於國立交通大學電子工程學系後,取得史坦福電機碩士、密西根安雅堡電機系統組博士、以及西雅圖華盛頓大學計算機工程博士後的經歷。他的研究興趣在電腦視覺、機器學習、以及人機互動領域,近年來基於深度學習在電腦視覺的突破,他致力於開發橫跨人工智慧不同子領域的系統,如自動影片文字描述(視覺x自然語言)、以及與人類行為互動的智慧機器(視覺 x 控制)。
https://telecombcn-dl.github.io/2018-dlcv/
Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of large-scale annotated datasets and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which were previously addressed with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or image captioning.
Summary of problems and research results on the problem of verifying multimedia content on the Internet. Includes results from the REVEAL and InVID research projects. Presented at the Technology Forum, Thessaloniki, May 16, 2018.
Mobile Interactive Hologram VerificationJens Grubert
Our presentation on mobile interactive hologram verification at ISMAR 2013 in Adelaide, Australia
ABSTRACT:
Verification of paper documents is an important part of checking
a person’s identity, authorization for access or simply establishing
a trusted currency. Many documents such as passports or paper
bills include holograms or other view-dependent elements that are
difficult to forge and therefore are used to verify the genuineness
of that document. View-dependent elements change their appearance
based both on the viewing direction as well as dominant light
sources, thus it requires special knowledge and training to accurately
distinguish original elements from forgeries. We present an
interactive application for mobile devices that integrates the recognition
of the documents with the interactive verification of viewdependent
elements. The system recognizes and tracks the paper
document, provides user guidance for view alignment and presents
a stored image of the element’s appearance depending on the current
view of the document also recording user decisions. We describe
how to model and capture the underlying spatially varying
BRDF representation of view-dependent elements. Furthermore,
we evaluate this approach within a user study and demonstrate that
such a setup captures images that are recognizable and that can be
correctly verified.
DeepFake Detection: Challenges, Progress and Hands-on Demonstration of Techno...Symeon Papadopoulos
Slides accompanying an online webinar on DeepFake Detection and a hands-on demonstration of the MeVer DeepFake Detection service. The webinar is supported by the US-Paris Tech Challenge award for our work on the InVID-WeVerify plugin.
Face Recognition System for Door UnlockingHassan Tariq
This is age of Modern Technology and it's becoming necessity
for everyone. Our project is on one of the most basic
daily life security system. As there was a time, when you
had to open the door by yourself or u needed a key of
some sort or a person for guarding some room.
our project changes that view, as we have automated
that old method. It's user friendly and no human interaction
is needed.Door unlocking to provide essential security to our homes, bank lockers , server rooms , private chambers and offices etc.
Presentation on Face detection and recognition - Credits goes to Mr Shriram, "https://www.hackster.io/sriram17ei/facial-recognition-opencv-python-9bc724"
For the full video of this presentation, please visit:
https://www.embedded-vision.com/platinum-members/luxoft/embedded-vision-training/videos/pages/may-2017-embedded-vision-summit
For more information about embedded vision, please visit:
http://www.embedded-vision.com
Alexey Rybakov, Senior Director for Embedded Systems at Luxoft, presents the "Deep Learning Beyond Cats and Cars: Developing a Real-life DNN-based Embedded Vision Product for Agriculture, Construction, Medical, or Retail" tutorial at the May 2017 Embedded Vision Summit.
By now we know very well how to design and train a neural network to recognize cats, dogs and cars. But what about real projects — for example, in agriculture, construction, medical, and retail? This how-to talk provides an overview of what it takes to design, train, and fine-tune a real-life DNN-based embedded vision solution. Rybakov explores algorithmic, data set, training, and optimization decisions that take you from proofs-of-concepts to solid, reliable, and highly optimized systems. This material is based on Luxoft's own successes, failures, and lessons learned while implementing embedded vision solutions.
Deepfakes: An Emerging Internet Threat and their DetectionSymeon Papadopoulos
Webinar talk in the context of the AI4EU Web Cafe. Recording of the talk available on: https://youtu.be/wY1rvseH1C8
Deepfakes have emerged for some time now as one of the largest Internet threats, and even though their primary use so far has been the creation of pornographic content, the risk of them being abused for disinformation purposes is growing by the day. Deepfake creation approaches and tools are continuously improving in terms of result quality and ease of use by non-experts, and accordingly the amount of deepfake content on the Internet is quickly growing. For that reason, approaches for deepfake detection are a valuable tool for media companies, social media platforms and ultimately citizens to help them tell authentic from deepfake generated content. In this presentation, I will be presenting a short overview of the developments in the field of deepfake detection, and present our lessons learned from working on the problem in the context of the Deepfake Detection Challenge and from developing a service for the H2020 WeVerify project.
Short panel presentation given in the context of the AI4EU WebCafe "The COVID-19 and Contact Tracing Apps" on June 23rd 2020, focusing on the problem of COVID-19 misinformation and how this could potentially affect the adoption of contact tracing apps.
Lecture given on January 28, 2019 to post-graduate students of the Computer Engineering and Media program, at the School of Journalism and Media, Aristotle University of Thessaloniki.
Presentation on the topic of sensing air-quality at city level based on Twitter data given at the IEEE Image, Video, and Multidimensional Signal Processing (IVMSP) 2018 workshop in Aristi, Greece.
Aggregating and Analyzing the Context of Social Media ContentSymeon Papadopoulos
Introduction to the Context Analysis and Aggregation service of InVID. Given at the Workshop on Content Verification Tools hosted by the journalists' association in Thessaloniki, Greece on June 6, 2018.
Tutorial for ACM Multimedia 2016, given together with Gerald Friedland, with contributions from Julia Bernd and Yiannis Kompatsiaris. The presentation covered an introduction to the problem of disclosing personal information through multimedia sharing, the associated security risks, methods for conducting multimodla inferences and technical frameworks that could help alleviate such risks.
Presentation of the joint participation between CERTH and CEA LIST in the MediaEval 2015 edition of the Retrieving Diverse Social Images Task in Wurzen, Germany on 14-15 September, 2015.
Presentation of the joint participation between CERTH and CEA LIST in the 2015 edition of the MediaEval Placing Task in Wurzen, Germany, September 14-15, 2015.
Presentation of the task overview in MediaEval 2015, Wurzen, Germany. Verifying Multimedia Use is about detecting tweets that carry misleading information and content.
Learning to Classify Users in Online Interaction NetworksSymeon Papadopoulos
Presentation given at ICCSS 2015, Helsinki, Finland. It illustrates an approach for classifying users of OSNs solely based on their interactions with other users.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Assure Contact Center Experiences for Your Customers With ThousandEyes
Deepfake Detection: The Importance of Training Data Preprocessing and Practical Considerations
1. DeepFake Detection: The Importance of Training Data
Preprocessing and Practical Considerations
Dr. Symeon (Akis) Papadopoulos – @sympap
MeVer Team @ Information Technologies Institute (ITI) /
Centre for Research & Technology Hellas (CERTH)
Joint work with Polychronis Charitidis, George Kordopatis-Zilos and
Yiannis Kompatsiaris
AI4Media Workshop on GANs for Media Content Generation, Oct 1, 2020
Media Verification
(MeVer)
2. DeepFakes
• Content, generated by AI, that seems
authentic to human eye
• Most common form: generation and
manipulation of human face
Source: https://en.wikipedia.org/wiki/Deepfake
Source: https://www.youtube.com/watch?v=iHv6Q9ychnA
Source: Media Forensics and DeepFakes: an overview
3. Manipulation types
Facial manipulations can
be categorised in four
main different groups:
• Entire face synthesis
• Attribute manipulation
• Identity swap
• Expression swap
Source: DeepFakes and Beyond: A Survey of Face Manipulation and Fake
Detection (Tolosana et al., 2020)
Tolosana, R., et al. (2020). Deepfakes and beyond:
A survey of face manipulation and fake
detection. arXiv preprint arXiv:2001.00179.
Verdoliva, L. (2020). Media forensics and deepfakes:
an overview. arXiv preprint arXiv:2001.06564.
Mirsky, Y., & Lee, W. (2020). The Creation and
Detection of Deepfakes: A Survey. arXiv preprint
arXiv:2004.11138.
4. WeVerify Project
• WeVerify aims at detecting disinformation in social media and expose
misleading and fabricated content
• Partners: Univ. Sheffield, OntoText, ATC, DW, AFP, EU DisinfoLab, CERTH
• A key outcome is a platform for collaborative content verification,
tracking, and debunking
• Currently, we are developing a deepfake detection service for the
WeVerify platform
• Participation in DeepFake Detection Challenge
https://weverify.eu/
5. DeepFake Detection Challenge
• Goal: detect videos with facial or voice manipulations
• 2,114 teams participated in the challenge
• Log Loss error evaluation on public and private validation sets
• Public evaluation contained videos with similar transformations as the
training set
• Private evaluation contained organic videos and videos with unknown
transformations from the Internet
• Our final standings:
• public leaderboard: 49 (top 3%) with 0.295 Log Loss error
• private leaderboard: 115 (top 5%) with 0.515 Log Loss error
Source: https://www.kaggle.com/c/deepfake-detection-challenge
6. DeepFake Detection Challenge - dataset
• Dataset of more than 110k videos
• Approx. 20k REAL and the rest are FAKE
• FAKE videos generated from the REAL
• Models used:
• DeepFake AutoEncoder (DFAE)
• Morphable Mask faceswap (MM/NN)
• Neural Talking Heads (NTH)
• FSGAN
• StyleGAN
Dolhansky, B., Bitton, J., Pflaum, B., Lu, J., Howes, R., Wang,
M., & Ferrer, C. C. (2020). The DeepFake Detection Challenge
Dataset. arXiv preprint arXiv:2006.07397.
7. Dataset preprocessing - Issues
• Face dataset quality depends on face extraction accuracy (Dlib,
mtcnn, facenet-pytorch, Blazeface)
• Generally all face extraction libraries generate a number of false
positive detections
• Manual tuning can improve the quality of the generated dataset
Deep learning
model
Face
extraction
Frame
extraction
Video
corpus
8. Noisy data creeping in the training set
• Extracting faces with 1 fps from Kaggle DeepFake Detection Challenge dataset
videos using pytorch implementation of MTCNN face detection
• Observation: False detections are less compared to true detections in a video
9. Our “noise” filtering approach
• Compute face embeddings for each detected face in video
• Similarity calculation between all face embeddings in a video → similarity graph construction
• Nodes represent faces and two faces are connected if their similarities are greater than 0.8 (solid lines)
• Drop components smaller than N/2 (e.g. component 2)
• N is the number of frames that contain face detections (true or false).
10. Advantages
• Simple and fast procedure
• No need for manual tuning of the face extraction settings
• Clusters of distinct faces in cases of multiple persons in the video
• This information can be utilized in various ways (e.g. predictions per face)
Faces extracted from multiple video frames
Component 1
Component 2
11. Experiments
• We trained multiple DeepFake detection models on the DFDC dataset
with and without (baseline) our proposed approach
• Three datasets: a) Celeb-DF, b) FaceForensics++, c) DFDC subset
• For evaluation we examined two aggregation approaches
• avg: prediction is the average of all face predictions
• face: prediction is the max prediction among different avg face predictions
• Results for the EfficientNet-B4 model in terms of Log loss error:
Pre-
processing
CelebDF FaceForensics++ DFDC
avg face avg face avg face
baseline 0,510 0,511 0,563 0,563 0,213 0,198
proposed 0,458 0,456 0,497 0,496 0,195 0,173
12. Our DFDC Approach - details
• Applied proposed preprocessing approach to clean the generated face dataset
• Face augmentation:
• Horizontal & vertical flip, random crop, rotation, image compression, Gaussian & motion
blurring, brightness, saturation & contrast transformation
• Trained three different models: a) EfficientNet-B3, b) EfficientNet-B4, c) I3D*
• Models trained on face level:
• I3d trained with 10 consecutive face images exploiting temporal information.
• EfficientNet models trained on single face images
• Per model:
• Added two dense layers with dropout after the backbone architecture with 256 and 1 units
• Used the sigmoid activation for the last layer
* ignoring the optical flow stream
13. Our DFDC approach – inference
pre-processing model inference post-processing
14. Lessons from other DFDC teams
• Most approaches ensemble multiple EfficientNet architectures (B3-B7) and
some of them were trained on different seeds
• ResNeXT was another architecture used by a top-performing solutions
combined with 3D architectures such as I3D, 3D ResNet34, MC3 & R2+1D
• Several approaches increased the margin of the detected facial bounding
box to further improve results.
• We used an additional margin of 20% but other works proposed a higher proportion.
• To improve generalization:
• Domain-specific augmentations: a) half face removal horizontally or vertically, b)
landmark (eyes, nose, or mouth) removal
• Mixup augmentations
15. Practical challenges
• Limited generalization
• This observation applies to most submissions. The winning team scored
0.20336 in public validation and only 0.42798 in the private (Log Loss)
• Overfitting
• The best submission in the public leaderboard scored 0.19207 but in the
private evaluation the error was 0.57468, leading to the 904-th position!
• Broad problem scope
• The term DeepFake may refer to every possible manipulation and generation
• Constantly increasing manipulation and generation techniques
• A detector is only trained with a subset of these manipulations
16. DeepFake Detection in the Wild
• Videos in the wild usually contain multiple scenes
• Only a subset of these scenes may contain DeepFakes
• Detection process might be slow for multi-shot videos (even short ones)
• Low quality videos
• Low quality faces tend to fool classifiers
• Small detected and fast-moving faces
• Usually lead to noisy predictions
• Changes in the environment
• Moving obstacles in front of the faces
• Changes in lighting
17. DeepFake Detection Service @ WeVerify
https://www.youtube.com/watch?v=cVljNV
V5VPw&ab_channel=TheFakening
18. More details at TTO 2020
Charitidis, P., Kordopatis-Zilos, G., Papadopoulos, S., & Kompatsiaris, Y.
(2020). Investigating the impact of preprocessing and prediction
aggregation on the DeepFake detection task. Proceedings of the
Conference for Truth and Trust Online (TTO) [to appear],
https://arxiv.org/abs/2006.07084
https://truthandtrustonline.com/
19. Thank you!
Dr. Symeon Papadopoulos
papadop@iti.gr
@sympap
Media Verification (MeVer)
https://mever.iti.gr/
@meverteam https://ai4media.eu/
https://weverify.eu/