Download to read offline
Uploaded byYiannis Kompatsiaris
Multimodal AI for Disinformation Detection: from Deep Learning to Foundational Models
Disinformation poses a critical threat to key societal values such as democracy, peace, health, and the economy. The rapid evolution of AI, particularly generative AI, has significantly amplified the creation and spread of disinformation, with deepfakes being a prime example. This talk will explore AI-driven approaches to counter disinformation, focusing on multimodal out-of-context misinformation detection and deepfake detection. We will also discuss the potential of Multimodal Foundation Models for fact-checking and highlight why technology alone is insufficient. Addressing disinformation effectively requires a multidisciplinary approach that considers human behavior, economic incentives, policy, and education.
More Related Content
Similar to Multimodal AI for Disinformation Detection: from Deep Learning to Foundational Models
Multimodal AI for Disinformation Detection: from Deep Learning to Foundational Models
- 1. Multimodal AI forDisinformation Detection Yiannis Kompatsiaris, CERTH-ITI, Director
- 2.
- 3. The Rise ofFake News https://trends.google.com/trends/explore?date=all&geo=US&q=fake%20news US Elections 2016 Volume for query “fake news” over time: A key milestone has been the US Elections in 2016, which marked the beginning of large-scale coordinated disinformation campaigns.
- 4.
- 5. …as old astime A challenge… high cost in both cases {disinformation} {misinformation}
- 6.
- 7. Much harder today •the rapid growth of social media and the Internet has made it easier for false information to spread quickly and widely • the rise of generative AI (e.g. deepfakes, ChatGPT) is making it increasingly difficult to distinguish between real and fake • political polarisation and the erosion of trust in traditional media sources has created an environment in which disinformation can thrive • financial models of online platforms (click-bait) • disinformation campaigns are often carried out by governments, political organisations, and other powerful groups with an agenda to manipulate public opinion and undermine democratic processes
- 8. • “The Supplyof Disinformation Will Soon Be Infinite” • Disinformation campaigns used to require a lot of human effort, but artificial intelligence will take them to a whole new level https://www.theatlantic.com/ideas/archive/2020/09/future-propaganda-will-be-computer-ge Much harder today
- 9. Much harder today Asingle adversary can operate thousands of AI personas, scheduling content and updating narrative frames instantaneously across an entire fleet of agents. Swarms of collaborative, malicious AI agents fusing LLM reasoning with multi-agent architectures Coordinating autonomously, infiltrating communities, and fabricating consensus at minimal cost
- 10. The impact canbe very high Key societal values are affected (not art, irony, parody since it attacks fundamental values/rights) • Misleading the public: the spread false or misleading information can impact individuals' decision-making and beliefs. • Undermining democracy: false or misleading information can be used to manipulate elections and undermine trust in democratic institutions • National internal and external security: national disasters and infowar • Exacerbating conflicts: disinformation can fuel social and political tensions, leading to increased conflict and violence • Damaging reputations: the distribution of various rumours and claims is frequently used to harm the reputation of individuals and organizations • Impairing public health: conspiracy theories and dangerous misinformation, like non- scientifically based medical advice about causes, symptoms, and treatments • Undermine Education: educational material containing false "scientific" information being recommended to children as "educational content".
- 11. The impact canbe very high • By 2028, enterprise spending on battling misinformation and disinformation will surpass $30 billion, cannibalizing 10% of marketing and cybersecurity budgets to combat a multifront threat • a December 2024 Gartner survey of 200 senior business and technology executives found that 72% of respondents identified misinformation, disinformation and malinformation as very or relatively important issues to their executive committees Gartner book World Without Truth
- 12. Universal Declaration ofHuman Rights • Whereas recognition of the inherent dignity and of the equal and inalienable rights of all members of the human family is the foundation of freedom, justice and peace in the world … • Article 19 Freedom of Opinion and Information • Article 21 Right to Participate in Government and in Free Elections • Article 25 Right to Adequate Living Standard (health and well-being) • Article 26 Right to Education
- 15. • Pseudo-science -that's presenting information as scientific fact that is not based on proper scientific methods, plus outright false information and conspiracy theories. • Examples of conspiracy theories are the existence of electricity-producing pyramids, the denial of human-caused climate change and the existence of aliens. • YouTube recommends these "bad science" videos to children alongside legitimate educational content. • The videos focused on wild claims, with sensationalist commentary, catchy titles and dramatic images to draw viewers in. The more people watch the videos, the more revenue - that's money - the channels get by allowing adverts to be shown on the screen. https://www.bbc.co.uk/newsround/66796495
- 16. Visual Disinformation andDeep Fakes
- 17. Visual disinformation isdangerous ● More persuasive than text ● Attracts more attention ● More tempting to share ● Can easily cross borders seeing is believing… Hameleers, M., Powell, T. E., Van Der Meer, T. G., & Bos, L. (2020). A picture paints a thousand lies? The effects and mechanisms of multimodal disinformation and rebuttals disseminated via social media. Political Communication, 37(2), 281-301. Dan, V., Paris, B., Donovan, J., Hameleers, M., Roozenbeek, J., van der Linden, S., & von Sikorski, C. (2021). Visual mis-and disinformation, social media, and democracy. Journalism & Mass Communication Quarterly, 98(3), 641-664. Thomson, T. J., Angus, D., Dootson, P., Hurcombe, E., & Smith, A. (2020). Visual mis/disinformation in journalism and public communications: current verification practices, challenges, and future opportunities. Journalism Practice, 1-25.
- 18.
- 20.
- 21. DeepFakes • Content, generatedby deep neural networks, that seems authentic to human eye • Most common form: generation and manipulation of human face Source: https://en.wikipedia.org/wiki/Deepfake Source: https://www.youtube.com/watch?v=iHv6Q9ychnA Source: Media Forensics and DeepFakes: an overview
- 22. A New Levelof Realism Back in 2021: • A lot of expertise, skill and time was needed • An impersonator who looks like the target (Miles Fisher) https://www.theverge.com/2021/3/5/22314980/tom-cruise-deepfake-tiktok- videos-ai-impersonator-chris-ume-miles-fisher
- 24. Google Veo 3(shark next gen) • Generates Convincing Deepfakes of Riots, Conflict, and Election Fraud. • Veo 3 videos can include dialogue, soundtracks and sound effects. • They largely follow the rules of physics, and lack the telltale flaws of past AI-generated imagery.
- 25. • Our studyfound that people are now able to distinguish between AI-generated and human- authored content only 51% of the time. • We also found that digital media characteristics such as authenticity, modality, and subject matter content, as well as observer demographics such as multilinguism and age, have been proven to further affect an individual’s detection capabilities. • Consequently, it has become vital to deploy alternative countermeasures to combat growing synthetic media misuse. https://cacm.acm.org/research/as-good-as-a-coin-toss- human-detection-of-ai-generated-content/
- 26. https://variety.com/2025/digital/news/deepfake-fraud-caused-200- million-losses-1236372068/ Financial losses fromdeepfake-enabled fraud exceeded $200 million during the first quarter of 2025, according to Resemble AI’s Q1 2025 Deepfake Incident Report, which was released on Thursday. The report suggests an “alarming” escalation and growing sophistication of deepfake- enabled attacks around the world. While 41% of targets for impersonation are public figures — primarily politicians followed by celebrities — the threat isn’t limited to these groups, as another 34% of targets are private citizens, the findings suggest.
- 27. https://spectrum.ieee.org/real-time-audio-deepfake- vishing It’s now possibleto create convincing real-time audio deepfakes using a combination of publicly available tools and affordable hardware. It seems you can’t always believe what you can hear, even if the source is a phone call with a person you’ve known for years.
- 28. DeepFakes and National(cyber)Security A 68-second deepfake video appeared in March 2022 during the third week of Russia’s invasion in Ukraine, depicting Ukrainian President Volodymyr Zelenskyy calling for the surrender of arms. The video appeared on a compromised Ukrainian news web site and was then widely circulated on social media. https://nypost.com/2022/03/17/deepfake-video-shows-volodymyr-zele nsky-telling-ukrainians-to-surrender/ A few days later a video was circulated on social media that supposedly showed Russian President Vladimir Putin announcing that the Russian military was surrendering. A tweet sharing the video with a caption prompted Russian soldiers to lay down their weapons and go home. https://www.snopes.com/fact-check/putin-deepfake-russian-surrender
- 29. Authorized deepfakes https://www.freethink.com/ hard-tech/deepfake-bruce-willis “In 2021,retired actor Bruce Willis gave video production firm DeepCake permission to use deepfake tech to insert his likeness into a series of advertisements — and the AI they trained for the project could allow him to keep entertaining fans well into the future.” “Our general mission is to help celebrities monetize their digital copies while being fully in control of their use and managing the rights.”
- 30. DeepFake Detection (Visual) ●CNN ● Visual Transformers (ViT) ● Capsule Networks Real/ Fake Common architectures Key Ideas • Detect abnormal changes in physiological signals e.g. head poses, eye blinking. • Exploit left-over manipulation artifacts Main Problem Poor generalization to new manipulation techniques.
- 31. Signs of aDeepFake for Detections (in 2020) • Different kinds of artifacts • Blurry areas around lips, hair, earlobs, earrings • Lack of symmetry • Lighting inconsistencies • Fuzzy background • Flickering (in video) https://apnews.com/article/bc2f19097a4c4fffaa00de6770b8a60d
- 32. Signs of aDeepFake (constantly changing) • “…while there are some artifacts today, these artifacts will eventually disappear as generative AI improves….” • “For several years, the earrings in AI-generated faces were often mis-matched.” • In this representative set of 12 images generated using Midjourney 6, this visual artifact appears to no longer exist.
- 33. Signs of aDeepFake (constantly changing) …heart-beat- related signals get lost during the deepfake generation process …this assumption is no longer valid for current deepfake methods…
- 34.
- 35. Multimodal misinformation Aneja, S.,Midoglu, C., Dang-Nguyen, D. T., Khan, S. A., Riegler, M., Halvorsen, P., ... & Adsumilli, B. (2022). ACM multimedia grand challenge on detecting cheapfakes. arXiv preprint arXiv:2207.14534. A low-cost but effective form of misinformation
- 36. Misinformation: False ormisleading information (regardless of intent to deceive). Multimodal Misinformation: Misinformation conveyed through multiple modalities (i.e. images and texts) arising through a falsified relationship or mismatch between the modalities. Visuals can amplify misinformation: Images and videos attract more attention, are shared more widely, and can make false claims more believable. Multimodal Misinformation Li, Y., & Xie, Y. (2020). Is a picture worth a thousand words? An empirical study of image content and social media engagement. Journal of marketing research, 57(1), 1-19. Newman, E. J., Garry, M., Bernstein, D. M., Kantner, J., & Lindsay, D. S. (2012). Nonprobative photographs (or words) inflate truthiness. Psychonomic Bulletin & Review, 19(5), 969-974.
- 37. Out-of-context misinformation “Repurposed imagesused to support misleading narratives.” ● Misleading narratives ● Amplification of bias ● Sensationalism and clickbait ● “CheapFakes” Aneja, S., Bregler, C., & Nießner, M. (2023, June). COSMOS: Catching Out-of-Context Image Misuse Using Self-Supervised Learning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 12, pp. 14084-14092).
- 38. Out-of-context misinformation detectionapproaches ● Deep Learning ● Evidence – based ● Large Foundation models ● Datasets – Evaluation ● Synthetic datasets ● Large Foundation models ● End to end fact checking
- 39. Multimodal Misinformation Detection:Deep Learning Input: Image-Text pair under examination Modality Representation Modality Fusion Optimization Output Truthful Out-of- context Claim Source: https://www.snopes.com/fact-check/glastonbury-greta-thunberg
- 40. Evidence-based Detection Claim EvidenceRetrieval Using Inverse Image Search (Google Images) Verdict “According to reports, the harrowing incident happened in Perugia in Italy. The bus in the video was an Irisbus Iveco Cityclass with an internal combustion engine and compressed natural gas (CNG) fuel tanks mounted on the roof.” “An electric-powered BasiGo bus bursting into flames on Karen Road in Kenya.” Claim Source: https://www.snopes.com/fact-check/electric-bus-fire-kenya Detector
- 41. Related Work: Datasets Syntheticdata: May lack the complexity and realism of real-world data. Annotated datasets: Often limited in scale and diversity. Weak annotations: Can introduce noise, inaccuracies, and label inconsistencies.
- 42.
- 43. SpotFake: Deep LearningDetection SpotFake: ● Visual encoder (VGG) ● Textual encoder (BERT) ● Classifier: Real or Fake 78% acc. on Twitter VMU dataset and 89% on Weibo dataset Singhal, S., Shah, R. R., Chakraborty, T., Kumaraguru, P., & Satoh, S. I. (2019, September). Spotfake: A multi-modal framework for fake news detection. In 2019 IEEE fifth international conference on multimedia big data (BigMM) (pp. 39-47). IEEE.
- 44. Deep Learning Detectionmethods
- 45. The COSMOS datasetand method Aneja, S., Bregler, C., & Nießner, M. (2023, June). COSMOS: catching out-of-context image misuse using self-supervised learning. In Proceedings of the AAAI conference on artificial intelligence (Vol. 37, No. 12, pp. 14084-14092). Random samples: Not realistic training
- 46. The NewsCLIPpings dataset Luo,G., Darrell, T., & Rohrbach, A. (2021, November). NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (pp. 6801-6817). Average human performance (crowdsourced): 65.6% (96.2% truthful, 35.0% out-of-context) Humans recognize only 35% of OCC cases
- 47.
- 48. Consistency-Checking Network (CCN) Abdelnabi,S., Hasan, R., & Fritz, M. (2022). Open-domain, content-based, multi-modal fact-checking of out-of-context images via online resources. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 14940-14949).
- 49. SNIFFER: Detection withLarge Vision-Language Models (LVLMs) Qi, P., Yan, Z., Hsu, W., & Lee, M. L. (2024). Sniffer: Multimodal large language model for explainable out-of-context misinformation detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 13052-13062). ● Multimodal Large Language Models (MLLMs) still lack sophistication in understanding and discovering the subtle cross- modal differences ● two-stage instruction tuning on Instruct-BLIP ○ links generic visual concepts with news-domain entities ○ OOC instruction data generated by GPT-4
- 50. Challenge: Relevant EvidenceDetection Papadopoulos, S. I., Koutlis, C., Papadopoulos, S., & Petrantonakis, P. C. (2025). Red-dot: Multimodal fact-checking via relevant evidence detection. IEEE Transactions on Computational Social Systems.
- 51. RED-DOT: Relevant EvidenceDetection Directed Transformer Evidence retrieval and re-ranking (NewsCLIPpings+ dataset) RED-DOT Architecture
- 52. RED-DOT: Key results Usingtop-1 re-ranked evidence is most effective; additional evidence can introduce noise (less informative). RED-DOT outperformed SotA performance on NewsCLIPings and VERITE (True vs OOC) Trained on NewsCLIPpings+ and evaluated on VERITE (True vs OOC) K,Z+ K,Z-
- 53.
- 54. Challenge: Model Generalizationfrom Synthetic to Real-World Recent approaches rely on synthetic training data (OOC or NEM). Progress is hard to assess with disparate synthetic test sets. > Comparative study on methods for generating synthetic training data (“Synthetic Misinformers”). Papadopoulos, S. I., Koutlis, C., Papadopoulos, S., & Petrantonakis, P. (2023, June). Synthetic misinformers: Generating and combating multimodal misinformation. In Proceedings of the 2nd ACM International Workshop on Multimedia AI against Disinformation (pp. 36-44). DT-Transformer: CLIP ViT B/32 and L/14 Transformer Encoder COSMOS Test set 850 Truthful, 850 False samples (real-world data) VisualNews ~1M image-text pairs from credible news organizations
- 55. Synthetic Misinformers OOC-based Methods ●Random sampling ● Similarity-based sampling NEM-based Methods ● Random entity swapping (from the same type) ● Rule-based entity swapping ● Similarity-based retrieval + Rule-based entity swapping Hybrid Methods
- 56. Synthetic Misinformers: Keyresults High accuracy on synthetic test data (NewsCLIPpings) Low generalization on real-world data (COSMOS) Unimodal bias in NEM-based and Hybrid methods.
- 57. Named Entity Manipulationsoften produce simplistic misinformation. Instead, we created the “MisCaption This!” datasets by leveraging LVLMs to generate more robust synthetic training data. Training on “MisCaption This!” improved performance by +10.4% on real-world data. “MisCaption This”: LVLM-generated miscaptioned images Papadopoulos, S. I., Koutlis, C., Papadopoulos, S., & Petrantonakis, P. C. (2025). Latent Multimodal Reconstruction for Misinformation Detection. arXiv preprint arXiv:2504.06010.
- 58. Latent Multimodal Reconstruction(LAMAR) Papadopoulos, S. I., Koutlis, C., Papadopoulos, S., & Petrantonakis, P. C. (2025). Latent Multimodal Reconstruction for Misinformation Detection. arXiv preprint arXiv:2504.06010. Developed LAMAR (Latent Multimodal Reconstruction): a Transformer-based architecture trained to reconstruct truthful caption embeddings. Achieved new state-of-the-art on VERITE (+5.6%), +8-10% compared to preview approach
- 59. Challenge: Accounting forUnimodal Bias Hypothesis: “Asymmetric multimodal misinformation” as a contributing factor to unimodal bias Estimate: Only 52% of COSMOS (test) samples show genuine multimodal misinformation (41% associative images, 7% reinforcing text) Problem: Biased or unimodal models can perform well artificially -> difficult to assess progress.
- 60. VERITE Benchmark (Verificationof Image-Text pairs) 338 truthful pairs, 338 “Miscaptioned” pairs, and 324 “Out-of-Context” pairs from Snopes and Reuters Excluding “Asymmetric” samples and leveraging “Modality Balancing” Papadopoulos, S. I., Koutlis, C., Papadopoulos, S., & Petrantonakis, P. C. (2024). Verite: a robust benchmark for multimodal misinformation detection accounting for unimodal bias. International Journal of Multimedia Information Retrieval, 13(1), 4.
- 61. VERITE Benchmark: Keyresults Evaluation on VERITE (trinary) Evaluation on synthetic test data (binary) Multimodal > Unimodal counterparts. “Modality balancing” reduces bias. No unimodal bias in “True vs. OOC” evaluation; but some bias remains in “True vs. MC” due to text artifacts. OOC methods: No unimodal bias. NEM-based data: High text-only accuracy signals unimodal bias, → though multimodal models still perform best.
- 62.
- 63. End-to-end fact-checking “End-to-End Fact-checking”: 1.Multimodal Evidence Retrieval 2. Claim Verification 3. Explanation Generation Yao, B. M., Shah, A., Sun, L., Cho, J. H., & Huang, L. (2023, July). End-to-end multimodal fact-checking and explan ation generation: A challenging dataset and mod els . In Proc. of 46th International ACM SIGIR Conference (pp. 2733-2743). S-BERT + CLIP BERT
- 64. Task objective: Developreal-time fact-checking tools to assist journalists and fact-checkers during political debates and interviews. End-to-End Pipeline 1. Monitor live speech 2. Identify claims that are worth examining * 3. Retrieve relevant information and articles from the web 4. Filter & re-rank results based on relevance and reliability 5. Generate assessment & explanation 6. Present results to the journalist Integrate open-source, efficient large language models, foundation models, search APIs, and task-optimized specialized models for effective real-time performance. * Currently working on (2), based on the “CLEF 2024 - CheckThat!” Challenge ELLIOT: Real-time, end-to-end fact-checking
- 65. Open-source, efficient LargeLanguage Models, Foundation Models: could prove useful for most tasks Identify check-worthy claims: Important because it is neither efficient nor feasible to fact-check every sentence. The system must prioritize “check-worthy” claims. Training Models using the CLEF 2024 CheckThat! Dataset. zero-shot and few-shot performance of LLMs. Apply Parameter-Efficient Fine-Tuning (PEFT) on LLMs for improved performance. Retrieve relevant articles via search APIs LLMs can help generate or refine search queries passed to the API. Filter and re-rank retrieved information based on reliance and reliability LLMs can help identify the most relevant and informative passages within articles. The final system will likely combine specialized trained models with LLM-based reasoning. ELLIOT: Real-time, end-to-end fact-checking
- 66. ELLIOT: Building Europe’s MultimodalGeneralist Foundation Models for Science and AI Dr Yiannis Kompatsiaris (ikom@iti.gr) ITI-CERTH, ELLIOT Coordinator
- 67. 68 Develop the nextgeneration of Open Multimodal (videos, images, text, 3D, sensor signals, industrial time series, satellite feeds, … ) Generalist & Fine-Tuned Trustworthy Foundation Models (MGFMs) – along with open datasets and methods for evaluating MGFMs in a fully reproducible manner, at a level & scale currently only feasible by big tech ELLIOT’s main objective
- 68. 69 30 partners, 12countries, 4 years, 28.5M € funding • 30 partners from 12 European countries • 4 years (1 July 2025 – 30 June 2029) • 25M € funding from the EC + 3.5M funding from the Swiss government
- 69. 70 ELLIOT concept • Inputs •Infrastructure • Activities • Outputs
- 70. 71 ELLIOT use cases 6Application Domains 11 Use Cases Variety of data modalities Diverse downstream tasks End user involvement
- 71. Conclusions and FutureDirections ● There has been significant progress in the field but challenges remain; i.e., model generalization, unimodal biases, and evaluation robustness. ● Need to expand task definition: include both out-of-context (OOC), miscaptioned (MC) images, and other manipulations. ● Synthetic datasets often lack real-world complexity; LVLM-generated data shows promise. ● Need for large-scale, real-world datasets with high quality (but not leaked) external evidence. ● Current research is focused on LVLM-based evidence retrieval and verification. ● Broaden focus beyond image-text pairs to audio, video, multilingual, and cross-lingual misinformation. ● Automated tools should be build to augment, not replace, human fact-checkers by prioritizing and assisting in evidence collection and large-scale monitoring. ● Continued interdisciplinary efforts and ethical vigilance are essential to advance robust and trustworthy detection systems.
- 72.
- 73. AFP • 2017: Onefact checker in Paris • 2022: 130 fact checkers, 80 countries, 24 languages
- 74. • OSINT &fact-checking: Collaboration on cross-border fact- checking and sharing best practices on a dedicated member-only platform. • Resource-sharing: Participating members will have access to intelligence and information on tools and workflows to enhance their OSINT and fact-checking capabilities. • Training & development: The network will provide courses, tutorials and interactive discussions for members – led by Eurovision News in collaboration with the EBU Academy – to expand their fact- checking and OSINT skills.
- 75.
- 76. What happens whenit becomes multiple on the hour? https://www.niemanlab.org/2024/05/indian-journalists- are-the-frontline-against-election-deepfakes/
- 77. Why Tools andFact Checking is not enough?
- 78. Human Behaviour relatedchallenges … participants' agreement with the news position was not attenuated by the explicit post hoc correction. Considering that information is judged as truth when it meets intuitive evaluation criteria (e.g., familiarity, compatibility with existing knowledge)… debunking its falsehood may not be sufficient to undo it. Confirmation Bias
- 79. Human Behaviour relatedchallenges Current psychological research posits that conspiracy theorists resist facts because their beliefs serve some internal need, such as belonging to a group, maintaining a sense of control over their circumstances or feeling special. https://spectrum.ieee.org/ai-conspiracy-theories
- 80. Human Behaviour relatedchallenges “An important conclusion of the study is that moral flexibility may be a reason political misinformation can be effective. ” “People are OK with politicians not telling the truth because they see it as morally justified, which is more concerning than if someone had tricked them (into believing a statement),” Moral Flexibility https://www.colorado.edu/today/2024/03/04/facts-ignored-truth-flexible-when-falsehoods-support-political-beliefs
- 81. Role of SocialMedia platforms ...”As it turns out, much like any video game, social media has a rewards system that encourages users to stay on their accounts and keep posting and sharing. Users who post and share frequently, especially sensational, eye-catching information, are likely to attract attention"… … The more moral outrageous language you use, the more inflammatory language, contemptuous language, the more indignation you use, the more it will get shared. So we are being rewarded for being division entrepreneurs. The better you are at innovating a new way to be divisive, we will pay you in more likes, followers and retweets.… Social Media Rewarding Scheme
- 82. Sophisticated Campaigns Narrative launderingis a process with the goal to conceal the origins and sourcing of false or misleading information 1. Placement, the initial posting of the false information relying on inauthentic social media accounts for this purpose. 2. Layering, the spread of information to more credible sources. Repetition brings a perception of credibility, employing both authentic and inauthentic social media accounts. 3. Integration, the information becomes endorsed by more credible and genuine sources and is widely disseminated by real users. Linvill, Darren and Warren, Patrick, "Infektion’s Evolution: Digital Technologies and Narrative Laundering" (2023). Media F orensics Hub Reports. 3, https://tigerprints.clemson.edu/mfh_reports/3 Narrative Laundering
- 83. “engagement at all costs”business model Financial – Bussinnes Aspects
- 84. “Making people hasn’tbeen that quick since Eden” https://www.theregister.com/2023/04/28/ tencent_digital_humans/ Providing Relevant Services Fraud Gathering donations
- 85. Financial Gains throughDisinformation https://www.washingtonpost.com/nation/2024/02/21/covid- misinformation-earnings/ “Four major nonprofits that rose to prominence during the coronavirus pandemic by capitalizing on the spread of medical misinformation collectively gained more than $118 million between 2020 and 2022, enabling the organizations to deepen their influence in statehouses, courtrooms and communities across the country, a Washington Post analysis of tax
- 86. What can bedone?
- 87.
- 88.
- 89. Μedia literacy toreach younger audiences One of the 8 recommendations by the American Psychological Association for countering fake narratives is to #prebunk misinformation ‘to inoculate susceptible audiences by building skills and resilience from an early age.’
- 90. Build Immunity againstDisinformation Exposure to manipulated content can prepare people to detect future manipulations
- 91. The country inoculatingagainst disinformation https://www.bbc.com/future/article/20220128-the-country-inoculating-against-disinformation
- 92. Understand human behavior “It’simportant to avoid a false dichotomy,” says van der Linden. “I suspect both needs and evidence play a role in persuading conspiracy believers.” https://spectrum.ieee.org/ai-conspiracy-theories Human behavior
- 93. Counter-disinformation policies The EUis working in close cooperation with online platforms to encourage them to promote authoritative sources, demote content that is fact-checked as false or misleading, and take down illegal content or content that could cause physical harm. Policies -
- 94.
- 95.
- 96. Counter-disinformation policies (DSA) “…These previous RFIs covered issues such as terrorist content, risk management related to civic discourse and election processes, and the protection of minors … Meta's risk assessment and mitigation measures reports, the protection of minors, elections and manipulated media.”
- 97.
- 98.
- 99. Key Challenges inAI against Disinformation ● Arms race nature of problem ○ Constantly improved AI models for synthetic media ○ New disinformation tactics ● Gap between research and end users ○ Journalists/fact-checkers need intuitive tools ○ Tools need to be robust and trustworthy ● Risks of discrimination ○ AI models might be biased against certain demographics ○ Use of AI models might be done in a biased manner ● Sustainability ○ AI costs a lot in terms of research, maintenance and deployment ○ Fighting disinformation is not a profitable business ○ Required contribution from various disciplines ○ Content Analytics - NLP, Machine Learning, Network Analysis, Big Data Architectures, Psychology – Social Sciences (patterns of presentation, sharing), Visualization ● Continual learning ● Out-of-domain generalization ● Explainability ● Human agency ● Robustness ● AI fairness ● Green AI ● Model compression ● Knowledge distillation
- 100.
- 101. With contributions from SymeonPapadopoulos papadop@iti.gr Senior Researcher, CERTH-ITI MeVer Group Leader Nikos Sarris nsarris@iti.gr Senior Researcher Stefanos-Iordanis Papadopoulos stefpapad@iti.gr MeVer Group
- 102. Thank you foryour attention! https://mklab.iti.gr Get in touch! Yiannis Kompatsiaris ikom@iti.gr
Editor's Notes
- #3 http://irevolution.net/2014/04/03/using-aidr-to-collect-and-analyze-tweets-from-chile-earthquake/
- #5 intentional (and often coordinated) efforts to spread misleading information to the public
- #21 Amy Adams: Lois Lane superman man of steel
- #31 Capsule: This provides the advantage of recognizing the whole entity by first recognizing its parts. The input to a capsule is the output (or features) from a CNN.
- #36 Optional slide! If you need more examples and context.
- #38 One type of multimodal misinformation: Out-of-context image-text pairs, refers to the visual content being detached from its original context, altering its intended meaning. Misleading narratives: Visuals can be manipulated or selectively chosen to support false narratives or misrepresent events, leading to distorted perceptions. Amplification of bias: Out-of-context disinformation can reinforce existing biases or stereotypes by presenting a distorted view of individuals, groups, or events. Sensationalism and clickbait: Out-of-context visuals are often used to generate sensational headlines or attract attention, increasing the likelihood of content sharing without critical evaluation. Example in Figure: (Red) denotes false captions and (green) shows the true captions along with year published. It shows two fact-checked examples where the two captions mean something very different, but describe the same parts of the image.
- #39 One type of multimodal misinformation: Out-of-context image-text pairs, refers to the visual content being detached from its original context, altering its intended meaning. Misleading narratives: Visuals can be manipulated or selectively chosen to support false narratives or misrepresent events, leading to distorted perceptions. Amplification of bias: Out-of-context disinformation can reinforce existing biases or stereotypes by presenting a distorted view of individuals, groups, or events. Sensationalism and clickbait: Out-of-context visuals are often used to generate sensational headlines or attract attention, increasing the likelihood of content sharing without critical evaluation. Example in Figure: (Red) denotes false captions and (green) shows the true captions along with year published. It shows two fact-checked examples where the two captions mean something very different, but describe the same parts of the image.
- #44 A simple and very common approach relies on early or late fusion (late in this case). Methods like “SpotFake” which utilizes two encoders, one visual (e.g VGG) and one textual encoder (e.g BERT), combine the features (e.g concatenation) and use a classifier (fully connected layer) for binary classification. Numerous papers have relied on similar architectures and experiment with different visual and textual encoders.
- #46 COSMOS is an example of an early approach to multimodal misinformation detection. It leverages self-supervised training, combining a vision model (Mask R-CNN) with a text model (Universal Sentence Encoder). The network is trained on triplets consisting of: image, matching/correct texts, and randomly selected out-of-context texts used as negative samples. However, because these negative samples are chosen randomly, there’s usually no meaningful relationship between the image and the mismatched text. This makes the training setup less realistic, and as a result, the model struggles to generalize to real-world misinformation, where out-of-context manipulations are often much more subtle.
- #47 To address this limitation, the NewsCLIPpings dataset introduces a more realistic approach by using pre-trained models to generate hard negative samples — that is, out-of-context image–text pairs. For example, a query about Angela Merkel is matched with its true image (top left), while additional images are retrieved based on text–image or text–text similarity, using various pre-trained models (CLIP, or SBERT, or ResNet) This method produces more challenging and realistic synthetic training data. Even human performance on this dataset was relatively low (~65%) highlighting the difficulty of the task.
- #49 Examining inconsistencies within image-text pairs alone isn’t always enough to verify their truthfulness. In practice, fact-checkers consult external information to cross-examine the claim. Here, the authors propose an automated approach that does the same, retrieving related evidence from the web using the NewsCLIPpings dataset as a base. They expand it by: Using the text to retrieve relevant images, and Using the image to retrieve related texts or articles. Their framework then assesses the consistency between the claim and the retrieved evidence (both text-to-text and image-to-image) in addition to the image–caption pair itself. + They introduce two “reasoning” modules: A visual reasoning module using ResNet-152 and a memory network (attention mechanism). A textual reasoning module using BERT, an LSTM, and a similar memory network. Each examines intra-modal relationships between the claim and its evidence. The outputs of these reasoning modules are concatenated with CLIP features and passed to a final classifier. This model achieves 84.7% accuracy, a substantial improvement over the 66% baseline without external knowledge.
- #50 Up until recently, most existing methods focused on assessing image-text consistency, but they often neglected to provide convincing explanations for their decisions. A more recent approach, called SNIFFER, introduces a two-stage verification framework: Internal checking : evaluates the consistency between the image and its accompanying text. External checking : assesses the relevance between the image-retrieved evidence and the original text. The results of these two analyses are then combined to produce a final judgment along with a composed reasoning-based explanation. SNIFFER is built on InstructBLIP and trained through two-stage instruction tuning: The first stage improves concept alignment, linking generic visual concepts with news-domain entities. The second stage fine-tunes the model using out-of-context (OOC) instruction data generated by GPT-4, enhancing its ability to discriminate subtle inconsistencies.
- #51 Our works so far did not leverage external evidence. And prior works on evidence-based detection assumed that all external information collected from the web to be relevant. for example: 10 images + 10 texts from the web -> attention mechanism -> prediction To address this limitation we created the Relevant Evidence Detection” (RED) a module is explicitly trained to discern whether each piece of evidence is relevant, to support or refute the claim. Example: caption “people tearing down a 5G tower to stop the spread of COVID, in China” -> retrieve articles and images from the web (based on reverse image search to get articles, and text search to get images) -> only one article title is relevant in this case: these were smart lamps, toppled by anti-surveillance demonstrators. Nothing to do with covid.
- #52 (left) first step of our method: re-rank the collected evidence (using pre-trained foundation language-vision models, i.e. OpenAI’s CLIP). Re-ranking gets the most relevant information at the top, in this case. Then: - We used hard negative sampling to define “irrelevant evidence” - We use CLIP features + Transformer encoder + modality fusion (a simple yet effective idea of combining operations: concatenation, addition, subtraction, multiplication) + multi-task learning where each piece of evidence is predicted as relevant or not relevant. - We explored multiple architectural variants (e.g., single or dual-stage) and mechanisms (such as “guided attention”).
- #53 (left) Variation of REDDOT : Using re-ranked evidence, a single items (1 image retrieved from text-> image search AND 1 article retrieved from reverse image search) tends to get the best performance. Additional items/evidence introduce noise (less informative) and reduce performance. -> importance of re-ranking and relevant-evidence-detection. (right) significant improvements over the SotA on the VERITE benchmark by up to 33.7%. AND evidence-based SotA on NewsCLIPings+ by up to 3%
- #55 (our 1st work) We observed two limitations on the recent literature: 1) lack of comparisons between different methods for generating OOC and 2) lack of evaluation on real-world misinformation. Difficult to assess progress in the field. We conducted a comparative study on existing and new “Synthetic Misinformers” and evaluated their performance onto real-world misinformation; (COSMOS benchmark) Pipeline: starting from a real news dataset (VisualNews : image - caption pairs), define a method for generating false/ooc samples. Encoder (i.e. CLIP) + Modality Fusion (i.e. concatenation) and/or Attention/Transformer (We used the Detection Transformer) → Evaluation on COSMOS
- #56 We explored various “Synthetic Misinformers” to create synthetic training datasets. Including both “OOC” (out-of-context)-based methods; simple random sampling, or similarity-based sampling and Named entity manipulations (NEM): 1) randomly swapping entities of the same type (dates with other dates, locations with other locations, etc), 2) swapping entities based on specific rules 3) combination of similarity-based (to maintain some semantic similarity) + rules Hybrid: combine OOC and NEM
- #57 We observe high performance on the synthetic test set (left fig). Using our DT-Transformer architecture, it reached 77 - 84% on the NewsCLIPpings dataset. However, this does not necessarily translate to good performance on real-world data (right figure:evaluation on the COSMOS benchmark). Highest performance was ~60% (binary classification) + we identified unimodal biases. Text-only methods competed and even outperformed multimodal ones (on a supposedly multimodal task!)
- #58 Afterwards, we shifted our focus to “miscaptioned images” which appeared to be a more challenging task. Due to the lack of large, annotated datasets of miscaptioned images, existing datasets are often synthetic, created by swapping named entities (e.g., replacing the date of an event with another date). To address this, we leveraged the power of Large Vision-Language Models (LVLMs) to create the “MisCaption This!” dataset, producing more realistic and diverse synthetic samples. In the figure: swapping named entities (e.g., locations or dates) often produced implausible claims, like placing the “Indian parliament” in “Bangladesh” (obvious epistemic contradiction) while the image showed Greek flags. This highlights the limits of surface-level substitutions. In contrast, LVLMs generate more nuanced and believable misinformation. For example, in this case it reframed an anti-austerity protest as pro-austerity, reinforcing the false narrative with phrases like “successful implementation” and “widespread approval”.
- #59 Furthermore, we developed a novel architecture: LAMAR (short for Latent Multimodal Reconstruction), a neural network trained to reconstruct truthful caption embeddings. This provides a strong auxiliary signal for detection and achieves state-of-the-art results on both synthetic datasets (e.g., NewsCLIPpings) and real-world evaluation benchmarks (VERITE). First, the original (truthful) caption is manipulated by an LVLM. Then, LAMAR takes the image and the manipulated caption embeddings as input and is tasked with recovering the original caption embedding. This reconstructed embedding is integrated into the final detection network alongside the fused multimodal representation. Our rationale: - LVLMs generate more diverse and realistic synthetic training data. - Reconstruction provides an auxiliary signal that captures cross-modal and intra-modal consistency, improving generalization.
- #60 (1) Initial motivation: datasets like NewsCLIPings are algorithmically created. Therefore good performance on these dataset does not necessarily translate to good performance on real-world misinformation detection. For this reason, have created the VERification of Image-TExt pairs” (VERITE) benchmark, to evaluate the generalizability of methods onto real-world misinfo. (2) second reason for creating VERITE: we identified unimodal bias in widely used datasets for multimodal misinformation: VMU Twitter and COSMOS. Where biased or unimodal models outperform their multimodal counterparts in a supposedly multimodal task. We designed the VERITE benchmark to mitigate this problem. We show how certain “asymmetries” exacerbate unimodal bias. Example: a claim pertains to “deceased people turning up to vote” and an image merely thematically related to the claim, is added primarily for cosmetic enhancement. Similarly, a manipulated image and a caption the simply repeats the depicted misinformation. Focusing on the dominant modality, a robust text-only or image-only detector would suffice for detecting misinformation; rendering the other modality inconsequential in the detection process. -> exacerbate unimodal bias In contrast, the example on the left: direct relationship between the text and the image. I.e. “grounds covered with garbage after Greta Thunberg’s speech in Glastonbury Music Festival in June 2022” and the image is used as evidence for the claim. (However this is taken out-of-context, the image is from 2015)
- #61 We collected fact-check articles (from Snopes and Reuters), applied a series of strict criteria, refined the false and truthful claims (as seen in the example). This process resulted in creating: “Truthful” image-text pairs, and “Miscaptioned” image-text pairs. Moreover, based on the truthful claim -> extracted relevant keywords -> retrieve some relevant but out-of-context image from search engine API. Advantage: most previous datasets were binary (true/false). We define three classes: True, Miscaptioned Images and Out-of-context pairs. + a unimodal or biased model can’t perform well on average. because the image is important for the claim (by avoiding asymmetric examples) + both modalities exist in both truthful and false samples, so a biased model can’t simply focus on one modality (modality balancing)
- #62 (left) evaluation on synthetic testing data (binary classification) OOC-based methods exhibit No unimodal bias. NEM-based methods: high text-only accuracy → signals unimodal bias, though multimodal models still perform best. (right) models trained for multi-class classification (True, VS Out-of-context, VS miscaptioned), datasets: on NewsCLIPings dataset (for OOC) + some “Miscaptioned” dataset. Evaluation on VERITE: Multimodal models consistently outperform their unimodal counterparts. -> “Modality balancing” reduces bias. No unimodal bias in “True vs. OOC” evaluation; but some bias remains in “True vs. MC” due to text artifacts.
- #64 From specific tasks, to end-to-end fact-checking: Prior methods relied on Image-Text pairs + tried to cross-examine them + identify internal inconsistencies. But fact-checking typically requires external information! -> End-to-end multimodal fact-checking The authors address multimodal fact checking with three subtasks: 1) multimodal evidence retrieval 2) claim verification 3) explanation generation And create the Mocheg dataset, consisting of 15L TEXTUAL claims along with collected evidence and explanations (provided by fact-checkers e.g. SNOPES). Evidence Retrieval: calculate the similarity between the input text claim and all texts in the “possible evidence” corpus. Rank the top-K texts based on Sentence BERT (S-BERT). Image retrieval: use CLIP to calculate the cross-modal (image-text) similarity between the CLAIM and ALL images. Retrieve the M most similar images. Use the Ranked Text Evidence, Ranked Image Evidence and Text Claim (concatenate S-BERT and CLIP features) for Claim verification. Use a BERT encoder-decoder for explanation generation. Limitations: they only provide textual claims, not multimodal misinformation (though they utilize multimodal evidence) + their evidence can be considered “leaked” since they are collected from within fact-checked articles. (very unrealistic scenario. does not generalize to new and emerging misinformation). Current works are trying to mitigate these limitations.
- #65 Next stage: ELLIOT. real-time , end to end fact-checking [“Open-source, efficient large language models, foundation models”: could prove useful for most tasks.] Monitor speech: Audio-to-Text models (We are not developing new models here. We rely on existing models) Identify check-worthy claims: Important because it is neither efficient nor feasible to fact-check every sentence. The system must prioritize “check-worthy” claims. We have trained small models using the CLEF 2024 CheckThat! Dataset. We are also experimenting with zero-shot and few-shot performance of LLMs. Next: apply Parameter-Efficient Fine-Tuning (PEFT) on LLMs for improved performance. Retrieve relevant articles via search APIs LLMs can help generate or refine search queries passed to the API. Filter and re-rank retrieved information based on reliance and reliability We will use predefined lists of credible sources (provided by project partners). LLMs can help identify the most relevant and informative passages within articles. The final system will likely combine specialized trained models with LLM-based reasoning.
- #66 Next stage: ELLIOT. real-time , end to end fact-checking [“Open-source, efficient large language models, foundation models”: could prove useful for most tasks.] Monitor speech: Audio-to-Text models (We are not developing new models here. We rely on existing models) Identify check-worthy claims: Important because it is neither efficient nor feasible to fact-check every sentence. The system must prioritize “check-worthy” claims. We have trained small models using the CLEF 2024 CheckThat! Dataset. We are also experimenting with zero-shot and few-shot performance of LLMs. Next: apply Parameter-Efficient Fine-Tuning (PEFT) on LLMs for improved performance. Retrieve relevant articles via search APIs LLMs can help generate or refine search queries passed to the API. Filter and re-rank retrieved information based on reliance and reliability We will use predefined lists of credible sources (provided by project partners). LLMs can help identify the most relevant and informative passages within articles. The final system will likely combine specialized trained models with LLM-based reasoning.