This document discusses a method for fusing the outputs of multiple image forensics algorithms to localize tampered regions in images. The authors develop an operation-wise attention fusion network (OwAF) that takes as input the localization maps from five forensic algorithms and produces a single, easier to interpret localization map. Evaluation on three datasets shows the OwAF approach outperforms individual algorithms and a competing fusion method in terms of localization accuracy. However, the authors note limitations in generalizability and future work to address this through increased training data and alternative fusion architectures.

1. Operation-wise Attention Network for
Tampering Localization Fusion
Polychronis Charitidis, Giorgos Kordopatis-Zilos, Symeon Papadopoulos, Ioannis
Kompatsiaris
MeVer Team @ Information Technologies Institute (ITI) /
Centre for Research & Technology Hellas (CERTH)
Content-Based Multimedia Indexing Conference, June 28-30, 2021

2. WeVerify Project
● Goals
○ Address the advanced content verification challenges
○ Social media and web content analysis for detection of disinformation
○ Exposure of misleading and fabricated content
○ Platform for collaborative, decentralised content verification, tracking, and debunking.
● Developed tools
○ DeepFake detection service
○ Image Verification Assistant

3. Image Verification Assistant
● Goal: forgery localization in images.
● Report from various image forensics algorithms.
○ JPEG based methods, Noise-based methods, Deep-learning based methods
○ Focuses on splicing and copy-move manipulations
● Inspect the multiple reports in tandem.
Tampered Image Mask Localizations from Forensic Algorithms
Source: DEFACTO dataset

4. Motivation
● Observations:
○ Many forensics output visualizations increases the complexity of the results, especially for
non-experts.
■ Each algorithm has requires specific knowledge for proper interpretation.
○ Some of these forensics results are complementary to each other so their combination could
potentially lead to better results
● Solution and contributions:
○ Develop a fully automatic fusion approach that is able to combine diverse forensics signals.
○ The combined result:
■ is more robust and accurate
■ is easier to interpret and requires no specialized knowledge
■ empower non-experts in image verification

5. Methodology
● For this work we select 5 forensics algorithms for fusion.
● These algorithms were selected among others based on their performance on
forgery localization datasets
○ ADQ1 and DCT that both base their detection on analysis of the JPEG compression, in the
transform domain
○ BLK and CAGI that base their detection on analysis of the JPEG compression in the spatial
domain
○ Splicebuster which is a noise-based detector
● Train a deep learning architecture to fuse the diverse tampering localization
algorithms
○ Fully automatic
○ Complex and diverse features
○ Availability of large-scale datasets

6. Models
● We considered two different models:
○ Eff-B4-Unet: A U-Net based architecture that uses Efficient-B4 as an encoder
○ Operation-wise Attention Fusion network (OwAF), which is an adapted image
restoration architecture
■ Operation-wise Attention layer:

7. Training and Evaluation process
● Training dataset:
○ DEFACTO dataset (Mahfoudi et al., 2019)
○ Contains various synthetic manipulations like splicing and copy-move
○ 15,000 tampered images / 75,000 forensics algorithms localizations
● Evaluation datasets:
○ DEFACTO test dataset
■ Contains 1000 tampered images
○ CASIA V2.0 dataset (Dong et al., 2013)
■ Contains 5,123 tampered images
○ The IFS-TC Image Forensics Challenge set
■ Contains 450 tampered images
● Compared our approach with another fusion approach (Iakovidou et al., 2020)
● Metrics: F1, IoU

8. Results on DEFACTO test dataset
MODELS MACRO-F1 IOU
BLK 0.463 0.053
ADQ1 0.573 0.123
CAGI 0.479 0.072
DCT 0.509 0.101
Splicebuster 0.554 0.087
Eff-B4-Unet 0.908 0.690
OwAF 0.912 0.707

9. Results on CASIA v2 dataset
MODELS MACRO-F1 IOU
BLK 0.509 0.089
ADQ1 0.573 0.130
CAGI 0.502 0.094
DCT 0.546 0.113
Splicebuster 0.576 0.093
Iakovidou et al. (2020) 0.598 0.166
OwAF 0.611 0.172

10. Results on IFS-TC dataset
MODELS MACRO-F1 IOU
BLK 0.459 0.063
ADQ1 0.485 0.076
CAGI 0.506 0.091
DCT 0.467 0.065
Splicebuster 0.560 0.129
Iakovidou et al. (2020) 0.549 0.112
OwAF 0.529 0.106

11. Discussion and Limitations
● The reported experimental results are promising and in many cases
outperform the individual forensics techniques.
● Our automatic approach outperforms a competing fusion approach in many
cases.
● The results of our approach are easier to interpret by non-experts.
● An important limitation of this work is the generalization ability of the
fusion model.
● Our approach performance depends on the performance of the individual
forensic algorithms.

12. Future work
● To deal with the generalization, we will try to increase the size of the training
dataset and include different manipulations from other datasets.
● We will experiment with task-specific regularization techniques, like
localization map dropout.
● We plan to experiment with multi-stream fusion architectures that besides the
forensics localization maps, will consider the input image itself.

13. Thank you!
Polychronis Charitidis / charitidis@iti.gr
Media Verification Team / https://mever.gr / @meverteam
WeVerify project / http://www.weverify.eu / @WeVerify