(Social) Multiimedia Forensics

Social Multimedia Forensics
Prof. Sebastiano Battiato
Dipartimento di Matematica e Informatica,
Università di Catania
Image Processing LAB – http://iplab.dmi.unict.it
iCTLab - www.ictlab.srl
battiato@dmi.unict.it
ICT Doctoral School, Trento - May 2017

Team UNICT
2 FULL PROFESSORS
1 ASSOCIATE
PROFESSOR
1 ASSISTANT
PROFESSOR
1 POST DOC
9 PH.D. STUDENTS
1 CONSULTANT
Computer Vision (First Person Vision)
Digital Forensics
Video Analytics (e.g. Video Surveillance, Digital Signage, etc)
Medical Imaging
Archeomatica

Research Contract with ParkSmart on «Road Traffic Analysis and Assisted Parking», 2015 (1 Phd
Fellowship)
Research Contract with JOL Wave Telecom on «Study, definition and development of an ultra-broadband
multi-device ecosystem for multimedia services” - 2014 (+1 Phd Fellowship)
Research Contract with JOL Wave Telecom on «Visual Sentiment Analysis», 2015 (+1 Phd Fellowship)
Research Contract with Centro Studi on «Computer Vision application on retail and DOOH», 2015 (2 Phd
Fellowship)
Joint IPLAB - iCTLab Phd Fellowship PON FSE-FESR 2014-2020 (Dottorati industriali).
Joint Lab with STMicroelectronics (Catania +1 Phd fellowship (2016)
Research Contract with Osram +1 Phd Fellowship (2016)
Research Contracts / Technological
Transfer

VISMAC - GRADO, JUNE 2016
• Support and technical advices to investigations, referring more
precisely on multimedia contents analysis (Images, Videos, etc.).
• Applied and basic R&D in partnership with private and public
institutions.
–Polizia Scientifica ( Direzione Nazionale Roma)
–RIS (Reparto Investigativo Speciale) - Carabinieri Messina
–IISFA (Italian Information System Forensics Association)
–Telefono Arcobaleno
–NIT (Nucleo Investigativo Telematico) – Siracusa
Forensics Partnerships & Projects

IPLab
Key Forensics Competences
Near Duplicate Image
Detection
Tampering detection
exploiting hash
signatures
Double JPEG compression
artifacts analysis for
forgery detection

Multimedia Forensics is based on the idea
that inherent traces (like digital fingerprints)
are left behind in a digital media during both
the creation phase and any other
successively process.

I cacciatori di bufale digitali: «Così
staniamo i falsi» - CorriereTV (2017)

Multimedia Forensics

Camera Ballistics
• Example:
• Forensic analysis of a smartphone: which pictures have been
generated on the device and which ones have been generated
by other devices and sent by messaging application or saved
from the internet
• We can identify:
• Type of device
• Maker and model
• Specific exemplar
• Which Device Has Created This Picture?

Current Trends: Point&Shoot
and Share…

https://ourworldindata.org/internet/

Social Networks Upload
https://cewe-photoworld.com/how-big-is-snapchat/ - 20/11/2016

Future of Imaging
Nikon

Sharing

Next Steps

Next Steps (2)

Social
Picture
Battiato, S., Farinella, G. M., Milotta, F. L., Ortis, A., Addesso, L., Casella, A., ... & Torrisi, G. (2016, June).
The Social Picture. In Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval
(pp. 397-400). ACM.

The Social Picture
S. Battiato, G.M. Farinella, F.L.M. Milotta, A. Ortis, L. Addesso, A. Casella, V. D’amico, G. Torrisi, “The
Social Picture”, In Proceedings of ACM International Conference on Multimedia Retrieval (ICMR)
2016, New York. ICT Doctoral School, Trento - May 2017

The Social Picture
S. Battiato, G. M. Farinella, F. L. M. Milotta, A. Ortis, L. Addesso, A. Casella, V. D'amico, G. Torrisi, The Social
Picture, ACM International Conference on Multimedia Retrieval 2016

Social (Multimedia) Forensics
• Image and Video Phylogeny
ReVeal project

Social (Multimedia) Forensics
• Uploading an image on a Social Network
- The process alters images
- Resize
- Rename
- Meta-Data deletion/editing
- Re-Compression
- NEW JPEG file Structure
M. Moltisanti, A. Paratore, S. Battiato, L. Saravo - Image Manipulation on Facebook for
Forensics Evidence – ICIAP 2015, LNCS 2015;
O. Giudice, A. Paratore, M. Moltisanti, S. Battiato - A Classification Engine for Image
Ballistics of Social Data – (Arxiv 2016 No. 1699257) http://arxiv.org/abs/1610.06347

Huawei (2016)

Social MF on Facebook
Preliminar studies involving:
- Different devices
- Dataset of images:
- Different Scene (outdoor artificial,
outdoor natural, indoor)
- Different Quality (resolution and
compression)
- Different upload setting
has proven that some invariance could be
guaranteed.
Some specific editing could be then traced and
used to retrieve useful info about image before
uploading.
Moltisanti, Paratore, Battiato, Saravo - Image Manipulation on Facebook for Forensics Evidence
– ICIAP 2015, LNCS 2015;

Social (Multimedia) Forensics (2)
- Each Social Network Service do different alterations
Resized
Proportionally
Squared
Image
O. Giudice, A. Paratore, M. Moltisanti, S. Battiato - A Classification Engine for Image Ballistics of
Social Data – (Arxiv 2016 No. 1699257) http://arxiv.org/abs/1610.06347ICT Doctoral School, Trento - May 2017

- Each Social Network Service makes different alterations
Social Data – (Arxiv 2016 No. 1699257) http://arxiv.org/abs/1610.06347

O. Giudice, A. Paratore, M. Moltisanti, S. Battiato - A Classification Engine for Image Ballistics of Social Data – (Arxiv
2016 No. 1699257) http://arxiv.org/abs/1610.06347

Social Network
Fingerprint
on Uploaded
Images

- Alterations are dependent to uploading client

• Social Image Ballistics (recover
image history)
Uploaded images
dataset

• Social Image Ballistics (recover image history)
Social Altered image dataset
- 10 Social Platforms
- Facebook, Google+, Instagram, Flickr, Tumblr, Twitter, Imgur,
Tinypic, Telegram, Whatsapp
- 2720 JPEG Images representing different subjects (natural, indoor,
outdoor)
- Dataset available at:
http://iplab.dmi.unict.it/DigitalForensics/social_image_forensics/

- On which Social Network was uploaded image I?
I Given a JPEG image I, the Social Image Ballistics task has the objective of
defining:
1) if there is a compatibility between the non-related JPEG elements of I
(i.e. filename, EXIF data) and the processing pipeline of SNSs;
2) if there is a compatibility between the JPEG elements of I and the
processing pipeline of SNSs;
3) which SNS is compatible with the JPEG elements of the image, with a
certain degree of confidence, and what is the uploading source in
terms of operating system (OS) and application.
Input Image

I
Input Image
Feature
Extraction

Representation of whole Dataset
Social Data – (Arxiv 2016 No. 1699257) http://arxiv.org/abs/1610.06347ICT Doctoral School, Trento - May 2017

Input Image
Feature
Extraction
• DQTs coeffs
• Image Size
• # EXIF
• # JPEG Markers
Anomaly
Detection
The Anomaly Detector excludes images not processed
by Social Network Platforms
Given a Similarity measure between features extracted
from images:
It is possible to build a distance matrix D of size N×N
where the element dij is equal to the distance
between the images Ii and Ij.
The Anomaly Detector is then defined as:

Input Image
Feature
Extraction
• DQTs coeffs
• Image Size
• # EXIF
• # JPEG Markers
Anomaly
Detection
SNS
Classification
Upload Client
Classification
Output: Not in our dataset
The image probably is not altered by a SNS
Image does not come from considered platforms

SNS
Classification
Upload Client
Classification
Where knn represent a k-nearest neightbour classifier based on
distance matrix D and dt is a decision tree.
A Decision tree builds classification in the form of a tree structure.
It breaks down a dataset into smaller and smaller subsets while at the
same time an associated decision tree is incrementally developed.
The final result is a tree with decision nodes. The algorithm used for
building the decision tree is the ID3

SNS
Classification
Upload Client
Classification
SNS
Consistency
Test
SNS Classification result is tested against known features
discovered in Image Dataset for the guessed SNS:
If conditions are met the classification is outputted, otherwise
the test is repeated for the next most probable prediction
from the SNS Classifier until the corresponding condition
is satisfied or the loop stalls on the same SNS prediction.
In this last case, the result of the classification is ”not sure”

Preliminar Results

DCT Analysis
• By studying histogram of DCT coefficient for the mode
before and after uploading a sample image on a SN
(Facebook, Twitter, FlickR) some regularities emerge
Caldelli, R., Becarelli, R., & Amerini, I. (2017). Image Origin Classification Based on Social
Network Provenance. IEEE Transactions on Information Forensics and Security, 12(6), 1299-1308.

Social Forensics: case study
• Industrial Intellectual Property leaking &
Ramson

• Industrial Intellectual Property leaking & Ramson
Dear Sebastiano,
We possess your newest patents. You can verify this by visiting
this link. You’ll find some images assessing that what I’m telling
you is true!
http://www.forumforramsons.com
You have 3 days to send 10 BTCs to this address:
3JKB1EWgDEg32aaQczg2H8jQCbQvtUjeas
Otherwise we’ll publish everything on the most popular Social
Networks starting from Google+

Social Forensics: case
study• Industrial Intellectual Property leaking &
Ramson
• Visiting the URL mentioned before…
PATENT n. 123456

• Three Days Later…
Dear Sebastiano,
You did not follow up with our request… We just posted some
images on Google+…
Now you have to send 20 BTCs to this address:
3JKB1EWgDEg32aaQczg2H8jQCbQvtUjeas
Otherwise we’ll publish everything on Facebook!!!

• Sebastiano knows that there are 3 suspects
• And has an evidence… This Image
PATENT n. 123456
Downloaded from Google+

• By means of Image Ballistics Tool:
PATENT n. 123456
It was processed by:
Google+

PATENT n. 123456
It was processed by:
Google+

PATENT n. 123456
But no EXIF

PATENT n. 123456
But no EXIF
No renaming for Google +

PATENT n. 123456
But no EXIF
No renaming for Google +
04 - Dw0KXG2.jpg

PATENT n. 123456
Let have a link for imgur
04 - Dw0KXG2.jpg

PATENT n. 123456
• The examiner downloads a new image from
imgur
≠
PATENT n. 123456

PATENT n. 123456
• By means of Image Ballistics Tool:

• And a new evidence!
PATENT n. 123456

• And a new evidence! PATENT n. 123456

• And a new evidence! PATENT n. 123456
PATENT n. 123456

• And a new evidence!
PATENT n. 123456
PATENT n. 123456

Future Works
• Extension of the involved dataset
• Continuous learning able to cope with updated policies of
each SNS
• New Media (Audio, Video)
• New SNS platform ( spanchat, Telegram, etc.)

Current trends

Body-Worn camera

Corriere.it 29/5/2016
Body Worn Camera

Furnari, Antonino, Giovanni Maria Farinella, and Sebastiano Battiato. "Recognizing Personal Contexts from Egocentric Images"
Proceedings of the IEEE International Conference on Computer Vision Workshops. 2015
A. Furnari, G. M. Farinella, S. Battiato, “Segmenting Egocentric Videos to Highlight Personal Locations of Interest”, IEEE International
Workshop on Egocentric (First-Person) Vision – in conjunction with the IEEE Conference on Computer Vision and Pattern
Recognition, Las Vegas, (2016)
A. Furnari, G. M. Farinella, S. Battiato, "Recognition of Personal Locations from Egocentric Videos" IEEE Transactions on Human-
Machine Systems, 2016
First Person Vision
Recognizing Personal Contexts
Datasets are available online:
http://iplab.dmi.unict.it/PersonalContexts/

Prof. Sebastiano Battiato
Dipartimento di Matematica e Informatica
University of Catania, Italy
Image Processing LAB – http://iplab.dmi.unict.it
iCTLab - www.ictlab.srl
battiato@dmi.unict.it

(Social) Multiimedia Forensics

Recommended

Recommended

More Related Content

Similar to (Social) Multiimedia Forensics

Similar to (Social) Multiimedia Forensics (20)

More from Sebastiano Battiato

More from Sebastiano Battiato (6)

Recently uploaded

Recently uploaded (20)

(Social) Multiimedia Forensics