This paper presents an overview of the Visual Privacy Task (VPT) of MediaEval 2014, its objectives, related dataset, and evaluation approaches. Participants in this task were required to implement a privacy filter or a combination of filters to protect various personal information regions in video sequences as provided. The challenge was to achieve an adequate balance between the degree of privacy protection, intelligibility (how much useful information is retained post privacy filtering), and pleasantness (how minimal were the adverse effects of filtering on the appearance of the video frames). The submissions from the eight (8) teams who participated in this task were evaluated subjectively by surveillance experts, practitioners, data protection experts and by naïve viewers using a crowdsourcing approach. http://ceur-ws.org/Vol-1263/mediaeval2014_submission_37.pdf

1. Visual Privacy Task
Overview
Atta Badii, Touradj Ebrahimi, Christian Fedorczak, Pavel Korshunov,
Tomas Piatrik, Volker Eiselein, Ahmed Al-Obaidi

2. Task Description
• To explore the possibilities to optimise the
process of privacy filtering so as to:
1. obscure personal visual information effectively
whilst,
2. keeping as much as possible of the ‘useful’
information that would enable a human viewer
to interpret the obscured video frame.
Slide 2

3. Participants
VVPPTT 22001144:: 1111 rreeggiissttrraattiioonnss,, 88 ppaarrttiicciippaannttss ,,

4. PEViD Dataset
21 annotated videos
• ~20s
• 25fps
• 1920x1080
• Indoor/Outdoor
• Daytime/Evening
• Featured actions include: Bag dropping, Pickpocketing,
Fighting, or simply Walking
• Ground truth consists of: annotated People, Faces,
Accessories, Hair regions, Skin regions
Privacy sensitive
region
Development Set = 9 videos
Evaluation Set = 12 videos
Sensitivity
level
Skin Medium (M)
Face High (H)
Hair Low (L)
Accessories Medium (M)
Person Low (L)

5. Subjective Evaluation Streams
The privacy protected video clips were evaluated by each of
three distinct communities of human evaluators:
i) Naive viewers (engaged via a crowdsourcing platform)
• 290 workers responded, 230 were found to have provided reliable
responses
ii) Video-analytics technology and privacy protection solutions
developers
• focus group consisted of (65) participants, (15) of them were
females
iii) Surveillance monitoring staff and law enforcement staff
• focus group comprised of (59) participants including (22) females
Slide 5

6. Evaluation Setup
• Six (6) video clips were pre-selected from each submission
and evaluated using the three (3) evaluation streams.
• A Questionnaire consisting of 12 questions had been carefully
designed to examine aspects related to privacy, intelligibility,
and pleasantness; this was used in stream 2 and 3.
• The First (5) questions were aimed at eliciting the opinions of
the evaluators re the Contents of the viewed videos. The
responses to these questions were considered with respect to
the ground truth.
• The rest of the questions were aimed at eliciting the Subjective
Opinions of the evaluators re the viewed videos.
• Stream 1 used a shortened version of the questionnaire with
(7) questions in total due to crowdsourcing constraints.
Slide 6

7. UI-REF based criteria
• Privacy Protection Level – How adequate was
the level of privacy protection achieved by the
filter across all testing video clips?
• Level of Intelligibility – How much ‘useful’
information that was retained in the video
frames after privacy filtering had been applied?
• Pleasantness of the resulting privacy filtered
video frames in terms of their ‘aesthetic’
perceptual appeal to human viewers. How
acceptable were any adverse aesthetic effects?
Perceived Effects, Side Effects, Cross-Effects, Affects
Slide 7

8. Stream 1 results
Slide 8

9. Stream 2 results
Slide 9

10. Stream 3 results
Slide 10

11. Median of the 3 streams
Slide 11

12. Thank You