Erik Krempel from Fraunhofer in Germany presented on Smart Video Surveillance Systems and privacy at the CRISP final conference in Brussels 16 March 2017.

1. © Fraunhofer
Smart Video Surveillance
and Privacy
 Dr.-Ing. Erik Krempel,
 Fraunhofer IOSB, Karlsruhe, Germany

2. © Fraunhofer IOSB
2
What can we do with video analytics…
SMART VIDEO SURVEILLANCE

3. © Fraunhofer IOSB
3
From Visual to Symbolic Video Surveillance
• The classic approach of video surveillance is visual video
monitoring by security staff.
• Today, video analytics are more and more in use as assistance systems. Mainly to
attract and sustain attention of the staff to possible points of interst.
• Current research is going towards useage of advanced video analytics for symbolic
situation awareness and „privacy-aware management by exception“
Classic Approach Intelligent Video Analytics
Privacy-aware Situation
Awareness Tools

4. © Fraunhofer IOSB
4
Video Analytics for Scene Understanding
Object Tracking
Activity Recognition
Semantic Scene
Understanding
Scene Context Recognition
Object DetectionImage Enhancement

5. © Fraunhofer IOSB
5
Current Research: Advanced Objekt Detection, Tracking
and Re-Identification
• Multi-camera multi-object detection and tracking of persons and vehicles
• indoor / outdoor applications with complex illumination
conditions
• self-calibrating cameras (automated geometric and color
calibration)
• New approaches on dynamic learning of objects‘ unique features

6. © Fraunhofer IOSB
6
Towards Activity Recognition for (Global) Scene
Understanding
• Classification of activities in videos.

7. © Fraunhofer IOSB
7
Person Retrieval
• Component that is looking for similar persons
in archive in a specific time interval
• Features of the person of interest
(color features and texture features) are compared to already computed
features of other tracks
• System generates a list
of tracks, sorted based
on their similarity to
the person of interest
• Operator can verify the
results visually

8. © Fraunhofer IOSB
8
Why do we need to certify smart video surveillance systems
CERTIFICATION

9. © Fraunhofer IOSB
9
Security
 What is the error rate of video processing algorithms?
 False positive rates
 False negative rates
 Is the system robust against environment conditions?
 Rain on the camera
 Light conditions
 Is the system safe against attacks?
 The systems keeps secrets for itself (Confidentiality)
 The systems stays operational in the presence of an attacker
(Availability)
 The systems detects the same results in the presence of an attacker
(Integrity)

10. © Fraunhofer IOSB
10
Trust / Transparency

11. © Fraunhofer IOSB
11
Trust
 How can you achieve real transparency in such complex systems?
 Is there a human preventing the system from making wrong decisions?
 Does the systems prevent discrimination?
© Axis Communications

12. © Fraunhofer IOSB
12
Efficiency dimension
 Is the system able to reduce crime / increase security?
 Is the system easy to use?
 Can you extend the system / use components of another manufacturer?

13. © Fraunhofer IOSB
13
Freedom Infringement
 Does the system prevent discrimination?
 Are video processing algorithms restricted to legal applications?
 Does the system prevent misuse by the operator?
 Security of the data processing…
 Secure against data theft…
 Purpose limitation…
 Errors by the systems…

14. © Fraunhofer IOSB
14
RESEARCH WORK
Privacy in smart video surveillance

15. © Fraunhofer IOSB
15
Situation-dependent video surveillance
 Alexander Roßnagel, Monika Desoi, and Gerrit Hornung (2011)

16. © Fraunhofer IOSB
16
Idea: Privacy-aware Surveillance Workflows
Default Mode: Optimized for privacy Assessment Mode:
Event-specific, privacy preserving
Event
detected
Event
confirmed
Event
resolved
Investigation Mode:
Event specific functions unlocked

17. © Fraunhofer IOSB
17
Default Mode: Optimized for Privacy
 Computer vision algorithms in background
 Abstract representation of observed environment visualized
 No access to video data
 No access to archived data
 Exposes as little information about observed persons as possible

18. © Fraunhofer IOSB
18
Assessment Mode: Event-specific, Privacy Preserving
Event
detected
 Assessment view according to event type
 Anonymized live video data released
 Possibly anonymized access to a limited buffer of recorded (video)
data
 Only selected cameras

19. © Fraunhofer IOSB
19
Investigation Mode: Event-specific Functions Unlocked
Event
confirmed
Event
resolved
 Allows additional privacy intrusions for investigation purposes
 I.e., retrieving the person who dropped a piece of luggage
 Restrict additional analyses to persons related to the event under
investigation

20. © Fraunhofer IOSB
20
IMPLEMENTATION
How to build systems with this concept…

21. © Fraunhofer IOSB
21
Security Use-Case

22. © Fraunhofer IOSB
22
Safety Use-Case: Prototype NurseEye

23. © Fraunhofer IOSB
23
Operator interaction
 Default mode:
 No Access
 Assessment mode:
 Access to
anonymized video
 Investigation mode:
 Full access to video
to help in
emergency handling

24. © Fraunhofer IOSB
24
Transparency
 All cameras come with displays showing the current mode and data
usage
 Displays become monitor for chat in the investigation mode
Default mode Assessment mode Investigation mode

25. © Fraunhofer IOSB
25
Video NurseEye

26. © Fraunhofer IOSB
26
TOOLS

27. © Fraunhofer IOSB
27
User Study on Anonymization Techniques: Introduction
 Which obfuscation technique should be used?
 Regarding privacy (identity leakage), utility and perceptual video
quality
ROI Anonymized person
Blurring Silhouette Edge detection PixelizationBlurring
(gray scale)

28. © Fraunhofer IOSB
28
Results: Subjective assessments
1 = silhouette; 2 = pixelization; 3,5 = edge detection; 4 gray scale blurring; 6-8 color blurring
Error bars represent one standard
deviation of the data sample
Perceived privacy protection and perceived image quality

29. © Fraunhofer IOSB
29
Video anonymisation
Edge detectionPixelization
Gaussian blurring Silhouette

30. © Fraunhofer IOSB
30
Adaptive „Privacy-Masking“ for Pan/Tilt/Zoom Camera
• Advanced Privacy Masking for semi-stationary PTZ-cameras
• Given position of camera over ground (or an elevation model of the site) optical distance to
objects in the scene is estimated
• Given a parameter for „privacy-preserving resolution“ video is pixelized or blurred
inhomoge-
nously depending on distance to object.
• Highly senstive areas (e.g. buildings / windows) can be blacked out of the stream by
dynamic adaptable polygons (depending on pan/tilt/zoom settings)

31. © Fraunhofer IOSB
31
Information Flow Tracking in NEST
Video &
Observation
archive
PEP
HMI
GeoViewer
Observations
PEP
Requests
HMI
StreamViewer
PEP
Image
exploitation
algorithm
Image
exploitation
algorithm
Image
exploitation
algorithm
PEP
IMG
Policy
Decision
Point
Policy
Information
Point
Policy
Observations
Control
PEP
Access
Control
OOWM
Association
PEP HLS-Alarms
PEP Classification
PEP Fusion

32. © Fraunhofer IOSB
32
Discussion
“Data is not an asset, it’s a liability”
-Marko Karppinen-
Contact Information:
Erik Krempel
Fraunhofer IOSB
Fraunhoferstr. 1
76131 Karlsruhe, Germany
erik.krempel@iosb.fraunhofer.de
+49-721-6091-292