Reversible visual privacy protection

1. 1 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Reversible visual privacy protection Touradj Ebrahimi touradj.ebrahimi@epfl.ch COST IC1206 Training School De-identification for privacy protection in multimedia content 7-11 October 2015, Limassol, Cyprus

2. 2 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Outline • Part I: – Motivations and context – Conventional privacy protection filters – Advanced privacy protection filters • Part II: – Visual privacy evaluation framework – Impact of new imaging modalities on privacy – Illustrative example

3. 3 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Motivation • People are increasingly exposed: – video surveillance – social media

4. 4 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Proliferation of video surveillance applications • Surveillance of sensitive locations – Embassies, airports, nuclear plants, military zone, border control, … • Intrusion detection – Residential surveillance, retail surveillance, … • Traffic control – Speed control • Access to places – Car license plate recognition in cities • Event detection – Child/Elderly care • Marketing/statistics – Customers habits – Number of visitors • …

5. 5 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Proliferation of social media applications

6. 6 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Main security solutions for visual content protection • Encryption – Secure communication – Conditional access • Integrity verification – Digital signature – Proof for lack of manipulation after capture

7. 7 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Alternatives to implement video surveillance with privacy • Fully automatic surveillance without intervention of human operators – False positives and false negatives • Encrypting the whole video – Not good for monitoring • Distorting/blocking sensitive regions – Impact on intelligibility • Reversible encryption/scrambling of sensitive regions with a key – Identification can take place when crime happens • Legal and best practices in video surveillance – Recorded materials locked in secure locations • Only extract/record needed information from the scene – MPEG-7 visual descriptors

8. 8 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Privacy-sensitive visual information • Predefined zones – Windows, doors – Bank teller – Casino playing tables – … • Automatic identification of Regions of Interest (ROI) – People in the scene – Human faces – Cars license plates – Moving objects – … • Advanced image/video analytics – Deep learning – Big data analytics

9. 9 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Social media/networks business model • User profiling • Targeted advertisement/marketing

10. 10 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Requirements for visual privacy protection • Maximize intelligibility • Minimize invasion of privacy • Visually pleasing • Reversible • Reasonable computational resources • Format preserving/independent • Reliable • Secure • …

11. 11 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Legacy solutions to visual privacy protection • Masking • Blur • Pixelization

12. 12 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Masking

13. 13 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Blur

14. 14 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Pixelization

15. 15 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne More recent solutions for privacy protection • (ROI) Encryption • (ROI) Scrambling

16. 16 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne ROI selective encryption

17. 17 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne ROI selective decryption

18. 18 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne ROI selective scrambling

19. 19 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Bitstream encryption • Selective encryption of the bitstream at packet level • One or more secret keys • Symmetric encryption – Packet body – Block cipher: e.g. AES packet encrypted packet private key

20. 20 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Scrambling approaches • Image-domain – Randomly flip bits in one or more bit planes • Pros – Very simple – Independent from the subsequent encoding scheme – Does not affect the bitstream syntax → standard compliance • Cons – Significantly alter statistics of video signal – Ensuing compression less efficient bitstreamimage Scrambling Encoder Transform Entropy Coding

21. 21 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Scrambling approaches • Transform-domain – Randomly flip sign of transform coefficients • Pros – Does not adversely affect subsequent entropy coding – Strength of scrambling can be controlled – Does not affect the bitstream syntax → standard compliance • Cons – Must be integrated inside the encoder bitstreamimage Transform Encoder Scrambling Entropy Coding

22. 22 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Scrambling approaches • Bitstream-domain – Randomly flip bits in bitstream • Pros – Applied on bitstream after encoding • Cons – Require parsing of bitstream – Difficult to guarantee syntax remains compliant and will not crash a decoder bitstreamimage Encoder Transform Entropy Coding Scrambling

23. 23 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Scrambling in JPEG (a) (b) DC PRNG pseudo-randomly inverse sign scrambled codestream public key assymetric encryption seed (c) DC PRNG pseudo-randomly inverse sign scrambled codestream public key assymetric encryption seed (d) DC PRNG pseudo-randomly inverse sign scrambled codestream public key assymetric encryption seed Figure 4 – AC coefficients scrambling: (a) 63 AC coefficients, (b) 60 AC coefficients, (c) 55 AC coefficients, (d) 48 AC coefficients. Straightforwardly, as the scrambling is merely flipping signs of selected coefficients, the technique requires negligible computational complexity. Clearly, the shape of the scrambled region is restricted to match the 8x8 DCT blocks boundaries.

24. 24 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Scrambling in JPEG (c) (d)

25. 25 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Scrambling in JPEG 2000 (JPSEC) • Codeblock-based bitstream domain scrambling    ≥ <+= → 900 900900mod)(' xxifx xxifxmxx x Preserve the markers in the bitstream; do not introduce erroneous markers x=current byte, y=preceding byte 1. If x=0xFF, no modification 2. If y=0xFF 3. Otherwise where m is an 8-bit pseudo- random number in [0x00,0x8F] where n is an 8-bit pseudo- random number in [0x00,0xFE] xFFnxxx 0mod)(' +=→ selective scrambling PRNG seed encryption encrypted seed scrambled codestream JPSEC codestream JPSEC syntax codestream quantizer selective scrambling PRNG seed wavelet transform arithmetic coder encryption encrypted seed scrambled codestream JPSEC codestream JPSEC syntax image Encoder Decoder

26. 26 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Scrambling in JPEG 2000 (JPSEC) • ROI-based wavelet domain scrambling – Arbitrary-shape regions • Exploit ROI mechanisms in JPEG 2000 Encoder Decoder quantizerwavelet transform arithmetic coder segmentation mask ? image down-shift wavelet coefficient PRNG seeds encrypt seeds ROI-based scrambled JPSEC code-stream scramble wavelet coefficient resolution level l < TI ? up-scale code-block distortion foreground objects background keys resolution level l ≥ TS ? yes no yesno inverse quantizer inv. wavelet transform arithmetic decoder coefficient < 2 s ? image up-shift wavelet coefficient PRNG seeds decrypt seeds ROI-based scrambled JPSEC code-stream unscramble wavelet coefficient foreground objects background keys resolution level l ≥ TS ? yes no Decoder

27. 27 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Scrambling in JPEG 2000

28. 28 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Scrambling in MPEG-4

31. 31 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Scrambling in H.264/AVC

32. 32 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Scrambling in H.264/AVC

33. 33 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne An existing product Scrambler Unscrambler

34. 34 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Scrambling in DVCScrambling in DVC • Key frame privacy (JPEG) – Scrambling in the transform domain on the DCT coefficients. – Driven by a Pseudo-Random Number Generator (PRNG) to pseudo- randomly invert the sign of the DCT Coefficients. • WZ frames DCT scrambler DVC scheme with privacy protection

35. 35 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Scrambling in DVCScrambling in DVC a) Key frame (JPEG). b) Wyner-Ziv transform domain scrambling.

36. 36 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne MPEG-7 camera XML scene description The MPEG-7 camera describes a scene in terms of semantic objects and of their properties

37. 37 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne MPEG-7 camera – Image analysis: segmentation, change detection, and tracking (implemented on the camera DSP). – MPEG-7 coder: scene description represented using MPEG-7 (XML). – MPEG-7 decoder: MPEG-7 description is parsed. Extraction of the information related to the specific applications.

38. 38 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne MPEG-7 camera <!-- ################################################## --!> <!-- DDL output for object 4 --!> <!-- ################################################## --!> <Object id="4"> <RegionLocator> <BoxPoly> Poly </BoxPoly> <Coords1> 237 222 </Coords1> <Coords2> 230 252 </Coords2> <Coords3> 240 286 </Coords3> <Coords4> 308 287 </Coords4> <Coords5> 312 284 </Coords5> </RegionLocator> <DominantColor> <ColorSpace> YUV </ColorSpace> <ColorValue1> 143.4 </ColorValue1> <ColorValue2> 123.3 </ColorValue2> <ColorValue3> 128.2 </ColorValue3> </DominantColor> <HomogeneousTexture> <TextureValue> 9.02 </TextureValue> </HomogeneousTexture> <MotionTrajectory> <TemporalInterpolation> <KeyFrame> 100 </KeyFrame> <KeyPos> 268.6 251.7 </KeyPos> <KeyFrame> 101 </KeyFrame> <KeyPos> 262.8 241.0 </KeyPos> ... <KeyFrame> 138 </KeyFrame> <KeyPos> 192.9 79.0 </KeyPos> </TemporalInterpolation> </MotionTrajectory> </Object> XML scene description

39. 39 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne MPEG-7 camera for video surveillance original frame segmentation mask bounding box

40. 40 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Existing product

41. 41 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Warping-based privacy filter • Compute map between original and shifted points • Interpolate in-between pixels with ‘cubic’ original points shifted points transformation

42. 42 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Recovery from warping • Know original points and seed for random algorithm to compute shifted points • Perform reverse mapping and interpolation shifted points original points transformation

43. 43 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne mild medium strong Facial features-based warping

44. 44 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne mild medium strong Unwarping

45. 45 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Morphing

46. 46 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Morphing

47. 47 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Privacy filter with false color

48. 48 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne JPEG Transmorphing JPEG Transcoder Mask matrix Sub-image Morphed JPEG image Original image Processed image − Sub-image embedded in APPn Markers T Reconstructed image 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 1 ! 1 ! 0 ! 0 ! 0 ! 0 ! 1 ! 1 ! 0 ! 1 ! 1 ! 1 ! 1 ! 0 ! 0 ! 1 ! 1 ! 1 ! 1 ! 1 ! 1 ! 1 ! 1 ! 0 ! 0 ! 1 ! 1 ! 1 ! 1 ! 0 ! 1 ! 1 ! 0 ! 0 ! 0 ! 0 ! 1 ! 1 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! 0 ! JPEG Transcoder Threshold t JPEG Transcoder

49. 49 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne JPEG Security and Privacy SOI APP1 (Exif) EOI SOI APP1 (Exif) EOI APP11 (protected metadata) JPEG-1 decoder JPEG Privacy & Security decoder APP1 (Exif) APP1 (Exif) original JPEG codestream JPEG compatible codestream with data protection Image Data Image data APP11 (protected image data) Image Data APP11 (protected metadata) Image data APP11 (protected image data) APP3 (JPSearch) APP3 (JPSearch) APP3 (JPSearch)

50. 50 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Homework! • Any non-reversible privacy protection filter can be converted into a reversible version! – Propose how this can be done!

51. 51 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Thanks for your attention! End of Part I

52. 52 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Outline • Part I: – Motivations and context – Conventional privacy protection filters – Advanced privacy protection filters • Part II: – Visual privacy evaluation framework – Impact of new imaging modalities on privacy – Illustrative example

53. 53 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Visual privacy evaluation framework

54. 54 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Privacy-intelligibility tradeoff • Protect privacy – obfuscate or remove personal information from the video • Perform surveillance – determine suspicious event/person, apprehend and prosecute criminals • Where is the balance?

55. 55 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Evaluation tools • Subjective evaluation • Crowdsourcing • Objective metrics

56. 56 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Subjective evaluation • Three naïve filters: blurring, pixelization, and masking • Dataset of 8 annotated videos – People acting normally or abnormally – Wearing glasses, scarf, hats, etc. – Blinking into the camera

57. 57 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Example

58. 58 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Questions asked to subjects race white asian I don’t know gender female male I don’t know glasses yes no I don’t know sunglasses yes no I don’t know scarf yes no I don’t know blinking yes no I don’t know Privacy Intelligibility

59. 59 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Subjective results 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 blurring filter pixelization filter masking filter Privacy

60. 60 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Facebook-based evaluations • The same experiment as for subjective evaluations • Facebook-based system – Shows videos – Collects answers – Trusted workers

61. 61 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Offline results Onlineresults blurring masking pixelization Crowdsourcing: privacy

62. 62 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Offline results Onlineresults blurring masking pixelization Crowdsourcing: intelligibility

63. 63 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne • The cheapest and the most scalable option • People counting in public transport – Face recognition is the metric of privacy – Face detection is the metric of intelligibility • An ideal privacy protection filter – Degrades face recognition – Does not affect face detection Objective evaluations

64. 64 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne • Increase strength of privacy filter • Note relative decrease in accuracy of face detection and recognition Evaluation examples

65. 65 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Detection Recognition Gaussian kernel size Accuracy Blurring, FERET dataset

66. 66 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Morphing, FERET dataset 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Accuracy Intensity strength interpolation_0 interpolation_0.2 interpolation_0.4 interpolation_0.6 interpolation_0.8 interpolation_1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Accuracy Intensity strength interpolation_0 interpolation_0.2 interpolation_0.4 interpolation_0.6 interpolation_0.8 interpolation_1 Recognition morphed Recognition recovered

67. 67 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Impact of new imaging modalities on privacy

68. 68 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne New imaging modalities • Ultra high definition (UHD)

69. 69 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne New imaging modalities • High dynamic range (HDR)

70. 70 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne New imaging modalities • Video from mini-drones

71. 71 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Ultra High Definition (UHD)

72. 72 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Walking example

73. 73 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Exchanging bag example

74. 74 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Fighting example

75. 75 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Subjective evaluations of privacy • 4K UHD Sony reference monitor • Evaluation questions about – People’s accessories – Main action – Visible items – Gender – Race • Accompanied questions on certainty

76. 76 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Measuring privacy %ofcorrectanswers

77. 77 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne High Dynamic Range (HDR)

78. 78 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Woman Elderly Wears sunglasses Woman?? Woman Young?? Woman Young?? Man?? Middle Age??Man?? Middle Age?? Man Middle Aged Man Middle Aged Woman Young Woman Young • HDR surveillance cameras • More details in the scene • More privacy intrusive? • HDR monitor is needed • Tone-mapped images Implications of HDR

79. 79 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne

83. 83 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Crowdsourcing evaluations of images • About 400 people participated • Evaluation questions about – Gender – Race – Age – Color of clothes – How many people

84. 84 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Crowdsourcing results

85. 85 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne (Mini-)drone surveillance

86. 86 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Drones & surveillance • Can collect sensitive data – Different heights – Different angles – Fly over fences • Bird-fly view • Harass and follow targets • Privacy protection?

87. 87 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Privacy protection filters • Blurring • Pixelization • Masking • Warping • Morphing

88. 88 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Privacy protection filters • Different strength levels – Mild – Noticeable – Obfuscating – Completely obfuscating

89. 89 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Crowdsourcing evaluations • Seven selected video contents – 5 privacy filters at 4 levels of strength + original = 21 versions of each content • 850 online workers from around the world (Microworkers platform) – Worker accesses one version of the content • Six questions about personal privacy and intelligibility of surveillance – Also, asked how certain is the answer

90. 90 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Evaluation framework • Questions – Activity in the scene – Number of people – Items (camera, wallet, etc.) – Accessories people wear (jacket, hat, helmet, sunglasses, etc.) – Gender – Ethnicity

91. 91 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Results – correctness Pixelization

92. 92 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Results – correctness Morphing

93. 93 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Results – certainty Pixelization

94. 94 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Results – certainty Morphing

95. 95 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Privacy-intelligibility tradeoff

96. 96 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne ProShare: A privacy-aware photo sharing platform

97. 97 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Illustrative example User 1 Client-side Notify User 2 Server-side Protect User 2 User 2 User 3 User 1User 2 Protect User 1 Friend relationship: User 1 & 2: ✔ User 1 & 3: ✔ User 2 & 3: ✖ Social Networking Services URL • Sender-side operations – Protection and upload • Server-side operations – Hosting and Access control • Recipient-side operations – Download and Reconstruction

98. 98 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne JPEG Privacy & Security

99. 99 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne JPEG Security and Privacy SOI APP1 (Exif) EOI SOI APP1 (Exif) EOI APP11 (protected metadata) JPEG-1 decoder JPEG Privacy & Security decoder APP1 (Exif) APP1 (Exif) original JPEG codestream JPEG compatible codestream with data protection Image Data Image data APP11 (protected image data) Image Data APP11 (protected metadata) Image data APP11 (protected image data) APP3 (JPSearch) APP3 (JPSearch) APP3 (JPSearch)

100. 100 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Thanks for your attention! End of Part II

Reversible visual privacy protection

You are reading a preview.

Reversible visual privacy protection

More Related Content

Similar to Reversible visual privacy protection

More from Touradj Ebrahimi

Featured

Related Books

Related Audiobooks