Privacy protection of visual information

1,201 views
1,093 views

Published on

Slides of my talk at MediaSense 2012, Dublin, Ireland, 21-22 May 2012.

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,201
On SlideShare
0
From Embeds
0
Number of Embeds
44
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Privacy protection of visual information

    1. 1. 1 Privacy protection of visual information Touradj Ebrahimi touradj.ebrahimi@epfl.ch MediaSense 2012 Dublin, Ireland 21-22 May 2012Multimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    2. 2. Video surveillance popularity 2• Rise in terrorism and crime – Globalization of good and bad• Political – Perception that the problem of crime and terrorism is addressed• Business and economy – New revenue models – Cost issues Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    3. 3. Potential abuses in video surveillance 3• Criminal abuse – Criminal misuse by law enforcement officers – Police official gathering information on a gay club to blackmail patrons• Institutional abuse – Spy upon and harass political activists (Civil Rights, Vietnam war) – Surveillance of political demonstrations• Personal usage – Police officers helping friends stalk women, track estranged girlfriends/ spouses• Discrimination – Racial discrimination towards people of color• Voyeurism – Bored male operators spying on women – Footage of public cameras made publicly available Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    4. 4. Civil liberty and right to privacy 4• Increased resistance to video surveillance• Several countries have set up or are in the process of setting up directives and guidelines to regulate video surveillance – EU – Directive 95/46/EC Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    5. 5. World Trade Center, 9/11 5 filmed by a Gas Station surveillance camera on September 10, 2001 filmed by an ATM surveillance camera on September 10, 2001Mohamed Atta Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    6. 6. Attack on London underground, July 7, 2005 6 On a reconnaissance mission two weeks before the attackMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    7. 7. Plot to attack trains in Germany, August 2006 7Two unexploded bombs found inluggage aboard two trainsBoth terrorists have been arrestedthanks to the video footage Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    8. 8. Proliferation of video surveillance applications 8• Surveillance of sensitive locations – Embassies, airports, nuclear plants, military zone, border control, …• Intrusion detection – Residential surveillance, retail surveillance, …• Traffic control – Speed control• Access to places – Car license plate recognition in cities• Event detection – Child/Elderly care• Marketing/statistics – Customers habits – Number of visitors• …Multimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    9. 9. Forensic video – legal admissibility 9• If the image is not inherently reliable, its admissibility in court is questionable• If a poor image is ruled admissible, it will be afforded little or no weight• For an image to be admissible, the prosecutor must prove that the image has not been altered – Lossy compression – Conditional replenishment – Enhancement• Original versus copy – Any digital image can be thought of as being ‘the original’ Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    10. 10. 10 Video surveillance dimensions• Technology• Business• Legal• SocialMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    11. 11. Video surveillance technologies 11• First generation – Analog – CCTV – Recording• Second generation – Digital/Hybrid – Recording – Computerized – IP wired/wirelessMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    12. 12. Video surveillance technologies 12• Third generation – Content analysis – Biometrics – Search – Unusual event detection• Forth generation – Pervasive – Distributed – Invisible – Multi-view – Ultra high definitionMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    13. 13. 13 Main security tools in video surveillance• Encryption – Secure communication – Conditional access• Integrity verification – Digital signature – Proof for lack of manipulation after captureMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    14. 14. 14Alternatives to implement video surveillance with privacy • Fully automatic surveillance without intervention of human operators – False positives and false negatives • Encrypting the whole video – Not good for monitoring • Distorting/blocking sensitive regions – Impact on intelligibility • Reversible encryption/scrambling of sensitive regions with a key – Identification can take place when crime happens • Legal and best practices in video surveillance – Recorded materials locked in secure locations • Only extract/record needed information from the scene – MPEG-7 visual descriptors Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    15. 15. Smart video surveillance 15 Video + Metadata RecordingMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    16. 16. Smart video surveillance 15 Video + Metadata RecordingMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    17. 17. Smart video surveillance 15 Video + Metadata RecordingMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    18. 18. Smart video surveillance 15 Video + Metadata RecordingMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    19. 19. Smart video surveillance 15 Video + Metadata RecordingMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    20. 20. Smart video surveillance 15 Video + Metadata RecordingMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    21. 21. Smart video surveillance 15 Video + Metadata RecordingMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    22. 22. Smart video surveillance 15 Video + Metadata Recording […011001…]Multimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    23. 23. Example: Smart video surveillance 16Multimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    24. 24. 17 Privacy-sensitive visual information• Predefined zones – Windows, doors – Bank teller – Casino playing tables – …• Automatic identification of Regions of Interest (ROI) – People in the scene – Human faces – Cars license plates – Moving objects – … Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    25. 25. 18 Legacy solutions to visual privacy protection• Masking• Blur• PixelizationMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    26. 26. 19 MaskingMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    27. 27. 20 BlurMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    28. 28. 21 PixelizationMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    29. 29. 22 More recent solutions for privacy protection• (ROI) Encryption• (ROI) ScramblingMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    30. 30. 23 ROI selective encryptionMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    31. 31. 24 ROI selective decryptionMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    32. 32. ROI selective scrambling 25Multimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    33. 33. Bitstream encryption 26• Selective encryption of the bitstream at packet level• One or more secret keys• Symmetric encryption – Packet body – Block cipher: e.g. AES packet private key encrypted packet Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    34. 34. Scrambling approaches 27• Image-domain – Randomly flip bits in one or more bit planes image Scrambling Transform Entropy Coding bitstream Encoder• Pros – Very simple – Independent from the subsequent encoding scheme – Does not affect the bitstream syntax → standard compliance• Cons – Significantly alter statistics of video signal – Ensuing compression less efficient Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    35. 35. Scrambling approaches 28• Transform-domain – Randomly flip sign of transform coefficients image Transform Scrambling Entropy Coding bitstream Encoder• Pros – Does not adversely affect subsequent entropy coding – Strength of scrambling can be controlled – Does not affect the bitstream syntax → standard compliance• Cons – Must be integrated inside the encoder Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    36. 36. Scrambling approaches 29• Bitstream-domain – Randomly flip bits in bitstream image Transform Entropy Coding Scrambling bitstream Encoder• Pros – Applied on bitstream after encoding• Cons – Require parsing of bitstream – Difficult to guarantee syntax remains compliant and will not crash a decoder Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    37. 37. 30 Scrambling in JPEG (a) (b) DC pseudo-randomly PRNG inverse sign seed assymetric scrambled encryption codestream public key (c) DC (d) DC pseudo-randomly pseudo-randomly PRNG PRNG inverse sign inverse sign seed seed assymetric scrambled assymetric scrambled encryption codestream encryption codestream public key public key Figure 4 – AC coefficients scrambling: (a) 63 AC coefficients, (b) 60 AC coefficients, (c) 55 AC coefficients, (d) 48 AC coefficients. Straightforwardly, as the scrambling is merely flipping signs of selected coefficients, the technique requires negligibleMultimedia Signal Processing GroupSwiss Federal Institute of Technology, scrambled region is restricted to match the 8x8 DCT blocks computational complexity. Clearly, the shape of the Lausanne boundaries.
    38. 38. 31 Scrambling in JPEGMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    39. 39. Scrambling in JPEG 2000 (JPSEC) 32 • Codeblock-based bitstream domain scrambling image wavelet quantizer selective arithmetic Encoder codestream selective scrambled Decoder transform scrambling coder scrambling codestream scrambled codestream PRNG JPSEC JPSEC PRNG JPSEC JPSEC syntax codestream syntax codestream encrypted encrypted seed seed seed encryption seed encryption Preserve the markers in the bitstream; do not introduce erroneous markers x=current byte, y=preceding byte 1. If x=0xFF, no modification 2. If y=0xFF where m is an 8-bit pseudo- random number in [0x00,0x8F] 3. Otherwise where n is an 8-bit pseudo- random number in [0x00,0xFE] Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    40. 40. Scrambling in JPEG 2000 (JPSEC) 33 • ROI-based wavelet domain scrambling – Arbitrary-shape regions • Exploit ROI mechanisms in JPEG 2000 wavelet quantizer Encoder image transform no resolution yes level l < TI ? keys ROI-based scrambled Decoder Decoder up-scale JPSEC code-stream code-block distortion decrypt arithmetic seeds decoder foreground foreground resolution seeds objects yes objects segmentation background yes resolution background level l level l coefficient ≥ TS ? mask ? ≥ TS ? < 2s ? scramble down-shift unscramble up-shift PRNG PRNG wavelet no wavelet wavelet wavelet no coefficient coefficient coefficient coefficientseeds encrypt arithmetic inverse inv. wavelet seeds coder quantizer transform ROI-based sc rambled keys JPSEC code-stream image Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    41. 41. Scrambling in JPEG 2000 34Multimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    42. 42. 35 Scrambling in MPEG-4Multimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    43. 43. 36 Scrambling in MPEG-4Multimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    44. 44. Scrambling in MPEG-4 37Multimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    45. 45. Scrambling in MPEG-4 37Multimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    46. 46. Scrambling in MPEG-4 37Multimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    47. 47. 38 Scrambling in H.264/AVCMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    48. 48. Scrambling in H.264/AVC 39Multimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    49. 49. 40 An existing product Scrambler UnscramblerMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    50. 50. 41 Scrambling in DVC• Key frame privacy (JPEG) – Scrambling in the transform domain on the DCT coefficients. – Driven by a Pseudo-Random Number Generator (PRNG) to pseudo- randomly invert the sign of the DCT Coefficients.• WZ frames DCT scrambler DVC scheme with privacy protection Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    51. 51. 42 Scrambling in DVC a) Key frame (JPEG). b) Wyner-Ziv transform domain scrambling.Multimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    52. 52. MPEG-7 camera 43 The MPEG-7 camera describes a scene in terms of semantic objects and of their properties XML scene descriptionMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    53. 53. MPEG-7 camera 44 – Image analysis: segmentation, change detection, and tracking (implemented on the camera DSP). – MPEG-7 coder: scene description represented using MPEG-7 (XML). – MPEG-7 decoder: MPEG-7 description is parsed. Extraction of the information related to the specific applications.Multimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    54. 54. MPEG-7 camera 45 <!-- ################################################## --!> <!-- DDL output for object 4 --!> <!-- ################################################## --!> <Object id="4"> <RegionLocator> <BoxPoly> Poly </BoxPoly> <Coords1> 237 222 </Coords1> <Coords2> 230 252 </Coords2> <Coords3> 240 286 </Coords3> <Coords4> 308 287 </Coords4> <Coords5> 312 284 </Coords5> </RegionLocator>XML scene <DominantColor>description <ColorSpace> YUV </ColorSpace> <ColorValue1> 143.4 </ColorValue1> <ColorValue2> 123.3 </ColorValue2> <ColorValue3> 128.2 </ColorValue3> </DominantColor> <HomogeneousTexture> <TextureValue> 9.02 </TextureValue> </HomogeneousTexture> <MotionTrajectory> <TemporalInterpolation> <KeyFrame> 100 </KeyFrame> <KeyPos> 268.6 251.7 </KeyPos> <KeyFrame> 101 </KeyFrame> <KeyPos> 262.8 241.0 </KeyPos> ... <KeyFrame> 138 </KeyFrame> <KeyPos> 192.9 79.0 </KeyPos> </TemporalInterpolation> </MotionTrajectory> </Object> Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    55. 55. MPEG-7 camera for video surveillance 46 original frame segmentation mask bounding boxMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    56. 56. 47 Existing productMultimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne
    57. 57. 48 Evaluation of privacy protection in video surveillance• Serious study of performance analysis of privacy protection solutions is lacking• It is paramount to validate privacy protection solutions against user and system requirements for privacy• Two approaches can be used – Performance analysis using subjective evaluations – Performance analysis using objective metrics Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    58. 58. 49 Pixelization• Naïve approach for privacy protection – Commonly used in television news and documentaries in order to obscure the faces of suspects, witnesses or bystanders to preserve their anonymity – Also used to censor nudity or to avoid unintentional product placement on television.• Consists in noticeably reducing resolution in ROI• Can be achieved by substituting a square block of pixels with its average• Drawback – Integrating pixels along trajectories over time may allow to partly recovering the concealed information – Irreversible Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    59. 59. 50 Gaussian Blur• Naïve approach for privacy protection• Removes details in ROI by applying a Gaussian low pass filter• Image is convolved with a 2D Gaussian function• Drawback – Irreversible Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    60. 60. 51 Scrambling by Random Sign Inversion• ROI-based transform-domain scrambling method• Scrambles the quantized transform coefficients of each 4x4 block of the ROI by pseudo-randomly flipping their sign• Advantages – Fully reversible – Same scrambled stream is transmitted to all users – Small impact in terms of coding efficiency – Requires a low computational complexity Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    61. 61. 52 Scrambling by Random Permutation• ROI-based transform-domain scrambling method• Random permutation to rearrange the order of transform coefficients in 4x4 blocks corresponding to ROI – Knuth shuffle to generate a permutation of n items with uniform random distribution• Advantages – Fully reversible – Same scrambled stream is transmitted to all users – Small impact in terms of coding efficiency – Requires a low computational complexity Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    62. 62. 53 Face Recognition - PCA• Principal Components Analysis (PCA) – Also known as eigenfaces – A linear transformation is applied to rotate feature vectors from the initially large and highly correlated subspace to a smaller and uncorrelated subspace – PCA has shown to be effective for face recognition – Firstly, it can be used to reduce the dimensionality of the feature space – Secondly, it eliminates statistical covariance in the transformed feature space – In other words, the covariance matrix for the transformed feature vectors is always diagonal Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    63. 63. 54 Face Recognition - LDA• Linear Discriminant Analysis (LDA) – LDA aims at finding a linear transformation which stresses differences between classes while lessening differences within classes (a class corresponds to all images of a given individual) – The resulting transformed subspace is linearly separable between classes – PCA is first performed to reduce the feature space dimensionality – LDA is then applied to further decrease the dimensionality while safeguarding the distinctive characteristics of the classes – The final subspace is obtained by multiplying the PCA and LDA basis vectors. Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    64. 64. 55 Face Identification and Evaluation System• Preprocessing – Reduces detrimental variations between images – Face alignment aligned using eye coordinates – Pixel values equalization, contrast and brightness normalization• Training – Create the subspace into which test images are subsequently projected and matched – Performed using a training set of images• Testing – A distance matrix is computed in the transformed subspace for all test images – Euclidian distance for PCA and soft distance for LDA – Two image sets are defined: – gallery set is made of known faces – probe set corresponds to faces to be recognized. Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    65. 65. 56 Face Identification and Evaluation System• Performance analysis – Generate cumulative match curve – For each probe image, the recognition rank is computed – rank 0 means that the best match is of the same subject – rank 1 means that the best match is from another person but the second best match is of the same subject – etc. – The cumulative match curve is obtained by summing the number of correct matches for each rank Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    66. 66. 57 Test Data• Grayscale Facial Recognition Technology (FERET) – Although it is not representative of typical video surveillance footage, this database is widely used for face recognition research – We consider a subset of 3368 images of frontal faces for which eye coordinates are available – Images have 256 by 384 pixels with eight-bit per pixel – We further consider two series of images denoted by ‘fa’ and ‘fb’ – ‘fa’ indicates a regular frontal image – ‘fb’ indicates an alternative frontal image, taken within seconds of the corresponding ‘fa’ image, where a different facial expression was requested from the subject.• Standard training, gallery and probe sets from the FERET test – Training set: 501 images from the ‘fa’ series – Gallery set: 1196 images from the ‘fa’ series – Probe set: 1195 images from the ‘fb’ series Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    67. 67. 58 Performance Analysis – Attack #1• Simple attack – Training and gallery sets are made of unaltered images – Probe set corresponds to images with privacy protection – In other words, altered images are merely processed by the face recognition algorithms without taking into account the fact that privacy protection tools have been applied. PCA LDA Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    68. 68. 59 Performance Analysis – Attack #1• For both PCA and LDA schemes applied on original images, recognition rate is superior to 70% at rank 0 (i.e. the best match is of the same subject as the probe), and superior to 90% at rank 50• When applying a Gaussian blur, the performance drops radically for LDA. However, recognition rate remains high for PCA with 56% success at rank 0• Pixelization fares worse. The recognition rate is 56% and 13% at rank 0 for PCA and LDA respectively• Results clearly show that both region-based transform-domain scrambling approaches are successful at hiding identity. The recognition rate is nearly 0% at rank 0, and remains below 10% at rank 50, for both PCA and LDA algorithms. In addition, it can be observed that both random sign inversion and random permutation schemes achieve nearly the same performance Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    69. 69. 60 Performance Analysis – Attack #2• More sophisticated attack – Privacy protection tools are now applied to all images in the training, gallery and probe sets – This corresponds to an attacker which gets access to protected data – Alternatively, an attacker may attempt replicating the alteration due to privacy protection techniques on his own training and gallery sets PCA LDA Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    70. 70. 61 Performance Analysis – Attack #2• With Gaussian blur, the performance remains nearly identical. It even improves slightly for LDA• Pixelization is not much better at hiding facial information. The recognition rate is still 45% and 17% at rank 0 for PCA and LDA respectively• Finally, both region-based transform-domain scrambling approaches are again successful at hiding identity. The recognition rate is nearly 0% at rank 0 for both PCA and LDA algorithms. Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    71. 71. 62 Thanks for your attention!Multimedia Signal Processing GroupSwiss Federal Institute of Technology, Lausanne

    ×