Successfully reported this slideshow.
Your SlideShare is downloading. ×

“12+ Image Quality Attributes that Impact Computer Vision,” a Presentation from Commonlands

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 50 Ad

“12+ Image Quality Attributes that Impact Computer Vision,” a Presentation from Commonlands

Download to read offline

For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2022/09/12-image-quality-attributes-that-impact-computer-vision-a-presentation-from-commonlands/

Max Henkart, Optics Consultant and Owner of Commonlands LLC, presents the “12+ Image Quality Attributes that Impact Computer Vision” tutorial at the May 2022 Embedded Vision Summit.

In this presentation, Henkart introduces key image quality metrics. He discusses how they impact the performance of geometric and CNN-based methods, along with providing key performance indicators (KPIs) and real-world examples to answer the following questions:
-Exposure: How does improper exposure impact your computer vision algorithms?
-Dynamic range: What is the difference between dynamic range and exposure? What’s the difference between HDR vs. WDR?
-Motion Blur: When does my camera hardware create motion artifacts?
-Resolution and Texture: How do the four types of resolution differ and degrade performance?
-Color and Shading: When does color influence computer vision performance?
-Noise: What are the different types of noise in my system and how do they impact performance?
-Image Artifacts: How do stray light, blemishes and fringing lead to edge cases?

For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2022/09/12-image-quality-attributes-that-impact-computer-vision-a-presentation-from-commonlands/

Max Henkart, Optics Consultant and Owner of Commonlands LLC, presents the “12+ Image Quality Attributes that Impact Computer Vision” tutorial at the May 2022 Embedded Vision Summit.

In this presentation, Henkart introduces key image quality metrics. He discusses how they impact the performance of geometric and CNN-based methods, along with providing key performance indicators (KPIs) and real-world examples to answer the following questions:
-Exposure: How does improper exposure impact your computer vision algorithms?
-Dynamic range: What is the difference between dynamic range and exposure? What’s the difference between HDR vs. WDR?
-Motion Blur: When does my camera hardware create motion artifacts?
-Resolution and Texture: How do the four types of resolution differ and degrade performance?
-Color and Shading: When does color influence computer vision performance?
-Noise: What are the different types of noise in my system and how do they impact performance?
-Image Artifacts: How do stray light, blemishes and fringing lead to edge cases?

Advertisement
Advertisement

More Related Content

More from Edge AI and Vision Alliance (20)

Advertisement

“12+ Image Quality Attributes that Impact Computer Vision,” a Presentation from Commonlands

  1. 1. 12+ Image Quality Attributes that Impact Computer Vision Max Henkart Imaging Optics & Camera Engineer, Founder Commonlands LLC
  2. 2. • Intro • Types of image quality • Review image quality metrics and the impact on CV • Summary Overview 2 © 2022 Commonlands LLC
  3. 3. Illumination & Object Characteristics Lens and Alignment Sensor & Image Capture Electronics Digital Image Processing Computer Vision Analytics, Localization, Prediction, etc. Image quality and computer vision require experts from multiple industries 3 Optical + Mech Engineering Electrical + Firmware + Image Quality Engineering Software Engineering © 2022 Commonlands LLC
  4. 4. Image quality in the field is fundamental to embedded computer vision performance 4 Su, et. al. “One Pixel Attack for Fooling Deep Neural Networks” © 2022 Commonlands LLC
  5. 5. Image quality in the field is fundamental to embedded computer vision performance 5 Su, et. al. “One Pixel Attack for Fooling Deep Neural Networks” “Nonetheless, we show some examples of situations where nearly imperceptible image modifications can result in dramatic perception changes. Even in applications without malicious people trying to trick your system, the natural world may be adversarial enough.” Pezzementi, et. al “Putting Image Manipulations in Context: Robustness Testing for Safe Perception” © 2022 Commonlands LLC
  6. 6. Objective • Independent of preference • Measurements with image quality test charts Subjective • Dependent on preference • Measurements through focus groups and other user feedback methods Two types of purpose-based image quality metrics are required to fully characterize an image 6 “Too Grainy” “Too Blurry” “Not Colorful” “SNR = 70db” “eSFR=20%@100lp/mm” “∆E=2” ? ? © 2022 Commonlands LLC
  7. 7. Computational-Based • Inputs are related to human visual cognition such as structure and color. • Includes content-aware image processing. • Directly related to image saliency in embedded computer vision. “Subjective Image Quality” Engineering-Based • Inputs are related to test charts, camera hardware, and image processing. • Independent of scene content and human visual quality assessment. “Objective Image Quality” Objective and subjective were coined before modern embedded vision systems 7 Reference: Zwanenberg, et. al., 2020, “Edge Detection Techniques for Quantifying Spatial Imaging System Performance and Image Quality” © 2022 Commonlands LLC
  8. 8. 1. Exposure + Motion Blur 2. Dynamic Range + Artifacts 3. Noise 4. Color 5. Shading 6. Resolution 7. Distortion 8. Texture Blur 9. Stray Light 10.Fringing + Blooming 11.Blemish 12.Dead Pixels 8 Let’s investigate how objective image quality metrics could impact your computer vision © 2022 Commonlands LLC
  9. 9. Related KPI • Exposure Value (EV) Reference: ISO12232 Exposure index combines the scene, lens, exposure length, gain, and image processing 9 Blasinski, etc. al. “Optimizing Image Acquisition Systems for Autonomous Driving” © 2022 Commonlands LLC
  10. 10. Motion blur is created by long exposure and/or imperfect high-dynamic range recombination 10 Pei, et. al. “Effects of Image Degradations to CNN-based Image Classification” Angular shift Blur Length © 2022 Commonlands LLC
  11. 11. 1. Exposure + Motion Blur 2. Dynamic Range + Artifacts 3. Noise 4. Color 5. Shading 6. Resolution 7. Distortion 8. Texture Blur 9. Stray Light 10.Fringing + Blooming 11.Blemish 12.Dead Pixels Let’s investigate how objective image quality metrics could impact your computer vision 11 © 2022 Commonlands LLC
  12. 12. Related KPI • Dynamic Range (dB) Reference: ISO21550 Camera dynamic range is the ratio of maximum to minimum signal, before saturation occurs 12 Hasinoff, et. al. “Burst photography for high dynamic range and low-light imaging on mobile cameras” © 2022 Commonlands LLC
  13. 13. Even modern HDR techniques can introduce other image quality artifacts 13 Robidoux, et. al. “End-to-end High Dynamic Range Camera Pipeline Optimization” © 2022 Commonlands LLC
  14. 14. 1. Exposure + Motion Blur 2. Dynamic Range + Artifacts 3. Noise 4. Color 5. Shading 6. Resolution 7. Distortion 8. Texture Blur 9. Stray Light 10.Fringing + Blooming 11.Blemish 12.Dead Pixels 14 Let’s investigate how objective image quality metrics could impact your computer vision © 2022 Commonlands LLC
  15. 15. Related KPIs • Signal-to-Noise Ratios (multiple types) • Noise Power Spectrum (Frequency) Reference: ISO15739 Noise is split into single pixel (temporal /random) and multi-pixel (spatial/pattern) 15 Dodge, et. al. “Understanding How Image Quality Affects Deep Neural Networks” © 2022 Commonlands LLC
  16. 16. 1. Exposure + Motion Blur 2. Dynamic Range + Artifacts 3. Noise 4. Color 5. Shading 6. Resolution 7. Distortion 8. Texture Blur 9. Stray Light 10.Fringing + Blooming 11.Blemish 12.Dead Pixels 16 Let’s investigate how objective image quality metrics could impact your computer vision © 2022 Commonlands LLC
  17. 17. Related KPI • ∆E (Color Accuracy) Reference: ISO17321 Color can impact edge contrast when using multiple channels and auto-white balance 17 Afifi+Brown. “ What Else Can Fool Deep Learning? Addressing Color Constancy Errors on Deep Neural Network Performance © 2022 Commonlands LLC
  18. 18. Related KPI • Contrast Detection Probability Reference: ISO12232 Dynamic range and color are closely related to tone mapping which impacts perception at every scale 18 Yeganeh, et. al. “Objective Quality Assessment of Tone-Mapped Images” © 2022 Commonlands LLC
  19. 19. 1. Exposure + Motion Blur 2. Dynamic Range + Artifacts 3. Noise 4. Color 5. Shading 6. Resolution 7. Distortion 8. Texture Blur 9. Stray Light 10.Fringing + Blooming 11.Blemish 12.Dead Pixels 19 Let’s investigate how objective image quality metrics could impact your computer vision © 2022 Commonlands LLC
  20. 20. Related KPIs • Luminance Non-uniformity • Lightness non-uniformity Reference: ISO17957 Luminance shading changes accuracy from center to edge of the field of view 20 Marc Levoy, ICCV 2015, “Extreme imaging using cell phones” © 2022 Commonlands LLC
  21. 21. Shading can include a radial color shift, impacting CV in different parts of the field 21 © 2022 Commonlands LLC
  22. 22. 1. Exposure + Motion Blur 2. Dynamic Range + Artifacts 3. Noise 4. Color 5. Shading 6. Resolution 7. Distortion 8. Texture Blur 9. Stray Light 10.Fringing + Blooming 11.Blemish 12.Dead Pixels 22 Let’s investigate how objective image quality metrics could impact your computer vision © 2022 Commonlands LLC
  23. 23. Resolution comes in many flavors 23 # Pixels Resolution Pixel Size Scene Field of View, Memory/Speed, On- board Functionality “Pixel Resolution” Image Quality Across an Edge in Object Space “Sharpness” Optical Resolution (MTF) System Resolution (SFR) Angular Resolution (deg/px) © 2022 Commonlands LLC
  24. 24. All types of resolution jointly impact the performance of embedded vision systems 24 Xu, et. al. “3D Human Shape and Pose from a Single Low-Resolution Image with Self-Supervised Learning” SOTA=State of the art as of Q1’20: “SPIN” © 2022 Commonlands LLC
  25. 25. Related KPIs • Edge SFR (eSFR), sinusoidal SFR (sSFR) • Lens MTF • Contrast Sensitivity Function (CSF) • Contrast Detection Probability (CDP) Reference: ISO12233, IEEE P2020 The Spatial Frequency Response (SFR) and contrast sensitivity are a corollary to “blur” 25 Vasiljevic, et. Al. “Examining the Impact of Blur on Recognition by Convolutional Networks” © 2022 Commonlands LLC
  26. 26. SFR can also characterize 10+ artifacts resulting from image compression quality 26 Zheng, et. Al “ Improving the Robustness of Deep Neural Networks via Stability Training” Example of Artifacts • Aliasing • Ringing • Blocking Reference: E. Allen, Thesis: “Image Quality Evaluation in Lossy Compressed Imaged” © 2022 Commonlands LLC
  27. 27. Related KPIs • # Pixels per ° • # Pixels per unit distance across an object Angular resolution defines the # of pixels each object has for feature extraction 27 Dollár, et. al. “Pedestrian Detection: An Evaluation of the State of the Art” © 2022 Commonlands LLC
  28. 28. 1. Exposure + Motion Blur 2. Dynamic Range + Artifacts 3. Noise 4. Color 5. Shading 6. Resolution 7. Distortion 8. Texture Blur 9. Stray Light 10.Fringing + Blooming 11.Blemish 12.Dead Pixels 28 Let’s investigate how objective image quality metrics could impact your computer vision © 2022 Commonlands LLC
  29. 29. Related KPI • % Distortion (Optical, TV, SMIA TV) Reference: ISO17850 Distortion is the change in angular resolution (magnification) across field 29 Pei, et. al. “Effects of Image Degradations to CNN-based Image Classification” Negative FΘ Rectilinear Angular resolution and perspective distortion at 45° Off Axis © 2022 Commonlands LLC
  30. 30. Related KPI • % Distortion (Optical, TV, SMIA TV) Reference: ISO17850 Distortion is the change in angular resolution (magnification) across field 30 Negative FΘ Rectilinear Angular resolution and perspective distortion at 45° Off Axis © 2022 Commonlands LLC
  31. 31. Related KPI • % Distortion (Optical, TV, SMIA TV) Reference: ISO17850 Distortion is the change in angular resolution (magnification) across field 31 Li, et.al. 2020, “ULSD: Unified Line Segment Detection across Pinhole, Fisheye, and Spherical Cameras” Rectilinear Spherical/Fisheye © 2022 Commonlands LLC
  32. 32. 1. Exposure + Motion Blur 2. Dynamic Range + Artifacts 3. Noise 4. Color 5. Shading 6. Resolution 7. Distortion 8. Texture Blur 9. Stray Light 10.Fringing + Blooming 11.Blemish 12.Dead Pixels 32 Let’s investigate how objective image quality metrics could impact your computer vision © 2022 Commonlands LLC
  33. 33. Related KPI • Texture SFR Reference: ISO19567 Texture SFR (and loss) results from noise reduction algorithms that filter high frequencies 33 Chen, et. Al. “Texture Construction Edge Detection Algorithm” © 2022 Commonlands LLC
  34. 34. 1. Exposure + Motion Blur 2. Dynamic Range + Artifacts 3. Noise 4. Color 5. Shading 6. Resolution 7. Distortion 8. Texture Blur 9. Stray Light 10.Fringing + Blooming 11.Blemish 12.Dead Pixels 34 Let’s investigate how objective image quality metrics could impact your computer vision © 2022 Commonlands LLC
  35. 35. Related KPIs • Glare spread function • Contrast detection probability Reference: ISO18844 Stray light from lenses create regions of low contrast and low detection probability 35 IEEE P2020 Whitepaper © 2022 Commonlands LLC
  36. 36. 1. Exposure + Motion Blur 2. Dynamic Range + Artifacts 3. Noise 4. Color 5. Shading 6. Resolution 7. Distortion 8. Texture Blur 9. Stray Light 10.Fringing + Blooming 11.Blemish 12.Dead Pixels 36 Let’s investigate how objective image quality metrics could impact your computer vision © 2022 Commonlands LLC
  37. 37. Chromatic aberration from lenses can result in artifacts around high contrast edges 37 Related KPIs • Chromatic Displacement Reference: ISO19084 Kang, “Automatic Removal of Chromatic Aberration from a Single Image” © 2022 Commonlands LLC
  38. 38. There are many types of color fringing, some result from blooming/cross-talk in sensor and tuning 38 Original images © 2022 Commonlands LLC
  39. 39. 1. Exposure + Motion Blur 2. Dynamic Range + Artifacts 3. Noise 4. Color 5. Shading 6. Resolution 7. Distortion 8. Texture Blur 9. Stray Light 10.Fringing + Blooming 11.Blemish 12.Dead Pixels 39 Let’s investigate how objective image quality metrics could impact your computer vision © 2022 Commonlands LLC
  40. 40. Image blemishes occur when dust /dirt / moisture are on the sensor or in / on the lens 40 Related KPIs • # and size of blemishes Reference: https://www.imatest.com/docs/blemish/ Michal Uricar “Let’s Get Dirty: GAN Based Data Augmentation for Camera Lens Soiling Detection in Autonomous Driving.” © 2022 Commonlands LLC
  41. 41. 1. Exposure + Motion Blur 2. Dynamic Range + Artifacts 3. Noise 4. Color 5. Shading 6. Resolution 7. Distortion 8. Texture Blur 9. Stray Light 10.Fringing + Blooming 11.Blemish 12.Dead Pixels 41 Let’s investigate how objective image quality metrics could impact your computer vision © 2022 Commonlands LLC
  42. 42. Dead pixels brings us full circle, as a real-world adversarial attack if no correction is performed 42 Su, et. al. “One Pixel Attack for Fooling Deep Neural Networks” © 2022 Commonlands LLC
  43. 43. Dead pixels brings us full circle, as a real-world adversarial attack if no correction is performed 43 Su, et. al. “One Pixel Attack for Fooling Deep Neural Networks” “Nonetheless, we show some examples of situations where nearly imperceptible image modifications can result in dramatic perception changes. Even in applications without malicious people trying to trick your system, the natural world, [your camera hardware, and your image processing pipeline] may be adversarial enough.” Pezzementi, et. al “Putting Image Manipulations in Context: Robustness Testing for Safe Perception” © 2022 Commonlands LLC
  44. 44. Where do I learn more about how image quality influences computer vision? Visit our website for the slides, the papers cited in this talk, plus related resources: -> Contact me at max.henkart@commonlands.com if working on a camera HW project or looking for lenses Resources through the Alliance • Felix Heide, Embedded Vision Summit 2018: • https://www.edge-ai-vision.com/2018/08/understanding-real-world-imaging-challenges-for-adas- and-autonomous-vision-systems-ieee-p2020-a-presentation-from-algolux/ Notable examples included on our reference page: IEEE P2020 Automotive Image Quality (Computer Vision) White Paper Electronic Imaging (January 2023) and Imaging.org University of Westminster & Nvidia’s Collaboration on Image Quality Metrics 44 https://commonlands.com/summit2022 © 2022 Commonlands LLC
  45. 45. Backup Slides
  46. 46. Contrast loss impacts both human visual perception and CNN-based methods. 46 Geirhos, et. al. “Comparing deep neural networks against humans: object recognition when the signal gets weaker” © 2022 Commonlands LLC
  47. 47. Pezzementi, et. al. “Putting Image Manipulations in Context: Robustness Testing for Safe Perception” • ADR= Average Detection Rate Quantitative Degradation of Agricultural Outdoor Detection Based on Image Quality 47 © 2022 Commonlands LLC
  48. 48. Karahan, et. al. “How Image Degradations Affect Deep CNN- based Face Recognition?” Quantitative Degradation of Facial Recognition Networks Based on Image Quality 48 © 2022 Commonlands LLC
  49. 49. Roy, et. Al. “Effects of Degradations on Deep Neural Network Architectures” Quantitative Degradation of A Variety of Images and Datasets 49 © 2022 Commonlands LLC
  50. 50. Example of CV impact • Must select line detection and/or dewarping methods carefully as camera to camera variations can throw off Hough transfroms and RANSAC • Fewer pixels for detection tasks at edges of negative FΘ lenses KPI • % Distortion (Optical, TV, SMIA TV) Reference: ISO17850 6) Distortion is the change in angular resolution (magnification) across field 50 Negative F-Θ Rectilinear Two 90° HFoV lenses on IMX477. Left 10% of image is shown. Original Images © 2022 Commonlands LLC

×