Ph.D. Candidate, Electrical and Computer Engineering,
Center for Intelligent Machines (CIM)
McGill University
(1) Time: Wednesday, Dec. 17th, 12:30-14:30 pm
(1) Location: faculty’s conference room, Isfahan University of Technology
(2) Time: Tuesday, Dec. 9th, 12:30-14:00pm
(2) Location: Room 212, School of Electrical and Computer Engineering of University of Tehran
Abstract:
Investigating low light imaging is of high importance in the field of color science from different perspectives. One of the most important challenges arises at low light levels is the issue of noise, or more generally speaking, low signal to noise ratio. In the present work, effects of different image sensor noises such as: photon noise, dark current noise, read noise, and quantization error are investigated on low light color measurements. In this regard, a typical image sensor is modeled and employed for this study. A detailed model of noise is considered in the process of implementing the image sensor model to guarantee the precision of the results. Several experiments have been performed over the implemented framework and the results show that: first, photon noise, read noise, and quantization error lead to uncertain measurements distributed around the noise free measurements and these noisy samples form an elliptical shape in the chromaticity diagram; second, even for an ideal image sensor, in very dark situations, stable measuring of color is impossible due to the physical limitation imposed by the fluctuations in photon emission rate; third, dark current noise reveals dynamic effects on color measurements by shifting their chromaticities towards the chromaticity of the camera black point; fourth, dark current dominates the other sensor noise types in the image sensor in terms of affecting measurements. Moreover, an SNR sensitivity analysis against the noise parameters is presented over different light intensities.
Mehdi Rezagholizadeh: Image Sensor Modeling: Color Measurement at Low Light Levels
1. Color Perception at Low Signal
to Noise Levels
By:
Mehdi REZAGHOLIZADEH
MCGILL UNIVERSITY
DEPARTMENT OF ELECTRICAL AND COMPUTER ENGINEERING
Supervisor:
Prof. James J. CLARK
Brief Overview
DECEMBER 2014
2. Statement of Problems
• Developing a Fast and Accurate Color
Constancy Method
• Color Appearance Modeling at Low Light
Levels
• Image Sensor Modeling and Color
Measurement at Low Light Levels
2
3. Definition of Color Constancy
• Discounting the illuminant effect on the color
of objects
• Color constancy is a great feature of our
visual system
3Computational Color Constancy
4. Importance of Color Constancy
• Applications:
Object Recognition
Image Enhancement
Robot Vision
Object Tracking
Photography and Film
Industry
The last three cases are
real-time applications of color
constancy.
4Computational Color Constancy
6. Problem of Color Constancy
• Image Formation Model:
ܴ = න ܧ ߣ ܵ ߣ, ݔ ߩ ߣ ݀ߣ
ܧ ߣ : the illuminant spectrum
ܵ(ߣ, :)ݔ the surface spectral reflectance function
at location .ݔ
ߩ(ߣ): the sensor spectral sensitivity
ܴ: the sensor response of ith channel
6Computational Color Constancy
7. Problem of Color Constancy
• The Transformation Imposed by a Change in Illumination:
ܴ = න ܧ ߣ ܵ ߣ, ݔ ߩ ߣ ݀ߣ
Givens:
ߩ(ߣ): the sensor spectral sensitivity
ܴ: the sensor response of ith channel
Unknowns:
ܧ ߣ : the illuminant spectrum
ܵ(ߣ, :)ݔ the surface spectral reflectance function at location .ݔ
Spectral Color Constancy Approaches try to find the entire spectrum
of the illuminant and the surface spectral reflectance function.
It is an ill-posed problem.
7Computational Color Constancy
8. Methods of Color Constancy
Computational
Color Constancy
Spectral Methods
Non-spectral
Methods
Static Methods
Gamut-Based
Learning-Based
8Computational Color Constancy
9. Non-Spectral Methods:
• MAIN Objective of this problem is to obtain:
ܴ
ௌ
= න ܵ ߣ, ݔ ߩ ߣ ݀ߣ
It is equivalent to obtaining the sensor responses when
ܧ ߣ = 1.
• Assuming that color of the illuminant can be estimated:
ܴ
ா
= න ܧ ߣ ߩ ߣ ݀ߣ
• The Transformation Imposed by an illuminant can be
obtained through an Over-simplification:
ܴ
ௌ
≅
ܴ
ܴ
ா =
ܧ ߣ ܵ ߣ, ݔ ߩ ߣ ݀ߣ
ܧ ߣ ߩ ߣ ݀ߣ
9Computational Color Constancy, March. 2014
10. Non-Spectral Methods:
Over-simplification leads to:
- Estimating the illuminant color
rather than
- Estimating the entire spectrum of the illuminant )ߣ(ܧ
• Corrective Transformation:
ܴଵ
௦
ܴଶ
௦
ܴଷ
௦
=
1
ܴଵ
ா 0 0
0
1
ܴଶ
ா 0
0 0
1
ܴଶ
ா
ܴଵ
ܴଶ
ܴଷ
10Computational Color Constancy, March. 2014
11. Problem II:
Color Appearance Modeling at Low Light Levels
• Color Appearance Model (CAM):
• An ideal color appearance model:
The output resembles human perception in all
conditions including different light levels
• Lack of a good color appearance model
for low light conditions
Color Perception at Low Signal to Noise Levels 11
TransformTransform
Tristimulus
values (RGB)
Perceptual attributes of
color:
lightness, hue, chroma
12. Biophysical Background
• Our eye can work in three different modes:
1- Photopic condition (Luminance>5 cd/m2 )
2- Mesopic condition (0.005<Luminance<5 cd/m2 )
3- Scotopic condition (Luminance<0.005 cd/m2 )
• Photopic Condition: (High Light Levels)
Color Perception at Low Signal to Noise Levels 12
• Mesopic Condition: (Low Light Levels)• Scotopic Condition: (Very Low Light Levels)
13. Background and Preliminaries
• Existing Models for Mesopic & Scotopic Vision:
Modeling Blue Shift in Moonlit scenes [1]
– Addresses scotopic vision by adding some blue to the initial image
– The output of this algorithm does not look natural and realistic
Cao’s Model of Mesopic Vision [2]
– It is a two stage model based on the gain control and cone opponent
mechanisms
– Model is fitted to the psychophysical experiment data
iCAM06 Tone Compression Model for Mesopic Vision [3]
– iCAM06 includes rod responses in a linear fashion
Shin’s Color Appearance Model [4]
– Boynton two-stage model is fitted to the behavioral experiment data
13
[1] S. M. Khan and S. N. Pattanaik, “Modeling blue shift in moonlit scenes by rod cone interaction,” Journal of VISION, vol. 4, no. 8,
2004.
[2] D. Cao, J. Pokorny, V. C. Smith, and A. J. Zele, “Rod contributions to color perception: linear with rod contrast," Vision research, vol.
48, no. 26, pp. 2586-2592, 2008.
[3] J. Kuang, G. M. Johnson, and M. D. Fairchild, “iCAM06: a rened image appearance model for HDR image rendering," Journal of
Visual Communication and Image Representation, vol. 18, no. 5, pp. 406 -414, 2007.
[4] J. Shin, N. Matsuki, H. Yaguchi, and S. Shioiri, “A color appearance model applicable in mesopic vision," Optical review, vol. 11, no.
4, pp. 272-278, 2004.
14. Physics & Color Perception
Color Perception at Low Signal to Noise Levels 14
Problem
• Lack of a good color appearance model
(CAM) for the low light conditions
Physics
• The basic physical principles governing the
probabilistic nature of color perception at
low light levels
Analysis
• Photon Detection and Color Perception at
low light levels
15. Proposed Method:
Maximum Entropy Spectral Modeling Approach to the
Low Light Levels Color Appearance Modeling
• Under very low light conditions:
The photoreceptor responses more uncertain
• Hypothesis:
Color Perception at Low Signal to Noise Levels 15
Visual Processing Center reconstructs
a part of the information being lost in the
projection of light spectra into the space
of photoreceptor responses
16. Proposed Method:
Maximum Entropy Spectral Modeling Approach to the
Low Light Levels Color Appearance Modeling
• The spectral theory of color perception [Clark and Skaff,
2009]:
Provides a tool to address the issues of uncertain
measurements
Estimates the spectral power distributions corresponding
to these uncertain measurements.
Color Perception at Low Signal to Noise Levels 16
17. Proposed Method:
Maximum Entropy Spectral Modeling Approach to the
Low Light Levels Color Appearance Modeling
• Spectral Model of Mesopic Vision
Clark and Skaff proposed a spectral model for color perception
which is valid for photopic conditions
During the mesopic condition, both cones and rods contribute to the
vision
Given the measurement vector ,ݎ we can model the rod intrusion
into the perception as follows:
ݎ = ߚ ݂
ߣ + ߦݓ݂ ߣ ߣ ݀ߣ + ߥ
ஃ
݅ ∈ {,ܮ ,ܯ ܵ}
- ࢌࢉ(ࣅ) and ࢌ࢘(ࣅ): cone and rod spectral sensitivity functions respectively
- (ࣅ): normalized mesopic spectral power distribution
- ߚ: intensity factor
- ࣈ [0, 1]: a parameter which determines relative rod intrusion
- ܹ = [ݓ
ݓெ
ݓௌ]: a diagonal matrix specifies the relative contribution of rod response to each cone
channel.
- ߥ: additive noise
17Color Perception at Low Signal to Noise Levels
18. Proposed Method:
Maximum Entropy Spectral Modeling Approach to the
Low Light Levels Color Appearance Modeling
• Spectral Model of Mesopic Vision
Given the measurement vector ,ݎ we can model the rod intrusion
into the perception as follows:
ݎ
= ߚ ݂
ߣ + ߦݓ
݂ ߣ ߣ ݀ߣ + ߥ
ஃ
݅ ∈ {,ܮ ,ܯ ܵ}
18
19. 19
Proposed Method:
Maximum Entropy Spectral Modeling Approach to the
Low Light Levels Color Appearance Modeling
An exponential family is employed to estimate ߣ :
̂ ߣ = exp (< ݂ ߣ , ߠ > −߰(ߠ))
݂ ߣ = ݂ ߣ + ߦܹ݂ ߣ
ߠ: parameter vector which should be estimated
߰(ߠ): normalizing function
Parameters can be estimated as follows:
ߠመ = min
ఏ
{ ߟො − ߟ ்
ߟ( ܣො − ߟ)} − ߛ})ߠ(ܪ
ߟ = ߚ/ݎ normalized measurement
:)ߠ(ܪ entropy function corresponding to ̂(ߣ)
A: positive definite matrix
ࢽ: regularization factor
• Spectral Model of Mesopic Vision
Color Perception at Low Signal to Noise Levels
20. Results:
• Simulation of Munsell patches
– surrounded by a white background
– viewed under different light levels from scotopic to
photopic.
Color Perception at Low Signal to Noise Levels 20
21. Image Sensor Modeling:
Color Measurement at Low Light Levels
By:
Mehdi REZAGHOLIZADEH
James J. CLARK
MCGILL UNIVERSITY
November 2014
22nd Color and Imaging Conference
22. The Presentation Outline:
Image Sensor Modeling: Color Measurement at Low Light Levels 22
Conclusion
Experiments and Results:
Preparation Caveats Analysis Experiment Scenarios
Solution: Image Sensor Modeling
Physical Background Noise Model Pixel Measurement Model
Introduction:
Motivation Statement of the Problem: Color Measurement at Low Light Levels
23. Introduction:
Motivation
Importance of Studying low light levels:
• Color Measurement at low light level
becomes more uncertain due to the
low signal to noise ratio
• Most of the theories, measures,
models and methods in color science
are developed for high intensities
• The quality of the human color vision
at low light levels is much better than
existing handy cameras
23Image Sensor Modeling: Color Measurement at Low Light Levels
Kirk, Adam G., and James F. O'Brien. "Perceptually based tone
mapping for low-light conditions." ACM Trans. Graph. 30.4 (2011): 42.
24. Introduction:
Statement of the Problem
24
Problem:
• What is the impact of noise at low light
levels on the color measurements of
imaging devices?
Image Sensor Modeling: Color Measurement at Low Light Levels
25. Applications of the Study:
Spectral Imaging
Image Processing
Low Light Photography
Characterizing the Noise of Image Sensors
Developing Denoising and Enhancement Algorithms
Photon Limited Imaging (biosensors, astronomy, etc)
25Image Sensor Modeling: Color Measurement at Low Light Levels
26. What Next…
26
Conclusion
Experiments and Results:
Preparation Caveats Analysis Experiment Scenarios
Solution: Image Sensor Modeling
Physical Background Noise Model Pixel Measurement Model
Introduction:
Motivation Statement of the Problem: Color Measurement at Low Light Levels
Image Sensor Modeling: Color Measurement at Low Light Levels
27. Physical Background
27
• Simulating the effect of Photon Noise (given the high
intensity description of the light):
• For each bin:
ܲ ݃(ߣ), ݊ ൌ
ఒ
ష ഊ
!
0
2
4
6
8
10
400
420
440
460
480
500
520
540
560
580
600
620
640
660
680
700
AveragePhotonCount:g(λ)
Wavelength (nm)
ᵟ
Image Sensor Modeling: Color Measurement at Low Light Levels
28. Physical Background
28
• A set of Poisson distributions (one for each bin)
characterizes the targeted light.
• To estimate the spectral radiance at a lower intensity:
• The estimated quantal spectral radiance:
ܮிே ߣ =
ܩ෨ி(ߣ)
ߜ
0
10
g(λ)
Wavelength (nm)
Image Sensor Modeling: Color Measurement at Low Light Levels
ܨ =
low intensity
high intensity
Draw samples from
ࡼ࢙ ࡲ ൈ ࢍ ࣅ
ࡺ ࡳ෩ࡲ ࣅ ~ࡼ࢙ሺࡲ ൈ ࢍሺࣅሻሻ
29. Simulation:
How Does Spectral Power Distribution Change with Intensity?
• The estimated spectral power distribution at
different intensities.
ܨ = 5 ൈ 10ିଵଶ
ܹܽݐݐ
ߜ ൌ 5 ݊݉
ݐ ൌ 0.2 ܿ݁ݏ
Color Perception at Low Signal to Noise Levels 29
30. Simulation:
How Does Spectral Power Distribution Change with Intensity?
• The estimated spectral power distribution at
different intensities.
Color Perception at Low Signal to Noise Levels 30
ܨ ൌ 5 ൈ 10ିଵଷ
ܹܽݐݐ
ߜ ൌ 5 ݊݉
ݐ ൌ 0.2 ܿ݁ݏ
31. Simulation:
How Does Spectral Power Distribution Change with Intensity?
• The estimated spectral power distribution at
different intensities.
Color Perception at Low Signal to Noise Levels 31
ܨ ൌ 5 ൈ 10ିଵସ
ܹܽݐݐ
ߜ ൌ 5 ݊݉
ݐ ൌ 0.2 ܿ݁ݏ
32. Image Sensor Modeling
32
• Image sensor pipeline (for a single channel):
Noise
Model
Photon Shot
Noise
Dark Current
Noise
Read Noise
Quantization
Noise
Image Sensor Modeling: Color Measurement at Low Light Levels
33. Image Sensor Modeling
33
• Image sensor pipeline (for a single channel):
Noise
Model
Photon Shot
Noise
Dark Current
Noise
Read Noise
Quantization
Noise
Image Sensor Modeling: Color Measurement at Low Light Levels
34. Image Sensor Modeling
34
• Image sensor pipeline (for a single channel):
Noise
Model
Photon Shot
Noise
Dark Current
Noise
Read Noise
Quantization
Noise
Image Sensor Modeling: Color Measurement at Low Light Levels
35. Image Sensor Modeling
35
• Image sensor pipeline (for a single channel):
Noise
Model
Photon Shot
Noise
Dark Current
Noise
Read Noise
Quantization
Noise
Image Sensor Modeling: Color Measurement at Low Light Levels
36. Image Sensor Modeling
36
• Image sensor pipeline (for a single channel):
Noise
Model
Photon Shot
Noise
Dark Current
Noise
Read Noise
Quantization
Noise
Image Sensor Modeling: Color Measurement at Low Light Levels
37. Image Sensor Modeling:
Noise Model
37
• Variations in the number of emitted photons
• Can be modeled by a Poisson Distribution
Photon Shot Noise
• The current produced inside the image sensor
• ܰௗ
(ߙ, ߚ)~ܲ(ݏ݅ ߪௗ
ଶ
)
Dark Current
Noise
• The noise in the readout circuit
• ܰௗ~ܰ(0, ߪௗ)
Read Noise
• The error introduced in the quantization step
Quantization
Noise
Image Sensor Modeling: Color Measurement at Low Light Levels
39. What Next…
39
Conclusion
Experiments and Results:
Preparation Caveats Analysis Experiment Scenarios
Solution: Image Sensor Modeling
Physical Background Noise Model Pixel Measurement Model
Introduction:
Motivation Statement of the Problem: Color Measurement at Low Light Levels
Image Sensor Modeling: Color Measurement at Low Light Levels
40. Experiments & Results:
Dataset and Preparation
Dataset: “A data set for Color Research”
By: Barnard et al.
Includes:
- The Sony DXC-930 sensor sensitivity curves
- The spectra and color measurements of 598 color samples
made by the Sony camera
40
[1] K. Barnard, L. Martin, B. Funt, and A. Coath, “A data set for color research,” Color Research & Application, vol. 27, no. 3, pp.
147-151, 2002.
41. Experiments & Results:
Dataset and Preparation
Preparation:
− 20 samples from the 598 color measurements are selected
for our experiments
− By scaling the initial spectra, the luminance values of color
samples are set to 100
41
− The luminance of each color
sample is modified by applying
the intensity factor, F.
Image Sensor Modeling: Color Measurement at Low Light Levels
42. Experiments & Results:
Caveats
42
• Temperature is assumed constant, hence the dark noise
parameters are fixed during the experiments.
• Noise model is additive
• The Sony DXC-930 camera is nearly linear for most of its
range, provided it is used with gamma disabled.
• Raw output images are considered for our analysis.
• The effects of reset noise, photodetector response
nonuniformity (PRNU), dark signal nonuniformity(DSNU) are
considered negligible.
Image Sensor Modeling: Color Measurement at Low Light Levels
44. Experiments & Results:
Scenario I: Ideal Image Sensor
44
Assumptions:
• Sensor is ideal (no internal noise in the model)
• Photon shot noise may corrupt the measurements
• log ܨ ∈ ሼ0, െ7, െ8, െ9, െ10, െ11, െ12, െ13, െ14ሽ
Image Sensor Modeling: Color Measurement at Low Light Levels
45. Experiments & Results:
Scenario I: Ideal Image Sensor
45Image Sensor Modeling: Color Measurement at Low Light Levels
Chromaticity of Measured Samples at
Different Light Levels
Magnified Result of the Data Point Indexed 3
at Different Intensity Factors
46. Experiments & Results:
Scenario II: Effects of Dark Current
46
Assumptions:
• Only photon shot noise and dark noise may corrupt
the measurements
• Only boundary color patches are used (index: 1-13)
• ܨ ∈ ሼ1, 0.5, 0.1, 0.05, 0.01, 0.005, 0.001ሽ
Image Sensor Modeling: Color Measurement at Low Light Levels
47. Experiments & Results:
Scenario II: Effects of Dark Current
47Image Sensor Modeling: Color Measurement at Low Light Levels
Chromaticity of Measured Samples at
Different Light Levels
Magnified Result of the Data Point Indexed 3
at Different Intensity Factors
48. Experiments & Results:
Scenario III: Real Image Sensor Model
48
Assumptions:
• A model of real image sensor is
considered
• ܨ ∈ ሼ1, 0.5, 0.1, 0.05, 0.01, 0.005, 0.001ሽ
Image Sensor Modeling: Color Measurement at Low Light Levels
49. Experiments & Results:
Scenario III: Real Image Sensor Model
49Image Sensor Modeling: Color Measurement at Low Light Levels
Chromaticity of Measured Samples at
Different Light Levels
Magnified Result of the Data Point Indexed 3
at Different Intensity Factors
50. Experiments & Results:
Comparing the Three Scenarios
50Image Sensor Modeling: Color Measurement at Low Light Levels
Scenario I Scenario II Scenario III
51. Experiments & Results:
Comparing the Three Scenarios
51Image Sensor Modeling: Color Measurement at Low Light Levels
Scenario I Scenario II Scenario III
52. What Next…
52
Conclusion
Experiments and Results:
Preparation Caveats Analysis Experiment Scenarios
Solution: Image Sensor Modeling
Physical Background Noise Model Pixel Measurement Model
Introduction:
Motivation Statement of the Problem: Color Measurement at Low Light Levels
Image Sensor Modeling: Color Measurement at Low Light Levels
53. Conclusion
53
− Photon noise
− read noise
− quantization error
The physical limitation
imposed by the photon noise
Dark current dominates the other sensor noise types
in the image sensor
Image Sensor Modeling: Color Measurement at Low Light Levels
Uncertain measurements distributed
around the noise free measurements
Dark
current
noise
dynamic
effects
on color
measur
ements
Shifting
chromaticities
towards the
camera black
point
1
2
3
4
Prevents stable measuring of color
(even for an ideal image sensor)
54. Image Sensor Modeling: Color Measurement at Low Light Levels 54
Thank You for Your Attention!
Questions…
55. Image Sensor Modeling
55
• Image sensor pipeline (for a single channel):
Noise
Model
Photon Shot
Noise
Dark Current
Noise
Read Noise
Quantization
Noise
Image Sensor Modeling: Color Measurement at Low Light Levels
56. Image Sensor Modeling:
Pixel Measurement Model
56
Output of the
Image Sensor
• ܩష: conversion gain (volts/݁ି
)
• ݂௦௧: saturation function of the sensor
• ܳ
ߣ : the quantum efficiency function of the sensor
• ܮிே: the quantal radiance at the intensity factor F (photons/sec/݉ଶ
/sr/nm)
ܸ
ߙ, ߚ ൌ ܩష ൈ ݂௦௧ ܶ ൈ න ܮிே ߙ, ߚ, ߣ ܳ
ߣ ݀ߣ ܶ ൈ ܰௗ
ሺߙ, ߚሻ
Image Sensor Modeling: Color Measurement at Low Light Levels