From Biological to Smart CMOS Imaging: Architectural approach

1. CMOS IMAGE SENSORS
From the Biological Eye to Smart Machine Vision
.
Fayçal Saffih, Ph.D. , Visiting scholar
Communication Research Laboratory (CRL)
McMaster University

2. Outline
 CMOS Imaging Technology
 New Advancements in Smart Cameras
 Applications in the Medical Domain
 Contributions and Future Research

3. CMOS Imaging Technology
InStat/MDR
 Two main electronic imaging technologies: Complementary Metal-
Oxide-Semiconductor (CMOS) and Charge-Coupled-Devices
(CCD)
 Total unit shipments of CMOS image sensors surpassed CCDs for
the first time in 2005. (In-STAT/MDR)
 CMOS image sensor shipments will grow at roughly seven times
the rate of CCDs through 2008 (In-Stat/MDR)

4. Why CMOS imaging is important ?
 Architectures flexibility allows a lot of flexibility of image acquisition in
CMOS imagers compared to their CCD counter parts.
 CMOS Imagers are well known for their low power (20-50 mW) compared to
CCD’s (2-5W) for the same pixel throughput. This is why CMOS imagers
are dominating portable devices market and wireless medical imaging .
 Integration of acquisition and data processing is only possible in CMOS
imaging technology.
 Compatibility with main stream CMOS technology. As such, every CMOS
foundry has CMOS image sensor division.

5. Research Contribution and
Future Work
Foveated Sampling Architectures
for CMOS Image Sensors
Ph. D. Thesis available at:
http://www.cs.yorku.ca/~visor/pdf/FaycalPhDThesis.pdf

6. Philosophical Statements
 Image Scanning Statement
 Image Scanning should be more adapted to image sampling/acquisition
rather than image display compatibility.

Image Information Acquisition is the priority of the imager.
 Image Sampling Statement
 As image resolutions get higher and with it the amount of transmitted
image data for display or processing, new architectures are needed for
down-scaling the sampling resolution for regions of reduced interest.
Innovative architectures are also needed to exploit the Human Visual
System for transmitting only the most important regions in the acquired
image.

Only Regions of Interest are sampled at highest imaging
requirement such as dynamic range, spatial resolution ... etc.

7. Why Foveated CMOS Imagers?
 Foveated image sampling architectures are needed due to:
1. Limited imaging system resources (memory, processing power & time)
2. Not all image areas are of the same degree of importance (or of interest)
3. Only area of interest are sampled at the highest resolutions, highest
dynamic range…etc.
4. Biologically inspired
“Nature does nothing uselessly.”
uselessly
–Aristotle, Politics
“The diversity of the phenomena of nature is so great, and the treasures
hidden in the heavens so rich, precisely in order that the human mind
shall never be lacking in fresh nourishment”
nourishment
–Kepler, from Carl Sagan’s book “Cosmos”

8. Why Foveated CMOS Imagers?
Ramon Cajal identified the retina basic anatomical
structure. Shown here is his sketch of the interconnectivity
configurations of photocells rodes (f) and cones (e) [4]

9. Our Research Contribution Outline
 Standard CMOS Image Sensor
 System Level Approach (Time Domain Fovea)
 Pyramidal CMOS Image Sensor

Architecture
 Physical Design

Scanning

Foveated Dynamic Range Enhancement
 High-Speed Imaging of Pyramidal CMOS Imager

Low Pyramidal Imager FPN Perception by HVS
 Pixel Level Approach (Spatial Domain Fovea)
 Multiresolution CMOS Image Sensor

Architecture

Multiresolution Active Pixel Sensor
 Results
 Conclusion and Future Work

10. The Classical Architecture
& Raster Scanning
Standard CMOS Image Sensor

11. Active Pixel Sensor Standard CMOS Image Sensor

12. The Seminar Outline
 Standard CMOS Image Sensor
 System Level Approach (Time Domain Fovea)
 Pyramidal CMOS Image Sensor

Architecture
 Physical Design

Scanning

Foveated Dynamic Range Enhancement
 High-Speed Imaging of Pyramidal CMOS Imager

Low Pyramidal Imager FPN Perception by HVS
 Pixel Level Approach (Spatial Domain Fovea)
 Multiresolution CMOS Image Sensor

Architecture

Multiresolution Active Pixel Sensor
 Results
 Conclusion and Future Work

13. System Level Approach
 Time Domain Approach:
 Because Dynamic Range enhancement can be
realised by multi-exposure concept, controlling the
topology of imager’s integration time profile may
lead to a non-uniform and interesting Dynamic
Range enhancement
 Design Approach:
 Keep the standard APS structure while changing the
Architectural Sampling structure
Pyramidal CMOS Image
Sensor

14. Pyramidal CMOS Image Sensor
 Architecture …
Classical CMOS Imager Architecture Pyramidal CMOS Imager Architecture
1D-Row Sampling 2D-Ring Sampling
Vertical Busses Diagonal Busses

15. Pyramidal CMOS Image Sensor
 Physical Design …
Physical Design

16. Pyramidal CMOS Image Sensor
 Scanning …
Rolling and Bouncing Scan

17. Foveated Dynamic Range Enhancement
 Mathematical Foundations: Prediction…
Definitions:
Integration time of a pixel (or of RST-
SEL shared sampling entities): Time
between consecutive RST & SEL
signals.
Ts: Pixel scanning time (0.1µs ~ 1µs)
Tspl: Row sampling time row
sampling including Vs sampling, reset,
Vr sampling (1µs ~ 10µs)
r: ring order
R: Total number of rings
Tin(r): Inward scan integration time.
Tout(r): Outward scan integration
time.
 r +1   r −1 
Tin( r , R, Ts , Tspl ) = 2  ∑ iTs + ( R − r )Tspl  + rTs , Tout ( r , Ts , Tspl ) = 2 ∑ s
 iT + ( r − 1)T  + rT
i =1 
i = R  spl s
i→i −1
 
 i →i +1
 

…and after simplification we get:
(
Tin ( r , R, Ts , Tspl ) = −Ts r 2 − 2Tspl r + R 2Ts + RTs + 2 RTspl )
Tout ( r , Ts , Tspl ) = Ts r 2 + 2Tspl r − 2Tspl

18. Foveated Dynamic Range Enhancement
 Mathematical Foundations: Prediction…
(
Tin ( r , R, Ts , Tspl ) = −Ts r 2 − 2Tspl r + R 2Ts + RTs + 2 RTspl )
Tout ( r , Ts , Tspl ) = Ts r 2 + 2Tspl r − 2Tspl
ToT ( R, Ts , Tspl ) = Tin( r , R, Ts , Tspl ) + Tout ( r , Ts , Tspl ) = R 2 Ts + R (Ts + 2Tspl ) − 2 Tspl

19. Foveated Dynamic Range Enhancement
 Mathematical Foundations: Prediction…
 max (Tin( r , R, Ts , Tspl ), Tout ( r , Ts , Tspl ) ) 
DRenh (r ) = 20 log  
 min (Tin ( r , R, Ts , Tspl ), Tout ( r , Ts , Tspl ) ) 
 
1 1 2 
FOVborder (α , R ) =  1 + α R ( R + 1) + α ( R + 1) − 1 , where α = Ts T spl
α 2 

20. Foveated Dynamic Range Enhancement
 Mathematical Foundations: Controllability…
Pinned Foveated Dynamic Range Enhancement

21. Foveated Dynamic Range Enhancement
 Mathematical Foundations: Advantage …
Dynamic Range is usually
associated to the number of
“meaningful” bits of the
digital output of an imager.
 DRdB 
DRbits = log 2 10 20



 
3D view of the pinned FDR enhancement expressed in binary bits.

22. Foveated Dynamic Range Enhancement
 Experimental Setup…
CMOS Pyramidal Imager under test inside black box

23. Foveated Dynamic Range Enhancement
 Experimental Results…
The model extraction will provide us:
1. Lmin: Minimum detectable light intensity
2. Lmax: Maximum saturation light intensity
3. Ss: Sensitivity of the imager
The Objective of this methodology is the extraction of:
1. The System electrical dynamic range
2. The Optical dynamic range enhancement
3. Verification of the mathematical predictions of dynamic range enhancement

24. Foveated Dynamic Range Enhancement
 Experimental Results…
Vcds in RMS voltage the whole pyramidal imager for 8 Slopes of linear regions of photon transfer curve
integration times

25. Foveated Dynamic Range Enhancement
 Experimental Results…
Measured and modeled values for Vcds Correlation of the model to the measured values

26. Foveated Dynamic Range Enhancement
 Experimental Results…
Definitions:
Most researchers in CMOS imagers
agree that Lmin is constant with
integration time.
Lmin ≈ 5Lux
Lmax is calculated based on the model.
It is at which light intensity the pixel
will reach saturation
System and Enhancement Dynamic Ranges Extraction
of the Pyramidal CMOS imager at 1MHz (800 Fps)
CalculDR = 20 Log ( Lmax Lmin )
 max( MaxDetLI inward , MaxDetLI outward ) 
OptDRMx _ MnDet (r ) = 20 log  
 min ( MaxDetLI inward , MaxDetLI outward ) 
 max (Tin( r , R, Ts , Tspl ), Tout ( r , Ts , Tspl ) ) 
Theory_DRenh = 20 Log  
 min (Tin ( r , R, T , T ), Tout ( r , T , T ) ) 
 s spl s spl 

27. Pyramidal CMOS Image Sensor
 Experimental Results: Ring Sampling & Blur Symmetrization
Motion Blur effect on rolling-shutter classical CMOS imagers Global shutter versus rolling shutter and motion blur
 Rolling shutter is used to separate integration time from frame time (speed)
 To cancel motion blur, Global shutter is used but mainly for still imaging applications
 To have a motion-blur free video (continuous) images, global shutter is not adequate due to :
Higher transmission rate requirements to transfer high frame rate images.
Higher power consumption
Excessive heat (increasing dark current).
The above factors are more pronounced with increasing CMOS imagers resolutions.

28. Pyramidal CMOS Image Sensor
 Experimental Results: Ring Sampling & Blur Symmetrization
The PixeLINK CMOS imager window (784x784) exposure (or integration) time = 43.2ms
The integration time the 64x64 pyramidal CMOS imager =
43.24ms

29. Foveated Dynamic Range Enhancement
 Experimental Results…
( + )x½= ,
Inward scan Outward scan Fused Image Rolling scan
Demonstration of Foveated Dynamic Range Enhancement at the Foveal Rings
( + )x½= ,
Inward scan Outward scan Fused Image Rolling scan
Demonstration of Foveated Dynamic Range Enhancement at the Peripheral Rings

30. Pyramidal CMOS Image Sensor
 High-Speed Imaging …
FRPyr R + 1.5 Inner ring
=8 R →∞ → 8
 
FRClass R+7
Outer ring
R=1024
 The Pyramidal CMOS imager is designed primarily for parallel ring sampling with 8 output channels
 This property of multi-channel readout is inherent to the pyramidal architecture in to contrast to multi-channel
technique included in high speed classical CMOS imagers.
 In serial readout of sampled rings:
 Pyramidal imager is faster at its fovea (central rings) than a classical CMOS imager of similar size of the
fovea.
 Pyramidal imager is very close in image sampling speed with the classical imager at higher imager resolutions.

31. Low Pyramidal Imager FPN Perception by
Human Visual System
Oblique effect found in the contrast sensitivity of the HVS at relatively high frequencies [94]
Average spatial power spectrum distribution of about 500 natural scenes [100]

32. Low Pyramidal Imager FPN Perception by
Human Visual System
Classical CMOS Imager FPN
2D-Fourier
Topology
Spectrum

33. Low Pyramidal Imager FPN Perception by
Human Visual System
Pyramidal CMOS Imager FPN
Topology

34. HVS Spatial Filter Application on
Pyramidal FPN images: Model Human Visual System
Low Pyramidal Imager FPN Perception by

35. HVS Spatial Filter Application on
Pyramidal FPN images: Results Human Visual System
Low Pyramidal Imager FPN Perception by

36. HVS Spatial Filter Application on
Pyramidal FPN images: Results Human Visual System
Low Pyramidal Imager FPN Perception by

37. HVS Spatial Filter Application on
Pyramidal FPN images: Results Human Visual System
Low Pyramidal Imager FPN Perception by

38. HVS Spatial Filter Application on
Pyramidal FPN images: Results Human Visual System
Low Pyramidal Imager FPN Perception by

39. Low Pyramidal Imager FPN Perception by
Human Visual System
HVS Spatial Filter Application on
Pyramidal FPN images: Conclusion
 FPN is known to increase with increasing sampling frequency
(or with decreasing integration time). This is why more
oblique stripes are more visible with increasing sampling
frequency.
 The positive value of the oblique FPN stripes infers the
presence of the FPN noise stripes in the tilted HVS filter and
its absence from the normal HVS filter. Therefore, the
pyramidal imager FPN noise is less perceived by normal
human observer in contrast to the normal CMOS imager
counter part.
 The importance of sampling architectures is proven again not
only to provide better imaging performances, but can make
some inherent limitation of CMOS image sensors, such as
higher FPN compared to CCD’s, less noticeable.

40. The Seminar Outline
 Standard CMOS Image Sensor
 System Level Approach (Time Domain)
 Pyramidal CMOS Image Sensor

Architecture
 Physical Design

Scanning

Foveated Dynamic Range Enhancement
 High-Speed Imaging of Pyramidal CMOS Imager

Low Pyramidal Imager FPN Perception by HVS
 Pixel Level Approach (Spatial Domain)
 Multiresolution CMOS Image Sensor

Architecture

Multiresolution Active Pixel Sensor
 Results
 Conclusion and Future Work

41. Pixel Level Approach
 Spatial Domain Approach:
 Only regions of interest are sampled at the highest possible resolution.
Other regions of less (or no) interest are sampled at lower resolutions.
 human Eye has variable interconnectivity configurations between the
photo-cells and the ganglion cells (processing neurons):
 one-to-many configuration for cones (highest resolution needed for
reading)
 Many-to-one configuration for rods (low resolution used for low
light vision)
 Design Approach:
 Keep the CMOS Image Sensor architecture while changing the APS
Sampling structure
Multiresolution CMOS Image Sensor

42. Multiresolution CMOS Image Sensor
 Image Sampled at highest resolution in all pixels
 Only Regions-Of-Interest (ROI) are kept at their highest resolution
(no-charge distribution)
 Regions-Of-Less-Interest (ROLI) are selectively down-resolved by
a charge-sharing mechanism to get a single value representing their
average.
 Previous implementations were chip and column level approaches
using the same concept.
 The suggested approach is pixel-based to ensure the expandability
of the architecture to any imager size without increasing
complexity.
 Further details about the mechanism implementation, layout design as
well as simulated Multiresolution foveated images are discussed in the
thesis.

43. The Seminar Outline
 Standard CMOS Image Sensor
 System Level Approach (Time Domain)
 Pyramidal CMOS Image Sensor

Architecture
 Physical Design

Scanning

Foveated Dynamic Range Enhancement
 High-Speed Imaging of Pyramidal CMOS Imager

Low Pyramidal Imager FPN Perception by HVS
 Pixel Level Approach (Spatial Domain)
 Multiresolution CMOS Image Sensor

Architecture

Multiresolution Active Pixel Sensor
 Results
 Conclusion and Future Work

44. Conclusion and Future Work
Conclusion:
The architecture flexibility of CMOS imaging technology was:
Key factor:
 In enhancing the optical dynamic range following a foveated profile
mimicking human eye (biological) vision.
Key factor:
 In minimizing the perception of the CMOS imager FPN noise by human
observer benefiting from HVS spatial discrimination limitation (Oblique
Effect)
Key factor:
 In developing similar multiresolution mechanisms as those found at retina of
the human eye which lead to an expandable multiresolution CMOS imager.
Future Work
Biological vision systems can give a great help in synthesising novel
architectures exhibiting more adaptability, to light intensity for higher ranges
and spatial resolving power. The Pyramidal CMOS imager and the
multiresolution CMOS imager are first steps towards this vision.

45. Thank you