A New In-Camera Imaging Model For Color Computer Vision And Its Application

A New In-Camera Imaging Model for
Color Computer Vision and its Application
Hai Ting LIN
Supervisor: Prof. Brown
National University of Singapore

May 9th, 2012

Joint work with:
Seon Joo Kim, Lu Zheng, Sabine Süsstrunk, Stephen Lin

Talk overview
• Motivation
– Modern digital cameras
– Cameras and computer vision
• Our work (ICCV’11)
– A new imaging pipeline for cameras
• Application of our pipeline
– sRGB to RAW
– Photo refinishing (and back)

Camera = light-measuring device
Illumination source
(Scene radiance)

Output (digital) image
Imaging System

Internal Image Plane
Scene Element

Simple models assume an image is a measurement of scene radiance.

From Digital Image Processing, Gonzales/Woods

Camera = light-measuring device
Shape-from-shading

•Shape from shading
•Intrinsic image
•Image Matching
•HDR Imaging
•Etc . . .

Lu et al, CVPR’10
Image Matching HDR Imaging

From Jon Mooser, CGIT Lab, USC From O’Reilly’s digital media forum

Onboard Photofinishing
“Secret Recipe” of a Camera

Three different cameras with same aperture, exposure,
white-balance and picture style, etc. . .

Can image values be treated as
physically meaningful values?

And if so, when and how?

Modern photography pipeline

Scene Radiance
Camera Output: sRGB

Pre-Camera
Starting point: Lens Filter In-Camera
reality (in radiance) Lens CCD response (RAW)
Shutter CCD Demosaicing (RAW)
Aperture
“Photo-finishing Processing”

8

Digital camera pipeline (early work)
“Radiometric Calibration”

E*k (RAW) f (x)

Unknown f . . . the camera’s non-linear response to RAW.

Debevec and Malik *SIG’97+

Accepted model

 Erx   erx   Erx   irx   f r (erx ) 
E  e  E  i   
=  f g (egx )
 gx 
 Ebx 
 gx  =
 ebx 
T  gx 
 Ebx 
 gx 
ibx 
   f b (ebx ) 
 
     
(RAW) T is a 3x3 matrix i is the sRGB
output and f is
a non-linear
function

Related work
Mann and Picard, SPIE’95

Fixed property of the camera
Debevec and Malik, SIG’97
Mitsunaga and Nayar, CVPR’99
Farid, TIP’01
Grossberg and Nayar, TPAMI’03
Lin et al, CVPR’04
…
Manders et al, ICIP’04
 erx   f r (irx ) 
1
Pal et al, CVPR’04
e   f 1 (i )  Lin et al, ICCV’05
 gx  =  g gx  Kim and Pollefeys, TPAMI’08
 ebx   f b1 (ibx ) 
   

Related work
Mann and Picard, SPIE’95

Fixed property of the camera
Debevec and Malik, SIG’97
Mitsunaga and Nayar, CVPR’99
Farid, TIP’01
Lin et al, CVPR’04
…
Manders et al, ICIP’04
 erx   f r (irx ) 
1
Pal et al, CVPR’04
e   f 1 (i )  Lin et al, ICCV’05
 gx  =  g gx  Kim and Pollefeys, TPAMI’08
 ebx   f b1 (ibx ) 
    Chakrabarti et al, BMVC’09

Chakrabarti et al conclusions:
 RAW is meaningful . . . .
 But, requires a 24 parameter model that is scene-dependent
to accurately go back from sRGB to RAW.

Accepted model

Is processing scene dependent?
Is this model good enough?
 irx   f r (erx )   erx   Erx 
i   f (e )  e  E 
 gx  =  g gx 
ibx   f (ebx ) 
 gx  =
 ebx 
T  gx 
 Ebx 
       

Observations
Where are the outliers?

Color rendering

Gamut Mapping

Gamut mapping is necessary because the
camera’s gamut
gamut of the camera’s color space is different
sRGB gamut from the gamut of sRGB.

Gamut mapping is a natural mechanism to
support picture styles, such as portrait,
landscape, etc.

Color rendering
Gamut mapping (color adjustment part)

From Canon webpage

Proposed a new model

RAW sRGB
Tw

Ts
White Balance (Tw) Gamut Mapping Tone Mapping
Camera RAW to sRGB (Ts) (h) (f) sRGB
(RAW) (JPEG)

Proposed a new model

RAW sRGB
Tw

Ts
(RAW) (JPEG)

 irx    Erx  
i   f (h  T  E  )
 gx    gx  
ibx   E  
    bx  

sRGB Image to RAW
 f 1 (irx )    erx     Erx  
 1     
 f (igx )  h  egx   h  T  E gx  
 f 1 (ibx ) 
=  
 e   = 
 E  

    bx     bx  

based on several sRGB-RAW image pairs,
• f-1 & T-1 are computed using less saturated points
• h-1 is computed with scatter point interpolation via radial basis func.

Canon EOS1Ds Mark III Picture Styles
Ts White Balance S Standard P Portrait L Landscape …
…
Tw1 Tw2 Tw3 Tw4 h1, f1 h2, f2 h3, f3

RAW sRGB
Tw

Ts
(RAW) (JPEG)

Gamut Mapping

Mapping is represented as a displacement map of the camera’s
original RGB value to its sRGB location.

Experiments : Mapping Image to RAW


input sRGB image ground truth RAW

Canon EOS1D


input sRGB image estimated RAW

Canon EOS1D


new model (f, T, h) old model (f, T)
We cannot handle
fully saturated points.
Canon EOS1D



Canon EOS550D



Canon EOS1D



Sony A200

Application: Photo Refinishing

…
Tw1 Tw2 Tw3 Tw4 h1, f1 h2, f2 h3, f3

RAW sRGB
Tw

Ts
(RAW) (JPEG)
What if you took a photo
with the wrong settings?

…
Tw1 Tw2 Tw3 Tw4 h1, f1 h2, f2 h3, f3

RAW sRGB
Tw -1

Ts -1

White Balance (Tw -1) Gamut Mapping Tone Mapping
Camera RAW to sRGB (Ts -1) (h -1) ( f -1) sRGB
(RAW) (JPEG)

…
Tw1 Tw2 Tw3 Tw4 h1, f1 h2, f2 h3, f3

RAW sRGB
Tw

Ts

Camera sRGB
Tw -1
(RAW) (JPEG)

Ts -1

White Balance (Tw -1) Gamut Mapping Tone Mapping
RAW to sRGB (T -1) (h -1) ( f -1)

Result - Canon EOS 1Ds Mark III

Input: cloudy WB + landscape style


Ground truth: fluorescent WB + standard style


Photoshop result


Refinished result


Input Ground truth

Photoshop Our refinished result


Input: tungsten WB + standard style


Ground truth: daylight WB + standard style


Our refinished result

Result – Nikon D200



Photoshop result


Refinished result


Input Ground truth

Photoshop Photo refinish

Result - Sony α200


Result - Sony α200

Photoshop result

Result - Sony α200

Our refinished result

Result - Sony α200

Input Ground truth

Photoshop Our Refinished Result

Conclusion
• Proposed a new camera processing model
– Gamut mapping was introduced
– Allowed us to accurately calibrate for scene modes
– Allowed for accurate remapping from sRGB back
to RAW
• Facilitated refinishing application
– Camera-specific refinishing
– Our result is what the camera would have
performed

A New In-Camera Imaging Model For Color Computer Vision And Its Application

More Related Content

Viewers also liked

Similar to A New In-Camera Imaging Model For Color Computer Vision And Its Application

More from IPALab

Recently uploaded

A New In-Camera Imaging Model For Color Computer Vision And Its Application