An implementation approach to flash photography and cartoonification in image processing

1. Bilateral filtering
Image processing project
Aryan Agal: 16D170004
Satya Sahoo: 16D170026

2. Bilateral filtering ● The Bilateral filter is a robust edge-preserving
filter introduced by Tomasi and Manduchi.
● It combines a classic low-pass filter with an
edgestopping function that attenuates the filter
kernel weights when the intensity difference
between pixels is large.

3. Bilateral filtering
Gaussian Blur Bilateral filtering

4. Bilateral vs Gauss smoothening visualised

5. Bilateral filtering

6. Bilateral filtering

7. Digital Photography with
Flash and No-Flash Image
Pairs

8. Digital
Photography with
Flash and No-Flash
Image Pairs
● Capturing the natural ambient
illumination in such low-light
environments
○ a long exposure time
○ open the aperture to let in more light
○ increase the camera’s gain
● Adding a flash circumvents these
problems
● The use of flash can also have a negative
impact on the lighting characteristics of
the environment.

9. How it works

10. Algorithm

11. Joint bilateral filter
● We modify the basic bilateral filter to compute the edge-stopping function using
the flash image F instead of A .

12. Denoising

13. Flash-To-Ambient Detail Transfer
● The joint bilateral filter cannot add detail that may be present in the flash image.
● To transfer this detail we begin by computing a detail layer from the flash image
as the following ratio:
● The ratio captures the local detail variation in F and is commonly called a quotient
image or ratio image in computer vision.
● Reduces haloing

14. ● Light from the flash can introduce
shadows and specularities into the
flash image.
● Within flash shadows, the image
may be as dim as the ambient
image and therefore suffer from
noise.
● May form artificial edges
Detecting Flash
Shadows and
Specularities

15. Flash Shadows
● To deal with noise and inter-
reflection, we compute the
shadow mask by looking for pixels
in which the difference between
the linearized flash and ambient
images is small:
Detecting Flash
Shadows and
Specularities
- continued

16. Flash Specularities
● Specular regions should be bright
in F and should therefore saturate
the image sensor. Hence, we look
for luminance values in the flash
image that are greater than 95% of
the range of sensor output values.
Detecting Flash
Shadows and
Specularities
- continued

17. Final Merge
● We now have 2 masks, these
masks can be contaminated by
noise, with small speckles, holes
and ragged edges.
● To clean up the mask, we use
morphological operations to erode
the speckles, and fill the holes. To
produce a conservative estimate,
we dilate the mask.
● Final mask M is obtained by taking
a union of both the masks.
Detecting Flash
Shadows and
Specularities
- continued

18. Cartoonification of an
image using bilateral
filtering

19. Cartoonification of
an image using
bilateral filtering
● One very obvious application of
bilateral filtering is the cartoonification
of images.
● The algorithm we have used in this
project is to first apply a bilateral filter
on an image, then apply an edge
detection method, and superimpose
the two images.
● The edge detection method used in
this project is using the sobel/prewitt
operator or canny edge detector, LoG,
DoG, etc.

20. RESULTS

21. VASE - Denoising

22. VASE - Flash-To-Ambient Detail Transfer
Very slight change, slight increase in detail

23. Continuous Flash Adjustment

24. Cartoonification
Original Cartoonified
Edge detection

25. Cartoonification
Original

26. Cartoonification
Cartoonified

27. Cartoonification
Edge detection

28. Cartoonification
Original

29. Cartoonification

30. Cartoonification
Edge detection

31. Algorithm fails in portrait images!
Original Bilateral
Edge detection Cartoonified

32. Further work
● Bilateral filter is expensive
● Brute-force computation is slow (several minutes) because of nested for
loops
● Non-linear, depends on image content (no FFT, no precomputation)
● Fast approximations exist
○ Significant loss of accuracy
○ No formal understanding of accuracy versus speed

33. Further work
● SEPARABLE BILATERAL FILTERING FOR FAST VIDEO PREPROCESSING -
Tuan Q. Pham, Lucas J. van Vliet(2005)
○ Present a separable implementation of the bilateral filter
○ 2-D bilateral kernel approximated as two separable, cheap 1-D parts
● Real-time O (1) bilateral filtering - Q Yang, KH Tan, N Ahuja(2009)
○ Proposed a new bilateral filtering algorithm with computational
complexity invariant to filter kernel size, so called O(1) or constant time
○ By showing that a bilateral filter can be decomposed into a number of
constant time spatial filters, the method yields a new class of constant
time bilateral filters that can have arbitrary spatial and arbitrary range
kernels

34. Further work
● Recursive bilateral filtering - Q.Yang (2012)
○ Computational complexity is linear in both input size and dimensionality
○ It is more efficient than the state-of-the-art bilateral filtering methods
that have a computational complexity of O(ND2)

35. Further work
● Flash shadows often appear at depth discontinuities between surfaces in the
scene. Using multiple flash photographs it may be possible to segment
foreground from background. Raskar et al. [2003] have explored this type of
approach to generate non-photorealistic renderings.
● May be possible to extend detail transfer technique to de-blur motion-blur
images. Jia et al. [2004]
● Joint optimization of our algorithms and the camera hardware design also a
possibility

36. ● Computational Photography Using a Pair of Flash/No-flash Images by
Iterative Guided Filtering [2011]
○ Iterative improvement of the guided image filter for F/NF photography
○ Proved to be effective for such applications as high dynamic range
compression, image matting, haze removal, and F/NF denoising etc.
Further work

37. ● Image Quilting for Texture Synthesis and Transfer [2001]
● Two-scale Tone Management for scale Tone Management for Photographic
Look[SIGGRAPH, 2006]
● Single Image Haze Removal Using Dark Channel Prior [2011]
Further work