Presentation of research paper "Interactive Photo Editing on Smartphones via Intrinsic Decomposition" presented at 42nd Annual Conference of the European Association for Computer Graphics (Eurographics’2021).

1. 1

2. 2

3. Recoloring Stylization
Retexturing Appearance Editing
Meka
et
al.,
SIGGRAPH
2016
Bonneel et al., Eurographics STAR, 2017 Beigpour et al., JPI, 2018 3

4. 4
Design Objectives Feasibility
• Accessibility – Possible on readily available
smartphones.
• Speed – All post-capture processing takes at most
few seconds.
• Interaction – Editing is possible at interactive rates.
• Quality – Rendered output looks plausible.
• Built-In Depth Sensors
• High-End GPU
• Graphics API

5. 5
Optimization-based Methods Learning-based Methods
-- Mostly are not real-time or interactive
-- Resource intensive
-- Limited to low image resolutions
-- Both class of methods assume only diffuse reflection
Bell
et
al.,
SIGGRAPH
2014
Input Albedo
Lettry
et
al.,
Pacific
Graphics
2018
Albedo
Input

6. 6

7. Diffuse + Depth Albedo
Shading
Output
Depth
RGB-D Input
Specularity
Removal
Intrinsic
Decomposition
Image
Editing
Specular
7

8. 8
Works quite well for images in a controlled setting.
Input Fu et al., 2019 Akashi et al., 2016 Shen et al., 2009
Ground Truth

9. 9
However, does not generalize well for real-world images.
Input Fu et al., 2019 Akashi et al., 2016 Shen et al., 2009

10. Intensity (In) Intensity (Out)
Chroma (In) Chroma (Out)
Input
Diffuse
Reducing positive coefficients of
subbands in the masked region
Specular
Mask
Inpainting missing information
by iterative bilateral hole-filling
High
Freq.
Low
Freq.
Sub-bands
10

11. 11
Diffuse + Depth Albedo
Shading
Output
Depth
RGB-D Input
Specularity
Removal
Intrinsic
Decomposition
Image
Editing
Specular
11

12. 𝐸 𝑎, 𝑠 =
1
2
𝜆𝑑𝐸𝑑 + 𝜆𝑟𝑎𝐸𝑟𝑎 + 𝜆𝑟𝑠𝐸𝑟𝑠 + 𝜆𝑠𝑝| 𝑎 |1
𝑎𝑟𝑔𝑚𝑖𝑛
𝑎, 𝑠 Data term Retinex-Albedo
term
Retinex-Shading
term
𝐿1- Regularization
of Albedo
Intrinsic Decomposition
12

13. Intrinsic Decomposition
Diffuse + Depth Shading
Albedo
13

14. 14

15. 15
Input Ours Fu et al., 2019 Akashi et al., 2016 Shen et al., 2009

16. 16
Input Only Low Freq. Reduced Only High Freq. Reduced Both Freq. Reduced Both Freq. Red. + Chroma Inpaint.

17. 17
Input Only Low Freq. Spec. Removal Only High Freq. Spec. Removal Specularity Enhanced

18. Input
18
Ours (RGB-D) Bell et al., 2014 (RGB) Jeon et al., 2014 (RGB-D) Lettry et al., 2018 (RGB)
Our results are comparable to existing techniques.

19. 19
Input Ours (RGB-D) Bell et al., 2014 (RGB) Jeon et al., 2014 (RGB-D) Lettry et al., 2018 (RGB)
Existing methods do not perform comparatively well for outside scenes

20. 20
𝐸 𝑎, 𝑠 =
1
2
𝜆𝑟𝑎𝐸𝑟𝑎 + 𝜆𝑑𝐸𝑑 + 𝜆𝑟𝑠𝐸𝑟𝑠 + 𝜆𝑠𝑝| 𝑎 |1
𝑎𝑟𝑔𝑚𝑖𝑛
𝑎, 𝑠 Data term
Retinex-Albedo
term
Retinex-Shading
term
𝐿1- Regularization
of Albedo
𝐸1 𝐸2 𝐸3
Input Albedo – Only 𝐸1 Albedo – 𝐸1+ 𝐸2 Albedo – 𝐸1+ 𝐸2 + 𝐸3

21. 21

22. Input Diffuse Specular
Albedo Shading
Diffuse Input
Silk
Mystique
Glass
God Rays
Toon
22

23. Mystique/Hulk
Input Glass
Silk Tattoo
23

24. Input Low-density Fog High-density Fog
24

25. Input God Rays God Rays + Fog
25

26. Input Comic Quickshift Halftone
26

27. 27
Input
Ground Truth Ours
Albedo
Edited
Albedo initialization with chromaticity leads to
color shifts in the edited results.
Input Diffuse
Our specular pixel identification step is not robust, thus can
potentially make the diffuse output darker.

28. 28
• Relax some of the existing assumptions such as -- single-color illuminant and only direct illumination.
• Perform super-resolution of depth map thereby enhancing the result quality.
• Make the specular mask generation more robust.
• A novel specularity removal method, that works for real-world images.
• A system for intrinsic decomposition of images on smartphones.
• Our GPU-based implementation allows for interactive image editing.
• We showcase a variety of applications.
Future Work