Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Decomposing Images into Layers with Advanced Color Blending [Pacific Graphics 2018]

290 views

Published on

https://koyama.xyz/project/color_unblending/

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Decomposing Images into Layers with Advanced Color Blending [Pacific Graphics 2018]

  1. 1. Decomposing Images into Layers with Advanced Color Blending Yuki Koyama & Masataka Goto
  2. 2. Quick Summary
  3. 3. Usage Scenario Import the resulting layers to Photoshop etc. Perform non-trivial edits (e.g., lighting-aware hue change) Input images except for the rightmost one courtesy of David Revoy
  4. 4. Background What is Advanced Color Blending?
  5. 5. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 7 Color Blending & Blend Mode A color-blend mode defines the mapping from the colors of the layers to the color that will be rendered Examples: multiply, overlay, color-dodge, normal, etc. …Available in
  6. 6. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 8 Digital Drawing & Effects with Color Blending David Revoy: “Painting with Blending-modes” https://youtu.be/AybFWViT-3Q Advanced (non-linear) color blending is used for creating interesting color effects Example:
 digital drawing using 
 the color-dodge mode
  7. 7. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 9 Digital Drawing & Effects with Color Blending It is important to combine various blend modes
  8. 8. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 10 Digital Drawing & Effects with Color Blending (N/A) normal multiply hard-light screen It is important to combine various blend modes normal multiply hard-light screen
  9. 9. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 11 Digital Drawing & Effects with Color Blending (N/A) normal multiply hard-light screen It is important to combine various blend modes normal multiply hard-light screen Note: every digital artist has his/her own way
 to combine blend modes
  10. 10. Our Motivation
  11. 11. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 13 Motivation: Color Unblending (N/A) normal multiply hard-light screennormal multiply hard-light screen It should be useful if an existing image can be decomposed into layers with an arbitrary combination of advanced color-blend modes Unblending Arbitrary combination of blend modes
  12. 12. Previous Work Layer Decomposition by Color Unblending
  13. 13. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 15 RGB-Space Geometry [Tan+, TOG 16] • Support only a specific linear blend mode (i.e., “normal”) • Heavily rely on the linearity of the blend mode • Cannot be (easily) extended for non-linear color-blend modes
  14. 14. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 16 Per-Pixel Optimization [Aksoy+, TOG 16; TOG 17] • Support only a specific linear blend mode (and a linear alpha addition)
  15. 15. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 17 Per-Pixel Optimization [Aksoy+, TOG 16; TOG 17] • Support only a specific linear blend mode (and a linear alpha addition) We generalize this method to support 
 advanced (non-linear) color-blend modes
  16. 16. Details of Our Method [1/2] Per-Pixel Unblending Optimization (Concept-Level Explanation)
  17. 17. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 19 Problem Setting: Per-Pixel Unblending Target image
  18. 18. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 20 Problem Setting: Per-Pixel Unblending Target image The pixel that we are going to handle now Target pixel color ⇥ r, g, b, ↵ ⇤
  19. 19. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 21 Problem Setting: Per-Pixel Unblending Target pixel color ⇥ r, g, b, ↵ ⇤
  20. 20. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 22 Problem Setting: Per-Pixel Unblending Target pixel color multiply screen normal … ⇥ r, g, b, ↵ ⇤ Un- blending
  21. 21. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 23 Problem Setting: Per-Pixel Unblending Target pixel color multiply screen normal … ⇥ r, g, b, ↵ ⇤ Un- blending Desirable color distribution: the distribution of RGB values that the decomposed layer contains Blend mode
  22. 22. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 24 Problem Setting: Per-Pixel Unblending Target pixel color multiply screen normal … ⇥ r, g, b, ↵ ⇤ Un- blending Input
  23. 23. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 25 Problem Setting: Per-Pixel Unblending Target pixel color multiply screen normal … ⇥ r, g, b, ↵ ⇤ ⇥ r1, g1, b1, ↵1 ⇤ ⇥ r2, g2, b2, ↵2 ⇤ ⇥ rn, gn, bn, ↵n ⇤ … Un- blending Input Output
  24. 24. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 26 Problem Setting: Per-Pixel Unblending Target pixel color multiply screen normal … ⇥ r, g, b, ↵ ⇤ ⇥ r1, g1, b1, ↵1 ⇤ ⇥ r2, g2, b2, ↵2 ⇤ ⇥ rn, gn, bn, ↵n ⇤ … Un- blending Input Output Composited pixel color ⇥ ˆr, ˆg,ˆb, ˆ↵ ⇤Blending
  25. 25. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 27 Problem Setting: Per-Pixel Unblending Composited pixel color Target pixel color multiply screen normal … ⇥ r, g, b, ↵ ⇤ ⇥ r1, g1, b1, ↵1 ⇤ ⇥ r2, g2, b2, ↵2 ⇤ ⇥ rn, gn, bn, ↵n ⇤ ⇥ ˆr, ˆg,ˆb, ˆ↵ ⇤ … Un- blending Blending Input Output Should equal
  26. 26. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 28 Problem Setting: Per-Pixel Unblending Composited pixel color Target pixel color multiply screen normal … ⇥ r, g, b, ↵ ⇤ ⇥ r1, g1, b1, ↵1 ⇤ ⇥ r2, g2, b2, ↵2 ⇤ ⇥ rn, gn, bn, ↵n ⇤ ⇥ ˆr, ˆg,ˆb, ˆ↵ ⇤ … Un- blending Blending Input Output Should equal Should be as consistent as possible
  27. 27. Example User Interaction
  28. 28. Input image courtesy of David Revoy
  29. 29. Basics (Not Our Method) Math of Advanced Color Blending
  30. 30. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 32 Descriptor of a Pixel Color Descriptor of the i-th layer’s pixel xi = ⇥ ri gi bi ↵i ⇤T = ⇥ cT i ↵i ⇤T 2 R4 Target pixel Layer Layer Layer
  31. 31. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 33 Descriptor of a Pixel Color Descriptor of the i-th layer’s pixel x = ⇥ xT 1 · · · xT n ⇤T 2 R4n Descriptor of all the layers’ pixels xi = ⇥ ri gi bi ↵i ⇤T = ⇥ cT i ↵i ⇤T 2 R4 Target pixel Layer Layer Layer
  32. 32. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 34 General Composition Operator ( ) for Compositing Two Layers ?xs xd General composition operator
  33. 33. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 35 xs rgb xd = f(cs, cd)↵s↵d + Y ↵s(1 ↵d)cs + Z↵d(1 ↵s)cd xs ↵ xd xs ↵ xd = X↵s↵d + Y ↵s(1 ↵d) + Z↵d(1 ↵s) General Composition Operator ( ) for Compositing Two Layers “SVG Compositing Specification” by W3C: https://www.w3.org/TR/2011/WD-SVGCompositing-20110315/ ?xs xd
  34. 34. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 36 f : [0, 1]3 ⇥ [0, 1]3 ! [0, 1]3 Blend function: Mode-specific function (can be non-linear) “SVG Compositing Specification” by W3C: https://www.w3.org/TR/2011/WD-SVGCompositing-20110315/ xs rgb xd = f(cs, cd)↵s↵d + Y ↵s(1 ↵d)cs + Z↵d(1 ↵s)cd xs ↵ xd xs ↵ xd = X↵s↵d + Y ↵s(1 ↵d) + Z↵d(1 ↵s) For example: General Composition Operator ( ) for Compositing Two Layers fnormal, fmultiply, fcolor-dodge, . . .
  35. 35. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 37 More-Than-Two-Layers Case: Recursive Layering multiply Final RGBA ˆxk = ⇢ x1 (k = 1) xk k ˆxk 1 (otherwise) color-dodge overlay x1 x2 x3 x4 ˆx4 ˆx3 ˆx2 Recursive rule:
  36. 36. Details of Our Method [2/2] Per-Pixel Unblending Optimization (Equation-Level Explanation)
  37. 37. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 39 Unblending Optimization Color reproducibility: The target pixel color should be reproduced by compositing the resulting layer colors Color consistency: Each pixel color should be as consistent to the specified color distribution as possible … Unblending Blending More consistent Less consistent Equal
  38. 38. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 40 Unblending Optimization Equality constraint: The target pixel color should be reproduced by compositing the resulting layer colors Objective: Each pixel color should be as consistent to the specified color distribution as possible Equality-constrained optimization problem min x2R4n E(x) s.t. C(x) = 0
  39. 39. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 41 Unblending Optimization Equality constraint: The target pixel color should be reproduced by compositing the resulting layer colors Objective: Each pixel color should be as consistent to the specified color distribution as possible Equality-constrained optimization problem min x2R4n E(x) s.t. C(x) = 0
  40. 40. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 42 Constraint Function Definition C(x) = (xn n (xn 1 n 1 (xn 2 n 2 (. . .)))) ctarget = 0 multiply color-dodge overlay …E.g.,
  41. 41. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 43 Constraint Function Definition C.f., [Aksoy+16; 17]: Assumed a special case of a linear model C(x) = nX i ↵ici ctarget = 0 C(x) = (xn n (xn 1 n 1 (xn 2 n 2 (. . .)))) ctarget = 0 multiply color-dodge overlay …E.g.,
  42. 42. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 44 Constraint Function Definition C.f., [Aksoy+16; 17]: Assumed a special case of a linear model C(x) = nX i ↵ici ctarget = 0 C(x) = (xn n (xn 1 n 1 (xn 2 n 2 (. . .)))) ctarget = 0 multiply color-dodge overlay … Our method is a generalization of [Aksoy+16; 17] E.g.,
  43. 43. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 45 Solving Equality-Constrained Optimization [1/2] Algorithm: The augmented Lagrangian method • Iteratively solve (unconstrained) optimizations while (not converged) { solve min x n E(x) T C(x) + ⇢ 2 kC(x)k2 o update and ⇢ }
  44. 44. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 46 Solving Equality-Constrained Optimization [1/2] Algorithm: The augmented Lagrangian method • Iteratively solve (unconstrained) optimizations Need the derivative of C(x) to efficiently solve this optimization while (not converged) { solve min x n E(x) T C(x) + ⇢ 2 kC(x)k2 o update and ⇢ }
  45. 45. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 47 Solving Equality-Constrained Optimization [2/2] @C(x) @xi = 8 >< >: I (i = k = 1) @ @xk (xk k ˆxk 1) (i = k 6= 1) @ˆxk 1 @xi · @ @ˆxk 1 (xk k ˆxk 1), (otherwise) It is possible to algorithmically calculate 
 the derivative of C(x) by recursively applying the chain rule
  46. 46. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 48 Solving Equality-Constrained Optimization [2/2] @C(x) @xi = 8 >< >: I (i = k = 1) @ @xk (xk k ˆxk 1) (i = k 6= 1) @ˆxk 1 @xi · @ @ˆxk 1 (xk k ˆxk 1), (otherwise) It is possible to algorithmically calculate 
 the derivative of C(x) by recursively applying the chain rule Our unblending optimization problem can be efficiently solved by gradient-based algorithms (e.g., L-BFGS)
  47. 47. Unblending using linear blend mode ➡ Identical to the results by [Aksoy+17] Unblending using various blend modes color-dodge multiply overlay Input image courtesy of David Revoy
  48. 48. Post-Processing Refinement for Smoothness
  49. 49. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 51 Refinement Step for Smoothness ■ Problem: Resulting layers may not be smooth because the unblending optimization is “per-pixel” (no inter-pixel consideration) ■ Solution: Performing a procedure called “matte refinement” [Aksoy+17] as a post processing Not smooth Smooth Input image courtesy of David Revoy
  50. 50. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 52 Refinement Step for Smoothness ■ Problem: Resulting layers may not be smooth because the unblending optimization is “per-pixel” (no inter-pixel consideration) ■ Solution: Performing a procedure called “matte refinement” [Aksoy+17] as a post processing to enforce smoothness Not smooth Smooth Input image courtesy of David Revoy
  51. 51. Usage Scenario Example: Photoshop
  52. 52. Input image courtesy of David Revoy
  53. 53. Applications #1 Lighting-Aware Editing of 
 an Existing Illustration
  54. 54. Input image courtesy of David Revoy
  55. 55. Applications #2 Bringing a Physical Painting
 into Digital Workflow
  56. 56. Other Applications
  57. 57. (N/A) multiply (N/A) multiply screen Intrinsic image decomposition: decomposing an image into reflectance and shading layers Adding gradation to an existing conference logo: extracting a “fill” layer from the logo and re-fill with gradation Input images are provided by [Innamorati+17]
  58. 58. Remixing existing illustrations: Harmonize colors of images from different scenes to put them together in a derivative work Input image courtesy of David Revoy
  59. 59. Summary
  60. 60. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 63 Summary ■ Solution for a new problem • Decomposing an image into layers with advanced (non- linear) color blending ■ Techniques • Generalizing Aksoy+’s method [16; 17] for handling advanced (non- linear) color blend modes ■ Future work • Automatic determination of optimal color distributions etc.
  61. 61. Decomposing Images into Layers with Advanced Color Blending Yuki Koyama & Masataka Goto We are preparing for releasing our source codes at GitHub so please check our project page later In preparation…
  62. 62. Decomposing Images into Layers with Advanced Color Blending Yuki Koyama & Masataka Goto
  63. 63. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 66 Two Main Usages of Layers For segmenting regions (not our focus) For blending colors (our focus) { {
  64. 64. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 67 Specialization of the Composite Operator Alpha blending model C.f., [Tan+16] Linear additive model C.f., [Aksoy+16; 18] xs rgb xd = ↵scs + (1 ↵s)↵dcd ↵s ↵ ↵d , ↵s ↵ ↵d = ↵s + (1 ↵s)↵d xs rgb xd = ↵scs + ↵dcd ↵s ↵ ↵d , ↵s ↵ ↵d = ↵s + ↵d f(cs, cd) = cs, X = Y = Z = 1 f(cs, cd) = cs + cd, X = 2, Y = Z = 1
  65. 65. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 68 Computational Cost 0 s 350 s 700 s 1,050 s 1,400 s 1 12 72 Number of CPU cores 0 s 125 s 250 s 375 s 500 s 0 k 500 k 1,000 k Number of pixels 0 s 40 s 80 s 120 s 160 s 2 3 4 5 6 Number of layers • Parallelization is effective • Increase cost along with the number of pixels • Increase cost along with the number of layers (due to the recursive calculation) 0 s 350 s 700 s 1,050 s 1,400 s 1 12 72 Number of CPU cores 0 s 125 s 250 s 375 s 500 s 0 k 500 k 1,000 k Number of pixels 0 s 40 s 80 s 120 s 160 s 2 3 4 5 6 Number of layers 0 s 350 s 700 s 1,050 s 1,400 s 1 12 72 Number of CPU cores 0 s 125 s 250 s 375 s 500 s 0 k 500 k 1,000 k Number of pixels 0 s 40 s 80 s 120 s 160 s 2 3 4 5 6 Number of layers
  66. 66. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 69 Fixing Deep Colorization via Layer Decomposition (N/A) overlay overlay (N/A) multiply Sketch & scribbles Deep-colorized image Composited image Final image with effects Our method enables a new digital painting workflow by effectively using such a deep-colorized image as a good starting point. Instead of directly editing the generated single-layered image, the user can decompose it into layers with familiar advanced color blending then individually edit each unsatisfactory layer by adding strokes, changing hue, etc.
  67. 67. Yuki Koyama and Masataka Goto | Decomposing Images into Layers with Advanced Color Blending | Comput. Graph. Forum (Pacific Graphics 2018) 70 Lighting-Aware Replacement of Background Hue change Original image Manual segmentation Re-composition after layer editing Composition with a different background (No edit)(No edit) normal normal overlay Decomposed layers and edits on themNaïve composition with a different background Example use of our method, in which the blue sky background is replaced with a sunset one. With our method, the character region is decomposed into three layers: the bottom two layers roughly correspond to albedo, and the top layer roughly corresponds to lighting by the blue sky represented as an overlay layer. To match the lighting in sunset, the top layer’s hue is modified. Finally, the layers are re-composited with the sunset background. Note that naïvely compositing the character region with the sunset background (the leftmost image) does not look good because lighting is mismatched. Input image courtesy of David Revoy.

×