Your SlideShare is downloading. ×
Lightspeed SIGGRAPH talk
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Lightspeed SIGGRAPH talk

5,119
views

Published on

Talk slides from the SIGGRAPH 2007 presentation of the paper "The Lightspeed Automatic Interactive Lighting Preview System" by Ragan-Kelley, et al.

Talk slides from the SIGGRAPH 2007 presentation of the paper "The Lightspeed Automatic Interactive Lighting Preview System" by Ragan-Kelley, et al.

Published in: Technology, Business

0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
5,119
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
54
Comments
0
Likes
5
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. The Lightspeed Automatic Interactive Lighting Preview System Jonathan Ragan-Kelley / MIT CSAIL Charlie Kilpatrick / ILM Brian Smith / ILM Doug Epps / Tippett Studio Paul Green / MIT CSAIL Christophe Héry / ILM Frédo Durand / MIT CSAIL
  • 2. Lighting Design ■ End of pipeline fixed geometry, viewpoint, material ■ Slow feedback 10-60 mins to render
  • 3. Lighting Design another hour later... 1 ■ End of pipeline fixed geometry, viewpoint, material ■ Slow feedback 10-60 mins to render
  • 4. Goal: Fast Lighting Preview Exploit redundancy between previews ■ geometry ■ view ■ material
  • 5. High-Level Approach ■ Precompute deep-framebuffer cache e.g. ... normal position diffuse specular texture texture ■ Preview dynamically on GPU as the user specifies new light parameters
  • 6. Prior Work
  • 7. Render Caching Parameterized Ray Tracing [Séquin & Smyrl 1989] G-Buffer [Saito & Takahashi 1990] Fast Relighting Engine [Gershbein & Hanrahan 2000] ■ Precomputed buffer (normals, texture) with BRDF*light reevaluated at runtime ■ Shortcomings ■ Simplistic shading ■ No antialiasing, motion blur, transparency
  • 8. Pixar’s Lpics [Pellacini et al. 2005] ■ Production shaders & lights (RenderMan) ■ Exploit programmable GPUs ■ No antialiasing, transparency ■ Requires extra shader-writing work ■ (final) RenderMan version + extra caching code ■ GPU preview version
  • 9. Specializing Shaders [Guenter et al. 1995] ■ Given dynamic parameters, automatically split shader code into: ■ static ■ dynamic ■ Simple language no control flow ■ Simple shaders Plate 1: Test image. Plate 1: Test image. Plate 2: Image obtained by reshading image of Plate 1. This Plate 2: Image obtained by reshading image of Plate 1. This took 12 seconds unspecialized, 0.5 seconds specialized. took 12 seconds unspecialized, 0.5 seconds specialized. Bibliography Bibliography [1] Andersen, Lars Ole. Self-applicable C Program [1] Andersen, Lars Ole. Self-applicable C Program [9] Jones, Neil D., Carsten K. Gomard, and Peter Sestoft. [9] Jones, Neil D., Carsten K. Gomard, and Peter Sestoft. Specialization. Proceedings of the ACM SIGPLAN Specialization. Proceedings of the ACM SIGPLAN Partial Evaluation and Automatic Program Generation. Partial Evaluation and Automatic Program Generation. Workshop on Partial Evaluation and Semantics-Based Workshop on Partial Evaluation and Semantics-Based Prentice-Hall, 1993. Prentice-Hall, 1993. Program Manipulation (San Francisco, California, June 12- Program Manipulation (San Francisco, California, June 12- [10] Mogensen, Torben. The Application of Partial Evaluation to [10] Mogensen, Torben. The Application of Partial Evaluation to 20, 1992). Yale University technical report 20, 1992). Yale University technical report YALEU/DCS/RR-909, 1992, 54-61. Ray-Tracing. Master's thesis, DIKU, University of Ray-Tracing. Master's thesis, DIKU, University of YALEU/DCS/RR-909, 1992, 54-61. Copenhagen, Denmark, 1986. Copenhagen, Denmark, 1986.
  • 10. Precomputed Radiance Transfer e Transfer for Real-Time Rendering requency Lighting Environments e.g. Sloan et al. 2002, Ng et al. 2003 Jan Kautz John Snyder x-Planck-Institut für Informatik Microsoft Research jnkautz@mpi-sb.mpg.de johnsny@microsoft.com use and hat cap- process, over the equency s global ect onto o actual ampling e rigidly ting and Figure 1: Precomputed, unshadowed irradiance from [34] (left) vs. our pherical precomputed transfer (right). The right model can be rendered at 129Hz Inappropriate for our application ntly on with self-shadowing and self-interreflection in any lighting environment. o a dot Glossy frequency lighting environments, using a low-order spherical ■ Large precomputation cost e further harmonic (SH) basis to represent such environments efficiently lighting without aliasing. The main idea is to represent how an object g points scatters this light onto itself or its neighboring space. ■ Largely ignore light functions m rigidly To describe our technique, assume initially we have a convex, ers. We diffuse object lit by an infinitely distant environment map. The roach. ■ Limitations on shading models object’s shaded “response” to its environment can be viewed as a chniques, transfer function, mapping incoming to outgoing radiance, which in this case simply performs a cosine-weighted integral. A more complex integral captures how a concave object shadows itself, where the integrand is multiplied by an additional transport factor tions are representing visibility along each direction. tunately, Our approach is to precompute for a given object the expensive environ- transport simulation required by complex transfer functions like 25], rad- shadowing. The resulting transfer functions are represented as a ple point dense set of vectors or matrices over its surface. Meanwhile, ndering. incident radiance need not be precomputed. The graphics hard- difficul- ware can dynamically sample incident radiance at a number of s of real points. Analytic models, such as skylight models [33] or simple
  • 11. Our Design Goals 1. High-performance preview ■ Low-latency feedback ■ Fast initial precomputation 2. Seamless integration ■ Same input: RenderMan scene & shaders ■ Same output: high quality image 3. Ease of implementation & maintenance
  • 12. System Overview
  • 13. System Overview RenderMan scene & shaders
  • 14. System Overview RenderMan cache & scene & preview shaders shaders automatic caching
  • 15. System Overview RenderMan cache & preview preview image scene & shaders shaders relighting automatic engine caching (gpu)
  • 16. System Overview RenderMan cache & preview preview image scene & shaders shaders relighting automatic engine caching (gpu) lighting parameters one-time precomputation iterative preview
  • 17. System Overview RenderMan cache & preview preview image scene & shaders shaders relighting automatic engine caching (gpu) lighting parameters Automatic for real production scenes ‣ specialization & translation ‣ cache compression
  • 18. System Overview RenderMan cache & preview preview image scene & shaders shaders relighting automatic engine caching (gpu) lighting parameters Automatic High fidelity for real production scenes antialiasing, motion blur, ‣ specialization & translation transparency ‣ cache compression ‣ indirect framebuffer
  • 19. Automatic Caching RenderMan cache & preview preview image scene & shaders shaders relighting automatic engine caching (gpu) lighting parameters
  • 20. Program Analysis
  • 21. Program Analysis mySurface(color
c)
{ albedo
=
c*Texture(U,V); shade
=
Light(L)*BRDF(N,L); return
albedo*shade; }
  • 22. Program Analysis mySurface(color
c)
{ dynamic albedo
=
c*Texture(U,V); shade
=
Light(L)*BRDF(N,L); return
albedo*shade; } what part of the code depends on the lighting parameter L?
  • 23. Program Analysis mySurface(color
c)
{ dynamic albedo
=
c*Texture(U,V); shade
=
Light(L)*BRDF(N,L); static return
albedo*shade; } everything else is static
  • 24. Program Analysis mySurface(color
c)
{ dynamic albedo
=
c*Texture(U,V); shade
=
Light(L)*BRDF(N,L); static return
albedo*shade; cache } values at the boundary are cached for reevaluation
  • 25. Program Analysis mySurface(color
c)
{ albedo
=
c*Texture(U,V); shade
=
Light(L)*BRDF(N,L); return
albedo*shade; }
  • 26. Program Analysis albedo shade mySurface(color
c)
{ albedo
=
c*Texture(U,V); shade
=
Light(L)*BRDF(N,L); return
albedo*shade; } c Texture Light BRDF U V L N
  • 27. Deep Framebuffer Generation mySurface(color
c)
{ albedo
=
c*Texture(U,V); shade
=
Light(L)*BRDF(N,L); return
albedo*shade; }
  • 28. Deep Framebuffer Generation caching shader mySurfaceCache(color
c)
{ albedo
=
c*Texture(U,V); cache(N); cache(albedo); }
  • 29. Deep Framebuffer Generation N albedo final deep renderer framebuffer cache caching shader mySurfaceCache(color
c)
{ albedo
=
c*Texture(U,V); cache(N); cache(albedo); }
  • 30. Deep Framebuffer Generation N albedo deep framebuffer cache caching shader
  • 31. Deep Framebuffer Generation N albedo deep framebuffer cache caching shader mySurfaceDynamic(vector
L)
{ N
=
lookup(); albedo
=
lookup(); shade
=
Light(L)*BRDF(N,L); preview return
albedo*shade; shader }
  • 32. Deep Framebuffer Generation N albedo deep framebuffer cache caching shader mySurfaceDynamic(vector
L)
{ N
=
lookup(); albedo
=
lookup(); shade
=
Light(L)*BRDF(N,L); preview return
albedo*shade; shader }
  • 33. Deep Framebuffer Generation N albedo deep framebuffer cache caching shader rendered shade image preview shader
  • 34. Automatic Caching
  • 35. Automatic Caching RenderMan scene scene preprocessor RenderMan shaders
  • 36. Automatic Caching RenderMan scene scene preprocessor program analysis RenderMan shaders
  • 37. Automatic Caching RenderMan scene scene preprocessor caching shaders program analysis RenderMan shaders
  • 38. Automatic Caching deep RenderMan framebuffer scene scene preprocessor cache x2 x1 final N P renderer caching shaders program analysis RenderMan shaders
  • 39. Automatic Caching deep RenderMan framebuffer scene scene preprocessor cache x2 x1 final N P renderer caching shaders program analysis RenderMan dynamic shaders preview shaders
  • 40. Automatic Caching challenge: cache size deep RenderMan framebuffer scene scene preprocessor cache x2 program analysis is final x1 N P conservative renderer caching shaders generates many cached terms program analysis (hundreds) RenderMan dynamic shaders preview shaders large caches (gigabytes)
  • 41. Cache Compression deep RenderMan framebuffer scene scene preprocessor cache x2 x1 final N P renderer caching shaders Remove redundant program analysis values after caching RenderMan dynamic shaders preview shaders
  • 42. Cache Compression deep RenderMan framebuffer scene scene preprocessor cache x2 x1 N final N cache P P renderer compression caching shaders Remove redundant program analysis values after caching RenderMan dynamic shaders preview shaders
  • 43. Cache Compression deep RenderMan framebuffer scene scene preprocessor cache x2 x1 N final N cache P P renderer compression caching shaders Remove redundant program analysis values after caching RenderMan shaders dynamic preview → 5x compression shaders
  • 44. System Overview deep RenderMan framebuffer scene scene preprocessor cache N P final cache renderer compr. comp. caching shaders program analysis RenderMan dynamic shaders preview shaders
  • 45. System Overview deep RenderMan framebuffer scene scene preprocessor cache N P final cache renderer comp. caching shaders program analysis RenderMan dynamic shaders preview shaders
  • 46. System Overview deep RenderMan framebuffer scene scene preprocessor cache N P final cache renderer comp. relighting caching shaders engine (gpu) program analysis preview image RenderMan dynamic shaders preview shaders lighting parameters
  • 47. System Overview antialiasing? RenderMan scene scene preprocessor deep framebuffer cache transparency? N P final cache renderer comp. relighting caching shaders engine (gpu) program analysis preview image RenderMan dynamic shaders preview shaders lighting parameters
  • 48. Classical Deep Framebuffer B A
  • 49. Classical Deep Framebuffer B A
  • 50. Classical Deep Framebuffer x2 deep x1 framebuffer N cache B P cache shade A Cache one sample per-pixel rendered Shade cache values to render image image
  • 51. Classical Deep Framebuffer x2 deep x1 framebuffer N cache B P cache shade A How can we do antialiasing? rendered image
  • 52. Classical Deep Framebuffer supersample image x2 x1 N B P cache shade A How can we do antialiasing? transparency?
  • 53. Classical Deep Framebuffer x4 supersample x3 cache N B P cache shade A supersample shading How can we do antialiasing? transparency?
  • 54. Classical Deep Framebuffer x4 x3 N B P ...? cache shade A How can we do antialiasing? ...? transparency?
  • 55. Our Indirect Framebuffer ■ Decouple shading from visibility ■ only supersample visibility ‣ Do what RenderMan does ■ Compress samples based on static visibility
  • 56. RenderMan / REYES B show *filtering* pixel color? A
  • 57. RenderMan / REYES B b1 show *filtering* pixel color? b2 a1 A dice into micropolygons
  • 58. RenderMan / REYES B b1 show *filtering* pixel color? b2 a1 A shade micropolygons
  • 59. RenderMan / REYES B b1 show *filtering* pixel color? b2 a1 A rasterize micropolygons
  • 60. RenderMan / REYES B b1 show *filtering* pixel color? b2 a1 A
  • 61. RenderMan / REYES 1 2 B a1 b2 b1 show *filtering* 1 2 pixel color? b2 3 a 1 4 3 4 A a1 a1 pixel = a1+b2+a1+a1
  • 62. RenderMan / REYES 1 2 B a1 b2 b1 show *filtering* 1 2 pixel color? b2 3 a 1 4 3 4 A a1 a1 pixel = a1+b2+a1+a1 4
  • 63. RenderMan / REYES 1 2 B a1 b2 b1 show *filtering* 1 2 pixel color? b2 3 a 1 4 3 4 A a1 a1 pixel = a1+b2+a1+a1 = 0.75*a1+0.25*b2 4
  • 64. RenderMan / REYES 1 2 B a1 b2 b1 show *filtering* 1 2 pixel color? b2 3 a 1 4 3 4 A a1 a1 pixel = ?
  • 65. RenderMan / REYES B b1 b2 a1 A pixel = ?
  • 66. RenderMan / REYES 1 B a1 b1 α=0.3 α=1.0 b1 0.3*a1+0.7*b1 1 b2 a1 A pixel = ?
  • 67. RenderMan / REYES 1 2 B a1 b1 b2 α=0.3 α=1.0 α=1.0 b1 2 0.3*a1+0.7*b1 1.0*b2 1 b2 a1 A pixel = ?
  • 68. RenderMan / REYES 1 2 B a1 b1 b2 α=0.3 α=1.0 α=1.0 b1 2 0.3*a1+0.7*b1 1.0*b2 1 b2 3 a 1 4 3 4 A a1 a1 b2 α=0.3 α=0.3 α=1.0 0.3*a1 0.3*a1+0.7*b1 pixel = ?
  • 69. RenderMan / REYES 1 2 B a1 b1 b2 α=0.3 α=1.0 α=1.0 b1 2 0.3*a1+0.7*b1 1.0*b2 1 b2 3 a 1 4 3 4 A a1 a1 b2 α=0.3 α=0.3 α=1.0 0.3*a1 0.3*a1+0.7*b1 pixel = 0.225*a1+0.35*b1+0.25*b2
  • 70. Indirect Framebuffer
  • 71. Indirect Framebuffer b1 a1b2
  • 72. Indirect Framebuffer b1 a1b2 x4 x4 deep x1 x3 x3 buffer N x2 x2 P P P a1 b1 b2 no longer image space (per-micropolygon)
  • 73. Indirect Framebuffer b1 a1b2 x4 x4 deep x1 x3 x3 buffer N x2 x2 P P P a1 b1 b2 shading shade samples shaded like a conventional deep-framebuffer
  • 74. Indirect Framebuffer b1 a1b2 indirect framebuffer x4 x4 deep x1 x3 x3 buffer N x2 x2 P P P a1 b1 b2 shading shade samples
  • 75. Indirect Framebuffer b1 a1b2 indirect framebuffer x4 x4 deep x1 x3 x3 buffer 0.225 0.35 0.25 N x2 x2 P P P a1 b1 b2 shading shade samples densely stored in vertex array, on GPU
  • 76. Indirect Framebuffer b1 a1b2 indirect framebuffer x4 x4 deep x1 x3 x3 buffer 0.225 0.35 0.25 N x2 x2 P P P a1 b1 b2 blend shading final shade samples pixel 4 subpixel samples, 2 transparent layers only 3 unique micropolygons
  • 77. Indirect Framebuffer b1 a1b2 indirect framebuffer x4 x4 deep x1 x3 x3 buffer 0.225 0.35 0.25 N x2 x2 P P P a1 b1 b2 blend shading final shade samples pixel 4 subpixel samples, 2 transparent layers only 3 unique micropolygons all contributions linearized into a single weight
  • 78. More samples, b1 Same cost a1b2 indirect framebuffer x4 x4 deep x1 x3 x3 buffer 0.242 0.331 0.245 N x2 x2 P P P a1 b1 b2 blend shading final shade samples pixel
  • 79. More samples, b1 Same cost a1b2 indirect framebuffer x4 x4 deep x1 x3 x3 buffer 0.242 0.331 0.245 N x2 x2 P P P a1 b1 b2 blend shading final shade samples pixel
  • 80. More samples, b1 Same cost a1b2 indirect framebuffer x4 x4 deep x1 x3 x3 buffer 0.242 0.331 0.245 N x2 x2 P P P a1 b1 b2 blend shading final shade samples pixel
  • 81. Indirect Framebuffer Results antialiasing motion blur transparency identical to RenderMan
  • 82. Indirect Framebuffer Results antialiasing motion blur transparency identical to RenderMan RenderMan 45M micropolygons 60M subpixel samples brute-force >100M samples Indirect framebuffer 10M deep fb +15M indirect fb 720x389, 64x supersampling
  • 83. System Architecture
  • 84. System Architecture RenderMan scene scene preprocessor RenderMan shaders
  • 85. System Architecture RenderMan scene scene preprocessor program analysis RenderMan shaders
  • 86. System Architecture RenderMan scene scene preprocessor caching shaders program preview analysis shaders RenderMan shaders
  • 87. System Architecture RenderMan scene scene preprocessor final renderer caching shaders program preview analysis shaders RenderMan shaders
  • 88. System Architecture RenderMan scene scene preprocessor deep fb indirect fb final cache x2 x1 renderer compr. N P caching shaders program preview analysis shaders RenderMan shaders
  • 89. System Architecture RenderMan scene scene preprocessor relighting engine (GPU) deep fb indirect fb final cache x2 x1 renderer compr. N P caching shaders program preview analysis shaders RenderMan shaders
  • 90. System Architecture RenderMan scene scene preprocessor relighting engine (GPU) deep fb indirect fb final cache x2 x1 renderer compr. N P caching shaders shade program preview analysis shaders RenderMan shaders
  • 91. System Architecture RenderMan scene scene preprocessor relighting engine (GPU) deep fb indirect fb rendered image final cache x2 x1 renderer compr. N P caching shaders shade blend program preview analysis shaders RenderMan shaders
  • 92. System Architecture RenderMan scene scene preprocessor relighting engine (GPU) deep fb indirect fb rendered image final cache x2 x1 renderer compr. N P caching shaders shade blend program preview analysis shaders lighting parameters RenderMan shaders one-time precomputation iterative preview
  • 93. Additional Features ■ Shadows ■ Subsurface Scattering ■ Performance enhancement ■ Progressive Refinement ■ Light Caching … ■ Tiling
  • 94. Results initial feedback: 0.05 sec full refinement: 0.7 sec Lightspeed 720x389 offline render: 5 mins
  • 95. Results initial feedback: 0.05 sec full refinement: 0.7 sec RenderMan Lightspeed 720x389 offline render: 5 mins
  • 96. Results
  • 97. Results caching: 16 mins initial feedback: 0.11 sec full refinement: 2.7 sec 914x389 offline render: 59 mins 13x13 antialiasing 42 lights
  • 98. Results caching: 16 mins initial feedback: 0.11 sec full refinement: 2.7 sec Lightspeed RenderMan 914x389 offline render: 59 mins 13x13 antialiasing 42 lights
  • 99. Results caching: 16 mins initial feedback: 0.11 sec full refinement: 2.7 sec Lightspeed RenderMan 914x389 offline render: 59 mins 13x13 antialiasing 42 lights
  • 100. Results
  • 101. Results
  • 102. Results
  • 103. Results
  • 104. Limitations ■ GPU programming model ■ Dynamic calls to external C code ■ Complex data structures ■ GPU limits (bandwidth, memory, registers) ■ Fully-accurate dynamic ray tracing ■ Unbounded dynamic loops ■ Additional features not yet implemented ■ Indirect diffuse ■ Deep shadows ■ Non-linear lights
  • 105. Discussion ■ Automatic and manual specialization (Lpics) both have advantages ■ Manual specialization allows hand optimization ■ Compiler requires up-front R&D, never perfect (especially on the GPU) ■ Saves significant time in production ■ Some material parameters are editable ■ Cache compression is key to practical automatic specialization much simpler than fancy static analysis ■ Indirect Framebuffer is powerful, scalable
  • 106. Summary ■ Interactive lighting preview milliseconds instead of hours ■ Automatic caching for our production scenes ■ Program analysis ■ Cache compression ■ Indirect framebuffer ■ Efficient antialiasing, motion blur, transparency ■ Progressive refinement ■ In use on current productions
  • 107. Acknowledgments Inception: Pat Hanrahan, Ujval Kapasi Compilers: Alex Aiken, John Kodumal Tippett: Aaron Luk, Davey Wentworth ILM: Alan Trombla, Ed Hanway, Dan Goldman, Steve Sullivan, Paul Churchill Images: Michael Bay, Dan Piponi Money: NSF, NVIDIA, MSR, Sloan & Ford fellowships
  • 108. Summary ■ Interactive lighting preview milliseconds instead of hours ■ Automatic caching for our production scenes ■ Program analysis ■ Cache compression ■ Indirect framebuffer ■ Efficient antialiasing, motion blur, transparency ■ Progressive refinement ■ In use on current productions
  • 109. EXTRAS
  • 110. Cache Compression deep framebuffer cache Remove redundant values after caching ■ non-varying same for many pixels ■ non-unique same within a pixel unique varying → 4-5x compression
  • 111. Cache Compression shader dynamic varying unique (static analysis) (compressed) generic surface 402 145 97 metallic paint 450 150 97
  • 112. Visibility Compression Indirect RenderMan Framebuffer scene res. samples shade subpix shade indirect robot 914x389 13x13 2.1M 32M 633k 1.6M robot 720x306 13x13 1.5M 21M 467k 3.8M (blur) pirate 640x376 4x4 2.5M 2.3M 327k 716k hairs 720x389 8x8 43M 58M 11M 17M

×