The Lighting Reconstruction Toolkit, an Image-Based Lighting approach used on The Matrix Reloaded and The Matrix Revolutions. Presented at Siggraph 2003

2. Lighting Reconstruction
for
“The Matrix” sequels
Haarm-Pieter Duiker
ESC Entertainment

3. Reconstructing Lighting Environments
• Achieve photo-real integration of visual
effects with live-action plates
• Directly recreate on-set lighting
environments
• Create a production-ready system
Motivation

4. • Chrome Ball Shooting Methodology
– Six Exposures per Chrome ball
– Three Exposures per Grey Ball
• Color Calibration Shooting Methodology
– Capture varying film response curves
Shooting Methodology

5. Elements

6. Elements

7. On-Set Exposures

8. Elements Screen Shots

9. • Recovery of Film Response Curve
• Linearization of Images
• Creation of High dynamic range imagery
Image Processing

10. Image Processing

11. Image Processing

12. Image Processing

13. Image Processing

14. • Light Extraction
– Differentiation between Key and Fill Lights
• Color Balancing
– Comparing Reference Object to Diffuse Grey Ball
Extracting Directional
Lights

15. Tools

16. Tools

17. Results Using a Single
Panorama

18. Results Using a Single
Panorama

19. Results Using a Single
Panorama

20. Results Using a Single
Panorama

21. • Static lighting throughout scene
– Good for daylight or night time scenes
– Bad for indoor scenes / anything with local lighting
effects
– Measurement of Key and Fill lighting from only one
direction.
Limitations of a Single
Panorama

22. • Multiple samples of single environment
• Photogrammetry reference
• Mechanism for integrating data
– Key Lights (High frequency)
– Fill Lights (Low frequency)
Reconstruct local
lighting environments

23. Source Data

24. Source Data

25. • Triangulation and photogrammetry
– Establish light location, direction, and size
• Fit Spot Light Model Parameters
– Panoramas as point samples of light
reflectance function
• Color
• Intensity
• Directional falloff
• Distance falloff
Fitting Key Lights

26. Interpolating Panoramas
• Scattered-Data Interpolation problem
– Delaunay triangulation of locations on-set
– Scaling perimeter of weight functions by distance
to neighbors
• Novel per-object lighting environment
– Dependent on object location and relative
panorama locations
Reconstructing Fill
Lighting

27. Results

28. Results

29. • Integration of geometry and lighting
• Characterization of varying reflectance across
surfaces
Subdividable Area
Lights

30. Stills

31. Stills

32. Stills

33. Maya Visualization

34. Stills

35. Color-Space and Compositing
• Compositing pipeline
– Linearization of all elements
– Automatic match of color and contrast
– Reliability for compositors and lighters
Side Benefits

36. Kim Libreri, George Borshukov, Christina
Tempelaar-Lietz, Andy Lomas, Oliver James,
Dan Piponi, Oystein Larsen, Ben Gunsberger,
J.P. Lewis
Credits / Thanks