Unity is enabling film and animation studios to revolutionize their pipelines with features developed specifically to empower storytellers who are creating linear and interactive content. Learn more about features such as Python, Shotgun, the Arbitrary Output Variables (AOV) used in Recorder for export, Alembic, and Universal Scene Description (USD). Speaker: Mathieu Muller - Unity Watch the session on YouTube: https://youtu.be/wrc3R-BoDGs

2. GenerativeArt—MadewithUnity
What’s ahead
for film and
animation with
Unity 2020
Mathieu Muller
Product Manager Film&TV, Unity
@the_f_key, @unity3d

3. 3
Edmond (Miam! + Arteacts Studio, 2019)

4. 4
3 years ago...

5. 5
Previz on Disney Jungle Book (Disney, MPC, Digital Domain, 2015)

6. 6
Ready Player One Virtual Production (Digital Domain, 2017)

7. 7
Sonder trailer (2016)

8. 8
Mr Carton (Tant Mieux Prod, 2017)

9. 9
Adam (2016)

10. In the
meantime...
10

11. From demo to toolset
11

12. 12
Adam Episode 2 (Oats, Neil Blomkamp, 2017)

13. 13
Adam Episode 3 (Oats, Neil Blomkamp, 2017)

14. Pushing real-time
graphics
14

15. 15
Book of the Dead (2018)

16. 16
Lexus demo (Unity, 2018)

17. From toolset to production
17

18. 18
Genesis Virtual production (MPC, 2018)

19. 19
Sonder (2018)
Watch
Watch here:
https://vimeo.com/253571683

20. 20
Sonder (Full movie – depuis hier!!!, 2018)
Watch
Watch here:
https://vimeo.com/253571683

21. 21
Sonder (2018)
Watch
Watch here:
https://vimeo.com/253571683
Watch
130+ people, in 7+ countries of the world, mostly
volunteers

22. 22
Timboo & Tuskar (VideoGyan, 2018)
https://www.youtube.com/watch?v=ImZBw2tA0JY&list=PLrIt3uECvxkpS93yaj85AFd4Vy4b0bRur&t=12s&index=7

23. 23
Crow The Legend (Baobab, 2018)

24. 24
Rainbow Crow, VR, Baobab (2018) Crow, The Legend, Baobab (2018) - Shot with VR assets in Unity
Crow The Legend (Baobab, 2018)

25. Scaling productions
25

26. 26
Lion King – Disney 2019 (Virtual production by Magnopus + MPC)
Video from: Inside The Lion King's VFX - BBC Click on Youtube

27. 27
Lion King – Disney 2019 (Virtual production by Magnopus + MPC)
Video from: Inside The Lion King's VFX - BBC Click on Youtube

28. 28
Big Hero 6 - Baymax Dreams - Too many Baymaxes (Disney TV + Unity, 2018)

29. Treasure Trekkers (Bwater Animation) – 52x11min TV series (2018-2020)

30. 30
Treasure Trekkers (Bwater) – 52x11min TV series (2018-2019)
China Transylvania Irland
Kenia Arizona Sahara desert
Ness lake Fantasy forest Under the ocean

31. 31
Treasure Trekkers (Bwater) – 52x11min TV series (2018-2019)

32. 32
Treasure Trekkers (Bwater) – 52x11min TV series (2018-2019)

33. Democratizing animation
33

34. 34
Little Mountain Animation – Norman’s Island (2019)

35. Internal
Demos
35
The butterfly Effect
BlackSmith
Adam
Book of the Dead
Lexus demo
The Heretic
BMW Real-time raytracing
Sherman
Jungle Book Previz
Mr Carton
Adam Episode 2
Adam Episode 3
Sonder
MPC Genesis
Heidi saison 2 (Studio 100)
Timboo & Tuskar
Baymax Dreams
Baobab Rainbow Crow
Disney Cycles
The Third Floor
Deltatre
Eurosport
DV Group
Miam! + Artefacts Studio
Wise Blue Studio
Gaumont Animation
Fauns
Atlas V
Umanimation
Big Company
Small
OnAnimation
The Mill
…
External
Productions

36. What we have
learned
36

37. 3D Rendering for episodic
content: real-time
rendering is good enough
or better than offline +
compositing.*
37
*Depends on how much hair and fur!
Short by Sparx - Offline rendering (2018)
Short by Sparx - Real-time rendering (2018)

38. Working in smaller group,
gain in creativity and
productivity (eg: less need
of storyboard)
38

39. No happy accidents.
People always control
what they do because
they see what they
get.
39
Premier Festival de films temps-réels (Unite Los Angeles, 2018)

40. Real-time engine different
from offline. Learning
phase is necessary.
40

41. Exchange with DCC and
production pipeline are
critical, as well as
organizing team work.
41

42. Pipeline: Formats and
standards (USD,
MaterialX,
OpenSubDiv,...) are
converging but more work
needs to be done both on
Unity and studios side.
42

43. Many possibilities to
explore and develop in
productions (virtual
production, mocap, live,
controllers, AR Walls,
realtime raytracing,
photogrammetry...).
43
Passage (Theoriz, 2018) – AR Wall
Disney Miraculous Live chat (Big Company 2019)

44. Estimated productivity
gain: 2x
44
Collaborative & parallel workflow
PreProduction =
Production
Instant review and
iteration
Creative decisions even late in the project
Artists work in
context of the
film amd other
departments

45. • The production energy •

46. What we do
46

47. Vision
47
Bring Real-time to all creators from the CG offline world.

48. Who and what
48
Location based
entertainment
High end location based VR,
theme parks rides, and
derived home entertainment.
Animated films
Animated TV series, shorts,
and feature film.
VFX & Virtual
production
Previz, on-set previz, post-
viz, tech viz, AR walls, until
actual production and post
production.
Advertisement
Multi-channel marketing
content creation including VR,
installations, digital signage,
and web configurators
Broadcast
Broadcast overlays, virtual
studios, AR walls, OTT/
streaming apps

49. 49
AA
Animation
AAA
Animation
All CG
users can
use Unity!
Virtual
production /
Location based
entertainment
Broadcast
VFX
Real-time
Capture (video,
mocap, camera,
VR...)
Rendering &
Animation
Production
PipelineDevelopment axis
XR

50. Infrastructure
Floating license, closed
network, Linux Unity
Editor on CentOS, Cloud
services
Pipeline
Integration with DCCs (roundtripping, live link,...),
python, asset management (eg: shotgun, Ftrack),
recorder, render passes, color pipeline, and formats
standards (eg: USD, MaterialX, OpenColorIO,
OpenTimelineIO,...), workflows for large teams,...
Graphics
Rendering to 11 (eg: Soft shadows, real-
time raytracing, high end post processing,
hair and fur, stacked lit shaders,...),
advanced camera (Cinemachine, physical
camera), HDRP VR
50
Animation
Advanced animation inside of Unity (free form
animation, custom deformers, all running with
ECS, new animation workflows,...), and
sequencing tool (Timeline)
Performance Capture
Plug any capture device
(mocap, virtual camera,...) and
record performances and re-
takes.
Video
Professional video IO (SDI, NDI,...),
advanced compositing (chroma key,
lens distorsion and calibration, node
based compositing,...)
6 pillars of development

51. Success in large scales
productions (eg: 500min)
51
Infrastructure
Floating license, closed
network, Linux Unity
Editor on CentOS, Cloud
services
Pipeline
Integration with DCCs (roundtripping, live link,...),
python, asset management (eg: shotgun, Ftrack),
recorder, render passes, color pipeline, and formats
standards (eg: USD, MaterialX, OpenColorIO,
OpenTimelineIO,...), workflows for large teams,...
Python + Shotgun in Unity (2019)

52. Continue to push graphics for more
ambitious visuals
52
Graphics
Rendering to 11 (eg: Soft shadows, motion
blur, real-time raytracing, high end post
processing, hair and fur, stacked lit shaders,
tissue, muscle, cloth simulation...),
advanced camera (Cinemachine, physical
camera), HDRP VR
Animation
Advanced animation inside of Unity (free form
animation, custom deformers, all running with
ECS, new animation workflows,...), and
sequencing tool (Timeline)
Unity Megacity demo (ECS demo)
Real-time raytracing
Fur

53. Virtual production
integrated to pipelines
53
Performance Capture
Plug any capture device
(mocap, virtual camera,...) and
record performances and re-
takes.
Video
Professional video IO (SDI, NDI,...),
advanced compositing (chroma key,
lens distorsion and calibration, node
based compositing,...)
Facial AR Remote
Digital Media Monarch Virtual production
DeckLink support

54. 54
Our film
tech
today

55. 55
Pipeline
MeshSync
(polygon mesh, blenshapes,
light, Camera, bones)
DCCs
(Maya, Max, Blender, Houdini, Motion Builder,
Substance, Photoshop,…)
FBX/USD/Alembic
roundtrip
(Mesh, Material, Stingray/Interactive
PBR material, Camera, Animation,
Custom attributes, Lights)
Video (+Encoding API in 2019.3-2020.1)
Image Sequence
Render passes (New in 2019.2!)
Audio Bakedown
PRODUCTION
PIPELINE
Python (2019.3)
Shotgun integration
(2019.3)
OpenTimelineIO
(EDL, XML.)
experimental 2019.3
Video editing -
Storyboard
(Premiere, Storyboard Pro,…)

56. 56
Python (2019.3)

57. 57
Shotgun (2019.3)

58. 58
Asset Interop (FBX, Alembic, USD import & export)
Alembic Material Mapper (Film Toolbox Package)
FBX Exporter Package

59. 59
Live Link = MeshSync
• Maya
• Max
• Motion
Builder
• Blender
• Modo
https://github.com/unity3d-
jp/MeshSync

60. 60
Rendering / Recording
• Image sequence
• Video (mp4, WebM, custom
encoders in 2019.3, FFmpeg )
• AOVs / Render passes
• Audio
And…
GameObjects!
https://github.com/keijiro/FFmpegRecorder

61. Kaptajn Bimse (2019)
61

62. 62
Editorial interop (private alpha)

63. Example of Editorial Pipeline in Unity
63
Sequence01 Sequence02
Shot10 Shot20 Shot30 Shot40
MEDIA01
Video
Snapshot
Real-time
Video Take03
Video Take02
Video Take01
Shot50EDITORIAL
Photography
REAL-TIME
MEDIA
Actors
Set, FX, Crowd,...
Actor1
Actor2, ...
Camera Anim
Dolly, crane,...
ACTOR1
PHOTOGRAPHY
ACTORS
Body Animation
Facial Animation
Cloth cache,...
BakedReal-time

64. 64
Artists tools
Shading/ Lookdev
ProBuilder ProGrid
EditorXR
Modeling/
Blocking
Set
dressing Layout Rigging/
Animation Lighting/CompEditing VFX
ShaderGraph VFXGraphTimeline
Terrain Tools
Anime ToonShader
Polybrush
Custom textures
Cinemachine
Post processing
2D animation
Runtime rigging

65. 65
Artists tools VFXGraph

66. 66
Rendering (HDRP)
Materials
Lighting
● Advanced physical based materials
○ Standard
○ SSS (Disney or Gimenez)
○ Transluency
○ Distortion
○ Anisotropy
○ Irridescence
○ Coat
○ Fabric
○ Hair
● Stacklit
● Tessellation, Built-in Decal System
● Shader graph integration
Physical based lighting
● Physical camera
● Real-time physical Directional, Point, Spot, Line, and
Rectangle lights (Lumens, Lux, Color temperature,...)
Volumetric Lighting
● Volumetric shadows
● Density volumes
Shadows
● Soft shadows
● Area light shadows
● Shadow atlas texture size
● Light layers (WIP)
Indirect Lighting:
● Progressive lightmapper (Editor)
● GPU accelerated ray-tracing for Baked Lightmap (Preview)
● Optics denoiser
● Importance sampling
● Pre-computed real-time GI with Enlighten (to be deprecated)
● Mixed lighting (direct real-time, indirect baked)
● Light probes
● Real-time reflection probes
● Planar reflections
Transmission and Sub surface scattering
Iridescence
Decal
PCSS shadows VS regular closeup
Volumetric fog
Global illumination
Tools
● Volume framework
● Decals
● Shader Graph
● VFX Graph
● VFX Toolbox
● AOV API
Post processing
● Tonemapping / Colorgrading
● AO
● Bloom
● Depth of field
● Vignette
● ….

67. 67

68. 68
Sherman (Unity, 2019) =>
Full project available!
https://create.unity3d.com
/sherman-unity-for-linear-
animation

69. 69
XR
Oculus Quest + URP
ARKit3
HDRP VR

70. Gloomy Eyes
70

71. 71
Hero Dad – WiseBlue Studios (2019)

72. Vtubers sitcom: TV Tokyo’s Watanuki-san-chino
(2019)
72
https://www.animationmagazine.net/digital-media/tv-tokyos-animated-vtuber-sitcom-made-with-blackmagic-design/

73. Future
73

74. 74
Improved and more standards support

75. 75

76. 76
Digital characters and performance capture

77. 77
Augmented reality everywhere!
TV Virtual Sets, Virtual Actors Location
Based
Entertainment
Live Events

78. 78
Roadmap

79. Convergence of Pipelines (“Hybridization”)
79
● Physical based shading, lighting, and physical camera
● Convergence of data and material formats (FBX, Alembic, USD, MDL, MaterialX, USDShade,…)
● Improvement and Standardization of real-time materials (Stacklit, GGX, Disney SSS)
● Real-time Raytracing
CHALLENGES:
● Large scenes, large number of triangles, large texture size
=> Hybrid pipeline (real-time and offline rendering)
Þ Data transformation pipeline
● Distributed teams, fast iterations, collaborative workflow
=> Cloud based production pipelines

80. Convergence of Distribution Platforms
80
● Google Stadia and Nvidia GeForce Now
Þ From buying hardware and installing games to Streaming Games!
● Streaming games vs Streaming Films?
=> Difference is a Game pad
● Netflix streaming games ? Steam streaming films? Or…
● 5G = Streaming anywhere

81. A New Entertainment Era
81
● Value is Consumption and User Acquisition, not # of sales
Þ New formats (short, long, interactive, asynchronous, social,…)
● Giants building differentiable catalogues
=> Need a lot more new IPs!
● Switching from linear to interactive is instantaneous
=> Natural transmedia
● Need for social entertainment
Þ VR, AR, LBE, Immersive Theatre

82. Questions?
More questions? Visit us on our booth!
Mathieu Muller
@the_f_key