Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ANYFACE*: Create Film Industry-Quality Facial Rendering & Animation Using Mainstream PCs | SIGGRAPH 2019 Technical Sessions

341 views

Published on

ANYFACE* brings film industry-quality facial rendering and animation to mainstream PC platforms using novel approaches to create face details and control microsurfaces. The solution enables users to create high-fidelity game character facial models using photogrammetry.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

ANYFACE*: Create Film Industry-Quality Facial Rendering & Animation Using Mainstream PCs | SIGGRAPH 2019 Technical Sessions

  1. 1. AnyFace Weibo Xie (NEXT) Kai Wang (Intel) SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST A Solution for Creating Film Industry Quality Facial Rendering and Animation on Mainstream Platforms
  2. 2. SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST Introduction
  3. 3. Agenda “Siren” for Game • Ultra Detail with Face Painting • Bone Based High Fidelity Facial Animation From “Siren” to AnyFace • Lossless Face Customization • Animation Retarget • Object Space Deformer Platform Optimizations Conclusion Q&A SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  4. 4. Digital Human • Just high quality feeling • Limited texture size • Limited mesh complexity Digital Human vs Game (Face Quality) • 100% match • Up to16k texture • Unlimited mesh complexity Game SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  5. 5. Material SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST 4K Base, Normal, Roughness, Wrinkle Maps
  6. 6. Agenda “Siren” for Game • Ultra Detail with Face Painting • Bone Based High Fidelity Facial Animation From “Siren” to AnyFace • Lossless Face Customization • Animation Retarget • Object Space Deformer Platform Optimizations Conclusion Q&A SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  7. 7. Rock Painting Painting face as a rock! SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  8. 8. Face Painting SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  9. 9. Face Masks SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  10. 10. 4 layers, 128 width detail normal SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  11. 11. 4 layers, 128 width detail normal, roughness SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  12. 12. 4 layers, 128 width detail normal, roughness, base cavity SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  13. 13. 4K normal, roughness SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  14. 14. Line (Vertical) Line (Horizontal) Dot_1 Dot_2 SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST Different Face Masks
  15. 15. SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST Shared Face Mask
  16. 16. Line (Vertical) Line (Horizontal) Dot_1 Dot_2 SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST Different Face Masks
  17. 17. SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST Shared Face Mask
  18. 18. Our Work
  19. 19. Agenda “Siren” for Game • Ultra Detail with Face Painting • Bone Based High Fidelity Facial Animation From “Siren” to AnyFace • Lossless Face Customization • Animation Retarget • Object Space Deformer Platform Optimizations Conclusion Q&A SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  20. 20. Blend Shape Based Facial Animation SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  21. 21. SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  22. 22. Digital Human • Just Neutral Face • 170+ Bones • Bone Animation Data • How to Paint Bone Weight? • Achieve Same Quality as BS? • 3-4 Months for Artist!!! Digital Human vs Game (Facial Animation) • 40+ Full Face Blend Shape • 700+ Spit Blend Shape • Rig-Logic Data Control Animation Game SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  23. 23. SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  24. 24. BlendShape 170 Bones AnimationSet Bone Indices Bone Weights Siren with SSDR SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  25. 25. Bones Facial AnimationSiren with Bone Based Facial Animation
  26. 26. Prove the Fidelity UE4 Runtime Compare • Unwrap Face Position to Texture • Compare Textures MeshCompare • https://github.com/dgovil/MeshCompare • Same Topology • Maya Plugin BlendShape 70 Bones 170 Bones SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  27. 27. Unwrapping Face Material SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  28. 28. Unwrapping Position BlendShape Unwrap Position SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  29. 29. 5000 Tris Unwrap Position Unwrapping Position SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  30. 30. Mesh Error Error SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  31. 31. Mesh Diff Matrix SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  32. 32. Agenda “Siren” for Game • Ultra Detail with Face Painting • Bone Based High Fidelity Facial Animation From “Siren” to AnyFace • Lossless Face Customization • Animation Retarget • Object Space Deformer Platform Optimizations Conclusion Q&A SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  33. 33. Full Face Morph
  34. 34. Face Part Split & Align SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST 100% to the target is our target!
  35. 35. https://www.russian3dscanner.com/wrap3-tutorials/ SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST Replace In Wrap3
  36. 36. Wrap Graph SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  37. 37. Face BS Split & Align (Wrap Replace)
  38. 38. More Variance https://www.3dscanstore.com/3d-body-parts-models SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  39. 39. Random Customization
  40. 40. Agenda “Siren” for Game • Ultra Detail with Face Painting • Bone Based High Fidelity Facial Animation From “Siren” to AnyFace • Lossless Face Customization • Animation Retarget • Object Space Deformer Platform Optimizations Conclusion Q&A SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  41. 41. Animation Retarget
  42. 42. Agenda “Siren” for Game • Ultra Detail with Face Painting • Bone Based High Fidelity Facial Animation From “Siren” to AnyFace • Lossless Face Customization • Animation Retarget • Object Space Deformer Platform Optimizations Conclusion Q&A SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  43. 43. N Heads N Hats SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  44. 44. N Heads One Hat SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  45. 45. • Unwrapping Face in Object Space • Delta Position Related to Neutral Face • Projecting Mesh Vertex to Face SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST Object Space Deformer
  46. 46. Unwrapping Face Material SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  47. 47. Unwrapping Neutral Face SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  48. 48. Delta Position SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  49. 49. 2nd UV for Unwrapping Face SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  50. 50. Projecting Mesh Vertex to Face SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  51. 51. Seam SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  52. 52. Better UV For Unwrapping Face SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  53. 53. Better UV for Hat SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  54. 54. Fixing the Seam SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  55. 55. Hat/Mask SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  56. 56. Glass Deformer SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  57. 57. SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST Hair Deformer
  58. 58. SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST Hair UV
  59. 59. Hair/Fuzz Deformer SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  60. 60. Agenda “Siren” for Game • Ultra Detail with Face Painting • Bone Based High Fidelity Facial Animation From “Siren” to AnyFace • Lossless Face Customization • Animation Retarget • Object Space Deformer Platform Optimizations Conclusion Q&A SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  61. 61. Platform Optimizations • Face Painting Optimization • Mesh LOD • Bone LOD • Deformer Optimization SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  62. 62. Face Painting Optimization • Detail Texture Packing • RG (Normal) • B (Roughness) • A (Diffuse Cavity) SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  63. 63. Detail Texture Packing SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  64. 64. Detail Texture Packing • Without Packing • 3 sampling for Base layer • 4 detail layers for each face • 3 sampling for each detail layer • 15 times! SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST • With Packing • 3 sampling for Base layer • 4 packed detail layers for each face • Reduced to 7 times!
  65. 65. Mesh LOD 40,000 Tris SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  66. 66. Mesh LOD 20,000 Tris SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  67. 67. Mesh LOD 10,000 Tris SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  68. 68. Mesh LOD 5,000 Tris SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  69. 69. 170 Bones 40,000 Tris SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  70. 70. 100 Bones 40,000 Tris SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  71. 71. 70 Bones 40,000 Tris SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  72. 72. Mesh Diff Matrix SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  73. 73. Compute Shader Time for Skinning (5 Faces) 0 50 100 150 200 250 MESH LOD0 MESH LOD1 MESH LOD2 MESH LOD3 MICROSECOND Bone_LOD0 Bone_LOD1 Bone_LOD2 Mesh LOD0 Mesh LOD1 Mesh LOD2 Mesh LOD3 Bone_LOD0 184 104 80 44 Bone_LOD1 194 99 68 40 Bone_LOD2 193 111 84 43SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST GPU Duration for 5 faces on Iris Pro 580 - Skinning
  74. 74. Base Pass Time Analysis (5 Faces) 0 500 1,000 1,500 2,000 2,500 MESH LOD0 MESH LOD1 MESH LOD2 MESH LOD3 MICROSECOND Bone_LOD0 Bone_LOD1 Bone_LOD2 Mesh LOD0 Mesh LOD1 Mesh LOD2 Mesh LOD3 Bone_LOD0 2092 1582 1408 1254 Bone_LOD1 2008 1467 1439 1151 Bone_LOD2 2015 1582 1347 1287SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST GPU Duration for 5 faces on Iris Pro 580 – Base Pass with material simplified
  75. 75. CPU Time (ms) for Animation (5 Faces) SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST Animation Time on Game Thread Animation Time on Worker Threads 1.33 1.11 1.07 0.86 1 0.79 4-core/8-thread 8-core/16-thread 0 0.2 0.4 0.6 0.8 1 1.2 1.4 LOD3 LOD2 LOD1 1.3 0.45 0.36 1.16 0.43 0.34 1.08 0.41 0.33 0 0.2 0.4 0.6 0.8 1 1.2 1.4 single-thread 4-core/8-thread 8-core/16-thread LOD3 LOD2 LOD1
  76. 76. Bone & Mesh Optimizations • Vertex number is a deciding factor to skinning time. • Proportional to computation cost (each vertex has 4 bone weights) • Bone number is basically not relevant. • Vertex number can largely affect to Base-pass time on some platforms. • Current bone matrices computation is on CPU side. • Greater impact for mesh complexity on GPU. • Greater impact for bone complexity on CPU. • Better performance on CPUs with more cores and further optimizations can be done. SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  77. 77. Deformer Optimizations • LOD Meshes for Deformer • Buffer Size Policy • 128 buffer for near shot • 32 for 5 meters away • Static delta texture for 15 meters away • RGBA32f->RGBA16f SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  78. 78. SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST 128x128 Buffer
  79. 79. SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST 32x32 Buffer
  80. 80. SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST 2x2 Buffer
  81. 81. GPU Time of Deformer (1 Face) SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST 317 237 164 130 RGBA16F 32x32 MESH LOD0 RGBA16F 32x32 MESH_LOD1 RGBA16F 32x32 MESH LOD2 RGBA16F 32x32 MESH LOD3 Deformer time for 1 face on Iris Pro 580• Mesh LOD is a deciding factor and lower mesh complexity largely improves the perf. • RT formats and buffer size have less impact on perf. • For RGBA16F 32x32, ~130 us per face is good for our current game titles.
  82. 82. Conclusion • Bring Siren to Game • Ultra Detail • Bone Based High Fidelity Facial Animation • One Hair/Attachment for Any Face • Customization • Different Platforms SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  83. 83. Future Work • Face Library Based on Our Tech • Further Work on Animation Retarget • Introducing More AI Elements • AI Face Painting • One Photo Avatar • Different Styles (e.g. Cartoon) • All Our Tech In A Game Production SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  84. 84. References • [unreal] “Digital Humans”, unreal • [Adelle] “Achieving High-Quality, Low-Cost Skin: An Environment Approach”, Adelle Bueno, Eduardo Mosena , GDC2017 • [Binh] “Smooth Skinning Decomposition with Rigid Bones”, Binh Huy Le , Zhigang Deng, SIGGRAPH Asia 2012 • [R3dS] “Wrap3 Tutorials”, R3dS • [Ryan] “Automated and Improved UV Dilation”, Ryan Brucks, 2016 • [dgovil] ”github.com/dgovil/MeshCompare”, dgovil, 2018 • [Andrew]” Photogrammetry and Star Wars Battlefront”, Andrew Hamilton, Kenneth Brown • [IDZ] https://software.intel.com/en-us SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  85. 85. • NEXT • Li Shen • Yu Gu • Vladislav Perge • Jun bai Li • Qiang Fu • Jiayang Yang • Li Jin • Xiaojun Ding • Jinjian Xu • Bo Yu • Shengjun Tang • Guofeng Chang • Intel • Cage Lu • Kai Wang Acknowledgements • Family SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST • Epic
  86. 86. Question? SIGGRAPH 2019 | LOS ANGLES | 28 JULY - 1 AUGUST
  87. 87. Legal Disclaimer Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at [intel.com]. Results have been estimated or simulated using internal Intel analysis or architectural simulation or modeling and provided to you for information purposes. Any differences in your system hardware, software or configuration may affect your actual performance. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors and Intel Integrated GPU. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more information go to www.intel.com/benchmarks. Optimization Notice: Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Configurations used for the perf data: • Macbook Pro with Intel Iris Graphics 550, 530, 515 some with fixed 850Mhz frequency and some with dynamic frequency • Latest Windows OS • macOS 10.15 Beta, macOS 10.14.5 • Intel® Gen11 processor graphics • All testing was performed at Intel Intel, the Intel logo, are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. © Intel Corporation.

×