Paris ACM Standards Tutorial Nov 2013


Published on

Paris ACM SIGGRAPH, in association with IRCAM and the CNRS, invites you to a conference on the subject of standards and open libraries in the animation and video game industries.

Published in: Technology, Art & Photos
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Paris ACM Standards Tutorial Nov 2013

  1. 1. Open Standards for Today’s Gaming Industry Erik Noreke VP of Business Development, Khronos Group © Copyright Khronos Group 2013 - Page 1
  2. 2. The Topics for this Afternoon • Why do we need open standards for gaming and animation? - What makes successful standards? • Graphics Acceleration Standards - Khronos acceleration API standards for low power and high performance gaming and animation • Bringing accelerated graphics into the Browser - WebGL – revolutionary access to the GPU in standard browsers • 3D Content Standards - How to make content re-usable without stifling innovation © Copyright Khronos Group 2013 - Page 2
  3. 3. Why Do We Need Standards? • Standards are interoperability interfaces so that compelling user experiences can be created inexpensively to build mass markets - Don’t slow growth with functionality fragmentation that adds no value • E.g. Wireless and IO standards - GSM/EDGE, UMTS/HSPA, LTE, IEEE 802.11, Bluetooth, USB … Standards drive mobile market growth by expanding device capabilities © Copyright Khronos Group 2013 - Page 3
  4. 4. Standards in the Real World Right time to Standardize? Vendor differences adding no value fragmentation is slowing growth – clear goals emerge for a standard Darwinian industry is still experimenting with what works and what doesn’t REFINE BY COMMITTEE Industry agrees on what to standardize – cooperative refinement from multiple viewpoints creates a robust solution DESIGN BY COMMITTEE Experimentation and design by committee can be slow and unfocused A good standard enables implementation innovation A bad standard stifles innovation and causes commoditization Every successful open standard has a de facto proprietary competitor and is open to competitive evolution Ecosystems seem to work best when both are healthy and evolving © Copyright Khronos Group 2013 - Page 4
  5. 5. Busting Some Standardization Myths • “Standards are slow to develop” - Time to productive ecosystems is the key … … rather than minimizing time to a proprietary specification - OpenCL 1.0 took just 6 months – intensive cooperation • “If I particpate in standards I ‘lose’ my IP” - Good IP Framework fully protects Members IP and the specification - Members agree not to assert IP claims against other Members or Adopters - License grant is VERY narrow – but protects the specification in practice • “Standards are boring” - A good standard is the industry coming together to solve real issues © Copyright Khronos Group 2013 - Page 5
  6. 6. Khronos Connects Software to Silicon ROYALTY-FREE, OPEN STANDARD APIs for advanced hardware acceleration Low level silicon to software interfaces needed on every platform Graphics, video, audio, compute, vision, sensor and camera processing Defines the forward looking roadmap for the silicon community Shipping on billions of devices across multiple operating systems Rigorous conformance tests for cross-vendor consistency Khronos is OPEN for any company to join and participate Acceleration APIs BY the Industry FOR the Industry © Copyright Khronos Group 2013 - Page 6
  7. 7. Courtesy of PC Gaming Alliance © Copyright Khronos Group 2013 - Page 7
  8. 8. OpenCL – Heterogeneous Computing • Native framework for programming diverse parallel computing resources - CPU, GPU, DSP – as well as hardware blocks(!) • Powerful, low-level flexibility - Foundational access to compute resources for higher-level engines, frameworks and languages • Embedded profile - No need for a separate “ES” spec - Reduces precision requirements A cross-platform, cross-vendor standard for harnessing all the compute resources in an SOC OpenCL Kernel OpenCL Code OpenCL Kernel Code OpenCL Kernel Code Kernel Code GPU DSP HW CPU CPU One code tree can be executed on CPUs, GPUs, DSPs and hardware. Dynamically interrogate system load and load balance work across available processors © Copyright Khronos Group 2013 - Page 8
  9. 9. OpenCL Overview • C Platform Layer API - Query, select and initialize compute devices • Kernel Language Specification - Subset of ISO C99 with language extensions - Well-defined numerical accuracy - IEEE 754 rounding with specified max error - Rich set of built-in functions: cross, dot, sin, cos, pow, log … • C Runtime API - Runtime or build-time compilation of kernels - Execute compute kernels across multiple devices • Memory management is explicit - Application must move data from host  global  local and back - Implementations can optimize data movement in unified memory systems © Copyright Khronos Group 2013 - Page 9
  10. 10. OpenCL Roadmap OpenCL HLM (High Level Model) High-level programming model, unifying host and device execution environments through language syntax for increased usability and broader optimization opportunities OpenCL 2.0 Provisional released! OpenCL 2.0 Significant enhancements to memory and execution models to expose emerging hardware capabilities and provide increased flexibility, functionality and performance to developers OpenCL SPIR 1.2 Provisional released! OpenCL SPIR (Standard Parallel Intermediate Representation) LLVM-based, low-level Intermediate Representation for IP Protection and as target back-end for alternative high-level languages © Copyright Khronos Group 2013 - Page 10
  11. 11. Mobile OpenCL Shipping • Android ICD extension released in latest extension specification - OpenCL implementations can be discovered and loaded as a shared object • Multiple implementations shipping in Android NDK - ARM, Imagination, Vivante, Qualcomm, Samsung … © Copyright Khronos Group 2013 - Page 11
  12. 12. OpenGL 3D API Family Tree ES3 is backward compatible so new features can be added incrementally Programmable vertex and fragment shaders Fixed function 3D Pipeline OpenGL ES 2.0 Content OpenGL ES 1.1 Content OpenGL ES 3.0 Content Mobile 3D WebGL 1.0 OpenGL ES 1.1 OpenGL ES 2.0 WebGL-Next OpenGL ES 3.0 ES-Next OpenGL ES 1.0 OpenGL 1.3 OpenGL 1.5 OpenGL 2.0 OpenGL 2.1 OpenGL 3.1 OpenGL 3.3 OpenGL 4.2 OpenGL 4.3 OpenGL 4.4 OpenGL 4.0 OpenGL 3.0 OpenGL 3.2 OpenGL 4.1 OpenGL 4.4 is a superset of DX11 Desktop 3D 2002 2003 2004 GL-Next 2005 2006 2007 2008 2009 2010 2011 2012 2013 © Copyright Khronos Group 2013 - Page 12
  13. 13. OpenGL ES 3.0 Highlights • Better looking, faster performing games and apps – at lower power - Incorporates proven features from OpenGL 3.3 / 4.x - 32-bit integers and floats in shader programs - NPOT, 3D textures, depth textures, texture arrays - Multiple Render Targets for deferred rendering, Occlusion Queries - Instanced Rendering, Transform Feedback … • Make life better for the programmer - Tighter requirements for supported features to reduce implementation variability • Backward compatible with OpenGL ES 2.0 - OpenGL ES 2.0 apps continue to run unmodified • Standardized Texture Compression - #1 developer request! © Copyright Khronos Group 2013 - Page 13
  14. 14. Accelerating OpenGL Innovation Bringing state-of-theart functionality to cross-platform graphics OpenGL 4.4 OpenGL 4.3 OpenGL 4.2 OpenGL 4.1 OpenGL 3.3/4.0 OpenGL 3.2 OpenGL 3.1 OpenGL 2.0 2004 OpenGL 2.1 2005 DirectX 9.0c 2006 2007 DirectX 10.0 OpenGL 3.0 2008 2009 DirectX 10.1 2010 DirectX 11 2011 2012 2013 DirectX 11.1 © Copyright Khronos Group 2013 - Page 14
  15. 15. OpenGL 4.3 Compute Shaders • Execute algorithmically general-purpose GLSL shaders - Can operate on uniforms, images and textures • Process graphics data in the context of the graphics pipeline - Easier than interoperating with a compute API IF processing ‘close to the pixel’ • Standard part of all OpenGL 4.3 implementations - Matches DX11 DirectCompute functionality Physics AI Simulation Ray Tracing Imaging Global Illumination © Copyright Khronos Group 2013 - Page 15
  16. 16. OpenGL Conformance Test Suite released!   Conformance submissions are required for GL 4.4 implementations  encouraged for earlier driver versions Shared codebase with OpenGL ES 3.0 CTS  additional desktop-specific tests  Core profile functionality  Enhancements underway to add more coverage © Copyright Khronos Group 2013 - Page 16
  17. 17. OpenCL and OpenGL Compute Shaders • OpenGL compute shaders and OpenCL support distinctly different use cases - OpenCL provides a significantly more powerful and complete compute solution 1. Full ANSI C programming of heterogeneous CPUs and GPUs 2. Utilize multiple processors 3. Precisely defined IEEE accuracy 1. Fine grain compute operations inside OpenGL 2. GLSL Shading Language 3. Execute on single GPU only Enhanced 3D Graphics apps “Shaders++” Imaging Video Physics AI Pure compute apps touching no pixels Compute Shaders © Copyright Khronos Group 2013 - Page 17
  18. 18. Active Gamers by Platform Courtesy of PC Gaming Alliance © Copyright Khronos Group 2013 - Page 18
  19. 19. Visual Sensor Revolution • Single sensor RGB cameras are just the start of the mobile visual revolution - IR sensors – LEAP Motion, eye-trackers • Multi-sensors: Stereo pairs -> Plenoptic array -> Depth cameras - Stereo pair can enable object scaling and enhanced depth extraction - Plenoptic Field processing needs FFTs and ray-casting • Hybrid visual sensing solutions - Different sensors mixed for different distances and lighting conditions • GPUs today – more dedicated ISPs tomorrow? Dual Camera Plenoptic Array Capri Structured Light 3D Camera LG Electronics Pelican imaging PrimeSense © Copyright Khronos Group 2013 - Page 19
  20. 20. OpenVX • Vision Hardware Acceleration Layer - Enables hardware vendors to implement accelerated imaging and vision algorithms - For use by high-level libraries or apps • Focus on enabling real-time vision - On mobile and embedded systems • Diversity of efficient implementations - From programmable processors, through GPUs to dedicated hardware pipelines Application OpenCV open source library Other higher-level CV libraries Dedicated hardware can help make vision processing performant and low-power enough for pervasive ‘always-on’ use Open source sample implementation Hardware vendor implementations © Copyright Khronos Group 2013 - Page 20
  21. 21. OpenVX - Power Efficient Vision Acceleration • Create vision processing graph for power and performance efficiency - Each Node can be implemented in software or accelerated hardware - Nodes may be fused by the implementation to eliminate memory transfers • EGLStreams can provide data and event interop with other APIs - BUT use of other Khronos APIs are not mandated • VXU Utility Library provides efficient access to single nodes - Open source implementation – easy way to start using OpenVX Native Camera Control OpenVX Node OpenVX Node OpenVX Node OpenVX Node Heterogeneous Processing © Copyright Khronos Group 2013 - Page 21
  22. 22. OpenVX and OpenCV are Complementary Governance Open Source Community Driven No formal specification Formal specification and full conformance tests Implemented by hardware vendors Scope Very wide 1000s of functions of imaging and vision Multiple camera APIs/interfaces Tight focus on hardware accelerated functions for mobile vision Use external camera API Conformance No Conformance testing Every vendor implements different subset Full conformance test suite / process Reliable acceleration platform Use Case Rapid prototyping Production deployment Efficiency Memory-based architecture Each operation reads and writes memory Sub-optimal power / performance Graph-based execution Optimized nodes and data transfer Highly efficient © Copyright Khronos Group 2013 - Page 22
  23. 23. Typical Imaging Pipeline • Pre- and Post-processing can be done on CPU, GPU, DSP… • ISP controls camera via 3A algorithms Auto Exposure (AE), Auto White Balance (AWB), Auto Focus (AF) • ISP may be a separate chip or within Application Processor Lens, sensor, aperture control 3A Bayer Pre-processing CMOS sensor Color Filter Array Lens RGB/YUV Image Signal Processor (ISP) Postprocessing App Need for advanced camera control API! © Copyright Khronos Group 2013 - Page 23
  24. 24. Camera Control API Goals • Provide functional portability for advanced camera applications - Reduce extreme fragmentation for ISVs wanting more than point and shoot • Generate image bursts with parameterized camera control and ISP control - For downstream processing by flexible combination of CPU, GPU and DSP • Control multiple sensors with multi-sensor synch and alignment - Stereo pairs, Plenoptic arrays, Depth Cameras • Enable system-wide sensor time-stamping - Synchronize MEMS and image sensor samples • This functionality is not available on any current platform APIs - Make this API align with future platform direction for easy adoption © Copyright Khronos Group 2013 - Page 24
  25. 25. Camera API Design Philosophy • C-language API starting from proven designs - e.g. FCAM, Android camera platform • Design alignment with widely used hardware standards - e.g. MIPI CSI • Focus on mobile, power-limited devices - But do not preclude other use cases such as automotive, surveillance, DSLR… • Minimize overlap and maximize interoperability with other Khronos APIs - But other Khronos APIs are not required • Provide support for vendor-specific extensions Group charter approved Apr13 First draft specification 2Q14 4Q13 Jul13 1Q14 Provisional specification Specification ratification Sample implementation and tests 3Q14 © Copyright Khronos Group 2013 - Page 25
  26. 26. Low Power Environment Scanning • Many sensor use cases would consume too much power to be running 24/7 - Environment aware use cases have to be very low power • ‘Scanners’ - very low power, always on, detect things in the environment - Trigger the next level of processing capability ARM 7 DSP GPU 1 MIP and accelerometers can detect someone in the vicinity Low power activation of camera to detect someone in field of view GPU acceleration for precision gesture processing © Copyright Khronos Group 2013 - Page 26
  27. 27. Sensor Industry Fragmentation … © Copyright Khronos Group 2013 - Page 27
  28. 28. StreamInput Sensor Fusion Stack Applications Platforms can provide increased access to improved sensor data stream – driving faster, deeper sensor usage by applications StreamInput implementations compete on sensor stream quality, reduced power consumption, environment triggering and context detection – enabling sensor subsystem vendors to increased ADDED VALUE OS Sensor OS APIs Middleware (E.g. Android SensorManager or iOS CoreMotion) Middleware engines need platformportable access to native, low-level sensor data stream (E.g. Augmented Reality engines, gaming engines) Low-level native API defines access to fused sensor data stream and context-awareness Sensor Sensor … Sensor Sensor Hub Hub Mobile or embedded platforms without sensor fusion APIs can provide direct application access to StreamInput Hardware transport interfaces are defined by each system, e.g. IIO or HID sensor © Copyright Khronos Group 2013 - Page 28
  29. 29. Khronos APIs for Virtual Reality VR needs not just advanced sensor processing, vision acceleration, computation and rendering - but also for all these subsystems to work efficiently together MEMS Sensors Sensor Fusion Application on CPUs, GPUs and DSPs Feature Tracking Precision timestamps on all sensor samples Advanced Camera Control and stream generation Audio Rendering EGLStream Stream frames between APIs 3D Rendering and Video Composition On GPU Camera Control API © Copyright Khronos Group 2013 - Page 29
  30. 30. Worldwide Game Users - 2012 WORLDWIDE GAME USERS BY REGION AND PLATFORM (IN MILLIONS) 500 400 300 200 100 0 N.America Europe Console PC Asia Dedicated Handheld Other Mobile Courtesy of PC Gaming Alliance © Copyright Khronos Group 2013 - Page 30
  31. 31. Leveraging Proven Native APIs into HTML5 • Khronos and W3C liaison - Leverage proven native API investments into the Web - Fast API development and deployment - Designed by the hardware community - Familiar foundation reduces developer learning curve HTML Canvas WebVX? Vision Processing Path Rendering Native APIs shipping or Khronos working group JavaScript API shipping, acceleration being developed or work underway WebCAM(!) WebStream? Sensor Fusion Camera control and video processing Camera Control JavaScript Native Possible future JavaScript APIs or acceleration © Copyright Khronos Group 2013 - Page 31
  32. 32. WebGL Implementation Anatomy Content downloaded from the Web. Middleware can make WebGL accessible to non-expert 3D programmers Content JavaScript, HTML, CSS, ... JavaScript Middleware Browser provides WebGL functionality alongside other HTML5 technologies - no plug-in required OS Provided Drivers. WebGL on Windows can use Google Angle to create conformant OpenGL ES 2.0 over DX9 HTML5 JavaScript CSS OpenGL ES 2.0 OpenGL DX9/Angle © Copyright Khronos Group 2013 - Page 32
  33. 33. WebGL Availability in Browsers Much WebGL content uses three.js library: - Microsoft – “where you have IE11, you have WebGL – turned on by default and working all the time” - Microsoft - WebGL also enabled for Windows applications - web app framework and web view - Apple - WebGL must be explicitly turned on MAC Safari and only exposed on iOS for iAds - Chrome OS - WebGL is the only cross-platform API to program the GPU - Google IO announcement - Chrome on Android will soon launch with WebGL © Copyright Khronos Group 2013 - Page 33
  34. 34. Cross-OS Portability HTML/CSS SDK C/C++ Dalvik (Java) HTML/CSS Objective C HTML/CSS HTML5 provides cross platform portability. GPU accessibility through WebGL available soon on ~90% mobile systems C# Preferred development environments not designed for portability DirectX Native code is portablebut apps must cope with different available APIs and libraries © Copyright Khronos Group 2013 - Page 34
  35. 35. WebGL First Wave Application Categories • Maps and Navigation • Modeling Tools and Repositories • Games • 3D Printing • Visualization • Music Videos and Promotion • Education • Photo Editors • Music Visualizers • Vision/Video Processing © Copyright Khronos Group 2013 - Page 35
  36. 36. Google Maps • All rendering (2D and 3D) in Google Maps uses WebGL © Copyright Khronos Group 2013 - Page 36
  37. 37. WebCL – Parallel Computing for the Web • JavaScript bindings to OpenCL APIs - Enables initiation of Kernels written in OpenCL C within the browser • Bindings stay close to the OpenCL standard - Maximum flexibility to provide a foundation for higher-level middleware - Minimal language modifications for 100% security and app portability - E.g. Mapping of CL memory objects into host memory space is not supported • API definition underway – public draft released - • Compelling use cases - Physics engines for WebGL games - Image and video editing in browser River Trail JavaScript binding to OpenCL APIs to enable initiation of OpenCL C kernels Data-parallel language extensions to JavaScript for offload at run-time © Copyright Khronos Group 2013 - Page 37
  38. 38. 3D Needs a Transmission Format! • Compression and streaming of 3D assets becoming essential - Mobile and connected devices need access to increasingly large asset databases • 3D is the last media type to define a compressed format - 3D is more complex – diverse asset types and use cases • Needs to be royalty-free - Avoid an ‘internet video codec war’ scenario • Eventually enable hardware implementations of successful codecs - High-performance and low power – but pragmatic adoption strategy is key Audio Video Images 3D MP3 H.264 JPEG ? ! An effective and widely adopted codec ignites previously unimagined opportunities for a media type © Copyright Khronos Group 2013 - Page 38
  39. 39. glTF Goals • Binary file format for efficient transmission for 3D assets - Reduce network bandwidth and minimize client processing overhead • Run-time neutral - DO NOT IMPLY OR MANDATE ANY RUN-TIME BEHAVIOR - Can be used by any app or run-time – usually WebGL accelerated • Scalable to handle compression and streaming - Though baseline format does not include compression • ‘Direct load efficiency’ for WebGL - Little or NO processing to drop glTF data into WebGL client • Carry conditioned data from any authoring format - Prototyping and optimizing efficient handling of COLLADA assets Authoring Playback A standards-based content pipeline for rich native and Web 3D applications © Copyright Khronos Group 2013 - Page 39
  40. 40. COLLADA and glTF Open Source Ecosystem OpenCOLLADA Importer/Exporter and COLLADA Conformance Tests On GitHUB Tool Interop COLLADA2GLTF Translator Other authoring formats Web-based Tools Pervasive WebGL deployment Three.js glTF Importer. Rest3D initiative © Copyright Khronos Group 2013 - Page 40
  41. 41. Conclusion • Gaming and Animation is a complex and expanding application domain and multiple standards across multiple domains are needed to enable the market • Advances in SOC silicon processing and associated APIs are about to enable Generated Reality to truly meet user expectations on all platforms • Now is a great time to get involved with the standards initiatives that effect your business • • © Copyright Khronos Group 2013 - Page 41