2013 Korean tour Jeju


Published on

Khronos toured Korea in November 2013. Erik Noreke VP of Business Development visited "Human Care Center Workshop" and "Interaction Standard Workshop". Additional details may be found on the Khronos Group event page https://www.khronos.org/news/events/korean-tour-2013

Published in: Technology, Art & Photos
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

2013 Korean tour Jeju

  1. 1. Technology Update Presented by: Erik Noreke, Khronos Group Vice President of Business Development November2013 © Copyright Khronos Group, 2013 - Page 1
  2. 2. Khronos Connects Software to Silicon ROYALTY-FREE, OPEN STANDARD APIs for advanced hardware acceleration Low level silicon to software interfaces needed on every platform Graphics, video, audio, compute, vision, sensor and camera processing Defines the forward looking roadmap for the silicon community Shipping on billions of devices across multiple operating systems Rigorous conformance tests for cross-vendor consistency Khronos is OPEN for any company to join and participate Acceleration APIs BY the Industry FOR the Industry © Copyright Khronos Group, 2013 - Page 2
  3. 3. Power is the New Limit to Performance • GPUs are much more power efficient than CPUs for data parallelism - When exploiting data parallelism can x10 as efficient – but can go further… • Lots of space for transistors on SOC – but can’t turn them all on at same time! - Would exceed Thermal Design Point • Dark Silicon - specialized hardware – only turned on when needed - Dedicated units can increase locality and parallelism of computation X100 Power Efficiency How do we provide access to this diversity of processors and hardware without horrible platform fragmentation? Standards! X10 X1 Dedicated Hardware Enabling new mobile use cases requires pushing computation onto GPUs and dedicated hardware GPU Compute Multi-core CPU Computation Flexibility © Copyright Khronos Group, 2013 - Page 3
  4. 4. OpenCL Built-in Kernels • Used to control non-OpenCL C-capable resources on an SOC – ‘Custom Devices’ - E.g. Video encode/decode, Camera ISP … • Represent functions of Custom Devices as an OpenCL kernel - Can enqueue Built-in Kernels to Custom Devices alongside standard OpenCL kernels • OpenCL run-time a powerful coordinating framework for ALL SOC resources - Programmable and custom devices controlled by one run-time Built-in kernels enable control of specialized processors and hardware from OpenCL run-time © Copyright Khronos Group, 2013 - Page 4
  5. 5. OpenCL Roadmap OpenCL-HLM (High Level Model) High-level programming model, unifying host and device execution environments through language syntax for increased usability and broader optimization opportunities OpenCL 2.0 Provisional released! OpenCL 2.0 Significant enhancements to memory and execution models to expose emerging hardware capabilities and provide increased flexibility, functionality and performance to developers OpenCL SPIR 1.2 Provisional released! OpenCL-SPIR (Standard Parallel Intermediate Representation) Exploring LLVM-based, low-level Intermediate Representation for IP Protection and as target back-end for alternative high-level languages © Copyright Khronos Group, 2013 - Page 5
  6. 6. Mobile OpenCL Shipping • Android ICD extension released in latest extension specification - OpenCL implementations can be discovered and loaded as a shared object • Multiple implementations shipping in Android NDK - ARM, Imagination, Vivante, Qualcomm, Samsung … © Copyright Khronos Group, 2013 - Page 6
  7. 7. OpenGL 3D API Family Tree ES3 is backward compatible so new features can be added incrementally Programmable vertex and fragment shaders Fixed function 3D Pipeline OpenGL ES 2.0 Content OpenGL ES 1.1 Content OpenGL ES 3.0 Content Mobile 3D WebGL 1.0 OpenGL ES 1.1 OpenGL ES 2.0 WebGL-Next OpenGL ES 3.0 ES-Next OpenGL ES 1.0 OpenGL 1.3 OpenGL 1.5 OpenGL 2.0 OpenGL 2.1 OpenGL 3.1 OpenGL 3.3 OpenGL 4.2 OpenGL 4.3 OpenGL 4.4 OpenGL 4.0 OpenGL 3.0 OpenGL 3.2 OpenGL 4.1 OpenGL 4.4 is a superset of DX11 Desktop 3D 2002 2003 2004 GL-Next 2005 2006 2007 2008 2009 2010 2011 2012 2013 © Copyright Khronos Group, 2013 - Page 7
  8. 8. OpenGL 4.3 Compute Shaders • Execute algorithmically general-purpose GLSL shaders - Can operate on uniforms, images and textures • Process graphics data in the context of the graphics pipeline - Easier than interoperating with a compute API IF processing ‘close to the pixel’ • Standard part of all OpenGL 4.3 implementations - Matches DX11 DirectCompute functionality Physics AI Simulation Ray Tracing Imaging Global Illumination © Copyright Khronos Group, 2013 - Page 8
  9. 9. OpenGL ES 3.0 Highlights • Better looking, faster performing games and apps – at lower power - Incorporates proven features from OpenGL 3.3 / 4.x - 32-bit integers and floats in shader programs - NPOT, 3D textures, depth textures, texture arrays - Multiple Render Targets for deferred rendering, Occlusion Queries - Instanced Rendering, Transform Feedback … • Make life better for the programmer - Tighter requirements for supported features to reduce implementation variability • Backward compatible with OpenGL ES 2.0 - OpenGL ES 2.0 apps continue to run unmodified • Standardized Texture Compression - #1 developer request! © Copyright Khronos Group, 2013 - Page 9
  10. 10. Visual Sensor Revolution • Single sensor RGB cameras are just the start of the mobile visual revolution - IR sensors – LEAP Motion, eye-trackers • Multi-sensors: Stereo pairs -> Plenoptic array -> Depth cameras - Stereo pair can enable object scaling and enhanced depth extraction - Plenoptic Field processing needs FFTs and ray-casting • Hybrid visual sensing solutions - Different sensors mixed for different distances and lighting conditions • GPUs today – more dedicated ISPs tomorrow? Dual Camera Plenoptic Array Capri Structured Light 3D Camera LG Electronics Pelican imaging PrimeSense © Copyright Khronos Group, 2013 - Page 10
  11. 11. OpenVX • Vision Hardware Acceleration Layer - Enables hardware vendors to implement accelerated imaging and vision algorithms - For use by high-level libraries or apps • Focus on enabling real-time vision - On mobile and embedded systems • Diversity of efficient implementations - From programmable processors, through GPUs to dedicated hardware pipelines Application OpenCV open source library Other higher-level CV libraries Dedicated hardware can help make vision processing performant and low-power enough for pervasive ‘always-on’ use Open source sample implementation Hardware vendor implementations © Copyright Khronos Group, 2013 - Page 11
  12. 12. OpenVX - Power Efficient Vision Acceleration • Create vision processing graph for power and performance efficiency - Each Node can be implemented in software or accelerated hardware - Nodes may be fused by the implementation to eliminate memory transfers • EGLStreams can provide data and event interop with other APIs - BUT use of other Khronos APIs are not mandated • VXU Utility Library provides efficient access to single nodes - Open source implementation – easy way to start using OpenVX Native Camera Control OpenVX Node OpenVX Node OpenVX Node OpenVX Node Heterogeneous Processing © Copyright Khronos Group, 2013 - Page 12
  13. 13. OpenVX and OpenCV are Complementary Governance Open Source Community Driven No formal specification Formal specification and full conformance tests Implemented by hardware vendors Scope Very wide 1000s of functions of imaging and vision Multiple camera APIs/interfaces Tight focus on hardware accelerated functions for mobile vision Use external camera API Conformance No Conformance testing Every vendor implements different subset Full conformance test suite / process Reliable acceleration platform Use Case Rapid prototyping Production deployment Efficiency Memory-based architecture Each operation reads and writes memory Sub-optimal power / performance Graph-based execution Optimized nodes and data transfer Highly efficient © Copyright Khronos Group, 2013 - Page 13
  14. 14. Typical Imaging Pipeline • Pre- and Post-processing can be done on CPU, GPU, DSP… • ISP controls camera via 3A algorithms Auto Exposure (AE), Auto White Balance (AWB), Auto Focus (AF) • ISP may be a separate chip or within Application Processor Lens, sensor, aperture control 3A Bayer Pre-processing CMOS sensor Color Filter Array Lens RGB/YUV Image Signal Processor (ISP) Postprocessing App Need for advanced camera control API! © Copyright Khronos Group, 2013 - Page 14
  15. 15. FCAM with Extensions • Sample time-stamping for synch between cameras and MEMS sensors • ISP model (including 3A) • Regions of Interest • Multiple cameras • Multiple ISPs • Re-entrant ISPs • Multiple output streams • Efficient memory allocation • Streaming rows (not just frames) • Image types - aligned with MIPI CSI specifications • Metadata & Statistics • Vendor extensions – specialized formats and capabilities © Copyright Khronos Group, 2013 - Page 15
  16. 16. Low Power Environment Scanning • Many sensor use cases would consume too much power to be running 24/7 - Environment aware use cases have to be very low power • ‘Scanners’ - very low power, always on, detect things in the environment - Trigger the next level of processing capability ARM 7 DSP GPU 1 MIP and accelerometers can detect someone in the vicinity Low power activation of camera to detect someone in field of view GPU acceleration for precision gesture processing © Copyright Khronos Group, 2013 - Page 16
  17. 17. Sensor Industry Fragmentation … © Copyright Khronos Group, 2013 - Page 17
  18. 18. StreamInput Sensor Fusion Stack Applications Platforms can provide increased access to improved sensor data stream – driving faster, deeper sensor usage by applications StreamInput implementations compete on sensor stream quality, reduced power consumption, environment triggering and context detection – enabling sensor subsystem vendors to increased ADDED VALUE OS Sensor OS APIs Middleware (E.g. Android SensorManager or iOS CoreMotion) Middleware engines need platformportable access to native, low-level sensor data stream (E.g. Augmented Reality engines, gaming engines) Low-level native API defines access to fused sensor data stream and context-awareness Sensor Sensor … Sensor Sensor Hub Hub Mobile or embedded platforms without sensor fusion APIs can provide direct application access to StreamInput Hardware transport interfaces are defined by each system, e.g. IIO or HID sensor © Copyright Khronos Group, 2013 - Page 18
  19. 19. Khronos APIs for Augmented Reality AR needs not just advanced sensor processing, vision acceleration, computation and rendering - but also for all these subsystems to work efficiently together MEMS Sensors Sensor Fusion Application on CPUs, GPUs and DSPs Vision Processing Precision timestamps on all sensor samples Advanced Camera Control and stream generation Audio Rendering EGLStream stream data between APIs 3D Rendering and Video Composition On GPU Camera Control API © Copyright Khronos Group, 2013 - Page 19
  20. 20. Leveraging Proven Native APIs into HTML5 • Khronos and W3C liaison - Leverage proven native API investments into the Web - Fast API development and deployment - Designed by the hardware community - Familiar foundation reduces developer learning curve HTML Canvas WebVX? Vision Processing Path Rendering Native APIs shipping or Khronos working group JavaScript API shipping, acceleration being developed or work underway WebCAM(!) WebStream? Sensor Fusion Camera control and video processing Camera Control JavaScript Native Possible future JavaScript APIs or acceleration © Copyright Khronos Group, 2013 - Page 20
  21. 21. Microsoft PhotoSynth2 • Demonstrated at Build 2013 http://channel9.msdn.com/Events/Build/2013/4-072 1:50 © Copyright Khronos Group, 2013 - Page 21
  22. 22. Cross-OS Portability HTML/CSS SDK C/C++ Dalvik (Java) HTML/CSS Objective C HTML/CSS HTML5 provides cross platform portability. GPU accessibility through WebGL available soon on ~90% mobile systems C# Preferred development environments not designed for portability DirectX Native code is portablebut apps must cope with different available APIs and libraries © Copyright Khronos Group, 2013 - Page 22
  23. 23. WebCL – Parallel Computing for the Web • JavaScript bindings to OpenCL APIs - Enables initiation of Kernels written in OpenCL C within the browser http://www.youtube.com/user/SamsungSISA#p/a/u/1/9Ttux1A-Nuc © Copyright Khronos Group, 2013 - Page 23
  24. 24. 3D Needs a Transmission Format! • Compression and streaming of 3D assets becoming essential - Mobile and connected devices need access to increasingly large asset databases • 3D is the last media type to define a compressed format - 3D is more complex – diverse asset types and use cases • Needs to be royalty-free - Avoid an ‘internet video codec war’ scenario • Eventually enable hardware implementations of successful codecs - High-performance and low power – but pragmatic adoption strategy is key Audio Video Images 3D MP3 H.264 JPEG ? ! An effective and widely adopted codec ignites previously unimagined opportunities for a media type © Copyright Khronos Group, 2013 - Page 24
  25. 25. COLLADA and glTF Open Source Ecosystem OpenCOLLADA Importer/Exporter and COLLADA Conformance Tests On GitHUB Tool Interop COLLADA2GLTF Translator Other authoring formats https://github.com/KhronosGroup/glTF Web-based Tools Pervasive WebGL deployment https://github.com/KhronosGroup/OpenCOLLADA https://github.com/KhronosGroup/COLLADA-CTS Three.js glTF Importer. Rest3D initiative © Copyright Khronos Group, 2013 - Page 25
  26. 26. Conclusion • Hardware acceleration is a complex application domain and needs multiple standards across diverse domains • Advances in SOC silicon processing and associated APIs to access them are about to enable mobile devices to truly meet user expectations • Now is a good time to get involved with the standards initiatives that effect your business • These slides and more details at www.khronos.org © Copyright Khronos Group, 2013 - Page 26