Enabling Augmented Reality 增强现实的实现
Beijing China, April 2013 北京 中国, 2013年4月
Erik Noreke
Technology Visionary
Vice Presiden...
Topics 主题
•  Components of Augmented Reality
•  Khronos standard APIs for Augmented Reality
•  Vision and sensor processin...
Main Components of AR 增强现实的主要组成部分
External
Data
外部数据

Data Access, Analysis
and Cloud Computing
数据获取,分析和云计算

External
Data...
Visual-based Augmented Reality 视觉为基础的增强现实
摄像机视频流发送至合成器
Camera video stream
sent to the compositor

GPS,
accelerometer,
gyr...
Khronos Standards
Unified Sensor and
Input Processing

统一传感器
和输入处理

15 Active Standards
15个现行标准

Plugin-free
3D Web Conten...
API Standard Evolution API标准的演进
DESKTOP
面

New API
technology first
evolves on high-end
platforms
全新API技术首先进入高
端平台

MOBILE...
Mobile Innovation Hot Spots 移动创新热点
•  New platform capabilities being driven by SILICON and APIs
•  芯片和API所带动的新平台性能
Consol...
How Many Sensors are in a Smartphone?
智能手机中有多少传感器?

8

•  Light 灯光
•  Proximity 邻近效应
•  2 cameras 2个照相机
•  3 microphones (...
Market Demand for Sensor Fusion API
传感融合API的市场需求
Innovative use of growing
sensor diversity
不断增长的传感多样性创新使用

Do NOT force t...
Portable Access to Sensor Fusion 传感融合的便携式获取
Apps request semantic sensor information

Spec expected in 2013
预计2013年推出规范

S...
OpenVX
•  Vision Hardware Acceleration Layer
-  Enables hardware vendors to implement accelerated
imaging and vision algor...
OpenVX Participants and Timeline 参与和计划
•  Aiming for provisional specification
in 1H 2013
-  Final specification 2H13

• 计...
OpenVX Execution Flow

执行流

•  OpenVX Graph for efficient execution
-  Each Node can be implemented in software or
acceler...
EGLStream – Streaming Images 流图片
•  EGL – originally embedded version of WGL
-  Abstraction layer to window systems and me...
EGLStream – Streaming Images 流图片

OpenMAX AL Media Player is the
EGLStream “Producer” and controls
production of frames.
O...
Example Use of Khronos APIs in AR
增强现实中Khronos API的使用
Positional
Sensors
位置传感

Audio
Rendering
音频渲染
Positional and
GPS Sen...
AR on the Web?

网页上的增强现实

•  Leverage native API investments into the Web
-  Faster API development and deployment
-  Fami...
Practical AR

现实的增强现实

AC Delco # 19181880
Info

$27.79
Add Part

1995 GMC K2500 7.4L – Error code 25

© Copyright Khronos...
In Summary 总结
•  APIs are key to enable AR applications on advanced hardware – APIs developed on
high-end hardware are now...
Upcoming SlideShare
Loading in …5
×

2013 April Meetup Asia - Enabling Augmented Reality

1,480 views

Published on

Enabling Augmented Reality talk in Beijing China by Erik Noreeke, Technology Visionary and Vice President of Business Development of The Khronos Group

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,480
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

2013 April Meetup Asia - Enabling Augmented Reality

  1. 1. Enabling Augmented Reality 增强现实的实现 Beijing China, April 2013 北京 中国, 2013年4月 Erik Noreke Technology Visionary Vice President Business Development, The Khronos Group © Copyright Khronos Group 2013 | Page 1
  2. 2. Topics 主题 •  Components of Augmented Reality •  Khronos standard APIs for Augmented Reality •  Vision and sensor processing for Augmented Reality •  Example Use of Khronos APIs in AR • 增强现实的组成 • 增强现实中的Khronos标准API • 增强现实的视觉和传感处理 • 增强现实中Khronos API使用案例 © Copyright Khronos Group 2013 | Page 2
  3. 3. Main Components of AR 增强现实的主要组成部分 External Data 外部数据 Data Access, Analysis and Cloud Computing 数据获取,分析和云计算 External Data 外部数据 External Data 外部数据 Data Comms Scene Analysis and Computation 场景分析和计算 Sensor Input 传感输入 Silicon Software Local Device 当地设备 Rendering 渲染 © Copyright Khronos Group 2013 | Page 3
  4. 4. Visual-based Augmented Reality 视觉为基础的增强现实 摄像机视频流发送至合成器 Camera video stream sent to the compositor GPS, accelerometer, gyro and camera images used to track the camera’s location and orientation GPS,加速计,回 转仪和相机图片用 于跟踪相机位置和 方位 Camera Tracking 相机跟踪 3D augmentations composited with video stream 3D 增大与视频 流合成 Camera-to-scene transform locks the 3D rendering to the real world 3D Augmentation Rendering 3D增强渲染 © Copyright Khronos Group 2013 | Page 4
  5. 5. Khronos Standards Unified Sensor and Input Processing 统一传感器 和输入处理 15 Active Standards 15个现行标准 Plugin-free 3D Web Content Parallel Computing Camera, Images and Streaming Media 照相机、图片、 流媒体 Embedded and Mobile 3D 嵌入和移动3D 2000 2003 3D Digital Asset Exchange format 3D 数字资产交 换格式 Cross platform desktop 3D 无插件3D网页 内容 并行计算 Hardware accelerated vision functions 硬件加速视觉功能 跨平台桌面 3D Web Compute Vector 2D 矢量 2D 2004 网页计算 Advanced Audio 高级音频 2005 2006 2008 2009 2011 © Copyright Khronos Group 2013 | Page 5
  6. 6. API Standard Evolution API标准的演进 DESKTOP 面 New API technology first evolves on high-end platforms 全新API技术首先进入高 端平台 MOBILE 移动 Mobile is the new platform for apps innovation. Mobile APIs unlock hardware and conserve battery life 手机是应用创新的新平台。移动AP I解脱了硬件和电池消耗的束缚 INTEROP, VISION AND SENSORS 交互操作,视觉和传感 Apps embrace mobility’s unique strengths and need complex, interoperating APIs with rich sensory inputs e.g. Augmented Reality 各种应用涵盖了移动设备的特殊性能并 需要复合功能的实现,于是带来了拥 有丰富传感输入的内部操作API,如 增强现实 WEB 网页 Diverse platforms – mobile, TV, embedded – means HTML5 will become increasingly important as a universal app platform 各种平台– 手机、电视、嵌入式HTML 5将越来越重要 © Copyright Khronos Group 2013 | Page 6
  7. 7. Mobile Innovation Hot Spots 移动创新热点 •  New platform capabilities being driven by SILICON and APIs •  芯片和API所带动的新平台性能 Console-Class 3D Performance, Quality, Controllers and TV connectivity 遥控类3D 性能,质量,遥控器和电视连接 •  Vision - Camera as sensor Computational Photography Gesture Processing Augmented Reality 视觉 – 摄像机传感器 计算性相片 姿势处理 增强现实 •  Media and Image Streaming Heterogeneous Parallel Processing •  媒体和图像流 •  构并行处理 Sensor Fusion Devices become ‘magically’ context aware – location, usage, position 传感器融合 设备成为“神奇的”文本感应 – 地点,使用,位置 HTML5 Web Apps that can be discovered on the Net and run on any platform 网页应用可以在网页上开发并在任 何平台上运行 © Copyright Khronos Group 2013 | Page 7
  8. 8. How Many Sensors are in a Smartphone? 智能手机中有多少传感器? 8 •  Light 灯光 •  Proximity 邻近效应 •  2 cameras 2个照相机 •  3 microphones (ultrasound) 3个麦克风(超声波) •  Touch 触摸 •  Position 位置 -  GPS -  WiFi (fingerprint) -  Cellular (tri-lateration) -  NFC, Bluetooth (beacons) •  Accelerometer 加速计 •  Magnetometer 磁力计 •  Gyroscope 回转仪 •  Pressure 压力 •  Temperature 温度 •  Humidity 湿度 19 © Copyright Khronos Group 2013 | Page 8
  9. 9. Market Demand for Sensor Fusion API 传感融合API的市场需求 Innovative use of growing sensor diversity 不断增长的传感多样性创新使用 Do NOT force the application developer to access individual sensors (unlike almost all other sensor APIs) Effective use of multiple interoperating sensors in one app 一个应用内多重互操作传 感的高效使用 High-level API enables sensor vendors to drive and deliver competitive sensor fusion innovation PORTABLE apps need to be isolated from sensor details 可移植应用需要从传感细节中分 离出来 不强迫应用 发者使用单独传感(这点与其 他大多数传感API是不同的) Application developers do not wish to be Sensor Fusion experts 应用 发者不想成为传感融合专家 高水平API可以让传感用户带动并提供具有 竞争力的传感融合创新 © Copyright Khronos Group 2013 | Page 9
  10. 10. Portable Access to Sensor Fusion 传感融合的便携式获取 Apps request semantic sensor information Spec expected in 2013 预计2013年推出规范 StreamInput defines possible requests, e.g. “Provide Skeleton Position” “Am I in an elevator?” 应用程序需要语义的传感信息 StreamInput定义了可能的需求,如“提供骨骼位置”,“我是 ⼀一个电梯吗?” Standardized Node Intercommunication Advanced Sensors Everywhere RGB and depth cameras, multi-axis motion/position, touch and gestures, microphones, wireless controllers, haptics keyboards, mice, track pads 高级传感无处不在 RGB和深度相机,多轴动作/位置,触摸和 姿势,麦克风,无线遥控,触觉键盘, mice,track pad Universal Timestamps Input Device Input Device Filter Node Filter Node Filter Node Input Device App Processing graph provides sensor data stream Apps Need Sophisticated Access to Sensor Data Without coding to specific sensor hardware 应用程序需要复杂的传感数据接入 无需编码到具体的传感硬件 Utilizes optimized, smart, sensor middleware Apps can gain ‘magical’ situational awareness 处理图形提供传感数据流 使用优化的,只能传感中间件 应用程序可以拥有“神奇的”感应能力 © Copyright Khronos Group 2013 | Page 10
  11. 11. OpenVX •  Vision Hardware Acceleration Layer -  Enables hardware vendors to implement accelerated imaging and vision algorithms -  For use by high-level libraries or apps •  Focus on enabling real-time vision -  On mobile and embedded systems Application 应用 OpenCV open source library 开源库 Other higher-level CV libraries 其他更高水平CV库 •  Diversity of efficient implementations -  From programmable processors, through GPUS to dedicated hardware pipelines OpenVX does not duplicate OpenCV functionality – JUST provides essential acceleration OpenVX不是 制OpenCV的功能 – 只是提供必要的加速 •  视觉硬件加速层 -  使硬件供应商可以执行加速成像和视觉算法 -  用于高水平库和应用程序 •  关注实现实时视觉 -  在移动和嵌入式系统上 Open source sample implementation? 开源实例执行? Hardware vendor implementations 硬件供应商执行 •  高效执行的多样化 -  从可编程处理器,通过GPUS到专门的硬件管线 Dedicated hardware can help make vision processing performant and low-power enough for pervasive ‘alwayson’ use 专 的硬件可以帮助是视觉处理实现性能,同时低能耗足 满足长期的持续使用 © Copyright Khronos Group 2013 | Page 11
  12. 12. OpenVX Participants and Timeline 参与和计划 •  Aiming for provisional specification in 1H 2013 -  Final specification 2H13 • 计划在2013年上半年推出建议版规范 -  最终版规范在2013年下半年推出 •  Itseez是工作组主席 •  Itseez is working group chair © Copyright Khronos Group 2013 | Page 12
  13. 13. OpenVX Execution Flow 执行流 •  OpenVX Graph for efficient execution -  Each Node can be implemented in software or accelerated hardware •  OpenVX Graph 用于高效的执行 -  每个Node都可以在软件或加速的硬件上执行 •  EGL provides data and event interop – with streaming -  BUT use of other Khronos APIs are not mandated •  EGL 提供数据和事件交互操作 – 通过流 -  但是,其他Khronos的API不受命令 •  VXU Utility Library provides efficient access to single nodes -  Open source implementation Camera Control & Image Processing Other Inputs 其他输入 UI and Display 用户界面和显示 相机控制和图片处理 OpenVX Node Compute Processing 计算处理 •  VXU 使用库提供高效的独立node获取 -  开源执行 OpenVX Node OpenVX Node OpenVX Node Compute Processing 计算处理 Other Outputs 其他输出 © Copyright Khronos Group 2013 | Page 13
  14. 14. EGLStream – Streaming Images 流图片 •  EGL – originally embedded version of WGL -  Abstraction layer to window systems and memory buffers •  Role has expanded to provide API interop – data and events -  EGLImages – single buffers that can be passed between APIs -  EGLStreams – provides stream of images – with AV sync -  Cross process EGLStreams – Producer and Consumer can be in different processes for performance or security – e.g. browser compositing process •  Android SurfaceTexture is a Java wrapper around EGLStreams -  Captures video decode or camera preview to OpenGL ES texture •  EGL – 来源于嵌入式版本的WGL -  窗口系统和内存缓冲的缩写层 •  功能扩展为提供API交互操作-数据和事件 -  EGLImages – 单独的缓冲可以在API间传递 -  EGLStreams – 提供图片流 – 通过AV同步 -  跨流程EGLStreams – 制造商和消费者可以在不同的功能或安全流程中 – 如浏览器编写流程 •  安卓SurfaceTexture是一个在EGLStreams周边的Java包装 -  捕捉视频解码或相机预览到OpenGL ES纹理 © Copyright Khronos Group 2013 | Page 14
  15. 15. EGLStream – Streaming Images 流图片 OpenMAX AL Media Player is the EGLStream “Producer” and controls production of frames. OpenMAX AL媒体播放器是 EGLStream“生产者”和产品框架控制 Camera File URL Etc. 相机文件 MEDIA PLAYER Object 媒体播放器目标 EGLStreams enables and hides details of video frame transport. Enables multiple buffering modes for different uses cases e.g.: FIFO and explicit latch/release EGLStreams实现并隐藏了视频框架传输细节。 实现了不用使用情况的多缓冲模式,如FIFO和 显式锁定/释放 EGLStream ? OpenGL ES GL_TEXTURE_EXTERNAL is the EGLStream “Consumer” and converts video format into RGB OpenGL ES texture OpenGL ES GL_TEXTURE_EXTERNAL 是EGLStream“消费者”,并将视频格 式转换成RGB OpenGL ES纹理 GL_TEXTURE_EXTERNAL © Copyright Khronos Group 2013 | Page 15
  16. 16. Example Use of Khronos APIs in AR 增强现实中Khronos API的使用 Positional Sensors 位置传感 Audio Rendering 音频渲染 Positional and GPS Sensor Data 位置和GPS传感数据 Synchronization and sensor fusion 同步和传感融合 Computer Vision/Tracking & Computational Photography 计算机视觉/跟踪&计算拍摄 Camera 相机 Camera Control Preprocess and generate video streams 相机控制流程并生成视频流 EGLStream Image stream to GPU EGLStream图片流 到GPU Position and Tracking Semantics 位置和跟踪语义 Tracked features 跟踪功能 Application on CPUs and GPUs 在CPU和GPU 上的应用 3D Rendering and Video Composition On GPU GPU上的3D渲染和视频编写 © Copyright Khronos Group 2013 | Page 16
  17. 17. AR on the Web? 网页上的增强现实 •  Leverage native API investments into the Web -  Faster API development and deployment -  Familiar foundation reduces developer learning curve •  Will this enable web based AR apps? -  Access to necessary hardware acceleration is there! •  本地API到网页上的投资 -  比API开发和执行的更快 -  对基础的熟悉降低了开发者的学习曲线 •  这将实现网页基础的增强现实应用程序吗? -  必要的硬件加速已经推出! Canvas WebMAX? Camera control and video processing 相机控制 视频处理 WebSL? Easy to use JavaScript Audio 使用简便 WebVX? Vision Processing 视觉处理 Device and Sensor APIs Device Orientation Working Groups 设备和传感API 设备主导工作组 JavaScript Native Native APIs shipping or 本地API已推出或 working group underway 工作组正在进行中 JavaScript API已推出 JavaScript API shipping or working group underway 或工作组正在进行中 Possible future 未来可能的 JavaScript APIs JavaScript API © Copyright Khronos Group 2013 | Page 17
  18. 18. Practical AR 现实的增强现实 AC Delco # 19181880 Info $27.79 Add Part 1995 GMC K2500 7.4L – Error code 25 © Copyright Khronos Group 2013 | Page 18
  19. 19. In Summary 总结 •  APIs are key to enable AR applications on advanced hardware – APIs developed on high-end hardware are now enabling AR on mobile devices •  APIs no longer exist alone – they interoperate and provide input AND output processing to form a complete platform for augmented reality •  Khronos is driving open standards for hardware acceleration Enabling portable AR applications in mobile devices! •  API是在高级硬件上实现增强现实应用的关键 – 在高端硬件上开发的API现在已经实现了移动 设备增强现实 •  API不再单独存在了 – 其交互操作并提供输入和输出处理,以为增强现实搭建一个完整平台 •  Khronos带动着硬件加速开放标准,以实现移动设备上的可移植增强现实应用程序! Connecting Software to Silicon © Copyright Khronos Group 2013 | Page 19

×