Multimedia Stack Overview
多媒体堆栈概述
Erik Noreke
Chair of OpenSL ES Working Group
Chair of OpenMAX AL Working Group
Peruser o...
Today’s Consumer Requirements 今日消费者的需求
•  Rich media applications and UI
-  Consumer decisions are driven by the multimedi...
Application Developer Requirements
应用程序开发者的需求
•  Use case driven design
-  For ease of use

•  使用案例带动设计发展
-  简化使用

•  Comp...
System Integration Requirements
系统整合的需求
•  Function driven design
-  For ease of integration
•  Comprehensive feature set
...
Multimedia Stack – Overview 多媒体堆栈概述
Applications
应用程序

Use cases
使用案例

Application Access
应用程序入口

Ease of use
简化使用

Multim...
Multimedia Stack – Overview 多媒体堆栈概述
Applications
应用程序

Use cases
使用案例

Application Access
应用程序入口

Ease of use
简化使用

Multim...
Multimedia Stack – Overview 多媒体堆栈概述
Applications
应用程序

Use cases
使用案例

Application Access
应用程序入口

Ease of use
简化使用

Multim...
Why Create Khronos Multimedia APIs?
为什么创建Khronos多媒体API?
Khronos APIs
•  Conformance Tests
•  Portable
•  IP Protection
•  ...
Streams – The Core of Multimedia 流 – 多媒体的核心
•  Streams are content delivery
channels
-  Video, Audio, Text data flow
-  Ad...
OpenMAX IL – Integration Layer 整合层
Applications
应用程序

Use cases
使用案例

Application Access
应用程序入口

Ease of use
简化使用

Multime...
OpenMAX IL – Managing Stream Flow 管理流
Component Networks

组件网络

•  Enables arbitrary multimedia pipelines by
•  通过一起插入块,实现...
OpenMAX IL Component 组件

4 bytes

•  Component abstracts a single
actor –
-  Decoder, effect or renderer

•  组件缩写成单独动作-  解...
OpenMAX IL Example Graph 示例图
•  Standardized component interfaces
enable flexible media graphs

•  标准化组件界面实现灵活的媒体图形

•  Co...
OpenSL ES
– Application Access 应用入口
OpenMAX AL
Applications
应用程序

Use cases
使用案例

Application Access
应用程序入口

Ease of use
简...
OpenSL ES
– Controlling Stream Flow 流控制
OpenMAX AL
•  Hardware and low level access is powerful
but complex
-  More power ...
OpenMAX AL

Focused on Streaming Media – Not Media Streams
关注流媒体 – 不是媒体流
•  Media playback and recording
-  Playback and r...
OpenSL ES – Powerful 3D Audio 强大的3D音频
•  Full range of effects and controls, including advanced 3D effects
such as Doppler...
OpenSL ES and OpenMAX AL
•  Working groups collaborated to define the common API functionality
•  Independent, Compatible,...
The OpenSL ES / OpenMAX AL Object 目标
•  Object represents a logical grouping of functionality
-  Acts on a stream or set o...
OpenMAX AL & OpenSL ES – Play Audio Shared Use Case
播放音频分享使用案例

© Copyright Khronos Group, 2013 - Page 20
OpenMAX AL & OpenSL ES – Play Video
Shared Use Case

© Copyright Khronos Group, 2013 - Page 21
Multimedia Stack 多媒体堆栈– Khronos APIs
Applications
应用程序

Use cases
使用案例

Application Access
应用程序入口

Ease of use
简化使用

Multi...
Stream Control vs. Stream Management - Audio playback
流控制和流管理 – 音频回放
OR
URI Data
Source

File Reader

Player

Audio
Decode...
Multimedia Stack 多媒体堆栈– Khronos APIs
Applications
应用程序

Use cases
使用案例

Application Access
应用程序入口

Ease of use
简化使用

Multi...
Example: 3D Playback of two audio files
示例:2个音频文件的3D回放

© Copyright Khronos Group, 2013 - Page 25
Example: Video Camera
示例:视频摄像机

© Copyright Khronos Group, 2013 - Page 26
Multimedia – Managing Streams 多媒体 – 管理流
•  OpenMAX IL manages the pipeline
-  Controls where and how the stream flows
•  O...
Thank You
谢谢
© Copyright Khronos Group, 2013 - Page 28
Upcoming SlideShare
Loading in …5
×

2013 April Meetup Asia - Multimedia Stack Overview

986 views

Published on

Multimedia Stack Overview talk by Erik Noreke, Chair of OpenSL ES working group, Chair of OpenMAX AL working Group, peruser of OpenMAX IL.

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

2013 April Meetup Asia - Multimedia Stack Overview

  1. 1. Multimedia Stack Overview 多媒体堆栈概述 Erik Noreke Chair of OpenSL ES Working Group Chair of OpenMAX AL Working Group Peruser of OpenMAX IL © Copyright Khronos Group, 2013 - Page 1
  2. 2. Today’s Consumer Requirements 今日消费者的需求 •  Rich media applications and UI -  Consumer decisions are driven by the multimedia experience -  Drives smart phone market •  Enhanced user experience -  Games -  Theater-like audio for video playback -  Streaming media services -  Mobile TV •  Availability of third party applications -  Consumers expected to purchase $35 billion of mobile applications by 2014 •  多媒体应用程序和用户界面 -  多媒体体验带动消费者决定 -  推动智能手机市场 •  增强的用户体验 -  游戏 -  视频播放拥有影院般的音响效果 -  流媒体服务 -  移动电视 •  第三方应用程序的实现 -  预计到2014年消费者对移动应用程序的购买力将达到350亿美元 © Copyright Khronos Group, 2013 - Page 2
  3. 3. Application Developer Requirements 应用程序开发者的需求 •  Use case driven design -  For ease of use •  使用案例带动设计发展 -  简化使用 •  Comprehensive feature set -  Including advanced audio and streaming media •  综合参数设置 -  包括高级音频和流媒体 •  Consistent APIs -  Same APIs for both hardware and software implementations •  API的一致性 -  同时可应用于硬件和软件的 同一个 •  Full, predictable functionality -  No need to guess what's available or how it is going to work •  完整可预测的性能 -  无需猜测什么是可以实现的 或者应该如何使用 •  Portable -  No need to port application for different platforms •  可移植性 -  不同平台间无需使用接口应 用程序 © Copyright Khronos Group, 2013 - Page 3
  4. 4. System Integration Requirements 系统整合的需求 •  Function driven design -  For ease of integration •  Comprehensive feature set -  Including advanced audio and streaming media •  Consistent APIs -  Same APIs regardless if using a software implementation or hardware acceleration •  Full, predictable functionality -  No need to guess about available functionality or how it is going to work •  Ease of Integration -  Expected set of components reduces integration time and costs •  功能带动设计 -  简化整合工作 •  综合参数设置 -  包括高级音频和流媒体 •  API一致性 -  相同的API同时可以使用在 软件执行和硬件加速上 •  完整的可预测的性能 -  无需猜测什么是可以实现 的或者应该如何使用 •  简化整合 -  未来会有整套零部件以降 低整合时间和成本 © Copyright Khronos Group, 2013 - Page 4
  5. 5. Multimedia Stack – Overview 多媒体堆栈概述 Applications 应用程序 Use cases 使用案例 Application Access 应用程序入口 Ease of use 简化使用 Multimedia Control Layer 多媒体控制层 Abstraction level 抽象水平 Mul$media  System  Access   多媒体系统入口 Ease of integration 简化整合 HW Integration / Implementation Layer 硬件整合/执行层 Choice of HW / SW 硬件/软件选择 System Hardware 系统硬件 HW  Accelera*on     硬件加速 Use case driven design 使用案例带动设计 Creates abstraction layer for application developer 为应用 发者创建抽象层 Function driven design with access regardless of implementation 通过各 执行的实现,功能带 动设计 Implementation of Multimedia functions in either software or hardware 软件和硬件中多媒体功能执行 © Copyright Khronos Group, 2013 - Page 5
  6. 6. Multimedia Stack – Overview 多媒体堆栈概述 Applications 应用程序 Use cases 使用案例 Application Access 应用程序入口 Ease of use 简化使用 Multimedia Control Layer 多媒体控制层 Abstraction level 抽象水平 Mul$media  System  Access   多媒体系统入口 Choice of HW / SW 硬件/软件选择 Operating Systems 操作系统 Ease of integration 简化整合 HW Integration / Implementation Layer 硬件整合/执行层 Application developers 应用程序 发者 System Hardware 系统硬件 Device Manufacturers 驱动制造商 HW  Accelera*on     硬件加速 © Copyright Khronos Group, 2013 - Page 6
  7. 7. Multimedia Stack – Overview 多媒体堆栈概述 Applications 应用程序 Use cases 使用案例 Application Access 应用程序入口 Ease of use 简化使用 Multimedia Control Layer 多媒体控制层 Abstraction level 抽象水平 Mul$media  System  Access   多媒体系统入口 Ease of integration 简化整合 HW Integration / Implementation Layer 硬件整合/执行层 Choice of HW / SW 硬件/软件选择 System Hardware 系统硬件 HW  Accelera*on     硬件加速 © Copyright Khronos Group, 2013 - Page 7
  8. 8. Why Create Khronos Multimedia APIs? 为什么创建Khronos多媒体API? Khronos APIs •  Conformance Tests •  Portable •  IP Protection •  Well defined layers of implementation Open Source Framework •  Free to change •  Every implementation unique •  GPL license •  Crosses implementation layers •  Predictable set of functionality •  Functionality dependent on implementation Khronos APIs 开源框架 •  符合性测试 •  自由更换 •  可移植性 •  每个执行都是不同的 •  IP 保护 •  GPL 授权 •  专业定义的执行层 •  跨执行层 •  可预测的整套性能 •  性能独立于执行 © Copyright Khronos Group, 2013 - Page 8
  9. 9. Streams – The Core of Multimedia 流 – 多媒体的核心 •  Streams are content delivery channels -  Video, Audio, Text data flow -  Additional metadata •  Controlling multimedia is about manipulating the streams -  Play rate, effects, source and destinations •  The streams are sent through various components -  Readers, parsers and decoders (codecs) -  Effects, schedulers and renderers -  Splitters and mixers •  Combined with HW clock synch and system controls this forms the core of the multimedia stack •  与硬件时钟同步和系统控制结合, 形成了多媒体堆栈的核心流是内容传递的渠道 •  流是通过各种零部件传输的 -  视频、音频、文本数据流 -  阅读器、解析器、解码器(编码器) -  其他meta数据 •  控制多媒体就是对流的控制 -  效果、调度程序、渲染器 -  分离器、混合器 -  播放速率、效果、来源、目的 © Copyright Khronos Group, 2013 - Page 9
  10. 10. OpenMAX IL – Integration Layer 整合层 Applications 应用程序 Use cases 使用案例 Application Access 应用程序入口 Ease of use 简化使用 Multimedia Control Layer 多媒体控制层 Abstraction level 抽象水平 Mul$media  System  Access   多媒体系统入口 Ease of integration 简化整合 HW Integration / Implementation Layer 硬件整合/执行层 Choice of HW / SW 硬件/软件选择 System Hardware 系统硬件 HW  Accelera*on     硬件加速 © Copyright Khronos Group, 2013 - Page 10
  11. 11. OpenMAX IL – Managing Stream Flow 管理流 Component Networks 组件网络 •  Enables arbitrary multimedia pipelines by •  通过一起插入块,实现任意多媒体管线 plugging blocks together -  Componentized architecture abstracts multimedia - 组件化结构缩写多媒体性能块界面 functionality block interfaces •  Wide variety of building blocks for imaging, •  拥有成像、视频和音频功能的各种建筑模块 video and audio functions -  Encode, decode, apply an effect, capture, render, - 编码、解码、效果应用、捕捉、渲染、 分离、混合等等 split, mix, etc •  Enables blocks from different sources to work together -  Blocks can be implemented in software or hardware Portable & Re-usable streaming media building blocks 可移植可再用的流媒体 搭建块 •  从不同来源共同实现块 - 块可以在软件和硬件内执行 © Copyright Khronos Group, 2013 - Page 11
  12. 12. OpenMAX IL Component 组件 4 bytes •  Component abstracts a single actor – -  Decoder, effect or renderer •  组件缩写成单独动作-  解码器、效果或渲染器 •  Standardized ports allows chaining of arbitrary components -  Including port tunneling for compatible components •  标准化接口实现任意组件的 链接 -  包括兼容组件的接口隧 道技术 •  Commands are sent to the component •  命令被发送到组件 nPortIndex •  从组件接收事件 nDataSize = 6 •  Events are received from the component offset padding OMX_BUFFERHEADERTYPE padding nSize = 28 OMX_OT nVersion eType != OMX_ExtraDataNone extra data •  接口有调用和回调 padding nSize = 24 OMX_OTH nVersion •  Ports have calls and callbacks nPortIndex Commands 命令 Input ports 输入接口 • Callbacks 回调 data ... pBuffer nOffset = 9 nAllocLen = 96 nFilledLen = 21 nFlags inc 0x40 ... eType = OMX_ExtraDataNone Events 事件 OMX IL Component 组件 nDataSize = 0 data padding unused Output ports 输出接口 • Calls 调用 • Callbacks 回调 © Copyright Khronos Group, 2013 - Page 12
  13. 13. OpenMAX IL Example Graph 示例图 •  Standardized component interfaces enable flexible media graphs •  标准化组件界面实现灵活的媒体图形 •  Components are chained into pipelines as needed for each stream •  包括多流同步 •  根据每个流的需求,组件被链接到不同的管线 •  Includes multi-stream synchronization OpenMAX IL Component Interfaces Enables components to be flexibly connected in any graph topology E.g. send dance beats to 3D application for visualization AAC Audio Audio Scheduler Plug-in RTSP (RTCP + RTP) Streaming Audio Renderer Speakers Video Scheduler Audio Decoder Video Renderer Display Clock for AV Sync Server Time Data Control MPEG4/ H.264 Video Video Decoder Decompressed Video Example: MPEG-4 video synchronized with AAC audio decode © Copyright Khronos Group, 2013 - Page 13
  14. 14. OpenSL ES – Application Access 应用入口 OpenMAX AL Applications 应用程序 Use cases 使用案例 Application Access 应用程序入口 Ease of use 简化使用 Multimedia Control Layer 多媒体控制层 Abstraction level 抽象水平 Mul$media  System  Access   多媒体系统入口 Ease of integration 简化整合 HW Integration / Implementation Layer 硬件整合/执行层 Choice of HW / SW 硬件/软件选择 System Hardware 系统硬件 HW  Accelera*on     硬件加速 © Copyright Khronos Group, 2013 - Page 14
  15. 15. OpenSL ES – Controlling Stream Flow 流控制 OpenMAX AL •  Hardware and low level access is powerful but complex -  More power than most application developers require •  硬件和低水平入口很强大但是很复杂 -  比大多数开发者所要求的更高效 •  Most application developers just want to playback and record media -  Specifying where the content comes from -  Specifying where the content should be rendered to -  Manipulate a few playback controls -  Have simple configurability •  -  -  -  -  •  OpenMAX AL and OpenSL ES provide… -  Simple high-level multimedia APIs for playback and recording use cases •  OpenMAX AL和OpenSL ES提供… -  简单的高水平多媒体API,用于回放和记录 使用案例 •  凭借跨平台可移植性 - 无需为每个平台重写应用程序 •  …With Cross-platform portability -  No need to rewrite an application for every platform 大多数应用开发者仅仅需要回放和记录媒体 内容具体来自哪 内容具体应该被渲染到哪 进行一些回放控制 简单的可配置性 © Copyright Khronos Group, 2013 - Page 15
  16. 16. OpenMAX AL Focused on Streaming Media – Not Media Streams 关注流媒体 – 不是媒体流 •  Media playback and recording -  Playback and recording of everything from video files to radio to camera input •  Full range of video effects and controls – including playback rate, post processing, and image manipulation -  Apply advanced effects to captured images and video before sending to friends •  Complete metadata support -  Read, Edit, and Set the metadata for most types of content •  媒体回放和记录 -  回放和记录从视频文件、音频到照相机输入获得的所有内容 •  完整的视频效果和控制 – 包括回放速率、后处理、图片管理 -  在与朋友分享之前,给捕捉图像和视频应用高级效果 •  完整的meta数据支持 -  大多数类型内容的Meta数据阅读、编辑、设置 © Copyright Khronos Group, 2013 - Page 16
  17. 17. OpenSL ES – Powerful 3D Audio 强大的3D音频 •  Full range of effects and controls, including advanced 3D effects such as Doppler and virtualization -  Experience rich, enhanced sound from locations other than the handset, even moving, for the ultimate multimedia experience •  Full 3D audio functionality enhances any gaming experience -  Perfect companion to OpenGL ES •  Designed to take full advantage of system capabilities •  各种效果和控制,包括高级3D效果,例如多普勒和虚拟 - 丰富体验、从地点加强音效,不仅仅是手持设备、甚至移动音效,用于多 媒体体验 •  完整3D音频功能加强各种游戏体验 - OpenGL ES的完美匹配 •  为了充分利用系统性能而设计 © Copyright Khronos Group, 2013 - Page 17
  18. 18. OpenSL ES and OpenMAX AL •  Working groups collaborated to define the common API functionality •  Independent, Compatible, Consistent and Distinct •  工作组合作共同定义共同的API性能 •  独立、可移植、一致的、相区别的 Video playback 3D Audio Audio Effects Advanced MIDI Buffer queues Audio Playback Audio Recording Basic MIDI Video recording Radio and RDS Camera Image capture & display © Copyright Khronos Group, 2013 - Page 18
  19. 19. The OpenSL ES / OpenMAX AL Object 目标 •  Object represents a logical grouping of functionality -  Acts on a stream or set of streams -  Player, recorder, mixer •  Object has a source and a destination -  Acts on a stream or set of streams as they flow through •  Objects have control interfaces -  Grouped methods and callbacks -  Play, Record, Seek •  目标代表一个逻辑功能群组 Dataflow - 作为一个流或一组流 - 播放器、记录器、混合器 •  目标有一个源和目的地 Source Object Destination - 当其流过时作为一个流或一组流 •  目标有控制界面 - 播放、记录、搜寻 Interfaces © Copyright Khronos Group, 2013 - Page 19
  20. 20. OpenMAX AL & OpenSL ES – Play Audio Shared Use Case 播放音频分享使用案例 © Copyright Khronos Group, 2013 - Page 20
  21. 21. OpenMAX AL & OpenSL ES – Play Video Shared Use Case © Copyright Khronos Group, 2013 - Page 21
  22. 22. Multimedia Stack 多媒体堆栈– Khronos APIs Applications 应用程序 Use cases 使用案例 Application Access 应用程序入口 Ease of use 简化使用 Multimedia Control Layer 多媒体控制层 Abstraction level 抽象水平 Mul$media  System  Access   多媒体系统入口 Ease of integration 简化整合 HW Integration / Implementation Layer 硬件整合/执行层 Choice of HW / SW 硬件/软件选择 System Hardware 系统硬件 HW  Accelera*on     硬件加速 © Copyright Khronos Group, 2013 - Page 22
  23. 23. Stream Control vs. Stream Management - Audio playback 流控制和流管理 – 音频回放 OR URI Data Source File Reader Player Audio Decoder Audio Effect Output Mix Audio Mixer Audio Sink Clock © Copyright Khronos Group, 2013 - Page 23
  24. 24. Multimedia Stack 多媒体堆栈– Khronos APIs Applications 应用程序 Use cases 使用案例 Application Access 应用程序入口 Ease of use 简化使用 Multimedia Control Layer 多媒体控制层 Abstraction level 抽象水平 Mul$media  System  Access   多媒体系统入口 Ease of integration 简化整合 HW Integration / Implementation Layer 硬件整合/执行层 Choice of HW / SW 硬件/软件选择 System Hardware 系统硬件 HW  Accelera*on     硬件加速 © Copyright Khronos Group, 2013 - Page 24
  25. 25. Example: 3D Playback of two audio files 示例:2个音频文件的3D回放 © Copyright Khronos Group, 2013 - Page 25
  26. 26. Example: Video Camera 示例:视频摄像机 © Copyright Khronos Group, 2013 - Page 26
  27. 27. Multimedia – Managing Streams 多媒体 – 管理流 •  OpenMAX IL manages the pipeline -  Controls where and how the stream flows •  OpenMAX AL and OpenSL ES manage the flow -  Controls when and how fast the stream flows •  Together they separate the domains of the multimedia stack -  A distinct integration layer -  Use case driven application access layer •  To form the basis for a complete multimedia stack -  Without limiting implementation or innovation •  OpenMAX IL管理管线 -  控制流的位置和方式 •  OpenMAX AL和OpenSL ES 管理流 -  控制流的位置和速度 •  通过分离多媒体堆栈的各个域 -  一个明确的整合层 -  使用案例带动应用入口层 •  从最基础的部分到完整的多媒体堆栈 -  对执行或创作完全没有限制 © Copyright Khronos Group, 2013 - Page 27
  28. 28. Thank You 谢谢 © Copyright Khronos Group, 2013 - Page 28

×