Uploaded byAlexy Joseph
A slightly deeper dive into Stagefright
This document provides an overview of Stagefright, the primary multimedia framework available on Android. It describes Stagefright's architecture and components, how it handles file playback, recording, and streaming. It also discusses hardware accelerated rendering, tweaking and optimizing Stagefright, and references additional resources. The document is presented by Alexy Mathew Joseph from Pathpartner and focuses on explaining how Stagefright works under the hood.
More Related Content
Similar to A slightly deeper dive into Stagefright
A slightly deeper dive into Stagefright
- 1. Stagefright – howit works Alexy Mathew Joseph
- 2. Who am I? Works at Pathpartner(pathpartnertech.com) as a Technical Architect. Working on multimedia frameworks and systems optimization for 8 years. Contact: galahad@gmail.com
- 3. Flow Overview Architecture File playback Recording Streaming Hardware accelerated rendering Tweaking and optimizing Questions?
- 4. Overview Multimedia frameworks - the weight lifters
- 5. Stagefright The primary multimedia framework available on Android Feature set Supports playback of multiple file formats Integrates with OpenMax codecs Recording RTSP streaming HTTP streaming
- 6. The scary blockdiagram +mSource MP3Decoder FileSource MPEG4Source MediaSource +mAudioSource DataSource +mVideoSource HTTPDataSour AVCDecoder ce +mSource MP3Source MediaExtractor OMXCodec AwesomePlayer IOMX MPEG4Extractor MP3Extractor TimedEventQueue OMX OMXClient AwesomeRenderer AudioPlayer AudioTrack AwesomeRem AwesomeLocalR (media OMXMaster IOMXRender framework) oteRenderer enderer 0...n TimeSource MediaPlayerBase:AudioSink SoftwareRen (media framework) OMXPluginBase OMXRender VideoRender der SystemTimeSource MediaPlayerInterface OMXPVCodecs HardwareRe (media framework) iSurface OMXPlugin StageFrightPlayer Plugin nder (surfaceflinger)
- 7. Components DataSource Media Extractor Media Source Media Buffer Meta Data OMXCodec Camera Source Media Writer
- 8. File playback Awesome Player File playback HTTP RTSP Components Timed queue AV sync
- 9. Caveats AV synch tweaks. Fast forward/rewind Codec configuration
- 10. Recording Consists of CameraSource, OMXCodec, MPEG4Writer.
- 11. Caveats Exchanging buffers with the camera Hardware codec integration and the troubles
- 12. Streaming support RTSP streaming – the basics
- 13. Thread Alooper:Looper Thread ALooper HandlerI AMessage ALooperRoster nfo 0..n registerHandler() mTarget post() registerHandler() unregisterHandler() postMessage() AHandler deliverMessage() onMessageReceived() MediaExtra looper() ctor ARTPAssembler +mNotifyMsg ARTSPCont AHandlerR ARTSPCon ARTPConne ARTPSource roller eflector<T> MyHan nection ction dler assembleMore() AMPEG4Audio Assembler 0..n +mNotifyMsg ARTPSe AAMRAss AAVCAssem AH263Asse AMPEG4Element MediaSource APacketSource TrackInfo aryAssembler ssion embler bler mbler MediaWriter ARTPWriter
- 14. Components ARTSPController ARTSPConnection ARTPConnection ARTPSource ARTPAssembler AAMRAssembler is an example
- 15. Gaps and furtherwork Jitter buffer Error correction and feedback RTCP handling
- 16. Hardware rendering VideoRenderer class Instantiating Hardware overlay
- 17. Tweaking and optimizing Copies Latency Slice mode Signaling errors from decoder
- 18. References http://freepine.blogspot.com/2010/01/overview-of- stagefrighter-player.html http://freepine.blogspot.com/2010/09/rtsp-in- stagefright-1.html http://freepine.blogspot.com/2010/09/rtsp-in- stagefright-2.html http://iamkcspa.pixnet.net/blog/category/1793816
- 19.
Editor's Notes
- #5 Give overview of multimedia frameworks and its use.
- #6 Mention about opencore and the transition from opencore to stagefright in froyo.Mention about move to OpenMax AL as an alternative in Ice cream sandwich.
- #8 DataSource – Baseclass for all data source. Sniffers are registered from here. This is what is used to find file/mime types. For e.g., FileExtractor, MPEG4DataSource etc.MediaExtractor – Talks to Data sources and parses data to extract media content. MPEG4Extractor is an example of this.MediaSource – The source of media data flowing through stagefright. Every component which needs to exchange media data have to be of this type. MPEG4Source is an example.MediaBuffer – object to pass data around. Supports reference counting, MIME types, timestamps etc. Also of interest is the MediaBufferGroup.MetaData – the ambassador. The structure used to exchange info b.w two components. Config info, Mime type, etcOMXCodec – the wrapper over the encoder and decoder components. Controls the whole life cycle of the OpenMax Codec (mention the life stages)CameraSource – Is the source node which feeds data for video recording. MediaWriter – sink node and the base class for all writers
- #9 Give overview of the flow from the top level media APIs to the bottom File playback is a sequence of initializing input nodes, finding data types, instantiating necessary decoders and using this info to instantiate renderers. A decode sequence involves extractors extracting the buffers and this gets is passed onto the OMXCodec. After decoding is complete, awesome renderer renders it using the renderer.TimedQueue is what is used to sleep on and achieve AV Sync. The clock gets updated from audio player depending on the amount of data played back.
- #11 What is important here is the passing around of config dataGive example of SKYPE
- #13 Will be describing the basics behind RTSP streaming here. The blocks involved.Has support for AMR, H.263, H.264, MPEG4.
- #15 ARTSP Controller is the media extractor, and the RTP stuff is taken care by ARTPConnection or ARTPSource. RTSP stuff is handled ARTSPConnection. Mention about looper and message posting.ARTSPConnection takes care of sending requests and receiving responses, ARTPConnection receives the packet, but the parsing is delegated to ARTPAssembler which assembles it using the codec specific assembler, through the ARTPSource.ARTPSession has a track info and the data is queued