A slightly deeper dive into Stagefright


Published on

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Give overview of multimedia frameworks and its use.
  • Mention about opencore and the transition from opencore to stagefright in froyo.Mention about move to OpenMax AL as an alternative in Ice cream sandwich.
  • DataSource – Baseclass for all data source. Sniffers are registered from here. This is what is used to find file/mime types. For e.g., FileExtractor, MPEG4DataSource etc.MediaExtractor – Talks to Data sources and parses data to extract media content. MPEG4Extractor is an example of this.MediaSource – The source of media data flowing through stagefright. Every component which needs to exchange media data have to be of this type. MPEG4Source is an example.MediaBuffer – object to pass data around. Supports reference counting, MIME types, timestamps etc. Also of interest is the MediaBufferGroup.MetaData – the ambassador. The structure used to exchange info b.w two components. Config info, Mime type, etcOMXCodec – the wrapper over the encoder and decoder components. Controls the whole life cycle of the OpenMax Codec (mention the life stages)CameraSource – Is the source node which feeds data for video recording. MediaWriter – sink node and the base class for all writers
  • Give overview of the flow from the top level media APIs to the bottom File playback is a sequence of initializing input nodes, finding data types, instantiating necessary decoders and using this info to instantiate renderers. A decode sequence involves extractors extracting the buffers and this gets is passed onto the OMXCodec. After decoding is complete, awesome renderer renders it using the renderer.TimedQueue is what is used to sleep on and achieve AV Sync. The clock gets updated from audio player depending on the amount of data played back.
  • What is important here is the passing around of config dataGive example of SKYPE
  • Will be describing the basics behind RTSP streaming here. The blocks involved.Has support for AMR, H.263, H.264, MPEG4.
  • ARTSP Controller is the media extractor, and the RTP stuff is taken care by ARTPConnection or ARTPSource. RTSP stuff is handled ARTSPConnection. Mention about looper and message posting.ARTSPConnection takes care of sending requests and receiving responses, ARTPConnection receives the packet, but the parsing is delegated to ARTPAssembler which assembles it using the codec specific assembler, through the ARTPSource.ARTPSession has a track info and the data is queued
  • A slightly deeper dive into Stagefright

    1. 1. Stagefright – how it works Alexy Mathew Joseph
    2. 2. Who am I? Works at Pathpartner(pathpartnertech.com) as a Technical Architect. Working on multimedia frameworks and systems optimization for 8 years. Contact: galahad@gmail.com
    3. 3. Flow Overview Architecture File playback Recording Streaming Hardware accelerated rendering Tweaking and optimizing Questions?
    4. 4. Overview Multimedia frameworks - the weight lifters
    5. 5. Stagefright The primary multimedia framework available on Android Feature set  Supports playback of multiple file formats  Integrates with OpenMax codecs  Recording  RTSP streaming  HTTP streaming
    6. 6. The scary block diagram +mSource MP3Decoder FileSource MPEG4Source MediaSource +mAudioSource DataSource +mVideoSource HTTPDataSour AVCDecoder ce +mSource MP3Source MediaExtractor OMXCodec AwesomePlayer IOMX MPEG4Extractor MP3Extractor TimedEventQueue OMX OMXClient AwesomeRenderer AudioPlayer AudioTrack AwesomeRem AwesomeLocalR (media OMXMaster IOMXRender framework) oteRenderer enderer 0...n TimeSource MediaPlayerBase:AudioSink SoftwareRen (media framework)OMXPluginBase OMXRender VideoRender der SystemTimeSource MediaPlayerInterfaceOMXPVCodecs HardwareRe (media framework) iSurface OMXPlugin StageFrightPlayer Plugin nder (surfaceflinger)
    7. 7. Components DataSource Media Extractor Media Source Media Buffer Meta Data OMXCodec Camera Source Media Writer
    8. 8. File playback Awesome Player  File playback  HTTP  RTSP Components  Timed queue  AV sync
    9. 9. Caveats AV synch tweaks. Fast forward/rewind Codec configuration
    10. 10. Recording Consists of CameraSource, OMXCodec, MPEG4Writer.
    11. 11. Caveats Exchanging buffers with the camera Hardware codec integration and the troubles
    12. 12. Streaming support RTSP streaming – the basics
    13. 13. Thread Alooper:Looper Thread ALooper HandlerI AMessage ALooperRoster nfo 0..n registerHandler() mTarget post() registerHandler() unregisterHandler() postMessage() AHandler deliverMessage() onMessageReceived()MediaExtra looper() ctor ARTPAssembler +mNotifyMsgARTSPCont AHandlerR ARTSPCon ARTPConne ARTPSource roller eflector<T> MyHan nection ction dler assembleMore() AMPEG4Audio Assembler 0..n +mNotifyMsg ARTPSe AAMRAss AAVCAssem AH263Asse AMPEG4ElementMediaSource APacketSource TrackInfo aryAssembler ssion embler bler mbler MediaWriter ARTPWriter
    14. 14. Components ARTSPController ARTSPConnection ARTPConnection ARTPSource ARTPAssembler  AAMRAssembler is an example
    15. 15. Gaps and further work Jitter buffer Error correction and feedback RTCP handling
    16. 16. Hardware rendering VideoRenderer class Instantiating Hardware overlay
    17. 17. Tweaking and optimizing Copies Latency Slice mode Signaling errors from decoder
    18. 18. References http://freepine.blogspot.com/2010/01/overview-of- stagefrighter-player.html http://freepine.blogspot.com/2010/09/rtsp-in- stagefright-1.html http://freepine.blogspot.com/2010/09/rtsp-in- stagefright-2.html http://iamkcspa.pixnet.net/blog/category/1793816
    19. 19. Questions?