A slightly deeper dive into Stagefright

22,323 views

Published on

A slightly deeper dive into Stagefright

  1. 1. Stagefright – how it works Alexy Mathew Joseph
  2. 2. Who am I? Works at Pathpartner(pathpartnertech.com) as a Technical Architect. Working on multimedia frameworks and systems optimization for 8 years. Contact: galahad@gmail.com
  3. 3. Flow Overview Architecture File playback Recording Streaming Hardware accelerated rendering Tweaking and optimizing Questions?
  4. 4. Overview Multimedia frameworks - the weight lifters
  5. 5. Stagefright The primary multimedia framework available on Android Feature set  Supports playback of multiple file formats  Integrates with OpenMax codecs  Recording  RTSP streaming  HTTP streaming
  6. 6. The scary block diagram +mSource MP3Decoder FileSource MPEG4Source MediaSource +mAudioSource DataSource +mVideoSource HTTPDataSour AVCDecoder ce +mSource MP3Source MediaExtractor OMXCodec AwesomePlayer IOMX MPEG4Extractor MP3Extractor TimedEventQueue OMX OMXClient AwesomeRenderer AudioPlayer AudioTrack AwesomeRem AwesomeLocalR (media OMXMaster IOMXRender framework) oteRenderer enderer 0...n TimeSource MediaPlayerBase:AudioSink SoftwareRen (media framework)OMXPluginBase OMXRender VideoRender der SystemTimeSource MediaPlayerInterfaceOMXPVCodecs HardwareRe (media framework) iSurface OMXPlugin StageFrightPlayer Plugin nder (surfaceflinger)
  7. 7. Components DataSource Media Extractor Media Source Media Buffer Meta Data OMXCodec Camera Source Media Writer
  8. 8. File playback Awesome Player  File playback  HTTP  RTSP Components  Timed queue  AV sync
  9. 9. Caveats AV synch tweaks. Fast forward/rewind Codec configuration
  10. 10. Recording Consists of CameraSource, OMXCodec, MPEG4Writer.
  11. 11. Caveats Exchanging buffers with the camera Hardware codec integration and the troubles
  12. 12. Streaming support RTSP streaming – the basics
  13. 13. Thread Alooper:Looper Thread ALooper HandlerI AMessage ALooperRoster nfo 0..n registerHandler() mTarget post() registerHandler() unregisterHandler() postMessage() AHandler deliverMessage() onMessageReceived()MediaExtra looper() ctor ARTPAssembler +mNotifyMsgARTSPCont AHandlerR ARTSPCon ARTPConne ARTPSource roller eflector<T> MyHan nection ction dler assembleMore() AMPEG4Audio Assembler 0..n +mNotifyMsg ARTPSe AAMRAss AAVCAssem AH263Asse AMPEG4ElementMediaSource APacketSource TrackInfo aryAssembler ssion embler bler mbler MediaWriter ARTPWriter
  14. 14. Components ARTSPController ARTSPConnection ARTPConnection ARTPSource ARTPAssembler  AAMRAssembler is an example
  15. 15. Gaps and further work Jitter buffer Error correction and feedback RTCP handling
  16. 16. Hardware rendering VideoRenderer class Instantiating Hardware overlay
  17. 17. Tweaking and optimizing Copies Latency Slice mode Signaling errors from decoder
  18. 18. References http://freepine.blogspot.com/2010/01/overview-of- stagefrighter-player.html http://freepine.blogspot.com/2010/09/rtsp-in- stagefright-1.html http://freepine.blogspot.com/2010/09/rtsp-in- stagefright-2.html http://iamkcspa.pixnet.net/blog/category/1793816
  19. 19. Questions?

×