Outline: a. MediaPlayer Subsystem b. Related Files c. MediaPlayer Frame of Playing Flow -StageFright and AwesomePlayer Relatin -AwesomePlayer Frame and Playing Flow d. Simple Playing Implement

1. Confidential
Media Playback Flow
Javid Hsu

2. Confidential
Outline
 MediaPlayer Subsystem
 Related Files
 MediaPlayer Frame of Playing Flow
─ StageFright and AwesomePlayer Relatin
─ AwesomePlayer Frame and Playing Flow
 Simple Playing Implement

3. Confidential
Binder
MediaPlayer Subsystem
Java
MediaPlayer
(frameworks/base/media/java/android/media)
libmedia_jni.so
JNI
libmedia.so
(frameworks/av/media/libmedia)
libmediaplayerservice.so
(frameworks/av/media/
libmediaplayerservice)
libstagefright.so
libstagefright_omx.so
libstagefright_soft_XXX.so
(codec)
(frameworks/av/media/
libstagefright/*)
(frameworks/base/media/jni)
libaudioflinger.so
(frameworks/av/services/
audioflinger)
libstagefrighthw.so
libOMXVideoEncoderXX.so
libOMXVideoDecoderXX.so
(vendor codec (hw codec))

4. Confidential
Related Files
libmedia_jni.so (frameworks/base/media/jni) libmedia.so (frameworks/av/media/libmedia)

5. Confidential
Related Files
libmediaplayerservice.so
(frameworks/av/media/libmediaplayerservice)
libstagefright.so
(frameworks/av/media/libstagefright)

6. Confidential
Related Files
libaudioflinger.so
(frameworks/av/services/audioflinger)

7. Confidential
MediaPlayer Frame of Playing Flow
 Stagefright and AwesomePlayer Relation

8. Confidential
MediaPlayer Frame of Playing Flow
Ref: http://blog.csdn.net/tx3344/article/details/8084912
SurfaceFlinger
AudioFlinger
 AwesomePlayer Frame and Playing Flow
Ref: http://blog.csdn.net/tx3344/article/details/8109185

9. Confidential
Simple Playing Implement
//For example:
//Simple Playing Implement:
MediaPlayer mp = new MediaPlayer();
mp.setDataSource(PATH_TO_FILE);
mp.prepare();
mp.start();
Ref: http://developer.android.com/reference/android/media/MediaPlayer.html
 MediaPlayer State Diagram

10. Confidential
Simple Playing Implement
//For example:
//Simple Playing Implement:
MediaPlayer mp = new MediaPlayer();
mp.setDataSource(PATH_TO_FILE);
mp.prepareAsync();
mp.start();

11. Confidential
New MediaPlayer
Applications
Applications Framework
• Initiation and Setup
Media Player
Application
android_media_MediaPlayer
MediaPlayer.java
android_media_MediaPlayer_native_init()
android_media_MediaPlayer_native_setup()
Static{
System.loadLibrary("media_jni");
native_init();
}
static void android_media_MediaPlayer_native_init (JNIEnv *env){
clazz = env->FindClass("android/media/MediaPlayer");
:
fields.post_event = env->GetStaticMethodID(clazz, "postEventFromNative",
"(Ljava/lang/Object;IIILjava/lang/Object;)V");
:
fields.context = env->GetFieldID(clazz, "mNativeContext", "I");
:
}
static void android_media_MediaPlayer_native_setup (JNIEnv *env, jobject thiz, jobject weak_this)
{
:
sp<MediaPlayer> mp = new MediaPlayer();
// create new listener and give it to MediaPlayer
sp<JNIMediaPlayerListener> listener = new JNIMediaPlayerListener(env, thiz, weak_this);
mp->setListener(listener);
// Stow our new C++ MediaPlayer in an opaque field in the Java object.
setMediaPlayer(env, thiz, mp);
}
static sp<MediaPlayer> setMediaPlayer (JNIEnv* env, jobject thiz, …)
{
Mutex::Autolock l(sLock);
:
env->SetIntField(thiz, fields.context, (int)player.get());
:
}

12. Confidential
Simple Playing Implement
//For example:
//Simple Playing Implement:
MediaPlayer mp = new MediaPlayer();
mp.setDataSource(PATH_TO_FILE);
mp.prepareAsync();
mp.start();

13. Confidential
Libraries
(libmedia.so)
MediaPlayer
player->setDataSource
IMediaPlayer
Media Player
Application
android_media_MediaPlayer
MediaPlayer.java
Applications
Applications Framework
MediaPlayerService::
Client
Media Set Data Source Flow (setDataSource)
 SetDataSource
 Using file type to choose
 setDataSource_pre(player_type
playerType)…..
 Using real player to set data source
setDataSource()
android_media_MediaPlayer_setDataSourceFD()
setDataSource()
status_t MediaPlayerService::Client::setDataSource(int fd, int64_t offset,
int64_t length)
{
:
player_type playerType = MediaPlayerFactory::getPlayerType(this,
fd,
offset,
length);
sp<MediaPlayerBase> p = setDataSource_pre(playerType);
if (p == NULL) {
return NO_INIT;
}
:
// now set data source
setDataSource_post(p, p->setDataSource(fd, offset, length));
return mStatus;
}

14. Confidential
Media Set Data Source Flow (setDataSource)
Get Player Type
Create Player
Instance
Player Set Data
Source
 Using file type to get player type
 get real player instance
• Load software and hardware codec
• Register default sniffers
• Create AwesomeEvents of onVideoEvent, onStreamDone, onBufferingUpdate,
onVideoLagUpdate, onThumbnailEvent, onCheckAudioStatus and
onAudioTearDownEvent.
• setNotifyCallback
 Using real player to set data source
• new FileSource(fd, …) insert to MediaExtractor::Create()
─ Sniff DataSource and get mine type
─ Get extractor with mime type
• Separate video (sp<MediaSource> mVideoTrack) and
audio (sp<MediaSource> mVideoSource) track

15. Confidential
Media Set Data Source Flow (setDataSource)
 Using file type to get player type
 get real player instance
• Load software and hardware codec
• Register default sniffers
• Create AwesomeEvents of onVideoEvent, onStreamDone, onBufferingUpdate,
onVideoLagUpdate, onThumbnailEvent, onCheckAudioStatus and
onAudioTearDownEvent.
• setNotifyCallback
 Using real player to set data source
• new FileSource(fd, …) insert to MediaExtractor::Create()
─ Sniff DataSource and get mine type
─ Get extractor with mime type
• Separate video (sp<MediaSource> mVideoTrack) and
audio (sp<MediaSource> mVideoSource) track
Get Player Type
Create Player
Instance
Player Set Data
Source

16. Confidential
Libraries
Media Set Data Source Flow (setDataSource)
Using file type to get player type
MediaPlayerFactory.cpp
-factory: IFactory*
-sFactoryMap
NuPlayerFactory
+createPlayer(): return new NuPlayerDriver
SonivoxPlayerFactory
+createPlayer(): return new MidiFile
TestPlayerFactory
+createPlayer(): return new TestPlayerStub
StagefrightPlayerFactory
+createPlayer(): return new StagefrightPlayer
(libmedia.so)
MediaPlayer
player->setDataSource
MediaPlayerFactory
IMediaPlayer
Media Player
Application
android_media_MediaPlayerMediaPlayer.java
Applications
Applications Framework
MediaPlayerService::
Client
a. playerType:getPlayerType()
MediaPlayerFactory:
StagefrightPlayerFactory
createPlayer()
return new StagefrightPlayer();
setDataSource()
android_media_MediaPlayer_setDataSourceFD()
setDataSource()
b. sp<MediaPlayerBase>:
createPlayer(playerType)
a. getPlayerType:
1. sFactoryMap add all player factory while MediaPlayerService
started.
2. scoreFactory function using compare file
to get corresponding player type:
a. NuPlayerFactory
if(!strncasecmp("http://", url, 7)
|| !strncasecmp("https://", url, 8)
|| !strncasecmp("file://", url, 7))
b. SonivoxPlayerFactory
if (!strncasecmp(url + start, FILE_EXTS[i], len))
static const char* const FILE_EXTS[] = { ".mid",
".midi",
".smf",
".xmf",
".mxmf",
".imy",
".rtttl",
".rtx",
".ota" };
c. StagefrightPlayerFactory
1. compare raw data header if is equal to 0x5367674f
2. Default Player Type

17. Confidential
Media Set Data Source Flow (setDataSource)
Get Player Type
 Using file type to get player type
 get real player instance
• Load software and hardware codec
• Register default sniffers
• Create AwesomeEvents of onVideoEvent, onStreamDone, onBufferingUpdate,
onVideoLagUpdate, onThumbnailEvent, onCheckAudioStatus and
onAudioTearDownEvent.
• setNotifyCallback
 Using real player to set data source
• new FileSource(fd, …) insert to MediaExtractor::Create()
─ Sniff DataSource and get mine type
─ Get extractor with mime type
• Separate video (sp<MediaSource> mVideoTrack) and
audio (sp<MediaSource> mVideoSource) track
Get Player Type
Create Player
Instance
Player Set Data
Source

18. Confidential
Media Set Data Source Flow (setDataSource)
Media Player
Application
Applications
android_media_MediaPlayer
Applications Framework
MediaPlayer.java
Libraries
• Load software and hardware codec
StagefrightPlayerMediaPlayerFactory
new StagefrightPlayer()
connect()
OMXClient
OMX
OMXMaster
OMX()
mMaster(new OMXMaster)
Software CodecHardware Codec
SoftOMXPlugin
AwesomePlayer
new AwesomePlayer()
libstagefrighthw.so

19. Confidential
Media Set Data Source Flow (setDataSource)
Media Player
Application
Applications
android_media_MediaPlayer
Applications Framework
MediaPlayer.java
Libraries
StagefrightPlayerMediaPlayerFactory
new StagefrightPlayer()
RegisterDefaultSniffers()
AwesomePlayer
new AwesomePlayer()
RegisterSniffer_l(SnifferFunc func)
MPEG4Extractor.cpp
MP3Extractor.cpp
bool SniffMP3(...)
:
:
1. SniffMPEG4 <--> MPEG4Extractor
2. SniffMatroska <--> MatroskaExtractor
3. SniffOgg <--> OggExtractor
4. SniffWAV <--> WAVExtractor
5. SniffFLAC <--> FLACExtractor
6. SniffAMR <--> AMRExtractor
7. SniffMPEG2TS <--> MPEG2TSExtractor
8. SniffMP3 <--> MP3Extractor
9. SniffAAC <--> AACExtractor
10. SniffMPEG2PS <--> MPEG2PSExtractor
11. SniffWVM <--> WVMExtractor
12. SniffAVI <--> AVIExtractor
All Sniff register in gSniffers
(List<SnifferFunc> gSniffers)
bool SniffMPEG4(...)
typedef bool (*SnifferFunc)(
const sp<DataSource> &source, String8 *mimeType,
float *confidence, sp<AMessage> *meta);
DataSource
• Register default sniffers
FileSource
DataSource
- static List<SnifferFunc> gSniffers
+static sp<DataSource> CreateFromUR()
+static void RegisterDefaultSniffers()
+static void RegisterSniffer_l()
+typedef bool (*SnifferFunc)()
+bool sniff()
+bool getUInt16()
+bool getUInt64()
+virtual ssize_t readAt()
+virtual String8 getMIMEType() const()
HTTPBase NuCachedSource2
:

20. Confidential
Media Set Data Source Flow (setDataSource)
Media Player
Application
Applications
android_media_MediaPlayer
Applications Framework
MediaPlayer.java
Libraries
StagefrightPlayerMediaPlayerFactory
new StagefrightPlayer() new AwesomePlayer()
AwesomePlayer
mVideoEvent
= new AwesomeEvent(this, &AwesomePlayer::onVideoEvent);
mStreamDoneEvent
= new AwesomeEvent(this, &AwesomePlayer::onStreamDone);
mBufferingEvent
= new AwesomeEvent(this, &AwesomePlayer::onBufferingUpdate);
mVideoLagEvent
= new AwesomeEvent(this, &AwesomePlayer::onVideoLagUpdate);
mThumbnailEvent
= new AwesomeEvent(this, &AwesomePlayer::onThumbnailEvent);
• Create AwesomeEvents of onVideoEvent …
Ref: http://blog.csdn.net/tx3344/article/details/8053106

21. Confidential
Applications
Applications Framework
Libraries
• set Notify Callback (before new MediaPlayer()) : AP -> HAL
APP : setOn##Listener(m##Listener);
MediaPlayer.java : setOn##Listener(listener){mOn##Listener=listener}
Media Player
Application
android_media_MediaPlayer_native_init(){
:
fields.post_event = env->GetStaticMethodID(clazz, "postEventFromNative",
"(Ljava/lang/Object;IIILjava/lang/Object;)V");
:
}
android_media_MediaPlayer
MediaPlayer.java
Static{
System.loadLibrary("media_jni");
native_init();
}
Media Set Data Source Flow (setDataSource)

22. Confidential
android_media_MediaPlayerMediaPlayer.java
(libmedia.so)
MediaPlayer
android_media_MediaPlayer_native_setup()
• set Notify Callback (new MediaPlayer()) : HAL -> C++
Applications
Applications Framework
Libraries
Media Player
Application
new MediaPlayer()
setListener(const sp<MediaPlayerListener>&listener)
{mListener = listener;}
listener = new JNIMediaPlayerListener(env, thiz, weak_this);
Media Set Data Source Flow (setDataSource)

23. Confidential
Media Player
Application
android_media_MediaPlayerMediaPlayer.java
MediaPlayerService
(libmedia.so)
MediaPlayer
player->setDataSource
MediaPlayerService::
Client
IMediaPlayer
• set Notify Callback (setDataSource(…)) : client -> service
Applications
Applications Framework
sp<Client> c
= new Client( const sp<MediaPlayerService>& service, pid_t pid,
int32_t connId, const p<IMediaPlayerClient>& client,
int audioSessionId, uid_t uid)
{
:
mClient = client;
:
}
service->create(this, mAudioSessionId)
Libraries
setDataSource()
android_media_MediaPlayer_setDataSourceFD()
setDataSource()
Media Set Data Source Flow (setDataSource)

24. Confidential
Libraries
MediaPlayerFactory
a. player_type:getPlayerType()
b. sp<MediaPlayerBase>:createPlayer(playerType, this, notify)
MediaPlayerService::
Client
• set Notify Callback (setDataSource(…)) : service -> player
StagefrightPlayer:public MediaPlayerInterface:
public MediaPlayerBase
void setNotifyCallback( void* cookie, notify_callback_f notifyFunc)
{
Mutex::Autolock autoLock(mNotifyLock);
mCookie = cookie; mNotify = notifyFunc;
}
void sendEvent(int msg, int ext1=0, int ext2=0, const Parcel *obj=NULL)
{
Mutex::Autolock autoLock(mNotifyLock);
if (mNotify) mNotify(mCookie, msg, ext1, ext2, obj);
}

Media Set Data Source Flow (setDataSource)
Media Player
Application
Applications
android_media_MediaPlayer
Applications Framework
MediaPlayer.java

StagefrightPlayer AwesomePlayer
mPlayer(new AwesomePlayer)
mPlayer->setListener(this)

25. Confidential
Libraries
(libmedia.so)
MediaPlayer
IMediaPlayer
Media Player
Application
android_media_MediaPlayer::
JNIMediaPlayerListenerMediaPlayer.java
Applications
Applications Framework
MediaPlayerService::
Client
• set Notify Callback (setDataSource(…)) : send message
AwesomePlayer.cpp
notifyListener_l(int msg, int ext1, int ext2)
{
:
listener->sendEvent(msg, ext1, ext2);
:
}
StagefrightPlayer:
public MediaPlayerInterface:
public MediaPlayerBase
Notify(int msg, …)
sendEvent(int msg, …)
Notify(int msg, …)
Notify((int msg, …)
postEventFromNative(int what, …)
• Media Event Type:
Media Set Data Source Flow (setDataSource)

26. Confidential
Media Set Data Source Flow (setDataSource)
 Using file type to get player type
 get real player instance
• Load software and hardware codec
• Register default sniffers
• Create AwesomeEvents of onVideoEvent, onStreamDone, onBufferingUpdate,
onVideoLagUpdate, onThumbnailEvent, onCheckAudioStatus and
onAudioTearDownEvent.
• setNotifyCallback
 Using real player to set data source
• new FileSource(fd, …) insert to MediaExtractor::Create()
─ Sniff DataSource and get mine type
─ Get extractor with mime type
• Separate video (sp<MediaSource> mVideoTrack) and
audio (sp<MediaSource> mVideoSource) track
Get Player TypeGet Player Type
Create Player
Instance
Player Set Data
Source

27. Confidential
Media Set Data Source Flow (setDataSource)
AwesomePlayer.cpp
setDataSource_l(dataSource)
MediaExtractor.cpp
Create(dataSource)
if (!strcasecmp(mime, MEDIA_MIMETYPE_CONTAINER_MPEG4)
|| !strcasecmp(mime, "audio/mp4")) {
ret = new MPEG4Extractor(source);
:
:
return ret;
DataSource.cpp
sniff(mimeType,…)
setDataSource_l(extractor)
extractor
 Using Real Player Set Data Source
• MediaExtractor::Create(const sp<DataSource> &source,...)
─ Sniff DataSource and get mine type
 Read at filesource
 Compabile ?
 Write mine type
─ Get extractor with mime type

28. Confidential
Media Set Data Source Flow (setDataSource)
─ Sniff DataSource and get mine type
 Read at filesource
 Compabile ?
 Write mine type
─ Get extractor with mime type
 Compabile ?
 Read at filesource: Ex: Golden Flower mp4
 Write mine type

29. Confidential
Media Set Data Source Flow (setDataSource)
 Using Real Player Set Data Source
• Separate video and audio track
Video track
Audio track

30. Confidential
Simple Playing Implement
//For example:
//Simple Playing Implement:
MediaPlayer mp = new MediaPlayer();
mp.setDataSource(PATH_TO_FILE);
mp.prepareAsync();
mp.start();

31. Confidential
AwesomePlayer.cpp
status_t prepareAsync(){
return prepareAsync_l();
}
status_t prepareAsync_l(){
new AsyncPrepareEvent;
postEvent();
return OK;
}
onPrepareAsyncEvent()
 initVideoDecoder
 initAudioDecoder
 notifyVideoSize_l
 notifyListener_l
AwesomePlayer.cpp
status_t prepare(){
prepareAsync_l(…)
condition.wait(Mutex)
}
status_t prepareAsync_l(){
:
:
}
onPrepareAsyncEvent()
 initVideoDecoder
 initAudioDecoder
 condition.broadcast();
• prepare • prepareAsync
 prepare V.S. prepareAsync (AwesomePlayer)
Media Set Data Source Flow (setDataSource)

32. Confidential
Libraries
MediaPlayerService::
Client
(libmedia.so)
MediaPlayer
IMediaPlayer
Media Player
Application
android_media_MediaPlayerMediaPlayer.java
Applications
Applications Framework
Media Set Data Source Flow (setDataSource)
 prepareAsync
prepareAsync()
android_media_MediaPlayer_prepareAsync()
prepareAsync()
mPlayer->prepareAsync()
StagefrightPlayer
prepareAsync() prepareAsync()
AwesomePlayer
AwesomePlayer::
initVideoDecoder()
AwesomePlayer::
initAudioDecoder()
TimedEventQueue::
start()

33. Confidential
Media Set Data Source Flow (setDataSource)
 initVideoDecoder
• How to choose codec
AwesomePlayer
mVideoSource
initVideoDecoder()
findMatchingCodecs(mime, componentName, …)
MediaCodecList : cat /etc/media_codecs.xml
OMXCodec
create(…MetaData, MediaSource, …)
makeComponentInstance(name, callbacks…)
Load the corresponding module and register callback
functions.
Callback Function:
OMX_CALLBACKTYPE OMXNodeInstance::kCallbacks = {
&OnEvent, &OnEmptyBufferDone, &OnFillBufferDone
};
libstagefrighthw.so

34. Confidential
Media Set Data Source Flow (setDataSource)
 Video Playback Flow
Ref: http://www.cnblogs.com/shakin/p/4729534.html

35. Confidential
Simple Playing Implement
//For example:
//Simple Playing Implement:
MediaPlayer mp = new MediaPlayer();
mp.setDataSource(PATH_TO_FILE);
mp.prepareAsync();
mp.start();

36. Confidential
Libraries
MediaPlayerService::
Client
(libmedia.so)
MediaPlayer
IMediaPlayer
Media Player
Application
android_media_MediaPlayer
MediaPlayer.java
Applications
Applications Framework
MediaPlay Start
 Start()
start()
android_media_MediaPlayer_start()
start()
mPlayer->start ()
StagefrightPlayer
start()
play()
AwesomePlayerAudioPlayer
start()
postVideoEvent_l()
AwesomePlayer::
onVideoEvent()

37. Confidential
Video Audio Synchronization
Playing point
Audio Samples
Video Samples
Start Point
The Time in AudioPlayer Real Playing
The Time in Audi Real Playing out
 Playback Time Line:
Playing point
latenessUs (VideoPlayer Lateness Time)

38. Confidential
The End