The document provides an overview of Android's multimedia framework. It discusses the architecture basics, including the display core, multimedia architecture, app development model, kernel drivers, HAL definitions and modules, system services like Stagefright, relevant apps, and references. The display core section specifically describes the display hardware, stack, kernel driver, HAL definition and module, SurfaceFlinger, and WindowManager.

1. 1
Android's Multimedia
Framework
September 2014
Karim Yaghmour
@karimyaghmour
karim.yaghmour@opersys.com

2. 2
These slides are made available to you under a Creative Commons Share-
Alike 3.0 license. The full terms of this license are here:
https://creativecommons.org/licenses/by-sa/3.0/
Attribution requirements and misc., PLEASE READ:
● This slide must remain as-is in this specific location (slide #2), everything
else you are free to change; including the logo :-)
● Use of figures in other documents must feature the below “Originals at”
URL immediately under that figure and the below copyright notice where
appropriate.
● You are free to fill in the “Delivered and/or customized by” space on the
right as you see fit.
● You are FORBIDEN from using the default “About” slide as-is or any of its
contents.
●
You are FORBIDEN from using any content provided by 3rd
parties without
the EXPLICIT consent from those parties.
(C) Copyright 2013-2014, Opersys inc.
These slides created by: Karim Yaghmour
Originals at: www.opersys.com/community/docs
Delivered and/or customized by

3. 3
About
● Author of:
● Introduced Linux Trace Toolkit in 1999
● Originated Adeos and relayfs (kernel/relay.c)
● Training, Custom Dev, Consulting, ...

4. 4
Agenda
● Architecture Basics
● Display Core
● Multimedia Architecture
● App Dev Model
● Kernel Drivers
● HAL definitions
● HAL modules
● System Services
● Stagefright
● Walkthrough
● Relevant Apps
● References and Pointers

5. 5
1. Architecture Basics
● Hardware used to run Android
● AOSP
● Binder
● System Services
● HAL

6. 6

7. 7

8. 8

9. 9

10. 10

11. 11
/frameworks/base/services/java/...
/frameworks/base/services/jni/
/hardware/libhardware/
/device/[MANUF.]/[DEVICE]
/sdk/emulator/
Kernel or module
/frameworks/base/core/...
AOSP-provided
ASL
Manuf.-provided
Manuf. license
Manuf.-provided
GPL-license

12. 12
2. Display Core
● Display Hardware
● Display stack in Android
● Kernel driver
● HAL definition
● HAL module
● Surface Flinger
● Window Manager

13. 13
2.1. Display Hardware
MMU
IOMMU

14. 14
2.2. Display stack in Android

15. 15

16. 16
From source.android.com - CC-BY 2.5

17. 17
From source.android.com - CC-BY 2.5

18. 18
2.3. Kernel driver
●
Video memory management
●
Mode setting
● Checking of parameters
● Motorola Xoom:
●
/dev/nvhdcp1
● /dev/nvhost-ctrl
● /dev/nvhost-display
● /dev/nvhost-dsi
●
/dev/nvhost-gr2d
● /dev/nvhost-gr3d
●
/dev/nvhost-isp
●
/dev/nvhost-mpe
● /dev/nvhost-vi
● /dev/nvmap
● /dev/tegra-crypto
●
/dev/tegra_avp
●
/dev/tegra_camera
●
/dev/tegra_fuse
● /dev/tegra_rpc
● /dev/tegra_sema
● ... whatever hides in hwcomposer HAL module

19. 19
2.3. HAL Definition
● /hardware/libhardware/include/hardware/hwcomposer.h
● struct hwc_procs:
● invalidate()
● vsync()
● struct hwc_composer_device:
● prepare()
● set()
● dump()
● registerProcs()
● query()
● *()

20. 20
2.5. HAL module
● Skeleton /hardware/libhardware/modules/hwcomposer.cpp
● /system/lib/hw/hwcomposer.BOARD.so
● /system/lib/hw/gralloc.BOARD.so
● Ex. - Mot Xoom:
● hwcomposer.tegra.so
● gralloc.tegra.so
● Surface Flinger hook:
● /frameworks/native/services/surfaceflinger/DisplayHardware
– HWComposer.cpp
– Provides fake vsync if none is provided in HW

21. 21
2.6. Surface Flinger
● Actual server:
● /frameworks/native/services/surfaceflinger
● Client side:
● /frameworks/native/libs/gui
● Client / Server interface:
● ISurfaceComposerClient.cpp
● ISurfaceComposer.cpp
● This is NOT an aidl'ed service
● All communication is manually
marshalled/unmarshalled

22. 22
2.7. Window Manager
● Server side:
● /frameworks/base/services/java/com/android/server/wm/
– WindowManagerService.java
– Session.java
● Client side:
● /frameworks/base/core/java/android/view/
– WindowManager.java
– WindowManagerImpl.java
– ViewRootImpl.java
● Interfaces:
● IWindowManager.aidl
● IWindowSession.aidl
● Parameters (incl. z-order):
● See WindowManager.java

23. 23
3. Multimedia Architecture
● OpenMAX
● Media Playback
● Audio
● Camera

24. 24
3.1. OpenMAX
From Khronos website

25. 25
3.2. Media Playback
From source.android.com - CC-BY 2.5

26. 26
3.3. Audio
From source.android.com - CC-BY 2.5

27. 27
3.4. Camera
From source.android.com - CC-BY 2.5

28. 28
From source.android.com - CC-BY 2.5

29. 29
From source.android.com - CC-BY 2.5

30. 30
4. App Dev Model
● Media Playback
● Audio Record
● Camera

31. 31
4.1. Media Playback
● developer.android.com sample:
Uri myUri = ....; // initialize Uri here
MediaPlayer mediaPlayer = new MediaPlayer();
mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);
mediaPlayer.setDataSource(getApplicationContext(), myUri);
mediaPlayer.prepare();
mediaPlayer.start();

32. 32
From developer.android.com - ASL

33. 33
4.2. Audio Record
● developer.android.com steps:
● Create a new instance of android.media.MediaRecorder.
● Set the audio source using MediaRecorder.setAudioSource(). You will probably
want to use MediaRecorder.AudioSource.MIC.
● Set output file format using MediaRecorder.setOutputFormat().
● Set output file name using MediaRecorder.setOutputFile().
● Set the audio encoder using MediaRecorder.setAudioEncoder().
● Call MediaRecorder.prepare() on the MediaRecorder instance.
● To start audio capture, call MediaRecorder.start().
● To stop audio capture, call MediaRecorder.stop().
● When you are done with the MediaRecorder instance, call
MediaRecorder.release() on it. Calling MediaRecorder.release() is always
recommended to free the resource immediately.

34. 34
From developer.android.com - ASL

35. 35
5. Kernel drivers
● Possibly V4Linux
● Possibly ALSA
● ... whatever hides in multimedia HAL modules

36. 36
6. HAL Definition
● /hardware/libhardware/include/hardware/:
● audio_effect.h
● audio.h
● audio_policy.h
● camera.h
● camera2.h
● camera3.h
● camera_common.h

37. 37
7. HAL module
● Audio skeletons in /hardware/libhardware/modules/audio
● USB audio in /hardware/libhardware/modules/usbaudio
● Default implementation in /hardware/libhardware/modules/camera
● /system/lib/hw/:
● audio.a2dp.default.so
● audio_policy.default.so
● audio_policy.mpq8064.so
● audio_policy.msm8960.so
● audio.primary.default.so
● audio.primary.mpq8064.so
● audio.primary.msm8960.so
● audio.usb.default.so
● camera.msm8960.so

38. 38
8. System Services
● Player
● Audio Flinger
● Audio Policy
● Camera

39. 39
8.1. Player
● Actual server:
● frameworks/av/media/libmediaplayerservice
● Client side:
● frameworks/av/media/libmedia
● Client / Server interface:
● IMediaPlayerClient.cpp
● IMediaPlayer.cpp
● IMediaPlayerService.cpp
● IMediaRecorderClient.cpp
● IMediaRecorder.cpp

40. 40
8.2. Audio Flinger
● Actual Server:
● frameworks/av/services/audioflinger
● Client side:
● frameworks/av/media/libmedia
● Client / Server interface:
● IAudioFlingerClient.cpp
● IAudioFlinger.cpp
● IAudioRecord.cpp
● iAudioTrack.cpp

41. 41
8.3. Audio Policy
● Actual Server:
● frameworks/av/services/audioflinger
● Client side:
● frameworks/av/media/libmedia
● Client / Server interface:
● IAudioPolicyService.cpp

42. 42
8.4. Camera
●
Actual Server:
● frameworks/av/services/camera/libcameraservice
● Client Side:
● frameworks/av/camera
● Client / Server interface:
● ICameraClient.cpp
● ICamera.cpp
● ICameraRecordingProxy.cpp
● ICameraRecordingProxyListener.cpp
● ICameraService.cpp
● ICameraServiceListener.cpp
● IProCameraCallbacks.cpp
● IProCameraUser.cpp

43. 43
8.5. Wrapper process
● All media system services run within
“mediaserver” system process.
● Started by init
● Sources:
● frameworks/av/media/mediaserver

44. 44
9. Stagefright
● Multimedia framework
● Houses AwesomePlayer
● Location:
● frameworks/av/media/libstagefright
● Support loadable HW codecs:
● libstagefrighthw.so

45. 45
10. Walkthrough

46. 46
10. Relevant Apps
● Music
● MusicFX
● Gallery2
● SpeechRecorder
● VideoEditor

47. 47
11. References and Pointers
● Google's “official” doumentation:
source.android.com/devices/media.html
source.android.com/devices/audio.html
source.android.com/devices/camera/camera.html
● “Use the source, Luke”
●
Developer doc:
● https://developer.android.com/guide/topics/media/mediaplayer.html
● https://developer.android.com/reference/android/media/MediaPlayer.html
● https://developer.android.com/guide/topics/media/audio-capture.html
● “Android Multimedia Framework”, picker - slideshare
●
“The Android media framework”, Bert Van Dam & Poornachandra Kallare, Android Builders Summit
2014
●
“Android’s Stagefright Media Player Architecture”, Ivo Krka,
http://quandarypeak.com/2013/08/androids-stagefright-media-player-architecture/
●
“Mastering the Android Media Framework”, Google I/O 2009
● https://www.khronos.org/openmax/

48. 48
Thank you ...
karim.yaghmour@opersys.com

49. 49
Acknowledgements:
● Some figures and snippets taken from Google's
Android documentation at
developer.android.com and source.android.com
distributed under the Apache 2.0 license and/or
CC-BY-SA 2.5.