Sound on Android is a topic that is rarely covered, which is why I wanted to shed some lights in my experience with the sound management APIs on Android. So in this talk, we'll touch on how sounds actually work programmatically, and we'll talk about how to play a sound on Android in the most simple way. We will also cover the principle of sound focus, what it is, how and when to use it. We'll see what we can do with MIDI as well. And finally, we'll go deeper in the rabbit hole and get introduced to the lower levels of sounds processing with the help of OpenSL ES and the SuperPowered library.

1. The Sounds of
Android
Let’s make some noise!
Android Makers 2018

2. Noise generator:
Yannick Lemin
• Android developer freelance
• GDG Brussels Organizer
• @android_leaks
• Noise Maker
• @theyann

3. Disclaimer

4. In the beginning…

5. d

6. So let’s make a fart
app…

7. So let’s make a fart
app…
…in 4 steps

8. Step 1 - Create a
new app

9. Step 1 - Create a
new app

10. Step 1 - Create a
new app
#Kotlin

11. Step 2 - Add a
button to the layout

12. Step 3 - Recording

13. Step 3 - Recording

14. Step 3.5 - Drop the
file in the project

15. Step 4 - Play the Media

16. Step 4 - Play the Media
MediaPlayer

17. Step 4.1 - MediaPlayer
class MainActivity : AppCompatActivity() {
private var player: MediaPlayer? = null
override fun onCreate(savedInstanceState: Bundle?) {
...
findViewById<View>(R.id.button_play).setOnClickListener { play() }
}
override fun onStop() {
super.onStop()
player?.release()
}
private fun play() {
player?.release()
player = MediaPlayer.create(this, R.raw.best_fart_ever).apply {
setOnCompletionListener { release() }
start()
}
}
}

18. MediaPlayer
• Very simple to use
• Exists since API1
• Handles resources, over
the network streaming,
even with DRM
PROS

19. MediaPlayer
• Very simple to use
• Exists since API1
• Handles resources, over
the network streaming,
even with DRM
PROS
• Not extensible
• Dependent on OS for bug
fixes and new features
CONS

20. Step 4 - Play the Media

21. ExoPlayer
Step 4 - Play the Media

22. class MainActivity : AppCompatActivity() {
private lateinit var player: ExoPlayer
override fun onCreate(savedInstanceState: Bundle?) {
...
player = ExoPlayerFactory.newSimpleInstance(this, DefaultTrackSelector())
}
...
private fun play() {
stop()
with (player) {
val source = ExtractorMediaSource.Factory(
DefaultDataSourceFactory(this@MainActivity, "fartheaven")
).createMediaSource(Uri.parse("asset:///best_fart_ever.m4a"))
prepare(source)
playWhenReady = true
}
}
}
Step 4.2 - ExoPlayer

23. ExoPlayer
• Much more flexible and
extensible
• Better suited for complex
use cases
• External library
• Handles caching, adaptive
playback, composition, etc
PROS

24. ExoPlayer
• Much more flexible and
extensible
• Better suited for complex
use cases
• External library
• Handles caching, adaptive
playback, composition, etc
PROS
• More complex to
apprehend
• MinSdk 16 (or 19 for
encryption support)
• Big library
CONS

25. Step 4 - Play the Media

26. Step 4 - Play the Media
SoundPool

27. Step 4 - Play the Media
SoundPool
… a pool full of sound?

28. private fun initSoundPool() {
soundPool = if (Build.VERSION.SDK_INT < Build.VERSION_CODES.LOLLIPOP) {
SoundPool(1, AudioManager.STREAM_MUSIC, 0)
} else {
val attributes = AudioAttributesCompat.Builder()
.setUsage(AudioAttributesCompat.USAGE_MEDIA)
.setContentType(AudioAttributesCompat.CONTENT_TYPE_SONIFICATION)
.build()
SoundPool.Builder()
.setAudioAttributes(attributes.unwrap() as AudioAttributes)
.setMaxStreams(1)
.build()
}
soundPool.setOnLoadCompleteListener({ _, _, status ->
loadComplete = status == 0
if (loadComplete) { play() }
})
soundId = soundPool.load(this, R.raw.best_fart_ever, 1)
}
Step 4.3 - SoundPool

29. private fun initSoundPool() {
soundPool = if (Build.VERSION.SDK_INT < Build.VERSION_CODES.LOLLIPOP) {
SoundPool(1, AudioManager.STREAM_MUSIC, 0)
} else {
val attributes = AudioAttributesCompat.Builder()
.setUsage(AudioAttributesCompat.USAGE_MEDIA)
.setContentType(AudioAttributesCompat.CONTENT_TYPE_SONIFICATION)
.build()
SoundPool.Builder()
.setAudioAttributes(attributes.unwrap() as AudioAttributes)
.setMaxStreams(1)
.build()
}
soundPool.setOnLoadCompleteListener({ _, _, status ->
loadComplete = status == 0
if (loadComplete) { play() }
})
soundId = soundPool.load(this, R.raw.best_fart_ever, 1)
}
Step 4.3 - SoundPool

30. private fun initSoundPool() {
soundPool = if (Build.VERSION.SDK_INT < Build.VERSION_CODES.LOLLIPOP) {
SoundPool(1, AudioManager.STREAM_MUSIC, 0)
} else {
val attributes = AudioAttributesCompat.Builder()
.setUsage(AudioAttributesCompat.USAGE_MEDIA)
.setContentType(AudioAttributesCompat.CONTENT_TYPE_SONIFICATION)
.build()
SoundPool.Builder()
.setAudioAttributes(attributes.unwrap() as AudioAttributes)
.setMaxStreams(1)
.build()
}
soundPool.setOnLoadCompleteListener({ _, _, status ->
loadComplete = status == 0
if (loadComplete) { play() }
})
soundId = soundPool.load(this, R.raw.best_fart_ever, 1)
}
Step 4.3 - SoundPool

31. private fun initSoundPool() {
soundPool = if (Build.VERSION.SDK_INT < Build.VERSION_CODES.LOLLIPOP) {
SoundPool(1, AudioManager.STREAM_MUSIC, 0)
} else {
val attributes = AudioAttributesCompat.Builder()
.setUsage(AudioAttributesCompat.USAGE_MEDIA)
.setContentType(AudioAttributesCompat.CONTENT_TYPE_SONIFICATION)
.build()
SoundPool.Builder()
.setAudioAttributes(attributes.unwrap() as AudioAttributes)
.setMaxStreams(1)
.build()
}
soundPool.setOnLoadCompleteListener({ _, _, status ->
loadComplete = status == 0
if (loadComplete) { play() }
})
soundId = soundPool.load(this, R.raw.best_fart_ever, 1)
}
Step 4.3 - SoundPool

32. private fun play() {
val audioManager = getSystemService(Context.AUDIO_SERVICE) as AudioManager
val actualVolume = audioManager.getStreamVolume(AudioManager.STREAM_MUSIC)
val maxVolume = audioManager.getStreamMaxVolume(AudioManager.STREAM_MUSIC)
val volume = (actualVolume / maxVolume).toFloat()
soundPool.play(soundId, volume, volume, priority, loopCount, speed)
}
Step 4.3 - SoundPool

33. Speaking of
recording …

34. MediaRecorder

35. MediaRecorder
fileName = "${externalCacheDir.absolutePath}/best_fart_ever.3gp"
private fun record() {
recorder = MediaRecorder().apply {
setAudioSource(MediaRecorder.AudioSource.MIC)
setOutputFormat(MediaRecorder.OutputFormat.THREE_GPP)
setOutputFile(fileName)
setAudioEncoder(MediaRecorder.AudioEncoder.AMR_NB)
try {
prepare()
start()
} catch (e: IOException) {
release()
}
}
}

36. To play it back
private fun play() {
player = MediaPlayer().apply {
try {
setOnCompletionListener { this@MainActivity.stop() }
setDataSource(fileName)
prepare()
start()
} catch (e: IOException) {
release()
}
}
}

37. AudioFocus

38. AudioFocus
…or how to make your audio app
not be a jackass and work well
with the platform

39. AudioFocus
Your App
Without

40. AudioFocus
Your App
Without

41. AudioFocus
Your App
Without

42. AudioFocus
Your App
Without

43. AudioFocus
Your App
With

44. AudioFocus
Your App
With

45. AudioFocus
Your App
With

46. AudioFocus
Your App
With

47. AudioFocus
Your App
With

48. 1. You need a
listener
class AudioFocusListener(...) : AudioManager.OnAudioFocusChangeListener {
override fun onAudioFocusChange(focusChange: Int) {
when(focusChange) {
AudioManager.AUDIOFOCUS_GAIN -> {
// start playing or reset volume
}
AudioManager.AUDIOFOCUS_LOSS_TRANSIENT_CAN_DUCK -> {
// reduce volume
}
AudioManager.AUDIOFOCUS_LOSS_TRANSIENT -> {
// pause without dropping focus
}
AudioManager.AUDIOFOCUS_LOSS -> {
// focus was lost
}
}
}
}

49. 1. You need a
listener
class AudioFocusListener(...) : AudioManager.OnAudioFocusChangeListener {
override fun onAudioFocusChange(focusChange: Int) {
when(focusChange) {
AudioManager.AUDIOFOCUS_GAIN -> {
// start playing or reset volume
}
AudioManager.AUDIOFOCUS_LOSS_TRANSIENT_CAN_DUCK -> {
// reduce volume
}
AudioManager.AUDIOFOCUS_LOSS_TRANSIENT -> {
// pause without dropping focus
}
AudioManager.AUDIOFOCUS_LOSS -> {
// focus was lost
}
}
}
}

50. 1. You need a
listener
class AudioFocusListener(...) : AudioManager.OnAudioFocusChangeListener {
override fun onAudioFocusChange(focusChange: Int) {
when(focusChange) {
AudioManager.AUDIOFOCUS_GAIN -> {
// start playing or reset volume
}
AudioManager.AUDIOFOCUS_LOSS_TRANSIENT_CAN_DUCK -> {
// reduce volume
}
AudioManager.AUDIOFOCUS_LOSS_TRANSIENT -> {
// pause without dropping focus
}
AudioManager.AUDIOFOCUS_LOSS -> {
// focus was lost
}
}
}
}

51. 1. You need a
listener
class AudioFocusListener(...) : AudioManager.OnAudioFocusChangeListener {
override fun onAudioFocusChange(focusChange: Int) {
when(focusChange) {
AudioManager.AUDIOFOCUS_GAIN -> {
// start playing or reset volume
}
AudioManager.AUDIOFOCUS_LOSS_TRANSIENT_CAN_DUCK -> {
// reduce volume
}
AudioManager.AUDIOFOCUS_LOSS_TRANSIENT -> {
// pause without dropping focus
}
AudioManager.AUDIOFOCUS_LOSS -> {
// focus was lost
}
}
}
}

52. 1. You need a
listener
class AudioFocusListener(...) : AudioManager.OnAudioFocusChangeListener {
override fun onAudioFocusChange(focusChange: Int) {
when(focusChange) {
AudioManager.AUDIOFOCUS_GAIN -> {
// start playing or reset volume
}
AudioManager.AUDIOFOCUS_LOSS_TRANSIENT_CAN_DUCK -> {
// reduce volume
}
AudioManager.AUDIOFOCUS_LOSS_TRANSIENT -> {
// pause without dropping focus
}
AudioManager.AUDIOFOCUS_LOSS -> {
// focus was lost
}
}
}
}

53. 2. Request the focus
private fun requestAudioFocus() {
audioFocusListener = AudioFocusListener(…)
val requestResult = audioManager.requestAudioFocus(audioFocusListener,
AudioManager.STREAM_MUSIC
AudioManager.AUDIOFOCUS_GAIN)
if (requestResult == AudioManager.AUDIOFOCUS_REQUEST_GRANTED) {
// start playing
} else {
// handle error
}
}

54. 2. Request the focus
… API >= 26
val attributes = AudioAttributesCompat.Builder()
.setContentType(AudioAttributesCompat.CONTENT_TYPE_MUSIC)
.setUsage(AudioAttributesCompat.USAGE_MEDIA)
.build()
// keep the request around
request = AudioFocusRequest.Builder(AudioManager.AUDIOFOCUS_GAIN)
.setAudioAttributes(attributes.unwrap as AudioAttributes)
.setOnAudioFocusChangeListener(audioFocusListener)
.build()
audioManager.requestAudioFocus(request)

55. Content Type and Usage
AudioAttributesCompat.CONTENT_TYPE_SPEECH
AudioAttributesCompat.CONTENT_TYPE_MUSIC
AudioAttributesCompat.CONTENT_TYPE_MOVIE
AudioAttributesCompat.CONTENT_TYPE_SONIFICATION
AudioAttributesCompat.USAGE_MEDIA
AudioAttributesCompat.USAGE_VOICE_COMMUNICATION
AudioAttributesCompat.USAGE_VOICE_COMMUNICATION_SIGNALLING
AudioAttributesCompat.USAGE_ALARM
AudioAttributesCompat.USAGE_NOTIFICATION
AudioAttributesCompat.USAGE_NOTIFICATION_RINGTONE
AudioAttributesCompat.USAGE_NOTIFICATION_COMMUNICATION_REQUEST
AudioAttributesCompat.USAGE_NOTIFICATION_COMMUNICATION_INSTANT
AudioAttributesCompat.USAGE_NOTIFICATION_COMMUNICATION_DELAYED
AudioAttributesCompat.USAGE_NOTIFICATION_EVENT
AudioAttributesCompat.USAGE_ASSISTANCE_ACCESSIBILITY
AudioAttributesCompat.USAGE_ASSISTANCE_NAVIGATION_GUIDANCE
AudioAttributesCompat.USAGE_ASSISTANCE_SONIFICATION
AudioAttributesCompat.USAGE_GAME
AudioAttributesCompat.USAGE_VIRTUAL_SOURCE
AudioAttributesCompat.USAGE_ASSISTANT

56. 3. Abandon when
you’re done
// API < 26
audioManager.abandonAudioFocus(audioFocusListener)
// API >= 26
audioManager.abandonAudioFocusRequest(request)

57. And how about a more
complex audio app?

58. MediaSession

59. MediaSession
…or how to control your
stream from anything

60. MediaSession
Source: https://medium.com/google-developers/understanding-mediasession-part-2-4-ad291fd32d60

61. MediaBrowserService

62. MediaBrowserService
…or how to expose your media
to the rest of the ecosystem

63. MediaBrowserService
…or how to expose your media
to the rest of the ecosystem
#ecosystem

64. Source : https://medium.com/google-developers/understanding-mediasession-part-4-4-dcc77c535f99

65. Source : https://medium.com/google-developers/understanding-mediasession-part-4-4-dcc77c535f99

66. Source : https://medium.com/google-developers/understanding-mediasession-part-4-4-dcc77c535f99

67. Source : https://medium.com/google-developers/understanding-mediasession-part-4-4-dcc77c535f99

68. Going Deeper…

69. AudioTrack

70. AudioTrack
… The lower level
MediaPlayer

71. AudioTrack API < 26
AudioTrack(AudioManager.STREAM_MUSIC,
sampleRate, // most likely 44.1 Khz
AudioFormat.CHANNEL_OUT_MONO,
AudioFormat.ENCODING_PCM_16BIT,
numberOfSamples,
AudioTrack.MODE_STATIC)

72. AudioTrack.Builder()
.setAudioAttributes(AudioAttributesCompat.Builder()
.setUsage(AudioAttributesCompat.USAGE_MEDIA)
.setContentType(AudioAttributesCompat.CONTENT_TYPE_MUSIC)
.build())
.setAudioFormat(AudioFormat.Builder()
.setEncoding(AudioFormat.ENCODING_PCM_16BIT)
.setSampleRate(sampleRate) // most likely 44.1Khz
.setChannelMask(AudioFormat.CHANNEL_OUT_MONO)
.build())
.setBufferSizeInBytes(bufferSize)
.build()
AudioTrack API >= 26

73. AudioTrack usage
fun play() {
val audioTrack = initAudioTrack()
val someSoundDataInByteArray = getSoundDataSomewhere()
audioTrack.write(someSoundsInByteArray, startOffset, bufferSize)
}

74. AudioRecord

75. AudioRecord
… The lower level
MediaRecorder

76. AudioEffect

77. AudioEffect
… applies effects on
audio!

78. Build-in Effects
• Equalizer
• Virtualizer
• BassBoost
• PresetReverb
• EnvironmentalReverb

79. Effect Usage
val reverb = PresetReverb(priority, audioSession).apply {
preset = PresetReverb.PRESET_LARGEHALL
enabled = true
...
}
audioTrack.apply {
attachAuxEffect(reverb.id)
setAuxEffectSendLevel(1.0F)
...
}

80. Standard

81. Standard
https://www.khronos.org/opensles/

82. Alternative SDK
https://superpowered.com

83. Last but not least

84. Last but not least

85. How does it work

86. How does it work
Controller

87. How does it work
Controller
Interface

88. How does it work
Controller
Computer
Interface

89. How does it work
Controller
Computer
Controller
Interface

90. How does it work
Controller
Computer
Controller
Interface
Controller

91. How does it work
Controller
Computer
Controller
Interface
Controller

92. On Android

93. THANK YOU :)

94. Good Reads
• Understanding MediaSession http://bit.ly/mediasession
• Building a Video Player app in Android http://bit.ly/
vplayerandroid
• ADB Podcast Episode 85 Focus on Audio http://bit.ly/
adbaudiofocus
• Styling Android’s MIDI posts http://bit.ly/midiandroid