Deep dive into Android’s audio latency problem
•
1 like•10,544 views
The document discusses audio latency issues on Android devices. It provides perspective on acceptable latency levels for different types of music (3-23ms) and reaction times (15-70ms). It then covers three classes of audio applications and how latency affects them differently. The bulk of the document dives into technical details of Android's audio stack, including available APIs (MediaPlayer, SoundPool, AudioTrack), hardware considerations (sampling rates, buffer sizes), and how Unity achieves low-latency audio versus a native implementation. It suggests experimenting with different buffer sizes can reduce latency but risks buffer underruns.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Deep dive into Android’s audio latency problem
Similar to Deep dive into Android’s audio latency problem (20)
Deep dive into Android’s audio latency problem
- 2. Deep dive into Android’s audio latency problem
- 3. Severity of the problem
- 6. In this talk, we will demystify together where exactly those time difference went to!
- 7. Some numbers, for perspective • Acceptable latency for musicians • Vocals : 3ms - voice is literally “in your head” • Drums & Percussions : 6ms - you sit right in front of them • Guitars : 12ms - may take time in the effect chains • Keyboards : 23ms - goes through several sound chambers’ acoustics Sound On Sound : Optimising The Latency Of Your PC Audio Interface https://www.soundonsound.com/techniques/optimising-latency-pc-audio-interface#para7
- 8. Some numbers, for perspective • Simple Reaction Time (SRT) Factors influencing the latency of simple reaction time David L. Woods, John M. Wyma, E. William Yund, Timothy J. Herron, and Bruce Reed https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4374455/
- 9. Some numbers, for perspective • Rhythm game’s highest (perfect) judgements, by player’s research • Dynamix 59ms • Deemo 50ms • Jubeat 41ms • Bang Dream! 40ms • Sound Voltex 33ms • Arcaea 25ms • Pop’n Music 25ms • Chunithm, Taiko no Tatsujin 33ms • VOEZ 30ms • Beatmania IIDX 20ms • Cytus 70ms • DDR 15ms Data compiled by @Sta_Light_ https://twitter.com/Sta_Light_/status/1020704143231500289
- 11. Why should I care?
- 12. 3 Classes of audio applications • Sequencer class - Latency does not matter • Music player : No one cares about latency. • Sequencer / Digital Audio Workstation : Able to compensate for the latency. • Instrument class - Latency may matter • Possible to “adapt to the latency”, just like a real instrument. • UI button sounds • Rhythm game class - Latency affects user experience greatly! Music is now tied to the gameplay. • Music games without response sound : compensate for latency is possible. • Music games with response sound : either perfect judgement with wrong response sound, or early judgement with in-time response sound.
- 13. Looper FL Studio Mobile Korg Gadget
- 14. Garage Band Korg iMS-20
- 15. Taiko no Tatsujin Groove Coaster Beatmania IIDX
- 17. Each device do “prefer” something • Native Sampling Rate • Older mostly 44000Hz, newer phones moved to 48000Hz • Optimal Buffer Size • The lower the better latency-wise. Bad phone cannot handle small buffers • Low Latency Feature (true / false) • Indicates a continuous output latency of 45 ms or less • Pro Audio Feature (true / false) • Indicates a continuous round-trip latency of 20 ms or less
- 18. Each device do “prefer” something Native Sampling Rate : 44100Hz Optimal Buffer Size : 640 Low Latency : False Pro Audio : False Native Sampling Rate : 44100Hz Optimal Buffer Size : 240 Low Latency : False Pro Audio : False Native Sampling Rate : 48000Hz Optimal Buffer Size : 192 Low Latency : True Pro Audio : True Go download a test app at https://superpowered.com/latency to see yours!
- 21. Introduction to all the API options! MediaPlayer SoundPool AudioTrack (Java) AudioTrack (C++) “Tracks” OpenSL ES AAudio Thread Thread Thread HAL Java C++ Hardware Abstraction Layer
- 22. Introduction to all the API options! MediaPlayer SoundPool AudioTrack (Java) AudioTrack (C++) “Tracks” OpenSL ES AAudio Thread Thread Thread HAL Java C++ Hardware Abstraction Layer
- 23. Introduction to all the API options! MediaPlayer SoundPool AudioTrack (Java) AudioTrack (C++) “Tracks” OpenSL ES AAudio Thread Thread Thread HAL Java C++ Hardware Abstraction Layer Let’s start from the lowest level So we know what higher level APIs ultimately maps to.
- 24. AudioTrack (C++) No matter what, it must comes down to this! • Our important question : How differently Unity used the AudioTrack than our “naive” native audio playing that results in that much latency?
- 25. A short while after Unity starts up…
- 27. Track Audio Stream Mixer Thread Audio Stream
- 29. Fast Native Sampling Rate : 44100Hz Optimal Buffer Size : 240 Low Latency : False Pro Audio : False
- 30. Mixer Threads • Fast Mixer • Up to 7 fast tracks • Mix normal mixer’s submix • Track attenuation (volume) • NO : • Resampling • Effects • Normal Mixer • Up to 32 tracks • Can play any sampling rate by sample rate conversion + effects Mixer Thread Fast Mixer Thread Normal Mixer Thread Normal Mixer Thread
- 31. 48000 Hz Mixer Thread 44100Hz 96000 Hz 44100Hz Sampling rate problem
- 33. adb shell dumpsys media.audio_flinger
- 34. Example Give me 15 tracks of 48000Hz, Buffer size 196 Native Sampling Rate : 48000Hz Optimal Buffer Size : 192 Low Latency : True Pro Audio : True
- 36. adb shell dumpsys media.audio_flinger
- 37. Phone
- 38. Your game !
- 39. How Unity achieve concurrency with just 1 AudioTrack ? • FMOD Mixer • Virtual tracks
- 40. (Normal) Mixer Thread24000 Hz FMOD Mixer (Latency!) 24000 Hz ? 44100 Hz 48000 Hz 22050 Hz (Normal) Track • Must wait until the end of frame to summarize `audioSource.Play()` (Latency!) • Calculate `AudioMixerGroup` buses and effects (Unity 5.0 +) (Latency!)
- 42. MediaPlayer SoundPool AudioTrack (Java) AudioTrack (C++) “Tracks” OpenSL ES AAudio Thread Thread Thread Java C++ Unity startup Usually gets normal track with 24000Hz rate (Latency!) Usually gets fast track with phone’s rate
- 43. OpenSL ES • An open source set of OpenGL-style audio commands. Implementers are free to maps to anything. • It maps to `AudioTrack` (C++) on Android. • Possible to get fast track.
- 44. “Project wilhelm” git clone https://android.googlesource.com/platform/frameworks/wilhelm
- 45. AudioTrack (Java) • `new AudioTrack()` = or • Still must feed audio stream. • Possible to get fast track. • `new AudioTrack()` x32 = crash
- 46. SoundPool • `new SoundPool()` does not instantiate any track yet. • Dynamically instantiate tracks as you play. (Can set a limit) Possible to get a fast track. • Get a track index back on every play. • Supports loading a file, storing multiple audio inside, loop, pause, resume, etc.
- 47. MediaPlayer • Cannot get fast track. • Full-blown media player with video support. • Not good for latency.
- 48. Fun experiment : Crashing Unity audio
- 50. Unity 2019.1.0a7
- 53. Latency wasted on bad AudioTrack
- 55. Buffer Size
- 57. ?
- 58. Buffer Underrun • Lower ring buffer size equals better latency, but with a risk of running out of data. • Exporting to PC build with Best Latency easily cause buffer underrun. • Exporting to Android with Best Latency usually works but I know a case that does not work.
- 59. Thank you!