Android Audio & OpenSL


Published on

Review of Android OpenSL implementation and how to use it for full duplex audio loop-back

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Android Audio & OpenSL

  1. 1. Android Audio with OpenSL Yossi Cohen DSP-IP
  2. 2. Overview <ul><li>Android Audio APIs </li></ul><ul><li>OpenSL Overview </li></ul><ul><li>OpenSL Android implementation (code) </li></ul>
  3. 3. Playing Audio on Android <ul><li>MediaPlayer </li></ul><ul><li>SoundPool </li></ul><ul><li>AudioTrack / AudioRecord (OpenMAX) </li></ul><ul><li>OpenSL </li></ul>
  4. 4. MediaPlayer <ul><li>Java Based, Rich API </li></ul><ul><li>Good for playing files and streams </li></ul><ul><li>Not Good for gaming sounds or audio conferencing duo to very large delay </li></ul>
  5. 5. OpenMAX <ul><li>OpenMAX, a Khronos multimedia integration API exposes AudioTrack() and AudioRecord() </li></ul><ul><li>Those functions enables the creation of “channels” which play or capture audio </li></ul><ul><li>Those API enables audio effects like reverb or distortion </li></ul><ul><li>This API also has a significant delay </li></ul>
  6. 6. OPENSL | ES <ul><li>Overview </li></ul>
  7. 7. OpenSL <ul><li>OpenSL is a Khronos Open Sound Library </li></ul><ul><li>OpenSL ES is OpenSL for embedded systems </li></ul><ul><li>Android 2.3 provides PARTIAL implementation of the OpenSL ES More on the limitations of And </li></ul><ul><li>More on the OpenSL ES functions supported by Android in this Article </li></ul>
  8. 8. OpenSL <ul><li>Advantages </li></ul><ul><ul><li>Enables Low level audio tweaking and control </li></ul></ul><ul><ul><li>Device independent between android phone (Android 2.3+) </li></ul></ul><ul><li>Disadvantages </li></ul><ul><ul><li>Doesn’t improve the long audio delay of android phones </li></ul></ul>
  9. 9. OpenSL Features <ul><li>OpenSL supports the following features </li></ul><ul><li>Android OpenSL Implementation does not support most of those features </li></ul>Doppler MIDI messages Preset Reverb Equalizer Pitch Volume Buffer Queues Rate Metadata extraction Environmental Reverb Virtualization Stereo widening LED & Vibra 3D positioning
  10. 10. Android and OpenSL Open SL OpenSL ES Android OpenSL ES
  11. 11. OpenSL, OpenMAX and Android <ul><li>Base on Android logging we noticed that OpenSL calls ALSA and OpenMAX API functions </li></ul>Applications Media silicon (audio HW, CPUs, DSPs) OpenMax (AudioTrack / AudioRecord) ALSA
  12. 12. Profiles <ul><li>OpenSL has three profiles: </li></ul><ul><ul><li>Phone </li></ul></ul><ul><ul><li>Music </li></ul></ul><ul><ul><li>Game </li></ul></ul>Phone Basic mobile phones Ring tone and alert tone playback (basic MIDI functionality), basic audio playback and record functionality, simple 2D audio games Game-centric mobile devices Advanced MIDI functionality, sophisticated audio capabilities such as 3D audio, audio effects, ability to handle buffers of audio, etc. Music-centric mobile devices High quality audio, ability to support multiple music audio codecs, audio streaming support Game Music
  13. 13. OpenSL API and Architecture <ul><li>OpenSL is a C API which implements a COM like API </li></ul><ul><li>Each object must implements the SLObjectItf Interface (like Iunknown in COM) </li></ul><ul><li>OpenSL is NOT a discrete component architecture like Directshow or OpenMax IL </li></ul><ul><li>OpenSL is not a channel like architecture like OpenMAX AL </li></ul>
  14. 14. OpenSL Objects <ul><li>After creation, object should be realized in order to use them. </li></ul>
  15. 15. OpenSL Architecture <ul><li>OpenSL uses a few objects </li></ul><ul><ul><li>Engine – a singleton object which must be implemented so we could use OpenSL </li></ul></ul><ul><ul><li>AudioPlayer – enables audio playing </li></ul></ul><ul><ul><li>OutputMixer – enable mixing and audio effects (such as reverb) </li></ul></ul><ul><ul><li>AudioRecorder – enable audio capturing from microphone. Since Android does NOT support OpenSL microphone enumeration & control functions so AudioRecorder is the only way to capture audio in OpenSL </li></ul></ul>
  17. 17. Android OpenSL Sample <ul><li>To use the sample, you should install the Android NDK. </li></ul><ul><li>Download Android NDK from no need to install just unzip </li></ul><ul><li>The samples is under: </li></ul><ul><ul><li>Android-ndk-r5b/Samples/native-audio </li></ul></ul>
  18. 18. Editing the project <ul><li>Select “Create new project” in eclipse select “android project” and follow the settings according to the screen capture: </li></ul>
  19. 19. Native Audio Overview <ul><li>Native Audio is a very simple (too simple) OpenSL sample. </li></ul><ul><li>It enables Half duplex utilization of the OpenSL, this enables either capturing a SINGLE 5 second buffer or playing of a single buffer. </li></ul><ul><li>What is missing? </li></ul><ul><li>Most voice applications will require Full Duplex behevior. The best sample for that is to create a loopback from Microphone to speakers. </li></ul>
  20. 20. Loopback sample <ul><li>For RT loopback behavior we need to add: </li></ul><ul><ul><li>Small buffers instead of 5 seconds buffers </li></ul></ul><ul><ul><li>A two threaded application one for capture and one for record </li></ul></ul><ul><li>Since this presentation is not about how to build native application on Android we will not create such app from scratch but use an existing one, HelloJNI </li></ul><ul><li>Import it from samples as we did for Native-Audio </li></ul>
  21. 21. Building an OpenSL application <ul><li>Add the following user permissions for Audio capture and play in the Manifest permission tab: </li></ul>
  22. 22. Application thread <ul><li>We will implement the recorder in the application thread. </li></ul><ul><li>Recorder is create from the objects in the diagram, each object is created() and Realized() </li></ul>
  23. 23. Code Example – Create Engine <ul><li>// create engine </li></ul><ul><li>__android_log_print(ANDROID_LOG_INFO, &quot;NativeAudio&quot;, &quot;initEngine&quot;); </li></ul><ul><li>  </li></ul><ul><li>SLEngineOption EngineOption[] = {(SLuint32) SL_ENGINEOPTION_THREADSAFE, (SLuint32) SL_BOOLEAN_TRUE}; </li></ul><ul><li>result = slCreateEngine( &engineObject, 1, EngineOption, 0, NULL, NULL); </li></ul><ul><li>__android_log_print(ANDROID_LOG_INFO, &quot;NativeAudio&quot;, &quot;create engine result %d&quot;, result); </li></ul><ul><li>assert(SL_RESULT_SUCCESS == result); </li></ul><ul><li>// realize the engine </li></ul><ul><li>result = (*engineObject)->Realize(engineObject, SL_BOOLEAN_FALSE); </li></ul><ul><li>assert(SL_RESULT_SUCCESS == result); </li></ul><ul><li>// get the engine interface, which is needed in order to create other objects </li></ul><ul><li>result = (*engineObject)->GetInterface(engineObject, SL_IID_ENGINE, &engineEngine); </li></ul><ul><li>assert(SL_RESULT_SUCCESS == result); </li></ul><ul><li>__android_log_print(ANDROID_LOG_INFO, &quot;NativeAudio&quot;, &quot;get engine result %d&quot;, result); </li></ul>
  24. 24. Play Thread creation <ul><li>result = pthread_create( &audioThread, NULL, AudioInOutThread, NULL); </li></ul><ul><li>After creating the recorder like in the native-Audio sample we will create the player thread </li></ul>void *AudioInOutThread( void *ptr ) { SLresult result; initPlayer(); // set the player's state to playing result = (*bqPlayerPlay)->SetPlayState(bqPlayerPlay, SL_PLAYSTATE_PLAYING); assert(SL_RESULT_SUCCESS == result); }
  25. 25. Player CallBack Function <ul><li>void bqPlayerCallback(SLAndroidSimpleBufferQueueItf bq, void *context) </li></ul><ul><li>{ </li></ul><ul><li>int playershift = playFrame*RECORDER_FRAME; SLresult result; </li></ul><ul><li>// enqueue another buffer </li></ul><ul><li>result = (*bqPlayerBufferQueue)->Enqueue(bqPlayerBufferQueue, &recorderBuffer[playershift++], RECORDER_FRAME*sizeof(short)); </li></ul><ul><li>} </li></ul><ul><li>We will enqueue the next sample in the callback (need to verify there is a next sample) </li></ul><ul><li>Recorder Enqueue is the same but also preform the first enqueue for the player to start it </li></ul>
  26. 26. Player Thread <ul><li>The player implements the following objects </li></ul>
  27. 27. Resources <ul><li>WiseAndroid Blog </li></ul><ul><li>Mobile Pearls Blog </li></ul><ul><li>MindTheRobot Blog </li></ul>
  28. 28. DSP-IP Contact information Download slides at: For course & Development consulting contact Mail : [email_address] Phone: +972-545-313092 Fax : +972-50- 8962910 <ul><ul><li>Yossi Cohen </li></ul></ul><ul><li>[email_address] +972-545-313092 </li></ul>