Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Android Audio & OpenSL


Published on

Review of Android OpenSL implementation and how to use it for full duplex audio loop-back

  • Be the first to comment

Android Audio & OpenSL

  1. 1. Android Audio with OpenSL Yossi Cohen DSP-IP
  2. 2. Overview <ul><li>Android Audio APIs </li></ul><ul><li>OpenSL Overview </li></ul><ul><li>OpenSL Android implementation (code) </li></ul>
  3. 3. Playing Audio on Android <ul><li>MediaPlayer </li></ul><ul><li>SoundPool </li></ul><ul><li>AudioTrack / AudioRecord (OpenMAX) </li></ul><ul><li>OpenSL </li></ul>
  4. 4. MediaPlayer <ul><li>Java Based, Rich API </li></ul><ul><li>Good for playing files and streams </li></ul><ul><li>Not Good for gaming sounds or audio conferencing duo to very large delay </li></ul>
  5. 5. OpenMAX <ul><li>OpenMAX, a Khronos multimedia integration API exposes AudioTrack() and AudioRecord() </li></ul><ul><li>Those functions enables the creation of “channels” which play or capture audio </li></ul><ul><li>Those API enables audio effects like reverb or distortion </li></ul><ul><li>This API also has a significant delay </li></ul>
  6. 6. OPENSL | ES <ul><li>Overview </li></ul>
  7. 7. OpenSL <ul><li>OpenSL is a Khronos Open Sound Library </li></ul><ul><li>OpenSL ES is OpenSL for embedded systems </li></ul><ul><li>Android 2.3 provides PARTIAL implementation of the OpenSL ES More on the limitations of And </li></ul><ul><li>More on the OpenSL ES functions supported by Android in this Article </li></ul>
  8. 8. OpenSL <ul><li>Advantages </li></ul><ul><ul><li>Enables Low level audio tweaking and control </li></ul></ul><ul><ul><li>Device independent between android phone (Android 2.3+) </li></ul></ul><ul><li>Disadvantages </li></ul><ul><ul><li>Doesn’t improve the long audio delay of android phones </li></ul></ul>
  9. 9. OpenSL Features <ul><li>OpenSL supports the following features </li></ul><ul><li>Android OpenSL Implementation does not support most of those features </li></ul>Doppler MIDI messages Preset Reverb Equalizer Pitch Volume Buffer Queues Rate Metadata extraction Environmental Reverb Virtualization Stereo widening LED & Vibra 3D positioning
  10. 10. Android and OpenSL Open SL OpenSL ES Android OpenSL ES
  11. 11. OpenSL, OpenMAX and Android <ul><li>Base on Android logging we noticed that OpenSL calls ALSA and OpenMAX API functions </li></ul>Applications Media silicon (audio HW, CPUs, DSPs) OpenMax (AudioTrack / AudioRecord) ALSA
  12. 12. Profiles <ul><li>OpenSL has three profiles: </li></ul><ul><ul><li>Phone </li></ul></ul><ul><ul><li>Music </li></ul></ul><ul><ul><li>Game </li></ul></ul>Phone Basic mobile phones Ring tone and alert tone playback (basic MIDI functionality), basic audio playback and record functionality, simple 2D audio games Game-centric mobile devices Advanced MIDI functionality, sophisticated audio capabilities such as 3D audio, audio effects, ability to handle buffers of audio, etc. Music-centric mobile devices High quality audio, ability to support multiple music audio codecs, audio streaming support Game Music
  13. 13. OpenSL API and Architecture <ul><li>OpenSL is a C API which implements a COM like API </li></ul><ul><li>Each object must implements the SLObjectItf Interface (like Iunknown in COM) </li></ul><ul><li>OpenSL is NOT a discrete component architecture like Directshow or OpenMax IL </li></ul><ul><li>OpenSL is not a channel like architecture like OpenMAX AL </li></ul>
  14. 14. OpenSL Objects <ul><li>After creation, object should be realized in order to use them. </li></ul>
  15. 15. OpenSL Architecture <ul><li>OpenSL uses a few objects </li></ul><ul><ul><li>Engine – a singleton object which must be implemented so we could use OpenSL </li></ul></ul><ul><ul><li>AudioPlayer – enables audio playing </li></ul></ul><ul><ul><li>OutputMixer – enable mixing and audio effects (such as reverb) </li></ul></ul><ul><ul><li>AudioRecorder – enable audio capturing from microphone. Since Android does NOT support OpenSL microphone enumeration & control functions so AudioRecorder is the only way to capture audio in OpenSL </li></ul></ul>
  17. 17. Android OpenSL Sample <ul><li>To use the sample, you should install the Android NDK. </li></ul><ul><li>Download Android NDK from no need to install just unzip </li></ul><ul><li>The samples is under: </li></ul><ul><ul><li>Android-ndk-r5b/Samples/native-audio </li></ul></ul>
  18. 18. Editing the project <ul><li>Select “Create new project” in eclipse select “android project” and follow the settings according to the screen capture: </li></ul>
  19. 19. Native Audio Overview <ul><li>Native Audio is a very simple (too simple) OpenSL sample. </li></ul><ul><li>It enables Half duplex utilization of the OpenSL, this enables either capturing a SINGLE 5 second buffer or playing of a single buffer. </li></ul><ul><li>What is missing? </li></ul><ul><li>Most voice applications will require Full Duplex behevior. The best sample for that is to create a loopback from Microphone to speakers. </li></ul>
  20. 20. Loopback sample <ul><li>For RT loopback behavior we need to add: </li></ul><ul><ul><li>Small buffers instead of 5 seconds buffers </li></ul></ul><ul><ul><li>A two threaded application one for capture and one for record </li></ul></ul><ul><li>Since this presentation is not about how to build native application on Android we will not create such app from scratch but use an existing one, HelloJNI </li></ul><ul><li>Import it from samples as we did for Native-Audio </li></ul>
  21. 21. Building an OpenSL application <ul><li>Add the following user permissions for Audio capture and play in the Manifest permission tab: </li></ul>
  22. 22. Application thread <ul><li>We will implement the recorder in the application thread. </li></ul><ul><li>Recorder is create from the objects in the diagram, each object is created() and Realized() </li></ul>
  23. 23. Code Example – Create Engine <ul><li>// create engine </li></ul><ul><li>__android_log_print(ANDROID_LOG_INFO, &quot;NativeAudio&quot;, &quot;initEngine&quot;); </li></ul><ul><li>  </li></ul><ul><li>SLEngineOption EngineOption[] = {(SLuint32) SL_ENGINEOPTION_THREADSAFE, (SLuint32) SL_BOOLEAN_TRUE}; </li></ul><ul><li>result = slCreateEngine( &engineObject, 1, EngineOption, 0, NULL, NULL); </li></ul><ul><li>__android_log_print(ANDROID_LOG_INFO, &quot;NativeAudio&quot;, &quot;create engine result %d&quot;, result); </li></ul><ul><li>assert(SL_RESULT_SUCCESS == result); </li></ul><ul><li>// realize the engine </li></ul><ul><li>result = (*engineObject)->Realize(engineObject, SL_BOOLEAN_FALSE); </li></ul><ul><li>assert(SL_RESULT_SUCCESS == result); </li></ul><ul><li>// get the engine interface, which is needed in order to create other objects </li></ul><ul><li>result = (*engineObject)->GetInterface(engineObject, SL_IID_ENGINE, &engineEngine); </li></ul><ul><li>assert(SL_RESULT_SUCCESS == result); </li></ul><ul><li>__android_log_print(ANDROID_LOG_INFO, &quot;NativeAudio&quot;, &quot;get engine result %d&quot;, result); </li></ul>
  24. 24. Play Thread creation <ul><li>result = pthread_create( &audioThread, NULL, AudioInOutThread, NULL); </li></ul><ul><li>After creating the recorder like in the native-Audio sample we will create the player thread </li></ul>void *AudioInOutThread( void *ptr ) { SLresult result; initPlayer(); // set the player's state to playing result = (*bqPlayerPlay)->SetPlayState(bqPlayerPlay, SL_PLAYSTATE_PLAYING); assert(SL_RESULT_SUCCESS == result); }
  25. 25. Player CallBack Function <ul><li>void bqPlayerCallback(SLAndroidSimpleBufferQueueItf bq, void *context) </li></ul><ul><li>{ </li></ul><ul><li>int playershift = playFrame*RECORDER_FRAME; SLresult result; </li></ul><ul><li>// enqueue another buffer </li></ul><ul><li>result = (*bqPlayerBufferQueue)->Enqueue(bqPlayerBufferQueue, &recorderBuffer[playershift++], RECORDER_FRAME*sizeof(short)); </li></ul><ul><li>} </li></ul><ul><li>We will enqueue the next sample in the callback (need to verify there is a next sample) </li></ul><ul><li>Recorder Enqueue is the same but also preform the first enqueue for the player to start it </li></ul>
  26. 26. Player Thread <ul><li>The player implements the following objects </li></ul>
  27. 27. Resources <ul><li>WiseAndroid Blog </li></ul><ul><li>Mobile Pearls Blog </li></ul><ul><li>MindTheRobot Blog </li></ul>
  28. 28. DSP-IP Contact information Download slides at: For course & Development consulting contact Mail : [email_address] Phone: +972-545-313092 Fax : +972-50- 8962910 <ul><ul><li>Yossi Cohen </li></ul></ul><ul><li>[email_address] +972-545-313092 </li></ul>