Your SlideShare is downloading. ×
0
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Android Audio & OpenSL
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Android Audio & OpenSL

24,406

Published on

Review of Android OpenSL implementation and how to use it for full duplex audio loop-back

Review of Android OpenSL implementation and how to use it for full duplex audio loop-back

0 Comments
14 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
24,406
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
454
Comments
0
Likes
14
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Android Audio with OpenSL Yossi Cohen DSP-IP
  • 2. Overview <ul><li>Android Audio APIs </li></ul><ul><li>OpenSL Overview </li></ul><ul><li>OpenSL Android implementation (code) </li></ul>
  • 3. Playing Audio on Android <ul><li>MediaPlayer </li></ul><ul><li>SoundPool </li></ul><ul><li>AudioTrack / AudioRecord (OpenMAX) </li></ul><ul><li>OpenSL </li></ul>
  • 4. MediaPlayer <ul><li>Java Based, Rich API </li></ul><ul><li>Good for playing files and streams </li></ul><ul><li>Not Good for gaming sounds or audio conferencing duo to very large delay </li></ul>
  • 5. OpenMAX <ul><li>OpenMAX, a Khronos multimedia integration API exposes AudioTrack() and AudioRecord() </li></ul><ul><li>Those functions enables the creation of “channels” which play or capture audio </li></ul><ul><li>Those API enables audio effects like reverb or distortion </li></ul><ul><li>This API also has a significant delay </li></ul>
  • 6. OPENSL | ES <ul><li>Overview </li></ul>
  • 7. OpenSL <ul><li>OpenSL is a Khronos Open Sound Library </li></ul><ul><li>OpenSL ES is OpenSL for embedded systems </li></ul><ul><li>Android 2.3 provides PARTIAL implementation of the OpenSL ES More on the limitations of And </li></ul><ul><li>More on the OpenSL ES functions supported by Android in this Article </li></ul>
  • 8. OpenSL <ul><li>Advantages </li></ul><ul><ul><li>Enables Low level audio tweaking and control </li></ul></ul><ul><ul><li>Device independent between android phone (Android 2.3+) </li></ul></ul><ul><li>Disadvantages </li></ul><ul><ul><li>Doesn’t improve the long audio delay of android phones </li></ul></ul>
  • 9. OpenSL Features <ul><li>OpenSL supports the following features </li></ul><ul><li>Android OpenSL Implementation does not support most of those features </li></ul>Doppler MIDI messages Preset Reverb Equalizer Pitch Volume Buffer Queues Rate Metadata extraction Environmental Reverb Virtualization Stereo widening LED &amp; Vibra 3D positioning
  • 10. Android and OpenSL Open SL OpenSL ES Android OpenSL ES
  • 11. OpenSL, OpenMAX and Android <ul><li>Base on Android logging we noticed that OpenSL calls ALSA and OpenMAX API functions </li></ul>Applications Media silicon (audio HW, CPUs, DSPs) OpenMax (AudioTrack / AudioRecord) ALSA
  • 12. Profiles <ul><li>OpenSL has three profiles: </li></ul><ul><ul><li>Phone </li></ul></ul><ul><ul><li>Music </li></ul></ul><ul><ul><li>Game </li></ul></ul>Phone Basic mobile phones Ring tone and alert tone playback (basic MIDI functionality), basic audio playback and record functionality, simple 2D audio games Game-centric mobile devices Advanced MIDI functionality, sophisticated audio capabilities such as 3D audio, audio effects, ability to handle buffers of audio, etc. Music-centric mobile devices High quality audio, ability to support multiple music audio codecs, audio streaming support Game Music
  • 13. OpenSL API and Architecture <ul><li>OpenSL is a C API which implements a COM like API </li></ul><ul><li>Each object must implements the SLObjectItf Interface (like Iunknown in COM) </li></ul><ul><li>OpenSL is NOT a discrete component architecture like Directshow or OpenMax IL </li></ul><ul><li>OpenSL is not a channel like architecture like OpenMAX AL </li></ul>
  • 14. OpenSL Objects <ul><li>After creation, object should be realized in order to use them. </li></ul>
  • 15. OpenSL Architecture <ul><li>OpenSL uses a few objects </li></ul><ul><ul><li>Engine – a singleton object which must be implemented so we could use OpenSL </li></ul></ul><ul><ul><li>AudioPlayer – enables audio playing </li></ul></ul><ul><ul><li>OutputMixer – enable mixing and audio effects (such as reverb) </li></ul></ul><ul><ul><li>AudioRecorder – enable audio capturing from microphone. Since Android does NOT support OpenSL microphone enumeration &amp; control functions so AudioRecorder is the only way to capture audio in OpenSL </li></ul></ul>
  • 16. CODING OPENSL IN ANDROID
  • 17. Android OpenSL Sample <ul><li>To use the sample, you should install the Android NDK. </li></ul><ul><li>Download Android NDK from http://developer.android.com/sdk/ndk/index.html no need to install just unzip </li></ul><ul><li>The samples is under: </li></ul><ul><ul><li>Android-ndk-r5b/Samples/native-audio </li></ul></ul>
  • 18. Editing the project <ul><li>Select “Create new project” in eclipse select “android project” and follow the settings according to the screen capture: </li></ul>
  • 19. Native Audio Overview <ul><li>Native Audio is a very simple (too simple) OpenSL sample. </li></ul><ul><li>It enables Half duplex utilization of the OpenSL, this enables either capturing a SINGLE 5 second buffer or playing of a single buffer. </li></ul><ul><li>What is missing? </li></ul><ul><li>Most voice applications will require Full Duplex behevior. The best sample for that is to create a loopback from Microphone to speakers. </li></ul>
  • 20. Loopback sample <ul><li>For RT loopback behavior we need to add: </li></ul><ul><ul><li>Small buffers instead of 5 seconds buffers </li></ul></ul><ul><ul><li>A two threaded application one for capture and one for record </li></ul></ul><ul><li>Since this presentation is not about how to build native application on Android we will not create such app from scratch but use an existing one, HelloJNI </li></ul><ul><li>Import it from samples as we did for Native-Audio </li></ul>
  • 21. Building an OpenSL application <ul><li>Add the following user permissions for Audio capture and play in the Manifest permission tab: </li></ul>
  • 22. Application thread <ul><li>We will implement the recorder in the application thread. </li></ul><ul><li>Recorder is create from the objects in the diagram, each object is created() and Realized() </li></ul>
  • 23. Code Example – Create Engine <ul><li>// create engine </li></ul><ul><li>__android_log_print(ANDROID_LOG_INFO, &amp;quot;NativeAudio&amp;quot;, &amp;quot;initEngine&amp;quot;); </li></ul><ul><li>  </li></ul><ul><li>SLEngineOption EngineOption[] = {(SLuint32) SL_ENGINEOPTION_THREADSAFE, (SLuint32) SL_BOOLEAN_TRUE}; </li></ul><ul><li>result = slCreateEngine( &amp;engineObject, 1, EngineOption, 0, NULL, NULL); </li></ul><ul><li>__android_log_print(ANDROID_LOG_INFO, &amp;quot;NativeAudio&amp;quot;, &amp;quot;create engine result %d&amp;quot;, result); </li></ul><ul><li>assert(SL_RESULT_SUCCESS == result); </li></ul><ul><li>// realize the engine </li></ul><ul><li>result = (*engineObject)-&gt;Realize(engineObject, SL_BOOLEAN_FALSE); </li></ul><ul><li>assert(SL_RESULT_SUCCESS == result); </li></ul><ul><li>// get the engine interface, which is needed in order to create other objects </li></ul><ul><li>result = (*engineObject)-&gt;GetInterface(engineObject, SL_IID_ENGINE, &amp;engineEngine); </li></ul><ul><li>assert(SL_RESULT_SUCCESS == result); </li></ul><ul><li>__android_log_print(ANDROID_LOG_INFO, &amp;quot;NativeAudio&amp;quot;, &amp;quot;get engine result %d&amp;quot;, result); </li></ul>
  • 24. Play Thread creation <ul><li>result = pthread_create( &amp;audioThread, NULL, AudioInOutThread, NULL); </li></ul><ul><li>After creating the recorder like in the native-Audio sample we will create the player thread </li></ul>void *AudioInOutThread( void *ptr ) { SLresult result; initPlayer(); // set the player&apos;s state to playing result = (*bqPlayerPlay)-&gt;SetPlayState(bqPlayerPlay, SL_PLAYSTATE_PLAYING); assert(SL_RESULT_SUCCESS == result); }
  • 25. Player CallBack Function <ul><li>void bqPlayerCallback(SLAndroidSimpleBufferQueueItf bq, void *context) </li></ul><ul><li>{ </li></ul><ul><li>int playershift = playFrame*RECORDER_FRAME; SLresult result; </li></ul><ul><li>// enqueue another buffer </li></ul><ul><li>result = (*bqPlayerBufferQueue)-&gt;Enqueue(bqPlayerBufferQueue, &amp;recorderBuffer[playershift++], RECORDER_FRAME*sizeof(short)); </li></ul><ul><li>} </li></ul><ul><li>We will enqueue the next sample in the callback (need to verify there is a next sample) </li></ul><ul><li>Recorder Enqueue is the same but also preform the first enqueue for the player to start it </li></ul>
  • 26. Player Thread <ul><li>The player implements the following objects </li></ul>
  • 27. Resources <ul><li>WiseAndroid Blog </li></ul><ul><li>Mobile Pearls Blog </li></ul><ul><li>MindTheRobot Blog </li></ul>
  • 28. DSP-IP Contact information Download slides at: www.dsp-ip.com For course &amp; Development consulting contact www.dsp-ip.com Mail : [email_address] Phone: +972-545-313092 Fax : +972-50- 8962910 <ul><ul><li>Yossi Cohen </li></ul></ul><ul><li>[email_address] +972-545-313092 </li></ul>

×