Android Audio & OpenSL
Upcoming SlideShare
Loading in...5
×
 

Android Audio & OpenSL

on

  • 23,754 views

Review of Android OpenSL implementation and how to use it for full duplex audio loop-back

Review of Android OpenSL implementation and how to use it for full duplex audio loop-back

Statistics

Views

Total Views
23,754
Views on SlideShare
23,716
Embed Views
38

Actions

Likes
13
Downloads
417
Comments
0

3 Embeds 38

http://www.dsp-ip.com 33
http://www.verious.com 3
http://www.linkedin.com 2

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

CC Attribution-NoDerivs LicenseCC Attribution-NoDerivs License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Android Audio & OpenSL Android Audio & OpenSL Presentation Transcript

  • Android Audio with OpenSL Yossi Cohen DSP-IP
  • Overview
    • Android Audio APIs
    • OpenSL Overview
    • OpenSL Android implementation (code)
  • Playing Audio on Android
    • MediaPlayer
    • SoundPool
    • AudioTrack / AudioRecord (OpenMAX)
    • OpenSL
  • MediaPlayer
    • Java Based, Rich API
    • Good for playing files and streams
    • Not Good for gaming sounds or audio conferencing duo to very large delay
  • OpenMAX
    • OpenMAX, a Khronos multimedia integration API exposes AudioTrack() and AudioRecord()
    • Those functions enables the creation of “channels” which play or capture audio
    • Those API enables audio effects like reverb or distortion
    • This API also has a significant delay
  • OPENSL | ES
    • Overview
  • OpenSL
    • OpenSL is a Khronos Open Sound Library
    • OpenSL ES is OpenSL for embedded systems
    • Android 2.3 provides PARTIAL implementation of the OpenSL ES More on the limitations of And
    • More on the OpenSL ES functions supported by Android in this Article
  • OpenSL
    • Advantages
      • Enables Low level audio tweaking and control
      • Device independent between android phone (Android 2.3+)
    • Disadvantages
      • Doesn’t improve the long audio delay of android phones
  • OpenSL Features
    • OpenSL supports the following features
    • Android OpenSL Implementation does not support most of those features
    Doppler MIDI messages Preset Reverb Equalizer Pitch Volume Buffer Queues Rate Metadata extraction Environmental Reverb Virtualization Stereo widening LED & Vibra 3D positioning
  • Android and OpenSL Open SL OpenSL ES Android OpenSL ES
  • OpenSL, OpenMAX and Android
    • Base on Android logging we noticed that OpenSL calls ALSA and OpenMAX API functions
    Applications Media silicon (audio HW, CPUs, DSPs) OpenMax (AudioTrack / AudioRecord) ALSA
  • Profiles
    • OpenSL has three profiles:
      • Phone
      • Music
      • Game
    Phone Basic mobile phones Ring tone and alert tone playback (basic MIDI functionality), basic audio playback and record functionality, simple 2D audio games Game-centric mobile devices Advanced MIDI functionality, sophisticated audio capabilities such as 3D audio, audio effects, ability to handle buffers of audio, etc. Music-centric mobile devices High quality audio, ability to support multiple music audio codecs, audio streaming support Game Music
  • OpenSL API and Architecture
    • OpenSL is a C API which implements a COM like API
    • Each object must implements the SLObjectItf Interface (like Iunknown in COM)
    • OpenSL is NOT a discrete component architecture like Directshow or OpenMax IL
    • OpenSL is not a channel like architecture like OpenMAX AL
  • OpenSL Objects
    • After creation, object should be realized in order to use them.
  • OpenSL Architecture
    • OpenSL uses a few objects
      • Engine – a singleton object which must be implemented so we could use OpenSL
      • AudioPlayer – enables audio playing
      • OutputMixer – enable mixing and audio effects (such as reverb)
      • AudioRecorder – enable audio capturing from microphone. Since Android does NOT support OpenSL microphone enumeration & control functions so AudioRecorder is the only way to capture audio in OpenSL
  • CODING OPENSL IN ANDROID
  • Android OpenSL Sample
    • To use the sample, you should install the Android NDK.
    • Download Android NDK from http://developer.android.com/sdk/ndk/index.html no need to install just unzip
    • The samples is under:
      • Android-ndk-r5b/Samples/native-audio
  • Editing the project
    • Select “Create new project” in eclipse select “android project” and follow the settings according to the screen capture:
  • Native Audio Overview
    • Native Audio is a very simple (too simple) OpenSL sample.
    • It enables Half duplex utilization of the OpenSL, this enables either capturing a SINGLE 5 second buffer or playing of a single buffer.
    • What is missing?
    • Most voice applications will require Full Duplex behevior. The best sample for that is to create a loopback from Microphone to speakers.
  • Loopback sample
    • For RT loopback behavior we need to add:
      • Small buffers instead of 5 seconds buffers
      • A two threaded application one for capture and one for record
    • Since this presentation is not about how to build native application on Android we will not create such app from scratch but use an existing one, HelloJNI
    • Import it from samples as we did for Native-Audio
  • Building an OpenSL application
    • Add the following user permissions for Audio capture and play in the Manifest permission tab:
  • Application thread
    • We will implement the recorder in the application thread.
    • Recorder is create from the objects in the diagram, each object is created() and Realized()
  • Code Example – Create Engine
    • // create engine
    • __android_log_print(ANDROID_LOG_INFO, "NativeAudio", "initEngine");
    •  
    • SLEngineOption EngineOption[] = {(SLuint32) SL_ENGINEOPTION_THREADSAFE, (SLuint32) SL_BOOLEAN_TRUE};
    • result = slCreateEngine( &engineObject, 1, EngineOption, 0, NULL, NULL);
    • __android_log_print(ANDROID_LOG_INFO, "NativeAudio", "create engine result %d", result);
    • assert(SL_RESULT_SUCCESS == result);
    • // realize the engine
    • result = (*engineObject)->Realize(engineObject, SL_BOOLEAN_FALSE);
    • assert(SL_RESULT_SUCCESS == result);
    • // get the engine interface, which is needed in order to create other objects
    • result = (*engineObject)->GetInterface(engineObject, SL_IID_ENGINE, &engineEngine);
    • assert(SL_RESULT_SUCCESS == result);
    • __android_log_print(ANDROID_LOG_INFO, "NativeAudio", "get engine result %d", result);
  • Play Thread creation
    • result = pthread_create( &audioThread, NULL, AudioInOutThread, NULL);
    • After creating the recorder like in the native-Audio sample we will create the player thread
    void *AudioInOutThread( void *ptr ) { SLresult result; initPlayer(); // set the player's state to playing result = (*bqPlayerPlay)->SetPlayState(bqPlayerPlay, SL_PLAYSTATE_PLAYING); assert(SL_RESULT_SUCCESS == result); }
  • Player CallBack Function
    • void bqPlayerCallback(SLAndroidSimpleBufferQueueItf bq, void *context)
    • {
    • int playershift = playFrame*RECORDER_FRAME; SLresult result;
    • // enqueue another buffer
    • result = (*bqPlayerBufferQueue)->Enqueue(bqPlayerBufferQueue, &recorderBuffer[playershift++], RECORDER_FRAME*sizeof(short));
    • }
    • We will enqueue the next sample in the callback (need to verify there is a next sample)
    • Recorder Enqueue is the same but also preform the first enqueue for the player to start it
  • Player Thread
    • The player implements the following objects
  • Resources
    • WiseAndroid Blog
    • Mobile Pearls Blog
    • MindTheRobot Blog
  • DSP-IP Contact information Download slides at: www.dsp-ip.com For course & Development consulting contact www.dsp-ip.com Mail : [email_address] Phone: +972-545-313092 Fax : +972-50- 8962910
      • Yossi Cohen
    • [email_address] +972-545-313092