Android Audio & OpenSL

Android Audio with OpenSL Yossi Cohen DSP-IP

Overview
Android Audio APIs
OpenSL Overview
OpenSL Android implementation (code)

Playing Audio on Android
MediaPlayer
SoundPool
AudioTrack / AudioRecord (OpenMAX)
OpenSL

MediaPlayer
Java Based, Rich API
Good for playing files and streams
Not Good for gaming sounds or audio conferencing duo to very large delay

OpenMAX
OpenMAX, a Khronos multimedia integration API exposes AudioTrack() and AudioRecord()
Those functions enables the creation of “channels” which play or capture audio
Those API enables audio effects like reverb or distortion
This API also has a significant delay

OPENSL | ES
Overview

OpenSL
OpenSL is a Khronos Open Sound Library
OpenSL ES is OpenSL for embedded systems
Android 2.3 provides PARTIAL implementation of the OpenSL ES More on the limitations of And
More on the OpenSL ES functions supported by Android in this Article

OpenSL
Advantages
Enables Low level audio tweaking and control
Device independent between android phone (Android 2.3+)
Disadvantages
Doesn’t improve the long audio delay of android phones

OpenSL Features
OpenSL supports the following features
Android OpenSL Implementation does not support most of those features
Doppler MIDI messages Preset Reverb Equalizer Pitch Volume Buffer Queues Rate Metadata extraction Environmental Reverb Virtualization Stereo widening LED & Vibra 3D positioning

Android and OpenSL Open SL OpenSL ES Android OpenSL ES

OpenSL, OpenMAX and Android
Base on Android logging we noticed that OpenSL calls ALSA and OpenMAX API functions
Applications Media silicon (audio HW, CPUs, DSPs) OpenMax (AudioTrack / AudioRecord) ALSA

Profiles
OpenSL has three profiles:
Phone
Music
Game
Phone Basic mobile phones Ring tone and alert tone playback (basic MIDI functionality), basic audio playback and record functionality, simple 2D audio games Game-centric mobile devices Advanced MIDI functionality, sophisticated audio capabilities such as 3D audio, audio effects, ability to handle buffers of audio, etc. Music-centric mobile devices High quality audio, ability to support multiple music audio codecs, audio streaming support Game Music

OpenSL API and Architecture
OpenSL is a C API which implements a COM like API
Each object must implements the SLObjectItf Interface (like Iunknown in COM)
OpenSL is NOT a discrete component architecture like Directshow or OpenMax IL
OpenSL is not a channel like architecture like OpenMAX AL

OpenSL Objects
After creation, object should be realized in order to use them.

OpenSL Architecture
OpenSL uses a few objects
Engine – a singleton object which must be implemented so we could use OpenSL
AudioPlayer – enables audio playing
OutputMixer – enable mixing and audio effects (such as reverb)
AudioRecorder – enable audio capturing from microphone. Since Android does NOT support OpenSL microphone enumeration & control functions so AudioRecorder is the only way to capture audio in OpenSL

CODING OPENSL IN ANDROID

Android OpenSL Sample
To use the sample, you should install the Android NDK.
Download Android NDK from http://developer.android.com/sdk/ndk/index.html no need to install just unzip
The samples is under:
Android-ndk-r5b/Samples/native-audio

Editing the project
Select “Create new project” in eclipse select “android project” and follow the settings according to the screen capture:

Native Audio Overview
Native Audio is a very simple (too simple) OpenSL sample.
It enables Half duplex utilization of the OpenSL, this enables either capturing a SINGLE 5 second buffer or playing of a single buffer.
What is missing?
Most voice applications will require Full Duplex behevior. The best sample for that is to create a loopback from Microphone to speakers.

Loopback sample
For RT loopback behavior we need to add:
Small buffers instead of 5 seconds buffers
A two threaded application one for capture and one for record
Since this presentation is not about how to build native application on Android we will not create such app from scratch but use an existing one, HelloJNI
Import it from samples as we did for Native-Audio

Building an OpenSL application
Add the following user permissions for Audio capture and play in the Manifest permission tab:

Application thread
We will implement the recorder in the application thread.
Recorder is create from the objects in the diagram, each object is created() and Realized()

Code Example – Create Engine
// create engine
__android_log_print(ANDROID_LOG_INFO, "NativeAudio", "initEngine");

SLEngineOption EngineOption[] = {(SLuint32) SL_ENGINEOPTION_THREADSAFE, (SLuint32) SL_BOOLEAN_TRUE};
result = slCreateEngine( &engineObject, 1, EngineOption, 0, NULL, NULL);
__android_log_print(ANDROID_LOG_INFO, "NativeAudio", "create engine result %d", result);
assert(SL_RESULT_SUCCESS == result);
// realize the engine
result = (*engineObject)->Realize(engineObject, SL_BOOLEAN_FALSE);
// get the engine interface, which is needed in order to create other objects
result = (*engineObject)->GetInterface(engineObject, SL_IID_ENGINE, &engineEngine);
__android_log_print(ANDROID_LOG_INFO, "NativeAudio", "get engine result %d", result);

Play Thread creation
result = pthread_create( &audioThread, NULL, AudioInOutThread, NULL);
After creating the recorder like in the native-Audio sample we will create the player thread
void *AudioInOutThread( void *ptr ) { SLresult result; initPlayer(); // set the player's state to playing result = (*bqPlayerPlay)->SetPlayState(bqPlayerPlay, SL_PLAYSTATE_PLAYING); assert(SL_RESULT_SUCCESS == result); }

Player CallBack Function
void bqPlayerCallback(SLAndroidSimpleBufferQueueItf bq, void *context)
{
int playershift = playFrame*RECORDER_FRAME; SLresult result;
// enqueue another buffer
result = (*bqPlayerBufferQueue)->Enqueue(bqPlayerBufferQueue, &recorderBuffer[playershift++], RECORDER_FRAME*sizeof(short));
}
We will enqueue the next sample in the callback (need to verify there is a next sample)
Recorder Enqueue is the same but also preform the first enqueue for the player to start it

Player Thread
The player implements the following objects

Resources
WiseAndroid Blog
Mobile Pearls Blog
MindTheRobot Blog

DSP-IP Contact information Download slides at: www.dsp-ip.com For course & Development consulting contact www.dsp-ip.com Mail : [email_address] Phone: +972-545-313092 Fax : +972-50- 8962910
Yossi Cohen
[email_address] +972-545-313092

Android Audio & OpenSL

by Yossi Cohen on May 04, 2011

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 33

Statistics

Android Audio & OpenSL Presentation Transcript