Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Multimedia Application
Development on Android


              KRK MOHAN
   Senior Engineer - Muvee Technologies
          ...
Agenda

• Multimedia and Android
• Video & Audio: MediaPlayer, MediaRecorder,
  JetPlayer, SoundPool, AudioTrack
• Images:...
Scope

• Introductory level
• Focus on what can and cannot be done with
  Android FroYo’s multimedia capabilities
• Refere...
Multimedia Apps

• Media Consumption - Slide Shows, Media
  Players,YouTube, facebook etc
• Media Production - Image Edito...
You will identify a great product when you see it

 Deep, Indulgent, Complete, Elegant (DICE)


                          ...
android.media
    • Contains classes to play audio/video content
       from raw resource files, file streams and JET
      ...
File formats and codecs

• File formats for recording media on device
 • 3GPP, MP4, RAW_AMR
• Codecs used for recording me...
The big 2

MediaPlayer and MediaRecorder classes encapsulate:

  File parsers for media file formats such as 3GP, MP4
  &
 ...
MediaPlayer - state diagram




                              From Android Developers -
                                  ...
Audio

• MediaPlayer & MediaRecorder
• SoundPool - multiple audio sources
• AudioTrack & AudioRecord- raw audio
  manipula...
Audio - MediaPlayer
Playing an audio resource
MediaPlayer mp = MediaPlayer.create(context, R.raw.sound_file_1);
mp.start();...
Audio - SoundPool
• Multiple sounds can be played from raw or
  compressed source. Sounds are identified
  by sound ids.
• ...
Audio - AudioTrack
• Used to play raw audio samples (PCM)
  stored in memory buffers or files
• Static mode for playing sou...
Audio - AudioTrack
AudioTrack audioTrack = new
AudioTrack(AudioManager.STREAM_MUSIC,                            
44100,   ...
Audio - JETPlayer
      • Interactive music playback by altering
         tempo, volume level etc of a series of MIDI
    ...
Video - MediaPlayer
mMediaPlayer = new MediaPlayer();
mMediaPlayer.setDataSource(localPath);           
mMediaPlayer.setDi...
Video - VideoView
mVideoView = (VideoView)
findViewById(R.id.surface_view);

mVideoView.setVideoPath(localPath);           ...
iMovie on Android!?

• Classes needed to process individual video
  frames
• API level 1 contained MediaPlayer.getFrame()
...
way forward...

• OMXCodec.h or a similar interface needs to
  be exposed via NDK
• This would facilitate comprehensive vi...
Graphics - OpenGL-ES
• 3D graphics API to process and render
  images/textures
• On Android
    GL10 loosely corresponds t...
GLSurfaceView
From Android Developers:
• Manages a surface, which is a special piece of memory that can
  be composited in...
OpenGL-ES Textures
• Create a Bitmap out of the image to be
  processed
• GLUtils class provides helper functions such
  a...
GLSurfaceView.Renderer
class MyRenderer implements GLSurfaceView.Renderer {
   
     public void onDrawFrame(GL10 gl) {
  ...
Together...
    • To begin rendering
       mGLSurfaceView = new GLSurfaceView(this);
       mGLSurfaceView.setRenderer(ne...
Misc OpenGL-ES
•   setDebugFlags(DEBUG_CHECK_GL_ERROR |
    DEBUG_LOG_GL_CALLS);

•   Use
    GLSurfaceView.setRenderMode(...
Bitmaps & image
           processing
• BitmapFactory.decode(filepath) returns a
  Bitmap of the image
• Use getPixels to a...
ImageView
 • Display PNG, JPG images
 • Apply effects like color tinting
 • View can be animated
 ImageView image = (Image...
Appendix - Animations

<animation-list xmlns:android="http://schemas.android.com/apk/res/android"
   android:oneshot="true...
Thank You!


    29
Upcoming SlideShare
Loading in …5
×

Gtug

2,496 views

Published on

SG-GTUG

Published in: Education
  • Be the first to comment

Gtug

  1. 1. Multimedia Application Development on Android KRK MOHAN Senior Engineer - Muvee Technologies www.muvee.com krk.mohan@gmail.com 1
  2. 2. Agenda • Multimedia and Android • Video & Audio: MediaPlayer, MediaRecorder, JetPlayer, SoundPool, AudioTrack • Images: OpenGL-ES graphics, Bitmaps, ImageView 2
  3. 3. Scope • Introductory level • Focus on what can and cannot be done with Android FroYo’s multimedia capabilities • References to APIDemos projects and developer site articles provided 3
  4. 4. Multimedia Apps • Media Consumption - Slide Shows, Media Players,YouTube, facebook etc • Media Production - Image Editors, Camera enhancements • Huge scope for improvement - great opportunity for developers 4
  5. 5. You will identify a great product when you see it Deep, Indulgent, Complete, Elegant (DICE) Guy Kawasaki “The Macintosh Way” 5
  6. 6. android.media • Contains classes to play audio/video content from raw resource files, file streams and JET content • Contains classes to record audio/video content from cameras or mic http://developer.android.com/intl/de/guide/topics/media/ index.html 6
  7. 7. File formats and codecs • File formats for recording media on device • 3GPP, MP4, RAW_AMR • Codecs used for recording media on device • H263, H264, MPEG4 (SP), AMR_NB 7
  8. 8. The big 2 MediaPlayer and MediaRecorder classes encapsulate: File parsers for media file formats such as 3GP, MP4 & Codecs to encode or decode media content Two media frameworks present in FroYo OpenCORE and StageFright 8
  9. 9. MediaPlayer - state diagram From Android Developers - MediaPlayer entry 9
  10. 10. Audio • MediaPlayer & MediaRecorder • SoundPool - multiple audio sources • AudioTrack & AudioRecord- raw audio manipulation • JetPlayer - interactive music 10
  11. 11. Audio - MediaPlayer Playing an audio resource MediaPlayer mp = MediaPlayer.create(context, R.raw.sound_file_1); mp.start(); Playing a file from file system MediaPlayer mp = new MediaPlayer(); mp.setDataSource(PATH_TO_FILE); mp.prepare(); mp.start(); 11
  12. 12. Audio - SoundPool • Multiple sounds can be played from raw or compressed source. Sounds are identified by sound ids. • More efficient than multiple MediaPlayers • Each sound can be assigned a priority • Adjustable playback frequency • Support for repeat mode 12
  13. 13. Audio - AudioTrack • Used to play raw audio samples (PCM) stored in memory buffers or files • Static mode for playing sounds that fit into memory and need to be played often with low overhead • Streaming mode for sounds that are too big to fit into memory 13
  14. 14. Audio - AudioTrack AudioTrack audioTrack = new AudioTrack(AudioManager.STREAM_MUSIC,                             44100,                                           AudioFormat.CHANNEL_CONFIGURATION_MONO,             AudioFormat.ENCODING_PCM_16BIT,                               musicLength, AudioTrack.MODE_STREAM); audioTrack.play(); audioTrack.write(music, 0, musicLength); //blocking Check out AudioRecord class Some examples at: http://emeadev.blogspot.com/2009/09/raw-audio-manipulation-in-android.html 14
  15. 15. Audio - JETPlayer • Interactive music playback by altering tempo, volume level etc of a series of MIDI clips • Used extensively in game development • JET Creator is used to author JET files http://developer.android.com/intl/de/guide/topics/media/jet/ jetcreator_manual.html 15
  16. 16. Video - MediaPlayer mMediaPlayer = new MediaPlayer(); mMediaPlayer.setDataSource(localPath);            mMediaPlayer.setDisplay(holder); mMediaPlayer.setOnCompletionListener(this); mMediaPlayer.setOnPreparedListener(this); mMediaPlayer.prepareAsync(); public void onPrepared(MediaPlayer m) { mMediaPlayer.start(); } public void onCompletion(MediaPlayer m) { mMediaPlayer.release(); }
  17. 17. Video - VideoView mVideoView = (VideoView) findViewById(R.id.surface_view); mVideoView.setVideoPath(localPath);            mVideoView.setMediaController (new MediaController(this)); mVideoView.requestFocus(); 17
  18. 18. iMovie on Android!? • Classes needed to process individual video frames • API level 1 contained MediaPlayer.getFrame() • MediaPlayer.snoop() is used in music visualization live wall paper, but is marked @hide 18
  19. 19. way forward... • OMXCodec.h or a similar interface needs to be exposed via NDK • This would facilitate comprehensive video processing and editing 19
  20. 20. Graphics - OpenGL-ES • 3D graphics API to process and render images/textures • On Android GL10 loosely corresponds to OpenGL-ES 1.0/1.1 GL20 loosely corresponds to OpenGL-ES 2.0 • Performance depends on GPU • No major issues with code portability across GPUs 20
  21. 21. GLSurfaceView From Android Developers: • Manages a surface, which is a special piece of memory that can be composited into the Android view system. • Manages an EGL display, which enables OpenGL to render into a surface. • Accepts a user-provided Renderer object that does the actual rendering. • Renders on a dedicated thread to decouple rendering performance from the UI thread. • Supports both on-demand and continuous rendering. • Optionally wraps, traces, and/or error-checks the renderer's OpenGL calls. 21
  22. 22. OpenGL-ES Textures • Create a Bitmap out of the image to be processed • GLUtils class provides helper functions such as: • texImage2D(int target, int level, int internalformat, Bitmap bitmap, int border) • target = GL10.GL_TEXTURE_2D, level = 0, internalformat = GL10.GL_RGB, GL_UNSIGNED_SHORT_5_6_5 • Compressed textures 22
  23. 23. GLSurfaceView.Renderer class MyRenderer implements GLSurfaceView.Renderer {     public void onDrawFrame(GL10 gl) {         // called repeatedly to animate the loaded texture }            public void onSurfaceChanged(GL10 gl, int width, int height) { } public void onSurfaceCreated(GL10 gl, EGLConfig config) { } } 23
  24. 24. Together... • To begin rendering mGLSurfaceView = new GLSurfaceView(this); mGLSurfaceView.setRenderer(new MyRenderer()); setContentView(mGLSurfaceView); • To pause rendering mGLSurfaceView.onPause(); •To resume rendering mGLSurfaceView.onResume(); For full example, checkout CubeMapActivity example in ApiDemos 24
  25. 25. Misc OpenGL-ES • setDebugFlags(DEBUG_CHECK_GL_ERROR | DEBUG_LOG_GL_CALLS); • Use GLSurfaceView.setRenderMode(RENDERMODE_WHEN_ DIRTY) & GLSurfaceView.requestRender()for passive applications • Use EGLChooser to select an appropriate EGL configuration • Minimize texture sizes and bit depth for greater performance • Use NDK wherever feasible to speed up the application 25
  26. 26. Bitmaps & image processing • BitmapFactory.decode(filepath) returns a Bitmap of the image • Use getPixels to access the image’s pixels • getPixels(int[] pixels, int offset, int stride, int x, int y, int width, int height) • Manipulate the pixels and use Bitmap.compress() to encode the processed pixels into a JPEG image 26
  27. 27. ImageView • Display PNG, JPG images • Apply effects like color tinting • View can be animated ImageView image = (ImageView) findViewById(R.id.image); image.setBackgroundResource(R.drawable.my_image); http://developer.android.com/intl/de/guide/topics/graphics/2d-graphics.html
  28. 28. Appendix - Animations <animation-list xmlns:android="http://schemas.android.com/apk/res/android" android:oneshot="true"> <item android:drawable="@drawable/rocket_thrust1" android:duration="200" /> <item android:drawable="@drawable/rocket_thrust2" android:duration="200" /> <item android:drawable="@drawable/rocket_thrust3" android:duration="200" /> </animation-list> ImageView rocketImage = (ImageView) findViewById(R.id.rocket_image); rocketImage.setBackgroundResource(R.drawable.rocket_thrust); rocketAnimation = (AnimationDrawable) rocketImage.getBackground(); rocketAnimation.start(); 28
  29. 29. Thank You! 29

×