Gtug

2,300 views
2,232 views

Published on

SG-GTUG

Published in: Education
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,300
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
85
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide





























  • Gtug

    1. 1. Multimedia Application Development on Android KRK MOHAN Senior Engineer - Muvee Technologies www.muvee.com krk.mohan@gmail.com 1
    2. 2. Agenda • Multimedia and Android • Video & Audio: MediaPlayer, MediaRecorder, JetPlayer, SoundPool, AudioTrack • Images: OpenGL-ES graphics, Bitmaps, ImageView 2
    3. 3. Scope • Introductory level • Focus on what can and cannot be done with Android FroYo’s multimedia capabilities • References to APIDemos projects and developer site articles provided 3
    4. 4. Multimedia Apps • Media Consumption - Slide Shows, Media Players,YouTube, facebook etc • Media Production - Image Editors, Camera enhancements • Huge scope for improvement - great opportunity for developers 4
    5. 5. You will identify a great product when you see it Deep, Indulgent, Complete, Elegant (DICE) Guy Kawasaki “The Macintosh Way” 5
    6. 6. android.media • Contains classes to play audio/video content from raw resource files, file streams and JET content • Contains classes to record audio/video content from cameras or mic http://developer.android.com/intl/de/guide/topics/media/ index.html 6
    7. 7. File formats and codecs • File formats for recording media on device • 3GPP, MP4, RAW_AMR • Codecs used for recording media on device • H263, H264, MPEG4 (SP), AMR_NB 7
    8. 8. The big 2 MediaPlayer and MediaRecorder classes encapsulate: File parsers for media file formats such as 3GP, MP4 & Codecs to encode or decode media content Two media frameworks present in FroYo OpenCORE and StageFright 8
    9. 9. MediaPlayer - state diagram From Android Developers - MediaPlayer entry 9
    10. 10. Audio • MediaPlayer & MediaRecorder • SoundPool - multiple audio sources • AudioTrack & AudioRecord- raw audio manipulation • JetPlayer - interactive music 10
    11. 11. Audio - MediaPlayer Playing an audio resource MediaPlayer mp = MediaPlayer.create(context, R.raw.sound_file_1); mp.start(); Playing a file from file system MediaPlayer mp = new MediaPlayer(); mp.setDataSource(PATH_TO_FILE); mp.prepare(); mp.start(); 11
    12. 12. Audio - SoundPool • Multiple sounds can be played from raw or compressed source. Sounds are identified by sound ids. • More efficient than multiple MediaPlayers • Each sound can be assigned a priority • Adjustable playback frequency • Support for repeat mode 12
    13. 13. Audio - AudioTrack • Used to play raw audio samples (PCM) stored in memory buffers or files • Static mode for playing sounds that fit into memory and need to be played often with low overhead • Streaming mode for sounds that are too big to fit into memory 13
    14. 14. Audio - AudioTrack AudioTrack audioTrack = new AudioTrack(AudioManager.STREAM_MUSIC,                             44100,                                           AudioFormat.CHANNEL_CONFIGURATION_MONO,             AudioFormat.ENCODING_PCM_16BIT,                               musicLength, AudioTrack.MODE_STREAM); audioTrack.play(); audioTrack.write(music, 0, musicLength); //blocking Check out AudioRecord class Some examples at: http://emeadev.blogspot.com/2009/09/raw-audio-manipulation-in-android.html 14
    15. 15. Audio - JETPlayer • Interactive music playback by altering tempo, volume level etc of a series of MIDI clips • Used extensively in game development • JET Creator is used to author JET files http://developer.android.com/intl/de/guide/topics/media/jet/ jetcreator_manual.html 15
    16. 16. Video - MediaPlayer mMediaPlayer = new MediaPlayer(); mMediaPlayer.setDataSource(localPath);            mMediaPlayer.setDisplay(holder); mMediaPlayer.setOnCompletionListener(this); mMediaPlayer.setOnPreparedListener(this); mMediaPlayer.prepareAsync(); public void onPrepared(MediaPlayer m) { mMediaPlayer.start(); } public void onCompletion(MediaPlayer m) { mMediaPlayer.release(); }
    17. 17. Video - VideoView mVideoView = (VideoView) findViewById(R.id.surface_view); mVideoView.setVideoPath(localPath);            mVideoView.setMediaController (new MediaController(this)); mVideoView.requestFocus(); 17
    18. 18. iMovie on Android!? • Classes needed to process individual video frames • API level 1 contained MediaPlayer.getFrame() • MediaPlayer.snoop() is used in music visualization live wall paper, but is marked @hide 18
    19. 19. way forward... • OMXCodec.h or a similar interface needs to be exposed via NDK • This would facilitate comprehensive video processing and editing 19
    20. 20. Graphics - OpenGL-ES • 3D graphics API to process and render images/textures • On Android GL10 loosely corresponds to OpenGL-ES 1.0/1.1 GL20 loosely corresponds to OpenGL-ES 2.0 • Performance depends on GPU • No major issues with code portability across GPUs 20
    21. 21. GLSurfaceView From Android Developers: • Manages a surface, which is a special piece of memory that can be composited into the Android view system. • Manages an EGL display, which enables OpenGL to render into a surface. • Accepts a user-provided Renderer object that does the actual rendering. • Renders on a dedicated thread to decouple rendering performance from the UI thread. • Supports both on-demand and continuous rendering. • Optionally wraps, traces, and/or error-checks the renderer's OpenGL calls. 21
    22. 22. OpenGL-ES Textures • Create a Bitmap out of the image to be processed • GLUtils class provides helper functions such as: • texImage2D(int target, int level, int internalformat, Bitmap bitmap, int border) • target = GL10.GL_TEXTURE_2D, level = 0, internalformat = GL10.GL_RGB, GL_UNSIGNED_SHORT_5_6_5 • Compressed textures 22
    23. 23. GLSurfaceView.Renderer class MyRenderer implements GLSurfaceView.Renderer {     public void onDrawFrame(GL10 gl) {         // called repeatedly to animate the loaded texture }            public void onSurfaceChanged(GL10 gl, int width, int height) { } public void onSurfaceCreated(GL10 gl, EGLConfig config) { } } 23
    24. 24. Together... • To begin rendering mGLSurfaceView = new GLSurfaceView(this); mGLSurfaceView.setRenderer(new MyRenderer()); setContentView(mGLSurfaceView); • To pause rendering mGLSurfaceView.onPause(); •To resume rendering mGLSurfaceView.onResume(); For full example, checkout CubeMapActivity example in ApiDemos 24
    25. 25. Misc OpenGL-ES • setDebugFlags(DEBUG_CHECK_GL_ERROR | DEBUG_LOG_GL_CALLS); • Use GLSurfaceView.setRenderMode(RENDERMODE_WHEN_ DIRTY) & GLSurfaceView.requestRender()for passive applications • Use EGLChooser to select an appropriate EGL configuration • Minimize texture sizes and bit depth for greater performance • Use NDK wherever feasible to speed up the application 25
    26. 26. Bitmaps & image processing • BitmapFactory.decode(filepath) returns a Bitmap of the image • Use getPixels to access the image’s pixels • getPixels(int[] pixels, int offset, int stride, int x, int y, int width, int height) • Manipulate the pixels and use Bitmap.compress() to encode the processed pixels into a JPEG image 26
    27. 27. ImageView • Display PNG, JPG images • Apply effects like color tinting • View can be animated ImageView image = (ImageView) findViewById(R.id.image); image.setBackgroundResource(R.drawable.my_image); http://developer.android.com/intl/de/guide/topics/graphics/2d-graphics.html
    28. 28. Appendix - Animations <animation-list xmlns:android="http://schemas.android.com/apk/res/android" android:oneshot="true"> <item android:drawable="@drawable/rocket_thrust1" android:duration="200" /> <item android:drawable="@drawable/rocket_thrust2" android:duration="200" /> <item android:drawable="@drawable/rocket_thrust3" android:duration="200" /> </animation-list> ImageView rocketImage = (ImageView) findViewById(R.id.rocket_image); rocketImage.setBackgroundResource(R.drawable.rocket_thrust); rocketAnimation = (AnimationDrawable) rocketImage.getBackground(); rocketAnimation.start(); 28
    29. 29. Thank You! 29

    ×