My INSURER PTE LTD - Insurtech Innovation Award 2024
Android OpenGL ES Game ImageGrabber Final Report
1. Android OpenGL ES Game
ImageGrabber Final Report
Jungsoo Nam
2015/03/26
namjungsoo@gmail.com
2. Agenda
• Development environment setup
• Development tools setup
• Build guide
• Source code review
• Java classes/methods
• C functions
• ELF architecture with flow chart
• Linux hooking mechanism
3. Development environment setup (1/2)
• Development tools setup
• Eclipse with ADT(Android Developer Tools) for Mac
• http://dl.google.com/android/adt/adt-bundle-mac-x86_64-20140702.zip
• SDK updates
• Requires android-18(4.4)
• NDK setup
• android-ndk-r10d-darwin-x86_64.bin
• Bash Profile setup
• vi .bash_profile
• export PATH=$PATH:/Users/yegam400/Downloads/android-ndk-r10d
4. Development environment setup (2/2)
• Build guide
• Download source codes(link to download)
• AngryBots.zip
• MediaWebEncoder.zip
• Import projects
• File – Import – Android – Existing Android Code into workspace
• NDK build
• Open terminal
• cd MediaWebEncoder
• ndk-build => libmediaweb.so
• In Eclipse, Right click at MediaWebEncoder project, click “Refresh (F5)”
• Run Unity3D app(UnityPlayerNativeActivity(=AngryBots))
• In Eclipse, Check project reference with MediaWebEncoder
5. Source code review – Java (1/3)
• com.medaweb.Grabber.VideoGrabber
• public
• startEncoder() – starts video recording
• stopEncoder() – stops video recording
• getRecordingEnabled() – gets recording enabled
• initialize() – initialize grabber
• private
• begin/endClear() – called when glClear() -> deprecated
• begin/endSwapBuffers() – called when eglSwapBuffer()
• prepareFrameBuffer() – called when glBindFrameBuffer()
6. Source code review – Java (2/3)
• com.android.grafika.gles – Reuse Grafika sources
• Texture2dProgram (modified for our implementation)
• draw2() – similar to draw()
• backupStates() – backup GLSL, arraybuffer and texture stage states
• setupStates() – same as draw()’s one
• restoreStates() – restore GLSL, arraybuffer and texture stage states
• To backup rendering states are the states which are changed by
setupState():
• 1. active texture unit
• 2. multi texture binding
• 3. current program binding
• 4. array buffer binding
• 5. vertex shader attributes
• 6. blend enabled
• 7. vertex shader attributes' pointer arrays
9. Source code review – C (2/5)
• mediaweb.c
• Called by VideoGrabber
• Called by Texture2dProgram
public class VideoGrabber {
static {
System.loadLibrary("mediaweb");
}
public static native void setInputSurface(long surfaceNativeHandle);
public static native int getWidth();
public static native int getHeight();
public static native void makeCurrent();
public static native void startNativeEncoder(int captureFbo);
public static native void stopNativeEncoder();
public static native void backupVertexAttribPointer(int index);
public static native void restoreVertexAttribPointer(int index, int size, int
type, int normalized, int stride);
10. Source code review – C (3/5)
• mediaweb.c
• public
• setInputSurface() - set MediaCodec input surface handle
• makeCurrent() – call eglMakeCurrent() with original surfaces
• backupVertexAttribPointer() - backup by glGetVertexAttribPointerv()
• restoreVertexAttribPointer() - restore by glVertexAttribPointer()
• startNativeEncoder() – start native side encoder with custom fbo handle
• stopNativeEncoder() – stop native side encoder
• private
• setupViewport() - calls prepareFramebuffer()
• do_hook2() - do_hook() wrapper. Hook a function
• so_entry() - hooks OpenGL ES functions
• JNI_OnLoad() - called when JNI initializing
These are not
supported by Java
OpenGL ES Wrapper
11. Source code review – C (4/5)
• mediaweb.c
• private
• hooked_glViewport()
• when first called, calls setupViewport() -> prepareFramebuffer()
• hooked_glClear()
• hooked glClear() function. rendering entry point -> deprecated.
• hooked_glBindFramebuffer()
• rendering entry point.
• If framebuffer == 0, then this is main fbo, so it changes mainfbo to capturefbo for capture when stateRecording
== 1.
• hooked_eglSwapBuffers()
• rendering end point.
• If stateRecording == 1, then calls beginSwapBuffers() -> orig_eglSwapBuffers() -> endSwapBuffers()
• hooked_eglMakeCurrent()
• when first called, remember main surfaces
• Unity3D callstack
• glClear()
• glBindFramebuffer()
• eglSwapBuffers()
4 times
12. Source code review – C (5/5)
• hook.c
• private
• get_module_base() – get certain module(*.so) base memory address from
/proc/self/maps
• find_got_entry_address_() – load certain module and find functions address
offset.(according to ELF file format)
• public
• do_hook() – entry of function hooking
• Replace entry_addr to hook_func
• Entry address = base address + offset address
13. ELF architecture with flow chart
• Document
• http://www.cs.bgu.ac.il/~caspl142/wiki.files/lab7/elf.pdf