Your SlideShare is downloading. ×
  • Like
Applied Computer Science Concepts in Android
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Applied Computer Science Concepts in Android


In this presentation, the unaware or indirect applications of essential computer science concepts are dicussed as showcase. Jim Huang presented in Department of Computer Science and Engineering, …

In this presentation, the unaware or indirect applications of essential computer science concepts are dicussed as showcase. Jim Huang presented in Department of Computer Science and Engineering, National Taiwan University.

Published in Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads


Total Views
On SlideShare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Applied Computer Science Concepts in AndroidJim Huang ( 黃敬群 )Developer & Co-Founder, 0xlab Dec 31, 2010 / CSIE, NTU
  • 2. Applied Computer Science Concepts in AndroidAndroid = A complete operating systemfor mobile computing, open sourced
  • 3. Rights to copy © Copyright 2010 0xlab contact@0xlab.orgAttribution – ShareAlike 3.0You are free Corrections, suggestions, contributions and translations are welcome! to copy, distribute, display, and perform the work to make derivative works Latest update:Dec 31, 2010 to make commercial use of the workUnder the following conditions Attribution. You must give the original author credit. Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one. For any reuse or distribution, you must make clear to others the license terms of this work. Any of these conditions can be waived if you get permission from the copyright holder.Your fair use and other rights are in no way affected by the above.License text:
  • 4. Agenda (1) Android Internals (2) Compiler: 2D/3D Graphics (3) OS: Telephony (4) Virtual Machine: DatabaseIn this presentation, the unaware or indirect applications ofIn this presentation, the unaware or indirect applications ofessential computer science concepts are dicussed as showcase.essential computer science concepts are dicussed as showcase.
  • 5. Android Internals… or, the low-level parts ...
  • 6. Android Internals• Android Low-Level system – Architecture View – Everything from “Zygote” – Key design concepts in Android – System components• Hardware Abstraction Layer (HAL) – GPS, RIL, Graphics (2D/3D), ...
  • 7. Android Low-level system• Architecture• Everything from “Zygote”• Key design concepts in Android
  • 8. Android Low-Level System [ Architecture ]
  • 9. GNU/Linux vs. Android Linux Android
  • 10. Android = (patched) Linux Kernel + special user-space
  • 11. Functional View Applications Gallery Phone Web Browser Google Maps ・・・・・ Android Framework Activity Window Content Notification View System Manager Manager Manager Manager Package Telephony Resource Location Manager Manager Manager Manager System Library Android RuntimeSurfaceFlinger OpenCORE SQLite Class Library Dalvik Virtual MachineOpenGL|ES AudioFlinger WebKit Skia OpenSSL bionic libc Freetype Linux Kernel
  • 12. Android Low-Level System [ Everything from Zygote ]
  • 13. Android Boot SequenceLow level initialization High level perspective (Android specific) ARM based Initialization process Target Board /etc/init.rc Android bootloader Start Android Service USB loader console adbd service manager Load Linux mountd debuggerd zygote mediaserver installd Start kernel flash_recovery - /init/main.c
  • 14. Process View User-space:: process Process PID : 1 PPID : 2 1 PPID : 0 spawn spawn (fork+exec)(fork+exec) Process Process Process spawn (fork+exec) PPID : 3 PID : 2 PID : 3 PPID : 1 PPID : 1 2 Process Process PID : 4 PID : 5 PPID : 2 PPID : 3 3 4 fork() 5 Copy on Write
  • 15. Role of Zygote
  • 16. Everything spawn from Zygote
  • 17. Android Low-Level System [ Key Design Concepts ]
  • 18. System InteractionIPC: Inter-Process Communication
  • 19. Activitieswindow widget Activities viewview group
  • 20. Activity state transition Task 1Home Activity1 Activity2 Activity3 1 2 3 Home Back Key KeyHome Activity4 Activity5 Activity6 4 5 6 Task 2
  • 21. Activity life cycle• Entire lifetime• Visible lifetime• Foreground lifetime
  • 22. Services Activity Activity Service Invoking specific service (provided by Media server) Home orActivity1 Back Activity2
  • 23. Broadcast Receivers ComponentsComponents ReceiverBroadcast SMS_RECEIVED Intent SMS_RECEIVED
  • 24. Content Providers Components Content Providers Interface Content Provider Class Components   A Preferences Content Resolver Files Components   B SQLite(DB) Content Resolver NetworkContent Providers, Intents, Resolver
  • 25. Components Intent Components Activity ActivityContext.startActivity() (Intents) orActivity.startActivityForResult() ServiceContext.startService()Context.bindService() or Broadcast BroadcastContext.sendBroadcast()Context.sendOrderedBroadcast() SystemContext.sendStickyBroadcast() 3 manifest Components 2 ActivityManager Class A Components Service Intent Intent 5 to class A 1 To Class A <class list> … class A 4 …
  • 26. Android Services in ActionContent Telephony Bluetooth Connectivity LocationManager Service Service Service ManagerWindow Activity Package Power …Manager Manager Manager Manager Add Service Service Audio Flinger Add Service Managerdaemons Surface Service Flinger usbd Manager adbd System runtime Serverdebuggerd rild Init Zygote Dalvik VM Kernel Android Service Linux Kernel Dalvik Framework
  • 27. IPC: Binder(1) Binder in Action  A pool of threads is associated to each service application to process incoming IPC (Inter-Process Communication).  Binder performs mapping of object between two processes.  Binder uses an object reference as an address in a process’s memory space.  Synchronous call, reference couting
  • 28. IPC: Binder(2)
  • 29. Understanding• Essential components – Compiler: Java – Virtual Machine: Dalvik – OS: Linux Kernel SSA form• Anything else?
  • 30. CompilersApplied in 2D/3D Graphics
  • 31. OpenGL Anywhere ARM® PowerVRCortex™- A8 SGXwith NEON VFP GraphicsC64x+ Other Display Peripherals Subsystem DSP
  • 32. OpenGL Graphics PipelineState Variables Front Buffer Geometry Fragment Pipeline PipelineCamera Position Light Sources F Etc. C G E D Back Buffer A C GL_TRIANGLES A B B GL_TRIANLE_FAN A C E F B D Texture Maps GL_TRIANGLE_STRIP
  • 33. Texturing ExampleBefore Texturing After Texturing
  • 34. Graphics Pipeline LOD selection Frustum Culling Application Portal Culling … Clipping VERTEX SHADERProgrammable Geometry Division by w Processing Assembly Primitive Viewport transform Backface culling Scan Conversion Rasterization FRAGMENT SHADER Output Output to Device
  • 35. Vertex and Fragment Shaders ( x, y, z, w ) ( x’, y’, z’, w’ ) ( nx, ny, nz ) ( nx’, ny’, nz’ ) ( s, t, r, q ) ( s’, t’, r’, q’ ) ( r, g, b, a ) ( r’, g’, b’, a’ ) VERTEX SHADER( x, y ) ( x, y )( r’, g’, b’, a’ ) ( r, g, b, a )( depth’ ) ( depth ) FRAGMENT SHADER
  • 36. Embedded code with no lighting OpenGL|ES• Added – Fixed-point and byte data• Retained – Vertex Transforms and Lighting (mostly) Embedded code with lighting – Multi-texturing (2D only) – Full Scene Antialiasing via Multisampling (Smooth Shaded) – Alpha blending Surface Normal Vectors
  • 37. OpenGL|ES 1.1 Example// Enable fixed-function shading (smooth or flat) glShadeModel(GL_SMOOTH);// Define the appearance of triangle surfaces glMaterialfv(GL_FRONT_AND_BACK, GL_AMBIENT, fMaterialAmbient); glMaterialf(GL_FRONT_AND_BACK, GL_SHININESS, Shininess);// Define the appearance and position of a light source glLightfv(GL_LIGHT0, GL_AMBIENT, fLightAmbient); glLightModelfv(GL_LIGHT_MODEL_AMBIENT, fAmbient); glLightfv(GL_LIGHT0, GL_POSITION, fLightPosition);// Set pointers to geometry and other attributes and draw it glVertexPointer(3, GL_FLOAT, 0, Vertices); glTexCoordPointer(2, GL_FLOAT, 0, TexCoords); glNormalPointer(GL_FLOAT, 0, NormalsPerVertex); glDrawArrays(GL_TRIANGLES, 0, Count); 38
  • 38. OpenGL|ES 2.0 Example// Create a vertex shader object, load source code and compile it hVertexShader = glCreateShader(GL_VERTEX_SHADER); glShaderSource(hVertexShader, 1, pVertexShaderSourceCode, NULL); glCompileShader(hVertexShader);// Create a shader program and attach the fragment and vertex shaders to it hProgram = glCreateProgram(); glAttachShader(hProgram, hFragmentShader); glAttachShader(hProgram, hVertexShader);// Link and load the new shader programs into the PowerVR SGX glLinkProgram(hProgram); glUseProgram(hProgram);// Set pointers to geometry and other attributes and send to the vertex shader glVertexAttribPointer(Index, 3, GL_FLOAT, GL_TRUE, Stride, pAttributes); glDrawArrays(GL_TRIANGLES, 0, Count); 39
  • 39. GPU = Graphics Processing Unit Pipelining 1 2 3  Number of stages Parallelism 1 2  Number of parallel processes 3 Parallelism + pipelining 1 2 3  Number of parallel pipelines 1 2 3 Operates on 4 tuples 1 2 3 – Position ( x, y, z, w ) – Color( red, green, blue, alpha ) – Texture Coordinates ( s, t, r, q )4 tuple ops, 1 clock cycle  SIMD [ Single Instruction Multiple Data ]
  • 40. Reyes(scanline,polygon) Raytracing Optimized by SIMD Hard to be optimized by SIMD Computing dependency Dynamic fast-paths required
  • 41. Specialize color space convertion takes lots of time. BGRA 444R → RGBA 8888Technique Speedup depends on src/dest format: – 5.4x speedup on average, 19.3x max speedup: (13.3MB/s to 257.7MB/s)
  • 42. LLVM = Low Level Virtual Machine
  • 43. Mandelbort is optimized through LLVM JIT up to 11x.
  • 44. Android OpenGL|ES
  • 45. Android SurfaceFlinger Properties  Can combine 2D/3D surfaces and surfaces from multiple applications  Surfaces passed as buffers via Binder IPC calls  Can use OpenGL ES and 2D hardware accelerator for its compositions  Double-buffering using page-flip
  • 46. UI Element GLSurfaceView javax .microedition .khronos .openglesView and Canvas → SurfaceView and Canvas → Surface android. view. View android. view. Surface android graphics. Canvas . Java Graphics JAVA is implemented Framework via JNI Native Framework Surface JNI Graphic JNI OpenGL JNI SurfaceFlinger skia OpenGL|ES Libui + Libui + SurfaceFlinger + SurfaceFlinger + libpixelflinger libpixelflinger libui Overlay Camera Surface Key / Event format EglWindows libpixelflinger FrameBuffer Event Input Driver Driver
  • 47. PixelFlinger : software renderer• Render functions: pointx, linex, recti, trianglex• Texture and color buffer: activeTexture, bindTexture, colorBuffer, readBuffer, depthBuffer, BindTextureLod• …• Device framebuffer functions: copyPixels, rasterPos2x, rasterPos2i• Optimizer: codeflinger (JIT assembler) I/SurfaceFlinger( 1931): OpenGL informations: I/SurfaceFlinger( 1931): vendor    : Android I/SurfaceFlinger( 1931): renderer : Android PixelFlinger 1.2 I/SurfaceFlinger( 1931): version   : OpenGL ES­CM 1.0
  • 48. GGLAssembler –GGLAssembler –codeflingers JIT Assemblercodeflingers JIT Assembler
  • 49. commit 77cadd2ffada95bb3279552e1a29f4bcf4012228 commit 77cadd2ffada95bb3279552e1a29f4bcf4012228 Author: Jim Huang <> Author: Jim Huang <> Date:   Wed Jan 13 01:01:18 2010 +0800 Date:   Wed Jan 13 01:01:18 2010 +0800     [libpixelflinger] Adds UXTB16 support to Pixelflinger     [libpixelflinger] Adds UXTB16 support to Pixelflinger     ...     ...     Uses UXTB16 to extract channels for SIMD operations, rather than creating and     Uses UXTB16 to extract channels for SIMD operations, rather than creating and     ANDing with masks. Saves a register and is faster on A8, as UXTB16 result can     ANDing with masks. Saves a register and is faster on A8, as UXTB16 result can     feed into first stage of multiply, unlike AND.     feed into first stage of multiply, unlike AND.               Also, used SMULWB rather than SMULBB, which allows removal of MOVs used to     Also, used SMULWB rather than SMULBB, which allows removal of MOVs used to     rescale results.     rescale results.               Code has been scheduled for A8 pipeline, specifically aiming to allow     Code has been scheduled for A8 pipeline, specifically aiming to allow     multiplies to issue in pipeline 0, for efficient dual issue operation.     multiplies to issue in pipeline 0, for efficient dual issue operation.               Testing on SpriteMethodTest (­for­android/) gives     Testing on SpriteMethodTest (­for­android/) gives     8% improvement (12.7 vs. 13.7 fps.)     8% improvement (12.7 vs. 13.7 fps.)libpixelflinger/codeflinger/ARMAssembler.cpplibpixelflinger/codeflinger/ARMAssembler.cpp@@ ­424,5 +424,15 @@ void ARMAssembler::SMLAW(int cc, int y,@@ ­424,5 +424,15 @@ void ARMAssembler::SMLAW(int cc, int y,     *mPC++ = (cc<<28) | 0x1200080 | (Rd<<16) | (Rn<<12) | (Rs<<8) | (y<<4) | Rm;     *mPC++ = (cc<<28) | 0x1200080 | (Rd<<16) | (Rn<<12) | (Rs<<8) | (y<<4) | Rm; } }  +#if 0+#if 0+#pragma mark ­+#pragma mark ­+#pragma mark Byte/half word extract and extend (ARMv6+ only)...+#pragma mark Byte/half word extract and extend (ARMv6+ only)...+#endif+#endif+++void ARMAssembler::UXTB16(int cc, int Rd, int Rm, int rotate)+void ARMAssembler::UXTB16(int cc, int Rd, int Rm, int rotate)+{+{+    *mPC++ = (cc<<28) | 0x6CF0070 | (Rd<<12) | ((rotate >> 3) << 10) | Rm;+    *mPC++ = (cc<<28) | 0x6CF0070 | (Rd<<12) | ((rotate >> 3) << 10) | Rm; opcode+}+}++ }; // namespace android }; // namespace android  
  • 50. libpixelflinger/codeflinger/ARMAssemblerProxy.cpplibpixelflinger/codeflinger/ARMAssemblerProxy.cpp@@ ­195,6 +195,10 @@ void ARMAssemblerProxy::SMLAW(int cc, int y, int Rd, int Rm, int Rs, int Rn) {@@ ­195,6 +195,10 @@ void ARMAssemblerProxy::SMLAW(int cc, int y, int Rd, int Rm, int Rs, int Rn) {     mTarget­>SMLAW(cc, y, Rd, Rm, Rs, Rn);     mTarget­>SMLAW(cc, y, Rd, Rm, Rs, Rn); } }  +void ARMAssemblerProxy::UXTB16(int cc, int Rd, int Rm, int rotate) {+void ARMAssemblerProxy::UXTB16(int cc, int Rd, int Rm, int rotate) {+    mTarget­>UXTB16(cc, Rd, Rm, rotate);+    mTarget­>UXTB16(cc, Rd, Rm, rotate);+}+}++   }; // namespace android }; // namespace android   libpixelflinger/codeflinger/texturing.cpp libpixelflinger/codeflinger/texturing.cpp @@ ­868,6 +869,106 @@ void GGLAssembler::filter24( @@ ­868,6 +869,106 @@ void GGLAssembler::filter24(      load(txPtr, texel, 0);      load(txPtr, texel, 0);  }  }     +#if __ARM_ARCH__ >= 6 +#if __ARM_ARCH__ >= 6 +// ARMv6 version, using UXTB16, and scheduled for Cortex­A8 pipeline +// ARMv6 version, using UXTB16, and scheduled for Cortex­A8 pipeline +void GGLAssembler::filter32( +void GGLAssembler::filter32( +        const fragment_parts_t& parts, +        const fragment_parts_t& parts, +        pixel_t& texel, const texture_unit_t& tmu, +        pixel_t& texel, const texture_unit_t& tmu, +        int U, int V, pointer_t& txPtr, +        int U, int V, pointer_t& txPtr, +        int FRAC_BITS) +        int FRAC_BITS) +{ +{ ... ... +    UXTB16(AL, temp, pixel, 0); +    UXTB16(AL, temp, pixel, 0); Reimplement filter32() +    if (round) { +    if (round) { +        ADD(AL, 0, u, u, imm(1<<(adjust­1))); +        ADD(AL, 0, u, u, imm(1<<(adjust­1))); +        MOV(AL, 0, u, reg_imm(u, LSR, adjust)); +        MOV(AL, 0, u, reg_imm(u, LSR, adjust)); +    } In optimized ASM +    } +    LDR(AL, pixellb, txPtr.reg, reg_scale_pre(offsetlb)); +    LDR(AL, pixellb, txPtr.reg, reg_scale_pre(offsetlb)); +    MUL(AL, 0, dh, temp, u); +    MUL(AL, 0, dh, temp, u); +    UXTB16(AL, temp, pixel, 8); +    UXTB16(AL, temp, pixel, 8); +    MUL(AL, 0, dl, temp, u); +    MUL(AL, 0, dl, temp, u); +    RSB(AL, 0, k, u, imm(0x100)); +    RSB(AL, 0, k, u, imm(0x100)); ... ... +#else +#else  void GGLAssembler::filter32(  void GGLAssembler::filter32(          const fragment_parts_t& parts,          const fragment_parts_t& parts,          pixel_t& texel, const texture_unit_t& tmu,          pixel_t& texel, const texture_unit_t& tmu,
  • 51. Operating SystemApplied in Telephony
  • 52. RIL → Telephony Radio Interface Layer(RIL)  HAL  Android TelephonyManager  baseband modem  Voice, Data, SMS, SIM, SIMToolkit  Android RIL  AT command RIL API  Android follows GSM TS 27.007 standard RIL implementation  Qualcomm supports CDMA & CDMA-GSM Multi-mode
  • 53. OS Kernels Monolithic Kernel Micro KernelTraditional UNIX, Linux, … Mach, L4, Symbian OS Exokernel
  • 54. L4 Microkernel• Developed by Jochen Liedtke in 1995. – German National Research Center for IT• Developed from scratch• 2nd generation microkernel• Small and Fast Kernel – 7 system calls – 12KB• In November 2005, NICTA announced that Qualcomm was deploying NICTAs L4 version on their Mobile Station Modem chipsets.
  • 55. Pipes and RPC
  • 56. Communications Processor L4 Interrupt Handler -> Send Message L4 microkernel RPC System- Server call Handler RPC Interrupt RPC RT task Requests fromShared Memory Application Processor to Modem
  • 57. Virtual MachineApplied in Database
  • 58. sqlite3_prepare()/*** Compile the UTF­8 encoded SQL statement zSql into a statement handle.*/int sqlite3_prepare(  sqlite3 *db,            /* Database handle. */  const char *zSql,       /* UTF­8 encoded SQL statement. */  int nBytes,             /* Length of zSql in bytes. */  sqlite3_stmt **ppStmt,  /* OUT: A pointer to the prepared                               statement */  const char** pzTail     /* OUT: End of parsed string */){...      sqlite3VdbeSetNumCols(sParse.pVdbe, 5);      sqlite3VdbeSetColName(sParse.pVdbe, 0, COLNAME_NAME, "addr", P3_STATIC);      sqlite3VdbeSetColName(sParse.pVdbe, 1, COLNAME_NAME, "opcode", P3_STATIC);      sqlite3VdbeSetColName(sParse.pVdbe, 2, COLNAME_NAME, "p1", P3_STATIC);      sqlite3VdbeSetColName(sParse.pVdbe, 3, COLNAME_NAME, "p2", P3_STATIC);      sqlite3VdbeSetColName(sParse.pVdbe, 4, COLNAME_NAME, "p3", P3_STATIC);… Opcode! SQLite Virtual Machine (VDBE) 61
  • 59. …And what about other platforms?The concept can be applied to all VM platformsApplication virtual machines (short list) .NET (CLR) Java Virtual Machine (JVM) Dalvik virtual machine (Google Android) PHP (Zend Engine) Flash Player / AIR - ActionScript Virtual Machine (AVM) SQLite virtual machine (VDBE; Virtual DataBase Engine) Perl virtual machine
  • 60. Inspiration
  • 61. Inspiration• Computer science concepts are applied everywhere in Android software stack.• Essential system software brings the further optimizations and feasibility of new technologies.• Advanced compiler techniques are already applied in every domain• Best Practice in Android
  • 62.