Porting and Binding Dalvik VM with GLibC
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Porting and Binding Dalvik VM with GLibC

on

  • 1,964 views

This presentation slide is aimed to represent my knowledge gain over 1 year experience in AOSP build system and Dalvik VM porting. It serves some fundamental aspect of Android's internals. Dalvik VM ...

This presentation slide is aimed to represent my knowledge gain over 1 year experience in AOSP build system and Dalvik VM porting. It serves some fundamental aspect of Android's internals. Dalvik VM porting attempts and methods are also included (Spoil: They are all fail).

PS: This slide is not complete, and is under updated.

Statistics

Views

Total Views
1,964
Views on SlideShare
1,953
Embed Views
11

Actions

Likes
3
Downloads
72
Comments
2

2 Embeds 11

https://www.linkedin.com 10
http://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Porting and Binding Dalvik VM with GLibC Presentation Transcript

  • 1. Porting and Binding DalvikVM with GlibC Pipat Methavanitpong Kunieda-Isshiki Laboratory November 26, 2013
  • 2. Outline         Introduction to Android Introduction to Dalvik VM Introduction to Bionic LibC Introduction to QEMU Merits of Porting GLibC Dalvik VM Porting Attempted Approaches Available Alternative Ports How to Setup AOSP Build System
  • 3. Introduction Android  Open Source OS for smart devices    Developed by Google Based on Linux Kernel   Smartphones, Tablets, Gadgets With several custom modules Ranked 1st among other Smart Device OSes  81% market share in Q3, 2013 – IDC Report
  • 4. Google I/O 2013: Keynote – May 15, 2013 1000 900 May-13 Mar-13 Jan-13 Nov-12 600 Sep-12 Jul-12 May-12 400 Mar-12 500 Jan-12 300 Nov-11 Sep-11 Jul-11 May-11 40 Mar-11 200 Jan-11 Nov-10 20 Sep-10 1 10 Jul-10 0 May-10 100 Mar-10 Jan-10 1,000,000 x Activations Recorded Android Activation 900 800 700 500 400 200 100 60 80
  • 5. Android API Level Name Chart Release Year  2009      2010     3.0 – Honeycomb 3.1 – Honeycomb 2012   2.2 – Froyo 2.3 – Gingerbread 2011   1.5 – Cupcake 1.6 – Donut 2.0 – Éclair 2.1 – Éclair 3.2 – Honeycomb 2013      4.0 – Ice-cream Sandwich 4.1 – Jellybean 4.2 – Jellybean 4.3 – Jellybean 4.4 – KitKat Wikipedia – Android Version History (cited Nov 25, 2013)
  • 6. Android System Architecture Google I/O 2008 – Dalvik Virtual Machine Internals E-Linux – Android Architecture (cited Nov 25, 2013)
  • 7. Android Low-Level Architecture Porting Android to Devices (cited Nov 25, 2013)
  • 8. Android Architecture Fancy Apps AOSP/packages Framework AOSP/dalvik AOSP/libcore AOSP/frameworks AOSP/externals Foundation AOSP/bionic AOSP/system AOSP/libnativehelper AOSP/hardware … K. Yaghmour, Embedded Android 1st edition, Chapter 2, Figure 2-1
  • 9. Introduction to Dalvik VM K. Yaghmour, Embedded Android 1st edition, Chapter 2, Figure 2-1
  • 10. Introduction to Dalvik VM  The core of Android Applications  All fancy Android applications are run by it  Register-based Process Virtual Machine  Think of running a Java application  Intermediate Language = Dalvik Bytecode  Executable File = Dalvik Executable (DEX)  A converted Java class done by “dx” tool  Reduce redundancy in variables
  • 11. Introduction to Dalvik VM  Variable Length ISA    1 Word = 2 bytes = 16 bits Fetched 1 word a time OP code is 8-bit length = Max of 256 instructions  Use only 246 instructions  Attached in the 1st word  Default Endianess is Little  Default OP code handles up to 65536 Virtual Registers (VRs)  Ideally infinite  VRs are 32-bit width  64-bit variable = concatenate 2 consecutive VRs together  Has Pseudo Instruction concept  Must be optimized by “dexopt” before run  4 Types of constant pool: String, Type, Field, Method  2 Types of prelinked offset: VTab offset, Field offset Source Android – Bytecode for the Dalvik VM (cited Nov 26, 2013) Source Android – Dalvik VM Instruction Formats (cited Nov 26, 2013)
  • 12. Introduction to Dalvik VM  2 Modes of Machine Code Interpretation (Mterp)    Fast – Based on manually written assembly code for each OP code Portable – Based on compiled generic C++ files for each OP code Enhancement  Java Native Interface   Just-In-Time compiler   Allow Dalvik VM to call native functions Prepare bytecode segments into machine code ahead during run-time Zygote  Mother process which    Create a Dalvik VM And preloads frequently used shared libraries New Dalvik VM starts by forking from this template Dalvik VM  Reduce loading time  Save memory space (Copy-On-Write) Forays in software development – Stack based vs Register based Virtual Machine Architecture, and the Dalvik VM (cited Nov 26, 2013) Linux For You – Virtual Machines For Abstraction: The Dalvik VM (cited Nov 26, 2013) Wikipedia – Just-in-time compilation (cited Nov 26, 2013)
  • 13. Stack-based Virtual Machine  Don’t name register in instuctions     Simple design Instruction width is short Redundancy in using same variables    Everything is queued up in its stack No name to registers = Copy on a stack repetitively More instructions used to complete a task But, total program size is smaller than Registerbased Y. Shi, D. Gregg, A. Beatty, Virtual Machine Showdown: Stack versus Registers, VEE’05 Forays in software development – Stack based vs Register based Virtual Machine Architecture, and the Dalvik VM (cited Nov 26, 2013) Linux For You – Virtual Machines For Abstraction: The Dalvik VM (cited Nov 26, 2013)
  • 14. Register-based Virtual Machine     Physical CPU analogy Explicitly naming register Instruction width is longer Can do complex register manipulation   Fast    No need to be in order like in Stack-based Fewer VM instructions An effect from larger instruction width can be neglected A bit memory consuming   Larger program size Normally require a lot of space for virtual registers Y. Shi, D. Gregg, A. Beatty, Virtual Machine Showdown: Stack versus Registers, VEE’05
  • 15. From Source Code to Execution Java Source Code javac Java Class dx DEX file dexopt Optimized DEX dalvikvm Forays in software development – Stack based vs Register based Virtual Machine Architecture, and the Dalvik VM (cited Nov 26, 2013)
  • 16. DEX Structure HEADER The Header (size of 0x70 bytes) STRING_IDS String Identifier List TYPE_IDS Type Identifier List PROTO_IDS Method Prototype Identifier List FIELD_IDS Field Identifier List METHOD_IDS Method Identifier List CLASS_DEFS Class Definition List DATA Data area for support contents above LINK_DATA Data used in statically linked files Source Android – Dalvik Executable Format (cited Nov 26, 2013)
  • 17. How to port Dalvik VM to new Architecture   Google mentions only for porting to other Android with different architecture  In short = Build for Target  Document in AOSP/dalvik/docs/porting-guide.html Steps  Create new Machine Interpreter (Mterp) in both C and its machine code for each instruction code  Create new Mterp config for your architecture  Update Mterp by AOSP/dalvik/vm/mterp/rebuild.sh      This assembles your written instruction codes InterpAsm-Your_Arch.S + InterpC-Your_Arch.cpp Modify Target build configuration in AOSP/dalvik/vm/Dvm.mk Set $(TARGET_ARCH) by lunch or export TARGET_ARCH=Your_Arch Build new libdvm.so by m libdvm
  • 18. libART (Android RunTime Library)  Android 4.4’s experimental virtual machine library  Disabled by default   Turn on by checking this option in Developer options Changing to libART from libDVM  Requires all apps to be recompiled = Restart + Super long first boot  Use Ahead-Of-Time (AOT) scheme instead of JIT  Precompile Dalvik Bytecode into machine language during installation  Take longer time to install for the first time  Applications run faster  Apps are in native form, so they are very ready to be executed  Saving more energy  Apps end faster = More PEs idle time  Occupy more storage space  Decompress to machine code Android Headlines – Google is Looking to Replace Dalvik with ART (cited Nov 26, 2013)
  • 19. Introduction to Bionic LibC   Derivation of BSD standard C library   Developed by Google  Small 2.11 GLibC 1,208,224 bytes   uClibc 0.9.30 243,938 bytes Upto 114% faster than GLibC License dietlibc No SysV IPC support Android has Binder IPC musl No wchar support  Has several LGPL kernel specific Linux BSD GPL MIT features GLibC, uClibc Not compatible with GLibC  JNI vs. Native C bound with GLibC Bionic ARM / x86 / MIPS Partially POSIX compatible  Optimized for low clock processors       Currently support for  Fast  Completely rewritten by Google  Bionic 2.1 424,235 bytes Based on Futexes  NetBSD (libc) + FreeBSD (libm)  Fast Pthread implementation No C++ exception support  Not full C++ STL  Not fully POSIX Thread compliant Wikipedia – Bionic (software) (cited Nov 25, 2013) ICCAS2012 – Benchmarking Java Application using JNI and Native C Application on Android Coding Relic – The Six Million Dollar LibC (cited Nov 25, 2013)
  • 20. Introduction to QEMU    Quick EMUlator Free and Open Source Hardware Virtualizer Can emulate  Whole system / board – Versatile Express-A9   Only CPU – x86, sparc, ARM   qemu-arm-softmmu -M vexpress-a9 qemu-i386 Has Kernel-based Virtual Machine (KVM) support  Accelerate emulation with same CPU architecture as its Host Wikipedia – QEMU (cited Nov 25, 2013)
  • 21. Merits of Porting GLibC Dalvik VM  Getting full capability of GLibC     C++ Exception handling Wide charactor handling POSIX Thread compliant Can be run on any ordinary Linux   GlibC is shipped with Linux by almost distros Consistence with other existing Linux programs    Can talk to each other under same SysV IPC -----------------------------------------------------------------What could come after this   Comparison between Bionic and GLibC Dalvik VM Porting GLibC Dalvik VM for TCT processor  GLibC porting to TCT is under going
  • 22. Porting Attempted Approaches Copy-Paste Bionic Dalvik VM 1.  Failed  Cannot find libraries Copy-Paste Bionic Dalvik VM + Bionic LibC 2.  Failed  Something low-level failed Copy-Paste Bionic Dalvik VM + Bionic LibC + Ashmem 3.  Failed  Something low-level failed (same output as previous attempt) GLibC - Dalvik VM porting 4.   Not done Make modification to build/core Makefiles  Only HOST configurations
  • 23. Testing Environment  ARM  Use QEMU to emulate VExpress-A9 with bare Linux kernel with Busybox shell   Use QEMU to emulate Vexpress-A9 with Ubuntu 13.04   http://balau82.wordpress.com/2010/03/27/busybox-for-arm-on-qemu/ http://releases.linaro.org/13.04/ubuntu/vexpress/ x86   Use QEMU to emulate QEMU’s x86 default board with Debian 7 Squeeze Run on Host machine directly
  • 24. 1st Attempt   Copy-Paste Bionic Dalvik VM Does not work   No Bionic LibC for it Error message is   bash: [Directory to Dalvik VM]/dalvikvm: No such file or directory This error message is ALSO OCCUR when run a GLibC linked program on an Android Emulator
  • 25. 2nd Attempt   Copy-Paste Bionic Dalvik VM Copy Bionic LibC to /system/lib  /system/bin/linker is hardcoded to link with /system/lib   Create dalvik-cache directory for Optimized DEX   /data/dalvik-cache Error message is   It is done by setting “rpath” flag Dalvik VM init failed (checked log file) There is no Log file left  No porting of liblog and Logger
  • 26. 3rd Attempt   Setting is same as 2nd attempt This attempt was tested only on   Emulated VExpress-A9 with bare Linux kernel with Busybox shell Linux kernel is shipped with Ashmem  Ashmem = Annonymous Shared Memory     Now shipped with Linux mainline residing in Staging directory It is said that Ashmem is needed for Dalvik VM porting Error message is   Sometimes, people call it Android Shared Memory Dalvik VM init failed (checked log file) Failure cause is still unknown Dalvik Virtual Machine on TS-7800 SBC (cited Nov 26, 2013)
  • 27. 4th Attempt   Host version of Dalvik VM is linked with GLibC Fake an attribute of Host: HOST_ARCH    GLibC linked Dalvik VM for that architecture Try with ARM architecture Modify several Makefiles to make sure AOSP builds for only Host side   Many Makefiles contains builds for Host and Target in one file m dalvik-host conscrypt-hostdex    Minimum Dalvik VM build for Host Create new Makefiles and Includes for new host architectures Still not complete  Configuration problems
  • 28. Modifying Guide  Build acp (Android’s cp tool) $ m acp  Modify build/core/config.mk Force use acp for x86 Host  From $ ACP := $(BUILD_OUT_EXECUTABLES)/acp$(BUILD_EXECUTABLE_SUFFIX)  To $ ACP := $(HOST_OUT_ROOT)/linux-x86/bin/acp$(BUILD_EXECUTABLE_SUFFIX)  Modify build/core/envsetup.mk to accept Host Architecture Overwrite  From $ BUILD_ARCH := $(HOST_ARCH)  To $ BUILD_ARCH := $(HOST_ARCH) $ ifneq ($(HOST_ARCH_OWR),) $ HOST_ARCH := $(HOST_ARCH_OWR) $ endif
  • 29. Guide  Build acp (Android’s cp tool) $ m acp  Modify build/core/config.mk Force use acp for x86 Host  From $ ACP := $(BUILD_OUT_EXECUTABLES)/acp$(BUILD_EXECUTABLE_SUFFIX)  To $ ACP := $(HOST_OUT_ROOT)/linux-x86/bin/acp$(BUILD_EXECUTABLE_SUFFIX)  Modify build/core/envsetup.mk to accept Host Architecture Overwrite  From $ BUILD_ARCH := $(HOST_ARCH)  To $ BUILD_ARCH := $(HOST_ARCH) $ ifneq ($(HOST_ARCH_OWR),) $ HOST_ARCH := $(HOST_ARCH_OWR) $ endif
  • 30. Guide  Modify build/tools/acp/Android.mk Disable acp build  From $ include $(BUILD_HOST_EXECUTABLE)  To $ ifeq ($(HOST_ARCH),x86) $ include $(BUILD_HOST_EXECUTABLE) $ endif  Create new build/core/combo/HOST_linux-arm.mk   Manually merge between HOST_linux-x86.mk and TARGET_linux-arm.mk Create new build/core/combo/include/arch/host_linuxarm/AndroidConfig.h  Manually merge between linux-x86, target_linux-x86, and target_linux-arm
  • 31. Available Alternative Ports  idvk – Independent Dalvik    Myriad Alien Dalvik   Run Android apps on non-Android devices IcedRobot   Not actually porting Just extract what Dalvik VM needs to be built from AOSP Use OpenJDK instead of Sun JDK which was under license problem between Google and Oracle libhybris.so    Overwrite Bionic symbol to solve incompatibility with GLibC Written by Mer developer Used by Sailfish OS and Ubuntu Touch
  • 32. How to Setup AOSP  1. 2. This guide assumes an environment of x86 Ubuntu-64bit 12.04++ Download and Install Oracle JDK6 (see later slide) Download and Install Other Required Software 1. Required Packages $ sudo apt-get install git gnupg flex bison gperf build-essential zip curl libc6-dev libncurses5-dev:i386 x11proto-core-dev libx11-dev:i386 libreadline6-dev:i386 libgl1-mesa-glx:i386 libgl1-mesa-dev g++-multilib mingw32 tofrodos python-markdown libxml2-utils xsltproc zlib1g-dev:i386 $ sudo ln -s /usr/lib/i386-linux-gnu-mesa-libGL.so.1 /usr/lib/i386-linux-gnu/libGL.so 2. Repo $ mkdir ~/bin $ PATH=~/bin:$PATH $ curl http://commondatastorage.googleapis.com/git-repo-downloads/repo > ~/bin/repo $ chmod a+x ~/bin/repo Source Android – Downloading and Building (cited Nov 26, 2013)
  • 33. How to Setup AOSP (cont.) Set up repo repository 3. $ mkdir AOSP $ cd AOSP $ repo init -u https://android.googlesource.com/platform/manifest Sync with server 3. $ repo sync Add GPG key 4. $ gpg --import 1. Copy the following key -----BEGIN PGP PUBLIC KEY BLOCK----- Version: GnuPG v1.4.2.2 (GNU/Linux) mQGiBEnnWD4RBACt9/h4v9xnnGDou13y3dvOx6/t43LPPIxeJ8eX9WB+8LLuROSV lFhpHawsVAcFlmi7f7jdSRF+OvtZL9ShPKdLfwBJMNkU66/TZmPewS4m782ndtw7 8tR1cXb197Ob8kOfQB3A9yk2XZ4ei4ZC3i6wVdqHLRxABdncwu5hOF9KXwCgkxMD u4PVgChaAJzTYJ1EG+UYBIUEAJmfearb0qRAN7dEoff0FeXsEaUA6U90sEoVks0Z wNj96SA8BL+a1OoEUUfpMhiHyLuQSftxisJxTh+2QclzDviDyaTrkANjdYY7p2cq /HMdOY7LJlHaqtXmZxXjjtw5Uc2QG8UY8aziU3IE9nTjSwCXeJnuyvoizl9/I1S5 jU5SA/9WwIps4SC84ielIXiGWEqq6i6/sk4I9q1YemZF2XVVKnmI1F4iCMtNKsR4 MGSa1gA8s4iQbsKNWPgp7M3a51JCVCu6l/8zTpA+uUGapw4tWCp4o0dpIvDPBEa9 b/aF/ygcR8mh5hgUfpF9IpXdknOsbKCvM9lSSfRciETykZc4wrRCVGhlIEFuZHJv aWQgT3BlbiBTb3VyY2UgUHJvamVjdCA8aW5pdGlhbC1jb250cmlidXRpb25AYW5k cm9pZC5jb20+iGAEExECACAFAknnWD4CGwMGCwkIBwMCBBUCCAMEFgIDAQIeAQIX gAAKCRDorT+BmrEOeNr+AJ42Xy6tEW7r3KzrJxnRX8mij9z8tgCdFfQYiHpYngkI 2t09Ed+9Bm4gmEO5Ag0ESedYRBAIAKVW1JcMBWvV/0Bo9WiByJ9WJ5swMN36/vAl QN4mWRhfzDOk/Rosdb0csAO/l8Kz0gKQPOfObtyYjvI8JMC3rmi+LIvSUT9806Up hisyEmmHv6U8gUb/xHLIanXGxwhYzjgeuAXVCsv+EvoPIHbY4L/KvP5x+oCJIDbk C2b1TvVk9PryzmE4BPIQL/NtgR1oLWm/uWR9zRUFtBnE411aMAN3qnAHBBMZzKMX LWBGWE0znfRrnczI5p49i2YZJAjyX1P2WzmScK49CV82dzLo71MnrF6fj+Udtb5+ OgTg7Cow+8PRaTkJEW5Y2JIZpnRUq0CYxAmHYX79EMKHDSThf/8AAwUIAJPWsB/M pK+KMs/s3r6nJrnYLTfdZhtmQXimpoDMJg1zxmL8UfNUKiQZ6esoAWtDgpqt7Y7s KZ8laHRARonte394hidZzM5nb6hQvpPjt2OlPRsyqVxw4c/KsjADtAuKW9/d8phb N8bTyOJo856qg4oOEzKG9eeF7oaZTYBy33BTL0408sEBxiMior6b8LrZrAhkqDjA vUXRwm/fFKgpsOysxC6xi553CxBUCH2omNV6Ka1LNMwzSp9ILz8jEGqmUtkBszwo G1S8fXgE0Lq3cdDM/GJ4QXP/p6LiwNF99faDMTV3+2SAOGvytOX6KjKVzKOSsfJQ hN0DlsIw8hqJc0WISQQYEQIACQUCSedYRAIbDAAKCRDorT+BmrEOeCUOAJ9qmR0l EXzeoxcdoafxqf6gZlJZlACgkWF7wi2YLW3Oa+jv2QSTlrx4KLM= =Wi5D -----END PGP PUBLIC KEY BLOCK----- 2. Then press Ctrl+D Source Android – Downloading and Building (cited Nov 26, 2013)
  • 34. How to Setup AOSP (cont. 2) 5. Setup AOSP environment $ . build/envsetup.sh 6. Select Target build $ lunch ** Example of building full options for generic ARM target $ . build/envsetup.sh $ lunch full-eng $ make -j8 -jN flag specifies N jobs to run simultaneously = Many cores processors benefit from it Source Android – Downloading and Building (cited Nov 26, 2013)
  • 35. Installing Oracle JDK6  Download JDK6 from Oracle  Execute the downloaded bin file = File extract  Make a directory for it   Copy the extracted files to that directory   sudo mkdir /usr/lib/jvm sudo mv jdk-1.6.0_xx /usr/lib/jvm/jdk-1.6.0_xx Add Java alternatives   sudo update-alternatives --install /usr/bin/java java /usr/lib/jvm/jdk-1.6.0_xx/bin/java 1   sudo update-alternatives --install /usr/bin/javac javac /usr/lib/jvm/jdk-1.6.0_xx/bin/javac 1 sudo update-alternatives --install /usr/bin/javaws javaws /usr/lib/jvm/jdk-1.6.0_xx/bin/javaws 1 Update Java alternatives   sudo update-alternatives --config java   sudo update-alternatives --config javac sudo update-alternatives --config javaws Enable Java plugin for Mozilla Firefox and Google Chrome  64-bit: sudo ln -s /usr/lib/jvm-1.6.0_xx/jre/lib/amd64/libnpjp2.so /usr/lib/mozilla/plugins  32-bit: sudo ln -s /usr/lib/jvm-1.6.0_xx/jre/lib/i386/libnpjp2.so /usr/lib/mozilla/plugins dev sniper – Ubuntu 12.04 – install sun jdk 6-7 (cited Nov 26, 2013)