Shorten Device Boot Time for Automotive IVI and Navigation Systems

5,060 views

Published on

Propose a practical approach of the mixture of
ARM hibernation (suspend to disk) and Linux
user-space checkpointing
– to shorten device boot time

Published in: Technology

Shorten Device Boot Time for Automotive IVI and Navigation Systems

  1. 1. Shorten Device Boot Time forAutomotive IVI and Navigation SystemsJim Huang ( 黃敬群 ) <jserv@0xlab.org>Dr. Shi-wu Lo <shiwulo@gmail.com>May 28, 2013 / Automotive Linux Summit (Spring)
  2. 2. Rights to copyAttribution – ShareAlike 3.0You are freeto copy, distribute, display, and perform the workto make derivative worksto make commercial use of the workUnder the following conditionsAttribution. You must give the original author credit.Share Alike. If you alter, transform, or build upon this work, you may distribute theresulting work only under a license identical to this one.For any reuse or distribution, you must make clear to others the license terms of this work.Any of these conditions can be waived if you get permission from the copyright holder.Your fair use and other rights are in no way affected by the above.License text: http://creativecommons.org/licenses/by-sa/3.0/legalcode© Copyright 2013 0xlabhttp://0xlab.org/contact@0xlab.orgCorrections, suggestions, contributions and translationsare welcome!Latest update: May 28, 2013
  3. 3. Goal of This Presentation• Propose a practical approach of the mixture ofARM hibernation (suspend to disk) and Linuxuser-space checkpointing– to shorten device boot time• An intrusive technique for Android/Linux– minimal init script and root file system changesare required• Boot time is one of the key factors for Automotive IVI– mentioned by “Linux Powered Clusters” and “Silver Bulletof Virtualization (Pitfalls, Challenges and Concerns)Continued” at ALS 2013– highlighted by “Boot Time Optimizations” at ALS 2012
  4. 4. About this presentation• joint development efforts of the following entities– 0xlab team - http://0xlab.org/– OSLab, National Chung Cheng University of Taiwan, ledby Dr. Shi-wu Lo. "Software platform on SoC" prject atITRI/ICL, supported by MOEA of Taiwan.– DMTCP Project at Northeastern University, led by Dr.Gene Cooperman• The implementation is public for reference– Kernel part: GNU GPLv2– Userspace: GNU LGPL• Some areas of the implementation might bepatented. We dont expect to discuss here.
  5. 5. Disclaimer of Warranties• Technical background knowledge in thispresentation is based on the experience with ourcustomers / partners such as CSR and MediaTek.– We share non-confidential parts to endorse thecollaboration of Linux ecosystem.• We make no warranty, either express or impliedwith respect to any product, and specially disclaimthe correctness or real-world behavior.
  6. 6. Agenda (1) Concepts: Analysis, Strategies(2) Boot Time Reduction:from traditional to our experiments(3) ARM Hibernation(4) Checkpointing(5) Mixed model
  7. 7. Concepts: Analysis & Strategies(no silver bullet, but you can optimize iteratively)
  8. 8. Boot Time Measurement
  9. 9. http://elinux.org/Boot_Time
  10. 10. Traditional Linux Environment:printk, initcall_debug, bootchart, strace, oprofile,perf, ..
  11. 11. Bootchart$ bootchart bootchart.tgz -f png
  12. 12. StraceTrace system calls during process execution and outputtiming information.$ strace -tt ls15:11:04.243357 execve("/bin/ls", ["ls"], [/* 51 vars */]) = 015:11:04.244252 brk(0) = 0x234f00015:11:04.244458 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT15:11:04.244676 mmap(NULL, 8192, PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f144479400015:11:04.244852 access("/etc/ld.so.preload", R_OK) = -1 ENOENT15:11:04.245096 open("/etc/ld.so.cache", O_RDONLY) = 3
  13. 13. OProfileOutput Example$ opreport --exclude-dependentCPU: PIII, speed 863.195 MHz (estimated)Counted CPU_CLK_UNHALTED events (clocks processor is nothalted)...450385 75.6634 cc1plus60213 10.1156 lyx29313 4.9245 XFree8611633 1.9543 as10204 1.7142 oprofiled7289 1.2245 vmlinux7066 1.1871 bash6417 1.0780 oprofile6397 1.0747 vim...
  14. 14. PerfRecording:Output$ perf record -a -f^C[ perf record: Woken up 1 times to write data ][ perf record: Captured and wrote 0.288 MB perf.data (~12567samples)]$ perf report --sort comm,dso,symbol|head -10# Events: 1K cycles## Overhead Command Shared Object Symbol# ........ ........... ...................... ............#35.47% firefox libxul.so [.] 0xc1b3a73.08% firefox libcairo.so.2.11000.2 [.] 0xff882.98% Xorg Xorg (deleted) [.] 0xe201c2.51% firefox firefox [.] 0x27261.49% Xorg [kernel.kallsyms] [k] find_vma0.93% perf_3.0.0 perf_3.0.0 [.] hex2u64
  15. 15. PerfTimechart$ perf timechart record$ perf timechart
  16. 16. Android Environment
  17. 17. BootchartOriginal “bootchartd” is not suitable on embeddeddevices.Android re-implemented in its “init”.
  18. 18. BootchartTo build:$ cd system/core/init$ touch init.c$ mm INIT_BOOTCHART=true
  19. 19. BootchartTo run:$ adb shell echo 120 > /data/bootchart-startRemember the /data directory must be write able duringboot.Use grab-bootchart.sh to retrieve the data.
  20. 20. StraceModify init.rc fromservice zygote /system/bin/app_process -Xzygote /system/bin --zygote --start-system-serverservice zygote /system/xbin/strace -tt -o/data/boot.strace /system/bin/app_process -Xzygote /system/bin --zygote --start-system-serverTo
  21. 21. LogcatAndroid log utilityCan output timing informationAdjust loglevel to 6 in init.rcDisplays time spent for each command
  22. 22. Dalvik Method TracerMethod tracer is built into DalvikUse DDMS or using calls inside source to collect data.// start tracing to "/sdcard/calc.trace"Debug.startMethodTracing("calc");// ...// stop tracingDebug.stopMethodTracing();
  23. 23. StopwatchNot real stopwatchA utility in Android Framework for measuring C++ code.Output result to system log#include <utils/StopWatch.h>…{StopWatch watch("blah");/* your codes here */}
  24. 24. Android Boot Time Analysis
  25. 25. Boot-loader InitUsually constant timeAvoid init hardware multiple timesEnsure to use maximum CPU frequencyUse faster NAND/MMC reading mechanism
  26. 26. Kernel InitMostly usual suspectsip_auto_configUSB initFlash driver initializationFullow the standard Kernel optimizing guide:http://elinux.org/Boot_TimeAvoid loading unneeded kernel module at boot time
  27. 27. Zygote Class PreloadingAndroid Framework has thousands of Java classesPreloaded by Zygote and instantiated in its heapTo improve Application startup time and save memoryControlled by resource: preloaded-classesframeworks/base/preloaded-classes
  28. 28. Zygote Class PreloadingCan use the tool in framework to adjust the list:$ adb logcat > logcat.txt$ java -p preload.jar Compile logcat.txt logcat.compiled$ java -p preload.jar PrintCsv logcat.compiledGoogle Android Developer Dianne Hackborn said:The content of the file is a “black art”You can adjust this as much as you likeBut the result maybe suboptimal
  29. 29. PackageManager Package ScannigEvery APK is scanned at boot timePackage management code is inefficientUses mmaped files means each access will cause pagefaultParseZipArchive() scans entire APK for only oneAndroidManifest.xml file
  30. 30. System Services StartingLast stage of Android bootStart every base serviceZygote start SystemServer processStart native service (SurfaceFlinger, AudioFlinger) firstStart each service sequentially
  31. 31. Boot Time Reduction
  32. 32. Boot Time Reduction:The traditional approaches mentioned by manydevelopers/speakers already
  33. 33. Qi Boot-loaderDeveloped by Openmoko and0xlabOnly one stage boot-loaderSmall footprint ~30KCurrently support− IMX31, Samsung 24xx,Beagleboard, PandaboardKISS concept− Boot device and load kernelBoot-loader Improvements:Write our faster one!QiBoot-oaderU-Boot + XLoaderSize ~30K ~270K+20KTime to Kernel < 1 s > 5sUsage Product EngineeringCode Simple ComplicatedAnother example:“Boot Time Optimizations”, Alexandre Belloni, ELCE 2012
  34. 34. Kernel Boot TimeFullow the standard Kernel optimizing guide:http://elinux.org/Boot_TimeMinimize kernel sizeUse compression or notEnable embedded optionsAvoid loading unneeded kernel module at boot time
  35. 35. Optimize Android InitParallize init tasksinsmod cannot be parallizedUse external scripts to init at backgroundStart services on demand
  36. 36. Optimize Class PreloadingTrade-off between preload class and application startuptimeSplit class to more packages to reduce dependencySave inited heap for later useShare heaps between zygote and children
  37. 37. Filesystem OptimizationAccording to reasearch by Linaro Kernel WGUse correct NAND configuration will improve theperformanceMMC controllers are often optimized for particularusage / filesystemAdjust the filesystem partition scheme
  38. 38. Toothpaste EffectObserved by Sony Developer Tim Bird“When you squeeze a tube of toothpaste, sometimes itjust moves the toothpaste somewhere else in the tube,and nothing actually comes out.”Example:“Trying to Improve Android Boot Time With Readahead”, Tim Bird,ABS 2011
  39. 39. Lessons learnt• Complex userspace environments like Androidusually contribute the most to boot time.– (reported) < 1s boot time often refers to minimalkernel + simplified initscript, which doesnt help• GregKH: "If you want a faster D-Bus, rewrite thedaemon / library, dont mess with the kernel." atALS 2013• Propose an intrusive technique for Android/Linux– Mixture of Hibernation and Userspacecheckpointing
  40. 40. Part I:Hibernation based Technologies
  41. 41. ARM Hibernation Solutions• swsup hibernation– Reference: “Extending the swsusp HibernationFramework to ARM”, Russell Dill at ELC 2013• QuickBoot (proprietary)• FastON
  42. 42. RAMHard disk4GBWrite out(seq.)ARM Hibernation Diagram (1)
  43. 43. ARM Hibernation Diagram (2)RAMHard disk4GBSwap space(/hibernationfile)
  44. 44. ARM Hibernation Diagram (3)RAMHard disk4GBRead-in(seq.)
  45. 45. ARM Hibernation Diagram (4)RAMHard disk4GB
  46. 46. QuickBootDeveloped by Japanesecompany, UbiquitousDemand loading ofrequired page from flashRequires deepintegration of hardwareand softwareNo visible information;No production record
  47. 47. FastONDeveloped by OSLab of National Chung ChengUniversity in Taiwan– Based on existing technologies thus requires littlemodifications to userspace– Release clean-pages before suspend– Swap out dirty-pages before save image– Image size reduced leads to faster resume time.Reference: “Swap-before-hibernate: a time efficientmethod to suspend an OS to a flash drive”, Dr.Shi-wu Lo
  48. 48. Memory AnalysisDirty pagesClean pagesworkingset1. Most of the memorystatus is unmodified(clean), with an exactcopy in secondarystorage.2. The size of working setis very very small.
  49. 49. FastON Diagram (1)flash driveWrite-out(rand.)
  50. 50. FastON Diagram (2)50Read-in(rand.) flash drive (fs &swp)Only working set isread and resumed!
  51. 51. FastONtimeThemachine isreadyRead the hibernation file(load the Linux kernel)The time wemeasured in the“experimentalresults”Swap in the data needed to“run” applications(load the working set)0The definition of “run” is we can start to operate theapplications very smoothly. (for example, scroll downand up, key in)
  52. 52. Boot Time: Original vs. FastONOriginal:FastON:sizeof(hiberfile)speedseq.+sizeof(workingset)speedrand.(16k)+ device_initsizeof(main_memory)speedseq.+ device_init
  53. 53. Boot Time: Original vs. FastONBoottimeComplexity / Memory sizeOriginalFastONOriginalThe hotpath
  54. 54. Boot Time: Original vs. FastON
  55. 55. Android Wakelocks & TuxOnIce
  56. 56. TuxOnIce PatchTuxOnIce (was Software Suspend 2) is a hibernationpatchsetCan save images to different locationsCan use different compresion algorithms
  57. 57. Suspend to Disk Resume
  58. 58. Android WakelocksAn aggressive approach to save device powerUse wakelocks to prevent device going suspendPorting TOI to Android has to deal with wakelocksbecause MMC driver might hold a wakelockSource available: http://gitorious.org/0xlab-kernel
  59. 59. Example: Android on BeagleboardOMAP353x; ARM Cortex-A8 600 Mhz; 256MB RAMKernel: 2.6.32 + TOI patchBootloader: QiAndroid 2.x system
  60. 60. Original Boot Sequence
  61. 61. Hibernation Boot
  62. 62. Further optimizations:No Need for full-featuredBootloader when resuming
  63. 63. R-LoaderNormal suspend-to-disk approach has many duplicatedeffortBoot-loader inits some hardwaresBoot-loader loads the normal kernel imageKernel inits some hardwares againKernel loads the suspended kernel imageKernel resumes, inits some hardwares again
  64. 64. R-Loader0xlab proposed “Resume-Loader”– A customized “snapshot” bootR-Loader inits some hardware then reads thesuspended kernel image as fast as possibleJump directly to the resume pointKernel will takeover the job and inits reset hardwares
  65. 65. Advanced Topics for integratingHibernation with AndroidSave the heap image (like core dump) of Zygote afterpreloading classesModify Dalvik to make hibernation image after systeminit and before Launcher startupParallize Android initCache & Share JITed code fragment
  66. 66. Part II:Userspace solution: Checkpointing
  67. 67. Hibernation looks fine, why another?• Hibernation is sometimes harmful to userspaceespecially for recovering from file system orupdating packages, that is crucial for Linux-based“smart” devices• Post-actions are still necessary: connectivity,firmware management, security, etc.• Broken drivers are impossible to work withhibernation framework, but they usually appearfrom chip vendors though.
  68. 68. Basic Idea of checkpointing:Process Migration• Process Migration (in past applications)– Distributed Load Balancing– Efficient Resource Utilization• Crash Recovery and Rollback Transaction– Useful to system adminNodeProcessesCommunication ChannelNodeProcessesNodeProcessesTCP/IPLibrary Library LibraryTCP/IP TCP/IPCheckpoint
  69. 69. Ideas about Checkpointing forAndroid/Linux• Resume to stored state for faster Android boot time• Better product field trial experience due to regularcheckpointing• Deploy problematic states for engineering analysis anddebugging transparently• Q&A stress test purpose
  70. 70. Communication state checkpointCheckpoint StateControlRhRtRecvbuffersStShSendbuffersShRt+1Timers,Options,etc.Rh St+1 ShRt+1 XXreceive()directaccessdirectaccessState for one socketRhRt...StSh...Rt+1RhShSt+1Timers,Options,etc.ControlRecv buffers Send bufferscopied_seqrcv_nxt snd_unawrite_seqLive Communication State• Acquire network stack locks tofreeze TCP processing• Save receive buffers usingsocket receive system call inpeek mode• Save send buffers by walkingkernel structures• Copy control state from kernelstructures• Modify two sequence numbersin saved state to reflect emptysocket buffers
  71. 71. Communication state restartControlLive Communication Statecopied_seqrcv_nxt snd_unawrite_seqStSh...Send buffersCheckpoint StateControlRhRtRecvbuffersStShSendbuffersShRt+1Timers,Options,etc.Rt+1 ShRt+1Rt+1ShTimers,Options,etc.RhRtRecvdatadirectupdateSt+1write()To App by interceptedreceive system callShState for one socket• Create a new socket• Copy control state incheckpoint to socket structure• Restore checkpointed sendbuffer data using the socketwrite call• Deliver checkpointed receivebuffer data to application ondemand
  72. 72. Existing Checkpointing mechanisms• CryoPID– http://cryopid.berlios.de/• BLCR (Berkeley Lab Checkpoint/Restart)– https://ftg.lbl.gov/projects/CheckpointRestart/• DMTCP– http://dmtcp.sourceforge.net/
  73. 73. Implementation Considerations• Checkpointing can be implemented in– kernel modifications + helpers in userspace– pure userspace• Introduce a virtualization layer groups processes into specificstates with private virtual name space– Intercepts system calls to expose only virtual identifiers (e.g.,vpid)– Preserves resource names and dependencies acrossmigration• Mechanism to checkpoint and restart states– User and kernel-level state– Primarily uses system call handlers– File system not saved or restored
  74. 74. DMTCP• Distributed Multi-Threaded CheckPointing.• Works with Linux Kernel 2.6.9 and later.• Supports sequential and multi-threaded computationsacross single/multiple hosts.• Entirely in user space (no kernel modules or rootprivilege).– Transparent (no recompiling, no re-linking).• Developed in Northeastern University and MIT andunder active development since 2006.• License: GNU LGPL (allows freely using and linking)
  75. 75. Process StructureCT = DMTCP checkpoint threadT = User ThreadDMTCPCoordinatorCTCTT1T1 T2Signal (USR2)Network SocketProcess 1 Process Ndmtcp_checkpoint <EXE> # starts coordinatordmtcp_command –c # talks to coordinatordmtcp_restart ckpt_<EXE>-*.dmtcp• Coordinator: a stateless synchronization server for thedistributed checkpointing algorithm.• Checkpoint/Restart performance related to size of memory,disk write speed, and synchronization.
  76. 76. How DMTCP works (1/4)• MTCP : component for checkpoint single-process• SIGUSR2: Used internally from checkpoint thread touser threads.CTT1 T2User programCheckpoint thread1. CT sends SIGUSR2 toeach threads for suspending2. Write checkpoint imageto disk3. Exit SIGUSR2 handler,and resume.
  77. 77. How DMTCP works (2/4)• LD_PRELOAD: Transparently preloads checkpoint libraries`dmtcphijack.so` which installs libc wrappers and checkpointingcode.• Wrappers: Only on less heavily used calls to libc– open, fork, exec, system, pipe, bind, listen,setsockopt, connect, accept, clone, close, ptsname,openlog, closelog, signal, sigaction, sigvec,sigblock, sigsetmask, sigprocmask, rt_sigprocmask,pthread_sigmask– Overhead is negligible....fd = open(path,flags);...User Programint open(const char *path,int flags){...funcs[_open](path, flags);...}dmtcphijack.soint open(const char *path,int flags){...}libc.so
  78. 78. How DMTCP works (3/4)• Additional wrappers when process id & thread idvirtualization is enabled– getpid, getppid, gettid, tcgetpgrp,tcsetprgrp, getgrp, setpgrp, getsid,setsid, kill, tkill, tgkill, wait, waitpid,waitid, wait3, wait4...pid = getpid();...User Programint getpid(){...real_pid = funcs[_getpid]();return pid_table[real_pid];}dmtcphijack.soint getpid(){...}libc.so
  79. 79. How DMTCP works (4/4)• Checkpoint image compression on-the-fly (default).• Currently only supports dynamically linking to libc.so.Support for static libc.a is feasible, but notimplemented.
  80. 80. Checkpoint under DMTCP(1/7)• dmtcphijack.so and libmtcp.so present in executable’smemory.– dmtcp_checkpoint <EXE>CTT1 T2CoordinatorUser programConnect to exist coordinatoror create a new coordinatorCTT1 T2User program
  81. 81. Checkpoint under DMTCP(2/7)• Ask coordinator process for checkpoint viadmtcp_command.– dmtcp_command -c• DMTCP also provides API to send command or querystatusCTT1 T2CoordinatorUser programcommandSend checkpointcommandCTT1 T2User program
  82. 82. Checkpoint under DMTCP(3/7)• Suspend user threads with SIGUSR2.CTT1 T2CoordinatorUser program1. send commandfor preparecheckpoint2. CT sends SIGUSR2 toeach threads for suspendingCTT1 T2User program
  83. 83. Checkpoint under DMTCP(4/7)• Pre-checkpoint stage• Synchronize every node and elect shared filedescriptor leaders.• Drain kernel buffers and do network handshake withpeers.CTT1 T2CoordinatorUser program2. Drain buffersCTT1 T2User programWait until all nodes are ready.1. Report all threadexcept CT are suspended
  84. 84. Checkpoint under DMTCP(5/7)• Write checkpoint to disk– One checkpoint file per process– ckpt_<EXE>_<uid>.dmtcpCTT1 T2CoordinatorUser programWrite checkpoint todisk separatelyCTT1 T2User programWait until all nodes ofcheckpoint are done
  85. 85. Checkpoint under DMTCP(6/7)• Post-Checkpint stage• Refill kernel buffersCTT1 T2CoordinatorUser programRefill buffer andre-handshake with peers.CTT1 T2User programWait until all nodes ofpost-checkpoint are done
  86. 86. Checkpoint under DMTCP(7/7)• Resume user threads.CTT1 T2CoordinatorUser programResume!CTT1 T2User programSend resume command
  87. 87. Restart under DMTCP(1/6)• Restart Process loads in memory.– dmtcp_restart ckpt_<EXE>_<uid>.dmtcpCTCoordinatordmtcp_restart1. Connect to exist coordinatoror create a new coordinator2. Load process to memory
  88. 88. Restart under DMTCP(2/6)• Fork user programCTCoordinatordmtcp_restartCTdmtcp_restart
  89. 89. Restart under DMTCP(3/6)• Reopen files and recreate ptys• Recreate and reconnect sockets• Rearrange file descriptors to initial layoutCTCoordinatordmtcp_restartWail all node recreate socketand reopen file1. Reopen file/ptys/socketsCTdmtcp_restart2. Rearrange FD via dup2
  90. 90. Restart under DMTCP(4/6)• Restore memory content.• Restore stack status for checkpoint thread.CTCoordinatordmtcp_restartCTdmtcp_restartCTCoordinatorCTUser programUser programdmtcp_restartmtcp_restartUser programexecvemmap + magic!!FDs will preserved across execve!
  91. 91. Restart under DMTCP(5/6)• Restore other threads.– Recreate thread and restore stack and context.– Restore back to the post-checkpint stage• Refill kernel bufferCTCoordinatorWail all nodes restoring are done1. Memory and threadcome back now!CTT1 T2T1 T2User programUser program2. Refill buffers
  92. 92. Restart under DMTCP(6/6)• Resume user threadsCTCoordinatorResume!CTT1 T2T1 T2User programUser program<user_func1><user_func2><user_funcN>...<sig-handler>stopthisthreadThread Call Stack...while (mtcp_state_value(&thread -> state)== ST_SUSPENDED) {mtcp_state_futex (&(thread -> state),FUTEX_WAIT,ST_SUSPENDED,NULL);}...
  93. 93. DMTCP WorkflowStart with dmtcpwrapperRunPre-CheckpointCheckpointPost-CheckpointRestartRe-open FDsRestore memoryRestore threadsProvide hookpoint!Get checkpointcommand
  94. 94. OS Features supported by DMTCP• Threads, mutexes/semaphores, fork, exec• Shared memory (via mmap), TCP/IP sockets, UNIXdomain sockets, pipes, ptys, terminal modes,ownership of controlling terminals, signal handlers,open and/or shared fds, I/O (including the readlinelibrary), parent-child process relationships, process id& thread id virtualization, session and process groupids, and more…
  95. 95. DMTCP/Android: Additional Features(LGPL; separated from Android)• ARM Architecture support– Verified on Samsung Galaxy S2 + Android 4.0• Binder IPC– Client: supported– Server: partially supported• Ashmem: supported• Logger: supported• Properties: supported• Wakelocks: Not supportedAlready merged in upstream DMTCP
  96. 96. Android Binder support for DMTCP• BinderConnection– Reopen /dev/binder and reset ioctl parameters– Restore the mmap region• Hijack the whole libbinder– Prevent libbinder from interpreting data twice– Implement necessary DMTCP hooks: preCheckpoint,postCheckpoint, postRestart• Re-initialize libbinder in postRestart• The server part is partially supported because binderserver is calling a blocked ioctl and blocking the wholecheckpoint process.– We implement an early checkpoint stage to suspendsuch kind of threads.
  97. 97. More extensions in DMTCP/Android• Improve the hook system in DMTCP– Original design only allows one set hook function.– Allow more than one set hook function inDMTCP/Android.• Implement per thread callback hook– Restore the DVM internal thread info• Add barrier and synchronize mechanisms to DMTCP– In order to make precise program checkpointing.
  98. 98. Android specific modifications• Reorder code in framework– registerZygoteSocket()• The socket is inherited from the parent process `init`,which implies we can not handle it in DMTCP.– Move few initializations later than the checkpoint processsince the current binder support is incomplete.• Reserve the ashmems file descriptor– Original behavior is to close the fd after mmap– DMTCP binds connection to one fd, so the connection will bedestroyed if that fd is closed.• Implement the missing PThread function in bionic libc– pthread_tryjoin_np is required by DMTCP,but it s not implemented in original bionic.
  99. 99. Technical Issues when modifyingDMTCP• ARM Architecture support is incomplete.• Different TLS implementation semantics between glibc andbionic libc– DMTCP/Android follows the techniques used in Android´sOpenGL ES package which links and defers to the slot ofTLS in bionic libc. Not elegant, but it works• PThread implementation expectation is quite different– AOSP master branch is merging libc from NetBSD, so itshould be better for compatibility.• Behavior of dynamic linker differs a lot in bionic libc.• Flags in dlopen() is not really functional.• The way to find symbol in bionic libc differs: weak symbol
  100. 100. Checkpoint for Zygote• Experiment environment:– Android on ARM Cortex-A9 dual (1GHz; Memory: 512 MB)with gzip without gzipCheckpoint time~8s ~4.5sRestart time~0.3s ~0.1sImage size~3M ~17M
  101. 101. --------- beginning of /dev/log/systemI/Vold ( 1270): Vold 2.1 (the revenge) firing upD/Vold ( 1270): Volume usb state changing -1 (Initializing) -> 0 (No-Media)I/Netd ( 1271): Netd 1.0 startingI/ ( 1275): ServiceManager: 0x8062b50I/ ( 1276): ServiceManager: 0x804fb98I/AudioFlinger( 1276): Loaded primary audio interface from LEGACY Audio HW HAL (audio)I/AudioFlinger( 1276): Using LEGACY Audio HW HAL (audio.primary) as the primary audio interface...D/AudioHardware( 1276): ### setVoiceVolume: 1.000000I/AudioPolicyService( 1276): [1276]Loaded audio policy from LEGACY Audio Policy HAL (audio_policy)E/BatteryService( 1382): usbOnlinePath not foundD/AndroidRuntime( 1902):D/AndroidRuntime( 1902): >>>>>> AndroidRuntime START com.android.internal.os.ZygoteInit <<<<<<D/AndroidRuntime( 1902): CheckJNI is ONI/SamplingProfilerIntegration( 1902): Profiling disabled.I/Zygote ( 1902): Preloading classes...D/dalvikvm( 1902): GC_EXPLICIT freed 35K, 85% free 399K/2560K, paused 0ms+0ms...I/Zygote ( 1902): ...preloaded 379 resources in 548ms.D/dalvikvm( 1902): GC_EXPLICIT freed 20K, 1% free 6417K/6467K, paused 0ms+0msI/Zygote ( 1902): ...preloaded 31 resources in 13ms.D/dalvikvm( 1902): GC_EXPLICIT freed 14K, 1% free 6418K/6467K, paused 0ms+0msD/dalvikvm( 1902): GC_EXPLICIT freed 5K, 1% free 6412K/6467K, paused 0ms+0msD/dalvikvm( 1902): GC_EXPLICIT freed <1K, 1% free 6412K/6467K, paused 0ms+2msI/dalvikvm( 1902): System server process 1911 has been created--------- beginning of /dev/log/systemI/Vold ( 1270): Vold 2.1 (the revenge) firing upD/Vold ( 1270): Volume usb state changing -1 (Initializing) -> 0 (No-Media)I/Netd ( 1271): Netd 1.0 startingI/ ( 1275): ServiceManager: 0x8062b50I/ ( 1276): ServiceManager: 0x804fb98I/AudioFlinger( 1276): Loaded primary audio interface from LEGACY Audio HWI/AudioFlinger( 1276): Using LEGACY Audio HW HAL (audio.primary) as the….D/AudioHardware( 1276): ### setVoiceVolume: 1.000000I/AudioPolicyService( 1276): [1276]Loaded audio policy from LEGACY Audio PD/dalvikvm( 1373): GC_EXPLICIT freed 14K, 1% free 6418K/6467K, paused 0D/dalvikvm( 1373): GC_EXPLICIT freed 5K, 1% free 6412K/6467K, paused 0mD/dalvikvm( 1373): GC_EXPLICIT freed <1K, 1% free 6412K/6467K, paused 0I/dalvikvm( 1373): System server process 1382 has been createdNormal bootup log message Bootup log message with restartObservations from logcat
  102. 102. Part III:Mixted Model
  103. 103. Mixed Model: Hibernation + Checkpointing• Basic Idea– Swap out dirty-pages before save image– Reducing image size leads to faster resume time– Only the “static” working set is suspended, other partsutilizes checkpointing– Room for keeping __broken__ device drivers• However, the state machine is relatively intricatebecause of application framework awareness.
  104. 104. Technical Challenges of Mixed Model• Have to modify Android/Tizen framework in order tofigure the proper checkpoint to perform.– It is not straightforward since recent application frameworksdepend on web rendering engine as the core component.– Working-set is hard to be minimized accordingly.• Separate the device initializations into two areas:– essential zone: taken for Hibernation– post-resuming: used for checkpointing, firmwaremanagement, and connectivity → problem of maintenance• Storage access time is crucial since we touch forseveral times– I/O scheduler modifications are under investigation
  105. 105. Timing diagram of Hibernation stage(when request speed > response speed, it means I/O operations are busy)
  106. 106. Time diagram of Device Resuming
  107. 107. Conclusion• No silver bullet for device boot time optimizations• Developing an intrusive technique for Android orcomplex Linux systems is possible for designaspects• The quality of device driver and file system causesessential impacts as well no matter how a newtechnique is introduced.• Mixed model can be used for the production ofdedicated systems such as digital TV andautomotive IVI.
  108. 108. Reference• Embedded Linux Wiki: http://elinux.org/Boot_Time• “DMTCP: An New Linux Checkpointing Mechanismfor Vanilla Universe Job”, Condor Project, University ofWisconsin-Madison• “Checkpointing using DMTCP, Condor, Matlab andFReD”, Gene Cooperman, Northeastern University, Boston• “Boot Time Optimizations”, Alexandre Belloni, ELCE 2012• “Extending the swsusp Hibernation Framework toARM”, Russell Dill, ELC 2013
  109. 109. http://0xlab.org

×