Linux boot-time
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,919
On Slideshare
1,890
From Embeds
29
Number of Embeds
3

Actions

Shares
Downloads
41
Comments
0
Likes
1

Embeds 29

https://twitter.com 19
http://eventifier.co 8
http://eventifier.com 2

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Andrea Righi - andrea@betterlinux.com Ottimizzare i tempi di boot di Linux
  • 2. Andrea Righi - andrea@betterlinux.com Agenda ● Overview ● Case study: Raspberry Pi ● Kernel optimizations ● rootfs optimizations ● Q/A
  • 3. Andrea Righi - andrea@betterlinux.com Overview of boot phases ● Firmware (boot-loader) ● Hardware probing ● Hardware initialization ● Kernel load and decompression ● Kernel execution ● Core init (start_kernel) ● Driver init (initcalls) ● User-space init ● /sbin/init ● RC scripts ● Application start (first user impression)
  • 4. Andrea Righi - andrea@betterlinux.com Generic concepts ● Identify boot time functionalities ● Measure boot time of each functionality ● Remove unnecessary functionalities ● Optimize required functions ● Defer loading of less-important features (modularization - LKM) ● Re-order initialization / parallelization ● Asynchronous initialization
  • 5. Andrea Righi - andrea@betterlinux.com Generic concepts ● Identify boot time functionalities ● Measure boot time of each functionality ● Remove unnecessary functionalities ● Optimize required functions ● Defer loading of less-important features (modularization - LKM) ● Re-order initialization / parallelization ● Asynchronous initialization Premature optimization is the root of all evil. -Donald Knuth
  • 6. Andrea Righi - andrea@betterlinux.com Optimization ● Remove features ● The fastest code is the code you don't run! ● Defer loading of less important features ● Modularization (LKM) ● Parallelization ● async initialization (kernel/async.c) ● Reduce probing delays ● Filesystem tricks
  • 7. Andrea Righi - andrea@betterlinux.com Case study ● Raspberry Pi ● Boot time functionality ● Reponsive ssh login ● Tools and techniques used to improve boot time ● Focusing on cold- boot optimizations
  • 8. Andrea Righi - andrea@betterlinux.com Boot steps ● RPi boot steps: ● bootcode.bin: 1st-stage boot loader from SoC BCM2835 ROM ● bootloader.bin: 2nd-stage boot loader from external SD card ● kernel.img: Linux ● rootfs: external SD card ● application: ssh (dropbear)
  • 9. Andrea Righi - andrea@betterlinux.com Boot steps ● Rpi boot steps: ● bootcode.bin: 1st-stage boot loader from SoC BCM2835 ROM ● bootloader.bin: 2nd-stage boot loader from external SD card ● kernel.img: Linux ● rootfs: external SD card ● application: ssh (dropbear)
  • 10. Andrea Righi - andrea@betterlinux.com Measure boot-time functionality (kernel) ● CONFIG_PRINTK_TIME ● Configure it in “Kernel hacking” section ● Adds timing informations to kernel messages (dmesg) # dmesg ... [ 0.022030] initcall bcm_mbox_init+0x0/0x38 returned 0 after 0 usecs [ 0.022054] calling bcm_power_init+0x0/0xa4 @ 1 [ 0.022065] bcm_power: Broadcom power driver [ 0.022080] bcm_power_open() -> 0 [ 0.022090] bcm_power_request(0, 8) [ 0.522766] bcm_mailbox_read -> 00000080, 0 [ 0.522783] bcm_power_request -> 0 [ 0.522812] initcall bcm_power_init+0x0/0xa4 returned 0 after 488281 usecs
  • 11. Andrea Righi - andrea@betterlinux.com Kernel initcall tracer ● Introduced in Linux 2.6.28 ● Add “initcall_debug” to the kernel boot options ● Allows to record the timings of each initcall in dmesg # dmesg | grep " initcall " | sed "s/ */ /g" | sort -n -t' ' -k8 | tail -5 [ 0.701759] initcall bcm2708_fb_init+0x0/0xc returned 0 after 32518 usecs [ 0.794306] initcall pty_init+0x0/0x3e0 returned 0 after 90313 usecs [ 0.660366] initcall init_kprobes+0x0/0x108 returned 0 after 94523 usecs [ 1.224203] initcall dwc_otg_driver_init+0x0/0xb8 returned 0 after 402667 usecs [ 0.518454] initcall bcm_power_init+0x0/0xac returned 0 after 488281 usecs
  • 12. Andrea Righi - andrea@betterlinux.com Kernel initcall tracer (example) ● dmesg > dmesg.log ● cat dmesg.log | perl scripts/bootgraph.pl > /boot/dmesg.svg
  • 13. Andrea Righi - andrea@betterlinux.com Raspbian ● http://www.raspbian.org ● General-purpose distro based on Debian ● Optimized for the Raspberry Pi ● compilation settings adjusted to produce hard-float code ● 2013-05-25-wheezy-raspbian.img
  • 14. Andrea Righi - andrea@betterlinux.com Raspbian: boot time # dmesg ... [ 3.107163] smsc95xx 1-1.1:1.0: eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1, smsc95xx USB 2.0 Ethernet, b8:27:eb:a9:d6:24 [ 5.664780] EXT4-fs (mmcblk0p2): recovery complete [ 5.677326] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null) [ 5.689427] VFS: Mounted root (ext4 filesystem) on device 179:2. [ 5.700523] devtmpfs: mounted [ 5.706028] Freeing init memory: 128K [ 6.257687] init start [ 7.337703] udevd[154]: starting version 175 [ 8.511002] Registered led device: led0 [ 16.041293] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null) [ 16.522927] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null) [ 25.553157] smsc95xx 1-1.1:1.0: eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1 [ 28.672178] Adding 102396k swap on /var/swap. Priority:-1 extents:2 across:507900k SS [ 30.436046] init done - Shell after ~6.3sec - SSH after ~30.5sec
  • 15. Andrea Righi - andrea@betterlinux.com sysv-rc-conf ● Run-level configuration for SysV like init scripts ● disable npt ● disable plymouth ● disable rsync ● disable x11-common and lightdm ● disable alsa-utils ● disable swap (dphys-swapfile) ● disable checkfs ● disable ntp
  • 16. Andrea Righi - andrea@betterlinux.com Static IP vs DHCP ● Assign a static IP to the board (save the time to get an IP via DHCP) ● Disable CONFIG_IP_PNP in the kernel .config ● Used to mount rootfs via NFS
  • 17. Andrea Righi - andrea@betterlinux.com Raspbian: boot time after simple optimizations # dmesg ... [ 3.132731] smsc95xx 1-1.1:1.0: eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1, smsc95xx USB 2.0 Ethernet, b8:27:eb:a9:d6:24 [ 5.730131] EXT4-fs (mmcblk0p2): recovery complete [ 5.742783] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null) [ 5.754881] VFS: Mounted root (ext4 filesystem) on device 179:2. [ 5.765958] devtmpfs: mounted [ 5.771460] Freeing init memory: 128K [ 6.310549] init start [ 7.389321] udevd[154]: starting version 175 [ 8.536932] calling leds_init+0x0/0x60 [led_class] @ 229 [ 8.537065] initcall leds_init+0x0/0x60 [led_class] returned 0 after 82 usecs [ 8.543802] calling gpio_led_driver_init+0x0/0xc [leds_gpio] @ 229 [ 8.544067] Registered led device: led0 [ 8.545242] initcall gpio_led_driver_init+0x0/0xc [leds_gpio] returned 0 after 1344 usecs [ 14.724169] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null) [ 15.202445] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null) [ 24.493346] smsc95xx 1-1.1:1.0: eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1 [ 24.991030] init done About 25 sec to login via ssh, instead of 30 sec (16% faster)
  • 18. Andrea Righi - andrea@betterlinux.com Build a minimal distro from scratch: requirements ● build toolchain (gcc, glibc, binutils) ● git://github.com/raspberrypi/tools.git ● HINT: use hard-float toolchain (arm-bcm2708hardfp-linux- gnueabi) ● Linux (kernel) ● git://github.com/raspberrypi/linux.git ● busybox (rootfs) ● git://busybox.net/busybox.git ● dropbear (ssh server) ● https://matt.ucc.asn.au/dropbear/
  • 19. Andrea Righi - andrea@betterlinux.com Kernel initcalls (before optimization) # dmesg | grep " initcall " | sed "s/ */ /g" | sort -n -t' ' -k8 ... [ 0.810927] initcall iscsi_transport_init+0x0/0x154 returned 0 after 446 usecs [ 1.225990] initcall bcm2835_thermal_driver_init+0x0/0xc returned 0 after 451 usecs [ 1.232472] initcall pm_qos_power_init+0x0/0xac returned 0 after 713 usecs [ 1.230686] initcall sysctl_ipv4_init+0x0/0x98 returned 0 after 755 usecs [ 0.810406] initcall vchiq_init+0x0/0x1dc returned 0 after 1565 usecs [ 1.228797] initcall sdhci_drv_init+0x0/0xc returned 0 after 1829 usecs [ 0.560116] initcall inet_init+0x0/0x260 returned 0 after 2739 usecs [ 0.808747] initcall loop_init+0x0/0x11c returned 0 after 4852 usecs [ 0.803722] initcall brd_init+0x0/0x1c0 returned 0 after 8230 usecs [ 0.540586] initcall genhd_device_init+0x0/0x84 returned 0 after 9765 usecs [ 0.556774] initcall chr_dev_init+0x0/0xd8 returned 0 after 12234 usecs [ 0.538982] initcall param_sysfs_init+0x0/0x1d4 returned 0 after 19531 usecs [ 0.701759] initcall bcm2708_fb_init+0x0/0xc returned 0 after 32518 usecs [ 0.794306] initcall pty_init+0x0/0x3e0 returned 0 after 90313 usecs [ 0.660366] initcall init_kprobes+0x0/0x108 returned 0 after 94523 usecs [ 1.224203] initcall dwc_otg_driver_init+0x0/0xb8 returned 0 after 402667 usecs [ 0.518454] initcall bcm_power_init+0x0/0xac returned 0 after 488281 usecs
  • 20. Andrea Righi - andrea@betterlinux.com Kernel optimizations (initcalls) ● disable kprobes (CONFIG_KPROBES): ~95ms ● reduce the number of PTYs: ~90ms ● CONFIG_LEGACY_PTY_COUNT=256 => CONFIG_LEGACY_PTY_COUNT=1 ● disable framebuffer: ~33ms ● remove loop device: ~5ms
  • 21. Andrea Righi - andrea@betterlinux.com Kernel initcalls (after optimization) # dmesg | grep " initcall " | sed "s/ */ /g" | sort -n -t' ' -k8 ... [ 0.570870] initcall tun_init+0x0/0x8c returned 0 after 259 usecs [ 0.568035] initcall des_generic_mod_init+0x0/0x10 returned 0 after 341 usecs [ 0.975476] initcall bcm2835_cpufreq_module_init+0x0/0xc returned 0 after 358 usecs [ 0.569678] initcall pty_init+0x0/0x194 returned 0 after 375 usecs [ 0.561542] initcall vc_mem_init+0x0/0x1b4 returned 0 after 393 usecs [ 0.974729] initcall bcm2835_thermal_driver_init+0x0/0xc returned 0 after 393 usecs [ 1.018751] initcall deferred_probe_initcall+0x0/0x6c returned 0 after 395 usecs [ 0.561078] initcall bcm2708_gpio_init+0x0/0xc returned 0 after 405 usecs [ 1.018028] initcall pm_qos_power_init+0x0/0x60 returned 0 after 506 usecs [ 0.559402] initcall inet_init+0x0/0x260 returned 0 after 1540 usecs [ 1.021006] initcall net_secret_init+0x0/0x1c returned 0 after 2145 usecs [ 1.024343] initcall initialize_hashrnd+0x0/0x1c returned 0 after 3052 usecs [ 0.557293] initcall chr_dev_init+0x0/0xd8 returned 0 after 10978 usecs [ 0.541305] initcall param_sysfs_init+0x0/0x19c returned 0 after 19531 usecs [ 1.016084] initcall sdhci_drv_init+0x0/0xc returned 0 after 39250 usecs [ 0.973739] initcall dwc_otg_driver_init+0x0/0xc4 returned 0 after 392536 usecs [ 0.522812] initcall bcm_power_init+0x0/0xa4 returned 0 after 488281 usecs
  • 22. Andrea Righi - andrea@betterlinux.com Kernel optimizations (other optimizations) ● preset loops_per_jiffy ● At each boot the kernel calibrates a delay loop (used later by the udelay() function) ● 1 jiffy = time between 2 timer interrupts ● CONFIG_HZ=100 => 250ms!!! ● loop ● disable console output ● remove “console=xxx” and add “quiet” to the kernel boot parameters ● use LZO kernel decompression (CONFIG_KERNEL_LZO) ● LZO is a compression algorithm that is much faster than gzip, at the cost of a slightly degrade compression ratio (+10%) ● reduce kernel size... ● A smaller kernel is faster to load, less code also means smaller working set (good for caches)
  • 23. Andrea Righi - andrea@betterlinux.com Kernel image (before => after)
  • 24. Andrea Righi - andrea@betterlinux.com rootfs ● Generated using busybox and dropbear ● Hints about busybox (reduce fork()s): ● CONFIG_FEATURE_SH_STANDALONE: use applets instead of fork/exec/wait external binaries ● CONFIG_FEATURE_SH_NOFORK: call <applet>_main() directly without spawning another task ● Build everything with -Os ● Use mklibs to strip system libraries (rootfs is mounted to /mnt) ● mklibs -v -D -d lib2 -L /mnt/lib --ldlib lib/ld-linux.so.3 --target=arm-bcm2708- linux-gnueabi /mnt/bin/* /mnt/sbin/* /mnt/usr/sbin/* /mnt/usr/bin/*; rm -rf lib; mv lib2 lib ● After all these stops total rootfs size is ~3.2MB
  • 25. Andrea Righi - andrea@betterlinux.com filesystem optimizations ● Split the filesystem into read-only portion and read/write portion ● Read-only filesystem mounts faster ● Use Squashfs + tmpfs (or aufs
  • 26. Andrea Righi - andrea@betterlinux.com root filesystem: boot time
  • 27. Andrea Righi - andrea@betterlinux.com Kernel image (before => after) 1.5GB 3.2MB
  • 28. Andrea Righi - andrea@betterlinux.com Demo: custom kernel+rootfs boot time ... [ 1.144025] Freeing init memory: 92K [ 1.228550] init start [ 1.249067] waiting eth0 to come up [ 1.345882] usb 1-1: new high-speed USB device number 2 using dwc_otg [ 1.346059] Indeed it is in host mode hprt0 = 00001101 [ 1.556942] hub 1-1:1.0: USB hub found [ 1.557074] hub 1-1:1.0: 3 ports detected [ 1.836029] usb 1-1.1: new high-speed USB device number 3 using dwc_otg [ 1.940066] smsc95xx v1.0.4 [ 1.951858] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1, smsc95xx USB 2.0 Ethernet, b8:27:eb:a9:d6:24 [ 2.086695] init done [ 3.406004] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1 - Shell after ~1.2 sec - SSH after ~3.4 sec
  • 29. Andrea Righi - andrea@betterlinux.com Conclusion ● Why a faster boot? ● Consumer electronics products require very fast boot times (digital camera, mobile phone, etc.) ● Fast recovery upon system failures (crashes) ● You can show to your friends your new awesome board booting in a couple of seconds ;-)
  • 30. Andrea Righi - andrea@betterlinux.com References ● “LPC: Booting Linux in 5 Seconds” - Arjan van de Ven's talk: http://lwn.net/Articles/299483/ ● elinux wiki - Tim's fastboot tools: http://elinux.org/Tims_Fastboot_Tools ● “Update on boot time reduction techniques” - Michael Opdenacker ● Linux: http:.//www.kernel.org ● Busybox: http://busybox.net ● Dropbear: https://matt.ucc.asn.au/dropbear/dropbear.html ● Raspberry Pi: http://www.raspberrypi.org ● http://it.emcelettronica.com/embedded-gnulinux-partendo-da-zero- test-sulla-raspberry-pi
  • 31. Andrea Righi - andrea@betterlinux.com Q/A ● You're very welcome! ● Twitter ● @arighi ● #bem2013