Email
Favorite
Favorited ×
Download
Embed
Private Content
Copy and paste this code into your blog or website Copy Customize Close
We have emailed the verification/download link to "".
Login to your email and click the link to download the file directly.

To request the link at a different email address, update it here. Close
Validation messages. Success message. Fail message.

Check your bulk/spam folders if you can't find our mail.

Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

+ Follow

Open Source Kan Sneller, the StartLog Study

by Albert Mietus on Feb 18, 2010

382 views

Historical, Linux is used on systems where boot-time is not relevant....

More…

Historical, Linux is used on systems where boot-time is not relevant.
With EQSLinux it has become easy to start using embedded Linux.
Now, we want to make to make Linux boot faster

Note: the 'show-notes' are full of typo's

Less

2 Embeds 7

http://albert.mietus.nl	6
http://www.slideshare.net	1

Statistics

Favorites: 0
Downloads: 0
Comments: 0
Embed Views: 7
Views on SlideShare: 375
Total Views: 382

Some terms Linux, Linux-systems and OpenSource embedded Linux (systems) Strictly speaking, ‘Linux’ is only a kernel; that is the lowest and deepest part of an (Unix alike) Operating System. To build an computer system more programs are needed. There exists a lot of OpenSource programs that run on Linux. Together they act as a ‘Linux System’. Well know Linux systems are ‘RedHat’, ‘Suse’ and a lot more. Usually the are called ‘distributions’ Most Linux systems function as some kind of server, for mail, for web-pages, or in the office. But there are other Linux systems too. E.g on super computers, on mainframe’s, or on embedded systems. These last ones, ‘the embedded Linux systems’, form only a small fraction of all Linux systems. And are subject to our study. Each embedded Linux system contains more programs the ‘Linux’ kernel only. The should be called ‘Embedded OpenSource systems’ as it would be a more accurate name; but nobody does. Although, it is possible to build an ‘Embedded OpenSource systems’ without Linux! By using with one of the (free/open/net/..)BSD Unix(alike) OpenSource kernels. Of with non-Unix-alike OpenSource Operating systems like ecos. Booting (embedded) Linux (systems) Linux is fundamentally different from (most) traditional RTOS’es. It boost different too. It is even hard to define where the boot process starts and where it ends. For kernel-programmers, it starts somewhere in the Linux kernel, at a place called start_kernel . And it ends when the kernel is ready to start the first process, called ‘init’. But the reality is different, or better: one can have a different view. For embedded systems, it is better to define the beginning at power-up. That is when the user starts his tool or toy. We think the boot is ended when the functionality of the embedded systems becomes available. So, we define the ending of the boot at the moment where the ‘ embedded application’ is started. (Or when there is no ‘embedded’-application: the login program). As we can see in this study, that means the system will start a lot of programs before that point. And the duration of each of those programs can add to the boot time. See next page “ The StartLog study” This document starts at page 3; the first two pages are used to introduce some terms. These are the notes of the (Dutch) presentation ‘OpenSource can become faster’ but can be read independent from that presentation. It isn’t a transcription of the that presentation, but contain extra details and background/side information.
(continue) To narrow the study a little, we use PC-hardware as a reference, and terms used with PC-hardware. Like ‘BIOS’; everybody knows about the ‘BIOS’; but only the PC’s use it. All (or almost all) hardware platform have similar software. Embedded systems too. Because those early stages are very specific, often propriety and hard to change, detailed views of the boot-sequence usually silently ignore that part. It is simply to much work to add it to each graph. But we all know it is there. Audience This presentation is is about booting Linux; or better: about booting embedded Linux systems. It is targeted towards programmers, or other technical informed people, with a background of embedded software/systems. Some general interest in Linux is assumed but no in-depth knowledge about Linux is required. This presentation does not explain ‘what embedded systems are’. For Linux-specialists which want to learn about ‘embedded’, other documents exist. EQSL (inux) and the ‘StartLog’ study In this study, we use PTS’ EQSLinux; the default embedded Linux version build by the E mbedded Q uick S tart L inux framework. It is easy to use, and it is our own. But the method we present is not ‘part’ of EQSL. All embedded Linux versions can use this concept. We even used it on a not-embedded version! And it should be easy to adapt it to other OpenSource Unix-alike operating systems.
Linux and booting embedded systems The history of booting Unix Linux is a kind of Unix; an operating systems which always has been used on ‘big’ computers. Systems that are fast and expensive, but are not famous about there boot-speed. A well known issue is ‘ fsck’ ; a tool to check all hard-drives on power-up and can run for minutes or longer. On those ‘big’ systems, a slow boot wasn’t a issue. The system is operated by an system administrator and who booted the system half a hour before regular users needed it. Another ‘solution’ is, to minimize the number of boots; systems running for (literally) years are no exceptions. Unix/Linux has the power and stability to to so. For embedded (Linux) systems, this is different. They are not ‘operated’, they are ‘used’. Some embedded systems are big and running ‘forever’. Then booting-speed is probably still not not important. Other embedded systems are booted frequently. Consumer electronics is a good example: people turn the power on and expect it will run instantly. Like a light-bulb. Most embedded systems are in-between. By example, car-electronic; starting a car-engine cost some time; time that probably can be used to boot the embedded system-computer. But nobody wants to wait a few minutes before driving of while the embedded computer is doing an ‘ fcsk’ . Car drivers even don’t know what ‘fsck’ means! Linux and OpenSource Linux is, as we all know, ‘OpenSource’. What does that imply? It means everybody, developers in particular, can see, study and modify the source code. It also implies that the current code is made to fit the majority of the developers working on and with it. As Linux is mainly used outside the field of embedded systems, relatively little of it is written to ‘ fit for ’ embedded systems. Booting the system is one of them. Often, attempts are make to speed-up the booting. Usually that is to please the server-admin, or the desktop user. On average, it becomes better. By example, running ‘fsck’ (see above) isn’t run on every boot. That is nice on average; worst case, it hasn’t change a lot. Most embedded developers only are interested in ‘worst case’, so ... Note: we use ‘Linux’ here as an example. As we will see, this issue applies to more OpenSource than Linux only.
Why is booting Linux so slow ? Compared to most embedded systems Linux is a slow starter. But compared to other, non-embedded systems it not particular slow. Most Unix systems have a similar boot-speed, as does as an average desktop system. But we focus on embedded systems. So, why does it need a lot of time? Abstraction Linux/Unix does run on a wide range of hardware. It used to demand ‘ much hardware ’; like a PDP-11: two banks of 64Kbyte of memory. That was ages ago. Any modern, generic CPU will do, from i386 to mainframe. (Linux does not run on really small systems, like 8bit CPU’s.) Accept the CPU, Linux abstracts other hardware to, like disks, networking and (internal) busses. And it doesn’t care how the system is used. Linuxcan be used in a production department of a financial centre and in some lab of a bunch of software developers. Those people have different demands however, in speed, reliability and cost (to name a few). For all those reasons, and more, Linux (booting) needs to be flexible, which cost performance. This flexibility is implemented in the boot phase. We may strip that that, for embedded systems; the system becomes less flexible (which isn’t needed on most embedded systems) and faster. Lack of interest As we have seen on the last sheet, Linux is made to fit most. And most software engineers do not work in the embedded field. So, there are only a (relative) few that share the need to make Linux boot fast(er). Besides, when someone is working in his (her) spare-time on Linux; most will work on parts they think are ‘hot’. And believe me, booting is not ‘sexy’. Last, Linux is often driven by functionality, not quality. In general, the quality of Linux is good. But, as each system, it has relative weak points. The build-process (the Makefiles and stuff) is one that is generally seen as ‘to be improved’ (and it is becoming better). The boot part is another one. By example, there are various ideas about how to boot a system. Even on PC hardware; ‘lilo’ and ‘grub’ are well known competitors for the early ‘ BOOT ’ phase. Sometimes, one (group of) developers create a nice framework, with hooks, flexibility and stuff. And then, others disagree (or don’t understand) and extent ( “enrich” ) it, with more code, more delay and more complexity. Not every change is a improvement! This especially happens in de domain of (see later) ‘start-up scripts’ which are easy to write.
Booting Linux, globally Booting Linux is much more complex then booting traditional embedded systems. As we have seen Linux, is flexible and generic to runs on several kinds of hardware. And for all kind of users. So, every brand of Unix/Linux, and even every individual system use another boot-sequence. Globally, we can recognize five phases The software that is run directly on power-on. On PC’s that is called ‘ BIOS ’; a more general term is P ower O n S elf T est. It is stored on ROM (or flash); the CPU ‘jumps’ to this code when it is reset. This means this code is very CPU and hardware-platform specific. Usually, the initial code is very limited. Typically, it has no knowledge about the file-system; so it can’t load the kernel, which is stored as a file. Therefore a BOOT-loader is needed. The ‘boot(loader)’ is loaded by ‘ POST ’, has knowledge of file-systems and will load the kernel. To be able to be found by POST , if it often loaded ad a pre-know location on the disk. The PC- BIOS does so, to start Windows, lilo, grub or any of the existing BOOT- loaders for PC. Like POST , BOOT is very hardware/platform depended. Some systems have a very powerful POST and have do not need so separate BOOT phase. This can apply for embedded Linux to, on specified hardware. When you design your own hardware, you have the option to skip BOOT (and put it in POST). Note: Often several sub-phases exist. E.g ‘grub’ has at least 2 to 3 stages. Eventually, the Linux kernel is loaded and run. Often, it will uncompress itself before it really starts. It begins with some CPU, or system, depended part, but most of the Linux-kernel is architecture-independent. The kernel will probe (look for /search for) hardware that is statically linked, and initialize it. Hardware that is not configured will be ignored. Equally, configured drivers for hardware that isn’t found will be ignored. It is not an error to have more or less hardware then is configured! Some hardware will be configured in the next phase. See next page
(continue) Most modern Unix systems, including Linux and other OpenSource operating systems, can use ‘loadable kernel modules ’; this is very much like a ‘DLL’, but for Unix/Linux kernels. Typically they are loaded one-by-one by the kernel and contain one driver each. Such a module/driver will probe for (its) hardware and initialize it. If the hardware isn’t found the module is ignored (and unloaded). Note: the code that will probe the hardware is part of the module. This implies a module has to be loaded to test if it is needed. it also implies it is ‘safe’ to remove (delete) unwanted modules-files. The kernel will not complain about missing them; it simple doesn’t know! Note: as some modules depend on others, and the dependencies can be cached (in a file), some warnings may be seen. The loading of the modules as described above isn’t done by the kernel itself. It is initiated by a start-up script. That script is (indirect) called by the init process; the main-routine of any unix system, When that process is terminated, the systems stops. Aside from the ‘load modules script’ there are numerous other start-up scripts . This is where Linux brands, other (free) Unix’s and systems differ(entiate). There are several concepts and places for these start-up or “ rc-scripts ”. Most (modern) Linux systems put the actual scripts in /etc/init.d/ and use (sym)links to /etc/rc { run-level } / , to select the files that needed in each run-level (See below). The order of starting the files is given by it name. Each file start with an S (for start) and a two digit number. They are run in alfanumerical order. Some systems handle script that ends in “ .sh ” slightly differently, they hope to win some speed. Note: The concepts of run-levels is an old “system-V” idea. Some Linux-experts are very fond of it. But personally, it do no see the advantage of it anymore ; the flexibility it provides is very limited and often misused. What results is a lot of links and unclearity . This specially is valid for embedded Linux systems: do not depend on it! Quick Wins Minimise bios options (Linux doesn’t use/need them) Remove the (configurable) time-outs like the boot-selection (grub) Be aware of ‘no-responds’ timeout! E.g. Using dhcp when there is no network can take 20 seconds ! Most time is spend in start-up scripts. Several of them are ‘slow’ (X, desktop, ...) and not existing/needed on embedded Linux systems. This makes an embedded Linux system ‘automatically’ faster then normal Linux systems. Details The module phase is technically part of the Start-up code; it is (virtually) absent when no kernel modules are configured The blue line in this graph is illustrative The ‘height’ of each phase denotes the upness of the system. On power-on it is zero. And when the system is usable, it is defined as 100%; this when is when the user can login (normal Linux) or “the embedded” application is started. Here, each phase takes a equal level of upness (20%).
Measuring the boot “ Clock-time” It is easy to measure the duration of the Linux boot. Simply watch the seconds-hand of the clock on the wand. It will give us a first impression. It will generally tell us we have to speed up the boot-process. This way of measurement, we call it ‘clock-time’, is used as a reference. It does not tell ( “hint” ) what to change, to gain the speed we need. External or Internal? It would be nice if it was possible to use some ‘external’ measurement-system. Like we use a scope to measure electronics. But that is not feasible. There is no current flowing, no pins to monitor or any other simple way to know what the software is doing. Even a hardware-emulation tool will no do. It will give detailed information about CPU-state, or possible debug/disassemble the (C) code only Not the big picture. Probably it can’t even see scripts? Therefore, we need to modify the system to monitor itself. As we have seen, there are several phases. Each with its own domain (knowledge) and each (more or less) independent from each other; in concepts, features, versions, etc. It is not possible to use one method for all. Minimize to maximize We use a pragmatic approach. The measurements are needed to improve the boot-speed. So there is no need to measure those parts we can’t (or will) change. Nor do we need to measure things we already now; like the duration of a time-out of a boot-selection-choise. We know it is there. And we know it is useful on a PC, possible also in a development lab. But eventually, we will switch it ‘off’, as the embedded system doesn’t need it. Also, we want a minimal change to the system. Because every change does influence the behaviour. But more important, while we want to be able to apply our measurement to each Linux system; at least during the development of the embedded system. Last, we do not have a need to measure very accurate. We want to spare seconds or more, so there is hardly any need to be sure about a (small, fast) step that takes 1 or 2 millisecond. Also we don’t care about some generic deacceleration; after we speed-up the boot, we will run the system at full speed again. At worst, the boot is a little to fast:-)
The ‘StartLog’ concept ‘ StartLog’ is a concept to measure the boot of an embedded Linux system. It concentrates on those parts that fall into the Linux domain: The kernel, the modules and the “rc-scripts”. With ‘StartLog’, it becomes possible to measure a systems, change (optimize) it, and re-measure. In this way, it becomes possible to evaluate the change. Furthermore, the measurements ‘hints’ places or steps where to optimize. We will see to that later. Working with ‘StartLog is: Fast With StartLog , the system will monitor itself during boot. It will notice when a program is started and store that in RAM. After boot, it is possible to read that data, transfer it to another system and process it. Because most work is done off-line, the real-time aspect of the boot is minimally changed. Simple Although powerful, StartLog is simple to implement. Only a handful of lines have to be added to the kernel, so any Linux version and any architecture can use it. Also it monitors almost all steps of the Linux-boot, including scripts and module loading. Note: To be exact, it will miss some steps, like rc-scripts that are ‘sourced’ instead of executed. In practise this in not a problem and is easy to compensate, if needed . Reliable StartLog has proven to be extremely reliable. We have modified the change to double and triple the amount of logging to measure. We have calculated the time-difference between steps (or loglines) and compared these step-durations between multiple measurements: in more then 90% of the cases this delta is less then one of two time-ticks. This means we can use StartLog to measure a system, change the system and re-measure it. Then we can compare those measurements and verify the change is an improvement. In short, StartLog is a combination of a small kernel patch, some (off-line) scripts and a procedure to measure and improve the boot(speed) of embedded Linux.
Details The ‘exec’ system call There are multiple reasons to use ‘exec’ (and not ‘fork’) to monitor In Linux, the fork system-call implementation is architecture dependent. This means the patch should be incorporated into multiple files, making ‘StartLog less simple. The exec system-call however is located in one file ( fs/exec.c ) Note: although there are several exec*() calls available to programs, the are all handled by the same kernel-routine: do_execve() Typically, programs are started by forking the current process; after which the ‘child’ will exec a new program. It is possible to do otherwise, but that isn’t done often. So, monitoring exec is almost the same as monitoring fork . When fork and exec are not combined, it is not trivial to say which one is better. Only theoretically, fork is a better indication for processes. But exec is better in monitoring programs. Only with exec , relevant information as program-name (filename) and arguments is available. Without, it is hard to chain the event (logline) to a human recognisable step. Storing data Each Unix/Linux system can store and show kernel messages, by using the “dmesg” system. Internally, a (fixed size) circular buffer is used by the kernel to store lines of information with a printf() alike function. In user space this buffer can be read. StartLog uses this to store an retrieve startLog-lines. To do so, we have to patch or configure the kernel to use a big buffer; one Mbyte will usually do. Note, a “bug” in the kernel-build system will sometimes overrule the specified (big) value and reset to a default (to small) value. Then a 2 nd patch is needed to fix this, otherwise the earliest loggings will be overwritten. Jiffies for time The Linux kernel has an counter to measure time, called jiffies. It doesn’t count time, but “ticks”. On most systems a jiffy is (about) 1 or 10 ms. Furthermore this counter does wrap (restart at 0, after some time), and possible does not start at zero. This is not a problem, it will be compensated by the off-line processing.
More details Retrieving and transferring the data We use the dmesg command to retrieve kernel messages from the kernel to user space. Some versions of dmesg, like the busybox version used in EQSL, use (by default) a limited buffer size. With the “-s” option, this can be overruled. Note however the shown value is the maximum allowed value. Specifying a value of more then 512K is silently ignored; ‘strange’ and ‘silly’, but that is Linux ... When network connected, use one of the netwerk-copy commando to transfer the data to a not embedded system, like a PC, a Unix system or (my personal choice) an Apple. The processing is done there Processing We use a combination of awk and Excel-macros for post-processing. As all script are simple, your own favourites will work equally well. Some scripts are generic, but usually they are adapted and hand-crafted to zoom in into specific details. Again, use what fits best. Quality Doing some extra measurements, we could measure/verify our patch added less then 0.5 ms to do_execve() . Only with the full patch, to log the arguments and environment to, the system becomes slow. But even then, when the system slows downs upto 25%, the relative times are still accurate. Note: the amount of slowdown is architecture and system depended. The given times are on our EmbeddedPC hardware; each call to prink cost 0.2ms with an extra 20µs per 10 characters to log. The Arm/Xscale system shows similar or faster results. Repeatability As we described before, the measurement is very repeatable, as it should be. Note however, Linux systems are influenced by the environment, specially the network! On the Gumstix by example, we have seen the systems will be slower the first time it is booted with a new (on board) flash image. Each first boot! Thereafter, we got repeatable results.
Measuring the boot of an embedded Linux system with StartLog The following pages show some graphs that are measured with StartLog. Below some hints are given to create your own ... The dmesg output contains more then only the log of StartLog. You can filter on “@PTS@startlog” to select StartLog-lines Collect dmesg output direct after the system is ready. The patch will continue monitoring the start of programs forever. To prevent the first loglines get lost, don’t wait to long before using the dmesg command Note: the counter in the first StartLog log-line should be zero. Else, data is missing. Trim de logdata where the system is “up”. We select everything up to the first occurrence of ‘getty’ Alternatively, it is possible to add a “ S99_TheSystemIsUp” marker-script It is possible to automate collecting data, by example in the above mention S99 script. Then it is “automatically” trimmed. Save the data by writing it to flash-disk or send it over the network. Don’t forget to post-process you logdata. Aside from selecting the correct lines, one should “recalculate” the jiffies. Save value of the first line and subtract it from each line (including the first). When the value is negative, add ‘maxint; to it. Remember to use “big numbers” for each calculation! Experiment a bit with each graph. Visualizing data isn’t rocket-science. Analyse the data, visualize it and experiment until the graph shows the details you need. Don’t get excited by graphing. Making “nice” graphs cost a lot of time. I did spend some time on the presented ones, for presentations reasons, only. For normal use, I keep them simple. They are needed to get the job done, not to please the eye.
The systems This sheet introduces some of the embedded Linux systems we have used; as well as some environmental influences. We use two kinds of hardware: an EmbeddedPC (i386 based, in various variations) and a ARM/Xscale base, (physically) very small system called ‘Gumstix’. All systems run the same kind of Linux: the demo version of PTS’ EQSLinux. Note: these graphs are not meant to show the ‘speed’ of the hardware, nor that of this Linux version. They are only generated to illustrate this presentation. ‘ Trident’ and ‘Start’ are EmbeddedPC’s; the former is a small scale embedded (or ‘industrial’) hardware system. ‘Start’, however is a (a bit old) normal PC, equipped with a CompactFlash card reader, compactFlash and without a real hard disk. In the ‘HDD’ variant the (normal) PC has got a normal hard disk again; to show the influence of CompactFlash (It is faster with CF!) Gumstix is an ARM/Xscale base embedded PC; it comes with it own version of Linux, which isn’t used (we use EQSLinux).It has onboard flash-memory With the ‘zonder network’ (no networking) variations, the Cable and/or the dhcp-server is removed (both give similar results). This test is done to show the influence of the environment: all systems become a lot slower. The measurement In this measurement we have booted the system and watched it behaviour. We use a normal clock (stopwatch) to measure some external visual moments. This isn’t the most reliable measurement but it comes close to real user- experience. As can be seen, the boot times are not acceptable for most embedded and/or consumer systems. But it is hard to see how to improve (aside from trivial waiting times). We need more detail In the following sheets; we use the ‘StartLog’ concept. To simplify the graphs we ignore everything before the first logline.
Timelines By drawing an XY-scatter graphic (jiffies versus logline-number) we get a kind of timeline. Here we define the system ‘up’ (or ‘ready’) when the getty process is started. That is when you can login. Loglines after that are ignored. The number of loglines (and so started programs) to get to that point can vary; as the graphs show. The EmbeddedPC needs about 1650 steps whereas the gumstix used a little under 300. This is due different (kernel) configuration. The Gumstix is configured for the limited hardware it can carry, so it has relative little drivers and very little (loadable kernel) modules. The EmbeddedPC on the other hand is configured for almost all hardware that will fit in a PC; and so it has hundreds of modules to load. For each module a few steps are needed. Also these graph demonstrates the duration of a ‘jiffy is architecture depend. In the top-graph more as 10-thousand jiffies are needed (always), whereas the lower graph stops before 4000! Using ‘clock-time’ we will find that a jiffy is about 1 ms on the EmbeddedPC and 10 on the Gumstix. So the former actually boots a little faster then the second. In practice we have hardly interest in the absolute numbers of both axes; we look to relative numbers. And, even more, to the angle of the line. What do we see? Primary, look to the angle of the line. A horizontal part means there are no programs started. Usually this implies a time-out or some other kind of waiting. Its a hint for speed-up. Remember, each dot is 10 to 100 of milliseconds wide. Most modern CPU’s can do a lot in that little time. And so, a steep line is possible. Where the line flattens the system is running idle. Or, another possibility, a program is started that “ loads” heavily. Maybe to heavy for a fast boot. Change and compare By drawing several measurements in one graph we can compare multiple versions of the software. Or see the influence of HW and/or environment. The top graph compares 4 similar systems. The light-blue line is a system with a (real) hard-disk instead of CompactFlash. Initially it is equal to the CF lines (it covers both the yellow and dark-blue ones) then it slows down! The bottom graph shows the influence of a dhcp-server; without it, the systems suddenly drops dead (the yellow line)
Bottlenecks As can be seen on the top-right pie-chart (the lower-left one is a ‘ dis-improvement ’) about a third of the boottime is used to load kernel modules. By statically linking those part into the kernel that can be improved. But then the kernel-load time will increase. Alternative, it is also possible to win (quite a lot) of time by optimizing the S05module-init-tools. New (not shown) measurements are needed to find the best (fastest) option. Also the networking part of booting is costly (even without the dhcp time-out ‘bug’). Optimizing that part is easier and probably more effective then e.g. the kernel. As can be seen on this graphs, one can hardly win any time with optimizing the web-server, the syslog-deamon or “cron”. An approach as “remove those less important services” is probably the wrong one. It will save (25+26+22=) less then a tenth of a second. Details Network timeout When there is no dhcp-server, no network, no cable, or for some reason there is no dhcp-responds, the dhcp-client (in S20networking ) has to wait until a timeout. In the example above, that takes (2091-487) extra jiffies. That is about 16 seconds Also, the S30thttpd script (a webserver) changes a few extra jiffies; but that is ignorable. Repeatability Aside from the timeout in this example; this graph also shows the repeatability of the measurement. But for the 2 (sub) phases described above, the numbers are exactly equal. This is valid in most graphs; although given the method of measurement a difference of 1 of 2 jiffies can be expected. Jiffies A jiffy is about 10 millisecond in this graph. How we know? Simply by watching a clock and dividing realtime by the total number of jiffies and round it.
Optimizing programs On the pie-chart of previous sheet, we can see which script (or generally: sub-phase) takes a lot of time. And optimize that. An alternative approach is to count program invocations. We display that in a bar-graph. As we can see, some programmes are called only once, and some are called often. In the graph we use both a linear and a logarithmic scale to display them. We can use this graph to select a program to optimize. Select a program that is used frequently; it has more impact then one that is used only once. Design warnings This graph can also be used to find ‘design-warnings’. They are not wrong, but not optimal too In the shown graphs, by example, the kernel tries to call the /sbin/hotplug program, which is not available. If the embedded system doesn’t need ‘ hotplug -functionality’ (to react when hardware is added to a running system) we can optimize the kernel and deselect it. Only removing (not installing) the hotplug program will do, but isn’t optimal. We also see both ‘ bash ’ and ‘ sh ’, both shells, are called during boot Bash and ‘sh’ are similar in functionality; ‘bash’ has more build-in statements and is better suited for interactive use. Sometimes ‘sh’ will be faster, sometimes ‘bash’ is (due its extended functionality, is has to start less subprograms). Using both ‘sh’ and ‘bash’ is suboptimal; one shell takes less memory, less disk-space, and is faster. The latter due the cashing functionality in Linux: loading a file twice is usually not needed. Note: StartLog only monitors the start of a program, not its termination. Therefore we can’t generally calculated the duration of a program. This is only possible when programs are called sequential, like the rc-scripts in previous example. It is easy to extent the logging with program-termination (use the process number as ID), then its becomes possible to calculate program duration.
(This sheets needs no notes)
(This sheets needs no notes)
(This sheets needs no notes)
(This sheets needs no notes)
(This sheets needs no notes)

Open Source Kan Sneller, the StartLog Study — Presentation Transcript

OpenSource kan sneller! Historical, Linux is used on systems where boot-time is not relevant. With EQSL inux it has become easy to start using embedded Linux. Now, we want to make to make Linux boot faster! But, Linux is complex ... It always has a thousand solutions ... And we even are not sure about the problem ... So, How to solve it? Time to investigate, to study, ...
Today
- Is there a problem?
- How does Linux boot?
- How to measure?
- Results and solutions
Typical Linux ...
- Linux is never designed for embedded systems
- Example: Booting one of our development-systems takes 143 seconds!
- On most embedded systems, that is not acceptable
Booting Linux is complex
- Flexibility
- Legacy
- Lack of design-vision
Booting Linux Systems
- There are 5 phases
time UP-ness (%) example development system : 143 seconds ! POST (bios) BOOT (grub) Kernel (vmlinuz) modules (*.ko) Start-up (rc-scripts) READY (login/application)
Booting Linux Systems
- There are 5 phases
time UP-ness (%) example development system : 143 seconds ! POST (bios) BOOT (grub) Kernel (vmlinuz) modules (*.ko) Start-up (rc-scripts) READY (login/application) Hidden sheet Needed for notes of prev sheet
How to measure?
- Reliable measuring the boot-steps is difficult
- Note: monitoring 143s in ms-resolution is (about) 500-meters of print-out!!
StartLog
- Concept
- Result: StartLog (patch & post-processing)
Details: patch
- Log system-call ‘exec’
- Use ‘ dmesg ’ storage
- Use ‘jiffies’ for time
- do_execve (... filename ... ) {
- static int PTS_startlog =0;
- // int PTS_i;
- printk( " @PTS@startlog=%07d,
- jiffies=%012lu, do_execve(%s) ",
- PTS_startlog ++, jiffies , filename );
- /* //Optional:
- for ( PTS _i =0; PTS _i < 9; PTS _i++) printk(" %s, ", argv [ PTS _i] );
- printk(” ; ");
- for ( PTS _i = 0; PTS _i < 9; PTS _i++) printk(" %s, ", envp [ PTS _i] );
- printk(" ) ”);
- */
- ...
- }
Details: processing & Quality
- Store Log
- Off-line
- Customisable
- Influence (real-time)
- Repeatability
Some Results
- The following sheets show some results.
All times (numbers) in jiffies All systems are NON-optimized!
A first impression
- Boot and clock- time does give some info , but
- We need more detail !
Some timelines
- Gumstix
- Embedded PC
Bottleneck && Network timeout
- Normal (top)
- Network Timeout (bottom)
/etc/rcS.d/S01mountvirtfs /etc/rcS.d/S05module-init-tools /etc/rcS.d/S10mountall /etc/rcS.d/S11RAMdisk /etc/rcS.d/S15hostname /etc/rcS.d/S17sysklogd /etc/rcS.d/S20networking /etc/rcS.d/S25cron /etc/rcS.d/S30thttpd Kernel
Program Count
- Often-exec’ed programs are good candidates to optimize
Summary (1/4)
- Problems & Solutions
Summary (2/4)
- Booting Linux
- Linux is OpenSource ...
Summary (3/4)
- StartLog
- Improve
Summary (4/4)
- Generic Quick Wins
- Advice
Questions and More info
- Although the sheets are overloaded with info, it’s only a fraction of what’s available.
- More info:
- Questions:

Open Source Kan Sneller, the StartLog Study

by Albert Mietus on Feb 18, 2010

Accessibility

Tags

Upload Details

Usage Rights

2 Embeds 7

Statistics

Open Source Kan Sneller, the StartLog Study — Presentation Transcript