Embedded linux barco-20121001

Internal Barco Training
September 24th / October 1st, 2012
Kubrick training room
Noordlaan 5, 8520 Kuurne | Belgium
Introduction to Embedded Linux for Engineering
Marc Leeman, VNG
Peter Korsgaard, DnA
2011

Contents
1 Introduction 1
1.1 Preconditions and Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.3 Some Hackable Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3.1 Marvell SheevaPlug . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3.2 Dreambox . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3.3 Linksys NSLU2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3.4 Bualo Linkstation Live . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3.5 Neo Freerunner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3.6 AzBox HD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2 Cross Compilation Toolchain 9
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 GNU Toolchain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.3 C Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.3.1 GNU C Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.3.2 uClibc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.4 Compilation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.4.1 Crosstool-NG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.4.2 Buildroot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.5 Hands On - Toolchain with Buildroot . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.5.1 Getting the Source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.5.2 Con

guring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.5.3 Finishing up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3 The Linux Boot Process 15
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.2 Step 1: The Boot Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.2.1 System startup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.2.2 Extracting the MBR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2.3 Stage 1 boot loader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2.4 Stage 2 boot loader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2.4.1 GRUB stage boot loaders . . . . . . . . . . . . . . . . . . . . . . . 18
3.2.5 Kernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.2.5.1 Manual boot in GRUB . . . . . . . . . . . . . . . . . . . . . . . . 20
3.2.5.2 decompress kernel output . . . . . . . . . . . . . . . . . . . . . . . 21
3.3 Step 2: init . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.3.1 Step 2.1: /etc/inittab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.4 Step 3: Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.5 Step 4: More inittab fun . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.6 Hands On . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
i

CONTENTS ii
3.7 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4 Boot Loaders 27
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.2 RedBoot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.3 Das U-Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.4 Barebox . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.6 Hands On - Explore U-Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.7 Hands On - Replace Bootloader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.7.1.1 Getting the Source . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.7.1.2 Con

guration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.7.1.3 Building and booting . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.7.1.4 A note about the
ash layout . . . . . . . . . . . . . . . . . . . . . 32
4.7.1.5 Adjusting the U-Boot environment . . . . . . . . . . . . . . . . . . 33
4.7.1.6 Fine tuning the Startup Behaviour . . . . . . . . . . . . . . . . . . 34
4.8 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5 The Linux Kernel 39
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.2 Timeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.3 Technical features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5.3.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5.3.2 Programming Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.3.3 Portability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.3.4 Versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.3.5 Getting the Source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.3.6 Source Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.3.7 Tracking Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.4 Hands On - Build Kernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.4.1 NFS - Network File System . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.4.2 Con

guration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.4.3 Upgrading a Kernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
5.5 Device Tree (Powerpc, Microblaze, only for now) . . . . . . . . . . . . . . . . . . . 47
5.5.1 Flash Mapping in the Device Tree . . . . . . . . . . . . . . . . . . . . . . . 49
5.5.2 What if Something Goes Wrong . . . . . . . . . . . . . . . . . . . . . . . . 50
5.5.2.1 What Is The Kernel Symbol Table? . . . . . . . . . . . . . . . . . 50
5.5.2.2 What Is An Oops? . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
5.5.2.3 What Does An Oops Have To Do With System.map? . . . . . . . 51
5.6 Device Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
5.6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
5.6.1.1 How to load a module . . . . . . . . . . . . . . . . . . . . . . . . . 52
5.6.1.2 Choosing the device type . . . . . . . . . . . . . . . . . . . . . . . 52
5.6.2 Busses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
5.6.2.1 Platform Bus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
5.6.2.2 PCI Bus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
5.6.3 A Real Life Barco Example . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
5.6.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
5.6.3.2 Initialisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
5.6.3.3 Registering a PCI Driver . . . . . . . . . . . . . . . . . . . . . . . 60
5.6.3.4 Assigning the I/O and Memory Spaces . . . . . . . . . . . . . . . 62
5.6.3.5 PCI Interrupts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
5.6.4 Adding a Character Interface . . . . . . . . . . . . . . . . . . . . . . . . . . 66

CONTENTS iii
5.6.4.1 Major and Minor Numbers . . . . . . . . . . . . . . . . . . . . . . 66
5.6.4.2 The Internal Representation of Device Numbers . . . . . . . . . . 66
5.6.4.3 Some Important Data Structures . . . . . . . . . . . . . . . . . . . 67
5.6.4.4 File Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5.6.4.5 Char Device Registration . . . . . . . . . . . . . . . . . . . . . . . 69
5.6.4.6 Open and Release . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
5.6.4.7 The Release Method . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.6.4.8 Read and Write . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
5.6.4.9 Ioctl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
5.6.5 Interrupt Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
5.6.5.1 Installing an Interrupt Handler . . . . . . . . . . . . . . . . . . . . 80
5.6.5.2 The /proc Interface . . . . . . . . . . . . . . . . . . . . . . . . . . 82
5.6.5.3 Fast and Slow Handlers . . . . . . . . . . . . . . . . . . . . . . . . 84
5.6.5.4 Implementing a Handler . . . . . . . . . . . . . . . . . . . . . . . . 84
5.6.5.5 Handler Arguments and Return Values . . . . . . . . . . . . . . . 85
5.6.5.6 Top and Bottom Halves . . . . . . . . . . . . . . . . . . . . . . . . 86
5.6.5.7 Interrupt Sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
5.6.5.8 Adding Your Driver in KCon

g . . . . . . . . . . . . . . . . . . . 91
5.6.6 Con

gure the Flash Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
5.6.6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
5.6.6.2 Flash Map Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
5.6.6.3 Combining Multiple Flash Chips (hardcoded) . . . . . . . . . . . . 96
5.6.6.4 Adding Your Driver in KCon

g . . . . . . . . . . . . . . . . . . . 98
5.7 Hands On - LED Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
5.7.1 Hardware Veri

cation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
5.7.2 Kernel Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
5.7.2.1 Platform Bus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
5.7.2.2 Hardware Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
5.7.2.3 Sysfs Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
5.7.2.4 Timers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
5.8 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
6 File Systems 112
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
6.2 Disk Based File Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
6.2.1 Choice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
6.3 Flash Based File Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
6.3.1 Choice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
6.4 Network File Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
6.5 Virtual File Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
6.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
6.7 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
7 Userspace 115
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
7.2 BusyBox . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
7.2.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
7.2.2 Con

guring and Building BusyBox . . . . . . . . . . . . . . . . . . . . . . . 117
7.2.3 Manual con

guration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
7.2.4 Hands On - Adding New Commands to BusyBox . . . . . . . . . . . . . . . 119
7.3 Dropbear . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
7.4 Build Systems and Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
7.4.1 Buildroot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
7.5 Hands On - Explore Buildroot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

CONTENTS iv
7.6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
8 Creating an image with a full Linux system 128
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
8.2 Preparing an ARM GNU/Debian based system on a GNU/Debian based build system128
8.3 Preparing an ARM GNU/Debian based system on a Non GNU/Debian based build
system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
8.4 Customising the ARM root

lesystem . . . . . . . . . . . . . . . . . . . . . . . . . 131
8.5 Starting up: Compiling the Linux kernel for NAND boot . . . . . . . . . . . . . . . 132
8.6 Starting up: Creating the base root

lesystem image . . . . . . . . . . . . . . . . . 133
8.7 Install GNU/Debian 6.0 on the internal NAND
ash . . . . . . . . . . . . . . . . . 134
8.7.1 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
8.8 Booting in the

nal system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
9 Hacking the SheevaPlug 141
9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
9.2 OpenOCD Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
9.2.1 Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
9.2.2 Target state handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
9.2.3 Memory access commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
9.2.4 Flash commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
9.3 Hacking the SheevaPlug . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
9.4 Hands On - Tweak System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
9.5 Flashing the system from the bootloader . . . . . . . . . . . . . . . . . . . . . . . . 150
9.5.1 GNU/Debian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
9.6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
10 Debugging with GDB 152
10.1 GDB and gdbserver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
10.2 gdb Remote debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
10.2.1 Major Dierences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
10.2.2 ELF and Binutil Background . . . . . . . . . . . . . . . . . . . . . . . . . . 153
10.3 Remote Debugging With GDB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
10.4 Talking Dirty with GDB and SSH Tunnelling . . . . . . . . . . . . . . . . . . . . . 158
10.5 SSH Tunnelling and GDB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
10.6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
A The GNU/Linux System 164
A.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
A.2 History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
A.3 Licensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
A.4 Linux and GNU/Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
A.5 Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
A.6 Development eorts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
A.7 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
A.8 Usability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
A.9 Market share . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
A.10 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
A.11 Installation on an existing platform . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
A.12 Demonstration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
A.13 Con

guration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
A.14 Programming on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
A.15 Portability of Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
A.16 Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172

CONTENTS v
A.17 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
B Setting up a Server 173
B.1 Setting up the NFS Root Filesystem . . . . . . . . . . . . . . . . . . . . . . . . . . 173
B.2 Set up a Firewall with a private address range. . . . . . . . . . . . . . . . . . . . . 175
B.2.1

rehol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
B.3 Con

gure your BDI probe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
B.3.1 Check the serial connection to the BDI . . . . . . . . . . . . . . . . . . . . 176
B.3.2 Activating BOOTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
B.3.3 Load/Update the BDI

rmware/logic . . . . . . . . . . . . . . . . . . . . . 177
B.3.4 Transmit the initial con

guration parameters . . . . . . . . . . . . . . . . . 177
B.3.5 Fixed Con

guarion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
B.3.6 Check con

guration and exit loader mode . . . . . . . . . . . . . . . . . . . 178
B.3.7 Summarising the upgrade procedure . . . . . . . . . . . . . . . . . . . . . . 178
C Miscellaneous Tools 180
C.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
C.2 Patch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
C.2.1 Applying a patch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
C.2.2 Creating a patch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
C.3 Quilt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
C.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
C.3.2 Some Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
C.3.3 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
D Network Con

guration 185
D.1 TCP/IP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
D.1.1 Static . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
D.1.2 DHCP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
D.1.2.1 IP address allocation . . . . . . . . . . . . . . . . . . . . . . . . . 185
D.1.2.2 Protocol Anatomy . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
D.1.2.3 DHCP and

rewalls . . . . . . . . . . . . . . . . . . . . . . . . . . 186
D.1.3 ZCIP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
D.1.4 DNS-SD uPnP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
D.2 Writing a simple web interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
D.2.1 What is CGI? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
D.2.2 Structure of a CGI Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
D.2.3 Reading the User's Form Input . . . . . . . . . . . . . . . . . . . . . . . . . 187
D.2.4 Sending the Response Back to the User . . . . . . . . . . . . . . . . . . . . 188
D.2.5 Haserl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
D.3 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189

List of Figures
1.1 Architecture of a Linux system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Marvell SheevaPlug . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Dreambox 7025 S . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.4 Linksys NSLU2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.5 Bualo Linkstation Live . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.6 Neo Freerunner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.7 The AzBox HD decoder, and much more... . . . . . . . . . . . . . . . . . . . . . . . 7
2.1 http://buildroot.net . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2 Con

guration Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3 Selecting a system wide path with a date-string avoids confusion and overwriting
existing toolchains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.1 A high level view of the Linux boot process . . . . . . . . . . . . . . . . . . . . . . 16
3.2 Anatomy of the MBR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.3 Anatomy of bzImage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.4 Major functions
ow for the Linux kernel x86 boot . . . . . . . . . . . . . . . . . . 20
5.1 The kernel.org website . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.2 make ARCH=arm menucon

g . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.3 make ARCH=arm gcon

g . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
5.4 Layout of a typical PCI System. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
5.5 The standardised PCI con

guration registers. . . . . . . . . . . . . . . . . . . . . . 57
5.6 The arguments to read. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5.7 Flash map for SVC mk II. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
7.1 Busybox con

guration screen. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
7.2 Winscp, a drag and drop interface to your embedded target . . . . . . . . . . . . . 121
10.1 Running ddd with a remote target. . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
10.2 Lab setup with a workstation on a LAN (10.x); public servers (150.158.231.x) and
embedded targets on the LAN. The gateway (niobe) is not directly accessible but
provides an ssh tunnel on port 22 to gemini on the LAN . . . . . . . . . . . . . . . 159
10.3 After putting the ssh tunnel in place, the connections on 150.158.231.13, port 4000
are forwarded over TCP to the target 10.2.4.10 on port 2200. . . . . . . . . . . . . 162
A.1 Richard Stallman, founder of the GNU project for a free operating system. . . . . 165
A.2 Linus Torvalds, creator of the Linux kernel. . . . . . . . . . . . . . . . . . . . . . . 166
A.3 A GNOME Desktop. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
C.1 Drawing a graphical dependency between patches with quilt. . . . . . . . . . . . . 184
vi

PREFACE
As an engineer, there is nothing more fun than poking in the internal of a system, to see how
it reacts and how it all works. It is very hard to cope with black boxes, we want to know what
causes output c when inputs a and b are provided. Even more; we want to know that a system is
designed good and if there is an error; we want to

x it.
An open source operating system allows to do exactly that: First of all, it gives us the freedom
to design the hardware platform we want, only limited by time and money; to add the peripherals
we care about and to con

t. Obviously, we'll encounter some bumps
along the road and we'll need to dig in to our designs; adding debugging code in the kernel - But
in the end, we can always get it to work.
One of the advantages, in our view, of Linux is that we can run the same software on all our
systems: From our servers that compile and manage our environment, over our desktops to our
embedded targets. As the requirements shrink1, so does our operating system. A typical server
installation quickly surpasses a couple of GB, while we can shrink the root

le system of an
embedded Linux system to a couple of hundred kB.
In this text, we tried to bundle some of the experience and techniques we've had with building
embedded Linux systems. We have tried to ensure everything is correct, but some errors are bound
to have slipped through. If you feel something is not correct or missing, you are invited to inform
us about it, so we can correct the text for future trainings.
Though a lot of text is original, some sections have been added or integrated that were accessible
from public sources. Wherever possible, you should be able to obtain the original text from the
references section at the end of each chapter.
We hope you have as much fun and as good a learning experience as we had while drafting
this text.
Flanders, August 2006, May 2008, December 2008, June 2009, August 2010, January 2011, Septem-ber
2012.
Marc Leeman Peter Korsgaard
1Taking an embedded processor does not mean that it has less capabilities than a desktop or server processor,
quite the contrary. A lot of functionality that is otherwise reserved for external peripherals is on the processor die
itself. As a rule, embedded processors will be clocked slower as the desktop and server counterparts and will be
re-designed to consume less power during operation.

Chapter 1
Introduction
1.1 Preconditions and Goals
Embedded Linux is a huge topic, that cannot realistically be covered in such a short session (even
if we would know it all). The idea of this training is therefore not to cover everything related to
building embedded Linux systems, but rather to provide an introduction to the subject and get
you up to speed as fast a possible.We will try to share the experience we have and to show what so-lutions
we have found to work. This is not to say that these are the only workable solutions though!
To limit the scope a bit and provide real life examples we focus on and base the examples on
the existing embedded Linux systems within Barco and the Marvell Sheevaplug. We also assume
that the reader is familiar with Linux on PC hardware. If not, have a look at appendix A.
1.2 System Overview
Like other embedded systems, the detailed architecture of embedded Linux systems vary a lot,
but certain basic components are common for all systems.
The basic hardware consists of a CPU, RAM, some kind of storage and a number of peripherals
for I/O.
Linux supports a long range of CPUs, but ARM and PowerPC processors are typically used
within Barco. Storage can also vary a lot: Disks, network, NOR/NAND/managed
ash, where

ash is the most commonly used solution. I/O peripherals probably have the most variation of
them all, but the most interesting from a Linux system design are UARTs and Ethernet MACs.
The software consists of a boot loader, a Linux kernel and one or more

le systems containing
the applications and libraries. Boot loaders are further described in chapter 4. Boot loaders are
important for bringing up a system, but once the kernel is loaded it is no longer active. The generic
architecture of a running Linux system can be seen in

gure 1.1.
Figure 1.1: Architecture of a Linux system
1

CHAPTER 1. INTRODUCTION 2
At the bottom we have the hardware. Right above it the kernel is located. The kernel is the
core part of the operating system, and its purpose is to manage hardware and provide high level
abstractions to the user level software. The kernel is (normally) the only software which talks
directly to the hardware. The kernel is further described in chapter 5. Above the kernel the user
space applications and static- or dynamically linked libraries are located. Libraries provide further
high level abstraction for applications than what is provided by the kernel. Libraries exists for just
about everything, but all Linux systems at least contain a C library1. User space applications are
further described in chapter 7.
Notice that this generic architecture is the same for all Linux systems, no matter if they are
server-, desktop- or embedded systems.
1.3 Some Hackable Examples
1.3.1 Marvell SheevaPlug
The Marvell SheevaPlug is a cheap, powerful device in a small form factor. It contains a 1.2GHz
Marvell Sheeva processor (ARMv5), 512MB DDR2 and 512MB NAND
ash, USB, gigabit ethernet
and a SDIO interface.
Figure 1.2: Marvell SheevaPlug
For development, it is also very interesting that the device comes with serial and JTAG access
(through USB) out of the box, making it very easy to get started with.
root@debian:~# cat /proc/cpuinfo
Processor : ARM926EJ-S rev 1 (v5l)
BogoMIPS : 1192.75
Features : swp half thumb fastmult edsp
CPU implementer : 0x56
CPU architecture: 5TE
CPU variant : 0x2
CPU part : 0x131
CPU revision : 1
Cache type : write-back
Cache clean : cp15 c7 ops
Cache lockdown : format C
Cache format : Harvard
I size : 16384
I assoc : 4
I line length : 32
I sets : 128
D size : 16384
1You could imagine a setup without it, but it wouldn't be very useful.

D assoc : 4
D line length : 32
D sets : 128
Hardware : Feroceon-KW
Revision : 0000
Serial : 0000000000000000
The price of a SheevaPlug is around 75 Euros.
1.3.2 Dreambox
The dreambox devices (see Figure 1.3) are very popular DVB (S/T/C) decoders that are all
running Linux. Since the code is open; a lot of alternative

rmwares are available on the internet;
oering more
exibility and functionality than the original

rmware.
root@dm7025:~ cat /proc/cpuinfo
system type : ATI XILLEON HDTV SUPERTOLL
processor : 0
cpu model : MIPS 4KEc V4.8
BogoMIPS : 297.98
wait instruction : yes
microsecond timers : yes
tlb_entries : 16
extra interrupt vector : yes
hardware watchpoint : yes
VCED exceptions : not available
VCEI exceptions : not available
Unfortunately; the Flemish DVB-C provider has chosen a closed box approach (generate rev-enue
based on trivial functionality like recording, pause, delayed playback, . . . ) and getting a
dreambox to run for cable TV is not that trivial in Flanders. There are reports that programming
the default box number (read with a JTAG probe) should work.
The most popular use is receiving DVB-S. Even though that the Satellite provider does not
support a dreambox; it is fully functional with the default

rmware in combination with a CI
(Common Interface) module or by replacing the

rmware with an alternative version that provides
a software CAM (Conditional Access Module).
Figure 1.3: Dreambox 7025 S
Depending on the model, a dreambox can be obtained from 300 Euro onwards.

1.3.3 Linksys NSLU2
Another extremely popular device up until recently is the Linksys NSLU2 (see Figure 1.4) (Network
Storage Link for USB 2.0). It oers out of the box a ARMv5 CPU, running from
ash. Via a web
interface; the user can con

gure the hard disks that can accessed via a number of network protocols
(e.g. NFS, Samba, . . . ).
The real interesting part of this device is that the hacker does not need to stick with the on
board
ash to build the system on. If a USB disk (or memory stick) is connected; the root

lesystem
can be stored on the external device; while the kernel boots from
ash. With this modi

cation;
the NSLU can serve as full
edged Linux server, keeping into account the hardware limitations of
e.g. 32 MB memory.
[marc@chiana ~]$ cat /proc/cpuinfo
Processor : XScale-IXP42x Family rev 1 (v5l)
BogoMIPS : 266.24
Features : swp half fastmult edsp
CPU architecture: 5TE
CPU variant : 0x0
CPU part : 0x41f
CPU revision : 1
Cache type : undefined 5
Cache clean : undefined 5
Cache lockdown : undefined 5
I size : 32768
I assoc : 32
I line length : 32
I sets : 32
D size : 32768
D assoc : 32
D line length : 32
D sets : 32
Hardware : Linksys NSLU2
Revision : 0000
Serial : 0000000000000000
When the external HDD is replaced by a
ash memory pen, the full power of the NSLU2 is
unleashed: a running Linux system can be used with as little as 4 Watt power consumption. Some
people use it for e.g. Domotics control (EIB), network access points for all kinds of USB devices,
ssh tunnel server, bittorrent downloader, . . .
The price of a NSLU2 used to be around 70 Euros.
1.3.4 Bualo Linkstation Live
Unfortunately, the NSLU2 is being made obsolete in the course of 2008, but a good candidate to
replace the niche left by the NSLU2 is the Bualo Linkstatation Live (see Figure 1.5).
Two of the drawbacks for the NSLU2 were the limited CPU clocking (133 or 266 for newer
devices) and only 32 MB of memory. In contrast, the Linkstation Live pictured here, has an ARM9
CPU core, clocked at 400 MHz and 128 MB of memory. Especially for running a home server; the
additional memory comes in handy for multiple concurrent processes.
Again, the stock

rmware can be replaced with GNU/Debian and support for the Feroceon
processor is included from kernl 2.6.27 onwards.

Figure 1.4: Linksys NSLU2
Processor : Feroceon rev 0 (v5l)
BogoMIPS : 266.24
Features : swp half thumb fastmult edsp
CPU architecture: 5TEJ
CPU variant : 0x0
CPU part : 0x926
CPU revision : 0
Cache lockdown : format C
I size : 32768
I assoc : 1
I line length : 32
I sets : 1024
D size : 32768
D assoc : 1
D line length : 32
D sets : 1024
Hardware : Buffalo Linkstation Pro/Live
Revision : 0000
Serial : 0000000000000000
As are real nice hacker feature; the case designers left a hole to connect a serial level converter
to; giving direct access to the U-Boot bootloader. It is enough to solder a 90 degree header on the
motherboard to get serial access on the device.
Depending on the size of the disk, the price of a Linkstation Live is anywhere between 100 to
200 Euros. Note that it can be cheaper buying a device with a small HDD and replace the HDD
with a larger one; than buying the Linkstation with the large disk in the

Figure 1.5: Bualo Linkstation Live
1.3.5 Neo Freerunner
The Neo FreeRunner (see Figure 1.6) (made by FIC) is a smartphone developed by the Openmoko
project. It is the successor to the

rst development phase smartphone Neo 1973, and is intended
for hackers, since it gives the user great customizability.
Processor : ARM920T rev 0 (v4l)
BogoMIPS : 199.47
Features : swp half thumb
CPU architecture: 4T
CPU variant : 0x1
CPU part : 0x920
CPU revision : 0
Cache lockdown : format A
I size : 16384
I assoc : 64
I line length : 32
I sets : 8
D size : 16384
D assoc : 64
D line length : 32
D sets : 8
Hardware : GTA02
Revision : 0360
Serial : 0000000000000000
The default OpenMoko distribution can be replaced by Debian.
The Freerunner costs about 350 Euro.

Figure 1.6: Neo Freerunner
1.3.6 AzBox HD
Just like the Dreambox devices, the AZBox is a DVB decoder based on Linux. It has full hardware
decoding of MPEG4; which allows you to basically decode almost any current video, audio or
image format on your box.
It allows you to add diskspace with Samba, eSata, USB; . . .
Figure 1.7: The AzBox HD decoder, and much more...
system type: Sigma Designs TangoX
processor: 0
cpu model: MIPS 4KEc V6.9
Initial bogomips: 296.96
wait instruction: yes
microsecond timers: yes
tlb_entries: 32
extra interrupt vector: yes
Hardware watchpoint: yes
ASES implemented: mips16
VCED exceptions: not available
VCEI exceptions: not available
System bus frequency: 200250000 Hz

CPU frequency: 300375000 Hz
DSP frequency: 300375000 Hz
At around 350 Euro, it is a lot cheaper than its Dreambox HD counterpart (DM 8000). As with
all Dreambox devices; the custom

rmwares use Software CAM (Conditional Access Module); to
keep track of the key negociation for image decoding. As such; keys can be shared over the network.

Chapter 2
Cross Compilation Toolchain
2.1 Introduction
Before we can get started with developing embedded Linux systems we need a toolchain suitable
for generating code for our embedded platform. Development can be done natively (E.G. on the
embedded system itself once it is bootstrapped), but by far the most common setup is to use a
cross compiler.
A cross compiler allows the developer to run the compilation on a much more powerful plat-form
(a multiuser server or a powerful desktop machine) instead of the slower and more resource
constrained embedded system.
This chapter describes how to con

gure and compile such a cross toolchain from sources. It is
possible to download pre-compiled cross toolchains like the ones included in ELDK, but even if
you are not going to compile the toolchain yourself, it can be very useful to know how it is done.
Just like on a desktop Linux system, the toolchain of choice for an embedded Linux system is
the GNU toolchain.
Compiling a program takes place by running a compiler on the build platform. The compiled
program will run on the host platform. Usually these two are the same; if they are dierent, the
process is called cross-compilation.
Typically the hardware architecture diers, like for example when compiling a program destined
for the PowerPC architecture on an x86-64 computer; but cross-compilation is also applicable when
only the operating system environment diers, as when compiling a FreeBSD program under Linux;
or even just the system library, as when compiling programs with uClibc on a glibc host.
The GNU/Autotools packages (i.e. autoconf, automake, and libtool) use the notion of a build
platform, a host platform, and a target platform.
The build platform is where the code is actually compiled.
The host platform is where the compiled code will execute.
The target platform usually only applies to compilers as it represents what type of object code
the package itself will produce (such as cross-compiling a cross-compiler); otherwise the
target platform setting is irrelevant.
Since we will be compiling a target

ts in under 1 MB, we cannot perform
the compilation on the target itself (limited in
ash). Even if we could; it would still be better
and faster to do this in a server class machine. Even when compiling for a target architecture that
is similar to the server/development environment, there are valid arguments for using a cross-compiler;
especially when the product is relatively long lived and there are no plans to upgrade
the operating systems' libc version1.
1This is not a good idea in any case, but it beats having to keep around that single version of the obsolete
RH7.0, merely for building the

CHAPTER 2. CROSS COMPILATION TOOLCHAIN 10
In this chapter, the building blocks will be laid out for the cross compiler speci

cally targeted
for small embedded systems.
First, gcc will be introduced, followed by glibc. gcc and glibc is the typical compiler combi-nation
that is used in most desktop systems. The following section will cover a smaller alternative
to glibc: uClibc. Finally, gdb (and gdbserver) is introduced.
These are the building blocks for the cross compilation toolchain that we need for our previously
introduced target. Manually hacking up a compiler can be a challenging task; but luckily there is
an easier way: Buildroot, which is a set of Make

les doing exactly this2.
2.2 GNU Toolchain
A minimal GNU toolchain consists of binutils, the GNU Compiler Collection (GCC), and a C
library.
Binutils are the binary utilities of the toolchain, i.e. the programs that work with the binary
and object

les. This includes the assembler, linker, archiver and a number of smaller more-or-less
obscure utilities.
GCC is the compiler itself. GCC contains front-ends for a lot of languages (C, C++, Java,
Ada, Objective C, Fortran, ..), but here we will only focus on the C compiler.
Last, but not least, a C library is needed. The C library is part of the con

guration and creation
of the GNU toolchain because part of the compiler con

guration depends on the chosen C library.
Due to this, GCC has to be compiled in two steps. First a bootstrap compiler is compiled,
which is then used to compile the C library, which in turn is used to compile the

nal compiler.
To summarise, a GNU toolchain con

guration depends on 3 high level choices:
Build and target CPU type
Build and target Operating System
C library to use
A con

guration could for example be: A cross compiler running on an x86 Linux PC which
creates executables for an embedded Linux system with a PowerPC processor using the uClibc
C library (see below). To keep track of all these con

guration parameters, the following naming
convention is normally used for the binaries:
target-cpu-target-os-target-c-library-toolname
e.g. the C compiler for the above would be called:
powerpc-linux-uclibc-gcc
Next to these major con

guration choices, some more subtle tweaking is still available. One
of the most important of these is
oating point mode. The compiler can either be con

gured to
generate hardware
oating point instructions or use a software
oating point emulation. Hardware

oating point instructions can be used even if the CPU doesn't have a FPU, but then the kernel
has to emulate it, which is a lot slower than soft
oat (10-100x).
2.3 C Library
What C library to use? Several options exists, the most popular being the GNU C library (Glibc)
and uClibc:
2There are a number of alternatives that will not be covered here

2.3.1 GNU C Library
Glibc is the GNU project's C standard library. It is free software and is available under the GNU
Lesser General Public License. The lead contributor and maintainer is Ulrich Drepper.
Glibc is what is used for practically all desktop and server Linux distributions. It is very
featureful and supports a lot of dierent hardware platforms and operating systems. Unfortunately
it is also very big (several MBs), which makes it less suitable for building small embedded Linux
systems.
2.3.2 uClibc
uClibc is a small C library intended for embedded Linux systems.
uClibc was created to support uClinux, a version of Linux not requiring a memory management
unit and thus suited for microcontrollers (hence the uC in the name), but now also runs on real
Linux.
uClibc is much smaller than Glibc, but still very much compatible. For most applications no
change to the source code is needed to use uClibc.
While Glibc is intended to fully support all relevant C standards across a wide range of plat-forms,
uClibc is speci

cally focused on embedded Linux. Features can be enabled or disabled
according to space requirements.
uClibc doesn't support other operating systems than Linux. It supports amongst others: i386,
ARM, AVR32, Black

n, h8300, m68k, Microblaze, MIPS, Nios/Nios2, PowerPC, SuperH, SPARC,
and x86-64 processors.
2.4 Compilation
As described above, the GNU toolchain is a big system consisting of several independent packages,
every version of which might not be compatible with each other without extra patches. Finding a
working combinations of all these packages and Compiling the toolchain by hand is not a simple
job.
Luckily there now exists scripts to automate it, crosstool(-NG) and buildroot.
2.4.1 Crosstool-NG
Crosstool-ng is a tool by Yann E. Morin, which makes it easy to create cross toolchains using
uClibc/Glibc/EGlibc. Crosstool-ng is nice, but it only creates toolchains, so we will here instead
focus on Buildroot (see chapter 7).
Notice that Crosstool-ng toolchains can be used with Buildroot through its external toolchain
support.
2.4.2 Buildroot
Buildroot is a set of Make

les and patches that allows to easily generate cross toolchains using
uClibc. Actually it is more than that, as it can also be used to build the complete userspace for a
system, but more about that in chapter 7.
2.5 Hands On - Toolchain with Buildroot
2.5.1 Getting the Source
While the hardware platform for the duration of the course will be Marvell SheevaPlug, most of
the Barco designs use Buildroot to create a toolchain and/or the target

lesystem. The central
website for Buildroot is http://www.buildroot.net, See

Figure 2.1: http://buildroot.net
Buildroot until recently didn't have releases on a regular basis, but that has luckily changed.
As for getting the source, we take the latest version available (or you can check out the sources
with git).
[mleeman@cypher code]$ wget http://www.buildroot.net/downloads/
buildroot-2012.08.tar.bz2
[mleeman@cypher code]$ tar jxf buildroot-2012.08.tar.bz2
[mleeman@cypher code]$ cd buildroot-2012.08
2.5.2 Con

guring
[mleeman@cypher buildroot-2012.08]$ make menuconfig
As for most projects that tackle the complexity of creating a kernel; the con

guration can be
done in detail; con

guring each and every component; or in a more coarse fashion. Since most of
the developers focus on small and fast, it can be assumed that the defaults are reasonable (this
has been veri

ed by experience).
At this point, only a toolchain is created; that is the compiler, the binutils, optionally gdb,
and the (uC)libc version that is heavily intertwined with the compiler. When gdb is enabled for
the host, gdbserver for the target needs to be enabled too (one without the other does not make
much sense). When browsing through the options, disable all the target packages.
Figure 2.2: Con

guration Interface
If we have an existing toolchain con

le from a previous build (see Figure 2.2), we
can load it in the con

guration tool that is modelled on the Linux kernel con

Figure 2.3: Selecting a system wide path with a date-string avoids confusion and overwriting
existing toolchains
Compilers and libc libraries improve and evolve over time. On the other hand, installing a new
toolchain, is changing the entire engine of your embedded development and needs to be done with
care. Therefore, adding a date string in system wide path (where the toolchain will be placed) is
added to avoid this. This way, users can play with dierent compilers by just changing the date
string in their $PATH environment variable (see Figure 2.3).
Exit and save the con

nal list of changed options is rather short:
[mleeman@cypher buildroot-2012.08]$ make savedefconfig
[mleeman@cypher buildroot-2012.08]$ cat defconfig
BR2_arm=y
BR2_arm926t=y
BR2_PACKAGE_GDB_SERVER=y
BR2_PACKAGE_GDB_HOST=y
make savedefconfig creates a defconfig

le from the full .config, with only the settings
that are changed from the default.
Run make, sit back and enjoy3:
[mleeman@cypher buildroot-2012.08]$ make
Buildroot will now download and compile all the packages. If a question is asked for input; just
opt for the default values.
Depending on the speed of you machine, this will take from about an hour to several hours to
compile (after all, the GCC compiler is compiled 3).
The result for the target is is a number of

les located in output/images. The number of

les
depends on the targets you selected for the root

lesystem. Typical targets are archive, ubifs,
ext2, jffs2, . . .
In order to use it, add these lines to the bottom of your ~/.bashrc
3You will need to con

gure wget to either use a proxy that does not require authentication and that uses the
Barco proxy as a parent; or con

gure the .wgetrc to use the proxy-user and proxy-password options.

PATH=/users/firmware/mleeman/Development/
buildroot-2012.08/buildroot-2012.08/
output/host/usr/bin:$PATH
export PATH
and re-source your .bashrc
[mleeman@neo buildroot]$ . ~/.bashrc
A

nal check of our toolchain should result in:
[mleeman@cypher bin]$ ./arm-unknown-linux-uclibcgnueabi-gcc -v
Using built-in specs.
COLLECT_GCC=./arm-unknown-linux-uclibcgnueabi-gcc
COLLECT_LTO_WRAPPER=/users/firmware/mleeman/Development/buildroot-2012.08/buildroot-2012.08/output/host/usr/libexec/gcc/arm-unkTarget: arm-unknown-linux-uclibcgnueabi
...
Thread model: posix
gcc version 4.7.1 (Buildroot 2012.08)
2.5.3 Finishing up
After creating the toolchain, we want to distribute it in a clean fashion to other machines of similar
architecture (e.g. colleagues debugging in the

eld with laptops).
In order to do that; select a location more suitable than a home directory
(/opt/barco/arm/20120911/toolchain uclibc arm/); and build the toolchain there.
Assuming you've built the toolchain on a comparable machine, use the following command to
package the toolchain in a Debian package:
[mleeman@neo buildroot-20120911]$ tar cvfz toolchain_arm_uclibc_20120911.tar.gz
/opt/barco/arm/20120911/
[mleeman@neo buildroot-20120911]$ fakeroot alien --fixperms
toolchain_arm_uclibc_20120911.tar.gz
toolchain-arm-uclibc-20120911_1-2_all.deb generated
Note that we put the time stamp in the package name, instead as in the version name; since
we want to allow dierent version to exist next to each other after installation. If not, installing a
package with a more recent version will replace (and remove) the other package.
2.6 References
Cross Compile: http://en.wikipedia.org/wiki/Cross-compile
Remote Debugging: http://www.cucy.net/lacp/archives/000024.html
GCC: http://gcc.gnu.org
Glibc: http://www.gnu.org/software/libc/
uClibc: http://www.uclibc.org
Crosstool-NG: http://ymorin.is-a-geek.org/projects/crosstool
Buildroot: http://buildroot.net
Embedded Linux Development Kit (ELDK): http://www.denx.de/wiki/DULG/ELDK

Chapter 3
The Linux Boot Process
In the beginning, there was GRUB (or maybe LILO) and GRUB loaded the kernel,
and kernel begat init, and init begat rc, and rc begat network and httpd and getty,
and getty begat login, and login begat shell and so on.
3.1 Introduction
This section will cover the boot process of most Linux distributions. Even though there are some
dierences between the distributions, the process is alike.
The process of booting a Linux system consists of a number of stages, but whether a x86,
x86-64 desktop, server or a deeply embedded processor is booted, the
ow is similar. In this
chapter, we will explore the Linux boot process from the initial bootstrap to the start of the

rst
user-space application. Along the way; several boot-related topics such as the bootloaders, kernel
decompression and RAM disks and other element of the Linux boot process will be introduced.
As an example, a GNU/Debian 6.0 (Squeeze) on a x86-64 will be used to explain the process;
but booting on x86, PowerPC, Sparc, . . . are more or less the same.
In modern computers the bootstrapping process begins with the CPU executing software con-tained
in ROM (for example, the BIOS of an IBM PC) at a prede

ned address (the CPU is
designed to execute this software after reset without outside help). This software contains rudi-mentary
functionality to search for devices eligible to participate in booting, and load a small
program from a special section (most commonly the boot sector) of the most promising device.
Boot loaders may face peculiar constraints, especially in size; for instance, on the IBM PC and
compatibles, the

rst stage of boot loaders must

rst 446 bytes of the Master Boot
Record, in order to leave room for the 64-byte partition table and the 2-byte AA55h 'signature',
which the BIOS requires for a proper boot loader.
Today's computers are equipped with facilities to simplify the boot process, but that doesn't
necessarily make it simple.
Figure 3.1 shows a high level view of the Linux boot process. In the next sections, each step
will be elaborated.
When a system is

rst booted, or is reset, the processor executes code at a well-known location.
In a personal computer (PC), this location is in the basic input/output system (BIOS), which is
stored in
ash memory on the motherboard. The central processing unit (CPU) in an embedded
system invokes the reset vector to start a program at a known address in
ash/ROM. In a lot of
Linux based embedded processors; the devices is boot at a well know address (e.g. 0x00000100 on
Chip Select 0 (CS0)). Placing the bootloader (e.g. U-Boot) on that location will start it.
In either case, the result is the same. Because PCs oer so much
exibility, the BIOS must
determine which devices are candidates for boot. We'll look at this in more detail later.
When a boot device is found, the

rst-stage boot loader is loaded into RAM and executed. This
boot loader is less than 512 bytes in length (a single sector), and its job is to load the second-stage
15

CHAPTER 3. THE LINUX BOOT PROCESS 16
Figure 3.1: A high level view of the Linux boot process
boot loader.
When the second-stage boot loader is in RAM and executing, a splash screen is commonly
displayed, and Linux and an optional initial RAM disk (temporary root

le system) are loaded
into memory. When the images are loaded, the second-stage boot loader passes control to the
kernel image and the kernel is decompressed and initialised. At this stage, the kernel checks and
initialises the system hardware, enumerates the attached hardware devices, mounts the root device,
and then loads the necessary kernel modules. When complete, the

rst user-space program (init)
starts, and high-level system initialisation is performed.
That's Linux boot in a nutshell. Now let's dig in a little further and explore some of the details
of the Linux boot process.
3.2 Step 1: The Boot Manager
The boot manager is a small program that resides mostly on the MBR1 1 and presents a menu
for choosing the Operating System (if more than one is present); kernel or boot options to boot.
In the regular, plain-old-booting-linux business, all the boot loader does is:
Load the kernel into memory
Optionally load a ramdisk called initrd containing stu like disk drivers
Pass the kernel arguments, of which we are only interested in runlevel and init
Start execution of the kernel.
3.2.1 System startup
The system startup stage depends on the hardware that Linux is being booted on. On an embedded
platform, a bootstrap environment is used when the system is powered on, or reset. Examples
include U-Boot, RedBoot, and MicroMonitor from Lucent. Embedded platforms are commonly
shipped with a boot monitor. These programs reside in special region of
ash memory on the
target hardware and provide the means to download a Linux kernel image into
ash memory and
subsequently execute it. In addition to having the ability to store and boot a Linux image, these
1Master Boot Record.

boot monitors perform some level of system test and hardware initialisation. In an embedded
target, these boot monitors commonly cover both the

rst- and second-stage boot loaders.
In a PC, booting Linux begins in the BIOS at address 0xFFFF0. The

rst step of the BIOS is
the power-on self test (POST). The job of the POST is to perform a check of the hardware. The
second step of the BIOS is local device enumeration and initialisation.
Given the dierent uses of BIOS functions, the BIOS is made up of two parts: the POST
code and runtime services. After the POST is complete, it is
ushed from memory, but the BIOS
runtime services remain and are available to the target operating system.
To boot an operating system, the BIOS runtime searches for devices that are both active
and bootable in the order of preference de

ned by the complementary metal oxide semiconductor
(CMOS) settings. A boot device can be a
oppy disk, a CD-ROM, a partition on a hard disk, a
device on the network, or even a USB
ash memory stick.
Commonly, Linux is booted from a hard disk, where the Master Boot Record (MBR) contains
the primary boot loader. The MBR is a 512-byte sector, located in the

rst sector on the disk
(sector 1 of cylinder 0, head 0). After the MBR is loaded into RAM, the BIOS yields control to it.
3.2.2 Extracting the MBR
As an exercise, the MBR can be inspected. Use these commands:
$ sudo dd if=/dev/sda of=mbr.bin bs=512 count=1
$ od -xa mbr.bin
The dd command, which needs to be run from root. Since is is a bad habit of logging into your
system as root; we use the sudo command that gives the user temporarily root permissions. dd
reads the

rst 512 bytes from /dev/sda (the

rst disk drive) and writes them to the mbr.bin

le.
The od command prints the binary

le in hex and ASCII formats.
3.2.3 Stage 1 boot loader
The primary boot loader that resides in the MBR is a 512-byte image containing both program
code and a small partition table (see Figure 3.2). The

rst 446 bytes are the primary boot loader,
which contains both executable code and error message text. The next sixty-four bytes are the
partition table, which contains a record for each of four partitions (sixteen bytes each). The MBR
ends with two bytes that are de

ned as the magic number (0xAA55). The magic number serves
as a validation check of the MBR.
The job of the primary boot loader is to

nd and load the secondary boot loader (stage 2). It
does this by looking through the partition table for an active partition. When it

nds an active
partition, it scans the remaining partitions in the table to ensure that they're all inactive. When
this is veri

ed, the active partition's boot record is read from the device into RAM and executed.
3.2.4 Stage 2 boot loader
The secondary, or second-stage, boot loader could be more aptly called the kernel loader. The task
at this stage is to load the Linux kernel and optional initial RAM disk.
The

rst- and second-stage boot loaders combined are called Linux Loader (LILO) or GRand
Uni

ed Bootloader (GRUB) in the x86 PC environment. Both alternatives are pretty well docu-mented,
elaborating on the options server little purpose here. Most of the options and con

le with a lot of the options explained in commentary (e.g. /boot/grub/menu.lst
for GRUB and /etc/lilo.conf for LILO). Some distribution have patched versions for including
graphical themes instead of the default minimalistic text or curses-alike approach. A dierence
that should be mentioned is that LILO requires to run the lilo command after modifying the
con

le; while current GRUB version do not: the changes in /boot/grub/menu.lst are
instantaneous.

Figure 3.2: Anatomy of the MBR
Because LILO has some disadvantages that were corrected in GRUB, let's look into GRUB.
The great thing about GRUB is that it includes knowledge of Linux

le systems. Instead of
using raw sectors on the disk, as LILO does, GRUB can load a Linux kernel from an ext2 or ext3

le system. It does this by making the two-stage boot loader into a three-stage boot loader. Stage
1 (MBR) boots a stage 1.5 boot loader that understands the particular

le system containing
the Linux kernel image. Examples include reiserfs stage1 5 (to load from a Reiser journaling

le
system) or e2fs stage1 5 (to load from an ext2 or ext3

le system). When the stage 1.5 boot loader
is loaded and running, the stage 2 boot loader can be loaded.
With stage 2 loaded, GRUB can, upon request, display a list of available kernels (de

ned
in /boot/grub/menu.lst). You can select a kernel and even amend it with additional kernel
parameters. Optionally, you can use a command-line shell for greater manual control over the
boot process.
With the second-stage boot loader in memory, the

le system is consulted, and the default
kernel image and initrd image are loaded into memory. With the images ready, the stage 2 boot
loader invokes the kernel image.
3.2.4.1 GRUB stage boot loaders
The /boot/grub directory contains the stage1, stage1.5, and stage2 boot loaders, as well as a
number of alternate loaders (for example, CR-ROMs use the iso9660 stage 1 5).

3.2.5 Kernel
With the kernel image in memory and control given from the stage 2 boot loader, the kernel stage
begins. The kernel image isn't so much an executable kernel, but a compressed kernel image. On
Linux systems, vmlinux is a statically linked executable

le that contains the Linux kernel in one
of the executable

le formats supported by Linux, including ELF, COFF and a.out. The vmlinux

le might be required for kernel debugging, generating symbol table or other operations, but must
be made bootable before being used as an operating system kernel by adding a multiboot header,
bootsector and setup routines.
Typically this is a zImage (compressed image, less than 512KB) or a bzImage (big compressed
image, greater than 512KB), that has been previously compressed with zlib. As the Linux kernel
matured, the size of the kernels generated by users grew beyond the limits imposed by some
architectures, where the space available to store the compressed kernel code is limited. The bzImage
(big zImage) format was developed to overcome this limitation by cleverly splitting the kernel over
discontiguous memory regions (see Figure 3.3). The bzImage format is still compressed using the
zlib algorithm2.
Figure 3.3: Anatomy of bzImage
At the head of this kernel image is a routine that does some minimal amount of hardware
setup and then decompresses the kernel contained within the kernel image and places it into high
memory. If an initial RAM disk image is present, this routine moves it into memory and notes it
for later use. The routine then calls the kernel and the kernel boot begins.
When the bzImage (for an x86 image) is invoked, you begin at ./arch/x86/boot/header.S in
the start assembly routine (see Figure 3.4 for the major
ow). This routine does some basic hard-ware
setup and invokes the startup 32 routine in ./arch/x86/boot/compressed/header.S. This
routine sets up a basic environment (stack, etc.) and clears the Block Started by Symbol (BSS).
The kernel is then decompressed through a call to a C function called decompress kernel (located
in ./arch/x86/boot/compressed/misc.c). When the kernel is decompressed into memory, it is
called. This is yet another startup 32 function, but this function is in ./arch/x86/kernel/header.S.
In the new startup 32 function (also called the swapper or process 0), the page tables are
initialised and memory paging is enabled. The type of CPU is detected along with any optional

oating-point unit (FPU) and stored away for later use. The start kernel function is then invoked
(init/main.c), which takes you to the non-architecture speci

c Linux kernel. This is, in essence,
the main function for the Linux kernel.
2Although there is the popular misconception that the bz- pre

x means that bzip2 compression is used (the
bzip2 package is often distributed with tools pre

xed with bz-, such as bzless, bzcat, etc.), this is not the case.

Figure 3.4: Major functions
ow for the Linux kernel x86 boot
With the call to start kernel, a long list of initialisation functions are called to set up inter-rupts,
perform further memory con

guration, and load the initial RAM disk. In the end, a call is
made to kernel thread (in ./arch/x86/kernel/process.c) to start the init function, which is
the

rst user-space process. Finally, the idle task is started and the scheduler can now take control
(after the call to cpu idle). With interrupts enabled, the pre-emptive scheduler periodically takes
control to provide multitasking.
During the boot of the kernel, the initial-RAM disk (initrd) that was loaded into memory
by the stage 2 boot loader is copied into RAM and mounted. This initrd serves as a temporary
root

le system in RAM and allows the kernel to fully boot without having to mount any physical
disks. Since the necessary modules needed to interface with peripherals can be part of the initrd,
the kernel can be very small, but still support a large number of possible hardware con

gurations.
After the kernel is booted, the root

le system is pivoted (via pivot root) where the initrd root

le system is unmounted and the real root

le system is mounted.
The initrd function allows you to create a small Linux kernel with drivers compiled as loadable
modules. These loadable modules give the kernel the means to access disks and the

le systems
on those disks, as well as drivers for other hardware assets. Because the root

le
system on a disk, the initrd function provides a means of bootstrapping to gain access to the
disk and mount the real root

le system. In an embedded target without a hard disk, the initrd
can be the

le system can be mounted via the Network File
System (NFS).
3.2.5.1 Manual boot in GRUB
From the GRUB command-line, you can boot a speci

c kernel with a named initrd image as
follows:
grub kernel /bzImage-2.6.22.6
[Linux-bzImage, setup=0x1400, size=0x29672e]
grub initrd /initrd-2.6.22.6.img
[Linux-initrd @ 0x5f13000, 0xcc199 bytes]

grub boot
Uncompressing Linux... Ok, booting the kernel.
If you don't know the name of the kernel to boot, just type a forward slash (/) and press the
Tab key. GRUB will display the list of kernels and initrd images.
3.2.5.2 decompress kernel output
The decompress kernel function is where you see the usual decompression messages emitted to
the display:
Uncompressing Linux... Ok, booting the kernel.
3.3 Step 2: init
After the kernel is booted and initialised, the kernel starts the

rst user-space application. This is
the

rst program invoked that is compiled with the standard C library. Prior to this point in the
process, no standard C applications have been executed.
The init argument the boot loader can pass to the kernel is the name of a program. Usually,
none is given, and the default, /sbin/init is used. But it need not be. Rarely do embedded systems
require the extensive initialisation provided by init (as con

gured through /etc/inittab). In
many cases, you can invoke a simple shell script that starts the necessary embedded applications.
A good example where /sbin/init is replaced by a script is in embedded systems; where a
read-only

lesystem is overlaid with another FS that is writable. The changes are written to a

ash

lesystem and during boot these changes are again overlaid on the RO

lesystem. In this
case, e.g. init=/etc/preinit is passed to the kernel as an argument.
#!/bin/sh
# script to do pivot root and allow the entire root filesystem to be
# written to
/sbin/insmod /lib/modules/$(uname -r)/kernel/fs/mini_fo/mini_fo.ko
/sbin/insmod /lib/modules/$(uname -r)/kernel/lib/zlib_deflate/zlib_deflate.ko
/sbin/insmod /lib/modules/$(uname -r)/kernel/fs/jffs2/jffs2.ko
if ! /bin/mount -t jffs2 -w -o noatime,nodiratime /dev/mtdblock7 /mnt/mtdblock7
then
/usr/bin/eraseall /dev/mtd7
/bin/mount -t jffs2 -w -o noatime,nodiratime /dev/mtdblock7 /mnt/mtdblock7
fi
mount -t mini_fo -o base=/,sto=/mnt/mtdblock7 / /mnt/mini_fo
cd /mnt/mini_fo
[ -e old_rootfs ] || mkdir -p old_rootfs
pivot_root . old_rootfs
exec /usr/sbin/chroot . /sbin/init
echo Oops, exec chroot didnt work! :( :( :(
exit 1
When the we pass the following parameter to the kernel: init=/bin/sh to the kernel, and then
a plain shell would be used instead of init.

What does the kernel do with init? It starts it. It's the only program the kernel itself starts,
everything else is started by init.
The regular Linux init will then read a

le called /etc/inittab to see what it has to do. The
format of that

le is somewhat involved and archaic, but it's not too complex3.
In order to understand the process of init, the concept of a runlevel needs to be introduced. A
runlevel is a state or mode, that is de

ned by the services that run in that mode. The runlevels
are derived from its Unix historical roots. Here services means services like sshd, network, ftpd
and, crond, . . .
Runlevels are needed because dierent systems can be used in dierent ways. Some services
are not available until the system is in a particular state or mode. Only when some lower services
are available, other higher services can be started/used.
Consider that your system disk, may be a LAN server and, is corrupted and you want to
repair it. In such situations, you do not expect other users to login to the system. Now you can
switch to runlevel 1 and perform the maintenance tasks on your disk. Since runlevel 1 doesn't
support network/multiuser login, other users cannot login to the system, when it is under main-tenance.
(i.e. When a low-level service

lesystem is not available, other high-level services such as
multiuser/network login cannot be started or used).
Linux has the following runlevels:
0 : Halt (Shutdown)
1 : Single User Mode
2 : Basic Multi-User mode without NFS
3 : Full Multi-User mode
4 : Not Used (User De

nable)
5 : Full Multi User Mode with X11 Login
6 : Reboot
Each runlevel runs a particular set of services. The list of all services in the system will be in
the /etc/init.d directory. There is a directory that corresponds to each runlevels.
For runlevel 0: /etc/rc0.d
For runlevel S: /etc/rcS.d
3A lot of embedded systems do not use the Sys-V init, but busybox init. The con

guration of the busybox
inittab

le is slightly dierent. Another option is initng. While classic init executes processes in sequence; and a
lot of these tasks are hardware dependent; the processor is idle while waiting the reply from the hardware. initng
tackles this by starting independent tasks in parallel, resulting in a faster boot-up; but a lot harder to con

gure
due to dependencies between tasks.

Each of these directory will contain many symbolic links. These links will point to the services
in the /etc/init.d directory. All these links will start with either an S or K. Each link is named
with a pre

x of K or S according to whether that particular service need to be killed or started
in that runlevel.
e.g.. Consider the following entries (symbolic links) in the directory /etc/rc0.d:
[mleeman@seraph ~]$ ls -1 /etc/rc0.d/
K11anacron
K11cron
K20autofs
K20courier-authdaemon
K20courier-mta
...
S50mdadm-raid
S60umountroot
S90halt
This directory corresponds to runlevel 0 which is shutdown. Here the services killall and
halt are started. All other services are killed. This can be seen since only killall and halt start
with S and all other entries start with K. You may wonder what if killall and halt services
start before the kill of all the other services. Unfortunately that doesnt happen. First all the kill
services in the directory will be executed, followed by the start services. If you need further info,
tweak into the /etc/init.d/rc

le which manages the start and stop of services when switching
runlevels.
The system starts, when init loads in an unde

ned state (sometimes called N), and then will
switch to one runlevel or another depending on what the runlevel argument from the bootloader
to the kernel was, and the contents of /etc/inittab.
For example, if the bootloader passed runlevel as 5, init will try to switch to that state. If no
runlevel argument was passed, it will use its default, which is in /etc/inittab
The default runlevel is de

le:
# The default runlevel.
id:3:initdefault:
By default it is set to runlevel 3 or 5 (when X11 is installed). It can be customized to your
needs4.
Some distributions (like Debian) de

ne a sysinit runlevel that is run

rst (/etc/rcS.d), and
starting as few processes as possible5.
Normally the only reason for the bootloader to pass an argument is if you want it to boot in an
unusual state, for example, a single-user mode for maintenance (runlevel 1), or with a replacement
init because of disk corruption (init=/bin/sh).
So, let's look at that

le in more detail.
3.3.1 Step 2.1: /etc/inittab
All lines starting with # are comments. The other lines are like this:
1:2345:respawn:/sbin/getty 38400 tty1
They have 4

elds, separated with colons, which mean (taken from the inittab(5) man page).
id : is a unique sequence of 1-4 characters which identi

es an entry in inittab (for versions of
sysvinit compiled with the old libc5 ( 5.2.18) or a.out libraries the limit is 2 characters).
4Alert: Be sure not to set the default to 0 or 6.
5In fact, Debian, as well as most of the distributions based on it, like Ubuntu, does not make any dierence
between runlevels 2 to 5, they are all there for the local admin to con

runlevels : lists the runlevels for which the speci

Embedded linux barco-20121001

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Embedded linux barco-20121001

Similar to Embedded linux barco-20121001 (20)

Recently uploaded

Recently uploaded (20)

Embedded linux barco-20121001