SlideShare a Scribd company logo
1 of 27
Download to read offline
Developping drivers on small machines
2013/09/24
Willy Tarreau
<willy@haproxy.com>
HAProxy / ALOHA R&D
http://www.haproxy.com/
1 of 27
Who am I ?
Lead architect at HAProxy France
Long time maintainer of very old kernels (2.4, 2.6.32)
Always interested in stuffing Linux anywhere :-)
2 of 27
Day to day job - the good and the ugly
Develop, test, optimize the HAProxy load balancer for extreme usages
with CPU usage at 85% system, you focus mostly on network stack and drivers
Develop tools to stress test haproxy and network stack
⇒ focus on efficiency and nothing else
Some recreation with easier tasks like GPIO, I2C, watchdogs, leds, LCD drivers for our ALOHA
appliances
3 of 27
Developing network products
Always requires multiple machines, minimum 3 :
the tested system must be saturated
the testers must not be saturated
4 of 27
Developing network products (cont'd)
CPUs and software become faster ⇒ hardware limits are hard to reach
Preferrably use 5+ machines (&switches) for reliable 20-60 Gbps tests
5 of 27
Developing network products (cont'd)
Cost and heat of hardware plus switches means they have to stay in the lab
Testing young features may require to update all kernels for each test (eg: TFO)
⇒ All this just to stress your code, how to simplify this ?
6 of 27
VMs ? not really suited for these tasks
Network workloads are highly sensitive to latency and jitter
Nothing reproducible ⇒ not suited for performance testing
SMP VMs do not always reproduce race bugs
Hardware emulation does not reflect reality (eg: bus latency, interrupt rates, inter-packet gaps,...)
Black-out on what the HV does (extra memory copies, merging, etc...)
Also VMs are not directly connectable to a breadboard :-)
7 of 27
Why not consider small HW alternatives ?
Portable code does not depend on specific hardware
Various architectures available: x86, MIPS, ARM, ...
don't be afraid of cross-compiling!
Smaller hardware is easier to saturate
⇒ less network nodes needed
Response time is emphasized (good for user interfaces)
ALOHA's WUI and APIs are developped on ALIX
Good reproducibility, sometimes even better than desktop PCs
Small caches / memory busses tend to magnify the impact of memory accesses
Single/dual issue CPUs, low GHz, 1..few cores, emphasizes effects of missed optimizations
⇒ Small HW behaves much like large HW with all numbers scaled down
8 of 27
Pros: feels much more "real" than VMs
Large choice of SMP hardware where concurrency really matters
Low latency access to onboard devices
⇒ most of the time is spent in YOUR code
Availability of Gigabit Ethernet ports, compatible with laptops/desktop PCs
⇒ 4 GigE ports on the AX3-4 :-)
Availability of a wide choice of busses (PCI, PCIe, USB, I2C, SPI, ...)
⇒ develop/experiment with standard hardware then move it to the target
9 of 27
Pros: Cost
I can afford to buy the hardware I want to play with, and have as many as I need (eg: 4-5 ALIX at home,
...)
I don't fear frying one (think GPIO, overclocking, stopping fans)
⇒ never fried one even after some soldering
I don't care about the risk of damage when carrying them in by bag
Sometimes the price/features ratio is attractive even for serious projects (eg: BBB)
Board Price Main features
BeagleBone Black $45 Cortex A8, HDMI, LAN, USB-powered
PC Engines ALIX 2D13 ~$100 x86, 3 LAN, mini-PCI, IDE
GlobalScale Mirabox ~$150 ARMv7, 2 GigE, PCIe, USB3
Some manufacturers are willing to donate hardware when you ask them!
10 of 27
Pros: Silence / Heat / Space / Weight / Power
Only fanless accepted, rare tolerance for very slow fans (Atom, XP-GP)
Heat : limited heating is the condition for no fan
No fan means no dust
Space : no dedicated room, it's easy to stack them on the desk
order short cables, but use power adapters with long cords
Not a single hard disk anymore at home nor in the lab
Weight : bring a few machines everywhere with you to test your ideas as they come.
Power : you don't want to carry the power adapters with you, and prefer to power the devices over USB
(LS/Dockstar/BBB only).
11 of 27
Pros: Build time
Kernel: start from xxx_defconfig and disable unused features
(eg: do you need SOUND ? WIRELESS ? BT ? SATA ? IPSEC ? IPV6 ? NETFILTER ?)
⇒ stripped down configs build in just a few tens of seconds on a core i5 laptop
Incredibly appreciated for "git bisect" (make clean/make) !
But no savings to expect on user land in general
Choose the compiler version and/or options that provide you the best balance between features and
build speed. Remember this is not the target platform.
12 of 27
Pros: Boot time, console access
PC: one minute from power on to prompt is not uncommon (worse with PXE)
Booting in 10 seconds is normal on small hardware.
Can do better for some usages using a minimal initramfs
Keep a list of very short copy-pastable boot commands
Boot loaders could be improved but are already great (everything editable)
Even the nastiest kernel panics are caught verbatim over the serial console
Pay attention to the console, must be accessible from outside, and use true RS232, 3..5V TTL or
self-powered USB.
13 of 27
Pros: Resistance to bad behaviour
No risk of frying a hard drive or losing BIOS settings with power cycles.
Low currents, much more resistant than PC hardware to soldering mistakes
Does not require any special packing even in a bag full of junk
BUT! be careful not to confuse 12V adapters with 5V ones, this can be fatal!
14 of 27
Pros: Wide availability of hardware settings
Sometimes you want to understand what limits your code
When you have the datasheets (still waiting for your response Marvell), you realize how much margin
you get on certain settings such as CPU speed and RAM latency.
Dockstar's RAM bandwidth may be doubled by setting registers
Various other things such as PCIe speed, component priorities, etc...
⇒ Helps figuring what your code is sensible to.
15 of 27
Pros: options for supporting standard components
USB is omni-present
Several devices offer PCIe busses
⇒ appreciated to run a PC NIC with the same driver as the
target (eg: igb runs pretty well on the mirabox) →
I2C is almost always available and usable to develop various I/O boards
Prototype of ALOHA rack duo power distribution board
16 of 27
Cons
ASM optimizations or extended instruction sets not always present (eg: bitops, SSE missing on Atom,
integer divide missing on many ARM CPUs, ...)
Hardware perf counters can differ or not exist, making profiling harder
Timing-dependant bugs often require the target system
Not all available boards are fully supported by mainline kernels, be careful
Some boards are hard to boot (complex cmdlines with many memory mappings, out of tree drivers,
such as the snowball ...)
Some boards have limited designs that can fool you into believing there is a problem with your code
(eg: 470 Mbps Gig NICs, Ethernet over USB, ...)
Some platform specific patches might not be available for your platform (eg: Netmap)
⇒ Using small platforms does not save you from testing on the target platform from time to
time.
17 of 27
Important
Don't waste your time trying to build your own toolchain : use crosstool-ng instead (and if it fails,
contribute back)
Don't waste your time trying to get your preferred development platform to work, use another one
instead.
Don't try to establish performance ratios between your development platform and the target. That
never works!
Don't try to optimize for the development platform, think about the target
Do not claim any big improvement before verifying it on the target
18 of 27
General hints
Prefer little endian and alignment traps to ensure maximum portability
Minimal kernel to save both build time and boot time
Minimal initramfs to save on boot time
Put your SSH pubkeys in the root account
Configure .ssh/config to connect using short names (eg: ssh c1/c2/s1/s2)
Ensure that the target rootfs has all the required tools
NFS root can be even better but limits network experimentations
Don't be ashamed to keep git repos with extremely ugly patches that will never reach mainline if they
save you some time and hassle.
⇒ Only issue is with git bisect.
And don't pollute upstream maintainers with your specific crap !
19 of 27
Kernel specific hints
make ARCH=$arch CROSS_COMPILE=$cross_prefix {oldconfig|zImage|modules}
Enable CONFIG_EARLY_PRINTK, CONFIG_IKCONFIG_PROC, CONFIG_DEVTMPFS_MOUNT
Prefer modules for your driver, but some drivers might have annoying side effects
Better have several light configs than a single fat one
Base your work on mainline and always keep one recent working version
Keep all your experimental kernel configs, kernel versions are not enough
Dissociate the kernel from the rootfs
have the kernel embed its own modules. Busybox in an initramfs also helps a lot.
A build script does everything and even appends DTB for multiple boards and uploads the kernels to
the TFTP area.
20 of 27
Kernel specific hints (cont'd)
Overview of my kernel build script
make prepare1.
make -j modules2.
make modules_install3.
tar cf initramfs/modules.tar $INSTALLDIR4.
make -j {bzImage|uImage|zImage}5.
optionally: make dtbs && mkimage6.
21 of 27
Userspace hints
Spend enough time optimizing the boot process, it will pay off very quickly.
Disable all unneeded services, even the rarely used ones
Disable anything that requires network connectivity
Rootfs may appear in various forms (RAM, CF, NAND, MicroSD, USB, NFS, ?)
Use fast passwords (if any) for the console, and none for sudo
Keep a local config if booting many similar machines over the network
Don't use the same IP address on multiple devices
And note MAC/IP/hostname somewhere for easier identification when you start to get lost (/etc/hosts,
DHCP configs, ...)
22 of 27
Userspace hints (con't)
Do not miss any single tool, install any required library etc...
⇒ you need to test, debug and measure your code!
Mandatory in the rootfs : strace, taskset, top, vmstat, perf, tcpdump, if_rate
Mandatory available : gdb (both native and cross)
Copy-pastable notes with setup commands, local scripts, ...
Sometimes useful to have some generic scripts that depend on the device they're running on (eg:
network setup) .
If playing a lot with the network, reserve an Ethernet port for admin if possible
23 of 27
Better, build your own rootfs!
You need :
/sbin/init : busybox is a good start for this
having a pre-init is even more convenient
/dev : use devtmpfs
/proc, /sys: mount them as usual
/bin/everything : busybox is fine here as well
⇒ My rescue rootfs contains dev, tmp, bin, sbin, lib, var, usr/bin, usr/sbin, proc, sys and starts with a
pre-init calling /busybox --install when mounting root fails.
Various other projects allow you to build fully functional rootfs in the 1..10 MB range (OpenWRT,
Buildroot, LFS, OpenEmbedded, ...)
24 of 27
Various boot choices
No one size fits all
Kernel options
Local storage : FS (grub, syslinux), NAND (u-boot, redboot)
Remote storage : PXE (pxelinux), TFTP (u-boot, redboot)
Rootfs options
Local storage : block device (ext2/3/4, squashfs, ...)
Local storage : MTD (eg: ubifs)
Remote storage : NFS
Loaded into RAM by boot loader from local/remote image (ext2/3/4, squashfs, ...)
Large initramfs embedded in the bzImage
Loading over true GigE is generally faster than local devices which are generally faster than 100
Mbps network.
25 of 27
Hardware suggestions
My platforms of choice at the moment :
Dockstar for GigE powered over USB (modded)
OpenBlocks AX3-4 for SMP and high performance networking
Mirabox when PCIe is needed, or to have two GigE ports in my pocket
ALIX for i586, except I2C (unreliable)
Atom N450 for x86_64 (but has a small fan!)
GuruPlug for I2C devs (modded)
⇒ Various combinations when building small labs on my desk
My ideal board would be the size of a BBB, have a dual-core Cortex A9, 2 true GigE ports, a few I/Os
and be powered over USB.
26 of 27
Conclusion
Developing on the kernel is easy when starting with hardware drivers, and can be really
cheap thanks to the wide choice of platforms. Everyone can hack in the kernel.
My everyday bag's contents
27 of 27

More Related Content

What's hot

Kernel Proc Connector and Containers
Kernel Proc Connector and ContainersKernel Proc Connector and Containers
Kernel Proc Connector and ContainersKernel TLV
 
DPDK Integration: A Product's Journey - Roger B. Melton
DPDK Integration: A Product's Journey - Roger B. MeltonDPDK Integration: A Product's Journey - Roger B. Melton
DPDK Integration: A Product's Journey - Roger B. Meltonharryvanhaaren
 
Kernel Recipes 2019 - Kernel documentation: past, present, and future
Kernel Recipes 2019 - Kernel documentation: past, present, and futureKernel Recipes 2019 - Kernel documentation: past, present, and future
Kernel Recipes 2019 - Kernel documentation: past, present, and futureAnne Nicolas
 
The 7 Deadly Sins of Packet Processing - Venky Venkatesan and Bruce Richardson
The 7 Deadly Sins of Packet Processing - Venky Venkatesan and Bruce RichardsonThe 7 Deadly Sins of Packet Processing - Venky Venkatesan and Bruce Richardson
The 7 Deadly Sins of Packet Processing - Venky Venkatesan and Bruce Richardsonharryvanhaaren
 
DPDK in Containers Hands-on Lab
DPDK in Containers Hands-on LabDPDK in Containers Hands-on Lab
DPDK in Containers Hands-on LabMichelle Holley
 
Kernel Recipes 2015: Kernel packet capture technologies
Kernel Recipes 2015: Kernel packet capture technologiesKernel Recipes 2015: Kernel packet capture technologies
Kernel Recipes 2015: Kernel packet capture technologiesAnne Nicolas
 
Firewalld : A New Interface to Your Netfilter Stack
Firewalld : A New Interface to Your Netfilter StackFirewalld : A New Interface to Your Netfilter Stack
Firewalld : A New Interface to Your Netfilter StackMahmoud Shiri Varamini
 
Kernel Recipes 2015: How to choose a kernel to ship with a product
Kernel Recipes 2015: How to choose a kernel to ship with a productKernel Recipes 2015: How to choose a kernel to ship with a product
Kernel Recipes 2015: How to choose a kernel to ship with a productAnne Nicolas
 
Linux Instrumentation
Linux InstrumentationLinux Instrumentation
Linux InstrumentationDarkStarSword
 
The Linux Block Layer - Built for Fast Storage
The Linux Block Layer - Built for Fast StorageThe Linux Block Layer - Built for Fast Storage
The Linux Block Layer - Built for Fast StorageKernel TLV
 
How Linux Processes Your Network Packet - Elazar Leibovich
How Linux Processes Your Network Packet - Elazar LeibovichHow Linux Processes Your Network Packet - Elazar Leibovich
How Linux Processes Your Network Packet - Elazar LeibovichDevOpsDays Tel Aviv
 
Enable DPDK and SR-IOV for containerized virtual network functions with zun
Enable DPDK and SR-IOV for containerized virtual network functions with zunEnable DPDK and SR-IOV for containerized virtual network functions with zun
Enable DPDK and SR-IOV for containerized virtual network functions with zunheut2008
 
Kernel Recipes 2017 - 20 years of Linux Virtual Memory - Andrea Arcangeli
Kernel Recipes 2017 - 20 years of Linux Virtual Memory - Andrea ArcangeliKernel Recipes 2017 - 20 years of Linux Virtual Memory - Andrea Arcangeli
Kernel Recipes 2017 - 20 years of Linux Virtual Memory - Andrea ArcangeliAnne Nicolas
 
OVS and DPDK - T.F. Herbert, K. Traynor, M. Gray
OVS and DPDK - T.F. Herbert, K. Traynor, M. GrayOVS and DPDK - T.F. Herbert, K. Traynor, M. Gray
OVS and DPDK - T.F. Herbert, K. Traynor, M. Grayharryvanhaaren
 
Intel® RDT Hands-on Lab
Intel® RDT Hands-on LabIntel® RDT Hands-on Lab
Intel® RDT Hands-on LabMichelle Holley
 
Tommaso Cucinotta - Low-latency and power-efficient audio applications on Linux
Tommaso Cucinotta - Low-latency and power-efficient audio applications on LinuxTommaso Cucinotta - Low-latency and power-efficient audio applications on Linux
Tommaso Cucinotta - Low-latency and power-efficient audio applications on Linuxlinuxlab_conf
 
Kernel Recipes 2015 - So you want to write a Linux driver framework
Kernel Recipes 2015 - So you want to write a Linux driver frameworkKernel Recipes 2015 - So you want to write a Linux driver framework
Kernel Recipes 2015 - So you want to write a Linux driver frameworkAnne Nicolas
 
Kernel Recipes 2019 - Analyzing changes to the binary interface exposed by th...
Kernel Recipes 2019 - Analyzing changes to the binary interface exposed by th...Kernel Recipes 2019 - Analyzing changes to the binary interface exposed by th...
Kernel Recipes 2019 - Analyzing changes to the binary interface exposed by th...Anne Nicolas
 

What's hot (20)

Kernel Proc Connector and Containers
Kernel Proc Connector and ContainersKernel Proc Connector and Containers
Kernel Proc Connector and Containers
 
DPDK Integration: A Product's Journey - Roger B. Melton
DPDK Integration: A Product's Journey - Roger B. MeltonDPDK Integration: A Product's Journey - Roger B. Melton
DPDK Integration: A Product's Journey - Roger B. Melton
 
Kernel Recipes 2019 - Kernel documentation: past, present, and future
Kernel Recipes 2019 - Kernel documentation: past, present, and futureKernel Recipes 2019 - Kernel documentation: past, present, and future
Kernel Recipes 2019 - Kernel documentation: past, present, and future
 
The 7 Deadly Sins of Packet Processing - Venky Venkatesan and Bruce Richardson
The 7 Deadly Sins of Packet Processing - Venky Venkatesan and Bruce RichardsonThe 7 Deadly Sins of Packet Processing - Venky Venkatesan and Bruce Richardson
The 7 Deadly Sins of Packet Processing - Venky Venkatesan and Bruce Richardson
 
DPDK in Containers Hands-on Lab
DPDK in Containers Hands-on LabDPDK in Containers Hands-on Lab
DPDK in Containers Hands-on Lab
 
100 M pps on PC.
100 M pps on PC.100 M pps on PC.
100 M pps on PC.
 
Kernel Recipes 2015: Kernel packet capture technologies
Kernel Recipes 2015: Kernel packet capture technologiesKernel Recipes 2015: Kernel packet capture technologies
Kernel Recipes 2015: Kernel packet capture technologies
 
Firewalld : A New Interface to Your Netfilter Stack
Firewalld : A New Interface to Your Netfilter StackFirewalld : A New Interface to Your Netfilter Stack
Firewalld : A New Interface to Your Netfilter Stack
 
Kernel Recipes 2015: How to choose a kernel to ship with a product
Kernel Recipes 2015: How to choose a kernel to ship with a productKernel Recipes 2015: How to choose a kernel to ship with a product
Kernel Recipes 2015: How to choose a kernel to ship with a product
 
Understanding DPDK
Understanding DPDKUnderstanding DPDK
Understanding DPDK
 
Linux Instrumentation
Linux InstrumentationLinux Instrumentation
Linux Instrumentation
 
The Linux Block Layer - Built for Fast Storage
The Linux Block Layer - Built for Fast StorageThe Linux Block Layer - Built for Fast Storage
The Linux Block Layer - Built for Fast Storage
 
How Linux Processes Your Network Packet - Elazar Leibovich
How Linux Processes Your Network Packet - Elazar LeibovichHow Linux Processes Your Network Packet - Elazar Leibovich
How Linux Processes Your Network Packet - Elazar Leibovich
 
Enable DPDK and SR-IOV for containerized virtual network functions with zun
Enable DPDK and SR-IOV for containerized virtual network functions with zunEnable DPDK and SR-IOV for containerized virtual network functions with zun
Enable DPDK and SR-IOV for containerized virtual network functions with zun
 
Kernel Recipes 2017 - 20 years of Linux Virtual Memory - Andrea Arcangeli
Kernel Recipes 2017 - 20 years of Linux Virtual Memory - Andrea ArcangeliKernel Recipes 2017 - 20 years of Linux Virtual Memory - Andrea Arcangeli
Kernel Recipes 2017 - 20 years of Linux Virtual Memory - Andrea Arcangeli
 
OVS and DPDK - T.F. Herbert, K. Traynor, M. Gray
OVS and DPDK - T.F. Herbert, K. Traynor, M. GrayOVS and DPDK - T.F. Herbert, K. Traynor, M. Gray
OVS and DPDK - T.F. Herbert, K. Traynor, M. Gray
 
Intel® RDT Hands-on Lab
Intel® RDT Hands-on LabIntel® RDT Hands-on Lab
Intel® RDT Hands-on Lab
 
Tommaso Cucinotta - Low-latency and power-efficient audio applications on Linux
Tommaso Cucinotta - Low-latency and power-efficient audio applications on LinuxTommaso Cucinotta - Low-latency and power-efficient audio applications on Linux
Tommaso Cucinotta - Low-latency and power-efficient audio applications on Linux
 
Kernel Recipes 2015 - So you want to write a Linux driver framework
Kernel Recipes 2015 - So you want to write a Linux driver frameworkKernel Recipes 2015 - So you want to write a Linux driver framework
Kernel Recipes 2015 - So you want to write a Linux driver framework
 
Kernel Recipes 2019 - Analyzing changes to the binary interface exposed by th...
Kernel Recipes 2019 - Analyzing changes to the binary interface exposed by th...Kernel Recipes 2019 - Analyzing changes to the binary interface exposed by th...
Kernel Recipes 2019 - Analyzing changes to the binary interface exposed by th...
 

Similar to Developping drivers on small machines

Kernel Recipes 2016 - Speeding up development by setting up a kernel build farm
Kernel Recipes 2016 - Speeding up development by setting up a kernel build farmKernel Recipes 2016 - Speeding up development by setting up a kernel build farm
Kernel Recipes 2016 - Speeding up development by setting up a kernel build farmAnne Nicolas
 
Kernel Recipes 2017 - Build farm again - Willy Tarreau
Kernel Recipes 2017 - Build farm again - Willy TarreauKernel Recipes 2017 - Build farm again - Willy Tarreau
Kernel Recipes 2017 - Build farm again - Willy TarreauAnne Nicolas
 
Deep learning: Hardware Landscape
Deep learning: Hardware LandscapeDeep learning: Hardware Landscape
Deep learning: Hardware LandscapeGrigory Sapunov
 
Module 4 Embedded Linux
Module 4 Embedded LinuxModule 4 Embedded Linux
Module 4 Embedded LinuxTushar B Kute
 
IMCSummit 2015 - Day 1 Developer Track - Evolution of non-volatile memory exp...
IMCSummit 2015 - Day 1 Developer Track - Evolution of non-volatile memory exp...IMCSummit 2015 - Day 1 Developer Track - Evolution of non-volatile memory exp...
IMCSummit 2015 - Day 1 Developer Track - Evolution of non-volatile memory exp...In-Memory Computing Summit
 
Embedded platform choices
Embedded platform choicesEmbedded platform choices
Embedded platform choicesTavish Naruka
 
AI Hardware Landscape 2021
AI Hardware Landscape 2021AI Hardware Landscape 2021
AI Hardware Landscape 2021Grigory Sapunov
 
Design of Software for Embedded Systems
Design of Software for Embedded SystemsDesign of Software for Embedded Systems
Design of Software for Embedded SystemsPeter Tröger
 
Ceph Day London 2014 - Best Practices for Ceph-powered Implementations of Sto...
Ceph Day London 2014 - Best Practices for Ceph-powered Implementations of Sto...Ceph Day London 2014 - Best Practices for Ceph-powered Implementations of Sto...
Ceph Day London 2014 - Best Practices for Ceph-powered Implementations of Sto...Ceph Community
 
Parallelism Processor Design
Parallelism Processor DesignParallelism Processor Design
Parallelism Processor DesignSri Prasanna
 
Dad i want a supercomputer on my next
Dad i want a supercomputer on my nextDad i want a supercomputer on my next
Dad i want a supercomputer on my nextAkash Sahoo
 
08 Supercomputer Fugaku
08 Supercomputer Fugaku08 Supercomputer Fugaku
08 Supercomputer FugakuRCCSRENKEI
 
The Deck by Phil Polstra GrrCON2012
The Deck by Phil Polstra GrrCON2012The Deck by Phil Polstra GrrCON2012
The Deck by Phil Polstra GrrCON2012Philip Polstra
 
01 linux-quick-start
01 linux-quick-start01 linux-quick-start
01 linux-quick-startNguyen Vinh
 
fpga1 - What is.pptx
fpga1 - What is.pptxfpga1 - What is.pptx
fpga1 - What is.pptxssuser0de10a
 
Computer Systems And Networks Configuration
Computer Systems And Networks ConfigurationComputer Systems And Networks Configuration
Computer Systems And Networks ConfigurationTara Daly
 
Embedded Linux Basics
Embedded Linux BasicsEmbedded Linux Basics
Embedded Linux BasicsMarc Leeman
 
DataStax: Extreme Cassandra Optimization: The Sequel
DataStax: Extreme Cassandra Optimization: The SequelDataStax: Extreme Cassandra Optimization: The Sequel
DataStax: Extreme Cassandra Optimization: The SequelDataStax Academy
 

Similar to Developping drivers on small machines (20)

Kernel Recipes 2016 - Speeding up development by setting up a kernel build farm
Kernel Recipes 2016 - Speeding up development by setting up a kernel build farmKernel Recipes 2016 - Speeding up development by setting up a kernel build farm
Kernel Recipes 2016 - Speeding up development by setting up a kernel build farm
 
Kernel Recipes 2017 - Build farm again - Willy Tarreau
Kernel Recipes 2017 - Build farm again - Willy TarreauKernel Recipes 2017 - Build farm again - Willy Tarreau
Kernel Recipes 2017 - Build farm again - Willy Tarreau
 
Deep learning: Hardware Landscape
Deep learning: Hardware LandscapeDeep learning: Hardware Landscape
Deep learning: Hardware Landscape
 
Module 4 Embedded Linux
Module 4 Embedded LinuxModule 4 Embedded Linux
Module 4 Embedded Linux
 
IMCSummit 2015 - Day 1 Developer Track - Evolution of non-volatile memory exp...
IMCSummit 2015 - Day 1 Developer Track - Evolution of non-volatile memory exp...IMCSummit 2015 - Day 1 Developer Track - Evolution of non-volatile memory exp...
IMCSummit 2015 - Day 1 Developer Track - Evolution of non-volatile memory exp...
 
Embedded platform choices
Embedded platform choicesEmbedded platform choices
Embedded platform choices
 
AI Hardware Landscape 2021
AI Hardware Landscape 2021AI Hardware Landscape 2021
AI Hardware Landscape 2021
 
Design of Software for Embedded Systems
Design of Software for Embedded SystemsDesign of Software for Embedded Systems
Design of Software for Embedded Systems
 
Ceph Day London 2014 - Best Practices for Ceph-powered Implementations of Sto...
Ceph Day London 2014 - Best Practices for Ceph-powered Implementations of Sto...Ceph Day London 2014 - Best Practices for Ceph-powered Implementations of Sto...
Ceph Day London 2014 - Best Practices for Ceph-powered Implementations of Sto...
 
Parallelism Processor Design
Parallelism Processor DesignParallelism Processor Design
Parallelism Processor Design
 
Dad i want a supercomputer on my next
Dad i want a supercomputer on my nextDad i want a supercomputer on my next
Dad i want a supercomputer on my next
 
08 Supercomputer Fugaku
08 Supercomputer Fugaku08 Supercomputer Fugaku
08 Supercomputer Fugaku
 
The Deck by Phil Polstra GrrCON2012
The Deck by Phil Polstra GrrCON2012The Deck by Phil Polstra GrrCON2012
The Deck by Phil Polstra GrrCON2012
 
01 linux-quick-start
01 linux-quick-start01 linux-quick-start
01 linux-quick-start
 
fpga1 - What is.pptx
fpga1 - What is.pptxfpga1 - What is.pptx
fpga1 - What is.pptx
 
Computer Systems And Networks Configuration
Computer Systems And Networks ConfigurationComputer Systems And Networks Configuration
Computer Systems And Networks Configuration
 
It332 Kaplan Unit 10
It332 Kaplan Unit 10It332 Kaplan Unit 10
It332 Kaplan Unit 10
 
Embedded Linux Basics
Embedded Linux BasicsEmbedded Linux Basics
Embedded Linux Basics
 
DataStax: Extreme Cassandra Optimization: The Sequel
DataStax: Extreme Cassandra Optimization: The SequelDataStax: Extreme Cassandra Optimization: The Sequel
DataStax: Extreme Cassandra Optimization: The Sequel
 
Clustering
ClusteringClustering
Clustering
 

More from Anne Nicolas

Kernel Recipes 2019 - Driving the industry toward upstream first
Kernel Recipes 2019 - Driving the industry toward upstream firstKernel Recipes 2019 - Driving the industry toward upstream first
Kernel Recipes 2019 - Driving the industry toward upstream firstAnne Nicolas
 
Kernel Recipes 2019 - No NMI? No Problem! – Implementing Arm64 Pseudo-NMI
Kernel Recipes 2019 - No NMI? No Problem! – Implementing Arm64 Pseudo-NMIKernel Recipes 2019 - No NMI? No Problem! – Implementing Arm64 Pseudo-NMI
Kernel Recipes 2019 - No NMI? No Problem! – Implementing Arm64 Pseudo-NMIAnne Nicolas
 
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernelKernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernelAnne Nicolas
 
Kernel Recipes 2019 - Metrics are money
Kernel Recipes 2019 - Metrics are moneyKernel Recipes 2019 - Metrics are money
Kernel Recipes 2019 - Metrics are moneyAnne Nicolas
 
Embedded Recipes 2019 - Knowing your ARM from your ARSE: wading through the t...
Embedded Recipes 2019 - Knowing your ARM from your ARSE: wading through the t...Embedded Recipes 2019 - Knowing your ARM from your ARSE: wading through the t...
Embedded Recipes 2019 - Knowing your ARM from your ARSE: wading through the t...Anne Nicolas
 
Kernel Recipes 2019 - GNU poke, an extensible editor for structured binary data
Kernel Recipes 2019 - GNU poke, an extensible editor for structured binary dataKernel Recipes 2019 - GNU poke, an extensible editor for structured binary data
Kernel Recipes 2019 - GNU poke, an extensible editor for structured binary dataAnne Nicolas
 
Embedded Recipes 2019 - Remote update adventures with RAUC, Yocto and Barebox
Embedded Recipes 2019 - Remote update adventures with RAUC, Yocto and BareboxEmbedded Recipes 2019 - Remote update adventures with RAUC, Yocto and Barebox
Embedded Recipes 2019 - Remote update adventures with RAUC, Yocto and BareboxAnne Nicolas
 
Embedded Recipes 2019 - Making embedded graphics less special
Embedded Recipes 2019 - Making embedded graphics less specialEmbedded Recipes 2019 - Making embedded graphics less special
Embedded Recipes 2019 - Making embedded graphics less specialAnne Nicolas
 
Embedded Recipes 2019 - Linux on Open Source Hardware and Libre Silicon
Embedded Recipes 2019 - Linux on Open Source Hardware and Libre SiliconEmbedded Recipes 2019 - Linux on Open Source Hardware and Libre Silicon
Embedded Recipes 2019 - Linux on Open Source Hardware and Libre SiliconAnne Nicolas
 
Embedded Recipes 2019 - From maintaining I2C to the big (embedded) picture
Embedded Recipes 2019 - From maintaining I2C to the big (embedded) pictureEmbedded Recipes 2019 - From maintaining I2C to the big (embedded) picture
Embedded Recipes 2019 - From maintaining I2C to the big (embedded) pictureAnne Nicolas
 
Embedded Recipes 2019 - Testing firmware the devops way
Embedded Recipes 2019 - Testing firmware the devops wayEmbedded Recipes 2019 - Testing firmware the devops way
Embedded Recipes 2019 - Testing firmware the devops wayAnne Nicolas
 
Embedded Recipes 2019 - Herd your socs become a matchmaker
Embedded Recipes 2019 - Herd your socs become a matchmakerEmbedded Recipes 2019 - Herd your socs become a matchmaker
Embedded Recipes 2019 - Herd your socs become a matchmakerAnne Nicolas
 
Embedded Recipes 2019 - LLVM / Clang integration
Embedded Recipes 2019 - LLVM / Clang integrationEmbedded Recipes 2019 - LLVM / Clang integration
Embedded Recipes 2019 - LLVM / Clang integrationAnne Nicolas
 
Embedded Recipes 2019 - Introduction to JTAG debugging
Embedded Recipes 2019 - Introduction to JTAG debuggingEmbedded Recipes 2019 - Introduction to JTAG debugging
Embedded Recipes 2019 - Introduction to JTAG debuggingAnne Nicolas
 
Embedded Recipes 2019 - Pipewire a new foundation for embedded multimedia
Embedded Recipes 2019 - Pipewire a new foundation for embedded multimediaEmbedded Recipes 2019 - Pipewire a new foundation for embedded multimedia
Embedded Recipes 2019 - Pipewire a new foundation for embedded multimediaAnne Nicolas
 
Kernel Recipes 2019 - ftrace: Where modifying a running kernel all started
Kernel Recipes 2019 - ftrace: Where modifying a running kernel all startedKernel Recipes 2019 - ftrace: Where modifying a running kernel all started
Kernel Recipes 2019 - ftrace: Where modifying a running kernel all startedAnne Nicolas
 
Kernel Recipes 2019 - Suricata and XDP
Kernel Recipes 2019 - Suricata and XDPKernel Recipes 2019 - Suricata and XDP
Kernel Recipes 2019 - Suricata and XDPAnne Nicolas
 
Kernel Recipes 2019 - Marvels of Memory Auto-configuration (SPD)
Kernel Recipes 2019 - Marvels of Memory Auto-configuration (SPD)Kernel Recipes 2019 - Marvels of Memory Auto-configuration (SPD)
Kernel Recipes 2019 - Marvels of Memory Auto-configuration (SPD)Anne Nicolas
 
Kernel Recipes 2019 - Formal modeling made easy
Kernel Recipes 2019 - Formal modeling made easyKernel Recipes 2019 - Formal modeling made easy
Kernel Recipes 2019 - Formal modeling made easyAnne Nicolas
 
Kernel Recipes 2019 - CVEs are dead, long live the CVE!
Kernel Recipes 2019 - CVEs are dead, long live the CVE!Kernel Recipes 2019 - CVEs are dead, long live the CVE!
Kernel Recipes 2019 - CVEs are dead, long live the CVE!Anne Nicolas
 

More from Anne Nicolas (20)

Kernel Recipes 2019 - Driving the industry toward upstream first
Kernel Recipes 2019 - Driving the industry toward upstream firstKernel Recipes 2019 - Driving the industry toward upstream first
Kernel Recipes 2019 - Driving the industry toward upstream first
 
Kernel Recipes 2019 - No NMI? No Problem! – Implementing Arm64 Pseudo-NMI
Kernel Recipes 2019 - No NMI? No Problem! – Implementing Arm64 Pseudo-NMIKernel Recipes 2019 - No NMI? No Problem! – Implementing Arm64 Pseudo-NMI
Kernel Recipes 2019 - No NMI? No Problem! – Implementing Arm64 Pseudo-NMI
 
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernelKernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
 
Kernel Recipes 2019 - Metrics are money
Kernel Recipes 2019 - Metrics are moneyKernel Recipes 2019 - Metrics are money
Kernel Recipes 2019 - Metrics are money
 
Embedded Recipes 2019 - Knowing your ARM from your ARSE: wading through the t...
Embedded Recipes 2019 - Knowing your ARM from your ARSE: wading through the t...Embedded Recipes 2019 - Knowing your ARM from your ARSE: wading through the t...
Embedded Recipes 2019 - Knowing your ARM from your ARSE: wading through the t...
 
Kernel Recipes 2019 - GNU poke, an extensible editor for structured binary data
Kernel Recipes 2019 - GNU poke, an extensible editor for structured binary dataKernel Recipes 2019 - GNU poke, an extensible editor for structured binary data
Kernel Recipes 2019 - GNU poke, an extensible editor for structured binary data
 
Embedded Recipes 2019 - Remote update adventures with RAUC, Yocto and Barebox
Embedded Recipes 2019 - Remote update adventures with RAUC, Yocto and BareboxEmbedded Recipes 2019 - Remote update adventures with RAUC, Yocto and Barebox
Embedded Recipes 2019 - Remote update adventures with RAUC, Yocto and Barebox
 
Embedded Recipes 2019 - Making embedded graphics less special
Embedded Recipes 2019 - Making embedded graphics less specialEmbedded Recipes 2019 - Making embedded graphics less special
Embedded Recipes 2019 - Making embedded graphics less special
 
Embedded Recipes 2019 - Linux on Open Source Hardware and Libre Silicon
Embedded Recipes 2019 - Linux on Open Source Hardware and Libre SiliconEmbedded Recipes 2019 - Linux on Open Source Hardware and Libre Silicon
Embedded Recipes 2019 - Linux on Open Source Hardware and Libre Silicon
 
Embedded Recipes 2019 - From maintaining I2C to the big (embedded) picture
Embedded Recipes 2019 - From maintaining I2C to the big (embedded) pictureEmbedded Recipes 2019 - From maintaining I2C to the big (embedded) picture
Embedded Recipes 2019 - From maintaining I2C to the big (embedded) picture
 
Embedded Recipes 2019 - Testing firmware the devops way
Embedded Recipes 2019 - Testing firmware the devops wayEmbedded Recipes 2019 - Testing firmware the devops way
Embedded Recipes 2019 - Testing firmware the devops way
 
Embedded Recipes 2019 - Herd your socs become a matchmaker
Embedded Recipes 2019 - Herd your socs become a matchmakerEmbedded Recipes 2019 - Herd your socs become a matchmaker
Embedded Recipes 2019 - Herd your socs become a matchmaker
 
Embedded Recipes 2019 - LLVM / Clang integration
Embedded Recipes 2019 - LLVM / Clang integrationEmbedded Recipes 2019 - LLVM / Clang integration
Embedded Recipes 2019 - LLVM / Clang integration
 
Embedded Recipes 2019 - Introduction to JTAG debugging
Embedded Recipes 2019 - Introduction to JTAG debuggingEmbedded Recipes 2019 - Introduction to JTAG debugging
Embedded Recipes 2019 - Introduction to JTAG debugging
 
Embedded Recipes 2019 - Pipewire a new foundation for embedded multimedia
Embedded Recipes 2019 - Pipewire a new foundation for embedded multimediaEmbedded Recipes 2019 - Pipewire a new foundation for embedded multimedia
Embedded Recipes 2019 - Pipewire a new foundation for embedded multimedia
 
Kernel Recipes 2019 - ftrace: Where modifying a running kernel all started
Kernel Recipes 2019 - ftrace: Where modifying a running kernel all startedKernel Recipes 2019 - ftrace: Where modifying a running kernel all started
Kernel Recipes 2019 - ftrace: Where modifying a running kernel all started
 
Kernel Recipes 2019 - Suricata and XDP
Kernel Recipes 2019 - Suricata and XDPKernel Recipes 2019 - Suricata and XDP
Kernel Recipes 2019 - Suricata and XDP
 
Kernel Recipes 2019 - Marvels of Memory Auto-configuration (SPD)
Kernel Recipes 2019 - Marvels of Memory Auto-configuration (SPD)Kernel Recipes 2019 - Marvels of Memory Auto-configuration (SPD)
Kernel Recipes 2019 - Marvels of Memory Auto-configuration (SPD)
 
Kernel Recipes 2019 - Formal modeling made easy
Kernel Recipes 2019 - Formal modeling made easyKernel Recipes 2019 - Formal modeling made easy
Kernel Recipes 2019 - Formal modeling made easy
 
Kernel Recipes 2019 - CVEs are dead, long live the CVE!
Kernel Recipes 2019 - CVEs are dead, long live the CVE!Kernel Recipes 2019 - CVEs are dead, long live the CVE!
Kernel Recipes 2019 - CVEs are dead, long live the CVE!
 

Recently uploaded

The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)codyslingerland1
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch TuesdayIvanti
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024Brian Pichman
 
Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfInfopole1
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNeo4j
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveIES VE
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and businessFrancesco Corti
 
How to release an Open Source Dataweave Library
How to release an Open Source Dataweave LibraryHow to release an Open Source Dataweave Library
How to release an Open Source Dataweave Libraryshyamraj55
 
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2DianaGray10
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechProduct School
 
20140402 - Smart house demo kit
20140402 - Smart house demo kit20140402 - Smart house demo kit
20140402 - Smart house demo kitJamie (Taka) Wang
 
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox
 
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationIntroduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationKnoldus Inc.
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarThousandEyes
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTopCSSGallery
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingMAGNIntelligence
 
UiPath Studio Web workshop series - Day 1
UiPath Studio Web workshop series  - Day 1UiPath Studio Web workshop series  - Day 1
UiPath Studio Web workshop series - Day 1DianaGray10
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0DanBrown980551
 
2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdfThe Good Food Institute
 

Recently uploaded (20)

The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch Tuesday
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024
 
Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdf
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4j
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and business
 
How to release an Open Source Dataweave Library
How to release an Open Source Dataweave LibraryHow to release an Open Source Dataweave Library
How to release an Open Source Dataweave Library
 
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
 
20140402 - Smart house demo kit
20140402 - Smart house demo kit20140402 - Smart house demo kit
20140402 - Smart house demo kit
 
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
 
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationIntroduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its application
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? Webinar
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development Companies
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced Computing
 
UiPath Studio Web workshop series - Day 1
UiPath Studio Web workshop series  - Day 1UiPath Studio Web workshop series  - Day 1
UiPath Studio Web workshop series - Day 1
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0
 
2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf
 

Developping drivers on small machines

  • 1. Developping drivers on small machines 2013/09/24 Willy Tarreau <willy@haproxy.com> HAProxy / ALOHA R&D http://www.haproxy.com/ 1 of 27
  • 2. Who am I ? Lead architect at HAProxy France Long time maintainer of very old kernels (2.4, 2.6.32) Always interested in stuffing Linux anywhere :-) 2 of 27
  • 3. Day to day job - the good and the ugly Develop, test, optimize the HAProxy load balancer for extreme usages with CPU usage at 85% system, you focus mostly on network stack and drivers Develop tools to stress test haproxy and network stack ⇒ focus on efficiency and nothing else Some recreation with easier tasks like GPIO, I2C, watchdogs, leds, LCD drivers for our ALOHA appliances 3 of 27
  • 4. Developing network products Always requires multiple machines, minimum 3 : the tested system must be saturated the testers must not be saturated 4 of 27
  • 5. Developing network products (cont'd) CPUs and software become faster ⇒ hardware limits are hard to reach Preferrably use 5+ machines (&switches) for reliable 20-60 Gbps tests 5 of 27
  • 6. Developing network products (cont'd) Cost and heat of hardware plus switches means they have to stay in the lab Testing young features may require to update all kernels for each test (eg: TFO) ⇒ All this just to stress your code, how to simplify this ? 6 of 27
  • 7. VMs ? not really suited for these tasks Network workloads are highly sensitive to latency and jitter Nothing reproducible ⇒ not suited for performance testing SMP VMs do not always reproduce race bugs Hardware emulation does not reflect reality (eg: bus latency, interrupt rates, inter-packet gaps,...) Black-out on what the HV does (extra memory copies, merging, etc...) Also VMs are not directly connectable to a breadboard :-) 7 of 27
  • 8. Why not consider small HW alternatives ? Portable code does not depend on specific hardware Various architectures available: x86, MIPS, ARM, ... don't be afraid of cross-compiling! Smaller hardware is easier to saturate ⇒ less network nodes needed Response time is emphasized (good for user interfaces) ALOHA's WUI and APIs are developped on ALIX Good reproducibility, sometimes even better than desktop PCs Small caches / memory busses tend to magnify the impact of memory accesses Single/dual issue CPUs, low GHz, 1..few cores, emphasizes effects of missed optimizations ⇒ Small HW behaves much like large HW with all numbers scaled down 8 of 27
  • 9. Pros: feels much more "real" than VMs Large choice of SMP hardware where concurrency really matters Low latency access to onboard devices ⇒ most of the time is spent in YOUR code Availability of Gigabit Ethernet ports, compatible with laptops/desktop PCs ⇒ 4 GigE ports on the AX3-4 :-) Availability of a wide choice of busses (PCI, PCIe, USB, I2C, SPI, ...) ⇒ develop/experiment with standard hardware then move it to the target 9 of 27
  • 10. Pros: Cost I can afford to buy the hardware I want to play with, and have as many as I need (eg: 4-5 ALIX at home, ...) I don't fear frying one (think GPIO, overclocking, stopping fans) ⇒ never fried one even after some soldering I don't care about the risk of damage when carrying them in by bag Sometimes the price/features ratio is attractive even for serious projects (eg: BBB) Board Price Main features BeagleBone Black $45 Cortex A8, HDMI, LAN, USB-powered PC Engines ALIX 2D13 ~$100 x86, 3 LAN, mini-PCI, IDE GlobalScale Mirabox ~$150 ARMv7, 2 GigE, PCIe, USB3 Some manufacturers are willing to donate hardware when you ask them! 10 of 27
  • 11. Pros: Silence / Heat / Space / Weight / Power Only fanless accepted, rare tolerance for very slow fans (Atom, XP-GP) Heat : limited heating is the condition for no fan No fan means no dust Space : no dedicated room, it's easy to stack them on the desk order short cables, but use power adapters with long cords Not a single hard disk anymore at home nor in the lab Weight : bring a few machines everywhere with you to test your ideas as they come. Power : you don't want to carry the power adapters with you, and prefer to power the devices over USB (LS/Dockstar/BBB only). 11 of 27
  • 12. Pros: Build time Kernel: start from xxx_defconfig and disable unused features (eg: do you need SOUND ? WIRELESS ? BT ? SATA ? IPSEC ? IPV6 ? NETFILTER ?) ⇒ stripped down configs build in just a few tens of seconds on a core i5 laptop Incredibly appreciated for "git bisect" (make clean/make) ! But no savings to expect on user land in general Choose the compiler version and/or options that provide you the best balance between features and build speed. Remember this is not the target platform. 12 of 27
  • 13. Pros: Boot time, console access PC: one minute from power on to prompt is not uncommon (worse with PXE) Booting in 10 seconds is normal on small hardware. Can do better for some usages using a minimal initramfs Keep a list of very short copy-pastable boot commands Boot loaders could be improved but are already great (everything editable) Even the nastiest kernel panics are caught verbatim over the serial console Pay attention to the console, must be accessible from outside, and use true RS232, 3..5V TTL or self-powered USB. 13 of 27
  • 14. Pros: Resistance to bad behaviour No risk of frying a hard drive or losing BIOS settings with power cycles. Low currents, much more resistant than PC hardware to soldering mistakes Does not require any special packing even in a bag full of junk BUT! be careful not to confuse 12V adapters with 5V ones, this can be fatal! 14 of 27
  • 15. Pros: Wide availability of hardware settings Sometimes you want to understand what limits your code When you have the datasheets (still waiting for your response Marvell), you realize how much margin you get on certain settings such as CPU speed and RAM latency. Dockstar's RAM bandwidth may be doubled by setting registers Various other things such as PCIe speed, component priorities, etc... ⇒ Helps figuring what your code is sensible to. 15 of 27
  • 16. Pros: options for supporting standard components USB is omni-present Several devices offer PCIe busses ⇒ appreciated to run a PC NIC with the same driver as the target (eg: igb runs pretty well on the mirabox) → I2C is almost always available and usable to develop various I/O boards Prototype of ALOHA rack duo power distribution board 16 of 27
  • 17. Cons ASM optimizations or extended instruction sets not always present (eg: bitops, SSE missing on Atom, integer divide missing on many ARM CPUs, ...) Hardware perf counters can differ or not exist, making profiling harder Timing-dependant bugs often require the target system Not all available boards are fully supported by mainline kernels, be careful Some boards are hard to boot (complex cmdlines with many memory mappings, out of tree drivers, such as the snowball ...) Some boards have limited designs that can fool you into believing there is a problem with your code (eg: 470 Mbps Gig NICs, Ethernet over USB, ...) Some platform specific patches might not be available for your platform (eg: Netmap) ⇒ Using small platforms does not save you from testing on the target platform from time to time. 17 of 27
  • 18. Important Don't waste your time trying to build your own toolchain : use crosstool-ng instead (and if it fails, contribute back) Don't waste your time trying to get your preferred development platform to work, use another one instead. Don't try to establish performance ratios between your development platform and the target. That never works! Don't try to optimize for the development platform, think about the target Do not claim any big improvement before verifying it on the target 18 of 27
  • 19. General hints Prefer little endian and alignment traps to ensure maximum portability Minimal kernel to save both build time and boot time Minimal initramfs to save on boot time Put your SSH pubkeys in the root account Configure .ssh/config to connect using short names (eg: ssh c1/c2/s1/s2) Ensure that the target rootfs has all the required tools NFS root can be even better but limits network experimentations Don't be ashamed to keep git repos with extremely ugly patches that will never reach mainline if they save you some time and hassle. ⇒ Only issue is with git bisect. And don't pollute upstream maintainers with your specific crap ! 19 of 27
  • 20. Kernel specific hints make ARCH=$arch CROSS_COMPILE=$cross_prefix {oldconfig|zImage|modules} Enable CONFIG_EARLY_PRINTK, CONFIG_IKCONFIG_PROC, CONFIG_DEVTMPFS_MOUNT Prefer modules for your driver, but some drivers might have annoying side effects Better have several light configs than a single fat one Base your work on mainline and always keep one recent working version Keep all your experimental kernel configs, kernel versions are not enough Dissociate the kernel from the rootfs have the kernel embed its own modules. Busybox in an initramfs also helps a lot. A build script does everything and even appends DTB for multiple boards and uploads the kernels to the TFTP area. 20 of 27
  • 21. Kernel specific hints (cont'd) Overview of my kernel build script make prepare1. make -j modules2. make modules_install3. tar cf initramfs/modules.tar $INSTALLDIR4. make -j {bzImage|uImage|zImage}5. optionally: make dtbs && mkimage6. 21 of 27
  • 22. Userspace hints Spend enough time optimizing the boot process, it will pay off very quickly. Disable all unneeded services, even the rarely used ones Disable anything that requires network connectivity Rootfs may appear in various forms (RAM, CF, NAND, MicroSD, USB, NFS, ?) Use fast passwords (if any) for the console, and none for sudo Keep a local config if booting many similar machines over the network Don't use the same IP address on multiple devices And note MAC/IP/hostname somewhere for easier identification when you start to get lost (/etc/hosts, DHCP configs, ...) 22 of 27
  • 23. Userspace hints (con't) Do not miss any single tool, install any required library etc... ⇒ you need to test, debug and measure your code! Mandatory in the rootfs : strace, taskset, top, vmstat, perf, tcpdump, if_rate Mandatory available : gdb (both native and cross) Copy-pastable notes with setup commands, local scripts, ... Sometimes useful to have some generic scripts that depend on the device they're running on (eg: network setup) . If playing a lot with the network, reserve an Ethernet port for admin if possible 23 of 27
  • 24. Better, build your own rootfs! You need : /sbin/init : busybox is a good start for this having a pre-init is even more convenient /dev : use devtmpfs /proc, /sys: mount them as usual /bin/everything : busybox is fine here as well ⇒ My rescue rootfs contains dev, tmp, bin, sbin, lib, var, usr/bin, usr/sbin, proc, sys and starts with a pre-init calling /busybox --install when mounting root fails. Various other projects allow you to build fully functional rootfs in the 1..10 MB range (OpenWRT, Buildroot, LFS, OpenEmbedded, ...) 24 of 27
  • 25. Various boot choices No one size fits all Kernel options Local storage : FS (grub, syslinux), NAND (u-boot, redboot) Remote storage : PXE (pxelinux), TFTP (u-boot, redboot) Rootfs options Local storage : block device (ext2/3/4, squashfs, ...) Local storage : MTD (eg: ubifs) Remote storage : NFS Loaded into RAM by boot loader from local/remote image (ext2/3/4, squashfs, ...) Large initramfs embedded in the bzImage Loading over true GigE is generally faster than local devices which are generally faster than 100 Mbps network. 25 of 27
  • 26. Hardware suggestions My platforms of choice at the moment : Dockstar for GigE powered over USB (modded) OpenBlocks AX3-4 for SMP and high performance networking Mirabox when PCIe is needed, or to have two GigE ports in my pocket ALIX for i586, except I2C (unreliable) Atom N450 for x86_64 (but has a small fan!) GuruPlug for I2C devs (modded) ⇒ Various combinations when building small labs on my desk My ideal board would be the size of a BBB, have a dual-core Cortex A9, 2 true GigE ports, a few I/Os and be powered over USB. 26 of 27
  • 27. Conclusion Developing on the kernel is easy when starting with hardware drivers, and can be really cheap thanks to the wide choice of platforms. Everyone can hack in the kernel. My everyday bag's contents 27 of 27