The document discusses managing custom kernel builds. It defines kernel versioning numbers and describes reasons for custom kernels such as better performance or supporting custom drivers. It recommends thoroughly testing new kernels and provides tips for automating building kernels across releases by branching the source code and rebasing patches. The document advises monitoring security issues and performance fixes for backporting and cautions against patching sensitive areas like scheduling code.

1. Manage custom kernel builds
Marian HackMan Marinov
CEO of 1H Ltd.
CTO of GetClouder Ltd.
<mm @ 1h .com >
ICQ: 7556201
Jabber: hackman@jabber.org
IRC: irc.freenode.net HackMan

2. Some definitions
3.X.Y

3. Some definitions
3.X.Y
➢ Major release (changes to this are rear)

4. Some definitions
3.X.Y
➢ Major release
➢ Minor release (the new Major :))

5. Some definitions
3.X.Y
➢ Major release
➢ Minor release
➢ Patch level (the new minor)

6. Some definitions
➢Mainline kernel - http://kernel.org
➢stable
➢release candidate (-rc)
➢next
➢long term support (LTS)
➢Distribution kernel
➢Debian
➢RedHat
➢SuSE
➢Project kernel
➢OpenVZ

7. Why do we need custom kernels?
➢Better performance
➢Faster boot times
➢More secure kernels(smaller attack serfice)
➢Monolithic kernels (does not work on machines
with SW RAID or LVM)
➢Support custom drivers
➢Support features that are not included in the
mainline
➢New features that are not supported by the
distribution kernels

8. Support custom drivers
➢network interface drivers
➢storage drivers
➢vendor supplied drivers
➢sometimes require specific kernel version
➢ they should be either back or forward ported
➢require changes in the kernel in order to build/load
properly
Note: this does not happen often

9. Support custom features
➢Kpatch (live kernel patching)
➢kGraft (live kernel patching)
➢AUFS (Docker)
➢BFQ (I/O scheduling)
➢GRsecurity

10. Problems with the custom features
➢Irregular releases
➢Required specific kernel versions
➢No release for the kernel you have chosen

11. How to choose a kernel to build upon
➢TEST

12. How to choose a kernel to build upon
➢TEST, TEST

13. How to choose a kernel to build upon
➢TEST, TEST, TEST

14. How to choose a kernel to build upon
➢TEST, TEST, TEST
➢Build every major release

15. How to choose a kernel to build upon
➢TEST, TEST, TEST
➢Build every major release
➢Try to experiment with x.y.1 or x.y.2 but not x.y.0
:)

16. How to choose a kernel to build upon
➢TEST, TEST, TEST
➢Build every major release
➢Try to experiment with x.y.1 or x.y.2 but not x.y.0
:)
➢The above is not a hard rule

17. How to choose a kernel to build upon
➢TEST, TEST, TEST
➢Build every major release
➢Try to experiment with x.y.1 or x.y.2 but not x.y.0
:)
➢The above is not a hard rule
➢Try LTS versions, but don't limit your self to
these kernels

18. How to choose a kernel to build upon
➢TEST, TEST, TEST
➢Build every major release
➢Try to experiment with x.y.1 or x.y.2 but not x.y.0
:)
➢The above is not a hard rule
➢Try LTS versions, but don't limit your self to
these kernels
➢Test the performance

19. Test the new Kernel performance
➢Look at your own load
➢Try to break the kernel
➢try to replay your normal load 100 or 1000 times
➢try to break the kernel with community benchmarks
➢Do not compare 32 and 64 bit kernels
➢Test the performance between two kernels in a
row
➢If you compare the performance between
different major releases, do not include the new
features

20. Test the new Kernel performance
➢I usually check if the following stay relatively the
same
➢/proc/loadavg
➢/proc/schedstat
➢is the memory allocation keeps similar values
➢I/O performance
➢Network performance (use tcpreplay)
➢hardware functionality
➢software compatibility

21. Feature automation
Initially
➢Create a branch with a name that should
include the name of the kernel version you are
using
example kernel: 3.17.2
example branch name: grsec-3.17.2

22. Feature automation
Second time
1. Check if you have a branch with the name of
your feature (grsec-3.17.2)
2. Check if you have a branch that includes the
name of the future and the version of your
current kernel
3. Check if there are newer versions of the kernel
4. If there are no new versions(tags)... don't do
anything
5. If there are new versions(tags)

23. Feature automation
6. Checkout the current branch into a new one named
with the new version
example: grsec-3.17.3
7. Rebase the new branch over the last released kernel
version
1. If there are errors, send me an e-mail
2. If no errors are detected, commit
3. Rename the branch to reflect the new kernel version
9. Do some cleanup from time to time :)
- You may move your archive branches into a separate
repository

24. Feature automation
Master
featureX-3.17.3
v3.18
v3.17.4
v3.17.3
v3.17.2

25. Feature automation
Master
TAG
featureX-3.17.3
v3.18
v3.17.4
v3.17.3
v3.17.2

26. Feature automation
Master
TAG
featureX-3.17.3
v3.18
v3.17.4
v3.17.3
v3.17.2
featureX-3.17.4

27. Feature automation
Master
TAG
featureX-3.17.3
v3.18
v3.17.4
v3.17.3
v3.17.2
TAG
featureX-3.17.4

28. Feature automation
Master
featureX-3.18
TAG
featureX-3.17.3
v3.18
v3.17.4
v3.17.3
v3.17.2
TAG
featureX-3.17.4

29. The actual management
➢ this works ONLY for your own patches
➢ keep a list with all your patches
➢ clone the Linus repo

30. The actual management
➢ git pull
➢ if there are new tags, create a test branch with the tag's
name:
v3.17.2 -> test-kernel-3.17.2
➢ for each patch branch that you keep
➢ checkout the current patch branch
➢ branch into a new name based on the new kernel version
➢ rebase
➢ checkout that branch
➢ copy my current kernel .config
➢ make olddefconfig (if it breaks, send me an e-mail)

31. Build
➢ Pull once a day or a little bit more often :)
➢ Build once a week or little bit more often :)
➢ If you are planning on upgrading this kernel, fix
found conflicts at least every two weeks

32. What is next?
➢ Handling CVEs
➢ Kernel patch and build
➢ Kpatch or kGraft patch module generation
➢ Load the kpatch or kgraft
➢ Monitor the Linux Kernel Mailing List(LKML) for
all subsystems of interest to you
➢ Do not try to monitor the LKML for everything...
➢ Backport the performance fixes made in new
versions

33. What not to patch?
➢ Assembly code
➢ Crypto stuff
➢ Limit the architectures you write patches for
➢ Be extremely careful with:
➢ cpu schedule
➢ vm scheduler
➢ i/o schedule

34. What not to patch?
Questions?
<mm @ 1h .com >
ICQ: 7556201
Jabber: hackman@jabber.org
IRC: irc.freenode.net HackMan
CEO of 1H Ltd.
CTO of GetClouder Ltd.