Minimizing energy consumption during CI/CD

1. Eco-friendly Linux kernel development
Minimizing energy consumption during CI/CD
Andrea Righi - Kernel Engineer at Canonical
Email: andrea.righi@canonical.com
5th
OSPM summit – Ancona - 2023

2. Agenda
●
Overview
●
Cost of CI/CD in the kernel
●
Use KernelCraft for fast and efficient CI/CD
●
Results
●
Demo
●
Q&A

3. Overview

4. ●
Lines of code:
$ find ( -name '*.[chS]' -o -name '*.rs' ) -exec wc -l {} ; | awk 'BEGIN{sum=0}{sum+=$1}END{print sum}'
32,190,298
●
Amount of patches applied last week:
$ git log --oneline v6.3-rc4..v6.3-rc5 | wc -l
244
●
Lines of code changed last week:
$ git diff --stat v6.3-rc4..v6.3-rc5 | tail -1
201 files changed, 1811 insertions(+), 913 deletions(-)
Linux kernel is big

5. Kernel development is difficult
● How can I reproduce a bug?
● How long does it take to build the kernel on my machine?
● How do I test this fix?
● Does my change cause problems on another arch?
● Does my change introduce a vulnerability?
● 😱

6. It’s not easy being a maintainer, either
● End users expect Linux distros to do proper testing
● Volume is massive

7. Cost of CI/CD in the kernel

8. Linux CI projects
● kselftests
● Syzbot
● 0-Day
● KernelCI
● CKI
● LKFT
● Fuego
● ...

9. Kernel CI workflow
●
Tests triggered periodically or by events (git push, patch on the ML, ...)
●
Create a source artifact
●
Build a fully-featured general-purpose kernel... 😴
●
Create packages for the new kernel (distro)
●
Deploy a test system (VM or bare metal)
●
Run tests
●
Collect results
●
Decommission a system

10. The problem of custom kernels
●
Ubuntu has a lot of custom kernels
●
Custom changes need to be tested (more than upstream changes)
●
Lots of kernels, systems and devices

11. Scalability
●
This approach doesn’t scale very well
●
More hardware! (expensive and not very energy efficient...)

12. Proposed solution

13. Proposed solution (in a nutshell)
●
Build minimal kernel to test individual features
●
Deploy a single golden system
●
Virtualize it among multiple VMs (in a CoW way)
●
Run tests in VM

14. Pros and cons
● Pros
●
A minimal kernel still allows to run tests effectively
●
More efficient in terms of time and energy
● Cons
●
Doesn’t work to test specific hardware
●
Kernel config coverage can be challenging
●
Identify the minimum subset of configs required to run the test(s)

15. KernelCraft

16. What is KernelCraft (kc)?
●
Build a kernel from source
●
Run it inside a virtualized snapshot of your live system
●
Based on virtme (by Andrew Lutomirski) and qemu / kvm

17. How does it work?
●
Generate a minimal kernel .config
●
Build the kernel from source in a couple of minutes
●
Run it inside a VM that is a snapshot of your live system (safe)
●
Host fs is exported to guest via 9p fs
●
Local changes are handled by overlayfs / tmpfs
●
Output is captured and can be fed to other tools

18. Results

19. Test
●
Hardware
●
Dell XPS 13
●
Test
●
Build kernel and run `uname -r` (avg of 10 kernel builds)
●
Metrics:
●
Elapsed time (sec)
●
Total power consumption (Joule)
●
Average power consumption (Watt)

20. Result: time (sec)
KernelCraft
Ubuntu
0 500 1000 1500 2000 2500 3000 3500 4000
Time (sec)

21. Result: total power consumption (Joule)
KernelCraft
Ubuntu
0 20000 40000 60000 80000 100000 120000
Power (J)

22. Result: average power (Watt)
KernelCraft
Ubuntu
0 5 10 15 20 25 30 35
Avg Power (W)

23. Result (summary)
●
v6.2-rc7 using kc
●
Time=371.749s, power=10,854.843 J (avg=29.19W)
●
2.6 kcal =~ half leaf of lettuce
●
v6.2-rc7 w/ Ubuntu config using make
●
Time=3481.965s, power=104,504.228 J (avg=30.013W)
●
25 kcal =~ half hard-boiled egg

24. Demo

25. commit 154e07c164859fc90bf4e8143f2f6c1af9f3a35e
Author: Andrea Righi <andrea.righi@canonical.com>
Date: Thu Mar 30 11:54:42 2023 +0200
l2tp: generate correct module alias strings
Commit 65b32f801bfb ("uapi: move IPPROTO_L2TP to in.h") moved the
definition of IPPROTO_L2TP from a define to an enum, but since
__stringify doesn't work properly with enums, we ended up breaking the
modalias strings for the l2tp modules:
$ modinfo l2tp_ip l2tp_ip6 | grep alias
alias: net-pf-2-proto-IPPROTO_L2TP
alias: net-pf-2-proto-2-type-IPPROTO_L2TP
alias: net-pf-10-proto-IPPROTO_L2TP
alias: net-pf-10-proto-2-type-IPPROTO_L2TP
Use the resolved number directly in MODULE_ALIAS_*() macros (as we
already do with SOCK_DGRAM) to fix the alias strings:
$ modinfo l2tp_ip l2tp_ip6 | grep alias
alias: net-pf-2-proto-115
alias: net-pf-2-proto-115-type-2
alias: net-pf-10-proto-115
alias: net-pf-10-proto-115-type-2
Moreover, fix the ordering of the parameters passed to
MODULE_ALIAS_NET_PF_PROTO_TYPE() by switching proto and type.
Fixes: 65b32f801bfb ("uapi: move IPPROTO_L2TP to in.h")
Link: https://lore.kernel.org/lkml/ZCQt7hmodtUaBlCP@righiandr-XPS-13-7390
Signed-off-by: Guillaume Nault <gnault@redhat.com>
Signed-off-by: Andrea Righi <andrea.righi@canonical.com>
Reviewed-by: Wojciech Drewek <wojciech.drewek@intel.com>
Tested-by: Wojciech Drewek <wojciech.drewek@intel.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

26. What’s next?

27. Open issues
●
Test coverage:
●
Minimal kernel doesn’t allow to test everything
●
We can’t easily perform tests that require to start services via systemd
●
fs performance over 9p is still not ideal
●
Nice improvement with:
●
00833408bb16 ("Merge tag '9p-for-6.1' of https://github.com/martinetd/linux")
●
virtme is almost an abandoned project
●
We are considering to fork and create a virtme-ng project

28. References

29. References
●
KernelCraft
https://github.com/arighi/kernelcraft
●
Virtme
https://github.com/arighi/virtme
https://github.com/amluto/virtme
●
All the benefits of CI/CD but “Greener”
https://www.perforce.com/blog/kw/green-software-development-ci-cd
●
Usage, Costs and Benefits of Continuous Integration in Open-Source Projects
http://cope.eecs.oregonstate.edu/papers/OpenSourceCIUsage.pdf

30. Questions?