Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Cookies for kernel developers


Published on

Maintaining stable Linux kernels isn't easy. This talk covers the work we are doing at Red Hat to automate the testing of Linux kernels and find bugs before they make it into stable kernels.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Cookies for kernel developers

  1. 1. Cookies for Kernel Developers Nikolai Kondrashov, Red Hat @spbnick Major Hayden, Red Hat @majorhayden Photo credit:
  2. 2. Who are we? Nikolai Kondrashov Senior Software Engineer Red Hat DIGImend project maintainer. Enjoys electronics and embedded as a hobby. Major Hayden Principal Software Engineer Red Hat Owner of too many domain names, including (Please do not give me any other ideas for domain names to buy.) ;)
  3. 3. Continuous Kernel Integration project A team from Red Hat's Base Operating Systems and Global QE ● China ● Czech Republic ● Finland ● United States “Cookie”
  4. 4. Maintaining stable kernels is a complex and difficult task Photo credit:
  5. 5. A developer writes a patchset that gets merged into the mainline* kernel tree * Patches that arrive here are included in the next major kernel release.
  6. 6. Time passes
  7. 7. The patchset becomes part of Greg’s* stable kernel release * Possibly the most efficient kernel developer on Earth
  8. 8. Time passes
  9. 9. A Linux distribution maintainer finds a bug or security problem* with that patch * After a lengthy investigation, bisection, and debugging
  10. 10. Distribution maintainer contacts the original developer
  11. 11. Time passes
  12. 12. By this time, the original developer can’t remember why they wrote the patch or what is in the patch
  13. 13. This is a frustrating, time consuming process.
  14. 14. What if we could find that problematic patch before it is ever merged? Photo credit: US Air Force
  15. 15. We built CI for kernel contributions
  16. 16. High-level process overview Clone the kernel tree, Apply patches (optional), Compile and test the kernel CI results go to kernel mailing lists as part of the original email thread + Watch git for commits, patchwork for patches
  17. 17. Building and testing Checkout, apply patches if any, compile the kernels Find hardware, install the kernels, run tests
  18. 18. Red Hat maintains many kernel tests and can run them on hard-to-get hardware platforms Including everyone’s favorite: IA-64! Photo credit: Wikipedia
  19. 19. Test suites onboarded so far PMTU discovery Connectathon NFS LTP lite USEX Libhugetlbfs KVM unit tests ...
  20. 20. Architectures x86_64 aarch64 ppc64le s390x IBM POWER8 IBM POWER9 INTEL AMD AppliedMicro Cavium Ampere Qualcomm Hisilicon IBM z12 IBM z13
  21. 21. Photo credit: Servers Mainframes Workstations Laptops Virtual machines Network cards Storage controllers Audio cards Infiniband adapters GPUs Platforms & Peripherals
  22. 22. What are we doing for upstream kernels today?
  23. 23. It all started in a conversation with Greg KH...
  24. 24. Coming upstream soon Merge Lint
  25. 25. Next tests++ trees++ latency--
  26. 26. Next open(planning) open(logs) open(docs)
  27. 27. Get your commits tested Write to We evaluate impact on RHEL RHELTREE You get your reports!
  28. 28. Have us run your tests Write to We evaluate impact on RHEL RHELTEST Together we add a wrapper for Beaker TEST ⚙ 🔧 You maintain it, we run it!
  29. 29. Why is Red Hat doing this?
  30. 30. We want better RHEL* *Red Hat Enterprise Linux Photo credit: Wikipedia
  31. 31. Get involved
  32. 32. Thank you! Ask some questions and get some (real) cookies. 🍪🍪🍪 Got questions after the talk? E-mail Visit (slides on the website soon!)