Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Continuous Kernel Integration

860 views

Published on

Adding CI for the kernel is not easy, but the Continuous Kernel Integration (CKI) team is aiming to do just that.

Published in: Technology
  • Get Now to Download This eBook === http://ebookdfsrewsa.justdied.com/2869303092-le-texas-par-la-queue.html
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Continuous Kernel Integration

  1. 1. Continuous Kernel Integration Major Hayden @majorhayden June 1, 2019 - Texas Linux Fest Photo: David Mark
  2. 2. Major Hayden Principal Software Engineer, Red Hat major@redhat.com 🤖 Kernel testing automator (CKI Project) 😺 Maintainer of icanhazip.com 📻 Ham radio operator (W5WUT) 🤦‍♂ Hoarder of domain names
  3. 3. It all started with a hunt for a new challenge. Photo: NASA
  4. 4. JOB INTERVIEW: “So we want you to help us do CI for the Linux kernel.” Photo: NASA
  5. 5. The team was moving quickly and I had a lot to learn. Photo: NASA
  6. 6. As it turns out, the Linux kernel is complicated. This creates lots of challenges.
  7. 7. CHALLENGE: It moves really quickly.
  8. 8. ≈ 14,000 changesets per release from over 1,700 different developers ≈ 150-300 are first time contributors Source: LWN
  9. 9. kubernetes, a very active project, has 45,264 changesets since 2014 (that’s equivalent to about three Linux kernel releases) Source: Stackalytics Photo: CNCF
  10. 10. From 5.0 to 5.1: 545,000 lines added 289,000 lines removed (net growth ≈ 256K lines) Source: LWN
  11. 11. CHALLENGE: The kernel development process is complex. Source: LWN
  12. 12. “If it’s not on the mailing list, then it didn’t happen.” Source: LWN
  13. 13. CHALLENGE: Merging a patchset into the kernel is not the same as a simple pull request.
  14. 14. Lifecycle of a kernel patch 2018-02-23: Submitted a patch for usb-serial 2018-02-26: Merged by subsystem maintainer 2018-02-28: Added to linux-next 2018-04-08: Released in stable 4.16.1 and 4.14.33 2018-06-03: Released in mainline 4.17 2018-10-21: Released in stable 3.16.60 9608e5c0f079 USB: serial: ftdi_sio: add RT Systems VX-8 cable
  15. 15. Linux does a lot of stuff that is difficult to test Photo: Wikipedia
  16. 16. When I talk about doing CI for the Linux kernel: “Should we really be doing that?” Source: LWN
  17. 17. Much more of the work involves winning over hearts and minds rather than writing good code. Source: LWN
  18. 18. Initial reaction: Source: LWN
  19. 19. After thinking it over for a little while: Source: LWN
  20. 20. OUR MISSION STATEMENT: Find bugs before they are merged and prevent them from being merged.
  21. 21. INITIAL GOALS: 1. Verify patchset merges cleanly 2. Ensure the kernel compiles 3. Boot the kernel and run LTP 4. Send a report to people who care Photo: pxhereLTP = Linux Test Project
  22. 22. 🐣 SUBSYSTEM* 🔥 LINUX-NEXT 🐧 MAINLINE 🐿 STABLE 🎉 RHEL KERNEL Simplified Kernel patch flow * There may be multiple layers of subsystem trees in some areas of the kernel, especially networking. 🤔 DEVELOPER
  23. 23. 🐣 SUBSYSTEM* 🔥 LINUX-NEXT 🐧 MAINLINE 🐿 STABLE 🎉 RHEL KERNEL Where we are now * There may be multiple layers of subsystem trees in some areas of the kernel, especially networking. 🤔 DEVELOPER 🛃 🛃 🛃
  24. 24. 🐣 SUBSYSTEM* 🔥 LINUX-NEXT 🐧 MAINLINE 🐿 STABLE 🎉 RHEL KERNEL Where we want to be * There may be multiple layers of subsystem trees in some areas of the kernel, especially networking. 🤔 DEVELOPER 🛃 🛃 🛃 🛃
  25. 25. Time to talk infrastructure
  26. 26. FIRST ITERATION: Jenkins pipeline on a single server Lots of CPU contention from kernel compiles (24 CPUs) We used the same gcc/glibc toolchain for everything Jenkins upgrades were often problematic Constantly reverting plugin upgrades
  27. 27. SECOND ITERATION: Jenkins pipeline with OpenShift Much easier to scale kernel compiles (128 CPUs) Each container had the right compiler toolchains for its compile job Jenkins upgrades and plugin upgrades were still rough Worker containers needed lots of extra Java components to talk to Jenkins
  28. 28. THIRD ITERATION: Self-hosted GitLab with OpenShift Tightly integrated repositories and CI with easy, controlled scaling GitLab runner containers could be stripped down to basic components We added integration testing for the pipeline code itself Easy runner migration to other OpenShift/Kubernetes deployments Access to 768 CPUs
  29. 29. We start testing kernels when certain events occur:
  30. 30. PIPELINE TRIGGER KERNEL REPO Baseline tests Checkout the current HEAD of the kernel tree and trigger GitLab pipelines to compile the kernel and run tests.
  31. 31. PIPELINE TRIGGER PATCHWORK Patchset tests Check for new patches in Patchwork and start testing pipelines in GitLab
  32. 32. PIPELINE TRIGGER STABLE PATCH QUEUE Stable patch queue tests Check for new patches the stable kernel patch queue and start testing pipelines in GitLab
  33. 33. Kernel test pipeline MERGE COMPILE (MULTI-ARCH) NATIVE ARCH BOOT/TESTING LINT (INTERNAL) REPORTING 🤔 MAILING LIST GITLAB RUNNER
  34. 34. Native architecture testing Kernels are booted on their native architecture using Beaker Beaker manages the pool of machines, runs tests, and provides results 44 kernel subsystem tests are open source (more are coming!) We test a few different architectures... Photo: Jonathan Cutrer
  35. 35. Photo: Wikipedia aarch64
  36. 36. ppc64 and ppc64le (IBM POWER) Photo: common.org
  37. 37. s390x (IBM Z) Photo: Wikipedia
  38. 38. x86_64 Photo: Wikipedia
  39. 39. CKI Statistics Over 100K individual tests per month Over 11K GitLab pipelines run Under 6 minutes to compile a kernel for 4 architectures Under 2 hours between patch proposal and feedback sent (stable)
  40. 40. The best part is that we are not alone in this endeavor.
  41. 41. Upstream involvement in automated kernel testing is growing 0day (Intel) CKI Project (Red Hat) Fuego (Sony) kernelci.org (Linux Foundation) Linaro Patchwork syzkaller/syzbot (Google) Yocto project (Linux Foundation)
  42. 42. How can I help?
  43. 43. Read our blog: https://cki-project.org/
  44. 44. Contribute to our tests! https://github.com/cki-project https://gitlab.com/cki-project
  45. 45. Join us at the kernel CI hackfest in Lisbon, Portugal after Linux Plumbers 2019! Photo: Wikipedia
  46. 46. So what’s with this background?
  47. 47. It’s a photo from the Curiosity Rover on Mars.
  48. 48. It’s a photo from the Curiosity Rover on Mars. EARTH
  49. 49. It’s also a metaphor for how much work still remains to be done with kernel testing. HOW MUCH WE’VE DONE SO FAR
  50. 50. Thank you! Major Hayden @majorhayden June 1, 2019 - Texas Linux Fest Photo: David Mark

×