Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Long-term Maintenance Model of Embedded Industrial Linux Distribution

869 views

Published on

To introduce a robust, secure and reliable platform for the industrial environments is a key challenge; moreover, the platform needs to survive for a long time (more than 10+ years). There are many good solutions aiming to meet these requirements, such as LTSI (Long Term Support Initiative) and CIP (Civil Infrastructure Platform). However, it still needs a high amount of maintenance and development costs in handling SoC/ hardware board in-house patch, non-upstream driver and keep source code consistent with different SoC and platform afterwards.
In this presentation, SZ Lin will introduce how to operate long-term maintenance model of embedded industrial Linux distribution. In addition, he will also address the building, deploying and testing architecture and workflow for producing a robust, secure and reliable platform.

Published in: Technology
  • Writing good research paper is quite easy and very difficult simultaneously. It depends on the individual skill set also. You can get help from research paper writing. Check out, please ⇒ www.HelpWriting.net ⇐
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Long-term Maintenance Model of Embedded Industrial Linux Distribution

  1. 1. Long-term Maintenance Model of Embedded Industrial Linux Distribution LINUXCON + CONTAINERCON + CLOUDOPEN CHINA 2017 SZ Lin (林上智) MXcore, Software Supervisor 06/19 2017
  2. 2. Confidential About Me SZ LIN (林上智)  Software Engineer at Moxa • Industrial-grade Linux Distribution  Debian Developer  Blog - https://szlin.me 2
  3. 3. Confidential Industrial/ Harsh Environments Including smart rail, smart grid, intelligent transportation, factory automation, oil & gas, marine, and more Confidential3
  4. 4. Confidential Smart Grid Smart Rail Smart Oil Field Smart Factory Smart Transportation Smart Marine Application Device Industrial Routers Industrial Ethernet Industrial Wireless LAN Network Infrastructure Confidential Embedded Computers Industrial Computing Industrial Linux Platform Serial Connectivity I/O Connectivity Video Connectivity Edge Connectivity 4
  5. 5. Confidential Smart Grid Smart Rail Smart Oil Field Smart Factory Smart Transportation Smart Marine Application Device Industrial Routers Industrial Ethernet Industrial Wireless LAN Network Infrastructure Confidential Embedded Computers Industrial Computing Industrial Linux Platform Serial Connectivity I/O Connectivity Video Connectivity Edge Connectivity 5 HardwareSoftware
  6. 6. Confidential Bug fixes Security fixes New userspace program backports New kernel features backports Industrial Linux Distribution Over 10+ years Maintenance release 6
  7. 7. Confidential Maintenance PhaseDevelopment Phase Hardware Survey Regular Update Kernel/ RFS Preparation 4321 CI/ CD/ LT 7
  8. 8. Confidential Development Phase Take action from the very beginning 8
  9. 9. Confidential Development Phase Hardware Survey Regular Update 4321 CI/ CD/ LT Kernel/ RFS Preparation 9
  10. 10. Confidential Why We Need Hardware Survey Image: https://upload.wikimedia.org/wikipedia/commons/5/5b/Linux_kernel_map.png 10
  11. 11. Confidential Hardware Survey – Key Components Short LifecycleLong Lifecycle (> 10 years) SoC e.g. ARM, MIPS Non-volatile memory e.g. NAND flash, eMMC, SD, SSD DRAMRTC Watchdog PMIC Cellular module Wi-Fi module PHY Switch module UART module WPAN module External LAN controller 11
  12. 12. Confidential Hardware Survey - Long-term Supply Action Pros Cons Build your own hardware components • Long-term support • Create hardware innovation Not cost-effective Sign long-term contract with hardware component provider • Long-term support Follow manufacturers’ regulations Do nothing Choose what you want Massive resource cost 12
  13. 13. Confidential Development Phase Hardware Survey Regular Update 4321 CI/ CD/ LT Kernel/ RFS Preparation 13
  14. 14. Confidential Kernel/ RFS Preparation 1 Choose proper RFS (Root filesystem) Stable root filesystem 2 Unified kernel All SoC is using the same version of kernel Choose proper kernel Long-term maintenance kernel version 3 Apply patches Build custom kernel 4 14
  15. 15. Confidential Choose Proper Kernel Long-term maintenance kernel version 15
  16. 16. Confidential SoC Board Support Package Kernel • Kernel version depends on SoC vendors – Well made but not well maintained • Contain lots of in-house patches – Errata patches – Specific feature patches – … • Different SoC might use different versions of kernel • The lifetime is unsure 16
  17. 17. Confidential LTS: Long Term Stable Kernel [1] Extend software uptime for stable kernel • Only accept bug fixes and security fixes img: https://www.kernel.org/category/releases.html 17
  18. 18. Confidential LTSI: Long Term Support Initiative [2] • Linux Foundation collaborative project – Based on LTS – Add another chance to include further patches on top of LTS – Auto Test framework – Same lifetime with LTS (yearly release and 2 years life time) 18
  19. 19. Confidential CIP (Civil Infrastructure Platform) [3] • Linux Foundation collaborative project – Support kernel and core package – Auto Test framework – Maintenance period • 10 years and more (10-20 years) 19
  20. 20. Confidential img: https://wiki.linuxfoundation.org/civilinfrastructureplatform/cipconferences 20
  21. 21. Confidential Comparison Table Version Maintenance Period (years) Features Latest version Root filesystem Maintainer SoC BSP kernel ? Bug fixes ? N SoC vendor kernel team LTS kernel 2 ~ ? • Bug fixes • Security fixes 4.9 N Kernel.org LTSI kernel LTS + 2 • Bug fixes • Security fixes • Specific features • New features 4.1 N LTSI CIP kernel 10 + • Bug fixes • Security fixes • Specific features • New features 4.4 Y (Project X) CIP 21
  22. 22. Confidential Choose proper RFS (Root filesystem) Stable root filesystem 22
  23. 23. Confidential img: http://lmelinux.net/wp-content/uploads/sites/2/2014/07/linux_distros_2013.jpg 23
  24. 24. Confidential Good system security [5] Everything is open Usually, fixed packages are uploaded within a few days Stability unstable → testing → stable Scalability Server, Desktop, Laptop, Embedded devices Long term support 5 more years by Debian-LTS project (i386, amd64, armel and armhf) Multiple architectures alpha, amd64, armel, armhf, aarch64, hppa, i386, ia64, mips, mipsel, powerpc, s390, and spar Why We Choose Debian [4] Incredible amounts of software Debian comes with over 43000 different pieces of software with free 24
  25. 25. Confidential Unified kernel All SoC is using the same version of kernel 25
  26. 26. Confidential Unified Kernel • Kernel supports multi-platform • Centralized topology – Use the same kernel version • Long-term maintenance – Apply patches through a single entry point • R&D resources focus – Modify Kernel configuration – Edit Device tree – Prepare SoC-related patches 26
  27. 27. Confidential27
  28. 28. Confidential Apply Patches Build custom kernel 28
  29. 29. Confidential29
  30. 30. Confidential30
  31. 31. Confidential31
  32. 32. Confidential
  33. 33. Confidential How to Find Out In-house Patch By Using Your Own Custom Kernel • Get hardware component errata/ datasheet • Get hardware component official driver if possible – It always surprises you • Refer to mainline kernel – It might be fixed already • Time to backport • Discuss the issue in subsystem mailing list or forum – Make the patch 33
  34. 34. Confidential Maintenance Phase Regular update and long-term testing 34
  35. 35. Confidential Maintenance Phase Hardware Survey Regular Update Kernel/ RFS Preparation 4321 CI/ CD/LT 35
  36. 36. Confidential Regular Update Mechanism 1 Well-testing Stable, stable and stable 2 Backported patches Include device drivers and new kernel features Upstream first Do not create in-house patch after fixing bug 3 Minimal modification Apply the patches only what we need 4 Bug fixes New feature patches Security fixes New device drivers Backported patches … 36
  37. 37. Confidential Backported Patches in Maintenance Phase • Try not to modify *ANY API* (if possible…) • Try not rename/ remove any member of structure – “Add” behavior is also risky • Regression test • Unit test • Do not apply lots of commits – Handle it carefully • git cherry-pick • git blame • git bisect • … 37
  38. 38. Confidential Maintenance Phase Hardware Survey Regular Update Kernel/ RFS Preparation 4321 CI/ CD/LT [13] More info: Building, Deploying and Testing an Industrial Linux Platform Open Source Summit Japan 2017 [13] 38
  39. 39. Confidential Industrial Linux Distribution CI CDLT Ecosystems for Industrial Linux Distribution CI : Continuous Integration CD : Continuous Delivery/ Deployment LT : Long-term Test 39
  40. 40. Confidential Master Develop Feature .. Merge Pull Request Fork Local Branch Patches Notification Internal/ External Developers CI/ CD Automatic Release Pipeline Maintainer Approval Pass Pass Y Y N N 40
  41. 41. Confidential CI/ CD Automatic Release Pipeline Building Testing Deploying 4321 Release Continuous Delivery Continuous Deployment Code Continuous Integration 41
  42. 42. Confidential Master Develop Merge Notification CI/ CD Automatic Release Pipeline Maintainer Approval Pass Pass Y Y N N 42 Image Deployment 24/ 7 Long-term Platform Test
  43. 43. Confidential Notification Test Results 43 * Test cases are managed by LAVA
  44. 44. Confidential 24/ 7 Long-term Platform Test Robustness Robustness is the ability of a computer system to cope with errors during execution and cope with erroneous input [12] Reliability Reliability is enhanced by features that help to avoid, detect and repair hardware faults [11] Security Quick response in resolving CVE/ vulnerabilities and attacks in platform Longevity Long-term support at least 10 years life cycle with bug fixes, new features and new hardware components 44 Endurance test Compatibility test …
  45. 45. Confidential 24/ 7 Long-term Platform Test Robustness Robustness is the ability of a computer system to cope with errors during execution and cope with erroneous input [12] Reliability Reliability is enhanced by features that help to avoid, detect and repair hardware faults [11] Security Quick response in resolving CVE/ vulnerabilities and attacks in platform Longevity Long-term support at least 10 years life cycle with bug fixes, new features and new hardware components 45 Fuzz testing [14][15][16] …
  46. 46. Confidential 24/ 7 Long-term Platform Test Reliability Reliability is enhanced by features that help to avoid, detect and repair hardware faults [11] Security Quick response in resolving CVE/ vulnerabilities and attacks in platform 46 Power failure test Reboot test …
  47. 47. Confidential 24/ 7 Long-term Platform Test Security Quick response in resolving CVE/ vulnerabilities and attacks in platform 47 Daily test for CVE [25] …
  48. 48. Confidential Tools Continuous Integration Jenkins [17] Continuous Delivery/ Deployment LAVA 2 [18] Test Case Management • Jenkins • LAVA 2 Version Control Git with gitlab [19] Static Program Analysis • Coding style • OWASP [17] • Infer [20] • Sonarqube [21] Dynamic Program Analysis • Gcov [22] • Valgrind [23] • Profiling tools [24] Security Testing OpenVAS [25] Fuzzing Testing • Syzkaller [14] • Trinity [15] • OSS-fuzz [16] 48
  49. 49. Confidential CI/ CD/ LT are concepts of software engineering instead of tools or procedures 49
  50. 50. Confidential50 Community Collaboration Upstream FirstPreparedness Planning 24/7 Testing Stable, stable and stable Conclusion
  51. 51. Confidential51
  52. 52. © Moxa Inc. All rights reserved. Thank You
  53. 53. Confidential References [1] https://www.kernel.org [2] http://ltsi.linuxfoundation.org [3] https://www.cip-project.org/ [4] https://www.debian.org/intro/why_debian.en.html [5] https://www.debian.org/security/index.en.html [6] http://ck.kolivas.org/patches/ [7] https://www.kernel.org/pub/linux/kernel/projects/rt/ [8] http://elinux.org/images/2/2e/MiniDebianConfJapan- Yoshi.pdf [9] https://openwrt.org/ [10] https://backports.wiki.kernel.org/index.php/Main_Page [11]https://en.wikipedia.org/wiki/Reliability,_availability_and_ serviceability 53
  54. 54. Confidential References [12] https://en.wikipedia.org/wiki/Robustness_(computer_science) [13] http://events.linuxfoundation.org/sites/events/files/slides/Build ing%2C%20Deploying%20and%20Testing%20an%20Industrial%20Linux% 20Platform.pdf [14] https://github.com/google/syzkaller [15] http://codemonkey.org.uk/projects/trinity/ [16] https://github.com/google/oss-fuzz [17] https://wiki.jenkins-ci.org/display/JENKINS/Plugins [18] https://www.linaro.org/initiatives/lava/ [19] https://about.gitlab.com/ [20] http://fbinfer.com/ [21] https://www.sonarqube.org/ [22] https://gcc.gnu.org/onlinedocs/gcc/Gcov.html [23] http://valgrind.org/ [24] https://perf.wiki.kernel.org/index.php/Main_Page 54
  55. 55. Confidential References [25] http://www.openvas.org/ [26] http://events.linuxfoundation.org/sites/events/files/slides/2017-02- 22_CIP-ELC-r7.pdf [27] http://events.linuxfoundation.org/sites/events/files/slides/PRE-trunk- ELCE-LongTermMaintenance.pdf [28] http://events.linuxfoundation.org/sites/events/files/slides/Efficient_ke rnel_backporting.pdf [29] http://events.linuxfoundation.org/sites/events/files/slides/LTSI- ELC2017-final.pdf [30] https://lwn.net/Articles/700530/ [31] Special thanks to Bruce Chen and Roy Hsu 55

×