This document discusses managing kernel vulnerabilities in the software development lifecycle. It covers choosing proper Linux kernel versions from trusted sources, maintaining kernels through upstream first methodology, hardening configurations, automated testing, vulnerability scanning, and community collaboration on security issues. The goal is to minimize risks and costs by addressing vulnerabilities early through a defined process.

1. Manage Kernel Vulnerabilities in
the Software Development Lifecycle
林上智 (SZ LIN)
szlin@debian.org
Date: 12th August, 2020

2. > 80 %
> 75 % 100 %
> 95 %
img src: https://kernel.org
src: https://www.linuxfoundation.org/about/
of the top one
million domains run
with Linux
of cloud-enabled
enterprises report
using Linux as their
primary cloud
platform
of new smartphones
sold run Android,
which is based on
the Linux kernel
of the top 500
supercomputers in
the world run on
Linux

3. Before Using Linux Kernel
Something you should know

4. 4
Copyright Patent
A patent gives its owner the right to
exclude others from making, using,
selling, and importing an invention
for a limited period of time, usually
twenty years.
src: https://en.wikipedia.org/wiki/Patent
Copyright is a legal right, that grants
the creator of an original work
exclusive rights to determine whether,
and under what conditions, this
original work may be used by others
src: https://en.wikipedia.org/wiki/Copyright

5. 5
Copyright Patent
A patent gives its owner the right to
exclude others from making, using,
selling, and importing an invention
for a limited period of time, usually
twenty years.
src: https://en.wikipedia.org/wiki/Patent
Copyright is a legal right, that grants
the creator of an original work
exclusive rights to determine whether,
and under what conditions, this
original work may be used by others
src: https://en.wikipedia.org/wiki/Copyright

6. Context
6
1400+
Members From
41 Countries
80%
of Fortune 100
Tech & Telecom
35,000+
Developers
Contributing Code
170+
Open Source
Projects
$16B
Shared
Value
Linux Foundation

7. 7
The OpenChain Project defines the key requirements of a
quality open source compliance program [1].

9. src:https://www.iso.org/standard/81039.html

10. 10
User Applications
GNU/ Linux
GNU C library
Init system
User
Space
Kernel
Space
Hardware and peripheral devices
Architecture-dependent firmware
Bootloader
Kernel
System call interface
Toolchain
Root filesystem
More info: Using open source software to
build an industrial-grade embedded Linux
platform from scratch
Open Source Summit Japan, 2019 [57]

11. End of LTS
11
Linux Kernel Releases
Mainline
Stable
(linux-stable-4.4)
v4.4
Stable
(linux-stable-4.19)
6+? years
v4.5 v4.19 v5.x
EOL
v4.4.x v4.4.y v4.4.z
v4.19.a v4.19.b
img src: https://en.wikipedia.org/wiki/Linux_kernel_version_history
End of LTS
6+? years

12. 27.8
60-90 Day 66,492 3,386,347
21,074
Mainline Kernel
Release Cycle
Million Lines Files Lines of New Codes
in 2019
Different Authors
12
src: https://www.phoronix.com/scan.php?page=news_item&px=Linux-Git-Stats-EOY2019
img src: https://kernel.org

13. Supply Chain Risk Management Practices
for Federal Information Systems and
Organizations
Special Publication 800-161 [4]
SM-9: Security requirements for
externally provided components
ISA/ IEC 62443-4-1 [5] NERCCIP-010-2 [6]
Configuration Change Management and
Vulnerability Assessments
img src: https://pixabay.com/illustrations/policies-standards-compliance-4720824/
13

14. src: https://www.ithome.com.tw/news/138633
2020-07-07發表
14

15. How to Manage
Vulnerabilities in
Linux Kernel?
15

16. Costs to Fix Software Defects at Different Stages of SDLC [7]
1x
5x
10x
15x
30x
0
5
10
15
20
25
30
35
Requirements Gathering
and Analysis/
Architectural Design
Coding/Unit Test Integration and
Component/RAISE
System Test
Early Customer
Feedback/Beta Test
Programs
Post-product
Release
16
X is a normalized unit of cost and can be expressed terms of person-hours, dollars, etc.

17. SDLC
Software
Development
Life Cycle
Requirement
Analysis
Design
Implementation
Testing
Maintenance
/ Evolution
17

18. SDLC
Software
Development
Life Cycle
Requirement
Analysis
Design
Implementation
Testing
Maintenance
/ Evolution
18

19. Scope
Schedule
Resources
Good enough
principle
KISS principle
Core technology
identification
Requirements Analysis
It’s imperative to collect, analyze, identify requirements for Linux
kernel and its configuration, it also reduces the unnecessary
maintenance effort related to security. Moreover, it provides
information for us to choose proper kernel source to fulfill our
requirements.
19

20. Requirements for the Civil Infrastructure Systems [8]
Industrial
Grade
• Reliability
• Functional Safety
• Security
• Real-time capabilities
Sustainability
Security
• Security &
vunerability
managment
• Firmware updates
• Minimize risk of
regressions
This has to be achieve with …
Development time
 Shorter development times for more complex system
Maintenance costs
 Low maintenance costs for commonly uses software
components
 Low commissioning and update costs
Development costs
 Don‘t re-invent the wheel
• Product life-cycles
of 10 – 60 years
20

21. SDLC
Software
Development
Life Cycle
Requirement
Analysis
Design
Implementation
Testing
Maintenance
/ Evolution
21

22. Choose Proper Linux Kernel
only from trusted sites
22

23. Category
Latest
version
Target Application Maintainer
Linux kernel 5.8
• Performance
• Resource Limited [9] [10]
Kernel.org
Preempt RT kernel 5.6
• Real-time
• Functional safety
• Resource Limited
Real Time Linux
collaborative project
*Real-time application [11][12]
23
*Grsecurity [13]

24. SoC Board Support Package Kernel
• Kernel version depends on SoC vendors
– Well made but not well maintained
• Contain lots of in-house patches
– Errata patches
– Specific feature patches
– …
• Different SoC might use different versions of kernel
• The lifetime is unsure
24

25. LTS: Long Term Stable Kernel [3]
Extend software uptime for stable kernel
• Only accept bug fixes and security fixes
img: https://www.kernel.org/category/releases.html
Retrieved 7th August
25

26. LTSI: Long Term Support Initiative [14]
• Linux Foundation collaborative project
– Based on LTS
– Add another chance to include further patches on top of LTS
– Auto Test framework
– Same lifetime with LTS (yearly release and 2 years life time)
26

27. CIP (Civil Infrastructure Platform) [16]
• Linux Foundation collaborative project
– Support kernel and core package
– Auto Test framework
– Maintenance period
• 10 years and more (10-20 years)
27
More info: CIP Kernel Team Activities to Accomplish Super Long Term Support
Embedded Linux Conference, 2020 [17]

28. CIP SLTS Kernel Releases
Mainline
Stable (linux-stable-4.4)
4.4
CIP SLTS 4.4 (linux-4.4.y-cip)
CIP SLTS 4.19 (linux-4.19.y-cip)
End of LTS
Stable (linux-stable-4.19)
Maintained by CIP
Maintained by
CIP Kernel
Maintainers
4.19
10 years
6 years
4 years
End of CIP SLTS
5.x
10 years
6 years
4 years
28

29. Speed and Efficiency : focus on differentiating parts
29

30. Linux Kernel Source Comparison Table
Version
Maintenance
Period
(years)
Features
Latest
Version
Supported Real-
time kernel
Maintainer
SoC
BSP kernel
? Bug fixes ? N SoC vendor kernel team
LTS
kernel
2 ~ ?
• Bug fixes
• Security fixes
5.4 N Kernel.org
LTSI kernel 2 ~ ?
• Bug fixes
• Security fixes
• Specific features
• New features
4.14 N
LTSI
(Linux Foundation Projects)
CIP
kernel
10 +
• Bug fixes
• Security fixes
• Specific features
• New features
4.19 Y
CIP
(Linux Foundation Projects)
30

31. ELISA: Safety-Critical Systems [17]
• Linux Foundation collaborative project
– Build and certify Linux-based safety-critical applications
– Define and maintain a common set of tools and processes
• SIL2LinuxMP [18] project and the Linux Foundation’s Real-Time Linux project
– IEC 61508
31

32. Year 2038 Problem [19][20]
• The time_t datatype is a data type in the ISO C library and
kernel structure defined for storing system time values.
• 32-bit system can represent dates from
 Dec 13 1901
 Jan 19th 2038
• It causes integer overflowing on
– 03:14:08 UTC 19 January 2038
32

33. Don’t choose rolling version
unless necessary
v4.4.1
Security fixes
Bug fixes
Upstream
rolling version
33
v4.4.2 v4.4.3

34. SDLC
Software
Development
Life Cycle
Requirement
Analysis
Design
Implementation
Testing
Maintenance
/ Evolution
34

35. Upstream First
35

36. Kernel inside the organization
Upstream
36
v4.4.1

37. Kernel inside the organization
In-house security or
bug patches
Upstream
37
v4.4.1
v4.4.1

38. Security fixes
Kernel inside the organization
Bug fixes
Upstream
38
v4.4.1 v4.4.2
v4.4.1
In-house security or
bug patches

39. Security fixes
Kernel inside the organization
In-house security or
bug patches
Bug fixes
Upstream
39
v4.4.1 v4.4.2
v4.4.1 v4.4.2

40. Security fixes
Kernel inside the organization
In-house security or
bug patches
Bug fixes
Upstream
40
v4.4.1 v4.4.2
Security fixes
v4.4.3
v4.4.1 v4.4.2 v4.4.3

41. Security fixes
Kernel inside the organization
In-house security or
bug patches
Bug fixes
Upstream
41
v4.4.1 v4.4.2
Security fixes
v4.4.3
v4.4.1 v4.4.2 v4.4.3

42. Security fixes
Kernel inside the organization
In-house security or
bug patches
Bug fixes
Upstream
42
v4.4.1 v4.4.2
Security fixes
v4.4.3
v4.4.1 v4.4.2 v4.4.3

43. • The project shares its results with the upstream
• The project fulfills longer time maintenance and
security fixes
• The project develops their code very quickly
• The project faces difficulties to backport upstream
patches due to conflicts as time goes by
43

44. Kernel Hardening –
Configuration Optimization
Secure the system by reducing its attack surface
44

45. 45

46. 46

47. 47

48. 48

49. 49

50. SDLC
Software
Development
Life Cycle
Requirement
Analysis
Design
Implementation
Testing
Maintenance
/ Evolution
50

51. For Stable Kernel Maintenance
• Automated Linux Kernel Testing [22][23]
– Detect, bisect, report and fix regressions on
upstream Kernel trees before release
– Short tests on many configurations
51
img src: https://kernelci.org/

52. img src: https://kernelci.org/
52

53. 53

54. Reproducible Builds [25]
• Create an independently-verifiable path from source to
binary
– Ensure builds have identical results
– Act as part of a chain of trust
– Prove the source code has not been tampered/modified
54

55. Continuous Integration • Jenkins [26]
• Jenkins X [27]
Continuous Delivery/ Deployment • LAVA 2 [28]
Distributed compiler service • icecc [29]
• GOMA [30][31]
• distcc [32]
Test Case Management • Jenkins
• LAVA 2
Version Control • Git with gitlab [33]
Static Program Analysis • checkpatch.pl [34]
• sparse [35][36]
• smatch [37]
Dynamic Program Analysis • Profiling tools [38]
Vulnerability Scanning • OpenVAS [39]
• Vuls [40]
Fuzzing Testing • Syzkaller [41]
• Trinity [42]
• perf_fuzzer [43]
More info:
Building, Deploying and Testing an
Industrial Linux Platform
Open Source Summit Japan 2017 [44]
55

56. SDLC
Software
Development
Life Cycle
Requirement
Analysis
Design
Implementation
Testing
Maintenance
/ Evolution
56

57. 0
400
800
1200
v5.4
v4.19
v4.14
v4.9
v4.4
Commit Counts per Month
Note: If a patch has an original patch, the date of the patch is that of the original one.
57

58. v4.19
v4.4
58
v4.9
v4.14

59. • cve-search [45]
• nvdtools [46]
• Distribution CVE tracker
• National vulnerability database [47]
• Upstream issue tracker or forum
Vulnerability Scanning – Component Level
59

60. Vulnerability Scanning – System Level
Security
Quick response in
resolving CVE/
vulnerabilities and
attacks in platform
Daily test for CVE
…
60

61. Vulnerability Management Framework
Dependency-Track [49]
SW360 [48]
61

62. Vulnerability Scanning – Source Code Level
62

63. • This project tracks the status of security issues, identified by CVE ID, in
mainline, stable, and other configured branches.
Introduction to "cip-kernel-sec”
63

64. Issue Format - YAML
64

65. Show via Web I/F
Mainline/LTS
cip-kernel-sec
Webview Command line view
Gather CVE Information for Kernel
Show via Command Line
65

66. cip-kernel-sec Web View
66

67. Linux Kernel Vulnerabilities = Bugs != CVEs
67

68. src: https://kernel-recipes.org/en/2019/talks/cves-are-dead-long-live-the-cve/
68

69. 69
src: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-19076

70. 70
src: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-19076

71. Community Collaboration
Different approach for
multiple target applications
Preparedness Planning
Testing and
well-maintenance
Conclusion

72. 72
Thank you

73. [1] https://www.openchainproject.org/
[2] https://www.iso.org/standard/81039.html
[3] https://www.kernel.org/
[4] https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-161.pdf
[5] https://webstore.iec.ch/preview/info_iec62443-4-1%7Bed1.0%7Den.pdf
[6] https://www.nerc.com/pa/Stand/Reliability%20Standards/CIP-010-2.pdf
[7] https://www.nist.gov/system/files/documents/director/planning/report02-3.pdf
[8] Industrial-grade Open Source Base Layer Development, Yoshitake Kobayashi, Urs
Gleim.
References

74. [9] https://tiny.wiki.kernel.org/start
[10] https://bootlin.com/pub/conferences/2017/jdll/opdenacker-embedded-linux-in-less-
than-4mb-of-ram/opdenacker-embedded-linux-in-less-than-4mb-of-ram.pdf
[11] https://xenomai.org/
[12] https://www.rtai.org/
[13] https://grsecurity.net/
[14] https://ltsi.linuxfoundation.org/
[15] https://events.linuxfoundation.org/wp-content/uploads/2017/11/Using-Linux-for-
Long-Term-Community-Status-and-the-Way-We-Go-OSS-Tsugikazu-Shibata.pdf
References

75. [16] https://www.cip-project.org/
[17] https://static.sched.com/hosted_files/ossna2020/d0/OSSNA2020-CIPKernelTeam-2.pdf
[17] https://elisa.tech/
[18] http://www.osadl.org/SIL2LinuxMP.sil2-linux-project.0.html
[19] http://elinux.org/images/6/6e/End_of_Time_--_Embedded_Linux_Conference_2015.pdf
[20] https://en.wikipedia.org/wiki/Year_2038_problem
[21] www.cvedetails.com/vulnerability-
list.php?vendor_id=33&product_id=47&version_id=261041&page=1&hasexp=0&opdos=0&ope
c=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&op
byp=0&opfileinc=0&opginf=0&cvssscoremin=0&cvssscoremax=0&year=2019&month=0&cweid
=0&order=3&trc=72&sha=53735ab937bcf3686d34f3999d8e47f304466007
References

76. [22] https://kernelci.org/
[23]
https://fosdem.org/2019/schedule/event/kernelci_a_new_dawn/attachments/slides/3300/export/even
ts/attachments/kernelci_a_new_dawn/slides/3300/gtucker_kernelci_fosdem_2019_v2_3_1024x768.pd
f
[24] https://kernelci.org/build/stable/branch/linux-4.19.y/kernel/v4.19.138/
[25] https://reproducible-builds.org/
[26] https://jenkins.io
[27] https://jenkins.io/projects/jenkins-x/
[28] https://validation.linaro.org/static/docs/v2/#
[29] https://github.com/icecc
[30] https://chromium.googlesource.com/infra/goma/server/
[31] https://chromium.googlesource.com/infra/goma/client
[32] https://github.com/distcc/distcc
[33] https://about.gitlab.com/
References

77. [34]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/scripts/checkpatch.p
l
[35] http://sparse.wiki.kernel.org/
[36] https://git.kernel.org/pub/scm/devel/sparse/sparse.git
[37] http://smatch.sourceforge.net/
[38] https://perf.wiki.kernel.org/index.php/Main_Page
[39] http://www.openvas.org/
[40] https://vuls.io/
[41] https://github.com/google/syzkaller
[42] http://codemonkey.org.uk/projects/trinity/
[43] http://web.eece.maine.edu/~vweaver/projects/perf_events/fuzzer/
References

78. [44] http://events.linuxfoundation.org/sites/events/files/slides/Build
ing%2C%20Deploying%20and%20Testing%20an%20Industrial%20Linux%
20Platform.pdf
[45] https://github.com/cve-search/cve-search
[46] https://github.com/facebookincubator/nvdtools
[47] https://nvd.nist.gov/
[48] https://www.eclipse.org/sw360/
[49] https://dependencytrack.org/
[50] https://www.cvedetails.com/version/261041/Linux-Linux-Kernel-4.19.html
[51] https://www.cvedetails.com/version/230587/Linux-Linux-Kernel-4.14.html
[52] https://www.cvedetails.com/version/205966/Linux-Linux-Kernel-4.9.html
[53] https://www.cvedetails.com/version/190796/Linux-Linux-Kernel-4.4.html
[54] https://gitlab.com/cip-project/cip-kernel/cip-kernel-sec
References

79. [55] https://icss20.sched.com/event/ZjMw/managing-vulnerabilities-in-open-source-
components-in-ics
[56] https://lore.kernel.org/lkml/20191204103955.63c4d9af@cakuba.netronome.com/
[57] https://ossalsjp19.sched.com/event/OVsf/using-open-source-software-to-build-an-
industrial-grade-embedded-linux-platform-from-scratch-sz-lin-moxa
References