Download to read offline
![Bill of Materials
on steroids
Reverse indexes
Library → Binaries [SCA tool]
O.S. API → Binaries [SAST tool]
Binary → Pipelines [artifact store]
Pipeline → Repo(s) [pipeline tool]
Pipeline
Binaries
Production
Library
Repo
deploy](https://image.slidesharecdn.com/softwarerotting-devopsconberlin-220622155611-f5031bf4/85/Software-rotting-DevOpsCon-Berlin-22-320.jpg)
![Dependencies
An average .NET project has 11 direct, and 76
indirect dependencies [Source: Snyk]
Project == nuget.org package
The average application contains 118 open-
source libraries [Source: Contrast Security]
Application Java/.NET/NodeJS](https://image.slidesharecdn.com/softwarerotting-devopsconberlin-220622155611-f5031bf4/85/Software-rotting-DevOpsCon-Berlin-30-320.jpg)
![1974
Continuing Change law
«A[n E-type] system
must be continually
adapted or it becomes
progressively less
satisfactory.»
Image source: WikiMedia](https://image.slidesharecdn.com/softwarerotting-devopsconberlin-220622155611-f5031bf4/85/Software-rotting-DevOpsCon-Berlin-39-320.jpg)
Uploaded byGiulio Vian
Software rotting - DevOpsCon Berlin
The document discusses the rapid decay of software and emphasizes the need for improved security approaches due to increasing vulnerabilities and exploitation incidents. It highlights the importance of better tooling, practices, and timely responses to security threats to manage the growing risk in software development. Recommendations include adopting automated patching, thorough regression testing, and standardized source control management practices to enhance security and efficiency.
More Related Content
Similar to Software rotting - DevOpsCon Berlin
More from Giulio Vian
![谷歌留痕技术教程[ 𝙩𝙤𝙥 𝟮𝟯𝟯. 𝙘 𝙤𝙢 ]](https://cdn.slidesharecdn.com/ss_thumbnails/top233-260130173900-2eb784f9-thumbnail.jpg?width=640&height=640&fit=bounds)
Software rotting - DevOpsCon Berlin
- 1. Software rotting Why youneed to change your approach to security… @giulio_vian https://www.getlatestversion.eu http://blog.casavian.eu https://www.slideshare.net/giuliov https://github.com/giuliov Giulio Vian 22 June 2022
- 2. Has any ofthese happened to you?
- 3. I bet so, becauseyou Deploy to Production
- 4.
- 5. Hardware spec: 1 KBRAM 4 KB ROM First computer Past employers Communities Giulio Vian Principal DevOps Engineer @giulio_vian giuliovdev@hotmail.com
- 6. Agenda What you aredoing today Ideas for improvement Is security a real problem?
- 7. What you aredoing today (probably)
- 8. High-level process CVE /Security triggers Developer Teams search Developer Teams fix Release Management deploy
- 9. Affected by Vulnerability Application stack Containerimages Virtual Machine images Application itself Application code Libraries Internal 3rd party Self-contained run-time Application Run-time OS libraries Image Self- contained search
- 10. Find code, easy? Multipleproduction branches release/* and hotfix/* Untagged releases SCA† tools pipeline-bound Rarely built code Pipeline does not work anymore † Software Composition Analysis search
- 11. Identify Vulnerabilities Static ApplicationSecurity Testing (SAST) Software Composition Analysis (SCA) Commercial Synopsys Black Duck, Snyk, WhiteSource Bolt, Sonatype Nexus Platform, JFrog Xray OSS npm audit OWASP Dependency Check Application Run-time OS libraries Ops Dev search
- 12. Fix code Scan multiplerepositories Patch code Regression test Can be automated? fix
- 13. What’s normal? Many teams Manyrepos My company has 3,000 repos across 100 teams, storing over 13 million lines of code, and using 2,800 pipelines A single vulnerability may affect 10s teams and 100s of repos Image: The Crowd For DMB 1 by Moses fix
- 14. Deployment hurdles Separation of Duties Sarbanes–Oxley HIPAA PCIDSS NIST 800-171 Maintenance windows† Image by Tom Staziker deploy
- 15.
- 16. Estimate Risk Probability ofan adverse cyber event Frequency of attack Availability of 0-day exploit Cost factors Number of systems to patch % of Components to build and redeploy Actuaries already have rich models search Image source: WikiMedia
- 17.
- 18. Good SCM Practices Standardize SCMto Git Single management system with rich API Standard naming for Production tags For branches, also Rich metadata via tagging features Repo owners Hotfix pipelines search Image by David Iliff from Wikimedia
- 19. Breadth of change Fix impactingmany systems at once Hundreds of concurrent pipelines Can your build & deploy tool auto-scale? Can your approval process scale? How fast can you rebuild a substantial portion of IT systems? fix Image source: public domain
- 20. Tooling Your SCA maygenerate code changes e.g. dependabot Build scripts can be massively edited git-xargs, auto-pr, multi-gitter, … Image: robotic arm in the Conrad Prebys Center for Chemical Genomics by Josh Baxt fix
- 21. Good patching practices Scan allrepos often Fast-track automated pipelines for all systems Thorough automated regression testing Expedite approval process fix Image source: public domain
- 22. Bill of Materials onsteroids Reverse indexes Library → Binaries [SCA tool] O.S. API → Binaries [SAST tool] Binary → Pipelines [artifact store] Pipeline → Repo(s) [pipeline tool] Pipeline Binaries Production Library Repo deploy
- 23. Redeploy. Every. Day. Simplest pattern Onceautomated patching is in place Zero-downtime deploy in place Consider pipeline resources Image: the gerbil wheel pose by dbgg1979 deploy
- 24. Expedite pipelines Separation of Duties Regulation/ audit requirement Slows 0-day patching Tightly controlled usage Automated checks Single commit with limited churn Additional approvers for quick turnaround Image courtesy of SpaceX deploy
- 25. Is security a realproblem? Image © Mediaset
- 26. Vulnerabilities over year Data:mitre.org
- 27. Zero-days exploits areincreasing Source: Google Project Zero
- 28. Open source dependency& vulnerability Source: Sonatype
- 29. Intermezzo: libraries &languages Source: Contrast Security
- 30. Dependencies An average .NETproject has 11 direct, and 76 indirect dependencies [Source: Snyk] Project == nuget.org package The average application contains 118 open- source libraries [Source: Contrast Security] Application Java/.NET/NodeJS
- 31. Open source JavaScriptvulnerabilities Source: Sonatype
- 32. Could be worse? Source:Snyk Source: Sonatype Mean Time to Update
- 33. Docker as anhidden dependency Source: Snyk
- 34. App Platform shift Chrome1 month patched after 14 days Node.JS 30 months (LTS) patched every 25 days 6 months Go 6 months patched every 26 days Two major releases supported. MongoDB 30 months patched every 5 weeks .NET 3 years (LTS) patched every 6 weeks 18 months Java 3 years (LTS) patched every 12 weeks 6 months
- 35. Wrapping-up Image: Three Stagesof Decay by Theen Moy
- 36. Value does nottell the whole story Business Feature Package Production Value
- 37. Consider Stop-Loss CVE SecurityPatch Production Stop-loss Image: Wikipedia Business Feature Package Production Value
- 38. Technical Inflation Unintended reduction in valueof a software product over time, independent of source code changes. Depreciation does not capture two elements: Unintentionality Value can be restored Image source: Max Pixel
- 39. 1974 Continuing Change law «A[nE-type] system must be continually adapted or it becomes progressively less satisfactory.» Image source: WikiMedia
- 40. Executive Summary Software decays rapidly, anddecay rate is speeding up. Security is the main force, but not the only one. We must improve tooling and practices to cope with this increased velocity. Technical Inflation helps Management understand what is going on. Image source: Public Domain
- 41. Questions? Next five slideslists bibliographic references
- 42.
- 43. References (2/5) https://heartbleed.com/ Why EveryBusiness Is a Software Business — Watts S. Humphrey Informit, Feb 22, 2002 http://www.informit.com/articles/article.aspx?p=25491 https://en.wikipedia.org/wiki/Watts_Humphrey https://www.sonatype.com/resources/state-of-the-software-supply-chain-2021 https://www.shopify.com/enterprise/global-ecommerce-statistics https://blog.cloudflare.com/popular-domains-year-in-review-2021/ https://radar.cloudflare.com/year-in-review-2021 https://snyk.io/blog/net-open-source-security-insights/ https://www.contrastsecurity.com/the-state-of-the-oss-report-2021 https://octoverse.github.com/static/github-octoverse-2020-security-report.pdf
- 44.
- 45.
- 46.
- 47.
Editor's Notes
- #3 …except that the next day, a new vulnerability has reached prime time, and … yes, Joe app is impacted! How often this happened to you? Is it happening more frequenty?
- #4 Software is not a problem if not deployed. The most secure computer is off and unplugged.
- #6 Who am I? I work at Unum, a Fortune 500 company, with more than a thousand people in IT. I studied DevOps for over 10 years and I speak at international conferences. Awarded by Microsoft as Most Valuable Professional on Azure DevOps category in the last few years. If you want to discuss today’s ideas or other DevOps topics you can reach me at Twitter as giulio_vian or email me directly. While I try to solve a new problem each day, some issues take years to go away.
- #9 How we run the process today? Publication of a CVE triggers the Security team in the organization, Security team instructs Dev Teams to fix application code as needed, code must be deployed to Production under Release Management team supervision A Release Management role may be required by SOX, Basilea, and similar regulation Deploy where? Production! We don’t care about the rest (although…), so we need to…
- #10 Joe stops and thinks: I need to look at my pom.xml (build.gradle, *.csproj, Makefile, package.json, … name it) for references to Log4J (or whatever is vulnerable). Oh, but I use SLF4J which in turns… indirect dependencies! I need a tool just to find all possible references recursively. Oh Oh, our Tomcat configuration is using Log4J! I must check more than my JAR file, says Joe. …and the next question is…
- #11 Here we discuss how to identify: 1. the code that needs to be patched 2. the pipeline that release that code in Production and some issues that one may face: If more than one branch can reach prod, which one you choose? How do you match the exact version of code? Software Composition Analysis kicks in only through pipelines? Is triggered by the deploy pipeline? The deploy pipeline hasn’t been used in months and doesn’t work anymore (e.g. a token expired, or there is no more an apt agent)
- #12 …are there tools to support me and detect vulnerabilities in the code I deliver? Yes, there are BLAH
- #13 The vulnerability could be a bad code pattern, use of an API, a vulnerable dependency; in any case we need to find the impacted code. We must scan all repositories that contain production code. Non-production repositories should be included in the search but listed separately to remove noise. Some patching can be easily automated, in particular library dependencies listed in project file (e.g. package.json, pom.xml,.csproj, …)
- #15 † Lack of blue/green, canary, rolling/progressive deployment
- #17 When I’ll be next We had billions of attacks!? Mostly intercepted email, though, one day it will be successful Ok but what might be the consequences of an attack?
- #19 First batch of crucial elements of a solution identifying the qualities required on Software Configuration Management (SCM). Moving to Git is a prerequisite because any modern development tool shifted to it. What about the rest? Consolidate all projects into a single SCM platform – GitHub, Azure DevOps, BitBucket, GitLab The one listed have rich API that enable automation and integration Recommend set of branch names and mandatory tag names to identify code that goes/matches production This does not translate that all teams use the same process (e.g. GitFlow) but that they use the same conventions for branch names and, especially for tags Modern systems allows to attach custom properties to Git repositories and Pipeline definitions A Tag/Label/Property can identify repo in use, archived, or just with experimental code Same for pipelines, there are builds with limited scope (quick CI) and builds that deploy to production: use Tag/Label/Property to distinguish
- #20 As mentioned, on a small scale, it is easy. Problems raise when you need to manage at scale: more than a few teams, repos, or pipeline. Consider the scenario where a single vulnerability impacts most of your applications (which is probable when you the majority of you code use the same platform, e.g. Log4J impacting all Java-based applications). You need to patch lots of repositories and deploy lots of components, each through a separate pipeline. In such scenario, you need new capabilities: Global editing tool Launch most pipelines in parallel (consider batching) Auto-scale build resources to sustain the spike Single-approval for the set of pipeline runs These aren’t offered by current systems.
- #21 Can be automated? <pause> To my knowledge there are some tools that do some of the work, like GitHub dependabot It scans sources and proposes changes via a pull-request mechanism It does not support all package manager, though, and some features requires GitHub And clearly we need to input which is the correct version to use. We have seen toolchain attacks were the fix was to rollback, haven’t we?
- #22 A crucial pattern to implement is the fast-track (expedite) pipeline. Every deployable component must have a pipeline that delivers just security fixes with as much regression testing as possible within a limited 2-hours timebox and as little manual steps (e.g. approvals) as possible. Tip: maybe you need some kind of incremental build to minimize build, test and deploy times.
- #23 Current tooling may offer some information but a well-rounded process lot of cross-reference data. Dependency management is a weak spot in general, SCA (Software Composition Analysis) can identify vulnerabilities in libraries. Use of API may be caught by security scans Artifact management tool can track the source (build) of binaries if properly used. Pipeline knows which repositories they use, what we need here is ability to call a REST API that tell us the dependency. If you can use such tools, great. Maybe you need to follow a bit of conventions and write some query tools. In the worst scenario, you have to build and maintain your own database.
- #25 Deploy where? Production! We don’t care about the rest (although…), so we need to… A Release Management role may be required by SOX, Basilea, and similar regulation But you need speed when it is a 0-day exploit. For example, you must be able to deploy a patch within hours of its release from a 3rd party (an OSS project or a vendor). fast-track (expedite) pipelines are not for normal usage: there should be some kind of trigger, like a new CVE, a communication from the Security team or upper management.
- #26 What is the way to solve this burning problem?
- #27 …they are not decreasing, quite the opposite. Increasing more than linearly!
- #29 …display the same pattern, even more. Why?
- #32 Modern app development is not using just a few suppliers like in the past. A study from Sonatype confirms that we use a lot more libraries, from all kind of sources, in particular Open Source libraries and attacks leverage this trend. The graphs illustrate Javascript scenario, but other languages…
- #33 Both graphs illustrate that we, as an industry, aren’t exactly great at reacting and fixing our applications. The one on the left is data about OSS projects. The one on the right is more interesting because based on telemetry data, a more significant insight on IT organizations.
- #35 .NET Core 3.1 3.1.0 December 3, 2019 3.1.22 December 14, 2021 got 22 patch releases in 3 years i.e. every 45 days/6 weeks Node v14 (Fermium) Active LTS start 2020-10-27 v14.15.0 2022-02-01, Version 14.19.0 total 19 releases in 463 days or 66 weeks i.e. every 24.4 days JDK 11 Java SE 11 (LTS)September 25, 2018 11.0.13+8 (GA), October 19th 2021 total 13 releases(updates) in 1121 days i.e. every 12.3 weeks or 86.2 days Go 1.16 released 2021-02-16 go1.16.14 (released 2022-02-10) total 14 updates in 360 days i.e. 26 days go1 (released 2012-03-28) -> go1.17 (released 2021-08-16) 17 major releases in 3429 days or 490 weeks MongoDB 5.0 5.0.0 - Jul 13, 2021 5.0.6 - January 31, 2022 total 6 releases in 203 days or 29 weeks i.e. every 4.8 weeks
- #36 What is the way to solve this burning problem?
- #37 Agile and DevOps focused on value-flow
- #40 «An E-program is written to perform some real-world activity; how it should behave is strongly linked to the environment in which it runs, and such a program needs to adapt to varying requirements and circumstances in that environment» “On understanding laws, evolution, and conservation in the large-program life cycle” Lehman M.M. - Journal of Systems and Software Vol. 1, 1979–1980, pp. 213-221
- #41 Today, I hope to convince you that we have serious problems in the way we patch and deploy applications, problems that we must address as an industry. At the core a perfectly working application today, is a huge risk tomorrow. That’s why I speak of decay and rotting, because it is not a slow process. Wear, erosion, rust… They do not convey the urgency and work required to preserve from decay. #1 unless you put it in a fridge or in a can, it starts smelling very soon #2 those other processes requires time, while rotting requires quick action to stop it I am not sure big an effort is to fix processes and tool to cope with security-related problems – the one this audience is acquainted to --. Security is the main driver, although not the only one. To change process and invest in tools, we have to speak to leadership/executive using a simple but effective vocabulary, so I suggest using the word inflation to convey the idea and start a discussion. As you may guessed, this presentation is a bit visionary, high-level, I will talk about industry trends and process not technology. For those interested in technology details, I recommend the sessions of my friends Michael Kaufmann and Matteo Emili. Now you have a couple of minutes to switch if you are not interested.
- #42 I am open for questions.