Software rotting - DevOpsCon Berlin
•Download as PPTX, PDF•
0 likes•21 views
Title: Software rotting or why you need to change your approach to security DevOpsCon Berlin 22 Jun 2022
Recommended
More Related Content
Similar to Software rotting - DevOpsCon Berlin
Similar to Software rotting - DevOpsCon Berlin (20)
More from Giulio Vian
More from Giulio Vian (20)
Recently uploaded
Recently uploaded (20)
Software rotting - DevOpsCon Berlin
- 1. Software rotting Why you need to change your approach to security… @giulio_vian https://www.getlatestversion.eu http://blog.casavian.eu https://www.slideshare.net/giuliov https://github.com/giuliov Giulio Vian 22 June 2022
- 2. Has any of these happened to you?
- 3. I bet so, because you Deploy to Production
- 5. Hardware spec: 1 KB RAM 4 KB ROM First computer Past employers Communities Giulio Vian Principal DevOps Engineer @giulio_vian giuliovdev@hotmail.com
- 6. Agenda What you are doing today Ideas for improvement Is security a real problem?
- 7. What you are doing today (probably)
- 8. High-level process CVE / Security triggers Developer Teams search Developer Teams fix Release Management deploy
- 9. Affected by Vulnerability Application stack Container images Virtual Machine images Application itself Application code Libraries Internal 3rd party Self-contained run-time Application Run-time OS libraries Image Self- contained search
- 10. Find code, easy? Multiple production branches release/* and hotfix/* Untagged releases SCA† tools pipeline-bound Rarely built code Pipeline does not work anymore † Software Composition Analysis search
- 11. Identify Vulnerabilities Static Application Security Testing (SAST) Software Composition Analysis (SCA) Commercial Synopsys Black Duck, Snyk, WhiteSource Bolt, Sonatype Nexus Platform, JFrog Xray OSS npm audit OWASP Dependency Check Application Run-time OS libraries Ops Dev search
- 12. Fix code Scan multiple repositories Patch code Regression test Can be automated? fix
- 13. What’s normal? Many teams Many repos My company has 3,000 repos across 100 teams, storing over 13 million lines of code, and using 2,800 pipelines A single vulnerability may affect 10s teams and 100s of repos Image: The Crowd For DMB 1 by Moses fix
- 14. Deployment hurdles Separation of Duties Sarbanes–Oxley HIPAA PCI DSS NIST 800-171 Maintenance windows† Image by Tom Staziker deploy
- 16. Estimate Risk Probability of an adverse cyber event Frequency of attack Availability of 0-day exploit Cost factors Number of systems to patch % of Components to build and redeploy Actuaries already have rich models search Image source: WikiMedia
- 18. Good SCM Practices Standardize SCM to Git Single management system with rich API Standard naming for Production tags For branches, also Rich metadata via tagging features Repo owners Hotfix pipelines search Image by David Iliff from Wikimedia
- 19. Breadth of change Fix impacting many systems at once Hundreds of concurrent pipelines Can your build & deploy tool auto-scale? Can your approval process scale? How fast can you rebuild a substantial portion of IT systems? fix Image source: public domain
- 20. Tooling Your SCA may generate code changes e.g. dependabot Build scripts can be massively edited git-xargs, auto-pr, multi-gitter, … Image: robotic arm in the Conrad Prebys Center for Chemical Genomics by Josh Baxt fix
- 21. Good patching practices Scan all repos often Fast-track automated pipelines for all systems Thorough automated regression testing Expedite approval process fix Image source: public domain
- 22. Bill of Materials on steroids Reverse indexes Library → Binaries [SCA tool] O.S. API → Binaries [SAST tool] Binary → Pipelines [artifact store] Pipeline → Repo(s) [pipeline tool] Pipeline Binaries Production Library Repo deploy
- 23. Redeploy. Every. Day. Simplest pattern Once automated patching is in place Zero-downtime deploy in place Consider pipeline resources Image: the gerbil wheel pose by dbgg1979 deploy
- 24. Expedite pipelines Separation of Duties Regulation / audit requirement Slows 0-day patching Tightly controlled usage Automated checks Single commit with limited churn Additional approvers for quick turnaround Image courtesy of SpaceX deploy
- 25. Is security a real problem? Image © Mediaset
- 26. Vulnerabilities over year Data: mitre.org
- 27. Zero-days exploits are increasing Source: Google Project Zero
- 28. Open source dependency & vulnerability Source: Sonatype
- 29. Intermezzo: libraries & languages Source: Contrast Security
- 30. Dependencies An average .NET project has 11 direct, and 76 indirect dependencies [Source: Snyk] Project == nuget.org package The average application contains 118 open- source libraries [Source: Contrast Security] Application Java/.NET/NodeJS
- 31. Open source JavaScript vulnerabilities Source: Sonatype
- 32. Could be worse? Source: Snyk Source: Sonatype Mean Time to Update
- 33. Docker as an hidden dependency Source: Snyk
- 34. App Platform shift Chrome 1 month patched after 14 days Node.JS 30 months (LTS) patched every 25 days 6 months Go 6 months patched every 26 days Two major releases supported. MongoDB 30 months patched every 5 weeks .NET 3 years (LTS) patched every 6 weeks 18 months Java 3 years (LTS) patched every 12 weeks 6 months
- 35. Wrapping-up Image: Three Stages of Decay by Theen Moy
- 36. Value does not tell the whole story Business Feature Package Production Value
- 37. Consider Stop-Loss CVE Security Patch Production Stop-loss Image: Wikipedia Business Feature Package Production Value
- 38. Technical Inflation Unintended reduction in value of a software product over time, independent of source code changes. Depreciation does not capture two elements: Unintentionality Value can be restored Image source: Max Pixel
- 39. 1974 Continuing Change law «A[n E-type] system must be continually adapted or it becomes progressively less satisfactory.» Image source: WikiMedia
- 40. Executive Summary Software decays rapidly, and decay rate is speeding up. Security is the main force, but not the only one. We must improve tooling and practices to cope with this increased velocity. Technical Inflation helps Management understand what is going on. Image source: Public Domain
- 41. Questions? Next five slides lists bibliographic references
- 43. References (2/5) https://heartbleed.com/ Why Every Business Is a Software Business — Watts S. Humphrey Informit, Feb 22, 2002 http://www.informit.com/articles/article.aspx?p=25491 https://en.wikipedia.org/wiki/Watts_Humphrey https://www.sonatype.com/resources/state-of-the-software-supply-chain-2021 https://www.shopify.com/enterprise/global-ecommerce-statistics https://blog.cloudflare.com/popular-domains-year-in-review-2021/ https://radar.cloudflare.com/year-in-review-2021 https://snyk.io/blog/net-open-source-security-insights/ https://www.contrastsecurity.com/the-state-of-the-oss-report-2021 https://octoverse.github.com/static/github-octoverse-2020-security-report.pdf
Editor's Notes
- …except that the next day, a new vulnerability has reached prime time, and … yes, Joe app is impacted! How often this happened to you? Is it happening more frequenty?
- Software is not a problem if not deployed. The most secure computer is off and unplugged.
- Who am I? I work at Unum, a Fortune 500 company, with more than a thousand people in IT. I studied DevOps for over 10 years and I speak at international conferences. Awarded by Microsoft as Most Valuable Professional on Azure DevOps category in the last few years. If you want to discuss today’s ideas or other DevOps topics you can reach me at Twitter as giulio_vian or email me directly. While I try to solve a new problem each day, some issues take years to go away.
- How we run the process today? Publication of a CVE triggers the Security team in the organization, Security team instructs Dev Teams to fix application code as needed, code must be deployed to Production under Release Management team supervision A Release Management role may be required by SOX, Basilea, and similar regulation Deploy where? Production! We don’t care about the rest (although…), so we need to…
- Joe stops and thinks: I need to look at my pom.xml (build.gradle, *.csproj, Makefile, package.json, … name it) for references to Log4J (or whatever is vulnerable). Oh, but I use SLF4J which in turns… indirect dependencies! I need a tool just to find all possible references recursively. Oh Oh, our Tomcat configuration is using Log4J! I must check more than my JAR file, says Joe. …and the next question is…
- Here we discuss how to identify: 1. the code that needs to be patched 2. the pipeline that release that code in Production and some issues that one may face: If more than one branch can reach prod, which one you choose? How do you match the exact version of code? Software Composition Analysis kicks in only through pipelines? Is triggered by the deploy pipeline? The deploy pipeline hasn’t been used in months and doesn’t work anymore (e.g. a token expired, or there is no more an apt agent)
- …are there tools to support me and detect vulnerabilities in the code I deliver? Yes, there are BLAH
- The vulnerability could be a bad code pattern, use of an API, a vulnerable dependency; in any case we need to find the impacted code. We must scan all repositories that contain production code. Non-production repositories should be included in the search but listed separately to remove noise. Some patching can be easily automated, in particular library dependencies listed in project file (e.g. package.json, pom.xml,.csproj, …)
- † Lack of blue/green, canary, rolling/progressive deployment
- When I’ll be next We had billions of attacks!? Mostly intercepted email, though, one day it will be successful Ok but what might be the consequences of an attack?
- First batch of crucial elements of a solution identifying the qualities required on Software Configuration Management (SCM). Moving to Git is a prerequisite because any modern development tool shifted to it. What about the rest? Consolidate all projects into a single SCM platform – GitHub, Azure DevOps, BitBucket, GitLab The one listed have rich API that enable automation and integration Recommend set of branch names and mandatory tag names to identify code that goes/matches production This does not translate that all teams use the same process (e.g. GitFlow) but that they use the same conventions for branch names and, especially for tags Modern systems allows to attach custom properties to Git repositories and Pipeline definitions A Tag/Label/Property can identify repo in use, archived, or just with experimental code Same for pipelines, there are builds with limited scope (quick CI) and builds that deploy to production: use Tag/Label/Property to distinguish
- As mentioned, on a small scale, it is easy. Problems raise when you need to manage at scale: more than a few teams, repos, or pipeline. Consider the scenario where a single vulnerability impacts most of your applications (which is probable when you the majority of you code use the same platform, e.g. Log4J impacting all Java-based applications). You need to patch lots of repositories and deploy lots of components, each through a separate pipeline. In such scenario, you need new capabilities: Global editing tool Launch most pipelines in parallel (consider batching) Auto-scale build resources to sustain the spike Single-approval for the set of pipeline runs These aren’t offered by current systems.
- Can be automated? <pause> To my knowledge there are some tools that do some of the work, like GitHub dependabot It scans sources and proposes changes via a pull-request mechanism It does not support all package manager, though, and some features requires GitHub And clearly we need to input which is the correct version to use. We have seen toolchain attacks were the fix was to rollback, haven’t we?
- A crucial pattern to implement is the fast-track (expedite) pipeline. Every deployable component must have a pipeline that delivers just security fixes with as much regression testing as possible within a limited 2-hours timebox and as little manual steps (e.g. approvals) as possible. Tip: maybe you need some kind of incremental build to minimize build, test and deploy times.
- Current tooling may offer some information but a well-rounded process lot of cross-reference data. Dependency management is a weak spot in general, SCA (Software Composition Analysis) can identify vulnerabilities in libraries. Use of API may be caught by security scans Artifact management tool can track the source (build) of binaries if properly used. Pipeline knows which repositories they use, what we need here is ability to call a REST API that tell us the dependency. If you can use such tools, great. Maybe you need to follow a bit of conventions and write some query tools. In the worst scenario, you have to build and maintain your own database.
- Deploy where? Production! We don’t care about the rest (although…), so we need to… A Release Management role may be required by SOX, Basilea, and similar regulation But you need speed when it is a 0-day exploit. For example, you must be able to deploy a patch within hours of its release from a 3rd party (an OSS project or a vendor). fast-track (expedite) pipelines are not for normal usage: there should be some kind of trigger, like a new CVE, a communication from the Security team or upper management.
- What is the way to solve this burning problem?
- …they are not decreasing, quite the opposite. Increasing more than linearly!
- …display the same pattern, even more. Why?
- Modern app development is not using just a few suppliers like in the past. A study from Sonatype confirms that we use a lot more libraries, from all kind of sources, in particular Open Source libraries and attacks leverage this trend. The graphs illustrate Javascript scenario, but other languages…
- Both graphs illustrate that we, as an industry, aren’t exactly great at reacting and fixing our applications. The one on the left is data about OSS projects. The one on the right is more interesting because based on telemetry data, a more significant insight on IT organizations.
- .NET Core 3.1 3.1.0 December 3, 2019 3.1.22 December 14, 2021 got 22 patch releases in 3 years i.e. every 45 days/6 weeks Node v14 (Fermium) Active LTS start 2020-10-27 v14.15.0 2022-02-01, Version 14.19.0 total 19 releases in 463 days or 66 weeks i.e. every 24.4 days JDK 11 Java SE 11 (LTS)September 25, 2018 11.0.13+8 (GA), October 19th 2021 total 13 releases(updates) in 1121 days i.e. every 12.3 weeks or 86.2 days Go 1.16 released 2021-02-16 go1.16.14 (released 2022-02-10) total 14 updates in 360 days i.e. 26 days go1 (released 2012-03-28) -> go1.17 (released 2021-08-16) 17 major releases in 3429 days or 490 weeks MongoDB 5.0 5.0.0 - Jul 13, 2021 5.0.6 - January 31, 2022 total 6 releases in 203 days or 29 weeks i.e. every 4.8 weeks
- What is the way to solve this burning problem?
- Agile and DevOps focused on value-flow
- «An E-program is written to perform some real-world activity; how it should behave is strongly linked to the environment in which it runs, and such a program needs to adapt to varying requirements and circumstances in that environment» “On understanding laws, evolution, and conservation in the large-program life cycle” Lehman M.M. - Journal of Systems and Software Vol. 1, 1979–1980, pp. 213-221
- Today, I hope to convince you that we have serious problems in the way we patch and deploy applications, problems that we must address as an industry. At the core a perfectly working application today, is a huge risk tomorrow. That’s why I speak of decay and rotting, because it is not a slow process. Wear, erosion, rust… They do not convey the urgency and work required to preserve from decay. #1 unless you put it in a fridge or in a can, it starts smelling very soon #2 those other processes requires time, while rotting requires quick action to stop it I am not sure big an effort is to fix processes and tool to cope with security-related problems – the one this audience is acquainted to --. Security is the main driver, although not the only one. To change process and invest in tools, we have to speak to leadership/executive using a simple but effective vocabulary, so I suggest using the word inflation to convey the idea and start a discussion. As you may guessed, this presentation is a bit visionary, high-level, I will talk about industry trends and process not technology. For those interested in technology details, I recommend the sessions of my friends Michael Kaufmann and Matteo Emili. Now you have a couple of minutes to switch if you are not interested.
- I am open for questions.