Successfully reported this slideshow.
Your SlideShare is downloading. ×

Hacking DevOps

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 16 Ad

Hacking DevOps

Incidents like the SolarWinds compromise show the extreme impact that a compromise of the software supply chain can have. DevOps pipelines often sit right at the heart of modern software supply chains. Used by development teams to increase the quality of their software and speed of delivery, these pipelines are also target-rich environments for attack. Additionally, they are often not as well protected as other software services. This talk will highlight common DevOps misconfigurations and how they can be leveraged by an attacker to escalate privileges, move laterally to other targets, and even perform supply chain compromises. Each example will also cover how to protect and defend against such an attack, and even how to use DevSecOps principles to protect the pipelines themselves. First presented at AvengerCon VII.

Incidents like the SolarWinds compromise show the extreme impact that a compromise of the software supply chain can have. DevOps pipelines often sit right at the heart of modern software supply chains. Used by development teams to increase the quality of their software and speed of delivery, these pipelines are also target-rich environments for attack. Additionally, they are often not as well protected as other software services. This talk will highlight common DevOps misconfigurations and how they can be leveraged by an attacker to escalate privileges, move laterally to other targets, and even perform supply chain compromises. Each example will also cover how to protect and defend against such an attack, and even how to use DevSecOps principles to protect the pipelines themselves. First presented at AvengerCon VII.

Advertisement
Advertisement

More Related Content

Recently uploaded (20)

Advertisement

Hacking DevOps

  1. 1. Hacking DevOps December 1, 2022 APPROVED FOR PUBLIC RELEASE; DISTRIBUTION UNLIMITED. CASE NUMBER 22-3754 © 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.
  2. 2. # whoami  Security Engineer: GSE #263, SANS Master’s Degree, 11+ security certifications  Hacker: Speaker at DEF CON Cloud & AppSec Villages  DevOps Engineer: Automating build, test, deployment, and monitoring  Systems Engineer: Focus on the overall system and process to deliver the system  Developer:  10+ years  Cloud Engineer: Designed and built both AWS and Azure environments for large teams © 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.
  3. 3. 3 Why should I care about DevOps?  DevOps is popular: ~83% adoption in one Puppet survey  Subverting this process is powerful because you can get inside an adversary’s development cycle to achieve superior visibility and persistence in their environment  Some of the attacks and defenses happen to open-source projects, providing an opportunity and a risk for “crowdsourced conflict”  Security is not consistently included in DevOps, if it is included at all © 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.
  4. 4. 4 Major Impacts of DevOps Compromises  SolarWinds Orion was modified during build time to include the malicious payload reaching many government organizations and costing millions to resolve  1500+ exposed Slack credentials on GitLab led to hackers being able to read proprietary info from major companies  A compromised JS package allowed remote code execution and exfiltration of package management credentials for up to 3.5 million installs © 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED. https://www.gao.gov/products/gao-22-104746
  5. 5. 5 Overview © 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED. https://dodcio.defense.gov/Portals/0/Documents/DoD%20Enterprise%20DevSecOps%20Reference%20Design%20v1.0_Public%20Release.pdf?ver=2019-09-26-115824-583
  6. 6. Protecting the Pipeline  Practice good DevOps for the pipeline itself (and related services) to ensure its integrity  The bootstrap problem can be somewhat mitigated and controlled through use of Infrastructure as Code (IaC) and continuous monitoring  Don’t forget the value of basic server hardening for internal resources  Consider using traditional tactics to compromise the pipeline, which can be a very valuable target for recon and increased access © 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED. 6
  7. 7. Source Control  Identity, Credential, & Access Management (ICAM)  Users – integrated with SSO?  Groups – where are they managed?  Use “Protected Branches” to enforce least privilege  Limit and/or protect potentially sensitive environment variables  Use one user’s access to pivot to or within SCM  Break developer expectations and insert malicious code into unprotected branches  Snoop on poorly protected environment variables, which may contain credentials  Exfiltrate development history to look for non-public vulnerabilities or incorrectly removed sensitive data or credentials © 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED. 7
  8. 8. Artifact Control  Permissions need to have a managed lifecycle  Who needs Read/Write  At what granularity are those permissions needed?  How are Non-Person Entities (NPEs) managed?  Have an artifact retention plan  Examples: left-pad, mimemagic  Use one actor’s access to pivot to additional data by inserting malicious artifacts  Disrupt operations or restoration capabilities by removing artifacts  If no artifact retention plan is in place, this can be done without internal access © 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED. 8
  9. 9. Pipeline  Limit access to modify sensitive or critical test steps  Alert and review when the pipeline changes  Include integrity checks when passing products or test results across security boundaries  Ensure sensitive details and credentials are appropriately secured  Ensure build steps are protected against disclosure of sensitive data  Change what the pipeline runs  Insert malicious steps  Remove security tests  Change test results  Change the built product to be different from the tested one  Find sensitive environment details or credentials  Pivot to ChatOps, test servers, prod servers  Recon for the rest of the DevOps process © 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED. 9
  10. 10. Build Servers  Manage which projects share build servers to limit the impact of a compromise  Use protected security boundaries between projects using a shared server  If relying on pre-installed software, verify its integrity  Access code and/or artifacts of new projects through poorly protected shared build servers  If a project uses pre-installed software or libraries, these can be compromised to produce false results © 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED. 10
  11. 11. Secrets  Manage permissions lifecycle  Put sensitive data into a Vault  For example: Passwords, Private Keys, API Tokens  Ensure secrets are not put into source control with a tool like Gitleaks  Avoid writing secrets to disk  Clear secrets when no longer needed, but be careful not to use these variables after they’re cleared  Find credentials and other sensitive data in:  Source code, using a tool like truffleHog  Configuration files  Environment variables in CI tools  Environment variables in test and production deployments © 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED. 11
  12. 12. Test & Evaluation  Limit/manage who can alter test cases and policy conditions  Consider more permissive development tests that become more strictly managed prior to making deployment decisions  Manage who has access to vulnerability information  N.B., developers need access to vulnerability results in order to remediate problems quickly  Monitor sensitive test steps for unusual activity  Change test results  Change what tests are run  Change pass/fail conditions  Exfiltrate results  For example, of security tests disclosing new vulnerabilities which may be used to further compromise the development and test environment © 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED. 12
  13. 13. ChatOps  Use a Privileged Access Management (PAM) solution to share credentials if it is necessary to do so  Practice good OPSEC if you don’t have control of the chat platform  Limit scope of access credentials provided to NPEs – for example to a #chatops-channel  Access to development discussions including bugs, vulnerabilities, architecture, schedules, and possibly credentials  May be able to trigger rebuild or redeploy, which could be useful to deny service or cover your tracks © 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED. 13
  14. 14. Infrastructure as Code  Test and validate infrastructure as code (IaC) using similar techniques to other code in the pipeline (e.g., static analysis)  Use IaC to consistently apply updates, including security patches  Consider adopting immutable infrastructure where infrastructure is destroyed and recreated instead of updated  Use IaC verification methods to monitor for unexpected changes to infrastructure and applications  Leverage IaC to reduce Mean Time To Recovery (MTTR) as part of a recovery plan  Compromising IaC provides a fantastic persistence mechanism  Include malicious infrastructure configurations to subvert application security  May be able to force deployment of older, vulnerable versions of the application  May be able to eliminate or bypass any mitigations and security services deployed as IaC © 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED. 14
  15. 15. 15 Key Takeaways  DevOps tools are:  Common  Contain valuable info for an attacker  Often poorly understood and protected  DevOps practices can be used to protect both the application being developed and the tooling used to evaluate it  Protecting development tooling is important for both open source and internal development teams  Failure to protect DevOps tools and processes can result in major impacts to an organization © 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.
  16. 16. Phillip Marlow pmarlow@mitre.org @wolramp linkedin.com/in/phillipmarlow © 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED. Thank You!

×