Secure Application Development in the Age of Continuous Delivery
•Download as PPTX, PDF•
0 likes•486 views
As delivered at LinuxCon and ContainerCon in Berlin 2016. Traditionally, when datacenter operators talk about application security, they've tended to focus on issues related to key management, firewalls and data access. By contrast, application developers have a security focus which is more aligned with code analysis and fuzzing techniques. The reality is, secure application deployment principles extend from the infrastructure layer through the application and include how the application is deployed. With the prevalence of continuous deployment of micro-services, it’s imperative to focus efforts on what attackers’ view as vulnerable; particularly in an environment where new exploits are being disclosed almost daily. In this session we’ll present: • How known vulnerabilities can make their way into production deployments • How deployment of vulnerable code can be minimized • How to determine the vulnerability status of a container • How to determine the risk associated with a specific package
Recommended
Recommended
More Related Content
What's hot
What's hot (18)
Similar to Secure Application Development in the Age of Continuous Delivery
Similar to Secure Application Development in the Age of Continuous Delivery (20)
More from Tim Mackey
More from Tim Mackey (19)
Recently uploaded
Recently uploaded (20)
Secure Application Development in the Age of Continuous Delivery
- 1. Secure Application Development in the Age of Continuous Delivery LinuxCon/ContainerCon Europe 2016 – Berlin Germany
- 2. #whoami – Tim Mackey Current roles: Senior Technical Evangelist; Occasional coder • Former XenServer Community Manager in Citrix Open Source Business Office Cool things I’ve done • Designed laser communication systems • Early designer of retail self-checkout machines • Embedded special relativity algorithms into industrial control system Find me • Twitter: @TimInTech ( https://twitter.com/TimInTech ) • SlideShare: slideshare.net/TimMackey • LinkedIn: www.linkedin.com/in/mackeytim
- 3. Standard Development Workflow DEVELOP BUILD PACKAGE DEPLOY PRODUCTION BUG TRACKING TEST AUTOMATION
- 5. Vulnerability Management Implies Data Breach Management 89% of data breaches had a financial or espionage motive Legal costs and forensics dominate remediation expenses Source: Verizon 2016 Data Breach Report
- 6. Attackers Decide What’s Valuable …
- 7. But security investment is often not aligned with actual risks
- 8. Anatomy of a New Attack Potential Attack Iterate Test against platforms Document Don’t forget PR department! Deploy
- 10. CLOSED SOURCE COMMERCIAL CODE • DEDICATED SECURITY RESEARCHERS • ALERTING AND NOTIFICATION INFRASTRUCTURE • REGULAR PATCH UPDATES • DEDICATED SUPPORT TEAM WITH SLA OPEN SOURCE CODE • “COMMUNITY”-BASED CODE ANALYSIS • MONITOR NEWSFEEDS YOURSELF • NO STANDARD PATCHING MECHANISM • ULTIMATELY, YOU ARE RESPONSIBLE Who is Responsible for Code and Security?
- 11. Knowledge is Key. Can You Keep Up? glibc Bug Reported July 2015 Vuln: CVE-2015-7547: glibc getaddrinfo stack-based buffer overflow
- 12. Knowledge is Key. Can You Keep Up? glibc Vuln Introduced May 2008 glibc Bug Reported July 2015 CVE-2015- 7547 CVE Assigned Feb 16-2016 Low Security Risk Vuln: CVE-2015-7547: glibc getaddrinfo stack-based buffer overflow
- 13. Knowledge is Key. Can You Keep Up? glibc Vuln Introduced May 2008 CVE-2015- 7547 CVE Assigned Feb 16-2016 glibc Bug Reported July 2015 National Vulnerability Database Vuln Published Feb 18-2016 Moderate Security Risk Low Security Risk Vuln: CVE-2015-7547: glibc getaddrinfo stack-based buffer overflow
- 14. Knowledge is Key. Can You Keep Up? glibc Vuln Introduced National Vulnerability Database Vuln Published You Find It May 2008 CVE-2015- 7547 CVE Assigned Feb 16-2016 Feb 18-2016 glibc Bug Reported July 2015 Patches Available You Fix It Highest Security Risk Moderate Security Risk Low Security Risk Vuln: CVE-2015-7547: glibc getaddrinfo stack-based buffer overflow
- 16. 0 500 1000 1500 2000 2500 3000 3500 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 Open Source Vulnerabilities Reported Per Year BDS-Enhanced Vulnerability Data nvd Reference: Black Duck Software KnowledgeBase, NVD Vulnerability Disclosures Trending Upward
- 17. 0 500 1000 1500 2000 2500 3000 3500 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 Open Source Vulnerabilities Reported Per Year nvd BDS-Enhanced Vulnerability Data Reference: Black Duck Software KnowledgeBase, NVD Vulnerability Disclosures Trending Upward
- 18. Vulnerabilities Enter Product and Become Deployed DEVELOP BUILD PACKAGE DEPLOY PRODUCTION BUG TRACKING TEST AUTOMATION
- 21. Primary goals • Protect against BIOS and firmware attacks • Protect cryptographic host state • Ensure valid hypervisor kernel • Validate launch of critical VMs • Attest to hosts’ trust state Implemented by • Intel Haswell and newer • Cryptographic hashes stored in TPM Intel TXT – Trusted Execution Protection - Foundational
- 22. Intel EPT – Extended Page Tables Page 0 … Page 13553 Page 13554 … … Page 126 Page 127 … Page 64589 Page 64590 Page 64591 Page 0 … Page 217 … Page 31289 … … Page 78924 … Page 97586 … 0→64589 13553→127 13554→64591 App Memory OS Memory TLB CR3 Virtual Machine 126→31289 127→0 64589→97586 64590→217 64591→78924 Host Memory EPT
- 23. Hypervisor Hypervisor Memory Introspection – Enabled by EPT Implementation Overview • Critical memory pages are assigned permissions in EPT • Exception handler defined in hypervisor • Shadow EPT defined with elevated privs Protects Against Attack Techniques • Rootkit injection • Buffer overflow • API hooking VM Kernel Memory Layout … Kernel Code (R/X) Driver Code (R/X) … Driver Data (R/W) Kernel Code (R/X) Kernel Data (R/W) … 126→31289 (R/X) 127→0 (R/X) 64589→97586 (R/W) 64590→217 (R/X) 64591→78924 (R/W) EPT#1 126→31289 (+W) 127→0 (+W) 64589→97586 (+X) 64590→217 (+W) 64591→78924 (+X) EPT#2 (Shadow) Exception Handler
- 24. Guest Guest Guest Guest Guest Critical Memory Access Critical Memory Access Critical Memory Access Critical Memory Access Critical Memory Access Networking StorageCompute Simplified Hypervisor Introspection Architecture Diagram Xen Project Hypervisor Control Domain (dom0) Security Appliance (domU) Memory Introspection Engine Direct Inspect APIs
- 25. Virtual Switches as Local Edge Protection – Silent Block Guest VM SSL access Attack silently blocked Virtual Switch Rules Ingress: HTTPS public Egress: Dynamic port to origin MySQL internal Private CIDR internal Port 22 access
- 26. Virtual Switches as Local Edge Protection – Traffic Monitor Guest VM SSL access Attack blocked with traffic log Virtual Switch Rules Ingress: HTTPS public Egress: Dynamic port to origin MySQL internal Private CIDR internal Port 22 access ovs Controller Log SSH Port 22 access Create port mirror for attacker Traffic Monitor Virtual Switch Rules Ingress: HTTPS public Egress: Dynamic port to origin MySQL internal Private CIDR internal Mirror: Port 22 to Traffic Monitor All attacker traffic to monitor
- 27. Guest VM Virtual Switches as Local Edge Protection – Quarantine Guest VM SSL access Attack quarantined with full log Virtual Switch Rules Ingress: HTTPS public Egress: Dynamic port to origin MySQL internal Private CIDR internal Port 22 access ovs Controller Log SSH Port 22 access Create port mirror for attacker Quarantine VM for attacker use Trigger replacement VM for farm Traffic Monitor Virtual Switch Rules Ingress: HTTPS attacker Egress: Dynamic port to origin Mirror: Port 22 to Traffic Monitor All attacker traffic to monitor
- 29. Are Containers Production Ready?
- 30. Container Use Cases Application containers • Hold a single application • Can follow micro-services, cloud native design pattern • Starting point for most container usage • Short lifespan, many per host System containers • Proxy for a VM • Insulate against core operating system • Perfect for legacy apps • Long lifespan, few per host MySQL Tomcat nginx Kernel MySQL Tomcat nginx Kernel
- 32. Baseline to Limit the Scope of Compromise • Enable Linux Security Modules • SELinux • --selinux-enabled on Docker engine, --security-opt=“label:profile” • AppArmor • -- security-opt=“apparmor:profile” • Apply Linux kernel security profiles • grsecurity, PaX and seccomp protections for ALSR and RBAC • Adjust privileged kernel capabilities • Reduce capabilities with --cap-drop • Beware –cap-add and –privileged=false, and CAP_SYS_ADMIN • Use a minimal Linux host OS • Red Hat Enterprise Linux Atomic Host 7, CoreOS, RancherOS, Intel Clear Linux, Alpine, etc • Reduce impact of noisy neighbors • Use cgroups to set CPU shares and memory
- 33. Red Hat Enterprise Linux Atomic Host 7 • What is Atomic Host? • Optimized RHEL7 variation designed for use with Docker • Uses SELinux for safeguards • Provides atomic upgrade and rollback capabilities via rpm-ostree • Pre-installed with Docker and Kubernetes • Atomic App and Atomic Nulecule • Provides a model for multi-container application definition • Supports Docker, Kubernetes, OpenShift and Mesos • OpenShift artifacts run natively or via atomic provider • Provides security compliance scan capabilities
- 34. Container Source Trust Red Hat Atomic Host AtomicApp AtomicApp AtomicApp Red Hat Registry MySQL Redis Jenkins Docker Hub DockerContainer DockerContainer DockerContainer DockerContainer DockerContainer Third Party and Custom Problem: Who to trust, and why? • Trusted source? • Unexpected image contents • Locked application layer versions (e.g. no yum update) • Layer dependencies (monolithic vs micro-services) • Validated when?
- 35. OpenSCAP vs. Black Duck Hub OpenSCAP • Profile driven compliance policy engine • Vendor vulnerability data is but one component of policy • Integrated directly with Red Hat Atomic • Usage: atomic scan --scanner openscap {container id} Black Duck Hub integration with Red Hat Atomic • Broad vulnerability data for most open source components • Covers vulnerability, license compliance and operational risk • Integrated with Red Hat Atomic • Rich tooling integration for development teams • Installed via: atomic install blackducksoftware/atomic • Usage: atomic scan --scanner blackduck {container id}
- 36. A DONATION HAPPILY GIVEN TO THE DEMO GODS
- 37. Gating Builds for Vulnerabilities DEVELOP BUILD PACKAGE BUG TRACKING VULNERABILITY ASSESMENT
- 38. Control Domain NetworkingCompute Storage Hypervisor Container VM Minimal OS Understanding Scope of Compromise – Protect From the Inside Container Container Container Container VM Minimal OS Container Container Container SecurityService Container
- 39. Understanding Scope of Compromise – Protect From the Inside Container VM Minimal OS Container Container Unpatched Container Compromised Container Compromised Container
- 40. Vulnerability Assessment Post Deployment DEVELOP BUILD PACKAGE DEPLOY PRODUCTION BUG TRACKING TEST AUTOMATION VULNERABILITY ASSESMENT
- 41. Risk Mitigation Shrinks Scope of Compromise Open source license compliance • Ensure project dependencies are understood Use of vulnerable open source components • Is component a fork or dependency? • How is component linked? Operational risk • Can you differentiate between “stable” and “dead”? • Is there a significant change set in your future? • API versioning • Security response process for project
- 42. Vulnerability Analysis Compliments SAST/DAST Vulnerability Analysis - Identifies vulnerable dependencies - 3000+ disclosures in 2015 - Most vulnerabilities found by researchers All possible security vulnerabilities Static and Dynamic Analysis - Discover common security patterns - Challenged by nuanced bugs - Focuses on your code; not upstream
- 43. 7 of the top 10 Software Companies (44 of the top 100) 6 of the top 8 Mobile Handset Vendors 6 of the top 10 Investment Banks 24 Countries 250+ Employees 1,800Customers Who is Black Duck Software? 27Founded 2002
- 44. 8,500 WEBSITES 350 BILLION LINES OF CODE 2,400 LICENSE TYPES 1.5 MILLION PROJECTS 76,000 VULNERABILITIES • Largest database of open source project information in the world. • Vulnerabilities coverage extended through partnership with Risk Based Security. • The KnowledgeBase is essential for identifying and solving open source issues. Comprehensive KnowledgeBase
- 45. Black Duck Hub Security Architecture Hub Scan1 File and Directory Signatures2 Open Source Component Identified 3 Hub Web Application Black Duck KnowledgeBase On Premise Black Duck Data Center
- 46. We Need Your Help Knowledge is power • Know what’s running and why • Define proactive vulnerability response process • Don’t let technology hype cycle dictate security Invest in defense in depth models • Don’t rely on perimeter security to do heavy lifting • Do look at hypervisor & container trends in security • Make developers and ops teams part of the solution • Focus attention on vulnerability remediation Together we can build a more secure data center
- 47. Free Tools to Help Free Docker Container Security Scanner • https://info.blackducksoftware.com/Security-Scan.html 14 Day Free Trial to Black Duck Hub • https://info.blackducksoftware.com/Demo.html
Editor's Notes
- http://www.istockphoto.com/photo/computer-crime-concept-gm516607038-89059287?st=9174601 http://www.verizonenterprise.com/verizon-insights-lab/dbir/2016/ Every year since 2008, Verizon have published a report on the attempted data breaches occurring within their data centers. For 2015, they found close to 90% of them had either a financial or espionage component to them. This report is well worth the read, and there are a few key findings in this report we should all be aware of. Costs of data breaches are heavily skewed towards legal consultation and forensics, and not to the public components of credit monitoring or lawsuits Despite some vulnerabilities having been public for years, there remain vulnerable components in use Some of those components simply may not have a patch forthcoming for a variety of reasons.
- Despite years of organizations spending energy protecting against attacks, it remains up to the attacker to define what’s valuable. Consider the case of ransomware. A police department in the town next to where I live was subjected to a raonsomeware attack. For roughly 500 USD in bitcoin, the attackers would decrypt the booking and evidence records they had just crypto locked. As an attacker, they likely had no knowledge of who they had attacked or what they had locked up. What mattered was the ransom, and that they had a police organization’s files didn’t factor into the equation.
- Big Idea: According to SAP research, most cyber attacks target the application layer, yet most security investment has been at the network layer. To best protect themselves from security breaches most IT organizations don’t necessarily need to spend more. They just need to spend smarter, investing in the areas that constitute the greatest risk. Question: How does your organization allocate IPSec spending? What % of your budget goes to application security v. network security? Why?
- Let’s take a little bit of time and look at how an attack is created. Potential attackers have a number of tools at their disposal, and use a number of different tactics. In this case, the attacker wishes to create an attack on a given component. In order to be effective, they have two primary models. First they can actively contribute code in a highly active area of the component with an objective of planting a back door of some form. The hope being that their code will fail to be recognized as suspect given how quickly the area of code is evolving. Second they can look for areas of code which are stable, and the longer they’ve bene stable, the better. The reason for this is simple, old code is likely written by someone who isn’t with the project any longer, or perhaps doesn’t recall all assumptions present at the time the code was written. After all, its been long understood that even with the best developers, assumptions change and old code doesn’t keep up. The goal in both cases being to create an attack against the component, so they test, and fail, and iterate against the component until they’re successful or move on. Assuming they’re successful, they create a deployment tool and document the tool for others. Of course, given the publicity received by some recent vulnerabilities, a little PR goes a long way. Now there are responsible researchers who follow a similar workflow, and they legitimately attempt to work with component creators to disclose vulnerabilities. They too will publish results, but are less interested in creating the an attack beyond a proof of concept. http://www.istockphoto.com/photo/person-in-hooded-sweater-using-a-laptop-on-wooden-table-gm464503138-58544934?st=cf78f31 http://www.istockphoto.com/photo/cloud-computing-gm518556682-90104967
- https://www.cesg.gov.uk/guidance/open-source-software-%E2%80%93-exploring-risk-good-practice-guide-38 If you’re using commercial software, the vendor is responsible for best practice deployment guidance, the notification of any security vulnerabilities and ultimately patches and workarounds for disclosed vulnerabilities. This is part of the deliverable they provide in return for their license fee. If you’re using open source software, that process becomes partly your responsibility. To illustrate the level of information you have to work with, let’s look at a media-wiki maintenance release from December 2015. “various special pages resulted in fata errors” – this clearly is something which needs resolution, but which pages? How do you test? “1.24.6 marks the end of support for 1.24.x” – this is good to know, but I hope it was published elsewhere. “However, 1.24.5 had issues (along with other versions) so it was thought fair to fix them” – This is a good thing, but can we expect this treatment in the future? From the title, we also have a fix for 1.23.x, but what other versions?
- https://sourceware.org/ml/libc-alpha/2016-02/msg00416.html https://sourceware.org/bugzilla/show_bug.cgi?id=18665 https://security.googleblog.com/2016/02/cve-2015-7547-glibc-getaddrinfo-stack.html https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-7547 (published via US-CERT) http://cve.mitre.org/cve/cna.html https://openclipart.org/detail/200681/primary-patch
- https://sourceware.org/ml/libc-alpha/2016-02/msg00416.html https://sourceware.org/bugzilla/show_bug.cgi?id=18665 https://security.googleblog.com/2016/02/cve-2015-7547-glibc-getaddrinfo-stack.html https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-7547 (published via US-CERT) http://cve.mitre.org/cve/cna.html https://openclipart.org/detail/200681/primary-patch
- https://sourceware.org/ml/libc-alpha/2016-02/msg00416.html https://sourceware.org/bugzilla/show_bug.cgi?id=18665 https://security.googleblog.com/2016/02/cve-2015-7547-glibc-getaddrinfo-stack.html https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-7547 (published via US-CERT) http://cve.mitre.org/cve/cna.html https://openclipart.org/detail/200681/primary-patch
- https://sourceware.org/ml/libc-alpha/2016-02/msg00416.html https://sourceware.org/bugzilla/show_bug.cgi?id=18665 https://security.googleblog.com/2016/02/cve-2015-7547-glibc-getaddrinfo-stack.html https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-7547 (published via US-CERT) http://cve.mitre.org/cve/cna.html https://openclipart.org/detail/200681/primary-patch https://www.youtube.com/watch?v=hkryI6eapOA
- https://software.intel.com/sites/default/files/Intel_R__TXT_Enabling_Guide.pdf
- Good reference for why this matters: http://www.rekall-forensic.com/posts/2014-10-03-vms.html
- The key to this technology is using the hypervisor, which already knows how to isolate VMs from one another in a performant way, to provide a virtual appliance, provided by a security vendor with the context required to monitor critical memory activity inside the VMs it’s protecting.
- If you assume from the outset your containers will be compromised, what would you do differently? What could you do to make life much harder to mount an attack from a compromised container? The first item everyone should have on their list is to enable the SELinux and AppArmor security modules for the distro you’re using as a base. For SELinux, you specify that it’s enabled on the command line for Docker Engine, and then on the container you specify a security option representing the profile you want to use. Now that you have some security modules in place, you’re in better shape, but you can still make it much harder to mount a proper attack. Consider for example an attacker who recognizes they’re in a container, and assumes that means there probably are multiple containers with the same profile. Armed with that knowledge, one of the easiest ways to make the life of an attacker harder is to randomize the memory load location. That’s where kernel security profiles come into play. Grsecurity, PaX and seccomp enable roles based access and address randomization upon load capabilities. Net of this, where the executable code lives in memory changes, and that makes it harder to know if you’ve created a viable attack or not. Now the Docker people have done a great job over the years in locking down the level of privileged access you get from within a container. In part, this is done using what’s known as kernel capabilities. Kernel capabilities offer a granular level of control over the types of operations you want the kernel to perform. Consistent with the concept of least privilege, you don’t want to ask for more rights than you need. If you find the defaults are providing more access than your application requires, you can pare things back using the –cap-drop option. Of course, it’s entirely possible you might need more rights, but if you find you need to disable priviledged access, or want to set the CAP_SYS_ADMIN flag, beware you’re effectively giving the container the equivalent of root access. Lastly, from a security perspective, you can choose to use a minimal Linux distro such as Atomic or CoreOS, but you still will want to pay attention to the options I’ve just outlined. So now that you’ve limited the access a potential attacker has to system services, you still have to contend with other types of havoc. A perfect example of this is the concept of a noisy neighbor. Most of us have had the experience of having someone in a neighboring space behave in an annoying manner. In the case of computing, the annoying neighbor can be one which consumes excessive memory or processing time. Limiting the scope of interference is very easy. All you need to do is define some CPU shares and memory limits for the container and set them during launch.
- Image source: US Navy 040814-N-5781F-033 Storekeeper 2nd Class Daniel Mina
- For example, if we have a datacenter filled with servers, and each server has virtual infrastructure which includes virtual networking and security services, we already have multiple layers of perimeter defenses. If we then add in container VMs which run a minimal OS upon which we load up some containers, we have a pretty good model for where some datacenters are today. If we then assume an attacker was able to compromise a container, now what? They’re on the other side of the various perimeters, and can attack from the inside. The big question being precisely how much damage could that attack create, and how would you know? This is the opportunity we have, and the people who are responsible for this are practitioners of a philosophy known as DevOps.
- Hub is based on a 3 part architecture: Scan Client – scans directories and artifact files creating “code prints” that uniquely but confidentially identify the files & directories contained in them Web Application – This is the main user interface and logic center for Hub. KnowledgeBase – This is a repository for open source component, license, and vulnerability information The code prints recoreded by the scan client are compared to reference code prints in the KnowledgeBase to identify open source components, versions, and origins. No source or binary code is ever uploaded.
- http://www.istockphoto.com/photo/strength-in-unity-gm514713440-88219133?st=af7fa36
- Docker Container Security Scanner https://info.blackducksoftware.com/Security-Scan.html 14 Day Free Trial to Black Duck Hub https://info.blackducksoftware.com/Demo.html Red Hat Atomic Host Integration (Requires Black Duck Hub) atomic scan --scanner blackduck [container]