Successfully reported this slideshow.
Your SlideShare is downloading. ×

Scale14x: Are today's foss security practices robust enough in the cloud era final

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 49 Ad

Scale14x: Are today's foss security practices robust enough in the cloud era final

Download to read offline

Recent vulnerabilities like Heartbleed and Shellshock have brought the security practices and track record of open-source projects into the spotlight. A project’s response to security issues has a major impact on how much risk end users are exposed to and how the project is perceived in the technology industry.

We will compare the security practices of key projects such as Linux, Docker, Xen Project, OpenStack and others. We will explore the trade-offs of different security practices, such as community trust, competing stakeholder interests, fairness and media coverage of vulnerabilities. Finally, we will explore the evolution of the Xen Project’s security process over the past 3 years as a case study. We will illustrate the trade-offs, pain points and unexpected issues we have experienced, to help other projects understand the pit-falls in designing robust security processes and help users of open source projects understand how open source projects manage security vulnerabilities.

Recent vulnerabilities like Heartbleed and Shellshock have brought the security practices and track record of open-source projects into the spotlight. A project’s response to security issues has a major impact on how much risk end users are exposed to and how the project is perceived in the technology industry.

We will compare the security practices of key projects such as Linux, Docker, Xen Project, OpenStack and others. We will explore the trade-offs of different security practices, such as community trust, competing stakeholder interests, fairness and media coverage of vulnerabilities. Finally, we will explore the evolution of the Xen Project’s security process over the past 3 years as a case study. We will illustrate the trade-offs, pain points and unexpected issues we have experienced, to help other projects understand the pit-falls in designing robust security processes and help users of open source projects understand how open source projects manage security vulnerabilities.

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Viewers also liked (20)

Advertisement

Similar to Scale14x: Are today's foss security practices robust enough in the cloud era final (20)

More from The Linux Foundation (20)

Advertisement

Recently uploaded (20)

Scale14x: Are today's foss security practices robust enough in the cloud era final

  1. 1. Lars Kurth Community Manger, Xen Project Chairman, Xen Project Advisory Board Director, Open Source/Xen Project, Citrix lars_kurth
  2. 2. Was a contributor to various projects Worked in parallel computing, tools, mobile and now virtualization Community guy for the Xen Project Working for Citrix Member of the group that develops XenServer Chairman of Xen Project Advisory Board
  3. 3. I I: Vulnerability Introduced D: Vulnerability Discovered D Discoverer exploits issue for his own purpose
  4. 4. I I: Vulnerability Introduced D: Vulnerability Discovered Discoverer reports security issues to security@yourproject D
  5. 5. A team-effort to ensure that … • All (known) doors are closed • All (known) doors are locked • All (known) windows are boarded up • Fences have no (known) weaknesses • …
  6. 6. XF R: Vulnerability Reported T: Triage A: Vulnerability Announced F: Fix Available X: Fix Deployed Vulnerability is known by the reporter and the security team Note: It may also be known and used by black hats Vulnerability is known publicly with no fix available Vulnerability is known publicly with fix available Basic Description R T A Patch/fix creation and validation
  7. 7. X R: Vulnerability Reported T: Triage P: Vulnerability Pre-disclosed A: Vulnerability Announced F: Fix Available X: Fix Deployed Vulnerability is known by the reporter and the security team Note: It may also be known and used by black hats Vulnerability is known about by a privileged and small group of users Vulnerability is known publicly A Pre-disclosure period R P Patch/fix creation and validation FT
  8. 8. Users Safe
  9. 9. Encourage discoverers to report security issues to security@yourproject Discoverers are in control You can’t stop them from releasing/using information A robust vulnerability process encourages discoverers to work with you
  10. 10. Ensure that your project fixes security issues as quickly as possible You don’t want unaddressed vulnerabilities
  11. 11. Exposure time to security issues is minimized A maximum of users* apply patches quickly Minimize risk
  12. 12. Linux Kernel/LXC/KVM if reported via OSS Security Linux Kernel/LXC/KVM if reported via security@kernel.org OpenStack, QEMU, … for low impact issues Full Linux Kernel/LXC/KVM if reported via OSS Security Distros Linux Distributions (both open source and commercial) QEMU, Libvirt, oVirt, ... OpenStack for intermediate to high impact issues OPNFV, OpenDayLight : process modeled on OpenStack Xen Project for all issues (also handles 3rd party issues, e.g. QEMU) Docker : states responsible disclosure; but policy docs empty / some CVEs Responsible Cloud Foundry : no clearly stated process; no published CVE’s CoreOS: just a mail to report issues Kubernetes: : just a mail to report issues (when I wrote this talk in Aug, no info) Not clearly stated Approach Used by Projects
  13. 13. Open-source software projects are often well intended, but security can take a back seat to making the code work. OpenDaylight, the multivendor software-defined networking (SDN) project, learned that the hard way last August after a critical vulnerability was found in its platform. It took until December for the flaw, called Netdump, to get patched … PC World, March 2015
  14. 14. Using the pre-dominant model as baseline Applies to Linux Distros, OSS Sec Distros, QEMU, … Mike Licht @ Flickr
  15. 15. A X Typically fixed time during which the security issue is handled secretly Depends on discoverer’s wishes R: Vulnerability Reported T: Triage P: Vulnerability Pre-disclosed A: Vulnerability Announced F: Fix Available X: Fix Deployed Vulnerability is known by the reporter and the security team Note: It may also be known and used by black hats Vulnerability is known about by a privileged and small group of users Vulnerability is known publicly Description, CVE allocation, … Pre-disclosure period R Patch/fix creation and validation FT P What can and can’t be done with privileged information can differ significantly between projects
  16. 16. mindfulness @ Flickr
  17. 17. F A XR Disclosure Time
  18. 18. Long disclosure times discredit responsible disclosure From a few days to many months Long disclosure times create a disincentive for reporters to work with you Increases the risk of 0 day exploits Pre-defined disclosure times help manage vendors Example later Most successful projects have a 2-3 weeks disclosure period
  19. 19. Assigning CVE numbers is best practice in by established projects and vendors in the Linux/Cloud ecosystem
  20. 20. CVE databases (such as www.cvedetails.com) can be used to evaluate your project This shows Xen Project CVE stats Before 2012, we didn’t have fewer vulnerabilities than after We just didn’t have a process requiring creation of CVEs
  21. 21. A fair comparison between projects/technologies using CVE data is not easily possible Not all projects/products create CVEs for all their issues Example: Linux/QEMU only do so for severe ones Policies are not always published Some projects don’t assign CVEs at all Some technologies/products cannot be easily identified in databases Example: KVM, LXC Sometimes CVEs can affect several products But are counted only against one Open source product definitions on cvedetails are often sloppy
  22. 22. Mike Licht @ Flickr
  23. 23. Description, CVE allocation, … A D Pre-disclosure period R Patch/fix creation and validation FT P What happens here depends on your process goals
  24. 24. Make sure that a fix is available before disclosure Make sure that downstream projects and products (e.g. distros) can package and test the fix in their environment Allow service providers that use your Software to start planning an upgrade (at scale this can take a week) Allow service providers that use your Software to deploy an upgrade before the embargo completes
  25. 25. What is allowed during pre-disclosure Who is privileged and trusted to be on the pre-disclosure mailing list Disclosure Time
  26. 26. Make sure that a fix is available before disclosure Make sure that downstream projects and products (e.g. distros) can package and test the fix in their environment Allow service providers that use your Software to start planning an upgrade (at scale this can take a week) Allow service providers that use your Software to deploy an upgrade before the embargo completesCloud Model Distro Model
  27. 27. Emerged recently! Recognizes the needs of service providers Pre-Cloud Computing! Services and their users are vulnerable immediately after disclosure
  28. 28. Approach Used by Projects Linux Kernel/LXC/KVM if reported via OSS Security Distros Linux Distributions (both open source and commercial) QEMU, Libvirt, oVirt, ... OpenStack for intermediate to high impact issues OPNFV, OpenDayLight : process modeled on OpenStack Xen Project for all issues (also handles 3rd party issues, e.g. QEMU) Docker: depends on severity, details only available on request
  29. 29. More Cloud/Service users than direct users of your software Example: AWS stated in 2014 that they have > 1M users (and a lot more instances) AliCloud claims that they have > 1M users …
  30. 30. Just imagine what the reputation damage would have been, if Xen had put AWS, Rackspace, SoftLayer, … users at real risk of a vulnerability. There were 100’s of stories at the time, despite the fact that users were never put at risk, but merely inconvenienced !
  31. 31. Pre-disclosure list membership: more members, more risk of leakage In the Distro Model, the number of privileged users is typically <10 In the Cloud Model, the number could be an order of magnitude higher (50-100) This increases risk of information being accidentally released
  32. 32. Restricting pre-disclosure list membership Restricting membership to large service providers to minimize risk That creates issues of “fairness” Which may be incompatible with your communities' values
  33. 33. How the Xen Project got to its Vulnerability Process xenproject.org/security-policy.html Moyan Brenn @ Flickr
  34. 34. 2011 2012 2013 2014 2015 2016 Goals: Allow fixing, packaging and testing; Allow service providers to prepare (but not deploy) during embargo Pre-disclosure: Membership biased towards distros & large service providers No predefined disclosure time 1.0
  35. 35. 2011 2012 2013 2014 2015 2016 July 2012: CVE-2012-0217, Intel SYSRET Affected FreeBSD, NetBSD, Solaris, Xen and Microsoft Windows A large pre-disclosure list member put pressure on key members of the Xen Project Community to get an embargo extension They eventually convinced the discoverer to request an extension 1.0
  36. 36. 2011 2012 2013 2014 2015 2016 Centered on: Predetermined disclosure schedule: 1 week to fix, 2 weeks embargo Who should be allowed on the pre-disclosure list Fairness issues between small and large service providers Direct vs. indirect Xen consumers The risk of larger pre-disclosure list membership 1.0
  37. 37. 2011 2012 2013 2014 2015 2016 Strongly recommended disclosure schedule Inclusive pre-disclosure list membership Changes to application procedure (based on checkable criteria) 1.0 2.0
  38. 38. 2011 2012 2013 2014 2015 2016 Sept 2014: CVE-2014-7118 Leading to the first Cloud Reboot AWS pre-announced cloud reboot to their customers Other vendors didn’t. Policy was interpreted differently by vendors. This highlighted ambiguities in the project’s security policy (what can/can’t be said/done during an embargo) 1.0 2.0
  39. 39. 2011 2012 2013 2014 2015 2016 Goals: Allow fixing, packaging and testing Allow service providers to prepare (and normally to deploy) during embargo Pre-disclosure: Clearer application criteria Public application process (transparency) Clear information on what is/is not allowed during an embargo (per XSA) Means for pre-disclosure list members to collaborate 1.0 2.0 3.0
  40. 40. 2011 2012 2013 2014 2015 2016 Conducted XSA-133 Retrospective upon request Process change: Earlier embargoed pre-disclosure without patches May 2015: CVE-2015-3456 First time we were affected by a branded bug QEMU bug, which was handled by several security teams: QEMU, OSS Distro Security, Oracle Security & Xen Project From a process perspective: were not able to provide a fix 2 weeks before the embargo date ended 1.0 2.0 3.0
  41. 41. Larger pre-disclosure list has not caused a single issues in two years of operating an inclusive approach We have not had a single 0-day vulnerability A well run vulnerability process builds trust Willingness to adapt to your stake-holders needs builds more trust It creates collaboration and understanding of stake-holders Fairness is a difficult issue There will always be practical issues, e.g. “interpretations of policy”, etc.
  42. 42. The Xen Project’s process is the only example case, where this issue has been tackled through a community consultation. To Contrast: OpenStack does not publish who is on their pre-disclosure list OpenStack does not have a formal application process Avoids dealing with the “fairness” issue head-on
  43. 43. Security stories are “hot” Xen is widely used, thus security stories “sell” It’s too easy for reporters to write a story Reporters just have to check our page, and know when the next story comes
  44. 44. Source: yanilavigne.net via domics.me
  45. 45. Very wide range of approaches vs. The reality that SW stacks contain many layers Consider the weakest link in your SW stack Best Practice appears to be emerging Older projects seem slow to change New projects, don’t build security management into their culture from the beginning New Post-Snowden era pressures How to effectively deal with media Hype?

Editor's Notes

  • Recent vulnerabilities like Heartbleed and Shellshock have brought the security practices and track record of open-source projects into the spotlight. A project’s response to security issues has a major impact on how much risk end users are exposed to and how the project is perceived in the technology industry. We will compare the security practices of key projects such as Linux, Docker, Xen Project, OpenStack and others.

    We will explore the trade-offs of different security practices, such as community trust, competing stakeholder interests, fairness and media coverage of vulnerabilities.

    Finally, we will explore the evolution of the Xen Project’s security process over the past 3 years as a case study. We will illustrate the trade-offs, pain points and unexpected issues we have experienced, to help other projects understand the pit-falls in designing robust security processes.
  • Love to travel to weird places and grow weird plants
  • 2M

    Anyone disagrees?

    Zombie Analogy to illustrate the one or other thing

    Vulnerabilities as an opening (an unlocked door, broken window, gap in your fence, …) for s zombie to get into your base.

    Base = Medevial town, moats, walls, gates, doors, … - compartmentalized SW architecture
  • Unless you get them to report it: it is game over for that particular bug
  • Unless you get them to report it: it is game over for that particular bug
  • So, let’s go explore that Walking Dead analogy, …
    Band of survivors
    Lived on the road escaping zombies, moving from one place to the next
    Their ranks swell, but not excessively so – they always loose people to zombies

    Eventually they find a prison,
    Kill the remaining zombies and secure it.
    Prison = lots of ” security features, which kind of makes this an interesting analogy.
    Over time our survivors becomes quite good at securing their prison, getting out into the world and getting supplies, …

    => TEAM EFFORT

    Eventually it all goes haywire, because they then end up at war with another group of survivors.
  • 3 MINS
  • P : instead of an announcement, you pre-disclose to a “special group of users”

    Bubble: hold that in your mind
  • What I wanted to do in this talk, is to look at Vulnerability Management.
    NOT at SW architecture, NOT at other techniques to minimize introduction of bugs, …

    Purpose of … = keep your users safe.
    Or at least that’s what it should be there for.

    I chose the picture above, because
    it’s from the Walking Dead comic/TV series – which I will continue to use as an analogy
    that little boy to be your user, who has to fend off security issues all by himself



  • I wanted to come back to the point about how to keep your users safe and what that means.

    MOST security flaws
    not be discovered by your DEVS,

    But Outsiders => One of the reasons v proc is to encourage reporters to work with you

    I think that side of things is well understood
  • Now you know of an issue

    And you need to get your developers to fix issues as fast as possible
    In the Xen Project, we have managed to do this typically in a week (in a few cases it took 2)
  • So now we have a fix and we can get this out to end users.

    And that’s where it get’s messy and where

    Different types of users:
    products built from your sw
    Services
    End-user

    Typically, unless you are and end-user project with auto-upgrade capability, you don’t have control of that part of the process.
  • So what do various projects actually do? Sample ….

    Linux: depends on how it is reported
    Predominant model for established projects Responsible disclosure (and in practice that applies to the kernel also)

    Distro/Cloud … explain that later

    Newer projects: not clear
  • NOTES:
    Wake-up call for ODL via Netdump
    First release
    OpenDayLight was not in production yet
    The issue spurred the project into creating a vulnerability process and security team *before* it does go into production
  • 15M
    ----
  • Resp : predominant model => detail

    Point out ORANGE
    What & Who?
  • Going to look at a couple of examples
  • 17M
  • Apple: 6 months (cross app resource vulnerability to allows access to keychain data, BIOS)

    Trade-off: disclosure time

  • Another trade-off
  • Refer to Intel talk at OSCON:
    makes the case that open source projects are more secure, because the proportion of high to severe issues is lower than in commercial products
  • 23M
  • Also depends on your user: this talk is about cloud security


  • Who should be on your predisclosure list
  • #1: High Value target
    #2: At scale, deploying fixes may take a week or two
  • So what do various projects actually do? Sample ….

    Linux: depends on how it is reported
    Predominant model for established projects Responsible disclosure (and in practice that applies to the kernel also)

    Distro/Cloud … explain that later

    Newer projects: not clear


  • This has not been an issue for the Xen Project, which has been operating an inclusive list for 2+ years
    Also: only issues for which fixes exist could normally be leaked
  • One thing you could do is to focus on a subset of service providers, e.g. large ones, or paying members, …
  • 30M
  • What was interesting, was a large vendor – call it V

    CEOs of members of the security team, maintainers and committers to get a 6 week extension

    Handed of handling off this CVE off to Mitre
  • Was: distros + VERY LARGE users

    Small service providers felt they could go out of business because of a sec issue, while large vendors reputation merely would be damaged

    What about service providers who are customers of a commercial distro? They shouldn’t be disadvantaged


  • An example of transparency
  • OpenStack process is similar to the Xen Project process

    If you don’t publish the list, fewer users want to join
  • 40M
  • After the cloud reboot, we were suddenly a high profile target for the tech press

    Every time we made a point release (which contained some XSA’s) the story was a security atory
    Every XSA is now a security story (regardless of whether it is Xen or QEMU)

  • TIMING: 16 MINUTES
  • TIMING: 16 MINUTES

×