DevOps, CLI, APIs, Oh My! Security Gone Agile

  • 241 views
Uploaded on

Traditional application security cannot keep pace with pace of change in applicaiton development - that model is dead. Move beyond the 5 stages of grief and get your agile security on. This talk …

Traditional application security cannot keep pace with pace of change in applicaiton development - that model is dead. Move beyond the 5 stages of grief and get your agile security on. This talk covers practices that helped the product security team at Rackspace keep up with the rate of change facing modern day application security teams.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
241
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
12
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • The purpose of this slide is to illustrate both the scale and scope of Rackspace in terms of the customers we serve and services we provide.
    Rackspace Hosting
    From both a Racker perspective and customer base perspective, we have seen dramatic growth.
    Our clients include more than 60% of the Fortune® 100. Today we have more than 190,000 customers in 120 countries.
    From a scope perspective, we have 8 DC’s serving our WW customer base and a broad portfolio of Cloud offerings….which is very important as it allows customers to mix and match our infrastructure to best suite the needs of their application. It drives flexibility.
    One of the achievements that we are most proud of is that Rackspace Hosting has been recognized by Fortune as one of the 100 best places to work not only in the United States, but in EMEA as well. People really like working here. What that means to customers is that we have a growing, stable workforce that is carefully selected not only for technical skills but also for how much each employee enjoys delivering exceptional service, and how well they match our culture and core values.
    OUR CULTURE AND THE EXCEPTIONAL SERVICE THAT WE BRAND AS FANATICAL SUPPORT MAKE THE DIFFERENCE BETWEEN GROWING AT, SAY, 5% A YEAR AND GROWING AT THE MUCH FASTER RATE THAT OUR COMPANY HAS EXPERIENCED IN THE LAST FEW YEARS
    Open Cloud - You’ll also note from looking at our logo in the upper left of the page….that we are branding our company as the “Open Cloud” company. As a founding member of the OpenStack community, we are committed to leveraging open technologies such as OpenStack to minimize technology lock-in and maximize flexibility. It means that we want to give our customers the choice to run their applications on infrastructure best suited to their particular application/workload.
    finally, Rackspace is the leader in Gartner’s Magic Quadrant for Managed Hosting….which really is a testament to our broad portfolio and the service we provide branded as Fanatical Support
    <number>
  • Vision
    Everyone at Rackspace can tell you our vision, a vision that we all support to become the world’s greatest service company. Our senior leadership is passionate about this. We refuse to accept mediocre. Once you accept less than great, you become “a phone company.” And, when was the last time you got great service from your mobile carrier or home phone company?
    PAUSE
    BUT, YOU CANNOT JUST HAVE A VISION TOO…
  • Vision
    Everyone at Rackspace can tell you our vision, a vision that we all support to become the world’s greatest service company. Our senior leadership is passionate about this. We refuse to accept mediocre. Once you accept less than great, you become “a phone company.” And, when was the last time you got great service from your mobile carrier or home phone company?
    PAUSE
    BUT, YOU CANNOT JUST HAVE A VISION TOO…
  • Vision
    Everyone at Rackspace can tell you our vision, a vision that we all support to become the world’s greatest service company. Our senior leadership is passionate about this. We refuse to accept mediocre. Once you accept less than great, you become “a phone company.” And, when was the last time you got great service from your mobile carrier or home phone company?
    PAUSE
    BUT, YOU CANNOT JUST HAVE A VISION TOO…
  • Vision
    Everyone at Rackspace can tell you our vision, a vision that we all support to become the world’s greatest service company. Our senior leadership is passionate about this. We refuse to accept mediocre. Once you accept less than great, you become “a phone company.” And, when was the last time you got great service from your mobile carrier or home phone company?
    PAUSE
    BUT, YOU CANNOT JUST HAVE A VISION TOO…
  • Vision
    Everyone at Rackspace can tell you our vision, a vision that we all support to become the world’s greatest service company. Our senior leadership is passionate about this. We refuse to accept mediocre. Once you accept less than great, you become “a phone company.” And, when was the last time you got great service from your mobile carrier or home phone company?
    PAUSE
    BUT, YOU CANNOT JUST HAVE A VISION TOO…

Transcript

  • 1. DevOps, CI, APIs, Oh My! Security Gone Agile Matt Tesauro, SANS AppSec 2014 – Austin, TX, February 2014
  • 2. Who am i? Matt Tesauro – Cloud Application Security Guy + OWASP Racker since October 2011 Rackspace’s Product Security Group Product Security Senior Engineer Work with developers and QE matt.tesauro@rackspace.com Former OWASP International Foundation Board Member and Treasurer Project Leader of OWASP Live CD / OWASP WTE matt.tesauro@owasp.org OWASP OpenStack Security Project RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 3. 3 ABOUT RACKSPACE 4,800+ RACKERS WE SERVE 60% 100 FORTUNE® OF THE OVER $1B Annualized Revenue 205,000+ CUSTOMERS 90,000+ SERVERS 26,000+ VM ≅70 PB STORED GLOBAL FOOTPRINT Founder OpenStack® Community Leader in Gartner ‘s Magic Quadrant for Managed Hosting CUSTOMERS IN 120+ COUNTRIES 9 WORLDWIDE DATA CENTERS PORTFOLIO OF HOSTED SOLUTIONS Dedicated - Cloud - Hybrid Named a Top Performer for Hosted Private Cloud by Forrester Research Inc. in “The Forrester Wave™: Q1 2013 RACKSPACE® HOSTING | WWW.RACKSPACE.COM
  • 4. DevOps, CI, APIs, Oh My! RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 5. A quick Overview of DevOps • The combination of traditional development activities with operations and testing (QA/QE) • Collaboration, communication and integration is key • Agile development model (sprints, scrum, stories…) • Release coordination and automation "DevOps" is an emerging set of principles, methods and practices for communication, collaboration and integration between software development (application/software engineering) and IT operations (systems administration/infrastructure) professionals. RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 6. CI, CD, CD, TDD and API CI == Continuous Integration CD == Continuous Deployment CD == Continuous Delivery TDD == Test Driven Development API == Application Programming Interface RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 7. THE PROBLEM • Cycle time for software is getting shorter • Continuous delivery is a goal • Scanning windows are not viable • First mover / first to market advantage RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 8. THE PROBLEM – or at least more • Traditional software development left little time to test • DevOps, Agile and Continuous Delivery squeeze those windows even more • New languages and programming methods aren’t making this better • Growth of interpreted languages with loose typing hurts static analysis efforts • Few automated tools to test APIs especially RESTful APIs • Little time for any testing, manual testing is doomed RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 9. 9 THE SOLUTION • Automated software testing • Automated operational infrastructure • Automated security testing RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 10. Think like a developer Sprints break software into little pieces… • Break your testing into little pieces • Use your threat model to know the crucial bits to test Long and short running tests • Testing time drives testing frequency • Code for tests needs to be optimized Smoke test versus full regression test • Smoke test early and often • Full regression tests on regular intervals RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 11. Maximize what you’ve got Make the most of your frameworks •Embrace, understand and fill gaps where necessary Make the best use of your time… • Make tests easily repeatable • Make tests easy to understand • Make tests abstract and combine-able • Ala carte tests for mixing and matching • Think about the Unix pipe | and its power RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 12. Under the constraints of DevOps, Continuous Deployment Your testing has to be nimble Dare I say…Agile In TDD, you know your code works when the tests pass In TD(S), you know your app has met the baseline when the tests pass RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 13. A time to morn... RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 14. 5 Stages of Grief This agile thing is a fad... Waterfall is the only way to produce quality software... RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 15. 5 Stages of Grief There's no way I can test in that time frame... If I see another freaking sticky note... RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 16. 5 Stages of Grief Well, I think I can test some of it in two days... I guess I can test it after its deployed to prod... RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 17. 5 Stages of Grief After that launch, I updated my LinkedIn profile... Game over man, GAME OVER... (Thanks Aliens) RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 18. 5 Stages of Grief So when can you add a story to work on that auth regression... After reviewing your deployment recipe, we filed a pull request to fix... RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 19. Fly thought those 5 stages by addressing... • Securing Infrastructure • Securing Apps and APIs • Securing Code RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 20. Securing Infrastructure RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 21. Automating Infrastructure • Declarative configuration language • Plain-text configuration in source control • Fully programmatic, no manual interactions RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 22. Chef for example Server / Hosted / Private 1. Solo Sys Admin 2. Server 3. Hosted 4. Private Hosted Node Node Node Node Node Node Node Node Node Node RACKSP ACE ® HOSTING Node Node Node Node Node | WWW.RACKS PACE.CO M
  • 23. Cookbooks, Stacks, Playbooks, ... • M ost have methods to bundle / share automation routines • You will have to write your own / customize • Good place to spend security cycles - Merge patches upstream for extra points. RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 24. Grouping & Tagging Apache MySql • Tagging your servers applies the required set of automation • A base set of for all servers • Each server can have multiple tags • M ap tags to security requirements Node Node Node Node Web Node Node Node Node DB Memcache Node Node Node Node Cache Monitoring RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 25. Inspector – you need one • For each group and/or tag • Review the recipe • Hook provisioning for post deploy review • Focus on checking for code compliance - Not perfection, bare minimums • Can include multiple facets - Security - Scalability - Compliance RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 26. Agent – one mole to rule them all • Add an agent to the standard deploy • Read-only helps sell to SysAdmin • Looks at the state of the system • Reports the state to the “mothership” • Add a dashboard to visualize state of infrastructure • Change policy, servers go red • Watch the board go green as patches roll-out • Roll your own or find a vendor RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 27. Turn Vuln scanning on its head • Add value for your ops teams • Subscribe and parse vuln emails for key software • Get this info during threat models or config mgmt • Provide an early warning and remove panic from software updates • Roll your own or find a vendor • Gmail + filters can work surprisingly well • Secunia VIM covers 40K+ products • Reverse the scan then report standard RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 28. Securing Apps & APIs RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 29. Findings directly to bug trackers • PDFs are great, bugs are better • Work with developer teams to submit bugs • Security category needs to exist • Bonus points if the bug tracker has an API • Security issues are now part of the normal work flow • Beware of death by backlog • Occasional security sprints • Learn how the team treats issues • ThreadFix is nice for metrics and pumping issues into issue trackers - http://code.google.com/p/threadfix/ RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 30. For the reticent: nag, nag, nag • Attach a SLA to each severity level for findings • Remediation plan vs Fixed • “Age” all findings against these SLAs • Politely warn when SLA dates are close • Walk up the Org chart as things get older • Bonus points for dashboards and bug tracker APIs • Get management sold first RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 31. Reports = Findings + Automation • Consider markup for findings • Markdown, Wiki Text, asciidoc • Pandoc to convert to whatever • HTML, PDF, .doc, .odt, ... • Keep testers writing the least possible • Template and re-use boiler plate items • New finding == new template for next time • Web app to keep things consistent • Create your own or maybe Dradis RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 32. Leverage existing consistencies • Requires consistent (generally automated) input • Find these and write some scripts • Automate the drudgery • Examples: • Automate finding/bug submission • Automate report PDF generation • API documentation to basic testing harness • Sec tool output – combine and convert RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 33. Securing Code RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 34. Start with the developers • Finding details have to be detailed enough to: • Reproduce the issue after 6 months • Allow QE to test the issue • Allow developers to find/fix the issue • Consider quick and dirty scripts to reproduce issue • Script to abuse an API • Web page of reflective XSS findings • Gauntlt - http://gauntlt.org/ • Once findings start flowing, look for training requests RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 35. Cherry pick what you look at • Threat Models are your friends • Focus on weak, unclear or suspicious areas • Focus on connections with external systems • Focus on format translations (XML to JSON) • When code changes in those areas, • Red flag it for review • Change +2 to +3 to before accepting pull request • Use search features in source code management • Start a list of problematic methods, calls, etc RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 36. No False Positives, period • If you can automate code review, you still must triage • 1 false positive == 100 valid bugs • If results aren't actionable, fail • Stick to diff analysis • Threat Modeling + “Scary Parts” + Code diffs == Quick triage of code changes • Automate where you can, iterate until you're happy • Need to build cred points with the dev teams RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 37. Quiet is better then wrong • Hire or befriend developers • Need to speak their language, not security's • Suggest requirements not implementation • Mitigation suggestions either generic or in the language the app is written in • Remember: Fast deploys also means fast fixes • Trying to shrink any vuln window not eliminate • Be prepared to retest / verify fix quickly RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 38. What is Rackspace's Product Security doing? RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 39. Securing Infrastructure • Rack has Chef, Puppet, Salt and Ansible, depending on the team • Reviewing the deployment scripts • Validating them with external vuln scans • Re-checks after bug fixes • Rack is using CloudPassage as a “mole” for some deployments • Also have some mole-like agents for one-offs • Rack has been conducting threat models ++ and using that info to watch for vulnerabilities RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 40. Securing Apps and APIs • Product Security finding workflow • PS team member find an issue • Documents it in Test Tracker app • Pushed finding(s) to ThreadFix • ThreadFix integrates with bug trackers • Metrics are driven off the ThreadFix database • We're re-implementing the nag, err reminder script for the new workflow • Using asciidoc markup for findings – easily creates PDFs, HTML, doc, reports based on templates RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 41. Securing Code • Rack is using Veracode if the language is supported • Self-service for the dev teams • Jenkins integration for submitting code to scan • API automation to pull findings into our workflow • PS team produces detailed finding blocks • Creates quick re-test scripts ad-hock • PS team holds trainings and has e-learning modules • PS team works with devs daily • Loaned to teams, attend stand-ups, … • PS “Dev Days” - team works on our automation RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 42. Key take aways • Automate, automate, automate • Look for “paper cuts” and fix those first • Finding workflow • Figure this out and standardize / optomize • Create systems which can grow organically • App is never done, its just created to easily be added to over time • Finding blocks become templates for next time • Learn to talk “dev” RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 43. Change is here and more is coming… "Whosoever desires constant success must change his conduct with the times." — Niccolo Machiavelli RACKSP ACE ® HOSTING | WWW.RACKS PACE.CO M
  • 44. ANY QUESTIONS? RACKSPACE® HOSTING US SALES: 1-800-961-2888 RACKSPACE® HOSTING | © RACKSPACE US, INC. | | | 5000 WALZEM ROAD | US SUPPORT: 1-800-961-4454 SAN ANTONIO, TX 78218 | WWW.RACKSPACE.COM RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. WWW.RACKSPACE.COM | 44