Testing at-cloud-speed sans-app-sec-austin-2013

302 views

Published on

  • Be the first to comment

  • Be the first to like this

Testing at-cloud-speed sans-app-sec-austin-2013

  1. 1. Testing at Cloud SpeedMatt Tesauro, SANS AppSec 2013– Austin, TX, April 2013
  2. 2. RACKSPACE® HOSTING | WWW.RACKSPACE.COMWho am i?Matt Tesauro – Cloud Application Security Guy + OWASP2matt.tesauro@rackspace.commatt.tesauro@owasp.orgRacker since October 2011Rackspace’s Product Security GroupProduct Security Engineering LeadWork with developers and QEFormer OWASP International Foundation BoardMember and TreasurerProject Leader ofOWASP Live CD / OWASP WTEOWASP OpenStack Security Project
  3. 3. RACKSPACE® HOSTING | WWW.RACKSPACE.COMABOUT RACKSPACE205,000+CUSTOMERS90,000+ SERVERS26,000+ VM≅70 PB STORED4,800+RACKERS9 WORLDWIDEDATA CENTERSGLOBALFOOTPRINTCUSTOMERS IN120+ COUNTRIES60%FORTUNE® 100OFTHEWE SERVEAnnualized RevenueOVER $1BPORTFOLIO OFHOSTED SOLUTIONSDedicated - Cloud - HybridLeader in Gartner ‘s Magic Quadrantfor Managed HostingFounderOpenStack® Community3Named a Top Performer for Hosted Private Cloudby Forrester Research Inc. in “The Forrester Wave™: Q1 2013
  4. 4. RACKSPACE® HOSTING | WWW.RACKSPACE.COM4SECURING APPS IN ADevOps WORLD
  5. 5. RACKSPACE® HOSTING | WWW.RACKSPACE.COMA quick Overview of DevOps• The combination of traditional development activities with operations andtesting (QA/QE)• Collaboration, communication and integration is key• Agile development model (sprints, scrum, …)• Release coordination and automation5"DevOps" is an emerging set of principles, methods and practices forcommunication, collaboration and integration between software development(application/software engineering) and IT operations (systemsadministration/infrastructure) professionals.
  6. 6. RACKSPACE® HOSTING | WWW.RACKSPACE.COMCI, CD, CD, TDD and APICI == Continuous IntegrationCD == Continuous DeploymentCD == Continuous DeliveryTDD == Test Driven DevelopmentAPI == Application Programming Interface6
  7. 7. RACKSPACE® HOSTING | WWW.RACKSPACE.COM7• Cycle time for software is gettingshorter• Continuous delivery is a goal• Scanning windows are not viable• First mover / first to marketadvantageTHE PROBLEM7
  8. 8. RACKSPACE® HOSTING | WWW.RACKSPACE.COM8THE PROBLEM – or at least more• Traditional software development left little time to test• DevOps, Agile and Continuous Delivery squeeze those windowseven more• New languages and programming methods aren’t makingthis better• Growth of interpreted languages with loose typinghurts static analysis efforts• Few automated tools to test APIs especiallyRESTful APIs• Little time for any testing, manual testing is doomed
  9. 9. RACKSPACE® HOSTING | WWW.RACKSPACE.COM9• Automated softwaretesting• Automated operationalinfrastructure• Automated securitytestingTHE SOLUTION9
  10. 10. RACKSPACE® HOSTING | WWW.RACKSPACE.COMThink like a developerSprints break software into little pieces…• Break your testing into little pieces• Use your threat model to know the crucial bits to test10Long and short running tests• Testing time drives testing frequency• Code for tests needs to be optimizedSmoke test versus full regression test• Smoke test early and often• Full regression tests on regular intervals
  11. 11. RACKSPACE® HOSTING | WWW.RACKSPACE.COMMaximize what you’ve gotMake the most of your frameworks•Embrace, understand and fill gaps where necessary11Make the best use of your time…• Make tests easily repeatable• Make tests easy to understand• Make tests abstract and combine-able• Ala carte tests for mixing and matching• Think about the Unix pipe | and its power
  12. 12. RACKSPACE® HOSTING | WWW.RACKSPACE.COMUnder the constraints of DevOps, Continuous Deployment12Your testing has to be nimbleDare I say…AgileIn TDD, you know your code workswhen the tests passIn TD(S), you know your app has metthe baseline when the tests pass
  13. 13. RACKSPACE® HOSTING | WWW.RACKSPACE.COMIts time to set the snail on fire!13➔Infrastructure➔App / API➔Code
  14. 14. RACKSPACE® HOSTING | WWW.RACKSPACE.COM14Securing Infrastructure
  15. 15. RACKSPACE® HOSTING | WWW.RACKSPACE.COM15Automating Infrastructure15• Declarative configuration language• Plain-text configuration in source control• Fully programmatic, no manual interactions
  16. 16. RACKSPACE® HOSTING | WWW.RACKSPACE.COM16Chef161. Solo2. Server3. Hosted4. Private HostedNodeNodeNodeNodeNodeNodeNodeNodeNodeNodeNodeNodeNodeNodeNodeSysAdminServer / Hosted / Private
  17. 17. RACKSPACE® HOSTING | WWW.RACKSPACE.COM17Cookbooks17• Most major softwarepackages havecookbooks• You will have to writeyour own / customize• Good place to spendsecurity cycles-Merge patches upstream forextra points.
  18. 18. RACKSPACE® HOSTING | WWW.RACKSPACE.COM18Grouping & Tagging18• Tagging yourservers appliesthe required setof recipes• A base set ofrecipes iscommon• Each server willhave multipletags set atbootstrap timeNodeNodeNodeNodeDBNodeNodeNodeNodeCacheNodeNodeNodeNodeWebApacheMonitoringMySqlMemcache
  19. 19. RACKSPACE® HOSTING | WWW.RACKSPACE.COM19Inspector – you need one19• For each group and/or tag• Review the recipe• Hook provisioning for postdeploy review• Focus on checking for codecompliance-Not perfection, bare minimums• Can include multiple facets-Security-Scalability-Compliance
  20. 20. RACKSPACE® HOSTING | WWW.RACKSPACE.COM20Agent – one mole to rule them all20• Add an agent to the standard deploy• Read-only helps sell to SysAdmin• Looks at the state of the system• Reports the state to the “mothership”• Add a dashboard to visualize state of infrastructure• Change policy, servers go red• Watch the board go green as patches roll-out• Roll your own or find a vendor
  21. 21. RACKSPACE® HOSTING | WWW.RACKSPACE.COM21Turn Vuln scanning on its head21• Add value for your ops teams• Subscribe and parse vuln emails for key software• Get this info during threat models• Provide an early warning and remove panic fromsoftware updates• Roll your own or find a vendor• Gmail + filters can work surprisingly well• Secunia VIM covers 40K+ products• Reverse the scan then report standard
  22. 22. RACKSPACE® HOSTING | WWW.RACKSPACE.COM22Securing Apps & APIs
  23. 23. RACKSPACE® HOSTING | WWW.RACKSPACE.COM23Findings directly to bug trackers23• PDFs are great, bugs are better• Work with developer teams to submit bugs• Security category needs to exist• Bonus points if the bug tracker has an API• Security issues are now part of the normal work flow• Beware of death by backlog• Occasional security sprints• Learn how the team treats issues
  24. 24. RACKSPACE® HOSTING | WWW.RACKSPACE.COM24For the reticent: nag, nag, nag24• Attach a SLA to each severity level for findings• Remediation plan vs Fixed• “Age” all findings against these SLAs• Politely warn when SLA dates are close• Walk up the Org chart as thingsget older• Bonus points for dashboards andbug tracker APIs• Get management sold first
  25. 25. RACKSPACE® HOSTING | WWW.RACKSPACE.COM25Reports = Findings + Automation25• Consider markup for findings• Markdown, Wiki Text, asciidoc• Pandoc to convert to whatever• HTML, PDF, .doc, .odt, ...• Keep testers writing the least possible• Template and re-use boiler plate items• New finding == new template for next time• Web app to keep things consistent• Create your own or maybe Dradis
  26. 26. RACKSPACE® HOSTING | WWW.RACKSPACE.COM26Leverage existing consistencies26• Requires consistent (generally automated) input• Find these and write some scripts• Automate the drudgery• Examples:• Automate finding/bug submission• Automate report PDF generation• API documentation to basic testing harness
  27. 27. RACKSPACE® HOSTING | WWW.RACKSPACE.COM27Securing Code
  28. 28. RACKSPACE® HOSTING | WWW.RACKSPACE.COM28Start with the developers28• Finding details have to be detailed enough to:• Reproduce the issue after 6 months• Allow QE to test the issue• Allow developers to find/fix the issue• Consider quick and dirty scripts to reproduce issue• Script to abuse an API• Web page of reflective XSS findings• Once findings start flowing, look for training requests
  29. 29. RACKSPACE® HOSTING | WWW.RACKSPACE.COM29Cherry pick what you look at29• Threat Models are your friends• Focus on weak, unclear or suspicious areas• Focus on connections with external systems• Focus on format translations (XML to JSON)• When code changes in those areas,• Red flag it for review• Change +2 to +3 to before accepting pull request• Use search features in source code management• Start a list of problematic methods, calls, etc
  30. 30. RACKSPACE® HOSTING | WWW.RACKSPACE.COM30No False Positives, period30• If you can automate code review, you still must triage• 1 false positive == 100 valid bugs• If results arent actionable, fail• Stick to diff analysis• Threat Modeling + “Scary Parts” + Code diffs== Quick triage of code changes• Automate where you can, iterate until youre happy• Need to build cred points with the dev teams
  31. 31. RACKSPACE® HOSTING | WWW.RACKSPACE.COM31Quiet is better then wrong31• Hire or befriend developers• Need to speak their language, not securitys• Suggest requirements not implementation• Mitigation suggestions either generic or in thelanguage the app is written in• Remember: Fast deploys also means fast fixes• Trying to shrink any vuln window not eliminate• Be prepared to retest / verify fix quickly
  32. 32. RACKSPACE® HOSTING | WWW.RACKSPACE.COMSo I was talking with a friend…He was bemoaning the pace of change and the speed at which softwarewas being pushed to production…32In essence, management has made the decision thatgetting their app out the door with possible bugs ismore valuable to the business then having strongassurance that the software has few or no significantbugs.You’ve got to up your game,get automated, agile andget on pace with your developers.
  33. 33. 33RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COMRACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. |WWW.RACKSPACE.COMANY QUESTIONS?Slides on slideshare – look for user “mtesauro”

×