Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Rebooting Software Development - OWASP AppSecUSA


Published on

If we are ever going to get ahead of the whack-a-mole security vulnerability game, we, as security professionals need to start getting involved more in the development of software. Let's review the origins of the traditional software development, and what assumptions are made. Then we'll review if those assumptions still hold for modern web applications, and what problems they cause, especially for security. Continuous deployment helps address these problems and allows for faster, more secure development. It's more than just "pushing code a lot", when done correctly it can be transformative to the organization. We'll discuss what continuous deployment is, how to get started, and what components are needed to make it successful, and secure.

Published in: Technology
  • Be the first to comment

Rebooting Software Development - OWASP AppSecUSA

  1. 1. Rebooting (secure) (web) software development with continuous deploymentNick Galbreath OWASP AppSec USA Austin, Oct 25, 2012@ngalbreath
  2. 2. The latest version of these slides Galbreath OWASP USA 2012-10-25 @ngalbreath
  3. 3. Presented at OWASP AppSec USAAustin, Texas, USAOctober 25, 2012 Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  4. 4. Based on a BSides Los Angeles presentation atHermosa Beach, August 18, 2012 Nick Galbreath OWASP USAThree Blocks 2012-10-25 from Conference. @ngalbreath
  5. 5. I took all these photos in NYC.Unless I didnt.In which case, they are from The Internet. Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  6. 6. Continuous Deployment? "Rebooting" software development?Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  7. 7. whoami Nick Galbreath @ngalbreath• Director of Engineering, Etsy • enterprise, fraud, security, fun • New Gig, but... Etsy sponsored my trip here• VP Engineering, “Company Confidential” • Stay tuned for details Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  8. 8. Context Alert! • My background is software development • Mostly in public, web-facing applications • Everything from C to PHP • Your mileage may vary if you are in different industries, with different risk profilesNick Galbreath OWASP USA 2012-10-25 @ngalbreath
  9. 9. Problem Statement • Web App vulnerabilities aren’t conceptually that hard and should be easy to deal with. • In spite (or because) of our efforts, security is an “end of line” process or whack-a-mole • Security education has been at best marginally useful to developers (in the large, your organization may be different). • How can we get ever get ahead?Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  10. 10. How did we get here?Nick Galbreath View from Hyatt Austin hotel room. OWASP USA 2012-10-25 @ngalbreath
  11. 11. The Software Product Model Code flows to functional groups. • Product Managers spec code • Engineers write code • QA tests code • Security tests code • Release engineers package code • Operations runs code Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  12. 12. High Distribution Cost The Software Product Model is designed for applications where the cost of distribution is high. Where “high” might be measure by risk, money, time, resources, customer annoyance. • Retail, CD/DVDs • Embedded or Exotic Hardware • Safety, Medical or Defense Systems • Operating Systems (phone or desktop) • Your Homework (1-time deploy)Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  13. 13. SPM -SDLC Release QA Ops QACommits freezeSpecs Development Bug Fix / New Specs Slush time in weeks or months Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  14. 14. SPM-Production Changes to Production a ng g B igB B an BigMajor Release Minor Releases Major Release New Features going live are 100% correlated with volume of changes to production. Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  15. 15. Nothing wrong here. Given high distribution costs, it makes senseNick Galbreath front-load USA development process to OWASP the 2012-10-25 @ngalbreath
  16. 16. WebApps Y2K • Mostly followed software product model since that’s all we knew • High barrier to entry • Specialized Hardware, software, people to get started • Lots of engineering to keep things running and scaling.Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  17. 17. True Story • “Can’t push out the spelling error change since it’s too risky” • “That code has already been through QA, it’s locked down.” • “Product has to prioritize that, else we aren’t touching it.” his ls T l e SmNick Galbreath OWASP USA 2012-10-25 @ngalbreath
  18. 18. WebApps 2012 • Almost no barrier to entry • Commodity hardware • “Learn PHP in 24 hours” • Scaling problems can be outsourced (sorta)Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  19. 19. WebApps 2012 and Cost of Distribution • Moving a few megabytes from source control to a few machines in production over a 1Gb or 10Gb link. • In other words... free!Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  20. 20. Given this and competitive /customer expectations, it’s not unreasonable to expect an SDLC moves faster than the Software Product ModelNick Galbreath OWASP USA 2012-10-25 @ngalbreath
  21. 21. On the other hand,WebApps 2012 have verydifferent failure cases Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  22. 22. The Nature of Failure • WebApps 2012 are data-driven. • and frequently have APIs, user-generated content, social features (unexpected use cases, new problems) • Failure might be due to algorithm problems, but... • ...more likely it’s bad user input, bad data in database, or operational load. • This means data added in the past might cause problems in the future. Complicated!Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  23. 23. And When It Happens • Rollback • Spend next week figuring out what changed, by whom, caused the breakage • Re-qa • Re-push • meanwhile new code is piling up.Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  24. 24. When SPM meets WebApp2012 • There is a long time between code-written to code- deployed. This “non-value added” steps for what should be low-cost changes. • Might be weeks or months before code deployed. • Feedback loop between code in dev and code in production broken. • When the bug/security report comes in, it’s likely the engineer is on a different project. • Any wonder that engineers don’t care for operations or security? Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  25. 25. Hypothesis • It is impossible to simulate the production environment in development, either to operational differences or data differences. • No amount of QA or Security Testing can prove you don’t have bugs, vulnerabilities, or won’t cause severe operational problems. • You have bugs and vulnerabilities right now your site.Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  26. 26. Conclusion:Youre Screwed! Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  27. 27. • Company wide push to move faster • Being a bottleneck isn’t acceptable. • Nor is giving up or saying “need more resources” • Engineers disengaged • Looming security disaster awaits • Whack-a-Mole doesn’t scaleNick Galbreath OWASP USA 2012-10-25 @ngalbreath
  28. 28. If we want tofix Security, we have toNick Galbreath fix Development. OWASP USA 2012-10-25 @ngalbreath
  29. 29. Continuous Deployment A System of Software Production Characterized By Numerous Small Changes to the Production Environment or That Crazy Shit That Etsy Does. And Google. And Facebook. And Amazon. And Twitter. And NetFlix. So maybe not that crazy.Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  30. 30. CD -Changes to Production new feature new feature New Features are not correlated with volume of changes to productionNick Galbreath OWASP USA 2012-10-25 @ngalbreath
  31. 31. Developers are responsibleand confident with their code.Nick Galbreath OWASP USA In 2012-10-25 @ngalbreath Production
  32. 32. What If You Had a Button that said DEPLOY •Pushes whatever is on HEAD/TRUNK to production. •In about a minute. •Anyone is allowed to push it. This button logs who performed the change, and what the change was, but no other rules or controls.Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  33. 33. Take 1: Fear • Likely no one is going to push it since they are afraid they’ll break something. • Meanwhile un-deployed code keeps piling up. ex. New hires are terrified of deploying an... HTML change! “but I don’t want to break Etsy!”Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  34. 34. Take 2: First Attempt • At some point, some brave sole will put their code on TRUNK, and push the button. • It’s likely someone else tells them that their feature blew up the site or doesn’t work, and to please role it back.Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  35. 35. Take 3: With Graphs • The developer learns that they’d don’t know how the code runs in production and they need some way of understanding how it works. • Enter Graphite/Ganglia/StatsD! measure-anything-measure-everything/ • Make it free to monitor anything in the application and expose that to everyone.Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  36. 36. Take 4: Push It • Repushing out code with fix, still causes some problem as witness by a graph falling off a cliff, but the developer was aware of it and was able to role back.Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  37. 37. Take 5: Isolation • Hmmm, the developer in reviewing the code notices that actually they are pushing a few bugs fixes, and some other minor features. • Maybe just pushing out a single bug fix one at time to help isolate the problem.Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  38. 38. Take 6: Success! • Yes! The developer pushed code and fixed a bug and made the site just that much better. • The secret about continuous deployment is small deltas that you or anyone can understand easily.Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  39. 39. Take 7: Dark Pushes • Now that the developer got the bugs out of the way, it time for the feature. • Let’s push out all the supporting files. By themselves they do nothing. By getting these out first, you isolate them as being “unlikely to cause a site problem” • Also now that they are on the trunk, others can look at them (easily).Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  40. 40. Take 8: Ramp-ups • Now it’s time to get that feature live. • Instead of a Big Bang, he’ll put a ‘ramp-up’ in the code. This will control how many people on the site will get the new feature. • Maybe start with “employees only” so his team can test in production. • Start at 1%, 5%, 10% and make sure things work, graphs are stable and work up to 100%. • if problem, can ramp-down or turn off.Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  41. 41. Take 8: Eliminate • Along the way you’ll get burned by little things, so, we’ll • A development environment that mimics prod as close as possible (won’t be exact) • Fast and stable unit and functional tests that are easy to run. If they are slow and flakey, no one will use them • Eliminate stupid bugs with commit or pre-commit static analysis. • Move QA/Security/Release checks as close as possible to the developer.Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  42. 42. Take 9: Communicate • As more people get use to it, you’ll need a way of co-ordinating releases among people. • IRC works well • Need set of conventions that match your risk levels. • At least developers are talking about releases!Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  43. 43. Take 10: Learn • No doubt along the way, serious mistakes will be made. Complex system failures will happen. • Learn from them. Do Post-Mortems. Do Root-Cause Analysis. • Recount what happened. • 99.99999999% of problems are caused by mistakes ... not maliciousness • How can the environment be changed so it doesn’t happen again? • Publish the results.Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  44. 44. Butt WhatAbout...Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  45. 45. What About That Guy Who Pushes at 3AM • That Guy who pushes at 3AM, and something goes wrong and wakes up all of operations with pagers going off will quickly learn this was a bad idea. • It’s about courtesy and respect. • Of course there are off-hours exception, in which That Guy should pre-inform everyone.Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  46. 46. What about... code reviews? • Yup, do them • Nothing here precludes code reviews. • In fact, it’s frequently easier to do since the reviewer doesn’t have to dork around with branches or tags.... they have all the dark code already on Trunk/Head • .. and the reviews are smaller and fasterNick Galbreath OWASP USA 2012-10-25 @ngalbreath
  47. 47. What about... security reviews? • Yup do ‘em. • Nothing here eliminates your existing review cycle.Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  48. 48. What about... Agile Methods? • (everyone does “agile differently” so hard to qualify this). • Agile methods frequently work to improve the spec-writing / development cycle • Or the spec / dev / qa cycle • But code still pools up waiting to go to production.Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  49. 49. What about Customer Service?Do they freak out with all the changes? • Remember, most changes either do nothing, or are trivial or are minor. • Feature launches always need to be co- ordinated with customer service(from audience question)Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  50. 50. So why did I tell you all this?Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  51. 51. That Engineer who previously didn’t push code is now sensitized that their code has consequences and are responsive to fix it. It’s amazing how interested engineers become in security when you find problems with their code when they are able to fix quickly themselves.Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  52. 52. Security Fixes can go out quickly. In addition, you know fixes can go out since they happen every day.Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  53. 53. You can repurpose the QA stack, graphing and log analysis for attack detection and vulnerability prevention. Need ideas? Check out these other presentations on fraud and security by Etsy: Galbreath OWASP USA 2012-10-25 @ngalbreath
  54. 54. While there is always whack-a-mole, you can focus on being a service organization and work on engineering to be secure by default.Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  55. 55. New Roles, Less Silos • Developers: works with operations • QA: works on making systems to empower people to write tests, static analysis, in-house consultancy on good test design • Release: tools to push code to production, system images. • Security: in house consultancy, security engineering, secure by default, detectionNick Galbreath OWASP USA 2012-10-25 @ngalbreath
  56. 56. So Continuous Deployment is Only for Websites? Wine Robot at DFW American Airlines lounge Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  57. 57. Google Chrome • Really made updates painless for the consumer. • Frequent changes “regularly” -- maybe not continuous but way faster than normal software product • Multiple channels of releases. • Config flags can turn on or off experimental features. • Works so well, many others are copying this approach.Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  58. 58. Apps • Due to cost of deployment being high (e.g. due to approval from Apple) • And due to diversity of destination (how many different types of hardware will it run on), hard to predict how well it work. • Put as much as you can into the release • Then read configs from internet to light up or turn off featuresNick Galbreath OWASP USA 2012-10-25 @ngalbreath
  59. 59. Chip Design • After this talk, I met an engineer who does hardware design. • All changes are tiny and then tested, then committed. • Any change too big is rejected. • Learned the hard way that big changes are impossible to understand and test.Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  60. 60. So What Now? Ice Skating at Rockefeller Center inNick Galbreath OWASP USA 2012-10-25 October. @ngalbreath
  61. 61. Security is in a Good Position to Force Change • Security bridges multiple disciplines: ops, dev, qa, release, business. • Unique position to make change. • When breach happens (in whatever the layer), we need to patch fast. • I hope that is not controversial.Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  62. 62. Start withNick Galbreath the Deploy Button OWASP USA 2012-10-25 @ngalbreath
  63. 63. It will change your SDLCNick Galbreath NYC Public Library OWASP USA 2012-10-25 @ngalbreath
  64. 64. Continuous DeploymentNick Galbreath OWASP USA 2012-10-25 @ngalbreath
  65. 65. Thanks!Nick Galbreath OWASP USA 2012-10-25 @ngalbreath
  66. 66. Nick Galbreath @ngalbreath nickg@client9.com Galbreath OWASP USA 2012-10-25 @ngalbreath