Fixing security by fixing software development


Published on

Fixing Security by Fixing Software Development Using Continuous Deployment
Do you have an effective release cycle? Is your process long and archaic? Long release cycle are typically based on assumptions we haven't seen since the 1980s and require very mature organizations to implement successfully. They can also disenfranchise developers from caring or even knowing about security or operational issues. Attend this session to learn more about an alternative approach to managing deployments through Continuous Deployment, otherwise known as Continuous Delivery. Find out how small, but frequent changes to the production environment can transform an organization’s development process to truly integrate security. Learn how to get started with continuous deployment and what tools and process are needed to make implementation within your organization a (security) success.

1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Fixing security by fixing software development

  1. 1. Fixing Security by Fixing Software DevelopmentUsing Continuous DeploymentNick Galbreath, VP Eng IPONWEB, 14-15, San Francisco, California
  2. 2. Follow AlongLatest version posted online:
  3. 3. Nick Galbreath (@ngalbreath)! Spoken at Black Hat, DEFCON, OWASP! Author of libinjection: advanced SQLidetection used in > 2 WAFs, 1 Honeypot! Book on cryptography! but really...! Engineering Management and SoftwareDevelopment for high growth startups.! Personal site
  4. 4. IPONWEB! customized online advertisinginfrastructure and exchanges! engineering offices in Moscow, withbusiness offices in London, New York andTokyo.
  5. 5. Original AbstractFixing Security by Fixing Software Development Using ContinuousDeploymentDo you have an effective release cycle? Is your process long and archaic? Longrelease cycle are typically based on assumptions we havent seen since the1980s and require very mature organizations to implement successfully. Theycan also disenfranchise developers from caring or even knowing about securityor operational issues. Attend this session to learn more about an alternativeapproach to managing deployments through Continuous Deployment, otherwiseknown as Continuous Delivery. Find out how small, but frequent changes to theproduction environment can transform an organization’s development processto truly integrate security. Learn how to get started with continuous deploymentand what tools and process are needed to make implementation within yourorganization a (security) success.
  6. 6. Well thats a bold statement..."Fixing Security by Fixing DevelopmentUsing Continuous Deployment"
  7. 7. and heres anotherFor web applications, our release-basedsoftware development lifecycle is stillbased on a pre-Internet model and isharmful to organizations and
particularly harmful for security.
  8. 8. What needs fixing?! SQLi dropped from #8 to #14 in the latestWhite Hat "The State of Web Security"report. Good news, right?! This means SQLi is only 7% of websites.Thats 1 in 15. And this is #14vulnerability!! And time to fix was on average 196 days. 
Thats embarrassing.Veracode claims 32% of incoming web applications have SQLi
  9. 9. Even worse...! Number 1 driver to fix security problems...
compliance.! Number 1 reason to not fix security is...
compliance.! Not.. ! keeping our employees and customers safe! protecting corporate interests.! improving quality! being good at what we do.
  10. 10. Security Products #1 .. in security bugsVeraCode: State ofSoftware Security, V4December 2011
  11. 11. Lets Just Give Up! “You could spend all your resources chasing suchthings as this,” William Ribich, the former president ofTechnology Solutions Group [ QintelIQ ], said in aninterview in January. Ribich, who retired in November 2009,shortly after the discovery of a major data theft, said heneeded to balance the uncertain risk that the hackers coulduse what they stole against a growing shopping list ofsecurity products and consulting fees.! "You finally have to reach a point where you say
’let’s move on,’” he said.
  12. 12. I would call that broken! But preventing SQLi isnt a technicallyhard problem.! And most security patches are very small.! How did we get here?
  13. 13. Software Product Model! Code flows between functional groups! Product Managers spec code! Engineers write code! QA engineers tests code! Release engineers package code! Operations runs code! ... and Security does something too.
  14. 14. High Distribution Cost! The Software Product Model is designedfor software where the cost of distributionis high. "High" might be financial, risk,time, resources, customer annoyance.! Retail, physical product, CD/DVD! Embedded of Exotic Hardware! Safety, Medical or Defense Systems! Operating Systems (desk or phone)! Homework (1-time deploy)
  15. 15. Software Product Lifecycle
  16. 16. Change to Production
  17. 17. Web Applications Year 2000! Mostly followed Software Product Modelsince thats all we knew.! High barrier to entry! Specialized Hardware, Software andPeople needed to get started.! Lots of engineering needed to keep thingsrunning.! (side note: CERT/CVE started in 1999)
  18. 18. True Story #1 ! "Cant push out the spelling error fix – itstoo risky"! "That code as already been through QA,its locked down."! "Product has to prioritize that change, elsewe arent touching it."
  19. 19. True Story #2! Well do an iteration, where we try to fix asmany things as possible.! This wont be a scheduled iteration, it willbe done because things are so bad! So the spelling error will get fixed... 
uhh, who knows when.
  20. 20. Web Applications 2013 ! Almost no barrier to entry! Commodity hardware! Programming not that hard! Scaling problems can 
be mostly outsourced
  21. 21. Cost of Distribution 2013! Frequently no compile step
or its very fast.! Moving to production a few kilobytes ormegabytes of code over 1Gbps, 10Gbpslink.! In other words... free
  22. 22. Failure is very different however! Most web applications are data-driven.! Frequently have social features, APIs,user-generated content.! Failures might be due to algorithmicproblems... but...! Most likely to due to user input, bad datain database or operational load.! this means data in past can causeproblems in the future.
  23. 23. Releases and Problems! When a web-release goes out, and hasproblems....! Next week is spent tracking down whochanged what, where.! Re-QA! Re-Push! meanwhile new code is piling up.
  24. 24. When SPM meets Web Apps! A long time between code being writtenand code being released.! Might be weeks or months! Feedback loop between code-in-dev andcode-in-production is broken! When security or bug reports come in, theauthor is likely on a different project.
  25. 25. Hypothesis! It is impossible to simulate the productionenvironment in development, either due tooperational differences or data differences.! No amount of QA or Security Testing canprove you dont have bugs, vulnerabilities,or cause severe operational problems.! You have bugs and vulnerabilities,
right now, in your application.
  26. 26. Impedance Mismatch! Easy to write code, +! Long release cycles +! Security as an end-of-line or out of bandprocess ==! no one cares
  27. 27. So the Answer is...! Going slower? Im sure your boss will lovethat suggestion! More steps and process? In other words,slower.! Asking for more people? Sure but goodluck hiring them. Doesnt scale.! Asking for more products? Since theothers have worked so well.
  28. 28. Continuous Deployment ! Also known as Continuous Delivery. ! A System of Software ProductionCharacterized by Numerous SmallChanges the Production Environment,initiated by the author of the change.You change it, you push it to prod.
  29. 29. Deployment != Feature Release
  30. 30. "Writing Software"! Software Developers think their job iswriting software.! And so, they love to make things perfectbefore anyone else sees it.! Impolite: "data hiding"! code is hiding on developers computer! or on some branch! in other words invisible until its ready.
  31. 31. Actually! The software engineers job is actuallywriting running software, that works well.! This idea is so alien, that companies haveto remind the engineers of this.
  32. 32. Rackspace Haikuwriting code is hard
if you cannot deployit does not matter@paulvx from DevOpsDays Austin 2013
  33. 33. Facebook "Move Fast and 
Break Things....Except "Push" (deployment system)via
  34. 34. Etsy MantraJust Ship
  35. 35. Todays goal! but for today the goal is getting thedeveloper to care about their code
in production.! If you dont have that, I dont think you canreally solve security problems.
  36. 36. How does this work?! Really? Developers push their own codeout?! How is this not a disaster.! How is this not a security disaster?
  37. 37. The Deploy Button! What is you had a button that said
 "DEPLOY"! That pushed to production, whatever iscurrent in your source control system.! And took about a minute! The change and who pressed the button islogged, but thats it.
  38. 38. Part 1: Fear! No one is going to push it ;-)! Meanwhile code is piling upReal example: A new hire I had at Etsy was afraid 
of deploying an HTML change that they made. "But I dont want to break the site!"
  39. 39. Part 2: First Push! Someone brave will press the button! And very likely the site will explode, and arollback will need to be done.! Theyll know since someone else will havetold them.
  40. 40. Part 3: With Graphs! Lets get all those operational graphs outin the open. And put them right next tothe button.
  41. 41. Part 4: Push #2! Repush! Site might still explode! But the developer is aware and canrollback.
  42. 42. Take 5: Isolation! Hmmm, the developer notices that in thechange set, a million things are going out.! Maybe just pushing out a smaller changewill help isolate the issue.
  43. 43. Take 6: Success!! Yes, the developer just pushed out somecode and made the site better.! The secret about continuous deploymentis small changes that can be easilyunderstood.
  44. 44. Take 7: Dark Pushes! Now we got some bugs fixed, lets push afeature.! First lets push out all the supporting files.Since they arent being called, they donothing and are safe to push out.! Now everyone can see them
  45. 45. Take 8: Getting the feature live! Instead of "all at once", we slowly ramp upa feature.! if (user_id % 20 == 0):
do new feature; ! we change change the percentage easilywith another code push.! or turn it down. Much nicer change log.! While the site didnt explode, its hard tosee if the feature is being used or not.
  46. 46. Take 9: Application Level Graphs! Allow developers to instrument their codeso they can see what is happening inproduction.! Enter StatsD and other UDP-based tools! Enter centralized logging and in-application method to make it easy to logproblems.
  47. 47. Take 10: Communication! So far good for one developer.! To scale up, youll need a system to allowdevelopers.! IRC-like tools work well (e.g. "the pushchannel") – skype, jabber, hangouts, etc
  48. 48. Along the way! Expose production logs to developers! Add in a staging-step where the codegoes to faction of the cluster, sodevelopers can test with real traffic! Try to make development closer to prod.! Make "smoke tests" to catch basic errors! Add syntax checkers to eliminate obviousissues. ! Use static analysis to find bugs
  49. 49. Mistakes will happen! Do postmortem analysis! Everyone thought they were doing theright thing at the time.! "How can the environment be changed toprevent this" and build tools to enforce it.! (Rarely can you truly change people)
  50. 50. Sounds good, 
but what about...
  51. 51. That guy who pushes at 3am! Courtesy and convention will convergevery quickly when the site goes down at3am and the developer starts gettingcalls ;-)! Of hours pushes of course can happen,when they notify operations.
  52. 52. What About Code Reviews?! Yes, please do them.! Nothing here prevents code reviews.! In fact code reviews are easier since! they are small! they are in mainline not some branch
  53. 53. What about Security Reviews! Please do them.! Nothing here eliminates architecturalplanning or review.! This actually doesnt change the SDLCvery much.
  54. 54. What about Agile Methods! (everyone seems to have a different idea ofwhat Agile is but..)! Agile methodologies typically work toimprove the business spec / developmentcycle. (are you building what the customerwants)! But doesnt address code deployment.! They are complimentary practices.
  55. 55. What about Customer Service?! "Dont they freak out with all thechanges?"! Remember: deployment != feature release! Most deployments do very little from thecustomer point of view! Feature releases (frequently controlled byramp-ups or flags) always needs to becoordinated with product and customerservice.
  56. 56. What about Compliance? PCI?! Let me tell you about compliance...! mechanism not policy
  57. 57. Back to Security
  58. 58. Obvious Benefit to Security! Security patches can go out quickly! You know this since they are now just partof a normal development cycle.
  59. 59. More Importantly! That Engineer who previously didn’t pushcode is now sensitized that their code hasconsequences and are responsive andempowered to fix it.! It’s amazing how interested engineersbecome in security when you findproblems with their code when they areable to fix quickly themselves.
  60. 60. Hack The Stack! A side effect of this you now have tools torepurpose for security and
monitoring of production! Note that most changes are not securityproblems.
  61. 61. Logging! Due to allow developers to see applicationlogging, its now very easy to instrumentthe application to log security events.! Or add logs to times when you are underattack.
  62. 62. Graphing! Make dashboards of! SQLi and XSS attacks! Every type of log-in failure! Core Dumps! Database Syntax Errors
  63. 63. Static Analysis ! You now have a place to insert them.! Work with QA group to add more codequality tests.
  64. 64. Post-Commit Checks! Alert on when sensitive areas of the codeare changed (auth)! Alert on crypto usage (why is developerusing MD5.. hmmm)! Alert your programming languages"dangerous functions"This allows you to engage the developer at the start of the cycle.
  65. 65. Faster is Better! You could do most of this in a normalrelease-cycle software lifecycle.! The difference is you are finding problemsat the start instead of 10m before thelaunch and telling everyone to stop.! The feedback loop works.
  66. 66. New Roles, Less Silos! Developers: works with operations! QA: works on building systems for testing,to empower others to write better tests! Release Engineering: tools to enable codeto flow faster! Security: in-house consultancy, secure-by-default architecture, monitoring
  67. 67. Getting Started
  68. 68. Goal: 50% reduction in deploy timestimes ! Whatever your state of deployment is, nomatter how many people are involved, nomatter how long it currently takes, make agoal of cutting it in half.! This is an easy sell to management just oncost basis.! Everything else flows from this.
  69. 69. Mechanism not Policy! Strive for the fastest deployment mechanism forpossible! But you define the "continuous" in ContinuousDeployment! Yes, Etsy was 60+ deploys per day, with each havingmultiple authors.! Current gig? we have rules of no more than 3 per weeksince our customer have asked for that, and onlydeployed at "low-tide"
  70. 70. In Other Contexts
  71. 71. In other contexts: Operations! How fast can you deploy OS changes toyou production environment?
  72. 72. In other contexts: IT! How fast can you deploy desktopchanges?
  73. 73. In other contexts: software product! here "production" might be getting codeinto the main branch and runningautomated build / test.! Its the flow of code: little changes vs big.
  74. 74. In other contexts: silicon! Continuous deployment already done forsilicon! wut?! Only small changes, with tests are allowedto be committed!! Big changes are rejected.
Learned the hard way that big changes arecompletely unmanageable
  75. 75. One more thing...
  76. 76. Your Attackers UseContinuous Deployment
  77. 77. Continuous DeploymentStart with 50% ReductionBuild the Deploy ButtonNick Galbreath ★ @ngalbreath ★ ★
  78. 78. © 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, itshould not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NOWARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.Fixing Security by Fixing Software Development Using Continuous DeploymentNick Galbreath ★ @ngalbreath ★ ★