Faster Secure Software Development with Continuous Deployment - PH Days 2013


Published on

Published in: Technology, Business
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Faster Secure Software Development with Continuous Deployment - PH Days 2013

  1. 1. Faster Secure SoftwareDevelopment with 
Continuous DeploymentNick GalbreathIPONWEBnickg@iponweb.netPHDays May 24, 2013Moscow, Russia
  2. 2. Follow AlongLatest version posted online:
  3. 3. Nick Galbreath (@ngalbreath)! Spoken at Black Hat, DEFCON, OWASP! Book on cryptography! but really...! Engineering Management and SoftwareDevelopment for high growth startups.! Personal site
  4. 4. libinjection! Author of libinjection! A very different way of doing SQLi detection! Right now in another room, VladimirVorontsov is showing how to bypass it (tobe fixed shortly)! Check it out and file bugs on github! Found a bypass?What databaseand query is neededto exploit it?RU ♥SQLi
  5. 5. IPONWEB! customized online advertising infrastructureand exchanges! engineering offices in Moscow, withbusiness offices in London, New York andTokyo.! YEAH IN MOSCOW! YEAH WE ARE HIRING! Send email to
  6. 6. Well thats a bold statement...Fixing Security by
Fixing Development
Using Continuous Deployment
  7. 7. and heres anotherFor web applications, our release-basedsoftware development lifecycle is stillbased on a pre-Internet model and isharmful to organizations and
particularly harmful for security.
  8. 8. What needs fixing?! SQLi dropped from #8 to #14 in the latestWhite Hat "The State of Web Security"report. Good news, right?! This means SQLi is only 7% of websites.Thats 1 in 15.
And this is the #14 vulnerability!! And time to fix was on average 196 days. 
Thats embarrassing.Veracode claims 32% of incoming web applications have SQLi
  9. 9. Even worse...! Number 1 driver to fix securityproblems...
compliance.! Number 1 reason to not fix security is...
compliance.! Not.. ! keeping our employees and customers safe! protecting corporate interests.! improving quality! being good at what we do.
  10. 10. Security Products #1 .. in security bugsVeraCode: State ofSoftware Security, V4December 2011Security Product74% Fail Rate
  11. 11. Lets Just Give Up! “You could spend all your resources chasingsuch things as this,” William Ribich, the formerpresident of Technology Solutions Group[ QinteliQ ], said in an interview in January.Ribich, who retired in November 2009, shortlyafter the discovery of a major data theft, said heneeded to balance the uncertain risk that thehackers could use what they stole against agrowing shopping list of security products andconsulting fees.! "You finally have to reach a point where yousay ’let’s move on,’” he said.^^^ The Russian and Chinese hackers did not move on ^^^^.
  12. 12. I would call that broken! But preventing SQLi isnt a technicallyhard problem.! And most security patches are very small.! How did we get here?
  13. 13. Software Product Model! Code flows between functional groups! Product Managers spec code! Engineers write code! QA engineers tests code! Release engineers package code! Operations runs code! ... and Security does something too.
  14. 14. High Distribution Cost! The Software Product Model is designed forsoftware where the cost of distribution ishigh. "High" might be financial, risk, time,resources, customer annoyance.! Retail, physical product, CD/DVD! Embedded of Exotic Hardware! Safety, Medical or Defense Systems! Operating Systems (desk or phone)! Homework (1-time deploy)
  15. 15. Software Product Lifecycle
  16. 16. Change to Production
  17. 17. Web Applications Year 2000! Mostly followed Software ProductModel since thats all we knew.! High barrier to entry! Specialized Hardware, Software andPeople needed to get started.! Lots of engineering needed to keepthings running.! (side note: CERT/CVE started in1999)
  18. 18. True Story #1 ! "Cant push out the spelling error fix– its too risky"! "That code as already been throughQA, its locked down."! "Product has to prioritize thatchange, else we arent touching it."
  19. 19. True Story #2! Well do an iteration, where we try to fixas many things as possible.! This wont be a scheduled iteration,
it will be done because so many thingsare piled up.! So the spelling error will get fixed... 
uhh, who knows when.
  20. 20. Web Applications 2013 ! Almost no barrier to entry! Commodity hardware! Programming not that hard! Scaling problems can 
be mostly outsourced
  21. 21. Cost of Distribution 2013! Frequently no compile step
or its very fast.! Moving to production a fewkilobytes or megabytes of codeover 1Gbps, 10Gbps link.! In other words... free
  22. 22. Failure is very different however! Most web applications are data-driven.! Frequently have social features, APIs,user-generated content.! Failures might be due to algorithmicproblems... but...! Most likely to due to user input, bad datain database or operational load.! this means data in past can causeproblems in the future.
  23. 23. Releases and Problems! When a web-release goes out, andhas problems....! Next week is spent tracking downwho changed what, where.! Re-QA! Re-Push! meanwhile new code is piling up.
  24. 24. When SPM meets Web Apps! A long time between code being writtenand code being released.! Might be weeks or months! Feedback loop between code-in-devand code-in-production is broken! When security or bug reports come in,the author is likely on a different project.
  25. 25. Hypothesis! It is impossible to simulate the productionenvironment in development, either due tooperational differences or datadifferences.! No amount of QA or Security Testing canprove you dont have bugs, vulnerabilities,or cause severe operational problems.! You have bugs and vulnerabilities,
right now, in your application.
  26. 26. Impedance Mismatch! Easy to write code, +! Long release cycles +! Security as an end-of-line or outof band process ==! no one cares! Something is going to break,and most people dont care.
  27. 27. Security Problems Will HappenMTTF ≪ ∞
  28. 28. So the Answer is...! Going slower? Im sure your bosswill love that suggestion! More steps and process? In otherwords, slower.! Asking for more people? Sure butgood luck hiring them. Doesntscale.! Asking for more products? Sincethe others have worked so well.
  29. 29. Continuous Deployment ! Also known as Continuous Delivery. ! A System of Software ProductionCharacterized by Numerous SmallChanges the Production Environment,initiated by the author of the change.You change it, you push it to prod.
  30. 30. Deployment != Feature ReleasetimechangeNew code goes out all the time.New features get turned on ina separate process.
  31. 31. "Writing Software"! Software Developers think their job iswriting software.! And so, they love to make things perfectbefore anyone else sees it.! Impolite: "data hiding"! code is hiding on developers computer! or on some branch! in other words invisible until its ready.
  32. 32. Actually! The software engineers job is actuallywriting running software, that works well.! This idea is so alien, that companies haveto remind the engineers of this.
  33. 33. Rackspace Haikuwriting code is hard
if you cannot deployit does not matter@paulvx from DevOpsDays Austin 2013
  34. 34. Facebooks Analog Labs Poster"Move Fast and 
Break Things....Except "Push" (deployment system)via
  35. 35. Etsy MantraJust Ship
  36. 36. Todays goal! but for today the goal is getting thedeveloper to care about their code
in production.! If you dont have that, I dont think you canreally solve security problems.
  37. 37. How does this work?! Really? 
Developers push their own codeout?! How is this not a disaster.! How is this not a security disaster?
  38. 38. The Deploy Button! What is you had a button that said
 "DEPLOY"! That pushed to production, whateveris current in your source controlsystem.! And took about a minute! The change and who pressed thebutton is logged, but thats it.
  39. 39. Part 1: Fear! No one is going to pushit ;-)! Meanwhile code is pilingupReal example: A new hire I had at Etsywas afraid of deploying an HTML changethat they made. "But I dont want to break the site!"
  40. 40. Part 2: First Push! Someone brave will press the button! And very likely the site will explode,and a rollback will need to be done.! Theyll know since someone else willhave told them.
  41. 41. Part 3: With Graphs! Lets get all those operational graphsout in the open. And put them rightnext to the button.
  42. 42. Part 4: Push #2! Repush! Site might still explode! But the developer is aware andcan rollback.
  43. 43. Take 5: Isolation! Hmmm, the developer notices that in thechange set, a million things are going out.! Maybe just pushing out a smaller changewill help isolate the issue.
  44. 44. Take 6: Success!! Yes, the developer just pushed outsome code and made the site better.! The secret about continuousdeployment is small changes thatcan be easily understood.
  45. 45. Take 7: Dark Pushes! Now we got some bugs fixed, lets pusha feature.! First lets push out all the supportingfiles. Since they arent being called,they do nothing and are safe to pushout.! Now everyone can see them
  46. 46. Take 8: Getting the feature live! Instead of "all at once", we slowly ramp up afeature.! if (user_id % 20 == 0):
do new feature; ! we change change the percentage easily withanother code push.! or turn it down. Much nicer change log.! While the site didnt explode, its hard to see ifthe feature is being used or not.
  47. 47. Take 9: Application Level Graphs! Allow developers to instrument theircode so they can see what ishappening in production.! Enter StatsD and otherUDP-based tools! Enter centralized logging and in-application method to make it easyto log problems.
  48. 48. Take 10: Communication! So far good for one developer.! To scale up, youll need a system to allowdevelopers.! IRC-like tools work well (e.g. "the pushchannel") – skype, jabber, hangouts, etc
  49. 49. Along the way! Expose production logs to developers! Add in a staging-step where the codegoes to faction of the cluster, sodevelopers can test with real traffic! Try to make development closer to prod.! Make "smoke tests" to catch basic errors! Add syntax checkers to eliminate obviousissues. ! Use static analysis to find bugs
  50. 50. Mistakes will happen! Do postmortem analysis! Everyone thought they were doingthe right thing at the time.! "How can the environment bechanged to prevent this" and buildtools to enforce it.! (Rarely can you truly change people)
  51. 51. Sounds good, 
but what about...
  52. 52. That guy who pushes at 3am! Courtesy and convention willconverge very quickly when the sitegoes down at 3am and thedeveloper starts getting calls ;-)! Of hours pushes of course canhappen, when they notify operations.
  53. 53. What About Code Reviews?! Yes, please do them.! Nothing here prevents code reviews.! In fact code reviews are easier since! they are small! they are in mainline not some branch
  54. 54. What about Security Reviews! Please do them.! Nothing here eliminatesarchitectural planning or review.! This actually doesnt change theSDLC very much.
  55. 55. What about Agile Methods! (everyone seems to have a different ideaof what Agile is but..)! Agile methodologies typically work toimprove the business spec / developmentcycle. (are you building what thecustomer wants)! But doesnt address code deployment.! They are complimentary practices.
  56. 56. What about Customer Service?! "Dont they freak out with all thechanges?"! Remember: deployment != feature release! Most deployments do very little from thecustomer point of view! Feature releases (frequently controlled byramp-ups or flags) always needs to becoordinated with product and customerservice.
  57. 57. What about Compliance? PCI?! Let me tell you about compliance...! mechanism not policy! compliance is a lot easier when its doneevery day instead of a once-a-year audit.
  58. 58. Back to Security
  59. 59. Obvious Benefit to Security! Security patches can go out quickly! You know this since they are nowjust part of a normal developmentcycle and code goes out regularly.! Why not clear out those low-prioritysecurity problems?
  60. 60. More Importantly! That Engineer who previously did notpush code is now sensitized that theircode has consequences and areresponsive and empowered to fix it.! It’s amazing how interested engineersbecome in security when you findproblems with their code when they areable to fix it quickly themselves.
  61. 61. New Security Math! Instead of focusing only on increasing MTTF,which will never be infinite! more firewalls, more process, more magic! You can focus on how fast can you detect faults,and how fast can you fix them.! How low can you go?! MTTD - Mean time to Detect! MTTR - Mean time to Repair
  62. 62. Hack The Stack! A side effect of this you now havetools to repurpose for security and
monitoring of production! Note that most changes are notsecurity problems.
  63. 63. Logging! Due to allow developers to seeapplication logging, its now veryeasy to instrument the application tolog security events.! Or add logs to times when you areunder attack.
  64. 64. Graphing! Make dashboards of! SQLi and XSS attacks! Every type of log-in failure! Core Dumps! Database Syntax Errors
  65. 65. Static Analysis ! You now have a place to insert them.! Work with QA group to add more codequality tests.
  66. 66. Post-Commit Checks! Alert on when sensitive areas of the codeare changed (auth, login)! Alert on crypto usage (why is developerusing MD5.. hmmm)! Alert on your programming languages"dangerous functions"This allows you to engage the developer at the start of the cycle.
  67. 67. Faster is Better! You could do most of this in a normalrelease-cycle software lifecycle.! The difference is you are findingproblems at the start instead of 10mbefore the launch and tellingeveryone to stop.! The feedback loop works.
  68. 68. New Roles, Less Silos! Developers: works with operations! QA: works on building systems fortesting, to empower others to writebetter tests! Release Engineering: tools to enablecode to flow faster! Security: in-house consultancy,secure-by-default architecture,monitoring
  69. 69. GettingStarted
  70. 70. Goal: 50% reduction in deploy time! Whatever your state of deployment is,no matter how many people areinvolved, no matter how long itcurrently takes, make a goal of cuttingit in half.! This is an easy sell to managementjust on cost basis.! Everything else flows from this.
  71. 71. Mechanism not Policy! Strive for the fastest deploymentmechanism for possible! But you define the "continuous" inContinuous Deployment! Yes, Etsy was 60+ deploys per day, witheach having multiple authors.! Current gig? we have rules of no morethan 3 per week since our customer haveasked for that, and only deployed at"low-tide"
  72. 72. In Other Contexts
  73. 73. In other contexts: Operations! How fast can you deploy OS changesto you production environment?! How fast can you deploy routerchanges?! How fast can you deploy patches tothe desktopYou probably dont do it that often sinceits really painful and time consuming!Thats exactly the problem.
  74. 74. In other contexts: software product! here "production" might be getting codeinto the main branch and runningautomated build / test.! Its the flow of code: little changes vs big.
  75. 75. In other contexts: silicon! Continuous deployment already done forsilicon! wut?! Only small changes, with tests areallowed to be committed!! Big changes are rejected.
Learned the hard way that big changesare completely unmanageable
  76. 76. One more thing...
  77. 77. Your Attackers UseContinuous Deployment
  78. 78. Continuous DeploymentStart with 50% ReductionBuild the Deploy Button
  79. 79. Thanksnickg@client9.com