Your SlideShare is downloading. ×
0
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Fixing security by fixing software development
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Fixing security by fixing software development

4,895

Published on

Fixing Security by Fixing Software Development Using Continuous Deployment …

Fixing Security by Fixing Software Development Using Continuous Deployment
Do you have an effective release cycle? Is your process long and archaic? Long release cycle are typically based on assumptions we haven't seen since the 1980s and require very mature organizations to implement successfully. They can also disenfranchise developers from caring or even knowing about security or operational issues. Attend this session to learn more about an alternative approach to managing deployments through Continuous Deployment, otherwise known as Continuous Delivery. Find out how small, but frequent changes to the production environment can transform an organization’s development process to truly integrate security. Learn how to get started with continuous deployment and what tools and process are needed to make implementation within your organization a (security) success.

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,895
On Slideshare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
9
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Fixing Security by Fixing Software DevelopmentUsing Continuous DeploymentNick Galbreath, VP Eng IPONWEB, http://www.iponweb.com/@ngalbreath http://www.client9.com/May 14-15, San Francisco, California
  • 2. Follow AlongLatest version posted online:  http://slidesha.re/144OIv3 http://www.client9.com/
  • 3. Nick Galbreath (@ngalbreath)! Spoken at Black Hat, DEFCON, OWASP! Author of libinjection: advanced SQLidetection used in > 2 WAFs, 1 Honeypot! Book on cryptography! but really...! Engineering Management and SoftwareDevelopment for high growth startups.! Personal site http://www.client9.com/
  • 4. IPONWEB! customized online advertisinginfrastructure and exchanges! engineering offices in Moscow, withbusiness offices in London, New York andTokyo.
  • 5. Original AbstractFixing Security by Fixing Software Development Using ContinuousDeploymentDo you have an effective release cycle? Is your process long and archaic? Longrelease cycle are typically based on assumptions we havent seen since the1980s and require very mature organizations to implement successfully. Theycan also disenfranchise developers from caring or even knowing about securityor operational issues. Attend this session to learn more about an alternativeapproach to managing deployments through Continuous Deployment, otherwiseknown as Continuous Delivery. Find out how small, but frequent changes to theproduction environment can transform an organization’s development processto truly integrate security. Learn how to get started with continuous deploymentand what tools and process are needed to make implementation within yourorganization a (security) success.
  • 6. Well thats a bold statement..."Fixing Security by Fixing DevelopmentUsing Continuous Deployment"
  • 7. and heres anotherFor web applications, our release-basedsoftware development lifecycle is stillbased on a pre-Internet model and isharmful to organizations and
particularly harmful for security.
  • 8. What needs fixing?! SQLi dropped from #8 to #14 in the latestWhite Hat "The State of Web Security"report. Good news, right?! This means SQLi is only 7% of websites.Thats 1 in 15. And this is #14vulnerability!! And time to fix was on average 196 days. 
Thats embarrassing.Veracode claims 32% of incoming web applications have SQLi
https://info.veracode.com/state-of-software-security-report-volume5.htmlhttps://reg.whitehatsec.com/WPstats0513
  • 9. Even worse...! Number 1 driver to fix security problems...
compliance.! Number 1 reason to not fix security is...
compliance.! Not.. ! keeping our employees and customers safe! protecting corporate interests.! improving quality! being good at what we do.
  • 10. Security Products #1 .. in security bugsVeraCode: State ofSoftware Security, V4December 2011
  • 11. Lets Just Give Up! “You could spend all your resources chasing suchthings as this,” William Ribich, the former president ofTechnology Solutions Group [ QintelIQ ], said in aninterview in January. Ribich, who retired in November 2009,shortly after the discovery of a major data theft, said heneeded to balance the uncertain risk that the hackers coulduse what they stole against a growing shopping list ofsecurity products and consulting fees.! "You finally have to reach a point where you say
’let’s move on,’” he said. http://www.bloomberg.com/news/2013-05-01/china-cyberspies-outwit-u-s-stealing-military-secrets.html
  • 12. I would call that broken! But preventing SQLi isnt a technicallyhard problem.! And most security patches are very small.! How did we get here?
  • 13. Software Product Model! Code flows between functional groups! Product Managers spec code! Engineers write code! QA engineers tests code! Release engineers package code! Operations runs code! ... and Security does something too.
  • 14. High Distribution Cost! The Software Product Model is designedfor software where the cost of distributionis high. "High" might be financial, risk,time, resources, customer annoyance.! Retail, physical product, CD/DVD! Embedded of Exotic Hardware! Safety, Medical or Defense Systems! Operating Systems (desk or phone)! Homework (1-time deploy)
  • 15. Software Product Lifecycle
  • 16. Change to Production
  • 17. Web Applications Year 2000! Mostly followed Software Product Modelsince thats all we knew.! High barrier to entry! Specialized Hardware, Software andPeople needed to get started.! Lots of engineering needed to keep thingsrunning.! (side note: CERT/CVE started in 1999)
  • 18. True Story #1 ! "Cant push out the spelling error fix – itstoo risky"! "That code as already been through QA,its locked down."! "Product has to prioritize that change, elsewe arent touching it."
  • 19. True Story #2! Well do an iteration, where we try to fix asmany things as possible.! This wont be a scheduled iteration, it willbe done because things are so bad! So the spelling error will get fixed... 
uhh, who knows when.
  • 20. Web Applications 2013 ! Almost no barrier to entry! Commodity hardware! Programming not that hard! Scaling problems can 
be mostly outsourced
(mostly)
  • 21. Cost of Distribution 2013! Frequently no compile step
or its very fast.! Moving to production a few kilobytes ormegabytes of code over 1Gbps, 10Gbpslink.! In other words... free
  • 22. Failure is very different however! Most web applications are data-driven.! Frequently have social features, APIs,user-generated content.! Failures might be due to algorithmicproblems... but...! Most likely to due to user input, bad datain database or operational load.! this means data in past can causeproblems in the future.
  • 23. Releases and Problems! When a web-release goes out, and hasproblems....! Next week is spent tracking down whochanged what, where.! Re-QA! Re-Push! meanwhile new code is piling up.
  • 24. When SPM meets Web Apps! A long time between code being writtenand code being released.! Might be weeks or months! Feedback loop between code-in-dev andcode-in-production is broken! When security or bug reports come in, theauthor is likely on a different project.
  • 25. Hypothesis! It is impossible to simulate the productionenvironment in development, either due tooperational differences or data differences.! No amount of QA or Security Testing canprove you dont have bugs, vulnerabilities,or cause severe operational problems.! You have bugs and vulnerabilities,
right now, in your application.
  • 26. Impedance Mismatch! Easy to write code, +! Long release cycles +! Security as an end-of-line or out of bandprocess ==! no one cares
  • 27. So the Answer is...! Going slower? Im sure your boss will lovethat suggestion! More steps and process? In other words,slower.! Asking for more people? Sure but goodluck hiring them. Doesnt scale.! Asking for more products? Since theothers have worked so well.
  • 28. Continuous Deployment ! Also known as Continuous Delivery. ! A System of Software ProductionCharacterized by Numerous SmallChanges the Production Environment,initiated by the author of the change.You change it, you push it to prod.
  • 29. Deployment != Feature Release
  • 30. "Writing Software"! Software Developers think their job iswriting software.! And so, they love to make things perfectbefore anyone else sees it.! Impolite: "data hiding"! code is hiding on developers computer! or on some branch! in other words invisible until its ready.
  • 31. Actually! The software engineers job is actuallywriting running software, that works well.! This idea is so alien, that companies haveto remind the engineers of this.
  • 32. Rackspace Haikuwriting code is hard
if you cannot deployit does not matter@paulvx from DevOpsDays Austin 2013
  • 33. Facebook "Move Fast and 
Break Things....Except "Push" (deployment system)via http://mitadmissions.org/blogs/entry/move_fast_and_break_things
  • 34. Etsy MantraJust Ship
  • 35. Todays goal! but for today the goal is getting thedeveloper to care about their code
in production.! If you dont have that, I dont think you canreally solve security problems.
  • 36. How does this work?! Really? Developers push their own codeout?! How is this not a disaster.! How is this not a security disaster?
  • 37. The Deploy Button! What is you had a button that said
 "DEPLOY"! That pushed to production, whatever iscurrent in your source control system.! And took about a minute! The change and who pressed the button islogged, but thats it.
  • 38. Part 1: Fear! No one is going to push it ;-)! Meanwhile code is piling upReal example: A new hire I had at Etsy was afraid 
of deploying an HTML change that they made. "But I dont want to break the site!"
  • 39. Part 2: First Push! Someone brave will press the button! And very likely the site will explode, and arollback will need to be done.! Theyll know since someone else will havetold them.
  • 40. Part 3: With Graphs! Lets get all those operational graphs outin the open. And put them right next tothe button.
  • 41. Part 4: Push #2! Repush! Site might still explode! But the developer is aware and canrollback.
  • 42. Take 5: Isolation! Hmmm, the developer notices that in thechange set, a million things are going out.! Maybe just pushing out a smaller changewill help isolate the issue.
  • 43. Take 6: Success!! Yes, the developer just pushed out somecode and made the site better.! The secret about continuous deploymentis small changes that can be easilyunderstood.
  • 44. Take 7: Dark Pushes! Now we got some bugs fixed, lets push afeature.! First lets push out all the supporting files.Since they arent being called, they donothing and are safe to push out.! Now everyone can see them
  • 45. Take 8: Getting the feature live! Instead of "all at once", we slowly ramp upa feature.! if (user_id % 20 == 0):
do new feature; ! we change change the percentage easilywith another code push.! or turn it down. Much nicer change log.! While the site didnt explode, its hard tosee if the feature is being used or not.
  • 46. Take 9: Application Level Graphs! Allow developers to instrument their codeso they can see what is happening inproduction.! Enter StatsD and other UDP-based tools! Enter centralized logging and in-application method to make it easy to logproblems.
  • 47. Take 10: Communication! So far good for one developer.! To scale up, youll need a system to allowdevelopers.! IRC-like tools work well (e.g. "the pushchannel") – skype, jabber, hangouts, etc
  • 48. Along the way! Expose production logs to developers! Add in a staging-step where the codegoes to faction of the cluster, sodevelopers can test with real traffic! Try to make development closer to prod.! Make "smoke tests" to catch basic errors! Add syntax checkers to eliminate obviousissues. ! Use static analysis to find bugs
  • 49. Mistakes will happen! Do postmortem analysis! Everyone thought they were doing theright thing at the time.! "How can the environment be changed toprevent this" and build tools to enforce it.! (Rarely can you truly change people)
  • 50. Sounds good, 
but what about...
  • 51. That guy who pushes at 3am! Courtesy and convention will convergevery quickly when the site goes down at3am and the developer starts gettingcalls ;-)! Of hours pushes of course can happen,when they notify operations.
  • 52. What About Code Reviews?! Yes, please do them.! Nothing here prevents code reviews.! In fact code reviews are easier since! they are small! they are in mainline not some branch
  • 53. What about Security Reviews! Please do them.! Nothing here eliminates architecturalplanning or review.! This actually doesnt change the SDLCvery much.
  • 54. What about Agile Methods! (everyone seems to have a different idea ofwhat Agile is but..)! Agile methodologies typically work toimprove the business spec / developmentcycle. (are you building what the customerwants)! But doesnt address code deployment.! They are complimentary practices.
  • 55. What about Customer Service?! "Dont they freak out with all thechanges?"! Remember: deployment != feature release! Most deployments do very little from thecustomer point of view! Feature releases (frequently controlled byramp-ups or flags) always needs to becoordinated with product and customerservice.
  • 56. What about Compliance? PCI?! Let me tell you about compliance...! mechanism not policy
  • 57. Back to Security
  • 58. Obvious Benefit to Security! Security patches can go out quickly! You know this since they are now just partof a normal development cycle.
  • 59. More Importantly! That Engineer who previously didn’t pushcode is now sensitized that their code hasconsequences and are responsive andempowered to fix it.! It’s amazing how interested engineersbecome in security when you findproblems with their code when they areable to fix quickly themselves.
  • 60. Hack The Stack! A side effect of this you now have tools torepurpose for security and
monitoring of production! Note that most changes are not securityproblems.
  • 61. Logging! Due to allow developers to see applicationlogging, its now very easy to instrumentthe application to log security events.! Or add logs to times when you are underattack.
  • 62. Graphing! Make dashboards of! SQLi and XSS attacks! Every type of log-in failure! Core Dumps! Database Syntax Errors
  • 63. Static Analysis ! You now have a place to insert them.! Work with QA group to add more codequality tests.
  • 64. Post-Commit Checks! Alert on when sensitive areas of the codeare changed (auth)! Alert on crypto usage (why is developerusing MD5.. hmmm)! Alert your programming languages"dangerous functions"This allows you to engage the developer at the start of the cycle.
  • 65. Faster is Better! You could do most of this in a normalrelease-cycle software lifecycle.! The difference is you are finding problemsat the start instead of 10m before thelaunch and telling everyone to stop.! The feedback loop works.
  • 66. New Roles, Less Silos! Developers: works with operations! QA: works on building systems for testing,to empower others to write better tests! Release Engineering: tools to enable codeto flow faster! Security: in-house consultancy, secure-by-default architecture, monitoring
  • 67. Getting Started
  • 68. Goal: 50% reduction in deploy timestimes ! Whatever your state of deployment is, nomatter how many people are involved, nomatter how long it currently takes, make agoal of cutting it in half.! This is an easy sell to management just oncost basis.! Everything else flows from this.
  • 69. Mechanism not Policy! Strive for the fastest deployment mechanism forpossible! But you define the "continuous" in ContinuousDeployment! Yes, Etsy was 60+ deploys per day, with each havingmultiple authors.! Current gig? we have rules of no more than 3 per weeksince our customer have asked for that, and onlydeployed at "low-tide"
  • 70. In Other Contexts
  • 71. In other contexts: Operations! How fast can you deploy OS changes toyou production environment?
  • 72. In other contexts: IT! How fast can you deploy desktopchanges?
  • 73. In other contexts: software product! here "production" might be getting codeinto the main branch and runningautomated build / test.! Its the flow of code: little changes vs big.
  • 74. In other contexts: silicon! Continuous deployment already done forsilicon! wut?! Only small changes, with tests are allowedto be committed!! Big changes are rejected.
Learned the hard way that big changes arecompletely unmanageable
  • 75. One more thing...
  • 76. Your Attackers UseContinuous Deployment
  • 77. Continuous DeploymentStart with 50% ReductionBuild the Deploy ButtonNick Galbreath ★ @ngalbreath ★ nickg@client9.com ★ http://www.client9.com/http://slidesha.re/144OIv3
  • 78. © 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, itshould not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NOWARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.Fixing Security by Fixing Software Development Using Continuous DeploymentNick Galbreath ★ @ngalbreath ★ http://www.client9.com ★ nickg@client9.com

×