DevOpsSec   Applying DevOps Principles to SecurityNick Galbreath nickg@etsy.com @ngalbreath   DevOpsDays, Austin Texas, Ap...
Slides! Video!• Originally presented on April 3, 2012• Latest Slides! Streaming Video!  http://client9.com/20120403• Relat...
whoami• Development background• Lots o’ startups, book, patents,blahblahblah• Director of Engineering at Etsy covering • S...
ContextMy biases for this talk is (Web) ApplicationSecurity, not classic Network Security or ITSecurity.
Double-click to edit• Double-click to edit           uhhhhhh....
Uhhhhhhh    What are“DevOps Principles”
Blah blah blah• Decentralization• Shared Resources• Risk based management• Catholic vs protestant methologies• Whitelist v...
Trust But Verify
...with theacknowledgement that• We are working in a complex system• And in complex systems failure happens• And failure c...
What does this mean for....People?Processes (workflow)?Machines?
An Only Slightly   Contrived Example• I trust MCR to run our network• I can verify this by looking at our dataporn• He tru...
Uhhhh....Why DevOpsSec and not DevOpsFoo?
Squeezed from Both Sides  Unreviewed Code going out,  Untrusted Data coming in DATA          UGH           CODE Makes stab...
Latent ProblemsThere are operational problems right nowjust not manifested.There are security problems right now justnot e...
Cultural Problems• Both have severe failure causes• Both Ops and Security have a “say no”  perception• “Operations” and “S...
Ok, back to theregularly scheduled   programming
DevOpsSec                              E 2   Applying DevOps Principles to Security            A KNick Galbreath nickg@ets...
MTTRMean Time To Resolve
SHIT HAPPENSSecurity problems will occur
How Fast Can You    Deploy or Rebuild• Your Firewall,VPN, Load Balancer• Your Operating System, Critical Servers• Your Dat...
Being able to deploy   quickly is my #1   security featureThis implies a standardized, automatedsystem and configuration ma...
I Call BullshitDoesn’t the rapid rate ofchange in a continuousintegration environmentmean things are less secure?         ...
We’ll rush that security fix.It will go out in next releasein about 6 weeks.                           former vendor at Etsy
MTTDMean Time To Detect
It’s ok if we have a few extrafiremen waiting around incase there is a fire                     I’m more concerned we       ...
Segmentation Faults• Why is your server falling over?• From the same IP address.• Over and over    Maybe time to patch? Al...
Database Syntax Errors          Almost game over here.    Whose is responsible for these anyways?Demand zero-tolerance for...
SQLi AttacksThe shittest check that works.Will undercount by at least 10xProTip #1: using regexp for sqli is failProTip #2...
And Graph ItSecurity is no longer a binary event.
Got that?Security is not a  binary event.   You are beingattacked constantly.THIS IS AWESOME
Attack Driven Testing       Security testing in dev is hard. Use actual attacks/probes to guide testing.• Server 500 error...
TESTINGIf infrastructure is code, then doesn’t it need testing too?
assertyour production    environment             Who knew that          writing solid C code is           similar to runni...
assert this• This page is always SSL• This page requires sign-in• This page never is publicly available• Google never craw...
Reuse your unit test framework to test production config
Reuse yourcontinuous integration     environment       ClamAV. Yes, we antivirus           our source code.
Other Topics in DevOpsSec
Post Mortems• All security issues are “P1” or “P2” (fix  now, or fix by end of week)• Even for internal applications.• All g...
Hiring• Strict no asshole policy.• Security is a services business.• “Product Security” is in-house consultancy.• Can you ...
Extend the Perimeter• Working on a training program for both in-  house employees, contractors, and external  vendors.• “D...
Nick Galbreath nickg@etsy.com @ngalbreath        DevOpsDays Austin Tx 2012http://client9.com/20120403 nickg@client9.com
Upcoming SlideShare
Loading in...5
×

DevOpsSec: Appling DevOps Principles to Security, DevOpsDays Austin 2012

6,953

Published on

From DevOpsDays Austin 2012.


Video: http://www.ustream.tv/recorded/21568549

Published in: Technology
4 Comments
24 Likes
Statistics
Notes
No Downloads
Views
Total Views
6,953
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
148
Comments
4
Likes
24
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • DevOpsSec: Appling DevOps Principles to Security, DevOpsDays Austin 2012

    1. 1. DevOpsSec Applying DevOps Principles to SecurityNick Galbreath nickg@etsy.com @ngalbreath DevOpsDays, Austin Texas, April 3, 2012http://client9.com/20120403 nickg@client9.com
    2. 2. Slides! Video!• Originally presented on April 3, 2012• Latest Slides! Streaming Video! http://client9.com/20120403• Related interview: http://youtu.be/Afd0u5DGxr8• Original video stream: http://www.ustream.tv/recorded/21568549
    3. 3. whoami• Development background• Lots o’ startups, book, patents,blahblahblah• Director of Engineering at Etsy covering • Security, Fraud, Biz Analytics, Email Infra, Internal Systems, and everything else not www.etsy.com “Enterprise” • Second time working with Allspaw!• “Oh you mean there is a name for this?”
    4. 4. ContextMy biases for this talk is (Web) ApplicationSecurity, not classic Network Security or ITSecurity.
    5. 5. Double-click to edit• Double-click to edit uhhhhhh....
    6. 6. Uhhhhhhh What are“DevOps Principles”
    7. 7. Blah blah blah• Decentralization• Shared Resources• Risk based management• Catholic vs protestant methologies• Whitelist vs. blacklist mentality• Transparency.
    8. 8. Trust But Verify
    9. 9. ...with theacknowledgement that• We are working in a complex system• And in complex systems failure happens• And failure can happen when everyone does nothing wrong• And given this, how can one increase reward and reduce risk for the business
    10. 10. What does this mean for....People?Processes (workflow)?Machines?
    11. 11. An Only Slightly Contrived Example• I trust MCR to run our network• I can verify this by looking at our dataporn• He trusts me that when things go wrong, the graphs won’t be used to burn him.• He can verify this by... seeing our Post Mortems in action (they are open at Etsy)
    12. 12. Uhhhh....Why DevOpsSec and not DevOpsFoo?
    13. 13. Squeezed from Both Sides Unreviewed Code going out, Untrusted Data coming in DATA UGH CODE Makes stability and responsibility “complicated”, even more so if there are walls between groups.
    14. 14. Latent ProblemsThere are operational problems right nowjust not manifested.There are security problems right now justnot exploited.
    15. 15. Cultural Problems• Both have severe failure causes• Both Ops and Security have a “say no” perception• “Operations” and “Security” are services groups but frequently not viewed as such
    16. 16. Ok, back to theregularly scheduled programming
    17. 17. DevOpsSec E 2 Applying DevOps Principles to Security A KNick Galbreath nickg@etsy.com @ngalbreath DevOpsDays, Austin Texas, April 3, 2012http://client9.com/20120403 nickg@client9.com
    18. 18. MTTRMean Time To Resolve
    19. 19. SHIT HAPPENSSecurity problems will occur
    20. 20. How Fast Can You Deploy or Rebuild• Your Firewall,VPN, Load Balancer• Your Operating System, Critical Servers• Your Database, server, schema, data• Your Application, patches• Any other configuration file in a consistent, sane manner
    21. 21. Being able to deploy quickly is my #1 security featureThis implies a standardized, automatedsystem and configuration management.
    22. 22. I Call BullshitDoesn’t the rapid rate ofchange in a continuousintegration environmentmean things are less secure? Well compare this to....
    23. 23. We’ll rush that security fix.It will go out in next releasein about 6 weeks. former vendor at Etsy
    24. 24. MTTDMean Time To Detect
    25. 25. It’s ok if we have a few extrafiremen waiting around incase there is a fire I’m more concerned we won’t know there is a fire until the house is burnt down Conversation between Chad Dickerson and Nick Galbreath, Etsy 2011
    26. 26. Segmentation Faults• Why is your server falling over?• From the same IP address.• Over and over Maybe time to patch? Also check out your server 500 errors.
    27. 27. Database Syntax Errors Almost game over here. Whose is responsible for these anyways?Demand zero-tolerance for database syntax errors.
    28. 28. SQLi AttacksThe shittest check that works.Will undercount by at least 10xProTip #1: using regexp for sqli is failProTip #2: write a unit test for this.
    29. 29. And Graph ItSecurity is no longer a binary event.
    30. 30. Got that?Security is not a binary event. You are beingattacked constantly.THIS IS AWESOME
    31. 31. Attack Driven Testing Security testing in dev is hard. Use actual attacks/probes to guide testing.• Server 500 errors• Core Dumps Can you automate• CSRF Failures verification?• XSS Attempts• Login Failures• Password Resets
    32. 32. TESTINGIf infrastructure is code, then doesn’t it need testing too?
    33. 33. assertyour production environment Who knew that writing solid C code is similar to running a complex system. Writing Solid Code Steve Maguire 1993
    34. 34. assert this• This page is always SSL• This page requires sign-in• This page never is publicly available• Google never crawls this page• This page is not being routed by the CDN• This port is never open
    35. 35. Reuse your unit test framework to test production config
    36. 36. Reuse yourcontinuous integration environment ClamAV. Yes, we antivirus our source code.
    37. 37. Other Topics in DevOpsSec
    38. 38. Post Mortems• All security issues are “P1” or “P2” (fix now, or fix by end of week)• Even for internal applications.• All get a post-mortem• Great educational experience and knowledge transfer
    39. 39. Hiring• Strict no asshole policy.• Security is a services business.• “Product Security” is in-house consultancy.• Can you take people who are interested and train them? TBD. http://www.sans.org/ https://www.owasp.org/
    40. 40. Extend the Perimeter• Working on a training program for both in- house employees, contractors, and external vendors.• “Device Tune-Up Day” -- employees bring their home computers in for a tune-up.
    41. 41. Nick Galbreath nickg@etsy.com @ngalbreath DevOpsDays Austin Tx 2012http://client9.com/20120403 nickg@client9.com
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×