Building a Modern Security
Engineering Organization
zane@signalsciences.com
@zanelackey
Who	
  is	
  this	
  guy	
  anyway?	
  
•  Built and led the Etsy Security Team
– Spoiler alert: what this presentation is...
This talk is a collection of lessons learned
from building and adapting a security
team
For security teams, the world has changed
in fundamental ways:
–  Code deployment is now near-instantaneous
For security teams, the world has changed
in fundamental ways:
–  Code deployment is now near-instantaneous
–  Merging of ...
For security teams, the world has changed
in fundamental ways:
–  Code deployment is now near-instantaneous
–  Merging of ...
Near-instantaneous deployment?
A	
  technical	
  diagram	
  of	
  tradi7onal	
  waterfall	
  code	
  deployment	
  	
  
What is this shifting to?
Etsy pushes to production 30 times a day
on average
Constant iteration in production via
feature flags, ramp ups, A/B testing
But doesn’t the
rapid rate of
change mean
things are less
secure?!
Actually,	
  the	
  opposite	
  is	
  
true	
  
They key to realize is vulnerabilities occur
in all development methodologies
…But there’s no such thing as an out-of-
ban...
They key to realize is vulnerabilities occur
in all development methodologies
…But there’s no such thing as an out-of-
ban...
Compared to:
“We’ll rush that security fix. It will go out …
in about 6 weeks.”
- Former vendor at Etsy
What makes continuous deployment safe?
Source:	
  h<p://www.slideshare.net/mikebri<ain/advanced-­‐topics-­‐in-­‐con7nuous-­‐deployment	
  
The same culture of graphing and
monitoring inherent to continuous
deployment can be used for security too
Surface security info for everyone, not just
the security team
“Don’t treat security as a binary event”
- @ngalbreath
Building	
  a	
  (k-­‐)rad	
  culture	
  
*Mullets	
  sold	
  separately	
  	
  
In the shift to continuous deployment,
speed increases by removing
organizational blockers
Trying to make security a blocker means
you get routed around
Instead, the focus becomes on
incentivizing teams to reach out to security
Keys to incentivizing conversation:
– Don’t be a jerk. This should be obvious, but
empathy needs to be explicitly set as a...
Keys to incentivizing conversation:
– Don’t be a jerk. This should be obvious, but
empathy needs to be explicitly set as a...
Keys to incentivizing conversation:
– Coherently explain impact. “This would
allow all our user data to be compromised if
...
Keys to incentivizing conversation:
– Coherently explain impact. “This would
allow all our user data to be compromised if
...
Keys to incentivizing conversation:
– Take the false positive hit yourself. Don’t
send unverified issues to dev and ops
tea...
Keys to incentivizing conversation:
– Take the false positive hit yourself. Don’t
send unverified issues to dev and ops
tea...
Access	
  restric7ons	
  
Startups begin with a simple access
control policy: Everyone can access
everything
As organization grow there will be more
pressure to institute access policies
The key to remember is don’t take away
capabilities
Methodology:
1.  Figure out what capability is needed
2.  Build an alternate way to perform the
needed function in a safe ...
Methodology:
1.  Figure out what capability is needed
2.  Build an alternate way to perform the
needed function in a safe ...
Methodology:
1.  Figure out what capability is needed
2.  Build an alternate way to perform the
needed function in a safe ...
Methodology:
1.  Figure out what capability is needed
2.  Build an alternate way to perform the
needed function in a safe ...
EX: SSH access to production systems
Security policy goal: Eliminate unneeded
access to production systems
–  Why do developers do it? Ex: To view error logs
–...
Security policy goal: Eliminate unneeded
access to production systems
–  Why do developers do it? Ex: To view error logs
–...
Security policy goal: Eliminate unneeded
access to production systems
–  Why do developers do it? Ex: To view error logs
–...
Security policy goal: Eliminate unneeded
access to production systems
–  Why do developers do it? Ex: To view error logs
–...
Increasing	
  a<acker	
  cost	
  
Specifically, some thoughts on:
–  Bug Bounties
–  Attack simulations/pentesting
Bug	
  Boun7es	
  
Bug bounties are tremendously useful. If
you’re not working towards launching one,
strongly consider it.
Common concerns about launching a
bounty:
1.  Budgetary concerns. Money is almost
never the main motivation for researcher...
Common concerns about launching a
bounty:
1.  Budgetary concerns. Money is rarely the
main motivation for participants, yo...
Common concerns about launching a
bounty:
1.  Budgetary concerns. Money is rarely the
main motivation for participants, yo...
The ultimate goals of a bug bounty are
threefold:
1.  Incentivize people to report issues to you
in the first place
2.  Dri...
The ultimate goals of a bug bounty are
threefold:
1.  Incentivize people to report issues to you
in the first place
2.  Dri...
The ultimate goals of a bug bounty are
threefold:
1.  Incentivize people to report issues to you
in the first place
2.  Dri...
Before you launch, record what vulnerability
classes you expect to see and what you don’t.
Compare this against the issues...
Before you launch, record what vulnerability
classes you expect to see and what you don’t.
Compare this against the issues...
Keep metrics on:
– Number of bugs reported and severities
– Time to remediation of reported issues
You want both of these ...
Practical considerations:
– Inform all teams before bounty launch,
especially non-engineering teams
•  Ex: Customer Suppor...
Practical considerations:
– Inform all teams before bounty launch,
especially non-engineering teams
•  Ex: Customer Suppor...
Practical considerations:
– Your first 2-3 weeks will be intense. Have as
many people as you can dedicated to triage
and re...
Practical considerations:
– Operationally review any helper systems for
scaling problems beforehand
•  When 10-100x traffic ...
Practical considerations:
– Operationally review any helper systems for
scaling problems beforehand
•  When 10-100x traffic ...
Practical considerations:
– Operationally review any helper systems for
scaling problems beforehand.
•  When 10-100x traffic...
XXX	
  
Running effective attack simulations
Problems with “pentesting” are well
understood in the offensive community
but not as well in the defensive community
Pentests typically result in a list of
enumerated known vulnerabilities to be
patched, not data on how a real attacker
wou...
Attack simulations should be done to learn
how attackers are likely to achieve goals
against your organization
NOT to show...
Use this attack data to focus where/how
to build detection mechanisms
From an organizational side, attack
simulations compliment vulnerability
enumeration/compliance/etc
Four keys to effective attack simulations:
1.  Goal oriented
•  “Obtain domain admin”, “read the CEOs email”,
“view credit ...
Four keys to effective attack simulations:
1.  Goal oriented
•  “Obtain domain admin”, “read the CEOs email”,
“view credit ...
Four keys to effective attack simulations:
3.  Simulate realistic compromise patterns
•  Start the attack team on a:
–  sta...
Four keys to effective attack simulations:
3.  Simulate realistic compromise patterns
•  Start the attack team on a:
–  sta...
The project output should be attack chains
showing how attack team went from A->B->C
to achieve goals, what steps they too...
Just as importantly, what steps they didn’t
take
Ex: “We didn’t try to find internal network diagrams
on your wiki because ...
Remember, the goal is to simulate realistic
attack behaviors and patterns that can be
used to enhance detection
In addition, simulate varying attack profiles
from quick & loud to quietly maintaining
persistence
Over multiple iterations learn what
behaviors overlap between attackers and
what strong signals of lateral movement in
you...
TL;DR
(The section formerly known as “Conclusions”)
•  Adapt security team culture to DevOps
and continuous deployment by:
– Surfacing security monitoring and metrics
– Incen...
Thanks!	
  
zane@signalsciences.com @zanelackey
Building a Modern Security Engineering Organization
Building a Modern Security Engineering Organization
Upcoming SlideShare
Loading in...5
×

Building a Modern Security Engineering Organization

10,773

Published on

Continuous deployment and the DevOps philosophy have forever changed the ways in which businesses operate. This talk with discuss how security adapts effectively to these changes, specifically covering:

- Practical advice for building and scaling modern AppSec and NetSec programs
- Lessons learned for organizations seeking to launch a bug bounty program
- How to run realistic attack simulations and learn the signals of compromise in your environment

Published in: Technology

Building a Modern Security Engineering Organization

  1. 1. Building a Modern Security Engineering Organization zane@signalsciences.com @zanelackey
  2. 2. Who  is  this  guy  anyway?   •  Built and led the Etsy Security Team – Spoiler alert: what this presentation is about •  Recently co-founded Signal Sciences to productize effective AppSec approaches
  3. 3. This talk is a collection of lessons learned from building and adapting a security team
  4. 4. For security teams, the world has changed in fundamental ways: –  Code deployment is now near-instantaneous
  5. 5. For security teams, the world has changed in fundamental ways: –  Code deployment is now near-instantaneous –  Merging of development and operations means more people with production access
  6. 6. For security teams, the world has changed in fundamental ways: –  Code deployment is now near-instantaneous –  Merging of development and operations means more people with production access –  Cost of attack has significantly dropped
  7. 7. Near-instantaneous deployment?
  8. 8. A  technical  diagram  of  tradi7onal  waterfall  code  deployment    
  9. 9. What is this shifting to?
  10. 10. Etsy pushes to production 30 times a day on average
  11. 11. Constant iteration in production via feature flags, ramp ups, A/B testing
  12. 12. But doesn’t the rapid rate of change mean things are less secure?!
  13. 13. Actually,  the  opposite  is   true  
  14. 14. They key to realize is vulnerabilities occur in all development methodologies …But there’s no such thing as an out-of- band patch in continuous deployment
  15. 15. They key to realize is vulnerabilities occur in all development methodologies …But there’s no such thing as an out-of- band patch in continuous deployment
  16. 16. Compared to: “We’ll rush that security fix. It will go out … in about 6 weeks.” - Former vendor at Etsy
  17. 17. What makes continuous deployment safe?
  18. 18. Source:  h<p://www.slideshare.net/mikebri<ain/advanced-­‐topics-­‐in-­‐con7nuous-­‐deployment  
  19. 19. The same culture of graphing and monitoring inherent to continuous deployment can be used for security too
  20. 20. Surface security info for everyone, not just the security team
  21. 21. “Don’t treat security as a binary event” - @ngalbreath
  22. 22. Building  a  (k-­‐)rad  culture   *Mullets  sold  separately    
  23. 23. In the shift to continuous deployment, speed increases by removing organizational blockers
  24. 24. Trying to make security a blocker means you get routed around
  25. 25. Instead, the focus becomes on incentivizing teams to reach out to security
  26. 26. Keys to incentivizing conversation: – Don’t be a jerk. This should be obvious, but empathy needs to be explicitly set as a core part of your teams culture.
  27. 27. Keys to incentivizing conversation: – Don’t be a jerk. This should be obvious, but empathy needs to be explicitly set as a core part of your teams culture. – Make realistic tradeoffs. Don’t fall in to the trap of thinking every issue is critical. •  Ex: Letting low risk issues ship with a reasonable remediation window buys you credibility for when things actually do need to be addressed immediately.
  28. 28. Keys to incentivizing conversation: – Coherently explain impact. “This would allow all our user data to be compromised if the attacker did X & Y” paints a clear picture, where “The input validation in this function is weak” does not.
  29. 29. Keys to incentivizing conversation: – Coherently explain impact. “This would allow all our user data to be compromised if the attacker did X & Y” paints a clear picture, where “The input validation in this function is weak” does not. – Reward communication with security team. T-Shirts, gift cards, and high fives all work (shockingly) well.
  30. 30. Keys to incentivizing conversation: – Take the false positive hit yourself. Don’t send unverified issues to dev and ops teams. When issues come in, have the secteam verify and make first attempt at patch. – Scale via team leads. Build relationships with technical leads from other teams so they make security part of their teams culture.
  31. 31. Keys to incentivizing conversation: – Take the false positive hit yourself. Don’t send unverified issues to dev and ops teams. When issues come in, have the secteam verify and make first attempt at patch. – Scale via team leads. Build relationships with technical leads from other teams so they make security part of their teams culture.
  32. 32. Access  restric7ons  
  33. 33. Startups begin with a simple access control policy: Everyone can access everything
  34. 34. As organization grow there will be more pressure to institute access policies
  35. 35. The key to remember is don’t take away capabilities
  36. 36. Methodology: 1.  Figure out what capability is needed 2.  Build an alternate way to perform the needed function in a safe way 3.  Transition the organization over to the safe way 4.  Alert on any usage of the old unsafe way
  37. 37. Methodology: 1.  Figure out what capability is needed 2.  Build an alternate way to perform the needed function in a safe way 3.  Transition the organization over to the safe way 4.  Alert on any usage of the old unsafe way
  38. 38. Methodology: 1.  Figure out what capability is needed 2.  Build an alternate way to perform the needed function in a safe way 3.  Transition the organization over to the safe way 4.  Alert on any usage of the old unsafe way
  39. 39. Methodology: 1.  Figure out what capability is needed 2.  Build an alternate way to perform the needed function in a safe way 3.  Transition the organization over to the safe way 4.  Alert on any usage of the old unsafe way
  40. 40. EX: SSH access to production systems
  41. 41. Security policy goal: Eliminate unneeded access to production systems –  Why do developers do it? Ex: To view error logs –  Build alternate approach: Send the logs to central logging service (ex: elasticsearch, splunk, etc) –  Publicize the new tooling to the organization –  After majority of transition, alert on any logins to production systems by non-sysops
  42. 42. Security policy goal: Eliminate unneeded access to production systems –  Why do developers do it? Ex: To view error logs –  Build alternate approach: Send the logs to central logging service (ex: elasticsearch, splunk, etc) –  Publicize the new tooling to the organization –  After majority of transition, alert on any logins to production systems by non-sysops
  43. 43. Security policy goal: Eliminate unneeded access to production systems –  Why do developers do it? Ex: To view error logs –  Build alternate approach: Send the logs to central logging service (ex: elasticsearch, splunk, etc) –  Publicize the new tooling to the organization –  After majority of transition, alert on any logins to production systems by non-sysops
  44. 44. Security policy goal: Eliminate unneeded access to production systems –  Why do developers do it? Ex: To view error logs –  Build alternate approach: Send the logs to central logging service (ex: elasticsearch, splunk, etc) –  Publicize the new tooling to the organization –  After majority of transition, alert on any logins to production systems by non-sysops
  45. 45. Increasing  a<acker  cost  
  46. 46. Specifically, some thoughts on: –  Bug Bounties –  Attack simulations/pentesting
  47. 47. Bug  Boun7es  
  48. 48. Bug bounties are tremendously useful. If you’re not working towards launching one, strongly consider it.
  49. 49. Common concerns about launching a bounty: 1.  Budgetary concerns. Money is almost never the main motivation for researchers, you can launch a bounty with just a hall of fame and still get great submissions. 2.  Risk of inviting attacks. You’re already getting attacked continuously, you’re just not getting the results.
  50. 50. Common concerns about launching a bounty: 1.  Budgetary concerns. Money is rarely the main motivation for participants, you can launch a bounty with just a hall of fame and still get great submissions. 2.  Risk of inviting attacks. You’re already getting attacked continuously, you’re just not getting the results.
  51. 51. Common concerns about launching a bounty: 1.  Budgetary concerns. Money is rarely the main motivation for participants, you can launch a bounty with just a hall of fame and still get great submissions. 2.  Risk of inviting attacks. It’s the Internet. You’re already getting pentested continuously, you’re just not receiving the report.
  52. 52. The ultimate goals of a bug bounty are threefold: 1.  Incentivize people to report issues to you in the first place 2.  Drive up cost of vulnerability discovery and exploitation for attackers 3.  Provide an external validation of if your security program is working (or not)
  53. 53. The ultimate goals of a bug bounty are threefold: 1.  Incentivize people to report issues to you in the first place 2.  Drive up cost of vulnerability discovery and exploitation for attackers 3.  Provide an external validation of if your security program is working (or not)
  54. 54. The ultimate goals of a bug bounty are threefold: 1.  Incentivize people to report issues to you in the first place 2.  Drive up cost of vulnerability discovery and exploitation for attackers 3.  Provide an external validation of where your security program is working (and where it’s not)
  55. 55. Before you launch, record what vulnerability classes you expect to see and what you don’t. Compare this against the issues actually reported.
  56. 56. Before you launch, record what vulnerability classes you expect to see and what you don’t. Compare this against the issues actually reported.
  57. 57. Keep metrics on: – Number of bugs reported and severities – Time to remediation of reported issues You want both of these metrics to trend down over time
  58. 58. Practical considerations: – Inform all teams before bounty launch, especially non-engineering teams •  Ex: Customer Support – Attacks will start almost immediately For Etsy bug bounty launch, time from announcement to first attack: 13min
  59. 59. Practical considerations: – Inform all teams before bounty launch, especially non-engineering teams •  Ex: Customer Support – Attacks will start almost immediately For Etsy bug bounty launch, time from announcement to first attack: 13min
  60. 60. Practical considerations: – Your first 2-3 weeks will be intense. Have as many people as you can dedicated to triage and response
  61. 61. Practical considerations: – Operationally review any helper systems for scaling problems beforehand •  When 10-100x traffic hits helper systems your security team uses, what falls over? – Money almost never the overriding factor, hall of fame is –  Researchers are generally great to interact with
  62. 62. Practical considerations: – Operationally review any helper systems for scaling problems beforehand •  When 10-100x traffic hits helper systems your security team uses, what falls over? – Money is almost never the main motivation for bounty participants, hall of fame credit is –  Researchers are generally great to interact with
  63. 63. Practical considerations: – Operationally review any helper systems for scaling problems beforehand. •  When 10-100x traffic hits helper systems your security team uses, what falls over? – Money is almost never the main motivation for bounty participants, hall of fame credit is –  Key to great researcher interaction is frequent and transparent communication
  64. 64. XXX   Running effective attack simulations
  65. 65. Problems with “pentesting” are well understood in the offensive community but not as well in the defensive community
  66. 66. Pentests typically result in a list of enumerated known vulnerabilities to be patched, not data on how a real attacker would operate against a given environment
  67. 67. Attack simulations should be done to learn how attackers are likely to achieve goals against your organization NOT to show compromise is possible (spoiler alert: it is.)
  68. 68. Use this attack data to focus where/how to build detection mechanisms
  69. 69. From an organizational side, attack simulations compliment vulnerability enumeration/compliance/etc
  70. 70. Four keys to effective attack simulations: 1.  Goal oriented •  “Obtain domain admin”, “read the CEOs email”, “view credit card data”, … •  Ask attack team for input on goals, they’ll come up with ones you didn’t think of 2.  Full ganization in scope •  Have attack team call a contact if they’re about to do something risky – several week simulat – ion
  71. 71. Four keys to effective attack simulations: 1.  Goal oriented •  “Obtain domain admin”, “read the CEOs email”, “view credit card data”, … •  Ask attack team for input on goals, they’ll come up with ones you didn’t think of 2.  Full organization in scope •  Have attack team call a contact if they’re about to do something risky –  Ex: Instead of throwing an exploit that lands “most of the time”, grant access to the target system with temporary credentials
  72. 72. Four keys to effective attack simulations: 3.  Simulate realistic compromise patterns •  Start the attack team on a: –  standard laptop/desktop to simulate phishing/clientside compromise –  database or web server to simulate SQL injection/RCE •  0days aren’t cheating, they’re reality. Attack team should be encouraged to use them. –  Break simulation down into iterations: •  Don’t spend the full engagement time on only round of testing, once one team achieve goal(s), then swap in new attack team to achieve the same goal(s) –  Ex: We try to run 3-4 iterations per several week simulation
  73. 73. Four keys to effective attack simulations: 3.  Simulate realistic compromise patterns •  Start the attack team on a: –  standard laptop/desktop to simulate phishing/clientside compromise –  database or web server to simulate SQL injection/RCE •  0days aren’t cheating, they’re reality. Attack team should be encouraged to use them. 4.  Break simulation down into iterations: •  Don’t spend the full engagement time on only round of testing, once one team achieve goal(s), then swap in new attack team to achieve the same goal(s) –  Ex: We try to run 3-4 iterations per several week simulation
  74. 74. The project output should be attack chains showing how attack team went from A->B->C to achieve goals, what steps they took and why
  75. 75. Just as importantly, what steps they didn’t take Ex: “We didn’t try to find internal network diagrams on your wiki because zone transfers were enabled so we could got enough data about your network from that”
  76. 76. Remember, the goal is to simulate realistic attack behaviors and patterns that can be used to enhance detection
  77. 77. In addition, simulate varying attack profiles from quick & loud to quietly maintaining persistence
  78. 78. Over multiple iterations learn what behaviors overlap between attackers and what strong signals of lateral movement in your environment look like
  79. 79. TL;DR (The section formerly known as “Conclusions”)
  80. 80. •  Adapt security team culture to DevOps and continuous deployment by: – Surfacing security monitoring and metrics – Incentivize discussions with the security team – When creating policy, don’t take away capabilities •  Drive up attacker cost through bug bounty programs, countering phishing, and running realistic attack simulations
  81. 81. Thanks!   zane@signalsciences.com @zanelackey
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×