1. Building a Modern Security
Engineering Organization
zane@signalsciences.com
@zanelackey

2. Who
is
this
guy
anyway?
• Built and led the Etsy Security Team
– Spoiler alert: what this presentation is about
• Recently co-founded Signal Sciences to
productize effective AppSec approaches

3. This talk is a collection of lessons learned
from building and adapting a security
team

4. For security teams, the world has changed
in fundamental ways:
– Code deployment is now near-instantaneous

5. For security teams, the world has changed
in fundamental ways:
– Code deployment is now near-instantaneous
– Merging of development and operations
means more people with production access

6. For security teams, the world has changed
in fundamental ways:
– Code deployment is now near-instantaneous
– Merging of development and operations
means more people with production access
– Cost of attack has significantly dropped

7. Near-instantaneous deployment?

8. A
technical
diagram
of
tradi7onal
waterfall
code
deployment

9. What is this shifting to?

10. Etsy pushes to production 30 times a day
on average

11. Constant iteration in production via
feature flags, ramp ups, A/B testing

12. But doesn’t the
rapid rate of
change mean
things are less
secure?!

13. Actually,
the
opposite
is
true

14. They key to realize is vulnerabilities occur
in all development methodologies
…But there’s no such thing as an out-of-
band patch in continuous deployment

15. They key to realize is vulnerabilities occur
in all development methodologies
…But there’s no such thing as an out-of-
band patch in continuous deployment

16. Compared to:
“We’ll rush that security fix. It will go out …
in about 6 weeks.”
- Former vendor at Etsy

17. What makes continuous deployment safe?

19. Source:
h<p://www.slideshare.net/mikebri<ain/advanced-­‐topics-­‐in-­‐con7nuous-­‐deployment

20. The same culture of graphing and
monitoring inherent to continuous
deployment can be used for security too

21. Surface security info for everyone, not just
the security team

23. “Don’t treat security as a binary event”
- @ngalbreath

24. Building
a
(k-­‐)rad
culture
*Mullets
sold
separately

25. In the shift to continuous deployment,
speed increases by removing
organizational blockers

26. Trying to make security a blocker means
you get routed around

27. Instead, the focus becomes on
incentivizing teams to reach out to security

28. Keys to incentivizing conversation:
– Don’t be a jerk. This should be obvious, but
empathy needs to be explicitly set as a core
part of your teams culture.

29. Keys to incentivizing conversation:
– Don’t be a jerk. This should be obvious, but
empathy needs to be explicitly set as a core
part of your teams culture.
– Make realistic tradeoffs. Don’t fall in to the
trap of thinking every issue is critical.
• Ex: Letting low risk issues ship with a reasonable
remediation window buys you credibility for
when things actually do need to be addressed
immediately.

30. Keys to incentivizing conversation:
– Coherently explain impact. “This would
allow all our user data to be compromised if
the attacker did X & Y” paints a clear picture,
where “The input validation in this function is
weak” does not.

31. Keys to incentivizing conversation:
– Coherently explain impact. “This would
allow all our user data to be compromised if
the attacker did X & Y” paints a clear picture,
where “The input validation in this function is
weak” does not.
– Reward communication with security
team. T-Shirts, gift cards, and high fives all
work (shockingly) well.

32. Keys to incentivizing conversation:
– Take the false positive hit yourself. Don’t
send unverified issues to dev and ops
teams. When issues come in, have the
secteam verify and make first attempt at
patch.
– Scale via team leads. Build relationships
with technical leads from other teams so
they make security part of their teams
culture.

33. Keys to incentivizing conversation:
– Take the false positive hit yourself. Don’t
send unverified issues to dev and ops
teams. When issues come in, have the
secteam verify and make first attempt at
patch.
– Scale via team leads. Build relationships
with technical leads from other teams so
they make security part of their teams
culture.

34. Access
restric7ons

35. Startups begin with a simple access
control policy: Everyone can access
everything

36. As organization grow there will be more
pressure to institute access policies

37. The key to remember is don’t take away
capabilities

38. Methodology:
1. Figure out what capability is needed
2. Build an alternate way to perform the
needed function in a safe way
3. Transition the organization over to the safe
way
4. Alert on any usage of the old unsafe way

39. Methodology:
1. Figure out what capability is needed
2. Build an alternate way to perform the
needed function in a safe way
3. Transition the organization over to the safe
way
4. Alert on any usage of the old unsafe way

40. Methodology:
1. Figure out what capability is needed
2. Build an alternate way to perform the
needed function in a safe way
3. Transition the organization over to the safe
way
4. Alert on any usage of the old unsafe way

41. Methodology:
1. Figure out what capability is needed
2. Build an alternate way to perform the
needed function in a safe way
3. Transition the organization over to the safe
way
4. Alert on any usage of the old unsafe way

42. EX: SSH access to production systems

43. Security policy goal: Eliminate unneeded
access to production systems
– Why do developers do it? Ex: To view error logs
– Build alternate approach: Send the logs to
central logging service (ex: elasticsearch,
splunk, etc)
– Publicize the new tooling to the organization
– After majority of transition, alert on any logins to
production systems by non-sysops

44. Security policy goal: Eliminate unneeded
access to production systems
– Why do developers do it? Ex: To view error logs
– Build alternate approach: Send the logs to
central logging service (ex: elasticsearch,
splunk, etc)
– Publicize the new tooling to the organization
– After majority of transition, alert on any logins to
production systems by non-sysops

45. Security policy goal: Eliminate unneeded
access to production systems
– Why do developers do it? Ex: To view error logs
– Build alternate approach: Send the logs to
central logging service (ex: elasticsearch,
splunk, etc)
– Publicize the new tooling to the organization
– After majority of transition, alert on any logins to
production systems by non-sysops

46. Security policy goal: Eliminate unneeded
access to production systems
– Why do developers do it? Ex: To view error logs
– Build alternate approach: Send the logs to
central logging service (ex: elasticsearch,
splunk, etc)
– Publicize the new tooling to the organization
– After majority of transition, alert on any logins to
production systems by non-sysops

47. Increasing
a<acker
cost

48. Specifically, some thoughts on:
– Bug Bounties
– Attack simulations/pentesting

49. Bug
Boun7es

50. Bug bounties are tremendously useful. If
you’re not working towards launching one,
strongly consider it.

51. Common concerns about launching a
bounty:
1. Budgetary concerns. Money is almost
never the main motivation for researchers,
you can launch a bounty with just a hall of
fame and still get great submissions.
2. Risk of inviting attacks. You’re already
getting attacked continuously, you’re just
not getting the results.

52. Common concerns about launching a
bounty:
1. Budgetary concerns. Money is rarely the
main motivation for participants, you can
launch a bounty with just a hall of fame
and still get great submissions.
2. Risk of inviting attacks. You’re already
getting attacked continuously, you’re just
not getting the results.

53. Common concerns about launching a
bounty:
1. Budgetary concerns. Money is rarely the
main motivation for participants, you can
launch a bounty with just a hall of fame
and still get great submissions.
2. Risk of inviting attacks. It’s the Internet.
You’re already getting pentested
continuously, you’re just not receiving the
report.

54. The ultimate goals of a bug bounty are
threefold:
1. Incentivize people to report issues to you
in the first place
2. Drive up cost of vulnerability discovery and
exploitation for attackers
3. Provide an external validation of if your
security program is working (or not)

55. The ultimate goals of a bug bounty are
threefold:
1. Incentivize people to report issues to you
in the first place
2. Drive up cost of vulnerability discovery and
exploitation for attackers
3. Provide an external validation of if your
security program is working (or not)

56. The ultimate goals of a bug bounty are
threefold:
1. Incentivize people to report issues to you
in the first place
2. Drive up cost of vulnerability discovery and
exploitation for attackers
3. Provide an external validation of where
your security program is working (and
where it’s not)

57. Before you launch, record what vulnerability
classes you expect to see and what you don’t.
Compare this against the issues actually
reported.

58. Before you launch, record what vulnerability
classes you expect to see and what you don’t.
Compare this against the issues actually
reported.

59. Keep metrics on:
– Number of bugs reported and severities
– Time to remediation of reported issues
You want both of these metrics to trend down
over time

60. Practical considerations:
– Inform all teams before bounty launch,
especially non-engineering teams
• Ex: Customer Support
– Attacks will start almost immediately
For Etsy bug bounty launch, time from
announcement to first attack: 13min

61. Practical considerations:
– Inform all teams before bounty launch,
especially non-engineering teams
• Ex: Customer Support
– Attacks will start almost immediately
For Etsy bug bounty launch, time from
announcement to first attack: 13min

62. Practical considerations:
– Your first 2-3 weeks will be intense. Have as
many people as you can dedicated to triage
and response

63. Practical considerations:
– Operationally review any helper systems for
scaling problems beforehand
• When 10-100x traffic hits helper systems your
security team uses, what falls over?
– Money almost never the overriding factor,
hall of fame is
– Researchers are generally great to interact
with

64. Practical considerations:
– Operationally review any helper systems for
scaling problems beforehand
• When 10-100x traffic hits helper systems your
security team uses, what falls over?
– Money is almost never the main motivation
for bounty participants, hall of fame credit is
– Researchers are generally great to interact
with

65. Practical considerations:
– Operationally review any helper systems for
scaling problems beforehand.
• When 10-100x traffic hits helper systems your
security team uses, what falls over?
– Money is almost never the main motivation
for bounty participants, hall of fame credit is
– Key to great researcher interaction is
frequent and transparent communication

66. XXX
Running effective attack simulations

67. Problems with “pentesting” are well
understood in the offensive community
but not as well in the defensive community

68. Pentests typically result in a list of
enumerated known vulnerabilities to be
patched, not data on how a real attacker
would operate against a given
environment

69. Attack simulations should be done to learn
how attackers are likely to achieve goals
against your organization
NOT to show compromise is possible
(spoiler alert: it is.)

70. Use this attack data to focus where/how
to build detection mechanisms

71. From an organizational side, attack
simulations compliment vulnerability
enumeration/compliance/etc

72. Four keys to effective attack simulations:
1. Goal oriented
• “Obtain domain admin”, “read the CEOs email”,
“view credit card data”, …
• Ask attack team for input on goals, they’ll come
up with ones you didn’t think of
2. Full ganization in scope
• Have attack team call a contact if they’re about
to do something risky
– several week simulat
– ion

73. Four keys to effective attack simulations:
1. Goal oriented
• “Obtain domain admin”, “read the CEOs email”,
“view credit card data”, …
• Ask attack team for input on goals, they’ll come
up with ones you didn’t think of
2. Full organization in scope
• Have attack team call a contact if they’re about
to do something risky
– Ex: Instead of throwing an exploit that lands “most of
the time”, grant access to the target system with
temporary credentials

74. Four keys to effective attack simulations:
3. Simulate realistic compromise patterns
• Start the attack team on a:
– standard laptop/desktop to simulate phishing/clientside
compromise
– database or web server to simulate SQL injection/RCE
• 0days aren’t cheating, they’re reality. Attack team
should be encouraged to use them.
– Break simulation down into iterations:
• Don’t spend the full engagement time on only round
of testing, once one team achieve goal(s), then swap
in new attack team to achieve the same goal(s)
– Ex: We try to run 3-4 iterations per several week
simulation

75. Four keys to effective attack simulations:
3. Simulate realistic compromise patterns
• Start the attack team on a:
– standard laptop/desktop to simulate phishing/clientside
compromise
– database or web server to simulate SQL injection/RCE
• 0days aren’t cheating, they’re reality. Attack team
should be encouraged to use them.
4. Break simulation down into iterations:
• Don’t spend the full engagement time on only round
of testing, once one team achieve goal(s), then swap
in new attack team to achieve the same goal(s)
– Ex: We try to run 3-4 iterations per several week
simulation

76. The project output should be attack chains
showing how attack team went from A->B->C
to achieve goals, what steps they took and
why

77. Just as importantly, what steps they didn’t
take
Ex: “We didn’t try to find internal network diagrams
on your wiki because zone transfers were enabled
so we could got enough data about your network
from that”

78. Remember, the goal is to simulate realistic
attack behaviors and patterns that can be
used to enhance detection

79. In addition, simulate varying attack profiles
from quick & loud to quietly maintaining
persistence

80. Over multiple iterations learn what
behaviors overlap between attackers and
what strong signals of lateral movement in
your environment look like

81. TL;DR
(The section formerly known as “Conclusions”)

82. • Adapt security team culture to DevOps
and continuous deployment by:
– Surfacing security monitoring and metrics
– Incentivize discussions with the security
team
– When creating policy, don’t take away
capabilities
• Drive up attacker cost through bug
bounty programs, countering phishing,
and running realistic attack simulations

83. Thanks!
zane@signalsciences.com @zanelackey