Fraser and Nick debate the relationship between DevOps and security. Nick argues security is too complex for DevOps approaches, while Fraser argues DevOps and security ultimately have the same goals of reducing risk and increasing value. They propose defining a "risk budget" to measure and manage risk like an "error budget", allowing more frequent deployments if risk is reduced through practices like testing and security engagement. Ultimately they agree DevOps and security need cooperation rather than separation, with security helping scale out practices while DevOps takes security responsibilities.

1. Join the conversation #devseccon
By Fraser Scott & Nick Drage
Epic Battle:
DevOps vs Security

2. Fraser:
Hello everyone and welcome to Epic Battle: DevOps vs Security.

3. Fraser:
Hi I’m Fraser, aka Player 1

4. Fraser:
I’m an ex-sysadmin, devops engineer. Now a cloud secops engineer at capital one. And I’m
a DevOps fanboi.

6. Nick:
Hi I’m Nick aka Player 2

7. Nick:
I’ve been sysadmin, security operations, pentester, and security practice architect, I’ve been
doing this since you were a teenager

8. Nick:
While I’ve enjoyed all the talks here today, and have the greatest respect for the speakers,
they’re all wrong.
DevOps and DevSecOps are just another silly management fad that just results in poorly
planned and hastily executed garbage like the Internet Of Things.

9. Nick:
Brian Krebs DDoS

11. Fraser:
DevOps and DevSecOps aren't management fads, they're a grassroots movement that
comes from engineers finding out that working together let’s them get shit done faster than
not working together. DevOps has just a hard time convincing management as much as it
does traditionalists.
Look, DevSecOps is just DevOps + Security. [CALMS slide] Some argue that Security is
automatically included in DevOps, and it should be, but it’s nice to have a term that includes
security so it doesn’t get forgotten about, hence DevSecOps.

12. Nick:
Yes, but the benefits of DevOps could easily be achieved if Developers, Operations, and
Security just talked to each other.

13. Nick:
Use online meetings

14. Nick:
Stop sitting in their own silos. And that way you’d still maintain separation of duties , access
with lowest level of privileges, and all the other benefits of dividing roles.

16. Dev Ops
Sec
DevOps
Sec
Fraser:
Right. Exactly! That’s the cultural aspect. DevOps doesn’t mean one unicorn engineer doing
all the things, it means breaking down the traditional silos. You might end up with a single
functional team that has a mixture of software engineers, operations engineers, QA and
security. Or maybe separate teams working together. The trick is getting the right people
involved earlier on.

17. Nick:
But if you still have multiple teams, then nothing has changed. People will still book pointless
meetings, email word documents around, argue about whose responsibility something is, not
read each other’s documentation etc.
( That guy, in the top left, with his black tshirt and earphones on… security )

18. Nick:
Multiple teams, nothing has changed, just gimmicks.

19. Nick:
Some more gimmicky than others

21. Fraser:
Possibly, yes, but it doesn’t have to work that way. Think of teams as microservices. They
can have well defined functions and “interfaces”. If you're constantly winging it or having to
read 100 page documents you're going to have a bad time. Automation has a big part to play
here because it removes the typical human barriers that introduce slowness and latency.
Instead of emailing some team a document containing changes for review, a git commit
could trigger automated tests that effectively carries out the decision making process the
person would have made. You’re breaking down the traditional silos through automation.
Instead of asking “how do i?”, go and look at the jobs and code.

22. Nick:
But automation is just a way of breaking things at scale.

23. Nick:
If you screw something up in an automated way, there’s nothing to stop it propagating out
and becoming a massive disaster.

25. https://xkcd.com/327/
Fraser:
Yes it’s true that things can go wrong, and with heavy automation things can go very wrong
at a large scale very quickly. But even a single manual command can bring down an entire
service, and that’s without any automation. Something like a DROP TABLE statement for
example. But with everything-as-code etc., you can also automate the detection and
response to problems. Automatically rollback changes whether they’re code or infrastructure.
Sure, there may be some brief downtime, or a percentage of users get some errors for a
while, but in order to maintain velocity and innovation you have to accept some risk of
failure.

26. Nick:
In security, failure is not an option! It’s one thing if a user can’t get their fix of cat pictures for
5 minutes, but it’s a whole other issue for their personal or financial details to be
compromised by some random hackers.

28. I
<3
failure
Fraser:
Agreed. Loss of availability is a best-case scenario for a security incident. But some failure is
inevitable. The question is, do you accept it and learn to leverage it to your advantage, or do
you pretend it doesn’t apply to you until one day everything is on fire. There have been an
increasingly large number of breaches lately, even from within organisations that had
relatively good security. Why?

29. Nick:
Then they didn’t security hard enough. They probably thought they had excellent security
because they had blinky boxes and were PCI compliant at some point in the past, but in
reality if they were compromised there was a weakness. They just need to get more staff and
put in more effort… security is a hungry process that needs to be fed.

31. Fraser:
Just throwing more staff at ineffective security processes doesn’t work. That’s like trying to
find a needle in a haystack by adding more hay. Yes, their security wasn’t perfect. But it also
never could be.

32. Nick:
No! Hackers only need to find one way in. We need to be perfect, at all times - once the
hackers are in, we’re done for.

34. Nick:
You don’t understand the threats we all face, you don’t have the cyber threat intelligence to
understand the cyber threat actors in the cyber threat landscape in the cyber threat space.

36. Nick:
I’ve got auditors to contend with, they’re a threat too. I’ve got to be secure, and compliant.

37. Nick:
I’ve got to minimise the attack surface of every interface thats under threat, and as everyone
and everything is a threat, that’s the attack surface of everything. I haven’t got time for fads.

39. Nick:
I’ve got to protect the confidentiality, integrity, and availability of all the company’s assets.
Company assets which undoubtedly come with default credentials and configurations, those
assets are a threat.
And I’ve got to find all those assets, which is only the first entry in the SANS Top 20.

40. Nick:
You’re a developer, you only have to contend with passive failure. A hard drive fails or
someone accidentally pushes some broken code. I'm up against the best sentient opponents
the Internet has to offer. Opponents that react and adapt.

42. Nick:
To do that I need to manage Firewalls

43. Nick:
Internal firewalls

44. Nick:
Ingress and egress filtering
Apparently allow users to talk to each other

45. Nick:
Hardened proxies

46. Nick:
Data Loss Prevention

47. Nick:
End Point Threat Management

48. Nick:
All this data needs to go into a Security Incident Event Management

49. Nick:
Which needs to be analysed by SOC analysts and hunt teams

50. Nick:
And I need a Data Lake to store it all. A Data Lake. Fraser, I don’t even know what a Data
Lake is!

52. Fraser:
Yes, security is hard. It's cat and mouse. But it's also unrealistic for security to control the
entire environment. At that point security will always be a blocker. Security needs to be part
of everyone’s job. I’m not saying that the security team is redundant, you’ll always need
subject matter expects. But security being a constant blocker just won't scale and would end
up stopping the organisation reaching it's goals and targets. Either that or you end up with
shadow IT.

53. Nick:
*I’M* Security, capital S, I’m the only one that knows best and gets to say “no” when needed.
How do I say no to people if everyone is doing security?

54. Department
of
No
CLOSED
Fraser:
You don’t. The department of “No” has been disbanded.

55. Nick:
If I don’t say no, the hacker’s will get in and then it’s game over, everything’s on fire.

56. Fraser:
It’s inevitable that at some point somebody’s going to get in, in some way. You have to
accept some level of risk. The only way you’re going to harden your environment to that
extent, is by stifling innovation and progress. We all know that the only way to fully secure a
device is to switch it off and bury it in concrete. There's more than just prevention, there's
detection, containing and remediation. Defense in depth right? An may one need to find one
way in, but once they're in they have to not make any mistakes for fear of being detected.

57. Nick:
If you catch me at a good time I might agree with you on risk, but risk analysis requires deep
contemplation over a long period of time by experienced and expensive risk consultants like
myself.

58. Nick:
It's impossible to build and deploy at the speed DevOps people want, and the same time as
stay on top of managing risk within the organisation.

60. https://landing.google.com/sre/interview/ben-treynor.html
Fraser:
Let’s step back a minute. Google has the concept of an “error budget”. The idea is that for a
given service, the SLA is determined, and the error budget is defined as 1 minutes the SLA.
Your spend is then the error budget minus the availability. So, let’s say for a given service, if
at the end of the quarter is it shown that the service is well within the SLA, so has had very
few availability issues and effectively hasn’t spent its error budget, then question is.. Could
they have pushed faster or innovated more? That budget has basically gone to waste. But if
they’d gone over budget, they’d probably have to slow down their deployments, writing more
or better tests etc. This way development and ops are both working off the same incentive -
spend that budget, and exactly that budget. No more, no less.

61. Nick:
SLAs are easy, things are up or they’re not.

62. Nick:
Risk is complex, you can’t analyse this complexity with a tool.

63. risk budget
Security meets Velocity
Fraser:
Well, what if you could define something like a “risk budget”? If you’re have to accept some
level of risk anyway, why not find a way to measure it and use that to align security and
deployment velocity. Again, automation and measurement is key here.

64. Nick:
OK, seeing as I’m winning by so much I’ll indulge you… how exactly would you do that?

65. Existing risk measurements
Documentation coverage
Unit Test coverage
Unit Test results
Integration Test coverage
Integration Test results
Fraser:
It probably depends on the organisation, but you could use existing measurements such as
documentation coverage, and unit and integration testing coverage and results. The
assumption here is that well tested, well documented code is probably of a higher quality
and therefore a higher security quality.

66. Other metrics
Continuous security testing
Code-driven Threat Modelling
Level of security engagement
Pentest results
% successful gate passes
Fraser:
You’d probably also want to measure security testing coverage and results. Maybe you’re
also generating threat modelling coverage and results, so that could be measured. Perhaps
you could measure the level of engagement the project has had with the security team. A
good annual pentest result could add a bonus to the risk budget for the service, and bad
pentest result could reduce the risk budget.

67. Fraser:
So let’s say a very important service has a risk budget of 1000 points per quarter. Currently
their deployments, based on all the above measurements, are coming in at a “cost” of 400
points per production deployment. That would mean they could release to production 2
times per quarter. Not very agile. If they wrote more and better security tests, expanded their
threat model coverage and involved the security team a bit sooner in the lifecycle, maybe
their cost would go down to 155 points. That would mean they could deploy to production
every two weeks.

68. Nick:
So what you’re saying is that if a platform is provably insecure then it can be altered less, but
if it’s provably secure, DevOps can deploy more often and add value?

69. Fraser:
Not provably insecure or secure, but provably less secure or more secure. If you think about
it, developers work with value. In fact, value is fundamental to devops. They’ll write a feature
because there is a hopefully high likelihood that the feature will have a positive impact on
sales or number of users etc, or whatever the key business metrics are. That sounds a lot like
risk, just that the impact is expected to be positive not negative.

70. Risk=
Impact x
Likelihood
Value=
Impact x
Likelihood
Nick:
OK, so if I’m going to buy into your way of thinking, which I might have to because we only
have twenty minutes, risk is negative impact times likelihood, and value is positive impact
times likelihood. So you’re saying that value-driven DevOps and risk-driven Security are
basically two sides of the same coin.

71. Fraser:
Yes, exactly! It’s all very YinYang. Two opposing sides that coexist. Both sides cannot exist
without the other.

72. Sometimes,
they are out
to get you
Fraser:
Devops has to take into account the negative side of human nature to succeed. It cannot
ignore security - security is everyone’s responsibility. It cannot pretend that there aren't
people out there who will actively try to attack and compromise their systems and data.

73. Trust,
But
Verify
Fraser:
Security has to utilise the positive side of human nature to succeed. Yes there are people out
there trying to get you, but not everyone is. Use a trust but verify model to scale out security
controls.

74. Nick:
So what you’re saying is that we can automate our security mindset into development and
operations, so we’re all thinking about security, rather than the separate security team only
enforcing security when they notice your project? So that would maybe free up security to
focus on thinking about the bigger picture rather than be bogged down in millions of
implementation details. They could focus on scaling out security not hoarding it.
So, according to the analogy I’ve been trying to force on this entire presentation, we both win

75. Nick:
...we all win o/

76. To DevOps:
Be patient with Security. They have to go
through a philosophical paradigm shift.
To Security:
Be patient with DevOps. They have the best
intentions and some pretty cool ideas.
To Everyone:
Remember, empathy goes both ways.
Fraser - first line
Nick - second line
Fraser - third line

77. Join the conversation #devseccon
Thanks!
Fraser Scott
@zeroXten
Nick Drage
@SonOfSunTzu
Image credits:
https://github.com/zeroXten/devseccon2016
Thanks!