Evident is a sponsor of the inaugural DevOps.com eBook titled Rugged DevOps: 10 Ways to Start Embedding Security into DevOps Patterns. Learn more about how to start moving toward a Rugged DevOps mentality through insights shared by security and DevOps experts, including Evident CEO Tim Prendergast, with reporter Ericka Chickowski.

1. RUGGED
DevOps:
10 WAYS TO START
EMBEDDING SECURITY
INTO DEVOPS
PATTERNS
BY ERICKA CHICKOWSKI

2. Security industry conferences and articles in the trade press
are fond of harping on the broken IT security model. They
point to growing breach statistics, embarrassing configuration
and patch management practices, and mounting vulnerability
exploits as ample evidence of the problems plaguing infosec.
It’s easy to point out problems,
but what about solutions?
Interestingly, as scary as DevOps may be
to the typical security pro, its fast iterations
and continuous delivery patterns may be just
the solution to fixing what’s broken in today’s
security model.
“Security pros need to understand that
this is the future of IT and of IT security,” says
Rich Mogull, analyst and CEO at security
analyst firm Securosis. “I see DevOps doing
nothing but improving security when done
right. Not only is software going to be more
secure, but if you learn from these techniques
it can help you get your day-to-day security
more efficient.”
But the devil is in the details. What exactly
does it take to do it right? How can security
operate within the DevOps context in order
to ensure a more sustainable and less
risky continuous delivery value chain? The
complete answers to those questions are
still being written. But the innovators in the
security and DevOps communities are finding
early success with a number of key practices
that contribute to what many security
pundits call a Rugged DevOps experience.

3. Attackers are already using their own
form of continuous delivery to overwhelm
the good guys. The reason security
teams can’t keep up is because the bad
guys have already figured out how to use
automation and cloud-style technologies
to scale up their attacks, says Tim
Prendergast, CEO of Evident.io and a
long-time DevOps and security
practitioner. Predergast is best
known for his former
role, leading up cloud
architecture and
security forAdobe.
“But the
defenders are
still relying
on this model
where you’ve
got a person
in front of a
console,” says
Prendergast.
“They’re just
outgunned and
outnumbered, so they
have to move into this
automation philosophy. They
have to make time for renovating their
approach and they have to be open to
new ideas.”
This means getting unstuck from
security patterns they’ve settled
into for the past 15 or 20 years, says
Prendergast.
“DevOps is bringing a whole new
approach to security to the table that’s
actually way better than what we used
to be doing,” he says, explaining that
security teams will be able to pivot more
quickly with attack trends if they learn
how to deploy platforms and protection
mechanisms in lean, iterative cycles
rather than relying on “big-bang releases.”
And rather than trying to be a
‘gating factor’ at the tail end of
enterprise development,
security should
be baked into
operational
practices and
automated
tooling
throughout the
development
pipeline.
“A DevOps
team will deploy
code and when
a bug pops up,
they’re very well
instrumented to
gather telemetry on the
bug, get it into repair and turn
a fix around the same day and redeploy,”
he says. This approach is in contrast to
the old model of rolling back the entire
deployment on the hunt for a pristine
deployment, delaying necessary
features for the sake of a whole list of
bugs that may take days or weeks to
take care of.
Understand
That Attackers
Already Deliver
Continuously
What is
Rugged
Software?
“Rugged” describes software development
organizations which have a culture of rapidly
evolving their ability to create available, survivable,
defensible, secure, and resilient software.
—From www.ruggedsoftware.org

4. The Rugged Manifesto
I am rugged and, more importantly,
my code is rugged.
I recognize that software has become
a foundation of our modern world.
I recognize the awesome responsibility
that comes with this foundational role.
I recognize that my code will be used in ways
I cannot anticipate, in ways it was not designed,
and for longer than it was ever intended.
I recognize that my code will be attacked by talented
and persistent adversaries who threaten our
physical, economic and national security.
I recognize these things –
and I choose to be rugged.
I am rugged because I refuse to be
a source of vulnerability or weakness.
I am rugged because I assure my code
will support its mission.
I am rugged because my code can face these
challenges and persist in spite of them.
I am rugged, not because it is easy, but because
it is necessary and I am up for the challenge.
-From www.ruggedsoftware.org

5. DevOps cycle times can seem
impossibly, dangerously fast to security
veterans, who may hold the belief that
code can’t possibly be secured without
days-long static and dynamic analysis.
But rule number one of
achieving a more rugged
DevOps process is not
taking the ball away
and going home
when you don’t get
your way.
“You don’t throw
up your hands
and say ‘If they’re
doing a daily build
and it takes four days
for feedback, I give
up,’” says Josh Corman,
CTO at Sonatype. “You
ask if there’s some sniff test or
some subset of tests we can do that
conforms to the compressed cycle time.
And maybe once a week, once a month
or once a quarter we do a deeper thing.”
Rather than entrenching themselves
against new waves of continuous
deployment and automation, security
professionals need to “get clever about
how to adapt,” says Corman.
Organizations are
finding as much as a
10x improvement
in development
speed through
continuous delivery.
They’re bringing
new features to
customers more
quickly, freeing up
resources to develop
new lines of business
fueled by digital
innovation and clearing
up their application backlogs.
Security should be participating.
“If you’re a speed bump, everyone will
go around you,” warns Mogull. “Security
leaders need to figure out the best way
to get into the rugged DevOps process.”
Don’t Throw
Up Your
Hands
SECURITYwas named as the number one DevOps obstacle
by 28 percent of enterprises

6. Before security pros even start to
think about automated testing, new
workflows or updated technology,
they should begin by exercising their
empathy muscles, says Corman.
One way to build that empathy
is to start thinking about
what the common
ground is between
DevOps-minded
practitioners
and security
professionals.
“We both hate
complexity. We
want to fail fast,
early and often
so we can iterate,“
says Corman. “And
we like to instrument
everything—we all have an
almost maniacal desire to get a
ton of actionable telemetry.”
Just as dev and ops
have had to bridge some
serious cultural gulfs to start
collaborating better, security
needs to meet the developers
and operations teams half way.
A longtime practitioner in both
operations and security, as well as the
primary developer of the open source
automated security test framework
Gauntlt, James Wickett recalls an eye-
opening interaction he once had with a
superior in an enterprise setting.
“My boss at the time said, ‘You’re
a security guy, so you probably just
want all the machines turned off and
unplugged,’” says Wickett, who is
currently a senior engineer at Signal
Sciences, a stealth-mode security firm
working to build products that work in
DevOps environments. “I was sort of
offended by that, because I’m
not an idiot and I’m not a
nihilist.”
Wickett, like many
other security folks,
understood that the
business was trying
to move forward
with its goals. But
the lesson was
that there may be
a reason for some of
that misapprehension.
As he puts it, a lot of
security professionals will
mutter something akin to ‘those
stupid developers’ as they put out
fires. If security pros want to move the
needle on risk management, they need
to help everyone come to a mutual
understanding of the time pressures
and restrictions on organizational
resources that each group faces,
Wickett says.
“In particular, the security people
need to know and learn and understand
how development works,” agrees Mogull.
“Obviously, there’s the operational side,
which they tend to be more familiar with.
But they also need to try to understand
developer’s terms and processes, and
use that to bring security into it. You
don’t need to be a developer, but you
need to know how they work.”
Start
With Mutual
Understanding
and Respect

7. Coming to a mutual understanding
could start with semantics. Cooperation
between security, developers and
operations staff could be improved
by overcoming some language
barriers, Corman says.
Take the Heartbleed
vulnerability as an
example. A security
person might see
the problem as a
potential breach, a
security risk or an
incident response
exercise. Meanwhile,
an ops person would
probably refer to it as
a break-fix or a service
interruption.
“ [Ops doesn’t] care about
security or risk, they’re concerned with
being measured on the five nines of
availability. They had an outage and they
had to create a remedy ticket and bring
all hands on deck to create a war room,”
Corman says.
From the perspective of developers or
enterprise architects, however,
something like the
Heartbleed response
would be referred
to as unplanned,
unscheduled work
that probably
caused them to
be late on their
sprint or their epic,
leaving business
frustrated.
Enlightened
security professionals
should understand how
security problems that they
identify eventually turn into break-fix
nightmares or unscheduled work that
holds up current development work.
Become
Effective
Translators
AWS Security is NOT self-Evident!
It’s no surprise that DevOps teams love the flexibility, agility,
and cost efficiency of AWS. Security and compliance teams,
however, struggle to adapt traditional tools, processes, and
frameworks to public cloud infrastructure and the rapid
change created by the DevOps continuous delivery model.
Builtbyateamthathassecuredsomeoftheworld’slargestpublic
cloud infrastructures, the Evident Security Platform provides
security and compliance automation for AWS infrastructure. ESP
bridges the gaps between DevOps & IT Sec, and makes security &
compliance an integrated, automated part of your DevOps.
The Security and Compliance IT Requires…
at the Speed DevOps Needs!
Secure your AWS Cloud Today!
For a free 14 day trial, visit us at www.evident.io or call us at +1 (855)
933-1337 to schedule a demo with one of our cloud security experts. AWS Security at the Speed of DevOps
Risk Analytics &
Threat Visibility
Continuous monitoring
and risk-based threat
analysis of all AWS
Accounts, Services,
and Regions.
Continuous
Compliance
Adaptively manage
compliance & automate
policy enforcement
across your entire AWS
infrastructure.
DevOps Ready
Integrates with the
tools your DevOps
teams love. Open and
extensible design with
RESTful API and SDK
support.

8. Automation is absolutely the lynchpin
to the whole strategy.
“At the speed people want to move,
reliance on manual testing just doesn’t
cut it,” explained Damon Edwards, host of
the DevOps Café podcast and long-time
DevOps evangelist, last year in a talk
at OWASPAppSecUSA. “The external
inspection by a third party who is going to
manually test something is not
going to be enough to catch
all of our problems.”
This does not
necessarily mean that
penetration tests go
away altogether, but they
may shift in importance
as automated tests
become the tip ofthe spear
and manual tests become a
backstop. At minimum, security
teams may want to start thinking ofways
to automatically integrate manual testing
results back into the pipeline.
“Even if you don’t do anything else,
whenever you shell out your $50,000
orwhateverfor your PCI compliance
testing, tell the penetration testers that
at the end, you won’t read a PDF,” Wickett
says. “Tell them the only output you will
accept is integration tests written in
Cucumber, Gauntlt, Behave orwhatever
you want your stack to be, even if it’s just
Python scripts that test different parts
of my application to verify whatever it is
that needs verifying. Tell them you want
code delivered at the end ofthe
engagement.”
All ofthis is, of course,
easier said than done. As
Mogull explains, security
teams will need to vet their
tools for DevOps speeds
and processes, weeding out
some, retuning others and
bringing in newtools that are
purpose built forthe new bar set
by the continuous delivery model.
“If you’re getting a lot offalse
positives using a static analysis tool
and that’s impeding devvelocity, that’s
a problem,” Mogull says. “Some tools
aren’t well suited forworking in this
environment.”
On the tools side, embedding
security into DevOps for a truly Rugged
DevOps experience requires that
security teams find ways to
test early and often by
automating as much
testing as they can.
As Wickett explains,
current security
testing cycles are
really geared toward
waterfall approaches,
even as the rest of the
IT industry has drifted into
Agile and DevOps. And so things
like risk assessments and analysis,
plus PCI compliance checks tend to be
completely out of sync with DevOps
cycles.
“We need better security testing
earlier on in the development
process and inside the build
pipeline,” Wickett says.
“Right after you commit
code, you should have
unit tests that run,
functional tests that run,
integration tests that
run and, at that point, you
should also have a suite of
security tests that run. Right
now, those tests usually are run on
an annual basis or as the product is just
about to ship.”
Test Early
and Often
Automate,
Automate,
Automate

9. AT THE SPEED
PEOPLE WANT
TO MOVE,
RELIANCE
ON MANUAL
TESTING
JUST
DOESN’T
CUT IT

10. But Rugged DevOps isn’t simply
about automating security testing.
As Corman explains, it’s about
fundamentally changing the design
process to account for security from
the start. This means thinking about
how to insert security into requirements
and architecture discussions before any
code is written and instituting threat
modeling in the earliest stages
of development.
“DevOps is an
amplifier—it will amplify
the good and the
bad,” Corman says.
“It’s incumbent upon
us to measure twice
and cut once. You
want to be doing
threat modeling
and making resilient
defensible infrastructure
choices.”
As Corman puts it, for
too long the app sec community
has been used to being “the caboose
at the end of the process.” While
engaging early in the architecture and
requirements phase won’t eliminate
after-the-fact testing, security is going
to get a higher yield this way.
“That’s what security people are
there for,” Mogull says. “They might
not be great programmers, but there
are places where they could really help
with some threat modeling to identify if
something could be really bad in the big
scheme of things.”
Getting involved early, and also
taking advantage of feature flag and
keyword alerts early in the
development process can
help the security team
home in on the more
sensitive parts of
the code or the
architecture that
may require a
closer look than
what automated
tests will provide.
“For example,
you’ve got to pay
close attention to user
authentication and, in
microservices architectures,
dealing with component
authentication and authorization,” he
says. “So the entire identity and access
management model, plus anything
dealing with API sanitization and crypto
are a huge deal and are probably the top
three things that’d keep me up at night
when I’m building stuff.”
Engaging at
Architecture and
Requirements

11. Improve application quality
and performance 42%
34%
29%
28%
27%
25%
19%
14%
Improve end customer experience
Increasing use of mobile devices
Pressures to release applications
more quickly
Increase collaboration between
development and operations teams
Need to develop/deploy cloud-based
applications
Increasingly complex IT infrastructure
Greater need for simultaneous
deployment across different platforms
DEVOPSDEMANDDRIVERS
5%Need to reduce IT costs

12. As security tries to find its place in the
DevOps workflow, one of the best things
CISOs can do is try to get security staff
embedded within dev or ops teams, or
more formally involved in a DevOps pilot
project.
“What you see with the DevOps
people is that they’re boundary
spanners and they assume that
their teammates are going to
bring something to the mix
that they can’t do on their
own, so they’re looking
for your complimentary
contribution,” Corman
says.
As these security
representatives
approach their peers
across the organization,
Corman suggests they
come at it with the hearts
of ambassadors. When you
visit their land you should be
observing more than you talk, trying
to understand their culture and speaking
their language, while also offering some
understanding about your own land.
Ultimately the goal should be to look for
ways to change the security dynamic to
Participate in
Cross-Functional
Teams

13. help speak in the universal language of
Rugged DevOps: speed of delivery.
“There should be a really big shift
for security people, moving from an
approver and requester relationship with
DevOps guys to more of a facilitative
and partnership relationship,” says
Prendergast, “where developers and
operations are helping with security
and learning along the way, and
security is able to mentor and continue
to pass responsibility to the DevOps
guys, because automation is making it
possible to remove the human element
from lot of processes.”
As Edwards explained in his OWASP
talk last year, security would do well to
learn from many QA professionals who
have embraced their role as toolsmiths
and mentors within the DevOps value
chain. As toolsmiths, they do their best
to develop a toolset that enables as
much self-service help as possible. And
as mentors, security should be helping
to identify security champions who
can lift some of the everyday security
mitigation workload off the security
team’s shoulders, allowing them to
focus on more strategic work.
Wickett related a story about how
cheaply a security empire building
can be won. He explained that Ken
Johnson, while at Living Social, ran an
internal capture-the-flag, bug-finding
competition. The prize was a specially
designed, very limited-edition t-shirt.The
program was simple but effective.
“That’s a unique way to get people
interested in security,” he says. “It helps
you identify people in your organization
who you can tap in the future and
whenever the winners wear the shirt,
people see it and are more aware of
security.”

14. In its role as toolsmith, the security
team can get some of its biggest
wins by finding ways to expose
operational security tooling
to the rest of the
organization, Wickett
says.
“Security folks
have a ton of really
great technical
knowledge that
other technical
people just don’t
have,” Wickett
says. “It’s a shame
that it is sort of locked
away in that part of the
organization, and it’s not
exposed or helping their coworkers with
self help.”
Finding ways to embed security
tooling into the general ops toolkit can
amplify the positive influence of small
security teams that otherwise can’t
extend their operational reach across
large dev or ops organizations.
Giving DevOps teammates
insight from security
tools allows them to
get more work done
on the infosec team’s
behalf.
“You’re putting a
broader workforce
with a great set
of knowledge and
operational capabilities
on the front line of
resolving security issues,”
Prendergast says. “Then if
they need to escalate up, they’d do
that through security, just like they’d
escalate operational issues through
engineering.”
Open Up
Your Toolset to
the Rest of the
Organization

15. Often the biggest obstacles
thrown up by security pros
about DevOps is what they
perceive as continuous
delivery’s inability to support
separation of duties and
audit-worthiness. But as
Mogull explains, that’s not
necessarily true. To security
pros, it might look like
developers are pushing code
into production, but there’s
a whole lot of automated
testing and approval processes
built into the commit cycles.
And everything along the
way is typically tracked and
measured, not necessarily for
security’s sake, but to improve
performance along the way.
“If you’re really doing this
well, you should have individual
user accounts and granular
logs of any changes made.
That’s essential for a good
development process in the
first place,” Mogull says. “But
it can really help from the
Rugged DevOps standpoint.”
Not only will these audit
trails help backtrack when
security issues pop up, but
they’ll also keep the auditors
at bay.
“What this has accidentally
created is an audit log of who
did what, when, where, how
and why,” Corman says. He
explains that all security has
to do is find ways to fine tune
this data collection to satisfy
specific regulatory compliance
demands. “If you’re going to
instrument everything, why
don’t you instrument things
in such a way that they can
create evidence at a moment’s
notice for any auditor?”
TakeAdvantage
of “Accidental”
Audit Trails

16.  http://www.devops.com
 https://twitter.com/devopsdotcom
 https://www.facebook.com/devopscom
 https://instagram.com/devopsdotcom
 https://plus.google.com/+Devopsdotcom/posts