A talk on how to apply Lean Software principles to information security. Also delivered at RSA 2015 and OWASP Austin. Converted to ppt from Keynote so sorry about the fonts.
4. @ERNESTMUELLER // THEAGILEADMIN.COM //
#LEANSECURITY
COMPANIES ARE SPENDING A GREAT
DEAL ON SECURITY, BUT WE READ
OF MASSIVE COMPUTER-RELATED
ATTACKS. CLEARLY SOMETHING IS
WRONG.
THE ROOT OF THE PROBLEM IS
TWOFOLD: WE’RE PROTECTING (AND
SPENDING MONEY ON PROTECTING)
THE WRONG THINGS, AND WE’RE
HURTING PRODUCTIVITY IN THE
PROCESS.
Source: Thinking Security (2005), Steven M. Bellovin
7. @ERNESTMUELLER // THEAGILEADMIN.COM //
#LEANSECURITY
WHAT IS AGILE?
• INDIVIDUALS AND INTERACTIONS
OVER PROCESSES AND TOOLS
• WORKING SOFTWARE
OVER COMPREHENSIVE DOCUMENTATION
• CUSTOMER COLLABORATION
OVER CONTRACT NEGOTIATION
• RESPONDING TO CHANGE
OVER FOLLOWING A PLAN
SOURCE: THE AGILE MANIFESTO
(HTTP://WWW.AGILEMANIFESTO.ORG/)
8. @ERNESTMUELLER // THEAGILEADMIN.COM //
#LEANSECURITY
WHY AGILE?
• 45% OF ORGANIZATIONS ARE USING AGILE ON A MAJORITY OF
THEIR TEAMS
ONLY 5% ARE NOT USING IT AT ALL
• AGILE RESULTS:
• ACCELERATE PRODUCT DELIVERY - 59%
• ENHANCE ABILITY TO MANAGE CHANGING PRIORITIES - 56%
• INCREASE PRODUCTIVITY - 53%
• ENHANCE SOFTWARE QUALITY - 46%
• ENHANCE DELIVERY PREDICTABILITY - 44%
SOURCE: VERSIONONE NINTH ANNUAL STATE OF AGILE SURVEY
(HTTPS://WWW.VERSIONONE.COM/PDF/STATE-OF-AGILE-DEVELOPMENT-SURVEY-NINTH.PDF)
11. @ERNESTMUELLER // THEAGILEADMIN.COM //
#LEANSECURITY
WHAT IS DEVOPS?
DEVOPS IS THE PRACTICE OF OPERATIONS AND
DEVELOPMENT ENGINEERS PARTICIPATING
TOGETHER IN THE ENTIRE SERVICE LIFECYCLE,
FROM DESIGN THROUGH THE DEVELOPMENT
PROCESS TO PRODUCTION SUPPORT.
DEVOPS IS ALSO CHARACTERIZED BY
OPERATIONS STAFF MAKING USE MANY OF THE
SAME TECHNIQUES AS DEVELOPERS FOR THEIR
SYSTEMS WORK.
SOURCE: THE AGILE ADMIN: WHAT IS DEVOPS?
HTTP://THEAGILEADMIN.COM/WHAT-IS-DEVOPS/
12. @ERNESTMUELLER // THEAGILEADMIN.COM //
#LEANSECURITY
WHY DEVOPS?• BY 2016 “DEVOPS WILL EVOLVE FROM A NICHE TO A MAINSTREAM
STRATEGY EMPLOYED BY 25% OF GLOBAL 2000 ORGANIZATIONS” -
GARTNER, MARCH 2015
• BENEFITS OF DEVOPS:
• NEW SOFTWARE/SERVICES THAT WOULD OTHERWISE NOT BE
POSSIBLE - 21%
• A REDUCTION IN TIME SPENT FIXING AND MAINTAINING APPLICATIONS
- 21%
• INCREASED COLLABORATION BETWEEN DEPARTMENTS - 21%
• AN INCREASE IN REVENUE - 19%
• IMPROVED QUALITY AND PERFORMANCE OF OUR DEPLOYED
APPLICATIONS - 19%
SOURCE: CA RESEARCH REPORT—DEVOPS: THE WORST-KEPT SECRET TO WINNING IN THE
APPLICATION ECONOMY (HTTP://REWRITE.CA.COM/US/ARTICLES/DEVOPS/RESEARCH-REPORT--
DEVOPS-THE-WORST-KEPT-SECRET-TO-WINNING-IN-THE-APPLICATION-ECONOMY.HTML)
13. @ERNESTMUELLER // THEAGILEADMIN.COM //
#LEANSECURITY
HIGH-PERFORMING IT
ORGANIZATIONS
EXPERIENCE 60X FEWER
FAILURES AND RECOVER
FROM FAILURE 168X FASTER
THAN THEIR LOWER-
PERFORMING PEERS. THEY
ALSO DEPLOY 30X MORE
FREQUENTLY WITH 200X
SHORTER LEAD TIMES.
15. @ERNESTMUELLER // THEAGILEADMIN.COM //
#LEANSECURITY
LEAN SOFTWARE
DEVELOPMENTSEVEN PRINCIPLES:
• ELIMINATE
WASTE
• AMPLIFY
LEARNING
• DECIDE AS LATE
AS POSSIBLE
• DELIVER AS FAST
AS POSSIBLE
• EMPOWER THE
TEAM
• BUILD INTEGRITY
IN
• SEE THE WHOLE
AN SOFTWARE DEVELOPMENT: AN AGILE TOOLKIT (2003), MARY AND TOM POPPENDIECK
16. @ERNESTMUELLER // THEAGILEADMIN.COM //
#LEANSECURITY
LEAN PRODUCT
DEVELOPMENT• BUILD-MEASURE-LEARN
• BUILD – MINIMUM VIABLE PRODUCT
• MEASURE – THE OUTCOME AND INTERNAL
METRICS
• LEARN – ABOUT YOUR PROBLEM AND YOUR
SOLUTION
• REPEAT – GO DEEPER WHERE IT’S NEEDED
SOURCE: LEAN STARTUP (2011), ERIC RIES
17. @ERNESTMUELLER // THEAGILEADMIN.COM //
#LEANSECURITY
WHY LEAN?
• BOTH DEVOPS AND AGILE BORROW KEY
CONCEPTS FROM LEAN MANUFACTURING, SO
IT'S ALL ABOUT COMMUNICATION AND
OPENNESS." -INFORMATIONWEEK
24. @ERNESTMUELLER // THEAGILEADMIN.COM //
#LEANSECURITY
“[RISK ASSESSMENT]
INTRODUCES A DANGEROUS
FALLACY: THAT STRUCTURED
INADEQUACY IS ALMOST AS
GOOD AS ADEQUACY AND
THAT UNDERFUNDED
SECURITY EFFORTS PLUS
RISK MANAGEMENT ARE
ABOUT AS GOOD AS
PROPERLY FUNDED SECURITY
WORK”SOURCE: THE TANGLED WEB (2011), MICHAEL ZALEWSKI
27. @ERNESTMUELLER // THEAGILEADMIN.COM //
#LEANSECURITY
A SECURITY MANAGEMENT SYSTEM PROVIDES OPTIMAL
VALUE TO THE ORGANIZATION IF IT:
• ACTIVELY SUPPORTS ACHIEVING THE BUSINESS AND
COMPLIANCE OBJECTIVES OF THE ORGANIZATION
(THE VARIABLE PART)
• IS AN EFFICIENT, AGILE AND INTEGRATED PROCESS,
CAPABLE OF DEALING WITH A DYNAMIC THREAT
ENVIRONMENT
• CONSUMES MINIMAL TIME AND RESOURCES
• RESULTS IN ADEQUATELY MANAGED SECURITY RISK,
IN LINE WITH THE RISK APPETITE OF THE
ORGANIZATION
• PROVIDES ONLY THE NECESSARY, YET ADEQUATE,
USER FRIENDLY, EFFICIENT AND MEASURABLE
SECURITY CONTROLS
SOURCE: JOHAN BAKKER, LEAN SECURITY MANAGEMENT WHITE PAPER
30. @ERNESTMUELLER // THEAGILEADMIN.COM //
#LEANSECURITY
THE AVERAGE TIME TO
DELIVER CORPORATE IT
PROJECTS HAS INCREASED
FROM ~8.5 MONTHS TO OVER
10 MONTHS IN THE LAST 5
YEARS
Revving up your Corporate RPMs, Fortune Magazine, Feb 1, 2016
31. @ERNESTMUELLER // THEAGILEADMIN.COM //
#LEANSECURITY
WHY ARE COMPANIES SO SLOW?
THE GROWTH OF CONTROL AND
RISK MANAGEMENT FUNCTIONS
WHICH IS TOO OFTEN POORLY
COORDINATED… RESULTING IN] A
PROLIFERATION OF NEW TASKS IN
THE AREAS OF COMPLIANCE,
PRIVACY AND DATA PROTECTION.
Revving up your Corporate RPMs, Fortune Magazine, Feb 1, 2016
33. @ERNESTMUELLER // THEAGILEADMIN.COM //
#LEANSECURITY
THE THREE WASTES
• MUDA - WORK WHICH ABSORBS RESOURCE
BUT ADDS NO VALUE
• MURI - UNREASONABLE WORK THAT IS
IMPOSED ON WORKERS AND MACHINES
• MURA - WORK COMING IN DRIBS AND
DRABS WITH SUDDEN PERIODS OF RUSH
RATHER THAN A CONSTANT OR REGULAR
FLOW, UNEVENNESS.
34. @ERNESTMUELLER // THEAGILEADMIN.COM //
#LEANSECURITY
SECURITY WASTE
MUDA COMES IN SEVEN FORMS:
• EXCESS INVENTORY - DUMPING YOUR THOUSAND
PAGE PDF OF VULNERABILITIES ON A BUSY TEAM.
PRIORITIZE AND LIMIT WORK IN PROGRESS (WIP)
• OVERPRODUCTION - SECURITY CONTROLS STEMMING
FROM FUD OR MISALIGNMENT WITH BUSINESS NEEDS
(NOT DEMANDED BY ACTUAL CUSTOMERS) - CF.
PHOENIX PROJECT
• EXTRA PROCESSING - FOR EXAMPLE, RELYING ON
COMPLIANCE TESTING RATHER THAN DESIGNING THE
PROCESS TO ELIMINATE PROBLEMS - HELP IT GET
BUILT RIGHT FIRST
35. @ERNESTMUELLER // THEAGILEADMIN.COM //
#LEANSECURITY
SECURITY WASTE
• HANDOFFS - LEVERAGE THE KNOWLEDGE OF THE TEAMS
DOING THE WORK AND COLLABORATE WITH THEM TO
BUILD SECURITY IN, INSTEAD OF THAT BEING SOME
OTHER TEAM’S JOB
• WAITING - LAG BETWEEN VALUE STEPS WAITING FOR
APPROVALS OR ANALYSES OR TICKET HANDLING - USE
SELF SERVICE AUTOMATION INSTEAD
• TASK SWITCHING - THE THOUSAND PAGE PDF AGAIN -
WORK WITH THEIR WORK INTAKE PROCESS NOT AGAINST
IT
• DEFECTS - FALSE POSITIVES AND FALSE NEGATIVES AND
JUST PLAIN UNIMPORTANT FINDINGS YOU REPORT
CAUSING ZERO-VALUE REWORK
40. @ERNESTMUELLER // THEAGILEADMIN.COM //
#LEANSECURITY
PERFORMANCE
• BROWSER EXTENSIONS FOR DEVS TO
UNDERSTAND PERFORMANCE PROBLEMS
• RESEARCH SHOWING PERFORMANCE TO
REVENUE CORRELATION
• SEARCHABLE LOGS EMITTING STATSD METRICS
• CONFERENCES COMBINING FRONT END DEVS AND
SYS ADMINS
• COMMITMENT TO INSTRUMENT AND GRAPH ALL
THE THINGS
41. @ERNESTMUELLER // THEAGILEADMIN.COM //
#LEANSECURITY
SECURITY
• BROWSER EXTENSIONS FOR DEVS TO
UNDERSTAND SECURITY PROBLEMS
• RESEARCH SHOWING SECURITY TO REVENUE
CORRELATION
• SEARCHABLE LOGS EMITTING STATSD METRICS
• CONFERENCES COMBINING DEVS, OPS, AND
SECURITY
• COMMITMENT TO INSTRUMENT AND GRAPH ALL
THE THINGS
42. @ERNESTMUELLER // THEAGILEADMIN.COM //
#LEANSECURITY
SEE THE WHOLE
• KEEP MEANINGFUL METRICS, MAKE THOSE
METRICS VISIBLE - IN CONTEXT OF
WORKERS’ TOOLCHAIN
• “LEAST PRIVILEGE” NEEDS TO BE
UNLEARNED SOMEWHAT IN MODERN
ORGANIZATIONS TO ALLOW EFFECTIVE
INFORMATION SHARING
• GET IN BUSINESS OF SHARING AND ADDING
VISIBILITY TO DEV AND TO OPS.
45. @ERNESTMUELLER // THEAGILEADMIN.COM //
#LEANSECURITY
– W. EDWARDS DEMING
“CEASE DEPENDENCE ON MASS
INSPECTION TO ACHIEVE QUALITY.
IMPROVE THE PROCESS AND BUILD
QUALITY INTO THE PRODUCT IN THE
FIRST PLACE."
48. @ERNESTMUELLER // THEAGILEADMIN.COM //
#LEANSECURITY
@slow @final
Feature: Look for cross site scripting (xss) using arachni
against a URL
Scenario: Using arachni, look for cross site scripting and
verify no issues are found
Given "arachni" is installed
And the following profile:
| name | value |
| url | http://localhost:8008 |
When I launch an "arachni" attack with:
"""
arachni —check=xss* <url>
"""
Then the output should contain "0 issues were detected."
Given
When
Then
What?
AN ATTACK LANGUAGE FOR DEVOPS
54. @ERNESTMUELLER // THEAGILEADMIN.COM //
#LEANSECURITY
ARE YOU “THAT
GUY?”• YOU ALREADY KNOW
YOU CAN’T MAKE
THINGS SECURE BY
YOURSELF
• YOU NEED EVERYONE
ELSE TO COOPERATE
WITH YOU
• BUT DOES IT SEEM
LIKE THE THINGS YOU
DO JUST ANGER
THEM?
55. @ERNESTMUELLER // THEAGILEADMIN.COM //
#LEANSECURITY
EMPOWER THE TEAM
• UNDERSTAND HUMAN
MOTIVATION
• NETFLIX AUTOMATION
CREATED SAFE PATHS
AS THE DEFAULT
• AUTOMATING PROCESS
REMOVES EMOTIONAL
CHARGE
Hello LASCON! James Wickett and I created this presentation for RSA’s Rugged DevOps track this year. James and I are both actively involved with the DevOps community (we run DevOpsDays Austin, coming in May). We blog together at theagileadmin.com.
DevOps changed my life and I’m here to share how understanding the same Lean techniques that have revolutionized development and operations can improve the effectiveness of your security work.
In the recent book by Steven Bellovin, “Thinking Security,” he points out that we are failing in our attempts to turn security spend into real security because we’re spending limited resources on the wrong things.
Even when we have the bad guys outnumbered, they still seem to slip through our fingers, not necessarily through any skill of their own.
To talk about Lean, we also want to talk about Agile and DevOps because they’re closely interrelated.
Most of you probably have an understanding of Agile. You have seen it done right, seen it done wrong… There’s a lot more to it, but this is the core manifesto.
Agile has become widespread in the industry. Why? Because it’s provided clear benefits to software development in organizations.
Next, DevOps!
We had a problem with development and operations being siloed and chasing opposing goals.
DevOps was conceptualized as a way to correct that. Tom Limoncelli’s book The Practice of Cloud System Administration says that “DevOps is the application of Agile methodology to system administration.”
We are no longer having conversations about whether it is or isn't a thing. We are now talking about DevOps adoption and how it’s quickly growing, and is showing the same kind of benefits to adopting organizations.
Gene Kim’s IT Revolution Press sponsors a yearly study on DevOps, compiled by a data scientist, and it reveals that companies are getting real ORDERS OF MAGNITUDE value out of devops.
And that brings us to Lean. Who’s already familiar with Lean in some context (manufacturing, Six Sigma, software development, or product development)?
What is Lean? Lean started off by revolutionizing the world of manufacturing, as pioneered by W. Edwards Deming, and later popularized in the Toyota Production System, but since then it has been adapted to software development. Its practices include value stream mapping, waste, pull, queueing theory, human motivation, measurement and visualization of metrics, TDD… We’ll go over many of these in the context of improving security work later in the presentation.
Eric Ries also applied lean principles to product development in his book Lean Startup, which characterizes the core loop inside the product development cycle as “Build – Measure – Learn.” Lean is about bringing your effort to bear on the item with the highest leverage at any given time.
The Puppet State of DevOps Report says: “One can describe DevOps as the pattern that emerges when you apply these same lean principles to technology.” Lean product, lean software, agile, and devops all come together into a single mutually reinforcing picture for a technology organization.
So let’s get to the security part of all this.
If you look at every new innovation, whether it’s Lean, devops, cloud, social, mobile, etc. only through the filter of how “it is a threat” to security then you’ve adopted a losing mindset out of the gate. We’re not going to spend any time on this because it’s a red herring.
Every single field has to innovate to stay relevant, and InfoSec doesn’t get a pass on that. IT operations had stagnated in innovation for a decade, and part of the impetus behind the adoption of DevOps is that just plain wasn’t acceptable any more. Being the BOFH saying “no” is no longer a viable approach; businesses won’t stand for it any more.
We’ll now examine six common challenges faced by InfoSec organizations and explain how you might be able to bring Lean to bear on implementing security more effectively in your organization. Each of these is very similar to a problem that we had in Ops and that seemed intractable before we found DevOps.
Each one of these issues is a perception you have probably heard from someone at some point. While these are not all fair, they are also not completely unfounded. The first complaint is that you’re just there to check boxes and don’t do much to make the apps and systems really more secure.
In his book on browser security, Michael Zalewski has a great intro covering the history of information security and he poignantly notes that at some point we decided risk management was able to fill the development and operations gaps we experienced. We became experts of structured inadequacy and wrapping problems with policies and “Accept the risk” statements.
This doesn’t create value. Let’s talk about value.
The foundation of Lean is to understand what the real value add steps are in the creation of your product or service. This is an example value stream mapping representing the value creation steps between the initiation of a process and delivery to the customer. In Lean Software, this is called the “Concept to Cash” flow.
So take a moment to think about this. Do you understand exactly what value you are creating and where? Is it just a compliance checkbox at the end of the process? Where could you be creating value?
Johan Bakker took a stab at what the true value of security management looks like to an organization. Note that it hinges on creating a solution that is custom to your organization. Whatever stock answers you got taught in Security School are not necessarily the value your customers need from you.
Where can you add real security value that improves the value of your organization’s core business? Years ago at National Instruments, I worked on the FPGA Compile Cloud product, a service where customers would upload their very sensitive IP to us to compile. By aligning our security work with end customer concerns, we had a security stance that was used as a positive marketing message and greatly facilitated adoption - which is another word for “sales”.
The second complaint is that security is just a bottleneck to getting “the real work done.”
Fortune Magazine reported just last month that the average time to deliver corporate IT Projects has increased from ~8.5 months to over 10 months in the last 5 years.
And the reason is… Security. Security has resulted in a proliferation of new work that can slow everything else down, if it’s “poorly coordinated.” What does that mean?
Waste. Lean focuses a lot on the identification and removal of waste; it’s the very first Lean principle. In today’s business environment time is a critical resource, and to be honest, Security is often guilty of squandering it.
Lean talks about three types of waste. Muda is the most discussed, though you can see how you might see muri (unreasonable demand, over-pressure) and mura (lack of flow) in security’s interaction with the rest of the organization.
The seven forms of muda can be seen in security operations. Here are a couple security-centric examples, but the takeaway is to analyze what you’re doing and identify the areas of waste in it.
Muda is separated into two types - “Type 1” is necessary - like compliance - but does not add value; “Type 2” is just plain unneeded. Even if you’re just focused on the “necessary” ones, do you really want to describe your career at family gatherings as “Type 1 waste?”
Your net value to the organization is the value you create minus the waste that you generate.
People are tempted to see security as a solution in search of a problem when they don’t see how it fits in to everything.
Security is everyone’s job, right?
In Operations, Performance used to be invisible and we would say performance was everyone’s job… And then no one did anything about it. Security has a lot of corollaries to application performance problems 5-10 years ago.
The Web and app performance field finally stopped passing the buck around and decided to help people actually see and address the problems, mainly by focusing on visualizing performance metrics directly in context to workers. Steve Souders, who served as head performance guru at Yahoo and Google, drove a movement to understand how to make high performance Web sites via tooling like YSlow, founding the Velocity conference, and urging companies like Microsoft and Google to share their research on performance impact to revenue.
We could do the same thing with security… Some of these things already exist, but are you using them?
This is an example of another Lean principle, “See the whole”; lean software development speaks extensively about metrics collection and visualization. You have to adopt a more open “broadcast” mentality to empower education and collaboration.
If you just add visibility to yourself and some executive manager - well, one of you better start writing the code, or no one else gives a rat’s ass.
It’s easy to get forced into reacting after there’s a breach, or even just after a vulnerability has gone live into production.
Deming addressed this issue very early in Lean. This gave rise to the “Build Integrity In” Lean software principle.
The more you can create fast feedback loops which detect and remediate security problems continually as part of your engineers’ normal work process, the less waste you generate.
Integrate into continuous integration and use test driven development (TDD) to rectify issues at the lowest waste point - upstream, or “shifted left” as the kids say nowadays. One tool that helps you do this is an open source project called gauntlt.
Many of you may have seen James present on gauntlt previously. It allows you to use the same gherkin syntax that several popular test driven development tools like cucumber and robot use to specify the security requirements of a piece of software and run it as a test along functional tests.
So as the code goes through the developers’ CI pipeline, it gets security tests applied to it prior to the build passing.
This provides continuous feedback at a very close point (not as close as the IDE, but close) - and therefore enables positive change with the least amount of waste.
There’s a whole workshop tutorial if you want to learn more about gauntlt.
Don’t wait till the end. Figure out what could be done in conjunction with each of those value creation steps.
It’s one thing to be a bottleneck due to constraints, but I think it’s fair to say many engineers consider security to be actively messing with them.
Adrian Cockroft from Netflix claimed they had “No process” in his talk at AppSec when it came here to Austin. What he really meant is that they have made doing the right thing a part of the systems everyone uses, so the perception is that there’s no process.
Use automation to build integrity in, don’t rely on mass inspection after things have already been done wrong.
Security is a product, like any other. And all products have to make tradeoffs about what they will do and what they won’t do.
If you listen to some folks you can’t “do security” without the direct backing of the CEO and a $1.2M budget to get the six or seven huge products you need, and that’s the way to achieve perfection. What if you can’t get that? Even if you can, many tools not only may be not a good fit for your needs, but also take a lot of time to implement. In a very real sense… Mo’ money, mo’ problems.
Lean Startup describes the solution to this efficiency problem. Rather than add additional waste via long analysis cycles, implement something small and fast, analyze the results, and iterate. Often layers of a couple imperfect items yields better security than one “perfect” item.
Security is a product. DevOps realized that performance, reliability, deployment, etc. was a product with defined customers and a need for active management and prioritization. Do the same thing with security. Manage it like a product manager manages all the other aspects of a product.
One of the reasons there’s been a DevOps track at this conference since 2012 is that not too many years ago we ops engineers were in the same situation and had all the exact same criticisms leveled at us. Operations was the at-best-invisible, beancounting bottleneck that was always a day late and a dollar short. But these time-tested principles has helped our entire industry begin to innovate its way out of that rut. Check them out and see if they can help you in the same way.
Thanks for your time! You can find more of our thoughts on DevOps, Lean, Security, and other topics at theagileadmin.com.