Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Rugged DevOps (eBook): 10 Ways to Start Embedding Security into DevOps Patterns


Published on

Evident is a sponsor of the inaugural eBook titled Rugged DevOps: 10 Ways to Start Embedding Security into DevOps Patterns. Learn more about how to start moving toward a Rugged DevOps mentality through insights shared by security and DevOps experts, including Evident CEO Tim Prendergast, with reporter Ericka Chickowski.

Published in: Technology
  • Be the first to comment

Rugged DevOps (eBook): 10 Ways to Start Embedding Security into DevOps Patterns

  2. 2. Security industry conferences and articles in the trade press are fond of harping on the broken IT security model. They point to growing breach statistics, embarrassing configuration and patch management practices, and mounting vulnerability exploits as ample evidence of the problems plaguing infosec. It’s easy to point out problems, but what about solutions? Interestingly, as scary as DevOps may be to the typical security pro, its fast iterations and continuous delivery patterns may be just the solution to fixing what’s broken in today’s security model. “Security pros need to understand that this is the future of IT and of IT security,” says Rich Mogull, analyst and CEO at security analyst firm Securosis. “I see DevOps doing nothing but improving security when done right. Not only is software going to be more secure, but if you learn from these techniques it can help you get your day-to-day security more efficient.” But the devil is in the details. What exactly does it take to do it right? How can security operate within the DevOps context in order to ensure a more sustainable and less risky continuous delivery value chain? The complete answers to those questions are still being written. But the innovators in the security and DevOps communities are finding early success with a number of key practices that contribute to what many security pundits call a Rugged DevOps experience.
  3. 3. Attackers are already using their own form of continuous delivery to overwhelm the good guys. The reason security teams can’t keep up is because the bad guys have already figured out how to use automation and cloud-style technologies to scale up their attacks, says Tim Prendergast, CEO of and a long-time DevOps and security practitioner. Predergast is best known for his former role, leading up cloud architecture and security forAdobe. “But the defenders are still relying on this model where you’ve got a person in front of a console,” says Prendergast. “They’re just outgunned and outnumbered, so they have to move into this automation philosophy. They have to make time for renovating their approach and they have to be open to new ideas.” This means getting unstuck from security patterns they’ve settled into for the past 15 or 20 years, says Prendergast. “DevOps is bringing a whole new approach to security to the table that’s actually way better than what we used to be doing,” he says, explaining that security teams will be able to pivot more quickly with attack trends if they learn how to deploy platforms and protection mechanisms in lean, iterative cycles rather than relying on “big-bang releases.” And rather than trying to be a ‘gating factor’ at the tail end of enterprise development, security should be baked into operational practices and automated tooling throughout the development pipeline. “A DevOps team will deploy code and when a bug pops up, they’re very well instrumented to gather telemetry on the bug, get it into repair and turn a fix around the same day and redeploy,” he says. This approach is in contrast to the old model of rolling back the entire deployment on the hunt for a pristine deployment, delaying necessary features for the sake of a whole list of bugs that may take days or weeks to take care of. Understand That Attackers Already Deliver Continuously What is Rugged Software? “Rugged” describes software development organizations which have a culture of rapidly evolving their ability to create available, survivable, defensible, secure, and resilient software. —From
  4. 4. The Rugged Manifesto I am rugged and, more importantly, my code is rugged. I recognize that software has become a foundation of our modern world. I recognize the awesome responsibility that comes with this foundational role. I recognize that my code will be used in ways I cannot anticipate, in ways it was not designed, and for longer than it was ever intended. I recognize that my code will be attacked by talented and persistent adversaries who threaten our physical, economic and national security. I recognize these things – and I choose to be rugged. I am rugged because I refuse to be a source of vulnerability or weakness. I am rugged because I assure my code will support its mission. I am rugged because my code can face these challenges and persist in spite of them. I am rugged, not because it is easy, but because it is necessary and I am up for the challenge. -From
  5. 5. DevOps cycle times can seem impossibly, dangerously fast to security veterans, who may hold the belief that code can’t possibly be secured without days-long static and dynamic analysis. But rule number one of achieving a more rugged DevOps process is not taking the ball away and going home when you don’t get your way. “You don’t throw up your hands and say ‘If they’re doing a daily build and it takes four days for feedback, I give up,’” says Josh Corman, CTO at Sonatype. “You ask if there’s some sniff test or some subset of tests we can do that conforms to the compressed cycle time. And maybe once a week, once a month or once a quarter we do a deeper thing.” Rather than entrenching themselves against new waves of continuous deployment and automation, security professionals need to “get clever about how to adapt,” says Corman. Organizations are finding as much as a 10x improvement in development speed through continuous delivery. They’re bringing new features to customers more quickly, freeing up resources to develop new lines of business fueled by digital innovation and clearing up their application backlogs. Security should be participating. “If you’re a speed bump, everyone will go around you,” warns Mogull. “Security leaders need to figure out the best way to get into the rugged DevOps process.” Don’t Throw Up Your Hands SECURITYwas named as the number one DevOps obstacle by 28 percent of enterprises
  6. 6. Before security pros even start to think about automated testing, new workflows or updated technology, they should begin by exercising their empathy muscles, says Corman. One way to build that empathy is to start thinking about what the common ground is between DevOps-minded practitioners and security professionals. “We both hate complexity. We want to fail fast, early and often so we can iterate,“ says Corman. “And we like to instrument everything—we all have an almost maniacal desire to get a ton of actionable telemetry.” Just as dev and ops have had to bridge some serious cultural gulfs to start collaborating better, security needs to meet the developers and operations teams half way. A longtime practitioner in both operations and security, as well as the primary developer of the open source automated security test framework Gauntlt, James Wickett recalls an eye- opening interaction he once had with a superior in an enterprise setting. “My boss at the time said, ‘You’re a security guy, so you probably just want all the machines turned off and unplugged,’” says Wickett, who is currently a senior engineer at Signal Sciences, a stealth-mode security firm working to build products that work in DevOps environments. “I was sort of offended by that, because I’m not an idiot and I’m not a nihilist.” Wickett, like many other security folks, understood that the business was trying to move forward with its goals. But the lesson was that there may be a reason for some of that misapprehension. As he puts it, a lot of security professionals will mutter something akin to ‘those stupid developers’ as they put out fires. If security pros want to move the needle on risk management, they need to help everyone come to a mutual understanding of the time pressures and restrictions on organizational resources that each group faces, Wickett says. “In particular, the security people need to know and learn and understand how development works,” agrees Mogull. “Obviously, there’s the operational side, which they tend to be more familiar with. But they also need to try to understand developer’s terms and processes, and use that to bring security into it. You don’t need to be a developer, but you need to know how they work.” Start With Mutual Understanding and Respect
  7. 7. Coming to a mutual understanding could start with semantics. Cooperation between security, developers and operations staff could be improved by overcoming some language barriers, Corman says. Take the Heartbleed vulnerability as an example. A security person might see the problem as a potential breach, a security risk or an incident response exercise. Meanwhile, an ops person would probably refer to it as a break-fix or a service interruption. “ [Ops doesn’t] care about security or risk, they’re concerned with being measured on the five nines of availability. They had an outage and they had to create a remedy ticket and bring all hands on deck to create a war room,” Corman says. From the perspective of developers or enterprise architects, however, something like the Heartbleed response would be referred to as unplanned, unscheduled work that probably caused them to be late on their sprint or their epic, leaving business frustrated. Enlightened security professionals should understand how security problems that they identify eventually turn into break-fix nightmares or unscheduled work that holds up current development work. Become Effective Translators AWS Security is NOT self-Evident! It’s no surprise that DevOps teams love the flexibility, agility, and cost efficiency of AWS. Security and compliance teams, however, struggle to adapt traditional tools, processes, and frameworks to public cloud infrastructure and the rapid change created by the DevOps continuous delivery model. Builtbyateamthathassecuredsomeoftheworld’slargestpublic cloud infrastructures, the Evident Security Platform provides security and compliance automation for AWS infrastructure. ESP bridges the gaps between DevOps & IT Sec, and makes security & compliance an integrated, automated part of your DevOps. The Security and Compliance IT Requires… at the Speed DevOps Needs! Secure your AWS Cloud Today! For a free 14 day trial, visit us at or call us at +1 (855) 933-1337 to schedule a demo with one of our cloud security experts. AWS Security at the Speed of DevOps Risk Analytics & Threat Visibility Continuous monitoring and risk-based threat analysis of all AWS Accounts, Services, and Regions. Continuous Compliance Adaptively manage compliance & automate policy enforcement across your entire AWS infrastructure. DevOps Ready Integrates with the tools your DevOps teams love. Open and extensible design with RESTful API and SDK support.
  8. 8. Automation is absolutely the lynchpin to the whole strategy. “At the speed people want to move, reliance on manual testing just doesn’t cut it,” explained Damon Edwards, host of the DevOps Café podcast and long-time DevOps evangelist, last year in a talk at OWASPAppSecUSA. “The external inspection by a third party who is going to manually test something is not going to be enough to catch all of our problems.” This does not necessarily mean that penetration tests go away altogether, but they may shift in importance as automated tests become the tip ofthe spear and manual tests become a backstop. At minimum, security teams may want to start thinking ofways to automatically integrate manual testing results back into the pipeline. “Even if you don’t do anything else, whenever you shell out your $50,000 orwhateverfor your PCI compliance testing, tell the penetration testers that at the end, you won’t read a PDF,” Wickett says. “Tell them the only output you will accept is integration tests written in Cucumber, Gauntlt, Behave orwhatever you want your stack to be, even if it’s just Python scripts that test different parts of my application to verify whatever it is that needs verifying. Tell them you want code delivered at the end ofthe engagement.” All ofthis is, of course, easier said than done. As Mogull explains, security teams will need to vet their tools for DevOps speeds and processes, weeding out some, retuning others and bringing in newtools that are purpose built forthe new bar set by the continuous delivery model. “If you’re getting a lot offalse positives using a static analysis tool and that’s impeding devvelocity, that’s a problem,” Mogull says. “Some tools aren’t well suited forworking in this environment.” On the tools side, embedding security into DevOps for a truly Rugged DevOps experience requires that security teams find ways to test early and often by automating as much testing as they can. As Wickett explains, current security testing cycles are really geared toward waterfall approaches, even as the rest of the IT industry has drifted into Agile and DevOps. And so things like risk assessments and analysis, plus PCI compliance checks tend to be completely out of sync with DevOps cycles. “We need better security testing earlier on in the development process and inside the build pipeline,” Wickett says. “Right after you commit code, you should have unit tests that run, functional tests that run, integration tests that run and, at that point, you should also have a suite of security tests that run. Right now, those tests usually are run on an annual basis or as the product is just about to ship.” Test Early and Often Automate, Automate, Automate
  10. 10. But Rugged DevOps isn’t simply about automating security testing. As Corman explains, it’s about fundamentally changing the design process to account for security from the start. This means thinking about how to insert security into requirements and architecture discussions before any code is written and instituting threat modeling in the earliest stages of development. “DevOps is an amplifier—it will amplify the good and the bad,” Corman says. “It’s incumbent upon us to measure twice and cut once. You want to be doing threat modeling and making resilient defensible infrastructure choices.” As Corman puts it, for too long the app sec community has been used to being “the caboose at the end of the process.” While engaging early in the architecture and requirements phase won’t eliminate after-the-fact testing, security is going to get a higher yield this way. “That’s what security people are there for,” Mogull says. “They might not be great programmers, but there are places where they could really help with some threat modeling to identify if something could be really bad in the big scheme of things.” Getting involved early, and also taking advantage of feature flag and keyword alerts early in the development process can help the security team home in on the more sensitive parts of the code or the architecture that may require a closer look than what automated tests will provide. “For example, you’ve got to pay close attention to user authentication and, in microservices architectures, dealing with component authentication and authorization,” he says. “So the entire identity and access management model, plus anything dealing with API sanitization and crypto are a huge deal and are probably the top three things that’d keep me up at night when I’m building stuff.” Engaging at Architecture and Requirements
  11. 11. Improve application quality and performance 42% 34% 29% 28% 27% 25% 19% 14% Improve end customer experience Increasing use of mobile devices Pressures to release applications more quickly Increase collaboration between development and operations teams Need to develop/deploy cloud-based applications Increasingly complex IT infrastructure Greater need for simultaneous deployment across different platforms DEVOPSDEMANDDRIVERS 5%Need to reduce IT costs
  12. 12. As security tries to find its place in the DevOps workflow, one of the best things CISOs can do is try to get security staff embedded within dev or ops teams, or more formally involved in a DevOps pilot project. “What you see with the DevOps people is that they’re boundary spanners and they assume that their teammates are going to bring something to the mix that they can’t do on their own, so they’re looking for your complimentary contribution,” Corman says. As these security representatives approach their peers across the organization, Corman suggests they come at it with the hearts of ambassadors. When you visit their land you should be observing more than you talk, trying to understand their culture and speaking their language, while also offering some understanding about your own land. Ultimately the goal should be to look for ways to change the security dynamic to Participate in Cross-Functional Teams
  13. 13. help speak in the universal language of Rugged DevOps: speed of delivery. “There should be a really big shift for security people, moving from an approver and requester relationship with DevOps guys to more of a facilitative and partnership relationship,” says Prendergast, “where developers and operations are helping with security and learning along the way, and security is able to mentor and continue to pass responsibility to the DevOps guys, because automation is making it possible to remove the human element from lot of processes.” As Edwards explained in his OWASP talk last year, security would do well to learn from many QA professionals who have embraced their role as toolsmiths and mentors within the DevOps value chain. As toolsmiths, they do their best to develop a toolset that enables as much self-service help as possible. And as mentors, security should be helping to identify security champions who can lift some of the everyday security mitigation workload off the security team’s shoulders, allowing them to focus on more strategic work. Wickett related a story about how cheaply a security empire building can be won. He explained that Ken Johnson, while at Living Social, ran an internal capture-the-flag, bug-finding competition. The prize was a specially designed, very limited-edition t-shirt.The program was simple but effective. “That’s a unique way to get people interested in security,” he says. “It helps you identify people in your organization who you can tap in the future and whenever the winners wear the shirt, people see it and are more aware of security.”
  14. 14. In its role as toolsmith, the security team can get some of its biggest wins by finding ways to expose operational security tooling to the rest of the organization, Wickett says. “Security folks have a ton of really great technical knowledge that other technical people just don’t have,” Wickett says. “It’s a shame that it is sort of locked away in that part of the organization, and it’s not exposed or helping their coworkers with self help.” Finding ways to embed security tooling into the general ops toolkit can amplify the positive influence of small security teams that otherwise can’t extend their operational reach across large dev or ops organizations. Giving DevOps teammates insight from security tools allows them to get more work done on the infosec team’s behalf. “You’re putting a broader workforce with a great set of knowledge and operational capabilities on the front line of resolving security issues,” Prendergast says. “Then if they need to escalate up, they’d do that through security, just like they’d escalate operational issues through engineering.” Open Up Your Toolset to the Rest of the Organization
  15. 15. Often the biggest obstacles thrown up by security pros about DevOps is what they perceive as continuous delivery’s inability to support separation of duties and audit-worthiness. But as Mogull explains, that’s not necessarily true. To security pros, it might look like developers are pushing code into production, but there’s a whole lot of automated testing and approval processes built into the commit cycles. And everything along the way is typically tracked and measured, not necessarily for security’s sake, but to improve performance along the way. “If you’re really doing this well, you should have individual user accounts and granular logs of any changes made. That’s essential for a good development process in the first place,” Mogull says. “But it can really help from the Rugged DevOps standpoint.” Not only will these audit trails help backtrack when security issues pop up, but they’ll also keep the auditors at bay. “What this has accidentally created is an audit log of who did what, when, where, how and why,” Corman says. He explains that all security has to do is find ways to fine tune this data collection to satisfy specific regulatory compliance demands. “If you’re going to instrument everything, why don’t you instrument things in such a way that they can create evidence at a moment’s notice for any auditor?” TakeAdvantage of “Accidental” Audit Trails
  16. 16.     