By show of hands, who here is security practitioner at their organization? (Some will raise their hands) By show of hands, how many of you are somewhat responsible for security? (Hopefully more will raise their hands) How many of you are Developers, Operations, Networking, Systems professionals? If you failed to raise your hand when we asked if you are responsible for security at your organization, then we need to help change your mind. EVERYONE is at least in part responsible for security. Have the security professionals raise their hands again… Point out how outnumbered they are compared to developers, operations, networking, and systems people. Security is a VAST domain and generally understaffed, overworked, and under-appreciated. Your security colleagues have to deal with regulatory concerns, document security, technology security, evaluating software, physical security, business continuity, disaster recovery planning, and more!
They would have to! They not only have to evaluate software being written by developers, they have to: Monitor and evaluate desktop security Document security Physical security E-Mail security Network Security Firewalls Disaster Recovery and Business Continuity Planning etc…
They’re understaffed and overwhelmed! Just like modern software development has been experiencing in the last few years, Information Security needs a GAME CHANGER!
For many years now I have been working in and around the practice of information security. From being a security auditor for the United States Navy, to getting my CISSP certification, to helping Red Hat customers learn about DevOps and how it doesn’t just start and end with Dev and Ops. I want to take you an a little journey of how a lowly security manager who was overwhelmed became a rock start, saved the company money, saved her sanity and family life, and became happier.
I know that the industry is using the terms DevOps and SecDevOps or DevSecOps these days, but I would like to submit that those terms are invalid. They imply that you put some Devs and some Ops people together with some security people and you’re done. The simple truth is that this is about knocking down communication and collaboration barriers throughout the enterprise. At Open Innovation Labs, we have been using the term “Long Lived Product Teams”, and the implication is that EVERYTHING which is needed for the long-term success of that product and team is part of the process. It may include Developers, Systems Administrators, Security Professionals, Network Engineers, Designers, Marketing, Legal, and perhaps even (GASP!) “END USERS”!!! Seriously, what would you call that? DevSecNetUserLegalDesignMarketingOps? Also, there is NO FINISH LINE… You’re never done until the product or the organization no longer exists! You can ALWAYS improve on what you have.
Let’s get back to Anne. She has been paying attention to the IT world around her, and she has noticed that people are doing a LOT of automation these days. Why can’t she automate security related things? Using tools like Jenkins and Selenium in conjunction with security tools like OWASP Zed Attack Proxy or Dependency Check or Snyk or SonarQube or MANY OTHER TOOLS!Most of the time, Anne’s job is just using some tool to evaluate the security of an application or a configuration. Why can’t that be automated?
Anne starts to think about her idea and realizes that it’s just too much! She doesn’t have time to automate all of these things and still handle her workload! She thought that she had a great idea, but the real-world implementation is just beyond reach. But Anne is tough and tenacious… She’s not going to give up that easily.
So Anne forms a plan… She’s going to approach one of the Agile Coaches at BigCorp and try to make a new friend. She’ll talk to that Agile Coach about how, if they work together, they can make it possible to reduce re-work, speed up delivery of new features, and help everyone become more capable in the realm of security. All of which are goals which should be exciting for ANY agile coach.They will start small, just automating one tool at first, and integrating that tool into the team’s Jenkins pipeline. Once that team integrates the tool, they can share that configuration with other teams. By starting small, the time investment is minimal, but the potential value is immense.
Anne’s first experiment was a ROARING success. The team whom she worked with now has a tool integrated into their Jenkins pipeline which scans their libraries for known vulnerabilities. They have even shared that snippet of pipeline code with a few other teams and they are all using it! AWESOME!
But how can we scale this and continue improving? It’s great that one thing is automated into the pipeline, but what about all of the other things which are constantly eating up her time and focus? How can scale?
Anne goes back and talks to her new friend, the agile coach. The agile coach explains that they have “Backlog Grooming” and “Sprint Planning” before the start of each of their sprints. It’s a chance for everyone to discuss their needs, the needs of the business, and negotiate to determine what will be worked on. Anne decides that she’s going to attend one of these sessions. During the Grooming and Sprint Planning, she gets a better idea of how the developers work together and prioritize work. She also gets a better understanding about what is being worked on. One developer is planning on adding a better authentication mechanism to the application, which Anne is SUPER excited about. But how can we test that? The developer tells Anne that they use Selenium to automate tests of the web application, and that it actually runs a real browser to try things out and verify functionality. Anne realizes that she does those sorts of things manually, but she runs her browser through Zed Attack Proxy. She asks the developer if they could run their tests through ZAP as well? The answer is YES!
From that moment on, Anne decides that she or one of her security teammates should attend Backlog Grooming and Sprint Planning sessions from time to time.
Over time, Anne and her security team start working more closely with the developers, network engineers, DBAs, Systems Administrators, etc… With small incremental improvements, most of the work which was eating up Anne’s valuable time gets automated and those automated configurations are shared across every team. Every time something new is automated, that is more time freed up for Anne and her team to think about and address new emerging threats. Additionally, the automation and pipelines help teach the other groups within BigCorp about security requirements. They get fast feedback about their work instead of having to wait days, weeks, or months for security to evaluate their products.
Anne and her security team have been attending agile ceremonies from time to time and they now better understand what is coming in the future, and they are able to provide guidance before teams implement something which doesn’t comply with security requirements. Also, when teams come up with new tools and libraries they want to use, they can just develop automated tests and scan them for vulnerabilities. She no longer has to say “NO” immediately, because now the people who were filling up her time with work are now acting almost as part of her team and simplifying the process.
Uh oh…. One of the applications which BigCorp is running was breached. Anne is panicking! The lawyers have descended like vultures, and Anne may lose her job. Security breaches at BigCorp are a serious business because of regulatory issues. Could all of Anne’s hard work in automating security have been for naught? Did her automation let something slip through?
Anne stops to think for a minute… All of her automations keep records of everything which was analyzed. She has extensive records of everything. As she sits down with the lawyers and auditors, she pulls up the automated analysis reports for everything; especially the compromised application.
As they are looking through the reports there is a ding on Anne’s phone. One of her security teammates has just run a static code analysis on the software which is used to run their application and it has identified a zero-day flaw! Nothing Anne could have done with security and pipelines would have caught the problem. Automation and pipelines cannot catch everything, but they can seriously improve your ability to respond to threats.
Because of the evidence and because Anne had done more than any other security manager in the company’s history; her job was safe. BigCorp could prove that it had done everything possible to prevent the breach, and therefore was not exposed to lawsuits.
Anne now can invest her valuable time in trying to detect, analyze, and mitigate threats like the zero-day flaw which had caused the company so much trouble. She has some of here security teammates start running static code analyses on the various open source tools which are used at BigCorp. Based on those findings, Anne is able to publish a number of papers in the security journals and she is lauded as being ahead of her time.
Anne is actually a real person though… Anne is one of my colleagues at Red Hat who works in Public Sector, consulting on government and DoD projects. The concepts and practices explained in her story are real practices we have helped customers to implement in their organizations, organizations like: The Department Of Homeland Security, Lockheed Martin, financial firms, banks, and many others. I’m not going to tell you that any of this is easy, but individual and incremental steps CAN BE. Start small, build incrementally, improve over time, and NEVER stop experimenting.
OWASP Dependency Check is one of the simplest tools you can add to a Java project in order to add some security to your pipeline. Simply adding this plugin to a Maven POM file and defining a threshold (failBuildOnCVSS) allows you to fail a build when critically vulnerable libraries are found in your project. In addition, you can capture reports for each build with a view of the known CVEs and CWEs for libraries you are using. For projects which are somewhat less active, you could even set up scheduled builds for your pipeline which will run and alert you to newly discovered vulnerabilities on a pro-active basis. Set the build to run as a “Canary Build” daily or weekly, even if there are no changes, and the plugin will check for newly discovered vulnerabilities in your existing libraries.
OWASP Dependency Check pulls identifying information from the NIST database of CVEs and CWEs every time you run a build. It’s ALWAYS as up-to-date as possible, so you will know very quickly if you are using a vulnerable library in your projects.
For NodeJS or Front-end projects, you can use `npm audit` to accomplish much the same as OWASP Dependency check in your pipeline.
SonarQube is a tool which can perform static code analysis on a number of different languages. It requires a deployed SonarQube server or an account on SonarCloud.com. In your pipeline, you can use the SonarQube Jenkins plugin to check the quality gate to ensure that you are meeting the defined standards for your project. SonarQube server comes with a set of very reasonable defaults for most projects. If the defaults do not meet your needs, it is quite easy to define your own quality profiles and quality gates for your projects or globally for the SonarQube instance.
If you are running integration tests against your applications, you can use OWASP Zed Attack Proxy to perform both passive and active scans of your systems. If you have ZAP deployed (As a container or installed VM) you simply set the appropriate proxy parameters and run your integration tests as normal. After running the tests you can then get the results from ZAP using their API.
Some possible suggested integrations for ZAP:
Using the “cloud” plugins for Jenkins, it is possible automate the installation, configuration, and testing of VMs and containers such that you can apply patches and verify everything is working before you deploy them to production. Every time that a regression is noticed when applying patches you can use this as an opportunity to improve your pipeline. One of the artifacts of your pipeline could even be an always-up-to-date “Gold Image” for use when deploying your new VMs and containers. As mentioned earlier, you can also use “Canary Builds” to ensure that you are always getting the latest patches and security configurations.
Pipelines and automation are not a finite set of concepts. Anything you imagine can probably be automated, though you will have to weigh the cost to automate against the value you derive. Each organization must determine that for themselves, but keep in mind that everything you automate is one less thing you have to do manually and frees up that much more time for you to do other (possible more important) things.
You can add tools like: Coverity OWASP TwistLock OpenSCAP Fortify GNS3 Istio Metasploit Nessus And on and on!!!
You’re only limited by your compute resources and how much time you are willing to invest, with the understanding that each small increment of time you invest frees up more time or improves your overall security posture.
Let’s address the elephant in the room… Some of you, whether consciously or unconsciously, think that this is all a bad idea. At some level, many people will see this as a way for their employer to downsize their: security team, developer teams, network engineering teams, systems administration teams, DBA teams, etc… This may be true for some companies. What is also true is that these are the cutting edge skills that the best employers are looking for today. Even if you avoid it for a while, it will happen regardless when someone new comes along. What you SHOULD be asking instead is “If I automate myself out of my current job, what’s the NEXT thing I would like to do”? If you’re anything like me, you don’t like doing the same old repetitive tasks over and over. I like to try new things, learn new skills, experiment with new technologies and concepts. That’s what gets me out of bed in the morning. IF you have that attitude, you can be VERY successful even after you automate your current workload, because you will be able to in-turn start to address deeper and possibly more satisfying itches. Every new skill you learn, and then learn to automate, increases your value in the job market. The day when a Machine Learning model can do your entire job may come, but I don’t believe it will be for a VERY long time, and in the meantime if you continue to automate your security processes, you can use your gained time to learn about how to build your own Machine Learning algorithms. It will take a VERY long time for AI and ML to even approach the level of understanding and skill which is inherent in your years of learning and experience.
A little security goes a long way in DevOps culture
WHO IS RESPONSIBLE FOR SECURITY?
“According to … NIST, the
average IT security professional
is doing the work of roughly
OR . . . How I stopped worrying and learned to love the
A Little Security Goes A
Long Way In DevOps Culture
Senior Consulting Engineer
Let me tell
you a story:
This is Anne, and she’s the information security manager at BigCorp.Anne is REALLY smart. She knows network security, disaster recovery, application
security, desktop security, understands different threat vectors, and even does
some security research when she has time.
The problem is that Anne doesn’t really HAVE much time. She’s too busy dealing
Anne is so overwhelmed handling her workload that she cannot even think about
emerging threats or scaling the company
This is probably why Anne gets frustrated when developers and engineers ask her if
they can use some new library or tool. She doesn’t have time to evaluate it!
But like I said, Anne is smart…. And she has an idea!
Integrating security into
automation is not going to solve all
of the problems. It IS, however,
going to reduce the frequency and
severity of those problems.
This story is
But the ideas and practices are real. An amalgamation of
real-life work we have been doing with customers for the last
2 years at Open Innovation Labs
Red Hat is the world’s leading provider of enterprise
open source software solutions. Award-winning
support, training, and consulting services make
Red Hat a trusted adviser to the Fortune 500.
And special thanks to my friend Anne Dalton, who was kind enough to be my reviewer for this content!