How to get the best out of DevSecOps - an operations perspective

© 2017 VERACODE INC. 1© 2017 VERACODE INC.
How to Get the Best Out
Of DevSecOps
The Operations Perspective

Introduction

© 2017 VERACODE INC. 3
About This Webinar
https://www.brighttalk.com/webcast/12807/252395
Colin Domoney
Senior Product Innovation Manager
@colindomoney

Further Reading
Kim, Gene, Kevin Behr,
and George Spafford.
2013. The Phoenix Project:
A Novel About IT, DevOps,
and Helping Your Business
Win.
Kim, Gene, Patrick
Debois, and John Willis.
2016. The Devops
Handbook: How to
Create World-Class
Agility, Reliability, and
Security in Technology
Organizations
Beyer, Betsy, Jennifer
Petoff, Chris Jones, and
Niall Richard Murphy.
Site Reliability
Engineering: How
Google Runs
Production Systems. 1
edition. O′Reilly, 2016.
Humble, Jez, and David
Farley. 2010. Continuous
Delivery: Reliable
Software Releases
Through Build, Test, and
Deployment Automation.
‘2016 State of DevOps Report’. 2017.
Puppet. Accessed January 23.
https://puppet.com/resources/white-
paper/2016-state-of-devops-report.

What is Dev(Sec)Ops

A Cultural Clash

What is Dev(Sec) Ops?
“DevOps is the practice of operations and development engineers
participating together in the entire service lifecycle,
from design through the development process to production support.”
“DevOps is also characterized by operations staff making
use many of the same techniques as developers for their
systems work.”
Source : ‘What Is DevOps?’ 2010. The Agile Admin. August 2.
https://theagileadmin.com/what-is-devops/.

The First Way : Systems Thinking
Source : Kim, Gene. 2012. ‘The Three Ways: The Principles Underpinning
DevOps’. IT Revolution. August 22. http://itrevolution.com/the-three-ways-
principles-underpinning-devops/.
• Never pass a known defect to downstream work centre
• Never allow local optimization to create global degradation
• Always seek to increase flow
• Always seek to achieve profound understanding of the system (per Deming)
The First Way emphasizes the performance of the entire system, as opposed to the
performance of a specific silo of work or department

The Second Way : Amplify Feedback Loops
The Second Way is about creating the right to left feedback loops.
• Understand and respond to all customers, internal and external
• Shorten and amplify all feedback loops
• Embed knowledge where you need it

The Third Way : Continual Experimentation
and Learning
• Allocate time for the improvement of daily work
• Create rituals that reward the team for taking risks
• Introduce faults into the system to increase resilience
The Third Way is about creating a culture that fosters two things: continual
experimentation, taking risks and learning from failure; and understanding that
repetition and practice is the prerequisite to mastery.

The Benefits of DevOps
• High-performing organizations are decisively outperforming their
lower-performing peers in terms of throughput.
• High performers have better employee loyalty,
as measured by employee Net Promoter Score (eNPS).
• Improving quality is everyone’s job.
• High performers spend 50 percent less time remediating security
issues than low performers.
• Taking an experimental approach to product development can
improve your IT and organizational performance.
• Undertaking a technology transformation initiative can produce
sizeable cost savings for any organization.
Source : ‘2016 State of DevOps Report’. 2017. Puppet. Accessed January
23. https://puppet.com/resources/white-paper/2016-state-of-devops-report.

DevOps Maturity Model
Initial
• Poor, ad hoc communication
• No automation
• Unpredictable, uncontrolled reactive process
Managed
• Managed communication, some shared decision making
• Siloed automation, no central infrastructure
• Processes are managed but not standardised
Defined
• Collaboration, shared decision making and accountability
• Central automated processes across the application lifecycle
• Processes are standardised across the organisation
Measured
• Collaboration-based processes are measured to identify inefficiencies and bottlenecks
• Collect and analyse metrics of the automated processes and measure against the business goals
• Visibility and predictability of entire process quality and performance
Optimised
• Effective knowledge sharing and individual empowerment
• Self-service automation, self-learning using analytics and self-remediation
• Process risk and cost optimisation

The Move to
DevOps
Market trends and enabling technologies

Before We Had DevOps

Cloud Technology and CI/CD Platforms
Cloud Technology CI/CD Platforms

Configuration Management Tools

What Makes a Good DevSecOps Solution?
•Provide security feedback as early as possible, in DevOps tools
Fail quickly, through automation
•Ease of use, actionable findings, speed, low FPs
Limit time-to-market impact
•Provide privacy early in SDLC, measure and assess teams, compliance and risk later
Support team autonomy with enterprise orchestration
•Microservices, Infrastructure as Code, leading edge languages and frameworks, Containerization
Adapt to latest practices & technologies
•Developer communities, small, consumable courses, open documentation, integrations in marketplaces
Support culture of learning & openness
•Provide feedback from Ops (Prod & QA) about risks/attacks in a way that is consumable by development
Provide operational visibility

The Impact to
Operations

Three Cornerstones
• Collaboration
– Very little crossover between teams, specific roles and responsibilities
– Collaborate flexibly through flexible tools (i.e.. Slack)
• Flexibility
– Previously focused on stability over everything else
– Modern organisations need to be flexible and responsive
• Automation
– Manual changes cannot keep pace with rapid turnaround times required
– Frees up resources for critical thinking tasks and problem solving

“Infrastructure as Code”
https://puppet.com/blog/what-is-infrastructure-as-code

Site Reliability Engineering - It’s all Software
Now
• Defined by Ben Treynor: "what happens when a software
engineer is tasked with what used to be called operations.
• The ideal SRE candidate is a coder who also has operational
and systems knowledge and likes to whittle down complex
tasks.
• Typically spend their time as follows:
– up to 50% of their time doing "ops" related work
– up to 50% of their time on development tasks such as new
features, scaling or automation
https://en.wikipedia.org/wiki/Site_reliability_engineering

Best Practices
for Securing
Operations

Control Your Source Code Repositories
• Continuous Deployment means any code checked in can
potentially reach production within minutes
• Best practices include:
– Splitting repositories
– Using Perforce for fine grained control
– Performing peer reviews on ‘pull requests’ to critical code

Protect Your Deployment Pipeline
• Continuous Deployment means that your pipeline is a critical piece of
infrastructure
• Best practices include:
– Hardening CI/CD systems to prevent compromise
– Review changes to prevent execution of unwanted code
– Test for suspicious API calls in unit tests or scripts
– Ensure CI/CD runs in isolated containers
– Ensure VCS credentials are ‘read only’

Using Security Testing Tools
Behavioural Driven Development
Security Testing
IDE Integrations for
Security Testing

Integrate Security Into Your Deployment
Pipeline – VSTS/TFS

Integrate Security Into Your Deployment
Pipeline - Jenkins

Security Telemetry in Applications
• Record all security relevant events such as:
– Successful and unsuccessful logins
– User password resets
– User e-mail address resets
– User credit card changes
• Monitor changes in ratios of success to failures
• Alert on events such as:
– Anomalous behaviour
– Sudden changes in values

Security Telemetry in the Environment
• Monitor environmental items and events such as:
– OS changes
– Security group changes
– Changes to configurations
– Cloud infrastructure changes
– Web server errors

Use the Right Tool for Job
https://www.slideshare.net/YuryChemerkin/zane-lackey-security-at-scale-web-application-security-in-a-
continuous-deployment-environment

Case Study : Security Telemetry at Etsy
• Nick Galbreath (Director of Engineering at Etsy, 2010):
– No dedicated fraud control or Infosec team
– Embedded telemetry with entire DevOps value stream
– Everyone was responsible for monitoring and alerting
• Example events:
– Abnormal program termination (segfaults)
– Database syntax error
– Indications of SQL attack

Automated Dashboards – “Measure All The
Things”

“Security Is Not A Binary Event”
https://www.slideshare.net/nickgsuperstar/devopssec-apply-devops-
principles-to-security

Logging for Security
• Logs are the ‘bread and butter’ of an IT Operations team
• Challenges when used in a security context:
– Delayed response to identity issues (delays in alerting)
– Limited data (no POST body, no header data)
– Limited context (disparate events in several locations)
Phillip Maddux, https://medium.com/@foospidy

Destructive Testing / Chaos Engineering
Chaos Engineering is the discipline of experimenting on a distributed system in order to
build confidence in the system’s capability to withstand turbulent conditions in production.
Four steps as follows:
• Build a hypothesis around steady-state behaviour.
• Vary real-world events.
• Run experiments in production.
• Automate experiments to run continuously.
“The best way to avoid failure is to fail constantly.”
- Jeff Atwood

Case Study : Netflix

Operating System Hardening
• Disable unused and/or guest accounts
• Run at level of least privilege
• Disable unused services
• Ensure automatic updates are enabled
• Ensure strong passwords are used
• Disable overly verbose logging
• Ensure backups are performed

Runtime Application Self Protection
• Very low false positives and false negatives.
• Requires no modification to application source code.
• Can report attack information into SIEM.
• Can be deployed onto legacy applications and platforms.
• Can execute in ‘monitor’ or ‘alert’ mode to identify attacks
without protecting the application (IAST)
Benefits of RASP
• An ‘agent’ that executes in parallel with an application and
provides run-time protection by monitoring traffic through the
application.
• Data propagation through the application to determine
whether input data is ‘tainted’ in its lifetime.
• If data is tainted then it is possible that the application is
under attack and the agent can then protect the application at
run-time.
What is RASP?

Change Management Process
• ITIL change management process defines three types of change:
– Standard (low-risk, follow standard process, can be automated)
– Normal (require approval by CAB, manual process)
– Emergency (high priority CAB)
• Too many changes are classified as ‘normal’
• DevOps best practice suggests:
– Try and make as much as possible ‘standard’ and auto-approve
– Optimise the CAB process for requests that remain as ‘normal’

Make a Commitment
• Learn how to code!
• Learn the ‘tools of the trade’ (Git, Ansible, etc.)
• Learn the basics with a test application i.e. WebGoat.Net
• Learn how a Version Control System works
• Automate a repetitive task
• Experience a ‘Day in the Life’ of a Developer
Security is Everyone’s Responsibility

How to get the best out of DevSecOps - an operations perspective

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to How to get the best out of DevSecOps - an operations perspective

Similar to How to get the best out of DevSecOps - an operations perspective (20)

Recently uploaded

Recently uploaded (20)

How to get the best out of DevSecOps - an operations perspective