OPTIMIZING THE OPS
IN DEVOPS
GORDON HAFF
Technology Evangelist, Red Hat
Cloud Expo Silicon Valley
3 November 2016
DevOps
FOCUS ON
CLOUD-NATIVE
APPLICATION
ARCHITECTURES
● Single-function units owned
by a team
● Bounded context
● Communicate through
lightweight APIs
Source: PWC
FOCUS ON IMPROVED
AND LESS ISOLATED
DEVELOPER WORKFLOWS
● Collaboration
● CI/CD
● Issue tracking
● Source code control
● Code review
● IDE
● xPaaS
Source: Mike McGarr, Netflix
AN OPPOSING VIEW
"I want to change my job because there is this horrible concept of
"pager duty" or "oncall". Where the developer has to be ready for
any issues that may occur. Are most software jobs like this? Is this
a norm? Where can I find software development positions without
such concepts?"
Anonymous Quora user
WE ALSO TALK
ABOUT CULTURE A LOT
● Empathy
● Trust
● Learning
● Cooperation
● Responsibility
DevOps
BUT WHAT ABOUT THE OPS IN DEVOPS?
DevOps
Biz
Sec
A FABLE
FOR
DEVOPS
NO OPS? (OR IS IT EVOLVED DEVOPS?)
"We have built tooling that removes many of the
operations tasks completely from the developer, and
which makes the remaining tasks quick and self
service. There is no ops organization involved in
running our cloud, no need for the developers to
interact with ops people to get things done, and less
time spent actually doing ops tasks than developers
would spend explaining what needed to be done to
someone else."
Adrian Cockroft, Netflix, 2012
FOCUS ON PROVIDING CORE SERVICES
AND GETTING OUT OF THE WAY
● Deploy a modern container platform
● Enable automated developer workflows
● Mitigate risk and automate security
MODERN PLATFORM
NEW CLOUD PLATFORM NEEDS
What? Why?
Scale-out to meet highly elastic service
requirements
Scale-up is not flexible or scalable enough to
meet changing business needs
Software-defined everything
Software functions running on standardized
hardware increase flexibility
Focused on applications composed of
loosely-coupled services
Large monolithic applications are fragile and
can’t be updated quickly
Enable lightweight iterative software
development and deployment
Modern applications are often short-lived and
require frequent refreshes/replacements
COMPREHENSIVE CLOUD-NATIVE INFRASTRUCTURE
Physical hardware
Container orchestration
Container-optimized Linux
Container/
services
Container/
services
Container/
services
Container/
services
Container/
services
Container/
services
Hybrid cloud management
Developertooling
Software-defined compute, storage, and networking
Public
clouds
OPENSTACK SOFTWARE-DEFINED INFRASTRUCTURE
MAKING CONTAINERS USEFUL:
ECOSYSTEM AND DEFACTO STANDARDS
1 Open Container Initiative (OCI)
2 Cloud Native Computing Foundation (CNCF)
OPERATED AT SCALE
• Different aspects of scale:
• Large scale workloads
• Diverse workloads (batch and services)
• Complex resource management (QoS,
latency sensitivity, etc.)
• Focus on lightweight containerized instances
• Orchestration and resource management
HYBRID MANAGEMENT SERVICES
SERVICE
AUTOMATION
Complete lifecycle and
operational management
that allows IT to remain in
control.
POLICY &
COMPLIANCE
Deploy across virtualization,
private cloud, public cloud and
container-based
environments.
UNIFIED HYBRID
MANAGEMENT
Draws on continuous
monitoring and deep
insights to raise alerts or
remediate issues.
Streamline complex service
delivery processes, saving
time and money.
OPERATIONAL
VISIBILITY
AUTOMATED
DEVELOPER WORKFLOWS
BUILD A PIPELINE
LOTS OF TOOLS FOR THE PIPELINE
gerrit
TRACK AND
VALIDATE
THIRD-PARTY
TOOLS
AND
COMPONENTS
MITIGATE RISK
AUTOMATE SECURITY
TRADITIONAL SECURITY
What we did The problem
Code audited for current compliance
New vulnerabilities constantly
discovered and exploited with no
opportunity for rapid remediation.
Applications and systems deployed on
“secured” platform
There is no perimeter.
Largely relied on checklists, written
processes, and manual actions
Limited throughput and prone to errors.
“Patch Tuesdays” last all month.
Primarily an end-of-process checkpoint Security is such a bottleneck!
DevSecOps
● Build on the mindset that "everyone is responsible for security"
● It’s the practice of building security into development processes
● Security as code
● Flips security from a defensive to an offensive posture that is both automated and
constant
BAKE IN SECURITY AND ASSURANCE
● Components built from source code using a secure, stable, reproducible build
environment
● Careful selection, configuration, and security tracking of packages
● Automated analysis and enforcement of security practices
● Active participation in upstream and community involvement
● Thoroughly validated vulnerability management process
INTEGRATED SECURITY
"Our goal as information security architects must be to
automatically incorporate security controls without manual
configuration throughout this cycle in a way that is as transparent
as possible to DevOps teams and doesn't impede DevOps agility,
but fulfills our legal and regulatory compliance requirements as
well as manages risk. "
DevSecOps: How to Seamlessly Integrate Security Into DevOps
Gartner. DevSecOps: How to Seamlessly Integrate Security Into DevOps. September 2016. G00315283
AUTOMATING SECURITY
CONFIGURATION
ERRORS
MISSINGPATCHES
CODINGMISTAKE
HUMAN ERROR
BAD OPSEC
SECURING CONTENT
EXAMPLE: CONTAINERS
A validated supply
chain helps ensure
use of tested and
patched software.
AN OPEN HYBRID CLOUD JOURNEY
Hybrid policy & management
Data, workflow, & API integration
Automation
Software-defined infrastructure
Legacy modernization
Self-service & flexibility
Optimized virtualization
Cloud migration
Orchestrated container platform
DevOps tooling
Mobile
Open Innovation Labs
Secured software supply chain
CREDITS
Dev: Nelson Pavlosky/flickr under CC http://www.flickr.com/photos/skyfaller/113796919/
Ops: Leonardo Rizzi/flickr under CC http://www.flickr.com/photos/stars6/4381851322/
Rainbows and Unicorns: http://kaigumo.deviantart.com/art/Unicorns-Fart-Rainbows-3-151273843
Piggy bank: https://www.flickr.com/photos/marcmos/3644751092
Stop: https://www.flickr.com/photos/r_grandmorin/6922697037
THANK YOU
plus.google.com/+RedHat
linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHatNews
TRADITIONAL SECURITY
What we did
Code audited for current compliance
Applications and systems deployed on
“secured” platform
Largely relied on checklists, written
processes, and manual actions
Primarily an end-of-process checkpoint
TRUSTED CONTAINER CONTENT
"From a security and governance perspective, trusting the
container image is a critical concern throughout the software
development lifecycle. Ensuring that images are signed and
originate from a trusted registry are solid security best practices. "
5 keys to conquering container security, Amir Jerbi, Infoworld
4 August 2016
http://www.infoworld.com/article/3104030/security/5-keys-to-docker-container-security.html
NoOps?
"This is part of what we call NoOps. The developers used to
spend hours a week in meetings with Ops discussing what they
needed, figuring out capacity forecasts and writing tickets to
request changes for the datacenter. Now they spend seconds
doing it themselves in the cloud."
Adrian Cockroft, Netflix, 2012
BACK TO ADRIAN
" We have built tooling that removes many of the operations tasks
completely from the developer, and which makes the remaining
tasks quick and self service. There is no ops organization involved
in running our cloud, no need for the developers to interact with
ops people to get things done, and less time spent actually doing
ops tasks than developers would spend explaining what needed
to be done to someone else. "
Adrian Cockroft, Netflix, 2012
Strategies for sourcing software
Wild West
Go ahead
and grab it!
Blacklist
Is it from a
known bad
source?
Whitelist
Is it a known good source?
Digitally signed/securely delivered
Rapid updates for vulnerabilities
Repeatable release processes
THE MOVE TO HYBRID INFRASTRUCTURES
BRINGS ADDITIONAL MANAGEMENT CHALLENGES
APPLICATION
ARCHITECTURE
INFRASTRUCTURE
PLATFORM
OPERATIONAL
MODEL
OPERATIONAL
CHALLENGES
Traditional Applications
Virtualization
Operational
Automation
Orchestration
Automation
Private Cloud
Scalable
Applications
Public Cloud
SaaS and PaaS
Cloud Native
Service
Brokering
Containers
Microservices
Self-service
Automated provisioning
Lifecycle management
Root cause analysis
Performance and
capacity management
Hybrid Management
Policy compliance
Quota enforcement
Chargeback
WHAT DEFINES A MODERN PLATFORM?
● Built through collaborative innovation in Linux and other open source communities
● Composed of integrated core software services
● Open container format, runtime, and orchestration
● Focused on large distributed system scale points
THE NEEDED MANAGEMENT SERVICES
SERVICE
AUTOMATION
Complete lifecycle and
operational management
that allows IT to remain in
control.
POLICY &
COMPLIANCE
Deploy across virtualization,
private cloud, public cloud and
container-based
environments.
UNIFIED HYBRID
MANAGEMENT
Draws on continuous
monitoring and deep
insights to raise alerts or
remediate issues.
Streamline complex service
delivery processes, saving
time and money.
OPERATIONAL
VISIBILITY
OPERATIONAL VISIBILITY CHALLENGES
Systems that are not being utilized
should be retired to reclaim resources.
Budgets are tight. We have to
make sure that we are utilizing
our systems efficiently.
Tracking problems across infrastructure
layers can be a challenge.
I’ve got to project infrastructure usage
out into the future for planning purposes.
CHALLENGES
LIFECYCLE MANAGEMENT
ROOT-CAUSE ANALYSIS CAPACITY MANAGEMENT
RESOURCE OPTIMIZATION
OPERATIONAL VISIBILITY WITH HYBRID MANAGEMENT
We now have complete lifecycle
management: provisioning, reconfiguration,
deprovisioning, and retirement.
Automatic resource optimization
intelligently places VMs and offers
right-sizing recommendations.
I can drill-down through infrastructure
layers to determine the root cause.
Resource tracking and trending aids in
capacity and what-if scenario planning.
CHALLENGES
LIFECYCLE MANAGEMENT
ROOT-CAUSE ANALYSIS CAPACITY MANAGEMENT
RESOURCE OPTIMIZATION

Optimizing the Ops in DevOps

  • 1.
    OPTIMIZING THE OPS INDEVOPS GORDON HAFF Technology Evangelist, Red Hat Cloud Expo Silicon Valley 3 November 2016
  • 3.
  • 4.
    FOCUS ON CLOUD-NATIVE APPLICATION ARCHITECTURES ● Single-functionunits owned by a team ● Bounded context ● Communicate through lightweight APIs Source: PWC
  • 5.
    FOCUS ON IMPROVED ANDLESS ISOLATED DEVELOPER WORKFLOWS ● Collaboration ● CI/CD ● Issue tracking ● Source code control ● Code review ● IDE ● xPaaS Source: Mike McGarr, Netflix
  • 6.
    AN OPPOSING VIEW "Iwant to change my job because there is this horrible concept of "pager duty" or "oncall". Where the developer has to be ready for any issues that may occur. Are most software jobs like this? Is this a norm? Where can I find software development positions without such concepts?" Anonymous Quora user
  • 7.
    WE ALSO TALK ABOUTCULTURE A LOT ● Empathy ● Trust ● Learning ● Cooperation ● Responsibility
  • 8.
    DevOps BUT WHAT ABOUTTHE OPS IN DEVOPS?
  • 9.
  • 10.
  • 11.
    NO OPS? (ORIS IT EVOLVED DEVOPS?) "We have built tooling that removes many of the operations tasks completely from the developer, and which makes the remaining tasks quick and self service. There is no ops organization involved in running our cloud, no need for the developers to interact with ops people to get things done, and less time spent actually doing ops tasks than developers would spend explaining what needed to be done to someone else." Adrian Cockroft, Netflix, 2012
  • 12.
    FOCUS ON PROVIDINGCORE SERVICES AND GETTING OUT OF THE WAY ● Deploy a modern container platform ● Enable automated developer workflows ● Mitigate risk and automate security
  • 13.
  • 14.
    NEW CLOUD PLATFORMNEEDS What? Why? Scale-out to meet highly elastic service requirements Scale-up is not flexible or scalable enough to meet changing business needs Software-defined everything Software functions running on standardized hardware increase flexibility Focused on applications composed of loosely-coupled services Large monolithic applications are fragile and can’t be updated quickly Enable lightweight iterative software development and deployment Modern applications are often short-lived and require frequent refreshes/replacements
  • 15.
    COMPREHENSIVE CLOUD-NATIVE INFRASTRUCTURE Physicalhardware Container orchestration Container-optimized Linux Container/ services Container/ services Container/ services Container/ services Container/ services Container/ services Hybrid cloud management Developertooling Software-defined compute, storage, and networking Public clouds
  • 16.
  • 17.
    MAKING CONTAINERS USEFUL: ECOSYSTEMAND DEFACTO STANDARDS 1 Open Container Initiative (OCI) 2 Cloud Native Computing Foundation (CNCF)
  • 18.
    OPERATED AT SCALE •Different aspects of scale: • Large scale workloads • Diverse workloads (batch and services) • Complex resource management (QoS, latency sensitivity, etc.) • Focus on lightweight containerized instances • Orchestration and resource management
  • 19.
    HYBRID MANAGEMENT SERVICES SERVICE AUTOMATION Completelifecycle and operational management that allows IT to remain in control. POLICY & COMPLIANCE Deploy across virtualization, private cloud, public cloud and container-based environments. UNIFIED HYBRID MANAGEMENT Draws on continuous monitoring and deep insights to raise alerts or remediate issues. Streamline complex service delivery processes, saving time and money. OPERATIONAL VISIBILITY
  • 20.
  • 21.
  • 22.
    LOTS OF TOOLSFOR THE PIPELINE gerrit
  • 23.
  • 24.
  • 25.
    TRADITIONAL SECURITY What wedid The problem Code audited for current compliance New vulnerabilities constantly discovered and exploited with no opportunity for rapid remediation. Applications and systems deployed on “secured” platform There is no perimeter. Largely relied on checklists, written processes, and manual actions Limited throughput and prone to errors. “Patch Tuesdays” last all month. Primarily an end-of-process checkpoint Security is such a bottleneck!
  • 26.
    DevSecOps ● Build onthe mindset that "everyone is responsible for security" ● It’s the practice of building security into development processes ● Security as code ● Flips security from a defensive to an offensive posture that is both automated and constant
  • 27.
    BAKE IN SECURITYAND ASSURANCE ● Components built from source code using a secure, stable, reproducible build environment ● Careful selection, configuration, and security tracking of packages ● Automated analysis and enforcement of security practices ● Active participation in upstream and community involvement ● Thoroughly validated vulnerability management process
  • 28.
    INTEGRATED SECURITY "Our goalas information security architects must be to automatically incorporate security controls without manual configuration throughout this cycle in a way that is as transparent as possible to DevOps teams and doesn't impede DevOps agility, but fulfills our legal and regulatory compliance requirements as well as manages risk. " DevSecOps: How to Seamlessly Integrate Security Into DevOps Gartner. DevSecOps: How to Seamlessly Integrate Security Into DevOps. September 2016. G00315283
  • 29.
  • 30.
    SECURING CONTENT EXAMPLE: CONTAINERS Avalidated supply chain helps ensure use of tested and patched software.
  • 31.
    AN OPEN HYBRIDCLOUD JOURNEY Hybrid policy & management Data, workflow, & API integration Automation Software-defined infrastructure Legacy modernization Self-service & flexibility Optimized virtualization Cloud migration Orchestrated container platform DevOps tooling Mobile Open Innovation Labs Secured software supply chain
  • 32.
    CREDITS Dev: Nelson Pavlosky/flickrunder CC http://www.flickr.com/photos/skyfaller/113796919/ Ops: Leonardo Rizzi/flickr under CC http://www.flickr.com/photos/stars6/4381851322/ Rainbows and Unicorns: http://kaigumo.deviantart.com/art/Unicorns-Fart-Rainbows-3-151273843 Piggy bank: https://www.flickr.com/photos/marcmos/3644751092 Stop: https://www.flickr.com/photos/r_grandmorin/6922697037
  • 33.
  • 34.
    TRADITIONAL SECURITY What wedid Code audited for current compliance Applications and systems deployed on “secured” platform Largely relied on checklists, written processes, and manual actions Primarily an end-of-process checkpoint
  • 35.
    TRUSTED CONTAINER CONTENT "Froma security and governance perspective, trusting the container image is a critical concern throughout the software development lifecycle. Ensuring that images are signed and originate from a trusted registry are solid security best practices. " 5 keys to conquering container security, Amir Jerbi, Infoworld 4 August 2016 http://www.infoworld.com/article/3104030/security/5-keys-to-docker-container-security.html
  • 36.
    NoOps? "This is partof what we call NoOps. The developers used to spend hours a week in meetings with Ops discussing what they needed, figuring out capacity forecasts and writing tickets to request changes for the datacenter. Now they spend seconds doing it themselves in the cloud." Adrian Cockroft, Netflix, 2012
  • 37.
    BACK TO ADRIAN "We have built tooling that removes many of the operations tasks completely from the developer, and which makes the remaining tasks quick and self service. There is no ops organization involved in running our cloud, no need for the developers to interact with ops people to get things done, and less time spent actually doing ops tasks than developers would spend explaining what needed to be done to someone else. " Adrian Cockroft, Netflix, 2012
  • 38.
    Strategies for sourcingsoftware Wild West Go ahead and grab it! Blacklist Is it from a known bad source? Whitelist Is it a known good source? Digitally signed/securely delivered Rapid updates for vulnerabilities Repeatable release processes
  • 39.
    THE MOVE TOHYBRID INFRASTRUCTURES BRINGS ADDITIONAL MANAGEMENT CHALLENGES APPLICATION ARCHITECTURE INFRASTRUCTURE PLATFORM OPERATIONAL MODEL OPERATIONAL CHALLENGES Traditional Applications Virtualization Operational Automation Orchestration Automation Private Cloud Scalable Applications Public Cloud SaaS and PaaS Cloud Native Service Brokering Containers Microservices Self-service Automated provisioning Lifecycle management Root cause analysis Performance and capacity management Hybrid Management Policy compliance Quota enforcement Chargeback
  • 40.
    WHAT DEFINES AMODERN PLATFORM? ● Built through collaborative innovation in Linux and other open source communities ● Composed of integrated core software services ● Open container format, runtime, and orchestration ● Focused on large distributed system scale points
  • 41.
    THE NEEDED MANAGEMENTSERVICES SERVICE AUTOMATION Complete lifecycle and operational management that allows IT to remain in control. POLICY & COMPLIANCE Deploy across virtualization, private cloud, public cloud and container-based environments. UNIFIED HYBRID MANAGEMENT Draws on continuous monitoring and deep insights to raise alerts or remediate issues. Streamline complex service delivery processes, saving time and money. OPERATIONAL VISIBILITY
  • 42.
    OPERATIONAL VISIBILITY CHALLENGES Systemsthat are not being utilized should be retired to reclaim resources. Budgets are tight. We have to make sure that we are utilizing our systems efficiently. Tracking problems across infrastructure layers can be a challenge. I’ve got to project infrastructure usage out into the future for planning purposes. CHALLENGES LIFECYCLE MANAGEMENT ROOT-CAUSE ANALYSIS CAPACITY MANAGEMENT RESOURCE OPTIMIZATION
  • 43.
    OPERATIONAL VISIBILITY WITHHYBRID MANAGEMENT We now have complete lifecycle management: provisioning, reconfiguration, deprovisioning, and retirement. Automatic resource optimization intelligently places VMs and offers right-sizing recommendations. I can drill-down through infrastructure layers to determine the root cause. Resource tracking and trending aids in capacity and what-if scenario planning. CHALLENGES LIFECYCLE MANAGEMENT ROOT-CAUSE ANALYSIS CAPACITY MANAGEMENT RESOURCE OPTIMIZATION