Reconstructing the SRE

Reconstructing the SRE
Bob Wise
CTO
Cloud Native Computing Team

Copyright © 2017 Samsung SDS Co., Ltd. All rights reserved
This presentation is intended to provide information concerning Samsung’s efforts around containers and container orchestration. We do
our best to make sure that information presented is accurate and fully up-to-date. However, the presentation may be subject to technical
inaccuracies, information that is not up-to-date or typographical errors. As a consequence, Samsung does not in any way guarantee the
accuracy or completeness of information provided on this presentation. Samsung reserves the right to make improvements, corrections
and/or changes to this presentation at any time.
The information in this presentation or accompanying oral statements may include forward-looking statements. These forward-looking
statements include all matters that are not historical facts, statements regarding the Samsung Data System' intentions, beliefs or current
expectations concerning, among other things, market prospects, growth, strategies, and the industry in which Samsung operates. By their
nature, forward-looking statements involve risks and uncertainties, because they relate to events and depend on circumstances that may or
may not occur in the future. Samsung cautions you that forward looking statements are not guarantees of future performance and that the
actual developments of Samsung, the market, or industry in which Samsung operates may differ materially from those made or suggested
by the forward-looking statements contained in this presentation or in the accompanying oral statements. In addition, even if the information
contained herein or the oral statements are shown to be accurate, those developments may not be indicative developments in future
periods.
Logos remain the property of their respective owners.
2

This presentation is intended to provide information concerning Samsung’s efforts around containers and container orchestration. We do
our best to make sure that information presented is accurate and fully up-to-date. However, the presentation may be subject to technical
inaccuracies, information that is not up-to-date or typographical errors. As a consequence, Samsung does not in any way guarantee the
accuracy or completeness of information provided on this presentation. Samsung reserves the right to make improvements, corrections
and/or changes to this presentation at any time.
The information in this presentation or accompanying oral statements may include forward-looking statements. These forward-looking
statements include all matters that are not historical facts, statements regarding the Samsung Data System' intentions, beliefs or current
expectations concerning, among other things, market prospects, growth, strategies, and the industry in which Samsung operates. By their
nature, forward-looking statements involve risks and uncertainties, because they relate to events and depend on circumstances that may or
may not occur in the future. Samsung cautions you that forward looking statements are not guarantees of future performance and that the
actual developments of Samsung, the market, or industry in which Samsung operates may differ materially from those made or suggested
by the forward-looking statements contained in this presentation or in the accompanying oral statements. In addition, even if the information
contained herein or the oral statements are shown to be accurate, those developments may not be indicative developments in future
periods.
Logos remain the property of their respective owners. So there.
3
Release the Kraken.

SDS - Cloud Native Computing Team
• Our #1 job is improving organizational velocity
– Delivering the business value of Kubernetes to you, fastest
• Rock solid Kubernetes cluster design and deployment, specific to you
• Optimized deployment pipelines and container strategy
• 24x7x365 Kubernetes operations so you can focus on your business
• Organizational consulting to rapidly adapt
• We are:
– Industry leaders in Operations Automation, Cluster Operations, and
Kubernetes Adoption
– Contributors and leaders on the Kubernetes project for 2+ years
– Maintainers of Kraken:production-grade cluster management
– Delivering this for Enterprise customers globally
4

Macro Trends
1. Macro Trend: Massive shifts in all industries towards
sophisticated and comprehensive automation to enable
competitive advantage
2. In all technology delivery “Cloud Native” architectures and
automation prevail
3. Outsourcing is the only path as companies cannot adapt and
acquire expertise internally
Copyright © 2017 Samsung SDS Co., Ltd. All rights reserved5

Root Causes: Deep Organizational Pain
• Tools and processes built for the era of bare metal and virtual
machines
• Dev teams under pressure to deliver new features quickly, not going
fast enough
• Operations teams trying to support hastily deployed features
– Quality issues
– Outages
– Constant firefighting
– Unhappy customers
– Employee retention issues
• Operations viewed as “just trying to block things”
• “Devops” has come to mean generalists who are overwhelmed
trying to handle everything

Cloud Native Has Dramatically Raised the Bar

Classic Enterprise IT is Especially Behind
From the highly recommended:
https://www.slideshare.net/adriancockcroft/dockercon-state-of-the-art-in-microservices

Cloud Native Market Realities
Enormous differences between companies in the ability to execute
on software product delivery have emerged on all three axes:
• Velocity
• Quality
• Efficiency
Copyright © 2017 Samsung SDS Co., Ltd. All rights reserved 2

Cloud Native Market Realities
The companies winning in all three of these categories share on
thing in common:
• Velocity
• Quality
• Efficiency
Theme: Using Cloud Native Approaches

(a) Containerized. Applications deployed in units that can be easily managed
and dealt with by everyone: developers, product managers, and operations.
(b) Dynamically managed. Automatically and responsively deployed by an
orchestration engine that considers customer experience and cost.
(c) Micro-services oriented. Loosely coupled and rapidly adaptable services
that can be innovated and deployed separately while the system as a whole
continues to operate.
12
What is Cloud Native?

Cloud Native Organizations Are High Performance Based On:
How They Behave
How They Measure

High Performance Orgs Embrace Change
14

High Performance Orgs Embrace Rapid
Change With Automation
15

Puppet Labs
2017 State of Devops Report
Highly Recommended Reading
https://puppet.com/resources/whitepaper/2017-state-of-devops-report

Monolithic
Complex Dependencies
Large, Centrally Controlled Teams
Months to Production
Waterfall Process
Manual QA
Manual Security Audits
Lots of Simple Parts
Independent Parts
Small, Independent Teams
Specialists not Generalists
SRE Pattern
Continuous Integration
Continuous Deployment
Automated QA
Ubiquitous Automation
Automated Security
Change Resistant Change Embracing
Embrace The Change

Monolithic
Waterfall Process
Manual QA
Independent Parts
Automated QA
Automated Security
Key Insight
Optimized for MTBF Optimized for MTTR

Monolithic
Waterfall Process
Manual QA
Independent Parts
Automated QA
Automated Security
Key Insight
Measured by MTBF Measured by MTTR
Mean Time Between Failure drives failure prevention and risk aversion
Mean Time To Repair drives responsiveness and allows risks

Independent Parts
Specialists, not Generalists
Adoption of the SRE Pattern
Automated QA
Automated Security
Architecture
(Microservices)
Org
Execution
Fundamentals

Independent Parts
Specialists, not Generalists
Adoption of the SRE Pattern
Automated QA
Automated Security
Architecture
(Microservices)
Org
Execution
Fundamentals
Cloud
Native

Devops Positives
• Breaking down silos
• No more “throwing it over the wall”
• Orientation to infrastructure as code
• Orientation to automation
• Valuing coding skills in operations
Devops fails:
– Company renames ops to devops.
– Improvement fail

Devops Anti-Patterns in Larger Orgs
• Rename ops to devops.
• Disband ops, “we just make the devs do it”
• Add a devops group with yet another silo between dev and ops
• Expect everyone to know everything
– “We only have full stack devops”
– This is unrealistic
Instead, we want a pattern for specialization with collaboration….

The SRE (Site Reliability Engineer)
• Helps product teams engineer for operability
– Architecture
– Tooling
• Owns capacity planning
• Ensures CI tooling is working and adequate
• Helps development go faster with positive assistance
• Spends a lot of time on developing tooling, typically monitoring,
CI, and operational analytics
• Ensure product management sets SLOs (service level objectives)
and error budgets
– Tracks and enforces

Ben Traynor – SRE VP at Google
“Outside Google, we often observe that there isn't parity of
esteem between the SWE and operations teams, which combines
poorly with the fact that they often have different incentives.
That's how we end up with the model that exists in the industry
today, where SWE teams write something and throw it over a wall
to the operations teams, who then try to make it work, and can't,
and throw it back, and so on.”

Ben Traynor – SRE VP at Google
“Outside Google, we often observe that there isn't parity of
esteem between the SWE and operations teams, which combines
poorly with the fact that they often have different incentives.
That's how we end up with the model that exists in the industry
today, where SWE teams write something and throw it over a
wall to the operations teams, who then try to make it work, and
can't, and throw it back, and so on.”

Dev
Ops
QA
CI
CD Monitoring
SRE

Evolving from Devops: Cluster Ops
Infra Ops
Cluster Ops
Dev
Cluster
Infra
App OpsApp App
Product Mgt
Infra OpsInfraInfra

Cluster Ops and the SRE Role
Infra Ops
Cluster Ops
Dev
Cluster
Infra
App OpsApp App
Product Mgt
Infra OpsInfraInfra
SRE

Technology Progression to Platforms…
Deconstruction Needed
Impact to the development
• Operations moving up the stack while Development is moving down the stack
• Still a lot complexity and separation between Operations/Development teams
• Outsource when economies of scale make it more beneficial to push to a 3rd
party
Bare Metal
OS
OS / Virtual
Machines
Application
Hypervisor
OS / Virtual
Machines
Application
Dependencies Dependencies
Bare Metal
OS
Dependencies
Application
Infrastructure
Infrastructure
IAAS
Application Application
PAAS
VM’s IAAS / PAASCo-Lo
Bare Metal
OS
Dependencies
Application
Infrastructure
DIY
OS / Virtual
Machines
OS / Virtual
Machines
3rd
Party
Operations
Development
Area of Responsibility

Organizational Trend to Outsourcing…
Deconstruction Needed
Improvements to Development Process.
• Faster release cycles means quicker to market.
• Smaller, more reliable deployments.
• Quicker recovery, reduce risk when issues occur. Shift in focus to
MTTR vs MTTF.
• Economies of scale are getting to the build vs buy decision
Infrastructure
Operations
Cluster
Operations
App Operations
Infrastructure
Operations
Cluster
Operations
App Operations
Infrastructure
Operations
Cluster
Operations
App Operations
IAAS
Application Application
PAAS
IAAS / PAAS
OS / Virtual
Machines
OS / Virtual
Machines
Future State
Kubernetes
Containers Pipeline
Alerting, Monitoring
Performance/Efficiency
Maintenance
Reporting
Speed to Market
Improvements to Organizational Velocity.
• Subject matter experts on Day 1
• No need to spin up a new operations team
• Allow development teams to focus on Product vs Tooling
• More cost efficient than doing it in-house

Multiple Factors
• Technology progression to Platforms
• Organizational Trend to Outsourcing
– Some SRE functions will be outside your org
• Best practices:
– Specialists over Generalists
– Collaboration over Silos

Reconstructing the SRE
SRE
ARE
Application
CRE
Cluster
IRE
Infra
ARE Concerns
• Application SLI/SLO
• Availability targets
• Pod autoscaling
• Capacity planning
• Deployments/Canaries
• Application CI/CD
• App packaging practices
• App SLI capture
• Application architecture
• Application monitoring
• Application logging
SLO – Service Level Objective
SLI – Service Level Indicator
CRE/IRE Concerns
• Cluster SLI/SLO
• Cluster Capacity
• Node autoscaling
• Control plane design
• Cluster upgrades
• Cluster utilization
• Under/over capacity
nodes
• Node specs
• Docker version
• Network
configuration
• Namespace design
• Failure domains
• Cluster monitoring
• Cluster logging

Kubernetes Factors

Kubernetes Conceptual
36
Node (Server)
Docker
Daemon
Kubernetes Node Controller
Node (Server)
Docker
Daemon
Node (Server)
Docker
Daemon
Kubernetes
Control
Plane
Status: Cluster is ready for work

Kubernetes
Control
Plane
37
To API: Run (1) of container X
Node (Server)
Docker
Daemon
Node (Server)
Docker
Daemon
Node (Server)
Docker
Daemon
1 of X
ReplicaSet (replication set)
Manages pod count

Node (Server)
Docker
Daemon
Node (Server)
Docker
Daemon
Node (Server)
Docker
Daemon
38
Status: Cluster is running X
Pod (X)
ContainerContainerContainer
Kubernetes
Control
Plane
1 of X

Orchestrating
Multiple
Applications
39
Kubernetes
Control
Plane
Node (Server)
Docker
Daemon
Node (Server)
Docker
Daemon
Node (Server)
Docker
Daemon
Pod
Pod
Pod
Pod
Pod
Pod
Pod
Pod
Pod
App 1
App 2
App 3
App 4
App 5

Kubernetes Namespace (Environments)
Service
Pod
Pod

main
Pod
Pod

main
Pod
Pod
main
Pod
Pod

main
Pod
Pod
main
Pod
Pod
CLASH

main
Pod
Pod
prod

main
Pod
Pod
prod
main
Pod
Pod
dev

Kubernetes Namespace
• Virtual cluster
• Fundamental admin unit, maps to different organizational
patterns:
– Resource name scoping, i.e.
• dev, qa, or prod
– Teams or projects
• Group1, group2
– Services
• What Quotas are attached to, and future permission-related
concepts
– Reflection of Google operational philosophy
– Micro/ACLs get too unwieldy

Kubernetes Namespaces: Key to the CRE/ARE Contract
• Namespaces are formal production entities, managed by CRE
• Dev pipelines that create/modify namespaces get CRE oversight
• RBAC configuration of namespaces is a critical role
• CRE owns namespace quotas
• ARE has broad permissions inside the namespace
• CRE owns the pod scheduling rules/contraints
(taints/tolerations)
• Reporting broken out by namespace

Managed ClusterOps Services
• Our goal is to enable Customers to focus entirely on their core business.
• Fastest path to production clusters
• Continuous monitoring and alerting
• Performance and efficiency analysis
• Cluster Readiness Evaluation
• Strategic Business Analysis
• POC Delivery
• Development Pipeline Planning
• Dedicated On-boarding Team
• Managed Cluster Operations
• Other services to accelerate benefits to
your business.
Professional Acceleration Services:
Our goal is to accelerate benefits to your business
from containers, cloud-native, and faster
deployments:
Hardware
Operations
Cluster
Operations
App Operations
Quickest and
and most cost
effective path
SDS – CNCT Commercial Offering

Contact and Project Info
• Bob Wise
– @countspongebob on twitter and github
– bob.wise@samsung.com
• CNCT home page:
– http://samsung-cnct.github.io/
• Kraken repo
– Production grade cluster management for Kubernetes
– open source, Apache license
– https://github.com/samsung-cnct/kraken
– #kraken channel on kubernetes slack

Reconstructing the SRE

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Reconstructing the SRE

Similar to Reconstructing the SRE (20)

Recently uploaded

Recently uploaded (20)

Reconstructing the SRE