DevOps teams struggle with complex automation pipelines that are difficult to maintain, duplicated pipeline code across services, and inability to scale SRE due to manual expert tasks required for resiliency testing and SLO analysis. Keptn addresses these problems by providing data-driven delivery and operations automation based on declarative configuration, separating concerns between the application plane defining processes and control plane executing tools, and event-driven automation that removes hard-coded dependencies.

1. Automating SRE from “Hello World”
to Enterprise Scale with
Andreas Grabner
DevOps Activist at Dynatrace
DevRel for Keptn
@grabnerandi, https://www.linkedin.com/in/grabnerandi
Star us @ https://github.com/keptn/keptn
Followus @keptnProject
Slack Us @ https://slack.keptn.sh
Visit us @ https://keptn.sh

2. 5
Confidential
Problem #1
„Our DevOps teams is building monolithic automation that becomes
more complex than the microservices we deploy!“

3. Confidential 6
Too complexautomationpipelines are too hard to maintain!
ChristianHeckelmann
Senior DevOps Engineer
2800
projects
966
CI/CDs
„I am constantlyreacting to
‚Pipeline Broken – please fix!‘“

4. 7
Confidential
Problem #2
„Our DevOps team builds and then copy/pastes tool integrations across
all our automation code due to lack of standards!“

5. Confidential 8
Duplicated Pipeline Code hard to maintain, extend & optimize
pipeline {
stages {
stage ('Depl oy to dev na mespac e') {
ste ps {
c ontain er(‘he lm’) {
}
}
}
stage ('Run tests' ) {
ste ps {
c ontain er(‘jm eter’) {
}
}
}
stage (‘Eval uate p erform ance’) {
ste ps {
c ontain er(‘cu rl’) {
}
}
}
if (e valuat ion.pa ssed) {
sta ge('De ploy t o stag ing') {
s teps {
conta iner(‘ helm’) {
}
}
}
}
}
}
pipeline {
stages {
stage ('Depl oy to dev na mespac e') {
ste ps {
c ontain er(‘ku stomiz e’) {
}
}
}
stage ('Run tests' ) {
ste ps {
c ontain er(‘jm eter’) {
}
}
}
stage (‘Eval uate p erform ance’) {
ste ps {
c ontain er(‘cu rl’) {
}
}
}
if (e valuat ion.pa ssed) {
sta ge('De ploy t o stag ing') {
s teps {
conta iner(‘ helm’) {
}
}
}
}
}
}
1 Service =
1 Pipeline
1 Project = x Pipelines n Teams = n*x Pipelines
pipeline {
stages {
stage ('Depl oy to dev na mespac e') {
ste ps {
c ontain er(‘he lm’) {
}
}
}
stage ('Run tests' ) {
ste ps {
c ontain er(‘jm eter’) {
}
}
}
stage (‘Eval uate p erform ance’) {
ste ps {
c ontain er(‘cu rl’) {
}
}
}
if (e valuat ion.pa ssed) {
sta ge('De ploy t o stag ing') {
s teps {
conta iner(‘ helm’) {
}
}
}
}
}
}
25
services
96
workloads
Dieter Ladenhauf
Senior ACE Engineer
„Onboarding or updating pipelines
is manual & error prone!‘“

6. 9
Confidential
Problem #3
“Our SRE teams can’t scale as resiliency testing and enforcing SLO
analysis is hard to automate!”

7. Confidential 10
Too much manual expert workon test execution & SLO Analysis
~90%of test reruns
9:1 ratio
script maintenancevs creation
only 10% projects
performance tested
Test Result Analysis
MonitoringConfiguration
~ 80%time
spent in manual...
Scripts Creation
SLO Report Generation
15-20
tests /
year
< 5
Apps
„We are limited in scaling SRE
due to manual expert tasks!‘“
Roman Ferstl
Managing Director

8. Confidential 11
3 examples of how Keptn solves these problems ...
Sumit Nagal
Principal Engineer
Roman Ferstl
Managing Director
ChristianHeckelmann
Senior DevOps Engineer
50+
Continuous SLO Validations
15x
Perf tests
10x
tested apps
~60
Services
14
Stages
Tool stack
SLO-based Quality Gate
Automation
Performance & Resiliance
Test Automation
Scaling SRE through
Delivery Automation

9. Confidential 12
And some nice testimonials fromour community ...

10. Confidential 13
Keptn: Data-DrivenDelivery& Operations Automation
You
(Dev/Ops/SRE)
bring your configuration
pick your use case
SLO-Quality
Gates
Progressive
Delivery
Auto-
Remediation
Declaration GitOps SLOs Standards
SLI/SLO shipyard runbook
SRE
Automation
workload
Monitoring Delivery Reliability Remediation
automates configuration and provides self-service for
through event-driven process orchestrationbased on
connect your tools

11. 14
Confidential
Keptn Architecture
Driven by new requirements in the cloud native world

12. Confidential 15
We lookedat the fundamential problems our users have
Mixed information about
• Process (build,deploy,test,evaluate, …)
• Target platform (k8s, …)
• Environments(dev, hardening,…)
• Tools (Terraform,Helm, hey, …)
No clear separation of concerns
• Developers
• Define which artifact to use
• Want fast feedback on their code
• DevOps Engineers
• Define which tools to use
• Ensure tools are properly configured
• Site ReliabilityEngineers
• Define delivery processes
• Define operations workflows

13. Confidential 16
Solution: Remove hard dependencies and integrations
Build
Prepare
Deploy
Test
Notify
Rollback
Config Mgmt.
Deploy
Test
Monitoring
ChatOps
Rollback

14. Confidential 17
Solution: Remove hard dependencies and integrations
Build
Prepare
Deploy
Test
Notify
Rollback
Config Mgmt.
Deploy
Test
Monitoring
ChatOps
Rollback
Eventing
Event:Deploy
Artifact:container1
Stage:Dev
Strategy:Blue/Green
which events to generate ➔ Process Definition who consumesevents ➔ Tool Definition

15. Confidential 18
Eventing
Solution:Keptn is built on an architecture that supports this paradigm
Application Plane (=Process Definition)
Define overall process for delivery and operations
Control Plane
Follow application logic and communicate/configure required services
API
Site Reliability
Engineer
DevOps
Developer
uniform.yaml
config-change*: helm
deploy*: JMeter
deploy-finish: Lighthouse
problem*: Remediation
all: Slack, Dynatrace
Execution Plane (=Tool Definition)
Deploy Service
(Helm, Jenkins …)
Test Service
(JMeter, Neotys, ..)
Validation Service
(Keptn Lighthouse …)
Remediation Service
(Keptn Remediation, SNOW …)
Config Service
(Git, …)
Monitoring Service
(Prometheus,
Dynatrace, …)
Artifact /
Microservice
config.change:artifact:x.y deploy.finished:http://service1 tests.finished:OK evaluation.done:98% Score problem.open:HighFailure
shipyard.yaml
- dev: direct, functional, SLO
- staging: B/G, perf, SLO
- prod: canary, real-user, SLA
remediation.yaml
- high-failure-rate:
- scaleup, rollback
- full-disk:
- cleandir;adjustlog-level

16. Confidential 19
Demo #1: SLO-Driven Progressive Delivery and Operations Automation with Keptn
Developer
Pull Request
(1)Deploy & Test
(2)Evaluate SLOs
(3)Auto-Promote
(4)Deploy & Test
(5)Evaluate SLOs
(7)Deploy Blue / Green
(8)Evaluate SLOs
(1)Action 1: Scale Up
(2)Evaluate SLOs
(3)Action 2: Roll Back
(6)Promote? (9)Toggle Blue/Green
#1 - IncreasedSpeedand Quality of ProgressiveDelivery through Automation #2 - Automate Operations
(10)Re-Evaluate SLOs (4)Evaluate SLOs
Closed-Loop Remediation
Observability

17. Confidential 21
User Example: ProgressiveDelivery with Keptn
Patrick Hofmann
Sr. Consultant
CI
CD

18. 22
Confidential
Data Driven Automation
SLOs to connect Ops with Dev

19. Confidential 23
RootCause: Lengthy manual approval
Build
Deployto
„Test“
Run Test
In „Test“ Manual Approval
Promote to
„Staging“
Functional: Test Result Trend Not Enough Performance: ManualComparison Is Slow Monitoring: Too much unstructed data
~30-60min
Which metrics are important
and which build is therefore better
Which data comesfrom my test
and is relevantfor business transactions
Is this regression impacting
key business use cases

20. Confidential 24
Learning from Google‘s SRE Practices
• Service Level Indicators(SLIs)
• Definition:Measurable Metrics as the base for evaluation
• Example:ErrorRate of Login Requests
• Service Level Objectives (SLOs)
• Definition:Bindingtargets for Service Level Indicators
• Example:Login Error Rate must be less than 2% over a 30 day period
• Service Level Agreements (SLAs)
• Definition:Business Agreement between consumer and provider typicallybasedon SLO
• Example:Logins must be reliable & fast (Error Rate, Response Time, Throughput)99% within a 30 daywindow
• Google Cloud YouTube Video
• SLIs, SLOs, SLAs, oh my! (class SRE implements DevOps): https://www.youtube.com/watch?v=tEylFyxbDLE
SLIs drive SLOs which inform SLAs

21. Confidential 25
Applying SRE Best Practices across the lifecycle
Authentication Service
0.89s
0.5%
May 2020 June 2020
0.61s
2.5%
1000/s 1600/s
Service X
xxs
xx% yys
yy%
xx/s yy/s
Production
Shift-Left
Continuous Delivery
Authentication Service
Commit
#1
Commit
#2
Commit
#3
Commit
#4
Service X
Quality Gates

22. Confidential 26
Explainer on SLI/SLO Validation as part of Continuous Delivery with Dynatrace & Keptn!
Overall Failure Rate
Query: builtin:service.errors.total
Test Step LOGIN Response Time
Query: calc:service.teststeprt:filter(Test, LOGIN)
Test Step LOGIN # Service Calls
Query: calc:service.testsvc:filter(tx, LOGIN)
<= 5%
<= 2%
<=150ms & <=+10%
<= 400ms
<= +0%
Build 1
0%
80ms
100ms
SLO: Overall Score Goal 90% 75%
Response Time 95th Perc
Query: builtin:service.responsetime(p95)
<=100ms
<= 250ms
SLO
SLIs (Service Level Indicators) warn
pass
1
100%
Build 2
4%
120ms
90ms
1
75%
Build 3
1%
90ms
120ms
2
62.5%
Build 4
0%
95ms
95ms
1
100%
Build 1 Build 2 Build 3 Build 4
$ keptn send event start-evaluation myproject myservice starttime=build1_deploy endtime=build1_testsdone
$ keptn send event start-evaluation myproject myservice starttime=build2_deploy endtime=build2_testsdone
$ keptn send event start-evaluation myproject myservice starttime=build3_deploy endtime=build3_testsdone
$ keptn send event start-evaluation myproject myservice starttime=build4_teststart endtime=build4_testsend
DevOps

23. Confidential 27
SLI/SLO-based evaluationimplementationin Keptn
SLIs defined per SLI Provider as YAML
SLI Provider specific queries,e.g: DynatraceMetrics Query
Quality Gates
...
Dynatrace Prometheus Neoload
Scores SLIs
Queries SLI
Providers with
SLI Definitions &
Timeframe
SLOs defined on Keptn Service Level as YAML
List of objectives with fixed or relative pass & warn criteria
indicators:
error_rate: "builtin:service.errors.total.count:merge(0):avg"
count_dbcalls: "calc:service.toptestdbcalls:merge(0):sum"
jvm_memory: "builtin:tech.jvm.memory.pool.committed:merge(0):sum"
objectives:
- sli: error_rate
pass:
- criteria:
- "<=1“ # We expect a max error rate of 1%
- sli: jvm_memory
- sli: count_dbcalls
pass:
- criteria:
- "=+2%" # We allow a 2% increase in DB Calls to previous runs
warning:
- criteria:
- "<=10" # We expect no more than 10 DB Calls per TX
total_score:
pass: "90%"
warning: "75%"
0.5 1.0 0.0 info
7/8
(87.5%)
4/8
(50%)
$ keptn start-evaluation 30m myservice sli.yaml slo.yaml
5 DB Calls 360MB 4.3% 123
SLI Value:
SLI Score:
Total Score
2
3
4
Tool X
1

24. Confidential 28
Demo: Automated SLI/SLO Validationbased on Dynatrace Dashboards
You: Just build a dashboard! : Automates the analysis!
15.5/16
(97%)
8/16
(50%)

25. Confidential 29
User Example: Automating Build Approvals using Keptn‘s SLIs/SLOs in GitLab
Christian Heckelmann
Senior Systems Engineer
87.5%: passed
AutomatedSLI/SLO based Quality Gates
Trigger Evaluation

26. 30
Confidential
Data Driven Operations
Remediation focused on your SLOs

27. Confidential 31
Keptn – Closed-LoopRemediation
version: 0.2.0
kind: Remediation
metadata:
name: remediation-ecommerce
spec:
remediations:
- problemType: Conversion Rate Dropped
actionsOnOpen:
- name: Scaling ReplicaSet by 1
action: scaling
values:
increment: +1
- name: Stop Ad Campaign
action: googleadtoggle
values:
enable: off
campaign: $campaignid
Problem: Conversion Rate Dropped
Root Cause: CPU Pressure
Get
remediation
action(s)
Execute
remediation
action(s)
Re-validate
SLO/BLO
Escalate
Scale Up
Stop
Campaign
1 2
1 2
1
2

28. Confidential 32
Too risky? Start in Pre-Prod leveraging Chaos Engineering to define & test Auto-Remediation
version: 0.2.0
kind: Remediation
metadata:
name: remediation-ecommerce
spec:
remediations:
- problemType: High CPU on ReportGen
actionsOnOpen:
- name: Stop Traffic
action: configureLoadBalancer
values:
action: stopTraffic
ip: $problem.hostIp
- name: Restart Process
action: executeAnsible
values:
script: restartProcess
process: $problem.processID
Problem: Slow ReportGen Service
Root Cause: High CPU on host
Get
remediation
action(s)
Execute
remediation
action(s)
Re-validate
SLO/BLO
Escalate
Stop
Traffic
Restart
Process
1 2
1 2
1
2

29. 33
Confidential
Let‘s wrap it up!

30. Confidential 34
Keptn: Data-DrivenDelivery& Operations Automation
You
(Dev/Ops/SRE)
bring your configuration
pick your use case
SLO-Quality
Gates
Progressive
Delivery
Auto-
Remediation
Declaration GitOps SLOs Standards
shipyard SLI/SLO runbook
SRE
Automation
workload
Monitoring Delivery Reliability Remediation
automates configuration and provides self-service for
through event-driven process orchestrationbased on
connect your tools

31. Confidential 35
Keptn 0.8 was just releasedearly March2021 and opens new use cases
• Multi-Cluster Setup
Separationof Control & Execution Plane
D
E
V
I
N
T
P
R
D

32. Confidential 36
Keptn 0.8 was just releasedearly March2021 and opens new use cases
• Shipyard v0.2.0: Explicit vs opinionated sequences

33. Confidential 37
Get started with our tutorials: tutorials.keptn.sh

34. Questions & Answers
Automating SRE from “Hello World”
to Enterprise Scale with
Andreas Grabner
DevOps Activist at Dynatrace
DevRel for Keptn
@grabnerandi, https://www.linkedin.com/in/grabnerandi
Star us @ https://github.com/keptn/keptn
Followus @keptnProject
Slack Us @ https://slack.keptn.sh
Visit us @ https://keptn.sh