Embracing service-level-objectives of your microservices in your Cl/CD
This presentation discusses defining service level objectives (SLOs) for microservices and using them to automate quality gates in continuous integration/continuous delivery (CI/CD) pipelines. It presents the open-source Keptn framework, which can provide quality gates for microservices running on Kubernetes clusters based on defined SLOs. The presentation covers how SLOs and service level indicators (SLIs) can help prevent bad code from reaching production and configure alerts, and demonstrates Keptn's automated SLO evaluation and a use case for performance as a self-service tool.
Embracing service-level-objectives of your microservices in your Cl/CD
1. Embracing service-level-objectives
of your microservices in your Cl/CD
Rob Jahn
Technical Partner Manager
Dynatrace
https://www.linkedin.com/in/robjahn/
https://events.nebulaworks.com/lightningtalk-april
2. Unfortunately, a robust CI/CD pipeline with well-defined tests does not guarantee a failure-safe
application in production. That is since many microservices are interwoven and dependent on each
other. Some of them have to handle hundreds of thousands of requests, while others are merely for
batch processing of data. Therefore, it is crucial to define the objectives and quality metrics of
individual services in terms of service level objectives as code.
• How service level indicators and service level objectives can help you set up automated quality gates in
a Cl/CD system to prevent bad code changes to ever reach production
• How quality criteria can be used to configure your alerts in monitoring solutions to be alerted on any
issues in your environments
• Present the open-source framework Keptn to provide quality gates for your microservice applications
running on your Kubernetes cluster.
More information about Keptn can be found on www.keptn.sh or on https://github.com/keptn
Embracing service-level-objectives of your microservices in your Cl/CD
3. Agenda
• Motivation / Inspiration
• Use Cases we provide in Keptn
• How it works
• How you can leverage & contribute
Embracing service-level-objectives of your microservices in your Cl/CD
4. API Testing Integration Testing Application Testing
Test a Component Test a System Test Real World
Continuous Testing Embedded in CI/CD Pipelines
Shift Left Performance
Continuous performance testing strategy
5. Challenge: Lengthy manual analysis and approval
Build Deploy to
„Test“
Run Test
In „Test“
Manual Approval
Promote to
„Staging“
Deploy to
„Test“
Functional: Test Result Trend Not Enough Performance: Manual Comparison Is Slow Monitoring: Too much unstructed data
Which build is better Which data comes from my testIs this a real problem
~30-60min
6. Solution - Automate the analysis
Error rate
Throughput + Response Time versus
objectives
Resources check
Test Failure
7. • An SLI is a service level indicator—a carefully defined
quantitative measure of some aspect of the level of
service that is provided.
• An SLO is a service level objective: a target value or
range of values for a service level that is measured by
an SLI. A natural structure for SLOs is thus SLI ≤
target, or lower bound ≤ SLI ≤ upper bound.
Inspiration #1 - Google SRE Handbook – Free online!
https://landing.google.com/sre/books/
8. Concept: Automated performance analysis by scoring
Spec files define:
• Service Level Indicators (SLI)
• Service Level Objectives (SLO)
Pass / Fail / Warning
=
Read the Spec
Retrieve
data
Calculate
Score
Datasource
1
2
3
4
9. Inspiration #2 – Mark Tomlinson
1 Create JIRA Ticket for Performance Feedback
Next Continuous Test Cycle picks up Request
Performance Feedback back on JIRA Ticket
2
3
1 2
3
10. Inspiration #3 – Thomas Steinmaurer
“Performance Signature”
for Build Nov 16
“Performance Signature”
for Build Nov 17
“Performance Signature”
for every Build
“Multiple Metrics”
compared to prev
Timeframe
Simple Regression Detection
per Metric
11. Define application delivery and
operations processes
declaratively
Use predefined CloudEvents to
separate the process from the
tools
Easy way to integrate and
switch between different tools
Blue/Green Deployments
Automated Quality Gates
Standardized communication protocol Keptn’s uniform
www.keptn.sh
an event-based control plane for continuous delivery and
automated operations for cloud-native applications
12. Use Case #1 – Automate SLO Evaluation
1 $ keptn start-evaluation timeframe service sli slo
Keptn pulls data from different data sources and Evaluates SLIs
Keptn delivers values and Score based on SLOs
2
3
1 2
3
7/8
(87.5%)
4/8
(50%)
13. Use Case #2 – Performance as a Self-Service
1 $ keptn deployment-complete my-service
Keptn
Keptn Delivers Score based on SLO
2
3
1 2
and Evaluates SLIsRuns Tests
3
7/8
(87.5%)
4/8
(50%)
21. As an engineer I can request performance
feedback on demand without having any
dependency to other teams or environments
Allow everyone to get performance
feedback from their latest builds
and production in an easy way
Performance Center of Excellence
Teams enable Self-Service to
Engineers (more scale)
Elevate every engineer to
become a Performance Expert
(right tools and guard rails)
Scale through automation, best
practices and standards
What is performance as a self-service?
22. Notification
Service
Run test
Performance as a self-service with Jira & Keptn
Performance Feedback
back on JIRA Ticket for
team review
1
2
7
From a JIRA story, invoke Keptn
via automation call (REST)
Dynatrace
service
Jmeter
Service
Jira
Service
SLO Validation
Service
Evaluate
targets
3
Push test context
to Dynatrace
4
Send result
notifications
5
Add Jira issue
with test results
6