GitOps, with tools like Argo and Flux, are preferred platform tools managing configuration in cloud native environments. But it is hard to troubleshoot a failed deployment of a complex application as there is no built-in deployment lifecycle observability, standardized hooks nor the concept of an application vs individual workloads. The CNCF project Keptn addresses those challenges by extending the Kubernetes Pod scheduler to provide OpenTelemetry Traces and Prometheus metrics for end-2-end deployment observability. Keptn introduces automated application-aware pre- and post-deployment lifecycle hooks to enforce dependency checks, send notifications or evaluates SLOs that otherwise need a custom K8s operator. Join this talk and learn how the Keptn Lifecycle Toolkit (KLT) Operator extends observability into GitOps deployments and how it enables declarative deployment lifecycle orchestration!

1. OpenTelemetry for GitOps:
Tracing Deployments from Git
Git Commit to Production on
K8s
Andreas Grabner
CNCF Ambassador, DevRel @ CNCF Keptn
Global DevRelLead @ Dynatrace
Keptn: https://lifecycle.keptn.sh

2. We are at a Dev Conference: Anybody knows my first programming language?

3. We are doing
LIVE Demos
using only
Open-Source
tooling 
If you stay you get to see YAML, a Terminal, graphs, traces and much more 
https://github.com/keptn-sandbox/klt-on-k3s-with-argocd

4. Let me get a quick pulse on those topics…
Hands-Up if you are using or intend to use …
DORA Metrics
1 2 3 5
4 6

5. 0: What is OpenTelemetry
1: Unmeasurable Change Impact
• “Its hard to troubleshoot failed GitOps deployments on K8s!”
• DEMO: DORA & Deployment Traces natively from K8s Pod Scheduler
2: Pipelines Gone Wild
• “Its hard adding tests, security, and SLO checks to deployments!”
• DEMO: Deployment Checks Natively in K8s Pod Scheduler
3: Observablity Tool Sprawl
• “Its hard connecting observability data to all GitOps tools!”
• DEMO: Unified Observability Data Access within K8s
Agenda: What is OpenTelemetry! And which observability challenges we can solve!

6. • A set of standardized, vendor-agnostic SDKs, APIs, and tools
• Ingests, transforms, and sends data to an Observability back-end
• Makes systems observable through Traces, Metrics, and Logs
• More signals in the works: real user monitoring, profiling …
What is OpenTelemetry and its architecture!
API
SDK
EXPORTER
OTLP
EXPORTER
COLLECTOR
Application
OTLP
Observability
Back-end
Collector
(optional)

7. OpenTelemetry is here to stay …
“By 2025, 70% of new
cloud-native applications
will adopt OpenTelemetry
for observability”,
Gartner Predicts 2022

8. OpenTelemetry’s first use case was Distributed Traces: Traces consist of Spans

9. Distributed Traces can then be analyzed in the backend observability tool / platform
Overview of all traces
Individual Trace
Single Span

10. To learn more about OpenTelemetry: https://opentelemetry.io
https://isitobservable.io/

11. 0: What is OpenTelemetry
1: Unmeasurable Change Impact
• “Its hard to troubleshoot failed GitOps deployments on K8s!”
• DEMO: DORA & Deployment Traces natively from K8s Pod Scheduler
2: Pipelines Gone Wild
• “Its hard adding tests, security, and SLO checks to deployments!”
• DEMO: Deployment Checks Natively in K8s Pod Scheduler
3: Observablity Tool Sprawl
• “Its hard connecting observability data to all GitOps tools!”
• DEMO: Unified Observability Data Access within K8s
Back to the agenda: Observability problems we can solve!

12. Various tools make changes to apps & infra
Platform Teams must build custom tooling to enable
app-aware Observability and traceability
Your teams use any app or infra delivery tool of choice
Keptn emits app-aware deployment lifecycle traces
(OTel) and app-aware DORA metrics (Prometheus)
Challenge #1: Unmeasurable Change Impact Solution #1: App-Aware Change Traceability
Declare Apps as CRDs
Keptn observes deployments app-aware
Business App1
Business App 2
Shared Services Shared Infra
Lifecycle Operator App2
App1 Svc Infra
But K8s lacks the concept of an app
Only knows who deployed change
“Its hard to troubleshoot failed deployments on K8s!” “Keptn: Telemetry for your end-2-end delivery process”

13. My-Application:2.0 **
Frontend-Svc:2.0
Backend-Svc:1.5
Storage-Svc:1.0
Post
Pre
Post
Pre
Post
Pre
Timespan & Result for each single deployment
Pre-App-Deployment
Post-App-Deployment
Timespan Time & Result for whole app deployment
* K8s Operators can leverage K8s webhooks and extend K8s scheduler for pre- and post-deployment hooks
** K8s doesn’t yet have a standard application concept but Delivery SIG is working on it
Observe: Metrics (DORA) & Traces
Some K8s technical details …

14. Demo #1: Deployment Tracing and DORA Metrics

15. Test Security Validate
Your app teams use various delivery tools
DevOps need to code all checks in every tool!
Unavoidable shortcuts increase quality & security risk!
Your app teams use delivery tool of choice
DevOps define pre- and post-deployment tasks once
Checks are always enforced K8s natively!
Challenge #2: Pipelines gone wild! Solution #2: Delivery checks native in K8s
Lifecycle Operator
Deploy straight to K8s
Dev Production Dev Production
Lifecycle Operator
Keptn prevents K8s
from deploying bad changes
No shortcuts possible
“Its hard adding tests, security, and SLO checks to deployments!” “Keptn: Event-driven delivery tasks natively in k8s”

16. My-Application:2.0 **
Frontend-Svc:2.0
Backend-Svc:1.5
Storage-Svc:1.0
Post
Pre
Post
Pre
Post
Pre
Timespan & Result for each single deployment
Pre-App-Deployment
Post-App-Deployment
Timespan Time & Result for whole app deployment
Tasks: Dependency, Env Health,
Certificates, Approval, ...
Evaluations: SLOs, Error Budgets ...
Timespan & result of
each task / evaluation
Tasks: Tests, Security Scans, Cleanups,
Promote ...
Evaluations: SLOs, User Experience,
Adoption ...
Timespan & result of
each task / evaluation
* K8s Operators can leverage K8s webhooks and extend K8s scheduler for pre- and post-deployment hooks
** K8s doesn’t yet have a standard application concept but Delivery SIG is working on it
Observe: Metrics (DORA) & Traces
Orchestrate: Pre-Deploy Orchestrate: Post-Deploy
Some K8s technical details continued …

17. Demo #2: Deployment Checks: Dependencies & SLOs
KLT 0.8 now also supports
custom containers

18. Your app teams use various observability tools
SREs need multiple tool-specific metric configs for
Auto-scaling, release validation, SLOs, alerting …
Metrics Server
Your app teams use observability tool of choice
SREs have a single standard-based configuration for
Auto-scaling, release validation, SLOs, alerting …
Keptn queries metrics from any provider define as CRDs …
… exposes them via K8s Metric API & Prometheus
Challenge #3: Observability Tool Sprawl! Solution #2: One Metric Standard for All!
“Its hard connecting observability data to all GitOps tools!” “Keptn: Unified way to access your observability data”

19. Demo #2: A Single Metric Standard for all K8s tools

20. “How do you connect your observability
data with all your GitOps tools?”
“How do you add tests, security, and SLO
checks to all your k8s deployments?”
“How do you measure success or
troubleshoot failed deployments?”
“Keptn: Unified way to access your
observability data”
“Keptn: Telemetry for your end-2-end
delivery process”
“Keptn: Event-driven delivery tasks
natively in k8s”
To wrap it up: 3 Solutions for 3 Observability challenges!

21. To learn more or try those things out

22. Thank you!
Questions?
Andreas Grabner
CNCF Ambassador, DevRel @ CNCF Keptn
Global DevRelLead @ Dynatrace
Learn more about Keptn: https://lifecycle.keptn.sh