SlideShare a Scribd company logo
Journey to Observability:
tracking every function execution in production
WHY
HOW
WHAT
Journey to Observability
Lucas Copi
Ingress Lead, IBM Cloud
Kubernetes Service
Background Architecture
Test in prod ≠ test only in prod
Understand the delta between pre-
production environments and production
environments
Can you adequately measure and
understand those differences in your
service? Do you know where to look?
Test in Prod
Were we able to
understand what our
code was doing in
prod vs pre-prod....
Issue in Prod
Not enough
data
Add metrics
or logs
Wait for the
issue in prod ....Not at all
• 4 Golden Signals for every operation
• Consistent metadata across all pillars of observability
• Automatic instrumentation
• Consistent and performant logging
Requirements
Need to be able to observe the
interactions between services
Need to be able to observe the
behavior of a single service
Orchestration Models
AND
Existing solutions are great at automatically
adding observability to the interactions
between services
Observing service internals requires unique
context aware instrumentation. Existing
instrumentation libraries add a
development burden without providing
value on top of observability.
Existing Solutions
Provides simple instrumentation of go services for metrics, logs, and tracing, by creating a
mutable context that is passed from parent to child functions within an execution flow.
Maintains metadata throughout a function call or an entire execution flow, providing
consistency throughout an execution flow and across the pillars of observability.
Chosen solution
Simplifies the instrumentation of logs, metrics and tracing:
• Maintains log row context throughout an execution flow and optimizes logging
performance by reducing fsyncs + related syscalls and expensive allocations/copies of
logger metadata
• Automatically records execution time for each instrumented function call in a histogram
with exemplars based on Trace ID when present.
• Automatically create a trace for each execution flow and a spans for each instrumented
function call, and includes helpers for utilizing and propagating incoming distributed
traces.
Functions are instrumented by adding a MethodStart context method at
the start of the function and deferring a call to the contexts MethodEnd
method with an optional pointer to an error.
Instrumentation
Software development isn't
linear
Software development isn't
only about features
Observability changed
our mindset
How can I observe this code?
How can I know it is working,
and how can I measure its
impact?
So... How can I understand my
code's performance?
Job contention and missed
callbacks isolated to improve
provisioning
Aggregate across user and
status errors to provide better
experiences in our services
How can I understand
my code's impact?
Impact of
observability on
customer outcomes
Impact of observability
on cycle time
2021
Kubernetes ALB
2021
Certificate Manager Support
2022
Secrets Manager Integration
2022
Secondary Cert Provider
2022
IPI / UPI for DNS Services
2022
New detailed Ingress Status
2023
Ingress domain (customizations,
multi-provider, CIS)
2023
Ingress UI
2023
Terraform Improvements
2023
Proxy-Protocol ROKS
2023
ALB Autoscaling
Questions?

More Related Content

Similar to OSMC 2023 | Journey to observability: tracking every function execution in production by Lucas Copi

The differing ways to monitor and instrument
The differing ways to monitor and instrumentThe differing ways to monitor and instrument
The differing ways to monitor and instrumentJonah Kowall
 
Continuous delivery @wcap 5-09-2013
Continuous delivery   @wcap 5-09-2013Continuous delivery   @wcap 5-09-2013
Continuous delivery @wcap 5-09-2013David Funaro
 
Modern apps in a microservices age May meet up Architecting for Innovation
Modern apps in a microservices age May meet up Architecting for InnovationModern apps in a microservices age May meet up Architecting for Innovation
Modern apps in a microservices age May meet up Architecting for InnovationAndrew Blades
 
Cognizant's Zero Deviation Life Cycle - an Overview
Cognizant's Zero Deviation Life Cycle - an OverviewCognizant's Zero Deviation Life Cycle - an Overview
Cognizant's Zero Deviation Life Cycle - an OverviewDr. Bippin Makoond
 
Why Distributed Tracing is Essential for Performance and Reliability
Why Distributed Tracing is Essential for Performance and ReliabilityWhy Distributed Tracing is Essential for Performance and Reliability
Why Distributed Tracing is Essential for Performance and ReliabilityAggregage
 
How to Migrate Applications Off a Mainframe
How to Migrate Applications Off a MainframeHow to Migrate Applications Off a Mainframe
How to Migrate Applications Off a MainframeVMware Tanzu
 
Monitoring Containerized Micro-Services In Azure
Monitoring Containerized Micro-Services In AzureMonitoring Containerized Micro-Services In Azure
Monitoring Containerized Micro-Services In AzureAlex Bulankou
 
2018 Pivotal DevOps Day_Pivotal 소개 및 세션 아젠다 소개
2018 Pivotal DevOps Day_Pivotal 소개 및 세션 아젠다 소개2018 Pivotal DevOps Day_Pivotal 소개 및 세션 아젠다 소개
2018 Pivotal DevOps Day_Pivotal 소개 및 세션 아젠다 소개VMware Tanzu Korea
 
2018 Pivotal DevOps Day_마이크로서비스 전환 방법론과 사례
2018 Pivotal DevOps Day_마이크로서비스 전환 방법론과 사례2018 Pivotal DevOps Day_마이크로서비스 전환 방법론과 사례
2018 Pivotal DevOps Day_마이크로서비스 전환 방법론과 사례VMware Tanzu Korea
 
Dev ops presentation
Dev ops presentationDev ops presentation
Dev ops presentationAhmed Kamel
 
SplunkLive! London 2016 Splunk for Devops
SplunkLive! London 2016 Splunk for DevopsSplunkLive! London 2016 Splunk for Devops
SplunkLive! London 2016 Splunk for DevopsSplunk
 
Solving the Hidden Costs of Kubernetes with Observability
Solving the Hidden Costs of Kubernetes with ObservabilitySolving the Hidden Costs of Kubernetes with Observability
Solving the Hidden Costs of Kubernetes with ObservabilityDevOps.com
 
Service quality monitoring system architecture
Service quality monitoring system architectureService quality monitoring system architecture
Service quality monitoring system architectureMatsuo Sawahashi
 
Diksha sda presentation
Diksha sda presentationDiksha sda presentation
Diksha sda presentationdikshagupta111
 
The Magic Of Application Lifecycle Management In Vs Public
The Magic Of Application Lifecycle Management In Vs PublicThe Magic Of Application Lifecycle Management In Vs Public
The Magic Of Application Lifecycle Management In Vs PublicDavid Solivan
 
From Monoliths to Microservices at Realestate.com.au
From Monoliths to Microservices at Realestate.com.auFrom Monoliths to Microservices at Realestate.com.au
From Monoliths to Microservices at Realestate.com.auevanbottcher
 
Feature drift monitoring as a service for machine learning models at scale
Feature drift monitoring as a service for machine learning models at scaleFeature drift monitoring as a service for machine learning models at scale
Feature drift monitoring as a service for machine learning models at scaleNoriaki Tatsumi
 

Similar to OSMC 2023 | Journey to observability: tracking every function execution in production by Lucas Copi (20)

The differing ways to monitor and instrument
The differing ways to monitor and instrumentThe differing ways to monitor and instrument
The differing ways to monitor and instrument
 
Continuous delivery @wcap 5-09-2013
Continuous delivery   @wcap 5-09-2013Continuous delivery   @wcap 5-09-2013
Continuous delivery @wcap 5-09-2013
 
Modern apps in a microservices age May meet up Architecting for Innovation
Modern apps in a microservices age May meet up Architecting for InnovationModern apps in a microservices age May meet up Architecting for Innovation
Modern apps in a microservices age May meet up Architecting for Innovation
 
Arunprakash Alagesan
Arunprakash AlagesanArunprakash Alagesan
Arunprakash Alagesan
 
Cognizant's Zero Deviation Life Cycle - an Overview
Cognizant's Zero Deviation Life Cycle - an OverviewCognizant's Zero Deviation Life Cycle - an Overview
Cognizant's Zero Deviation Life Cycle - an Overview
 
Why Distributed Tracing is Essential for Performance and Reliability
Why Distributed Tracing is Essential for Performance and ReliabilityWhy Distributed Tracing is Essential for Performance and Reliability
Why Distributed Tracing is Essential for Performance and Reliability
 
How to Migrate Applications Off a Mainframe
How to Migrate Applications Off a MainframeHow to Migrate Applications Off a Mainframe
How to Migrate Applications Off a Mainframe
 
Monitoring Containerized Micro-Services In Azure
Monitoring Containerized Micro-Services In AzureMonitoring Containerized Micro-Services In Azure
Monitoring Containerized Micro-Services In Azure
 
2018 Pivotal DevOps Day_Pivotal 소개 및 세션 아젠다 소개
2018 Pivotal DevOps Day_Pivotal 소개 및 세션 아젠다 소개2018 Pivotal DevOps Day_Pivotal 소개 및 세션 아젠다 소개
2018 Pivotal DevOps Day_Pivotal 소개 및 세션 아젠다 소개
 
2018 Pivotal DevOps Day_마이크로서비스 전환 방법론과 사례
2018 Pivotal DevOps Day_마이크로서비스 전환 방법론과 사례2018 Pivotal DevOps Day_마이크로서비스 전환 방법론과 사례
2018 Pivotal DevOps Day_마이크로서비스 전환 방법론과 사례
 
Open Digital Framework from TMFORUM
Open Digital Framework from TMFORUMOpen Digital Framework from TMFORUM
Open Digital Framework from TMFORUM
 
Dev ops presentation
Dev ops presentationDev ops presentation
Dev ops presentation
 
The Devops Handbook
The Devops HandbookThe Devops Handbook
The Devops Handbook
 
SplunkLive! London 2016 Splunk for Devops
SplunkLive! London 2016 Splunk for DevopsSplunkLive! London 2016 Splunk for Devops
SplunkLive! London 2016 Splunk for Devops
 
Solving the Hidden Costs of Kubernetes with Observability
Solving the Hidden Costs of Kubernetes with ObservabilitySolving the Hidden Costs of Kubernetes with Observability
Solving the Hidden Costs of Kubernetes with Observability
 
Service quality monitoring system architecture
Service quality monitoring system architectureService quality monitoring system architecture
Service quality monitoring system architecture
 
Diksha sda presentation
Diksha sda presentationDiksha sda presentation
Diksha sda presentation
 
The Magic Of Application Lifecycle Management In Vs Public
The Magic Of Application Lifecycle Management In Vs PublicThe Magic Of Application Lifecycle Management In Vs Public
The Magic Of Application Lifecycle Management In Vs Public
 
From Monoliths to Microservices at Realestate.com.au
From Monoliths to Microservices at Realestate.com.auFrom Monoliths to Microservices at Realestate.com.au
From Monoliths to Microservices at Realestate.com.au
 
Feature drift monitoring as a service for machine learning models at scale
Feature drift monitoring as a service for machine learning models at scaleFeature drift monitoring as a service for machine learning models at scale
Feature drift monitoring as a service for machine learning models at scale
 

Recently uploaded

OC Streetcar Final Presentation-Downtown Santa Ana
OC Streetcar Final Presentation-Downtown Santa AnaOC Streetcar Final Presentation-Downtown Santa Ana
OC Streetcar Final Presentation-Downtown Santa AnaRahsaan L. Browne
 
Oracle Database Administration I (1Z0-082) Exam Dumps 2024.pdf
Oracle Database Administration I (1Z0-082) Exam Dumps 2024.pdfOracle Database Administration I (1Z0-082) Exam Dumps 2024.pdf
Oracle Database Administration I (1Z0-082) Exam Dumps 2024.pdfSkillCertProExams
 
Writing Sample 2 -Bridging the Divide: Enhancing Public Engagement in Urban D...
Writing Sample 2 -Bridging the Divide: Enhancing Public Engagement in Urban D...Writing Sample 2 -Bridging the Divide: Enhancing Public Engagement in Urban D...
Writing Sample 2 -Bridging the Divide: Enhancing Public Engagement in Urban D...Rahsaan L. Browne
 
05232024 Joint Meeting - Community Networking
05232024 Joint Meeting - Community Networking05232024 Joint Meeting - Community Networking
05232024 Joint Meeting - Community NetworkingMichael Orias
 
527598851-ppc-due-to-various-govt-policies.pdf
527598851-ppc-due-to-various-govt-policies.pdf527598851-ppc-due-to-various-govt-policies.pdf
527598851-ppc-due-to-various-govt-policies.pdfrajpreetkaur75080
 
The Canoga Gardens Development Project. PDF
The Canoga Gardens Development Project. PDFThe Canoga Gardens Development Project. PDF
The Canoga Gardens Development Project. PDFRahsaan L. Browne
 
Cymulate (Breach and Attack Simulation).
Cymulate (Breach and Attack Simulation).Cymulate (Breach and Attack Simulation).
Cymulate (Breach and Attack Simulation).luckyk1575
 
Pollinator Ambassador Earth Steward Day Presentation 2024-05-22
Pollinator Ambassador Earth Steward Day Presentation 2024-05-22Pollinator Ambassador Earth Steward Day Presentation 2024-05-22
Pollinator Ambassador Earth Steward Day Presentation 2024-05-22LHelferty
 
Deciding The Topic of our Magazine.pptx.
Deciding The Topic of our Magazine.pptx.Deciding The Topic of our Magazine.pptx.
Deciding The Topic of our Magazine.pptx.bazilnaeem7
 
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docx
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docxThe Influence and Evolution of Mogul Press in Contemporary Public Relations.docx
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docxMogul Press
 
Hi-Tech Industry 2024-25 Prospective.pptx
Hi-Tech Industry 2024-25 Prospective.pptxHi-Tech Industry 2024-25 Prospective.pptx
Hi-Tech Industry 2024-25 Prospective.pptxShivamM16
 
Breathing in New Life_ Part 3 05 22 2024.pptx
Breathing in New Life_ Part 3 05 22 2024.pptxBreathing in New Life_ Part 3 05 22 2024.pptx
Breathing in New Life_ Part 3 05 22 2024.pptxFamilyWorshipCenterD
 
123445566544333222333444dxcvbcvcvharsh.pptx
123445566544333222333444dxcvbcvcvharsh.pptx123445566544333222333444dxcvbcvcvharsh.pptx
123445566544333222333444dxcvbcvcvharsh.pptxgargh1099
 
Understanding Poverty: A Community Questionnaire
Understanding Poverty: A Community QuestionnaireUnderstanding Poverty: A Community Questionnaire
Understanding Poverty: A Community Questionnairebazilnaeem7
 
ServiceNow CIS-Discovery Exam Dumps 2024
ServiceNow CIS-Discovery Exam Dumps 2024ServiceNow CIS-Discovery Exam Dumps 2024
ServiceNow CIS-Discovery Exam Dumps 2024SkillCertProExams
 

Recently uploaded (15)

OC Streetcar Final Presentation-Downtown Santa Ana
OC Streetcar Final Presentation-Downtown Santa AnaOC Streetcar Final Presentation-Downtown Santa Ana
OC Streetcar Final Presentation-Downtown Santa Ana
 
Oracle Database Administration I (1Z0-082) Exam Dumps 2024.pdf
Oracle Database Administration I (1Z0-082) Exam Dumps 2024.pdfOracle Database Administration I (1Z0-082) Exam Dumps 2024.pdf
Oracle Database Administration I (1Z0-082) Exam Dumps 2024.pdf
 
Writing Sample 2 -Bridging the Divide: Enhancing Public Engagement in Urban D...
Writing Sample 2 -Bridging the Divide: Enhancing Public Engagement in Urban D...Writing Sample 2 -Bridging the Divide: Enhancing Public Engagement in Urban D...
Writing Sample 2 -Bridging the Divide: Enhancing Public Engagement in Urban D...
 
05232024 Joint Meeting - Community Networking
05232024 Joint Meeting - Community Networking05232024 Joint Meeting - Community Networking
05232024 Joint Meeting - Community Networking
 
527598851-ppc-due-to-various-govt-policies.pdf
527598851-ppc-due-to-various-govt-policies.pdf527598851-ppc-due-to-various-govt-policies.pdf
527598851-ppc-due-to-various-govt-policies.pdf
 
The Canoga Gardens Development Project. PDF
The Canoga Gardens Development Project. PDFThe Canoga Gardens Development Project. PDF
The Canoga Gardens Development Project. PDF
 
Cymulate (Breach and Attack Simulation).
Cymulate (Breach and Attack Simulation).Cymulate (Breach and Attack Simulation).
Cymulate (Breach and Attack Simulation).
 
Pollinator Ambassador Earth Steward Day Presentation 2024-05-22
Pollinator Ambassador Earth Steward Day Presentation 2024-05-22Pollinator Ambassador Earth Steward Day Presentation 2024-05-22
Pollinator Ambassador Earth Steward Day Presentation 2024-05-22
 
Deciding The Topic of our Magazine.pptx.
Deciding The Topic of our Magazine.pptx.Deciding The Topic of our Magazine.pptx.
Deciding The Topic of our Magazine.pptx.
 
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docx
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docxThe Influence and Evolution of Mogul Press in Contemporary Public Relations.docx
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docx
 
Hi-Tech Industry 2024-25 Prospective.pptx
Hi-Tech Industry 2024-25 Prospective.pptxHi-Tech Industry 2024-25 Prospective.pptx
Hi-Tech Industry 2024-25 Prospective.pptx
 
Breathing in New Life_ Part 3 05 22 2024.pptx
Breathing in New Life_ Part 3 05 22 2024.pptxBreathing in New Life_ Part 3 05 22 2024.pptx
Breathing in New Life_ Part 3 05 22 2024.pptx
 
123445566544333222333444dxcvbcvcvharsh.pptx
123445566544333222333444dxcvbcvcvharsh.pptx123445566544333222333444dxcvbcvcvharsh.pptx
123445566544333222333444dxcvbcvcvharsh.pptx
 
Understanding Poverty: A Community Questionnaire
Understanding Poverty: A Community QuestionnaireUnderstanding Poverty: A Community Questionnaire
Understanding Poverty: A Community Questionnaire
 
ServiceNow CIS-Discovery Exam Dumps 2024
ServiceNow CIS-Discovery Exam Dumps 2024ServiceNow CIS-Discovery Exam Dumps 2024
ServiceNow CIS-Discovery Exam Dumps 2024
 

OSMC 2023 | Journey to observability: tracking every function execution in production by Lucas Copi

  • 1. Journey to Observability: tracking every function execution in production
  • 2. WHY HOW WHAT Journey to Observability Lucas Copi Ingress Lead, IBM Cloud Kubernetes Service
  • 4. Test in prod ≠ test only in prod Understand the delta between pre- production environments and production environments Can you adequately measure and understand those differences in your service? Do you know where to look? Test in Prod
  • 5. Were we able to understand what our code was doing in prod vs pre-prod.... Issue in Prod Not enough data Add metrics or logs Wait for the issue in prod ....Not at all
  • 6. • 4 Golden Signals for every operation • Consistent metadata across all pillars of observability • Automatic instrumentation • Consistent and performant logging Requirements
  • 7. Need to be able to observe the interactions between services Need to be able to observe the behavior of a single service Orchestration Models AND
  • 8. Existing solutions are great at automatically adding observability to the interactions between services Observing service internals requires unique context aware instrumentation. Existing instrumentation libraries add a development burden without providing value on top of observability. Existing Solutions
  • 9. Provides simple instrumentation of go services for metrics, logs, and tracing, by creating a mutable context that is passed from parent to child functions within an execution flow. Maintains metadata throughout a function call or an entire execution flow, providing consistency throughout an execution flow and across the pillars of observability. Chosen solution Simplifies the instrumentation of logs, metrics and tracing: • Maintains log row context throughout an execution flow and optimizes logging performance by reducing fsyncs + related syscalls and expensive allocations/copies of logger metadata • Automatically records execution time for each instrumented function call in a histogram with exemplars based on Trace ID when present. • Automatically create a trace for each execution flow and a spans for each instrumented function call, and includes helpers for utilizing and propagating incoming distributed traces.
  • 10. Functions are instrumented by adding a MethodStart context method at the start of the function and deferring a call to the contexts MethodEnd method with an optional pointer to an error. Instrumentation
  • 11. Software development isn't linear Software development isn't only about features Observability changed our mindset How can I observe this code? How can I know it is working, and how can I measure its impact?
  • 12. So... How can I understand my code's performance?
  • 13. Job contention and missed callbacks isolated to improve provisioning Aggregate across user and status errors to provide better experiences in our services How can I understand my code's impact?
  • 15. Impact of observability on cycle time 2021 Kubernetes ALB 2021 Certificate Manager Support 2022 Secrets Manager Integration 2022 Secondary Cert Provider 2022 IPI / UPI for DNS Services 2022 New detailed Ingress Status 2023 Ingress domain (customizations, multi-provider, CIS) 2023 Ingress UI 2023 Terraform Improvements 2023 Proxy-Protocol ROKS 2023 ALB Autoscaling