SlideShare a Scribd company logo
1 of 68
Download to read offline
HOSTED BY
99.9% of your traces are trash 🚮
Paige Cruz
Senior Developer Advocate at Chronosphere
Distributed tracing is still finding its footing in many organizations
today, one challenge to overcome is the data volume - keeping
100% of your traces is expensive and unnecessary. Enter sampling -
head vs tail how do you decide? Let’s look at the design of Sifter and
get familiar with why tail-based sampling is the way to enact a
cost-effective tracing solution while actually increasing the system’s
observability.
Tracing Today
A Very Brief History of Tracing
2010
Google
publishes
Dapper paper
Dapper
A Very Brief History of Tracing
2012
Company
formerly known
as Twitter
releases Zipkin
Zipkin
A Very Brief History of Tracing
2015
Uber releases
Jaeger
Jaeger
CNCF accepts
OT
OpenTracing
A Very Brief History of Tracing
2017
Google promotes
OpenCensus
OpenCensus
A Very Brief History of Tracing
2019
Baggage!
W3C
OC + OT = OTel
OpenTelemetry
A Very Brief History of Tracing
2021
Tracing for all!
OTel GA
A Very Brief History of Tracing
2010
Dapper
2015
Jaeger
2019
W3C
2017
OpenCensus
2021
OTel GA
2012
Zipkin
OpenTracing OpenTelemetry
that’s me!
How to scale
Collectors?
What sampling
strategy makes
sense?
How do I get
other teams to
adopt tracing?
Broken
Economics of
Tracing
I can’t even
Factors that make tracing $$$$
■ High throughput = more traces generated
■ # of services instrumented
● more services = more spans
■ More spans = bigger traces
■ # of labels
● More labels = bigger spans
■ Egress costs sending to vendor
■ Cross-AZ costs & storage costs (self-hosting)
In practice, sampling
rates can be as low as
0.01%
Heads or Tails?
Sampling Strategies
No Sampling
■ Best for very low throughput services/endpoints
■ Specific high impact time-boxed use cases
● A/B testing of a new endpoint
● Customer troubleshooting
● etc…
Head
🚮
Rate Limit
TRACES
Probabilistic
Targeted
Adaptive
Head Based Sampling
■ Efficient for systems with high throughput
■ Minimal impact on service performance
■ Lower financial cost
■ Easy to get started with
■ Tends to be configured statically requiring
re-deployments to take effect
Tail
🚮
Tail Based Sampling
■ Works best for
● systems with low throughput
● systems with high throughput combined with
head based sampling
■ Very configurable + easy to reason about
■ Higher financial and operational cost
Which traces should we keep?
Which traces should we discard?
What constitutes an “interesting” trace?
Edge case code
paths
Infrequent
request types
Anomalous events
Tomorrow’s Tracing
P99 Conf Template
#1A1047 #00E5FF
#753bf0 #FF2CDF
#2B53F9
Font usage Share Tech or Roboto
Color Palette
Table
Column 1 Column 2 Column 3 Column 5
Data 1 Data 2 Data 3 Data 4
Data 5 Data 6 Data 7 Data 8
#667EEA
P99 Conf Template
#1A1047
r26 g16 b71
c100 m100 y34 b45
Pantone 275c
#00E5FF
r0 g229 b255
c60 m0 y9 b0
#753bf0
r117 g59 b240
c79 m77 y0 b0
#FF2CDF
r255 g44 b223
C31 m78 y0 b0
#2B53F9
r38 g24 b250
c91 m75 y0 b0
Pantone 2727c
Color Palette Details
#667EEA
r102 g126 b234
c60 m35 y0 b5
Pantone 659c
P99 Conf Template
<here is some code>
<styling>
<use consolas for font when displaying code>
<don’t go below 12pt font size>
Slide title with white background
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer auctor eros eu
faucibus sodales. Nunc dictum, urna id blandit pretium, mauris velit pulvinar ligula,
interdum blandit sem tortor eget dolor.
■ Bullet 1
● Bullet 2
■ Bullet 3
Slide title with white background
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer auctor eros eu
faucibus sodales. Nunc dictum, urna id blandit pretium, mauris velit pulvinar ligula,
interdum blandit sem tortor eget dolor.
■ Bullet 1
● Bullet 2
■ Bullet 3
Slide title with white background
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer auctor eros eu
faucibus sodales. Nunc dictum, urna id blandit pretium, mauris velit pulvinar ligula,
interdum blandit sem tortor eget dolor.
■ Bullet 1
● Bullet 2
■ Bullet 3
Centered Large Text
Clear slide for diagram with caption
Paige Cruz
paigerduty@chronosphere.io
@paigerduty@hachyderm.io
www.paigerduty.com
Thank you! Let’s connect.

More Related Content

Similar to 99.9% of Your Traces are Trash

Similar to 99.9% of Your Traces are Trash (20)

Scaling AI in production using PyTorch
Scaling AI in production using PyTorchScaling AI in production using PyTorch
Scaling AI in production using PyTorch
 
Building scalable cloud-native applications (Sam Vanhoutte at Codit Azure Paa...
Building scalable cloud-native applications (Sam Vanhoutte at Codit Azure Paa...Building scalable cloud-native applications (Sam Vanhoutte at Codit Azure Paa...
Building scalable cloud-native applications (Sam Vanhoutte at Codit Azure Paa...
 
WALD: A Modern & Sustainable Analytics Stack
WALD: A Modern & Sustainable Analytics StackWALD: A Modern & Sustainable Analytics Stack
WALD: A Modern & Sustainable Analytics Stack
 
Mike Tuche, CEO of Talend: Enabling the Data Driven Enterprise
Mike Tuche, CEO of Talend: Enabling the Data Driven EnterpriseMike Tuche, CEO of Talend: Enabling the Data Driven Enterprise
Mike Tuche, CEO of Talend: Enabling the Data Driven Enterprise
 
#Interactive Session by Seema Kohli, "Test Leadership in the Era of Artificia...
#Interactive Session by Seema Kohli, "Test Leadership in the Era of Artificia...#Interactive Session by Seema Kohli, "Test Leadership in the Era of Artificia...
#Interactive Session by Seema Kohli, "Test Leadership in the Era of Artificia...
 
[Paris merge world tour] Perforce Keynote
[Paris   merge world tour] Perforce Keynote[Paris   merge world tour] Perforce Keynote
[Paris merge world tour] Perforce Keynote
 
Pivotal - Advanced Analytics for Telecommunications
Pivotal - Advanced Analytics for Telecommunications Pivotal - Advanced Analytics for Telecommunications
Pivotal - Advanced Analytics for Telecommunications
 
FIWARE Wednesday Webinars - Machine Learning with Cosmos and Spark
FIWARE Wednesday Webinars - Machine Learning with Cosmos and SparkFIWARE Wednesday Webinars - Machine Learning with Cosmos and Spark
FIWARE Wednesday Webinars - Machine Learning with Cosmos and Spark
 
Data analytics at a petabyte scale final
Data analytics at a petabyte scale   finalData analytics at a petabyte scale   final
Data analytics at a petabyte scale final
 
E2E Life Cycle Intelligence & Channel Strategy @ TPC
E2E Life Cycle Intelligence & Channel Strategy @ TPCE2E Life Cycle Intelligence & Channel Strategy @ TPC
E2E Life Cycle Intelligence & Channel Strategy @ TPC
 
JMRA Ray Poynter 2017
JMRA Ray Poynter 2017JMRA Ray Poynter 2017
JMRA Ray Poynter 2017
 
Data Culture Series - Keynote & Panel - Reading - 12th May 2015
Data Culture Series  - Keynote & Panel - Reading - 12th May 2015Data Culture Series  - Keynote & Panel - Reading - 12th May 2015
Data Culture Series - Keynote & Panel - Reading - 12th May 2015
 
External Aerodynamic Optimization Using ANSYS Mesh Morphing
External Aerodynamic Optimization Using ANSYS Mesh MorphingExternal Aerodynamic Optimization Using ANSYS Mesh Morphing
External Aerodynamic Optimization Using ANSYS Mesh Morphing
 
Lean kanban India 16
Lean kanban India 16Lean kanban India 16
Lean kanban India 16
 
Future Architecture of Streaming Analytics: Capitalizing on the Analytics of ...
Future Architecture of Streaming Analytics: Capitalizing on the Analytics of ...Future Architecture of Streaming Analytics: Capitalizing on the Analytics of ...
Future Architecture of Streaming Analytics: Capitalizing on the Analytics of ...
 
Charles sonigo - Demuxed 2018 - How to be data-driven when you aren't Netflix...
Charles sonigo - Demuxed 2018 - How to be data-driven when you aren't Netflix...Charles sonigo - Demuxed 2018 - How to be data-driven when you aren't Netflix...
Charles sonigo - Demuxed 2018 - How to be data-driven when you aren't Netflix...
 
Building a cutting-edge data processing environment on a budget
Building a cutting-edge data processing environment on a budgetBuilding a cutting-edge data processing environment on a budget
Building a cutting-edge data processing environment on a budget
 
Outside-in Planning Processes. A Case for Change at ASCM Conference
Outside-in Planning Processes. A Case for Change at ASCM Conference Outside-in Planning Processes. A Case for Change at ASCM Conference
Outside-in Planning Processes. A Case for Change at ASCM Conference
 
2020 Testing Trends: Top Predictions for QA Teams to Watch, Join, and Lead
2020 Testing Trends: Top Predictions for QA Teams to Watch, Join, and Lead2020 Testing Trends: Top Predictions for QA Teams to Watch, Join, and Lead
2020 Testing Trends: Top Predictions for QA Teams to Watch, Join, and Lead
 
OpenSIPS Summit, Open Source Telecom Software Survey 2022
OpenSIPS Summit, Open Source Telecom Software Survey 2022OpenSIPS Summit, Open Source Telecom Software Survey 2022
OpenSIPS Summit, Open Source Telecom Software Survey 2022
 

More from Paige Cruz

More from Paige Cruz (14)

Power Up with Podman - Kubernetes Community Day LA
Power Up with Podman - Kubernetes Community Day LAPower Up with Podman - Kubernetes Community Day LA
Power Up with Podman - Kubernetes Community Day LA
 
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdf
 
OTel Orientation: How to Train Teams (OTel in Practice)
OTel Orientation: How to Train Teams (OTel in Practice)OTel Orientation: How to Train Teams (OTel in Practice)
OTel Orientation: How to Train Teams (OTel in Practice)
 
Avoiding Alert Bankruptcy and Burnout
 Avoiding Alert Bankruptcy and Burnout Avoiding Alert Bankruptcy and Burnout
Avoiding Alert Bankruptcy and Burnout
 
Tracing Adventures from PR - Production
Tracing Adventures from PR - ProductionTracing Adventures from PR - Production
Tracing Adventures from PR - Production
 
Threat Modeling in the Cloud
Threat Modeling in the CloudThreat Modeling in the Cloud
Threat Modeling in the Cloud
 
There's No Place Like Production
There's No Place Like ProductionThere's No Place Like Production
There's No Place Like Production
 
Taming Feral DevOps
Taming Feral DevOps Taming Feral DevOps
Taming Feral DevOps
 
SRECon23 Cognitive Apprenticeship in Action_ Alert Triage Hour of Power
SRECon23 Cognitive Apprenticeship in Action_ Alert Triage Hour of PowerSRECon23 Cognitive Apprenticeship in Action_ Alert Triage Hour of Power
SRECon23 Cognitive Apprenticeship in Action_ Alert Triage Hour of Power
 
Pushing Observability Uphill - The Single “Pain” of Glass
Pushing Observability Uphill - The Single “Pain” of GlassPushing Observability Uphill - The Single “Pain” of Glass
Pushing Observability Uphill - The Single “Pain” of Glass
 
Power Up with Podman
Power Up with PodmanPower Up with Podman
Power Up with Podman
 
Intro to Instrumentation
Intro to InstrumentationIntro to Instrumentation
Intro to Instrumentation
 
From Cardinal(ity) Sins to Cost-Efficient Metrics Aggregation
From Cardinal(ity) Sins to Cost-Efficient Metrics AggregationFrom Cardinal(ity) Sins to Cost-Efficient Metrics Aggregation
From Cardinal(ity) Sins to Cost-Efficient Metrics Aggregation
 
3rd Wave Observability: Open or Bust
3rd Wave Observability: Open or Bust 3rd Wave Observability: Open or Bust
3rd Wave Observability: Open or Bust
 

Recently uploaded

IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
Enterprise Knowledge
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
giselly40
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
Joaquim Jorge
 

Recently uploaded (20)

Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
 
Tech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdfTech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdf
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 

99.9% of Your Traces are Trash

  • 1. HOSTED BY 99.9% of your traces are trash 🚮 Paige Cruz Senior Developer Advocate at Chronosphere
  • 2. Distributed tracing is still finding its footing in many organizations today, one challenge to overcome is the data volume - keeping 100% of your traces is expensive and unnecessary. Enter sampling - head vs tail how do you decide? Let’s look at the design of Sifter and get familiar with why tail-based sampling is the way to enact a cost-effective tracing solution while actually increasing the system’s observability.
  • 4. A Very Brief History of Tracing 2010 Google publishes Dapper paper Dapper
  • 5. A Very Brief History of Tracing 2012 Company formerly known as Twitter releases Zipkin Zipkin
  • 6. A Very Brief History of Tracing 2015 Uber releases Jaeger Jaeger CNCF accepts OT OpenTracing
  • 7. A Very Brief History of Tracing 2017 Google promotes OpenCensus OpenCensus
  • 8. A Very Brief History of Tracing 2019 Baggage! W3C OC + OT = OTel OpenTelemetry
  • 9. A Very Brief History of Tracing 2021 Tracing for all! OTel GA
  • 10. A Very Brief History of Tracing 2010 Dapper 2015 Jaeger 2019 W3C 2017 OpenCensus 2021 OTel GA 2012 Zipkin OpenTracing OpenTelemetry
  • 12.
  • 13.
  • 14.
  • 15. How to scale Collectors? What sampling strategy makes sense? How do I get other teams to adopt tracing?
  • 16.
  • 18.
  • 20. Factors that make tracing $$$$ ■ High throughput = more traces generated ■ # of services instrumented ● more services = more spans ■ More spans = bigger traces ■ # of labels ● More labels = bigger spans ■ Egress costs sending to vendor ■ Cross-AZ costs & storage costs (self-hosting)
  • 21.
  • 22. In practice, sampling rates can be as low as 0.01%
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29.
  • 30.
  • 31.
  • 32.
  • 33.
  • 34. No Sampling ■ Best for very low throughput services/endpoints ■ Specific high impact time-boxed use cases ● A/B testing of a new endpoint ● Customer troubleshooting ● etc…
  • 35. Head
  • 36. 🚮
  • 41. Head Based Sampling ■ Efficient for systems with high throughput ■ Minimal impact on service performance ■ Lower financial cost ■ Easy to get started with ■ Tends to be configured statically requiring re-deployments to take effect
  • 42. Tail
  • 43. 🚮
  • 44. Tail Based Sampling ■ Works best for ● systems with low throughput ● systems with high throughput combined with head based sampling ■ Very configurable + easy to reason about ■ Higher financial and operational cost
  • 45.
  • 46. Which traces should we keep? Which traces should we discard? What constitutes an “interesting” trace?
  • 47.
  • 48.
  • 49. Edge case code paths Infrequent request types Anomalous events
  • 50.
  • 51.
  • 53.
  • 54.
  • 55.
  • 56. P99 Conf Template #1A1047 #00E5FF #753bf0 #FF2CDF #2B53F9 Font usage Share Tech or Roboto Color Palette Table Column 1 Column 2 Column 3 Column 5 Data 1 Data 2 Data 3 Data 4 Data 5 Data 6 Data 7 Data 8 #667EEA
  • 57. P99 Conf Template #1A1047 r26 g16 b71 c100 m100 y34 b45 Pantone 275c #00E5FF r0 g229 b255 c60 m0 y9 b0 #753bf0 r117 g59 b240 c79 m77 y0 b0 #FF2CDF r255 g44 b223 C31 m78 y0 b0 #2B53F9 r38 g24 b250 c91 m75 y0 b0 Pantone 2727c Color Palette Details #667EEA r102 g126 b234 c60 m35 y0 b5 Pantone 659c
  • 58. P99 Conf Template <here is some code> <styling> <use consolas for font when displaying code> <don’t go below 12pt font size>
  • 59. Slide title with white background Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer auctor eros eu faucibus sodales. Nunc dictum, urna id blandit pretium, mauris velit pulvinar ligula, interdum blandit sem tortor eget dolor. ■ Bullet 1 ● Bullet 2 ■ Bullet 3
  • 60. Slide title with white background Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer auctor eros eu faucibus sodales. Nunc dictum, urna id blandit pretium, mauris velit pulvinar ligula, interdum blandit sem tortor eget dolor. ■ Bullet 1 ● Bullet 2 ■ Bullet 3
  • 61. Slide title with white background Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer auctor eros eu faucibus sodales. Nunc dictum, urna id blandit pretium, mauris velit pulvinar ligula, interdum blandit sem tortor eget dolor. ■ Bullet 1 ● Bullet 2 ■ Bullet 3
  • 63.
  • 64.
  • 65.
  • 66.
  • 67. Clear slide for diagram with caption