Distributed tracing is still finding its footing in many organizations today, one challenge to overcome is the data volume - keeping 100% of your traces is expensive and unnecessary. Enter sampling - head vs tail how do you decide? Let’s review the tradeoffs associated with different types of sampling strategies and how they can be mixed and matched.
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
99.9% of Your Traces are Trash
1. HOSTED BY
99.9% of your traces are trash 🚮
Paige Cruz
Senior Developer Advocate at Chronosphere
2. Distributed tracing is still finding its footing in many organizations
today, one challenge to overcome is the data volume - keeping
100% of your traces is expensive and unnecessary. Enter sampling -
head vs tail how do you decide? Let’s look at the design of Sifter and
get familiar with why tail-based sampling is the way to enact a
cost-effective tracing solution while actually increasing the system’s
observability.
20. Factors that make tracing $$$$
■ High throughput = more traces generated
■ # of services instrumented
● more services = more spans
■ More spans = bigger traces
■ # of labels
● More labels = bigger spans
■ Egress costs sending to vendor
■ Cross-AZ costs & storage costs (self-hosting)
34. No Sampling
■ Best for very low throughput services/endpoints
■ Specific high impact time-boxed use cases
● A/B testing of a new endpoint
● Customer troubleshooting
● etc…
41. Head Based Sampling
■ Efficient for systems with high throughput
■ Minimal impact on service performance
■ Lower financial cost
■ Easy to get started with
■ Tends to be configured statically requiring
re-deployments to take effect
44. Tail Based Sampling
■ Works best for
● systems with low throughput
● systems with high throughput combined with
head based sampling
■ Very configurable + easy to reason about
■ Higher financial and operational cost
45.
46. Which traces should we keep?
Which traces should we discard?
What constitutes an “interesting” trace?
56. P99 Conf Template
#1A1047 #00E5FF
#753bf0 #FF2CDF
#2B53F9
Font usage Share Tech or Roboto
Color Palette
Table
Column 1 Column 2 Column 3 Column 5
Data 1 Data 2 Data 3 Data 4
Data 5 Data 6 Data 7 Data 8
#667EEA
58. P99 Conf Template
<here is some code>
<styling>
<use consolas for font when displaying code>
<don’t go below 12pt font size>
59. Slide title with white background
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer auctor eros eu
faucibus sodales. Nunc dictum, urna id blandit pretium, mauris velit pulvinar ligula,
interdum blandit sem tortor eget dolor.
■ Bullet 1
● Bullet 2
■ Bullet 3
60. Slide title with white background
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer auctor eros eu
faucibus sodales. Nunc dictum, urna id blandit pretium, mauris velit pulvinar ligula,
interdum blandit sem tortor eget dolor.
■ Bullet 1
● Bullet 2
■ Bullet 3
61. Slide title with white background
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer auctor eros eu
faucibus sodales. Nunc dictum, urna id blandit pretium, mauris velit pulvinar ligula,
interdum blandit sem tortor eget dolor.
■ Bullet 1
● Bullet 2
■ Bullet 3