Are you looking at your organization's efforts to enter or expand into the cloud native landscape and feeling a bit daunted by the vast expanse of information surrounding cloud native observability? When you're moving so fast with agile practices across your DevOps, SRE's, and platform engineering teams, it's no wonder this can seem a bit confusing. Unfortunately, the choices being made have a great impact on both your business, your budgets, and the ultimate success of your cloud native initiatives. That hasty decision up front leads to big headaches very quickly down the road. In this talk, I'll introduce the problem facing everyone with cloud native observability followed by 3 common mistakes that I'm seeing organizations make and how you can avoid them! Key takeaways - This session is never the same twice as you the audience / attendees choose from a list of cloud native observability pitfalls that DevOps have to contend with in their daily cloud native lives! Super engaging and fun to tour the challenges that interest you most!

1. chronosphere.io
chronosphere.io
Choose Your Own Adventure
Eric D. Schabell
Director Evangelism
@ericschabell{@fosstodon.org}
Cloud Native Observability Pitfalls

2. chronosphere.io
Cloud Native Observability

3. chronosphere.io
Cloud Native

4. chronosphere.io
Data volume
Experiment:
- Hello World app on 4 node
Kubernetes cluster with
Tracing, End User Metrics
(EUM), Logs, Metrics
(containers / nodes)
- 30 days == +450 GB

5. chronosphere.io

6. chronosphere.io
Cloud Native at Scale

7. chronosphere.io
Observability…

8. chronosphere.io
Cloud Native Observability
at Scale

9. chronosphere.io
O11y at Scale (need)

10. chronosphere.io
Picking Your Pitfalls
1. Ignoring existing landscape
2. Focusing on The Pillars
3. Sneaky sprawling mess
4. Controlling costs
5. The protocol jungle
6. Underestimating cardinality
(Click on a pitfall to jump to that section)

11. chronosphere.io
1. Ignoring existing
landscape

12. chronosphere.io
If they can’t
see me…
they can’t
hurt me...

13. chronosphere.io

14. chronosphere.io
Prometheus for metrics, alerting, queries

15. chronosphere.io
Prometheus auto discovery

16. chronosphere.io
Manual instrumentation (java client lib)

17. chronosphere.io
Short link: bit.ly/prom-
workshop

18. chronosphere.io
Applications (Java)
OTel Auto Instrumentation (libraries)
OTel API
OTel SDK
OTel Collector
OTLP
OTLP
OTLP
OpenTelemetry (Auto) instrumentation

19. chronosphere.io
Host
Observability Backend
(Prometheus, Jaeger, Fluent Bit, etc.),
Applications
OTel Auto Instrumentation
OTel API
OTel SDK
OTel Collector Agent
OTLP
OTLP
OTLP
OTLP
OTLP
OpenTelemetry Collector (agent)

20. chronosphere.io
Host
Host
Host
Observability Backend
(Prometheus, Jaeger, Fluent Bit, etc.),
Applications
OTel Auto Instrumentation
OTel API
OTel SDK
OTel Collector Agent
OTLP
OTLP
OTLP
OTLP
Collector (gateway)
OTel Collector Gateway

21. chronosphere.io
Short link: bit.ly/opentelemetry-
workshop

22. chronosphere.io
1. Ignoring existing landscape
2. Focusing on The Pillars
3. Sneaky sprawling mess
4. Controlling costs
5. The protocol jungle
6. Underestimating cardinality
(Click on a pitfall to jump to that section, or jump to end)
Picking Your Next Pitfall

23. chronosphere.io
2. Focusing on The Pillars

24. chronosphere.io
Pillars Phases

25. chronosphere.io
Developer
Technology
Bottom up

26. chronosphere.io
Pillar problems…

27. chronosphere.io
Car is on fire…

28. chronosphere.io
Better outcomes…
Faster remediation…
Easier detection…
Happier customers…

29. chronosphere.io
Phase 1
Know something is
happening as fast
as possible…

30. chronosphere.io
Phase 2
Triage with specific
information…

31. chronosphere.io
Phase 3
Understand to
ensure never
happens again…

32. chronosphere.io

33. chronosphere.io

34. chronosphere.io
1. Ignoring existing landscape
2. Focusing on The Pillars
3. Sneaky sprawling mess
4. Controlling costs
5. The protocol jungle
6. Underestimating cardinality
(Click on a pitfall to jump to that section, or jump to end)
Picking Your Next Pitfall

35. chronosphere.io
3. Sneaky sprawling mess

36. chronosphere.io
Over 66% of organizations
use more than 10 different
observability tools
– ESG report over exploding data volumes

37. chronosphere.io

38. chronosphere.io
Know
Triage
Understand

39. chronosphere.io

40. chronosphere.io
1. Ignoring existing landscape
2. Focusing on The Pillars
3. Sneaky sprawling mess
4. Controlling costs
5. The protocol jungle
6. Underestimating cardinality
(Click on a pitfall to jump to that section, or jump to end)
Picking Your Next Pitfall

41. chronosphere.io
4. Controlling costs

42. chronosphere.io
“It’s remarkable how common
this situation is, where an
organization is paying more for
their observability data, than
they do for their production
infrastructure.”

43. chronosphere.io
O11y data storage costs
are broken.
Keeping everything
model?

44. chronosphere.io
Know the cost of
observability
metrics data?

45. chronosphere.io
DATA
COLLECTION
CONTROL PLANE
PURPOSE-BUILT DATA STORES
PER TELEMETRY TYPE
CHRONOSPHERE LENS
Align cost to value Single Tenanted Architecture w/
99.99% Reliability
Turns raw data into generated
insights for each user
Customer Environment Chronosphere SaaS Platform
METRICS
|
LOGS
|
TRACES
|
EVENTS
Ingest all your data from
any source

46. chronosphere.io

47. chronosphere.io
1. Ignoring existing landscape
2. Focusing on The Pillars
3. Sneaky sprawling mess
4. Controlling costs
5. The protocol jungle
6. Underestimating cardinality
(Click on a pitfall to jump to that section, or jump to end)
Picking Your Next Pitfall

48. chronosphere.io
5. The protocol jungle

49. chronosphere.io
Without open standards,
you’ll not find a way back…

50. chronosphere.io

51. chronosphere.io
Host
Observability Backend
(Prometheus, Jaeger, Fluent Bit, etc.),
Applications
OTel Auto Instrumentation
OTel API
OTel SDK
OTel Collector Agent
OTLP
OTLP
OTLP
OTLP
OTLP
OpenTelemetry Collector (agent)

52. chronosphere.io
Prometheus for metrics, alerting, queries

53. chronosphere.io
1. Ignoring existing landscape
2. Focusing on The Pillars
3. Sneaky sprawling mess
4. Controlling costs
5. The protocol jungle
6. Underestimating cardinality
(Click on a pitfall to jump to that section, or jump to end)
Picking Your Next Pitfall

54. chronosphere.io
6. Underestimating cardinality

55. chronosphere.io
The struggle is real
“I don't yet collect spans/traces because I can hardly get our devs to care about basic metrics, let alone
traces.”
“This is a large enterprise with approx. 1000 developers. Cultivating a culture of engineering that cares
about availability is a challenge that we need to solve alongside any technical implementations.”

56. chronosphere.io
10 hours
on average, per week,
trying to triage and
understand incidents -
a quarter of a
40 hour work week

57. chronosphere.io
33%
said those issues
disrupted their
personal life
39%
admitting they are
frequently
stressed out

58. chronosphere.io
Cloud Native
Observability
at Scale

59. chronosphere.io
DATA
COLLECTION
CONTROL PLANE
PURPOSE-BUILT DATA STORES
PER TELEMETRY TYPE
CHRONOSPHERE LENS
Align cost to value Single Tenanted Architecture w/
99.99% Reliability
Turns raw data into generated
insights for each user
Customer Environment Chronosphere SaaS Platform
METRICS
|
LOGS
|
TRACES
|
EVENTS
Ingest all your data from
any source

60. chronosphere.io
1. Ignoring existing landscape
2. Focusing on The Pillars
3. Sneaky sprawling mess
4. Controlling costs
5. The protocol jungle
6. Underestimating cardinality
(Click on a pitfall to jump to that section, or jump to end)
Picking Your Next Pitfall

61. chronosphere.io
What should be
the #1 item on
cloud wishlist?
What should be #1 item on your
cloud native observability
wishlist?

62. chronosphere.io
chronosphere.io
Questions?
Eric D. Schabell
Director Evangelism
@ericschabell{@fosstodon.org}