Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
What to Upload to SlideShare
What to Upload to SlideShare
Loading in …3
×
1 of 25

Introduction to Open Telemetry as Observability Library

0

Share

Download to read offline

My talks on OpenInfra Asia 2021

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Introduction to Open Telemetry as Observability Library

  1. 1. Introduction to Open Telemetry -Your Universal Telemetry Library- Tonny Adhi Sabastian (tonny.sabastian@gojek.com) Open Infrastructure Day Asia - September 11th, 2021
  2. 2. About Me ● Currently : Senior DevOps Engineer at MAPAN (GoTo Group) - https://www.mapan.id (2018 - Now) ○ We’re hiring, please visit https://www.career.mapan.id/ ○ Send your CV to mapan-recruitment@gojek.com ○ Interest in our DevOps engineering ? PM me via mail ● Former Lead Engineer for University of Indonesia - Data Center (2012 - 2018) - https://ui.ac.id ● Former Lecturer at Faculty of Computer Science , University of Indonesia (2014 - 2018) ○ Teaching Subject : Distributed System , System Programming, Modern IT Infrastructure ○ Research Subject : IoT , Deep Packet Inspection, Aeronautical Telecommunication Network, Linux System Performance ● Co-founding two startup during 2012 - 2018 , (Rumio - AR Platform & Peentar - IoT Platform) - now defunct ● Former volunteer system administrator for kambing.ui.ac.id - one of Indonesia largest F/OSS repository ; now defunct (2009 - 2018) ● Contact : tonny.sabastian@gojek.com / tonny@segmentationfault.xyz
  3. 3. Agenda ● Observability 101 ● Meet OpenTelemetry ● Instrumenting with OpenTelemetry - Tracing with Go Example - ● OpenTelemetry @ MAPAN
  4. 4. Observability 101
  5. 5. The Observability Triad Do you know what your codes are doing on 2AM when your payment service is down ? Monitoring Tracing Logging
  6. 6. Metrics - It’s About The Number ● Each metric reports a number ● The monitoring system tracks those numbers over time ● Identify anomalous behavior and wake someone up ● Recognizes trends for your infrastructure scaling ● Four Golden Signals - ( Latency - Traffic - Saturation - Errors )
  7. 7. Tracing - Finding Your Bottleneck ● Uniquely identify Requests and Spans ● Spans carry contextual metadata ● Runtime reports start/end timestamps for each span ● Span Aggregator find out how long everything took
  8. 8. Logging - How to debug your production at 2AM ● Convey appropriate urgency ● Provide enough information ● Not overwhelming ● Structured for searching not reading
  9. 9. Meet OpenTelemetry
  10. 10. OpenTelemetry ● OpenCensus + OpenTracing = OpenTelemetry ● Provides a unified set of open standards API, SDK, and middleware for generating and exporting Trace , Metrics, (and logging - incoming) ● Enabling quick bootstrap of observability into your system ● Contributor from various major vendor and wide community support source : https://opentelemetry.io
  11. 11. OpenTelemetry Architecture (1) ● API : Used to generate telemetry data ● SDK : Implementation of API in various languages ○ Tracer : responsible for creating and tracking span - distributed tracing ○ Meter : responsible for creating and tracking metrics ● OpenTelemetry Proto : protobuf implementation on semantic convention and OpenTelemetry Protocol (OTLP) source : https://blog.newrelic.com/product-news/what-is-opentelemetry/
  12. 12. OpenTelemetry Architecture (2) ● Collector : middleware service on receiving, processing, and exporting telemetry data to various telemetry backend (ex : jaeger, prometheus, datadog, etc). It’s acting as telemetry pipeline ● Collector composed of following components : ○ receivers : receive telemetry data ○ processors : telemetry data additional processing ○ exporters : send telemetry data to final backend ● Collector can be deployed as Agent or Gateway source : https://blog.newrelic.com/product-news/what-is-opentelemetry/
  13. 13. Instrumenting with OpenTelemetry
  14. 14. Tracing API Concepts ● TracerProvider is the entry point of the API. It’s providing access to Tracers ○ TracerProvider register global configuration of exporter, tracing attributes and sampler ● Tracer is the class responsible for creating Spans ● Spans is the API to trace an operation within your code
  15. 15. Tracing SDK Concepts ● Sampler , controls the number of traces capture and send to backend ○ Sampling Decision Values [ DROP | RECORD_ONLY | RECORD_AND_SAMPLE ] ○ SDK Built In Samplers [ AlwaysOn | AlwaysOff | TraceIDRatioBased ] ● SpanProcessor , hook for span start and stop invocation ○ SDK Built In Processor [ ExportFormatConverter | Batcher ] ○ Only invoked when isRecording is true ● SpanExporter, defines interfaces for specific backend exporters (ex : Jaeger, OTLP, Lighstep, Datadog)
  16. 16. Go Instrumentation - Set Dependencies OpenTelemetry Go Dependencies
  17. 17. Go Instrumentation - Create Tracer ● Set SpanExporter using OpenTelemetry Protocol via otlpgrpc ● Set initial SpanAttributes, such as your ServiceName ● Deploy TracerProvider with our established SpanExporter, Sampler and initial SpanAttributes
  18. 18. Go Instrumentation - Adding Span ● Create a Tracer and bind it to a TracerProvider ● Bind the Tracer to a Span ● Set additional SpanAttributes. Perhaps you want to trace your http request ● Track an event on your function. Add a SpanEvent to our Span
  19. 19. Go Instrumentation - Trace from your HTTP Router ● Create http handler function with http request path as its span context ● Bind this function with your http router
  20. 20. Go Instrumentation - Spawn your Tracer ● Create a context and bind it to your Tracer ● Bind your http handler function to your http router. This will create your root span
  21. 21. Go Instrumentation - Span Result on Jaeger
  22. 22. Go Instrumentation - OpenTelemetry @ MAPAN ● Wrap OpenTelemetry into our Observability Library ● Wrap our Observability Library into our HTTP Server Library ● Simple Observability Bootstraping via server start options
  23. 23. Demo
  24. 24. Instrumentation Demo ● Demo Repository in Golang https://github.com/tonnyadhi/distributed-tracing-example ● Contains two services ○ Weather Service, fetch weather report from backend and print it in JSON ○ OWM Service, fetch weather report from OpenWeathermap, simplify it and Sent it to Weather Service ● Distributed Tracing is set between two services using OpenTelemetry HTTP Traces ● Span are sent to OpenTelemetry Collector. The collector will export the span to Jaeger and Sentry
  25. 25. Thank You

×