SlideShare a Scribd company logo
1 of 37
Download to read offline
Built-in OpenTelemetry support
in Elasticsearch clients
Greg Kalapos
● Works at Elastic since 2018
● Created the Elastic .NET APM Agent
(completely OpenSource)
● Now focuses on OpenTelemetry
https://twitter.com/gregkalapos
Demo
Out of the box, built-in telemetry in
Elasticsearch clients based on
OpenTelemetry
What is OpenTelemetry?
● CNCF Project
● Collection of APIs, SDKs, and tools.
● To instrument, generate, collect, and
export telemetry data
● Metrics, logs, and traces
Signals
● Traces
● Metrics
● Logs
● (Profiling)
API
SDK
Agent / Auto Inst.
SDK
Context
Propagation
Semantic
Conventions
Collector
OTLP
…
Observability Backends
https://opentelemetry.io website:
UBIQUITOUS OBSERVABILITY . . .
imgflip.com
Wos isn des?!
Anatomy of a modern application
public class HomeController
{
public async Task<IActionResult> Index()
{
return View();
}
}
Anatomy of a modern application
public class HomeController : Controller
{
public async Task<IActionResult> Index()
{
return View();
}
}
Frameworks and libraries
public class HomeController : Controller
{
ElasticsearchClient _client = new(CloudId, new ApiKey(ApiKey));
public async Task<IActionResult> Index()
{
var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index"));
var doc = response.Source;
return View(doc);
}
}
Anatomy of a modern application
Frameworks and libraries
public class HomeController : Controller
{
ElasticsearchClient _client = new(CloudId, new ApiKey(ApiKey));
IProducer<string, string> producer = new ProducerBuilder<string, string>(config).Build();
public async Task<IActionResult> Index()
{
var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index"));
var deliveryResult = await producer.ProduceAsync(topic, message);
var doc = response.Source;
return View(doc);
}
}
Anatomy of a modern application
Frameworks and libraries
Observability
Things we want to see
HTTP GET - Index 723 ms
Elasticsearch - Get - my_index 360 ms
Kafka - Send 280 ms
Traces
Logs /
Events
[2023-11-06T15:23:40:152][INFO] HomeController - GET - /index - 200
[2023-11-06T15:23:40:167][INFO] ES Client - GET - my_index - 1
[2023-11-06T15:23:40:167][DEBUG] ES Client - Selected node MyNode-2
[2023-11-06T15:23:40:167][INFO] Kafka - Sending record to topic XYZ
[2023-11-06T15:23:40:167][WARN] Kafka - Failed sending, retrying …
Metrics
How do we get the data?
Instrumentation!
public async Task<IActionResult> Index()
{
var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index"));
var deliveryResult = await producer.ProduceAsync(topic, message);
var doc = response.Source;
return View(doc);
}
How do we get the data?
Instrumentation!
public async Task<IActionResult> Index()
{
using (var httpSpan = tracer.StartActiveSpan("HTTP GET Index"))
{
var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index"));
var deliveryResult = await producer.ProduceAsync(topic, message);
var doc = response.Source;
return View(doc);
}
}
HTTP GET - Index 723 ms
Traces
How do we get the data?
Instrumentation!
public async Task<IActionResult> Index()
{
using (var httpSpan = tracer.StartActiveSpan("HTTP GET Index"))
{
using (var esSpan = tracer.StartActiveSpan("ElasticSearch Get my_index"))
{
var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index"));
}
var deliveryResult = await producer.ProduceAsync(topic, message);
var doc = response.Source;
return View(doc);
}
}
HTTP GET - Index 723 ms
Elasticsearch - Get - my_index 360 ms
Traces
How do we get the data?
Instrumentation!
public async Task<IActionResult> Index()
{
using (var httpSpan = tracer.StartActiveSpan("HTTP GET Index"))
{
using (var esSpan = tracer.StartActiveSpan("ElasticSearch Get my_index"))
{
var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index"));
}
using (var kafkaSpan = tracer.StartActiveSpan("Kafka Send"))
{
var deliveryResult = await producer.ProduceAsync(topic, message);
}
var doc = response.Source;
return View(doc);
}
}
HTTP GET - /index 723 ms
Elasticsearch - Get - my_index 360 ms
Kafka - Send 280 ms
Traces
Photo by Andrea Piacquadio - pexels.com
LET’S DO IT FOR ALL THE
DEPENDENCIES …
… YOU WOULD GET ALL OF THIS FOR FREE
IMAGINE …
imgflip.com
imgflip.com
This is a solved problem!
WAIT !
imgflip.com
YOU’RE RIGHT!
imgflip.com
BUT, …
OTel Language
Team
OTel auto-instrumentation / Agents today
dependency
uses
instruments
Library
Application
Library
Team
Icons by icons8.com
Instrumentation
Module
…
auto-instrumentation / agent
LET ME WRITE THE
LOGS FOR YOU
OTEL DEV
WHAT ?
LIBRARY DEV imgflip.com
uses
OTel auto-instrumentation / Agents today
dependency
instruments
Library
Application
…
Library
Team
OTel Language
Team
Icons by icons8.com
Instrumentation
Module
auto-instrumentation / agent
uses
OTel auto-instrumentation / Agents today
The ownership problem
dependency
instruments
Library
Application
Instrumentation
Module
auto-instrumentation / agent
…
Library
Team
OTel Language
Team
● misplaced expertise ● scalability
Icons by icons8.com
● maintenance ● technical complexity
… OBSERVABILITY WOULD BE TRULY UBIQUITOUS
IMAGINE …
imgflip.com
Photo by Işıl - pexels.com
UBIQUITOUS
OBSERVABILITY
→
BAKED-IN LIBRARY
INSTRUMENTATION
Baked-in Instrumentation
Aligned ownership
dependency
uses
Icons by icons8.com
Application
uses
Library
Baked-in
Instrumentation
Library
Team
OTel Language
Team
SDK / Agent …
auto-instrumentation / agent
API
Our story at Elastic with
Elasticsearch clients
Semantic Conventions
● Common names for different kinds of operations and data.
● Common naming scheme that can be standardized across a
codebase, libraries, and platforms.
● https://opentelemetry.io/docs/concepts/semantic-conventions/
Semantic Conventions
for ElasticSearch
https://opentelemetry.io/docs/specs/semconv/database/elasticsearch/
Demo
● Same setup with Elastic APM
Implementation in specific language clients
● Adding reference to the OpenTelemetry API
● Implement built-in tracing
Implementing built-in tracing
Challenges
● Think from the perspective of the users of the
library
○ Don't over-engineer
● Be careful about span names
○ Pay attention to low cardinality
● Instrumentation always comes with overhead
○ Only instrument relevant code-path
○ Avoid code paths that are being called
excessively
● Do not collect sensitive data (by default)
Implementing built-in tracing
List of PRs
● Semantic Conventions:
https://github.com/open-telemetry/semantic-conventions/
pull/23
● Java:
https://github.com/elastic/elasticsearch-java/pull/588
● .NET:
https://github.com/elastic/elasticsearch-net/pull/6990
● Ruby:
https://github.com/elastic/elasticsearch-ruby/pull/2179
Implementing built-in tracing
List of PRs
● RabbitMQ .NET:
https://github.com/rabbitmq/rabbitmq-dotnet-client/pull/1
261
Future of native instrumentation
● Realistically: there will always be a mix of native + external
instrumentation
● OTel registry: https://opentelemetry.io/ecosystem/registry/

More Related Content

Similar to OSMC 2023 | Built-in OpenTelemetry support in Elasticsearch clients by Greg Kalapos

Building intranet applications with ASP.NET AJAX and jQuery
Building intranet applications with ASP.NET AJAX and jQueryBuilding intranet applications with ASP.NET AJAX and jQuery
Building intranet applications with ASP.NET AJAX and jQueryAlek Davis
 
Serverless Single Page Apps with React and Redux at ItCamp 2017
Serverless Single Page Apps with React and Redux at ItCamp 2017Serverless Single Page Apps with React and Redux at ItCamp 2017
Serverless Single Page Apps with React and Redux at ItCamp 2017Melania Andrisan (Danciu)
 
OpenStack Workshop - WECode Harvard Conference
OpenStack Workshop - WECode Harvard ConferenceOpenStack Workshop - WECode Harvard Conference
OpenStack Workshop - WECode Harvard ConferenceIccha Sethi
 
Apache Spark™ + IBM Watson + Twitter DataPalooza SF 2015
Apache Spark™ + IBM Watson + Twitter DataPalooza SF 2015Apache Spark™ + IBM Watson + Twitter DataPalooza SF 2015
Apache Spark™ + IBM Watson + Twitter DataPalooza SF 2015Mike Broberg
 
Scalable Open-Source IoT Solutions on Microsoft Azure
Scalable Open-Source IoT Solutions on Microsoft AzureScalable Open-Source IoT Solutions on Microsoft Azure
Scalable Open-Source IoT Solutions on Microsoft AzureMaxim Ivannikov
 
Why apache Flink is the 4G of Big Data Analytics Frameworks
Why apache Flink is the 4G of Big Data Analytics FrameworksWhy apache Flink is the 4G of Big Data Analytics Frameworks
Why apache Flink is the 4G of Big Data Analytics FrameworksSlim Baltagi
 
Introduction to interactive data visualisation using R Shiny
Introduction to interactive data visualisation using R ShinyIntroduction to interactive data visualisation using R Shiny
Introduction to interactive data visualisation using R Shinyanamarisaguedes
 
Logging for Production Systems in The Container Era
Logging for Production Systems in The Container EraLogging for Production Systems in The Container Era
Logging for Production Systems in The Container EraSadayuki Furuhashi
 
System insight without Interference
System insight without InterferenceSystem insight without Interference
System insight without InterferenceTony Tam
 
Apache Flink Adoption at Shopify
Apache Flink Adoption at ShopifyApache Flink Adoption at Shopify
Apache Flink Adoption at ShopifyYaroslav Tkachenko
 
LINQ 2 SQL Presentation To Palmchip And Trg, Technology Resource Group
LINQ 2 SQL Presentation To Palmchip  And Trg, Technology Resource GroupLINQ 2 SQL Presentation To Palmchip  And Trg, Technology Resource Group
LINQ 2 SQL Presentation To Palmchip And Trg, Technology Resource GroupShahzad
 
Web_of_Things_2013
Web_of_Things_2013Web_of_Things_2013
Web_of_Things_2013Max Kleiner
 
ThroughTheLookingGlass_EffectiveObservability.pptx
ThroughTheLookingGlass_EffectiveObservability.pptxThroughTheLookingGlass_EffectiveObservability.pptx
ThroughTheLookingGlass_EffectiveObservability.pptxGrace Jansen
 
How and why we evolved a legacy Java web application to Scala... and we are s...
How and why we evolved a legacy Java web application to Scala... and we are s...How and why we evolved a legacy Java web application to Scala... and we are s...
How and why we evolved a legacy Java web application to Scala... and we are s...Katia Aresti
 
Getting Started with Innoslate
Getting Started with InnoslateGetting Started with Innoslate
Getting Started with InnoslateElizabeth Steiner
 
Easing offline web application development with GWT
Easing offline web application development with GWTEasing offline web application development with GWT
Easing offline web application development with GWTArnaud Tournier
 
UniRx - Reactive Extensions for Unity(EN)
UniRx - Reactive Extensions for Unity(EN)UniRx - Reactive Extensions for Unity(EN)
UniRx - Reactive Extensions for Unity(EN)Yoshifumi Kawai
 
Near real-time anomaly detection at Lyft
Near real-time anomaly detection at LyftNear real-time anomaly detection at Lyft
Near real-time anomaly detection at Lyftmarkgrover
 
C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...
C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...
C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...DataStax Academy
 

Similar to OSMC 2023 | Built-in OpenTelemetry support in Elasticsearch clients by Greg Kalapos (20)

Building intranet applications with ASP.NET AJAX and jQuery
Building intranet applications with ASP.NET AJAX and jQueryBuilding intranet applications with ASP.NET AJAX and jQuery
Building intranet applications with ASP.NET AJAX and jQuery
 
Serverless Single Page Apps with React and Redux at ItCamp 2017
Serverless Single Page Apps with React and Redux at ItCamp 2017Serverless Single Page Apps with React and Redux at ItCamp 2017
Serverless Single Page Apps with React and Redux at ItCamp 2017
 
OpenStack Workshop - WECode Harvard Conference
OpenStack Workshop - WECode Harvard ConferenceOpenStack Workshop - WECode Harvard Conference
OpenStack Workshop - WECode Harvard Conference
 
Apache Spark™ + IBM Watson + Twitter DataPalooza SF 2015
Apache Spark™ + IBM Watson + Twitter DataPalooza SF 2015Apache Spark™ + IBM Watson + Twitter DataPalooza SF 2015
Apache Spark™ + IBM Watson + Twitter DataPalooza SF 2015
 
Scalable Open-Source IoT Solutions on Microsoft Azure
Scalable Open-Source IoT Solutions on Microsoft AzureScalable Open-Source IoT Solutions on Microsoft Azure
Scalable Open-Source IoT Solutions on Microsoft Azure
 
Why apache Flink is the 4G of Big Data Analytics Frameworks
Why apache Flink is the 4G of Big Data Analytics FrameworksWhy apache Flink is the 4G of Big Data Analytics Frameworks
Why apache Flink is the 4G of Big Data Analytics Frameworks
 
Introduction to interactive data visualisation using R Shiny
Introduction to interactive data visualisation using R ShinyIntroduction to interactive data visualisation using R Shiny
Introduction to interactive data visualisation using R Shiny
 
Logging for Production Systems in The Container Era
Logging for Production Systems in The Container EraLogging for Production Systems in The Container Era
Logging for Production Systems in The Container Era
 
System insight without Interference
System insight without InterferenceSystem insight without Interference
System insight without Interference
 
Apache Flink Adoption at Shopify
Apache Flink Adoption at ShopifyApache Flink Adoption at Shopify
Apache Flink Adoption at Shopify
 
LINQ 2 SQL Presentation To Palmchip And Trg, Technology Resource Group
LINQ 2 SQL Presentation To Palmchip  And Trg, Technology Resource GroupLINQ 2 SQL Presentation To Palmchip  And Trg, Technology Resource Group
LINQ 2 SQL Presentation To Palmchip And Trg, Technology Resource Group
 
Web_of_Things_2013
Web_of_Things_2013Web_of_Things_2013
Web_of_Things_2013
 
ThroughTheLookingGlass_EffectiveObservability.pptx
ThroughTheLookingGlass_EffectiveObservability.pptxThroughTheLookingGlass_EffectiveObservability.pptx
ThroughTheLookingGlass_EffectiveObservability.pptx
 
How and why we evolved a legacy Java web application to Scala... and we are s...
How and why we evolved a legacy Java web application to Scala... and we are s...How and why we evolved a legacy Java web application to Scala... and we are s...
How and why we evolved a legacy Java web application to Scala... and we are s...
 
Getting Started with Innoslate
Getting Started with InnoslateGetting Started with Innoslate
Getting Started with Innoslate
 
Easing offline web application development with GWT
Easing offline web application development with GWTEasing offline web application development with GWT
Easing offline web application development with GWT
 
UniRx - Reactive Extensions for Unity(EN)
UniRx - Reactive Extensions for Unity(EN)UniRx - Reactive Extensions for Unity(EN)
UniRx - Reactive Extensions for Unity(EN)
 
Eclipse e4
Eclipse e4Eclipse e4
Eclipse e4
 
Near real-time anomaly detection at Lyft
Near real-time anomaly detection at LyftNear real-time anomaly detection at Lyft
Near real-time anomaly detection at Lyft
 
C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...
C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...
C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...
 

Recently uploaded

The Influence and Evolution of Mogul Press in Contemporary Public Relations.docx
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docxThe Influence and Evolution of Mogul Press in Contemporary Public Relations.docx
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docxMogul Press
 
2024-05-15-Surat Meetup-Hyperautomation.pptx
2024-05-15-Surat Meetup-Hyperautomation.pptx2024-05-15-Surat Meetup-Hyperautomation.pptx
2024-05-15-Surat Meetup-Hyperautomation.pptxnitishjain2015
 
Deciding The Topic of our Magazine.pptx.
Deciding The Topic of our Magazine.pptx.Deciding The Topic of our Magazine.pptx.
Deciding The Topic of our Magazine.pptx.bazilnaeem7
 
Breathing in New Life_ Part 3 05 22 2024.pptx
Breathing in New Life_ Part 3 05 22 2024.pptxBreathing in New Life_ Part 3 05 22 2024.pptx
Breathing in New Life_ Part 3 05 22 2024.pptxFamilyWorshipCenterD
 
Microsoft Fabric Analytics Engineer (DP-600) Exam Dumps 2024.pdf
Microsoft Fabric Analytics Engineer (DP-600) Exam Dumps 2024.pdfMicrosoft Fabric Analytics Engineer (DP-600) Exam Dumps 2024.pdf
Microsoft Fabric Analytics Engineer (DP-600) Exam Dumps 2024.pdfSkillCertProExams
 
Understanding Poverty: A Community Questionnaire
Understanding Poverty: A Community QuestionnaireUnderstanding Poverty: A Community Questionnaire
Understanding Poverty: A Community Questionnairebazilnaeem7
 
ACM CHT Best Inspection Practices Kinben Innovation MIC Slideshare.pdf
ACM CHT Best Inspection Practices Kinben Innovation MIC Slideshare.pdfACM CHT Best Inspection Practices Kinben Innovation MIC Slideshare.pdf
ACM CHT Best Inspection Practices Kinben Innovation MIC Slideshare.pdfKinben Innovation Private Limited
 
ServiceNow CIS-Discovery Exam Dumps 2024
ServiceNow CIS-Discovery Exam Dumps 2024ServiceNow CIS-Discovery Exam Dumps 2024
ServiceNow CIS-Discovery Exam Dumps 2024SkillCertProExams
 
DAY 0 8 A Revelation 05-19-2024 PPT.pptx
DAY 0 8 A Revelation 05-19-2024 PPT.pptxDAY 0 8 A Revelation 05-19-2024 PPT.pptx
DAY 0 8 A Revelation 05-19-2024 PPT.pptxFamilyWorshipCenterD
 
Databricks Machine Learning Associate Exam Dumps 2024.pdf
Databricks Machine Learning Associate Exam Dumps 2024.pdfDatabricks Machine Learning Associate Exam Dumps 2024.pdf
Databricks Machine Learning Associate Exam Dumps 2024.pdfSkillCertProExams
 

Recently uploaded (10)

The Influence and Evolution of Mogul Press in Contemporary Public Relations.docx
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docxThe Influence and Evolution of Mogul Press in Contemporary Public Relations.docx
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docx
 
2024-05-15-Surat Meetup-Hyperautomation.pptx
2024-05-15-Surat Meetup-Hyperautomation.pptx2024-05-15-Surat Meetup-Hyperautomation.pptx
2024-05-15-Surat Meetup-Hyperautomation.pptx
 
Deciding The Topic of our Magazine.pptx.
Deciding The Topic of our Magazine.pptx.Deciding The Topic of our Magazine.pptx.
Deciding The Topic of our Magazine.pptx.
 
Breathing in New Life_ Part 3 05 22 2024.pptx
Breathing in New Life_ Part 3 05 22 2024.pptxBreathing in New Life_ Part 3 05 22 2024.pptx
Breathing in New Life_ Part 3 05 22 2024.pptx
 
Microsoft Fabric Analytics Engineer (DP-600) Exam Dumps 2024.pdf
Microsoft Fabric Analytics Engineer (DP-600) Exam Dumps 2024.pdfMicrosoft Fabric Analytics Engineer (DP-600) Exam Dumps 2024.pdf
Microsoft Fabric Analytics Engineer (DP-600) Exam Dumps 2024.pdf
 
Understanding Poverty: A Community Questionnaire
Understanding Poverty: A Community QuestionnaireUnderstanding Poverty: A Community Questionnaire
Understanding Poverty: A Community Questionnaire
 
ACM CHT Best Inspection Practices Kinben Innovation MIC Slideshare.pdf
ACM CHT Best Inspection Practices Kinben Innovation MIC Slideshare.pdfACM CHT Best Inspection Practices Kinben Innovation MIC Slideshare.pdf
ACM CHT Best Inspection Practices Kinben Innovation MIC Slideshare.pdf
 
ServiceNow CIS-Discovery Exam Dumps 2024
ServiceNow CIS-Discovery Exam Dumps 2024ServiceNow CIS-Discovery Exam Dumps 2024
ServiceNow CIS-Discovery Exam Dumps 2024
 
DAY 0 8 A Revelation 05-19-2024 PPT.pptx
DAY 0 8 A Revelation 05-19-2024 PPT.pptxDAY 0 8 A Revelation 05-19-2024 PPT.pptx
DAY 0 8 A Revelation 05-19-2024 PPT.pptx
 
Databricks Machine Learning Associate Exam Dumps 2024.pdf
Databricks Machine Learning Associate Exam Dumps 2024.pdfDatabricks Machine Learning Associate Exam Dumps 2024.pdf
Databricks Machine Learning Associate Exam Dumps 2024.pdf
 

OSMC 2023 | Built-in OpenTelemetry support in Elasticsearch clients by Greg Kalapos

  • 1. Built-in OpenTelemetry support in Elasticsearch clients
  • 2. Greg Kalapos ● Works at Elastic since 2018 ● Created the Elastic .NET APM Agent (completely OpenSource) ● Now focuses on OpenTelemetry https://twitter.com/gregkalapos
  • 3. Demo Out of the box, built-in telemetry in Elasticsearch clients based on OpenTelemetry
  • 4. What is OpenTelemetry? ● CNCF Project ● Collection of APIs, SDKs, and tools. ● To instrument, generate, collect, and export telemetry data ● Metrics, logs, and traces
  • 5. Signals ● Traces ● Metrics ● Logs ● (Profiling) API SDK Agent / Auto Inst. SDK Context Propagation Semantic Conventions Collector OTLP … Observability Backends
  • 7. UBIQUITOUS OBSERVABILITY . . . imgflip.com Wos isn des?!
  • 8. Anatomy of a modern application public class HomeController { public async Task<IActionResult> Index() { return View(); } }
  • 9. Anatomy of a modern application public class HomeController : Controller { public async Task<IActionResult> Index() { return View(); } } Frameworks and libraries
  • 10. public class HomeController : Controller { ElasticsearchClient _client = new(CloudId, new ApiKey(ApiKey)); public async Task<IActionResult> Index() { var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index")); var doc = response.Source; return View(doc); } } Anatomy of a modern application Frameworks and libraries
  • 11. public class HomeController : Controller { ElasticsearchClient _client = new(CloudId, new ApiKey(ApiKey)); IProducer<string, string> producer = new ProducerBuilder<string, string>(config).Build(); public async Task<IActionResult> Index() { var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index")); var deliveryResult = await producer.ProduceAsync(topic, message); var doc = response.Source; return View(doc); } } Anatomy of a modern application Frameworks and libraries
  • 12. Observability Things we want to see HTTP GET - Index 723 ms Elasticsearch - Get - my_index 360 ms Kafka - Send 280 ms Traces Logs / Events [2023-11-06T15:23:40:152][INFO] HomeController - GET - /index - 200 [2023-11-06T15:23:40:167][INFO] ES Client - GET - my_index - 1 [2023-11-06T15:23:40:167][DEBUG] ES Client - Selected node MyNode-2 [2023-11-06T15:23:40:167][INFO] Kafka - Sending record to topic XYZ [2023-11-06T15:23:40:167][WARN] Kafka - Failed sending, retrying … Metrics
  • 13. How do we get the data? Instrumentation! public async Task<IActionResult> Index() { var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index")); var deliveryResult = await producer.ProduceAsync(topic, message); var doc = response.Source; return View(doc); }
  • 14. How do we get the data? Instrumentation! public async Task<IActionResult> Index() { using (var httpSpan = tracer.StartActiveSpan("HTTP GET Index")) { var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index")); var deliveryResult = await producer.ProduceAsync(topic, message); var doc = response.Source; return View(doc); } } HTTP GET - Index 723 ms Traces
  • 15. How do we get the data? Instrumentation! public async Task<IActionResult> Index() { using (var httpSpan = tracer.StartActiveSpan("HTTP GET Index")) { using (var esSpan = tracer.StartActiveSpan("ElasticSearch Get my_index")) { var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index")); } var deliveryResult = await producer.ProduceAsync(topic, message); var doc = response.Source; return View(doc); } } HTTP GET - Index 723 ms Elasticsearch - Get - my_index 360 ms Traces
  • 16. How do we get the data? Instrumentation! public async Task<IActionResult> Index() { using (var httpSpan = tracer.StartActiveSpan("HTTP GET Index")) { using (var esSpan = tracer.StartActiveSpan("ElasticSearch Get my_index")) { var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index")); } using (var kafkaSpan = tracer.StartActiveSpan("Kafka Send")) { var deliveryResult = await producer.ProduceAsync(topic, message); } var doc = response.Source; return View(doc); } } HTTP GET - /index 723 ms Elasticsearch - Get - my_index 360 ms Kafka - Send 280 ms Traces
  • 17. Photo by Andrea Piacquadio - pexels.com LET’S DO IT FOR ALL THE DEPENDENCIES …
  • 18. … YOU WOULD GET ALL OF THIS FOR FREE IMAGINE … imgflip.com
  • 19. imgflip.com This is a solved problem! WAIT !
  • 22. OTel Language Team OTel auto-instrumentation / Agents today dependency uses instruments Library Application Library Team Icons by icons8.com Instrumentation Module … auto-instrumentation / agent
  • 23. LET ME WRITE THE LOGS FOR YOU OTEL DEV WHAT ? LIBRARY DEV imgflip.com
  • 24. uses OTel auto-instrumentation / Agents today dependency instruments Library Application … Library Team OTel Language Team Icons by icons8.com Instrumentation Module auto-instrumentation / agent
  • 25. uses OTel auto-instrumentation / Agents today The ownership problem dependency instruments Library Application Instrumentation Module auto-instrumentation / agent … Library Team OTel Language Team ● misplaced expertise ● scalability Icons by icons8.com ● maintenance ● technical complexity
  • 26. … OBSERVABILITY WOULD BE TRULY UBIQUITOUS IMAGINE … imgflip.com
  • 27. Photo by Işıl - pexels.com UBIQUITOUS OBSERVABILITY → BAKED-IN LIBRARY INSTRUMENTATION
  • 28. Baked-in Instrumentation Aligned ownership dependency uses Icons by icons8.com Application uses Library Baked-in Instrumentation Library Team OTel Language Team SDK / Agent … auto-instrumentation / agent API
  • 29. Our story at Elastic with Elasticsearch clients
  • 30. Semantic Conventions ● Common names for different kinds of operations and data. ● Common naming scheme that can be standardized across a codebase, libraries, and platforms. ● https://opentelemetry.io/docs/concepts/semantic-conventions/
  • 32. Demo ● Same setup with Elastic APM
  • 33. Implementation in specific language clients ● Adding reference to the OpenTelemetry API ● Implement built-in tracing
  • 34. Implementing built-in tracing Challenges ● Think from the perspective of the users of the library ○ Don't over-engineer ● Be careful about span names ○ Pay attention to low cardinality ● Instrumentation always comes with overhead ○ Only instrument relevant code-path ○ Avoid code paths that are being called excessively ● Do not collect sensitive data (by default)
  • 35. Implementing built-in tracing List of PRs ● Semantic Conventions: https://github.com/open-telemetry/semantic-conventions/ pull/23 ● Java: https://github.com/elastic/elasticsearch-java/pull/588 ● .NET: https://github.com/elastic/elasticsearch-net/pull/6990 ● Ruby: https://github.com/elastic/elasticsearch-ruby/pull/2179
  • 36. Implementing built-in tracing List of PRs ● RabbitMQ .NET: https://github.com/rabbitmq/rabbitmq-dotnet-client/pull/1 261
  • 37. Future of native instrumentation ● Realistically: there will always be a mix of native + external instrumentation ● OTel registry: https://opentelemetry.io/ecosystem/registry/