SlideShare a Scribd company logo
Built-in OpenTelemetry support
in Elasticsearch clients
Greg Kalapos
● Works at Elastic since 2018
● Created the Elastic .NET APM Agent
(completely OpenSource)
● Now focuses on OpenTelemetry
https://twitter.com/gregkalapos
Demo
Out of the box, built-in telemetry in
Elasticsearch clients based on
OpenTelemetry
What is OpenTelemetry?
● CNCF Project
● Collection of APIs, SDKs, and tools.
● To instrument, generate, collect, and
export telemetry data
● Metrics, logs, and traces
Signals
● Traces
● Metrics
● Logs
● (Profiling)
API
SDK
Agent / Auto Inst.
SDK
Context
Propagation
Semantic
Conventions
Collector
OTLP
…
Observability Backends
https://opentelemetry.io website:
UBIQUITOUS OBSERVABILITY . . .
imgflip.com
Wos isn des?!
Anatomy of a modern application
public class HomeController
{
public async Task<IActionResult> Index()
{
return View();
}
}
Anatomy of a modern application
public class HomeController : Controller
{
public async Task<IActionResult> Index()
{
return View();
}
}
Frameworks and libraries
public class HomeController : Controller
{
ElasticsearchClient _client = new(CloudId, new ApiKey(ApiKey));
public async Task<IActionResult> Index()
{
var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index"));
var doc = response.Source;
return View(doc);
}
}
Anatomy of a modern application
Frameworks and libraries
public class HomeController : Controller
{
ElasticsearchClient _client = new(CloudId, new ApiKey(ApiKey));
IProducer<string, string> producer = new ProducerBuilder<string, string>(config).Build();
public async Task<IActionResult> Index()
{
var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index"));
var deliveryResult = await producer.ProduceAsync(topic, message);
var doc = response.Source;
return View(doc);
}
}
Anatomy of a modern application
Frameworks and libraries
Observability
Things we want to see
HTTP GET - Index 723 ms
Elasticsearch - Get - my_index 360 ms
Kafka - Send 280 ms
Traces
Logs /
Events
[2023-11-06T15:23:40:152][INFO] HomeController - GET - /index - 200
[2023-11-06T15:23:40:167][INFO] ES Client - GET - my_index - 1
[2023-11-06T15:23:40:167][DEBUG] ES Client - Selected node MyNode-2
[2023-11-06T15:23:40:167][INFO] Kafka - Sending record to topic XYZ
[2023-11-06T15:23:40:167][WARN] Kafka - Failed sending, retrying …
Metrics
How do we get the data?
Instrumentation!
public async Task<IActionResult> Index()
{
var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index"));
var deliveryResult = await producer.ProduceAsync(topic, message);
var doc = response.Source;
return View(doc);
}
How do we get the data?
Instrumentation!
public async Task<IActionResult> Index()
{
using (var httpSpan = tracer.StartActiveSpan("HTTP GET Index"))
{
var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index"));
var deliveryResult = await producer.ProduceAsync(topic, message);
var doc = response.Source;
return View(doc);
}
}
HTTP GET - Index 723 ms
Traces
How do we get the data?
Instrumentation!
public async Task<IActionResult> Index()
{
using (var httpSpan = tracer.StartActiveSpan("HTTP GET Index"))
{
using (var esSpan = tracer.StartActiveSpan("ElasticSearch Get my_index"))
{
var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index"));
}
var deliveryResult = await producer.ProduceAsync(topic, message);
var doc = response.Source;
return View(doc);
}
}
HTTP GET - Index 723 ms
Elasticsearch - Get - my_index 360 ms
Traces
How do we get the data?
Instrumentation!
public async Task<IActionResult> Index()
{
using (var httpSpan = tracer.StartActiveSpan("HTTP GET Index"))
{
using (var esSpan = tracer.StartActiveSpan("ElasticSearch Get my_index"))
{
var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index"));
}
using (var kafkaSpan = tracer.StartActiveSpan("Kafka Send"))
{
var deliveryResult = await producer.ProduceAsync(topic, message);
}
var doc = response.Source;
return View(doc);
}
}
HTTP GET - /index 723 ms
Elasticsearch - Get - my_index 360 ms
Kafka - Send 280 ms
Traces
Photo by Andrea Piacquadio - pexels.com
LET’S DO IT FOR ALL THE
DEPENDENCIES …
… YOU WOULD GET ALL OF THIS FOR FREE
IMAGINE …
imgflip.com
imgflip.com
This is a solved problem!
WAIT !
imgflip.com
YOU’RE RIGHT!
imgflip.com
BUT, …
OTel Language
Team
OTel auto-instrumentation / Agents today
dependency
uses
instruments
Library
Application
Library
Team
Icons by icons8.com
Instrumentation
Module
…
auto-instrumentation / agent
LET ME WRITE THE
LOGS FOR YOU
OTEL DEV
WHAT ?
LIBRARY DEV imgflip.com
uses
OTel auto-instrumentation / Agents today
dependency
instruments
Library
Application
…
Library
Team
OTel Language
Team
Icons by icons8.com
Instrumentation
Module
auto-instrumentation / agent
uses
OTel auto-instrumentation / Agents today
The ownership problem
dependency
instruments
Library
Application
Instrumentation
Module
auto-instrumentation / agent
…
Library
Team
OTel Language
Team
● misplaced expertise ● scalability
Icons by icons8.com
● maintenance ● technical complexity
… OBSERVABILITY WOULD BE TRULY UBIQUITOUS
IMAGINE …
imgflip.com
Photo by Işıl - pexels.com
UBIQUITOUS
OBSERVABILITY
→
BAKED-IN LIBRARY
INSTRUMENTATION
Baked-in Instrumentation
Aligned ownership
dependency
uses
Icons by icons8.com
Application
uses
Library
Baked-in
Instrumentation
Library
Team
OTel Language
Team
SDK / Agent …
auto-instrumentation / agent
API
Our story at Elastic with
Elasticsearch clients
Semantic Conventions
● Common names for different kinds of operations and data.
● Common naming scheme that can be standardized across a
codebase, libraries, and platforms.
● https://opentelemetry.io/docs/concepts/semantic-conventions/
Semantic Conventions
for ElasticSearch
https://opentelemetry.io/docs/specs/semconv/database/elasticsearch/
Demo
● Same setup with Elastic APM
Implementation in specific language clients
● Adding reference to the OpenTelemetry API
● Implement built-in tracing
Implementing built-in tracing
Challenges
● Think from the perspective of the users of the
library
○ Don't over-engineer
● Be careful about span names
○ Pay attention to low cardinality
● Instrumentation always comes with overhead
○ Only instrument relevant code-path
○ Avoid code paths that are being called
excessively
● Do not collect sensitive data (by default)
Implementing built-in tracing
List of PRs
● Semantic Conventions:
https://github.com/open-telemetry/semantic-conventions/
pull/23
● Java:
https://github.com/elastic/elasticsearch-java/pull/588
● .NET:
https://github.com/elastic/elasticsearch-net/pull/6990
● Ruby:
https://github.com/elastic/elasticsearch-ruby/pull/2179
Implementing built-in tracing
List of PRs
● RabbitMQ .NET:
https://github.com/rabbitmq/rabbitmq-dotnet-client/pull/1
261
Future of native instrumentation
● Realistically: there will always be a mix of native + external
instrumentation
● OTel registry: https://opentelemetry.io/ecosystem/registry/

More Related Content

Similar to OSMC 2023 | Built-in OpenTelemetry support in Elasticsearch clients by Greg Kalapos

Building intranet applications with ASP.NET AJAX and jQuery
Building intranet applications with ASP.NET AJAX and jQueryBuilding intranet applications with ASP.NET AJAX and jQuery
Building intranet applications with ASP.NET AJAX and jQuery
Alek Davis
 
Serverless Single Page Apps with React and Redux at ItCamp 2017
Serverless Single Page Apps with React and Redux at ItCamp 2017Serverless Single Page Apps with React and Redux at ItCamp 2017
Serverless Single Page Apps with React and Redux at ItCamp 2017
Melania Andrisan (Danciu)
 
OpenStack Workshop - WECode Harvard Conference
OpenStack Workshop - WECode Harvard ConferenceOpenStack Workshop - WECode Harvard Conference
OpenStack Workshop - WECode Harvard Conference
Iccha Sethi
 
Apache Spark™ + IBM Watson + Twitter DataPalooza SF 2015
Apache Spark™ + IBM Watson + Twitter DataPalooza SF 2015Apache Spark™ + IBM Watson + Twitter DataPalooza SF 2015
Apache Spark™ + IBM Watson + Twitter DataPalooza SF 2015
Mike Broberg
 
Scalable Open-Source IoT Solutions on Microsoft Azure
Scalable Open-Source IoT Solutions on Microsoft AzureScalable Open-Source IoT Solutions on Microsoft Azure
Scalable Open-Source IoT Solutions on Microsoft Azure
Maxim Ivannikov
 
Why apache Flink is the 4G of Big Data Analytics Frameworks
Why apache Flink is the 4G of Big Data Analytics FrameworksWhy apache Flink is the 4G of Big Data Analytics Frameworks
Why apache Flink is the 4G of Big Data Analytics Frameworks
Slim Baltagi
 
Introduction to interactive data visualisation using R Shiny
Introduction to interactive data visualisation using R ShinyIntroduction to interactive data visualisation using R Shiny
Introduction to interactive data visualisation using R Shiny
anamarisaguedes
 
Logging for Production Systems in The Container Era
Logging for Production Systems in The Container EraLogging for Production Systems in The Container Era
Logging for Production Systems in The Container Era
Sadayuki Furuhashi
 
System insight without Interference
System insight without InterferenceSystem insight without Interference
System insight without Interference
Tony Tam
 
Apache Flink Adoption at Shopify
Apache Flink Adoption at ShopifyApache Flink Adoption at Shopify
Apache Flink Adoption at Shopify
Yaroslav Tkachenko
 
LINQ 2 SQL Presentation To Palmchip And Trg, Technology Resource Group
LINQ 2 SQL Presentation To Palmchip  And Trg, Technology Resource GroupLINQ 2 SQL Presentation To Palmchip  And Trg, Technology Resource Group
LINQ 2 SQL Presentation To Palmchip And Trg, Technology Resource Group
Shahzad
 
Web_of_Things_2013
Web_of_Things_2013Web_of_Things_2013
Web_of_Things_2013
Max Kleiner
 
ThroughTheLookingGlass_EffectiveObservability.pptx
ThroughTheLookingGlass_EffectiveObservability.pptxThroughTheLookingGlass_EffectiveObservability.pptx
ThroughTheLookingGlass_EffectiveObservability.pptx
Grace Jansen
 
How and why we evolved a legacy Java web application to Scala... and we are s...
How and why we evolved a legacy Java web application to Scala... and we are s...How and why we evolved a legacy Java web application to Scala... and we are s...
How and why we evolved a legacy Java web application to Scala... and we are s...
Katia Aresti
 
Getting Started with Innoslate
Getting Started with InnoslateGetting Started with Innoslate
Getting Started with Innoslate
Elizabeth Steiner
 
Easing offline web application development with GWT
Easing offline web application development with GWTEasing offline web application development with GWT
Easing offline web application development with GWT
Arnaud Tournier
 
UniRx - Reactive Extensions for Unity(EN)
UniRx - Reactive Extensions for Unity(EN)UniRx - Reactive Extensions for Unity(EN)
UniRx - Reactive Extensions for Unity(EN)
Yoshifumi Kawai
 
Eclipse e4
Eclipse e4Eclipse e4
Eclipse e4
Chris Aniszczyk
 
Near real-time anomaly detection at Lyft
Near real-time anomaly detection at LyftNear real-time anomaly detection at Lyft
Near real-time anomaly detection at Lyft
markgrover
 
C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...
C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...
C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...
DataStax Academy
 

Similar to OSMC 2023 | Built-in OpenTelemetry support in Elasticsearch clients by Greg Kalapos (20)

Building intranet applications with ASP.NET AJAX and jQuery
Building intranet applications with ASP.NET AJAX and jQueryBuilding intranet applications with ASP.NET AJAX and jQuery
Building intranet applications with ASP.NET AJAX and jQuery
 
Serverless Single Page Apps with React and Redux at ItCamp 2017
Serverless Single Page Apps with React and Redux at ItCamp 2017Serverless Single Page Apps with React and Redux at ItCamp 2017
Serverless Single Page Apps with React and Redux at ItCamp 2017
 
OpenStack Workshop - WECode Harvard Conference
OpenStack Workshop - WECode Harvard ConferenceOpenStack Workshop - WECode Harvard Conference
OpenStack Workshop - WECode Harvard Conference
 
Apache Spark™ + IBM Watson + Twitter DataPalooza SF 2015
Apache Spark™ + IBM Watson + Twitter DataPalooza SF 2015Apache Spark™ + IBM Watson + Twitter DataPalooza SF 2015
Apache Spark™ + IBM Watson + Twitter DataPalooza SF 2015
 
Scalable Open-Source IoT Solutions on Microsoft Azure
Scalable Open-Source IoT Solutions on Microsoft AzureScalable Open-Source IoT Solutions on Microsoft Azure
Scalable Open-Source IoT Solutions on Microsoft Azure
 
Why apache Flink is the 4G of Big Data Analytics Frameworks
Why apache Flink is the 4G of Big Data Analytics FrameworksWhy apache Flink is the 4G of Big Data Analytics Frameworks
Why apache Flink is the 4G of Big Data Analytics Frameworks
 
Introduction to interactive data visualisation using R Shiny
Introduction to interactive data visualisation using R ShinyIntroduction to interactive data visualisation using R Shiny
Introduction to interactive data visualisation using R Shiny
 
Logging for Production Systems in The Container Era
Logging for Production Systems in The Container EraLogging for Production Systems in The Container Era
Logging for Production Systems in The Container Era
 
System insight without Interference
System insight without InterferenceSystem insight without Interference
System insight without Interference
 
Apache Flink Adoption at Shopify
Apache Flink Adoption at ShopifyApache Flink Adoption at Shopify
Apache Flink Adoption at Shopify
 
LINQ 2 SQL Presentation To Palmchip And Trg, Technology Resource Group
LINQ 2 SQL Presentation To Palmchip  And Trg, Technology Resource GroupLINQ 2 SQL Presentation To Palmchip  And Trg, Technology Resource Group
LINQ 2 SQL Presentation To Palmchip And Trg, Technology Resource Group
 
Web_of_Things_2013
Web_of_Things_2013Web_of_Things_2013
Web_of_Things_2013
 
ThroughTheLookingGlass_EffectiveObservability.pptx
ThroughTheLookingGlass_EffectiveObservability.pptxThroughTheLookingGlass_EffectiveObservability.pptx
ThroughTheLookingGlass_EffectiveObservability.pptx
 
How and why we evolved a legacy Java web application to Scala... and we are s...
How and why we evolved a legacy Java web application to Scala... and we are s...How and why we evolved a legacy Java web application to Scala... and we are s...
How and why we evolved a legacy Java web application to Scala... and we are s...
 
Getting Started with Innoslate
Getting Started with InnoslateGetting Started with Innoslate
Getting Started with Innoslate
 
Easing offline web application development with GWT
Easing offline web application development with GWTEasing offline web application development with GWT
Easing offline web application development with GWT
 
UniRx - Reactive Extensions for Unity(EN)
UniRx - Reactive Extensions for Unity(EN)UniRx - Reactive Extensions for Unity(EN)
UniRx - Reactive Extensions for Unity(EN)
 
Eclipse e4
Eclipse e4Eclipse e4
Eclipse e4
 
Near real-time anomaly detection at Lyft
Near real-time anomaly detection at LyftNear real-time anomaly detection at Lyft
Near real-time anomaly detection at Lyft
 
C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...
C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...
C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...
 

Recently uploaded

Gamify it until you make it Improving Agile Development and Operations with ...
Gamify it until you make it  Improving Agile Development and Operations with ...Gamify it until you make it  Improving Agile Development and Operations with ...
Gamify it until you make it Improving Agile Development and Operations with ...
Ben Linders
 
AWS User Group Torino 2024 #3 - 18/06/2024
AWS User Group Torino 2024 #3 - 18/06/2024AWS User Group Torino 2024 #3 - 18/06/2024
AWS User Group Torino 2024 #3 - 18/06/2024
Guido Maria Nebiolo
 
The Intersection between Competition and Data Privacy – OECD – June 2024 OECD...
The Intersection between Competition and Data Privacy – OECD – June 2024 OECD...The Intersection between Competition and Data Privacy – OECD – June 2024 OECD...
The Intersection between Competition and Data Privacy – OECD – June 2024 OECD...
OECD Directorate for Financial and Enterprise Affairs
 
Genesis chapter 3 Isaiah Scudder.pptx
Genesis    chapter 3 Isaiah Scudder.pptxGenesis    chapter 3 Isaiah Scudder.pptx
Genesis chapter 3 Isaiah Scudder.pptx
FamilyWorshipCenterD
 
2 December UAE National Day - United Arab Emirates
2 December UAE National Day - United Arab Emirates2 December UAE National Day - United Arab Emirates
2 December UAE National Day - United Arab Emirates
UAE Ppt
 
Proposal: The Ark Project and The BEEP Inc
Proposal: The Ark Project and The BEEP IncProposal: The Ark Project and The BEEP Inc
Proposal: The Ark Project and The BEEP Inc
Raheem Muhammad
 
Using-Presentation-Software-to-the-Fullf.pptx
Using-Presentation-Software-to-the-Fullf.pptxUsing-Presentation-Software-to-the-Fullf.pptx
Using-Presentation-Software-to-the-Fullf.pptx
kainatfatyma9
 
2023 Ukraine Crisis Media Center Financial Report
2023 Ukraine Crisis Media Center Financial Report2023 Ukraine Crisis Media Center Financial Report
2023 Ukraine Crisis Media Center Financial Report
UkraineCrisisMediaCenter
 
SASi-SPi Science Policy Lab Pre-engagement
SASi-SPi Science Policy Lab Pre-engagementSASi-SPi Science Policy Lab Pre-engagement
SASi-SPi Science Policy Lab Pre-engagement
Francois Stepman
 
ServiceNow CIS-ITSM Exam Dumps & Questions [2024]
ServiceNow CIS-ITSM Exam Dumps & Questions [2024]ServiceNow CIS-ITSM Exam Dumps & Questions [2024]
ServiceNow CIS-ITSM Exam Dumps & Questions [2024]
SkillCertProExams
 
2023 Ukraine Crisis Media Center Finance Balance
2023 Ukraine Crisis Media Center Finance Balance2023 Ukraine Crisis Media Center Finance Balance
2023 Ukraine Crisis Media Center Finance Balance
UkraineCrisisMediaCenter
 
Legislation And Regulations For Import, Manufacture,.pptx
Legislation And Regulations For Import, Manufacture,.pptxLegislation And Regulations For Import, Manufacture,.pptx
Legislation And Regulations For Import, Manufacture,.pptx
Charmi13
 
Bridging the visual gap between cultural heritage and digital scholarship
Bridging the visual gap between cultural heritage and digital scholarshipBridging the visual gap between cultural heritage and digital scholarship
Bridging the visual gap between cultural heritage and digital scholarship
Inesm9
 
Prsentation for VIVA Welike project 1semester.pptx
Prsentation for VIVA Welike project 1semester.pptxPrsentation for VIVA Welike project 1semester.pptx
Prsentation for VIVA Welike project 1semester.pptx
prafulpawar29
 
IEEE CIS Webinar Sustainable futures.pdf
IEEE CIS Webinar Sustainable futures.pdfIEEE CIS Webinar Sustainable futures.pdf
IEEE CIS Webinar Sustainable futures.pdf
Claudio Gallicchio
 
The Intersection between Competition and Data Privacy – KEMP – June 2024 OECD...
The Intersection between Competition and Data Privacy – KEMP – June 2024 OECD...The Intersection between Competition and Data Privacy – KEMP – June 2024 OECD...
The Intersection between Competition and Data Privacy – KEMP – June 2024 OECD...
OECD Directorate for Financial and Enterprise Affairs
 
The Intersection between Competition and Data Privacy – COLANGELO – June 2024...
The Intersection between Competition and Data Privacy – COLANGELO – June 2024...The Intersection between Competition and Data Privacy – COLANGELO – June 2024...
The Intersection between Competition and Data Privacy – COLANGELO – June 2024...
OECD Directorate for Financial and Enterprise Affairs
 
怎么办理(lincoln学位证书)英国林肯大学毕业证文凭学位证书原版一模一样
怎么办理(lincoln学位证书)英国林肯大学毕业证文凭学位证书原版一模一样怎么办理(lincoln学位证书)英国林肯大学毕业证文凭学位证书原版一模一样
怎么办理(lincoln学位证书)英国林肯大学毕业证文凭学位证书原版一模一样
kekzed
 
ACTIVE IMPLANTABLE MEDICAL DEVICE IN EUROPE
ACTIVE IMPLANTABLE MEDICAL DEVICE IN EUROPEACTIVE IMPLANTABLE MEDICAL DEVICE IN EUROPE
ACTIVE IMPLANTABLE MEDICAL DEVICE IN EUROPE
Charmi13
 
Data Processing in PHP - PHPers 2024 Poznań
Data Processing in PHP - PHPers 2024 PoznańData Processing in PHP - PHPers 2024 Poznań
Data Processing in PHP - PHPers 2024 Poznań
Norbert Orzechowicz
 

Recently uploaded (20)

Gamify it until you make it Improving Agile Development and Operations with ...
Gamify it until you make it  Improving Agile Development and Operations with ...Gamify it until you make it  Improving Agile Development and Operations with ...
Gamify it until you make it Improving Agile Development and Operations with ...
 
AWS User Group Torino 2024 #3 - 18/06/2024
AWS User Group Torino 2024 #3 - 18/06/2024AWS User Group Torino 2024 #3 - 18/06/2024
AWS User Group Torino 2024 #3 - 18/06/2024
 
The Intersection between Competition and Data Privacy – OECD – June 2024 OECD...
The Intersection between Competition and Data Privacy – OECD – June 2024 OECD...The Intersection between Competition and Data Privacy – OECD – June 2024 OECD...
The Intersection between Competition and Data Privacy – OECD – June 2024 OECD...
 
Genesis chapter 3 Isaiah Scudder.pptx
Genesis    chapter 3 Isaiah Scudder.pptxGenesis    chapter 3 Isaiah Scudder.pptx
Genesis chapter 3 Isaiah Scudder.pptx
 
2 December UAE National Day - United Arab Emirates
2 December UAE National Day - United Arab Emirates2 December UAE National Day - United Arab Emirates
2 December UAE National Day - United Arab Emirates
 
Proposal: The Ark Project and The BEEP Inc
Proposal: The Ark Project and The BEEP IncProposal: The Ark Project and The BEEP Inc
Proposal: The Ark Project and The BEEP Inc
 
Using-Presentation-Software-to-the-Fullf.pptx
Using-Presentation-Software-to-the-Fullf.pptxUsing-Presentation-Software-to-the-Fullf.pptx
Using-Presentation-Software-to-the-Fullf.pptx
 
2023 Ukraine Crisis Media Center Financial Report
2023 Ukraine Crisis Media Center Financial Report2023 Ukraine Crisis Media Center Financial Report
2023 Ukraine Crisis Media Center Financial Report
 
SASi-SPi Science Policy Lab Pre-engagement
SASi-SPi Science Policy Lab Pre-engagementSASi-SPi Science Policy Lab Pre-engagement
SASi-SPi Science Policy Lab Pre-engagement
 
ServiceNow CIS-ITSM Exam Dumps & Questions [2024]
ServiceNow CIS-ITSM Exam Dumps & Questions [2024]ServiceNow CIS-ITSM Exam Dumps & Questions [2024]
ServiceNow CIS-ITSM Exam Dumps & Questions [2024]
 
2023 Ukraine Crisis Media Center Finance Balance
2023 Ukraine Crisis Media Center Finance Balance2023 Ukraine Crisis Media Center Finance Balance
2023 Ukraine Crisis Media Center Finance Balance
 
Legislation And Regulations For Import, Manufacture,.pptx
Legislation And Regulations For Import, Manufacture,.pptxLegislation And Regulations For Import, Manufacture,.pptx
Legislation And Regulations For Import, Manufacture,.pptx
 
Bridging the visual gap between cultural heritage and digital scholarship
Bridging the visual gap between cultural heritage and digital scholarshipBridging the visual gap between cultural heritage and digital scholarship
Bridging the visual gap between cultural heritage and digital scholarship
 
Prsentation for VIVA Welike project 1semester.pptx
Prsentation for VIVA Welike project 1semester.pptxPrsentation for VIVA Welike project 1semester.pptx
Prsentation for VIVA Welike project 1semester.pptx
 
IEEE CIS Webinar Sustainable futures.pdf
IEEE CIS Webinar Sustainable futures.pdfIEEE CIS Webinar Sustainable futures.pdf
IEEE CIS Webinar Sustainable futures.pdf
 
The Intersection between Competition and Data Privacy – KEMP – June 2024 OECD...
The Intersection between Competition and Data Privacy – KEMP – June 2024 OECD...The Intersection between Competition and Data Privacy – KEMP – June 2024 OECD...
The Intersection between Competition and Data Privacy – KEMP – June 2024 OECD...
 
The Intersection between Competition and Data Privacy – COLANGELO – June 2024...
The Intersection between Competition and Data Privacy – COLANGELO – June 2024...The Intersection between Competition and Data Privacy – COLANGELO – June 2024...
The Intersection between Competition and Data Privacy – COLANGELO – June 2024...
 
怎么办理(lincoln学位证书)英国林肯大学毕业证文凭学位证书原版一模一样
怎么办理(lincoln学位证书)英国林肯大学毕业证文凭学位证书原版一模一样怎么办理(lincoln学位证书)英国林肯大学毕业证文凭学位证书原版一模一样
怎么办理(lincoln学位证书)英国林肯大学毕业证文凭学位证书原版一模一样
 
ACTIVE IMPLANTABLE MEDICAL DEVICE IN EUROPE
ACTIVE IMPLANTABLE MEDICAL DEVICE IN EUROPEACTIVE IMPLANTABLE MEDICAL DEVICE IN EUROPE
ACTIVE IMPLANTABLE MEDICAL DEVICE IN EUROPE
 
Data Processing in PHP - PHPers 2024 Poznań
Data Processing in PHP - PHPers 2024 PoznańData Processing in PHP - PHPers 2024 Poznań
Data Processing in PHP - PHPers 2024 Poznań
 

OSMC 2023 | Built-in OpenTelemetry support in Elasticsearch clients by Greg Kalapos

  • 1. Built-in OpenTelemetry support in Elasticsearch clients
  • 2. Greg Kalapos ● Works at Elastic since 2018 ● Created the Elastic .NET APM Agent (completely OpenSource) ● Now focuses on OpenTelemetry https://twitter.com/gregkalapos
  • 3. Demo Out of the box, built-in telemetry in Elasticsearch clients based on OpenTelemetry
  • 4. What is OpenTelemetry? ● CNCF Project ● Collection of APIs, SDKs, and tools. ● To instrument, generate, collect, and export telemetry data ● Metrics, logs, and traces
  • 5. Signals ● Traces ● Metrics ● Logs ● (Profiling) API SDK Agent / Auto Inst. SDK Context Propagation Semantic Conventions Collector OTLP … Observability Backends
  • 7. UBIQUITOUS OBSERVABILITY . . . imgflip.com Wos isn des?!
  • 8. Anatomy of a modern application public class HomeController { public async Task<IActionResult> Index() { return View(); } }
  • 9. Anatomy of a modern application public class HomeController : Controller { public async Task<IActionResult> Index() { return View(); } } Frameworks and libraries
  • 10. public class HomeController : Controller { ElasticsearchClient _client = new(CloudId, new ApiKey(ApiKey)); public async Task<IActionResult> Index() { var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index")); var doc = response.Source; return View(doc); } } Anatomy of a modern application Frameworks and libraries
  • 11. public class HomeController : Controller { ElasticsearchClient _client = new(CloudId, new ApiKey(ApiKey)); IProducer<string, string> producer = new ProducerBuilder<string, string>(config).Build(); public async Task<IActionResult> Index() { var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index")); var deliveryResult = await producer.ProduceAsync(topic, message); var doc = response.Source; return View(doc); } } Anatomy of a modern application Frameworks and libraries
  • 12. Observability Things we want to see HTTP GET - Index 723 ms Elasticsearch - Get - my_index 360 ms Kafka - Send 280 ms Traces Logs / Events [2023-11-06T15:23:40:152][INFO] HomeController - GET - /index - 200 [2023-11-06T15:23:40:167][INFO] ES Client - GET - my_index - 1 [2023-11-06T15:23:40:167][DEBUG] ES Client - Selected node MyNode-2 [2023-11-06T15:23:40:167][INFO] Kafka - Sending record to topic XYZ [2023-11-06T15:23:40:167][WARN] Kafka - Failed sending, retrying … Metrics
  • 13. How do we get the data? Instrumentation! public async Task<IActionResult> Index() { var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index")); var deliveryResult = await producer.ProduceAsync(topic, message); var doc = response.Source; return View(doc); }
  • 14. How do we get the data? Instrumentation! public async Task<IActionResult> Index() { using (var httpSpan = tracer.StartActiveSpan("HTTP GET Index")) { var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index")); var deliveryResult = await producer.ProduceAsync(topic, message); var doc = response.Source; return View(doc); } } HTTP GET - Index 723 ms Traces
  • 15. How do we get the data? Instrumentation! public async Task<IActionResult> Index() { using (var httpSpan = tracer.StartActiveSpan("HTTP GET Index")) { using (var esSpan = tracer.StartActiveSpan("ElasticSearch Get my_index")) { var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index")); } var deliveryResult = await producer.ProduceAsync(topic, message); var doc = response.Source; return View(doc); } } HTTP GET - Index 723 ms Elasticsearch - Get - my_index 360 ms Traces
  • 16. How do we get the data? Instrumentation! public async Task<IActionResult> Index() { using (var httpSpan = tracer.StartActiveSpan("HTTP GET Index")) { using (var esSpan = tracer.StartActiveSpan("ElasticSearch Get my_index")) { var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index")); } using (var kafkaSpan = tracer.StartActiveSpan("Kafka Send")) { var deliveryResult = await producer.ProduceAsync(topic, message); } var doc = response.Source; return View(doc); } } HTTP GET - /index 723 ms Elasticsearch - Get - my_index 360 ms Kafka - Send 280 ms Traces
  • 17. Photo by Andrea Piacquadio - pexels.com LET’S DO IT FOR ALL THE DEPENDENCIES …
  • 18. … YOU WOULD GET ALL OF THIS FOR FREE IMAGINE … imgflip.com
  • 19. imgflip.com This is a solved problem! WAIT !
  • 22. OTel Language Team OTel auto-instrumentation / Agents today dependency uses instruments Library Application Library Team Icons by icons8.com Instrumentation Module … auto-instrumentation / agent
  • 23. LET ME WRITE THE LOGS FOR YOU OTEL DEV WHAT ? LIBRARY DEV imgflip.com
  • 24. uses OTel auto-instrumentation / Agents today dependency instruments Library Application … Library Team OTel Language Team Icons by icons8.com Instrumentation Module auto-instrumentation / agent
  • 25. uses OTel auto-instrumentation / Agents today The ownership problem dependency instruments Library Application Instrumentation Module auto-instrumentation / agent … Library Team OTel Language Team ● misplaced expertise ● scalability Icons by icons8.com ● maintenance ● technical complexity
  • 26. … OBSERVABILITY WOULD BE TRULY UBIQUITOUS IMAGINE … imgflip.com
  • 27. Photo by Işıl - pexels.com UBIQUITOUS OBSERVABILITY → BAKED-IN LIBRARY INSTRUMENTATION
  • 28. Baked-in Instrumentation Aligned ownership dependency uses Icons by icons8.com Application uses Library Baked-in Instrumentation Library Team OTel Language Team SDK / Agent … auto-instrumentation / agent API
  • 29. Our story at Elastic with Elasticsearch clients
  • 30. Semantic Conventions ● Common names for different kinds of operations and data. ● Common naming scheme that can be standardized across a codebase, libraries, and platforms. ● https://opentelemetry.io/docs/concepts/semantic-conventions/
  • 32. Demo ● Same setup with Elastic APM
  • 33. Implementation in specific language clients ● Adding reference to the OpenTelemetry API ● Implement built-in tracing
  • 34. Implementing built-in tracing Challenges ● Think from the perspective of the users of the library ○ Don't over-engineer ● Be careful about span names ○ Pay attention to low cardinality ● Instrumentation always comes with overhead ○ Only instrument relevant code-path ○ Avoid code paths that are being called excessively ● Do not collect sensitive data (by default)
  • 35. Implementing built-in tracing List of PRs ● Semantic Conventions: https://github.com/open-telemetry/semantic-conventions/ pull/23 ● Java: https://github.com/elastic/elasticsearch-java/pull/588 ● .NET: https://github.com/elastic/elasticsearch-net/pull/6990 ● Ruby: https://github.com/elastic/elasticsearch-ruby/pull/2179
  • 36. Implementing built-in tracing List of PRs ● RabbitMQ .NET: https://github.com/rabbitmq/rabbitmq-dotnet-client/pull/1 261
  • 37. Future of native instrumentation ● Realistically: there will always be a mix of native + external instrumentation ● OTel registry: https://opentelemetry.io/ecosystem/registry/