At Elastic, we recently added OpenTelemetry support to most of our OpenSource Elasticsearch clients. This talk will tell the story on how we got there and what we learned along the way. Elasticsearch clients exist in multiple languages (Java, .NET, PHP, Ruby, etc.), therefore we also created Semantic Conventions to make sure all Elasticsearch client instrumentations behave in the same way. Attendees will learn about how to instrument existing libraries with OpenTelemetry and they will also learn how to interact with the community and collaborate on creating Semantic Conventions for specific technologies.

1. Built-in OpenTelemetry support
in Elasticsearch clients

2. Greg Kalapos
● Works at Elastic since 2018
● Created the Elastic .NET APM Agent
(completely OpenSource)
● Now focuses on OpenTelemetry
https://twitter.com/gregkalapos

3. Demo
Out of the box, built-in telemetry in
Elasticsearch clients based on
OpenTelemetry

4. What is OpenTelemetry?
● CNCF Project
● Collection of APIs, SDKs, and tools.
● To instrument, generate, collect, and
export telemetry data
● Metrics, logs, and traces

5. Signals
● Traces
● Metrics
● Logs
● (Profiling)
API
SDK
Agent / Auto Inst.
SDK
Context
Propagation
Semantic
Conventions
Collector
OTLP
…
Observability Backends

6. https://opentelemetry.io website:

7. UBIQUITOUS OBSERVABILITY . . .
imgflip.com
Wos isn des?!

8. Anatomy of a modern application
public class HomeController
{
public async Task<IActionResult> Index()
{
return View();
}
}

9. Anatomy of a modern application
public class HomeController : Controller
{
public async Task<IActionResult> Index()
{
return View();
}
}
Frameworks and libraries

10. public class HomeController : Controller
{
ElasticsearchClient _client = new(CloudId, new ApiKey(ApiKey));
public async Task<IActionResult> Index()
{
var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index"));
var doc = response.Source;
return View(doc);
}
}
Anatomy of a modern application
Frameworks and libraries

11. public class HomeController : Controller
{
ElasticsearchClient _client = new(CloudId, new ApiKey(ApiKey));
IProducer<string, string> producer = new ProducerBuilder<string, string>(config).Build();
public async Task<IActionResult> Index()
{
var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index"));
var deliveryResult = await producer.ProduceAsync(topic, message);
var doc = response.Source;
return View(doc);
}
}
Anatomy of a modern application
Frameworks and libraries

12. Observability
Things we want to see
HTTP GET - Index 723 ms
Elasticsearch - Get - my_index 360 ms
Kafka - Send 280 ms
Traces
Logs /
Events
[2023-11-06T15:23:40:152][INFO] HomeController - GET - /index - 200
[2023-11-06T15:23:40:167][INFO] ES Client - GET - my_index - 1
[2023-11-06T15:23:40:167][DEBUG] ES Client - Selected node MyNode-2
[2023-11-06T15:23:40:167][INFO] Kafka - Sending record to topic XYZ
[2023-11-06T15:23:40:167][WARN] Kafka - Failed sending, retrying …
Metrics

13. How do we get the data?
Instrumentation!
public async Task<IActionResult> Index()
{
var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index"));
var deliveryResult = await producer.ProduceAsync(topic, message);
var doc = response.Source;
return View(doc);
}

14. How do we get the data?
Instrumentation!
public async Task<IActionResult> Index()
{
using (var httpSpan = tracer.StartActiveSpan("HTTP GET Index"))
{
var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index"));
var deliveryResult = await producer.ProduceAsync(topic, message);
var doc = response.Source;
return View(doc);
}
}
HTTP GET - Index 723 ms
Traces

15. How do we get the data?
Instrumentation!
public async Task<IActionResult> Index()
{
using (var httpSpan = tracer.StartActiveSpan("HTTP GET Index"))
{
using (var esSpan = tracer.StartActiveSpan("ElasticSearch Get my_index"))
{
var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index"));
}
var deliveryResult = await producer.ProduceAsync(topic, message);
var doc = response.Source;
return View(doc);
}
}
HTTP GET - Index 723 ms
Elasticsearch - Get - my_index 360 ms
Traces

16. How do we get the data?
Instrumentation!
public async Task<IActionResult> Index()
{
using (var httpSpan = tracer.StartActiveSpan("HTTP GET Index"))
{
using (var esSpan = tracer.StartActiveSpan("ElasticSearch Get my_index"))
{
var response = await _client.GetAsync<MyDoc>(1, idx => idx.Index("my_index"));
}
using (var kafkaSpan = tracer.StartActiveSpan("Kafka Send"))
{
var deliveryResult = await producer.ProduceAsync(topic, message);
}
var doc = response.Source;
return View(doc);
}
}
HTTP GET - /index 723 ms
Elasticsearch - Get - my_index 360 ms
Kafka - Send 280 ms
Traces

17. Photo by Andrea Piacquadio - pexels.com
LET’S DO IT FOR ALL THE
DEPENDENCIES …

18. … YOU WOULD GET ALL OF THIS FOR FREE
IMAGINE …
imgflip.com

19. imgflip.com
This is a solved problem!
WAIT !

20. imgflip.com
YOU’RE RIGHT!

21. imgflip.com
BUT, …

22. OTel Language
Team
OTel auto-instrumentation / Agents today
dependency
uses
instruments
Library
Application
Library
Team
Icons by icons8.com
Instrumentation
Module
…
auto-instrumentation / agent

23. LET ME WRITE THE
LOGS FOR YOU
OTEL DEV
WHAT ?
LIBRARY DEV imgflip.com

24. uses
OTel auto-instrumentation / Agents today
dependency
instruments
Library
Application
…
Library
Team
OTel Language
Team
Icons by icons8.com
Instrumentation
Module
auto-instrumentation / agent

25. uses
OTel auto-instrumentation / Agents today
The ownership problem
dependency
instruments
Library
Application
Instrumentation
Module
auto-instrumentation / agent
…
Library
Team
OTel Language
Team
● misplaced expertise ● scalability
Icons by icons8.com
● maintenance ● technical complexity

26. … OBSERVABILITY WOULD BE TRULY UBIQUITOUS
IMAGINE …
imgflip.com

27. Photo by Işıl - pexels.com
UBIQUITOUS
OBSERVABILITY
→
BAKED-IN LIBRARY
INSTRUMENTATION

28. Baked-in Instrumentation
Aligned ownership
dependency
uses
Icons by icons8.com
Application
uses
Library
Baked-in
Instrumentation
Library
Team
OTel Language
Team
SDK / Agent …
auto-instrumentation / agent
API

29. Our story at Elastic with
Elasticsearch clients

30. Semantic Conventions
● Common names for different kinds of operations and data.
● Common naming scheme that can be standardized across a
codebase, libraries, and platforms.
● https://opentelemetry.io/docs/concepts/semantic-conventions/

31. Semantic Conventions
for ElasticSearch
https://opentelemetry.io/docs/specs/semconv/database/elasticsearch/

32. Demo
● Same setup with Elastic APM

33. Implementation in specific language clients
● Adding reference to the OpenTelemetry API
● Implement built-in tracing

34. Implementing built-in tracing
Challenges
● Think from the perspective of the users of the
library
○ Don't over-engineer
● Be careful about span names
○ Pay attention to low cardinality
● Instrumentation always comes with overhead
○ Only instrument relevant code-path
○ Avoid code paths that are being called
excessively
● Do not collect sensitive data (by default)

35. Implementing built-in tracing
List of PRs
● Semantic Conventions:
https://github.com/open-telemetry/semantic-conventions/
pull/23
● Java:
https://github.com/elastic/elasticsearch-java/pull/588
● .NET:
https://github.com/elastic/elasticsearch-net/pull/6990
● Ruby:
https://github.com/elastic/elasticsearch-ruby/pull/2179

36. Implementing built-in tracing
List of PRs
● RabbitMQ .NET:
https://github.com/rabbitmq/rabbitmq-dotnet-client/pull/1
261

37. Future of native instrumentation
● Realistically: there will always be a mix of native + external
instrumentation
● OTel registry: https://opentelemetry.io/ecosystem/registry/