GraphQL is convenient for developers because it allows them to ask for exactly what they want and get the data they need without hassle. But running GraphQL in production can be incredibly challenging. Pinpointing the exact source of a GraphQL error is like searching for a needle in a haystack. It requires sifting through layers of data fetching logic, resolver functions and upstream services, hoping to stumble upon the culprit. But OpenTelemetry can help... Recording: https://youtu.be/IkEUJjRBCbo?feature=shared

1. What could go wrong
with a GraphQL query and
can OpenTelemetry help?
Sonja Chevre & Ahmet Soormally

3. How is GraphQL different from your typical REST API?
What does a query & schema look like?
What are the general use-cases for GraphQL?
GraphQL 101
DECLARATIVE QUERY LANGUAGE FOR THE INTERNET
What kinds of problems does GraphQL solve?
1
2
3
4

4. SELECT
authors.name,
posts.title
FROM
posts
LEFT JOIN authors
ON authors.id = posts.author_id
WHERE posts.status = ‘published’;
SQL
GET /posts?status=published
GET /authors/2
GET /authors/7
GET /authors/9
…
REST
query {
postsByStatus(status: "published") {
title
author {
name
}
}
}
GRAPHQL
GraphQL 101

5. SELECT
authors.name,
posts.title
FROM
posts
LEFT JOIN authors
ON authors.id = posts.author_id
WHERE posts.status = ‘published’;
SQL
GET /posts?status=published
GET /authors/2
GET /authors/7
GET /authors/9
REST
query {
postsByStatus(status: "published") {
title
author {
name
}
}
}
GRAPHQL
Overfetching
GRAPHQL 101

6. SELECT
authors.name,
posts.title
FROM
posts
LEFT JOIN authors
ON authors.id = posts.author_id
WHERE posts.status = ‘published’;
SQL
GET /posts?status=published
GET /authors/2
GET /authors/7
GET /authors/9
…
REST
query {
postsByStatus(status: "published") {
title
author {
name
}
}
}
}
GRAPHQL
Underfetching
GRAPHQL 101

7. type Query {
getPostsByStatus(status: String!): [Post]
}
type User {
id: ID!
name: String!
email: String!
posts: [Post]
comments: [Comment]
}
type Post {
title: String
content: String
status: String
author: User
comments: [Comment]
}
type Comment {
id: ID!
author: User
comment: String
}
Schema
query {
getPostsByStatus(status: "published") {
title
author {
name
}
}
}
Operation
{
“data”: [
{
“title”: “abc”,
“author”: {
“name”: “Ahmet”
}
},
{
“title”: “def”,
“author”: {
“name”: “Sonja”
}
},
]
}
Response
GraphQL 101

8. Kube travel - Server side
type Photo {
id: ID!
country: ID
src: String
title: String
}
type Country {
name: String
code: ID!
capital: String
currency: String
photos: [Photo]
weather: [Weather]
}
type Weather {
description: String
temperature: Int
}
type Query {
getCountries: [Country]
getCountry(code: ID!): Country
}
Countries service
GraphQL (SAAS)
Image service
REST (SAAS)
Weather service
REST
Node.js (express)

9. Kube travel app
React app
type Photo {
id: ID!
country: ID
src: String
title: String
}
type Country {
name: String
code: ID!
capital: String
currency: String
photos: [Photo]
weather: [Weather]
}
type Weather {
description: String
temperature: Int
}
type Query {
getCountries: [Country]
getCountry(code: ID!): Country
}
Countries service
GraphQL (SAAS)
Image service
REST (SAAS)
Weather service
REST
Node.js (express)

10. type Photo {
id: ID!
country: ID
src: String
title: String
}
type Country {
name: String
code: ID!
capital: String
currency: String
photos: [Photo]
weather: [Weather]
}
type Weather {
description: String
temperature: Int
}
type Query {
getCountries: [Country]
getCountry(code: ID!): Country
}
Countries service
GraphQL (SAAS)
Image service
REST (SAAS)
Kube travel API product
Weather service
REST
Weather app
Booking app
Gallery app
…
Node.js (express)
GraphQL
capable API
gateway

11. How do we monitor GraphQL in
production?
Let’s apply the RED method to GraphQL
Duration - distributions of the amount of time each request takes.
Errors - the number of failed requests per second.
Rate - the number of requests, per second, your services are serving.
R
E
D

12. Collector
Node.js (express)
Countries service
GraphQL (SAAS)
Image service
REST (SAAS)
Adding OpenTelemetry to Kube Travel
Weather service
REST
GraphQL capable
API gateway
Spans

13. Instrumentation libraries
available for GraphQL
Instrumenting GraphQL with
OpenTelemetry

14. GraphQL instrumentation

15. An end-to-end
distributed trace in
Jaeger

16. RED metrics from traces with Jaeger
Collector
Node.js (express)
Spans
Metrics
Traces
Span Metrics Connector creates two metrics based on spans:
Calls_total
Counts the total number of spans, including error spans. Call counts are
differentiated from errors via the status_code label. Errors are identified as
any time series with the label status_code = "STATUS_CODE_ERROR".
Latency
latency_count: The total number of data points across all buckets in the
histogram.
latency_sum: The sum of all data point values.
latency_bucket: A collection of n time series (where n is the number of
latency buckets) for each latency bucket identified by an le (less than or equal
to) label. The latency_bucket counter with lowest le and le >= span
latency will be incremented for each span.
Span
metrics
connector

18. Resolver Errors
Errors
Upstream errors
1
2

20. Something is going on…
25% error rate!

21. Causing GraphQL to return a
500 http status code
Weather service returning a
400 http status code

22. The GraphQL query that is
causing the error

24. Everything looks fine

25. All good!

26. GraphQL errors
The request returns HTTP
status code 200 even though
the response contains an error
Here is the error!

27. Adding GraphQL to error spans
Is there a semantic
convention we can use?

28. Report the error to
the active span
with manual
instrumentation

29. The error is now
recorded in the span
Including error details

30. Still no error
But we can use
Prometheus to report error
rates for GraphQL queries

31. How can we detect performance issues?
Is tracking the latency
for a the single graphql
endpoint good enough?
PERFORMANCE

33. How can we detect performance issues?
Is tracking the latency
for a the single graphql
endpoint good enough?
PERFORMANCE

34. Query depth
Query complexity
Cyclic queries
Performance issues
N+1
1
2
3
4

35. N+1 issue
PERFORMANCE
n
1

36. 27 HTTP GET calls to the
weather service for this
GraphQL query!

37. Detecting N+1 queries
PERFORMANCE
On average, 1 GraphQL query
makes 24 outgoing requests

38. Cyclic queries
PERFORMANCE

39. Expensive Queries
PERFORMANCE
{
a {
b {
c
}
}
}
Depth 3
{
a b c {
d e {
f
}
}
}
Complexity 6

40. What information do we have on our spans?
Full query (doesn’t
match the semantic
conventions)
Field name (doesn’t
match the semantic
conventions either)
PERFORMANCE

41. OpenTelemetry collector
configuration to export
additional span metrics

42. Let try to group them by query
PERFORMANCE

43. Using operation type and name
but not available in our
instrumentation library
but not available in our
instrumentation library

44. Using operation type and name
query continents 300 ms
query continents 2400 ms
mutation updateCountry 500 ms

45. Adding a client identification
query continents mobile 600 ms
query continents react 3500 ms
query continents weather-app 550 ms

46. RED metrics approach of monitoring needs to be extended to report GraphQL specifics
(query type, operation name, query depth, query complexity)
GraphQL errors are missing from semantic conventions and from instrumentation
needed to detect for errors
and for error based sampling
What have we learnt?
Semantic conventions are not always respected by instrumentation libraries
4
3
2
OpenTelemetry is helpful for monitoring and troubleshooting GraphQL queries, BUT:
1

48. Thank you!
Come talk to us to continue the discussion
or reach out:
Sonja Chevre
@SonjaChevre
linkedin.com/in/sonjachevre
Ahmet Soormally
@SoormallyAhmet
linkedin.com/in/ahmet-soormally