How we automatically build and maintain a search index using GraphQL. Presented at GraphQL Summit 2019.

1. GraphQL
Search
Artem Shtatnov
GraphQL Summit 2019

2. Escape Room Game?

3. How can we quickly search and aggregate
data spread across multiple sources?

4. Roadmap
Indexing
the Data
Keeping It
Updated
Data Flow Learnings

5. Marketing Tech at Netflix
Creatives

6. Our Data
query getAllCreatives() {
creative {
id
...AllCreativeFields
show {
...AllShowFields
titleRanking {
...AllTitleRankingFields
}
}
comments {
...AllCommentFields
}
}
}
Creative Service
Show Service
Title Ranking Algo
Comment Service

7. 50 Million

8. Roadmap
Indexing
the Data
Keeping It
Updated
Data Flow Learnings

9. Indexing Data
● Iterate over all creatives
● Call our GraphQL server
with a query
● Put the result into
Elasticsearch
query getCreativesById($creativeIds: [UUID]) {
creative(creativeIds: $creativeIds) {
id
...AllCreativeFields
show {
...AllShowFields
titleRanking {
...AllTitleRankingFields
}
}
comments {
...AllCommentFields
}
}
}

10. After Our Data is Indexed
● Sort, group, and filter the data on arbitrary fields
● Provide type-ahead suggestions
● Implement faceted search
● Display results with infinite scroll
● Our page loads much faster

11. Roadmap
Indexing
the Data
Keeping It
Updated
Data Flow Learnings

12. Keeping the Index Up To Date
● If the creative changes, then reindex that creative
● if the show changes, then reindex the creative with that
show id
● If the title ranking changes, then find the related show, use
the show to find the creative, and reindex it
● if a comment changes, then reindex the creative with that
comment id
If our data changes, we need to reindex

13. GraphQL Defines Relationships in Our Data
Title Ranking
Show
Creative
Comments

14. Auto Search Indexer
Title Ranking
Show
Creative
Comments
1. Change event

15. Auto Search Indexer
1. Change event
2. Fan out one level
Title Ranking
Show
Creative
Comments

16. Auto Search Indexer
1. Change event
2. Fan out one level
3. Search all indexed
documents
Title Ranking
Show
Creative
Comments
Elasticsearch

17. Auto Search Indexer
1. Change event
2. Fan out one level
3. Search all indexed
documents
4. Rebuild and reindex
documents
Title Ranking
Show
Creative
Comments

18. Auto Search Indexer
1. Change event
2. Fan out one level
3. Search all indexed
documents
4. Rebuild and reindex
documents
rebuild
search
fan out
change

19. Roadmap
Indexing
the Data
Keeping It
Updated
Data Flow Learnings

20. Data Flow: Before Indexing
BrowserGraphQL
aggregation on page load (read)
Creative
Service
Show Service
Title Ranking
Algo
Commenting
Service

21. Data Flow: After Indexing
BrowserGraphQL
Creative
Service
Show Service
Title Ranking
Algo
Commenting
Service
Elasticsearch
aggregation on data change (write)

22. Data Flow: Abstract Elasticsearch
BrowserGraphQL
Creative
Service
Show Service
Title Ranking
Algo
Commenting
Service
Elasticsearch
GraphQL can choose its data source

23. Search becomes a plugin that you can enable
on a GraphQL server with minimal configuration

24. Roadmap
Indexing
the Data
Keeping It
Updated
Data Flow Learnings

25. Initial Setup of the Search Indexer
Search
Indexer
GraphQL
Kafka
Elasticsearch
1. Use GraphQL - graph edges must be bidirectional
2. Listen to change events
3. Index to a search database

26. 1. Define a GraphQL query
2. Done
How to Build a New Index

27. Search Indexer Performance
Fan out Search Rebuild+ +Change +
< 500ms
=

28. More Information on Monday
medium.com/netflix-techblog

29. GraphQL Search Indexing
● Robust search and instant page loads
● Minimal refactoring on the client side
● Configuration driven and automated
+

30. Thank You
Artem Shtatnov
medium.com/netflix-techblog