The journey at gutefrage from a 17s MySQL query to a 90ms Elasticsearch query. The special ingredient is the request cache of Elasticsearch.

1. How We
(Almost) Forgot
Lambda Architecture and
Used Elasticsearch

2. Michael Stockerl
Data Engineer
michael.stockerl@gutefrage.net
@stockerlm

3. Gutefrage is
the question-answering plattform,
where millions of people help each
other.

4. Views per month
140 Mio.
Registered Users
4 Mio.
Questions
16.5 Mio.
Answers
63 Mio.

5. gutefrage architecture:
The big picture

6. App Analytics

7. App Analytics

8. App Analytics

9. Lambda Architecture Example: Answer Score
● Better sorting
● Hide bad answers
● Google Thin Content (SEO)

10. Incoming Event
Batch view
Live view
Join when read
Implemented with Lambda Architecture

11. Incoming Event
Batch view
Live view
Join when read
The batch layer

12. Incoming Event
Batch view
Live view
Join when read
The Speed Layer

13. Incoming Event
Batch view
Live view
Join when read
The Serving Layer

14. Learnings
+ Reads are fast
+ Spark helps building a Lambda Architecture
- Still duplicate code and complexity
- Each change needs an update of the batch view

15. Recent problem:
A new point system with user ranking

16. Points based on Feedback

17. Points based on Feedback
Like Question
Most Helpful Answer
Say Thank You Rate the Answer

18. Overall Ranking

19. Tag Ranking

20. Overall ranking with MySQL
SELECT
user_id,
SUM(points) as score
FROM event_log
WHERE created_at BETWEEN now() AND 90 Days ago
GROUP BY user_id
ORDER BY score DESC

21. First results of performance test
● Some queries were fast enough
● BUT: 17 - 20 seconds queries in worst case scenario

22. Solution:
Aggregations of Elasticsearch

23. Aggregations in Elasticsearch
The aggregations framework helps provide aggregated data based on a search
query. It is based on simple building blocks called aggregations, that can be
composed in order to build complex summaries of the data.
elasticsearch documentation

24. Aggregation for Top User List
"aggregations": {
"top_users": {
"terms": {
"field": "user_id",
"size": 100,
"shard_size": 2000,
"order": {
"total_score": "desc"
}
},
"total_score": {
"sum": {
"field": "score"
}
}
}
}

25. Aggregation for Top User List
"aggregations": {
"top_users": {
"terms": {
"field": "user_id",
"size": 100,
"shard_size": 2000,
"order": {
"total_score": "desc"
}
},
"total_score": {
"sum": {
"field": "score"
}
}
}
}
groupBy

26. Aggregation for Top User List
"aggregations": {
"top_users": {
"terms": {
"field": "user_id",
"size": 100,
"shard_size": 2000,
"order": {
"total_score": "desc"
}
},
"total_score": {
"sum": {
"field": "score"
}
}
}
}
order by

27. Aggregation for Top User List
"aggregations": {
"top_users": {
"terms": {
"field": "user_id",
"size": 100,
"shard_size": 2000,
"order": {
"total_score": "desc"
}
},
"total_score": {
"sum": {
"field": "score"
}
}
}
}
tune accuracy

28. Solution:
Aggregations of Elasticsearch
Query response times:
Worst case 2.5 seconds (MySQL 17s)

29. Request cache
● Search on local shards
● Cache local
● Invalidated on changes
● Hits.total, aggregations and suggestions

30. Request cache
● Search on local shards
● Cache local
● Invalidated on changes
● Hits.total, aggregations and suggestions
➔ Too much updates
➔ A lot of cache misses

31. Split data:
● Data of today: use index template to create index with first event
● Historical data: index without changes
Incoming Event
historical data
data of today

32. Use filtered aliases to select data of time range
Incoming Event
historical data
data of today
today
90days
filtered alias

33. Use cached results from historical data
Incoming Event
historical data
data of today
today
90days
filtered alias
Cache
_search?request_cache=true
service

34. The next day
Incoming Event
historical data
data of yesterday
today
90days
filtered alias
Cache
_search?request_cache=true
service
data of today

35. Merge the old indices
Incoming Event
historical data
data of yesterday
today
90days
filtered alias
Cache
_search?request_cache=true
service
data of today

36. Warm cache already in merge job
Incoming Event
historical data
data of today
today
90days
filtered alias
Cache
_search?request_cache=true
service

37. Query response times:
Worst case 90ms (MySQL 17s)

38. Learnings:
● Improved internal reindex framework
● Alias are always your friends
● Request cache FTW
● Cache miss, when you use index name instead of alias (?)
● Results may not be 100% accurate (but no problem for us)

39. Questions?

40. We’re hiring…
● Web-Developer
● We are looking for experts in the area of Search and
NLP interested in supporting us for a couple of days!
Please get in touch. :)