The document discusses Elasticsearch and how it was used to improve the performance of a transaction report used by tens of thousands of users daily. It needed to be faster loading. Elasticsearch provided near real-time latency, fast aggregations, and accessible individual transaction records while being flexible, scalable, and able to handle increased load. It produced results in under 200ms and allowed easy scaling by adding nodes. Key features included its REST API and ability to add new data fields with little work.

1. Elasticsearch
@Affiliate Window

2. Richard He
Team Falcon

3. A transaction report used by tens of thousands of
users every day that took too long to load
Our problem

5. Near real-time latency
Really fast aggregations
Accessible individual transaction records
Flexible
Scalable
What we wanted

6. The solution - Producing

7. The solution -Consuming

8. When shit hits the fan

9. Next steps

10. A good architect maximises the number of
decisions NOT made
by Uncle Bob Martin

11. New data fields can be added with almost no work
Average response time is <200ms
Easy to scale by just adding nodes
REST API
Why Elasticsearch

12. Things that are good to know…

13. Transaction Log - Refresh - Flush

18. 1.x – fsync every 5 seconds
2.x – on request (default) / fsync every 5 seconds
Transaction log durability

19. "refresh_interval" : "-1”
"refresh_interval" : ”1s”
"refresh_interval" : ”15s"
Refresh interval
A refresh makes data searchable and a refresh interval decides how often
that happens

20. Filters vsQueries (v 1.7)
A filter askes yes / no questions, a query ask how well does this document
match

21. 76ms 1385ms

22. Filters vsQueries
Disable analysed field if not needed
Use a filter if a query is not required

23. DocValues
Set a smaller heap size (4-16G on a 64G machine)
More RAM is now available for File System Cache
Enabled by default in version 2
Does not work with analysed field
On-disk data structure that allows fast aggregation without JVM Heap
overhead

24. Aggregations are approximate

28. Wrong!

29. The solution
Size = 0
The cost

30. Split Brain

31. M
4
52
1

32. 1
2
3
4 5

33. M
2
M
4 5

34. M
2
M
4 5

35. 1
2
M
4
5

36. Doc values
https://www.elastic.co/guide/en/elasticsearch/reference/current/doc-values.html
https://www.elastic.co/guide/en/elasticsearch/guide/current/_deep_dive_on_doc_values.html
Aggregation
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html
Update Index settings
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-update-settings.html
Important config changes
https://www.elastic.co/guide/en/elasticsearch/guide/1.x/_important_configuration_changes.html
https://www.elastic.co/guide/en/elasticsearch/guide/current/translog.htmlhttps://github.com/elastic/elasticsearch/issues/11287
Transaction log
https://www.elastic.co/guide/en/elasticsearch/guide/current/translog.html
https://github.com/elastic/elasticsearch/issues/11287
Resources

37. Questions