Taking Elasticsearch From 0 to 88mph
Everyone wants their Elasticsearch cluster to index and search faster, but optimizing both and finding the balance between the two can be tricky. At Kenna Security, we use Elasticsearch to store over 3 billion vulnerabilities for our clients. All that data needs to be quickly accessible so clients can assess their cyber security risk. At the same time the data is constantly changing. On average, we update 200+ million documents a day which means indexing speed is also a top priority. In the early days our cluster could barely keep up. Nodes would fall over constantly, indexing queues would get backed up for days, and searches timed out about 50% of the time. Fixing all of these issues did not happen overnight. However, with a lot of testing, tweaking, and a few “OH crap!” moments we were able to build a stable, 21 node cluster that now meets all of our indexing and searching demands. In this talk I will share the insights we gained and the strategies we used to scale our cluster and hopefully that advice will save others some time and frustration as they grow their own.
More Related Content
What's hot
What's hot(20)
Similar to Taking Elasticsearch From 0 to 88mph
Similar to Taking Elasticsearch From 0 to 88mph (20)
![[2D1]Elasticsearch 성능 최적화](https://cdn.slidesharecdn.com/ss_thumbnails/2d1elasticsearch-140929192211-phpapp02-thumbnail.jpg?width=768&fit=bounds)
![[2 d1] elasticsearch 성능 최적화](https://cdn.slidesharecdn.com/ss_thumbnails/2d1elasticsearch-140930233029-phpapp02-thumbnail.jpg?width=768&fit=bounds)
More from Molly Struve
More from Molly Struve(13)
Recently uploaded
Recently uploaded(20)
Taking Elasticsearch From 0 to 88mph
- 1. Taking Elasticsearch from 0 to 88 mph By: Molly Struve
- 2. 500 million documents 1 million processed daily
- 3. Over 3 billion documents 200 million processed daily
- 4. The average company has... 60 thousand assets 24 million vulnerabilities?
- 10. In-memory buffer
- 11. Toggle the Refresh Interval curl -XPUT 'localhost:9200/my_index/_settings' -d '{ "index" : { "refresh_interval" : "30s" } }'
- 12. In addition... ● ?refresh=wait_for option ● Manual refresh ○ curl -XPOST 'localhost:9200/my_index/_refresh'
- 13. Speed up Indexing ● Toggle the refresh interval1 2 3 Speed Up Searching 4 5 6 7
- 14. ● 200 thousand assets ● 100 million vulnerabilities
- 17. Bulk Processing
- 18. POST _bulk
- 19. When bulk processing your data... ● Start with batches of 100 and double the size from there until indexing time plateaus ● Too large requests can put memory pressure on Elasticsearch so keep it under a couple tens of megabytes
- 20. Bulk process data Speed up Indexing Toggle the refresh interval1 2 3 Speed Up Searching 4 5 6 7
- 22. 429 Too Many Requests
- 24. 8 Threads 4 Threads Shard 2 Shard 3 Shard 4 4 Threads 4 Threads Shard 1 Shard 2 Shard 3 Shard 4 Shard 1 Shard 1 Shard 2 Shard 2 2 Threads 2 Threads Shard 1 Shard 3 Shard 4 Shard 4 Shard 3
- 25. Routing shard = hash(_routing) % number_of_primary_shards PUT my_index/_doc/1?routing=custom { "title": "This is a document" } document _id custom
- 26. 4 Threads 2 Threads 2 Threads Shard 1 Shard 2 Shard 2 Shard 1 Shard 3 Shard 4 Shard 4 Shard 3 Route 1 Route 2 Route 2 Route 1 Route 3 Route 4 Route 4 Route 3
- 27. Parent -> Child Asset -> Vulnerabilities PUT my_index/_vulnerability/1?parent=2 { "title": "This is a document" }
- 28. Route your documents Bulk process data Speed up Indexing Toggle the refresh interval1 2 3 Speed Up Searching 4 5 6 7
- 31. Group Your Data
- 32. Client 1 Client 2 Client 3 Client 4
- 33. Speed Up Searching Group your data Route your documents Bulk process data Speed up Indexing Toggle the refresh interval1 2 3 4 5 6 7
- 34. Does NOT score documents and only cares if the document matches the search criteria or not Queries Scores documents based on how well they match the search criteria. Filters Easy/Fast Hard/Slow Easy/Fast Hard/Slow
- 35. 2.x 5.x
- 41. GET /_search { "query": { "bool" : { "must" : { "term" : { "user" : "kimchy" } }, "filter": { "term" : { "tag" : "tech" } } } } }
- 42. Use filters whenever possible Speed Up Searching Group your data Route your documents Bulk process data Speed up Indexing Toggle the refresh interval1 2 3 4 5 6 7
- 44. Store IDs as keywords
- 45. Why? ● Numeric mapping types are optimized for RANGE queries ● Keyword mapping types are optimized for TERM queries
- 47. 30% increase in search speed
- 48. Store IDs as keywords Use filters whenever possible Speed Up Searching Group your data Route your documents Bulk process data Speed up Indexing Toggle the refresh interval1 2 3 4 5 6 7
- 51. Don’t let your users slow you down
- 53. Define keywords for searching
- 54. Don’t let your users slow you down Store IDs as keywords Use filters whenever possible Speed Up Searching Group your data Route your documents Bulk process data Speed up Indexing Toggle the refresh interval1 2 3 4 5 6 7
- 55. Questions?