Taking Elasticsearch From 0 to 88mph

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Taking Elasticsearch from
0 to 88 mph
By: Molly Struve

500 million documents
1 million processed daily

Over 3 billion documents
200 million processed daily

The average company has...
60 thousand
assets
24 million
vulnerabilities?

MySQL Elasticsearch

Refresh Interval

In-memory buffer

In-memory buffer

$Toggle the Refresh Interval curl -XPUT 'localhost:9200/my_index/_settings' -d '{ "index" : { "refresh_interval" : "30s" } ...$

In addition...
● ?refresh=wait_for option
● Manual refresh
○ curl -XPOST 'localhost:9200/my_index/_refresh'

Speed up Indexing
● Toggle the refresh interval1
2
3
Speed Up Searching
4
5
6
7

● 200 thousand assets
● 100 million
vulnerabilities

Bulk Processing

POST _bulk

When bulk processing your data...
● Start with batches of 100 and double the size from there until indexing time
plateaus
...

Bulk process data
Speed up Indexing
Toggle the refresh interval1
2
3
Speed Up Searching
4
5
6
7

MySQL
Elasticsearch
Elasticsearch
MySQL

429 Too Many Requests

Route Your Documents

8 Threads 4 Threads
Shard 2
Shard 3
Shard 4
4 Threads
4 Threads
Shard 1
Shard 2
Shard 3
Shard 4
Shard 1
Shard 1
Shard 2
Sh...

$Routing shard = hash(_routing) % number_of_primary_shards PUT my_index/_doc/1?routing=custom { "title": "This is a documen...$

4 Threads
2 Threads
2 Threads
Shard 1
Shard 2
Shard 2
Shard 1
Shard 3
Shard 4
Shard 4
Shard 3
Route 1
Route 2
Route 2
Rout...

$Parent -> Child Asset -> Vulnerabilities PUT my_index/_vulnerability/1?parent=2 { "title": "This is a document" }$

Route your documents
Bulk process data
Speed up Indexing
Toggle the refresh interval1
2
3
Speed Up Searching
4
5
6
7

Speed up Searching

Typical Logging Cluster
logstash_2018.09.01
logstash_2018.09.02
logstash_2018.09.03
logstash_2018.09.04
Search

Group Your Data

Client 1 Client 2 Client 3 Client 4

Speed Up Searching
Group your data
Route your documents
Bulk process data
Speed up Indexing
Toggle the refresh interval1
2...

Does NOT score documents
and only cares if the
document matches the search
criteria or not
Queries
Scores documents based ...

2.x 5.x

Use Filters Whenever
Possible

Filters Are Friends!
Filters

$GET /_search { "query": { "bool" : { "must" : { "term" : { "user" : "kimchy" } }, "filter": { "term" : { "tag" : "tech" } ...$

Use filters whenever possible
Speed Up Searching
Group your data
Route your documents
Bulk process data
Speed up Indexing
...

Store IDs as keywords

Why?
● Numeric mapping types are optimized for RANGE
queries
● Keyword mapping types are optimized for TERM
queries

30% increase in
search speed

Store IDs as keywords
Use filters whenever possible
Speed Up Searching
Group your data
Route your documents
Bulk process d...

Don’t let your users slow
you down

Define keywords for searching

Don’t let your users slow you down
Store IDs as keywords
Use filters whenever possible
Speed Up Searching
Group your data
...

Questions?

Contact
https://www.linkedin.com/in/mollystruve/
https://github.com/mstruve
@molly_struve
molly.struve@gmail.com