The document describes challenges faced when building a search application using Elasticsearch to index and search 6 billion documents. The initial approach of using default shard counts and indexing strategies led to out of memory errors and slow searches. Key problems identified were high field data usage bringing down nodes, searching all indices being slow, and the garbage collector being unable to free enough memory. Improvements involved right-sizing the number of shards, monitoring and reducing field data, targeting specific indices in searches, changing garbage collectors, and dedicating hardware roles.

1. elasticsearchelasticsearch
from the trenchesfrom the trenches
Jai Jones
jaij@slalom.com

2. about meabout me
solution architect at slalom
enjoy building search apps
7+ years Lucene
2+ years Hibernate Search
~2 years Elasticsearch

3. agendaagenda
the ask
initial approach
problems
next steps
lessons learned
improvements
questions

4. the askthe ask
search 6 billions docs in under 1.5 sec
index 2 millions new docs / day
export billions of docs to CSV files
index and search docs in realtime
use search throughout the application
free text search
faceted navigation
suggestions
dashboards

5. free text searchfree text search

6. faceted navigationfaceted navigation
drill down

7. suggestionssuggestions

8. dashboardsdashboards

9. hardwarehardware
used "large" servers
servers had lots of CPUs & RAM
non-RAIDed spinning disks
5 dedicated nodes
all nodes store data
all nodes are master
all nodes sort & aggregate
clustercluster
initial approachinitial approach

10. shardsshards
used the default shard count
5 primary + 1 replica
unlimited primary shards / node
indicesindices
data was chronological
used the time-based index strategy
weekly indices for transaction logs
daily indices for audit logs
initial approachinitial approach

11. memorymemory
dedicated 31 GB to the jvm heap
used remaining memory for file system cache
turned off linux process swapping
maxed out linux file descriptors
used G1 Garbage Collector
initial approachinitial approach
index mappingsindex mappings
indexed all fields
stored big documents with 60+ fields
nested documents
parent-child relationships

12. searchessearches
searched all indices
used query_string searches
searched all fields
sorted & aggregated on any field
range queries
parent-child queries
GET /index-*/_search
"query_string" : {
"query": "+(eggplant | potato)",
"default_field": "_all",
"default_operator": "and"
}
initial approachinitial approach

13. problemsproblems
OutOfMemoryError
field data exceeded jvm heap
shard count was in the thousands
garbage collector could not free memory
CircuitBreakerException
field data exceeded jvm heap
search results exceeded jvm heap
slow searches (latency increased from seconds to minutes)
nodes became unresponsive
frequent GC pauses
early signs

14. cluster downcluster down
index corruption
data loss
nodes failed to restart

15. next stepsnext steps
shard capacityshard capacity
understand data & searches
size based on actual usage
field datafield data
monitor
identify the producers
reduce usage
searchsearch
identify bottlenecks
optimize
clustercluster
find failure points
make topology changes
make hardware changes
identify and fix problems...

16. shard capacityshard capacity
1 shard can handle a lot of data
actually it held ~5x more data
didn't need 5 shards per index
did't need weekly/daily indices
learned...learned...
shard is the unit of scale
how much data can a single shard hold?
find the single shard breaking point
1. loaded a single shard with data
2. ran typical searches
3. recorded search response time
4. repeated until response time became unacceptable

17. field datafield data
which fields and indices are using a lot of field data?
use the stats API to find out
fields used for sorting & aggregation
high cardinality fields
id-cache for parent-child relationships
field data is loaded first time field is accessed
field data is maintained per-index
field data is not GC'd
culprits...culprits...
# Node Stats
curl -XGET 'http://localhost:9200/_nodes/stats/indices/fielddata?human'
# Indices Stat
curl -XGET 'http://localhost:9200/_stats/fielddata/?human'

18. searchsearch
searching all indices is slow, CPU
intensive and causes field data to
be loaded for every index
# Searches all indices
/indexname-*/_search
# Search specific indices
/indexname-2015/_search
query_string is flexible but allows
inefficient searches like leading
wildcard searches and searches
_all fields by default
{
"query_string" : {
"default_field" : "_all",
"allow_leading_wildcard" "true",
"query" : "this AND that OR thus"
}
}
what are the bottlenecks and resource killers?

19. clustercluster
field data used up 70-90% of the heap memory
not much heap left for node & shard management
stop the world Garbage Collector (GC) pauses made the
cluster unresponsive
nodes dropped out of the cluster
the G1 GC had longer pauses than the CMS GC
sorting, aggregations, id-cache for parent-child
relationships used up a lot of heap memory
managing too many shards used a lot of heap memory
why is the cluster crashing?

20. lessons learned...lessons learned...
number of shards / node should not exceed the number of CPU cores
figure out the single shard capacity
monitor field data usage
field data usage is permanent and does not get garbage collected
too high field data usage will bring down the cluster
search specific indices by target date range
tune and test all search API searches
split cluster into data, client and master nodes
use the default ES JVM settings and garbage collector

21. hardwarehardware
used "large" servers
servers had lots of CPUs & RAM
non-RAIDed spinning disks
put master and client nodes on same servers
5 8 dedicated nodes
all nodes are master dedicated master nodes
all nodes store data dedicated data nodes
all nodes sort & aggregate dedicated client nodes
clustercluster
improvementsimprovements

22. shardsshards
default shard count didn't work
5 1 primary + 1 replica
unlimited primary shards / node # of primary
shards less than # of CPU cores
indicesindices
data was chronological
used the time-based index strategy
weekly monthly indices for transaction logs
daily monthly indices for audit logs
improvementsimprovements

23. memorymemory
dedicated 31 GB to the jvm heap
used remaining memory for file system cache
turned off linux process swapping
maxed out linux file descriptors
used new G1 GC used stable CMS GC
improvementsimprovements

24. index mappingsindex mappings
indexed all 40 fields
stored big documents with 60+ fields
nested documents
parent-child relationships
used field aliases to define alternate
fields used in sorting and aggregation
used doc_value on sortable &
aggregation fields
changed boolean data type to string
"field": {
"index": "no"
}
# uses field data
"fieldA": {
"type": "boolean"
}
# uses doc_value (no field data)
"fieldA": {
"type": "string",
"index": "analyzed",
"fields": {
"raw" : {
"type" : "string",
"index" : "not_analyzed",
"fielddata": {
"format": "doc_values"
}
}
}
}
improvementsimprovements

25. searchessearches
search all indices target specific
indices
query_string simple_query_string
search on all some fields
sorting & aggregations on all low
cardinality fields
range queries filters
parent-child nested queries
added query timeouts
GET /index-201501/_search
"simple_query_string" : {
"query": "+(eggplant | potato)",
"fields": ["field1", "field2"],
"default_operator": "and"
}
improvementsimprovements

26. Questions?Questions?

27. Thank YouThank You