Elasticsearch in production, or an overview of things you want to know about before happening upon them in production.

1. Elasticsearch in Production
!
Alex Brasetvik
alex@found.no
@alexbrasetvik

2. Elasticsearch in Production
!
Alex Brasetvik
alex@found.no
@alexbrasetvik

3. Who?
Co-founder of Found AS
8+ years search, 3+ Elasticsearch
Herding hundreds of Elasticsearch clusters

4. Agenda

5. Agenda
• Anti-patterns
• Memory / Resource Usage
• Distributed problems
• Security
• Client concerns
• Changing a cluster

6. found.no/foundation
Elasticsearch in Production Elasticsearch as a NoSQL Database
Intro to Function Scoring All About Analyzers
Securing your Elasticsearch Cluster

11. Snapshot / Restore
Circuit breakers
Document values
Aggregations
Distributed percolation
Suggesters
…

12. Anti-Patterns

13. Arbitrary Keys
• “Schema Free”
• One field per value
• Ever-growing cluster state
acls:
1234: READ
42: WRITE

14. Heavy Updating
• Update = Delete + Reindex
• Be careful with counters

15. Slow queries
• WHERE foo ILIKE ‘%bar%’
• {“query_string”: {“query”: “foo:*bar*”}}
• Don’t ask for 3300 results :)

16. Arbitrary searches
query:
filtered:
filter:
term:
user_id: 42
query:
[user’s query here]

17. Memory

18. Memory
• Field caches
• Filter caches
• Page caches
• Aggregations
• Index building

19. Page Cache
• Keeping index pages in memory
• Can’t have too much
• Outgrow: Gradual slowdown

20. Heap Space
• Memory used by Elasticsearch process
• Field / Filter caches
• Aggregations

22. Time Bomb

23. Time Bomb

24. OutOfMemoryError
Woah there
I ate all the memories
Your cluster may or may not work any more

25. OutOfMemory
• Growing too big
• Selecting too big timespan in Kibana
• Document ingestion peak

26. Preventing OOMs
• Have enough memory :-)
• Understand your search’s memory profile
• Bulk / Circuit breaker settings
• Monitoring
• Document values

27. Marvel
( /_stats )

31. "my_field": {
"type": "string",
"fielddata": {
"format": "doc_values"
}
}

32. Document Values
• Rely on page cache
• Only caches doc values actually used

33. Sizing

34. Sizing
• Test, don’t guess
• Start big, scale down
• Index, search, monitor

38. Glitch Meltdown

39. Glitch Meltdown

42. • Tie-breaker can be a cheap master-node
• Applies to data centers / availability zones too

43. Data-only
nodes
Master-only
nodes

45. Jepsen

46. Jepsen
• Kyle Kingsbury’s series on distributed systems
• Distributed systems are hard
• aphyr.com

47. Security

48. Security
• “Not my job!” – Elasticsearch
• That’s fine!

49. Dynamic Scripts
!
• Scoring
• Aggregations
• Updating

50. Dynamic Scripts
Runtime.getRuntime().exec(…)

51. Dynamic Scripts
Runtime.getRuntime().exec(…)
<script src=“http://127.0.0.1:9200/_search?callback=capture&…

52. Security
!
• Disable dynamic scripts (On by default in ≤1.1)
• Mind index patterns
• Even then, don’t accept arbitrary requests

53. Client Concerns

54. Client Concerns
• Connection pools
• Idempotent requests
• Have sane syncing/indexing strategies

56. # BOOM !

57. Cluster changes

58. Cluster changes
• Make new nodes join existing cluster
• No rolling restarts
• Easy rollback if things go bad

59. v1.0.0 v1.0.1

60. Cluster changes
• Test first
• Mind recover_*-settings

61. Multi-Cluster Workflows
• Snapshot/Restore
• Operations across clusters
• Swap clusters!
• Works well with good syncing strategy

62. • Rolling restarts: Risky, fast
• Grow and shrink: Less risky, copies lots of data
• Multiple clusters: Least risky, copies lots of data

63. Misc
• Same JVM
• ulimits
• Unicast
• Kernel-settings like IO-scheduler

64. ?
@foundsays