Elasticsearch easily lets you develop amazing things, and it has gone to great lengths to make Lucene's features readily available in a distributed setting. However, when it comes to running Elasticsearch in production, you still have a fairly complicated system on your hands: a system with high demands on network stability, a huge appetite for memory, and a system that assumes all users are trustworthy. This talk will cover some of the lessons we've learned from securing and herding hundreds of Elasticsearch clusters.

1. Elasticsearch in production
!
Konrad Beiske
konrad@found.no
@beiske

2. Who?
Senior software engineer of Found AS
Working with Elasticsearch for 2 years
Herding hundreds of Elasticsearch clusters

3. Agenda

4. Agenda
• Anti-patterns
• Memory / Resource Usage
• Distributed problems
• Security
• Client concerns
• Changing a cluster

5. found.no/foundation

10. Snapshot / Restore
Circuit breakers
Document values
Aggregations
Distributed percolation
Suggesters
…

11. Snapshot / Restore
Circuit breakers
Document values
Aggregations
Distributed percolation
Suggesters
…

12. Anti-Patterns

13. Arbitrary Keys
• “Schema Free”
• One field per value
acls:
• Ever-growing cluster state
1234: READ
42: WRITE

14. Heavy Updating
• Update = Delete + Reindex
• Be careful with counters

15. Slow queries
• WHERE foo ILIKE ‘%bar%’
• {“query_string”: {“query”: “foo:*bar*”}}

16. Arbitrary searches
query:
filtered:
filter:
term:
user_id: 42
query:
[user’s query here]

18. Time Bomb

19. Memory

20. Memory
• Field caches
• Filter caches
• Page caches
• Aggregations
• Index building

21. Page Cache
• Keeping index pages in memory
• Can’t have too much
• Outgrow: Gradual slowdown

22. Heap Space
• Memory used by Elasticsearch process
• Field / Filter caches
• Aggregations

23. Time Bomb

24. Time Bomb

25. OutOfMemoryError
Woah there
I ate all the memories
Your cluster may or may not work any more

26. OutOfMemory
• Growing too big
• Selecting too big timespan in Kibana
• Document ingestion peak

27. Preventing OOMs
• Have enough memory :-)
• Understand your search’s memory profile
• Bulk / Circuit breaker settings
• Monitoring
• Document values

28. Marvel
( /_stats )

31. Document Values

32. "my_field": {
"type": "string",
"fielddata": {
"format": "doc_values"
}
}

33. Sizing

34. Sizing
• Test, don’t guess
• Start big, scale down
• Index, search, monitor

38. Glitch Meltdown

40. Glitch Meltdown

41. Glitch Meltdown

42. Glitch Meltdown

43. Glitch Meltdown

46. • Tie-breaker can be a cheap master-node
• Applies to data centers / availability zones too

47. Data-only
nodes
Master-only
nodes

49. Jepsen

50. Jepsen
• Kyle Kingsbury’s series on distributed systems
• Distributed systems are hard
• aphyr.com

51. Security

52. Security
• “Not my job!” – Elasticsearch
• That’s fine!

53. Dynamic Scripts
!
• Scoring
• Aggregations
• Updating

54. Dynamic Scripts
Runtime.getRuntime().exec(…)

55. Security
!
• Disable dynamic scripts
• Mind index patterns
• Even then, don’t accept arbitrary requests

56. Client Concerns

57. Client Concerns
• Connection pools
• Idempotent requests
• Have sane syncing/indexing strategies

58. # BOOM !

59. Cluster changes

60. Cluster changes
• Make new nodes join existing cluster
• No rolling restarts
• Easy rollback if things go bad

61. v1.0.0 v1.0.1

62. v1.0.0 v1.0.1

63. v1.0.0 v1.0.1

64. v1.0.0 v1.0.1

65. v1.0.0 v1.0.1

66. Cluster changes
• Test first
• Mind recover_*-settings

67. Multi-Cluster Workflows
• Snapshot/Restore
• Operations across clusters
• Swap clusters!
• Works well with good syncing strategy

68. Misc
• Same JVM
• ulimits
• Unicast and cluster name
• SSD? noop-scheduler

69. Learn More!
Follow
@foundsays
!
found.no/foundation
@beiske