Elasticsearch in Production (London version)
Upcoming SlideShare
Loading in...5
×
 

Elasticsearch in Production (London version)

on

  • 2,659 views

Elasticsearch in production, or an overview of things you want to know about before happening upon them in production.

Elasticsearch in production, or an overview of things you want to know about before happening upon them in production.

Statistics

Views

Total Views
2,659
Views on SlideShare
984
Embed Views
1,675

Actions

Likes
5
Downloads
16
Comments
0

13 Embeds 1,675

http://www.elasticsearch.org 1447
http://feedly.com 130
https://twitter.com 27
http://digg.com 21
http://newsblur.com 16
http://www.newsblur.com 12
http://www.slideee.com 6
http://www.inoreader.com 6
http://new.elasticsearch.com 4
http://www.hanrss.com 2
http://new.es.dev 2
https://reader.aol.co.uk 1
http://www.linkedin.com 1
More...

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Elasticsearch in Production (London version) Elasticsearch in Production (London version) Presentation Transcript

  • Elasticsearch in Production ! Alex Brasetvik alex@found.no @alexbrasetvik
  • Elasticsearch in Production ! Alex Brasetvik alex@found.no @alexbrasetvik
  • Who? Co-founder of Found AS 8+ years search, 3+ Elasticsearch Herding hundreds of Elasticsearch clusters
  • Agenda
  • Agenda • Anti-patterns • Memory / Resource Usage • Distributed problems • Security • Client concerns • Changing a cluster
  • found.no/foundation Elasticsearch in Production Elasticsearch as a NoSQL Database Intro to Function Scoring All About Analyzers Securing your Elasticsearch Cluster
  • Snapshot / Restore Circuit breakers Document values Aggregations Distributed percolation Suggesters …
  • Anti-Patterns
  • Arbitrary Keys • “Schema Free” • One field per value • Ever-growing cluster state acls: 1234: READ 42: WRITE
  • Heavy Updating • Update = Delete + Reindex • Be careful with counters
  • Slow queries • WHERE foo ILIKE ‘%bar%’ • {“query_string”: {“query”: “foo:*bar*”}} • Don’t ask for 3300 results :)
  • Arbitrary searches query: filtered: filter: term: user_id: 42 query: [user’s query here]
  • Memory
  • Memory • Field caches • Filter caches • Page caches • Aggregations • Index building
  • Page Cache • Keeping index pages in memory • Can’t have too much • Outgrow: Gradual slowdown
  • Heap Space • Memory used by Elasticsearch process • Field / Filter caches • Aggregations
  • Time Bomb
  • Time Bomb
  • OutOfMemoryError Woah there I ate all the memories Your cluster may or may not work any more
  • OutOfMemory • Growing too big • Selecting too big timespan in Kibana • Document ingestion peak
  • Preventing OOMs • Have enough memory :-) • Understand your search’s memory profile • Bulk / Circuit breaker settings • Monitoring • Document values
  • Marvel ( /_stats )
  • "my_field": { "type": "string", "fielddata": { "format": "doc_values" } }
  • Document Values • Rely on page cache • Only caches doc values actually used
  • Sizing
  • Sizing • Test, don’t guess • Start big, scale down • Index, search, monitor
  • Glitch Meltdown
  • Glitch Meltdown
  • • Tie-breaker can be a cheap master-node • Applies to data centers / availability zones too
  • Data-only nodes Master-only nodes
  • Jepsen
  • Jepsen • Kyle Kingsbury’s series on distributed systems • Distributed systems are hard • aphyr.com
  • Security
  • Security • “Not my job!” – Elasticsearch • That’s fine!
  • Dynamic Scripts ! • Scoring • Aggregations • Updating
  • Dynamic Scripts Runtime.getRuntime().exec(…)
  • Dynamic Scripts Runtime.getRuntime().exec(…) <script src=“http://127.0.0.1:9200/_search?callback=capture&…
  • Security ! • Disable dynamic scripts (On by default in ≤1.1) • Mind index patterns • Even then, don’t accept arbitrary requests
  • Client Concerns
  • Client Concerns • Connection pools • Idempotent requests • Have sane syncing/indexing strategies
  • # BOOM !
  • Cluster changes
  • Cluster changes • Make new nodes join existing cluster • No rolling restarts • Easy rollback if things go bad
  • v1.0.0 v1.0.1
  • Cluster changes • Test first • Mind recover_*-settings
  • Multi-Cluster Workflows • Snapshot/Restore • Operations across clusters • Swap clusters! • Works well with good syncing strategy
  • • Rolling restarts: Risky, fast • Grow and shrink: Less risky, copies lots of data • Multiple clusters: Least risky, copies lots of data
  • Misc • Same JVM • ulimits • Unicast • Kernel-settings like IO-scheduler
  • ? @foundsays