Scaling Massive Elasticsearch Clusters

  • 1,460 views
Uploaded on

 

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,460
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
61
Comments
0
Likes
8

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Scaling Massive ElasticSearch Clusters Rafał Kuć – Sematext International @kucrafal @sematext sematext.com
  • 2. Who Am I • „Solr 3.1 Cookbook” author • Sematext software engineer • Solr.pl co-founder • Father and husband :-) Copyright 2012 Sematext Int’l. All rights reserved
  • 3. What Will I Talk About ? • ElasticSearch scaling • Indexing thousands of documents per second • Performing queries in tens of milliseconds • Controling shard and replica placement • Handling multilingual content • Performance testing • Cluster monitoring Copyright 2012 Sematext Int’l. All rights reserved
  • 4. The Challenge • More than 50 millions of documents a day • Real time search • Less than 200ms average query latency • Throughput of at least 1000 QPS • Multilingual indexing • Multilingual querying Copyright 2012 Sematext Int’l. All rights reserved
  • 5. Why ElasticSearch ? • Written with NRT and cloud support in mind • Uses Lucene and all its goodness • Distributed indexing with document distribution control out of the box • Easy index, shard and replicas creation on live cluster Copyright 2012 Sematext Int’l. All rights reserved
  • 6. Index Design • Several indices (at least one index for each day of data) • Indices divided into multiple shards • Multiple replicas of a single shard • Real-time, synchronous replication • Near-real-time index refresh (1 to 30 seconds) Copyright 2012 Sematext Int’l. All rights reserved
  • 7. Shard Deployment Problems • Multiple shards per node • Replicas on the same nodes as shards • Not evenly distributed shards and replicas • Some nodes being hot, while others are cold Copyright 2012 Sematext Int’l. All rights reserved
  • 8. Default Shard Deployment ElasticSearch Cluster Node 1 Node 2 Node 3 Shard 1 Shard 2 Shard 3 Replica 1 Replica 2 Replica 3 Copyright 2012 Sematext Int’l. All rights reserved
  • 9. What Can We Do With Shards Then ? • Contol shard placement with node tags: – index.routing.allocation.include.tag – index.routing.allocation.exclude.tag • Control shard placement with nodes IP addresses: – cluster.routing.allocation.include._ip – cluster.routing.allocation.exclude._ip • Specified on index or cluster level • Can be changed on live cluster ! Copyright 2012 Sematext Int’l. All rights reserved
  • 10. Shard Allocation Examples • Cluster level: curl -XPUT localhost:9200/_cluster/settings -d '{ "persistent" : { "cluster.routing.allocation.exclude._ip" : "192.168.2.1" } }' • Index level: curl -XPUT localhost:9200/sematext/ -d '{ "index.routing.allocation.include.tag" : "nodeOne,nodeTwo" }' Copyright 2012 Sematext Int’l. All rights reserved
  • 11. Number of Shards Per Node • Allows one to specify number of shards per node • Specified on index level • Can be changed on live indices • Example: curl -XPUT localhost:9200/sematext -d '{ "index.routing.allocation.total_shards_per_node" : 2 }' Copyright 2012 Sematext Int’l. All rights reserved
  • 12. Controlled Shard Deployment ElasticSearch Cluster Node 1 Node 2 Node 3 Shard 1 Shard 2 Shard 3 Replica 1Replica 2 Replica 3 Copyright 2012 Sematext Int’l. All rights reserved
  • 13. Does Routing Matters ? • Controls target shard for each document • Defaults to hash of a document identifier • Can be specified explicitly (routing parameter) or as a field value (a bit less performant) • Can take any value • Example: curl -XPUT localhost:9200/sematext/test/1?routing=1234 -d '{ "title" : "Test routing document" }' Copyright 2012 Sematext Int’l. All rights reserved
  • 14. Indexing the Data ElasticSearch Cluster Node 1 Node 2 Node 3 Shard 1 Shard 2 Shard 3 Replica 1 Replica 2 Replica 3 Indexing application Copyright 2012 Sematext Int’l. All rights reserved
  • 15. How We Indexed Data ElasticSearch Cluster Node 1 Node 2 Shard 1 Shard 2 Node 3 Indexing application Copyright 2012 Sematext Int’l. All rights reserved
  • 16. Nodes Without Data • Nodes used only to route data and queries to other nodes in the cluster • Such nodes don’t suffer from I/O waits (of course Data Nodes don’t suffer from I/O waits all the time) • Not default ElasticSearch behavior • Setup by setting node.data to false Copyright 2012 Sematext Int’l. All rights reserved
  • 17. Multilingual Indexing • Detection of document's language before sending it for indexing • With, e.g. Sematext LangID or Apache Tika • Set known language analyzers in configuration or mappings • Set analyzer during indexing (_analyzer field) Copyright 2012 Sematext Int’l. All rights reserved
  • 18. Multilingual Indexing Example curl -XPUT localhost:9200/sematext/test/10 -d '{ "title" : "Test document", "langId" : "english" }' { "test" : { "_analyzer" : { "path" : "langId" }, "properties" : { "id" : { "type" : "long", "store" : "yes", "precision_step" : "0" }, "title" : { "type" : "string", "store" : "yes", "index" : "analyzed" }, "langId" : { "type" : "string", "store" : "yes", "index" : "not_analyzed" } } } } Copyright 2012 Sematext Int’l. All rights reserved
  • 19. Multilingual Queries • Identify language of query before its execution (can be problematic) • Query analyzer can be specified per query (analyzer parameter): curl -XGET localhost:9200/sematext/_search?q=let+AND+me&analyzer=english Copyright 2012 Sematext Int’l. All rights reserved
  • 20. Query Performance Factors – Lucene level • Refresh interval – Defaults to 1 second – Can be specified on cluster or index level – curl -XPUT localhost:9200/_settings -d '{ "index" : { "refresh_interval" : "600s" } }' • Merge factor – Defaults to 10 – Can be specified on cluster or index level – curl -XPUT localhost:9200/_settings -d '{ "index" : { "merge.policy.merge_factor" : 30 } }' Copyright 2012 Sematext Int’l. All rights reserved
  • 21. Let’s Talk About Routing Once Again • Routes a query to a particular shard • Speeds up queries depending on number of shards for a given index • Have to be specified manualy with routing parameter during query • routing parameter can take any value: curl -XGET 'localhost:9200/sematext/_search?q=test&routing=2012-02-16' Copyright 2012 Sematext Int’l. All rights reserved
  • 22. Querying ElasticSearch – No Routing Shard 1 Shard 2 Shard 3 Shard 4 Shard 5 Shard 6 Shard 7 Shard 8 ElasticSearch Index Application Copyright 2012 Sematext Int’l. All rights reserved
  • 23. Shard 1 Shard 2 Shard 3 Shard 4 Shard 5 Shard 6 Shard 7 Shard 8 ElasticSearch Index Application Querying ElasticSearch – With Routing Copyright 2012 Sematext Int’l. All rights reserved
  • 24. Performance Numbers Queries without routing (200 shards, 1 replica) #threads Avg response time Throughput 90% line Median CPU Utilization 1 3169ms 19,0/min 5214ms 2692ms 95 – 99% Queries with routing (200 shards, 1 replica) #threads Avg response time Throughput 90% line Median CPU Utilization 10 196ms 50,6/sec 642ms 29ms 25 – 40% 20 218ms 91,2/sec 718ms 11ms 10 – 15% Copyright 2012 Sematext Int’l. All rights reserved
  • 25. Scaling Query Throughput – What Else ? • Increasing the number of shards for data distribution • Increasing the number of replicas • Using routing • Avoid always hitting the same node and hotspotting it Copyright 2012 Sematext Int’l. All rights reserved
  • 26. FieldCache and OutOfMemory • ElasticSearch default setup doesn’t limit field data cache size Copyright 2012 Sematext Int’l. All rights reserved
  • 27. FieldCache – What We Can do With It ? • Keep its default type and set: – Maximum size (index.cache.field.max_size) – Expiration time (index.cache.field.expire) • Change its type: – soft (index.cache.field.type) • Change your data: – Make your fields less precise (ie: dates) – If you sort or facet on fields think if you can reduce fields granularity • Buy more servers :-) Copyright 2012 Sematext Int’l. All rights reserved
  • 28. FieldCache After Changes Copyright 2012 Sematext Int’l. All rights reserved
  • 29. Additional Problems We Encountered • Rebalancing after full cluster restarts – cluster.routing.allocation.disable_allocation – cluster.routing.allocation.disable_replica_allocation • Long startup and initialization • Faceting with strings vs faceting on numbers on high cardinality fields Copyright 2012 Sematext Int’l. All rights reserved
  • 30. JVM Optimization • Remember to leave enough memory to OS for cache • Make GC frequent ans short vs. rare and long – -XX:+UseParNewGC – -XX:+UseConcMarkSweepGC – -XX:+CMSParallelRemarkEnabled • -XX:+AlwaysPreTouch (for short performance tests) Copyright 2012 Sematext Int’l. All rights reserved
  • 31. Performance Testing • Data – How much data do I need ? – Choosing the right queries • Make changes – One change at a time – Understand the impact of the change • Monitor your cluster (jstat, dstat/vmstat, SPM) • Analyze your results Copyright 2012 Sematext Int’l. All rights reserved
  • 32. ElasticSearch Cluster Monitoring • Cluster health • Indexing statistics • Query rate • JVM memory and garbage collector work • Cache usage • Node memory and CPU usage Copyright 2012 Sematext Int’l. All rights reserved
  • 33. Cluster Health Copyright 2012 Sematext Int’l. All rights reserved Node restart
  • 34. Indexing Statistics Copyright 2012 Sematext Int’l. All rights reserved
  • 35. Query Rate Copyright 2012 Sematext Int’l. All rights reserved
  • 36. JVM Memory and GC Copyright 2012 Sematext Int’l. All rights reserved
  • 37. Cache Usage Copyright 2012 Sematext Int’l. All rights reserved
  • 38. CPU and Memory Copyright 2012 Sematext Int’l. All rights reserved
  • 39. Summary • Controlling shard and replica placement • Indexing and querying multilingual data • How to use sharding and routing and not to tear your hair out • How to test your cluster performance to find bottle-necks • How to monitor your cluster and find problems right away Copyright 2012 Sematext Int’l. All rights reserved
  • 40. We Are Hiring ! • Dig Search ? • Dig Analytics ? • Dig Big Data ? • Dig Performance ? • Dig working with and in open – source ? • We’re hiring world – wide ! http://sematext.com/about/jobs.html Copyright 2012 Sematext Int’l. All rights reserved
  • 41. How to Reach Us • Rafał Kuć – Twitter: @kucrafal – E-mail: rafal.kuc@sematext.com • Sematext – Twitter: @sematext – Website: http://sematext.com • Graphs used in the presentation are from: – SPM for ElasticSearch (http://sematext.com/spm) Copyright 2012 Sematext Int’l. All rights reserved
  • 42. Thank You For Your Attention