Download as PDF, PPTX
![37
/_cluster/allocation/explain
…
"allocate_explanation" : "cannot allocate because allocation
is not permitted to any of the nodes”,
…
{
"decider" : "filter",
"decision" : "NO",
"explanation" : "node does not match index setting
[index.routing.allocation.include]
filters [_name:"non_existent_node"]"
}
…](https://image.slidesharecdn.com/elasticsearchelasticon2017-170405093831/75/Elastic-ON-2017-Recap-37-2048.jpg)
![38
/_cluster/allocation/explain
…
"unassigned_info" : {
"reason" : "NODE_LEFT",
"at" : "2017-01-04T18:03:28.464Z",
"details" : "node_left[OIWe8UhhThCK0V5XfmdrmQ]",
"last_allocation_status" : "no_valid_shard_copy"
},
"can_allocate" : "no_valid_shard_copy",
"allocate_explanation" : "cannot allocate because a
previous copy of the primary shard existed but can no longer
be found on the nodes in the cluster"
…](https://image.slidesharecdn.com/elasticsearchelasticon2017-170405093831/75/Elastic-ON-2017-Recap-38-2048.jpg)
Uploaded byMatias Cascallares
Elastic{ON} 2017 Recap
The document provides a comprehensive recap of Elasticsearch developments in 2017, highlighting its growth to 85,000 members and various performance enhancements in version 5.x. Key features discussed include full-text search improvements, operational optimizations, and introductions of tools like the Java high-level HTTP client and cross-cluster search capabilities. It also outlines significant upgrades, such as rolling upgrades from version 5 to 6, emphasizing efficiency and ease of use in Elasticsearch's evolving architecture.
More Related Content
Similar to Elastic{ON} 2017 Recap
Elastic{ON} 2017 Recap
- 1.
- 3.
- 4.
- 5.
- 6. 6 • Faster • Friendlier •Smaller • Smarter • Safer 26 October 2016 Elasticsearch 5.0.0
- 7. Throughput with onereplica on two nodes, with auto-generated IDs Append-only indexing 7 0 8 15 23 30 v2.4.2 v5.2.1 master K docs/s
- 8. What’s new in5.x?
- 9.
- 10. Range Fields &Queries
- 11. 11 What’s on atElasticon tomorrow between 11am and 2pm?
- 12.
- 13.
- 14. 14 Wednesday 11am -2pm - INTERSECTS
- 15. 15 Wednesday 11am -2pm - CONTAINS
- 16. 16 Wednesday 11am -2pm - WITHIN
- 17.
- 18. 18 { "city": { "type": "string", "index":"analyzed", "fields": { "city.keyword": { "type": "string", "index": "not_analyzed" } } } }
- 19. 19 { "city": { "type": "text" "fields":{ "city.keyword": { "type": "keyword" } } } }
- 20. 20 { "city": { "type": "text" "fields":{ "city.keyword": { "type": "keyword" } } } } Full text queries Full text analysis
- 21. 21 { "city": { "type": "text" "fields":{ "city.keyword": { "type": "keyword" } } } } Keyword queries Aggregations Sorting
- 22. 22 { "city": { "type": "text" "fields":{ "city.keyword": { "type": "keyword" } } } } No analysis
- 23. 23 San Francisco SAN FRANCISCO sanfrancisco San franciscO
- 24. 24 San Francisco SAN FRANCISCO sanfrancisco San franciscO san francisco Normalizer
- 25.
- 26.
- 27. 27 NY NYC New York New YorkCity } Synonyms
- 28.
- 29.
- 30.
- 31. 31 Synonym Filter: (ny|nyc|new), (is|york),(old,city) ny is old nyc new york city Synonym Graph Filter:
- 32.
- 33. 33 Query Optimizations • Smarterquery caching • Faster geo, range, and nested queries • Unified highlighter • Field collapsing • Cancellable searches • Partitioned term aggs
- 34.
- 35. 35 When your clusteris RED… /_cat/allocation /_cat/indices /_cat/nodes /_cat/recovery /_cat/shards /_cluster/health /_cluster/state /{index}/_shard_stores /_cluster/settings /_node/stats /{index}/_settings /_node
- 36. 36 When your clusteris RED… /_cluster/allocation/explain
- 37. 37 /_cluster/allocation/explain … "allocate_explanation" : "cannotallocate because allocation is not permitted to any of the nodes”, … { "decider" : "filter", "decision" : "NO", "explanation" : "node does not match index setting [index.routing.allocation.include] filters [_name:"non_existent_node"]" } …
- 38. 38 /_cluster/allocation/explain … "unassigned_info" : { "reason": "NODE_LEFT", "at" : "2017-01-04T18:03:28.464Z", "details" : "node_left[OIWe8UhhThCK0V5XfmdrmQ]", "last_allocation_status" : "no_valid_shard_copy" }, "can_allocate" : "no_valid_shard_copy", "allocate_explanation" : "cannot allocate because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster" …
- 39. 39 /_cluster/allocation/explain … "rebalance_explanation" : "cannotrebalance as no target node exists that can both allocate this shard and improve the cluster balance", "node_allocation_decisions" : [ { "node_id" : "oE3EGFc8QN-Tdi5FFEprIA", "node_name" : "node_t1", "transport_address" : "127.0.0.1:9401", "node_decision" : "worse_balance", "weight_ranking" : 1 } …
- 40.
- 41. 41 Java REST Client- behind the scenes • Came late to the party… • Isn’t nearly as extensive as the Transport Client • Maintaining a transport protocol based client causes a massive engineering overhead • It’s a “second” entry point into the system • Complicates distinguishing between clients and nodes
- 42. 42 Java low-level HTTPclient • Released in 5.0.0 • JSON strings only • Resilient, but not user friendly due to the lack of a higher level API
- 43. 43 Java high-level HTTPclient • IDE friendly • Similar API to Transport Client - easy migration • Based on low-level REST client • Support CRUD & Search • Previews in 5.5 • Depends on elasticsearch-core
- 44.
- 45. 45 How the TribeNode works Cluster Sales Master Nodes Data Node Data Node Data Node Cluster R&D Master Nodes Data Node Data Node Data Node
- 46. 46 How the TribeNode works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node Cluster R&D Master Nodes Data Node Data Node Data Node tribe: t1: cluster.name: sales t2: cluster.name: r_and_d
- 47. 47 How the TribeNode works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node
- 48. 48 How the TribeNode works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node t2 Node Client
- 49. 49 How the TribeNode works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node t2 Node ClientCluster State Cluster State
- 50. 50 How the TribeNode works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node t2 Node Client Cluster State Cluster State
- 51. 51 How the TribeNode works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node t2 Node Client Merged Cluster State
- 52. 52 How the TribeNode works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node t2 Node Client Merged Cluster State Kibana
- 53. 53 Problems With Howthe Tribe Node works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node t2 Node Client Merged Cluster State Kibana
- 54. 54 Problems With Howthe Tribe Node works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node t2 Node Client Merged Cluster State Kibana Static Configuration tribe: t1: cluster.name: sales t2: cluster.name: r_and_d
- 55. 55 Problems With Howthe Tribe Node works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node t2 Node Client Kibana Merged Cluster State Connections to All Nodes
- 56. 56 Problems With Howthe Tribe Node works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node t2 Node Client Kibana Merged Cluster State Frequent cluster state updates
- 57. 57 Problems With Howthe Tribe Node works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node t2 Node Client Kibana Merged Cluster State Index names must be unique
- 58. 58 Problems With Howthe Tribe Node works Cluster Sales Master Nodes Data Node Data Node Data Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node t2 Node Client Merged Cluster State Tribe Node Kibana No master node No index creation
- 59. 59 Problems With Howthe Tribe Node works Cluster Sales Master Nodes Data Node Data Node Data Node Tribe Node t1 Node Client Cluster R&D Master Nodes Data Node Data Node Data Node t2 Node Client Merged Cluster State Kibana Reduce results from many shards
- 60. The Tribe Nodeis Dead
- 61.
- 62. Minimal viable solution tosupersede tribe 62
- 63. Reduces the problemdomain to query execution 63
- 64. Cluster related informationis reduced to a namespace 64
- 65. 65 How Cross-Cluster searchworks Cluster Sales Master Nodes Data Node Data Node Data Node Cluster R&D Master Nodes Data Node Data Node Data Node
- 66. 66 How Cross-Cluster searchworks Cluster Sales Master Nodes Data Node Data Node Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Any node can perform cross-cluster search
- 67. 67 How Cross-Cluster searchworks Cluster Sales Master Nodes Data Node Data Node Data Node Optional dedicated cross-cluster search cluster Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node
- 68. 68 How Cross-Cluster searchworks Cluster Sales Master Nodes Data Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node PUT _cluster/settings { "transient": { "search.remote": { "sales.seeds": "10.0.0.1:9300", “r_and_d.seeds”: "10.1.0.1:9300" } } } Dynamic settings Optional dedicated cross-cluster search cluster
- 69. 69 How Cross-Cluster searchworks Cluster Sales Master Nodes Data Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node No cluster state updates Optional dedicated cross-cluster search cluster
- 70. 70 How Cross-Cluster searchworks Cluster Sales Master Nodes Data Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node Kibana Optional dedicated cross-cluster search cluster
- 71. 71 How Cross-Cluster searchworks Cluster Sales Master Nodes Data Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node Kibana Can create indices Optional dedicated cross-cluster search cluster
- 72. 72 How Cross-Cluster searchworks Cluster Sales Master Nodes Data Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node Kibana Optional dedicated cross-cluster search cluster
- 73. 73 How Cross-Cluster searchworks Cluster Sales Master Nodes Data Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node Kibana Few lightweight connections Optional dedicated cross-cluster search cluster
- 74. 74 How Cross-Cluster searchworks Cluster Sales Master Nodes Data Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node Kibana Index namespacing GET sales:*,r_and_d:logs*/_search { "query": { … } } Optional dedicated cross-cluster search cluster
- 75. 75 How Cross-Cluster searchworks Cluster Sales Master Nodes Data Node Data Node Data Node Master/Data Node Cluster R&D Master Nodes Data Node Data Node Data Node Master/Data Node Kibana With many shards Batched Reduce Phase Optional dedicated cross-cluster search cluster
- 76.
- 77.
- 78.
- 79.
- 80. 80 Doc Values • Columnarstore • Fast access to a field’s value for many documents. • Used for aggregations, sorting, scripting, and some queries • Written to disk at index time. • Cached in the file-system cache
- 81. 81 Doc Values -Dense Values Segment 1 Docs Field 1 Field 2 1 One A 2 Two B 3 Three C Segment 2 Docs Field 1 Field 2 1 Four D
- 82. 82 Doc Values -Dense Values Segment 1 Docs Field 1 Field 2 1 One A 2 Two B 3 Three C Segment 2 Docs Field 1 Field 2 1 Four D Merged Segment 3 Docs Field 1 Field 2 1 One A 2 Two B 3 Three C 4 Four D
- 83. 83 Doc Values -Sparse Values Segment 1 Docs Field 1 Field 2 1 One A 2 Two B 3 Three C Segment 2 Docs Field 3 Field 4 Field 5 1 Foo Bar Baz
- 84. 84 Doc Values -Sparse Values Segment 1 Docs Field 1 Field 2 1 One A 2 Two B 3 Three C Segment 2 Docs Field 3 Field 4 Field 5 1 Foo Bar Baz Merged Segment 3 Docs Field 1 Field 2 Field 3 Field 4 Field 5 1 One A Null Null Null 2 Two B Null Null Null 3 Three C Null Null Null 4 Null Null Foo Bar Baz
- 85.
- 86.
- 87. 87 Index sorting • Sortindex by e.g. weight, recency, or popularity • Ultra-fast search - can terminate once enough hits found
- 88. 88 Index sorting • Sortindex by e.g. weight, recency, or popularity • Ultra-fast search - can terminate once enough hits found • Even helps with total count and aggregations • Sort index by low cardinality terms - faster search • Better sparse index compression • Slower indexing, good for static indices
- 89.
- 90. 90 Sequence Numbers • InternalFeature • Every operation gets a sequence number • In 6.0: Fast replica recovery on active indices • Lays groundwork for: • Primary-Replica syncing when Primary fails • Cross Data-Centre Recovery • Changes API
- 91.
- 92.
- 93. 93 Rolling Upgrades • Upgradefrom 5.latest to 6.latest, without a full cluster restart • Why now and not earlier? • Testing needs to be ready • The team and the code must be ready • Growing user-base and faster release cycles required less painful upgrades
- 94. 94 Rolling Upgrades • Whatis 5.latest? • It’s the latest release of 5.x that is GA once 6.0.0 goes GA • All 6.x releases will allow upgrading from that 5.x release • There might be subsequent 5.x releases that are also eligible for upgrades to 6.x
- 95. 95 Rolling Upgrades • Caveats: •If using security, must have TLS enabled • Reserve the right to require full cluster restart in the future - but only if absolutely necessary • All nodes must be upgraded to 5.latest in order to upgrade • Indices created in 2.x still need to be reindexed before upgrading to 6.x
- 96. Cross Major VersionSearch v6.0.0
- 97. 97 Cross Major VersionSearch v5.2.0 Kibana Master Nodes Data Node Data Node
- 98. 98 Cross Major VersionSearch v5.2.0 Kibana v6.0.0 Master Nodes Data Node Data Node Master Nodes Data Node Data Node
- 99. 99 Cross Major VersionSearch v5.2.0 Master Nodes Data Node Data Node v6.0.0 v5.latest Kibana Master Nodes Data Node Data Node
- 100. 100 Cross Major VersionSearch v5.2.0 Master Nodes Data Node Data Node v6.0.0 Kibana Master Nodes Data Node Cross Cluster Client v5.latest
- 101.
- 102. 102 Elasticsearch SQL • SupportElasticsearch features as much as possible • Support standard SQL as much as possible • Same experience out of the box as Elasticsearch, which means • Lightweight & fast • Easy to pick up • Still Elasticsearch :)
- 103.
- 104.
- 105.
- 106.
- 107.
- 108.
- 109.
- 110.
- 111.
- 112.
- 113. 113 Logstash Monitoring • Newvisualization application on top of Elasticsearch data • Use Case: • live infographics • presentations with live data feeds • highly customized reports • Currently, in the prototyping phase • Release date: TBD
- 114.
- 115.
- 116. 116 Beats Packetbeat Network data Filebeat Log files Winlogbeat WindowsEvent Logs Heartbeat Uptime monitoring +40 community Beats Metricbeat Metrics
- 117.
- 118. 118 Heartbeat host! ! !Your app! OS! TCP/TLS connect! ICMPping! HTTP/S request!
- 119.
- 120. 120 Metricbeat modules MySQL Memcac he PHP- FPM CEPH Zoo keeper GolangDocker Apache Kafka HAProx y SystemRedis Couchb ase NGINX Postgre s Prometh eus Jolokia
- 121.
- 122.
- 123.
- 124.
- 125. 125 Persistent Queues • Survive(temporary) machine failures • Limited impact on performance • View stats in Monitoring UI
- 126.
- 127.
- 128.
- 129.