Downloaded 79 times
![Playing with ElasticSearch
Distributed Model
• Number of shards is the scaling unit [ #shards > #nodes ]
• each one is a separate Lucene index thus, many per-index settings are available
• Moving shards around is faster than splitting them (no reindex)
• Replicas also serves reads, allowing to scale search
• # of replicas can be updated dynamically after index creation
Node 1
user (0)
user (1)
Node 2
user1 (0)
user (1)
Node 3
user (0)
user2 (0)
Automatic Discovery Protocol
Replica
Shard
Tuesday, 12 March 13](https://image.slidesharecdn.com/elasticsearchv5-130903113316-/75/elasticsearch-14-2048.jpg)
![Playing with ElasticSearch
Index Aliases
curl -X POST 'http://localhost:9200/_aliases' -d '{
"actions" : [
{
"add" : {
“index” : “users”,
“alias” : “user_1”,
“filter” : { “term” : { “user” : “1” } },
“routing” : “1”
}
} ]
}'
Indexing and search happens on the alias, with automatic use of routing and filtering
Tuesday, 12 March 13](https://image.slidesharecdn.com/elasticsearchv5-130903113316-/75/elasticsearch-15-2048.jpg)
![Playing with ElasticSearch
Index Aliases
curl -X POST 'http://localhost:9200/_aliases' -d
' {
"actions" : [ { "add" : {
"index" : "user_1",
"alias" : "users"
}
},
{ "add" : {
"index" : "user_2",
"alias" : "users"
}
} ]
}'
users
user_1
user_2
curl -X GET "http://localhost:9200/users/_search?q=..."
Tuesday, 12 March 13](https://image.slidesharecdn.com/elasticsearchv5-130903113316-/75/elasticsearch-16-2048.jpg)
![REST API : http://host:port/[index]/[type]/[_action/id]
HTTP Methods : GET, POST, PUT, DELETE
Playing with ElasticSearch
Tuesday, 12 March 13](https://image.slidesharecdn.com/elasticsearchv5-130903113316-/75/elasticsearch-23-2048.jpg)
![REST API : http://host:port/[index]/[type]/[_action/id]
HTTP Methods : GET, POST, PUT, DELETE
Playing with ElasticSearch
Some Definitions.....
• index -> Like a database
• type -> Like a table
• id -> Like a row in a table
Tuesday, 12 March 13](https://image.slidesharecdn.com/elasticsearchv5-130903113316-/75/elasticsearch-24-2048.jpg)
![Playing with ElasticSearch
REST API : http://host:port/[index]/[type]/_action/id]
HTTP Methods : GET, POST, PUT, DELETE
curl -X POST "http://localhost:9200/articles/article/1" -d '
{
"title" : "ElasticSearch Understands JSON!",
"body" : "ElasticSearch not only “works” with JSON, it understands it! Let’s first ...",
"published_on" : "2013/02/06 10:00:00",
"tags" : ["search", "json"],
"author" : {
"first_name" : "Bruce",
"last_name" : "Croft",
"email" : "bruce@croft.org"
}
}'
request
Tuesday, 12 March 13](https://image.slidesharecdn.com/elasticsearchv5-130903113316-/75/elasticsearch-25-2048.jpg)
![Playing with ElasticSearch
REST API : http://host:port/[index]/[type]/_action/id]
HTTP Methods : GET, POST, PUT, DELETE
curl -X POST "http://localhost:9200/articles/article/1" -d '
{
"title" : "ElasticSearch Understands JSON!",
"body" : "ElasticSearch not only “works” with JSON, it understands it! Let’s first ...",
"published_on" : "2013/02/06 10:00:00",
"tags" : ["search", "json"],
"author" : {
"first_name" : "Bruce",
"last_name" : "Croft",
"email" : "bruce@croft.org"
}
}'
{
"ok":true,
"_index":"articles",
"_type":"article",
"_id":"1",
"_version":1
}
requestresponse
Tuesday, 12 March 13](https://image.slidesharecdn.com/elasticsearchv5-130903113316-/75/elasticsearch-26-2048.jpg)
![Playing with ElasticSearch
REST API : http://host:port/[index]/[type]/_action/id]
HTTP Methods : GET, POST, PUT, DELETE
request
curl -X GET "http://localhost:9200/articles/_search?q=author.first_name:BRUCE"
Tuesday, 12 March 13](https://image.slidesharecdn.com/elasticsearchv5-130903113316-/75/elasticsearch-27-2048.jpg)
![Playing with ElasticSearch
REST API : http://host:port/[index]/[type]/_action/id]
HTTP Methods : GET, POST, PUT, DELETE
{
"took":1,
"timed_out":false,
"_shards":{"total":5,"successful":5,"failed":0},
"hits":{
"total":1,
"max_score":0.30685282,
"hits":[{
"_index":"articles",
"_type":"article",
"_id":"1",
"_score":0.30685282,
"_source" :
{
"title" : "ElasticSearch Understands JSON!",
"body" : "ElasticSearch not only “works” with JSON, it understands it! Let’s first ...",
"published_on" : "2013/02/06 10:00:00",
"tags" : ["search", "json"],
"author" : {
"first_name" : "Bruce",
"last_name" : "Croft",
"email" : "bruce@croft.org"
}
} } ] } }
request
curl -X GET "http://localhost:9200/articles/_search?q=author.first_name:BRUCE"
response
Tuesday, 12 March 13](https://image.slidesharecdn.com/elasticsearchv5-130903113316-/75/elasticsearch-28-2048.jpg)
![Playing with ElasticSearch
REST API : http://host:port/[index]/[type]/_action/id]
HTTP Methods : GET, POST, PUT, DELETE
{
"took":1,
"timed_out":false,
"_shards":{"total":5,"successful":5,"failed":0},
"hits":{
"total":1,
"max_score":0.30685282,
"hits":[{
"_index":"articles",
"_type":"article",
"_id":"1",
"_score":0.30685282,
"_source" :
{
"title" : "ElasticSearch Understands JSON!",
"body" : "ElasticSearch not only “works” with JSON, it understands it! Let’s first ...",
"published_on" : "2013/02/06 10:00:00",
"tags" : ["search", "json"],
"author" : {
"first_name" : "Bruce",
"last_name" : "Croft",
"email" : "bruce@croft.org"
}
}
} ] } }
request
curl -X GET "http://localhost:9200/articles/_search?q=author.first_name:BRUCE"
response
Location & ID
Document Source
Total number of documents
Tuesday, 12 March 13](https://image.slidesharecdn.com/elasticsearchv5-130903113316-/75/elasticsearch-29-2048.jpg)
Uploaded bySatish Mohan
elasticsearch
The document discusses ElasticSearch, an open source search engine and database. It describes how ElasticSearch allows data to flow from various sources into an index using Rivers. It also explains key ElasticSearch concepts like shards, replicas, and index aliases that improve scalability and performance. The document provides examples of ElasticSearch REST API calls for indexing, searching, and retrieving documents.
elasticsearch
- 1. Satish Mohan Your Data, YourSearch Tuesday, 12 March 13
- 2. Enterprises today arecollecting and have access to more data points in their ecosystem then ever. Tuesday, 12 March 13
- 3. File Store Example File/ Folder Navigation Integration - Mount Points Limited Metadata Hierarchical Structure Regular File Store Tuesday, 12 March 13
- 4. • Find adocument from December 2011 about transfer containing proposal and David • Find the document received from John containing David and transfer • Find the revisions of transfer document File Store Example File / Folder Navigation Integration - Mount Points Limited Metadata Hierarchical Structure Tuesday, 12 March 13
- 5. • Find adocument from December 2011 about transfer containing proposal and David • Find the document received from John containing David and transfer • Find the revisions of transfer document File Store Example File / Folder Navigation Integration - Mount Points Limited Metadata Hierarchical Structure Collections / Documents Local / Distributed Integrations Semantic Metadata Declarative Queries Automatic Indexing Provenance Automatic Organization Virtual Collections Regular File Store Intelligent File Store Tuesday, 12 March 13
- 6. ElasticSearch is anopen source, scalable, distributed, cloud-ready, highly-available full-text search engine and database with powerful aggregation features, communicating by JSON over RESTful HTTP, based on Apache Lucene. Tuesday, 12 March 13
- 7. Structured Data Unstructured DataData Refinery Message Queues Inverted index Transaction Log Versioning Source Document Capture & Curate Index Streams Analyse Search MemoryShared FS FS + MemoryLocal FS Document Store Playing with ElasticSearch Tuesday, 12 March 13
- 8. Playing with ElasticSearch Rivers •Data flows from sources using Rivers • Continues to add data as it flows • Can be added, removed, configured dynamically Tuesday, 12 March 13
- 9. Playing with ElasticSearch Rivers •Data flows from sources using Rivers • Continues to add data as it flows • Can be added, removed, configured dynamically ES NodeData Source Data Source Data Source River River River ES Index Tuesday, 12 March 13
- 10. Playing with ElasticSearch Rivers •Data flows from sources using Rivers • Continues to add data as it flows • Can be added, removed, configured dynamically ES NodeData Source Data Source Data Source River River River ES Index Tuesday, 12 March 13
- 11. Playing with ElasticSearch RiverModules • CouchDB • JDBC • MongoDB • Solr • Wikipedia • Jira • Twitter • CSV • ActiveMQ • FileSystem • RabbitMQ • SysInfo • NSQ • Logs • RSS • LDAP Tuesday, 12 March 13
- 12. Playing with ElasticSearch Index •Describes document structure to the search engine • Automatically created with sensible defaults • Explicit mapping can be provided (generally, a good idea) • Simple: • string, integer/long, float/double, boolean, and null • Complex: • array, object Tuesday, 12 March 13
- 13. Structured Data Unstructured DataData Refinery Message Queues Inverted index Transaction Log Versioning Source Document Shards Replication Load Balancing Nodes Distributed Capture & Curate Index Streams Analyse Search MemoryShared FS FS + MemoryLocal FS Document Store Playing with ElasticSearch Tuesday, 12 March 13
- 14. Playing with ElasticSearch DistributedModel • Number of shards is the scaling unit [ #shards > #nodes ] • each one is a separate Lucene index thus, many per-index settings are available • Moving shards around is faster than splitting them (no reindex) • Replicas also serves reads, allowing to scale search • # of replicas can be updated dynamically after index creation Node 1 user (0) user (1) Node 2 user1 (0) user (1) Node 3 user (0) user2 (0) Automatic Discovery Protocol Replica Shard Tuesday, 12 March 13
- 15. Playing with ElasticSearch IndexAliases curl -X POST 'http://localhost:9200/_aliases' -d '{ "actions" : [ { "add" : { “index” : “users”, “alias” : “user_1”, “filter” : { “term” : { “user” : “1” } }, “routing” : “1” } } ] }' Indexing and search happens on the alias, with automatic use of routing and filtering Tuesday, 12 March 13
- 16. Playing with ElasticSearch IndexAliases curl -X POST 'http://localhost:9200/_aliases' -d ' { "actions" : [ { "add" : { "index" : "user_1", "alias" : "users" } }, { "add" : { "index" : "user_2", "alias" : "users" } } ] }' users user_1 user_2 curl -X GET "http://localhost:9200/users/_search?q=..." Tuesday, 12 March 13
- 17. Structured Data Unstructured DataData Refinery Message Queues Inverted index Transaction Log Versioning Source Document Shards Replication Load Balancing Nodes Distributed Capture & Curate Index Streams Analyse Search Transport HTTP WebSockets Thrift ZeroMQ memcached TCP MemoryShared FS FS + MemoryLocal FS Document Store Playing with ElasticSearch Tuesday, 12 March 13
- 18. Structured Data Unstructured DataData Refinery Message Queues Inverted index Transaction Log Versioning Source Document Data Sources Tokenisers Retrieval Models Structured Results Language Bindings Transport Shards Replication Load Balancing Nodes Distributed Capture & Curate Index Streams Analyse Search Transport HTTP WebSockets Thrift ZeroMQ memcached TCP Modules Extend MemoryShared FS FS + MemoryLocal FS Document Store Playing with ElasticSearch Tuesday, 12 March 13
- 19. Structured Data Unstructured DataData Refinery Message Queues Inverted index Transaction Log Versioning Source Document Data Sources Tokenisers Retrieval Models Structured Results Language Bindings Transport Shards Replication Load Balancing Nodes Distributed Zen EC2 Capture & Curate Index Streams Analyse Search Transport HTTP WebSockets Thrift ZeroMQ memcached TCP Modules Extend Discovery MemoryShared FS FS + MemoryLocal FS Document Store Playing with ElasticSearch Tuesday, 12 March 13
- 20. Structured Data Unstructured DataData Refinery Message Queues Inverted index Transaction Log Versioning Source Document Data Sources Tokenisers Retrieval Models Structured Results Language Bindings Transport Shards Replication Load Balancing Nodes Distributed Zen EC2 mvel Python Groovy Javascript Capture & Curate Index Streams Analyse Search Transport HTTP WebSockets Thrift ZeroMQ memcached TCP Modules Extend Script Discovery MemoryShared FS FS + MemoryLocal FS Document Store Playing with ElasticSearch Tuesday, 12 March 13
- 21. Structured Data Unstructured DataData Refinery Message Queues Inverted index Transaction Log Versioning Source Document Data Sources Tokenisers Retrieval Models Structured Results Language Bindings Transport Shards Replication Load Balancing Nodes Distributed Zen EC2 mvel Python Groovy Javascript Capture & Curate Index Streams Analyse Search Transport HTTP WebSockets Thrift ZeroMQ memcached TCP Modules Extend Script Monitor Discovery MemoryShared FS FS + MemoryLocal FS Document Store Playing with ElasticSearch Tuesday, 12 March 13
- 22. Structured Data Unstructured DataData Refinery Message Queues Inverted index Transaction Log Versioning Source Document Data Sources Tokenisers Retrieval Models Structured Results Language Bindings Transport Shards Replication Load Balancing Nodes Distributed Zen EC2 mvel Python Groovy Javascript Capture & Curate Index Streams Analyse Search Transport HTTP WebSockets Thrift ZeroMQ memcached TCP Modules Extend Script Monitor Discovery RESTful MemoryShared FS FS + MemoryLocal FS Document Store Playing with ElasticSearch Tuesday, 12 March 13
- 23. REST API :http://host:port/[index]/[type]/[_action/id] HTTP Methods : GET, POST, PUT, DELETE Playing with ElasticSearch Tuesday, 12 March 13
- 24. REST API :http://host:port/[index]/[type]/[_action/id] HTTP Methods : GET, POST, PUT, DELETE Playing with ElasticSearch Some Definitions..... • index -> Like a database • type -> Like a table • id -> Like a row in a table Tuesday, 12 March 13
- 25. Playing with ElasticSearch RESTAPI : http://host:port/[index]/[type]/_action/id] HTTP Methods : GET, POST, PUT, DELETE curl -X POST "http://localhost:9200/articles/article/1" -d ' { "title" : "ElasticSearch Understands JSON!", "body" : "ElasticSearch not only “works” with JSON, it understands it! Let’s first ...", "published_on" : "2013/02/06 10:00:00", "tags" : ["search", "json"], "author" : { "first_name" : "Bruce", "last_name" : "Croft", "email" : "bruce@croft.org" } }' request Tuesday, 12 March 13
- 26. Playing with ElasticSearch RESTAPI : http://host:port/[index]/[type]/_action/id] HTTP Methods : GET, POST, PUT, DELETE curl -X POST "http://localhost:9200/articles/article/1" -d ' { "title" : "ElasticSearch Understands JSON!", "body" : "ElasticSearch not only “works” with JSON, it understands it! Let’s first ...", "published_on" : "2013/02/06 10:00:00", "tags" : ["search", "json"], "author" : { "first_name" : "Bruce", "last_name" : "Croft", "email" : "bruce@croft.org" } }' { "ok":true, "_index":"articles", "_type":"article", "_id":"1", "_version":1 } requestresponse Tuesday, 12 March 13
- 27. Playing with ElasticSearch RESTAPI : http://host:port/[index]/[type]/_action/id] HTTP Methods : GET, POST, PUT, DELETE request curl -X GET "http://localhost:9200/articles/_search?q=author.first_name:BRUCE" Tuesday, 12 March 13
- 28. Playing with ElasticSearch RESTAPI : http://host:port/[index]/[type]/_action/id] HTTP Methods : GET, POST, PUT, DELETE { "took":1, "timed_out":false, "_shards":{"total":5,"successful":5,"failed":0}, "hits":{ "total":1, "max_score":0.30685282, "hits":[{ "_index":"articles", "_type":"article", "_id":"1", "_score":0.30685282, "_source" : { "title" : "ElasticSearch Understands JSON!", "body" : "ElasticSearch not only “works” with JSON, it understands it! Let’s first ...", "published_on" : "2013/02/06 10:00:00", "tags" : ["search", "json"], "author" : { "first_name" : "Bruce", "last_name" : "Croft", "email" : "bruce@croft.org" } } } ] } } request curl -X GET "http://localhost:9200/articles/_search?q=author.first_name:BRUCE" response Tuesday, 12 March 13
- 29. Playing with ElasticSearch RESTAPI : http://host:port/[index]/[type]/_action/id] HTTP Methods : GET, POST, PUT, DELETE { "took":1, "timed_out":false, "_shards":{"total":5,"successful":5,"failed":0}, "hits":{ "total":1, "max_score":0.30685282, "hits":[{ "_index":"articles", "_type":"article", "_id":"1", "_score":0.30685282, "_source" : { "title" : "ElasticSearch Understands JSON!", "body" : "ElasticSearch not only “works” with JSON, it understands it! Let’s first ...", "published_on" : "2013/02/06 10:00:00", "tags" : ["search", "json"], "author" : { "first_name" : "Bruce", "last_name" : "Croft", "email" : "bruce@croft.org" } } } ] } } request curl -X GET "http://localhost:9200/articles/_search?q=author.first_name:BRUCE" response Location & ID Document Source Total number of documents Tuesday, 12 March 13
- 30. Structured Data Unstructured DataData Refinery Message Queues Inverted index Transaction Log Versioning Source Document Data Sources Tokenisers Retrieval Models Structured Results Language Bindings Transport Shards Replication Load Balancing Nodes Distributed Zen EC2 mvel Python Groovy Javascript Capture & Curate Index Streams Analyse Search Transport HTTP WebSockets Thrift ZeroMQ memcached TCP Modules Extend Script Monitor Discovery RESTful Micro Apps MemoryShared FS FS + MemoryLocal FS Document Store Playing with ElasticSearch Tuesday, 12 March 13
- 31. Structured Data Unstructured DataData Refinery Message Queues Inverted index Transaction Log Versioning Source Document Data Sources Tokenisers Retrieval Models Structured Results Language Bindings Transport Shards Replication Load Balancing Nodes Distributed Zen EC2 mvel Python Groovy Javascript HTML5/CSS3 Javascript Capture & Curate Index Streams Analyse Search Transport HTTP WebSockets Thrift ZeroMQ memcached TCP Modules Extend Script Monitor Discovery RESTful Micro Apps MemoryShared FS FS + MemoryLocal FS Document Store Playing with ElasticSearch Tuesday, 12 March 13
- 32. Micro Applications Rich, interactivesingle-page web applications powered by JavaScript, HTML and CSS. Tuesday, 12 March 13
- 33. Micro Applications Tuesday, 12March 13
- 34. Micro Applications Tuesday, 12March 13
- 35. Micro Applications Tuesday, 12March 13
- 36. Micro Applications Tuesday, 12March 13
- 37. Micro Applications Rich, interactivesingle-page web applications powered by JavaScript, HTML and CSS. • A self-described framework for ambitious applications • Rails-inspired “convention over configuration” approach • High level abstractions, two-way binding and auto-updating templates Data Model ControllerRouter View Model Model Controller View View View Tuesday, 12 March 13
- 38. Micro Applications Rich, interactivesingle-page web applications powered by JavaScript, HTML and CSS. • A self-described framework for ambitious applications • Rails-inspired “convention over configuration” approach • High level abstractions, two-way binding and auto-updating templates • Ember Data • Client side storage adapter • Provides a common interface to persist application data • RESTful HTTP service - primary endpoint • Browser’s localStorage • Emerging web databases such as IndexedDB Data Model ControllerRouter View Model Model Controller View View View Tuesday, 12 March 13
- 39. Playing with ElasticSearch MoreFeatures..... • document oriented • load balancing • versioning • plugins • parent/child docs • more_like_this • scripting • multi_field mapping • dynamic mapping templates • percolation • bulk indexing • facets • geo location • index aliases • auto-complete • ngrams & edge-ngrams • histograms • rivers Tuesday, 12 March 13
- 40. Structured Data Unstructured DataData Refinery Message Queues Inverted index Transaction Log Versioning Source Document Data Sources Tokenisers Retrieval Models Structured Results Language Bindings Transport Shards Replication Load Balancing Nodes Distributed Zen EC2 mvel Python Groovy Javascript HTML5/CSS3 Javascript Capture & Curate Index Streams Analyse Search Transport HTTP WebSockets Thrift ZeroMQ memcached TCP Modules Extend Script Monitor Discovery RESTful Micro Apps MemoryShared FS FS + MemoryLocal FS Document Store An alternative that would allow scientists or even casual users to perform analysis of distributed data regardless of where the data resides. Tuesday, 12 March 13
- 41. Search is theprimary interface for getting information today. Let’s build on it. Search DiscoverAnalyse Tuesday, 12 March 13
- 42.
- 43. Data Management Tools- Challenges • Interactive queries, data exploration or iterative query refinement poses significant challenges for current methods • Building and running jobs and queries requires deep understanding of cluster size and structure, job performance, etc. • Time-consuming to set up, deploy and use Tuesday, 12 March 13