elasticsearch
Upcoming SlideShare
Loading in...5
×
 

elasticsearch

on

  • 1,809 views

Search-oriented architecture using elasticsearch

Search-oriented architecture using elasticsearch

Statistics

Views

Total Views
1,809
Slideshare-icon Views on SlideShare
1,784
Embed Views
25

Actions

Likes
6
Downloads
61
Comments
0

3 Embeds 25

http://localhost 14
http://www.linkedin.com 8
https://www.linkedin.com 3

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    elasticsearch elasticsearch Presentation Transcript

    • Satish Mohan Your Data, Your Search Tuesday, 12 March 13
    • Enterprises today are collecting and have access to more data points in their ecosystem then ever. Tuesday, 12 March 13
    • File Store Example File / Folder Navigation Integration - Mount Points Limited Metadata Hierarchical Structure Regular File Store Tuesday, 12 March 13
    • • Find a document from December 2011 about transfer containing proposal and David • Find the document received from John containing David and transfer • Find the revisions of transfer document File Store Example File / Folder Navigation Integration - Mount Points Limited Metadata Hierarchical Structure Tuesday, 12 March 13
    • • Find a document from December 2011 about transfer containing proposal and David • Find the document received from John containing David and transfer • Find the revisions of transfer document File Store Example File / Folder Navigation Integration - Mount Points Limited Metadata Hierarchical Structure Collections / Documents Local / Distributed Integrations Semantic Metadata Declarative Queries Automatic Indexing Provenance Automatic Organization Virtual Collections Regular File Store Intelligent File Store Tuesday, 12 March 13
    • ElasticSearch is an open source, scalable, distributed, cloud-ready, highly-available full-text search engine and database with powerful aggregation features, communicating by JSON over RESTful HTTP, based on Apache Lucene. Tuesday, 12 March 13
    • Structured Data Unstructured Data Data Refinery Message Queues Inverted index Transaction Log Versioning Source Document Capture & Curate Index Streams Analyse Search MemoryShared FS FS + MemoryLocal FS Document Store Playing with ElasticSearch Tuesday, 12 March 13
    • Playing with ElasticSearch Rivers • Data flows from sources using Rivers • Continues to add data as it flows • Can be added, removed, configured dynamically Tuesday, 12 March 13
    • Playing with ElasticSearch Rivers • Data flows from sources using Rivers • Continues to add data as it flows • Can be added, removed, configured dynamically ES NodeData Source Data Source Data Source River River River ES Index Tuesday, 12 March 13
    • Playing with ElasticSearch Rivers • Data flows from sources using Rivers • Continues to add data as it flows • Can be added, removed, configured dynamically ES NodeData Source Data Source Data Source River River River ES Index Tuesday, 12 March 13
    • Playing with ElasticSearch River Modules • CouchDB • JDBC • MongoDB • Solr • Wikipedia • Jira • Twitter • CSV • ActiveMQ • FileSystem • RabbitMQ • SysInfo • NSQ • Logs • RSS • LDAP Tuesday, 12 March 13
    • Playing with ElasticSearch Index • Describes document structure to the search engine • Automatically created with sensible defaults • Explicit mapping can be provided (generally, a good idea) • Simple: • string, integer/long, float/double, boolean, and null • Complex: • array, object Tuesday, 12 March 13
    • Structured Data Unstructured Data Data Refinery Message Queues Inverted index Transaction Log Versioning Source Document Shards Replication Load Balancing Nodes Distributed Capture & Curate Index Streams Analyse Search MemoryShared FS FS + MemoryLocal FS Document Store Playing with ElasticSearch Tuesday, 12 March 13
    • Playing with ElasticSearch Distributed Model • Number of shards is the scaling unit [ #shards > #nodes ] • each one is a separate Lucene index thus, many per-index settings are available • Moving shards around is faster than splitting them (no reindex) • Replicas also serves reads, allowing to scale search • # of replicas can be updated dynamically after index creation Node 1 user (0) user (1) Node 2 user1 (0) user (1) Node 3 user (0) user2 (0) Automatic Discovery Protocol Replica Shard Tuesday, 12 March 13
    • Playing with ElasticSearch Index Aliases curl -X POST 'http://localhost:9200/_aliases' -d '{ "actions" : [ { "add" : { “index” : “users”, “alias” : “user_1”, “filter” : { “term” : { “user” : “1” } }, “routing” : “1” } } ] }' Indexing and search happens on the alias, with automatic use of routing and filtering Tuesday, 12 March 13
    • Playing with ElasticSearch Index Aliases curl -X POST 'http://localhost:9200/_aliases' -d ' { "actions" : [ { "add" : { "index" : "user_1", "alias" : "users" } }, { "add" : { "index" : "user_2", "alias" : "users" } } ] }' users user_1 user_2 curl -X GET "http://localhost:9200/users/_search?q=..." Tuesday, 12 March 13
    • Structured Data Unstructured Data Data Refinery Message Queues Inverted index Transaction Log Versioning Source Document Shards Replication Load Balancing Nodes Distributed Capture & Curate Index Streams Analyse Search Transport HTTP WebSockets Thrift ZeroMQ memcached TCP MemoryShared FS FS + MemoryLocal FS Document Store Playing with ElasticSearch Tuesday, 12 March 13
    • Structured Data Unstructured Data Data Refinery Message Queues Inverted index Transaction Log Versioning Source Document Data Sources Tokenisers Retrieval Models Structured Results Language Bindings Transport Shards Replication Load Balancing Nodes Distributed Capture & Curate Index Streams Analyse Search Transport HTTP WebSockets Thrift ZeroMQ memcached TCP Modules Extend MemoryShared FS FS + MemoryLocal FS Document Store Playing with ElasticSearch Tuesday, 12 March 13
    • Structured Data Unstructured Data Data Refinery Message Queues Inverted index Transaction Log Versioning Source Document Data Sources Tokenisers Retrieval Models Structured Results Language Bindings Transport Shards Replication Load Balancing Nodes Distributed Zen EC2 Capture & Curate Index Streams Analyse Search Transport HTTP WebSockets Thrift ZeroMQ memcached TCP Modules Extend Discovery MemoryShared FS FS + MemoryLocal FS Document Store Playing with ElasticSearch Tuesday, 12 March 13
    • Structured Data Unstructured Data Data Refinery Message Queues Inverted index Transaction Log Versioning Source Document Data Sources Tokenisers Retrieval Models Structured Results Language Bindings Transport Shards Replication Load Balancing Nodes Distributed Zen EC2 mvel Python Groovy Javascript Capture & Curate Index Streams Analyse Search Transport HTTP WebSockets Thrift ZeroMQ memcached TCP Modules Extend Script Discovery MemoryShared FS FS + MemoryLocal FS Document Store Playing with ElasticSearch Tuesday, 12 March 13
    • Structured Data Unstructured Data Data Refinery Message Queues Inverted index Transaction Log Versioning Source Document Data Sources Tokenisers Retrieval Models Structured Results Language Bindings Transport Shards Replication Load Balancing Nodes Distributed Zen EC2 mvel Python Groovy Javascript Capture & Curate Index Streams Analyse Search Transport HTTP WebSockets Thrift ZeroMQ memcached TCP Modules Extend Script Monitor Discovery MemoryShared FS FS + MemoryLocal FS Document Store Playing with ElasticSearch Tuesday, 12 March 13
    • Structured Data Unstructured Data Data Refinery Message Queues Inverted index Transaction Log Versioning Source Document Data Sources Tokenisers Retrieval Models Structured Results Language Bindings Transport Shards Replication Load Balancing Nodes Distributed Zen EC2 mvel Python Groovy Javascript Capture & Curate Index Streams Analyse Search Transport HTTP WebSockets Thrift ZeroMQ memcached TCP Modules Extend Script Monitor Discovery RESTful MemoryShared FS FS + MemoryLocal FS Document Store Playing with ElasticSearch Tuesday, 12 March 13
    • REST API : http://host:port/[index]/[type]/[_action/id] HTTP Methods : GET, POST, PUT, DELETE Playing with ElasticSearch Tuesday, 12 March 13
    • REST API : http://host:port/[index]/[type]/[_action/id] HTTP Methods : GET, POST, PUT, DELETE Playing with ElasticSearch Some Definitions..... • index -> Like a database • type -> Like a table • id -> Like a row in a table Tuesday, 12 March 13
    • Playing with ElasticSearch REST API : http://host:port/[index]/[type]/_action/id] HTTP Methods : GET, POST, PUT, DELETE curl -X POST "http://localhost:9200/articles/article/1" -d ' { "title" : "ElasticSearch Understands JSON!", "body" : "ElasticSearch not only “works” with JSON, it understands it! Let’s first ...", "published_on" : "2013/02/06 10:00:00", "tags" : ["search", "json"], "author" : { "first_name" : "Bruce", "last_name" : "Croft", "email" : "bruce@croft.org" } }' request Tuesday, 12 March 13
    • Playing with ElasticSearch REST API : http://host:port/[index]/[type]/_action/id] HTTP Methods : GET, POST, PUT, DELETE curl -X POST "http://localhost:9200/articles/article/1" -d ' { "title" : "ElasticSearch Understands JSON!", "body" : "ElasticSearch not only “works” with JSON, it understands it! Let’s first ...", "published_on" : "2013/02/06 10:00:00", "tags" : ["search", "json"], "author" : { "first_name" : "Bruce", "last_name" : "Croft", "email" : "bruce@croft.org" } }' { "ok":true, "_index":"articles", "_type":"article", "_id":"1", "_version":1 } requestresponse Tuesday, 12 March 13
    • Playing with ElasticSearch REST API : http://host:port/[index]/[type]/_action/id] HTTP Methods : GET, POST, PUT, DELETE request curl -X GET "http://localhost:9200/articles/_search?q=author.first_name:BRUCE" Tuesday, 12 March 13
    • Playing with ElasticSearch REST API : http://host:port/[index]/[type]/_action/id] HTTP Methods : GET, POST, PUT, DELETE { "took":1, "timed_out":false, "_shards":{"total":5,"successful":5,"failed":0}, "hits":{ "total":1, "max_score":0.30685282, "hits":[{ "_index":"articles", "_type":"article", "_id":"1", "_score":0.30685282, "_source" : { "title" : "ElasticSearch Understands JSON!", "body" : "ElasticSearch not only “works” with JSON, it understands it! Let’s first ...", "published_on" : "2013/02/06 10:00:00", "tags" : ["search", "json"], "author" : { "first_name" : "Bruce", "last_name" : "Croft", "email" : "bruce@croft.org" } } } ] } } request curl -X GET "http://localhost:9200/articles/_search?q=author.first_name:BRUCE" response Tuesday, 12 March 13
    • Playing with ElasticSearch REST API : http://host:port/[index]/[type]/_action/id] HTTP Methods : GET, POST, PUT, DELETE { "took":1, "timed_out":false, "_shards":{"total":5,"successful":5,"failed":0}, "hits":{ "total":1, "max_score":0.30685282, "hits":[{ "_index":"articles", "_type":"article", "_id":"1", "_score":0.30685282, "_source" : { "title" : "ElasticSearch Understands JSON!", "body" : "ElasticSearch not only “works” with JSON, it understands it! Let’s first ...", "published_on" : "2013/02/06 10:00:00", "tags" : ["search", "json"], "author" : { "first_name" : "Bruce", "last_name" : "Croft", "email" : "bruce@croft.org" } } } ] } } request curl -X GET "http://localhost:9200/articles/_search?q=author.first_name:BRUCE" response Location & ID Document Source Total number of documents Tuesday, 12 March 13
    • Structured Data Unstructured Data Data Refinery Message Queues Inverted index Transaction Log Versioning Source Document Data Sources Tokenisers Retrieval Models Structured Results Language Bindings Transport Shards Replication Load Balancing Nodes Distributed Zen EC2 mvel Python Groovy Javascript Capture & Curate Index Streams Analyse Search Transport HTTP WebSockets Thrift ZeroMQ memcached TCP Modules Extend Script Monitor Discovery RESTful Micro Apps MemoryShared FS FS + MemoryLocal FS Document Store Playing with ElasticSearch Tuesday, 12 March 13
    • Structured Data Unstructured Data Data Refinery Message Queues Inverted index Transaction Log Versioning Source Document Data Sources Tokenisers Retrieval Models Structured Results Language Bindings Transport Shards Replication Load Balancing Nodes Distributed Zen EC2 mvel Python Groovy Javascript HTML5/CSS3 Javascript Capture & Curate Index Streams Analyse Search Transport HTTP WebSockets Thrift ZeroMQ memcached TCP Modules Extend Script Monitor Discovery RESTful Micro Apps MemoryShared FS FS + MemoryLocal FS Document Store Playing with ElasticSearch Tuesday, 12 March 13
    • Micro Applications Rich, interactive single-page web applications powered by JavaScript, HTML and CSS. Tuesday, 12 March 13
    • Micro Applications Tuesday, 12 March 13
    • Micro Applications Tuesday, 12 March 13
    • Micro Applications Tuesday, 12 March 13
    • Micro Applications Tuesday, 12 March 13
    • Micro Applications Rich, interactive single-page web applications powered by JavaScript, HTML and CSS. • A self-described framework for ambitious applications • Rails-inspired “convention over configuration” approach • High level abstractions, two-way binding and auto-updating templates Data Model ControllerRouter View Model Model Controller View View View Tuesday, 12 March 13
    • Micro Applications Rich, interactive single-page web applications powered by JavaScript, HTML and CSS. • A self-described framework for ambitious applications • Rails-inspired “convention over configuration” approach • High level abstractions, two-way binding and auto-updating templates • Ember Data • Client side storage adapter • Provides a common interface to persist application data • RESTful HTTP service - primary endpoint • Browser’s localStorage • Emerging web databases such as IndexedDB Data Model ControllerRouter View Model Model Controller View View View Tuesday, 12 March 13
    • Playing with ElasticSearch More Features..... • document oriented • load balancing • versioning • plugins • parent/child docs • more_like_this • scripting • multi_field mapping • dynamic mapping templates • percolation • bulk indexing • facets • geo location • index aliases • auto-complete • ngrams & edge-ngrams • histograms • rivers Tuesday, 12 March 13
    • Structured Data Unstructured Data Data Refinery Message Queues Inverted index Transaction Log Versioning Source Document Data Sources Tokenisers Retrieval Models Structured Results Language Bindings Transport Shards Replication Load Balancing Nodes Distributed Zen EC2 mvel Python Groovy Javascript HTML5/CSS3 Javascript Capture & Curate Index Streams Analyse Search Transport HTTP WebSockets Thrift ZeroMQ memcached TCP Modules Extend Script Monitor Discovery RESTful Micro Apps MemoryShared FS FS + MemoryLocal FS Document Store An alternative that would allow scientists or even casual users to perform analysis of distributed data regardless of where the data resides. Tuesday, 12 March 13
    • Search is the primary interface for getting information today. Let’s build on it. Search DiscoverAnalyse Tuesday, 12 March 13
    • Tuesday, 12 March 13
    • Data Management Tools - Challenges • Interactive queries, data exploration or iterative query refinement poses significant challenges for current methods • Building and running jobs and queries requires deep understanding of cluster size and structure, job performance, etc. • Time-consuming to set up, deploy and use Tuesday, 12 March 13