ElasticSearch at berlinbuzzwords 2010

You Know, For Search

lightning talk

lightning talk kick ass

lightning talk kick ass ~joke

ping 127.0.0.1

ping 127.0.0.1 icmq_seq=0 name=shay.banon

ping 127.0.0.1 icmq_seq=0 name=shay.banon icmq_seq=1 a.k.a=kimchy

ping 127.0.0.1 icmq_seq=0 name=shay.banon icmq_seq=1 a.k.a=kimchy icmq_seq=2 blog=http://kimchy.org

ping 127.0.0.1 icmq_seq=0 name=shay.banon icmq_seq=1 a.k.a=kimchy icmq_seq=2 blog=http://kimchy.org icmq_seq=3 twitter=kimchy

ping elasticsearch

ping elasticsearch icmq_seq=0 tag=open_source (apache)

ping elasticsearch icmq_seq=0 tag=open_source (apache) icmq_seq=1 tag=distributed

ping elasticsearch icmq_seq=0 tag=open_source (apache) icmq_seq=1 tag=distributed icmq_seq=2 tag=RESTful

ping elasticsearch icmq_seq=0 tag=open_source (apache) icmq_seq=1 tag=distributed icmq_seq=2 tag=RESTful icmq_seq=3 tag=(no_sql) search_engine

ping elasticsearch icmq_seq=0 tag=open_source (apache) icmq_seq=1 tag=distributed icmq_seq=2 tag=RESTful icmq_seq=3 tag=(no_sql) search_engine icmq_seq=4 site=http://www.elasticsearch.com

open_source

open_source apache_2

open_source apache_2 http://github.com/elasticsearch/elasticsearch

open_source apache_2 http://github.com/elasticsearch/elasticsearch fork, push, send_pull

your_data

your_data => your_search

your_data => your_search a product should natively talk the domain model

your_data => your_search a product should natively talk the domain model and not the other way around

recipes for good domain model language

recipes for good domain model language - document_oriented

recipes for good domain model language - document_oriented - schema free

recipes for good domain model language - document_oriented - schema free - include json

recipes for good domain model language - document_oriented - schema free - include json de_facto open_web standard

lets build ourself an amazon store

lets build ourself an amazon store (we will get to the cloud laters)

ﬁrst thing we have are books

this is how they look like { "book" : { "isbn" : "0812504321", "name" : "Call of the Wild", "author" : { "ﬁrst_name" : "Jack", "last_name" : "London" }, "pages" : 128, "tag" : ["ﬁction", "children"] } }

lets index it $ curl -XPUT host:9200/amazon/book/0812504321 -d ‘ { "book" : { "isbn" : "0812504321", "name" : "Call of the Wild", "author" : { "ﬁrst_name" : "Jack", "last_name" : "London" }, "pages" : 128, "tag" : ["ﬁction", "children"] } } ‘

... àmazon` is the _index name $ curl -XPUT host:9200/amazon/book/0812504321 -d ‘ { "book" : { "isbn" : "0812504321", "name" : "Call of the Wild", "author" : { "first_name" : "Jack", "last_name" : "London" }, "pages" : 128, "tag" : ["fiction", "children"] } } ‘

... `book` is the _type $ curl -XPUT host:9200/amazon/book/0812504321 -d ‘ { "book" : { "isbn" : "0812504321", "name" : "Call of the Wild", "author" : { "ﬁrst_name" : "Jack", "last_name" : "London" }, "pages" : 128, "tag" : ["ﬁction", "children"] } } ‘

... `0812504321` is the _id $ curl -XPUT host:9200/amazon/book/0812504321 -d ‘ { "book" : { "isbn" : "0812504321", "name" : "Call of the Wild", "author" : { "ﬁrst_name" : "Jack", "last_name" : "London" }, "pages" : 128, "tag" : ["ﬁction", "children"] } } ‘

search for all ﬁction books $ curl -XGET host:9200/amazon/book/_search?q=tag:ﬁction‘

search for all fiction books $ curl -XGET host:9200/amazon/book/_search?q=tag:fiction‘ {"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"hits": [{"_index":"amazon","_type":"book","_id":"0812504321", "_source" : { "book" : { "isbn" : "0812504321", "name" : "Call of the Wild", "author" : { "first_name" : "Jack", "last_name" : "London" }, "pages" : 128, "tag" : ["fiction", "children"] } }}]}}

... number of total hits $ curl -XGET host:9200/amazon/book/_search?q=tag:fiction‘ {"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"hits": [{"_index":"amazon","_type":"book","_id":"0812504321", "_source" : { "book" : { "isbn" : "0812504321", "name" : "Call of the Wild", "author" : { "first_name" : "Jack", "last_name" : "London" }, "pages" : 128, "tag" : ["fiction", "children"] } }}]}}

... each hit has the `_index` it came from $ curl -XGET host:9200/amazon/book/_search?q=tag:fiction‘ {"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"hits": [{"_index":"amazon","_type":"book","_id":"0812504321", "_source" : { "book" : { "isbn" : "0812504321", "name" : "Call of the Wild", "author" : { "first_name" : "Jack", "last_name" : "London" }, "pages" : 128, "tag" : ["fiction", "children"] } }}]}}

... each hit has the `_type` it belongs to $ curl -XGET host:9200/amazon/book/_search?q=tag:fiction‘ {"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"hits": [{"_index":"amazon","_type":"book","_id":"0812504321", "_source" : { "book" : { "isbn" : "0812504321", "name" : "Call of the Wild", "author" : { "first_name" : "Jack", "last_name" : "London" }, "pages" : 128, "tag" : ["fiction", "children"] } }}]}}

... each hit has its `_id` $ curl -XGET host:9200/amazon/book/_search?q=tag:fiction‘ {"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"hits": [{"_index":"amazon","_type":"book","_id":"0812504321", "_source" : { "book" : { "isbn" : "0812504321", "name" : "Call of the Wild", "author" : { "first_name" : "Jack", "last_name" : "London" }, "pages" : 128, "tag" : ["fiction", "children"] } }}]}}

... each hit has the actual _source document $ curl -XGET host:9200/amazon/book/_search?q=tag:fiction‘ {"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"hits": [{"_index":"amazon","_type":"book","_id":"0812504321", "_source" : { "book" : { "isbn" : "0812504321", "name" : "Call of the Wild", "author" : { "first_name" : "Jack", "last_name" : "London" }, "pages" : 128, "tag" : ["fiction", "children"] } }}]}}

... each hit has the actual _source document $ curl -XGET host:9200/amazon/book/_search?q=tag:fiction‘ {"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"hits": [{"_index":"amazon","_type":"book","_id":"0812504321", "_source" : { "book" : { "isbn" : "0812504321", "name" : "Call of the Wild", "author" : { "first_name" : "Jack", (can be disabled) "last_name" : "London" }, "pages" : 128, "tag" : ["fiction", "children"] } }}]}}

next, we have cds { "cd" : { "asin" : "B00192IV0O", "name" : "THE E.N.D. (Energy Never Dies)", "artist" : "Black Eyed Peas", "label" : "Interscope", "release_date": "2009-06-09", "tag" : ["hip-hop", "pop-rap"] } }

lets index it as well $ curl -XPUT host:9200/amazon/cd/B00192IV0O -d ‘ { "cd" : { "asin" : "B00192IV0O", "name" : "THE E.N.D. (Energy Never Dies)", "artist" : "Black Eyed Peas", "label" : "Interscope", "release_date": "2009-06-09", "tag" : ["hip-hop", "pop-rap"] } } ‘

... same _index, `amazon` $ curl -XPUT host:9200/amazon/cd/B00192IV0O -d ‘ { "cd" : { "asin" : "B00192IV0O", "name" : "THE E.N.D. (Energy Never Dies)", "artist" : "Black Eyed Peas", "label" : "Interscope", "release_date": "2009-06-09", "tag" : ["hip-hop", "pop-rap"] } } ‘

... new _type, `cd` $ curl -XPUT host:9200/amazon/cd/B00192IV0O -d ‘ { "cd" : { "asin" : "B00192IV0O", "name" : "THE E.N.D. (Energy Never Dies)", "artist" : "Black Eyed Peas", "label" : "Interscope", "release_date": "2009-06-09", "tag" : ["hip-hop", "pop-rap"] } } ‘

search for all hip-hop cds $ curl -XGET host:9200/amazon/cd/_search?q=tag:hip-hop‘

search across _type(s)

search across _type(s) search on both `cd` and `book` using `name` $ curl -XGET ‘host:9200/amazon/cd,book/_search?q=name:call’

search across _type(s) search on both `cd` and `book` using `name` $ curl -XGET ‘host:9200/amazon/cd,book/_search?q=name:call’ search on `_all` types using `name` $ curl -XGET ‘host:9200/amazon/_search?q=name:call’

search across _type(s) search on both `cd` and `book` using `name` $ curl -XGET ‘host:9200/amazon/cd,book/_search?q=name:call’ search on `_all` types using `name` $ curl -XGET ‘host:9200/amazon/_search?q=name:call’ search on specific fields types $ curl -XGET ‘host:9200/amazon/_search?q=book.author.first_name:jack OR cd.artist:jack‘

book and cd can be indices as well $ curl -XPUT host:9200/book/info/0812504321 -d ‘ { "isbn" : "0812504321", "name" : "Call of the Wild", "author" : { "ﬁrst_name" : "Jack", "last_name" : "London" }, "pages" : 128, "tag" : ["ﬁction", "children"] } ‘

... `book` is now the _index $ curl -XPUT host:9200/book/info/0812504321 -d ‘ { "isbn" : "0812504321", "name" : "Call of the Wild", "author" : { "ﬁrst_name" : "Jack", "last_name" : "London" }, "pages" : 128, "tag" : ["ﬁction", "children"] } ‘

... ìnfo` is the _type $ curl -XPUT host:9200/book/info/0812504321 -d ‘ { "isbn" : "0812504321", "name" : "Call of the Wild", "author" : { "first_name" : "Jack", "last_name" : "London" }, "pages" : 128, "tag" : ["fiction", "children"] } ‘

cd is an index as well $ curl -XPUT host:9200/cd/info/B00192IV0O -d ‘ { "asin" : "B00192IV0O", "name" : "THE E.N.D. (Energy Never Dies)", "artist" : "Black Eyed Peas", "label" : "Interscope", "release_date": "2009-06-09", "tag" : ["hip-hop", "pop-rap"] } ‘

search across _index(es)

search across _index(es) search on both `cd` and `book` using `name` $ curl -XGET ‘host:9200/cd,book/info/_search?q=name:call’

search across _index(es) search on both `cd` and `book` using `name` $ curl -XGET ‘host:9200/cd,book/info/_search?q=name:call’ search on `_all` indices using `name` $ curl -XGET ‘host:9200/_search?q=name:call’

query_dsl

query_dsl done with the your_data part

query_dsl done with the your_data part => now to the your_search part

... term_query { “term” : { “name” : “call” } }

... term_query, also with type { “term” : { “name” : “call” } } { “term” : { “book.name” : “call” } }

... range_query { “range” : { “pages” : { “from” : 200, “to” : 300 } } }

... range_query, also with type { “range” : { “pages” : { “from” : 200, “to” : 300 } } } { “range” : { “book.pages” : { “from” : 200, “to” : 300 } } }

... many more queries { “preﬁx” : { “book.name” : “ca” } }

... many more queries { “preﬁx” : { “book.name” : “ca” } } { “wildcard” : { “book.name” : “c*ll” } }

... many more queries { “preﬁx” : { { “book.name” : “ca” “ﬁeld” : { } “book.name” : “+call +wild” } } { } “wildcard” : { “book.name” : “c*ll” } }

... many more queries { “preﬁx” : { { “book.name” : “ca” “ﬁeld” : { } “book.name” : “+call +wild” } } { } “wildcard” : { “book.name” : “c*ll” } } { “query_string” : { “query” : “+call +wild” } }

... and, of course, bool_query { “bool” : { “must” : [ { “ﬁeld” : { “book.name” : “+call +wild” } }, { “range” : { “book.pages” : { “gte” : 200 } } } ], “must_not” : [ { “term” : { “book.author.ﬁrst_name” : “jack” } } ], “should” : [ { “term” : { “book.author.last_name” : “london” } } ] } }

ﬁlters faster than queries

ﬁlters faster than queries no scoring

ﬁlters faster than queries no scoring cached for even faster access

... sample ﬁlters (wrapped as constant_score query) { “constant_score” : { “ﬁlter” : { “term” : { “name” : “call” } } } }

... sample filters (wrapped as constant_score query) { “constant_score” : { “filter” : { “term” : { { “name” : “call” “constant_score” : { } “filter” : { } “range” : { } “book.pages” : { } “from” : 100, “to” : 200 } } } } }

... easily combined with queries { “ﬁltered” : { “query” : { “query_string” : { “query” : “+call +wild” } } “ﬁlter” : { “range” : { “book.pages” : { “from” : 100, “to” : 200 } } } } }

used in a search request $ curl -XPOST ‘host:9200/amazon/cd/_search‘ -d ‘ { “query” : { “ﬁeld” : { “name” : “+call + wild” } } } ‘

other search features

other search features highlighting

other search features highlighting facets

other search features highlighting retrieve speciﬁc ﬁelds facets

other search features highlighting retrieve specific fields facets _all field

other search features highlighting retrieve specific fields facets _all field scrolling

distributed

automatic shard allocation

... start 1st node node 1

... create an index with 2 shards, 1 replica node 1 PUT /amazon { “index.number_of_shards” : 2, “index.number_of_replicas” : 1 }

... create an index with 2 shards, 1 replica node 1 1 PUT /amazon { “index.number_of_shards” : 2, “index.number_of_replicas” : 1 }

... create an index with 2 shards, 1 replica node 1 1 2 PUT /amazon { “index.number_of_shards” : 2, “index.number_of_replicas” : 1 }

... start 2nd node node 1 node 2 1 1 2 2

... start 3rd and 4th node 1 node 2 node 3 node 4 1 1 2 2

... index a document node 1 node 2 node 3 node 4 1 1 2 2 PUT /amazon/book/1 { ... }

... index a document => hashed to 1st shard node 1 node 2 node 3 node 4 1 1 2 2 PUT /amazon/book/1 { ... }

... index a document => replicated node 1 node 2 node 3 node 4 1 1 2 2 PUT /amazon/book/1 { ... }

... index another document node 1 node 2 node 3 node 4 1 1 2 2 PUT /amazon/book/2 { ... }

... index another document => hashed to 2nd shard node 1 node 2 node 3 node 4 1 1 2 2 PUT /amazon/book/2 { ... }

... index another document => replicated node 1 node 2 node 3 node 4 1 1 2 2 PUT /amazon/book/2 { ... }

search node 1 node 2 node 3 node 4 1 1 2 2 GET /amazon/_search?q=name:call

search => scatter node 1 node 2 node 3 node 4 1 1 2 2 GET /amazon/_search?q=name:call

search => gather node 1 node 2 node 3 node 4 1 1 2 2 GET /amazon/_search?q=name:call

create another index node 1 node 2 node 3 node 4 1 1 2 2 PUT /amazon2 { “index.number_of_shards” : 1, “index.number_of_replicas” : 1 }

create another index node 1 node 2 node 3 node 4 1 1 1 1 2 2 PUT /amazon2 { “index.number_of_shards” : 1, “index.number_of_replicas” : 1 }

create another index node 1 node 2 node 3 node 4 1 1 1 1 2 2 almost all settings are index based

per document consistency

per document consistency - you index it, its there

per document consistency - you index it, its there - no need to commit / ﬂush

per document consistency - you index it, its there - no need to commit / ﬂush - uses a transaction log

(near) real_time search

(near) real_time search - automatic, 1 second refresh rate

(near) real_time search - automatic, 1 second refresh rate - there’s an api for that

(near) real_time search - automatic, 1 second refresh rate - there’s an api for that - POST /_refresh

long_term persistency

long_term persistency - similar to apple time_machine

long_term persistency - similar to apple time_machine - or data_grid write_behind

long_term persistency - similar to apple time_machine - or data_grid write_behind - write changes to index / translog

long_term persistency - similar to apple time_machine - or data_grid write_behind - write changes to index / translog - to a shared persistent storage

long_term persistency - similar to apple time_machine - or data_grid write_behind - write changes to index / translog - to a shared persistent storage - asynchronously (reliable)

long_term persistency - similar to apple time_machine - or data_grid write_behind - write changes to index / translog - to a shared persistent storage - asynchronously (reliable) - low requirements from storage

long_term persistency - similar to apple time_machine - or data_grid write_behind - write changes to index / translog - to a shared persistent storage - asynchronously (reliable) - low requirements from storage - cluster meta_data persisted

long_term persistency ... storage options

long_term persistency ... storage options - shared ﬁle system

long_term persistency ... storage options - shared ﬁle system - no need for locking, etc ...

long_term persistency ... storage options - shared ﬁle system - no need for locking, etc ... - hadoop, using HDFS

long_term persistency ... storage options - shared ﬁle system - no need for locking, etc ... - hadoop, using HDFS - cloud (aws_s3, rackspace_cloud_ﬁles)

long_term persistency ... node storage

long_term persistency ... node storage - considered transient

long_term persistency ... node storage - considered transient - can be recovered from gateway

long_term persistency ... node storage - considered transient - can be recovered from gateway - can be stored on

long_term persistency ... node storage - considered transient - can be recovered from gateway - can be stored on - local ﬁle system

long_term persistency ... node storage - considered transient - can be recovered from gateway - can be stored on - local ﬁle system - heap (jvm) memory

long_term persistency ... node storage - considered transient - can be recovered from gateway - can be stored on - local ﬁle system - heap (jvm) memory - native (os) memory

long_term persistency ... node storage - considered transient - can be recovered from gateway - can be stored on - local ﬁle system - heap (jvm) memory - native (os) memory - fs & memory combination

native cloud support

native cloud support in the cloud

native cloud support in the cloud machine removed / added more dynamically

native cloud support in the cloud machine removed / added more dynamically machine fail more “dynamically”

native cloud support in the cloud machine removed / added more dynamically machine fail more “dynamically” local storage is wiped

native cloud support ... storage

native cloud support ... storage - local storage is wiped

native cloud support ... storage - local storage is wiped - use external storage (aws ebs)

native cloud support ... storage - local storage is wiped - use external storage (aws ebs) - needs to be per machine / shard

native cloud support ... storage - local storage is wiped - use external storage (aws ebs) - needs to be per machine / shard - can get wiped as well ...

native cloud support ... storage - local storage is wiped - use external storage (aws ebs) - needs to be per machine / shard - can get wiped as well ... - mmm..., ok, snapshot to aws s3

native cloud support ... storage - local storage is wiped - use external storage (aws ebs) - needs to be per machine / shard - can get wiped as well ... - mmm..., ok, snapshot to aws s3 - expensive => ebs & s3

native cloud support ... storage - local storage is wiped - use external storage (aws ebs) - needs to be per machine / shard - can get wiped as well ... - mmm..., ok, snapshot to aws s3 - expensive => ebs & s3 - snapshot interval problematic

native cloud support ... the elastic way

native cloud support ... the elastic way - use long term persistency

native cloud support ... the elastic way - use long term persistency - directly into cloud blob storage

native cloud support ... the elastic way - use long term persistency - directly into cloud blob storage - aws s3, rackspace cloud_ﬁles

native cloud support ... the elastic way - use long term persistency - directly into cloud blob storage - aws s3, rackspace cloud_ﬁles - reliable & asynchronous

native cloud support ... discovery

native cloud support ... discovery - no multicast

native cloud support ... discovery - no multicast - resort to unicast discovery

native cloud support ... discovery - no multicast - resort to unicast discovery - who’s my special nodes ...

native cloud support ... discovery - no multicast - resort to unicast discovery - who’s my special nodes ... - require persistent ip’s

native cloud support ... discovery - no multicast - resort to unicast discovery - who’s my special nodes ... - require persistent ip’s - complicate operations

native cloud support ... the elastic way

native cloud support ... the elastic way - discovery support multicast & unicast

native cloud support ... the elastic way - discovery support multicast & unicast - but also support cloud discovery

native cloud support ... the elastic way - discovery support multicast & unicast - but also support cloud discovery - use cloud_provider API

native cloud support ... the elastic way - discovery support multicast & unicast - but also support cloud discovery - use cloud_provider API - ... to get the current list of nodes

ping end_session

ping end_session icmq_seq=0 desc=brief overview of elasticsearch

ping end_session icmq_seq=0 desc=brief overview of elasticsearch icmq_seq=1 version=current version is 0.8

ping end_session icmq_seq=0 desc=brief overview of elasticsearch icmq_seq=1 version=current version is 0.8 icmq_seq=2 notice=is beta

ping end_session icmq_seq=0 desc=brief overview of elasticsearch icmq_seq=1 version=current version is 0.8 icmq_seq=2 notice=is beta so expect bugs ...

ping end_session icmq_seq=0 desc=brief overview of elasticsearch icmq_seq=1 version=current version is 0.8 icmq_seq=2 notice=is beta so expect bugs ... they are actively ﬁxed

ping end_session icmq_seq=0 desc=brief overview of elasticsearch icmq_seq=1 version=current version is 0.8 icmq_seq=2 notice=is beta so expect bugs ... they are actively ﬁxed as new features are being added

ping end_session icmq_seq=0 desc=brief overview of elasticsearch icmq_seq=1 version=current version is 0.8 icmq_seq=2 notice=is beta so expect bugs ... they are actively ﬁxed as new features are being added icmq_seq=3 join=mailing list, suggestions, code

ping end_session icmq_seq=0 desc=brief overview of elasticsearch icmq_seq=1 version=current version is 0.8 icmq_seq=2 notice=is beta so expect bugs ... they are actively ﬁxed as new features are being added icmq_seq=3 join=mailing list, suggestions, code icmq_seq=4 thanks!

ping audience

http://log.medcl.net	163
http://www.slideshare.net	80
http://devvideos.com	37
http://es-bbs.medcl.net	37
http://iwiki.eur.ad.sag	14
http://www.progredi.com	8
http://adiefatlady.posterous.com	8
http://devslides.com	4
http://s.medcl.com	2
http://s.medcl.net	2
http://web.archive.org	2
http://noahinsider.com	2
http://facebook.slideshare.com	1
http://www.taaza.com	1
http://www.onlydoo.com	1
http://www.devslides.com	1
https://iwiki.eur.ad.sag	1

ElasticSearch at berlinbuzzwords 2010

by elasticsearch on Jun 13, 2010

Accessibility

Categories

Upload Details

Usage Rights

17 Embeds 364

Statistics

ElasticSearch at berlinbuzzwords 2010 Presentation Transcript