99 problems but the search aint one, confoo 2011, andrei zmievski

99 Problems, ButThe Search Ain’t OneAndrei Zmievski • ConFoo •!Mar 9, 2011

who am i? curl http://localhost:9200/speaker/info/andrei{“name”: “Andrei Zmievski”, “works”: “Analog Co-op”, “projects”: [“PHP”, “PHP-GTK”, “Smarty”, “Unicode/i18n”], “likes”: [“coding”, “beer”, “brewing”, “photography”], “twitter”: “@a”, “email”: “andrei@zmievski.org”}

what is elasticsearch?a search engine for the NoSQL generation domain-driven distributed RESTful Hitchhiker’s Guide to the Galaxy (no, really)

document modeldocument-orientedJSON-basedschema-free

enginebased on Lucenemulti-tenancydistributed, out of the box

3 easy steps

1. index !"#$%&()*+%,--./00$1!2$,13-/45660!17803.92:9#0;%&<= >request %%%%?72@9?/%?A7<#9B%C@B9D3:B?E %%%%?-2$:?/%?44%(#1F$9@3E%F"-%-,9%*92#!,%AB7==-%)79?E %%%%?$B:93?/%G?!1<B7H?E%?F99#?E%?.,1-1H#2.,I?JE %%%%?-KB--9#?/%?2?E %%%%?,9BH,-?/%;LM N= >response %%%%?1:?/-#"9 %%%%?OB7<9P?/?!178? %%%%?O-I.9?/?3.92:9#? %%%%?OB<?/?;? N

2. searchrequest !"#$%,--./00$1!2$,13-/45660!17803.92:9#0O392#!,QRSF99# >%?-11:?%/%TE %%?O3,2#<3?%/%> %%%%?-1-2$?%/%;E %%%%?3"!!9338"$?%/%;E %%%%?82B$9<?%/%6 %%NE %%?,B-3?%/%> %%%%?-1-2$?%/%;Eresponse %%%%?@2PO3!1#9?%/%6UV46LM64E %%%%?,B-3?%/%G%> %%%%%%?OB7<9P?%/%?!178?E %%%%%%?O-I.9?%/%?3.92:9#?E %%%%%%?OB<?%/%?;?E %%%%%%?O3!1#9?%/%6UV46LM64E %%%%%%?O31"#!9?%/% > %%%%?72@9?/%?A7<#9B%C@B9D3:B?E %%%%?-2$:?/%?44%(#1F$9@3E%F"-%-,9%*92#!,%AB7=-%)79?E %%%%?$B:93?/%G?!1<B7H?E%?F99#?E%?.,1-1H#2.,I?JE %%%%?-KB--9#?/%?2?E %%%%?,9BH,-?/%;LM N%N%J%N%N

2. searchrequest !"#$%,--./00$1!2$,13-/45660!17803.92:9#0O392#!,QRSF99# >%?-11:?%/%TE %%?O3,2#<3?%/%> %%%%?-1-2$?%/%;E %%%%?3"!!9338"$?%/%;E %%%%?82B$9<?%/%6 %%NE total number of hits %%?,B-3?%/%> !!!!"#$#%&"!!()response %%%%?@2PO3!1#9?%/%6UV46LM64E %%%%?,B-3?%/%G%> %%%%%%?OB7<9P?%/%?!178?E %%%%%%?O-I.9?%/%?3.92:9#?E %%%%%%?OB<?%/%?;?E %%%%%%?O3!1#9?%/%6UV46LM64E %%%%%%?O31"#!9?%/% > %%%%?72@9?/%?A7<#9B%C@B9D3:B?E %%%%?-2$:?/%?44%(#1F$9@3E%F"-%-,9%*92#!,%AB7=-%)79?E %%%%?$B:93?/%G?!1<B7H?E%?F99#?E%?.,1-1H#2.,I?JE %%%%?-KB--9#?/%?2?E %%%%?,9BH,-?/%;LM N%N%J%N%N

2. searchrequest !"#$%,--./00$1!2$,13-/45660!17803.92:9#0O392#!,QRSF99# >%?-11:?%/%TE %%?O3,2#<3?%/%> %%%%?-1-2$?%/%;E %%%%?3"!!9338"$?%/%;E %%%%?82B$9<?%/%6 %%NE %%?,B-3?%/%> %%%%?-1-2$?%/%;E the index of the docresponse %%%%?@2PO3!1#9?%/%6UV46LM64E %%%%?,B-3?%/%G%> !!!!!!"*+,-./"!!"0$,1") %%%%%%?O-I.9?%/%?3.92:9#?E %%%%%%?OB<?%/%?;?E %%%%%%?O3!1#9?%/%6UV46LM64E %%%%%%?O31"#!9?%/% > %%%%?72@9?/%?A7<#9B%C@B9D3:B?E %%%%?-2$:?/%?44%(#1F$9@3E%F"-%-,9%*92#!,%AB7=-%)79?E %%%%?$B:93?/%G?!1<B7H?E%?F99#?E%?.,1-1H#2.,I?JE %%%%?-KB--9#?/%?2?E %%%%?,9BH,-?/%;LM N%N%J%N%N

2. searchrequest !"#$%,--./00$1!2$,13-/45660!17803.92:9#0O392#!,QRSF99# >%?-11:?%/%TE %%?O3,2#<3?%/%> %%%%?-1-2$?%/%;E %%%%?3"!!9338"$?%/%;E %%%%?82B$9<?%/%6 %%NE %%?,B-3?%/%> %%%%?-1-2$?%/%;Eresponse %%%%?@2PO3!1#9?%/%6UV46LM64E %%%%?,B-3?%/%G%> the type of the doc %%%%%%?OB7<9P?%/%?!178?E !!!!!!"*#23."!!"43.%5.6") %%%%%%?OB<?%/%?;?E %%%%%%?O3!1#9?%/%6UV46LM64E %%%%%%?O31"#!9?%/% > %%%%?72@9?/%?A7<#9B%C@B9D3:B?E %%%%?-2$:?/%?44%(#1F$9@3E%F"-%-,9%*92#!,%AB7=-%)79?E %%%%?$B:93?/%G?!1<B7H?E%?F99#?E%?.,1-1H#2.,I?JE %%%%?-KB--9#?/%?2?E %%%%?,9BH,-?/%;LM N%N%J%N%N

2. searchrequest !"#$%,--./00$1!2$,13-/45660!17803.92:9#0O392#!,QRSF99# >%?-11:?%/%TE %%?O3,2#<3?%/%> %%%%?-1-2$?%/%;E %%%%?3"!!9338"$?%/%;E %%%%?82B$9<?%/%6 %%NE %%?,B-3?%/%> %%%%?-1-2$?%/%;Eresponse %%%%?@2PO3!1#9?%/%6UV46LM64E %%%%?,B-3?%/%G%> %%%%%%?OB7<9P?%/%?!178?E %%%%%%?O-I.9?%/%?3.92:9#?E !!!!!!"*+-"!!"(") the id of the doc %%%%%%?O3!1#9?%/%6UV46LM64E %%%%%%?O31"#!9?%/% > %%%%?72@9?/%?A7<#9B%C@B9D3:B?E %%%%?-2$:?/%?44%(#1F$9@3E%F"-%-,9%*92#!,%AB7=-%)79?E %%%%?$B:93?/%G?!1<B7H?E%?F99#?E%?.,1-1H#2.,I?JE %%%%?-KB--9#?/%?2?E %%%%?,9BH,-?/%;LM N%N%J%N%N

2. searchrequest !"#$%,--./00$1!2$,13-/45660!17803.92:9#0O392#!,QRSF99# >%?-11:?%/%TE %%?O3,2#<3?%/%> %%%%?-1-2$?%/%;E %%%%?3"!!9338"$?%/%;E %%%%?82B$9<?%/%6 %%NE %%?,B-3?%/%> %%%%?-1-2$?%/%;Eresponse %%%%?@2PO3!1#9?%/%6UV46LM64E %%%%?,B-3?%/%G%> %%%%%%?OB7<9P?%/%?!178?E %%%%%%?O-I.9?%/%?3.92:9#?E !!!!!!"*+-"!!"(") %%%%%%?O3!1#9?%/%6UV46LM64E the hit score %%%%%%?O31"#!9?%/% > %%%%?72@9?/%?A7<#9B%C@B9D3:B?E %%%%?-2$:?/%?44%(#1F$9@3E%F"-%-,9%*92#!,%AB7=-%)79?E %%%%?$B:93?/%G?!1<B7H?E%?F99#?E%?.,1-1H#2.,I?JE %%%%?-KB--9#?/%?2?E %%%%?,9BH,-?/%;LM N%N%J%N%N

2. searchrequest !"#$%,--./00$1!2$,13-/45660!17803.92:9#0O392#!,QRSF99# >%?-11:?%/%TE %%?O3,2#<3?%/%> %%%%?-1-2$?%/%;E %%%%?3"!!9338"$?%/%;E %%%%?82B$9<?%/%6 %%NE %%?,B-3?%/%> %%%%?-1-2$?%/%;Eresponse %%%%?@2PO3!1#9?%/%6UV46LM64E %%%%?,B-3?%/%G%> %%%%%%?OB7<9P?%/%?!178?E %%%%%%?O-I.9?%/%?3.92:9#?E !!!!!!"*+-"!!"(") %%%%%%?O3!1#9?%/%6UV46LM64E %%%%%%?O31"#!9?%/% the original doc contents 7 !!!!",%8."!"9,-6.+!:8+.;45+") !!!!"#%&5"!"<<!=6$>&.84)!>?#!#@.!A.%60@!9+,B#!C,.") !!!!"&+5.4"!D"0$-+,E")!">..6")!"3@$#$E6%3@2"F) !!!!"#G+##.6"!"%") !!!!"@.+E@#"!(HI J%N%J%N%N

2. searchrequest !"#$%,--./00$1!2$,13-/45660!17803.92:9#0O392#!,QRSF99# >%"#$$5"!!K) %%?O3,2#<3?%/%> %%%%?-1-2$?%/%;E the execution time %%%%?3"!!9338"$?%/%;E %%%%?82B$9<?%/%6 %%NE %%?,B-3?%/%> %%%%?-1-2$?%/%;Eresponse %%%%?@2PO3!1#9?%/%6UV46LM64E %%%%?,B-3?%/%G%> %%%%%%?OB7<9P?%/%?!178?E %%%%%%?O-I.9?%/%?3.92:9#?E %%%%%%?OB<?%/%?;?E %%%%%%?O3!1#9?%/%6UV46LM64E %%%%%%?O31"#!9?%/% > %%%%?72@9?/%?A7<#9B%C@B9D3:B?E %%%%?-2$:?/%?44%(#1F$9@3E%F"-%-,9%*92#!,%AB7=-%)79?E %%%%?$B:93?/%G?!1<B7H?E%?F99#?E%?.,1-1H#2.,I?JE %%%%?-KB--9#?/%?2?E %%%%?,9BH,-?/%;LM N%N%J%N%N

3. proﬁtthat’s up to you

distributed modelprovides: performance resiliency (high-availability)

shardsa portion of the document spaceeach one is a separate Lucene index thus, many per-index settings are availabledocument is sharded by its _id value but can be assigned (routed) to a shard deterministically

zero-conf discoveryzen (multicast and unicast)cloud (EC2 via API)

auto-routingmaster node: maintains cluster state reassigns shards if nodes leave/join clusterany node can process the search requestthe query is handled via scatter-gather mechanism

replicaseach shard can have 1 or more replicas# of replicas can be updated dynamically afterindex creationreplicas can be used for querying in parallel

shard allocation node 1 start with a single node

shard allocation node 1 person1 person2 PUT /person { “index”: { “number_of_shards”: 2, “number_of_replicas”: 1 }}

shard allocation node 1 node 2 person1 person1 person2 person2 start the second node

shard allocationnode 1 node 2 node 3 node 4person1 person1person2 person2 start 2 more nodes

shard allocationnode 1 node 2 node 3 node 4person1 person1 person2 person2 start 2 more nodes

document shardingnode 1 node 2 node 3 node 4person1 person1 person2 person2 PUT /person/info/1 {…}

document sharding node 1 node 2 node 3 node 4 person1 person1 person2 person2 PUT /person/info/1hashed to shard 1 {…}

document shardingnode 1 node 2 node 3 node 4person1 person1 person2 person2 replicated PUT /person/info/1 {…}

document shardingnode 1 node 2 node 3 node 4person1 person1 person2 person2 PUT /person/info/2 {…}

document shardingnode 1 node 2 node 3 node 4person1 person1 person2 person2hashed to shard 2 PUT /person/info/2 {…}

document shardingnode 1 node 2 node 3 node 4person1 person1 person2 person2 replicated PUT /person/info/2 {…}

scatter-gathernode 1 node 2 node 3 node 4person1 person1 person2 person2 GET /person/_search?q=name:thomas

shard allocationnode 1 node 2 node 3 node 4person1 person1 person2 person2 GET /person/_search?q=name:thomas

transactional modelper-document consistencyno need to commit/ﬂushuses write-ahead transaction logwrite consistency (W) can be controlled one, quorum, or all

(near) real-time search1 second refresh rate by default_refresh API also

index storagenode data considered transientcan be stored in local ﬁle system, JVM heap,native OS memory, or FS & memory combinationpersistent storage requires a gateway

gatewayspersistent store for cluster state and indicesasynchronous, translog-based write strategyallows full recovery if a cluster restart is neededsupported gateways: local shared FS Hadoop via HDFS S3

mappingdescribes document structure to the searchengineautomatically created with sensible defaultsexplicit mapping can be provided (generally, agood idea)can run into merge conﬂicts

mappingimportant meta ﬁelds: _source _allthere are more

mapping typessimple: string, integer/long, ﬂoat/double, boolean, and null)complex: array, object

sample mappingdocument >?"39#?/%%%%%%?<9#B!:?E %?-B-$9?/%%%%%?W17X-%(27B!?E %?-2H3?/%%%%%%G?.#18B$B7H?E%?<9F"HHB7H?E%?.,.?JE %?.13-W2-9?/%%?56;6&;5&55+;M/;Y/;5?E %?.#B1#B-I?/%%5N >?.13-?/%> %%?.#1.9#-B93?%/%>mapping %%%%?"39#?/%%%%%%>?-I.9?/%?3-#B7H?E%?B7<9P?/%?71-O272$IZ9<?NE %%%%?@9332H9?/%%%>?-I.9?/%?3-#B7H?E%[F113-/%;UVNE %%%%?-2H3?/%%%%%%>?-I.9?/%?3-#B7H?E%?B7!$"<9OB7O2$$?/%?71?NE %%%%?.13-W2-9?%/%>?-I.9?%/%?<2-9?E%[3-1#9/%[71NE %%%%?.#B1#B-I?%/%>?-I.9?%/%?B7-9H9#?N NNN

analyzersbreak down (tokenize) and normalize ﬁelds duringindexing and query strings at search timeanalyzer = tokenizer + token ﬁlters (0 or more)*-27<2#<%A72$IZ9#%S%%%*-27<2#<%+1:97BZ9#%]%%%%%%%*-27<2#<%+1:97%^B$-9#%]%%%%%%%_1K9#!239%+1:97%^B$-9#%]%%%%%%%*-1.%+1:97%^B$-9#

analyzers analyzers, tokenizers, and ﬁlters can be customizedmapping elasticsearch.yml B7<9P/ %%272$I3B3/ %%%%272$IZ9#/ %%%%%%.?&%,E/ %%%%%%%%-I.9/%!"3-1@ %%%%%%%%-1:97BZ9#/%3-27<2#< %%%%%%%%8B$-9#/%G3-27<2#<E%$1K9#!239E%3-1.E %%%%%%%%%%%%%%%%%23!BB81$<B7HE%.1#-9#*-9@J ` ?-B-$9?/%>?-I.9?/%?3-#B7H?E%?272$IZ9#?/%?9"$27H?NE `

API conventionsappend ?pretty=true to get readable JSONboolean values: false/0/off = false, rest is trueJSONP support via callback parameter

API structurehttp://host:port/[index]/[type]/[_action/id] GET http://es:9200/_status GET http://es:9200/twitter/_status POST http://es:9200/twitter/tweet/1 GET http://es:9200/twitter/tweet/1

API structurehttp://host:port/[index]/[type]/[_action/id] GET http://es:9200/twitter/tweet/_search GET http://es:9200/twitter/user/_search GET http://es:9200/twitter/tweet,user/_search GET http://es:9200/twitter,facebook/_search GET http://es:9200/_search

API query example>%%%%?R"9#I?/%>%%%%%%%%?8B$-9#9<?/%>%%%%%%%%%%%%?R"9#I?/%>%%%%%%%%%%%%%%%%?R"9#IO3-#B7H?/%>%%%%%%%%%%%%%%%%%%%%?R"9#I?/%?811%F2#?E%%%%%%%%%%%%%%%%%%%%?<982"$-O1.9#2-1#?/%?AaW?E%%%%%%%%%%%%%%%%%%%%?8B9$<3?/%G?-B-$9?E%?<93!#B.-B17?JE%%%%%%%%%%%%%%%%%%%%?F113-?/%5U6%%%%%%%%%%%%%%%%N%%%%%%%%%%%%NE%%%%%%%%%%%%?8B$-9#?/%>%%%%%%%%%%%%%%%%?#27H9?/%>?<2-9?/%>?H-?/%?56;;&6T&64?NN%%%%%%%%%%%%N%%%%%%%%N%%%%NE%%%%?8#1@/%;6E%%%%?3BZ9?/%;6N

API {core}index searchbulk querydelete from/size pagingdelete by query sortget highlightingcount selective ﬁelds

API {indices}create optimizedelete snapshotopen/close update settingsget/put/delete analyzemapping statusrefresh ﬂush

Query DSLterm / terms query_stringrange default_operatorpreﬁx analyzerbool phrase_slopfuzzy etcwildcard

ﬁltersshare some similar features with queries (term,range, etc)why use a ﬁlter?

filtersfaster than queriescached (depends on the filter) the cache is used for different queries against the same filterno scoringmore useful ones: term, terms, range, prefix, and,or, not, exists, missing, query

facetsprovide aggregated data based on the searchrequestterms, histogram, date histogram, range,statistical, and more

geo searchimplemented as ﬁlters (and a facet) geo_distance geo_bounding_box geo_polygon

interfacesRESTJava /!GroovyLanguage clients (REST/Thrift): pyes, PHP (standalone and symfony), Ruby, PerlFlume sink implementation

data importES is not the primary data store (usually)to import/synchronize data: write an agent (Gearman, message queues, etc) use rivers (CouchDB, RabbitMQ, Twitter)

10 more featuresversioning load balancing nodesindex aliases pluginsparent/child docs more_like_thisscripting multi_ﬁeld mappingdynamic mapping percolationtemplates

Referenceshttp://github.com/elasticsearch/elasticsearchhttp://groups.google.com/a/elasticsearch.com/group/users/IRC: #elasticsearch on irc.freenode.nettwitter: @elasticsearch

http://www.berejeb.com	69
http://www.devetdesign.com	9
http://feeds.feedburner.com	3

99 problems but the search aint one, confoo 2011, andrei zmievski

by Bachkoutou Toutou on Mar 10, 2011

Accessibility

Upload Details

Usage Rights

3 Embeds 81

Statistics

99 problems but the search aint one, confoo 2011, andrei zmievski Presentation Transcript