Introduction to elasticsearch fast lane

Introduction elasticsearch
@somkiat
SPRINT3R
Siam Chamnankit Co., Ltd., Odd-e (Thailand) Co., Ltd. and Alliance

realtime, search
and analytics
engine
SPRINT3R
Lucene
Distributed
Scalability
open-source
documentation
JSON
API
RESTFul
document store
High-Availability
JAVA
Plug-ins

History
SPRINT3R
First version 0.4 in Feb 2010
Rewrite from Compass project
Add scalability
Created by Shay Banon
Current version 1.7.1

Who use ?
SPRINT3R
https://www.elastic.co/use-cases

SPRINT3R
• Search repositories, users, issues, pull request
• Search sourcecode 130 พันล้านบรรทัด
• Track alerts, events และ logs

SPRINT3R
• ใ๡้ Full text seach + geolocation
• ใ๡้ feature More-like-this ในการหาคําถาม และ คําตอบ

Clients
SPRINT3R
• Java
• PHP
• Ruby
• Python
• JavaScript
• Go
• Scala
• .Net
• Clojure
• Erlang
• NodeJS • R
http://www.elasticsearch.org/guide/en/elasticsearch/client/community/current/clients.html

https://github.com/github-sprint3r/
elasticsearch_workshop
SPRINT3R

Let’s start
SPRINT3R

Installation …
SPRINT3R

elasticsearch
SPRINT3R
• real-time
• distributed
• search
• analytics

SPRINT3R
mapping
analysis query dsl

SPRINT3R
query dsl
• flexible
• powerful
• query language

SPRINT3R
queries
• relevance
filters
• boolean yes/no
• full text • exact values
• not cached • cached
• slower • faster

SPRINT3R
Filter first
query remaining documents

SPRINT3R
GET /_search

SPRINT3R
GET /_search
{ 
“query” : { . . . } 
}

SPRINT3R
GET /_search
{ 
“query” : { “match” : { “title” : “search” } } 
}

SPRINT3R
GET /_search
{ 
“query” : { “match_all” : { } } 
}

SPRINT3R
GET /_search
{ 
“query” : {  
“filtered” : {
“query” : { … },
“filter” : { … }
}
} 
}

SPRINT3R
GET /_search
{ 
“query” : {  
“filtered” : {
“query” : { “match” : { “title” : “search” } },
“filter” : { … }
}
} 
}

SPRINT3R
GET /_search
{ 
“query” : {  
“filtered” : {
“query” : { “match” : { “title” : “search” } },
“filter” : { “term” : { “status” : “active” } }
}
} 
}

SPRINT3R
GET /_search
{ 
“query” : {  
“filtered” : {
“query” : { “match_all” : { } },
“filter” : { “term” : { “status” : “active” } }
}
} 
}

SPRINT3R
How data is indexed ?

SPRINT3R
{
"title": "Quick brown rabbits",
"content": "Brown rabbits are commonly seen"
}
{
"title": "Keeping pets healthy",
"content": "My quick brown fox eats rabbits on  
a regular basis"
}

SPRINT3R
where content like
“%brown%fox%”

SPRINT3R
Analysis

SPRINT3R
{
"title": [quick,brown,rabbits],
"content": [brown,rabbits,are,commonly,seen]
}
{
"title": [keeping,pets,healthy],
"content": [my,quick,brown,fox,eats,rabbits,on  
,a,regular,basis]
}

SPRINT3R
field: content
Term Document 1 Document 2
a
are
basis
brown
commonly
eats
fox
my
on
quick
rabbits
regular
seen

SPRINT3R
Inverted indexed

SPRINT3R
where field = “value”

SPRINT3R
where field contains “value”

SPRINT3R
where field contains “value”
term filter
“term” : {
“title” : “brown”
}

SPRINT3R
GET /_search
{ 
“query” : {  
“filtered” : {
“query” : { “match_all” : { } },
“filter” : { “term” : { “title” : “brown” } }
}
} 
}

SPRINT3R
field: title
Term Document 1 Document 2
brown
healthy
keeping
pets
quick
rabbits

SPRINT3R
Result
bitset[ 1, 0 ]
cache as “title:brown”

SPRINT3R
where field IN [“value1”, … ]

SPRINT3R
terms filter
“terms” : {
“title” : [“quick”, “pets”]
}
where field IN [“value1”, … ]

SPRINT3R
Result
bitset[ 1, 1 ]
cache as “title:quick title:pets”

SPRINT3R
where field >= “value 1”
and field < “value 2”

SPRINT3R
where field >= “value 1”
and field < “value 2”
range filter

SPRINT3R
“range” : {
“content” : {
“gte” : “a”,
“lt” : “m”
}
}

SPRINT3R
Result
bitset[ 1, 1 ]
cache as “content:[a TO m]”

SPRINT3R
AND … OR … NOT

SPRINT3R
AND … OR … NOT
bool filter

SPRINT3R
“bool” : {
“must” : [ <filters> ],
“should” : [ <filters> ],
“must_not” : [ <filters> ],
}

SPRINT3R
how relevant is this term ?

SPRINT3R
how relevant is this term ?
term query

SPRINT3R
GET /_search
{ 
“query” : {  
“term” : {
“title” : “brown”
}
} 
}

SPRINT3R
how relevant is this document ?

SPRINT3R
Relevant score
Term in this document
more is better

SPRINT3R
Relevant score
Term in all documents
less is better

SPRINT3R
Relevant score
How long of this document ?
shorter is better

SPRINT3R
AND … OR … NOT
bool query

SPRINT3R
“bool” : {
“must” : [ <filters> ], #AND
“should” : [ <filters> ], ???
“must_not” : [ <filters> ], #NOT
}

SPRINT3R
“bool” : {
“must” : [ <filters> ], #AND
“should” : [ <filters> ], ???
“must_not” : [ <filters> ], #NOT
“minimum_should_match” : ?
}

SPRINT3R
bool filter -> True | False
bool query -> _score

SPRINT3R
match query

SPRINT3R
match query
high level query
understand mapping and analysis

SPRINT3R
match query
1. analyze query string
2. rewrite query

SPRINT3R
{ “match” : { “title” : “QUICK!” } }
title:quick
{ “term” : { “title” : “quick” } }

SPRINT3R
{ “match” : { “title” : “QUICK FOX!” } }
title:quick OR title:fox
{ “bool” : {
“should” : [
{ “term” : { “title” : “quick” } },
{ “term” : { “title” : “fox” } }
]
}}

SPRINT3R
Need all words must match ?

SPRINT3R
GET /_search
{ 
“query” : {  
“term” : {
“title” : “QUICK FOX!”
}
} 
}

SPRINT3R
GET /_search
{ 
“query” : {  
“term” : {
“title” : {
“query” : “QUICK FOX!”,
“operator” : “and”
}
}
} 
}

SPRINT3R
GET /_search
{
"query": {
"bool": {
"must": [
{ "term": { "title" : "quick" } },
{ "term": { "title" : "fox" } }
]
}
}
}

SPRINT3R
GET /_search
{ 
“query” : {  
“match_pharse” : {
“title” : {
“query” : “QUICK BROWN”
}
}
} 
}

SPRINT3R
GET /_search
{ 
“query” : {  
“title” : {
“query” : “QUICK RABBITS”
}
}
} 
}

SPRINT3R
GET /_search
{ 
“query” : {  
“title” : {
“query” : “BROWN QUICK”
}
}
} 
}

SPRINT3R
Limit long tail ?

SPRINT3R
GET /_search
{ 
“query” : {  
“term” : {
“title” : {
“query” : “QUICK BROWN FOX!”,
“minimum_should_match” : “75”%
}
}
} 
}

SPRINT3R
GET /_search
{
"query": {
"bool": {
"should": [
{ "term": { "title" : "quick" } },
{ "term": { "title" : "brown" } },
{ "term": { "title" : "fox" } }
],
“minimum_should_match” : 2
}
}
}

SPRINT3R
Aggregation API
analytic your data
explore your data

Introduction to elasticsearch fast lane

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (20)

Similar to Introduction to elasticsearch fast lane

Similar to Introduction to elasticsearch fast lane (18)

More from Somkiat Puisungnoen

More from Somkiat Puisungnoen (20)

Recently uploaded

Recently uploaded (20)

Introduction to elasticsearch fast lane