Дмитрий Жлобо, Ruby and Rails Developer в Twinslash «Использование Elasticsearch для организации поиска по сайту» Организация качественного поиска на сайте – сложная и нетривиальная задача. В своем докладе Дмитрий расскажет о том, как ее решить с помощью Elasticsearch. Будет рассмотрено, как Elasticsearch работает с текстом или другими данными: от анализа и индексации документов до поиска и агрегации. По шагам и на примерах будет показано, как настроить поиск, учитывающий, например, морфологию и фонетику русского языка. Также Дмитрий расскажет, как все это использовать в приложениях на Ruby, как организовать добавление документов в индекс и др.

2. elasticsearch

3. me
Developer at Twinslash
email: dima.zhlobo@gmail.com
skype: dima.zhlobo
github: proghat
twitter: @proghat
Dmitry Zhlobo

4. search is hard
● speed vs. relevancy

5. search is hard
● speed vs. relevancy
● real time

6. search is hard
● speed vs. relevancy
● real time
● different kinds of data

7. usual approach
● SELECT * FROM posts WHERE `body`
LIKE '%query%'

8. usual approach
● SELECT * FROM posts WHERE `body`
LIKE '%query%'
● gem 'thinking-sphinx'
…
Article.search(params[:q])

9. usual approach
● SELECT * FROM posts WHERE `body`
LIKE '%query%'
● gem 'thinking-sphinx'
…
Article.search(params[:q])

10. how search works?
● document 1:
flexible and powerful open source, distributed real-
time search and analytics engine for the cloud...
● document 2:
Apache Mahout has implementations of a wide
range of machine learning and data mining...
● document 3:
Our core algorithms for clustering, classification and
batch based collaborative filtering are implemented
on top of Apache Hadoop using the MapReduce...

11. how search works?
data
mapreduce
learning
classification
recommenders
analysis

12. how search works?
data
mapreduce
learning
classification
recommenders
analysis
1 2 3
1 2
2 3
2 3
2
3

13. how search works?
data
mapreduce
learning
classification
recommenders
analysis
1 2 3
1 2
2 3
2 3
2
3

14. how search works?
data
mapreduce
learning
classification
recommenders
analysis
1 2 3
1 2
2 3
2 3
2
3

15. how search works?
data
mapreduce
learning
classification
recommenders
analysis
1 2 3
1 2
2 3
2 3
2
3

16. elasticsearch

17. ● full text search
● real time data
● opensource
● restful api
● distributed
● schema free & document oriented
elasticsearch

18. analysis
Flexible and powerful <strong>search</strong> engine

19. analysis
Flexible and powerful <strong>search</strong> engine
char
filters
Mapping
HTML Strip
Pattern Replace

20. analysis
char
filters
Flexible and powerful search engine

21. analysis
char
filters
tokenizer
Flexible and powerful search engine
Path Hierarchy
Keyword
Letter
Lowercase
NGram
Standard
Whitespace
Pattern
Edge NGram

22. analysis
char
filters
tokenizer
Flexible and powerful search engine

23. analysis
char
filters
tokenizer
Flexible and powerful search engine
token
filters
Stop
Lowercase
Snowball
Synonym
TrimUnique
Normalization
Stemmer
Shingle
Truncate
Reverse

24. analysis
tokenizer
token
filters
char
filters
flexible powerful search engine

25. russian morphology
“Он отлично рассказал о лучшем поисковом движке”

26. russian morphology
“Он отлично рассказал о лучшем поисковом движке”
отлично отличный

27. russian morphology
“Он отлично рассказал о лучшем поисковом движке”
отлично отличный рассказать

28. russian morphology
“Он отлично рассказал о лучшем поисковом движке”
отлично отличный рассказать хороший

29. russian morphology
“Он отлично рассказал о лучшем поисковом движке”
отлично отличный рассказать хороший
поисковый

30. russian morphology
“Он отлично рассказал о лучшем поисковом движке”
отлично отличный рассказать хороший
движокпоисковый

31. russian morphology
“Он отлично рассказал о лучшем поисковом движке”
отлично отличный рассказать хороший
движокпоисковый

32. russian morphology
“Он отлично рассказал о лучшем поисковом движке”
отлично отличный рассказать хороший
движок
“хороший поисковой движок”
хороший движок
поисковый
поисковый

33. russian morphology
“Он отлично рассказал о лучшем поисковом движке”
отлично отличный рассказать хороший
движок
“хороший поисковой движок”
хороший
поисковый
движокпоисковый

34. phonetic analysis

36. phonetic analysis
Eyjafjallajökull

37. phonetic analysis
Eyjafjallajökull
Eiyafyalayokul
iofiolDkul

38. analysis
char
filters
tokenizer
token
filters

39. analysis
char
filters
tokenizer
token
filters
analyzer

40. analysis
char
filters
tokenizer
token
filters
analyzer
● has name

41. analysis
char
filters
tokenizer
token
filters
analyzer
● has name
● reusable

42. analysis
char
filters
tokenizer
token
filters
analysis:
analyzer:
rus_morphology:
type: "custom"
char_filter: ["html_strip"]
tokenizer: "standard"
filter: ["lowercase", "russian_morphology", "stopwords"]

43. getting started
# Add document to index
curl -XPOST "localhost:9200/posts/post/1" -d '{ title: "Search" }'

44. getting started
# Add document to index
curl -XPOST "localhost:9200/posts/post/1" -d '{ title: "Search" }'

45. getting started
# Add document to index
curl -XPOST "localhost:9200/posts/post/1" -d '{ title: "Search" }'

46. getting started
# Add document to index
curl -XPOST "localhost:9200/posts/post/1" -d '{ title: "Search" }'

47. # Add document to index
curl -XPOST "localhost:9200/posts/post/1" -d '{ title: “Search" }'
curl -XPOST "localhost:9200/posts/post/whatever" -d '{ title: "ES" }'
getting started

48. # Add document to index
curl -XPOST "localhost:9200/posts/post/1" -d '{ title: “Search" }'
curl -XPOST "localhost:9200/posts/post/whatever" -d '{ title: "ES" }'
# Search documents
curl -XGET "localhost:9200/posts/_search?q=data"
curl -XGET "localhost:9200/posts/_search?q=title:elasticsearch"
getting started

49. # Add document to index
curl -XPOST "localhost:9200/posts/post/1" -d '{ title: “Search" }'
curl -XPOST "localhost:9200/posts/post/whatever" -d '{ title: "ES" }'
# Search documents
curl -XGET "localhost:9200/posts/_search?q=data"
curl -XGET "localhost:9200/posts/_search?q=title:elasticsearch"
# Update and delete documents
curl -XPUT "localhost:9200/posts/post/1" -d '{ title: “Data" }'
curl -XDELETE "localhost:9200/posts/post/whatever"
getting started

50. mapping
{
repository: "elasticsearch/elasticsearch",
description: "Open Source, Distributed, RESTful Search Engine",
maintainer: {
name: "Shay Banon",
email: "kimchy@gmail.com"
},
languages: ["Java", "Shell"],
created_at: "2010-02-08"
}

51. mapping
{
repository: "elasticsearch/elasticsearch",
description: "Open Source, Distributed, RESTful Search Engine",
maintainer: {
name: "Shay Banon",
email: "kimchy@gmail.com"
},
languages: ["Java", "Shell"],
created_at: "2010-02-08"
}
repository: {
type: "string"
}

52. mapping
{
repository: "elasticsearch/elasticsearch",
description: "Open Source, Distributed, RESTful Search Engine",
maintainer: {
name: "Shay Banon",
email: "kimchy@gmail.com"
},
languages: ["Java", "Shell"],
created_at: "2010-02-08"
}
repository: {
type: "string",
boost: 5
}

53. mapping
{
repository: "elasticsearch/elasticsearch",
description: "Open Source, Distributed, RESTful Search Engine",
maintainer: {
name: "Shay Banon",
email: "kimchy@gmail.com"
},
languages: ["Java", "Shell"],
created_at: "2010-02-08"
}
repository: {
type: "string",
boost: 5,
analyzer: "repo_name"
}
repo_name: {
tokenizer: "letter",
filter: ["lowercase","phonetic"]
}

54. mapping
{
repository: "elasticsearch/elasticsearch",
description: "Open Source, Distributed, RESTful Search Engine",
maintainer: {
name: "Shay Banon",
email: "kimchy@gmail.com"
},
languages: ["Java", "Shell"],
created_at: "2010-02-08"
}
description: {
type: "string"
}

55. mapping
{
repository: "elasticsearch/elasticsearch",
description: "Open Source, Distributed, RESTful Search Engine",
maintainer: {
name: "Shay Banon",
email: "kimchy@gmail.com"
},
languages: ["Java", "Shell"],
created_at: "2010-02-08"
}
description: {
type: "string",
analyzer: "english_text"
}
english_text: {
tokenizer: "standard",
filter:
["lowercase", "stopwords", "snowball"]
}

56. mapping
{
repository: "elasticsearch/elasticsearch",
description: "Open Source, Distributed, RESTful Search Engine",
maintainer: {
name: "Shay Banon",
email: "kimchy@gmail.com"
},
languages: ["Java", "Shell"],
created_at: "2010-02-08"
}
maintainer: {
properties: {
}
}

57. mapping
{
repository: "elasticsearch/elasticsearch",
description: "Open Source, Distributed, RESTful Search Engine",
maintainer: {
name: "Shay Banon",
email: "kimchy@gmail.com"
},
languages: ["Java", "Shell"],
created_at: "2010-02-08"
}
maintainer: {
properties: {
name: {
type: "string"
}
}
}

58. mapping
{
repository: "elasticsearch/elasticsearch",
description: "Open Source, Distributed, RESTful Search Engine",
maintainer: {
name: "Shay Banon",
email: "kimchy@gmail.com"
},
languages: ["Java", "Shell"],
created_at: "2010-02-08"
}
maintainer: {
properties: {
name: {
type: "string",
analyzer: "phonetic"
}
}
}
phonetic: {
tokenizer: "standard",
filter: ["lowercase", "stopwords",
"beidermorse"]
}

59. mapping
{
repository: "elasticsearch/elasticsearch",
description: "Open Source, Distributed, RESTful Search Engine",
maintainer: {
name: "Shay Banon",
email: "kimchy@gmail.com"
},
languages: ["Java", "Shell"],
created_at: "2010-02-08"
}
maintainer: {
properties: {
email: {
type: "string"
}
}
}

60. mapping
{
repository: "elasticsearch/elasticsearch",
description: "Open Source, Distributed, RESTful Search Engine",
maintainer: {
name: "Shay Banon",
email: "kimchy@gmail.com"
},
languages: ["Java", "Shell"],
created_at: "2010-02-08"
}
maintainer: {
properties: {
email: {
type: "string",
index: "not_analyzed"
}
}
}

61. mapping
{
repository: "elasticsearch/elasticsearch",
description: "Open Source, Distributed, RESTful Search Engine",
maintainer: {
name: "Shay Banon",
email: "kimchy@gmail.com"
},
languages: ["Java", "Shell"],
created_at: "2010-02-08"
}
language: {
type: "string"
}

62. mapping
{
repository: "elasticsearch/elasticsearch",
description: "Open Source, Distributed, RESTful Search Engine",
maintainer: {
name: "Shay Banon",
email: "kimchy@gmail.com"
},
languages: ["Java", "Shell"],
created_at: "2010-02-08"
}
language: {
type: "string",
analyzer: "programming_lang"
}
programming_lang: {
tokenizer: "keyword",
filter: ["lowercase"]
}

63. mapping
{
repository: "elasticsearch/elasticsearch",
description: "Open Source, Distributed, RESTful Search Engine",
maintainer: {
name: "Shay Banon",
email: "kimchy@gmail.com"
},
languages: ["Java", "Shell"],
created_at: "2010-02-08"
}
created_at: {
type: "date",
format: "YYYY-MM-DD"
}

64. mapping
curl -XPOST "localhost:9200/repositories" -d '
settings: {
analysis: {
analyzer: { ... },
filter: { ... }
}
},
mappings: {
repository: {
properties: { ... }
}
}'

65. mapping
curl -XPOST "localhost:9200/repositories" -d '...'
curl -XPOST "localhost:9200/repositories/repository" -d '
{
repository: "elasticsearch/elasticsearch",
description: "Open Source, Distributed, RESTful Search Engine",
maintainer: {
name: "Shay Banon",
email: "kimchy@gmail.com"
},
languages: ["Java", "Shell"],
created_at: "2010-02-08"
}'

66. search
curl -XGET "localhost:9200/repositories/repository/_search?q=engine"

67. search
curl -XPOST "localhost:9200/repositories/repository/_search" -d '
{
query: {
match: {
description: "search"
}
}
}'

68. search
"hits" : {
"total" : 3,
"hits" : [
{
"_score" : 0.22295055,
"_source" : { repository: "elasticsearch/elasticsearch" }
}, {
"_score" : 0.22295055,
"_source" : { repository: "ankane/searchkick" }
}, {
"_score" : 0.095891505,
"_source" : { repository: "karmi/tire" }
}
]
}

69. search
curl -XPOST "localhost:9200/repositories/repository/_search" -d '
{
query: {
match: {
_all: "elasticsearch"
}
}
}'

70. search
"hits" : {
"total" : 2,
"hits" : [
{
"_score" : 5.46875,
"_source" : { repository: "elasticsearch/elasticsearch" }
}, {
"_score" : 0.04746387,
"_source" : { repository: "karmi/tire" }
}
]
}

71. facets
curl -XPOST "localhost:9200/repositories/repository/_search" -d '
{
query: {
match: {
_all: "search"
}
},
facets: {
language: {
terms: {
field: "languages"
}
}
}
}'

72. facets
"hits" : {
"total" : 2,
"hits" : [ ... ]
},
"facets" : {
"language" : {
"terms" : [
{ "term" : "ruby", "count" : 2 },
{ "term" : "shell", "count" : 1 },
{ "term" : "java", "count" : 1 }
]
}
}

73. filters
curl -XPOST "localhost:9200/repositories/repository/_search" -d '
{
query: {
filtered: {
query: {
match: {
_all: "search"
}
},
filter: {
term: { "languages": "java" }
}
}
}
}'

74. filters
"hits" : {
"total" : 1,
"hits" : [
{
"_score" : 5.46875,
"_source" : { repository: "elasticsearch/elasticsearch" }
}
]
}

75. performance and scaling

76. performance and scaling
elasticsearch is web scale

77. random facts
● bulk operations
● real time
● highlights
● geo types and geo distance facets
● attachments
● “did you mean?” and completions
● common terms
● filters and caching
● river

78. You know. For search.