Elasticsearch - SEARCH & ANALYZE DATA IN REAL TIME

1. ELASTICSEARCH
SEARCH & ANALYZE DATA IN REAL TIME*
Piotr Pelczar • github • stackoverflow
Wrocław 2017, Eurobank
freeimages.com v 1.2

2. AGENDA
You will find out:
• purpose
• how data is stored and searched
• features + 3rd party
• architecture
• usecase on production

3. AGENDA
You will not find out:
• production ready configuration: HA, repl, sharding
• monitoring
• production internal bottlenecks / failure recovery
• ELK stack – elasticsearch + logstash + kibana
• comparison of ES i Solr/Sphinx

4. PURPOSE
• NoSQL
• Databse for full text search
• More reads then writes
every document update is a creation of a new one
• No transactions – BASE instead of ACID

5. FULL-TEXT SEARCH
• full-text search (FTS) refers to techniques of efficient
search the data simillar to natural language text
• search is performed using full text index
• under the hood ES is powered by Apache Lucene

6. FULL-TEXT SEARCH
FTS is available in Oracle, MsSQL, MySQL, but...
• KILLER FEATURE: ES enables to customize the proces
of building the full text index
• features like:
– autocomplete
– „Did you mean?” based Levenstein distance
– indexing one field in a several ways

7. ES IS POWERED BY LUCENE
Features added:
• clustering
• sharding (horizontal scaling)
• replication (copy of shards)
• versioning
• non-full-text indices
• REST API
https://pl.pinterest.com/pin/528328600014757803/

8. BASE vs ACID
• Atomicity
• Consistency
• Isolation
• Durability
• Basically Available
• Soft state
• Eventual consistency
http://wallpaperswide.com/domino_effect-wallpapers.html

9. TERMINOLOGY
• Cluster
• Node
• Index (collection of docs) <-> (?) tablespace/database
• Type (partition of index) <-> (?) table/collection
• Document (JSON format)
• Shard & Replicas
https://wallpaperscraft.com/download/london_philharmonic_orchestra_scene_show_play_conductor_8925/2560x1080

10. DOCUMENT-ORIENTED
I SCHEMA-FREE
• data is stored as documents
• documents are unstructured *
* by default, but there is a possibility to require strict o partly strict schema in the type
definition in index
• all fields are indexed in full-text by default *
* this behaviour is fully configurable, data type can be changed or
the field can be ignored in full-text index
http://www.shximai.com/education-wallpapers.html

11. INVERTED INDEX

12. RANKING FORMULA
BM25 similarity function
https://www.slideshare.net/Hadoop_Summit/t-435p212leauv2
1. http://ipl.cs.aueb.gr/stougiannis/bm25_2.html
2. https://en.wikipedia.org/wiki/Tf%E2%80%93idf
q - query
t – term
d - document

13. ANALYZERS
Analysis - the process of converting text into tokens or
terms which are added to the inverted index for
searching. Analysis is performed by an analyzer.
• Index time analyser
• Search time analyser
http://wallpaperswide.com/glasses_and_book-wallpapers.html

14. ANALYZERS
1. Tokenizing a block into terms
2. Normalizing (reducing) terms into root form
• Every field in document type can have own analyser
• Fields can be indexed by several analysers
(multi-fields)
https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-
fields.html#_multi_fields_with_multiple_analyzers
http://wallpaperswide.com/glasses_and_book-wallpapers.html

15. ANALYZERS
1. Character Filters
– html and entities
– triming
2. Tokenizer
3. Token Filters
– stopwords
– Stemmer (root form)
– Phonetic, n-grams
– Synonim
– Patten capture
http://wallpaperswide.com/glasses_and_book-wallpapers.html
Input
Filter
Tokenizer
Token
filter
Index

16. ANALYZERS
Polish analyser – Stempel
sudo bin/plugin install analysis-stempel
http://wallpaperswide.com/glasses_and_book-wallpapers.html

17. CUSTOM ANALYSERS
PUT /index_name
{
"settings": {
"analysis": {
"analyzer": {
"polskie_slowa": {
"type": "custom",
"tokenizer": "standard",
"filter": ["lowercase", "stopwords_polska", "polish_stem", "asciifolding"],
"char_filter": ["html_strip"]
}
},
"filter": {
"stopwords_polska": {
"type": "stop",
"stopwords": ["a", "aby", ...]
}
}
}
}
// ...

18. TESTING ANALYSER
GET /_analyze
{
"analyzer": "standard",
"text": "Text to analyze"
}
{
"tokens": [
{
"token": "text",
"start_offset": 0,
"end_offset": 4,
"type": "<ALPHANUM>",
"position": 1
},
// ..

19. INSERT
PUT /website/blog/123
{
"title": "My first blog entry",
"text": "Just trying this out...",
"date": "2014/01/01"
}
Custom ID - PUT
POST /website/blog/
Auto ID - POST

20. INSERT
By default:
• There is no document schema
• All fields are indexed in full-text index
• Type of encountered previously unknown field
is determined by the first value that appeared
(dynamic mapping)
https://www.elastic.co/guide/en/elasticsearch/guide/current/dynamic-mapping.html
https://studyinthestates.dhs.gov/2017/01/new-interim-final-guidance-open-for-comment-until-feb-27

21. INSERT – DYNAMIC MAPPING
dyanmic
- true (fields out of schema are allowed)
- false (new fields are just ignored)
- strict (exception, when unknown field)
https://studyinthestates.dhs.gov/2017/01/new-interim-final-guidance-open-for-comment-until-feb-27

22. INSERT – DYNAMIC MAPPING
PUT /my_index
{
"mappings": {
"my_type": {
"dynamic": "strict",
"properties": {
"title": { "type": "string"},
"stash": {
"type": "object",
"dynamic": true
}
}
}
}
}

23. MAPPING
Data types:
• Core
– numeric, date, boolean, binary, string (keyword, fulltext)
• Complex
– array, object, nested (array of objects)
– geo
• Multi-fields
– e.x. date as raw date and full-text value (movie title YEAR), or field
with multiple analysers
• Specialized
– ip, completion, …

24. MAPPING
"mappings": {
"news": {
"dynamic": "strict",
"properties": {
"title": {
"type": "string",
"analyzer": "polskie_slowa",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
},
"date_published": {
"type": "date",
"index": "not_analyzed"
},
multifield

25. MAPPING
"mappings": {
"news": {
"dynamic": "strict",
"properties": {
"stories": {
"type": "nested",
"dynamic": "strict",
"properties": {
"id": {
"type": "long",
"index": "not_analyzed"
},
"title": {
"type": "string",
"analyzer": "polskie_slowa"
}
}
}

26. MAPPING
"mappings": {
"news": {
"dynamic": "strict",
"properties": {
"media": {
"type": "object",
"dynamic": "strict",
"properties": {
"gallery_has": {
"type": "boolean",
"index": "not_analyzed"
},
"video_has": {
"type": "boolean",
"index": "not_analyzed"
},
"poll_has": {
"type": "boolean",
"index": "not_analyzed"
},

27. GET
GET /website/blog/123
{
"_index" : "website",
"_type" : "blog",
"_id" : "123",
"_version" : 1,
"found" : true,
"_source" : {
"title": "My first blog entry",
"text": "Just trying this out...",
"date": "2014/01/01"
}
}
Status code: 200, 404

28. DELETE
DELETE /website/blog/123

29. UPDATE
PUT /website/blog/123
{
"title": "My first blog entry",
"text": "Just trying this out...",
"date": "2014/01/01"
}
POST /website/blog/123/_update
{
"title": "My first blog entry„
}

30. UPDATE – IMMUTABLE DOCS
• Documents are immutable, because Lucene
segments are immutable
• The new version of documents are created
• Old version is marked in .del file
• Previous version is still searchable, but is
removed from search result in the runtime
until cleanup process
https://studyinthestates.dhs.gov/2017/01/new-interim-final-guidance-open-for-comment-until-feb-27

31. URI SEARCH
GET /index/_search?q=user:kimchy
GET /index/type1,type2/_search?q=user:kimchy
GET /index1,index2/type/_search?q=user:kimchy
GET /_all/type/_search?q=user:kimchy
GET /_search?q=user:kimchy

32. URI SEARCH
{
"timed_out": false,
"took": 62,
"hits":{
"total" : 1,
"max_score": 1.3862944,
"hits" : [
{
"_index" : "twitter",
"_type" : "tweet",
"_id" : "0",
"_score": 1.3862944,
"_source" : {
"user" : "kimchy",
"date" : "2009-11-15T14:12:12",
"message" : "trying out Elasticsearch",
"likes": 0
}

33. TERM QUERY
GET /index/type/_search
{
"query" : {
"term" : { "user" : "kimchy" }
}
}

34. SEARCH
Search segments:
• must-match
• should-match (scoring)
• fuzzy-query (Levenstein desitnation)
• filter (without scoring, very fast)
• limit/offset

35. TERM QUERY & BOOST
"query": {
"bool": {
"should" : [
{
"match": { "title": { "query": "myśliwy", "boost": 1 }
},
],
"filter": {
"and" : [
{
"term": {
"media.gallery_has": false
}
}
]
}

36. HIGHLIGHTING
GET /_search
{
"query" : {
"match": { "content": "kimchy" }
},
"highlight" : {
"fields" : {
"content" : {}
}
}
}

37. SUGGESTING
POST music/_search?pretty
{
"suggest": {
"song-suggest" : {
"prefix" : "nir",
"completion" : {
"field" : "suggest"
}
}
}
}

38. SUGGESTING
"suggest": {
"song-suggest" : [ {
"text" : "nir",
"offset" : 0,
"length" : 3,
"options" : [ {
"text" : "Nirvana",
"_index": "music",
"_type": "song",
"_id": "1",
"_score": 1.0,
"_source": {
"suggest": ["Nevermind", "Nirvana"]
}
} ]

39. SCORE FUNCTIONS
• Weight
• Field Value factor
• Decay functions
https://www.elastic.co/guide/en/elasticsearch/guide/current/decay-functions.html

40. SCORE FUNCTIONS - DECAY
Model functions:
• Gauss
• Exp
• Linear
• Multivalue
– 2d
– 3d
– n …

41. SCORE FUNCTIONS - DECAY
"function_score": {
"query": { ... },
"score_mode": "multiply", // how functions are compared
"boost_mode": "multiply", // how functions has impact to
original score
"functions": {
"date_published": {
"origin": " ... "
"offset": " ... "
"scale": " ... "
}
}

42. CHANGE
https://www.entrepreneur.com/article/269669

43. REINDEX
POST _reindex
{
"source": {
"index": "twitter"
},
"dest": {
"index": "new_twitter"
}
}
Every index change needs index rebuild (reindex) :
• Change data types, schema
• Analyser modification
• Shard key, numer of shards (no way to rebalance data)

44. REINDEX
POST _reindex
{
"source": {
"index": "twitter",
"type": "tweet",
"query": {
"term": {
"user": "kimchy"
}
}
},
"dest": {
"index": "new_twitter"
}
}
2x more space is needed, but there is the possibility to
do query width limit and offset and delete data from
old index in the meantime.
Application have to query
both indices/aliases during
reindex process and filter
duplicates in the runtime.

45. Index Aliases and Zero Downtime
POST /_aliases
{
"actions" : [
{ "remove" : { "index" : "news_v1",
"alias" : "news_view" } },
{ "add" : { "index" : "news_v2",
"alias" : "news_view" } }
]
}
https://www.elastic.co/guide/en/
elasticsearch/guide/current/index-aliases.html

46. * REAL TIME, NEAR REALTIME (NRT)
This means is there is a slight latency
(normally one second) from the time you index a
document until the time it becomes searchable.
http://all-free-download.com/wallpapers/animals/horse_racing_wallpaper_horses_animals_wallpaper_382.html

47. NEAR REALTIME (NRT)
1. Data have to be analysed
2. There are caveats with Lucene segments

48. NRT i BASE
In the system there is no global-lock
-> append-only, immutable
-> no transactions
-> >> response, availability => PROFIT!
-> data could be not visible immediately
Stale data, but any data...
http://wallpaperswide.com/domino_effect-wallpapers.html

49. NEAR REALTIME (NRT)
• New index is created in new Lucene Segments periodicaly
• Creation new Lucene Segments are called refresh
• 1 sec by default - NRT
• INSERT -> GET can respond 404.

50. NEAR REALTIME (NRT)
• Lucene allows to open a new Segment and search in it
without Commit
• Commit makes Lucene Segment immutable
• Document is always added to a new Segment that temporarily
resides in memory (file system cache) and are searchable
• To save the Segment fsync is needed - expensive
• Data are not stored on disk immediately, but there is no global
lock (BASE)

51. PERSISTENCE - TRANSLOG

52. PERSISTENCE - TRANSLOG
1. New documents are stored in translog and in-memory buffer
(at this point there are not searchable)
2. At the periodically refresh process they are copied to a new
Segment that resides also in memory
– from now, document are searchable
3. When Segment Commit occurs (fsync’ed into disk), the
document is removed from translog
4. Commit and translog cleanup is called flush
periodically or when translog is too big
http://sixthjudicialdistrict.sleekup.com/wp-content/uploads/2015/08/Judge-holding-gavel.jpg

53. PERSISTENCE - TRANSLOG
1. Translog is configurable per index
– translog is fsynce’d every 5 sec by default
– after every insert/update/delete/index
– translog is committed after bulk-insert – worth to use
2. Can be configured as async with interval:
PUT /my_index/_settings
{
"index.translog.durability": "async",
"index.translog.sync_interval": "5s"
}
http://sixthjudicialdistrict.sleekup.com/wp-content/uploads/2015/08/Judge-holding-gavel.jpg

54. PER-OPERATION PERSISTENCE
http://sixthjudicialdistrict.sleekup.com/wp-content/uploads/2015/08/Judge-holding-gavel.jpg

55. SCALING
sharding and replication

56. NODES IN CLUSTER
• There is one primary shard (in replicas)
• Shard in any node can become the primary

57. CRUD IN CLUSTER
• Request can be handled by any node
this node will coordinate the request
• CRUD is performed on Primary Shard first and
replicated to Replicas

58. CONSISTENCY
• Quorum by default (majority of Shard copies)
– floor( (primary + number_of_replicas) / 2 ) + 1
– Can be:
• Quorum
• One
• All
• Defining timeout is recommened (1min by default)
– Elasticsearch will be waiting until all needed responses
appear. In the case of timeout an application should take
the decision what to do
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#index-wait-
for-active-shards

59. READ FROM CLUSTER
• Any node can receive request and coordinates it
• Results will be fetched from nodes
By default, coordinate node will choose different shard copy on every
request in order to rebalance reads (round robin)

60. NODES IN CLUSTER
• Every node knows where data lives (information
about shard key), so can route the request
for client this approach is transparent, can talk with
any node it want
• If client keep connections to multiple nodes, there is
no Single Point of Failure
• Round-robin, to distribute the load

61. DISTRIBUTED READ
QUERY PHASE / FETCH PHASE
https://www.elastic.co/guide/en/elasticsearch/guide/current/distributed-search.html

62. CONNECTIONS
• REST API
– connection should be kept alive (choose proper library)
• Native client
– Binary protocol
– Designed fot inter-node communication

63. DOCUMENT VERSIONING
• Optimistic concurency tool
• Version types
– Internal (1, 2, 3 …)
– external or external_gt
– external_gte
POST /index/type?version=TIMESTAMP

64. DEMO

65. Search in whole ElasticSearch dataset personalized for specific user
(based on neo4j graph relations)
https://neo4j.com/developer/elastic-search/

67. Thanks! Q&A
http://doineedbackup.com/customer-acquisition-channels-for-saas/

68. Thanks! Q&A
https://highspark.co/how-to-end-a-presentation/