This document summarizes Grzegorz Kolpuc's work developing search and analytics solutions at Thomson Reuters using the Elastic Stack. It describes: 1. Developing distributed search engines using Elasticsearch for full-text search, filtering, caching and plugins. 2. Creating the Marvel tool for debugging and monitoring Elasticsearch clusters. 3. Implementing cross-cluster search, analytics for aggregation, grouping and metrics, as well as geo-location features like distance filtering. 4. Using scripting for custom scoring, filtering and analytics beyond standard queries. 5. Developing alerting to notify users of data changes through Elasticsearch percolation.

2. Grzegorz Kolpuc
@gkolpuc
https://pl.linkedin.com/pub/grzegorz-kolpuc/55/b7/700
grzegorzkolpuc@gmail.com

3. Event Platform

6. Financial & Risk IP & Science
Legal News
Tax & Accounting Technology & OPS

8. 60,000+
EMPLOYEES
10,000+
IN TECHNOLOGY

9. 1200+
EMPLOYEES IN GDYNIA
150+
IN TECHNOLOGY

10. Eikon

12. Elastic Development
Implementing big data search

13. Distributed Search Engine
Full-text search
Analytics

14. open source
document-oriented
based on Lucene
distributed
search
analytics

15. full-text search
filtering (exact matches, ranges, geo)
cacheable

16. Plugins
API extensions
scripting
analysis
scripting

17. Plugins
API extensions
scripting
analysis
scripting

19. Marvel

21. Native client
fluent API
single request
bulk requests
bulk processor

22. 1. Highlight offset

23. standard highlighter
"highlight": {
"StrEvntBriefBlob": [
"continues to see <b>pressure</b> on
internal growth"
]
}
"snippets": [
{
"snippet": "continues to see
<b>pressure</b> on
internal growth",
"startOff": 2082,
"endOff": 2126,
"pages": [0],
"snippetID": 1
}
]
fvhEx highlighter

24. 2. Cross cluster search
cluster_threecluster_twocluster_one
newsfilings
research
events

25. Tribe Node
cluster_threecluster_twocluster_one
newsfilings
research
events
Tribe Node
tribe:
t1:
cluster.name: cluster_one
t2:
cluster.name: cluster_two
t2:
cluster.name: cluster_three

26. 3. Analytics
grouping search result
statistics
metrics

27. "aggregations": {
"events_grouped": {
"terms": {"field": "group"},
"aggregations": {
"top_events": {
"top_hits": {
"size": 10,
"sort": [
{"date": {"order": "desc"}}
]
}
}
}
}
}
"aggs": {
"level2": {
"date_histogram": {
"field": "timestamp",
"interval": "day",
"order": {"_key": "desc"}
},
"aggs": {
"level3": {
"terms": {"field": "host"},
"aggs": {
"level4": {
"terms": {"field":
"apiRequest.appId"}
}
}
}}}}

28. 4. Geo search
distance filter
polygon filter
bounding box filter
shape query
"location": {
"type": "geo_shape",
"tree": "quadtree",
"precision": "1m"
}
"position": {
"type": "geo_point"
}

29. 5. Scripting
if nothing else works
decrease performance

30. ‘total count of snippets’
{
"query": {
"function_score": {
"query": {
"query_string": {"query":
"Average "Running five""}
},
"script_score": {>>>>},
"boost_mode": "replace"
}
},
"sort": [{"_score": {}}]
}
"script_score": {
"script_id": "scount6",
"lang": "groovy",
"params": {
"terms": ["average"],
"phases": [
{
"distance": 1,
"phase": ["running","five"]
}
],
"fields": [{"field": "StrEvntTranscriptBlob",
"grp": 111}]
}
},
"boost_mode": "replace"

31. 6. Alerting
users subscribed to data
notify users when data changes

32. Elastic Search
Percolate
Elastic Search
index documentsquery

33. Any Cons?

34. Low throughput
org.elasticsearch.common.util.concurrent.EsRejectedExecutionException
org.elasticsearch.cluster.block.ClusterBlockException

35. {
"error":
"NoShardAvailableActionException[[std_events_sem_021215][5]
null]",
“status": 503
}

36. Support

37. Reindex
Apply mapping changes online
Updatable number of shards in the index
Shard splitting

39. 15+
Clusters
3+
percolate clusters

40. ~75
nodes per cluster
250+
indices

41. 16+
TB of indexed data
7,000,000,000+
documents

42. 100,000+
queries per day (EVENTS)
500,000+
queries per day (expected for research)

43. Q?