1. Side by Side with
Elasticsearch & Solr
Part 2 -­ Performance & Scalability
Radu Gheorghe
Rafał Kuć

2. Who are we?
Radu Rafał
Logsene
cftweia

3. Last year

4. Last year conclusions

5. One year progress
Facet by function
https://issues.apache.org/jira/browse/SOLR-­1581
Analytics component
https://issues.apache.org/jira/browse/SOLR-­5302
Solr as standalone application
https://issues.apache.org/jira/browse/SOLR-­4792
top_hits aggregation
https://github.com/elastic/elasticsearch/pull/6124
minimum_should_match on has_child
https://github.com/elastic/elasticsearch/pull/6019
filters aggregation
https://github.com/elastic/elasticsearch/pull/6118

6. That's not all
JSON facets
https://issues.apache.org/jira/browse/SOLR-­7214
Backup + restore
https://issues.apache.org/jira/browse/SOLR-­5750
Streaming aggregations
https://issues.apache.org/jira/browse/SOLR-­7082
Moving averages aggregation
https://github.com/elastic/elasticsearch/pull/10002
Computation on aggregations
https://github.com/elastic/elasticsearch/pull/9876
Cluster state diff support
https://github.com/elastic/elasticsearch/pull/6295
Cross data center replication
https://issues.apache.org/jira/browse/SOLR-­6273
Shadow replicas
https://github.com/elastic/elasticsearch/pull/8976

7. This year’s agenda
Horizontal scaling
Products use-­case
Logs use-­case

8. Horizontal scaling in theory
Node
shard

9. Horizontal scaling in theory
Node
shard
Node
shard

10. Horizontal scaling in theory
Node
shard
Node
shard
Node
shard

11. Horizontal scaling in theory
Node
shard shard
shard shard
Node
shard shard
shard shard
Node
shard shard
shard shard

12. Horizontal scaling in theory
Node
shard shard
shard shard
replica
replica
replica
replica
Node
shard shard
shard shard
replica
replica
replica
replica
Node
shard shard
shard shard
replica
replica
replica
replica

13. Horizontal scaling -­ the API
Create / remove replicas on the fly -­
Collections API
Moving shards around the cluster using
add / delete replica
Create / remove replicas on the fly -­
Update Indices Settings
Moving shards around the cluster using
Cluster Reroute API
Shard splitting using
Collections API
Automatic shard balancing
by default
Migrating data with a given routing key
to another collection using API
Shard allocation awareness & rule
based shard placement

14. The products -­ assumptions
Steady data growth
Common, known
data structure
Large QPS
Spikes in traffic

15. Hardware and data
2 x EC2 c3.large instances
(2vCPU, 3.5GB RAM,
2x16GB SSD in RAID0)
vs
Wikipedia

16. Test requests
One, common query
https://github.com/sematext/berlin-­buzzwords-­samples/tree/master/2015
Dictionary of common and
uncommon terms
JMeter hammering

17. Product search @ 5 threads

18. The products -­ summary
prepare
for high
QPS
plan for
data
growth
Both are fast, but with wikipedia
<

19. The products -­ summary
prepare
for high
QPS
plan for
data
growth
Both are fast, but with wikipedia
<

20. Hardware and data (2nd try)
2 x EC2 c3.large instances
(2vCPU, 3.5GB RAM,
2x16GB SSD in RAID0)
vs
Video
search
video video
video video
video video
video
video

21. Real product search @ 20 threads

22. Real product search @ 20 threads

23. The products – real summary
prepare
for high
QPS
plan for
data
growth
With video both are fast and
configuration matters

24. The logs -­ assumptions
Logs
Logs
Logs
Logs
Logs
Logs
Logs
Logs
Logs
Logs
Logs
Logs
Logs
Logs
Logs
Logs
Logs
Logs Logs
Logs
Lots of data
No updates
Low query count
Time oriented
data

25. Hardware and data
2 x EC2 c3.large instances
(2vCPU, 3.5GB RAM,
2x16GB SSD in RAID0)
vs
Logs
Logs
Logs
Logs
Logs
Logs
Logs
Logs
Logs
Logs
Logs
Logs
Logs
Logs
Logs
Logs
Logs
Logs Logs
Logs
Apache logs

26. Tuning
refresh_interval: 5s
doc_values: truedocValues: true
soft autocommit: 5s
catch all field: on _all: enabled
hard autocommit: 200mb flush_threshold_size: 200mb

27. Test requests
Filters Aggregations/Facets
filter by client IP date histogram
filter by word in user
agent
top 10 response codes
wildcard filter on domain # of unique IPs
top IPs per response per time
https://github.com/sematext/berlin-­buzzwords-­samples/tree/master/2015

28. Test runs
1. Write throughput
2. Capacity of a single index
3. Capacity with time-­based indices on
hot/cold setup

29. Single index write throughput

30. Single index @ max EPS

31. Single index @ 400 EPS

32. Time-­based indices
smaller indices
lighter indexing
easier to isolate hot data from cold data
easier to relocate
bigger indices
less RAM
less management overhead
smaller cluster state

33. Hot / Cold in practice
cron job does everything
creates indices in advance, on the hot
nodes (createNodeSet property)
Uses node properties (e.g. node.tag)
index templates + shard allocation
awareness = new indices go to hot
nodes automagically
Optimizes old indices, creates N new
replicas of each shard on the cold nodes
Removes all the replicas from the hot
nodes
Cron job optimizes old indices and
changes shard allocation attributes =>
shards get moved and Do on the cold
nodes

34. Time-­based: 2 hot and 2 cold
Before
AfterAfter
Before

35. Time-­based: 2 hot and 2 cold

36. The logs -­ summary
Faceting
use time
based indices
use hot / cold
nodes policy
Filtering

37. One summary to rule them all
Differences in configuration often matter
more than differences in products
Before the tests we expected results to be
totally opposite in both use cases
Do your own tests with your data and
queries before jumping into conclusions

38. We are hiring
Dig Search?
Dig Analytics?
Dig Big Data?
Dig Performance?
Dig Logging?
Dig working with and in open – source?
We’re hiring world – wide!
http://sematext.com/about/jobs.html

39. Call me, maybe?
Radu Gheorghe
@radu0gheorghe
radu.gheorghe@sematext.com
Rafał Kuć
@kucrafal
rafal.kuc@sematext.com
Sematext
@sematext
http://sematext.com