Battle of the Giants round 2

Battle of the Giants
Rafał Kuć – Sematext Group, Inc.
@kucrafal @sematext sematext.com

Ich bin ein…
Sematext consultant & engineer
Solr Cookbook series author
„ElasticSearch Server” author
„Mastering ElasticSearch” author
Solr.pl co-founder
Father and husband 
Copyright 2013 Sematext Group. Inc. All rights reserved

Under the Hood
Lucene 4.3Lucene 4.3

Expectations
Scalability
Fault toleranance
High availablity
Features
Manageability
Ease of installation
Tools
Support

Expectations vs Reality
Only ElasticSearch nodes
Single leader
Solr + ZooKeeper
Leader per shard
Distributed
Fault tolerant
Automatic leader election

All Time Top Committers

Active Contributors

The Code

The Mailing Lists

Trends

Collection vs Index
Collections and Indices can be spread among
different nodes in the cluster
Collection – main
logical index
Index – main
logical structure

Apache Solr Index Structure
Field and types defined in schema
Automatic value copying
Dynamic fields
Custom similarity
Custom postings format
Multiple document types require shared schema
Can be read using API

ElasticSearch Index Structure
Schema - less
Fields and types defined with HTTP API
Multi – field support
Nested and parent – child documents
Custom similarity
Custom postings format
Multiple document with different structure
Can be read and written using API

Shards and Replicas
Many shards
0 or more replicas
Replica can become leader
Replicas can be created on
live cluster

Configuration
Static in solrconfig.xml
Can be reloaded with
core reload
Static in elasticsearch.yml
Changable at runtime

Discovery
Zen DiscoveryApache Zookeeper

Solr & ZooKeeper
Requires additional software
Prevents split – brain situations
Holds collections configurations
ZooKeeper ensemble needed

ElasticSearch Zen Discovery
Automatic node discovery
Multicast and unicast discovery methods
Automatic master detection
Two - way failure detection

HTTP FTW
HTTP REST API in ElasticSearch or Query String
for simple queries
HTTP with Query String in Apache Solr
Both provide specialized Java API

Results Grouping
Group on:
field value
query result
function query

Prospective Search
Called Percolator
Matches documents to stored queries

Full Text Search Capabilities
Variety of queries
Control score calculation
Different query parsers
Advanced Lucene queries

Score Calculation
Leverage Lucene scoring
Control importance of:
documents
queries
terms
phrases
Similiarity configuration

Apache Solr and Score Influence
Index - time boosting
Query - time
Term boosts
Field boosts
Phrases boost
Function queries
Sub-queries used for boosting

ElasticSearch and Score Influence
Index - time
Query - time
Different queries provide different boost controls
Can calculate distributed term frequencies
Negative and Positive boosting queries
Custom score filters
Scripts

ElasticSearch Query Rescore
Reorders top N hits by using other query
Executed on shards before results are returned
to the node handling it
Not executed with scan and count

ElasticSearch Nested Objects
Indexed as separate documents
Stored in the same part of index as root doc
Hidden from standard queries and filters
Need appropriate queries and filters (nested)
Top level documents can be sorted on the basis
of nested ones

Solr Parent – Child Relationship
Used at query time
Multi core joins possible
select?q={!join from=parent to=id}color:Yellow

ElasticSearch Parent – Child
Proper indexing required
Indexed as separate documents
Standard queries don’t return child documents
Retrieve parent docs using queries and filters
(has_child, has_parent, top_children)

Filters
Used to narrown down query results
Good candidates for caching and reuse
Addictive
Can use different query parsers
Can use local params
Narrows down faceting results
Defined using Query DSL
Can be used for score calculation
Doesn’t narrow down faceting
results

Faceting
Terms
Range & query
Terms statistics
Spatial distance
Pivot Histograms

Real Time Or Not ?
Get not yet indexed docs from transaction log
Don’t need searcher reopening
Separate Get and
Multi Get API
Separate Realtime Get
Handler

Data Handling
Single and batch indexing supported
JSON in / JSON out
(and YAML)
Different formats allowed
(XML, JSON, CSV, binary)

Partial Document Updates
Not based on LUCENE-3837
Server-side doc reindexing
Both servers use versioning
Decreases network traffic

Apache Solr Partial Doc Update
Sent to the standard update handler
Requires _version_ field
curl 'localhost:8983/solr/update?commit=true' -H
'Content-type:application/json' -d '[ {
"id" : "12345",
"enabled" : {
"set" : true
}
} ]'

ElasticSearch Partial Doc Update
Special end – point exposed - _update
Supports parameters like routing, parent,
replication, percolate, etc (similar to Index API)
Uses scripts to perform document updates
curl -XPOST 'localhost:9200/sematext/test/12345/_update' -d '{
"script" : "ctx._source.enabled = enabled",
"params" : {
"enabled" : true
}
}'

Solr Collections API
Collection
creation
reload
deletion
shards splitting

ElasticSearch Indices REST API
Index
creation
deletion
closing and opening
refreshing
existence checking

Apache Solr Shard Splitting
admin/collections?action=SPLITSHARD&collection=collection1&shard=shard1

Cluster State Monitoring
Multiple MBeans exposed by
JMX
Multiple REST end – points
exposed to get different
statistics

ElasticSearch Statistics API
Health and state check
Nodes information
Cache statistics
Segments information
Index information
Mappings information
SPM – „One to rule them all”

ElasticSearch Cluster Settings Update
Control
rebalancing
recovery
allocation
Change cluster configuration properties

ElasticSearch Custom Shard Allocation
Cluster level:
Index level:
curl -XPUT localhost:9200/_cluster/settings -d '{
"persistent" : {
"cluster.routing.allocation.exclude._ip" : "192.168.2.1"
}
}'
curl -XPUT localhost:9200/sematext/_settings/ -d '{
"index.routing.allocation.include.tag" : "nodeOne,nodeTwo"
}'

Moving Shards and Replicas
Move shards between nodes on demand
curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
"commands" : [
{"move" : {"index" : "sematext", "shard" : 0, "from_node" : "node1",
"to_node" : "node2"}},
{"allocate" : {"index" : "sematext", "shard" : 1, "node" : "node3"}}
]
}'

The Verdict

And The Winner Is ?

We Are Hiring !
Dig Search ?
Dig Analytics ?
Dig Big Data ?
Dig Performance ?
Dig working with and in open – source ?
We’re hiring world – wide !
http://sematext.com/about/jobs.html

Rafał Kuć
@kucrafal
rafal.kuc@sematext.com
Sematext
@sematext
http://sematext.com
http://blog.sematext.com
ElasticSearch Server 25% off:
MREESS25
Thank You !

Battle of the Giants round 2

More Related Content

What's hot

Viewers also liked

Similar to Battle of the Giants round 2

Recently uploaded

Battle of the Giants round 2