Presented by Stephane Gamard, Chief Technology Officer, Searchbox
In this presentation we aim to show how to make a high availability Solr cloud with 4.1 using only Solr and a few bash scripts. The goal is to present an infrastructure which is self healing using only cheap instances based on ephemeral storage. We will start by providing a comprehensive overview of the relation between collections, Solr cores, shardes, and cluster nodes. We continue by an introduction to Solr 4.x clustering using zookeeper with a particular emphasis on cluster state status/monitoring and solr collection configuration. The core of our presentation will be demonstrated using a live cluster.
We will show how to use cron and bash to monitor the state of the cluster and the state of its nodes. We will then show how we can extend our monitoring to auto generate new nodes, attach them to the cluster, and assign them shardes (selecting between missing shardes or replication for HA). We will show that using a high replication factor it is possible to use ephemeral storage for shards without the risk of data loss, greatly reducing the cost and management of the architecture. Future work discussions, which might be engaged using an open source effort, include monitoring activity of individual nodes as to scale the cluster according to traffic and usage.
6. Lucene revolu+on 2013
Solr Clustering
6
A6er 4.x:
Nodes
Automa+c Rou+ng
Simple Provisioning
Node
Monitoring
Node Node Node
ZK
NodeNode Node
ZK
ZK
LB LB
Thank you to the SolrCloud Team !!!
7. Lucene revolu+on 2013
What is SolrCloud?
7
Backward compa=bility
• Plain old Solr (with Lucene 4.x)
• Same schema
• Same solrconfig
• Same plugins
Some plugins might need update (distrib)
10. Lucene revolu+on 2013
What is SolrCloud?
10
Automa=c Rou=ng
Node
Monitoring
Node Node Node
ZK
NodeNode Node
ZK
ZK
LB LB
• Smart client connect to ZK
• Any node can forward a
requests to node that can
process it
11. Lucene revolu+on 2013
What is SolrCloud?
11
Collec=on API
• Abstrac+on level
• An index is a collec+on
• A collec+on is a set of shards
• A shard is a set of cores
• CRUD API for collec+on
“Collec?ons represents a set of cores with
iden)cal configura?on. The set of cores of
a collec?on covers the en?re index”
12. Lucene revolu+on 2013
What is SolrCloud?
12
Node
Core
Shard
Collec=on Abstrac+on level of interac+on & config
Scaling factor for collec+on size (numShards)
Scaling factor for QPS (replica?onFactor)
Scaling factor for cluster size (liveNodes)
=> SolrCloud is highly geared toward horizontal scaling
19. Lucene revolu+on 2013 19
SolrCloud -‐ Core Sizing
Heuris=cally inferred from “experience”
• Size on shard, not collec+on
• Do NOT starve resources on nodes
• Senle for JVM/Disk sizing
• Large amount of spare disk (op+mize)
RAM Disk
3 G 60 G
22. Lucene revolu+on 2013 22
SolrCloud -‐ Provisioning
Stand-‐by nodes
• Automa+cally assigned as replica
• provides a metric of HA
Node addi=on * (self healing)
• Scheduled check on cluster conges+on
• Automa+cally spawn new nodes per need
23. Lucene revolu+on 2013 23
SolrCloud -‐ Conclusion
Using SolrCloud is like juggling
• Gets bener with prac+ce
• There is always some magic leq
• Could become very overwhelming
• When it fails you loose your balls
Test -‐> Test -‐> Test -‐> some more Tests -‐> Test
24. Lucene revolu+on 2013 24
What would make our current SolrCloud cluster
even more awesome:
• Balance/distribute core based on machine
load
• Standby core (replicas not serving request
and auto-‐shurng down
Next Steps
25. Lucene revolu+on 2013 25
Requirement for solrCloud:
• Solr Mailing list: solr-‐user@lucene.apache.org
Further informa+on
• blogs & feed: hnp://www.searchbox.com/blog/
• Searchbox email: contact@searchbox.com
Further Informa+on
26. Lucene revolu+on 2013
CONFERENCE PARTY
The Tipsy Crow: 770 5th Ave
Starts after Stump The Chump
Your conference badge gets
you in the door
TOMORROW
Breakfast starts at 7:30
Keynotes start at 8:30
CONTACT
Stephane Gamard
stephane.gamard@searchbox.com
26Lucene revolu+on 2013