This document discusses deploying and managing Apache Solr at scale. It introduces the Solr Scale Toolkit, an open source tool for deploying and managing SolrCloud clusters in cloud environments like AWS. The toolkit uses Python tools like Fabric to provision machines, deploy ZooKeeper ensembles, configure and start SolrCloud clusters. It also supports benchmark testing and system monitoring. The document demonstrates using the toolkit and discusses lessons learned around indexing and query performance at scale.

1. Deploying and managing Solr at Scale

2. Who am I?
• Anshum Gupta, Apache Lucene/Solr committer,
Lucidworks Employee.
• Interested in search and related stuff.
• Apache Lucene since 2006 and Solr since 2010.
• Organizations I am or have been a part of:

3. Apache Solr has a huge install base and tremendous momentum
most widely used search
solution on the planet.
8M+
total downloads
Solr is both established & growing
250,000+
monthly downloads
Solr has tens of thousands
of applications in production.
You use Solr everyday.
2500+open Solr jobs.
Activity Summary
30 Day summary
Dec 06, 2014 - Jan 05, 2015
• 135 Commits
• 17 Contributors
via https://www.openhub.net/p/solr
12 Month Summary
Jan 5, 2014 — Jan 5, 2015
• 1363 Commits
• 30 Contributors

4. Getting started with Solr
• Download
• Untar/Unzip
• bin/solr start -e cloud -noprompt
• open http://localhost:8983/solr

5. Recent usability improvements
• Start scripts
• Schema APIs
• Config API - Register custom handlers using API
• Status APIs and more….

6. SolrCloud Architecture
Shard 1
(leader)
Followers
Shard 2
(leader)
Followers
ZooKeeper
Ensemble
Multiple Nodes = Need for Coordination

7. Production scale?
• Zk ensemble. NOT embedded
• Multiple nodes
• Manually (or script) the 4 steps for each node?

8. Solr Scale Toolkit
• Open Source!
• Fabric (Python) toolset for deploying and managing SolrCloud
clusters in the cloud
• Code to support benchmark tests (Pig script for data generation /
indexing, JMeter samplers)
• EC2 for now, more cloud providers coming soon via Apache
libcloud
• No *need* to know Python!

9. The building blocks: A lot of python!
• boto – Python API for AWS (EC2, S3, etc)
• Fabric – Python-based tool for automating system admin tasks over SSH
• pysolr – Python library for Solr (sending commits, queries, ...)
• kazoo – Python client tools for ZooKeeper
• Supporting Cast:
• JMeter – run tests, generate reports
• collectd – system monitoring
• Logstash4Solr – log aggregation
• JConsole/VisualVM – monitor JVM during indexing / queries

10. Overview of features:
• Provisioning N machine instances in EC2
• Configuring / starting ZooKeeper (1 to n servers)
• Configuring / starting N Solr instances in cloud
mode (M x N nodes)
• Integrating with Logstash4Solr and other
supporting services, e.g. collectd
• Day-to-day operations on an existing cluster

11. N X M SolrCloud Nodes
ZK Host N
Node 1: Custom AMI
Architecture
Solr-Scale-Toolkit
SiLK
ZK Host 1
ZooKeeper 1
ZK Ensemble
Meta Node
Solr Node 1: 8983
core
core
core
Solr Node N: 89xx
core
core
core
ZooKeeper N
X M such machines
system monitoring
of M machines w/
collectd and JMX

12. Provisioning cluster nodes
• Custom built AMI (one for PV instances and one for HVM instances) – Amazon Linux
• Dedicated disk per Solr node
• Launch and then poll status until they are live
• Verify SSH connectivity
• Tag each instance with a cluster ID and username
fab new_ec2_instances:test1,n=3,instance_type=m3.xlarge

13. Deploy ZooKeeper ensemble
• Two options to use the ensemble:
• Provision 1 to N nodes when you launch Solr cluster
• use existing named ensemble
• Fabric command simply creates the myid files and zoo.cfg file for the
ensemble
• and some cron scripts for managing snapshots
• Basic health checking of ZooKeeper status:
• echo srvr | nc localhost 2181
fab
new_zk_ensemble:zk1,n=3

14. Deploy SolrCloud cluster
• Uses bin/solr in Solr 4.10 to control Solr nodes
• Set system props: jetty.port, host, zkHost, JVM opts
• One or more Solr nodes per machine
• JVM mem opts dependent on instance type and # of Solr nodes per
instance
• Optionally configure log4j.properties to append messages to Rabbitmq
for SiLK integration
fab
new_solrcloud:test1,zk=zk1,nodesPerHost=2

15. Demo
• Launch ZooKeeper Ensemble
• 3 nodes to establish quorum
• Launch SolrCloud cluster
• Create new collection and index some docs
• Run a healthcheck on the collection

16. Dashboards

17. Other useful stuff
• patch from a local build.
• fab mine: See clusters I’m running (or for other users too)
• fab kill_mine: Terminate all instances I’m running
• fab ssh_to: Quick way to SSH to one of the nodes in a cluster
• fab stop/recover/kill: Basic commands for controlling specific
Solr nodes in the cluster
• fab jmeter: Execute a JMeter test plan against your cluster
• Example test plan and Java sampler is included with the source

18. Testing Methodology
• Transparent repeatable results
• Ideally hoping for something owned by the community
• Synthetic docs ~ 1K each on disk, mix of field types
• Data set created using code borrowed from PigMix
• English text fields generated using a Zipfian distribution
• Java 1.7u67, Amazon Linux, r3.2xlarge nodes
• enhanced networking enabled, placement group, same AZ
• Stock Solr (cloud) 4.10
• Using custom GC tuning parameters and auto-commit settings
• Use Elastic MapReduce to generate indexing load
• As many nodes as I need to drive Solr!

19. Indexing performance
Cluster Size # of Shards # of Replicas Reducers Time (secs) Docs / sec
10 10 1 48 1762 73,780
10 10 2 34 3727 34,881
10 20 1 48 1282 101,404
10 20 2 34 3207 40,536
10 30 1 72 1070 121,495
10 30 2 60 3159 41,152
15 15 1 60 1106 117,541
15 15 2 42 2465 52,738
15 30 1 60 827 157,195
15 30 2 42 2129 61,062

20. Indexing performance lessons
• Solr has no built-in throttling support – will accept work until it
falls over; need to build this into your indexing application
logic
• Oversharding helps parallelize indexing work and gives you an
easy way to add more hardware to your cluster
• GC tuning is critical
• Auto-hard commit to keep transaction logs manageable
• Auto soft-commit to see docs as they are indexed
• Replication is expensive! (Work in progress, SOLR-6816)

21. Query Performance
• Still a work in progress!
• Sustained QPS & Execution time of 99th Percentile
• Stable: ~5,000 QPS / 99th at 300ms while indexing ~10,000 docs / sec
• Using the TermsComponent to build queries based on the terms in each
field.
• Harder to accurately simulate user queries over synthetic data
• Need mix of faceting, paging, sorting, grouping, boolean clauses, range
queries, boosting, filters (some cached, some not), etc ...
• Start with one server (1 shard) to determine baseline query performance.
• Look for inefficiencies in your schema and other config settings

22. More on query performance…
• Higher risk of full GC pauses (facets, filters, sorting)
• Use optimized data structures (DocValues) for facet / sort fields, Trie-
based numeric fields for range queries, facet.method=enum for low
cardinality fields
• Add more replicas; load-balance
• -Dhttp.maxConnections=## (default = 5, increase to accommodate
more threads sending queries)
• Avoid increasing ZooKeeper client timeout ~ 15000 (15 seconds) is
about right
• Don’t just keep throwing more memory at Java! –Xmx128G

23. Roadmap
• Not just AWS
• No need for custom AMI, configurable download
paths and versions.

24. Questions?

25. References
• Solr scale toolkit
• Blog: http://lucidworks.com/blog/introducing-
the-solr-scale-toolkit/
• Podcast: http://solrcluster.podbean.com/e/tim-
potter-on-the-solr-scale-toolkit/
• github: https://github.com/LucidWorks/solr-
scale-tk

26. Connect @
http://www.twitter.com/anshumgupta
http://www.linkedin.com/in/anshumgupta/
anshum@apache.org