How SolrCloud Changes the User Experience In a Sharded Environment


Published on

Presented by Erick Erickson, Lucid Imagination - See conference video -

The next major release of Solr (4.0) will include "SolrCloud", which provides new distributed capabilities for both in-house and externally-hosted Solr installations. Among the new capabilities are: Automatic Distributed Indexing, High Availability and Failover, Near Real Time searching and Fault Tolerance. This talk will focus, at a high level, on how these new capabilities impact the design of Solr-based search applications primarily from infrastructure and operational perspectives.

Published in: Technology

How SolrCloud Changes the User Experience In a Sharded Environment

  1. 1. How SolrCloud Changes the Erick Erickson, LucidUser Experience In a ImaginationSharded EnvironmentLucene Revolution, 9-May-2012
  2. 2. Who am I?!   “Erick is just some guy, you know” •  Your geekiness score is increased if you know where that quote comes from, and your age is hinted at!   30+ years in the programming business, mostly as a developer!   Currently employed by Lucid Imagination in Professional Services •  I get to see how various organizations interpret “search” and I’m amazed at the different problems Solr is used to solve!   Solr/Lucene committer!!   Sailor, anybody need crew for sailboat delivery? 2
  3. 3. What we’ll cover!   Briefly, what else is coming in 4.0! SolrCloud (NOT Solr-in-the-cloud), upcoming in 4.0 •  What it is •  Why you may care!   Needs SolrCloud addresses •  DR/HA •  Distributed indexing •  Distributed searching!   I’m assuming basic familiarity with Solr 3
  4. 4. I’m not the implementer, Mark is!   Well, Mark Miller and others!   Mark’s talk (tomorrow) is a deeper technical dive, I recommend it highly •  Anything I say that contradicts anything Mark says, believe Mark − After all, he wrote much of the code!   Mark insisted on the second slide after this one 4
  5. 5. 5
  6. 6. 6
  7. 7. When and Where can we get 4.0?!   When will it be released? Hopefully 2012 •  Open Source; have you ever tried herding cats? •  Alpha/Beta planned, this is unusual •  3.6 probably last 3x release!   How usable are nightly builds? •  LucidWorks Enterprise runs on trunk, so trunk is quite stable and in production!   There’s lots of new stuff! •  “unstable” doesn’t really mean unstable code − Changing APIs, index format may change!   Nightly builds:!   Source code and build instructions: HowToContribute 7
  8. 8. Cool stuff in addition to SolrCloud in 4.0 8
  9. 9. Other cool 4.0 (trunk) features!   Similarity calculations decoupled from Lucene. !   Scoring is pluggable !   There are several different OOB implementations now (e.g. BM25)!   FST (Finite State Automata/Transducer) based work. Speed and size improvements !   FST for fuzzy queries, 100x faster (McCandless’ blog)!   You can plug in your own index codec. See pulsing and SimpleTextCodec. This is really your own index format •  Can be done on a per field basis •  Text output as an example!   Much more efficient in-memory structures!   NRT (Near Real Time) searching and “soft commits”!   Spatial (LSP) rather than spatial contrib 9
  10. 10. More cool new features!   Adding PivotFacetComponent for Hierarchical faceting. See Yoniks presentation, “useful URLs” section!   Pseudo-join queries – See Yonik’s presentation URL in “useful URLs” section!   New Admin UI!   Can’t over-emphasize the importance of CHANGES.txt •  Solr •  Lucene •  Please read them when upgrading. Really 10
  11. 11. SolrCloud setup and use 11
  12. 12. What is SolrCloud! SolrCloud is a set of new distributed capabilities in Solr that: •  Automatically distributes updates (i.e. indexes documents) to the appropriate shard •  Uses transaction logs for robust update recovery •  Automatically distributes searches in a sharded environment •  Automatically assigns replicas to shards when available •  Supports Near Real Time searching (NRT) •  Uses Zookeeper as a repository for cluster state 12
  13. 13. Common pain points (why you may care)!   Every large organization seems to have a recurring set of issues: •  Sharding – have to do it yourself, usually through SolrJ or similar. •  Capacity expansion – what to do when you need more capacity •  System status – getting alerts when machines die •  Replication – configuration •  Finding recently-indexed data – everyone wants “real time” − Often not as important as people think, but... •  Inappropriate configuration − Trying for “real time” by replicating every 5 seconds − Committing every document/second/packet − Mismatched schema or config files on masters and slaves 13
  14. 14. Common Pain Points (Why you may care)!   Maintaining different configuration files (and coordinating them) for masters and slaves! SolrCloud addresses most of these.! SolrCloud is currently “a work in progress” 14
  15. 15. Typical sharding setup Indexing  ! Multiple Indexers! Query Slaves •  1 or more per indexer! Yes, you can shard & distribute Load  Balancer   Searching  
  16. 16. Steps to set this up!   Figure out how many shards required!   Configure all masters, which may be complex •  Point your indexing at the appropriate master!   Configure all slaves •  Configure distributed searching •  Make sure the slaves point at the correct master •  Find out where you mis-configured something, e.g. “I’m getting duplicate documents”.. Because you indexed the same doc to two shards? •  Deal with your manager wanting to know why the doc she just indexed isn’t showing up in the search (replication delay) •  Rinse, Repeat… 16
  17. 17. How is this different with SolrCloud?!   Decide how many shards you need!   Ask the ops folks how many machines you can have!   Start your servers: •  On the Zookeeper machine (s): java -Dbootstrap_confdir=./solr/conf - DzkRun –DnumShards=### -jar start.jar •  On all the other machines: java –DzkHost=<ZookeeperMachine:port> [,<ZookeeperMachine:port>…] -jar start.jar!   Index any way you want •  To any machine you want, perhaps in parallel!   Send search to any machine you want!   Note: Demo uses embedded Zookeeper •  Most production installations will probably use “ensembles” 17
  18. 18. Diving a little deeper (indexing) 18
  19. 19. Diving a little deeper (indexing)!   How are shard machines assigned? •  It’s magic, ask Mark. •  As each machine is started, it’s assigned shard N+1 until numShards is reached •  The information is recorded in Zookeeeper where it’s available to all!   How are leaders elected? •  Initially, on a first-come-first-served basis, so at initial setup each shard machine will be a leader (numShards == num available machines)!   How are replicas assigned? •  See above (magic), but conceptually it’s on a “round robin” basis •  As each machine is started for the first time, it’s assigned to the shard with the fewest replicas (tie-breaking on lowest shard ID) 19
  20. 20. Assigning machines ZK Host(s ) Leader shard1-DnumShards=3-Dbootstrap_confdir=./solr/conf-DzkHost=<host>:<port>[,<host>:<port>] 20
  21. 21. Assigning machines ZK Host(s ) Leader Leader shard1 shard2-DzkHost=<host>:<port>[,<host>:<port>] 21
  22. 22. Assigning machines ZK Host(s ) Leader Leader Leader shard1 shard2 shard3-DzkHost=<host>:<port>[,<host>:<port>]At this point you can index and search, you have one machine/shard 22
  23. 23. Assigning machines ZK Host(s ) Leader Leader Leader shard1 shard2 shard3 Replica shard1-DzkHost=<host>:<port>[,<host>:<port>] 23
  24. 24. Assigning machines ZK Host(s ) Leader Leader Leader shard1 shard2 shard3 Replica Replica shard1 shard2-DzkHost=<host>:<port>[,<host>:<port>] 24
  25. 25. Assigning machines ZK Host(s ) Leader Leader Leader shard1 shard2 shard3 Replica Replica Replica shard1 shard2 shard3-DzkHost=<host>:<port>[,<host>:<port>] 25
  26. 26. Diving a little deeper (indexing)!   Let’s break this up a bit!   There really aren’t any masters/slaves in SolrCloud •  “Leaders” and “replicas”. Leaders are automatically elected − Leaders are just a replica with some coordination responsibilities for the associated replicas •  If a leader goes down, one of the associated replicas is elected as the new leader •  You don’t have to do anything for this to work!   When you send a document to a machine for indexing the code (DistributedUpdateProcessor) does several things: •  If I’m a replica, forward the request to my leader •  If I’m a leader − determine which shard each document should go to and forwards the doc (in batches of 10 presently) to that leader − Indexes any documents for this shard to itself and replicas 26
  27. 27. Diving a little deeper (indexing)!   When new machines are added and get assigned to a shard •  Probably an old-style replication will occur initially, it’s most efficient for bulk updates − This doesn’t require user intervention •  Any differences between the replication and the current state of the leader will be replayed from the transaction log until the new machine’s index is identical to the leader •  When this is complete, search requests are forwarded to the new machine 27
  28. 28. Diving a little deeper (indexing)!   Transaction log, huh?!   A record of updates is kept in the “transaction log”. This allows for more robust indexing •  Any time the indexing process in interrupted, any uncommitted updates can be replayed from the transaction log!   Synchronizing replicas has some heuristics applied. •  If there are “a lot” of updates (currently 100) to be synchronized, then an old-style replication is triggered •  Otherwise, the transaction log is “replayed” to synchronize the replica 28
  29. 29. Diving a little deeper (indexing)!   “Soft commits”, huh?!   Solr 4.0 introduces the idea of “soft commits” to handle “near real time” searching •  Historically, Solr required a “commit” to close segments. At that point: − New searchers were opened so those documents could be seen − Slaves couldn’t search new documents until after replication!   Think of soft commits as adding documents to an in-memory, writeable segment •  On a hard commit, the currently-open segment is closed and the in- memory structures are reset!   Soft commits can happen as often as every second!   Soft commits (and NRT) are used by SolrCloud, but can be used outside of the SolrCloud framework 29
  30. 30. Diving a little deeper (searching) and all the rest 30
  31. 31. Diving a little deeper (searching)!   Searching “just happens” •  There’s no distinction between masters and slaves, so any request can be sent to any machine in the cluster!   Searching is NRT. Since replication isn’t as significant now, this is automatic •  There is a small delay while the documents are forwarded to all the replicas!   Shard information does not need to be configured in Solr configuration files 31
  32. 32. Diving a little deeper (the rest)!   Capacity expansion!   System status!   Replication!   NRT!   Zookeeper 32
  33. 33. Capacity expansion!   Whew! Let’s say that you have your system running just fine, and you discover that you are running close to the edge of your capacity. What do you need to do to expand capacity? •  Install Solr on N more machines •  Start them up with the –DzkHost parameter •  Register them with your fronting load balancer •  Sit back and watch the magic!   Well, what about reducing capacity •  Shut the machines down 33
  34. 34. System Status!   There is a new Admin UI that graphically shows the state of your cluster, especially active machines!   But overall, sending alerts etc. isn’t in place today, although it’s under discussion 34
  35. 35. Replication!   But we’ve spent a long time understanding replication!!   Well, it’s largely irrelevant now. When using SolrCloud, replication is automatically handled •  This includes machines being temporarily down. When they come back up, SolrCloud re-synchronizes them with the master and forwards queries to them after they are synchronized •  This includes temporary glitches (say your network burps) 35
  36. 36. Finding Recently-indexed Docs (NRT)!   NRT has been a long time coming, but it’s here!   Near Real Time because there are still slight delays from 2 sources •  Until a “soft commit” happens, which can be every second •  Some propagation delay while incoming index requests are: − Perhaps forwarded to the shard leader − Forwarded to the proper shard − Forwarded to the replicas from the shard leader •  But these delays probably won’t be noticed 36
  37. 37. Zookeeper! ZooKeeper is “a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.”!   A lot of complexity for maintaining Solr installations is solved with Zookeeper!   Zookeeper is the repository for cluster state information!   See: 37
  38. 38. Using Zookeeper with SolrCloud!   The –DzkRun flag (in the demo) causes an embedded Zookeeper server to run in that server •  Simple to use in the tutorials, but probably not the right option for production •  An enterprise installation will probably run Zookeeper as an “ensemble”, external to Solr servers!   Zookeeper works on a quorum model where N/2+1 Zookeepers must be running •  It’s best to run an odd number of them (and three or more!) to avoid Zookeeper being a single point of failure!   Yes, setting up Zookeeper and making SolrCloud aware of them is an added bit of complexity, but TANSTAAFL (more age/geek points if you know where that comes from) 38
  39. 39. Gotchas!   This is new and changing •  Optimistic locking not fully in place yet •  At least one machine/shard must be running.!   _version_ is a magic field, don’t change it!   It’s a whole new world, some of your infrastructure is obsolete!   We’re on the front end of the learning curve!   Some indexing speed penalty!   This is trunk, index formats may change etc. 39
  40. 40. Useful URLs!   The Solr Wiki:!   Source code, builds, etc:!   Main Solr/Lucene website:!   Really good blogs: •  Simon Willnauer: •  Mike McCandless: •  Lucid Imagination:!   Lucene Spatial Playground/Spatial4J: 40
  41. 41. More useful URLs! DocumentWriterPerThread (DWPT) writeup (Simon Willnauer): you-have-i-can-use-them!/!   FST and fuzzy query 100X faster: times-faster.html!   Solr Cloud: •  NOT Solr-in-the-cloud!   Lucene JIRA:!   Solr JIRAs: 41
  42. 42. Even more useful URLs! Yonik Seeley presentations: •  See particularly the LuceneRevolution2011 presentation, re: pivot faceting.!   Grant Ingersoll’s memory estimator prototype (trunk)!   Memory improvements:!   Zookeeper 42
  43. 43. Thank You, Questions? Erick Erickson Erick.Erickson@lucidim