This document discusses scaling Apache Solr search clusters using Kubernetes. It describes running Solr Cloud clusters within Kubernetes namespaces, using stateful sets and headless services. It also covers monitoring Solr metrics with Prometheus and Grafana, exposing custom metrics via the Prometheus adapter, and implementing horizontal pod autoscaling in Kubernetes based on Solr metrics. Special cases around node removal and collection-specific scaling are also addressed. Lucidworks Managed Search is presented as an alternative to managing Solr clusters manually in Kubernetes.
5. 5
Kubernetes (K8s)
System for running and coordinating containerized applications across a cluster of machines
referred from: https://cloud.google.com/kubernetes-engine/kubernetes-comic/
6. 6
Run Solr Cloud within Kubernetes
Load Balancer / Ingress
(external IPs / custom route names)
Headless Service
for internal
communication
Service
IP is exposed
solr-0 solr-1 solr-2
…more solr pods
Service
IP is exposed
Headless Service
for internal
communication
zk-0 zk-1 zk-2
pv-0 pv-1 pv-2
pv-0 pv-1 pv-2
Stateful Set Stateful Set
HTTP Traffic
pv
persistent volume
7. 7
Solr Cloud Clusters Within Kubernetes
Namespace A Namespace B Namespace C
Namespace D Namespace E
Single
Kubernetes
Cluster
Solr Cloud
Cluster
Solr Cloud
Cluster
Solr Cloud
Cluster
Solr Cloud
Cluster
Solr Cloud
Cluster
17. 17
K8s Horizontal Pod Autoscaling
solr-0 solr-1 solr-2
Stateful Set
Horizontal Pod Autoscaling
(HPA) on metric ‘M’
If metric ‘M’ value > X,
Add more pods
avg metric ‘M’ value < X
solr-0 solr-1 solr-2
Stateful Set
solr-3
until avg metric ‘M’ value < X
19. 19
Solr Autoscaling
• Autoscaling in Solr provide balanced and stable cluster for various cluster change events.
• Achieved by satisfying a set of rules and sorting preferences
to select the target of cluster management operations.
S - A
S - B
R - 1
R - 1
R - 2
C
R - 2
Set solr-autoscaling policy and trigger
such that -
Add one replica to each shard of
collection
● if new Node is added
C - collection
S - shard
R - replica
20. 20
Solr Autoscaling
Set solr-autoscaling policy and trigger such that -
Add one replica to each shard of collection
● if new Node is added "node_added_trigger":{
"event":"nodeAdded",
"waitFor":60,
"preferredOperation":"ADDREPLICA",
"actions":[
{
"name":"compute_plan",
"class":"solr.ComputePlanAction"},
{
"name":"execute_plan",
"class":"solr.ExecutePlanAction"}],
}
Trigger Policy
{
"cluster-policy":[
{
"node":"#ANY",
"replica":"<2",
"shard":"#EACH"}
]
}
Cluster Policy
21. 21
Handling Special Cases
● What happens when nodes are removed due to scaling?
Is there a potential of losing data?
○ All replicas of collection gets hosted on scaled nodes.
● Do we need to scale on each collection on the entire cluster?
How to configure scaling on specific collection or set of collections with
common attribute?
● How does the Node Added Trigger behaves when a node gets restarted?
Replicas will be added to each shard of each collection where it was
previously not.