Many Scylla maintenance operations require significant data movement between database nodes in a cluster. It is not an easy task to make the management operations efficient while maintaining minimum impact on the workload all the time. In this talk, we will share how we made those maintenance operations easier, safer and faster with the new Scylla features and improvements, e.g., seedless, repair based node operations, smarter off-strategy compaction, io bandwidth limiter for repair and compaction, parallel repair in Scylla Manger and more.

1. How We Made Scylla
Maintenance Easier Safer and Faster
Asias He

2. Asias He
Software Developer, ScyllaDB
Asias He is a long-time open source developer who previously worked on
Debian Project, Solaris Kernel, KVM Virtualization for Linux and OSv
unikernel. He now works on Seastar and Scylla.
2

3. Agenda
▪ Motivation
▪ Scylla maintenance improvements
▪ Summary
3

4. Motivation
4

5. Motivation
▪ Maintenance is an important job to operate Scylla
▪ Require significant data movement
▪ Maintain minimum impact on your workload
▪ Set a goal to make operating Scylla as easy as possible
5

6. Towards better elasticity and
maintainability
▪ Scylla scales horizontally and vertically
▪ Users enjoy the performance of Scylla
▪ But maintenance has rough edges
• Some maintenance operations are complicated and not
resilient enough
• Compaction causes longer time to add node
6

7. Scylla Maintenance
Improvements
7

8. Seedless
▪ Seed nodes
• Help the discovery of the cluster and propagation of gossip
• Special role breaks “every node is symmetric” architecture
• Special treatment and procedures
▪ The seed concept is removed (Scylla 4.3)
• Seed config --seed-provider-parameters
• Remains for compatibility
• Serves as contact points
• Used once when adding new node to the cluster
8

9. Seedless
▪ Simplified cluster operations
• No more special treatment for seed nodes
• Seed node bootstraps
• Dangerous auto_bootstrap is gone
• No need to replace a dead seed node
• No need to promote a non-seed to seed
• No need to update seed config after node is bootstrapped
• Easier planing
• No more selecting 2 or 3 nodes per DC as seed nodes
▪ More details on blog post
9

10. Improvements in row-level repair
▪ Introduced since Scylla 3.1
• Repair at the granularity of row instead of partitions
▪ Continuous performance and stability improvements
• Generate RF -1 times less sstables during repair (Scylla 4.4)
• Before: one sstable per peer nodes per vnode
• After: one sstable for all peer nodes per vnode
• Reduce reactor stalls to reduce latency impact
• Futurize operations take time
• Switch to btree_set to store repair_hash
10
Node2
Node1
Node 3
SST for Node1
SST for Node2
Node2
Node1
Node 3
SST for Node1
& Node2

11. Improvements in row-level repair
▪ Continuous performance and stability
improvements
• Use UUID to identify a repair job
• Synchronous API to query repair status
• Progress report on metrics
▪ More row level repair internals on blog post
11

12. Repair based node operations
▪ Use repair for all node operations
• Use row level repair as the underlying mechanism to
sync data between nodes instead of streaming
• Single mechanism for all the node operations
• Bootstrap / replace / rebuild
• Decommission / removenode
• Repair
The
Speaker’s
camera
displays
here
12

13. Repair based node operations
▪ Significant improvements on performance and data safety
• Resumable
• Resume from previous failed bootstrap ops
• Consistency
• Latest replica is guaranteed
• Simplified
• No need to run repair before or after node operations like replace and removenode
• Unified
• All node ops use the same underlying mechanism
▪ More details:
The
Speaker’s
camera
displays
here
13

14. Redesigned replace operation
▪ Replace an existing dead node
• Common operation to recover a node
▪ Replace operation before Scylla 4.1
• The replace operation takes time to sync data
• Replacing node takes no writes during the operation
• Missing the writes during that time
• Repair is needed after the replace operation for the missing writes
14

15. Redesigned replace operation
▪ Simpler faster and safer replace operation
(Scylla 4.1)
• No missing writes on the replacing node
• Ensured by the “replacing node takes writes” feature
• Replacing node has latest replica
• Ensured by the “repair based node ops” feature
• Note: Disabled by default, use --enable-repair-based-
node-ops true to enable
• With the above two features
• No repair is needed before or after the replace operation
15

16. Make removenode safe by default
▪ Current removenode works in “best-effort mode”
• Remove a dead node from the cluster
• Succeed even if part of the data owned by removed
node has not been synced to new owners
• Might cause data consistency issues
16

17. Make removenode safe by default
▪ With “safe mode” feature (Scylla 4.4)
• Fail the operation if it is not safe
• By default, require all relevant nodes to be alive
• Allow ignore dead nodes explicitly to run in best-effort mode
• E.g., nodetool removenode --ignore-nodes node1,node2
• More resilient in case of error
• Automatically revert to previous state
17

18. ▪ Introduce off-strategy compaction
• Work-in-Progress
• Sstables generated by node operations
are kept in a separate data set
• Compact them together and integrate to main
set when node operation is done
• Less compaction work during node operations
• Faster to add / remove node
Smarter compaction during
node operations
18
No compaction
during replace with
off-strategy
Compaction during
replace without off-
strategy

19. ▪ Introduce bandwidth and IOPS
limiter for repair and compaction
• Work-in-Progress
• Limit the max disk bandwidth and
IOPS allowed
• More control over speed of repair
and other node operations like
add or remove node
• Less impact on your workload
Better control of repair and
compaction intensity
19
20 MB/s Cap
10 MB/s Cap
40 MB/s Cap

20. ▪ Backup throughput is bounded
▪ De-duplicates at file level
▪ Scylla manager and the agent runs under cgroup
to reduce interfere with Scylla Core
Scylla Manager Backup and
Repair improvements
20

21. ▪ Faster repair with Scylla Manager 2.2
• Run repair jobs on multiple nodes in parallel
• Ensures that each node takes part in at most one Scylla
repair job at all times
• Allow adjustment of repair intensity to a node capabilities
• Changing the repair speed in flight
• Small table optimization
• More on blog
Scylla Manager Backup and
Repair improvements
21

22. Summary
22

23. Summary
▪ We covered recent improvements to make
maintenance easier safer and faster
• Seedless
• Repair based node operations & redesigned replace / removenode operation
• Off-strategy compaction & IO rate limiter
• Scylla Manger improvements
▪ We will continue to invest more to make operating scylla
as easy as possible
• Add / remove multiple nodes in parallel
• Raft based schema and topology change
• And more …
23

24. Thank You
@asias_he
asias@scylladb.com
Asias He
24

25. Download Scylla Open Source:
scylladb.com/download
Talk to an expert:
scylladb.com/consultation
Take a test drive:
scylladb.com/test-drive
Experience Scylla for Yourself
25