Optimizing Write-Intensive Database Workloads Masterclass: Strategies for Optimizing Write Performance

Editor's Notes

• #1 Felipe's presentation covered write-intensive database workloads and, more importantly, on why writes become difficult as we scale. The central theme of this presentation is that write performance isn't controlled by a single knob. It's the result of several interacting decisions: how data is distributed, how we access it, how we batch writes, how storage is organized, and how we allocate compute and I/O resources. So rather than looking for one magic optimization, we're going to walk through the design and operational choices that have the biggest impact.
• #2 We're going to cover: the fundamental reasons writes become difficult in different DBs data distribution and schema design then contention and batching and finally storage and resource considerations The goal is to leave you with a set of tradeoffs you can apply when looking at a write-heavy workload.
• #3 Before we get into the technical material, there's also an interactive component to the masterclass. You can put questions into the chat during the session, and the five best questions will win a copy of Database Performance at Scale. So if something comes up as we're going through the examples, don't wait until the end, put the question in the chat.
• #5 The first topic I want to review is: that write performance is multifactorial. When we talk about database write performance, it's tempting to reduce the problem to something like CPU, disk latency, network throughput. But htere's much more a write has to go through. The db has to determine where the data is allocated, coordinate the operation, then the db persist it ( manage in-memory structures) before acknowledging the write. At some point, it also has to organize the resulting data on disk, and compete with the rest of the workload for resources. So when a write-heavy system isn't performing well, we need to ask: which part of that chain is actually limiting us? That distinction becomes particularly important at scale, because improving one component can simply expose another bottleneck. The rest of this presentation is essentially a tour through those different factors and how to optimize them.
• #6 This reinforces the same point before we move into the first concrete problem: why writes struggle at scale. The important takeaway here is that scaling the database doesn't automatically eliminate write bottlenecks. The architecture determines where those bottlenecks appear.
• #7 There are different reasons writes struggle depending on the database architecture. On the relational world, there are several very familiar constraints: a single-writer model, ACID locking, WAL and flushes, and ultimately a compute ceiling. The important thing here isn't that relational databases can't handle writes, they absolutely can. The point is that the architecture introduces particular constraints around how much write concurrency and coordination it can handle. Distributed and NoSQL DBs approach the problem differently, but they don't make the problem disappear. Instead, we're dealing with distributed-systems considerations, different design approaches, potentially single-master writers, distributed coordination, and the tradeoffs associated with the CAP theorem (Consistency, Availability, Partition tolerance). So when we move to a distributed architecture, we're changing the set of constraints we're working with. Which leads to the ScyllaDB architecture: if we're going to distribute writes, we need to understand exactly how data gets distributed through the system.
• #8 Now let's look at how data distribution works in ScyllaDB. At the highest level, we start with the token range (all the possible outputs of the Murmur3 hash). Those token ranges are assigned to nodes. Nodes are then associated with racks, which correspond to zones in cloud deployments. That's the high-level architecture. Then, within each node, we have shards. And spread across those shards shards, we have tablets. The reason this matters for write performance is that distribution isn't simply about spreading data across machines. The workload also has to be distributed appropriately within the machines. So we're progressively dividing the work: token ranges determine where data goes in the cluster, nodes provide the machine-level distribution, and shards provide distribution across CPU cores within a node. That's why ScyllaDB's shard-per-core architecture is designed for performance: the way we design our keys and access patterns ultimately determines where the work lands within the cluster.
• #9 Distribution solves one problem, but it creates another design responsibility: making sure the workload actually distributes. The first problem is hot partitions. If access patterns are skewed, some partitions receive disproportionately more traffic than others. The image represents a Zipfian distribution as example: a small number of items receive a very large share of the activity. At the shard level, this can become a situation of high concurrency. One shard can become much busier than the others, leading to increased latency and reducing overall performance. The second major issue is large partitions. If data continues accumulating without a bound, a partition can become increasingly expensive to work with. And then we have poor key choices. The workload itself can create concentration. For example, time-based accumulation, power users, bots. So the key question when designing a write-heavy workload is not just ‘does my partition key uniquely identify the data?’ You should rather answer the question: does this key distribute the actual workload?
• #10 This brings us to the first major group of practical techniques: design decisions that control how the workload is distributed. We're going to start with one of the most direct ways of addressing skew and large partitions: synthetic partitioning, or bucketing.
• #11 One way to tackle problematic distribution is synthetic partitioning, also called bucketing. The idea is to add an artificial, predictable component to the partition key. And there are some ways you can achieve that: One is time: we can add a time component such as a date. Another is a hash-based bucket—for example, deriving a bucket from a click ID. And another is adding an attribute combination to the partitioning strategy. The important thing is that we're introducing another dimension into the partition key so that data doesn't all accumulate into one logical partition. The example here makes that concrete. The naïve schema uses channel_id as the partition key, with message_id as the clustering key. If a channel receives a large and continuous volume of messages, everything for that channel accumulates in one partition. <advance slide> With bucketing, we introduce bucket into the partition key: (channel_id, bucket). Now messages for the same channel can be distributed across multiple buckets rather than accumulating into a single partition. The tradeoff is that we've made the data model more complex. The application now needs to understand the bucket as part of the access pattern. To recap: bucketing isn't free. It's a tradeoff of simpler partition key for better, predictable distribution.
• #12 Distribution is one side of the problem. Another is contention. One recommendation here is not to overuse LWTs. The reason is that LWTs introduce additional operations and round trips. For write-heavy workloads, another approach is to design for idempotency instead. Append-only patterns are particularly useful because they're easier to reason about when failures occur. It's also important to use client-side timestamps. And there are some patterns we want to avoid in a write-intensive workload: read-modify-write and row locking. The general principle is to avoid making every write depend on additional coordination or on reading existing state first. If we can design a write so that it can be safely repeated, that simplifies failure handling and reduces the need for coordination.
• #13 Another major lever for write-heavy workloads is batching. But batching is one of those optimizations where the details matter enormously. So let's look at when batching helps and where the limits are.
• #14 Batching is particularly relevant when we're writing intensive sets of similar data points. For example: IoT metrics per device, bulk user operations, and event streaming. The benefit is that we can reduce the per-statement overhead. That includes round trips, transaction-related overhead, WAL flushes, and the overhead associated with handling individual retries. But there's an important caveat: batching doesn't mean that failure handling disappears. If a single condition fails, we may still have to retry. There's also an expectation of out all-or-nothing aspect to operations. They either complete or fail. That is not always true, depending on the type of operations. So the tradeoff is straightforward: batching can reduce overhead and improve efficiency, but it changes the unit at which operations are handled and can introduce different failure and atomicity considerations.
• #15 Now we get into the practical rules for high-performance batching. First, we have the distinction between logged and unlogged batches. Second, keep batches small. The deck gives two limits: a warning at 128 kilobytes and a failure threshold at 1024 kilobytes. These are tunable, but the recommendation is to be careful about changing them. And the most important best practice is: only batch to the same partition. That's the key performance-oriented recommendation. The example shown is a batch containing several operations against different users. The important thing to notice is that these operations don't all target the same partition. For performance-oriented batching, we want the batch to stay within the same partition. Batches are automatically converted to unlogged batches, while retaining atomic behavior and being fast. So the lesson isn't simply ‘use batches.’ It's ‘use the right kind of batch, keep it small, and avoid using batching as a way to combine unrelated partitions.
• #16 Here I want to highlight the most important batching recommendation. If we're optimizing for performance, keep the batch within the same partition. That keeps the operation localized instead of turning batching into a distributed operation across multiple partitions, which then involves multiple shards or even nodes. So when you code an application leveraging batches, one of the first questions I'd ask is: are these writes actually going to the same partition? If they're not, batching may not give us the behavior we're looking for.
• #17 So far we've looked mostly at application and data-model decisions. Now let's move down the stack into the storage layer and the resources underneath the workload. We'll start with compaction.
• #18 Compaction strategy is another major consideration for write-heavy workloads. LCS has high write amplification and can compact super aggressively, so it isn't the focus for these write-heavy scenarios. As for STCS… at this point you should just use ICS. ICS is the best choice for general-purpose ingestion, with a lot of tunables. And TWCS is particularly suited to time-based workloads. For TWCS, the assumptions matter: TTL is required, the workload is continuously ingesting data, and the workload is append-only, without overwrites or deletes. So there isn't one universally correct compaction strategy here. The choice depends on the workload characteristics: especially whether we're dealing with general-purpose ingestion or a time-based, TTL-driven, append-only workload.
• #19 Compression introduces another performance tradeoff. The deck highlights recent developments around LZ4 and ZStdWithDictsCompressor, with potentially very high compression ratios—up to 90%, depending on the workload. ZStd is tunable in terms of the effort required, with levels from 1 to 9. For a write-heavy workload, the recommendation here is to find a good balance rather than simply maximizing compression effort. The suggested starting point is ZStd with dictionaries at level 3. Then measure the resulting compression ratio. There's also a second dimension: network compression. That can be particularly valuable in cloud environments because reducing network traffic can also reduce cloud costs. So compression isn't simply a CPU-versus-disk tradeoff. The deck is pointing out that storage efficiency and network efficiency can both matter.
• #20 Now let's talk about resource utilization. One point that can look surprising if you're coming from other database systems is the statement that 100% CPU utilization in ScyllaDB is a feature. The point is that the IO scheduler is doing its job, while CPU is being used for several things at once: commitlog and memtable flushes, compactions, and the user workload. So seeing CPU close to 100% doesn't automatically mean the system is unhealthy. But we also have to think about workload prioritization. If online reads and writes need to remain very fast while batch workloads are also running, we can segregate resources and prioritize them differently. Online reads and writes being super fast, while batch reads and writes are best effort and lower priority. And finally, the underlying hardware matters: fast NICs and very fast NVMe storage are part of the picture. So again, the key idea is that write performance is a resource-management problem as well as a data-model problem.
• #21 Disk utilization is another area where workload characteristics determine the appropriate strategy. For overwrite-heavy workloads, use ICS together with the space amplification goal setting. For delete-heavy workloads, the recommendation is space amplification goals combined with a lower tombstone compaction interval. For time-series workloads, TWCS is recommended, with approximately 12 to 15 windows. The number and size of windows is calculated by TTL divided by 15 for the approximate range of each window. Then there are additional considerations: Use compression, ICS, and tablets. Finally, for ScyllaDB X Cloud, up to 90% disk utilization is supported, together with fast autoscaling for uninterrupted ingestion. The broader point is that disk utilization can't be considered independently from the workload. The right compaction strategy and storage configuration depend on whether we're overwriting, deleting, or continuously ingesting time-series data.
• #22 Let's wrap it all together. There are a few principles behind all of these recommendations. Write-heavy workloads consume resources. The first thing we need to get right is distribution. Then we want to minimize unnecessary coordination, use batching carefully, tune storage behavior to the workload, and make sure resources are appropriately isolated.
• #23 Let me leave you with five practical takeaways. First: write-heavy workloads are resource hungry. There's no single resource we can assume is irrelevant. Second: design for distribution. Your partitioning strategy needs to distribute not just data, but the workload. Third: don't overuse coordinator-heavy operations. Where possible, design writes to minimize unnecessary coordination and favor patterns such as idempotency. Fourth: batch carefully. Batching can reduce overhead, but the details matter—keep batches small and, for high-performance batching, keep them to the same partition. And fifth: tune compaction and compression to the workload. Time-series ingestion, overwrites, deletes, and general-purpose ingestion have different characteristics, and the storage strategy needs to reflect that. Finally: resource isolation matters. If online workloads and batch workloads compete for the same resources, workload prioritization can help preserve the performance characteristics that matter to the application. So if you're troubleshooting or designing a write-heavy system, don't start by looking for a single database setting. Start with the workload: how is the data distributed, how are writes being issued, what coordination is required, how are they being batched, what does the storage workload look like, and which resources are competing with each other? Those decisions together determine the write performance you get.