Tackling your own database performance challenges is serious business. For a change of pace, let’s have some fun learning from other teams’ performance predicaments. Join us for an interactive session where we dissect four specific database performance challenges faced by teams considering or using ScyllaDB. For each dilemma, we'll: - Examine the context and technical requirements - Talk about potential solutions and cover the pros and cons of each - Disclose what approach the team took, and how it worked out About the speaker: Felipe is an IT specialist with years of experience on distributed systems and open-source technologies. He is one of the co-authors of "Database Performance at Scale", an Open Access, freely available publication for individuals interested on improving database performance. At ScyllaDB, he works as a Solution Architect.

1. Dissecting Real-World
Database Performance
Dilemmas
Felipe Mendes, Solution Architect at ScyllaDB

2. Why Disney
Moved from
DynamoDB
to ScyllaDB
Yichen Wei, Senior
Software Engineer
Adam Drennan, Senior
Software Engineer
So You've
Lost Quorum:
Lessons From
Accidental
Downtime
Bo Ingram, Staff Software
Engineer, Persistence
Infrastructure
Inside Expedia's
Migration to
ScyllaDB for
Change Data
Capture
Jean Carlo Rivera Ura,
NoSQL Database Engineer III
MongoDB
vs ScyllaDB:
Tractian’s
Experience with
Real-Time ML
JP Voltani, Director of
Engineering

3. Poll
How much time did you spend
troubleshooting performance issues
in your database?

4. + For data-intensive applications that require high
throughput and predictable low latencies
+ Close-to-the-metal design takes full advantage of
modern infrastructure
+ >5x higher throughput
+ >20x lower latency
+ >75% TCO savings
+ Compatible with Apache Cassandra and Amazon
DynamoDB
+ DBaaS/Cloud, Enterprise and Open Source
solutions
The Database for Gamechangers
4
“ScyllaDB stands apart...It’s the rare product
that exceeds my expectations.”
– Martin Heller, InfoWorld contributing editor and reviewer
“For 99.9% of applications, ScyllaDB delivers all the
power a customer will ever need, on workloads that other
databases can’t touch – and at a fraction of the cost of
an in-memory solution.”
– Adrian Bridgewater, Forbes senior contributor

5. 5
+400 Gamechangers Leverage ScyllaDB
Seamless experiences
across content + devices
Digital experiences at
massive scale
Corporate fleet
management
Real-time analytics 2,000,000 SKU -commerce
management
Video recommendation
management
Threat intelligence service
using JanusGraph
Real time fraud detection
across 6M transactions/day
Uber scale, mission critical
chat & messaging app
Network security threat
detection
Power ~50M X1 DVRs with
billions of reqs/day
Precision healthcare via
Edison AI
Inventory hub for retail
operations
Property listings and
updates
Unified ML feature store
across the business
Cryptocurrency exchange
app
Geography-based
recommendations
Global operations- Avon,
Body Shop + more
Predictable performance for
on sale surges
GPS-based exercise
tracking
Serving dynamic live
streams at scale
Powering India's top
social media platform
Personalized
advertising to players
Distribution of game
assets in Unreal Engine

6. Hi! Nice to e-Meet Ya!
Felipe Mendes, Solution Architect at ScyllaDB
+ Published Author on Linux and Databases
+ Helps teams solve their most challenging problems
+ Years of experience with Linux and distributed systems

7. Agenda + Hunting down a Latency Problem
+ Ticking Time-Series Bomb
+ Hot Facts, Cold Insights
+ Holistic Performance View
7

8. Hunting Down a Latency
Problem
Uncovering a Multi-Region Performance Challenge
8

9. Customer evaluated ScyllaDB and was happy with the results
+ Initial testing:
+ 3 node cluster (AWS us-east1)
+ P99 < 15ms
+ Using gocql driver:
+ Following query best-practices
+ Making use of gocql.DataCentreHostFilter
+ All application queries using a LOCAL_* ConsistencyLevel
9
A Latency Sensitive Workload for AdTech

10. Final production requirements were multi-region (AWS us-west-1)
+ Followed the Adding a New DC to an ScyllaDB Cluster procedure
+ Latency went through the roof!
10
The PROBLEM

11. 11
Why?!

12. + We realized that latencies only affected a specific ScyllaDB scheduling class
12
A Few Data Points
+ A path for resolution: main query class P99 driven by the Network RTT

13. + nodetool setlogginglevel query_processing trace:
13
Whoops! Tracking down individual queries
+ Seems like we are getting close. Ideas?
for i in *; do egrep -Hi "SELECT|UPDATE|INSERT|DELETE" $i | awk -F':' '{ print $1, $NF }'; done | sort -V | uniq -c
215 (cached) "SELECT * FROM system_auth.roles WHERE role = ?" (cassandra)
1 (cached) "SELECT * FROM system_distributed.service_levels;" ()
304 (cached) "SELECT * FROM system_auth.roles WHERE role = ?" (cassandra)
3 (cached) "SELECT * FROM system_distributed.service_levels;" ()
323 (cached) "SELECT * FROM system_auth.roles WHERE role = ?" (cassandra)
2 (cached) "SELECT * FROM system_distributed.service_levels;" ()
1 (cached) "SELECT id, data, written_at, version FROM system.batchlog LIMIT 128" ()
7 (cached) "SELECT * FROM system_distributed.service_levels;" ()
1 (cached) "SELECT id, data, written_at, version FROM system.batchlog LIMIT 128" ()
6 (cached) "SELECT * FROM system_distributed.service_levels;" ()
1 (cached) "SELECT id, data, written_at, version FROM system.batchlog LIMIT 128" ()
6 (cached) "SELECT * FROM system_distributed.service_levels;" ()

14. + From ScyllaDB:
14
The answer lies within the code (and in our docs!)
+ From Enable Authorization:
db::consistency_level
password_authenticator::consistency_for_user(std::string_view role_name) {
if (role_name == DEFAULT_USER_NAME) {
return db::consistency_level::QUORUM;
}
return db::consistency_level::LOCAL_ONE;
}

15. Ticking Time-series bomb
Rushing against time
15

16. Log retention use case in production
+ ScyllaDB Cloud cluster in GCP
+ 6 node cluster
+ Latency within bounds, except during repairs
16
Major Streaming Company

17. Cluster repairs were taking long to complete
+ Some nodes started to run out of disk space
+ Customer was under a time-sensitive "freeze" period
17
The PROBLEM
4% free space!

18. + Compaction was unable to keep up with the rate of incoming files:
18
An interesting symptom

19. 19
What would you do?

20. + What we know thus far:
+ Nodes are running out of disk space
+ Compaction is unable to catch up
+ Repair is taking long
20
Data Modeling Review

21. + Making use Jaeger as an integration:
+ Business-critical application metrics
+ Time to Live (TTL) for automatic data expiration
+ Spreads data using a "bucket" technique, sorted by time
+ Circling back to the basics:
21
Data Modeling Review
$ cqlsh -e "DESC SCHEMA" | egrep -i "compaction =" | sort -h | uniq -c
43 AND compaction = {'class': 'IncrementalCompactionStrategy'}
33 AND compaction = {'class': 'SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
55 AND compaction = {'class': 'TimeWindowCompactionStrategy', 'compaction_window_size': '1',
'compaction_window_unit': 'HOURS'}
+ 131 tables to run through … 😢
+ … but TWCS tables seem interesting 💪

22. + TTL 259200 (in seconds) == 30 days
+ Split in 1 hour buckets
+ 30 * 24 buckets = 720 windows!
22
Diving Deeper
CREATE TABLE x.y (
<...>
) WITH crc_check_chance = 1.0
AND compaction = {'class': 'TimeWindowCompactionStrategy', 'compaction_window_size': '1',
'compaction_window_unit': 'HOURS'}
AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND default_time_to_live = 2592000
AND gc_grace_seconds = 10800

23. 23
Understanding the problem
TWCS
Picture from: https://www.pythian.com/blog/proposal-for-a-new-cassandra-cluster-key-compaction-strategy

24. 24
After schema changes
+ Repair completed, memory utilization reduced, performance improved!

25. 25
+ Customer explained they were using Jaeger integration "stock" settings
+ We reported and addressed jaegertracing/jaeger/4561 upstream:
+ "With ScyllaDB (and likely Cassandra too), increasing TTL leads to large numbers of
sstables"
+ Felipe introduced the "twcs_max_window_count" tunable in ScyllaDB:
+ Safemode - Introduce TimeWindowCompactionStrategy Guardrails
+ Available starting at ScyllaDB Open Source 5.2 & Enterprise 2023.1
+ Takeaways
+ Be suspicious of activities taking a long time to complete
+ Review how third-party integrations works under the hood
+ Rest assured with ScyllaDB Cloud expertise to back you up!
Post event findings (and diligence)

26. Hot Facts, Cold Insights
Data Thermodynamics & ScyllaDB
26

27. Engaged with us after losing data in HBase
+ On-prem bare-metal deployment
+ HUGE storage footprint – Petabyte range
+ Tiered storage or similar requirement:
+ "hot" – frequently accessed data or;
+ "cold" – least accessed data
+ No retention periods, data is forever stored
27
Worldwide Feed Aggregator

28. Which deployment and replication strategies to follow?
+ Single cluster, dual hot/cold DC?
+ Separate clusters, observable replication?
+ Should we use CDC for replication?
+ To dual-write or not?
+ How to evict data from the "hot" cluster?
28
Challenges

29. 29
Let's talk
strategies

30. Reasons were plenty:
+ An on-premise deployment makes it fair difficult to effectively leverage it
+ Decommissioning the previous HBase infrastructure wasn't an option
+ Performance:
+ "Local" Object Storage latencies were suboptimal
+ Cloud ones would result in:
+ Network RTT penalty
+ Internet traffic ($$)
+ Need to come up with data tiering/replication strategies anyway…
+ … and still figure out how to enable their applications to work with an Object
Store
30
Why not just use Object Storage?

31. Plan-ahead:
+ ScyllaDB specialized cache fully allocates server's memory (LSA)
+ Important on-disk components (SSTable metadata) needs to be stored
+ 1:30 for performance, 1:100 as an upper bound limit
31
Memory and Storage Limits

32. Ultimately, they decided to apply the following strategy:
+ Application primarily writes to "hot" DC;
+ Replication from "hot" to "cold" is asynchronously handled by ScyllaDB
+ After their cut-off period, simply remove all replicas from the "hot" DC:
32
ScyllaDB Replication
ALTER KEYSPACE replicated_keyspace WITH replication = {'class':
'NetworkTopologyStrategy', 'hot': 0, 'cold': 3};
+ Similar strategy already employed in HBase, zero friction point

33. Performance requires an holistic view:
+ An overlooked setting may be the culprit
to performance problems
+ Integrations won't always follow best practices
+ Be sure to take the most of your underlying
(and available) infrastructure
+ Context is fundamental to guide your decisions
33
Summary

34. Q&A
ScyllaDB Cloud
Start free trial
scylladb.com/cloud
Feb 14-15 | VIRTUAL EVENT
scylladb.com/summit
Virtual Workshop
February 22, 2024
scylladb.com/events

35. Thank you
for joining us today.
@scylladb scylladb/
slack.scylladb.com
@scylladb company/scylladb/
scylladb/