Optimizing ScyllaDB Performance via Observability

1. Optimizing ScyllaDB Performance via Observability Amnon Heiman, Principal Software Engineer

2. Amnon Heiman Over 20 years of experience in software development for large scale systems. Holds a BA and MSc in Computer Science from the Technion-Machon Technologi Le’ Israel and an MBA from Tel Aviv University.

3. ■ ScyllaDB Monitoring 101 ■ Monitoring while developing ■ Performance degradation ■ Alerts Agenda

4. ScyllaDB Monitoring 101

5. ScyllaDB Monitoring Stack Components ■ By default - Container Based ■ Prometheus ■ ScyllaDB ■ Node-exporter ■ Grafana-Loki ■ Alert-Manager ■ Grafana ■ Can read ScyllaDB tables

6. ScyllaDB Monitoring Stack Components ScyllaDB ScyllaDB ScyllaDB

7. Monitoring While Developing

8. Development With NoSQL Databases ■ Think about the queries ﬁrst ■ Create a data model ■ Indexing ■ Consistency/Availability ■ Replication ■ Test with traﬃc ■ Validate after installation

9. Read/Write With Replication 9 R1 R2 R3 Client Coordinator Node 5 Nodes Cluster Replication Factor 3

10. Always Use Token-Aware Drivers 10 R1 R2 R3 Client Coordinator Node 5 Nodes Cluster Replication Factor 3

11. CQL Optimization

12. Non-Prepared Statements

13. Always Prepare Your Statements ■ Performance ■ The driver can ﬁnd a replica node ■ The server only parses the query once ■ Security ■ CQL Injection user_lookup_stmt = session.prepare("SELECT * FROM users WHERE user_id=?") users = [] for user_id in user_ids_to_query: user = session.execute(user_lookup_stmt, [user_id]) users.append(user) for user_id in user_ids_to_query: user = session.execute("select * from users WHERE user_id=" + user_id)

14. Reverse Order Reads ■ You can sort SELECT results by the clustering key ■ For best results, sort by the deﬁned order

15. Consistency Level Consistency - Under what conditions a read after a write will return that write ■ Prefer CL LOCAL QUORUM ■ Never use CL ANY ■ Avoid using CL ALL ■ Use CL ONE with care

16. Common Pitfalls ■ Be careful with allow ﬁltering ■ Be careful with reverse order reads ■ Be careful from full range scan ■ Use Paging for queries that return a lot of data

17. Performance Degradation ScyllaDB Was Fine and Now It’s Not

18. Overview Dashboard

19. Overview Dashboard ■ Hardware failures ■ Timeout and latencies spikes - system overload ■ Imbalance - hardware failure or hot partition ■ Advisor table - check for known issues

20. Alerts

21. ScyllaDB Monitoring Stack Components ScyllaDB ScyllaDB ScyllaDB

22. Alerts and the Alertmanager ■ Alert source ■ Prometheus - Metrics based alerts ■ Loki - Logs based alerts ■ Alertmanager Distribute Alerts to other systems ■ Email ■ Pagerduty ■ Slack ■ webhooks ■ Preset rules out of the box

23. Thank You Stay in Touch Amnon Heiman amnon@scylladb.com amnonheiman amnonh amnon-heiman-939421

Optimizing ScyllaDB Performance via Observability

You are reading a preview.

Check these out next

Optimizing ScyllaDB Performance via Observability

More Related Content

Similar to Optimizing ScyllaDB Performance via Observability (20)

More from ScyllaDB (20)

Recently uploaded (20)