Scylla Summit 2018: Scylla Feature Talks - Scylla Streaming and Repair Updates

•Download as PPTX, PDF•

1 like•1,544 views

Streaming is an internal operation that moves data from node to node over a network which. It is the foundation of various Scylla cluster operations, e.g., add node, decommission node and rebuild node. Repair is another important operation that detects the mismatch between multiple replicas on different nodes and synchronize the replicas. In this talk we will cover recent changes and performance improvements to streaming and repair. We will introduce the new Scylla streaming and the brand new row level repair that will be released in the upcoming scylla releases.

Software

What is streaming in Scylla
▪ Internal operation to move data
▪ Foundation of various Scylla cluster
operations:
▪ E.g., add node, decommission node,
rebuild node, repair node

Problems in current streaming
▪ Uses RPC Calls to stream data
one by one
▪ Write to memtable on the
receiver

Improving streaming
▪ Skip memtable and write to
sstables directly
• Less memory consumption
• Less CPU consumption
• Bigger and fewer SSTables
• Less compactions
▪ Use RPC Stream to send
streaming data
• Less RPC calls

Streaming test results
▪ Time to rebuild:
• 170 mins to 50 mins, 70%
▪ Streaming bandwidth:
• 36 MiB/s to 123 MiB/s, 240%
Test setup:
DB Nodes: 3
Instance Type: i3.xlarge / i3.8xlarge
Replication Factor (RF): 3
Consistency Level (CL): QUORUM
Compaction Strategy: Size-Tiered
Dataste: 1B keys per node

What is repair in Scylla
▪ Important maintenance
operation
▪ Detect mismatch between
replicas on different nodes
▪ Fix the mismatch for the replicas

Problems in partition level repair
▪ Partition level
• 100 partitions per checksum
• Single row mismatch causes 100 partitions
to be synced
• Even a single partition can be large
• Unnecessary on wire data transmission
▪ Two stage repair
• Checksum to find the mismatch
• Streaming to fix the mismatch
• Read data twice (checksum + streaming)
Partition 1
Partition 2
Partition 100
...
Checksum

Introduce the row level repair
▪ Row level
• One row per checksum
• Set reconciliation algorithms
• Only exchange mismatched rows
• No unnecessary data transmission
▪ Single stage repair
• Reuse data read for checksum
• Without streaming service
• Read data once from disk
Row 1
Row 2
Row N
...
Checksum
Checksum
Checksum

Repair test results
▪ Completely unsynced
• Time to repair: 87 mins to 70 mins, 20%
▪ Completely synced
• Time to repair: 43 mins to 20 mins, 45%
▪ 99.9% synced
• Time to repair: 211 mins to 44 mins, 80%
• Amount of data sent on wire:
• Before: tx= 162 GiB, rx = 90 GiB
• After: tx= 1.15 GiB, rx = 0.57 GiB
• Improvement: tx = 99%, rx = 99%
DB Nodes: 3
Instance Type: i3.xlarge
Replication Factor (RF): 3
Consistency Level (CL): QUORUM
Compaction Strategy: Size-Tiered
Dataset: 1B keys per node
Graphs

Thank You
Any Questions ?
Please stay in touch
asias@scylladb.com
@asias_he

When ingesting large amounts of data into a Scylla cluster, we would like the ingestion to proceed as quickly as possible, but not quicker. We explain how over-eager ingestion could result in a buildup of queues of background writes, possibly to the point of depleting available memory. We then explain how Scylla avoids this risk by automatically slowing down well-behaving applications to the best possible ingestion rate (“flow control”). For applications which cannot be slowed down, Scylla still achieves the highest possible throughput by quicky rejecting excess requests (“admission control”). In this talk we investigate the different causes of queue buildup during writes, including consistency-level lower than “ALL” and materialized views, and review the mechanisms which Scylla uses to automatically solve this problem.

Scylla Summit 2018: Keeping Your Latency SLAs No Matter What!

ScyllaDB

Scylla Summit 2018: Make Scylla Fast Again! Find out how using Tools, Talent,...

ScyllaDB

Scylla strives to deliver high throughput at low, consistent latencies under any scenario. But in the field things can and do get slower than one would like. Some of those issues come from bad data modelling and anti-patterns. Some others from lack of resources and bad system configuration, and in rare cases even product malfunction. But how to tell them apart? And once you do, how to understand how to fix your application or reconfigure your system? Scylla has a rich ecosystem of tools available to answer those questions and in this talk we’ll discuss the proper use of some of them and how to take advantage of each tool’s strength. We will discuss real examples using tools like CQL tracing, nodetool commands, the Scylla monitor and others.

Seastar Summit 2019 Keynote

ScyllaDB

Seastar is a framework for disk, network, compute, and multicore intensive applications such as databases and filesystems. It treats multicore CPUs and disk I/O as asynchronous entities like networking, replacing locks with message passing. This provides benefits like high throughput, low latency, and control over where throughput and latency occur. The keynote discussed Seastar's approach to scheduling, opportunities around coroutines, and its goals for modules, stream revamping, and task co-execution. Compatibility policies were outlined emphasizing community involvement in supported compilers, APIs, and architectures.

Scylla Summit 2018: Consensus in Eventually Consistent Databases

ScyllaDB

Eventually consistent databases choose to remain available under failure, allowing for conflicting data to be stored in different replicas (later repaired by background processes). Weakening the consistency guarantees improves not only availability, but also performance, as the number of replicas involved in a given operation can be minimized. There are, however, use-cases that require the opposite trade-off. Indeed, Apache Cassandra and Scylla provide Lightweight Transactions (LWT), which allow single-key linearizable updates. The mechanism underlying LWT is asynchronous consensus. In this talk, we'll describe the characteristics and requirements of Scylla's consensus implementation, and how it enables strongly consistent updates. We will also cover how consensus can be applied to other aspects of the system, such as schema changes, node membership, and range movements, in order to improve their reliability and safety. We will thus show that an eventually consistent database can leverage consensus without compromising either availability or performance.

Scylla Summit 2022: Operating at Monstrous Scales: Benchmarking Petabyte Work...

ScyllaDB

ScyllaDB is a distributed database designed to scale horizontally and vertically — in theory. What about in practice? ScyllaDB’s Benny Halevy, Director, Software Engineering, will take you through the process and results of benchmarking our NoSQL database at the petabyte level, showing how you can use advanced features like workload prioritization to control priorities of transactional (read-write) and analytic (read-only) queries on the same cluster with smooth and predictable performance. To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.

Scylla’s Journey Towards Being an Elastic Cloud Native Database

ScyllaDB

Scylla Summit 2018: Introducing ValuStor, A Memcached Alternative Made to Run...

ScyllaDB

In this presentation, we share approaches to replacing RAM-only caching infrastructure while achieving high performance against a persistent datastore. Memcached has proven very popular, but it also requires its users to sacrifice reliability, scalability, redundancy, availability, and security. To address these issues, Sensaphone implemented a memcached replacement called ValuStor, an easy-to-use key-value database client layer written in C++ that works well with Scylla. ValuStor includes features like client-side write queues, multi-threading support, automatic adaptive consistency, and support for multiple data types (including JSON).

Discover the new features and capabilities of Scylla Open Source 5.0 directly from the engineers who developed it. This second block of lightning talks will cover the following topics: - New IO Scheduler and Disk Parallelism - Per-Service-Level Timeouts - Better Workload Estimation for Backpressure and Out-of-Memory Conditions - Large Partition Handling Improvements - Optimizing Reverse Queries To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.

Seastar / ScyllaDB, or how we implemented a 10-times faster Cassandra

Tzach Livyatan

Seastar Summit 2019 vectorized.io

ScyllaDB

Vectorized is presenting on their RPC system and Redpanda product. They are focused on operational simplicity, safety, and 10x the performance of Kafka. Their RPC goals include using PODs instead of IDLs, isolation, avoiding translation costs, and embracing futures. Their measurements show their RPC system is up to 7x faster than flatbuffers for nested data structures and has lower latency than alternative systems.

ScyllaDB @ Apache BigData, may 2016

Tzach Livyatan

Scylla is a new open source NoSQL database that is compatible with Apache Cassandra but provides significantly higher performance through a redesign that takes advantage of modern hardware. Scylla is capable of over 1.8 million operations per second per node with predictable low latencies. It uses an architecture with shard-per-core and reactor programming that avoids locks and threads for near-linear scaling. Scylla also has its own efficient unified cache and I/O scheduler that maximize throughput and allow it to outperform Cassandra on benchmarks by an order of magnitude. Scylla is fully compatible with Cassandra and aims to build an open source community around ongoing core database improvements.

How Opera Syncs Tens of Millions of Browsers and Sleeps Well at Night

ScyllaDB

Opera chose Scylla over Cassandra to sync the data of millions of browsers to a back-end data repository. The results of the migration and further optimizations they made in their stack helped Opera to gain better latency/throughput and lower resources usage beyond their expectations. Attend this session to learn how to Migrate your data in a sane way, without any downtime Connect a Python+Django web app to Scylla, how to use intranode sharding to improve your application

Scylla Summit 2016: Compose on Containing the Database

ScyllaDB

This document discusses how Compose applies containerization best practices to provide database services. It outlines the "Twelve Factors of Stateful Apps" that guide Compose's architecture. These include running databases and data in separate containers, using environment variables for configuration, scaling containers vertically before adding nodes, and collecting logs and metrics within the deployment. By applying these factors, Compose can reliably deploy a range of database technologies like MongoDB, PostgreSQL, and now ScyllaDB across its platform.

Using ScyllaDB with JanusGraph for Cyber Security

ScyllaDB

Scylla Summit 2022: Building Zeotap's Privacy Compliant Customer Data Platfor...

ScyllaDB

Customer Data Platforms, commonly called CDPs, form an integral part of the marketing stack powering Zeotap's Adtech and Martech use-cases. The company offers a privacy-compliant CDP platform, and ScyllaDB is an integral part. Zeotap's CDP demands a mix of OLTP, OLAP, and real-time data ingestion, requiring a highly-performant store. In this presentation, Shubham Patil, Lead Software Engineer, and Safal Pandita, Senior Software Engineer at Zeotap will share how ScyllaDB is powering their solution and why it's a great fit. They begin by describing their business use case and the challenges they were facing before moving to ScyllaDB. Then they cover their technical use-cases and requirements for real-time and batch data ingestions. They delve into our data access patterns and describe their data model supporting all use cases simultaneously for ingress/egress. They explain how they are using Scylla Migrator for our migration needs, then describe their multiregional, multi-tenant production setup for onboarding more than 130+ partners. Finally, they finish by sharing some of their learnings, performance benchmarks, and future plans. To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.

Scylla Summit 2016: Scylla at Samsung SDS

ScyllaDB

Scylla Summit 2016: Why Kenshoo is about to displace Cassandra with Scylla

ScyllaDB

Scylla Summit 2018: How We Made Large Partition Scans Over Two Times Faster

ScyllaDB

Fetching large amount of data in a single query is a longstanding pain for applications. Queries that return a significant amount of data have to be paged, in other words, split into multiple subqueries that return data little by little. In both Scylla and Apache Cassandra, paging is stateless: each subquery is independent of each other and can even be sent to different replicas. Because of that, all the work done in the previous subqueries will not be reused causing a reduction from the maximum expected throughput. In this talk we are going to examine the problems with the previous stateless paging implementation and introduce the new stateful paging implementations that brings vast improvements in the throughput of large partition scans.

How we got to 1 millisecond latency in 99% under repair, compaction, and flus...

ScyllaDB

Scylla is an open source reimplementation of Cassandra which performs up to 10X with drop in-replacement compatibility. At ScyllaDB, performance matters but even more importantly, stable performance under any circumstances. A key factor for our consistent performance is our reliance on userspace schedulers. Scheduling in userspace allows the application, the database in our case to have better control on the different priorities each task has and to provide an SLA to selected operations. Scylla used to have an I/O scheduler and recently won a CPU scheduler. At ScyllaDB, we make architectural decisions that provide not only low latencies but consistently low latencies at higher percentiles. This begins with our choice of language and key architectural decisions such as not using the Linux page-cache, and is fulfilled by autonomous database control, a set of algorithms, which guarantees that the system will adapt to changes in the workload. In the last year, we have made changes to Scylla that provide latencies that are consistent in every percentile. In this talk, Dor Laor will recap those changes and discuss what ScyllaDB is doing in the future.

How We Made Scylla Maintenance Easier, Safer and Faster

ScyllaDB

Many Scylla maintenance operations require significant data movement between database nodes in a cluster. It is not an easy task to make the management operations efficient while maintaining minimum impact on the workload all the time. In this talk, we will share how we made those maintenance operations easier, safer and faster with the new Scylla features and improvements, e.g., seedless, repair based node operations, smarter off-strategy compaction, io bandwidth limiter for repair and compaction, parallel repair in Scylla Manger and more.

Comparing Apache Cassandra 4.0, 3.0, and ScyllaDB

ScyllaDB

How does Cassandra 4.0’s performance compare to Cassandra 3.x’s? What’s been fixed in 4.0 and what remains unchanged? Should you upgrade or consider other options? Join us for a webinar where we’ll answer these questions and more, based on our extensive benchmarks comparing Cassandra 4.0 against Cassandra 3.11. We’ll also share how the new release of Cassandra stacks up against Scylla Open Source. You’ll learn the rationale and results for our head-to-head comparisons, including: - Throughput under various loads - Comparison of long-tail (p95, p99) latencies - Improvements to operations such as compactions If you are considering upgrading your existing infrastructure from Cassandra 3.11, or if you are considering a new wide column database for a greenfield deployment, this is a session you won’t want to miss!

Back to the future with C++ and Seastar

Tzach Livyatan

Seastar is an open source framework that provides highly scalable and asynchronous distributed applications. It uses a shared-nothing architecture with no locks or threads to achieve linear scaling across cores. Applications built on Seastar can handle millions of connections and I/O operations in parallel. It uses an asynchronous programming model based on promises and futures with zero-copy networking and disk I/O for high performance.

10 Devops-Friendly Database Must-Haves - Dor Laor, ScyllaDB - DevOpsDays Tel ...

DevOpsDays Tel Aviv

This document discusses 10 must-have features for a devops-friendly database: 1) being open source, 2) supporting multi-tenancy, 3) allowing isolated test and development environments, 4) integrating with big data technologies, 5) being resilient during failures, 6) being extensible through export/import and scripting, 7) automatically handling tuning and scaling, 8) providing consistent tail latency, 9) easily backing up and restoring data, and 10) always being available across multiple data centers. It promotes ScyllaDB as a database that has these features by providing Cassandra compatibility at Redis speeds.

Scylla: 1 Million CQL operations per second per server

Avi Kivity

FireEye & Scylla: Intel Threat Analysis Using a Graph Database

ScyllaDB

FireEye believes in intelligence driven cyber security. Their legacy system used PostgreSQL with a custom graph database system to store and facilitate analysis of threat intelligence data. As their user base increased they ran into scaling issues requiring a system redesign with a new platform. This presentation will focus on the bac kend systems and migration path to a new technology stack using JanusGraph running on top of Scylla plus Elasticsearch. Using Scylladb turned out to be a game-changer in terms of performance and the types of analysis our application is able to do effortlessly.

Deploying Kafka at Dropbox, Mark Smith, Sean Fellows

confluent

At Dropbox we are currently handling approximately 10,000,000 messages per second at peak across our handful of Kafka clusters. The largest of which has hit throughputs of 7,000,000 per second (~30 Gbps) on only 20 nodes. We’ll walk you through the steps we took to get where we are, the design that works for us — and those that didn’t. We’ll talk about the tooling we had to build and what we want to see exist. We’ll dive deeper into configuration and provide a blueprint you can follow. We’ll talk about the trials and tribulations of using Kafka — including ways we’ve set our clusters on fire, ways we’ve lost data, ways we’ve turned our hairs gray, and ways we’ve heroically saved the day for our users. Finally, we’ll spend time on some of the work we’re doing to handle consumer coordination across our many different systems and to integrate Kafka into a well established corporate infrastructure. (I.e., making Kafka “”play nice”” with everybody.)

DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Hakka Labs

The document discusses reasons for reimplementing messaging queue functionality instead of using an existing solution like RabbitMQ or Kafka. It notes that reimplementing provides full understanding and control, allowing for quick fixes and easy addition of features without needing a daemon. However, it also acknowledges that reimplementing takes a long time to reach stability. The document then describes the specific requirements for the reimplemented messaging queue and provides a brief overview of its design and performance.

Full Stack Load Testing

Terral R Jordan

A talk I gave at the Boston Web Performance Meetup in August 2014. Performance is one of the most challenging issues in modern web app design, in large part because modeling, testing, and validating performance before deploying to production is so challenging. While many ops teams have nailed down the problem of re-creating pre-production environments that closely mimic production, those environments frequently rely on known-good components beyond the application code itself: AWS ELB, F5 load balancers, CDNs, Varnish, and more. Testing plug-in components like that can be challenging, because their performance characteristics don't directly align with application metrics. - How many simultaneous users can my load balancer support? - What sort of network load will I put on my CDN (i.e., how much will it cost?) - How do different user behavior patterns affect performance? In this meetup, we'll introduce a novel tool in this toolbox: tcpreplay, an open-source tool for replaying packet capture files back at an application. By replaying user traffic to a staging environment, you can test the effects of - Network saturation to the load balancer - High numbers of users / IPs - Lots of traffic to your other monitoring tools!

Tuning kafka pipelines

Sumant Tambe

Kafka is a high-throughput, fault-tolerant, scalable platform for building high-volume near-real-time data pipelines. This presentation is about tuning Kafka pipelines for high-performance. Select configuration parameters and deployment topologies essential to achieve higher throughput and low latency across the pipeline are discussed. Lessons learned in troubleshooting and optimizing a truly global data pipeline that replicates 100GB data under 25 minutes is discussed.

What's hot

Scylla Summit 2022: Scylla 5.0 New Features, Part 1

ScyllaDB

Seastar / ScyllaDB, or how we implemented a 10-times faster Cassandra

Tzach Livyatan

Seastar Summit 2019 vectorized.io

ScyllaDB

ScyllaDB @ Apache BigData, may 2016

Tzach Livyatan

How Opera Syncs Tens of Millions of Browsers and Sleeps Well at Night

ScyllaDB

Scylla Summit 2016: Compose on Containing the Database

ScyllaDB

Using ScyllaDB with JanusGraph for Cyber Security

ScyllaDB

Scylla Summit 2022: Building Zeotap's Privacy Compliant Customer Data Platfor...

ScyllaDB

Scylla Summit 2016: Scylla at Samsung SDS

ScyllaDB

Scylla Summit 2016: Why Kenshoo is about to displace Cassandra with Scylla

ScyllaDB

Scylla Summit 2018: How We Made Large Partition Scans Over Two Times Faster

ScyllaDB

How we got to 1 millisecond latency in 99% under repair, compaction, and flus...

ScyllaDB

How We Made Scylla Maintenance Easier, Safer and Faster

ScyllaDB

Comparing Apache Cassandra 4.0, 3.0, and ScyllaDB

ScyllaDB

Back to the future with C++ and Seastar

Tzach Livyatan

10 Devops-Friendly Database Must-Haves - Dor Laor, ScyllaDB - DevOpsDays Tel ...

DevOpsDays Tel Aviv

Scylla: 1 Million CQL operations per second per server

Avi Kivity

FireEye & Scylla: Intel Threat Analysis Using a Graph Database

ScyllaDB

Deploying Kafka at Dropbox, Mark Smith, Sean Fellows

confluent

DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Hakka Labs

What's hot (20)

Scylla Summit 2022: Scylla 5.0 New Features, Part 1

Seastar / ScyllaDB, or how we implemented a 10-times faster Cassandra

Seastar Summit 2019 vectorized.io

ScyllaDB @ Apache BigData, may 2016

How Opera Syncs Tens of Millions of Browsers and Sleeps Well at Night

Scylla Summit 2016: Compose on Containing the Database

Using ScyllaDB with JanusGraph for Cyber Security

Scylla Summit 2022: Building Zeotap's Privacy Compliant Customer Data Platfor...

Scylla Summit 2016: Scylla at Samsung SDS

Scylla Summit 2016: Why Kenshoo is about to displace Cassandra with Scylla

Scylla Summit 2018: How We Made Large Partition Scans Over Two Times Faster

How we got to 1 millisecond latency in 99% under repair, compaction, and flus...

How We Made Scylla Maintenance Easier, Safer and Faster

Comparing Apache Cassandra 4.0, 3.0, and ScyllaDB

Back to the future with C++ and Seastar

10 Devops-Friendly Database Must-Haves - Dor Laor, ScyllaDB - DevOpsDays Tel ...

Scylla: 1 Million CQL operations per second per server

FireEye & Scylla: Intel Threat Analysis Using a Graph Database

Deploying Kafka at Dropbox, Mark Smith, Sean Fellows

DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ

Similar to Scylla Summit 2018: Scylla Feature Talks - Scylla Streaming and Repair Updates

Full Stack Load Testing

Terral R Jordan

Tuning kafka pipelines

Sumant Tambe

HBaseCon2017 gohbase: Pure Go HBase Client

HBaseCon

Aerospike & GCE (LSPE Talk)

Sayyaparaju Sunil

This document discusses database performance characteristics and benchmarks Aerospike on Google Compute Engine (GCE). It finds that with 50 nodes, Aerospike achieved a median latency of 7ms and 83% of requests under 16ms latency for 1 million writes per second. CPU utilization was only 50-60% due to overhead. Network bottlenecks were identified, and optimizations like DPDK helped achieve 4.2 million reads per second with 90% under 4ms latency. Live migrations can impact highly consistent databases and their applications. Local SSDs provide good performance as an alternative to RAM and were benchmarked positively with Aerospike.

Scylla Summit 2016: Outbrain Case Study - Lowering Latency While Doing 20X IO...

ScyllaDB

Tuning the Kernel for Varnish Cache

Per Buer

Project Slides for Website 2020-22.pptx

AkshitAgiwal1

This document describes a resource efficient ternary content addressable memory (TCAM) architecture. It uses a pipelined layer architecture to increase operating frequency. The TCAM was implemented on an FPGA with a size of 64x32 and achieved a 10.52% higher speed, 7.62% lower power, and 50% lower resource utilization compared to existing designs. When implemented in ASIC using a 45nm process, the proposed TCAM achieved 121.10% higher speed, 70.12% lower power, and an 18.7x smaller area compared to existing designs.

Release it! - Takeaways

Manuela Grindei

1. A major airline experienced an outage when engineers performed a planned database failover that unexpectedly caused all check-in kiosks and IVR servers to go down after 3 hours, costing the company hundreds of thousands in delays and expenses. 2. An online retailer crashed on launch day when their new system was overwhelmed by high traffic, with sessions consuming resources due to session replication on every request. They added throttling and blocking to stabilize the system. 3. Zero downtime deployments can be achieved by expanding capacity, doing rolling upgrades of each server one by one, and finally cleaning up old components once complete.

Broken Performance Tools

C4Media

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1N4GN6z. Brendan Gregg focuses on broken tools and metrics instead of the working ones. Metrics can be misleading, and counters can be counter-intuitive. Gregg includes advice and methodologies for verifying new performance tools, understanding how they work, and using them successfully. Filmed at qconsf.com. Brendan Gregg is a senior performance architect at Netflix, where he does large scale computer performance design, analysis, and tuning. He is the author of multiple technical books including Systems Performance published by Prentice Hall, and received the USENIX LISA Award for Outstanding Achievement in System Administration.

Couchbase live 2016

Pierre Mavro

Pierre Mavro from Criteo discussed Couchbase usage at their company. Criteo has over 100 Couchbase clusters storing over 90TB of data serving up to 25 million queries per second. They benchmarked Couchbase and found network bandwidth and replicas increased latency. To improve, Criteo monitored latency, split workloads across clusters, automated operations, and tuned Couchbase and systems. Their changes helped Couchbase scale for Criteo's large workload.

Surge 2013: Maximizing Scalability, Resiliency, and Engineering Velocity in t...

Coburn Watson

Nick Fisk - low latency Ceph

ShapeBlue

This document discusses optimizing Ceph latency through hardware design. It finds that CPU frequency has a significant impact on latency, with higher frequencies resulting in lower latencies. Testing shows 4KB write latency of 2.4ms at 900MHz but 694us at higher frequencies. The document also discusses how CPU power states that wake slowly, like C6 at 85us, can negatively impact latency. Overall it advocates designing hardware with fast CPUs and avoiding slower cores or dual sockets to minimize latency in Ceph deployments.

Red Hat Storage Day Seattle: Stabilizing Petabyte Ceph Cluster in OpenStack C...

Red_Hat_Storage

Cisco uses Ceph for storage in its OpenStack cloud platform. The initial Ceph cluster design used HDDs which caused stability issues as the cluster grew to petabytes in size. Improvements included throttling client IO, upgrading Ceph versions, moving MON metadata to SSDs, and retrofitting journals to NVMe SSDs. These steps stabilized performance and reduced recovery times. Lessons included having clear stability goals and automating testing to prevent technical debt from shortcuts.

Scaling with sync_replication using Galera and EC2

Marco Tusa

Managing Open vSwitch Across a Large Heterogenous Fleet

andyhky

Open vSwitch (OVS) is one of the more popular ways to provide VM connectivity in OpenStack. Rackspace has been using Open vSwitch in production since late 2011. In this session, we will detail the challenges faced with managing and upgrading Open vSwitch across a large heterogenous fleet. Finally, we will share some of the tools we have created to monitor OVS availability and performance. Specific topics covered will include: Why upgrade OVS? Measuring OVS Minimizing downtime with upgrades Bridge fail modes Kernel module gotchas Monitoring OVS

Part9-congestion.pptx

Olivier Bonaventure

Neutron scaling

Vinay Bannai

Much Faster Networking

C4Media

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1VfmmLC. David Riddoch talks about the technologies that make very high performance networking possible on commodity servers and networks, with a special focus on kernel bypass technologies including sockets acceleration and NFV. These techniques give user-space applications direct access the network adapter hardware, making possible sub-microsecond latencies and millions of messages per second per thread. Filmed at qconlondon.com. David Riddoch leads the development of the Solarflare Open IP stack.

Apache HBase Low Latency

Nick Dimiduk

Scaling Kubernetes to Support 50000 Services.pptx

thaond2

Scaling Kubernetes to Support 50,000 Services The challenges of scaling Kubernetes include high API server load from managing a large number of services, pods and endpoints. Solutions tested include batch processing requests to reduce QPS, and restructuring iptables rules into a search tree for faster routing. IPVS was also tested and showed significantly better performance than iptables for large scale deployments, with constant latency for adding rules and higher network bandwidth. With these techniques, Kubernetes has been scaled to support over 50,000 services.

Similar to Scylla Summit 2018: Scylla Feature Talks - Scylla Streaming and Repair Updates (20)

Full Stack Load Testing

Tuning kafka pipelines

HBaseCon2017 gohbase: Pure Go HBase Client

Aerospike & GCE (LSPE Talk)

Scylla Summit 2016: Outbrain Case Study - Lowering Latency While Doing 20X IO...

Tuning the Kernel for Varnish Cache

Project Slides for Website 2020-22.pptx

Release it! - Takeaways

Broken Performance Tools

Couchbase live 2016

Surge 2013: Maximizing Scalability, Resiliency, and Engineering Velocity in t...

Nick Fisk - low latency Ceph

Red Hat Storage Day Seattle: Stabilizing Petabyte Ceph Cluster in OpenStack C...

Scaling with sync_replication using Galera and EC2

Managing Open vSwitch Across a Large Heterogenous Fleet

Part9-congestion.pptx

Neutron scaling

Much Faster Networking

Apache HBase Low Latency

Scaling Kubernetes to Support 50000 Services.pptx

More from ScyllaDB

Optimizing NoSQL Performance Through Observability

ScyllaDB

ScyllaDB has the potential to deliver impressive performance and scalability. The better you understand how it works, the more you can squeeze out of it. But before you squeeze, make sure you know what to monitor! Watch our experienced Postgres developer work through monitoring and performance strategies that help him understand what mistakes he’s made moving to NoSQL. And learn with him as our database performance expert offers friendly guidance on how to use monitoring and performance tuning to get his sample Rust application on the right track. This webinar focuses on using monitoring and performance tuning to discover and correct mistakes that commonly occur when developers move from SQL to NoSQL. For example: - Common issues getting up and running with the monitoring stack - Using the CQL optimizations dashboard - Common issues causing high latency in a node - Common issues causing replica imbalance - What a healthy system looks like in terms of memory - Key metrics to keep an eye on This isn’t “Death-by-Powerpoint.” We’ll walk through problems encountered while migrating a real application from Postgres to ScyllaDB – and try to fix them live as well.

Event-Driven Architecture Masterclass: Challenges in Stream Processing

ScyllaDB

Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...

ScyllaDB

Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...

ScyllaDB

Developer Data Modeling Mistakes: From Postgres to NoSQL

ScyllaDB

See where an RDBMS-pro’s intuition leads him astray – and learn practical tips for the data modeling transition ScyllaDB has the potential to deliver impressive performance and scalability. The better you understand how it works, the more you can squeeze out of it. However, developers new to high-performance NoSQL intuitively shoot themselves in the foot with respect to things like table design, query design, indexing, and partitioning. Watch where our experienced Postgres developer intuitively falls into traps that hurt performance and scalability. And learn with him as our database performance expert offers friendly guidance on navigating all the unexpected behaviors that tend to trip up RDBMS experts. This webinar focuses on common data modeling and querying mistakes that occur when developers move from SQL to NoSQL. For example: - Understanding query first design principles - Planning for schema evolution - Steering clear of common pitfalls and anti-patterns - Assessing data access patterns This isn’t “Death-by-Powerpoint.” We’ll walk through problems encountered while migrating a real application from Postgres to ScyllaDB – and try to fix them live as well.

What Developers Need to Unlearn for High Performance NoSQL

ScyllaDB

See where an RDBMS-pro’s intuition leads him astray – and learn practical tips for the transition ScyllaDB has the potential to deliver impressive performance and scalability. The better you understand how it works, the more you can squeeze out of it. However, developers new to high-performance NoSQL intuitively shoot themselves in the foot with respect to things like table design, query design, indexing, and partitioning. Watch where our experienced Postgres developer intuitively falls into traps that hurt performance and scalability. And learn with him as our database performance expert offers friendly guidance on navigating all the unexpected behaviors that tend to trip up RDBMS experts. Our first webinar of this series will cover common mistakes with practices such as: - Translating the data model to NoSQL - Optimizing table design - Optimizing query performance - Planning for partitioning This isn’t “Death-by-Powerpoint.” We’ll walk through problems encountered while migrating a real application from Postgres to ScyllaDB – and try to fix them live as well.

Low Latency at Extreme Scale: Proven Practices & Pitfalls

ScyllaDB

Expert tips on how to maximize your database performance at scale Untangle the complexity of achieving database performance at scale. Join this webinar to discover commonly overlooked ways to get predictable low latency, even at extreme scale. Our Solution Architects will walk you through the strategies and pitfalls learned by working on thousands of real-world distributed database projects, many reaching 1M OPS with single-digit MS latencies. In addition to offering clear recommendations, we’ll also explain the process behind how we arrived at them – so you can benefit from the lessons learned by other teams. We’ll cover how to: - Design and deploy a large-scale distributed database cluster - Optimize your clients’ interactions with it - Expand the cluster horizontally and globally - Ensure it survives whatever disasters the world throws at it

Dissecting Real-World Database Performance Dilemmas

ScyllaDB

Tackling your own database performance challenges is serious business. For a change of pace, let’s have some fun learning from other teams’ performance predicaments. Join us for an interactive session where we dissect four specific database performance challenges faced by teams considering or using ScyllaDB. For each dilemma, we'll: - Examine the context and technical requirements - Talk about potential solutions and cover the pros and cons of each - Disclose what approach the team took, and how it worked out About the speaker: Felipe is an IT specialist with years of experience on distributed systems and open-source technologies. He is one of the co-authors of "Database Performance at Scale", an Open Access, freely available publication for individuals interested on improving database performance. At ScyllaDB, he works as a Solution Architect.

Beyond Linear Scaling: A New Path for Performance with ScyllaDB

ScyllaDB

Linear scaling (sometimes near linear scaling) is often mentioned in several benchmarks, articles and product comparisons as proof that a given technology and algorithmic optimizations perform better than another. But is that really what performance is all about, and should you even care? This webinar discusses performance beyond linear scalability, including what typically matters more when running high throughput and low latency workloads at scale. We'll cover how ScyllaDB offers unparalleled performance and share our insights on: - The hidden aspects of linear scaling - When linear scaling matters most and when it’s simply irrelevant - Often overlooked considerations for optimizing and measuring distributed systems performance Watch now to learn from our experience (and lessons learned) in building the fastest NoSQL database in the world.

Dissecting Real-World Database Performance Dilemmas

ScyllaDB

Navigating Complex Database Performance Hurdles Tackling your own database performance challenges is serious business. For a change of pace, let’s have some fun learning from other teams’ performance predicaments. Join us for an interactive session where we dissect 4 specific database performance challenges faced by teams considering or using ScyllaDB. For each dilemma: - The presenters will describe the context and technical requirements - Together, we’ll talk about potential solutions and cover the pros and cons of each - Finally, we’ll disclose what approach the team took, and how it worked out Throughout the event, we’ll have opportunities to win ScyllaDB swag and prizes! Come prepared to engage in lively discussions and gain valuable insight into database performance strategies.

Database Performance at Scale Masterclass: Workload Characteristics by Felipe...

ScyllaDB

Database Performance at Scale Masterclass: Database Internals by Pavel Emelya...

ScyllaDB

Database Performance at Scale Masterclass: Driver Strategies by Piotr Sarna

ScyllaDB

Replacing Your Cache with ScyllaDB

ScyllaDB

This document discusses replacing external caching solutions with using the internal caching capabilities of ScyllaDB. It provides examples of companies that improved performance, reduced costs and complexity by moving from Redis or Elasticsearch with an external cache to using ScyllaDB's embedded cache instead. The document also outlines some of the advantages of ScyllaDB's cache like improved latency, coherency with the database and observability compared to external caching layers.

Powering Real-Time Apps with ScyllaDB_ Low Latency & Linear Scalability

ScyllaDB

Discover how your team can achieve low latency at the extreme scale that your data-intensive applications require. We’ll walk you through an example of how ScyllaDB scales linearly to achieve 1M and then 2M OPS – with <1ms P99 latency. We’ll cover how this works on a sample realtime app (an ML feature store), share best practices for performance, and talk about the most important tradeoffs you’ll need to negotiate. Join us to learn: - Why and how to ensure your database takes full advantage of your cloud infrastructure - What architectural considerations matter most for high throughput and low latency - Key factors to consider when selecting a high-performance database

7 Reasons Not to Put an External Cache in Front of Your Database.pptx

ScyllaDB

This document discusses the pros and cons of placing an external cache in front of a database. It introduces Tomasz Grabiec and Tzach Livyatan from ScyllaDB and describes ScyllaDB's optimized internal caching design. External caches can increase latency and costs while ignoring the database's context and workload knowledge. ScyllaDB embeds its cache to minimize overhead and ensure data and query awareness. The document shares customer examples that improved performance and reduced costs by moving from cached databases to ScyllaDB.

Getting the most out of ScyllaDB

ScyllaDB

Expert tips on how to maximize your database potential If you’re considering or getting started with ScyllaDB, you’re probably intrigued by its potential to achieve high throughput and predictable low latency at a reasonable cost. So how do you ensure that you’re maximizing that potential for your team’s specific workloads and use case? This webinar offers practical advice for navigating the various decision points you’ll face as you assess whether ScyllaDB is a good fit for your team and later roll it out into production. We’ll cover the most critical considerations, tradeoffs, and recommendations related to: - Infrastructure selection - ScyllaDB configuration - Client-side setup - Data modeling

NoSQL Database Migration Masterclass - Session 2: The Anatomy of a Migration

ScyllaDB

NoSQL Database Migration Masterclass - Session 3: Migration Logistics

ScyllaDB

NoSQL Data Migration Masterclass - Session 1 Migration Strategies and Challenges

ScyllaDB

More from ScyllaDB (20)

Optimizing NoSQL Performance Through Observability

Event-Driven Architecture Masterclass: Challenges in Stream Processing

Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...

Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...

Developer Data Modeling Mistakes: From Postgres to NoSQL

What Developers Need to Unlearn for High Performance NoSQL

Low Latency at Extreme Scale: Proven Practices & Pitfalls

Dissecting Real-World Database Performance Dilemmas

Beyond Linear Scaling: A New Path for Performance with ScyllaDB

Dissecting Real-World Database Performance Dilemmas

Database Performance at Scale Masterclass: Workload Characteristics by Felipe...

Database Performance at Scale Masterclass: Database Internals by Pavel Emelya...

Database Performance at Scale Masterclass: Driver Strategies by Piotr Sarna

Replacing Your Cache with ScyllaDB

Powering Real-Time Apps with ScyllaDB_ Low Latency & Linear Scalability

7 Reasons Not to Put an External Cache in Front of Your Database.pptx

Getting the most out of ScyllaDB

NoSQL Database Migration Masterclass - Session 2: The Anatomy of a Migration

NoSQL Database Migration Masterclass - Session 3: Migration Logistics

NoSQL Data Migration Masterclass - Session 1 Migration Strategies and Challenges

Recently uploaded

GreenCode-A-VSCode-Plugin--Dario-Jurisic

Green Software Development

GraphSummit Paris - The art of the possible with Graph Technology

Neo4j

Vitthal Shirke Java Microservices Resume.pdf

Vitthal Shirke

原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样

mz5nrf0n

原版一模一样【微信：741003700 】【美国纽约州立大学奥尔巴尼分校毕业证学位证书】【微信：741003700 】学位证，留信认证（真实可查，永久存档）offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。本公司拥有海外各大学样板无数，能完美还原海外各大学 Bachelor Diploma degree, Master Degree Diploma 1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700 留信网认证的作用: 1:该专业认证可证明留学生真实身份 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才

How to write a program in any programming language

Rakesh Kumar R

Orion Context Broker introduction 20240604

Fermin Galan

A Study of Variable-Role-based Feature Enrichment in Neural Models of Code

Aftab Hussain

Understanding variable roles in code has been found to be helpful by students in learning programming -- could variable roles help deep neural models in performing coding tasks? We do an exploratory study. - These are slides of the talk given at InteNSE'23: The 1st International Workshop on Interpretability and Robustness in Neural Software Engineering, co-located with the 45th International Conference on Software Engineering, ICSE 2023, Melbourne Australia

Enterprise Resource Planning System in Telangana

NYGGS Automation Suite

Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics. To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/

UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions

Peter Muessig

The UI5 tooling is the development and build tooling of UI5. It is built in a modular and extensible way so that it can be easily extended by your needs. This session will showcase various tooling extensions which can boost your development experience by far so that you can really work offline, transpile your code in your project to use even newer versions of EcmaScript (than 2022 which is supported right now by the UI5 tooling), consume any npm package of your choice in your project, using different kind of proxies, and even stitching UI5 projects during development together to mimic your target environment.

E-commerce Development Services- Hornet Dynamics

Hornet Dynamics

Essentials of Automations: The Art of Triggers and Actions in FME

Safe Software

In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation. We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios. Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!

Energy consumption of Database Management - Florina Jonuzi

Green Software Development

Graspan: A Big Data System for Big Code Analysis

Aftab Hussain

We built a disk-based parallel graph system, Graspan, that uses a novel edge-pair centric computation model to compute dynamic transitive closures on very large program graphs. We implement context-sensitive pointer/alias and dataflow analyses on Graspan. An evaluation of these analyses on large codebases such as Linux shows that their Graspan implementations scale to millions of lines of code and are much simpler than their original implementations. These analyses were used to augment the existing checkers; these augmented checkers found 132 new NULL pointer bugs and 1308 unnecessary NULL tests in Linux 4.4.0-rc5, PostgreSQL 8.3.9, and Apache httpd 2.2.18. - Accepted in ASPLOS ‘17, Xi’an, China. - Featured in the tutorial, Systemized Program Analyses: A Big Data Perspective on Static Analysis Scalability, ASPLOS ‘17. - Invited for presentation at SoCal PLS ‘16. - Invited for poster presentation at PLDI SRC ‘16.

openEuler Case Study - The Journey to Supply Chain Security

Shane Coughlan

Top Features to Include in Your Winzo Clone App for Business Growth (4).pptx

rickgrimesss22

Automated software refactoring with OpenRewrite and Generative AI.pptx.pdf

timtebeek1

May Marketo Masterclass, London MUG May 22 2024.pdf

Adele Miller

E-commerce Application Development Company.pdf

Hornet Dynamics

Fundamentals of Programming and Language Processors

Rakesh Kumar R

ALGIT - Assembly Line for Green IT - Numbers, Data, Facts

Green Software Development

Recently uploaded (20)

GreenCode-A-VSCode-Plugin--Dario-Jurisic

GraphSummit Paris - The art of the possible with Graph Technology

Vitthal Shirke Java Microservices Resume.pdf

原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样

How to write a program in any programming language

Orion Context Broker introduction 20240604

A Study of Variable-Role-based Feature Enrichment in Neural Models of Code

Enterprise Resource Planning System in Telangana

UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions

E-commerce Development Services- Hornet Dynamics

Essentials of Automations: The Art of Triggers and Actions in FME

Energy consumption of Database Management - Florina Jonuzi

Graspan: A Big Data System for Big Code Analysis

openEuler Case Study - The Journey to Supply Chain Security

Top Features to Include in Your Winzo Clone App for Business Growth (4).pptx

Automated software refactoring with OpenRewrite and Generative AI.pptx.pdf

May Marketo Masterclass, London MUG May 22 2024.pdf

E-commerce Application Development Company.pdf

Fundamentals of Programming and Language Processors

ALGIT - Assembly Line for Green IT - Numbers, Data, Facts

Scylla Summit 2018: Scylla Feature Talks - Scylla Streaming and Repair Updates

1. Scylla Streaming and Repair Updates Asias He Software Developer

2. Presenter bio Asias He is a software developer with over 10 years of experience in programming. In the past, he worked on Debian Project, Solaris Kernel, KVM Virtualization for Linux, OSv unikernel. He now works on Seastar and ScyllaDB.

3. Streaming

4. What is streaming in Scylla ▪ Internal operation to move data ▪ Foundation of various Scylla cluster operations: ▪ E.g., add node, decommission node, rebuild node, repair node

5. Problems in current streaming ▪ Uses RPC Calls to stream data one by one ▪ Write to memtable on the receiver

6. Improving streaming ▪ Skip memtable and write to sstables directly • Less memory consumption • Less CPU consumption • Bigger and fewer SSTables • Less compactions ▪ Use RPC Stream to send streaming data • Less RPC calls

7. Streaming test results ▪ Time to rebuild: • 170 mins to 50 mins, 70% ▪ Streaming bandwidth: • 36 MiB/s to 123 MiB/s, 240% Test setup: DB Nodes: 3 Instance Type: i3.xlarge / i3.8xlarge Replication Factor (RF): 3 Consistency Level (CL): QUORUM Compaction Strategy: Size-Tiered Dataste: 1B keys per node

8. Repair

9. What is repair in Scylla ▪ Important maintenance operation ▪ Detect mismatch between replicas on different nodes ▪ Fix the mismatch for the replicas

10. Problems in partition level repair ▪ Partition level • 100 partitions per checksum • Single row mismatch causes 100 partitions to be synced • Even a single partition can be large • Unnecessary on wire data transmission ▪ Two stage repair • Checksum to find the mismatch • Streaming to fix the mismatch • Read data twice (checksum + streaming) Partition 1 Partition 2 Partition 100 ... Checksum

11. Introduce the row level repair ▪ Row level • One row per checksum • Set reconciliation algorithms • Only exchange mismatched rows • No unnecessary data transmission ▪ Single stage repair • Reuse data read for checksum • Without streaming service • Read data once from disk Row 1 Row 2 Row N ... Checksum Checksum Checksum

12. Repair test results ▪ Completely unsynced • Time to repair: 87 mins to 70 mins, 20% ▪ Completely synced • Time to repair: 43 mins to 20 mins, 45% ▪ 99.9% synced • Time to repair: 211 mins to 44 mins, 80% • Amount of data sent on wire: • Before: tx= 162 GiB, rx = 90 GiB • After: tx= 1.15 GiB, rx = 0.57 GiB • Improvement: tx = 99%, rx = 99% DB Nodes: 3 Instance Type: i3.xlarge Replication Factor (RF): 3 Consistency Level (CL): QUORUM Compaction Strategy: Size-Tiered Dataset: 1B keys per node Graphs

13. Thank You Any Questions ? Please stay in touch asias@scylladb.com @asias_he

Scylla Summit 2018: Scylla Feature Talks - Scylla Streaming and Repair Updates

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Scylla Summit 2018: Scylla Feature Talks - Scylla Streaming and Repair Updates

Similar to Scylla Summit 2018: Scylla Feature Talks - Scylla Streaming and Repair Updates (20)

More from ScyllaDB

More from ScyllaDB (20)

Recently uploaded

Recently uploaded (20)

Scylla Summit 2018: Scylla Feature Talks - Scylla Streaming and Repair Updates