.NET developers have a lot of options when it comes to databases these days. Apache Cassandra is a scalable, fault-tolerant database that has already found its way into more than 25% of the Fortune 100 and continues to grow in popularity. But what makes it different from the myriad of other options available? In this talk, we’ll take a deep dive into Cassandra and learn about:
- Cassandra’s internals and how it works
- CQL (the SQL-like query language for Cassandra)
- Data Modeling like a pro
- Tools available for developers
- Writing .NET code that talks to Cassandra
If there’s time and interest, we’ll finish up with how some companies are already using Cassandra to power services you probably interact with in your daily life. You’ll leave with all the tools you need to start build highly available .NET applications and services on top of Cassandra.
Scylla Summit 2022: How to Migrate a Counter Table for 68 Billion RecordsScyllaDB
In this talk, we will discuss Happn's war story about migrating a Cassandra 2.1 cluster containing more than 68 Billion records in a counter table to ScyllaDB Open Source.
To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.
Deletes Without Tombstones or TTLs (Eric Stevens, ProtectWise) | Cassandra Su...DataStax
Deleting data from Cassandra has several challenges, and existing solutions (tombstones or TTLs) have limitations that make them unusable or untenable in certain circumstances. We'll explore the cases where existing deletion options fail or are inadequate, then describe a solution we developed which deletes data from Cassandra during standard or user-defined compaction, but without resorting to tombstones or TTL's.
About the Speaker
Eric Stevens Principal Architect, ProtectWise, Inc.
Eric is the principal architect, and day one employee of ProtectWise, Inc., specializing in massive real time processing and scalability problems. The team at ProtectWise processes, analyzes, optimizes, indexes, and stores billions of network packets each second. They look for threats in real time, but also store full fidelity network data (including PCAP), and when new security intelligence is received, automatically replay existing network history through that new intelligence.
Storing time series data with Apache CassandraPatrick McFadin
If you are looking to collect and store time series data, it's probably not going to be small. Don't get caught without a plan! Apache Cassandra has proven itself as a solid choice now you can learn how to do it. We'll look at possible data models and the the choices you have to be successful. Then, let's open the hood and learn about how data is stored in Apache Cassandra. You don't need to be an expert in distributed systems to make this work and I'll show you how. I'll give you real-world examples and work through the steps. Give me an hour and I will upgrade your time series game.
Myths of Big Partitions (Robert Stupp, DataStax) | Cassandra Summit 2016DataStax
Large partitions shall no longer be a nightmare. That is the goal of CASSANDRA-11206.
100MB and 100,000 cells per partition is the recommended limit for a single partition in Cassandra up to 3.5. Exceeding these limits can cause a lot of trouble. Repairs and compactions could fail and reads cause out-of-memory failures.
This talk provides a deep-dive of the reasons for the previous limitations, why exceeding these limitations caused trouble, how the improvements in Cassandra 3.6 helps with big partitions and why you should not blindly let your partitions get huge.
About the Speaker
Robert Stupp Solution Architect, DataStax
Robert is working as a Solutions Architect at DataStax and is also a Committer to Apache Cassandra. Before joining DataStax he worked with his customers to architect and build distributed systems using Cassandra and has a long experience in building distributed backend systems mostly using Java as the preferred language of choice.
Bucket your partitions wisely - Cassandra summit 2016Markus Höfer
When we talk about bucketing we essentially talk about possibilities to split cassandra partitions in several smaller parts, rather than having only one large partition.
Bucketing of cassandra partitions can be crucial for optimizing queries, preventing large partitions or to fight TombstoneOverwhelmingException which can occur when creating too many tombstones.
In this talk I want to show how to recognize large partitions during datamodeling. I will also show different strategies we used in our projects to create, use and maintain buckets for our partitions.
Scylla Summit 2022: How to Migrate a Counter Table for 68 Billion RecordsScyllaDB
In this talk, we will discuss Happn's war story about migrating a Cassandra 2.1 cluster containing more than 68 Billion records in a counter table to ScyllaDB Open Source.
To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.
Deletes Without Tombstones or TTLs (Eric Stevens, ProtectWise) | Cassandra Su...DataStax
Deleting data from Cassandra has several challenges, and existing solutions (tombstones or TTLs) have limitations that make them unusable or untenable in certain circumstances. We'll explore the cases where existing deletion options fail or are inadequate, then describe a solution we developed which deletes data from Cassandra during standard or user-defined compaction, but without resorting to tombstones or TTL's.
About the Speaker
Eric Stevens Principal Architect, ProtectWise, Inc.
Eric is the principal architect, and day one employee of ProtectWise, Inc., specializing in massive real time processing and scalability problems. The team at ProtectWise processes, analyzes, optimizes, indexes, and stores billions of network packets each second. They look for threats in real time, but also store full fidelity network data (including PCAP), and when new security intelligence is received, automatically replay existing network history through that new intelligence.
Storing time series data with Apache CassandraPatrick McFadin
If you are looking to collect and store time series data, it's probably not going to be small. Don't get caught without a plan! Apache Cassandra has proven itself as a solid choice now you can learn how to do it. We'll look at possible data models and the the choices you have to be successful. Then, let's open the hood and learn about how data is stored in Apache Cassandra. You don't need to be an expert in distributed systems to make this work and I'll show you how. I'll give you real-world examples and work through the steps. Give me an hour and I will upgrade your time series game.
Myths of Big Partitions (Robert Stupp, DataStax) | Cassandra Summit 2016DataStax
Large partitions shall no longer be a nightmare. That is the goal of CASSANDRA-11206.
100MB and 100,000 cells per partition is the recommended limit for a single partition in Cassandra up to 3.5. Exceeding these limits can cause a lot of trouble. Repairs and compactions could fail and reads cause out-of-memory failures.
This talk provides a deep-dive of the reasons for the previous limitations, why exceeding these limitations caused trouble, how the improvements in Cassandra 3.6 helps with big partitions and why you should not blindly let your partitions get huge.
About the Speaker
Robert Stupp Solution Architect, DataStax
Robert is working as a Solutions Architect at DataStax and is also a Committer to Apache Cassandra. Before joining DataStax he worked with his customers to architect and build distributed systems using Cassandra and has a long experience in building distributed backend systems mostly using Java as the preferred language of choice.
Bucket your partitions wisely - Cassandra summit 2016Markus Höfer
When we talk about bucketing we essentially talk about possibilities to split cassandra partitions in several smaller parts, rather than having only one large partition.
Bucketing of cassandra partitions can be crucial for optimizing queries, preventing large partitions or to fight TombstoneOverwhelmingException which can occur when creating too many tombstones.
In this talk I want to show how to recognize large partitions during datamodeling. I will also show different strategies we used in our projects to create, use and maintain buckets for our partitions.
All about Zookeeper and ClickHouse Keeper.pdfAltinity Ltd
ClickHouse clusters depend on ZooKeeper to handle replication and distributed DDL commands. In this Altinity webinar, we’ll explain why ZooKeeper is necessary, how it works, and introduce the new built-in replacement named ClickHouse Keeper. You’ll learn practical tips to care for ZooKeeper in sickness and health. You’ll also learn how/when to use ClickHouse Keeper. We will share our recommendations for keeping that happy as well.
Producer Performance Tuning for Apache KafkaJiangjie Qin
Kafka is well known for high throughput ingestion. However, to get the best latency characteristics without compromising on throughput and durability, we need to tune Kafka. In this talk, we share our experiences to achieve the optimal combination of latency, throughput and durability for different scenarios.
Everything You Always Wanted to Know About Kafka’s Rebalance Protocol but Wer...confluent
Apache Kafka is a scalable streaming platform with built-in dynamic client scaling. The elastic scale-in/scale-out feature leverages Kafka’s “rebalance protocol” that was designed in the 0.9 release and improved ever since then. The original design aims for on-prem deployments of stateless clients. However, it does not always align with modern deployment tools like Kubernetes and stateful stream processing clients, like Kafka Streams. Those shortcoming lead to two mayor recent improvement proposals, namely static group membership and incremental rebalancing (which will hopefully be available in version 2.3). This talk provides a deep dive into the details of the rebalance protocol, starting from its original design in version 0.9 up to the latest improvements and future work. We discuss internal technical details, pros and cons of the existing approaches, and explain how you configure your client correctly for your use case. Additionally, we discuss configuration tradeoffs for stateless, stateful, on-prem, and containerized deployments.
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of FacebookThe Hive
This presentation describes the reasons why Facebook decided to build yet another key-value store, the vision and architecture of RocksDB and how it differs from other open source key-value stores. Dhruba describes some of the salient features in RocksDB that are needed for supporting embedded-storage deployments. He explains typical workloads that could be the primary use-cases for RocksDB. He also lays out the roadmap to make RocksDB the key-value store of choice for highly-multi-core processors and RAM-speed storage devices.
The Missing Manual for Leveled Compaction Strategy (Wei Deng & Ryan Svihla, D...DataStax
In this presentation, we will look into JIRAs, JavaDocs and system log entries to gain a deeper understanding on how LCS works under the hood. We will explain what scenarios don't work well for LCS and (more importantly) why. We will leverage legacy TRACE/DEBUG level log for compaction related objects as well as some newer compaction logging information introduced in C* 3.6 (CASSANDRA-10805) to gain better insights.
About the Speakers
Wei Deng Solutions Architect, DataStax
Solutions Architect for DataStax. I have a strong interest in big data, cloud application and distributed computing practices.
C* Summit 2013: How Not to Use Cassandra by Axel LiljencrantzDataStax Academy
At Spotify, we see failure as an opportunity to learn. During the two years we've used Cassandra in our production environment, we have learned a lot. This session touches on some of the exciting design anti-patterns, performance killers and other opportunities to lose a finger that are at your disposal with Cassandra.
Apache Kafka becoming the message bus to transfer huge volumes of data from various sources into Hadoop.
It's also enabling many real-time system frameworks and use cases.
Managing and building clients around Apache Kafka can be challenging. In this talk, we will go through the best practices in deploying Apache Kafka
in production. How to Secure a Kafka Cluster, How to pick topic-partitions and upgrading to newer versions. Migrating to new Kafka Producer and Consumer API.
Also talk about the best practices involved in running a producer/consumer.
In Kafka 0.9 release, we’ve added SSL wire encryption, SASL/Kerberos for user authentication, and pluggable authorization. Now Kafka allows authentication of users, access control on who can read and write to a Kafka topic. Apache Ranger also uses pluggable authorization mechanism to centralize security for Kafka and other Hadoop ecosystem projects.
We will showcase open sourced Kafka REST API and an Admin UI that will help users in creating topics, re-assign partitions, Issuing
Kafka ACLs and monitoring Consumer offsets.
Modeling Data and Queries for Wide Column NoSQLScyllaDB
Discover how to model data for wide column databases such as ScyllaDB and Apache Cassandra. Contrast the differerence from traditional RDBMS data modeling, going from a normalized “schema first” design to a denormalized “query first” design. Plus how to use advanced features like secondary indexes and materialized views to use the same base table to get the answers you need.
Presentation at Strata Data Conference 2018, New York
The controller is the brain of Apache Kafka. A big part of what the controller does is to maintain the consistency of the replicas and determine which replica can be used to serve the clients, especially during individual broker failure.
Jun Rao outlines the main data flow in the controller—in particular, when a broker fails, how the controller automatically promotes another replica as the leader to serve the clients, and when a broker is started, how the controller resumes the replication pipeline in the restarted broker.
Jun then describes recent improvements to the controller that allow it to handle certain edge cases correctly and increase its performance, which allows for more partitions in a Kafka cluster.
Running Apache Kafka in production is only the first step in the Kafka operations journey. Professional Kafka users are ready to handle all possible disasters - because for most businesses having a disaster recovery plan is not optional.
In this session, we’ll discuss disaster scenarios that can take down entire Kafka clusters and share advice on how to plan, prepare and handle these events. This is a technical session full of best practices - we want to make sure you are ready to handle the worst mayhem that nature and auditors can cause.
Visit www.confluent.io for more information.
Building ClickHouse and Making Your First Contribution: A Tutorial_06.10.2021Altinity Ltd
ClickHouse is open source. You can build it yourself. What’s more, you can make it better! In this webinar, we’ll demonstrate how to pull the ClickHouse code from Github and build it. We’ll then walk through how to contribute a new feature to ClickHouse by developing, testing, and pushing a pull request through the community merge process. There will be demos and ample time for questions. Join us to get started as a ClickHouse developer!
Cassandra by example - the path of read and write requestsgrro
This article describes how Cassandra handles and processes requests. It will help you to get a better impression about Cassandra's internals and architecture. The path of a single read request as well as the path of a single write request will be described in detail.
Cassandra Day NY 2014: Getting Started with the DataStax C# DriverDataStax Academy
So you’ve grabbed the latest 2.0 beta of DataStax C# driver from NuGet. Now what? In this talk, Luke will walk you through some of the basics of the C# driver--how to bootstrap the driver and connect to a cluster, execute statements, and retrieve result sets. Wondering what the difference between a PreparedStatement and a SimpleStatement is? Not sure what the appropriate lifetime for a Cluster or a Session object is and whether you should reuse one (from multiple threads)? What about ADO.NET and LINQ support? We’ll cover this and more, so that you can get on with building applications on top of Cassandra and .NET.
Getting started with DataStax .NET Driver for CassandraLuke Tillman
Video of this presentation from Cassandra Day Seattle is here: https://www.youtube.com/watch?v=sbs6YExxYqc&index=6&list=PLqcm6qE9lgKIgRKG0d-NEvYw9qYOztbci
So you’ve grabbed the latest 2.0 version of the DataStax C# driver from NuGet. Now what? In this talk, Luke will walk you through some of the basics of the C# driver--how to bootstrap the driver and connect to a cluster, execute CQL, and retrieve the results. Wondering what the difference between a PreparedStatement and a SimpleStatement is? Not sure what the appropriate lifetime is for a Cluster or a Session object? What about ADO.NET and LINQ support? We’ll cover this and more, so that you can get on with building applications on top of Cassandra. Even if you’re not a C# developer (or think that C# is the handiwork of the devil), many of the concepts we’ll cover will help you get started with the other DataStax drivers as well (Python, Java, and C++).
All about Zookeeper and ClickHouse Keeper.pdfAltinity Ltd
ClickHouse clusters depend on ZooKeeper to handle replication and distributed DDL commands. In this Altinity webinar, we’ll explain why ZooKeeper is necessary, how it works, and introduce the new built-in replacement named ClickHouse Keeper. You’ll learn practical tips to care for ZooKeeper in sickness and health. You’ll also learn how/when to use ClickHouse Keeper. We will share our recommendations for keeping that happy as well.
Producer Performance Tuning for Apache KafkaJiangjie Qin
Kafka is well known for high throughput ingestion. However, to get the best latency characteristics without compromising on throughput and durability, we need to tune Kafka. In this talk, we share our experiences to achieve the optimal combination of latency, throughput and durability for different scenarios.
Everything You Always Wanted to Know About Kafka’s Rebalance Protocol but Wer...confluent
Apache Kafka is a scalable streaming platform with built-in dynamic client scaling. The elastic scale-in/scale-out feature leverages Kafka’s “rebalance protocol” that was designed in the 0.9 release and improved ever since then. The original design aims for on-prem deployments of stateless clients. However, it does not always align with modern deployment tools like Kubernetes and stateful stream processing clients, like Kafka Streams. Those shortcoming lead to two mayor recent improvement proposals, namely static group membership and incremental rebalancing (which will hopefully be available in version 2.3). This talk provides a deep dive into the details of the rebalance protocol, starting from its original design in version 0.9 up to the latest improvements and future work. We discuss internal technical details, pros and cons of the existing approaches, and explain how you configure your client correctly for your use case. Additionally, we discuss configuration tradeoffs for stateless, stateful, on-prem, and containerized deployments.
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of FacebookThe Hive
This presentation describes the reasons why Facebook decided to build yet another key-value store, the vision and architecture of RocksDB and how it differs from other open source key-value stores. Dhruba describes some of the salient features in RocksDB that are needed for supporting embedded-storage deployments. He explains typical workloads that could be the primary use-cases for RocksDB. He also lays out the roadmap to make RocksDB the key-value store of choice for highly-multi-core processors and RAM-speed storage devices.
The Missing Manual for Leveled Compaction Strategy (Wei Deng & Ryan Svihla, D...DataStax
In this presentation, we will look into JIRAs, JavaDocs and system log entries to gain a deeper understanding on how LCS works under the hood. We will explain what scenarios don't work well for LCS and (more importantly) why. We will leverage legacy TRACE/DEBUG level log for compaction related objects as well as some newer compaction logging information introduced in C* 3.6 (CASSANDRA-10805) to gain better insights.
About the Speakers
Wei Deng Solutions Architect, DataStax
Solutions Architect for DataStax. I have a strong interest in big data, cloud application and distributed computing practices.
C* Summit 2013: How Not to Use Cassandra by Axel LiljencrantzDataStax Academy
At Spotify, we see failure as an opportunity to learn. During the two years we've used Cassandra in our production environment, we have learned a lot. This session touches on some of the exciting design anti-patterns, performance killers and other opportunities to lose a finger that are at your disposal with Cassandra.
Apache Kafka becoming the message bus to transfer huge volumes of data from various sources into Hadoop.
It's also enabling many real-time system frameworks and use cases.
Managing and building clients around Apache Kafka can be challenging. In this talk, we will go through the best practices in deploying Apache Kafka
in production. How to Secure a Kafka Cluster, How to pick topic-partitions and upgrading to newer versions. Migrating to new Kafka Producer and Consumer API.
Also talk about the best practices involved in running a producer/consumer.
In Kafka 0.9 release, we’ve added SSL wire encryption, SASL/Kerberos for user authentication, and pluggable authorization. Now Kafka allows authentication of users, access control on who can read and write to a Kafka topic. Apache Ranger also uses pluggable authorization mechanism to centralize security for Kafka and other Hadoop ecosystem projects.
We will showcase open sourced Kafka REST API and an Admin UI that will help users in creating topics, re-assign partitions, Issuing
Kafka ACLs and monitoring Consumer offsets.
Modeling Data and Queries for Wide Column NoSQLScyllaDB
Discover how to model data for wide column databases such as ScyllaDB and Apache Cassandra. Contrast the differerence from traditional RDBMS data modeling, going from a normalized “schema first” design to a denormalized “query first” design. Plus how to use advanced features like secondary indexes and materialized views to use the same base table to get the answers you need.
Presentation at Strata Data Conference 2018, New York
The controller is the brain of Apache Kafka. A big part of what the controller does is to maintain the consistency of the replicas and determine which replica can be used to serve the clients, especially during individual broker failure.
Jun Rao outlines the main data flow in the controller—in particular, when a broker fails, how the controller automatically promotes another replica as the leader to serve the clients, and when a broker is started, how the controller resumes the replication pipeline in the restarted broker.
Jun then describes recent improvements to the controller that allow it to handle certain edge cases correctly and increase its performance, which allows for more partitions in a Kafka cluster.
Running Apache Kafka in production is only the first step in the Kafka operations journey. Professional Kafka users are ready to handle all possible disasters - because for most businesses having a disaster recovery plan is not optional.
In this session, we’ll discuss disaster scenarios that can take down entire Kafka clusters and share advice on how to plan, prepare and handle these events. This is a technical session full of best practices - we want to make sure you are ready to handle the worst mayhem that nature and auditors can cause.
Visit www.confluent.io for more information.
Building ClickHouse and Making Your First Contribution: A Tutorial_06.10.2021Altinity Ltd
ClickHouse is open source. You can build it yourself. What’s more, you can make it better! In this webinar, we’ll demonstrate how to pull the ClickHouse code from Github and build it. We’ll then walk through how to contribute a new feature to ClickHouse by developing, testing, and pushing a pull request through the community merge process. There will be demos and ample time for questions. Join us to get started as a ClickHouse developer!
Cassandra by example - the path of read and write requestsgrro
This article describes how Cassandra handles and processes requests. It will help you to get a better impression about Cassandra's internals and architecture. The path of a single read request as well as the path of a single write request will be described in detail.
Cassandra Day NY 2014: Getting Started with the DataStax C# DriverDataStax Academy
So you’ve grabbed the latest 2.0 beta of DataStax C# driver from NuGet. Now what? In this talk, Luke will walk you through some of the basics of the C# driver--how to bootstrap the driver and connect to a cluster, execute statements, and retrieve result sets. Wondering what the difference between a PreparedStatement and a SimpleStatement is? Not sure what the appropriate lifetime for a Cluster or a Session object is and whether you should reuse one (from multiple threads)? What about ADO.NET and LINQ support? We’ll cover this and more, so that you can get on with building applications on top of Cassandra and .NET.
Getting started with DataStax .NET Driver for CassandraLuke Tillman
Video of this presentation from Cassandra Day Seattle is here: https://www.youtube.com/watch?v=sbs6YExxYqc&index=6&list=PLqcm6qE9lgKIgRKG0d-NEvYw9qYOztbci
So you’ve grabbed the latest 2.0 version of the DataStax C# driver from NuGet. Now what? In this talk, Luke will walk you through some of the basics of the C# driver--how to bootstrap the driver and connect to a cluster, execute CQL, and retrieve the results. Wondering what the difference between a PreparedStatement and a SimpleStatement is? Not sure what the appropriate lifetime is for a Cluster or a Session object? What about ADO.NET and LINQ support? We’ll cover this and more, so that you can get on with building applications on top of Cassandra. Even if you’re not a C# developer (or think that C# is the handiwork of the devil), many of the concepts we’ll cover will help you get started with the other DataStax drivers as well (Python, Java, and C++).
.NET developers have a lot of options when it comes to databases these days. Apache Cassandra is a scalable, fault-tolerant database that has already found its way into more than 25% of the Fortune 100 and continues to grow in popularity. But what makes it different from the myriad of other options available? In this talk, we’ll take a deep dive into Cassandra and learn about:
- Cassandra’s internals and how it works
- CQL (the SQL-like query language for Cassandra)
- Data Modeling like a pro
- Tools available for developers
- Writing .NET code that talks to Cassandra
If there’s time and interest, we’ll finish up with how some companies are already using Cassandra to power services you probably interact with in your daily life. You’ll leave with all the tools you need to start build highly available .NET applications and services on top of Cassandra.
Cassandra is pretty awesome, sure I am biased, but it rocks. Always on, tuneable consistency and multi-master architecture? Let’s get our web scale on and build a highly available app that never goes down!
Hold on a second. There is one key piece of the puzzle that has a massive impact on your applications availability: the client driver.
In this talk we will go through the how to best configure your clients to make the most of failure handling and tuneable consistency in Cassandra.
Introduction to Data Modeling with Apache CassandraLuke Tillman
Relational systems have always been built on the premise of modeling relationships. As you will see, static schema, one-to-one, many-to-many still have a place in Cassandra. From the familiar, we’ll go into the specific differences in Cassandra and tricks to make your application fast and resilient.
Building your First Application with CassandraLuke Tillman
You’ve heard the talks, followed the tutorials, and done the research. You are a font of Cassandra knowledge. Now it’s time to change the world! (Or at least build something to make your boss happy). In this talk we’ll walk through the process of building KillrVideo, an open source video sharing website where users can upload and share videos, rate them, comment on them, and more. By looking at a real application, we’ll talk about architectural decisions, how the application drives the data model, some pro tips when using the DataStax drivers, and some lessons learned from mistakes made along the way. You’ll leave this session ready to start building your next application (world-changing or otherwise) with Cassandra.
Apache Cassandra is a scalable, fault-tolerant database that has found its way into more than 25% of the Fortune 100 and continues to enjoy significant adoption in the marketplace. In this talk we'll introduce you to Cassandra, explore some of its internals, and discuss CQL (the SQL-like query language for Cassandra). We'll finish by talking about how some companies are using it for services you probably interact with in your daily life. You'll leave with all the tools you need to start exploring Cassandra on your own.
Event Sourcing with Cassandra (from Cassandra Japan Meetup in Tokyo March 2016)Luke Tillman
In this talk, we'll dive into the event sourcing API used to persist actor state in Akka and talk about how we build a data model to support it in Cassandra. At first, the data model seems pretty straightforward, but the more we dig in, the more we see that a couple of classic Cassandra anti-patterns are pushing us close to the Pit of Despair. We'll come up with a way to avoid these problems so we can go on building distributed systems happily ever after with Akka and Cassandra.
Avoiding the Pit of Despair - Event Sourcing with Akka and CassandraLuke Tillman
With Akka you take a complicated system and break it down into lots of smaller units (actors) that communicate by passing messages. A single actor system can easily scale to millions or tens of millions of actors running on many machines. As actors process messages, they build up internal state, and many times we want that state persisted somewhere. In this talk, we'll dive into the event sourcing API used to persist actor state in Akka and talk about how we build a data model to support it in Cassandra. At first, the data model seems pretty straightforward, but the more we dig in, the more we see that a couple of classic Cassandra anti-patterns are pushing us close to the Pit of Despair. We'll come up with a way to avoid these problems so we can go on building distributed systems happily ever after with Akka and Cassandra.
Relational Scaling and the Temple of Gloom (from Cassandra Summit 2015)Luke Tillman
You're building the next big thing. It will attract hundreds of thousands of users and make so much cash, Gordon Gecko would blush. You just know that if you build it, they will come. But what happens when all those users do show up? Will you spend your time adding the new features they're clamoring for, or will you be scrambling to make sure your relational database doesn't die hard? In this talk, we'll take a look at some of the risky business we undertake to try and scale our relational databases and the problems we run into. Then we'll talk about how Cassandra is different and some of the knobs you control to turn things up to 11. If you're new to Cassandra and are looking for an introduction, come from a relational database background, or you just want to see how many 80s movie references we can cover in 40 minutes, then don your favorite fedora and come for an excellent adventure.
From Monolith to Microservices with Cassandra, gRPC, and Falcor (from Cassand...Luke Tillman
Transitioning a legacy monolithic application to microservices is a daunting task by itself and it only gets more complicated as you start to dig through all the libraries and frameworks out there meant to help. In this talk, we'll cover the transition of a real Cassandra-based application to a microservices architecture using Grpc from Google and Falcor from Netflix. (Yes, Falcor is more than just a magical luck dragon from an awesome 80's movie.) We'll talk about why these technologies were a good fit for the project as well as why Cassandra is often a great choice once you go down the path of microservices. And since all the code for the project is open source, you'll have plenty to dig into afterwards.
Slides for the talk "Cassandra and Spark: Love at First Sight" given at Texas Linux Fest 2015. Gives an introduction to both Cassandra and Spark and how they work together.
Scylla Summit 2016: Outbrain Case Study - Lowering Latency While Doing 20X IO...ScyllaDB
Outbrain is the world's largest content discovery program. Learn about their use case with Scylla where they lowered latency while doing 20X IOPS of Cassandra.
C* Summit 2013: Netflix Open Source Tools and Benchmarks for Cassandra by Adr...DataStax Academy
Netflix has updated and added new tools and benchmarks for Cassandra in the last year. In this talk we will cover the latest additions and recipes for the Astyanax Java client, updates to Priam to support Cassandra 1.2 Vnodes, plus newly released and upcoming tools that are all part of the NetflixOSS platform. Following on from the Cassandra on SSD on AWS benchmark that was run live during the 2012 Summit, we've been benchmarking a large write intensive multi-region cluster to see how far we can push it. Cassandra is the data storage and global replication foundation for the Cloud Native architecture that runs Netflix streaming for 36 Million users. Netflix is also offering a Cloud Prize for open source contributions to NetflixOSS, and there are ten categories including Best Datastore Integration and Best Contribution to Performance Improvements, with $10K cash and $5K of AWS credits for each winner. We'd like to pay you to use our free software!
Cyclone DDS Unleashed: Scalability in DDS and Dealing with Large SystemsZettaScaleTechnology
Welcome to the second episode of the "Cyclone DDS Unleashed" webinar series.
During this session, Lex Heerink (our Software Architect) discusses some of the challenges that users face when using DDS to build large systems and some of the design and deployment choices that can be made to address these issues.
If your robots work in busy factories or warehouses and you want to level up their efficiency, this session is for you.
If you have any questions and you want to reach out, you can send us an email at contact@zettascale.tech or join our Discord channel: https://discord.gg/6GwdBxntxt
You can read more about Cyclone DDS on our website: https://www.zettascale.tech/product/cyclone
Stay up to date with the latest news:
Twitter: https://twitter.com/zettascaletech
LinkedIn: https://www.linkedin.com/company/zettascaletech/
Website: https://www.zettascale.tech/
Newsletter: http://eepurl.com/igPw31
Leveraging Cassandra for real-time multi-datacenter public cloud analyticsJulien Anguenot
iland has built a global data warehouse across multiple data centers, collecting and aggregating data from core cloud services including compute, storage and network as well as chargeback and compliance. iland's warehouse brings actionable intelligence that customers can use to manipulate resources, analyze trends, define alerts and share information.
In this session, we would like to present the lessons learned around Cassandra, both at the development and operations level, but also the technology and architecture we put in action on top of Cassandra such as Redis, syslog-ng, RabbitMQ, Java EE, etc.
Finally, we would like to share insights on how we are currently extending our platform with Spark and Kafka and what our motivations are.
iland Internet Solutions: Leveraging Cassandra for real-time multi-datacenter...DataStax Academy
iland has built a global data warehouse across multiple data centers, collecting and aggregating data from core cloud services including compute, storage and network as well as chargeback and compliance. iland's warehouse brings actionable intelligence that customers can use to manipulate resources, analyze trends, define alerts and share information.
In this session, we would like to present the lessons learned around Cassandra, both at the development and operations level, but also the technology and architecture we put in action on top of Cassandra such as Redis, syslog-ng, RabbitMQ, Java EE, etc.
Finally, we would like to share insights on how we are currently extending our platform with Spark and Kafka and what our motivations are.
SignalFx: Making Cassandra Perform as a Time Series DatabaseDataStax Academy
SignalFx ingests, processes runs analytics against, (and ultimately stores) massive numbers of time series streaming in parallel into our service which provides an analytics-based monitoring platform for modern applications.
We've chose to build our time series database (TSDB) on Cassandra for it's read and write performance at high load. This presentation will go over our evolution of optimizations to squeeze the most performance out of the TSDB to date and some steps we'll be taking in the future.
Making Cassandra Perform as a Time Series Database - Cassandra Summit 15SignalFx
SignalFx engineer Paul Ingram presented these slides at Cassandra Summit 2015.
SignalFx ingests, processes runs analytics against, (and ultimately stores) massive numbers of time series streaming in parallel into our service which provides an analytics-based monitoring platform for modern applications.
We've chose to build our time series database (TSDB) on Cassandra for it's read and write performance at high load. This presentation will go over our evolution of optimizations to squeeze the most performance out of the TSDB to date and some steps we'll be taking in the future.
Read more: http://blog.signalfx.com/making-cassandra-perform-as-a-tsdb
This presentation is from the Gophercon-India where we talked about how to design a concurrent high performance database client in go language. We talked about how we use goroutines and channels to our advantages. we also talked about how to use pools for efficient memory utilization.
Geographically dispersed perconaxtra db cluster deploymentMarco Tusa
Geographically Dispersed Percona XtraDB Cluster Deployment
Percona XtraDB Cluster is a very robust, high performing and widly used solution to answer to High Availability needs. But it can be very challinging when we are in the need to deploy the cluster over a geographically disperse area.
This presentation will briefely discuss what is the right approach to sucessfully deploy PXC when in the need to cover multiple geographical sites, close and far.
- What is PXC and what happens in a set of node when commit
- Let us clarify, geo dispersed
- What to keep in mind then
- how to measure it correctly
- Use the right way (sync/async)
- Use help like replication_manager
AWS December 2015 Webinar Series - Design Patterns using Amazon DynamoDBAmazon Web Services
If you’re familiar with relational databases, designing your app to use a NoSQL database like DynamoDB may be new to you. In this webinar, we’ll walk you through common data design patterns for a variety of applications to help you learn how to design a schema, then store and retrieve the data with DynamoDB. We will discuss the benefits of using DynamoDB to develop mobile, web, IoT, and gaming apps.
Learning Objectives:
Learn schema design best practices with DynamoDB across multiple use cases, including gaming, AdTech, IoT, and others
Who Should Attend:
Architects, Developers, and SysOps interested in learning how to design NoSQL schemas to support mobile, web, IoT, AdTech, and gaming apps.
Familiarity with DynamoDB is helpful
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
A Deep Dive into Apache Cassandra for .NET Developers
1. A Deep Dive into Apache Cassandra for
.NET Developers
Luke Tillman (@LukeTillman)
Language Evangelist at DataStax
2. Who are you?!
• Evangelist with a focus on the .NET Community
• Long-time .NET Developer
• Recently presented at Cassandra Summit 2014 with Microsoft
• Very Recent Denver Transplant
2
3. Why should I care?
• Cassandra Core Principles
– Ease of Use
– Massive Scalability
– High Performance
– Always Available
• Growth!
3
DB Engines Rankings (January 2014)
http://db-engines.com/en/ranking
4. 1 What is Cassandra and how does it work?
2 Cassandra Query Language (CQL)
3 Data Modeling Like a Pro
4 .NET Driver for Cassandra
5 Tools and Code
4
6. What is Cassandra?
• A Linearly Scaling and Fault Tolerant Distributed Database
• Fully Distributed
– Data spread over many nodes
– All nodes participate in a cluster
– All nodes are equal (masterless)
– No SPOF (shared nothing)
6
7. What is Cassandra?
• Linearly Scaling
– Have More Data? Add more nodes.
– Need More Throughput? Add more nodes.
7
http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html
8. What is Cassandra?
• Fault Tolerant
– Nodes Down != Database Down
– Datacenter Down != Database Down
8
9. What is Cassandra?
• Fully Replicated
• Clients write local
• Data syncs across WAN
• Replication Factor per DC
9
US Europe
Client
10. Cassandra and the CAP Theorem
• The CAP Theorem limits what distributed systems can do
• Consistency
• Availability
• Partition Tolerance
• Limits? “Pick 2 out of 3”
• Cassandra is an AP system that is Eventually Consistent
10
11. Two knobs control Cassandra fault tolerance
• Replication Factor (server side)
– How many copies of the data should exist?
11
Client
B
AD
C
AB
A
CD
D
BC
Write A
RF=3
12. Two knobs control Cassandra fault tolerance
• Consistency Level (client side)
– How many replicas do we need to hear from before we acknowledge?
12
Client
B
AD
C
AB
A
CD
D
BC
Write A
CL=QUORUM
Client
B
AD
C
AB
A
CD
D
BC
Write A
CL=ONE
13. Consistency Levels
• Applies to both Reads and Writes (i.e. is set on each query)
• ONE – one replica from any DC
• LOCAL_ONE – one replica from local DC
• QUORUM – 51% of replicas from any DC
• LOCAL_QUORUM – 51% of replicas from local DC
• ALL – all replicas
• TWO
13
14. Consistency Level and Speed
• How many replicas we need to hear from can affect how quickly
we can read and write data in Cassandra
14
Client
B
AD
C
AB
A
CD
D
BC
5 µs ack
300 µs ack
12 µs ack
12 µs ack
Read A
(CL=QUORUM)
15. Consistency Level and Availability
• Consistency Level choice affects availability
• For example, QUORUM can tolerate one replica being down and
still be available (in RF=3)
15
Client
B
AD
C
AB
A
CD
D
BC
A=2
A=2
A=2
Read A
(CL=QUORUM)
16. Consistency Level and Eventual Consistency
• Cassandra is an AP system that is Eventually Consistent so
replicas may disagree
• Column values are timestamped
• In Cassandra, Last Write Wins (LWW)
16
Client
B
AD
C
AB
A
CD
D
BC
A=2
Newer
A=1
Older
A=2
Read A
(CL=QUORUM)
Christos from Netflix: “Eventual Consistency != Hopeful Consistency”
https://www.youtube.com/watch?v=lwIA8tsDXXE
17. Writes in the cluster
• Fully distributed, no SPOF
• Node that receives a request is the Coordinator for request
• Any node can act as Coordinator
17
Client
B
AD
C
AB
A
CD
D
BC
Write A
(CL=ONE)
Coordinator Node
18. Writes in the cluster – Data Distribution
• Partition Key determines node placement
18
Partition Key
id='pmcfadin' lastname='McFadin'
id='jhaddad' firstname='Jon' lastname='Haddad'
id='ltillman' firstname='Luke' lastname='Tillman'
CREATE TABLE users (
id text,
firstname text,
lastname text,
PRIMARY KEY (id)
);
19. Writes in the cluster – Data Distribution
• The Partition Key is hashed using a consistent hashing function
(Murmur 3) and the output is used to place the data on a node
• The data is also replicated to RF-1 other nodes
19
Partition Key
id='ltillman' firstname='Luke' lastname='Tillman'
Murmur3
id: ltillman Murmur3: A
B
AD
C
AB
A
CD
D
BC
RF=3
20. Hashing – Back to Reality
• Back in reality, Partition Keys actually hash to 128 bit numbers
• Nodes in Cassandra own token ranges (i.e. hash ranges)
20
B
AD
C
AB
A
CD
D
BC
Range Start End
A 0xC000000..1 0x0000000..0
B 0x0000000..1 0x4000000..0
C 0x4000000..1 0x8000000..0
D 0x8000000..1 0xC000000..0
Partition Key
id='ltillman' Murmur3 0xadb95e99da887a8a4cb474db86eb5769
21. Writes on a single node
• Client makes a write request
Client
UPDATE users
SET firstname = 'Luke'
WHERE id = 'ltillman'
Disk
Memory
22. Writes on a single node
Client
UPDATE users
SET firstname = 'Luke'
WHERE id = 'ltillman'
Commit Log
id='ltillman', firstname='Luke'
…
…
Disk
Memory
• Data is appended to the Commit Log
• Append only, sequential IO == FAST
23. Writes on a single node
Client
UPDATE users
SET firstname = 'Luke'
WHERE id = 'ltillman'
Commit Log
id='ltillman', firstname='Luke'
…
…
Disk
Memory
Memtable for Users Some
Other
Memtableid='ltillman' firstname='Luke' lastname='Tillman'
• Data is written/merged into Memtable
24. Writes on a single node
Client
UPDATE users
SET firstname = 'Luke'
WHERE id = 'ltillman'
Commit Log
id='ltillman', firstname='Luke'
…
…
Disk
Memory
Memtable for Users Some
Other
Memtableid='ltillman' firstname='Luke' lastname='Tillman'
• Server acknowledges to client
• Writes in C* are FAST due to simplicity, log structured storage
25. Writes on a single node
Client
UPDATE users
SET firstname = 'Luke'
WHERE id = 'ltillman'
Data Directory
Disk
Memory
Memtable for Users Some
Other
Memtableid='ltillman' firstname='Luke' lastname='Tillman'
Some
Other
SSTable
SSTable
#1 for
Users
SSTable
#2 for
Users
• Once Memtable is full, data is flushed to disk as SSTable (Sorted
String Table)
26. Compaction
• Compactions merge and unify data in our SSTables
• SSTables are immutable, so this is when we consolidate rows
26
SSTable
#1 for
Users
SSTable
#2 for
Users
SSTable #3 for
Users
id='ltillman'
firstname='Lucas'
(timestamp=Older)
lastname='Tillman'
id='ltillman' firstname='Luke' lastname='Tillman'
id='ltillman'
firstname='Luke'
(timestamp=Newer)
27. Reads in the cluster
• Same as writes in the cluster, reads are coordinated
• Any node can be the Coordinator Node
27
Client
B
AD
C
AB
A
CD
D
BC
Read A
(CL=QUORUM)
Coordinator Node
28. Reads on a single node
• Client makes a read request
28
Client
SELECT firstname, lastname
FROM users
WHERE id = 'ltillman'
Disk
Memory
29. Reads on a single node
• Data is read from (possibly multiple) SSTables and merged
29
Client
SELECT firstname, lastname
FROM users
WHERE id = 'ltillman'
Disk
Memory
SSTable #1 for Users
id='ltillman'
firstname='Lucas'
(timestamp=Older)
lastname='Tillman'
SSTable #2 for Users
id='ltillman'
firstname='Luke'
(timestamp=Newer)
firstname='Luke' lastname='Tillman'
30. Reads on a single node
• Any unflushed Memtable data is also merged
30
Client
SELECT firstname, lastname
FROM users
WHERE id = 'ltillman'
Disk
Memory
firstname='Luke' lastname='Tillman'
Memtable
for Users
31. Reads on a single node
• Client gets acknowledgement with the data
• Reads in Cassandra are also FAST but often limited by Disk IO
31
Client
SELECT firstname, lastname
FROM users
WHERE id = 'ltillman'
Disk
Memory
firstname='Luke' lastname='Tillman'
32. Compaction - Revisited
• Compactions merge and unify data in our SSTables, making
them important to reads (less SSTables = less to read/merge)
32
SSTable
#1 for
Users
SSTable
#2 for
Users
SSTable #3 for
Users
id='ltillman'
firstname='Lucas'
(timestamp=Older)
lastname='Tillman'
id='ltillman' firstname='Luke' lastname='Tillman'
id='ltillman'
firstname='Luke'
(timestamp=Newer)
34. Data Structures
• Keyspace is like RDBMS Database or Schema
• Like RDBMS, Cassandra uses Tables to store data
• Partitions can have one row (narrow) or multiple
rows (wide)
34
Keyspace
Tables
Partitions
Rows
35. Schema Definition (DDL)
• Easy to define tables for storing data
• First part of Primary Key is the Partition Key
CREATE TABLE videos (
videoid uuid,
userid uuid,
name text,
description text,
tags set<text>,
added_date timestamp,
PRIMARY KEY (videoid)
);
36. Schema Definition (DDL)
• One row per partition (familiar)
CREATE TABLE videos (
videoid uuid,
userid uuid,
name text,
description text,
tags set<text>,
added_date timestamp,
PRIMARY KEY (videoid)
);
name ...
Keyboard Cat ...
Nyan Cat ...
Original Grumpy Cat ...
videoid
689d56e5- …
93357d73- …
d978b136- …
37. Clustering Columns
• Second part of Primary Key is Clustering Columns
• Clustering columns affect ordering of data (on disk)
• Multiple rows per partition
37
CREATE TABLE comments_by_video (
videoid uuid,
commentid timeuuid,
userid uuid,
comment text,
PRIMARY KEY (videoid, commentid)
) WITH CLUSTERING ORDER BY (commentid DESC);
38. Clustering Columns – Wide Rows (Partitions)
• Use of Clustering Columns is where the (old) term “Wide Rows”
comes from
38
videoid='0fe6a...'
userid=
'ac346...'
comment=
'Awesome!'
commentid='82be1...'
(10/1/2014 9:36AM)
userid=
'f89d3...'
comment=
'Garbage!'
commentid='765ac...'
(9/17/2014 7:55AM)
CREATE TABLE comments_by_video (
videoid uuid,
commentid timeuuid,
userid uuid,
comment text,
PRIMARY KEY (videoid, commentid)
) WITH CLUSTERING ORDER BY (commentid DESC);
39. Inserts and Updates
• Use INSERT or UPDATE to add and modify data
• Both will overwrite data (no constraints like RDBMS)
• INSERT and UPDATE functionally equivalent
39
INSERT INTO comments_by_video (
videoid, commentid, userid, comment)
VALUES (
'0fe6a...', '82be1...', 'ac346...', 'Awesome!');
UPDATE comments_by_video
SET userid = 'ac346...', comment = 'Awesome!'
WHERE videoid = '0fe6a...' AND commentid = '82be1...';
40. TTL and Deletes
• Can specify a Time to Live (TTL) in seconds when doing an
INSERT or UPDATE
• Use DELETE statement to remove data
• Can optionally specify columns to remove part of a row
40
INSERT INTO comments_by_video ( ... )
VALUES ( ... )
USING TTL 86400;
DELETE FROM comments_by_video
WHERE videoid = '0fe6a...' AND commentid = '82be1...';
41. Querying
• Use SELECT to get data from your tables
• Always include Partition Key and optionally Clustering Columns
• Can use ORDER BY and LIMIT
• Use range queries (for example, by date) to slice partitions
41
SELECT * FROM comments_by_video
WHERE videoid = 'a67cd...'
LIMIT 10;
43. Cassandra Data Modeling
• Requires a different mindset than RDBMS modeling
• Know your data and your queries up front
• Queries drive a lot of the modeling decisions (i.e. “table per
query” pattern)
• Denormalize/Duplicate data at write time to do as few queries
as possible come read time
• Remember, disk is cheap and writes in Cassandra are FAST
43
44. Getting to Know Your Data
44
User
id
firstname
lastname
email
password
Video
id
name
description
location
preview_image
tags
features
Comment
comment
id
adds
timestamp
posts
timestamp
1
n
n
1
1
n
n
m
rates
rating
45. Application Workflows
45
User Logs
into site
Show basic
information
about user
Show videos
added by a
user
Show
comments
posted by a
user
Search for a
video by tag
Show latest
videos added
to the site
Show
comments
for a video
Show ratings
for a video
Show video
and its
details
46. Queries come from Workflows
46
Users
User Logs
into site
Find user by email
address
Show basic
information
about user
Find user by id
Comments
Show
comments
for a video
Find comments by
video (latest first)
Show
comments
posted by a
user
Find comments by
user (latest first)
Ratings
Show ratings
for a video Find ratings by video
47. Users – The Relational Way
• Single Users table with all user data and an Id Primary Key
• Add an index on email address to allow queries by email
User Logs
into site
Find user by email
address
Show basic
information
about user
Find user by id
48. Users – The Cassandra Way
User Logs
into site
Find user by email
address
Show basic
information
about user
Find user by id
CREATE TABLE user_credentials (
email text,
password text,
userid uuid,
PRIMARY KEY (email)
);
CREATE TABLE users (
userid uuid,
firstname text,
lastname text,
email text,
created_date timestamp,
PRIMARY KEY (userid)
);
49. Modeling Relationships – Collection Types
• Cassandra doesn’t support JOINs, but your data will still have
relationships (and you can still model that in Cassandra)
• One tool available is CQL collection types
CREATE TABLE videos (
videoid uuid,
userid uuid,
name text,
description text,
location text,
location_type int,
preview_image_location text,
tags set<text>,
added_date timestamp,
PRIMARY KEY (videoid)
);
50. Modeling Relationships – Client Side Joins
50
CREATE TABLE videos (
videoid uuid,
userid uuid,
name text,
description text,
location text,
location_type int,
preview_image_location text,
tags set<text>,
added_date timestamp,
PRIMARY KEY (videoid)
);
CREATE TABLE users (
userid uuid,
firstname text,
lastname text,
email text,
created_date timestamp,
PRIMARY KEY (userid)
);
Currently requires query for video,
followed by query for user by id based
on results of first query
51. Modeling Relationships – Client Side Joins
• What is the cost? Might be OK in small situations
• Do NOT scale
• Avoid when possible
51
54. .NET and Cassandra
• Open Source (on GitHub), available via NuGet
• Bootstrap using the Builder and then reuse the ISession object
Cluster cluster = Cluster.Builder()
.AddContactPoint("127.0.0.1")
.Build();
ISession session = cluster.Connect("killrvideo");
54
55. .NET and Cassandra
• Executing CQL with SimpleStatement
• Sync and Async API available for executing statements
• Use Async API for executing queries in parallel
var videoId = Guid.NewGuid();
var statement = new SimpleStatement("SELECT * FROM videos WHERE videoid = ?",
videoId);
RowSet rows = await session.ExecuteAsync(statement);
55
56. .NET and Cassandra
• Getting values from a RowSet is easy
• Rowset is a collection of Row (IEnumerable<Row>)
RowSet rows = await _session.ExecuteAsync(statement);
foreach (Row row in rows)
{
var videoId = row.GetValue<Guid>("videoid");
var addedDate = row.GetValue<DateTimeOffset>("added_date");
var name = row.GetValue<string>("name");
}
56
57. CQL 3 Data Types to .NET Types
• Full listing available in driver docs (http://www.datastax.com/docs)
CQL 3 Data Type .NET Type
bigint, counter long
boolean bool
decimal, float float
double double
int int
uuid, timeuuid System.Guid
text, varchar string (Encoding.UTF8)
timestamp System.DateTimeOffset
varint System.Numerics.BigInteger
58. .NET and Cassandra - PreparedStatement
• SimpleStatement useful for one-off CQL execution (or when
dynamic CQL is a possibility)
• Use PreparedStatement for better performance
• Pay the cost of Prepare once (server roundtrip)
• Save the PreparedStatement instance and reuse
PreparedStatement prepared = session.Prepare(
"SELECT * FROM user_credentials WHERE email = ?");
59. .NET and Cassandra - PreparedStatement
• Bind variable values to get BoundStatement for execution
• Execution only has to send variable values
• You will use these all the time
• Remember: Prepare once, bind and execute many
BoundStatement bound = prepared.Bind("luke.tillman@datastax.com");
RowSet rows = await _session.ExecuteAsync(bound);
60. .NET and Cassandra - BatchStatement
• Add Simple/Bound statements to a batch
BoundStatement bound = prepared.Bind(video.VideoId, video.Name);
var simple = new SimpleStatement(
"UPDATE videos SET name = ? WHERE videoid = ?"
).Bind(video.Name, video.VideoId);
// Use an atomic batch to send over all the mutations
var batchStatement = new BatchStatement();
batchStatement.Add(bound);
batchStatement.Add(simple);
RowSet rows = await _session.ExecuteAsync(batch);
61. .NET and Cassandra - BatchStatement
• Batches are Logged (atomic) by default
• Use when you want a group of mutations (statements) to all
succeed or all fail (denormalizing at write time)
• Really large batches are an anti-pattern (and Cassandra will
warn you)
• Not a performance optimization for bulk-loading data
62. .NET and Cassandra – Statement Options
• Options like Consistency Level and Retry Policy are available at
the Statement level
• If not set on a statement, driver will fallback to defaults set when
building/configuring the Cluster
62
IStatement bound =
prepared.Bind("luke.tillman@datastax.com")
.SetPageSize(100)
.SetConsistencyLevel(ConsistencyLevel.LocalOne)
.SetRetryPolicy(new DefaultRetryPolicy())
.EnableTracing();
63. Lightweight Transactions (LWT)
• Use when you don’t want writes to step on each other
• AKA Linearizable Consistency
• Serial Isolation Level
• Be sure to read the fine print: has a latency cost associated
with using it, so use only where needed
• The canonical example: unique user accounts
64. Lightweight Transactions (LWT)
• Returns a column called [applied] indicating success/failure
• Different from the relational world where you might expect an
Exception (i.e. PrimaryKeyViolationException or similar)
var statement = new SimpleStatement("INSERT INTO user_credentials (email,
password) VALUES (?, ?) IF NOT EXISTS");
statement = statement.Bind("user1@killrvideo.com", "Password1!");
RowSet rows = await _session.ExecuteAsync(statement);
var userInserted = rows.Single().GetValue<bool>("[applied]");
65. Automatic Paging
• The Problem: Loading big result sets into memory is a recipe
for disaster (OutOfMemoryExceptions, etc.)
• Better to load and process a large result set in pages (chunks)
• Doing this manually with Cassandra prior to 2.0 was a pain
• Automatic Paging makes paging on a large RowSet
transparent
66. Automatic Paging
• Set a page size on a statement
• Iterate over the resulting RowSet
• As you iterate, new pages are fetched transparently when the
Rows in the current page are exhausted
• Will allow you to iterate until all pages are exhausted
boundStatement = boundStatement.SetPageSize(100);
RowSet rows = await _session.ExecuteAsync(boundStatement);
foreach (Row row in rows)
{
}
67. Typical Pager UI in a Web Application
• Show page of records in UI and allow user to navigate
68. Typical Pager UI in a Web Application
• Automatic Paging – this is not the feature you are looking for
69. .NET and Cassandra
• Mapping results to DTOs: if you like using CQL for querying, try
Mapper component (formerly CqlPoco package)
public class User
{
public Guid UserId { get; set; }
public string Name { get; set; }
}
// Create a mapper from your session object
var mapper = new Mapper(session);
// Get a user by id from Cassandra or null if not found
var user = client.SingleOrDefault<User>(
"SELECT userid, name FROM users WHERE userid = ?", someUserId);
69
70. .NET and Cassandra
• Mapping results to DTOs: if you like LINQ, use built-in LINQ
provider
[Table("users")]
public class User
{
[Column("userid"), PartitionKey]
public Guid UserId { get; set; }
[Column("name")]
public string Name { get; set; }
}
var user = session.GetTable<User>()
.SingleOrDefault(u => u.UserId == someUserId)
.Execute();
70
73. Installing Cassandra
• Windows installer is super easy way to do development and
testing on your local machine
• Production Cassandra deployments on Linux (Windows
performance parity is coming in 3.0)
74. In the Box – Command Line Tools
• Use cqlsh REPL for running CQL against your cluster
74
75. In the Box – Command Line Tools
• Use nodetool for information on your cluster (and lots more)
• On Windows, available under apache-cassandrabin folder
75
76. DevCenter
• Get it on DataStax
web site
(www.datastax.com)
• GUI for Cassandra
development (think
SQL Server
Management Studio
for Cassandra)
76
77. Sample Code – KillrVideo
• Live demo available at http://www.killrvideo.com
– Written in C#
– Live Demo running in Azure
– Open source: https://github.com/luketillman/killrvideo-csharp
77
78. Questions? Overtime (use cases)?
Follow me for updates or to ask questions later: @LukeTillman
78
81. Some Common Use Case Categories
• Product Catalogs and Playlists
• Internet of Things (IoT) and
Sensor Data
• Messaging (emails, IMs, alerts,
comments)
• Recommendation and
Personalization
• Fraud Detection
• Time series and temporal
ordered data
http://planetcassandra.org/apache-cassandra-use-cases/
82. The “Slide Heard Round the World”
• From Cassandra
Summit 2014, got a
lot of attention
• 75,000+ nodes
• 10s of PBs of data
• Millions ops/s
• One of the largest
known Cassandra
deployments
82
83. Spotify
• Streaming music web service
• > 24,000,000 music tracks
• > 50TB of data in Cassandra
Why Cassandra?
• Was PostgreSQL, but hit scaling
problems
• Multi Datacenter Availability
• Integration with Spark for data
processing and analytics
Usage
• Catalog
• User playlists
• Artists following
• Radio Stations
• Event notifications
83
http://planetcassandra.org/blog/interview/spotify-scales-to-the-top-of-the-charts-with-apache-cassandra-at-40k-requestssecond/
84. eBay
• Online auction site
• > 250TB of data, dozens of nodes,
multiple data centres
• > 6 billion writes, > 5 billion reads
per day
Why Cassandra?
• Low latency, high scale, multiple data
centers
• Suited for graph structures using
wide rows
Usage
• Building next generation of
recommendation engine
• Storing user activity data
• Updating models of user interests in
real time
84
http://planetcassandra.org/blog/5-minute-c-interview-ebay/
85. FullContact
• Contact management: from multiple
sources, sync, de-dupe, APIs available
• 2 clusters, dozens of nodes, running
in AWS
• Based here in Denver
Why Cassandra?
• Migated from MongoDB after
running into scaling issues
• Operational simplicity
• Resilience and Availability
Usage
• Person API (search by email, Twitter
handle, Facebook, or phone)
• Searched data from multiple sources
(ingested by Hadoop M/R jobs)
• Resolved profiles
85
http://planetcassandra.org/blog/fullcontact-readies-their-search-platform-to-scale-moves-from-mongodb-to-apache-cassandra/
86. Instagram
• Photo-sharing, video-sharing and
social networking service
• Originally AWS (Now Facebook data
centers?)
• > 20k writes/second, >15k
reads/second
Why Cassandra?
• Migrated from Redis (problems
keeping everything in memory)
• No painful “sharding” process
• 75% reduction in costs
Usage
• Auditing information – security,
integrity, spam detection
• News feed (“inboxes” or activity feed)
– Likes, Follows, etc.
86
http://planetcassandra.org/blog/instagram-making-the-switch-to-cassandra-from-redis-75-instasavings/
Summit 2014 Presentation: https://www.youtube.com/watch?v=_gc94ITUitY
87. Netflix
• TV and Movie streaming service
• > 2700+ nodes on over 90 clusters
• 4 Datacenters
• > 1 Trillion operations per day
Why Cassandra?
• Migrated from Oracle
• Massive amounts of data
• Multi datacenter, No SPOF
• No downtime for schema changes
Usage
• Everything! (Almost – 95% of DB use)
• Example: Personalization
– What titles do you play?
– What do you play before/after?
– Where did you pause?
– What did you abandon watching after 5
minutes?
87
http://planetcassandra.org/blog/case-study-netflix/
Summit 2014 Presentation: https://www.youtube.com/watch?v=RMSNLP_ORg8&index=43&list=UUvP-AXuCr-naAeEccCfKwUA
88. Go forth and build awesome things!
Follow me for updates or to ask questions later: @LukeTillman
88