This article describes how Cassandra handles and processes requests. It will help you to get a better impression about Cassandra's internals and architecture. The path of a single read request as well as the path of a single write request will be described in detail.
Introduction to Cassandra: Replication and ConsistencyBenjamin Black
A short introduction to replication and consistency in the Cassandra distributed database. Delivered April 28th, 2010 at the Seattle Scalability Meetup.
A Deep Dive into Apache Cassandra for .NET DevelopersLuke Tillman
.NET developers have a lot of options when it comes to databases these days. Apache Cassandra is a scalable, fault-tolerant database that has already found its way into more than 25% of the Fortune 100 and continues to grow in popularity. But what makes it different from the myriad of other options available? In this talk, we’ll take a deep dive into Cassandra and learn about:
- Cassandra’s internals and how it works
- CQL (the SQL-like query language for Cassandra)
- Data Modeling like a pro
- Tools available for developers
- Writing .NET code that talks to Cassandra
If there’s time and interest, we’ll finish up with how some companies are already using Cassandra to power services you probably interact with in your daily life. You’ll leave with all the tools you need to start build highly available .NET applications and services on top of Cassandra.
Instaclustr has a diverse customer base including Ad Tech, IoT and messaging applications ranging from small start ups to large enterprises. In this presentation we share our experiences, common issues, diagnosis methods, and some tips and tricks for managing your Cassandra cluster.
About the Speaker
Brooke Jensen VP Technical Operations & Customer Services, Instaclustr
Instaclustr is the only provider of fully managed Cassandra as a Service in the world. Brooke Jensen manages our team of Engineers that maintain the operational performance of our diverse fleet clusters, as well as providing 24/7 advice and support to our customers. Brooke has over 10 years' experience as a Software Engineer, specializing in performance optimization of large systems and has extensive experience managing and resolving major system incidents.
Scylla allows us to create highly performant and scalable systems. However, to achieve good results and prevent our Scylla cluster from being overloaded, we need to properly write our client application and configure the driver. Join this session to learn some practical tips that can help you make your applications faster and more available.
From: DataWorks Summit 2017 - Munich - 20170406
HBase hast established itself as the backend for many operational and interactive use-cases, powering well-known services that support millions of users and thousands of concurrent requests. In terms of features HBase has come a long way, overing advanced options such as multi-level caching on- and off-heap, pluggable request handling, fast recovery options such as region replicas, table snapshots for data governance, tuneable write-ahead logging and so on. This talk is based on the research for the an upcoming second release of the speakers HBase book, correlated with the practical experience in medium to large HBase projects around the world. You will learn how to plan for HBase, starting with the selection of the matching use-cases, to determining the number of servers needed, leading into performance tuning options. There is no reason to be afraid of using HBase, but knowing its basic premises and technical choices will make using it much more successful. You will also learn about many of the new features of HBase up to version 1.3, and where they are applicable.
Introduction to Cassandra: Replication and ConsistencyBenjamin Black
A short introduction to replication and consistency in the Cassandra distributed database. Delivered April 28th, 2010 at the Seattle Scalability Meetup.
A Deep Dive into Apache Cassandra for .NET DevelopersLuke Tillman
.NET developers have a lot of options when it comes to databases these days. Apache Cassandra is a scalable, fault-tolerant database that has already found its way into more than 25% of the Fortune 100 and continues to grow in popularity. But what makes it different from the myriad of other options available? In this talk, we’ll take a deep dive into Cassandra and learn about:
- Cassandra’s internals and how it works
- CQL (the SQL-like query language for Cassandra)
- Data Modeling like a pro
- Tools available for developers
- Writing .NET code that talks to Cassandra
If there’s time and interest, we’ll finish up with how some companies are already using Cassandra to power services you probably interact with in your daily life. You’ll leave with all the tools you need to start build highly available .NET applications and services on top of Cassandra.
Instaclustr has a diverse customer base including Ad Tech, IoT and messaging applications ranging from small start ups to large enterprises. In this presentation we share our experiences, common issues, diagnosis methods, and some tips and tricks for managing your Cassandra cluster.
About the Speaker
Brooke Jensen VP Technical Operations & Customer Services, Instaclustr
Instaclustr is the only provider of fully managed Cassandra as a Service in the world. Brooke Jensen manages our team of Engineers that maintain the operational performance of our diverse fleet clusters, as well as providing 24/7 advice and support to our customers. Brooke has over 10 years' experience as a Software Engineer, specializing in performance optimization of large systems and has extensive experience managing and resolving major system incidents.
Scylla allows us to create highly performant and scalable systems. However, to achieve good results and prevent our Scylla cluster from being overloaded, we need to properly write our client application and configure the driver. Join this session to learn some practical tips that can help you make your applications faster and more available.
From: DataWorks Summit 2017 - Munich - 20170406
HBase hast established itself as the backend for many operational and interactive use-cases, powering well-known services that support millions of users and thousands of concurrent requests. In terms of features HBase has come a long way, overing advanced options such as multi-level caching on- and off-heap, pluggable request handling, fast recovery options such as region replicas, table snapshots for data governance, tuneable write-ahead logging and so on. This talk is based on the research for the an upcoming second release of the speakers HBase book, correlated with the practical experience in medium to large HBase projects around the world. You will learn how to plan for HBase, starting with the selection of the matching use-cases, to determining the number of servers needed, leading into performance tuning options. There is no reason to be afraid of using HBase, but knowing its basic premises and technical choices will make using it much more successful. You will also learn about many of the new features of HBase up to version 1.3, and where they are applicable.
Scylla Summit 2022: Scylla 5.0 New Features, Part 1ScyllaDB
Discover the new features and capabilities of Scylla Open Source 5.0 directly from the engineers who developed it. This second block of lightning talks will cover the following topics:
- New IO Scheduler and Disk Parallelism
- Per-Service-Level Timeouts
- Better Workload Estimation for Backpressure and Out-of-Memory Conditions
- Large Partition Handling Improvements
- Optimizing Reverse Queries
To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.
Storing time series data with Apache CassandraPatrick McFadin
If you are looking to collect and store time series data, it's probably not going to be small. Don't get caught without a plan! Apache Cassandra has proven itself as a solid choice now you can learn how to do it. We'll look at possible data models and the the choices you have to be successful. Then, let's open the hood and learn about how data is stored in Apache Cassandra. You don't need to be an expert in distributed systems to make this work and I'll show you how. I'll give you real-world examples and work through the steps. Give me an hour and I will upgrade your time series game.
Deletes Without Tombstones or TTLs (Eric Stevens, ProtectWise) | Cassandra Su...DataStax
Deleting data from Cassandra has several challenges, and existing solutions (tombstones or TTLs) have limitations that make them unusable or untenable in certain circumstances. We'll explore the cases where existing deletion options fail or are inadequate, then describe a solution we developed which deletes data from Cassandra during standard or user-defined compaction, but without resorting to tombstones or TTL's.
About the Speaker
Eric Stevens Principal Architect, ProtectWise, Inc.
Eric is the principal architect, and day one employee of ProtectWise, Inc., specializing in massive real time processing and scalability problems. The team at ProtectWise processes, analyzes, optimizes, indexes, and stores billions of network packets each second. They look for threats in real time, but also store full fidelity network data (including PCAP), and when new security intelligence is received, automatically replay existing network history through that new intelligence.
Cassandra concepts, patterns and anti-patternsDave Gardner
An introduction to the fundamental concepts behind Apache Cassandra. This talk explains the engineering principles that make Cassandra such an attractive choice for building highly resilient and available systems and then goes on to explain how to use it - covering basic data modelling patterns and anti-patterns.
Scylla Summit 2022: How to Migrate a Counter Table for 68 Billion RecordsScyllaDB
In this talk, we will discuss Happn's war story about migrating a Cassandra 2.1 cluster containing more than 68 Billion records in a counter table to ScyllaDB Open Source.
To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.
All about Zookeeper and ClickHouse Keeper.pdfAltinity Ltd
ClickHouse clusters depend on ZooKeeper to handle replication and distributed DDL commands. In this Altinity webinar, we’ll explain why ZooKeeper is necessary, how it works, and introduce the new built-in replacement named ClickHouse Keeper. You’ll learn practical tips to care for ZooKeeper in sickness and health. You’ll also learn how/when to use ClickHouse Keeper. We will share our recommendations for keeping that happy as well.
Understanding How CQL3 Maps to Cassandra's Internal Data StructureDataStax
CQL3 is the newly ordained, canonical, and best-practices means of interacting with Cassandra. Indeed, the Apache Cassandra documentation itself declares that the Thrift API as “legacy” and recommends that CQL3 be used instead. But I’ve heard several people express their concern over the added layer of abstraction. There seems to be an uncertainty about what’s really happening inside of Cassandra.
In this presentation we will open up the hood and take a look at exactly how Cassandra is treating CQL3 queries. Our first stop will be the Cassandra data structure itself. We will briefly review the concepts of keyspaces, columnfamilies, rows, and columns. And we will explain where this data structure excels and where it does not. Composite rowkeys and columnnames are heavily used with CQL3, so we'll cover their functionality as well.
We will then turn to CQL3. I will demonstrate the basic CQL syntax and show how it maps to the underlying data structure. We will see that CQL actually serves as a sort of best practices interface to the internal Cassandra data structure. We will take this point further by demonstrating CQL3 collections (set, list, and map) and showing how they are really just a creative use of this same internal data structure.
Attendees will leave with a clear, inside-out understanding of CQL3 and will be able use CQL with a confidence that they are following best-practices.
This talk delves into the many ways that a user has to use HBase in a project. Lars will look at many practical examples based on real applications in production, for example, on Facebook and eBay and the right approach for those wanting to find their own implementation. He will also discuss advanced concepts, such as counters, coprocessors and schema design.
Bucket your partitions wisely - Cassandra summit 2016Markus Höfer
When we talk about bucketing we essentially talk about possibilities to split cassandra partitions in several smaller parts, rather than having only one large partition.
Bucketing of cassandra partitions can be crucial for optimizing queries, preventing large partitions or to fight TombstoneOverwhelmingException which can occur when creating too many tombstones.
In this talk I want to show how to recognize large partitions during datamodeling. I will also show different strategies we used in our projects to create, use and maintain buckets for our partitions.
Amazon Elastic Block Store (Amazon EBS) provides persistent block level storage volumes for use with Amazon EC2 instances. In this technical session, we conduct a detailed analysis of the differences among the three types of Amazon EBS block storage: General Purpose (SSD), Provisioned IOPS (SSD), and Magnetic. We discuss how to maximize Amazon EBS performance, with a special eye towards low-latency, high-throughput applications like databases. We discuss Amazon EBS encryption and share best practices for Amazon EBS snapshot management. Throughout, we share tips for success.
Scylla Summit 2022: The Future of Consensus in ScyllaDB 5.0 and BeyondScyllaDB
Beyond the immediate schema changes supported in Scylla Open Source 5.0, learn how the Raft consensus infrastructure will enable radical new capabilities. Discover how it will enable more dynamic topology changes, tablets, immediate consistency, better and faster elasticity, and simplification to repair operations.
To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.
Scylla Summit 2022: Scylla 5.0 New Features, Part 2ScyllaDB
Discover the new features and capabilities of Scylla Open Source 5.0 directly from the engineers who developed it. This first block of lightning talks will cover the following topics:
- Repair Based Node Operations
- Repair Based Tombstone Garbage Collection
- Compaction Improvements
- Range Tombstone Improvements
To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.
Scylla Summit 2022: Scylla 5.0 New Features, Part 1ScyllaDB
Discover the new features and capabilities of Scylla Open Source 5.0 directly from the engineers who developed it. This second block of lightning talks will cover the following topics:
- New IO Scheduler and Disk Parallelism
- Per-Service-Level Timeouts
- Better Workload Estimation for Backpressure and Out-of-Memory Conditions
- Large Partition Handling Improvements
- Optimizing Reverse Queries
To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.
Storing time series data with Apache CassandraPatrick McFadin
If you are looking to collect and store time series data, it's probably not going to be small. Don't get caught without a plan! Apache Cassandra has proven itself as a solid choice now you can learn how to do it. We'll look at possible data models and the the choices you have to be successful. Then, let's open the hood and learn about how data is stored in Apache Cassandra. You don't need to be an expert in distributed systems to make this work and I'll show you how. I'll give you real-world examples and work through the steps. Give me an hour and I will upgrade your time series game.
Deletes Without Tombstones or TTLs (Eric Stevens, ProtectWise) | Cassandra Su...DataStax
Deleting data from Cassandra has several challenges, and existing solutions (tombstones or TTLs) have limitations that make them unusable or untenable in certain circumstances. We'll explore the cases where existing deletion options fail or are inadequate, then describe a solution we developed which deletes data from Cassandra during standard or user-defined compaction, but without resorting to tombstones or TTL's.
About the Speaker
Eric Stevens Principal Architect, ProtectWise, Inc.
Eric is the principal architect, and day one employee of ProtectWise, Inc., specializing in massive real time processing and scalability problems. The team at ProtectWise processes, analyzes, optimizes, indexes, and stores billions of network packets each second. They look for threats in real time, but also store full fidelity network data (including PCAP), and when new security intelligence is received, automatically replay existing network history through that new intelligence.
Cassandra concepts, patterns and anti-patternsDave Gardner
An introduction to the fundamental concepts behind Apache Cassandra. This talk explains the engineering principles that make Cassandra such an attractive choice for building highly resilient and available systems and then goes on to explain how to use it - covering basic data modelling patterns and anti-patterns.
Scylla Summit 2022: How to Migrate a Counter Table for 68 Billion RecordsScyllaDB
In this talk, we will discuss Happn's war story about migrating a Cassandra 2.1 cluster containing more than 68 Billion records in a counter table to ScyllaDB Open Source.
To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.
All about Zookeeper and ClickHouse Keeper.pdfAltinity Ltd
ClickHouse clusters depend on ZooKeeper to handle replication and distributed DDL commands. In this Altinity webinar, we’ll explain why ZooKeeper is necessary, how it works, and introduce the new built-in replacement named ClickHouse Keeper. You’ll learn practical tips to care for ZooKeeper in sickness and health. You’ll also learn how/when to use ClickHouse Keeper. We will share our recommendations for keeping that happy as well.
Understanding How CQL3 Maps to Cassandra's Internal Data StructureDataStax
CQL3 is the newly ordained, canonical, and best-practices means of interacting with Cassandra. Indeed, the Apache Cassandra documentation itself declares that the Thrift API as “legacy” and recommends that CQL3 be used instead. But I’ve heard several people express their concern over the added layer of abstraction. There seems to be an uncertainty about what’s really happening inside of Cassandra.
In this presentation we will open up the hood and take a look at exactly how Cassandra is treating CQL3 queries. Our first stop will be the Cassandra data structure itself. We will briefly review the concepts of keyspaces, columnfamilies, rows, and columns. And we will explain where this data structure excels and where it does not. Composite rowkeys and columnnames are heavily used with CQL3, so we'll cover their functionality as well.
We will then turn to CQL3. I will demonstrate the basic CQL syntax and show how it maps to the underlying data structure. We will see that CQL actually serves as a sort of best practices interface to the internal Cassandra data structure. We will take this point further by demonstrating CQL3 collections (set, list, and map) and showing how they are really just a creative use of this same internal data structure.
Attendees will leave with a clear, inside-out understanding of CQL3 and will be able use CQL with a confidence that they are following best-practices.
This talk delves into the many ways that a user has to use HBase in a project. Lars will look at many practical examples based on real applications in production, for example, on Facebook and eBay and the right approach for those wanting to find their own implementation. He will also discuss advanced concepts, such as counters, coprocessors and schema design.
Bucket your partitions wisely - Cassandra summit 2016Markus Höfer
When we talk about bucketing we essentially talk about possibilities to split cassandra partitions in several smaller parts, rather than having only one large partition.
Bucketing of cassandra partitions can be crucial for optimizing queries, preventing large partitions or to fight TombstoneOverwhelmingException which can occur when creating too many tombstones.
In this talk I want to show how to recognize large partitions during datamodeling. I will also show different strategies we used in our projects to create, use and maintain buckets for our partitions.
Amazon Elastic Block Store (Amazon EBS) provides persistent block level storage volumes for use with Amazon EC2 instances. In this technical session, we conduct a detailed analysis of the differences among the three types of Amazon EBS block storage: General Purpose (SSD), Provisioned IOPS (SSD), and Magnetic. We discuss how to maximize Amazon EBS performance, with a special eye towards low-latency, high-throughput applications like databases. We discuss Amazon EBS encryption and share best practices for Amazon EBS snapshot management. Throughout, we share tips for success.
Scylla Summit 2022: The Future of Consensus in ScyllaDB 5.0 and BeyondScyllaDB
Beyond the immediate schema changes supported in Scylla Open Source 5.0, learn how the Raft consensus infrastructure will enable radical new capabilities. Discover how it will enable more dynamic topology changes, tablets, immediate consistency, better and faster elasticity, and simplification to repair operations.
To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.
Scylla Summit 2022: Scylla 5.0 New Features, Part 2ScyllaDB
Discover the new features and capabilities of Scylla Open Source 5.0 directly from the engineers who developed it. This first block of lightning talks will cover the following topics:
- Repair Based Node Operations
- Repair Based Tombstone Garbage Collection
- Compaction Improvements
- Range Tombstone Improvements
To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.
What Every Developer Should Know About Database Scalabilityjbellis
Replication. Partitioning. Relational databases. Bigtable. Dynamo. There is no one-size-fits-all approach to scaling your database, and the CAP theorem proved that there never will be. This talk will explain the advantages and limits of the approaches to scaling traditional relational databases, as well as the tradeoffs made by the designers of newer distributed systems like Cassandra. These slides are from Jonathan Ellis's OSCON 09 talk: http://en.oreilly.com/oscon2009/public/schedule/detail/7955
ContextSpace is working to develop and support an open source implementation of the Camunda core engine that persists all if its data to Cassandra. This development addresses issues of ACID as well as approaches to lock management. ContextSpace plans to integrate this implementation with its own product offering in order to expose data and events generated from its identity, security, roles, messaging and contextual user activities to be managed by Camunda-driven business processes.
This talk will cover what really goes in the read path, from the driver all the way to reading from SSTables on disk. This includes selecting a coordinator, paging, selecting replicas, handling failures, handling inconsistency, index lookups, bloom filters, and more.
About the Speaker
Tyler Hobbs Cassandra Committer, Datastax
I've been a Cassandra committer for two years, and have worked on Cassandra-related projects for the last six years.
Further discussion on Data Modeling with Apache Cassandra. Overview of formal data modeling techniques as well as practical. Real-world use cases and associated data models.
Next Generation Databases mostly addressing some of the points: being non-relational, distributed, open-source and horizontally scalable. The original intention has been modern web-scale databases. The movement began early 2009 and is growing rapidly. Often more characteristics apply such as: schema-free, easy replication support, simple API, eventually consistent / BASE (not ACID), a huge amount of data and more. So the misleading term "nosql" (the community now translates it mostly with "not only sql") should be seen as an alias to something like the definition above.
Cassandra's Sweet Spot - an introduction to Apache CassandraDave Gardner
Slides from my NoSQL Exchange 2011 talk introducing Apache Cassandra. This talk explained the fundamental concepts of Cassandra and then demonstrated how to build a simple ad-targeting application using PHP, with a focus on data modeling.
Video of talk: http://skillsmatter.com/podcast/home/cassandra/js-2880
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
"Impact of front-end architecture on development cost", Viktor Turskyi
Cassandra by example - the path of read and write requests
1. Cassandra by example - the path of read and
write requests
Abstract
This article describes how Cassandra handles and processes requests. It will help you to get a better
impression about Cassandra's internals and architecture. The path of a single read request as well as
the path of a single write request will be described in detail. This description is based on a single data
center Cassandra V1.1.4 cluster (default store configuration).
Example data model
Please consider that this article is not an introduction to the Cassandra model. In the examples below
a column family hotel is used. In short, a column family is analogous to tables of the relational
database approach. Each hotel record or row is identified by a unique key. The columns of a hotel
row include the hotel name as well as the category of the hotel.
The column family hotel lives inside the keyspace book_a_hotel . A keyspace can be described by
analogy as a tablespace or database.
Thrift
The common way to access Cassandra is using Thrift. Thrift is a language-independent RPC protocol
originally developed at Facebook and contributed to Apache. Although Thrift is widely supported by
the most popular programming languages the Cassandra project suggests using higher level
Cassandra clients such as Hector or Astyanax instead the raw Thrift-based API. In general these high
level clients try to hide the underlying middleware protocol.
Gregor Roth Cassandra by example - the path of read and write requests 1
2. The listing below shows a simple query by using the Hector client library V1.1.
// [1] prepare the client (cluster)
Cluster cluster = HFactory.getOrCreateCluster("TestClstr", "172.39.126.14, 172.39.126.93, 172.39.126.52");
Keyspace keyspaceOperator = HFactory.createKeyspace("book_a_hotel", cluster);
// [2] create the query (fetching the column category)
SliceQuery<String, String, String> query = HFactory.createSliceQuery(keyspaceOperator,
AsciiSerializer.get(), StringSerializer.get(), StringSerializer.get());
query.setColumnFamily("hotel");
query.setKey("26813445");
query.setColumnNames("category");
// [3] perform the request
QueryResult<ColumnSlice<String, String>> result = query.execute();
ColumnSlice<String, String> row = result.get();
String category = row.getColumnByName("category").getValue();
//...
// [4] release the client (cluster)
cluster.getConnectionManager().shutdown();
In the first line of the listing a set of server IP addresses is passed over by creating a Hector Cluster
object. The server address identifies a single Cassandra node. A collection of independent Cassandra
nodes (the Cassandra cluster) represents the Cassandra database. Within this cluster all nodes are
peers. No master node or something like that exists.
The client is free to connect any Cassandra node to perform any request. In the listing above 3
addresses are configured. This does not mean that the Cassandra cluster consist of 3 nodes. It just
defines that the client will communicate with these nodes only.
The connected Cassandra node plays two roles, potentially. In each case the connected node is the
coordinator node which is responsible to handle the dedicated request. Furthermore the connected
node will be a replica store node, if the node is responsible to store a replica of the requested data.
For instance the requested Pavillon Nation hotel record of the example above does not have to be
stored on the connected node. Often the coordinator node has to send sub requests to other replica
nodes to be able to handle the request. As shown in the diagram below the notes 172.39.126.14,
172.39.126.93 and 172.39.126.52 would not able to serve a Pavillon Nation query in a direct way
without sub requesting other nodes.
Gregor Roth Cassandra by example - the path of read and write requests 2
3. Please consider that a coordinator node and a replica node is a role description of a Cassandra node
in context of a dedicated read or write operation. All Cassandra nodes can be a coordinator node as
well as a replica node.
Hector uses a round-robin strategy to select the node to use. By executing the example query Hector
first connects one of the configured nodes. The connect request will be handled on the server-side by
the CassandraServer .
By default the CassandraServer is bound to server port 9160 during the start sequence of a
Cassandra node. The CassandraServer implements Cassandra's Thrift interface which defines remote
procedure methods such as set_keyspace(…) or get_slice(…).This meansCassandra's Thrift interface is
stateful, implicitly. The Hector client has to call the remote method set_keyspace(..) first to assign the
keyspace book_a_hotel to the current connection session. After assigning the keyspace the
get_slice(..) can be called to request the columns of the Pavillon Nation hotel.
However, you are not forced to use Thrift to access Cassandra. Several alternative open-source
connectors such as REST-based connectors exist.
Determining the replica nodes
The CassandraServer is responsible to handle the client-server communication only. Internally, the
CassandraServer calls the local StorageProxy class to process the request. The StorageProxy
implements the coordinator logic. The coordinator logic includes determining the replica notes for
the request row key as well as requesting these replica nodes.
By default a RandomPartitioner is used to determine the replica nodes for the row key of the
request. The RandomPartinitoner spreads the data records (rows) evenly across the Cassandra nodes
which are arranged in a circular ring. Within this ring each node is assigned to a range of hash values
(tokens). To determine the first replica, the MD5 hash of the row key will be calculated and the node
will be selected where the key hash maps with the assigned token range.
Gregor Roth Cassandra by example - the path of read and write requests 3
4. For instance the token of the Pavillon Nation's row key 26813445 is
91851936251452796391746312281860607309. This token is within the token range of node
172.39.126.86 which means that node 172.39.126.86 is responsible to store a replica of the Pavillon
Nation record.
In most case a replica is stored by more than one node which depends on the key space's replication
factor. For instance a replication factor 2 means the clockwise next node of the ring will store the
replica, too. If replication level is 3, the next of the next will also store the replica and so forth.
Processing a read request
The handle a read request the StorageProxy (which is the coordinator of the request) determines the
replica nodes as described above. Additionally, the StorageProxy checks that enough replica nodes
are alive to handle the read request. If this is true, the replica nodes will be sorted by proximity
(closest node first) and the first replica node will be called to get the requested row data.
In contrast to the thrift-based client-server communication the Cassandra nodes interchange data by
using a message-oriented tcp-based protocol. This means the StorageProxy will get the requested
row data by using Cassandra's messaging protocol.
Calling other replica nodes depends on the consistency level. The consistency level is specified by the
client request. If consistency level ONE is required, no further replica nodes will be called. If
consistency level QUORUM is required, in total (replication_factor / 2) + 1 replica nodes
will be called.
In contrast to the first full-data read call all additional calls are digest calls. A digest call queries a
single MD5 hash of all column names, values and timestamps instead requesting the complete row
data. The hashes of all calls, including the first one will be compared together. If a hash does not
match, the replicas will be inconsistent and the out-of-date replicas will be auto-repaired during the
Gregor Roth Cassandra by example - the path of read and write requests 4
5. read process. To do this, a full-data read request will be sent to the additional nodes, the most recent
version of data will be computed and the diff will be sent to out-of-date replicas.
Occasionally all replica nodes for the row key will be called independent of the requested consistency
level. This depends on the column family's read_repair_chance parameter. This configuration
parameter specifies the probability with which read repairs should be invoked. The default value of
0.1 means that a read repair is performed 10%. However, the client response will always be
answered regarding to the requested consistency level. Additional work will be done in background.
A read_repair_chance parameter larger the 0 ensures that frequently read data remains consistent
even though only consistency level ONE is required. The row becomes consistent eventually.
Performing the local data query
As already mentioned above, a dedicated messaging protocol is used for inter-node communication.
Similar to the CassandraServer the MessagingService will be started during the start sequence of a
Cassandra node, too. By default the MessagingService in bound to server port 7000.
The replica node will receive the read call from the coordinator node through the replica node's
MessagingService. However, the MessagingService will not access the local store in a direct way. To
read and write data locally, the ColumnFamilyStore has to be used. Roughly speaking, the
ColumFamilyStore represents the underlying local store of a dedicated column family.
Please consider that a coordinator node can also be in role replica node. This will be true, if the client
calls node 172.39.126.52 to get the Mister bed city row instead of the Pavillon Nation row in the
example above. In this case the StorageProxy of the coordinator node will not call the
Gregor Roth Cassandra by example - the path of read and write requests 5
6. MessagingService of the same node. To avoid remote calls to the same node, the StorageProxy will
call the ColumnFamilyStore in the same way the MessagingServices does to access local data.
By processing a query the ColumnFamilyStore will try to read the requested row data through the
row cache, if the row cache is activated for the column family. The row cache holds the entire row
and is deactivated per default. If the row cache contains the requests row data, no disk IO will be
required. The query will be served very fast by performing in-memory operations only. However, an
activated row cache causes that the full row have to be fetched internally even though a sub set of
columns is requested. For this reasons the row cache is often less efficient for large rows and small
sub set queries.
If the request row isn't cached, the Memtables and the SSTables (sorted strings table) have to be
read. Memtables and SSTables are maintained per column family. SSTables are data files containing
row data fragments and only allow appending data. A Memtable is an in-memory table which buffers
writes. If the Memtable is full, it will be written to disk as a new SSTable file in background. For this
reason the columns of the requested Pavillon Nation row could be fragmented over several SSTables
and unflushed Memtables. For instance one SSTable book_a_hotel-hotel-he-1-Data.db could contain
the initial inserted columns ‘name’= ‘Pavillon Nation’ and ‘category’=’4’ of the Pavillon Nation row.
Another SSTable book_a_hotel-hotel-he-2-Data.db (or Memtable) could contain the updated
category column ‘category’=’5’.
If an SSTable exists for the requested column family, first the associated (key-scoped) Bloom filter of
the SSTable file will be read to avoid unnecessary disk IO. For each SSTable the ColumnFamilyStore
holds an in-memory structure called SSTableReader which contains metadata as wells as the Bloom
filter of the underlying SSTable file. The Bloom filter indicates that the dedicated SSTable could
contain a row data fragment (false positive are possible, false negative not). If this is true, the key
cache will be requested to get the seek position. If not found, the on-disk index will have to be
scanned. The fetched seek position will be added to the key cache in this case. Based on the seek
position the row data fragment will be read from the SSTable file. The data fragments of the SSTables
and Memtables will be merged together by using the column timestamp and the requested row data
will be returned to the caller.
Gregor Roth Cassandra by example - the path of read and write requests 6
7. Processing an write request
To insert, update or delete a row Cassandra's mutate method has to be called. The listing below
shows such a mutate call by using the Hector client.
//...
// [1.b] create and perform an update
Mutator<String> mutator = HFactory.createMutator(keyspaceOperator, AsciiSerializer.get());
mutator.addInsertion("26813445", "hotel",
HFactory.createColumn("category", "5", StringSerializer.get(), StringSerializer.get()));
MutationResult result = mutator.execute();
//...
The write path is very the same to the read path. Similar to the read request a write request also
includes the required consistency level. However, the coordinator node tries to send a write request
including the mutated columns to all replica nodes for the row key.
First, the StorageProxy of the coordinator node checks if enough replica notes for the row key are
alive regarding to the requested consistency level. If this is true, the write request will be sent to the
living replica nodes. If not, an error response will be returned. Write requests to temporarily failed
replica nodes will be scheduled as a hinted handoff. This means that a hint will be written locally
instead calling the failed node. Once the failed replica node is back the hint will be sent to this node
to perform the write operation. By sending the hints the failed nodes becomes consistent to the
other nodes. Please consider that hints will not longer store locally, if the failed node is dead longer
than 1 hour (config param max_hint_window_in_ms).
The coordinator node returns the response to the client as soon as the replica nodes conforming to
the consistency level have confirmed the update (a hinted write will not count towards the
requested consistency level). The updates of the other replica nodes will still be executed in
background. If an error occurs by updating the replica nodes conforming to the consistency level, an
error response will be returned. However, in this case the already updated nodes will not be
reverted. Cassandra does not support distributed transactions, and hence it does not support a
distributed rollback.
The write operation supports an additional consistency level ANY which means that the mutated
columns have to be written to at least one node regardless of whether this node is a replica node for
the key or not. In contrast to consistency level ONE the write will also succeed, if a hinted handoff is
written (by the coordinator node). However, in this case the mutated columns will not be readable
until the responsible replica nodes have recovered.
Gregor Roth Cassandra by example - the path of read and write requests 7
8. Performing the local update
Similar to the local data query a local update is triggered by handling a message through the
MessagingService or by the StorageProxy. However, in contrast to the read path, first a commit log
entry will be written for durability reasons. By default the commit log entry will be written in
background asynchronously.
The mutated columns will also be written into the in-memory Memtable of the column family. After
inserting the changes the local update is completed.
However, the memory size of a Memtable is limited. If the max size is exceeded, the Memtable will
be written to disk as a new SSTable. This is done by a background thread which checks the current
size of all unflushed Memtables of all ColumnFamilies, periodically. If a Memtable exceeds the max
size, the background thread replaces the current Memtable by a new one. The old Memtable will be
marked as pending flush and will be flushed by another thread. Under certain circumstances several
pending Memtables for a column family could exists. After writing the Memtable to disk a new
SSTableReader referring the written SSTable is created and added to the ColumnFamilyStore. Once
written, the SSTable file is immutable. By default the SSTable data will be compressed
(SnappyCompression).
Compacting
The SSTable file includes the modified columns of the row including their timestamps as well as
additional row meta data. For instance the meta data section includes a (column name-scoped)
Bloom Filter which is used to reduce disk IO by fetching columns by name.
To reduce fragmentation and save space, SSTable files will be merged into a new SSTable file,
occasionally. This compaction will be triggered by a background thread, if the compaction threshold
is exceeded. The compaction threshold can be set for each column family.
Gregor Roth Cassandra by example - the path of read and write requests 8
9. About the author
Gregor Roth works as a software architect at United Internet group, a leading European Internet
Service Provider to which GMX, 1&1, and Web.de belong. His areas of interest include software and
system architecture, enterprise architecture management, distributed computing, and development
methodologies.
Gregor Roth Cassandra by example - the path of read and write requests 9