1000 node Cassandra cluster on Amazon's EKS?

•

0 likes•76 views

Come here about our experience scaling Cassandra on EKS to over 1000 nodes and 20 million transactions per second. This session will cover the lessons learned, successes, failures, and tools used to get there. This talk was given by Matt Overstreet for DoK Day Europe @ KubeCon 2022.

Technology

1,000 Cassandra nodes on
Kubernetes?
Matt Overstreet
DoK Day Europe 2022 @ KubeCon 1

Who am I?
Matt Overstreet
Field CTO, Cloud
DataStax
"Our Mission is to Connect every
developer in the world to the power of
Apache Cassandra, with the freedom to
run their data on any device and in any
cloud."
DoK Day Europe 2022 @ KubeCon 2

What did we do?
Built a 1,200 node K8ssandra cluster on Kubernetes
And used Kubernetes jobs to bench test it with NoSQLBench
DoK Day Europe 2022 @ KubeCon 3

A what now?
Kubernetes - You know this one, and if you are watching today
you know about the history of using it for stateful stuff like
databases.
Apache Cassandra - An open source database that runs some
of the worlds biggest "fast data", and geo-distributed
workloads.
K8ssandra - A way to run Cassandra on Kubernetes.
NoSQLBench - A reliable way to benchmark servers.
DoK Day Europe 2022 @ KubeCon 4

Why do we care?
The first rule of distributed system
troubleshooting: Compare the configs
Config drift between machines in
distributed software causes weird
issues.
! "
DoK Day Europe 2022 @ KubeCon 5

What did K8ssandra bring to the party?
• Configuration consistency for the cluster
• Ease of Ops - 1,000 nodes would have taken weeks if
hand configured (Shout out to Amazon EKS )
Setup for other Kubernetes goodness!
• Consolidated logging
• Full stack visibility with a service mesh
DoK Day Europe 2022 @ KubeCon 6

What worked?
DoK Day Europe 2022 @ KubeCon 7

What worked?
• 1,200+ nodes/1 engineer/1 week
• Petabyte+ of data/tens of millions of reads and writes per
second/p75 latencies under 10ms
• Destroyed 800 nodes in 5 minutes using the combined
superpowers of helm update and Amazon Elastic
Kubernetes Service
! " #
• Scaling up the load test through Kubernetes was super easy
DoK Day Europe 2022 @ KubeCon 8

What didn’t?
DoK Day Europe 2022 @ KubeCon 9

What didn’t?
Gotchas with K8ssandra
• K8ssandra (v1) monitoring wasn’t up to the task, tweaks are
required for large clusters
• K8ssandra needs a little love when deciding how to
coordinate new Cassandra node "racks" with their
K8ssandra pod startup order
DoK Day Europe 2022 @ KubeCon 10

What didn’t?
Gotchas with EKS (Hyperscalers in
general)
• Resource limits on AWS EKS were often a surprise
• If you are using cloud hosted Kubernetes, think twice before
using an autoscaler
!
• Know before you grow... IP addresses can run out quickly
for large Kubernetes clusters with single or limited pod
DoK Day Europe 2022 @ KubeCon 11

What did I miss?
Thanks!
Me matt.overstreet@datastax.com | @omnifroodle - twitter
K8ssandra https://K8ssandra.io | @k8ssandra - twitter
DataStax https://www.datastax.com
DoK Day Europe 2022 @ KubeCon 12

Similar to 1000 node Cassandra cluster on Amazon's EKS?

Day 2 Kubernetes - Tools for Operability (QConSF)

bridgetkromhout

Zenko & MetalK8s @ Dublin Docker Meetup, June 2018

Laure Vergeron

As a data professional, you are the glue that makes cross-platform integrations possible. With the increase in adoption of hybrid cloud architectures, Kafka is an increasingly relevant tool for building data pipelines between platforms and accelerating delivery on cloud projects. Early exposure to Kafka on Azure capabilities gives you an edge to build better mousetraps at the design phase. Customers already running Kafka on premises and are looking to extend Kafka systems to Azure can get started quickly with Confluent Cloud. Additionally, DevOps for self-managed options can be easily scalable with Ansible for Virtual Machines or containers via Azure Kubernetes Services or Azure Container Instances. This session is presented from the Microsoft Solution Architect perspective by Israel Ekpo, Microsoft Cloud Solution Architect and Alicia Moniz, Microsoft MVP. They will cover use cases and scenarios, along with key Azure integration points and architecture patterns.

Confluent On Azure: Why you should add Confluent to your Azure toolkit | Alic...

HostedbyConfluent

Multiple clusters exist in most Kubernetes environments today, and number of clusters will increase overtime. The reasons for having multiple Kubernetes clusters are many, for example, overcoming scale limits, reducing complexity, geo separation, redundancy and having separate production, staging, and development environments. Once you have multiple K8S clusters, it can be useful to have the ability to easily move or duplicate workloads across these different clusters. Kubernetes does not have a native method to allow migration or duplication of workloads across clusters. Fortunately, there are tools that provide this functionality. In this presentation we will explore the different uses cases for cross cluster migration, and what is involved, and how these migration tools work. We'll cover some popular uses cases, such as, Disaster Recovery, Test/Dev, and performance testing. Migration could entail moving the entire cluster, or individual workloads. The components that need to be moved would include configuration and resources stored in etcd, and persistent data residing on PVCs. We'll cover the uses cases and challenges for migration, and run through an example of using one of these migration tools. This talk was given by Ryan Kaw for DoK Day Europe @ KubeCon 2022.

The many uses of Kubernetes cross cluster migration of persistent data

DoKC

Link: https://youtu.be/J3JiwW5FIAI https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) Multiple clusters exist in most Kubernetes environments today, and number of clusters will increase overtime. The reasons for having multiple Kubernetes clusters are many, for example, overcoming scale limits, reducing complexity, geo separation, redundancy and having separate production, staging, and development environments. Once you have multiple K8S clusters, it can be useful to have the ability to easily move or duplicate workloads across these different clusters. Kubernetes does not have a native method to allow migration or duplication of workloads across clusters. Fortunately, there are tools that provide this functionality. In this presentation we will explore the different uses cases for cross cluster migration, and what is involved, and how these migration tools work. We'll cover some popular uses cases, such as, Disaster Recovery, Test/Dev, and performance testing. Migration could entail moving the entire cluster, or individual workloads. The components that need to be moved would include configuration and resources stored in etcd, and persistent data residing on PVCs. We'll cover the uses cases and challenges for migration, and run through an example of using one of these migration tools.

The many uses of Kubernetes cross cluster migration of persistent data

DoKC

Kubernetes 101

Crevise Technologies

On CloudStack, Docker, Kubernetes, and Big Data…Oh my ! By Sebastien Goasguen...

Radhika Puthiyetath

[WSO2Con Asia 2018] Deploying Applications in K8S and Docker

WSO2

Day 2 Kubernetes - Tools for Operability (Velocity London Meetup)

bridgetkromhout

Docker is an open platform for developers and system administrators to build, ship and run distributed applications. Using Docker, companies in Jordan have been able to build powerful system architectures that allow speeding up delivery, easing deployment processes and at the same time cutting major hosting costs. Osama Jaber shares his experience at ArabiaWeather in how they moved away from AWS to a highly-redundant, high-performance and low-cost solution using docker and other open-source technologies.

JOSA TechTalks - Downgrade your Costs

Jordan Open Source Association

In this talk, we will leverage all cloud native stacks and tools to build Camel Quarkus routes natively using GraalVM native-image on Tekton pipeline and deploy these routes to Kubernetes cluster with Knative installed. We will dive into the following topics in the talk: - Introduction to Camel - Introduction to Camel Quarkus - Introduction to GraalVM Native Image - Introduction to Tekon - Introduction to Knative - Demo shows how to deploy end to end a Camel Quarkus route which include the following steps: - Look at whole deployment pipeline for Cloud Native Camel Quarkus routes - Build Camel Quarkus routes with GraalVM native-image on Tekton pipeline. - Deploy Camel Quarkus routes to Kubernetes cluster with Knative Targeted Audience: Users with basic Camel knowledge

Build and Deploy Cloud Native Camel Quarkus routes with Tekton and Knative

Omar Al-Safi

Unleashing k8 s to reduce complexities of an entire middleware platform

Lakmal Warusawithana

If you are considering a lift-and-shift from on-prem to public cloud, this talk is for you. Our team runs a centralized build farm for Nokia's software division. The build farm consists of a fleet of Jenkins Masters, a Kubernetes cluster, artifact storage, and various back-office services for monitoring and automation. This talk gives an overview of how we migrated our build farm from an on-prem OpenStack-based datacenter to AWS. It was a successful migration, but we made several mistakes along the way and would like to share what we learned. Stack: Ansible, AWX, Terraform, Jenkins, Artifactory, Prometheus, Grafana, Elasticsearch, Kubernetes, AWS EKS https://cfp.cfgmgmtcamp.be/2020/talk/EGJCKT/

Migrating a build farm from on-prem to AWS

Claes Buckwalter

Building and deploying an analytic service on Cloud is a challenge. A bigger challenge is to maintain the service. In a world where users are gravitating towards a model where cluster instances are to be provisioned on the fly, in order for these to be used for analytics or other purposes, and then to have these cluster instances shut down when the jobs get done, the relevance of containers and container orchestration is more important than ever. Container orchestrators like Kubernetes can be used to deploy and distribute modules quickly, easily, and reliably. The intent of this talk is to share the experience of building such a service and deploying it on a Kubernetes cluster. In this talk, we will discuss all the requirements which an enterprise grade Hadoop/Spark cluster running on containers bring in for a container orchestrator. This talk will cover in details how Kubernetes orchestrator can be used to meet all our needs of resource management, scheduling, networking, and network isolation, volume management, etc. We will discuss how we have replaced our home grown container orchestrator with Kubernetes which used to manage the container lifecycle and manage resources in accordance to our requirements. We will also discuss the feature list as container orchestrator which is helping us deploy and patch 1000s of containers and also a list which we believe need improvement or can be enhanced in a container orchestrator. Speaker Rachit Arora, SSE, IBM

Why Kubernetes as a container orchestrator is a right choice for running spar...

DataWorks Summit

Supporting Hadoop in containers takes much more than the very primitive support Docker provides using the Storage Plugin. A production scale Hadoop deployment inside containers needs to honor anti/affinity, fault-domain and data-locality policies. Kubernetes alone, with primitives such as StatefulSets and PersitentVolumeClaims, is not sufficient to support a complex data-heavy application such as Hadoop. One needs to think about this problem more holistically across containers, networking and storage stacks. Also, constructs around deployment, scaling, upgrade etc in traditional orchestration platforms is designed for applications that have adopted a microservices philosophy, which doesn't fit most Big Data applications across the ingest, store, process, serve and visualization stages of the pipeline. Come to this technical session to learn how to run and manage lifecycle of containerized Hadoop and other applications in the data analytics pipeline efficiently and effectively, far and beyond simple container orchestration. #BigData, #NoSQL, #Hortonworks, #Cloudera, #Kafka, #Tensorflow, #Cassandra, #MongoDB, #Kudu, #Hive, #HBase, PARTHA SEETALA, CTO, Robin Systems.

Containerized Hadoop beyond Kubernetes

DataWorks Summit

Run Cloud Native MySQL NDB Cluster in Kubernetes

Bernd Ocklin

As the demand for real-time data processing continues to grow, so too do the challenges associated with building production-ready applications that can handle large volumes of data and handle it quickly. In this talk, we will explore common problems faced when building real-time applications at scale, with a focus on a specific use case: detecting and responding to cyclist crashes. Using telemetry data collected from a fitness app, we’ll demonstrate how we used a combination of Apache Kafka and Python-based microservices running on Kubernetes to build a pipeline for processing and analyzing this data in real-time. We'll also discuss how we used machine learning techniques to build a model for detecting collisions and how we implemented notifications to alert family members of a crash. Our ultimate goal is to help you navigate the challenges that come with building data-intensive, real-time applications that use ML models. By showcasing a real-world example, we aim to provide practical solutions and insights that you can apply to your own projects. Key takeaways: An understanding of the common challenges faced when building real-time applications at scale Strategies for using Apache Kafka and Python-based microservices to process and analyze data in real-time Tips for implementing machine learning models in a real-time application Best practices for responding to and handling critical events in a real-time application

Data Engineer's Lunch #86: Building Real-Time Applications at Scale: A Case S...

Anant Corporation

Metal-k8s presentation by Julien Girardin @ Paris Kubernetes Meetup

Laure Vergeron

NoSQL databases like MongoDB, Elasticsearch, and Cassandra are synonymous with scalability, search, and developer agility. But there’s a downside...having to give up the ease and comfort of SQL. Or do you? Join this webcast to learn how the newest databases, like CrateDB and CockroachDB deliver the benefits of NoSQL with the ease of SQL by building SQL engines on top of custom NoSQL technology stacks. Database industry veteran Andy Ellicott, who helped launch Vertica, VoltDB, Cloudant, and now with Crate.io, will provide a no-BS view of current DBMS architectures and predictions for the future of data. If you’re a DBMS user, this webcast will help you make sense of a very crowded DBMS market and make better-informed decisions for your new tech stacks.

Webinar: The Future of SQL

Crate.io

25 12 18 meetup - road to k8s

Daniel Borovsky

Similar to 1000 node Cassandra cluster on Amazon's EKS? (20)

Day 2 Kubernetes - Tools for Operability (QConSF)

Zenko & MetalK8s @ Dublin Docker Meetup, June 2018

Confluent On Azure: Why you should add Confluent to your Azure toolkit | Alic...

The many uses of Kubernetes cross cluster migration of persistent data

Kubernetes 101

On CloudStack, Docker, Kubernetes, and Big Data…Oh my ! By Sebastien Goasguen...

[WSO2Con Asia 2018] Deploying Applications in K8S and Docker

Day 2 Kubernetes - Tools for Operability (Velocity London Meetup)

JOSA TechTalks - Downgrade your Costs

Build and Deploy Cloud Native Camel Quarkus routes with Tekton and Knative

Unleashing k8 s to reduce complexities of an entire middleware platform

Migrating a build farm from on-prem to AWS

Why Kubernetes as a container orchestrator is a right choice for running spar...

Containerized Hadoop beyond Kubernetes

Run Cloud Native MySQL NDB Cluster in Kubernetes

Data Engineer's Lunch #86: Building Real-Time Applications at Scale: A Case S...

Metal-k8s presentation by Julien Girardin @ Paris Kubernetes Meetup

Webinar: The Future of SQL

25 12 18 meetup - road to k8s

Distributed Vector Databases - What, Why, and How

DoKC

Is It Safe? Security Hardening for Databases Using Kubernetes Operators - Robert Hodges, Altinity Thanks to the Operator Pattern, Kubernetes is now an outstanding platform to run databases. But to quote Marathon Man, "is it safe?" This talk is a top-level review of the database security problem in Kubernetes, standard ways that operators can mitigate threats, and a wallet-sized checklist of security features you should look for in any operator you use. Our talk is practical and focused on needs of Kubernetes developers. Join us!

Is It Safe? Security Hardening for Databases Using Kubernetes Operators

DoKC

Stop Worrying and Keep Querying, Using Automated Multi-Region Disaster Recovery - Shivani Gupta, Elotl & Sergey Pronin, Percona Disaster Recovery(DR) is critical for business continuity in the face of widespread outages taking down entire data centers or cloud provider regions. DR relies on deployment to multiple locations, data replication, monitoring for failure and failover. The process is typically manual involving several moving parts, and, even in the best case, involves some downtime for end-users. A multi-cluster K8s control plane presents the opportunity to automate the DR setup as well as the failure detection and failover. Such automation can dramatically reduce RTO and improve availability for end-users. This talk (and demo) describes one such setup using the open source Percona Operator for PostgreSQL and a multi-cluster K8s orchestrator. The orchestrator will use policy driven placement to replicate the entire workload on multiple clusters (in different regions), detect failure using pluggable logic, and do failover processing by promoting the standby as well as redirecting application traffic

Stop Worrying and Keep Querying, Using Automated Multi-Region Disaster Recovery

DoKC

Transforming Data Processing with Kubernetes: Journey Towards a Self-Serve Data Mesh - Rakesh Subramanian Suresh & Jainik Vora, Intuit This presentation explores how Intuit uses Kubernetes with Domain-Driven Design and Data Mesh principles to transform its data processing landscape, crucial for its AI-driven expert platform. We will discuss the importance of clean data in developing robust generative artificial intelligence and how Intuit is addressing this through the creation of paved paths for data platforms running on Kubernetes. We'll examine the challenges and solutions in managing 100,000 data pipelines and 1000+ engineers interacting with data, highlighting the need for scalable solutions. We'll also discuss how Intuit uses Kubernetes to build its batch and stream processing platform, overcoming hurdles in data pipeline deployment, scheduling, orchestration, and dependency management. We'll conclude by emphasizing how this transformation, based on treating data as a product, has improved decision-making speed and accuracy across the organization and fostered a more efficient, collaborative data culture.

Transforming Data Processing with Kubernetes: Journey Towards a Self-Serve Da...

DoKC

The State of Stateful on Kubernetes - Stateful Workloads in Kubernetes: A Deep Dive - Kaslin Fields & Michelle Au, Google As a platform for distributed computing, Kubernetes enables users to run their workloads across machines. However data has gravity, and when workloads in Kubernetes have to share data with other applications, managing the application’s requirements can get more tricky. In this talk, we will explore what "Stateful" means from Kubernetes' perspective. We will discuss the different types of stateful workloads, and the challenges of deploying them on Kubernetes. We will also look at the features that exist in Kubernetes to support stateful workloads, as well as the features that are in the works. Key Takeaways: What is a stateful workload from Kubernetes’ perspective? What are the challenges of deploying stateful workloads on Kubernetes? What features exist in Kubernetes to support stateful workloads? What features are in the works to support stateful workloads better in the future?

The State of Stateful on Kubernetes

DoKC

Colocating Data Workloads and Web Services on Kubernetes to Improve Resource Utilization - He Cao, ByteDance Recently, more and more data workloads are running on top of Kubernetes, such as ETL processes, Spark and Flink jobs, and more. These workloads typically exhibit high resource utilization and remain relatively stable over time. In contrast, web services often exhibit tidal patterns, characterized by significant fluctuations in resource utilization. The resource model of vanilla Kubernetes is static, which can lead to low resource utilization accumulated over 24 hours. In this talk, He will introduce how ByteDance uses Katalyst to colocate data workloads and online services on Kubernetes to improve resource utilization. In addition, He will explain how Katalyst ensures the QoS of these workloads through QoS-aware scheduling, service profiling, multi-dimensional resource isolation, real-time container resource adjustment, and more. In ByteDance, Katalyst has been deployed on 500,000+ nodes with tens of millions of cores, and has improved daily resource utilization from 20% to 60%.

Colocating Data Workloads and Web Services on Kubernetes to Improve Resource ...

DoKC

Make Your Kafka Cluster Production-Ready - Jakub Scholz, Red Hat Kubernetes became the de-facto standard for running cloud-native applications. And more and more users turn to it also to run stateful applications such as Apache Kafka. While there are different tools such as Helm charts or operators which can get you quickly up and running, there is often still a long way to make sure the Kafka cluster is production-ready. This talk will take you through the main aspects you should consider for your Kafka cluster and will cover things such as resource management, storage, scheduling, rolling updates, or reliability. It will show you how to do it using the Strimzi operator, but the lessons learned will apply also to any other Kafka cluster. If you are interested in production-ready Apache Kafka on Kubernetes, this is a talk for you.

Make Your Kafka Cluster Production-Ready

DoKC

Dynamic Large Scale Spark on Kubernetes: Empowering the Community with Argo Workflows and Argo Events - Ovidiu Valeanu, AWS & Vara Bonthu, Amazon Are you eager to build and manage large-scale Spark clusters on Kubernetes for powerful data processing? Whether you are starting from scratch or considering migrating Spark workloads from existing Hadoop clusters to Kubernetes, the challenges of configuring storage, compute, networking, and optimizing job scheduling can be daunting. Join us as we unveil the best practices to construct a scalable Spark clusters on Kubernetes, with a special emphasis on leveraging Argo Workflows and Argo Events. In this talk, we will guide you through the journey of building highly scalable Spark clusters on Kubernetes, using the most popular open-source tools. We will showcase how to harness the potential of Argo Workflows and Argo Events for event-driven job scheduling, enabling efficient resource utilization and seamless scalability. By integrating these powerful tools, you will gain better control and flexibility for executing Spark jobs on Kubernetes.

Dynamic Large Scale Spark on Kubernetes: Empowering the Community with Argo W...

DoKC

Run PostgreSQL in Warp Speed Using NVMe/TCP in the Cloud - Sagy Volkov, Lightbits PostgreSQL as a SQL engine can accommodate a very high-transaction rate, but as your data grows and the number of connections and queries increases, there is a challenge for the storage to keep up with the SQL engine. To the rescue comes NVMe over TCP (or NVMe/TCP). Developed by Lightbits Labs in 2016 and donated to the Linux community, it is the next evaluation of using NVMe based storage over TCP Fabric. NVMe/TCP simplifies how you interact with remote NVMe devices (targets) and allows your PostgreSQL storage to consume fast storage very easily. In this session I will explain the core concept of the NVMe/TCP protocol, current storage providers that can use it, how you can consume it in Kubernetes (super easy), and discuss the possibilities of using NVMe/TCP in the cloud. The session will also include a performance comparison of a few storage that are available in AWS and even a live demo of how PostgreSQL can run super fast - warp speed fast - in AWS.

Run PostgreSQL in Warp Speed Using NVMe/TCP in the Cloud

DoKC

An explanation of how ING deals with local persistence at scale in secure and compliant manner for Elastic and Prometheus workloads today and other Data Services in the future. In more detail we will elaborate on the following topics How we solve local persistence Type of workloads now and in the future Typical requirements for a banking environment Automation Scale Resilience Security / Compliance Service offering / demarcation About Tor and Luuk: Tor and Luuk are experienced engineers working at ING for over 10 years and working in the Kubernetes area for the last 5 years. They are specialized in and responsible for the Data Services OpenShift clusters in ING and have a strong focus on resilience, automation and security.

ING Data Services hosted on ICHP DoK Amsterdam 2023

DoKC

A small walkthrough of projects within the dutch government running Data(bases) on OpenShift. This talk shares success stories, provides a proven recipe to `get it done` and debunks some of the FUD. About Sebastiaan: I have always been a weird DBA, trying to combine Databases with out-of-the-box thinking and a DevOps mindset. Around 2016 I fell in love with both Postgres and Kubernetes, and I then committed my life to enabling Dutch organisations with running their Database workloads CloudNative. Over the last few years I worked as a private contractor for 2 large government agencies doing exactly that, and I want to share my and others (success stories) hoping to enable and inspire Data on Kubernetes adoption.

Implementing data and databases on K8s within the Dutch government

DoKC

https://go.dok.community/slack https://dok.community/ Link: https://youtu.be/n_thXwyJNSU ABSTRACT OF THE TALK Deploying Stateless applications is easy but this is not the case for Stateful applications. StatefulSets are the K8s API object that helps to manage stateful application. Learn about what Stateful sets are, how to create, How it differs from Deployments. KEY TAKE-AWAYS FROM THE TALK This talk is focused on basics of StatefulSet, how StatefulSet differs from Deployments, How to manage Stateful app using StatefulSet

StatefulSets in K8s - DoK Talks #154

DoKC

Link: https://youtu.be/cegd3Exg05w https://go.dok.community/slack https://dok.community/ Gabriele Bartolini - Vice President/CTO of Cloud Native and Kubernetes, EDB ABSTRACT OF THE TALK Imagine this: you have a virtual infrastructure based on Kubernetes, made up of virtual data centers, possibly spread across multiple Kubernetes clusters and regions. Your infrastructure could even be hosted on premises or on different cloud service providers. Infrastructure as Code is a requirement. You’ve been tasked to run Postgres databases, alongside your applications. The good news is that you can leverage a fully open source stack with Kubernetes, PostgreSQL and the CloudNativePG operator, and deploy your Postgres database in the same way you deploy applications. Join me in this webinar to discover the key role that you have to make this succeed, starting from day 0 through day 2 operations. I’ll share some examples and best practices for running Postgres databases in Kubernetes, before peeking at the new features we are developing for the months to come.

Running PostgreSQL in Kubernetes: from day 0 to day 2 with CloudNativePG - Do...

DoKC

Link: https://youtu.be/Y-1uFVKDfgY https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK This talk concerns performing analytical tasks with Apache Superset with ClickHouse as the data backend. ClickHouse is a super fast database for analytical tasks, and Apache Superset is an Apache Software foundation project meant for data visualization and exploration. Performing analytical tasks using this combo is super fast since both the software are designed to be scalable and capable of handling data of petabyte scale.

Analytics with Apache Superset and ClickHouse - DoK Talks #151

DoKC

Link: https://youtu.be/EFaRyl4HmmE https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK If you are running or planning a multi-cloud or even a multi-cluster environment, there are several considerations in implementing a data protection solution – especially if you plan on an organic home-grown, do-it-yourself option. This talk will highlight challenges and best practices around centralized management of configuration, credentials, compliance across multiple accounts, regions, providers etc. We will also highlight the deviations in CSI driver implementations of various storage vendors and cloud providers. Finally, we will cover the various recovery options available in the market today. Kubernetes cloud services are popular since they mitigate, but do not eliminate, the difficulties of operating a Kubernetes environment. This is especially true for protecting the stateful configuration and data of your Kubernetes applications, where the inherent high-availability and infrastructure as code are not a substitute for have cloud-native backup and disaster recovery capabilities. Further, many companies now have multi-cloud strategies for their cloud-native applications. These challenges can be addressed with backup applications that are both Kubernetes managed service and multi-cloud aware in order to snapshot, copy, restore, and migrate Kubernetes workloads (resources and data) running on AKS, EKS and GKE. Capturing information from cloud accounts and how the cluster and storage resources are configured allows 1) centralized visibility into all cloud accounts and the clusters and resources in the accounts including for compliance; 2) cross-account, cross-cluster, and cross-region data restores; 3) automation of the cluster and data restores including for Dev, Test, and Production recovery use cases. BIO Sebastian Glab is a Cloud Architect for CloudCasa and he resides in Poland. He is responsible for integrating the different cloud providers with the CloudCasa service, and making sure that all clusters in the cloud service get discovered and protected. In his free time, he plays volleyball and develops his own projects. Martin Phan is the Field CTO in North America for CloudCasa by Catalogic Software. With over 20+ years of experience in the software-industry, he takes pride in supporting, developing, implementing, and selling enterprise software and data protection solutions to help customer solve their backup and recovery challenges. KEY TAKE-AWAYS FROM THE TALK 1) Challenges and best practices around centralized management of configuration, credentials, compliance across multiple accounts, regions, providers etc. 2) Advantages of cloud awareness and Kubernetes managed service awareness for application and data recovery and security 3) Examples of overcoming Container Storage Interface (CSI) deviations 4) Various recovery options available in the market today.

Overcoming challenges with protecting and migrating data in multi-cloud K8s e...

DoKC

Link: https://youtu.be/YVXEpcSclwY https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK In a continuation of a talk given at DoK day at KubeCon EU 2022, join Dinesh Majrekar, Civo's CTO as they walk through their evaluation process of the CNCF Storage market. Civo offers managed Kubernetes clusters powered by K3s to customers around the world. We manage thousands of Virtual Machines and stateful customer data within multiple data centres across several continents. In late 2021, Civo had the opportunity to evaluate the CNCF storage landscape to move to a new technology stack. During the migration project, Civo evaluated Mayastor, Ondat, Ceph and Longhorn against the following metrics: Scalability Performance Ease of Support Attendants will see practical examples on how they could carry out their own similar evaluation and see some of the results of the Civo research project. BIO Dinesh is CTO at Civo. Having worked in the hosting industry for many years, Dinesh has a passion for creating solutions that operate at scale. This not only applies to the technology stack, but for nurturing engineers through their career.

Evaluating Cloud Native Storage Vendors - DoK Talks #147

DoKC

Link: https://youtu.be/qUW8LkxYayc https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK How do you make sure your Stateful Workloads remain available when your Kubernetes infrastructure updates? This talk will discuss different strategies of upgrading a Kubernetes cluster, and how you can manage risk for your workload. The talk will showcase demos of each upgrade strategy. BIO Peter is a Senior Software Engineer on GKE at Google. He works on improving Kubernetes for Stateful workloads. His main focus is on enhancing the Kubernetes ecosystem for high availability applications. KEY TAKE-AWAYS FROM THE TALK The mechanics of different upgrade strategies, when to apply a particular upgrade strategy depending on your Stateful workload and how to mitigate risk to your application’s availability.

Kubernetes Cluster Upgrade Strategies and Data: Best Practices for your State...

DoKC

Link: https://youtu.be/AjvwG53yLMY https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK Stateful workloads are the heart of any application, yet they remain confusing and complicated even to daily K8s practitioners. That’s why many organizations shy away from migrating their data - their prized possession - to the unfamiliar stateful realm of Kubernetes. After meeting with many organizations in the adoption phase, I discovered what works best, what to avoid, and how critical it is to gain confidence and the right knowledge in order to successfully adopt stateful workloads. In this talk I will demonstrate how to optimally adopt Kubernetes and stateful workloads in a few steps, based on what I’ve learned from observing dozens of different adoption journeys. If you are taking your first steps in data on K8s or contemplating where to start - this talk is for you! BIO - A Developer turned Solution Architect. - Working at Komodor, a startup building the first K8s-native troubleshooting platform. - Love everything in infrastructure: storage, networks & security - from 70’s era mainframes to cloud-native. - All about “plan well, sleep well”. KEY TAKE-AWAYS FROM THE TALK - Understand how critical stateful workloads are for any system, and that the key challenges to migrating it to Kubernetes are knowledge and confidence. - How to build the foundational knowledge required to overcome adoption challenges by creating a learning path for individuals and teams. - How to gain confidence to run stateful workloads on Kubernetes with support from the community (and yourself!)

We will Dok You! - The journey to adopt stateful workloads on k8s

DoKC

Link: https://youtu.be/Pi5ueyl_1jU https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK During my first talk for DoK community I want to walk you through the world of NoSQL database MongoDB and Kubernetes Operators - Community Edition, Enterprise Edition (MongoDB and Ops Manager on K8s), and Atlas operator, highlight the most important capabilities, talk about use cases and challenges, the theory will be mixed with a live demos! BIO I'm a SRE / NoSQL / DevOps professional. I hold CKA, CKAD, CKS, also I’m MongoDB Certified DBA and MongoDB Champion. I have experience with multiple cloud providers, Kubernetes, different types of K8s operators (Strimzi, RabbitMQ Cluster Operator), but especially MongoDB K8s Operator. I also work with KEDA. Since 2017, I have been a speaker at MongoDB conferences all around the world (USA, China, Europe). KEY TAKE-AWAYS FROM THE TALK I would like to share the best practices of running NoSQL database - MongoDB on Kubernetes also I want to show how to manage Atlas (MongoDB cloud) via K8s operator https://www.mongodb.com/developer/community-champions/arkadiusz-borucki/

Mastering MongoDB on Kubernetes, the power of operators

DoKC

Link: https://youtu.be/KUipuM3UJF4 https://go.dok.community/slack https://dok.community/ DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) Kubernetes comes with a lot of useful features like Volumes and StatefulSets, which make running stateful workloads simple. Interestingly, when combined with the right tools, these features can make Kubernetes very valuable for developers wanting to run massive production databases in development! This is exactly what was seen at "Extendi". The developers at Extendi deal with a large amount of data in their production Kubernetes clusters. But when developing locally, they didn't have an easy way of replicating this data. This replication was needed because it allowed developers to test new features instantaneously without worrying if they would work as expected when pushed to production. But replicating a 100Gb+ production database for development wasn't turning out to be an easy task! This is where leveraging Kubernetes + remote development environments came to the rescue. Running data on Kubernetes turned out to be way faster than any of the traditional approaches because of Kubernetes' ability to handle stateful workloads exceptionally well. And since Extendi already used Kubernetes in production - the setup process was fairly simple. This talk will cover practical steps on how leveraging Kubernetes based development environments allowed dev teams at Extendi to run production data on Kubernetes during development using features like Volume Snapshots, having a huge positive impact on developer productivity. ----- Arsh is a Developer Experience Engineer at Okteto. He is an active contributor to the upstream Kubernetes project and was awarded the Kubernetes Contributor Award for his contributions in 2021. Arsh has written blogs and spoken about different topics in the cloud-native ecosystem at various conferences before, including KubeCon + CloudNativeCon + Open Source Summit China 2021. He has also been on the Kubernetes Release Team since the 1.23 release. He also serves as the New Contributor Ambassador for the Documentation Special Interest Group of the Kubernetes project and continuously mentors new folks in the community. Previously, he worked at VMware and was an active contributor to other CNCF projects, including cert-manager and Kyverno. ----- Lapo is a Software Engineer currently leading the development team of a Social Listening and Audience Intelligence platform. He started coding at the early age of 14 and since he turned his passion into a real job, he has always been looking for boosting his knowledge by constantly researching for newer and newer technologies. Active on Ruby Open Source projects ----- Ramiro Berrelleza is one of the founders of Okteto. He has spent most of his career (and his free time) building cloud services and developer tools. Before starting Okteto, Ramiro was an Architect at Atlassian and a Software Engineer at Microsoft Azure.

Leveraging Running Stateful Workloads on Kubernetes for the Benefit of Develo...

DoKC

More from DoKC (20)

Distributed Vector Databases - What, Why, and How

Is It Safe? Security Hardening for Databases Using Kubernetes Operators

Stop Worrying and Keep Querying, Using Automated Multi-Region Disaster Recovery

Transforming Data Processing with Kubernetes: Journey Towards a Self-Serve Da...

The State of Stateful on Kubernetes

Colocating Data Workloads and Web Services on Kubernetes to Improve Resource ...

Make Your Kafka Cluster Production-Ready

Dynamic Large Scale Spark on Kubernetes: Empowering the Community with Argo W...

Run PostgreSQL in Warp Speed Using NVMe/TCP in the Cloud

ING Data Services hosted on ICHP DoK Amsterdam 2023

Implementing data and databases on K8s within the Dutch government

StatefulSets in K8s - DoK Talks #154

Running PostgreSQL in Kubernetes: from day 0 to day 2 with CloudNativePG - Do...

Analytics with Apache Superset and ClickHouse - DoK Talks #151

Overcoming challenges with protecting and migrating data in multi-cloud K8s e...

Evaluating Cloud Native Storage Vendors - DoK Talks #147

Kubernetes Cluster Upgrade Strategies and Data: Best Practices for your State...

We will Dok You! - The journey to adopt stateful workloads on k8s

Mastering MongoDB on Kubernetes, the power of operators

Leveraging Running Stateful Workloads on Kubernetes for the Benefit of Develo...

Recently uploaded

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

wesley chun

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

UK Journal

Real Time Object Detection Using Open CV

Khem

Partners Life - Insurer Innovation Award 2024

The Digital Insurer

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Principled Technologies

The presentation explores the development and application of artificial intelligence (AI) from its inception to its current status in the modern world. The term "artificial intelligence" was first coined by John McCarthy in 1956 to describe efforts to develop computer programs capable of performing tasks that typically require human intelligence. This concept was first introduced at a conference held at Dartmouth College, where programs demonstrated capabilities such as playing chess, proving theorems, and interpreting texts. In the early stages, Alan Turing contributed to the field by defining intelligence as the ability of a being to respond to certain questions intelligently, proposing what is now known as the Turing Test to evaluate the presence of intelligent behavior in machines. As the decades progressed, AI evolved significantly. The 1980s focused on machine learning, teaching computers to learn from data, leading to the development of models that could improve their performance based on their experiences. The 1990s and 2000s saw further advances in algorithms and computational power, which allowed for more sophisticated data analysis techniques, including data mining. By the 2010s, the proliferation of big data and the refinement of deep learning techniques enabled AI to become mainstream. Notable milestones included the success of Google's AlphaGo and advancements in autonomous vehicles by companies like Tesla and Waymo. A major theme of the presentation is the application of generative AI, which has been used for tasks such as natural language text generation, translation, and question answering. Generative AI uses large datasets to train models that can then produce new, coherent pieces of text or other media. The presentation also discusses the ethical implications and the need for regulation in AI, highlighting issues such as privacy, bias, and the potential for misuse. These concerns have prompted calls for comprehensive regulations to ensure the safe and equitable use of AI technologies. Artificial intelligence has also played a significant role in healthcare, particularly highlighted during the COVID-19 pandemic, where it was used in drug discovery, vaccine development, and analyzing the spread of the virus. The capabilities of AI in healthcare are vast, ranging from medical diagnostics to personalized medicine, demonstrating the technology's potential to revolutionize fields beyond just technical or consumer applications. In conclusion, AI continues to be a rapidly evolving field with significant implications for various aspects of society. The development from theoretical concepts to real-world applications illustrates both the potential benefits and the challenges that come with integrating advanced technologies into everyday life. The ongoing discussion about AI ethics and regulation underscores the importance of managing these technologies responsibly to maximize their their benefits while minimizing potential harms.

Artificial Intelligence: Facts and Myths

Joaquim Jorge

Increase engagement and revenue with Muvi Live Paywall! In this presentation, we will explore the five key benefits of using Muvi Live Paywall to monetize your live streams. You'll learn how Muvi Live Paywall can help you: Monetize your live content easily: Set up pay-per-view access to your live streams and start generating revenue from your content. Increase audience engagement: Provide exclusive, premium content behind the paywall to keep your viewers engaged. Gain valuable viewer insights: Track viewer data and analytics to better understand your audience and tailor your content accordingly. Reduce content piracy: Muvi Live Paywall's security features help protect your content from unauthorized distribution. Streamline your workflow: The all-in-one platform simplifies the process of managing and monetizing your live streams. With Muvi Live Paywall, you can take control of your live stream monetization and create a sustainable business model for your content. Learn more about Muvi Live Paywall and start generating revenue from your live streams today!

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

Roshan Dwivedi

Artificial Intelligence Chap.5 : Uncertainty

Khushali Kathiriya

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

GenAI Risks & Security Meetup 01052024.pdf

lior mazor

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Neo4j

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Drew Madelung

Recently uploaded (20)

Powerful Google developer tools for immediate impact! (2023-24 C)

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Real Time Object Detection Using Open CV

Partners Life - Insurer Innovation Award 2024

Strategies for Landing an Oracle DBA Job as a Fresher

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Boost PC performance: How more available memory can improve productivity

Artificial Intelligence: Facts and Myths

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

Artificial Intelligence Chap.5 : Uncertainty

Exploring the Future Potential of AI-Enabled Smartphone Processors

GenAI Risks & Security Meetup 01052024.pdf

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Boost Fertility New Invention Ups Success Rates.pdf

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Data Cloud, More than a CDP by Matt Robison

How to Troubleshoot Apps for the Modern Connected Worker

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

1000 node Cassandra cluster on Amazon's EKS?

1. 1,000 Cassandra nodes on Kubernetes? Matt Overstreet DoK Day Europe 2022 @ KubeCon 1

2. Who am I? Matt Overstreet Field CTO, Cloud DataStax "Our Mission is to Connect every developer in the world to the power of Apache Cassandra, with the freedom to run their data on any device and in any cloud." DoK Day Europe 2022 @ KubeCon 2

3. What did we do? Built a 1,200 node K8ssandra cluster on Kubernetes And used Kubernetes jobs to bench test it with NoSQLBench DoK Day Europe 2022 @ KubeCon 3

4. A what now? Kubernetes - You know this one, and if you are watching today you know about the history of using it for stateful stuff like databases. Apache Cassandra - An open source database that runs some of the worlds biggest "fast data", and geo-distributed workloads. K8ssandra - A way to run Cassandra on Kubernetes. NoSQLBench - A reliable way to benchmark servers. DoK Day Europe 2022 @ KubeCon 4

5. Why do we care? The first rule of distributed system troubleshooting: Compare the configs Config drift between machines in distributed software causes weird issues. ! " DoK Day Europe 2022 @ KubeCon 5

6. What did K8ssandra bring to the party? • Configuration consistency for the cluster • Ease of Ops - 1,000 nodes would have taken weeks if hand configured (Shout out to Amazon EKS ) Setup for other Kubernetes goodness! • Consolidated logging • Full stack visibility with a service mesh DoK Day Europe 2022 @ KubeCon 6

7. What worked? DoK Day Europe 2022 @ KubeCon 7

8. What worked? • 1,200+ nodes/1 engineer/1 week • Petabyte+ of data/tens of millions of reads and writes per second/p75 latencies under 10ms • Destroyed 800 nodes in 5 minutes using the combined superpowers of helm update and Amazon Elastic Kubernetes Service ! " # • Scaling up the load test through Kubernetes was super easy DoK Day Europe 2022 @ KubeCon 8

9. What didn’t? DoK Day Europe 2022 @ KubeCon 9

10. What didn’t? Gotchas with K8ssandra • K8ssandra (v1) monitoring wasn’t up to the task, tweaks are required for large clusters • K8ssandra needs a little love when deciding how to coordinate new Cassandra node "racks" with their K8ssandra pod startup order DoK Day Europe 2022 @ KubeCon 10

11. What didn’t? Gotchas with EKS (Hyperscalers in general) • Resource limits on AWS EKS were often a surprise • If you are using cloud hosted Kubernetes, think twice before using an autoscaler ! • Know before you grow... IP addresses can run out quickly for large Kubernetes clusters with single or limited pod DoK Day Europe 2022 @ KubeCon 11

12. What did I miss? Thanks! Me matt.overstreet@datastax.com | @omnifroodle - twitter K8ssandra https://K8ssandra.io | @k8ssandra - twitter DataStax https://www.datastax.com DoK Day Europe 2022 @ KubeCon 12

1000 node Cassandra cluster on Amazon's EKS?

Recommended

Recommended

More Related Content

Similar to 1000 node Cassandra cluster on Amazon's EKS?

Similar to 1000 node Cassandra cluster on Amazon's EKS? (20)

More from DoKC

More from DoKC (20)

Recently uploaded

Recently uploaded (20)

1000 node Cassandra cluster on Amazon's EKS?