Make Your Kafka Cluster Production-Ready

Experiences building a distributed shared log on RADOS - Noah Watkins

Ceph Community

Under The Hood Of A Shard-Per-Core Database Architecture

Most databases are based on architectures that pre-date advances to modern hardware. This results in performance issues, the need to overprovision, and a high total cost of ownership. In this webinar, we will discuss the advances to modern server technology and take a deep dive into ScyllaDB’s shard-per-core architecture and our asynchronous engine, the Seastar framework. Join us to learn how Seastar (and ScyllaDB): - Avoid locks and contention on the CPU level - Bypass kernel bottlenecks - Implement its per-core shared-nothing autosharding mechanism - Utilize modern storage hardware - Leverage NUMA to get the best RAM performance - Balance your data across CPUs and nodes for the best and smoothest performance Plus we’ll cover the advantages of unlocking vertical scalability.

Oslo Vancouver Project Update

Ben Nemec

Open stack@ebaySubbu Allamaraju

Scala & Spark(1.6) in Performance Aspect for Scala Taiwan

Jimin Hsieh

Imagine being able to stand up thousands of tenants with thousands of apps, running thousands of Docker-formatted container images and routes, all on a self-healing cluster. Now, take that one step further with all of those images being updatable through a single upload to the registry, and with zero downtime. In this session, Steve Gordon of the Red Hat OpenStack Platform team will show you just that. Steve will walk through a recent benchmarking deployment using the Cloud Native Computing Foundation’s (CNCF) new 1,000 node cluster with OpenStack and Red Hat’s OpenShift Container Platform, the enterprise-ready Kubernetes for developers.

BRKDCT-2445

Rohit Agarwalla

One of the key areas of contributions from Cisco within OpenStack has been in the evolution and the development of the OpenStack Networking Service - Neutron. Using Neutron's Modular Layer 2 (ML2) plug-in and advanced services framework, Cisco has integrated products and solutions with the networking service to simplify the deployment of highly scalable, manageable and performant networks. Through this session we will cover and provide details of reference as well as the various OpenStack Neutron plugins/drivers for hardware and software Cisco products including the Nexus physical and virtual switches, UCS FI, CSR 1kv, ASR1K, CPNR and Application Policy Infrastructure Controller (APIC). We will further discuss different OpenStack networking architecture, deployments and understand Cisco's community code contribution that enable and support IPv6 and NFV related features in Neutron.

Scylla on Kubernetes: Introducing the Scylla Operator

How can Kubernetes be best used to automate the deployment, scaling, and various operations of a Scylla database? Enter Kubernetes Operators, the way to combine domain-specific knowledge about Scylla with the automation framework of Kubernetes. In this presentation, we will quickly explore what Kubernetes is and why it works so well, highlight the pain points of running Scylla with just Kubernetes primitives, and show how we extended Kubernetes so that it can correctly operate a Scylla database. Finally, we will show the Scylla Operator in action and show how easily you can spin up a Scylla cluster with just one command.

Kafka on Kubernetes—From Evaluation to Production at Intuit

(Shrinand Javadekar, Intuit Inc.) Kafka Summit SF 2018 Kubernetes is fast becoming the platform of choice for running distributed, containerized applications in the cloud. It has great features for availability, scalability, monitoring, ease of deployment, a rich set of tools and an extremely fast-growing ecosystem that is making it ever more useful. However, running stateful applications such as Kafka on Kubernetes is not a common practice today. At Intuit, we took an experimentation and data-driven approach for evaluating Kafka on Kubernetes in AWS. In this talk, we will provide details of our functional and non-functional requirements, the experimental configuration and the details of the evaluation. The evaluation process included functional tests for producing/consuming messages, network isolation tests, cross-region tests as well as performance and stress tests. We will focus on the problems we ran into and how we addressed them. This talk will demonstrate a Kubernetes cluster running Kafka along with the details of how each component is configured. Specifically, we will cover the Kafka and ZooKeeper StatefulSets, the ConfigMaps used for storing the server.properties used by all brokers, the service objects for enabling access to the brokers, securing the data and, last but not least, integration with Splunk and Wavefront for logging and monitoring respectively.

Rook - cloud-native storage

Karol Chrapek

Introduction to Apache Kafka

Shiao-An Yuan

Capital One Delivers Risk Insights in Real Time with Stream Processing

Speakers: Ravi Dubey, Senior Manager, Software Engineering, Capital One + Jeff Sharpe, Software Engineer, Capital One Capital One supports interactions with real-time streaming transactional data using Apache Kafka®. Kafka helps deliver information to internal operation teams and bank tellers to assist with assessing risk and protect customers in a myriad of ways. Inside the bank, Kafka allows Capital One to build a real-time system that takes advantage of modern data and cloud technologies without exposing customers to unnecessary data breaches, or violating privacy regulations. These examples demonstrate how a streaming platform enables Capital One to act on their visions faster and in a more scalable way through the Kafka solution, helping establish Capital One as an innovator in the banking space. Join us for this online talk on lessons learned, best practices and technical patterns of Capital One’s deployment of Apache Kafka. -Find out how Kafka delivers on a 5-second service-level agreement (SLA) for inside branch tellers. -Learn how to combine and host data in-memory and prevent personally identifiable information (PII) violations of in-flight transactions. -Understand how Capital One manages Kafka Docker containers using Kubernetes. Watch the recording: https://videos.confluent.io/watch/6e6ukQNnmASwkf9Gkdhh69?.

Introduction to apache kafka

Samuel Kerrien

Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...

javier ramirez

QuestDB es una base de datos open source de alto rendimiento. Mucha gente nos comentaba que les gustaría usarla como servicio, sin tener que gestionar las máquinas. Así que nos pusimos manos a la obra para desarrollar una solución que nos permitiese lanzar instancias de QuestDB con provisionado, monitorización, seguridad o actualizaciones totalmente gestionadas. Unos cuantos clusters de Kubernetes más tarde, conseguimos lanzar nuestra oferta de QuestDB Cloud. Esta charla es la historia de cómo llegamos ahí. Hablaré de herramientas como Calico, Karpenter, CoreDNS, Telegraf, Prometheus, Loki o Grafana, pero también de retos como autenticación, facturación, multi-nube, o de a qué tienes que decir que no para poder sobrevivir en la nube.

14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...

Athens Big Data

SouJava May 2020: Apache Camel 3 - the next generation of enterprise integration

Claus Ibsen

In this session, we'll discuss: - What’s Apache Camel: An overview of Camel and what you use it for and why you should care. - Camel 3: Demos of how Camel 3, Camel K and Camel Quarkus all work together, and will provide insights into Camel’s role in the next major release of Red Hat Integration products. - Camel K: This serverless integration platform provides low-code/no-code capabilities, where integrations can be snapped together quickly using the powers from integration patterns and Camel’s extensive set of connectors. - Camel Quarkus: Using Knative (the fast runtime of Quarkus) and Camel K brings awesome serverless features, such as auto-scaling, scaling to zero, and event-based communication, with great integration capabilities from Apache Camel. You will also hear about the latest Camel sub-project Camel Kafka Connectors which makes it possible to use all the Camel components as Kafka Connect connectors. Finally we bring details of the roadmap for what is coming up in the Camel projects. And after the presentation we have about 30 minutes of QA answering all the questions from the audience.

Building a Messaging Solutions for OVHcloud with Apache Pulsar_Pierre Zemb

StreamNative

OVHcloud is the biggest European cloud provider. From dedicated servers to Managed Kubernetes, from VMware® based Hosted Private Cloud to OpenStack-based Public Cloud, we have over 1.4 million customers worldwide. Internally, we have been running Apache Kafka for years, and despite all the skills obtained operating multiples clusters with millions of messages per second, we decided to shift and build the foundation of our 'topic-as-a-service' product called ioStream on Apache Pulsar. In this talk, you will have the insights of why we decided to use Apache Pulsar instead of Apache Kafka as the core of ioStream. We will tell you our journey to use Apache Pulsar, from our deployments to the management, what did work and what did not.

An Introduction to Apache Kafka

Amir Sedighi

HKG15-401: Ceph and Software Defined Storage on ARM servers

Linaro

HKG15-401: Ceph and Software Defined Storage on ARM servers --------------------------------------------------- Speaker: Yazen Ghannam Steve Capper Date: February 12, 2015 --------------------------------------------------- ★ Session Summary ★ Running Ceph in the colocation, ongoing optimizations -------------------------------------------------- ★ Resources ★ Pathable: https://hkg15.pathable.com/meetings/250828 Video: https://www.youtube.com/watch?v=RdZojLL7ttk Etherpad: http://pad.linaro.org/p/hkg15-401 --------------------------------------------------- ★ Event Details ★ Linaro Connect Hong Kong 2015 - #HKG15 February 9-13th, 2015 Regal Airport Hotel Hong Kong Airport --------------------------------------------------- http://www.linaro.org http://connect.linaro.org

Redpanda and ClickHouse

Altinity Ltd

A Container Stack for Openstack - OpenStack Silicon Valley

Stephen Gordon

OpenStack is an Infrastructure as a Service offering that provides a powerful abstraction layer for interacting with your datacenter infrastructure, supported by a wide array of pluggable drivers for existing physical and virtual infrastructure investments. In this session, you’ll learn how OpenStack is evolving to integrate with the Linux, Docker, Kubernetes stack to provide the ideal infrastructure platform for modern containerized applications. You’ll learn how you can modernize application delivery using the Linux, Docker, Kubernetes stack provided by Red Hat while seamlessly using the authentication, network, and storage infrastructure services provided by an underlying OpenStack cloud.

How we got to 1 millisecond latency in 99% under repair, compaction, and flus...

Scylla is an open source reimplementation of Cassandra which performs up to 10X with drop in-replacement compatibility. At ScyllaDB, performance matters but even more importantly, stable performance under any circumstances. A key factor for our consistent performance is our reliance on userspace schedulers. Scheduling in userspace allows the application, the database in our case to have better control on the different priorities each task has and to provide an SLA to selected operations. Scylla used to have an I/O scheduler and recently won a CPU scheduler. At ScyllaDB, we make architectural decisions that provide not only low latencies but consistently low latencies at higher percentiles. This begins with our choice of language and key architectural decisions such as not using the Linux page-cache, and is fulfilled by autonomous database control, a set of algorithms, which guarantees that the system will adapt to changes in the workload. In the last year, we have made changes to Scylla that provide latencies that are consistent in every percentile. In this talk, Dor Laor will recap those changes and discuss what ScyllaDB is doing in the future.

Distributed Vector Databases - What, Why, and How

Distributed Vector Databases - What, Why, and How - Steve Pousty, VMware In the last two years, AI machine learning has exploded in prominence. One of the key concepts used in the modeling and storage of AI is vectors. Feeling like you should learn more and how you would use them in your data work? Wondering how you would run this distributed on Kubernetes? Then have I got a talk for you! We will start by explaining the concept of (embedding) vectors and how they are used in the AI life cycle. From there we will go into putting them into a database. We will cover the use cases where this technology makes sense. As opposed to an RDBMS, vector databases are more tightly focused and optimized for particular use cases. To ground this discussion in something more concrete, there will be hands-on demos throughout the talk. You will see the advantages to running distributed vector databases on Kubernetes infrastructure. Bring your favorite Kube infrastructure and leave with hands-on experience running AI infrastructure on Kubernetes.

Is It Safe? Security Hardening for Databases Using Kubernetes Operators

Is It Safe? Security Hardening for Databases Using Kubernetes Operators - Robert Hodges, Altinity Thanks to the Operator Pattern, Kubernetes is now an outstanding platform to run databases. But to quote Marathon Man, "is it safe?" This talk is a top-level review of the database security problem in Kubernetes, standard ways that operators can mitigate threats, and a wallet-sized checklist of security features you should look for in any operator you use. Our talk is practical and focused on needs of Kubernetes developers. Join us!

Similar to Make Your Kafka Cluster Production-Ready

From Fixed-Function to Programmable Switching Chip for Network Packet Broker ...

Junho Suh

RocksDB meetup

Javier González

How is Kafka so Fast?

Ricardo Paiva

Containers for the Enterprise: Delivering OpenShift on OpenStack for Performa...

Stephen Gordon

BRKDCT-2445

Rohit Agarwalla

Scylla on Kubernetes: Introducing the Scylla Operator

Kafka on Kubernetes—From Evaluation to Production at Intuit

Rook - cloud-native storage

Karol Chrapek

Introduction to Apache Kafka

Shiao-An Yuan

Capital One Delivers Risk Insights in Real Time with Stream Processing

Introduction to apache kafka

Samuel Kerrien

Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...

javier ramirez

14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...

Athens Big Data

SouJava May 2020: Apache Camel 3 - the next generation of enterprise integration

Claus Ibsen

Building a Messaging Solutions for OVHcloud with Apache Pulsar_Pierre Zemb

StreamNative

An Introduction to Apache Kafka

Amir Sedighi

HKG15-401: Ceph and Software Defined Storage on ARM servers

Linaro

Redpanda and ClickHouse

Altinity Ltd

A Container Stack for Openstack - OpenStack Silicon Valley

Stephen Gordon

How we got to 1 millisecond latency in 99% under repair, compaction, and flus...

Similar to Make Your Kafka Cluster Production-Ready (20)

From Fixed-Function to Programmable Switching Chip for Network Packet Broker ...

RocksDB meetup

How is Kafka so Fast?

Containers for the Enterprise: Delivering OpenShift on OpenStack for Performa...

BRKDCT-2445

Scylla on Kubernetes: Introducing the Scylla Operator

Kafka on Kubernetes—From Evaluation to Production at Intuit

Rook - cloud-native storage

Introduction to Apache Kafka

Capital One Delivers Risk Insights in Real Time with Stream Processing

Introduction to apache kafka

Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...

14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...

SouJava May 2020: Apache Camel 3 - the next generation of enterprise integration

Building a Messaging Solutions for OVHcloud with Apache Pulsar_Pierre Zemb

An Introduction to Apache Kafka

HKG15-401: Ceph and Software Defined Storage on ARM servers

Redpanda and ClickHouse

A Container Stack for Openstack - OpenStack Silicon Valley

How we got to 1 millisecond latency in 99% under repair, compaction, and flus...

More from DoKC

Distributed Vector Databases - What, Why, and How

Is It Safe? Security Hardening for Databases Using Kubernetes Operators

Stop Worrying and Keep Querying, Using Automated Multi-Region Disaster Recovery

Stop Worrying and Keep Querying, Using Automated Multi-Region Disaster Recovery - Shivani Gupta, Elotl & Sergey Pronin, Percona Disaster Recovery(DR) is critical for business continuity in the face of widespread outages taking down entire data centers or cloud provider regions. DR relies on deployment to multiple locations, data replication, monitoring for failure and failover. The process is typically manual involving several moving parts, and, even in the best case, involves some downtime for end-users. A multi-cluster K8s control plane presents the opportunity to automate the DR setup as well as the failure detection and failover. Such automation can dramatically reduce RTO and improve availability for end-users. This talk (and demo) describes one such setup using the open source Percona Operator for PostgreSQL and a multi-cluster K8s orchestrator. The orchestrator will use policy driven placement to replicate the entire workload on multiple clusters (in different regions), detect failure using pluggable logic, and do failover processing by promoting the standby as well as redirecting application traffic

Transforming Data Processing with Kubernetes: Journey Towards a Self-Serve Da...

Transforming Data Processing with Kubernetes: Journey Towards a Self-Serve Data Mesh - Rakesh Subramanian Suresh & Jainik Vora, Intuit This presentation explores how Intuit uses Kubernetes with Domain-Driven Design and Data Mesh principles to transform its data processing landscape, crucial for its AI-driven expert platform. We will discuss the importance of clean data in developing robust generative artificial intelligence and how Intuit is addressing this through the creation of paved paths for data platforms running on Kubernetes. We'll examine the challenges and solutions in managing 100,000 data pipelines and 1000+ engineers interacting with data, highlighting the need for scalable solutions. We'll also discuss how Intuit uses Kubernetes to build its batch and stream processing platform, overcoming hurdles in data pipeline deployment, scheduling, orchestration, and dependency management. We'll conclude by emphasizing how this transformation, based on treating data as a product, has improved decision-making speed and accuracy across the organization and fostered a more efficient, collaborative data culture.

The State of Stateful on Kubernetes

The State of Stateful on Kubernetes - Stateful Workloads in Kubernetes: A Deep Dive - Kaslin Fields & Michelle Au, Google As a platform for distributed computing, Kubernetes enables users to run their workloads across machines. However data has gravity, and when workloads in Kubernetes have to share data with other applications, managing the application’s requirements can get more tricky. In this talk, we will explore what "Stateful" means from Kubernetes' perspective. We will discuss the different types of stateful workloads, and the challenges of deploying them on Kubernetes. We will also look at the features that exist in Kubernetes to support stateful workloads, as well as the features that are in the works. Key Takeaways: What is a stateful workload from Kubernetes’ perspective? What are the challenges of deploying stateful workloads on Kubernetes? What features exist in Kubernetes to support stateful workloads? What features are in the works to support stateful workloads better in the future?

Colocating Data Workloads and Web Services on Kubernetes to Improve Resource ...

Colocating Data Workloads and Web Services on Kubernetes to Improve Resource Utilization - He Cao, ByteDance Recently, more and more data workloads are running on top of Kubernetes, such as ETL processes, Spark and Flink jobs, and more. These workloads typically exhibit high resource utilization and remain relatively stable over time. In contrast, web services often exhibit tidal patterns, characterized by significant fluctuations in resource utilization. The resource model of vanilla Kubernetes is static, which can lead to low resource utilization accumulated over 24 hours. In this talk, He will introduce how ByteDance uses Katalyst to colocate data workloads and online services on Kubernetes to improve resource utilization. In addition, He will explain how Katalyst ensures the QoS of these workloads through QoS-aware scheduling, service profiling, multi-dimensional resource isolation, real-time container resource adjustment, and more. In ByteDance, Katalyst has been deployed on 500,000+ nodes with tens of millions of cores, and has improved daily resource utilization from 20% to 60%.

Dynamic Large Scale Spark on Kubernetes: Empowering the Community with Argo W...

Dynamic Large Scale Spark on Kubernetes: Empowering the Community with Argo Workflows and Argo Events - Ovidiu Valeanu, AWS & Vara Bonthu, Amazon Are you eager to build and manage large-scale Spark clusters on Kubernetes for powerful data processing? Whether you are starting from scratch or considering migrating Spark workloads from existing Hadoop clusters to Kubernetes, the challenges of configuring storage, compute, networking, and optimizing job scheduling can be daunting. Join us as we unveil the best practices to construct a scalable Spark clusters on Kubernetes, with a special emphasis on leveraging Argo Workflows and Argo Events. In this talk, we will guide you through the journey of building highly scalable Spark clusters on Kubernetes, using the most popular open-source tools. We will showcase how to harness the potential of Argo Workflows and Argo Events for event-driven job scheduling, enabling efficient resource utilization and seamless scalability. By integrating these powerful tools, you will gain better control and flexibility for executing Spark jobs on Kubernetes.

Run PostgreSQL in Warp Speed Using NVMe/TCP in the Cloud

Run PostgreSQL in Warp Speed Using NVMe/TCP in the Cloud - Sagy Volkov, Lightbits PostgreSQL as a SQL engine can accommodate a very high-transaction rate, but as your data grows and the number of connections and queries increases, there is a challenge for the storage to keep up with the SQL engine. To the rescue comes NVMe over TCP (or NVMe/TCP). Developed by Lightbits Labs in 2016 and donated to the Linux community, it is the next evaluation of using NVMe based storage over TCP Fabric. NVMe/TCP simplifies how you interact with remote NVMe devices (targets) and allows your PostgreSQL storage to consume fast storage very easily. In this session I will explain the core concept of the NVMe/TCP protocol, current storage providers that can use it, how you can consume it in Kubernetes (super easy), and discuss the possibilities of using NVMe/TCP in the cloud. The session will also include a performance comparison of a few storage that are available in AWS and even a live demo of how PostgreSQL can run super fast - warp speed fast - in AWS.

The Kubernetes Native Database

Link: https://www.youtube.com/watch?v=D8kJCvsHD9Q&list=PLHgdNuGxrJt04Fwaip9aDYvXrbRSmc5HZ&index=12 https://go.dok.community/slack https://dok.community/ From DoK Day NA 2022 (https://www.youtube.com/watch?v=YWTa-DiVljY&list=PLHgdNuGxrJt04Fwaip9aDYvXrbRSmc5HZ) In the software industry we’re fond of terms that define major trends, like “cloud native”, “Kubernetes native” and “serverless”. As more and more organizations move stateful workloads to Kubernetes, we’ve started to see these terms applied to data infrastructure, where they can get overtaken by marketing hype unless we work to define them. In this talk, we’ll examine two different databases, TiDB and Apache Cassandra, in order to identify what it means for a database to be Kubernetes native and why it matters. We’ll look at points including: - The differences between cloud native, Kubernetes native, and serverless - How databases become Kubernetes native - Benefits of Kubernetes native databases - How Kubernetes can better support databases ----- Jeff has worked as a software engineer and architect in multiple industries and as a developer advocate helping engineers get up to speed on Apache Cassandra. He's involved in multiple open source projects in the Cassandra and Kubernetes ecosystems including Stargate and K8ssandra. Jeff is the author of the O’Reilly books “Cassandra: The Definitive Guide" and “Managing Cloud Native Data on Kubernetes".

ING Data Services hosted on ICHP DoK Amsterdam 2023

An explanation of how ING deals with local persistence at scale in secure and compliant manner for Elastic and Prometheus workloads today and other Data Services in the future. In more detail we will elaborate on the following topics How we solve local persistence Type of workloads now and in the future Typical requirements for a banking environment Automation Scale Resilience Security / Compliance Service offering / demarcation About Tor and Luuk: Tor and Luuk are experienced engineers working at ING for over 10 years and working in the Kubernetes area for the last 5 years. They are specialized in and responsible for the Data Services OpenShift clusters in ING and have a strong focus on resilience, automation and security.

Implementing data and databases on K8s within the Dutch government

A small walkthrough of projects within the dutch government running Data(bases) on OpenShift. This talk shares success stories, provides a proven recipe to `get it done` and debunks some of the FUD. About Sebastiaan: I have always been a weird DBA, trying to combine Databases with out-of-the-box thinking and a DevOps mindset. Around 2016 I fell in love with both Postgres and Kubernetes, and I then committed my life to enabling Dutch organisations with running their Database workloads CloudNative. Over the last few years I worked as a private contractor for 2 large government agencies doing exactly that, and I want to share my and others (success stories) hoping to enable and inspire Data on Kubernetes adoption.

StatefulSets in K8s - DoK Talks #154

https://go.dok.community/slack https://dok.community/ Link: https://youtu.be/n_thXwyJNSU ABSTRACT OF THE TALK Deploying Stateless applications is easy but this is not the case for Stateful applications. StatefulSets are the K8s API object that helps to manage stateful application. Learn about what Stateful sets are, how to create, How it differs from Deployments. KEY TAKE-AWAYS FROM THE TALK This talk is focused on basics of StatefulSet, how StatefulSet differs from Deployments, How to manage Stateful app using StatefulSet

Running PostgreSQL in Kubernetes: from day 0 to day 2 with CloudNativePG - Do...

Link: https://youtu.be/cegd3Exg05w https://go.dok.community/slack https://dok.community/ Gabriele Bartolini - Vice President/CTO of Cloud Native and Kubernetes, EDB ABSTRACT OF THE TALK Imagine this: you have a virtual infrastructure based on Kubernetes, made up of virtual data centers, possibly spread across multiple Kubernetes clusters and regions. Your infrastructure could even be hosted on premises or on different cloud service providers. Infrastructure as Code is a requirement. You’ve been tasked to run Postgres databases, alongside your applications. The good news is that you can leverage a fully open source stack with Kubernetes, PostgreSQL and the CloudNativePG operator, and deploy your Postgres database in the same way you deploy applications. Join me in this webinar to discover the key role that you have to make this succeed, starting from day 0 through day 2 operations. I’ll share some examples and best practices for running Postgres databases in Kubernetes, before peeking at the new features we are developing for the months to come.

Analytics with Apache Superset and ClickHouse - DoK Talks #151

Link: https://youtu.be/Y-1uFVKDfgY https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK This talk concerns performing analytical tasks with Apache Superset with ClickHouse as the data backend. ClickHouse is a super fast database for analytical tasks, and Apache Superset is an Apache Software foundation project meant for data visualization and exploration. Performing analytical tasks using this combo is super fast since both the software are designed to be scalable and capable of handling data of petabyte scale.

Overcoming challenges with protecting and migrating data in multi-cloud K8s e...

Link: https://youtu.be/EFaRyl4HmmE https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK If you are running or planning a multi-cloud or even a multi-cluster environment, there are several considerations in implementing a data protection solution – especially if you plan on an organic home-grown, do-it-yourself option. This talk will highlight challenges and best practices around centralized management of configuration, credentials, compliance across multiple accounts, regions, providers etc. We will also highlight the deviations in CSI driver implementations of various storage vendors and cloud providers. Finally, we will cover the various recovery options available in the market today. Kubernetes cloud services are popular since they mitigate, but do not eliminate, the difficulties of operating a Kubernetes environment. This is especially true for protecting the stateful configuration and data of your Kubernetes applications, where the inherent high-availability and infrastructure as code are not a substitute for have cloud-native backup and disaster recovery capabilities. Further, many companies now have multi-cloud strategies for their cloud-native applications. These challenges can be addressed with backup applications that are both Kubernetes managed service and multi-cloud aware in order to snapshot, copy, restore, and migrate Kubernetes workloads (resources and data) running on AKS, EKS and GKE. Capturing information from cloud accounts and how the cluster and storage resources are configured allows 1) centralized visibility into all cloud accounts and the clusters and resources in the accounts including for compliance; 2) cross-account, cross-cluster, and cross-region data restores; 3) automation of the cluster and data restores including for Dev, Test, and Production recovery use cases. BIO Sebastian Glab is a Cloud Architect for CloudCasa and he resides in Poland. He is responsible for integrating the different cloud providers with the CloudCasa service, and making sure that all clusters in the cloud service get discovered and protected. In his free time, he plays volleyball and develops his own projects. Martin Phan is the Field CTO in North America for CloudCasa by Catalogic Software. With over 20+ years of experience in the software-industry, he takes pride in supporting, developing, implementing, and selling enterprise software and data protection solutions to help customer solve their backup and recovery challenges. KEY TAKE-AWAYS FROM THE TALK 1) Challenges and best practices around centralized management of configuration, credentials, compliance across multiple accounts, regions, providers etc. 2) Advantages of cloud awareness and Kubernetes managed service awareness for application and data recovery and security 3) Examples of overcoming Container Storage Interface (CSI) deviations 4) Various recovery options available in the market today.

Evaluating Cloud Native Storage Vendors - DoK Talks #147

Link: https://youtu.be/YVXEpcSclwY https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK In a continuation of a talk given at DoK day at KubeCon EU 2022, join Dinesh Majrekar, Civo's CTO as they walk through their evaluation process of the CNCF Storage market. Civo offers managed Kubernetes clusters powered by K3s to customers around the world. We manage thousands of Virtual Machines and stateful customer data within multiple data centres across several continents. In late 2021, Civo had the opportunity to evaluate the CNCF storage landscape to move to a new technology stack. During the migration project, Civo evaluated Mayastor, Ondat, Ceph and Longhorn against the following metrics: Scalability Performance Ease of Support Attendants will see practical examples on how they could carry out their own similar evaluation and see some of the results of the Civo research project. BIO Dinesh is CTO at Civo. Having worked in the hosting industry for many years, Dinesh has a passion for creating solutions that operate at scale. This not only applies to the technology stack, but for nurturing engineers through their career.

Kubernetes Cluster Upgrade Strategies and Data: Best Practices for your State...

Link: https://youtu.be/qUW8LkxYayc https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK How do you make sure your Stateful Workloads remain available when your Kubernetes infrastructure updates? This talk will discuss different strategies of upgrading a Kubernetes cluster, and how you can manage risk for your workload. The talk will showcase demos of each upgrade strategy. BIO Peter is a Senior Software Engineer on GKE at Google. He works on improving Kubernetes for Stateful workloads. His main focus is on enhancing the Kubernetes ecosystem for high availability applications. KEY TAKE-AWAYS FROM THE TALK The mechanics of different upgrade strategies, when to apply a particular upgrade strategy depending on your Stateful workload and how to mitigate risk to your application’s availability.

We will Dok You! - The journey to adopt stateful workloads on k8s

Link: https://youtu.be/AjvwG53yLMY https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK Stateful workloads are the heart of any application, yet they remain confusing and complicated even to daily K8s practitioners. That’s why many organizations shy away from migrating their data - their prized possession - to the unfamiliar stateful realm of Kubernetes. After meeting with many organizations in the adoption phase, I discovered what works best, what to avoid, and how critical it is to gain confidence and the right knowledge in order to successfully adopt stateful workloads. In this talk I will demonstrate how to optimally adopt Kubernetes and stateful workloads in a few steps, based on what I’ve learned from observing dozens of different adoption journeys. If you are taking your first steps in data on K8s or contemplating where to start - this talk is for you! BIO - A Developer turned Solution Architect. - Working at Komodor, a startup building the first K8s-native troubleshooting platform. - Love everything in infrastructure: storage, networks & security - from 70’s era mainframes to cloud-native. - All about “plan well, sleep well”. KEY TAKE-AWAYS FROM THE TALK - Understand how critical stateful workloads are for any system, and that the key challenges to migrating it to Kubernetes are knowledge and confidence. - How to build the foundational knowledge required to overcome adoption challenges by creating a learning path for individuals and teams. - How to gain confidence to run stateful workloads on Kubernetes with support from the community (and yourself!)

Mastering MongoDB on Kubernetes, the power of operators

Link: https://youtu.be/Pi5ueyl_1jU https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK During my first talk for DoK community I want to walk you through the world of NoSQL database MongoDB and Kubernetes Operators - Community Edition, Enterprise Edition (MongoDB and Ops Manager on K8s), and Atlas operator, highlight the most important capabilities, talk about use cases and challenges, the theory will be mixed with a live demos! BIO I'm a SRE / NoSQL / DevOps professional. I hold CKA, CKAD, CKS, also I’m MongoDB Certified DBA and MongoDB Champion. I have experience with multiple cloud providers, Kubernetes, different types of K8s operators (Strimzi, RabbitMQ Cluster Operator), but especially MongoDB K8s Operator. I also work with KEDA. Since 2017, I have been a speaker at MongoDB conferences all around the world (USA, China, Europe). KEY TAKE-AWAYS FROM THE TALK I would like to share the best practices of running NoSQL database - MongoDB on Kubernetes also I want to show how to manage Atlas (MongoDB cloud) via K8s operator https://www.mongodb.com/developer/community-champions/arkadiusz-borucki/

Leveraging Running Stateful Workloads on Kubernetes for the Benefit of Develo...