Slides from the talk on Running Kafka on Kubernetes by Avinash Upadhyaya and Ashwin Venkatesan of Platformatory at the Apache Kafka Bengaluru July 2023 meetup.
This talk will provide an introduction to concerns around running Apache Kafka on top of K8S and the operator pattern. It will cover a comparative view of operators available as well as experiential guidance around operations at scale
Slides from the talk on lessons on running Kafka on Kubernetes by Pavan Keshavamurthy and Avinash Upadhyaya of Platformatory at the Apache Kafka Mumbai July 2023 meetup.
Look at various tooling around running Apache Kafka on Kubernetes and cover best practices for running a distributed system such as Kafka on Kubernetes.
Confluent Operator as Cloud-Native Kafka Operator for KubernetesKai Wähner
Agenda:
- Cloud Native vs. SaaS / Serverless Kafka
- The Emergence of Kubernetes
- Kafka on K8s Deployment Challenges
- Confluent Operator as Kafka Operator
- Q&A
Confluent Operator enables you to:
Provisioning, management and operations of Confluent Platform (including ZooKeeper, Apache Kafka, Kafka Connect, KSQL, Schema Registry, REST Proxy, Control Center)
Deployment on any Kubernetes Platform (Vanilla K8s, OpenShift, Rancher, Mesosphere, Cloud Foundry, Amazon EKS, Azure AKS, Google GKE, etc.)
Automate provisioning of Kafka pods in minutes
Monitor SLAs through Confluent Control Center or Prometheus
Scale Kafka elastically, handle fail-over & Automate rolling updates
Automate security configuration
Built on our first hand knowledge of running Confluent at scale
Fully supported for production usage
Putting Kafka In Jail – Best Practices To Run Kafka On Kubernetes & DC/OSLightbend
Apache Kafka–part of Lightbend Fast Data Platform–is a distributed streaming platform that is best suited to run close to the metal on dedicated machines in statically defined clusters. For most enterprises, however, these fixed clusters are quickly becoming extinct in favor of mixed-use clusters that take advantage of all infrastructure resources available.
In this webinar by Sean Glover, Fast Data Engineer at Lightbend, we will review leading Kafka implementations on DC/OS and Kubernetes to see how they reliably run Kafka in container orchestrated clusters and reduce the overhead for a number of common operational tasks with standard cluster resource manager features. You will learn specifically about concerns like:
* The need for greater operational knowhow to do common tasks with Kafka in static clusters, such as applying broker configuration updates, upgrading to a new version, and adding or decommissioning brokers.
* The best way to provide resources to stateful technologies while in a mixed-use cluster, noting the importance of disk space as one of Kafka’s most important resource requirements.
* How to address the particular needs of stateful services in a model that natively favors stateless, transient services.
18th Athens Big Data Meetup - 2nd Talk - Run Spark and Flink Jobs on KubernetesAthens Big Data
Title: Run Spark and Flink Jobs on Kubernetes
Speaker: Chaoran Yu (https://linkedin.com/in/chaoran-yu-97b1144a/)
Date: Thursday, November 14, 2019
Event: https://meetup.com/Athens-Big-Data/events/265957761/
Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...HostedbyConfluent
At Wells-Fargo, we move 150 TB of logs data from our syslogs to Splunk forwarders that get indexed and organized for analytic queries. As we modernize and migrate our applications to our hybrid cloud the performance expectations for this infrastructure will proportionately increase. Those improvements include the resilience of the end to end infrastructure. First, we decoupled the applications from their logging interface through a loglibrary which split the streams of logs from their sources to KAFKA which routed them to two separate destinations Splunk and ELK respectively. We also used prometheus and grafana for monitoring the metrics. We also deployed KAFKA, Splunk, ELK, Prometheus and Grafana on the Kubernetes clusters. Confluent had released a version of KAFKA without Zookeeper and replaced its functionality with Quorum Controller. The Quorum-Controller version exhibited better disposability one of the 12factors that's important for Cloud-Nativeness. We packaged this version into a Kubernetes operator called Keda and deployed this for auto-scaling. We tested this to simulate the amount of logdata that we typically generate in production. Based on the above we have also implemented distributed tracing and help make it just as resilient. We will share our lessons learnt, the patterns and practices to modernize both our underlying runtime platforms and our applications with highly performing and resilient event-driven architectures.
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...confluent
Microservices, events, containers, and orchestrators are dominating our vernacular today. As operations teams adapt to support these technologies in production, cloud-native platforms like Pivotal Cloud Foundry and Kubernetes have quickly risen to serve as force multipliers of automation, productivity and value.
Apache Kafka® is providing developers a critically important component as they build and modernize applications to cloud-native architecture.
This talk will explore:
• Why cloud-native platforms and why run Apache Kafka on Kubernetes?
• What kind of workloads are best suited for this combination?
• Tips to determine the path forward for legacy monoliths in your application portfolio
• Demo: Running Apache Kafka as a Streaming Platform on Kubernetes
Kubecon 2023 EU - KServe - The State and Future of Cloud-Native Model ServingTheofilos Papapanagiotou
KServe is a cloud-native open source project for serving production ML models built on CNCF projects like Knative and Istio. In this talk, we’ll update you on KServe’s progress towards 1.0, the latest developments, such as ModelMesh and InferenceGraph, and its future roadmap. We’ll discuss the Kubernetes design patterns used in KServe to achieve the core ML inference capability, as well as the design philosophy behind KServe and how it integrates the CNCF ecosystem so you can walk up and down the stack to use features to meet your production model deployment requirements. The well-designed InferenceService interface encapsulates the complexity of networking, lifecycle, server configurations and allows you to easily add serverless capabilities to model servers like TensorFlow Serving, TorchServe, and Triton on CPU/GPU. You can also turn on full service mesh mode to secure your InferenceServices. We’ll walk through different scenarios to show how you can quickly start with KServe and evolve to a production-ready setup with scalability, security, observability, and auto-scaling acceleration using CNCF projects like Knative, Istio, SPIFFE/SPIRE, OpenTelemetry, and Fluid.
Slides from the talk on lessons on running Kafka on Kubernetes by Pavan Keshavamurthy and Avinash Upadhyaya of Platformatory at the Apache Kafka Mumbai July 2023 meetup.
Look at various tooling around running Apache Kafka on Kubernetes and cover best practices for running a distributed system such as Kafka on Kubernetes.
Confluent Operator as Cloud-Native Kafka Operator for KubernetesKai Wähner
Agenda:
- Cloud Native vs. SaaS / Serverless Kafka
- The Emergence of Kubernetes
- Kafka on K8s Deployment Challenges
- Confluent Operator as Kafka Operator
- Q&A
Confluent Operator enables you to:
Provisioning, management and operations of Confluent Platform (including ZooKeeper, Apache Kafka, Kafka Connect, KSQL, Schema Registry, REST Proxy, Control Center)
Deployment on any Kubernetes Platform (Vanilla K8s, OpenShift, Rancher, Mesosphere, Cloud Foundry, Amazon EKS, Azure AKS, Google GKE, etc.)
Automate provisioning of Kafka pods in minutes
Monitor SLAs through Confluent Control Center or Prometheus
Scale Kafka elastically, handle fail-over & Automate rolling updates
Automate security configuration
Built on our first hand knowledge of running Confluent at scale
Fully supported for production usage
Putting Kafka In Jail – Best Practices To Run Kafka On Kubernetes & DC/OSLightbend
Apache Kafka–part of Lightbend Fast Data Platform–is a distributed streaming platform that is best suited to run close to the metal on dedicated machines in statically defined clusters. For most enterprises, however, these fixed clusters are quickly becoming extinct in favor of mixed-use clusters that take advantage of all infrastructure resources available.
In this webinar by Sean Glover, Fast Data Engineer at Lightbend, we will review leading Kafka implementations on DC/OS and Kubernetes to see how they reliably run Kafka in container orchestrated clusters and reduce the overhead for a number of common operational tasks with standard cluster resource manager features. You will learn specifically about concerns like:
* The need for greater operational knowhow to do common tasks with Kafka in static clusters, such as applying broker configuration updates, upgrading to a new version, and adding or decommissioning brokers.
* The best way to provide resources to stateful technologies while in a mixed-use cluster, noting the importance of disk space as one of Kafka’s most important resource requirements.
* How to address the particular needs of stateful services in a model that natively favors stateless, transient services.
18th Athens Big Data Meetup - 2nd Talk - Run Spark and Flink Jobs on KubernetesAthens Big Data
Title: Run Spark and Flink Jobs on Kubernetes
Speaker: Chaoran Yu (https://linkedin.com/in/chaoran-yu-97b1144a/)
Date: Thursday, November 14, 2019
Event: https://meetup.com/Athens-Big-Data/events/265957761/
Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...HostedbyConfluent
At Wells-Fargo, we move 150 TB of logs data from our syslogs to Splunk forwarders that get indexed and organized for analytic queries. As we modernize and migrate our applications to our hybrid cloud the performance expectations for this infrastructure will proportionately increase. Those improvements include the resilience of the end to end infrastructure. First, we decoupled the applications from their logging interface through a loglibrary which split the streams of logs from their sources to KAFKA which routed them to two separate destinations Splunk and ELK respectively. We also used prometheus and grafana for monitoring the metrics. We also deployed KAFKA, Splunk, ELK, Prometheus and Grafana on the Kubernetes clusters. Confluent had released a version of KAFKA without Zookeeper and replaced its functionality with Quorum Controller. The Quorum-Controller version exhibited better disposability one of the 12factors that's important for Cloud-Nativeness. We packaged this version into a Kubernetes operator called Keda and deployed this for auto-scaling. We tested this to simulate the amount of logdata that we typically generate in production. Based on the above we have also implemented distributed tracing and help make it just as resilient. We will share our lessons learnt, the patterns and practices to modernize both our underlying runtime platforms and our applications with highly performing and resilient event-driven architectures.
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...confluent
Microservices, events, containers, and orchestrators are dominating our vernacular today. As operations teams adapt to support these technologies in production, cloud-native platforms like Pivotal Cloud Foundry and Kubernetes have quickly risen to serve as force multipliers of automation, productivity and value.
Apache Kafka® is providing developers a critically important component as they build and modernize applications to cloud-native architecture.
This talk will explore:
• Why cloud-native platforms and why run Apache Kafka on Kubernetes?
• What kind of workloads are best suited for this combination?
• Tips to determine the path forward for legacy monoliths in your application portfolio
• Demo: Running Apache Kafka as a Streaming Platform on Kubernetes
Kubecon 2023 EU - KServe - The State and Future of Cloud-Native Model ServingTheofilos Papapanagiotou
KServe is a cloud-native open source project for serving production ML models built on CNCF projects like Knative and Istio. In this talk, we’ll update you on KServe’s progress towards 1.0, the latest developments, such as ModelMesh and InferenceGraph, and its future roadmap. We’ll discuss the Kubernetes design patterns used in KServe to achieve the core ML inference capability, as well as the design philosophy behind KServe and how it integrates the CNCF ecosystem so you can walk up and down the stack to use features to meet your production model deployment requirements. The well-designed InferenceService interface encapsulates the complexity of networking, lifecycle, server configurations and allows you to easily add serverless capabilities to model servers like TensorFlow Serving, TorchServe, and Triton on CPU/GPU. You can also turn on full service mesh mode to secure your InferenceServices. We’ll walk through different scenarios to show how you can quickly start with KServe and evolve to a production-ready setup with scalability, security, observability, and auto-scaling acceleration using CNCF projects like Knative, Istio, SPIFFE/SPIRE, OpenTelemetry, and Fluid.
Deploying Anything as a Service (XaaS) Using Operators on KubernetesAll Things Open
Presented by: Jeff Spahr
Presented at the All Things Open 2021
Raleigh, NC, USA
Raleigh Convention Center
Abstract: Kubernetes has long since solved compute as a service, but what if you want to deploy higher level services without reimplementing the finer details of how to scale, cluster, and upgrade those services? Custom Resource Definitions (CRDs) allow users to expand the Kubernetes API to create resources like 'kind: elasticsearch' or 'kind: mariadb'. Operators manage those CRDs and take on orchestration and lifecycle management of those services.
In this talk I'll cover the what and why of Operators on Kubernetes with a focus on what real world problems this solves for Kubernetes end users. I'll walk through deploying operators for common high level services that make up a production application.
The XaaS walkthrough and demo will include some of the following technologies:
* Cloud Services (EC2, S3)
* Databases (MariaDB, Vitess, Elasticsearch)
* Load balancers (F5, NGINX)
* Streaming (Kafka, RabbitMQ)
You'll leave this session with a foundation to start offering XaaS to your end users.
Event Streaming Architectures with Confluent and ScyllaDBScyllaDB
Jeff Bean will lead a discussion of event-driven architectures, Apache Kafka, Kafka Connect, KSQL and Confluent Cloud. Then we'll talk about some uses of Confluent and Scylla together, including a co-deployment with Lookout, ScyllaDB and Confluent in the IoT space, and the upcoming native connector.
Kubernetes has become the defacto standard as a platform for container orchestration. Its ease of extending and many integrations has paved the way for a wide variety of data science and research tooling to be built on top of it.
From all encompassing tools like Kubeflow that make it easy for researchers to build end-to-end Machine Learning pipelines to specific orchestration of analytics engines such as Spark; Kubernetes has made the deployment and management of these things easy. This presentation will showcase some of the larger research tools in the ecosystem and go into how Kubernetes has enabled this easy form of application management.
This talk discusses the core concepts behind the Kubernetes extensibility model. We are going to see how to implement new CRDs, operators and when to use them to automate the most critical aspects of your Kubernetes clusters.
We believe that the popularity of Kubernetes derives from its ability to adapt and improve the infrastructure in which is deployed. I'll explain how this is done
[WSO2Con EU 2018] Deploying Applications in K8S and DockerWSO2
Within the last four years container technologies have become very popular. A lot of companies and developers are now using containers to ship their applications. Docker provides an easy-to-use packaging model to bundle the application. However in many cases, a single container is not enough to run an application. It requires multiple containers, scaled into multiple host machines to become a production grade deployment. Kubernetes is an open source system for automating deployment, scaling, and management of containerized applications. It groups containers that make up an application into logical units for easy management and discovery. This presentation discusses best practices of deploying application in Docker and Kubernetes while discussing Docker and Kubernetes concepts.
Kubernetes has been a key component for many companies to reduce technical debt in infrastructure by:
• Fostering the Adoption of Docker
• Simplifying Container Management
• Onboarding Developers On Infrastructure
• Unlocking Continuous Integration and Delivery
During this meetup we are going to discuss the following topics and share some best practices
• What's new with Kubernetes 1.3
• Generate Cluster Configuration using CloudFormation
• Deploy Kubernetes Clusters on AWS
• Scaling the Cluster
• Integrating Ingress with Elastic Load Balancer
• Using Internal ELB's as Kubernetes' Service
• Using EBS for persistent volumes
• Integrating Route53
Disaster Recovery Options Running Apache Kafka in Kubernetes with Rema Subra...HostedbyConfluent
Active-Active, Active-Passive, and stretch clusters are hallmark patterns that have been the gold standard in Apache Kafka® disaster recovery architectures for years. Moving to Kubernetes requires unpacking these patterns and choosing a configuration that allows you to meet the same RTO and RPO requirements.
In this talk, we will cover how Active-Active/Active-Passive modes for disaster recovery have worked in the past and how the architecture evolves with deploying Apache Kafka on Kubernetes. We'll also look at how stretch clusters sitting on this architecture give a disaster recovery solution that's built-in!
Armed with this information, you will be able to architect your new Apache Kafka Kubernetes deployment (or retool your existing one) to achieve the resilience you require.
In Apache Cassandra Lunch #41: Apache Cassandra Lunch #41: Cassandra on Kubernetes - Docker/Kubernetes/Helm Part 1, we discuss Cassandra on Kubernetes and give an introduction to Docker, Kubernetes, and Helm.
Accompanying Blog: https://blog.anant.us/apache-cassandra-lunch-41-cassandra-on-kubernetes-docker-kubernetes-helm-part-1/
Accompanying YouTube: https://youtu.be/-I8cKQO_Qr0
Sign Up For Our Newsletter: http://eepurl.com/grdMkn
Join Cassandra Lunch Weekly at 12 PM EST Every Wednesday: https://www.meetup.com/Cassandra-DataStax-DC/events/
Cassandra.Link:
https://cassandra.link/
Follow Us and Reach Us At:
Anant:
https://www.anant.us/
Awesome Cassandra:
https://github.com/Anant/awesome-cassandra
Cassandra.Lunch:
https://github.com/Anant/Cassandra.Lunch
Email:
solutions@anant.us
LinkedIn:
https://www.linkedin.com/company/anant/
Twitter:
https://twitter.com/anantcorp
Eventbrite:
https://www.eventbrite.com/o/anant-1072927283
Facebook:
https://www.facebook.com/AnantCorp/
In this session, Kiran gives a talk about the rich ecosystem of tools (cmk, CAPC, Terraform, Ansible, Packer, csbench, mbx), that support Cloudstack.
Find out how the various tools work and how easy it is to integrate with Apache CloudStack.
This session provides a great way to speed up CloudStack adoption and improve performance by saving valuable time.
-----------------------------------------
The CloudStack India User Group 2024 took place in Hyderabad on 23rd February. The conference, arranged by a group of volunteers from the Apache CloudStack Community, saw multiple sessions held about the cloud orchestration platform and its latest advancements.
Running Apache Spark Jobs Using KubernetesDatabricks
Apache Spark has introduced a powerful engine for distributed data processing, providing unmatched capabilities to handle petabytes of data across multiple servers. Its capabilities and performance unseated other technologies in the Hadoop world, but while Spark provides a lot of power, it also comes with a high maintenance cost, which is why we now see innovations to simplify the Spark infrastructure.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
More Related Content
Similar to A Primer Towards Running Kafka on Top of Kubernetes.pdf
Deploying Anything as a Service (XaaS) Using Operators on KubernetesAll Things Open
Presented by: Jeff Spahr
Presented at the All Things Open 2021
Raleigh, NC, USA
Raleigh Convention Center
Abstract: Kubernetes has long since solved compute as a service, but what if you want to deploy higher level services without reimplementing the finer details of how to scale, cluster, and upgrade those services? Custom Resource Definitions (CRDs) allow users to expand the Kubernetes API to create resources like 'kind: elasticsearch' or 'kind: mariadb'. Operators manage those CRDs and take on orchestration and lifecycle management of those services.
In this talk I'll cover the what and why of Operators on Kubernetes with a focus on what real world problems this solves for Kubernetes end users. I'll walk through deploying operators for common high level services that make up a production application.
The XaaS walkthrough and demo will include some of the following technologies:
* Cloud Services (EC2, S3)
* Databases (MariaDB, Vitess, Elasticsearch)
* Load balancers (F5, NGINX)
* Streaming (Kafka, RabbitMQ)
You'll leave this session with a foundation to start offering XaaS to your end users.
Event Streaming Architectures with Confluent and ScyllaDBScyllaDB
Jeff Bean will lead a discussion of event-driven architectures, Apache Kafka, Kafka Connect, KSQL and Confluent Cloud. Then we'll talk about some uses of Confluent and Scylla together, including a co-deployment with Lookout, ScyllaDB and Confluent in the IoT space, and the upcoming native connector.
Kubernetes has become the defacto standard as a platform for container orchestration. Its ease of extending and many integrations has paved the way for a wide variety of data science and research tooling to be built on top of it.
From all encompassing tools like Kubeflow that make it easy for researchers to build end-to-end Machine Learning pipelines to specific orchestration of analytics engines such as Spark; Kubernetes has made the deployment and management of these things easy. This presentation will showcase some of the larger research tools in the ecosystem and go into how Kubernetes has enabled this easy form of application management.
This talk discusses the core concepts behind the Kubernetes extensibility model. We are going to see how to implement new CRDs, operators and when to use them to automate the most critical aspects of your Kubernetes clusters.
We believe that the popularity of Kubernetes derives from its ability to adapt and improve the infrastructure in which is deployed. I'll explain how this is done
[WSO2Con EU 2018] Deploying Applications in K8S and DockerWSO2
Within the last four years container technologies have become very popular. A lot of companies and developers are now using containers to ship their applications. Docker provides an easy-to-use packaging model to bundle the application. However in many cases, a single container is not enough to run an application. It requires multiple containers, scaled into multiple host machines to become a production grade deployment. Kubernetes is an open source system for automating deployment, scaling, and management of containerized applications. It groups containers that make up an application into logical units for easy management and discovery. This presentation discusses best practices of deploying application in Docker and Kubernetes while discussing Docker and Kubernetes concepts.
Kubernetes has been a key component for many companies to reduce technical debt in infrastructure by:
• Fostering the Adoption of Docker
• Simplifying Container Management
• Onboarding Developers On Infrastructure
• Unlocking Continuous Integration and Delivery
During this meetup we are going to discuss the following topics and share some best practices
• What's new with Kubernetes 1.3
• Generate Cluster Configuration using CloudFormation
• Deploy Kubernetes Clusters on AWS
• Scaling the Cluster
• Integrating Ingress with Elastic Load Balancer
• Using Internal ELB's as Kubernetes' Service
• Using EBS for persistent volumes
• Integrating Route53
Disaster Recovery Options Running Apache Kafka in Kubernetes with Rema Subra...HostedbyConfluent
Active-Active, Active-Passive, and stretch clusters are hallmark patterns that have been the gold standard in Apache Kafka® disaster recovery architectures for years. Moving to Kubernetes requires unpacking these patterns and choosing a configuration that allows you to meet the same RTO and RPO requirements.
In this talk, we will cover how Active-Active/Active-Passive modes for disaster recovery have worked in the past and how the architecture evolves with deploying Apache Kafka on Kubernetes. We'll also look at how stretch clusters sitting on this architecture give a disaster recovery solution that's built-in!
Armed with this information, you will be able to architect your new Apache Kafka Kubernetes deployment (or retool your existing one) to achieve the resilience you require.
In Apache Cassandra Lunch #41: Apache Cassandra Lunch #41: Cassandra on Kubernetes - Docker/Kubernetes/Helm Part 1, we discuss Cassandra on Kubernetes and give an introduction to Docker, Kubernetes, and Helm.
Accompanying Blog: https://blog.anant.us/apache-cassandra-lunch-41-cassandra-on-kubernetes-docker-kubernetes-helm-part-1/
Accompanying YouTube: https://youtu.be/-I8cKQO_Qr0
Sign Up For Our Newsletter: http://eepurl.com/grdMkn
Join Cassandra Lunch Weekly at 12 PM EST Every Wednesday: https://www.meetup.com/Cassandra-DataStax-DC/events/
Cassandra.Link:
https://cassandra.link/
Follow Us and Reach Us At:
Anant:
https://www.anant.us/
Awesome Cassandra:
https://github.com/Anant/awesome-cassandra
Cassandra.Lunch:
https://github.com/Anant/Cassandra.Lunch
Email:
solutions@anant.us
LinkedIn:
https://www.linkedin.com/company/anant/
Twitter:
https://twitter.com/anantcorp
Eventbrite:
https://www.eventbrite.com/o/anant-1072927283
Facebook:
https://www.facebook.com/AnantCorp/
In this session, Kiran gives a talk about the rich ecosystem of tools (cmk, CAPC, Terraform, Ansible, Packer, csbench, mbx), that support Cloudstack.
Find out how the various tools work and how easy it is to integrate with Apache CloudStack.
This session provides a great way to speed up CloudStack adoption and improve performance by saving valuable time.
-----------------------------------------
The CloudStack India User Group 2024 took place in Hyderabad on 23rd February. The conference, arranged by a group of volunteers from the Apache CloudStack Community, saw multiple sessions held about the cloud orchestration platform and its latest advancements.
Running Apache Spark Jobs Using KubernetesDatabricks
Apache Spark has introduced a powerful engine for distributed data processing, providing unmatched capabilities to handle petabytes of data across multiple servers. Its capabilities and performance unseated other technologies in the Hadoop world, but while Spark provides a lot of power, it also comes with a high maintenance cost, which is why we now see innovations to simplify the Spark infrastructure.
Similar to A Primer Towards Running Kafka on Top of Kubernetes.pdf (20)
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
5. - More gluttony for torture
- Surprisingly simpler than
configuring
server.properties by hand
(or ansible)
- (if done well)
You want to run Kafka on K8S?
6. The Operator
Pattern in a
summary
- Kubernetes operator watches a CR type and takes application-specific actions to make the
current state match the desired state in that resource
- Implement domain-specific knowledge using Kubernetes
- Allows managing complex applications using the Kubernetes API and the kubectl interface
8. Scope of coverage:
A mental model on
Kubernetes
Operators for kafka
- Operator Core
- Custom Resources
- Workload Type
- Networking
- Storage
- Security
- Authentication
- Authorization
- Operational Features
- Balancing
- Monitoring
- Disaster Recovery
- Scale up/out
- Deployments & Rollouts
- Extensibility
9. Security: What is a typical requirement for kafka?
● Auto generate certificates for TLS and mTLS between brokers and other internal components
● Natively support authentication mechanism such as SASL/PLAIN, SASL/SCRAM,
SASL/OAUTHBEARER, SASL/GSSAPI
● Authorization with ACLs - Provide user management capabilities using the k8s API
10. Operations: What is a typical requirement for kafka?
● Re-balancing partitions when the load on the brokers is uneven, broker is added/removed
● Monitoring cluster health with JMX metrics
● Rolling upgrades with no downtime
● Replicate data across clusters
● Rack awareness for durability
11. Confluent For
Kubernetes(CFK)
● Confluent Platform on Kubernetes
● Based on experience of running Kafka on
Kubernetes for Confluent Cloud
● Uses StatefulSets for restoring a Kafka pod with
the same Kafka broker ID, configuration, and
persistent storage volumes if a failure occurs.
● Provides server properties, JVM, and Log4j
configuration overrides for customization of all
Confluent Platform components.
● Complete granular RBAC
● Support for credential management systems,
such as Hashicorp Vault, to inject sensitive
configurations in memory to Confluent
deployments
● Supports tiered storage
● Supports multi-region
12. Strimzi
● Open source, CNCF sandbox project
● Implement security in a Kubernetes-native
fashion
● Uses StrimziPodSets to overcome challenges of
StatefulSets
○ Add/remove broker arbitrarily
○ Stretch cluster across k8s clusters
○ Different configurations and volumes for different
brokers
● KafkaBridge for a RESTful HTTP interface
13. Koperator (Banzai
Cloud)
● Open-source core component of Banzai Cloud
Supertubes
○ most of the compelling features and integrations
are only available as part of the Supertubes Core
or Supertubes Pro product suites
● Envoy based load balancing for external access
● Uses pods instead of StatefulSets, in order to
○ modify the configuration of unique Brokers
○ remove specific Brokers from clusters
○ use multiple Persistent Volumes for each Broker
15. Prescriptive Advise
- As with all things, k8s: It is important to setup
resource constraints (CPU, MemLimits)
- Generally advised to have Kafka nodes tainted
to NoSchedule and run on a dedicated basis.
- = no binpack nodes
- For most real-life use-cases, CRs are a starting
point. Will need to be or packaged to “platform
recipes” with different components, orienting
some level of tenancy around the brokers as
well as the components
- Typically a higher order Helm chart, preferably
with GitOps style deployments
- Prospective users must also think about operator
tenancy itself. Could be a global operator or a
namespaced operator
16. Key Takeaways
- Running Kafka on K8S can be a lot of toil,
without an operator. If you are running Kafka at
scale (and not on a managed service), consider
running one. It will save you time, money &
sanity
- You can make a choice based on your
environment, features (or the lack thereof),
licensing and other specialized purposes
- YMMV with Operator CRs. Each operator has its
own opinion based on the realities it was
designed for
- Kafka is ultimately not “k8s native”. The operator
only provides so much operational sugar
- As a result, there are several shoehorning
mechanisms (such as config overrides to inject
component properties, builtin); Full expressivity of
the workload doesn’t quite exist
- All operators provide comparable performance
