Deploying a secured Flink cluster on Kubernetes
•Download as PPTX, PDF•
2 likes•194 views
One of the main sources of concerns when switching to the container paradigm is security. When dealing with big amounts of sensitive customer data it’s very important to be able to guarantee that the data is transported safely between the different components of the system. This talk will focus on how to setup a Flink cluster to run on a Kubernetes environment taking into account all security aspects to ensure secured communication between the nodes of the cluster, state backend and also for job submission, all taking advantage of Kubernetes tools.
Report
Share
Report
Share
Recommended
Deploying Flink on Kubernetes - David Anderson
Kubernetes has rapidly established itself as the de facto standard for orchestrating containerized infrastructures. And with the recent completion of the refactoring of Flink's deployment and process model known as FLIP-6, Kubernetes has become a natural choice for Flink deployments. In this talk we will walk through how to get Flink running on Kubernetes
Kubernetes and Istio
Istio Introduction
Setup
Shopping Portal Microservice Deployment
Canary Deployment
Routing Rules based on User Agent and Weight
Distributed Tracing
Visualizing Metrics
Sf bay area Kubernetes meetup dec8 2016 - deployment models
I talk about deploying complex, multi-layer applications in Kuberentes.
I describe how Kubernetes AppController project (https://github.com/Mirantis/k8s-AppController) can be leveraged to enhance such deployments
NetflixOSS for Triangle Devops Oct 2013
My @TriangleDevops talk from 2013-10-17. I covered the work that led us to @NetflixOSS (Acme Air), the work we did on the cloud prize (NetflixOSS on IBM SoftLayer/RightScale) and the @NetflixOSS platform (Karyon, Archaius, Eureka, Ribbon, Asgard, Hystrix, Turbine, Zuul, Servo, Edda, Ice, Denominator, Aminator, Janitor/Conformity/Chaos Monkeys of the Simian Army).
Gatekeeper: API gateway
In this WebHack talk I shared my experience about microservices, Docker, Kubernetes and Kong, an API gateway by Mashape. Since they are based on a real working system, this slides is majorly for how to build the whole thing up, not about detailed internal implementation. Although I included some details and reference in order to make it more comprehensive.
"[WORKSHOP] K8S for developers", Denis Romanuk
"It seems that perfection is attained, not when there is nothing more to add, but when there is nothing more to take away." - Antoine de Saint Exupéry
We can talk a lot about Kubernetes. But does the whole info is really need for just begining? Let's take away everything, except the really needed, and show a way for ones, who still uses only docker.
Let's consider:
Why do developers need Kubernetes and what does it looks like?
An attempt to automate ops, or why does it was a reinvention of Kubernetes
What does kube consists of?
How to get a Kubernetes?
From docker-compose to c
What is a Helm and why it's difficult without one?
What is "local development on Kubernetes"?
CRD & Operators
Docker Kubernetes Istio
Docker Kubernetes Istio
Understanding Docker and creating containers.
Container Orchestration based on Kubernetes
Blue Green Deployment, AB Testing, Canary Deployment, Traffic Rules based on Istio
Docker at Shopify: From This-Looks-Fun to Production by Simon Eskildsen (Shop...
Since July 2014 Shopify's been serving thousands of requests per second of production web traffic from Docker containers. This was an 8 month effort, with multiple pivots of direction from the team—and we're only getting started. This talk covers the lessons learned through the trial and error of an in-flight architecture redesign, spanning hundreds of hosts, as well as the technical vision of the future of our platform.
Recommended
Deploying Flink on Kubernetes - David Anderson
Kubernetes has rapidly established itself as the de facto standard for orchestrating containerized infrastructures. And with the recent completion of the refactoring of Flink's deployment and process model known as FLIP-6, Kubernetes has become a natural choice for Flink deployments. In this talk we will walk through how to get Flink running on Kubernetes
Kubernetes and Istio
Istio Introduction
Setup
Shopping Portal Microservice Deployment
Canary Deployment
Routing Rules based on User Agent and Weight
Distributed Tracing
Visualizing Metrics
Sf bay area Kubernetes meetup dec8 2016 - deployment models
I talk about deploying complex, multi-layer applications in Kuberentes.
I describe how Kubernetes AppController project (https://github.com/Mirantis/k8s-AppController) can be leveraged to enhance such deployments
NetflixOSS for Triangle Devops Oct 2013
My @TriangleDevops talk from 2013-10-17. I covered the work that led us to @NetflixOSS (Acme Air), the work we did on the cloud prize (NetflixOSS on IBM SoftLayer/RightScale) and the @NetflixOSS platform (Karyon, Archaius, Eureka, Ribbon, Asgard, Hystrix, Turbine, Zuul, Servo, Edda, Ice, Denominator, Aminator, Janitor/Conformity/Chaos Monkeys of the Simian Army).
Gatekeeper: API gateway
In this WebHack talk I shared my experience about microservices, Docker, Kubernetes and Kong, an API gateway by Mashape. Since they are based on a real working system, this slides is majorly for how to build the whole thing up, not about detailed internal implementation. Although I included some details and reference in order to make it more comprehensive.
"[WORKSHOP] K8S for developers", Denis Romanuk
"It seems that perfection is attained, not when there is nothing more to add, but when there is nothing more to take away." - Antoine de Saint Exupéry
We can talk a lot about Kubernetes. But does the whole info is really need for just begining? Let's take away everything, except the really needed, and show a way for ones, who still uses only docker.
Let's consider:
Why do developers need Kubernetes and what does it looks like?
An attempt to automate ops, or why does it was a reinvention of Kubernetes
What does kube consists of?
How to get a Kubernetes?
From docker-compose to c
What is a Helm and why it's difficult without one?
What is "local development on Kubernetes"?
CRD & Operators
Docker Kubernetes Istio
Docker Kubernetes Istio
Understanding Docker and creating containers.
Container Orchestration based on Kubernetes
Blue Green Deployment, AB Testing, Canary Deployment, Traffic Rules based on Istio
Docker at Shopify: From This-Looks-Fun to Production by Simon Eskildsen (Shop...
Since July 2014 Shopify's been serving thousands of requests per second of production web traffic from Docker containers. This was an 8 month effort, with multiple pivots of direction from the team—and we're only getting started. This talk covers the lessons learned through the trial and error of an in-flight architecture redesign, spanning hundreds of hosts, as well as the technical vision of the future of our platform.
Kubernetes Concepts And Architecture Powerpoint Presentation Slides
The document provides an overview of Kubernetes concepts and architecture. It begins with an introduction to containers and microservices architecture. It then discusses what Kubernetes is and why organizations should use it. The remainder of the document outlines Kubernetes components, nodes, development processes, networking, and security measures. It provides descriptions and diagrams explaining key aspects of Kubernetes such as architecture, components like Kubelet and Kubectl, node types, and networking models.
KubeCon EU 2016: Kubernetes meets Finagle for Resilient Microservices
Finagle is an open-source, high-volume RPC client library, handling millions of QPS at companies like Twitter, Pinterest and Soundcloud. In this talk, we demonstrate how Finagle can be applied to Kubernetes applications via linkerd, an open-source, standalone Finagle proxy. By deploying linkerd as a sidecar container or with DaemonSets, we show how polyglot multi-service applications running in Kubernetes can be “wrapped” in Finagle’s operational model, adding connection pooling, load-balancing, failure detection, and failover mechanisms to existing applications with minimal code change. We demonstrate how linkerd communicates with the Kubernetes API and how the resulting systems perform under load and adverse conditions.
Sched Link: http://sched.co/6BhW
Play Support in Cloud Foundry
- Play 2.0 is a web framework for Java and Scala that simplifies development by embracing HTTP rather than fighting it
- It takes a new approach to building web apps in Java by not being built on top of servlet APIs and using an asynchronous programming model
- Developing, testing, and deploying a Play app locally and to CloudFoundry involves creating a project, running it locally, and pushing the compiled code to CloudFoundry which automatically detects and supports Play apps
Kubernetes-Presentation-Syed-Murtaza-Hassan
Kubernetes is an open-source system and is quickly becoming the new standard for automating deployment, scaling, and management of containerized applications.
In the presentation we will have a high-level overview of the most important components of Kubernetes and how they fit together. We will start with having an overview of Container and Orchestration and what Kubernetes is capable of and how it helps in automating deployment and scaling software in the cloud. Afterwards we will discuss Kubernetes objects (Pod, ReplicaSet, Deployment, Services, Namespaces) with some examples.
Kubernetes persistence 101
In this meetup, Oleg, CTO at Kublr, walks you through the basics of K8s persistence management functionality and how it can be used to simplify managing persistent applications across different environments - in the cloud or on premise. Oleg will use a demo environment with clusters in different clouds to show K8s persistence in practice.
We will cover:
• Persistent data abstractions in K8s: persistent volumes (PV) and their attributes
• PV specifics in different clouds
• Using PV in K8s: persistent volume claims (PVC) and storage classes (SC)
• Automatic volume provisioning
• Persistence and scheduling interrelationships
• Practical examples
Kubernetes (K8s) is a powerful and flexible open source container orchestration system. The power of K8s comes from its modularity and simplicity of basic concepts. Each of these basic concepts build on the other and, from the most basic elements to more advanced ones, each is responsible for its own well-defined logic and behavior.
fabric8 ... and Docker, Kubernetes & OpenShift
This presentation gives an Overview about Kubernetes and OpenShift, their differences and what fabric8 (fabric8.io) adds to the mix
Kubernetes Ingress 101
This presentation explains the basics of Kubernetes ingress traffic management functionality, and how it can be used to simplify managing applications across different environments - in the cloud or on premise.
Kubernetes security
This document provides an overview of Kubernetes security concepts including the Kubernetes attack surface, TLS certificates, securing the Kubelet and etcd, authentication, authorization, admission controllers, tooling landscape, pod security policies, network policies, and secrets. It discusses key configuration recommendations and checks for securing different components like disabling privileged mode, setting TLS certificates, CIS benchmarks for etcd, and authentication methods in Kubernetes. The different types of admission controllers and CNCF security tooling are also briefly introduced.
Microservices, Kubernetes and Istio - A Great Fit!
Microservices and containers are now influencing application design and deployment patterns. Sixty percent of all new applications will use cloud-enabled continuous delivery microservice architectures and containers. Service discovery, registration, and routing are fundamental tenets of microservices. Kubernetes provides a platform for running microservices. Kubernetes can be used to automate the deployment of Microservices and leverage features such as Kube-DNS, Config Maps, and Ingress service for managing those microservices. This configuration works fine for deployments up to a certain size. However, with complex deployments consisting of a large fleet of microservices, additional features are required to augment Kubernetes.
Using Docker EE to Scale Operational Intelligence at Splunk
With more than 14,000 customers in 110+ countries, Splunk is the market leader in analyzing machine data to deliver operational intelligence for security, IT and the business. Our rapid growth as a company meant that our Infrastructure Engineering Team, responsible for all the common tooling, build and test systems and frameworks utilized by the Splunk engineers, was bogged down with a sprawl of virtual machines and physical servers that were becoming incredibly difficult to manage. And as our customer’s demand for data has grown, testing at the scale of petabytes/day has become our new normal. We needed a reliable and scalable “Test Lab” for functional and performance testing.
With Docker Enterprise Edition, our engineers are able to create small test stacks on their laptop just as easily as creating multi-petabyte stacks in our Test Lab. Support for Windows, Role Based Access Control and having support for both the orchestration platform and the container engine were key in deciding to go with Docker over other solutions.
In this talk, we will cover the architecture, tooling, and frameworks we built to manage our workloads, which have grown to run on over 600 bare-metal servers, with tens of thousands of containers being created every day. We will share the lessons learned from running at scale. Lastly, we will demonstrate how we use Splunk to monitor and manage Docker Enterprise Edition.
Kubernetes 101
This document provides an introduction and overview of Kubernetes for deploying and managing containerized applications at scale. It discusses Kubernetes' key features like self-healing, dynamic scaling, networking and efficient resource usage. It then demonstrates setting up a Kubernetes cluster on AWS and deploying a sample application using pods, deployments and services. While Kubernetes provides many benefits, the document notes it requires battle-testing to be production-ready and other topics like logging, monitoring and custom autoscaling solutions would need separate discussions.
The Mushroom Cloud Effect or What Happens When Containers Fail? by Alois Mayr...
This document discusses the "mushroom cloud effect" that can occur when containers fail in highly dynamic container environments. It describes how a failure in one container due to a lack of disk space on the host led to cascading failures that affected many dependent services. The failure spread as container health checks failed and orchestration rescheduled containers, eventually exhausting disk space and preventing any new containers from running. Automated monitoring is needed to pinpoint the root cause of such cascading failures in complex systems with many interdependent containers and services.
Clocker - The Docker Cloud Maker
This document provides an overview and demonstration of Clocker, an open source tool for managing Docker clouds and deploying composite applications on Docker. It discusses Clocker's components including its use of Brooklyn for application management and jclouds for provisioning. It also covers Clocker's features such as container placement strategies, networking using Weave, and roadmap items like support for Docker Swarm and improved networking.
Fabric8 - Being devOps doesn't suck anymore
Fabric8 is a tool that aims to reduce the gap between development and operations by allowing developers to deploy and manage applications from development through production. It uses profiles and containers to deploy applications in a unified way across environments. Fabric8 supports various containers like Tomcat, Docker, and OpenShift. It includes integrated monitoring via Hawt.io and a shell to help with scripting and diagnostics. The goal is to make developers more self-sufficient by enabling them to take on more operations tasks in their own sandboxes.
Devops - Microservice and Kubernetes
This document provides an overview of microservices and various methodologies for deploying microservices. It begins with an introduction comparing monolithic and microservice architectures. Next, it discusses considerations for microservice design like scalability and complexity. It then covers challenges of deploying microservices like monitoring and scaling. The document proceeds to introduce Kubernetes as a container orchestration tool for deploying microservices. It includes descriptions of various Kubernetes components like pods, deployments, services, and ingress. Finally, it demonstrates deploying microservices locally using Minikube to simulate a Kubernetes cluster on a single node.
Deep Dive into Kubernetes - Part 1
This presentation includes information on Kubernetes Architecture, Container Orchestration, Internal Routing, External Routing, Configuration Management, Credentials Management, Persistent Volumes, Rolling Out Updates, Autoscaling, Package Management, and a Hello World example using Helm.
Kubernetes Networking 101
Kubernetes (K8s) is a powerful, flexible and portable open source framework for distributed containerized applications delivery and management. An important part of the services provided by most Kubernetes clusters is the containers’ networking stack. In most cases and for many applications it “just works”, but this seeming simplicity is backed by a complex stack of technologies that provide many capabilities beyond the basics.
This presentation accompanies the meetup and webinar where Oleg Chunikhin, CTO at Kublr, shows how Kubernetes networking stack works, describes main components, interfaces and extensibility options.
What is covered:
- general notions of Kubernetes networking - Pods and Network Policies
- implementation of Kubernetes networking - CNI, CNI plugins, and Linux network namespaces
- some Kubernetes CNI providers: Calico, Weave, Flanel, and Canal
- K8S networking extensibility for advanced and “exotic” use-cases with Multus CNI plugin as an example
Scaling docker with kubernetes
Kubernetes is a container cluster manager that aims to provide a platform for automating deployment, scaling, and operations of application containers across clusters of machines. It uses pods as the basic building block, which are groups of application containers that share storage and networking resources. Kubernetes includes control planes for replication, scheduling, and services to expose applications. It supports deployment of multi-tier applications through replication controllers, services, labels, and pod templates.
Containers Docker Kind Kubernetes Istio
Building Cloud-Native App Series - Part 7 of 11
Microservices Architecture Series
Containers Docker Kind Kubernetes Istio
- Pods
- ReplicaSet
- Deployment (Canary, Blue-Green)
- Ingress
- Service
Enabling ceph-mgr to control Ceph services via Kubernetes
The document discusses enabling Ceph management services through Kubernetes using Rook and Ceph-mgr. Rook allows deploying Ceph in a containerized way on Kubernetes for simplified management. Ceph-mgr allows controlling Ceph services and integrating with Kubernetes through Rook. This provides multiple ways to consume Ceph based on needs, from simple storage with Rook to full control with Ceph tools. Upcoming improvements will reduce management complexity through automation.
(DVO305) Turbocharge YContinuous Deployment Pipeline with Containers
This document outlines best practices for using containers in a continuous delivery pipeline. It recommends using containers with tools like Docker, Docker Compose, Amazon ECS, Jenkins, and AWS CodePipeline to build, test, and deploy applications. The workflow involves developing code in a source code repository, building Docker images, running tests inside containers, and deploying containers to production using Amazon ECS and AWS services for automation and orchestration of the pipeline. Demo applications and architectures are presented to illustrate container-based continuous delivery.
AWS re:Invent 2016: Service Integration Delivery and Automation Using Amazon ...
Through a combination of Amazon ECS and open source technologies, customers are able to build portable CI/CD pipelines on AWS. As container based deployments become more complex, they require additional rigging for integration. In this session, we show how popular Apache products like Kakfa, Storm, and Zookeeper are being deployed on top of Amazon ECS. We hear from HERE, a provider of mapping data, technologies, and services to the automotive, consumer, and enterprise sectors about an approach that leverages Consul from Hashicorp and Amazon ECS clusters for short-cycle deployments and tag-based environment promotion.
More Related Content
What's hot
Kubernetes Concepts And Architecture Powerpoint Presentation Slides
The document provides an overview of Kubernetes concepts and architecture. It begins with an introduction to containers and microservices architecture. It then discusses what Kubernetes is and why organizations should use it. The remainder of the document outlines Kubernetes components, nodes, development processes, networking, and security measures. It provides descriptions and diagrams explaining key aspects of Kubernetes such as architecture, components like Kubelet and Kubectl, node types, and networking models.
KubeCon EU 2016: Kubernetes meets Finagle for Resilient Microservices
Finagle is an open-source, high-volume RPC client library, handling millions of QPS at companies like Twitter, Pinterest and Soundcloud. In this talk, we demonstrate how Finagle can be applied to Kubernetes applications via linkerd, an open-source, standalone Finagle proxy. By deploying linkerd as a sidecar container or with DaemonSets, we show how polyglot multi-service applications running in Kubernetes can be “wrapped” in Finagle’s operational model, adding connection pooling, load-balancing, failure detection, and failover mechanisms to existing applications with minimal code change. We demonstrate how linkerd communicates with the Kubernetes API and how the resulting systems perform under load and adverse conditions.
Sched Link: http://sched.co/6BhW
Play Support in Cloud Foundry
- Play 2.0 is a web framework for Java and Scala that simplifies development by embracing HTTP rather than fighting it
- It takes a new approach to building web apps in Java by not being built on top of servlet APIs and using an asynchronous programming model
- Developing, testing, and deploying a Play app locally and to CloudFoundry involves creating a project, running it locally, and pushing the compiled code to CloudFoundry which automatically detects and supports Play apps
Kubernetes-Presentation-Syed-Murtaza-Hassan
Kubernetes is an open-source system and is quickly becoming the new standard for automating deployment, scaling, and management of containerized applications.
In the presentation we will have a high-level overview of the most important components of Kubernetes and how they fit together. We will start with having an overview of Container and Orchestration and what Kubernetes is capable of and how it helps in automating deployment and scaling software in the cloud. Afterwards we will discuss Kubernetes objects (Pod, ReplicaSet, Deployment, Services, Namespaces) with some examples.
Kubernetes persistence 101
In this meetup, Oleg, CTO at Kublr, walks you through the basics of K8s persistence management functionality and how it can be used to simplify managing persistent applications across different environments - in the cloud or on premise. Oleg will use a demo environment with clusters in different clouds to show K8s persistence in practice.
We will cover:
• Persistent data abstractions in K8s: persistent volumes (PV) and their attributes
• PV specifics in different clouds
• Using PV in K8s: persistent volume claims (PVC) and storage classes (SC)
• Automatic volume provisioning
• Persistence and scheduling interrelationships
• Practical examples
Kubernetes (K8s) is a powerful and flexible open source container orchestration system. The power of K8s comes from its modularity and simplicity of basic concepts. Each of these basic concepts build on the other and, from the most basic elements to more advanced ones, each is responsible for its own well-defined logic and behavior.
fabric8 ... and Docker, Kubernetes & OpenShift
This presentation gives an Overview about Kubernetes and OpenShift, their differences and what fabric8 (fabric8.io) adds to the mix
Kubernetes Ingress 101
This presentation explains the basics of Kubernetes ingress traffic management functionality, and how it can be used to simplify managing applications across different environments - in the cloud or on premise.
Kubernetes security
This document provides an overview of Kubernetes security concepts including the Kubernetes attack surface, TLS certificates, securing the Kubelet and etcd, authentication, authorization, admission controllers, tooling landscape, pod security policies, network policies, and secrets. It discusses key configuration recommendations and checks for securing different components like disabling privileged mode, setting TLS certificates, CIS benchmarks for etcd, and authentication methods in Kubernetes. The different types of admission controllers and CNCF security tooling are also briefly introduced.
Microservices, Kubernetes and Istio - A Great Fit!
Microservices and containers are now influencing application design and deployment patterns. Sixty percent of all new applications will use cloud-enabled continuous delivery microservice architectures and containers. Service discovery, registration, and routing are fundamental tenets of microservices. Kubernetes provides a platform for running microservices. Kubernetes can be used to automate the deployment of Microservices and leverage features such as Kube-DNS, Config Maps, and Ingress service for managing those microservices. This configuration works fine for deployments up to a certain size. However, with complex deployments consisting of a large fleet of microservices, additional features are required to augment Kubernetes.
Using Docker EE to Scale Operational Intelligence at Splunk
With more than 14,000 customers in 110+ countries, Splunk is the market leader in analyzing machine data to deliver operational intelligence for security, IT and the business. Our rapid growth as a company meant that our Infrastructure Engineering Team, responsible for all the common tooling, build and test systems and frameworks utilized by the Splunk engineers, was bogged down with a sprawl of virtual machines and physical servers that were becoming incredibly difficult to manage. And as our customer’s demand for data has grown, testing at the scale of petabytes/day has become our new normal. We needed a reliable and scalable “Test Lab” for functional and performance testing.
With Docker Enterprise Edition, our engineers are able to create small test stacks on their laptop just as easily as creating multi-petabyte stacks in our Test Lab. Support for Windows, Role Based Access Control and having support for both the orchestration platform and the container engine were key in deciding to go with Docker over other solutions.
In this talk, we will cover the architecture, tooling, and frameworks we built to manage our workloads, which have grown to run on over 600 bare-metal servers, with tens of thousands of containers being created every day. We will share the lessons learned from running at scale. Lastly, we will demonstrate how we use Splunk to monitor and manage Docker Enterprise Edition.
Kubernetes 101
This document provides an introduction and overview of Kubernetes for deploying and managing containerized applications at scale. It discusses Kubernetes' key features like self-healing, dynamic scaling, networking and efficient resource usage. It then demonstrates setting up a Kubernetes cluster on AWS and deploying a sample application using pods, deployments and services. While Kubernetes provides many benefits, the document notes it requires battle-testing to be production-ready and other topics like logging, monitoring and custom autoscaling solutions would need separate discussions.
The Mushroom Cloud Effect or What Happens When Containers Fail? by Alois Mayr...
This document discusses the "mushroom cloud effect" that can occur when containers fail in highly dynamic container environments. It describes how a failure in one container due to a lack of disk space on the host led to cascading failures that affected many dependent services. The failure spread as container health checks failed and orchestration rescheduled containers, eventually exhausting disk space and preventing any new containers from running. Automated monitoring is needed to pinpoint the root cause of such cascading failures in complex systems with many interdependent containers and services.
Clocker - The Docker Cloud Maker
This document provides an overview and demonstration of Clocker, an open source tool for managing Docker clouds and deploying composite applications on Docker. It discusses Clocker's components including its use of Brooklyn for application management and jclouds for provisioning. It also covers Clocker's features such as container placement strategies, networking using Weave, and roadmap items like support for Docker Swarm and improved networking.
Fabric8 - Being devOps doesn't suck anymore
Fabric8 is a tool that aims to reduce the gap between development and operations by allowing developers to deploy and manage applications from development through production. It uses profiles and containers to deploy applications in a unified way across environments. Fabric8 supports various containers like Tomcat, Docker, and OpenShift. It includes integrated monitoring via Hawt.io and a shell to help with scripting and diagnostics. The goal is to make developers more self-sufficient by enabling them to take on more operations tasks in their own sandboxes.
Devops - Microservice and Kubernetes
This document provides an overview of microservices and various methodologies for deploying microservices. It begins with an introduction comparing monolithic and microservice architectures. Next, it discusses considerations for microservice design like scalability and complexity. It then covers challenges of deploying microservices like monitoring and scaling. The document proceeds to introduce Kubernetes as a container orchestration tool for deploying microservices. It includes descriptions of various Kubernetes components like pods, deployments, services, and ingress. Finally, it demonstrates deploying microservices locally using Minikube to simulate a Kubernetes cluster on a single node.
Deep Dive into Kubernetes - Part 1
This presentation includes information on Kubernetes Architecture, Container Orchestration, Internal Routing, External Routing, Configuration Management, Credentials Management, Persistent Volumes, Rolling Out Updates, Autoscaling, Package Management, and a Hello World example using Helm.
Kubernetes Networking 101
Kubernetes (K8s) is a powerful, flexible and portable open source framework for distributed containerized applications delivery and management. An important part of the services provided by most Kubernetes clusters is the containers’ networking stack. In most cases and for many applications it “just works”, but this seeming simplicity is backed by a complex stack of technologies that provide many capabilities beyond the basics.
This presentation accompanies the meetup and webinar where Oleg Chunikhin, CTO at Kublr, shows how Kubernetes networking stack works, describes main components, interfaces and extensibility options.
What is covered:
- general notions of Kubernetes networking - Pods and Network Policies
- implementation of Kubernetes networking - CNI, CNI plugins, and Linux network namespaces
- some Kubernetes CNI providers: Calico, Weave, Flanel, and Canal
- K8S networking extensibility for advanced and “exotic” use-cases with Multus CNI plugin as an example
Scaling docker with kubernetes
Kubernetes is a container cluster manager that aims to provide a platform for automating deployment, scaling, and operations of application containers across clusters of machines. It uses pods as the basic building block, which are groups of application containers that share storage and networking resources. Kubernetes includes control planes for replication, scheduling, and services to expose applications. It supports deployment of multi-tier applications through replication controllers, services, labels, and pod templates.
Containers Docker Kind Kubernetes Istio
Building Cloud-Native App Series - Part 7 of 11
Microservices Architecture Series
Containers Docker Kind Kubernetes Istio
- Pods
- ReplicaSet
- Deployment (Canary, Blue-Green)
- Ingress
- Service
Enabling ceph-mgr to control Ceph services via Kubernetes
The document discusses enabling Ceph management services through Kubernetes using Rook and Ceph-mgr. Rook allows deploying Ceph in a containerized way on Kubernetes for simplified management. Ceph-mgr allows controlling Ceph services and integrating with Kubernetes through Rook. This provides multiple ways to consume Ceph based on needs, from simple storage with Rook to full control with Ceph tools. Upcoming improvements will reduce management complexity through automation.
What's hot (20)
Kubernetes Concepts And Architecture Powerpoint Presentation Slides
Kubernetes Concepts And Architecture Powerpoint Presentation Slides
KubeCon EU 2016: Kubernetes meets Finagle for Resilient Microservices
KubeCon EU 2016: Kubernetes meets Finagle for Resilient Microservices
Microservices, Kubernetes and Istio - A Great Fit!
Microservices, Kubernetes and Istio - A Great Fit!
Using Docker EE to Scale Operational Intelligence at Splunk
Using Docker EE to Scale Operational Intelligence at Splunk
The Mushroom Cloud Effect or What Happens When Containers Fail? by Alois Mayr...
The Mushroom Cloud Effect or What Happens When Containers Fail? by Alois Mayr...
Enabling ceph-mgr to control Ceph services via Kubernetes
Enabling ceph-mgr to control Ceph services via Kubernetes
Similar to Deploying a secured Flink cluster on Kubernetes
(DVO305) Turbocharge YContinuous Deployment Pipeline with Containers
This document outlines best practices for using containers in a continuous delivery pipeline. It recommends using containers with tools like Docker, Docker Compose, Amazon ECS, Jenkins, and AWS CodePipeline to build, test, and deploy applications. The workflow involves developing code in a source code repository, building Docker images, running tests inside containers, and deploying containers to production using Amazon ECS and AWS services for automation and orchestration of the pipeline. Demo applications and architectures are presented to illustrate container-based continuous delivery.
AWS re:Invent 2016: Service Integration Delivery and Automation Using Amazon ...
Through a combination of Amazon ECS and open source technologies, customers are able to build portable CI/CD pipelines on AWS. As container based deployments become more complex, they require additional rigging for integration. In this session, we show how popular Apache products like Kakfa, Storm, and Zookeeper are being deployed on top of Amazon ECS. We hear from HERE, a provider of mapping data, technologies, and services to the automotive, consumer, and enterprise sectors about an approach that leverages Consul from Hashicorp and Amazon ECS clusters for short-cycle deployments and tag-based environment promotion.
Best Practices for Running Kafka on Docker Containers
Docker containers provide an ideal foundation for running Kafka-as-a-Service on-premises or in the public cloud. However, using Docker containers in production environments for Big Data workloads using Kafka poses some challenges – including container management, scheduling, network configuration and security, and performance.
In this session at Kafka Summit in August 2017, Nanda Vijyaydev of BlueData shared lessons learned from implementing Kafka-as-a-Service with Docker containers.
https://kafka-summit.org/sessions/kafka-service-docker-containers
ITB2019 Multi-language / multi-OS communication using RabbitMQ - Wil de Bruin
Our web hosting company is using many different services for managing e-mail, spam-filters, DNS, domain registrations, SSL registrations, ticket systems and more. Some of these services have well defined Web API’s, others can only be managed by simple command line scripts.
In this session we will explain how we tried to automate the various workflows by using a messaging system such as RabbitMQ for communication between our cfml based customer control panel and these services.
Multi-language/multi-OS communication using RabbitMQ
This document discusses using RabbitMQ for multi-language and multi-OS communication. It provides an overview of messaging concepts in RabbitMQ, describes a use case for a web hosting panel that requires communication between different systems, and demonstrates setting up producers and consumers in ColdFusion and other languages to send and receive messages via RabbitMQ. While ColdFusion RabbitMQ producers are straightforward, consuming messages can be challenging due to limitations in ColdFusion.
Continuous Delivery with Docker and Amazon ECS
- The document outlines best practices for using containers in continuous delivery solutions with Docker and Amazon ECS, including using containers to provide predictable environments, fast feedback, and quick feature rollouts.
- It demonstrates a development and deployment workflow using AWS CodeCommit for source control, Jenkins for builds and testing, and Amazon ECS for deployment, with CodePipeline automating the workflow.
- Live demos were presented showing building and deploying a Rails application using Docker Compose locally and on Amazon ECS, running tests with Jenkins, and integrating the stages with AWS CodePipeline.
컴퓨팅 서비스 업데이트 - EC2, ECS, Lambda (김상필) :: re:Invent re:Cap Webinar 2015
- The document summarizes updates to Amazon EC2, EC2 Container Service, and AWS Lambda computing services.
- For EC2, new X1 instances with over 100 vCPUs and 2 TB memory were announced for in-memory applications. New T2.nano instances and dedicated hosts were also mentioned.
- For ECS, a new container registry service was highlighted. Scheduler improvements and expanded Docker configuration options were noted.
- For Lambda, added support for Python, longer function durations, scheduled functions, and versioning were summarized.
Openstack days sv building highly available services using kubernetes (preso)
This document discusses Google Cloud Platform's Kubernetes and how it can be used to build highly available services. It provides an overview of Kubernetes concepts like pods, labels, replica sets, volumes, and services. It then describes how Kubernetes Cluster Federation allows deploying applications across multiple Kubernetes clusters for high availability, geographic scaling, and other benefits. It outlines how to create clusters, configure the federated control plane, add clusters to the federation, deploy federated services and backends, and perform cross-cluster service discovery.
Orchestrating Docker with Terraform and Consul by Mitchell Hashimoto
Terraform is a tool for building and safely iterating on infrastructure, while Consul provides service discovery, monitoring and orchestration. In this talk we discuss using Terraform and Consul together to build a Docker-based Service Oriented Architecture at scale. We use Consul to provide the runtime control plane for the datacenter, and Terraform is used to modify the underlying infrastructure to allow for elastic scalability.
How we scale DroneCi on demand
We are sharing our process of migrating to the container based DroneCI platform and our lessons learned when scaling it up for an active open source project like ownCloud. Our journey started with a static legacy CI system, which was gradually replaced with, at first, a static DroneCI infrastructure. Over the course of half a year, we further more migrated to a cloud provider in order to dynamically scale the CI system based on the build volume. The lessons learned during this journey, were transformed and contributed to the DroneCI project and resulted in the DroneCI autoscaler - which allows for automatic scaling of infrastructure resources with common cloud providers.
WebSphere and Docker
Presentation given to the UK WebSphere User Group on 24 April 2016 giving a recap and update on integration between WebSphere Application Server and Docker. It covers both Liberty and the traditional application server.
TurboCharge Your Continuous Delivery Pipeline with Containers - Pop-up Loft
This document discusses how to use containers to turbocharge a continuous delivery pipeline. It recommends using Docker and Docker Compose to define and run multi-container applications. Jenkins can be used to run tests directly in Docker containers. Amazon ECS and AWS Elastic Beanstalk can deploy containers across infrastructure. AWS CodePipeline can orchestrate the full software release workflow from source to deployment.
Automating Software Development Life Cycle - A DevOps Approach
The document discusses DevOps and provides an overview of the key concepts. It describes how DevOps aims to bring development, operations, and business teams together through automating processes, continuous monitoring, and breaking down silos between teams. The document then covers various DevOps tools and technologies like version control systems, build tools, configuration management, virtualization, and continuous integration/deployment practices.
Production Ready Containers from IBM and Docker
Containers are quickly becoming the default foundation for modern applications. As a public cloud provider, IBM has been an early champion of containers in the cloud and has built an enterprise ready container service as part of IBM Bluemix. IBM has a long heritage of supporting, contributing to, and building offerings on top of open technologies and IBM carries this commitment to the open development of container solutions by being an active/founding member of the Open Containers Initiative and Cloud Native Computing Foundation. In this session, we will explore the enduring commitment to open technology as well as the advantages of using a pure containers service where the user has access to total solution life cycle management through integration of lessons learned, cutting edge enhancements/development and end-to-end support on the user's underlying infrastructure.
We will explore topics such as exploiting bare metal servers, applying overlay networking to containers, ensuring isolation and security in a truly multi-tenant container environment and managing a global service deployment.
Building Bizweb Microservices with Docker
This document discusses using Docker to build and deploy microservices for a Bizweb application. It proposes using Docker Swarm mode for orchestration along with Netflix OSS components like Eureka, Zuul and Ribbon to provide service discovery, API gateway functionality and load balancing between microservices. Continuous integration is achieved using Jenkins to build Docker images from code commits, tag them and push to a private Docker registry. When code is committed, Jenkins triggers deployment of updated services to a Docker Swarm cluster for testing or production. This approach aims to reduce delivery time, simplify deployment and scaling of microservices.
AWS re:Invent 2016: Securing Container-Based Applications (CON402)
Containers have had an incredibly large adoption rate since Docker was launched, especially from the developer community, as it provides an easy way to package, ship, and run applications. Securing your container-based application is now becoming a critical issue as applications move from development into production. In this session, you learn ways to implement storing secrets, distributing AWS privileges using IAM roles, protecting your container-based applications with vulnerability scans of container images, and incorporating automated checks into your continuous delivery workflow.
AWS re:Invent 2016: Securing Container-Based Applications (CON402)
This document discusses securing container-based applications. It covers container and OS security best practices like using Linux namespaces and cgroups for isolation, reducing the container attack surface, and hardening container images. It also discusses securing the container lifecycle through vulnerability scanning, configuration governance with Amazon ECS, and using secrets management. Finally, it shows how to automate security deployments through the CI/CD pipeline and tools like CloudFormation and CodeDeploy.
Deploying on Kubernetes - An intro
So, you have developed your application, and have built containers for each of its components, now you need to deploy them in a container orchestration platform. You have heard about Kubernetes, but have yet to take the plunge and learn how to use it effectively. Say no more fam.
In this talk I will start by describing the Kubernetes resources, how they map to usual application components, and best practices on Kubernetes deployments.
Kubernetes: training micro-dragons for a serious battle
My second version of how to use kubernetes effectively as a platform for developing microservice applications.
20170831 - Greg Palmier: Terraform & AWS at Tempus
Terraform is used at Tempus to define AWS infrastructure as code using modules and blueprints. Common utilities like a shared VPC are defined for re-use, while environment-specific configurations are generated. Applications are deployed using modules that define resources like load balancers, auto-scaling groups, and databases, allowing standardized deployment across environments in about 5 minutes. Monitoring and automation is implemented through CloudWatch, ASGs, and tools like Jenkins and PagerDuty.
Similar to Deploying a secured Flink cluster on Kubernetes (20)
(DVO305) Turbocharge YContinuous Deployment Pipeline with Containers
(DVO305) Turbocharge YContinuous Deployment Pipeline with Containers
AWS re:Invent 2016: Service Integration Delivery and Automation Using Amazon ...
AWS re:Invent 2016: Service Integration Delivery and Automation Using Amazon ...
Best Practices for Running Kafka on Docker Containers
Best Practices for Running Kafka on Docker Containers
ITB2019 Multi-language / multi-OS communication using RabbitMQ - Wil de Bruin
ITB2019 Multi-language / multi-OS communication using RabbitMQ - Wil de Bruin
Multi-language/multi-OS communication using RabbitMQ
Multi-language/multi-OS communication using RabbitMQ
컴퓨팅 서비스 업데이트 - EC2, ECS, Lambda (김상필) :: re:Invent re:Cap Webinar 2015
컴퓨팅 서비스 업데이트 - EC2, ECS, Lambda (김상필) :: re:Invent re:Cap Webinar 2015
Openstack days sv building highly available services using kubernetes (preso)
Openstack days sv building highly available services using kubernetes (preso)
Orchestrating Docker with Terraform and Consul by Mitchell Hashimoto
Orchestrating Docker with Terraform and Consul by Mitchell Hashimoto
TurboCharge Your Continuous Delivery Pipeline with Containers - Pop-up Loft
TurboCharge Your Continuous Delivery Pipeline with Containers - Pop-up Loft
Automating Software Development Life Cycle - A DevOps Approach
Automating Software Development Life Cycle - A DevOps Approach
AWS re:Invent 2016: Securing Container-Based Applications (CON402)
AWS re:Invent 2016: Securing Container-Based Applications (CON402)
AWS re:Invent 2016: Securing Container-Based Applications (CON402)
AWS re:Invent 2016: Securing Container-Based Applications (CON402)
Kubernetes: training micro-dragons for a serious battle
Kubernetes: training micro-dragons for a serious battle
20170831 - Greg Palmier: Terraform & AWS at Tempus
20170831 - Greg Palmier: Terraform & AWS at Tempus
Recently uploaded
Webinar On-Demand: Using Flutter for Embedded
Flutter is a popular open source, cross-platform framework developed by Google. In this webinar we'll explore Flutter and its architecture, delve into the Flutter Embedder and Flutter’s Dart language, discover how to leverage Flutter for embedded device development, learn about Automotive Grade Linux (AGL) and its consortium and understand the rationale behind AGL's choice of Flutter for next-gen IVI systems. Don’t miss this opportunity to discover whether Flutter is right for your project.
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
Learn about the latest innovations in and around OpenUI5/SAPUI5: UI5 Tooling, UI5 linter, UI5 Web Components, Web Components Integration, UI5 2.x, UI5 GenAI.
Recording:
https://www.youtube.com/live/MSdGLG2zLy8?si=INxBHTqkwHhxV5Ta&t=0
Enums On Steroids - let's look at sealed classes !
These are slides for my session"Enums On Steroids - let's look at sealed classes !" - delivered among others, on Devoxx UK 2024 conference
Transforming Product Development using OnePlan To Boost Efficiency and Innova...
Ready to overcome challenges and drive innovation in your organization? Join us in our upcoming webinar where we discuss how to combat resource limitations, scope creep, and the difficulties of aligning your projects with strategic goals. Discover how OnePlan can revolutionize your product development processes, helping your team to innovate faster, manage resources more effectively, and deliver exceptional results.
Boost Your Savings with These Money Management Apps
A money management app can transform your financial life by tracking expenses, creating budgets, and setting financial goals. These apps offer features like real-time expense tracking, bill reminders, and personalized insights to help you save and manage money effectively. With a user-friendly interface, they simplify financial planning, making it easier to stay on top of your finances and achieve long-term financial stability.
Microservice Teams - How the cloud changes the way we work
A lot of technical challenges and complexity come with building a cloud-native and distributed architecture. The way we develop backend software has fundamentally changed in the last ten years. Managing a microservices architecture demands a lot of us to ensure observability and operational resiliency. But did you also change the way you run your development teams?
Sven will talk about Atlassian’s journey from a monolith to a multi-tenanted architecture and how it affected the way the engineering teams work. You will learn how we shifted to service ownership, moved to more autonomous teams (and its challenges), and established platform and enablement teams.
WMF 2024 - Unlocking the Future of Data Powering Next-Gen AI with Vector Data...
Vector databases are transforming how we handle data, allowing us to search through text, images, and audio by converting them into vectors. Today, we'll dive into the basics of this exciting technology and discuss its potential to revolutionize our next-generation AI applications. We'll examine typical uses for these databases and the essential tools
developers need. Plus, we'll zoom in on the advanced capabilities of vector search and semantic caching in Java, showcasing these through a live demo with Redis libraries. Get ready to see how these powerful tools can change the game!
How Can Hiring A Mobile App Development Company Help Your Business Grow?
ToXSL Technologies is an award-winning Mobile App Development Company in Dubai that helps businesses reshape their digital possibilities with custom app services. As a top app development company in Dubai, we offer highly engaging iOS & Android app solutions. https://rb.gy/necdnt
Modelling Up - DDDEurope 2024 - Amsterdam
What to do when you have a perfect model for your software but you are constrained by an imperfect business model?
This talk explores the challenges of bringing modelling rigour to the business and strategy levels, and talking to your non-technical counterparts in the process.
14 th Edition of International conference on computer vision
About the event
14th Edition of International conference on computer vision
Computer conferences organized by ScienceFather group. ScienceFather takes the privilege to invite speakers participants students delegates and exhibitors from across the globe to its International Conference on computer conferences to be held in the Various Beautiful cites of the world. computer conferences are a discussion of common Inventions-related issues and additionally trade information share proof thoughts and insight into advanced developments in the science inventions service system. New technology may create many materials and devices with a vast range of applications such as in Science medicine electronics biomaterials energy production and consumer products.
Nomination are Open!! Don't Miss it
Visit: computer.scifat.com
Award Nomination: https://x-i.me/ishnom
Conference Submission: https://x-i.me/anicon
For Enquiry: Computer@scifat.com
WWDC 2024 Keynote Review: For CocoaCoders Austin
Overview of WWDC 2024 Keynote Address.
Covers: Apple Intelligence, iOS18, macOS Sequoia, iPadOS, watchOS, visionOS, and Apple TV+.
Understandable dialogue on Apple TV+
On-device app controlling AI.
Access to ChatGPT with a guest appearance by Chief Data Thief Sam Altman!
App Locking! iPhone Mirroring! And a Calculator!!
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
The UI5 tooling is the development and build tooling of UI5. It is built in a modular and extensible way so that it can be easily extended by your needs. This session will showcase various tooling extensions which can boost your development experience by far so that you can really work offline, transpile your code in your project to use even newer versions of EcmaScript (than 2022 which is supported right now by the UI5 tooling), consume any npm package of your choice in your project, using different kind of proxies, and even stitching UI5 projects during development together to mimic your target environment.
The Rising Future of CPaaS in the Middle East 2024
Explore "The Rising Future of CPaaS in the Middle East in 2024" with this comprehensive PPT presentation. Discover how Communication Platforms as a Service (CPaaS) is transforming communication across various sectors in the Middle East.
What is Continuous Testing in DevOps - A Definitive Guide.pdf
Once an overlooked aspect, continuous testing has become indispensable for enterprises striving to accelerate application delivery and reduce business impacts. According to a Statista report, 31.3% of global enterprises have embraced continuous integration and deployment within their DevOps, signaling a pervasive trend toward hastening release cycles.
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
Charla impartida en el evento de "KuberTENes Birthday Bash Guadalajara" para celebrar el 10mo. aniversario de Kubernetes #kuberTENes #celebr8k8s #k8s
ppt on the brain chip neuralink.pptx
A neural network is a machine learning program, or model, that makes decisions in a manner similar to the human brain, by using processes that mimic the way biological neurons work together to identify phenomena, weigh options and arrive at conclusions.
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM 🐰
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
IBM watsonx Code Assistant for Z, our latest Generative AI-assisted mainframe application modernization solution. Mainframe (IBM Z) application modernization is a topic that every mainframe client is addressing to various degrees today, driven largely from digital transformation. With generative AI comes the opportunity to reimagine the mainframe application modernization experience. Infusing generative AI will enable speed and trust, help de-risk, and lower total costs associated with heavy-lifting application modernization initiatives. This document provides an overview of the IBM watsonx Code Assistant for Z which uses the power of generative AI to make it easier for developers to selectively modernize COBOL business services while maintaining mainframe qualities of service.
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0
Recently uploaded (20)
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
Enums On Steroids - let's look at sealed classes !
Enums On Steroids - let's look at sealed classes !
Transforming Product Development using OnePlan To Boost Efficiency and Innova...
Transforming Product Development using OnePlan To Boost Efficiency and Innova...
Boost Your Savings with These Money Management Apps
Boost Your Savings with These Money Management Apps
Microservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we work
WMF 2024 - Unlocking the Future of Data Powering Next-Gen AI with Vector Data...
WMF 2024 - Unlocking the Future of Data Powering Next-Gen AI with Vector Data...
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
The Rising Future of CPaaS in the Middle East 2024
The Rising Future of CPaaS in the Middle East 2024
What is Continuous Testing in DevOps - A Definitive Guide.pdf
What is Continuous Testing in DevOps - A Definitive Guide.pdf
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
Deploying a secured Flink cluster on Kubernetes
- 1. Deploying a secured Flink cluster on Kubernetes Edward Rojas Clavijo Software Engineer IBM France
- 2. Use case • Multiple sources of data => Kafka • Different storage targets • Multiple environments • Sensitive customer data => Secured • Highly available • Fault tolerant
- 3. Zookeeper Job Manager HA Task Manager Task Manager Task Manager Task Manager Task Manager Storage checkpoints / savepoints Flink Client Flink Client Flink Client Submit job Flink Architecture – use case 1.5.3
- 4. Flink on Kubernetes JM Service TM Flink Client TM TM JM JM Deployment TM Deployment Pods • Job Manager Deployment • Maintain 1 replica • Job Manager Service • Task Manager Deployment • Maintain n replicas • One Pod per Flink Job • Fixed amount of jobs • Endless stateful jobs • ConfigMap for flink conf
- 5. Flink SSL/TLS • Certificates generation • Dynamic IP • TM Dynamic hostnames JM Service TM Flink Client TM TM JM JM Deployment TM Deployment Pods
- 6. Flink SSL/TLS • Hostname validation • Task managers => Statefulset • Predictable hostnames • flink-tm-0.flink-tm-svc.<namespace>.svc.cluster.local • flink-tm-1.flink-tm-svc.<namespace>.svc.cluster.local JM Service TM Flink Client TM TM JM JM Deployment TM Statefulset Pods TM Service TM service • Wildcard certificate
- 7. Flink SSL/TLS • One single purpose certificate • Truststore and Keystore with only that certificate • Deactivate hostname validation Expose as secrets Mount on Pods base64 encoded
- 9. Sources and Sinks - Kerberos • Keytab file exposed as k8s secret • Configuration properties on ConfigMap => env vars • Realm, Principal, KDC • Dockerfile • COPY krb5.conf template – /etc/krb5.conf - set access rights • COPY core-site.xml template • Docker entrypoint: • Set krb5.conf • Flag for each service • Set flink conf properties • Set hadoop conf • Kafka consumer/producer • sasl.kerberos.service.name • security.protocol => SASL_PLAINTEXT
- 10. Sources and Sinks - SSL • Expose certificates as secrets => mount on pods • Dockerfile • Set access rights to $JAVA_HOME/lib/security/cacerts • Docker entrypoint: • Flag for each service • Import certificate into pods trustore keytool -importcert -file ${CERT_FILE} -alias ${certName} -keystore $JAVA_HOME/lib/security/cacerts - noprompt -storepass changeit
- 11. Sources and Sinks - SSL • Elasticsearch 6.x => REST • ElasticsearchSink – only Flink 1.6 => Custom ElasticsearchSink Flink 1.6 =>
- 12. Checkpoints / Savepoints • Persistent volume • NFS • Encryption in-flight and at-rest • Encrypt Job state => Custom State Serializer PV TM TM TM JM
- 13. What next? • Enforce with k8s security config • Use RBAC • Restrict access to kubectl • Use Network Policy • Pod security policy • Single job cluster -> Flink 1.6
- 14. Thanks for your attention! Edward Rojas Clavijo
Editor's Notes
- Generic