1) The document describes an Azure Resource Manager (ARM) template for deploying OpenShift Enterprise on Azure. It provisions masters, infra nodes, and worker nodes with load balancing and storage.
2) The ARM template automates the entire deployment process through nested templates for each resource and Bash scripts for configuration. It handles naming, load balancing, storage, networking, and more.
3) The goal is to create a production-ready reference architecture for OpenShift on Azure and automate the deployment process through the ARM template. Current work focuses on deployment, storage, authentication, and documentation. Future work includes additional features and integrations.
This document provides an introduction to Kubernetes including:
- What Kubernetes is and what it does including abstracting infrastructure, providing self-healing capabilities, and providing a uniform interface across clouds.
- Key concepts including pods, services, labels, selectors, and namespaces. Pods are the atomic unit and services provide a unified access method. Labels and selectors are used to identify and group related objects.
- The Kubernetes architecture including control plane components like kube-apiserver, etcd, and kube-controller-manager. Node components include kubelet and kube-proxy. Optional services like cloud-controller-manager and cluster DNS are also described.
Java one kubernetes, jenkins and microservicesChristian Posta
This document discusses microservices with Docker, Kubernetes and Jenkins. It provides an overview of Kubernetes concepts like pods, replication controllers, services and labels. It also discusses how Kubernetes can help manage containers across multiple hosts and address challenges of scaling, avoiding port conflicts and keeping containers running. The document promotes using Jenkins and Kubernetes for continuous integration and delivery of containerized microservices applications. It recommends Fabric8 as a tool that can help create and deploy microservices on Kubernetes.
Slides from the talk given to the Startup Berlin Slack Group that demonstrates how TruckIN is implementing its continuous delivery workflow using technologies and open-source tools.
Topics that are covered: Automated Cloud Provisioning (Network, Subnets, VMs, Kubernetes Cluster, Firewall, Disks, Credentials, Private Docker Registry); Configuration Management (Salt Stack), Continuous Integration (Jenkins CI), Continuous Delivery/Deployment (Salt API/Reactor + Kubernetes) to a Google Cloud Kubernetes Cluster, Remote Application Debugging, Managing Google Cloud Kubernetes Cluster, Logging, Monitoring and ChatOps (Slack and operable.io)
OpenStack on Kubernetes (BOS Summit / May 2017 update)rhirschfeld
This document discusses using Kubernetes as an underlay platform for OpenStack. Some key points:
1. Kubernetes is becoming more widely used and understood by operators compared to OpenStack. Using Kubernetes as an underlay could improve simplicity, stability, and upgrade processes for OpenStack.
2. There are still many technical challenges to address, such as networking, storage, tooling to manage OpenStack on Kubernetes, and ensuring containers meet Kubernetes' immutable infrastructure requirements.
3. Using Kubernetes as an underlay risks further confusing the messaging around OpenStack by implying Kubernetes is more stable or a replacement target. Clear communication will be important to avoid undermining OpenStack.
Helm is a package manager for Kubernetes that allows for easy installation, upgrade, and management of Kubernetes applications. It provides repeatability, reliability, and simplifies deploying applications across multiple Kubernetes environments. Helm originated from an internal hackathon at Deis and was jointly developed by Google and Deis. It is now maintained by the Cloud Native Computing Foundation. Helm consists of a client that interacts with the Tiller server running inside the Kubernetes cluster to manage application lifecycles using charts, which are packages containing Kubernetes resource definitions.
Learn from the dozens of large-scale deployments how to get the most out of your Kubernetes environment:
- Container images optimization
- Organizing namespaces
- Readiness and Liveness probes
- Resource requests and limits
- Failing with grace
- Mapping external services
- Upgrading clusters with zero downtime
1) The document describes an Azure Resource Manager (ARM) template for deploying OpenShift Enterprise on Azure. It provisions masters, infra nodes, and worker nodes with load balancing and storage.
2) The ARM template automates the entire deployment process through nested templates for each resource and Bash scripts for configuration. It handles naming, load balancing, storage, networking, and more.
3) The goal is to create a production-ready reference architecture for OpenShift on Azure and automate the deployment process through the ARM template. Current work focuses on deployment, storage, authentication, and documentation. Future work includes additional features and integrations.
This document provides an introduction to Kubernetes including:
- What Kubernetes is and what it does including abstracting infrastructure, providing self-healing capabilities, and providing a uniform interface across clouds.
- Key concepts including pods, services, labels, selectors, and namespaces. Pods are the atomic unit and services provide a unified access method. Labels and selectors are used to identify and group related objects.
- The Kubernetes architecture including control plane components like kube-apiserver, etcd, and kube-controller-manager. Node components include kubelet and kube-proxy. Optional services like cloud-controller-manager and cluster DNS are also described.
Java one kubernetes, jenkins and microservicesChristian Posta
This document discusses microservices with Docker, Kubernetes and Jenkins. It provides an overview of Kubernetes concepts like pods, replication controllers, services and labels. It also discusses how Kubernetes can help manage containers across multiple hosts and address challenges of scaling, avoiding port conflicts and keeping containers running. The document promotes using Jenkins and Kubernetes for continuous integration and delivery of containerized microservices applications. It recommends Fabric8 as a tool that can help create and deploy microservices on Kubernetes.
Slides from the talk given to the Startup Berlin Slack Group that demonstrates how TruckIN is implementing its continuous delivery workflow using technologies and open-source tools.
Topics that are covered: Automated Cloud Provisioning (Network, Subnets, VMs, Kubernetes Cluster, Firewall, Disks, Credentials, Private Docker Registry); Configuration Management (Salt Stack), Continuous Integration (Jenkins CI), Continuous Delivery/Deployment (Salt API/Reactor + Kubernetes) to a Google Cloud Kubernetes Cluster, Remote Application Debugging, Managing Google Cloud Kubernetes Cluster, Logging, Monitoring and ChatOps (Slack and operable.io)
OpenStack on Kubernetes (BOS Summit / May 2017 update)rhirschfeld
This document discusses using Kubernetes as an underlay platform for OpenStack. Some key points:
1. Kubernetes is becoming more widely used and understood by operators compared to OpenStack. Using Kubernetes as an underlay could improve simplicity, stability, and upgrade processes for OpenStack.
2. There are still many technical challenges to address, such as networking, storage, tooling to manage OpenStack on Kubernetes, and ensuring containers meet Kubernetes' immutable infrastructure requirements.
3. Using Kubernetes as an underlay risks further confusing the messaging around OpenStack by implying Kubernetes is more stable or a replacement target. Clear communication will be important to avoid undermining OpenStack.
Helm is a package manager for Kubernetes that allows for easy installation, upgrade, and management of Kubernetes applications. It provides repeatability, reliability, and simplifies deploying applications across multiple Kubernetes environments. Helm originated from an internal hackathon at Deis and was jointly developed by Google and Deis. It is now maintained by the Cloud Native Computing Foundation. Helm consists of a client that interacts with the Tiller server running inside the Kubernetes cluster to manage application lifecycles using charts, which are packages containing Kubernetes resource definitions.
Learn from the dozens of large-scale deployments how to get the most out of your Kubernetes environment:
- Container images optimization
- Organizing namespaces
- Readiness and Liveness probes
- Resource requests and limits
- Failing with grace
- Mapping external services
- Upgrading clusters with zero downtime
GCP - Continuous Integration and Delivery into Kubernetes with GitHub, Travis...Oleg Shalygin
Kubernetes provides an automated platform to deployment, scaling and operations of applications across a cluster of hosts. Complementing Kubernetes with a series of build scripts in conjunction with Travis-CI, GitHub, Artifactory, and Google Cloud Platform, we can take code from a merged pull request to a deployed environment with no manual intervention on a highly scaleable and robust infrastructure.
This document provides an overview of Cloud Spanner including:
1. What Cloud Spanner is and how it compares to other database offerings.
2. Key product highlights such as it being fully managed, providing relational database capabilities at massive scale with strong consistency, and high availability.
3. Common use cases such as user data, order management, and electronic medical records.
4. Details on Spanner's architecture including splits, TrueTime, reads/writes, and Paxos.
5. Current areas of focus such as new features, developer productivity, and growing the open source ecosystem.
OpenShift is Red Hat's container application platform that provides a full-stack platform for deploying and managing containerized applications. It is based on Docker and Kubernetes and provides additional capabilities for self-service, automation, multi-language support, and enterprise features like authentication, centralized logging, and integration with Red Hat's JBoss middleware. OpenShift handles building, deploying, and scaling applications in a clustered environment with capabilities for continuous integration/delivery, persistent storage, routing, and monitoring.
We are on the cusp of a new era of application development software: instead of bolting on operations as an after-thought to the software development process, Kubernetes promises to bring development and operations together by design.
Kubernetes and Cloud Native Update Q4 2018CloudOps2005
This year’s final set of Kubernetes and Cloud Native meetups just took place. They kicked off in Kitchener-Waterloo on November 29th, and continued in Montreal December 3rd, Ottawa December 4th, Toronto December 5th, and Quebec December 6th. In preparation for the upcoming KubeCon and CloudNativeCon in Seattle, a wide range of open source solutions were discussed and, as always, beer and pizza provided. Ayrat Khayretdinov began each meetup with an update of Kubernetes and the Cloud Native landscape.
Presentation delivered at LinuxCon China 2017.
Zephyr is an upstream open source project for places where Linux is too big to fit. This talk will overview the progress we've made in the first year towards the projects goals around incorporating best of breed technologies into the code base, and building up the community to support multiple architectures and development environments. We will share our roadmap, plans and the challenges ahead of the us and give an overview of the major technical challenges we want to tackle in 2017.
Kubernetes is an open source container cluster orchestration platform founded by Google. This presentation covers an overview of it's main concepts, plus how it fits into Google Cloud Platform. This was delivered by Kit Merker at DevNexus 2015 in Atlanta.
Learn about the challenges the come with deploying and operating Kubernetes at scale and how the Mesosphere DC/OS Kubernetes integration helps solve them.
During this presentation, Joerg Schad discusses:
1. Common challenges associated with getting a Kubernetes cluster up and running
2. The basics of running Kubernetes on Mesosphere DC/OS
3. How failure recovery works with the DC/OS-Kubernetes solution
Kubernetes is a fast-paced project and things move really fast. In deploying applications, you have several options like raw YAML files, Helm, or Operator but what are the pros and cons of each?
This talk will explore the right ways to manage your production applications through seamless installation, the patch fixes, and upgrades. Several demos will be used on a live cluster to illustrate how things can be done the right way that makes life very easy for the DevOps.
Top 3 reasons why you should run your Enterprise workloads on GKESreenivas Makam
This deck covers top 3 reasons why Google Kubernetes engine is best suited to run containerized workloads. The reasons covered are Security, Observability and Maturity.
The Operator Pattern - Managing Stateful Services in KubernetesQAware GmbH
Cloud Native Night, January 2018, Mainz: Talk by Jakob Karalus (@krallistic, IT Consultant at codecentric)
Join our Meetup: https://www.meetup.com/de-DE/Cloud-Native-Night
Abstract: While it's easy to deploy stateless application with Kubernetes, it's harder for stateful software. Since applications often require custom functionality that Kubernetes can't provide, developers want to add more specialized patterns like automatic backups, failover or rebalancing to their Kubernetes deployments. In this talk, we will look at the Operator Pattern and other possibilities to extend the functionality of Kubernetes and how to use them to operate stateful applications.
** Kubernetes Certification Training: https://www.edureka.co/kubernetes-certification **
This Edureka tutorial on "Kubernetes Architecture" will give you an introduction to popular DevOps tool - Kubernetes, and will deep dive into Kubernetes Architecture and its working. The following topics are covered in this training session:
1. What is Kubernetes
2. Features of Kubernetes
3. Kubernetes Architecture and Its Components
4. Components of Master Node and Worker Node
5. ETCD
6. Network Setup Requirements
DevOps Tutorial Blog Series: https://goo.gl/P0zAfF
Building Clustered Applications with Kubernetes and DockerSteve Watt
This document discusses building clustered applications with Kubernetes and Docker. It provides an overview of Kubernetes, including its architecture and components. It then demonstrates how to install Kubernetes, define and deploy pods, add replication controllers and services. It discusses using volumes for persistence, including different volume types like GlusterFS. Finally, it touches on debugging and provides contact information for following up.
Kubecon US 2019: Kubernetes Multitenancy WG Deep DiveSanjeev Rampal
This document provides an overview and agenda for a presentation on secure multitenancy in Kubernetes. It discusses what Kubernetes multitenancy is, available solutions, architectural models for multitenancy including namespace grouping and virtual Kubernetes clusters. It also covers community initiatives for multitenancy control plane including tenant controllers and hierarchical namespaces. The document outlines benchmarking categories and a proposed baseline reference implementation for multitenancy including control plane, data plane, and network isolation techniques.
Enabling ceph-mgr to control Ceph services via Kubernetesmountpoint.io
The document discusses enabling Ceph management services through Kubernetes using Rook and Ceph-mgr. Rook allows deploying Ceph in a containerized way on Kubernetes for simplified management. Ceph-mgr allows controlling Ceph services and integrating with Kubernetes through Rook. This provides multiple ways to consume Ceph based on needs, from simple storage with Rook to full control with Ceph tools. Upcoming improvements will reduce management complexity through automation.
This document discusses various methods for accessing Kubernetes pods including through API server proxies, port forwarding, and logs/attach. It begins with an overview of how pods expose access points via kubectl. Examples are provided for using kubectl proxy to access pod endpoints, port forwarding to proxy local ports to pods, and retrieving logs and attaching to pods' stdin/stdout/stderr. The raw kubectl option and debugging with increased log levels are also covered.
KubeCon CloudNativeCon Seattle 2019 Recap - General overview and also summary of some of the application deployment track (App sig, Operator Framework, Helm, Kustomize, CNAB).
[Spark Summit 2017 NA] Apache Spark on KubernetesTimothy Chen
This document summarizes a presentation about running Apache Spark on Kubernetes. It discusses how Spark jobs can be scheduled and run on Kubernetes, including scheduling the driver and executor pods. Key points of the design include the Kubernetes scheduler backend for Spark and components like the file staging server. The roadmap outlines upcoming support for features like Spark Streaming and improvements to dynamic allocation.
Apache Spark on Kubernetes Anirudh Ramanathan and Tim ChenDatabricks
Kubernetes is a fast growing open-source platform which provides container-centric infrastructure. Conceived by Google in 2014, and leveraging over a decade of experience running containers at scale internally, it is one of the fastest moving projects on GitHub with 1000+ contributors and 40,000+ commits. Kubernetes has first class support on Google Cloud Platform, Amazon Web Services, and Microsoft Azure.
Unlike YARN, Kubernetes started as a general purpose orchestration framework with a focus on serving jobs. Support for long-running, data intensive batch workloads required some careful design decisions. Engineers across several organizations have been working on Kubernetes support as a cluster scheduler backend within Spark. During this process, we encountered several challenges in translating Spark considerations into idiomatic Kubernetes constructs. In this talk, we describe the challenges and the ways in which we solved them. This talk will be technical and is aimed at people who are looking to run Spark effectively on their clusters. The talk assumes basic familiarity with cluster orchestration and containers.
GCP - Continuous Integration and Delivery into Kubernetes with GitHub, Travis...Oleg Shalygin
Kubernetes provides an automated platform to deployment, scaling and operations of applications across a cluster of hosts. Complementing Kubernetes with a series of build scripts in conjunction with Travis-CI, GitHub, Artifactory, and Google Cloud Platform, we can take code from a merged pull request to a deployed environment with no manual intervention on a highly scaleable and robust infrastructure.
This document provides an overview of Cloud Spanner including:
1. What Cloud Spanner is and how it compares to other database offerings.
2. Key product highlights such as it being fully managed, providing relational database capabilities at massive scale with strong consistency, and high availability.
3. Common use cases such as user data, order management, and electronic medical records.
4. Details on Spanner's architecture including splits, TrueTime, reads/writes, and Paxos.
5. Current areas of focus such as new features, developer productivity, and growing the open source ecosystem.
OpenShift is Red Hat's container application platform that provides a full-stack platform for deploying and managing containerized applications. It is based on Docker and Kubernetes and provides additional capabilities for self-service, automation, multi-language support, and enterprise features like authentication, centralized logging, and integration with Red Hat's JBoss middleware. OpenShift handles building, deploying, and scaling applications in a clustered environment with capabilities for continuous integration/delivery, persistent storage, routing, and monitoring.
We are on the cusp of a new era of application development software: instead of bolting on operations as an after-thought to the software development process, Kubernetes promises to bring development and operations together by design.
Kubernetes and Cloud Native Update Q4 2018CloudOps2005
This year’s final set of Kubernetes and Cloud Native meetups just took place. They kicked off in Kitchener-Waterloo on November 29th, and continued in Montreal December 3rd, Ottawa December 4th, Toronto December 5th, and Quebec December 6th. In preparation for the upcoming KubeCon and CloudNativeCon in Seattle, a wide range of open source solutions were discussed and, as always, beer and pizza provided. Ayrat Khayretdinov began each meetup with an update of Kubernetes and the Cloud Native landscape.
Presentation delivered at LinuxCon China 2017.
Zephyr is an upstream open source project for places where Linux is too big to fit. This talk will overview the progress we've made in the first year towards the projects goals around incorporating best of breed technologies into the code base, and building up the community to support multiple architectures and development environments. We will share our roadmap, plans and the challenges ahead of the us and give an overview of the major technical challenges we want to tackle in 2017.
Kubernetes is an open source container cluster orchestration platform founded by Google. This presentation covers an overview of it's main concepts, plus how it fits into Google Cloud Platform. This was delivered by Kit Merker at DevNexus 2015 in Atlanta.
Learn about the challenges the come with deploying and operating Kubernetes at scale and how the Mesosphere DC/OS Kubernetes integration helps solve them.
During this presentation, Joerg Schad discusses:
1. Common challenges associated with getting a Kubernetes cluster up and running
2. The basics of running Kubernetes on Mesosphere DC/OS
3. How failure recovery works with the DC/OS-Kubernetes solution
Kubernetes is a fast-paced project and things move really fast. In deploying applications, you have several options like raw YAML files, Helm, or Operator but what are the pros and cons of each?
This talk will explore the right ways to manage your production applications through seamless installation, the patch fixes, and upgrades. Several demos will be used on a live cluster to illustrate how things can be done the right way that makes life very easy for the DevOps.
Top 3 reasons why you should run your Enterprise workloads on GKESreenivas Makam
This deck covers top 3 reasons why Google Kubernetes engine is best suited to run containerized workloads. The reasons covered are Security, Observability and Maturity.
The Operator Pattern - Managing Stateful Services in KubernetesQAware GmbH
Cloud Native Night, January 2018, Mainz: Talk by Jakob Karalus (@krallistic, IT Consultant at codecentric)
Join our Meetup: https://www.meetup.com/de-DE/Cloud-Native-Night
Abstract: While it's easy to deploy stateless application with Kubernetes, it's harder for stateful software. Since applications often require custom functionality that Kubernetes can't provide, developers want to add more specialized patterns like automatic backups, failover or rebalancing to their Kubernetes deployments. In this talk, we will look at the Operator Pattern and other possibilities to extend the functionality of Kubernetes and how to use them to operate stateful applications.
** Kubernetes Certification Training: https://www.edureka.co/kubernetes-certification **
This Edureka tutorial on "Kubernetes Architecture" will give you an introduction to popular DevOps tool - Kubernetes, and will deep dive into Kubernetes Architecture and its working. The following topics are covered in this training session:
1. What is Kubernetes
2. Features of Kubernetes
3. Kubernetes Architecture and Its Components
4. Components of Master Node and Worker Node
5. ETCD
6. Network Setup Requirements
DevOps Tutorial Blog Series: https://goo.gl/P0zAfF
Building Clustered Applications with Kubernetes and DockerSteve Watt
This document discusses building clustered applications with Kubernetes and Docker. It provides an overview of Kubernetes, including its architecture and components. It then demonstrates how to install Kubernetes, define and deploy pods, add replication controllers and services. It discusses using volumes for persistence, including different volume types like GlusterFS. Finally, it touches on debugging and provides contact information for following up.
Kubecon US 2019: Kubernetes Multitenancy WG Deep DiveSanjeev Rampal
This document provides an overview and agenda for a presentation on secure multitenancy in Kubernetes. It discusses what Kubernetes multitenancy is, available solutions, architectural models for multitenancy including namespace grouping and virtual Kubernetes clusters. It also covers community initiatives for multitenancy control plane including tenant controllers and hierarchical namespaces. The document outlines benchmarking categories and a proposed baseline reference implementation for multitenancy including control plane, data plane, and network isolation techniques.
Enabling ceph-mgr to control Ceph services via Kubernetesmountpoint.io
The document discusses enabling Ceph management services through Kubernetes using Rook and Ceph-mgr. Rook allows deploying Ceph in a containerized way on Kubernetes for simplified management. Ceph-mgr allows controlling Ceph services and integrating with Kubernetes through Rook. This provides multiple ways to consume Ceph based on needs, from simple storage with Rook to full control with Ceph tools. Upcoming improvements will reduce management complexity through automation.
This document discusses various methods for accessing Kubernetes pods including through API server proxies, port forwarding, and logs/attach. It begins with an overview of how pods expose access points via kubectl. Examples are provided for using kubectl proxy to access pod endpoints, port forwarding to proxy local ports to pods, and retrieving logs and attaching to pods' stdin/stdout/stderr. The raw kubectl option and debugging with increased log levels are also covered.
KubeCon CloudNativeCon Seattle 2019 Recap - General overview and also summary of some of the application deployment track (App sig, Operator Framework, Helm, Kustomize, CNAB).
[Spark Summit 2017 NA] Apache Spark on KubernetesTimothy Chen
This document summarizes a presentation about running Apache Spark on Kubernetes. It discusses how Spark jobs can be scheduled and run on Kubernetes, including scheduling the driver and executor pods. Key points of the design include the Kubernetes scheduler backend for Spark and components like the file staging server. The roadmap outlines upcoming support for features like Spark Streaming and improvements to dynamic allocation.
Apache Spark on Kubernetes Anirudh Ramanathan and Tim ChenDatabricks
Kubernetes is a fast growing open-source platform which provides container-centric infrastructure. Conceived by Google in 2014, and leveraging over a decade of experience running containers at scale internally, it is one of the fastest moving projects on GitHub with 1000+ contributors and 40,000+ commits. Kubernetes has first class support on Google Cloud Platform, Amazon Web Services, and Microsoft Azure.
Unlike YARN, Kubernetes started as a general purpose orchestration framework with a focus on serving jobs. Support for long-running, data intensive batch workloads required some careful design decisions. Engineers across several organizations have been working on Kubernetes support as a cluster scheduler backend within Spark. During this process, we encountered several challenges in translating Spark considerations into idiomatic Kubernetes constructs. In this talk, we describe the challenges and the ways in which we solved them. This talk will be technical and is aimed at people who are looking to run Spark effectively on their clusters. The talk assumes basic familiarity with cluster orchestration and containers.
Global azurebootcamp2019vancouver aks_presentation_by_ashprasad_arjavprasadashishpd
Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. It was originally developed by Google and is now maintained by the Cloud Native Computing Foundation. The presentation showed how to deploy an ASP.Net Core application to Azure Kubernetes Service (AKS) using a DevOps approach and then how to scale the application on AKS by adding more nodes.
There is increased interest in using Kubernetes, the open-source container orchestration system for modern, stateful Big Data analytics workloads. The promised land is a unified platform that can handle cloud native stateless and stateful Big Data applications. However, stateful, multi-service Big Data cluster orchestration brings unique challenges. This session will delve into the technical gaps and considerations for Big Data on Kubernetes.
Containers offer significant value to businesses; including increased developer agility, and the ability to move applications between on-premises servers, cloud instances, and across data centers. Organizations have embarked on this journey to containerization with an emphasis on stateless workloads. Stateless applications are usually microservices or containerized applications that don’t “store” data. Web services (such as front end UIs and simple, content-centric experiences) are often great candidates as stateless applications since HTTP is stateless by nature. There is no dependency on the local container storage for the stateless workload.
Stateful applications, on the other hand, are services that require backing storage and keeping state is critical to running the service. Hadoop, Spark and to lesser extent, noSQL platforms such as Cassandra, MongoDB, Postgres, and mySQL are great examples. They require some form of persistent storage that will survive service restarts...
Speakers
Anant Chintamaneni, VP Products, BlueData
Nanda Vijaydev, Director Solutions, BlueData
Kubernetes Architecture - beyond a black box - Part 1Hao H. Zhang
This is part 1 of my Kubernetes architecture deep-dive slide series.
I have been working with Kubernetes for more than a year, from v1.3.6 to v1.6.7, and I am a CNCF certified Kubernetes administrator. Before I move on to something else, I would like to summarize and share my knowledges and take-aways about Kubernetes, from a software engineer perspective.
This set of slides is a humble dig into one level below your running application in production, revealing how different components of Kubernetes work together to orchestrate containers and present your applications to the rest of the world.
The slides contains 80+ external links to Kubernetes documentations, blog posts, Github issues, discussions, design proposals, pull requests, papers, source code files I went through when I was working with Kubernetes - which I think are valuable for people to understand how Kubernetes works, Kubernetes design philosophies and why these design came into places.
Running secured Spark job in Kubernetes compute cluster and integrating with ...DataWorks Summit
This presentation will provide technical design and development insights to run a secured Spark job in Kubernetes compute cluster that accesses job data from a Kerberized HDFS cluster. Joy will show how to run a long-running machine learning or ETL Spark job in Kubernetes and to access data from HDFS using Kerberos Principal and Delegation token.
The first part of this presentation will unleash the design and best practices to deploy and run Spark in Kubernetes integrated with HDFS that creates on-demand multi-node Spark cluster during job submission, installing/resolving software dependencies (packages), executing/monitoring the workload, and finally disposing the resources at the end of job completion. The second part of this presentation covers the design and development details to setup a Spark+Kubernetes cluster that supports long-running jobs accessing data from secured HDFS storage by creating and renewing Kerberos delegation tokens seamlessly from end-user's Kerberos Principal.
All the techniques covered in this presentation are essential in order to set up a Spark+Kubernetes compute cluster that accesses data securely from distributed storage cluster such as HDFS in a corporate environment. No prior knowledge of any of these technologies is required to attend this presentation.
Speaker
Joy Chakraborty, Data Architect
On CloudStack, Docker, Kubernetes, and Big Data…Oh my ! By Sebastien Goasguen...Radhika Puthiyetath
Sebastien Goasguen is a developer who works on Apache CloudStack and other open source projects related to cloud computing, containers, and big data. He gave a talk covering CloudStack, Docker, Kubernetes, CoreOS, and how various technologies can work together for managing distributed applications and infrastructure. He also discussed the evolving landscape of cloud computing and how big data solutions fit within that landscape.
Storage Requirements and Options for Running Spark on KubernetesDataWorks Summit
In a world of serverless computing users tend to be frugal when it comes to expenditure on compute, storage and other resources. Paying for the same when they aren’t in use becomes a significant factor. Offering Spark as service on cloud presents very unique challenges. Running Spark on Kubernetes presents a lot of challenges especially around storage and persistence. Spark workloads have very unique requirements of Storage for intermediate data, long time persistence, Share file system and requirements become very tight when it same need to be offered as a service for enterprise to mange GDPR and other compliance like ISO 27001 and HIPAA certifications.
This talk covers challenges involved in providing Serverless Spark Clusters share the specific issues one can encounter when running large Kubernetes clusters in production especially covering the scenarios related to persistence.
This talk will help people using Kubernetes or docker runtime in production and help them understand various storage options available and which is more suitable for running Spark workloads on Kubernetes and what more can be done
This document discusses storage requirements for running Spark workloads on Kubernetes. It recommends using a distributed file system like HDFS or DBFS for distributed storage and emptyDir or NFS for local temp scratch space. Logs can be stored in emptyDir or pushed to object storage. Features that would improve Spark on Kubernetes include image volumes, flexible PV to PVC mappings, encrypted volumes, and clean deletion for compliance. The document provides an overview of Spark, Kubernetes benefits, and typical Spark deployments.
[DevDay 2017] OpenShift Enterprise - Speaker: Linh Do - DevOps Engineer at Ax...DevDay.org
This session discusses OpenShift Enterprise (or OpenShift Container Platform). OpenShift Container Platform is Red Hat's on-premise private platform as a service product, built around a core of application containers powered by Docker, with orchestration and management provided by Kubernetes, on a foundation of Red Hat Enterprise Linux.
This document discusses running MySQL on Kubernetes with Percona Kubernetes Operators. It provides an introduction to cloud native applications and Kubernetes. It then discusses the benefits and challenges of running MySQL on Kubernetes compared to database-as-a-service options. It introduces Percona Kubernetes Operators for MySQL, which help manage and configure MySQL deployments on Kubernetes. Finally, it discusses how to deploy MySQL with the Percona Kubernetes Operators, including prerequisites, connectivity, architecture, high availability, and monitoring.
Deploying Anything as a Service (XaaS) Using Operators on KubernetesAll Things Open
This document discusses deploying software-as-a-service (XaaS) applications using operators on Kubernetes. It defines operators as collections of custom resource definitions and controllers that manage the lifecycle of those resources. Operators can deploy applications and dependencies within or outside the Kubernetes cluster. The document provides examples of when to use operators for internal resources like databases, as well as managed cloud services. It also discusses where to find operators and how to deploy common ones like Elasticsearch, AWS services, and Kafka.
Centralizing Kubernetes and Container OperationsKublr
While developers see and realize the benefits of Kubernetes, how it improves efficiencies, saves time, and enables focus on the unique business requirements of each project; InfoSec, infrastructure, and software operations teams still face challenges when managing a new set of tools and technologies, and integrating them into an existing enterprise infrastructure.
These meetup slides go over what’s needed for a general architecture of a centralized Kubernetes operations layer based on open source components such as Prometheus, Grafana, ELK Stack, Keycloak, etc., and how to set up reliable clusters and multi-master configuration without a load balancer. It also outlines how these components should be combined into an operations-friendly enterprise Kubernetes management platform with centralized monitoring and log collection, identity and access management, backup and disaster recovery, and infrastructure management capabilities. This presentation will show real-world open source projects use cases to implement an ops-friendly environment.
Check out this and more webinars in our BrightTalk channel: https://goo.gl/QPE5rZ
Building Cloud-Native Applications with Kubernetes, Helm and KubelessBitnami
This document discusses building cloud-native applications with Kubernetes, Helm, and Kubeless. It introduces cloud-native concepts like containers and microservices. It then explains how Kubernetes provides container orchestration and Helm provides application packaging. Finally, it discusses how Kubeless enables serverless functionality on Kubernetes.
In order to provide prompt results and efficiently deal with data-intensive workloads, Big Data applications execute their jobs on compute slots across large clusters. Also, for optimal performance, these applications should be as close as possible to the data they use. Data-aware scheduling is the way to achieve that optimization and can conveniently be set up using Kubernetes. We’ll present two different use cases: First, we’ll make use of how Big Data applications like Hadoop and Spark can use their native HDFS protocol for data-aware scheduling. Second, we’ll demonstrate an efficient way to write a data-aware scheduler for Kubernetes that satisfies not just your application’s requirements, but also keeps your admins happy. As a bonus, it’ll also allows us to run data-aware scheduling on applications other than Big Data.
Event: Kubernetes Meetup Rhein-Neckar, 18.10.2017
Speaker: Johannes M. Scheuermann
weiter Tech-Vorträge: https://www.inovex.de/de/content-pool/vortraege/
Tech-Artikel in unserem Blog: https://www.inovex.de/blog/
Docker kubernetes fundamental(pod_service)_190307Inhye Park
The document discusses several challenges with traditional IT infrastructure including lack of agility due to long development times, aging infrastructure with outdated hardware and software, and high costs associated with monolithic architectures. It then introduces containers and microservices as ways to address these challenges by enabling faster development and deployment, using modern infrastructure, and developing applications in a more modular way. Key concepts covered include containerizing existing applications, rearchitecting apps for scale with containers, and moving to a container platform and microservices.
A Primer on Kubernetes and Google Container EngineRightScale
Docker and other container technologies offer the promise of improved productivity and portability. Kubernetes is one of the leading cluster management systems for Docker and powers the Google Container Engine managed service.
-A review of key Linux container concepts
-The role of Kubernetes in deploying Docker-based applications
-Primer on Google Container Service
-How RightScale works with containers and clusters
Moby is an open source project providing a "LEGO set" of dozens of components, the framework to assemble them into specialized container-based systems, and a place for all container enthusiasts to experiment and exchange ideas.
One of these assemblies is Docker CE, an open source product that lets you build, ship, and run containers.
This talk will explain how you can leverage the Moby project to assemble your own specialized container-based system, whether for IoT, cloud or bare metal scenarios.
We will cover Moby itself, the framework, and tooling around the project, as well as many of it’s components: LinuxKit, InfraKit, containerd, SwarmKit, Notary.
Then we will present a few use cases and demos of how different companies have leveraged Moby and some of the Moby components to create their own container-based systems.
Video at https://www.youtube.com/watch?v=kDp22YkD6WY
Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.
“An Outlook of the Ongoing and Future Relationship between Blockchain Technologies and Process-aware Information Systems.” Invited talk at the joint workshop on Blockchain for Information Systems (BC4IS) and Blockchain for Trusted Data Sharing (B4TDS), co-located with with the 36th International Conference on Advanced Information Systems Engineering (CAiSE), 3 June 2024, Limassol, Cyprus.
Monitoring and Managing Anomaly Detection on OpenShift.pdfTosin Akinosho
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
AI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdfTechgropse Pvt.Ltd.
In this blog post, we'll delve into the intersection of AI and app development in Saudi Arabia, focusing on the food delivery sector. We'll explore how AI is revolutionizing the way Saudi consumers order food, how restaurants manage their operations, and how delivery partners navigate the bustling streets of cities like Riyadh, Jeddah, and Dammam. Through real-world case studies, we'll showcase how leading Saudi food delivery apps are leveraging AI to redefine convenience, personalization, and efficiency.
HCL Notes and Domino License Cost Reduction in the World of DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
CAKE: Sharing Slices of Confidential Data on BlockchainClaudio Di Ciccio
Presented at the CAiSE 2024 Forum, Intelligent Information Systems, June 6th, Limassol, Cyprus.
Synopsis: Cooperative information systems typically involve various entities in a collaborative process within a distributed environment. Blockchain technology offers a mechanism for automating such processes, even when only partial trust exists among participants. The data stored on the blockchain is replicated across all nodes in the network, ensuring accessibility to all participants. While this aspect facilitates traceability, integrity, and persistence, it poses challenges for adopting public blockchains in enterprise settings due to confidentiality issues. In this paper, we present a software tool named Control Access via Key Encryption (CAKE), designed to ensure data confidentiality in scenarios involving public blockchains. After outlining its core components and functionalities, we showcase the application of CAKE in the context of a real-world cyber-security project within the logistics domain.
Paper: https://doi.org/10.1007/978-3-031-61000-4_16
Programming Foundation Models with DSPy - Meetup SlidesZilliz
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
Infrastructure Challenges in Scaling RAG with Custom AI modelsZilliz
Building Retrieval-Augmented Generation (RAG) systems with open-source and custom AI models is a complex task. This talk explores the challenges in productionizing RAG systems, including retrieval performance, response synthesis, and evaluation. We’ll discuss how to leverage open-source models like text embeddings, language models, and custom fine-tuned models to enhance RAG performance. Additionally, we’ll cover how BentoML can help orchestrate and scale these AI components efficiently, ensuring seamless deployment and management of RAG systems in the cloud.
OpenID AuthZEN Interop Read Out - AuthorizationDavid Brossard
During Identiverse 2024 and EIC 2024, members of the OpenID AuthZEN WG got together and demoed their authorization endpoints conforming to the AuthZEN API
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
GraphRAG for Life Science to increase LLM accuracyTomaz Bratanic
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
8. Containers
• Repeatable Builds and
Workflows
• Application Portability
• High Degree of Control over
Software
• Faster Development Cycle
• Reduced dev-ops load
• Improved Infrastructure
Utilization
libs
app
kernel
libs
app
libs
app
libs
app
9. • Based on Google's experience running containers in
production for over 15 years
• Large OSS Community - 1200+ contributors and 45k+
commits
• Ecosystem and Partners - 100+ organizations involved
• One of the top 100 projects overall on GitHub - 23k+
stars
Statistics
14. Controllers
● Drive current state -> desired state
● Act independently
● Recurring pattern in the system
Examples:
● Deployment
● DaemonSet
● StatefulSet
observe
diff
act
16. • Resource sharing between batch, serving and stateful
workloads
– Streamlined developer experience
– Reduced operational costs
– Improved infrastructure utilization
• Kubernetes and the Container Ecosystem
– Lots of addon services: third-party logging, monitoring,
and security tools
– For example, the Istio project, announced May 24, by IBM,
Google and Lyft
Why Kubernetes?
19. • Beta recently announced at Spark Summit 2017
• Google, Haiwen, Hyperpilot, Intel, Palantir, Pepperdata,
Red Hat, and growing.
Spark on Kubernetes
https://github.com/apache-spark-on-k8s/spar
k
Spark Core
Kubernetes Standalone YARN Mesos
GraphX SparkSQL MLlib Streaming
20. Spark on Kubernetes
Kubernetes
Integration
Container images with dependencies baked
in
Files from GCS/S3/HDFS/HTTP
File Staging Server
Staged files and
JARs
Several ways of running Spark Jobs along with their dependencies
on Kubernetes
21. Spark on Kubernetes
Spark Core Kubernetes Scheduler
Backend
Kubernetes Clusternew executors
remove executors
configuration
• Resource Requests
• Authnz
• Communication with K8s
22. State of Spark
Spark Streaming
Spark Shell
Client Mode
Python/R support
Cluster Mode
Java/Scala
Support
Dynamic
Allocation
Local File Staging High Availability
Spark SQL
GraphX MLlib
Dec 2016
Development
Began
Mar 2017
Alpha
Release
June 2017
Beta
Release
Nov 2016
Design
= supported but
untested
= not yet
supported
23. • Community driven effort to get HDFS running well on
Kubernetes
• Uses a helm chart to install onto a cluster
• Identified and solved several problems around data
locality when running Spark Jobs
HDFS on Kubernetes
https://github.com/apache-spark-on-k8s/kubernetes-HDFS
24. HDFS on Kubernetes
node A node B
Driver Pod Executor Pod 1 Executor Pod 2
10.0.0.2
196.0.0.5 196.0.0.6
10.0.0.3 10.0.1.2
Namenode Pod Datanode Pod 1 Datanode Pod 2
HDFS on Kubernetes -- Lessons Learned [Public]
Kimoon Kim (PepperData)
25. State of HDFS
• HDFS with basic data locality works!
• Future Work
– Remaining data locality issues -- rack locality, node
preference, etc
– Performance benchmarks and testing
– Kerberos support
– Namenode HA
27. • Pipelines feature many other components.
• All of the below must run well on K8s
– Cassandra
– Kafka
– Zookeeper
– Elasticsearch, Kibana, etc
Data Pipelines are complicated!