Why run Postgres in Kubernetes?
•
0 likes•121 views
Postgres should run inside your Kubernetes cluster. Yes, inside, not outside Kubernetes. After all, a database should be seen as an application, a special type of application - for which it is legitimate to require an additional level of care and attention. However, the small price you pay for this is worth the return on investment that your organization receives by running microservice applications, plus database combos, inside your Cloud Native infrastructure. In this session, I will cover why it is important to make such a decision. Discover the challenges and the opportunities that running Postgres inside Kubernetes presents. I will discuss what we all expect from a good Postgres operator, including self-healing, high availability, scalability, backup and recovery, performance, and - last but not least - security. I will then go beyond technical aspects to spark conversations about the holistic improvements that running a Cloud Native database brings to your organization. This talk was given by Gabriele Bartolini for DoK Day Europe @ KubeCon 2022.
Report
Share
Report
Share
Download to read offline
Recommended
Scalable Clusters On Demand
At Opendoor, we do a lot of big data processing, and use Spark and Dask clusters for the computations. Our machine learning platform is written in Dask and we are actively moving data ingestion pipelines and geo computations to PySpark. The biggest challenge is that jobs vary in memory, cpu needs, and the load in not evenly distributed over time, which causes our workers and clusters to be over-provisioned. In addition to this, we need to enable data scientists and engineers run their code without having to upgrade the cluster for every request and deal with the dependency hell.
To solve all of these problems, we introduce a lightweight integration across some popular tools like Kubernetes, Docker, Airflow and Spark. Using a combination of these tools, we are able to spin up on-demand Spark and Dask clusters for our computing jobs, bring down the cost using autoscaling and spot pricing, unify DAGs across many teams with different stacks on the single Airflow instance, and all of it at minimal cost.
Cloud Native PostgreSQL - APJ
EDB Cloud Native Postgres includes database container images and a Kubernetes Operator that manage the lifecycle of a database from deployment to operations. This Kubernetes Operator for Postgres is written by EDB entirely from scratch in the Go language and relies exclusively on the Kubernetes API.
Attend this webinar to learn about:
- DevOps & Cloud Native
- Overview of Cloud Native Postgres
- Storage for Postgres workloads in Kubernetes
- Using Cloud Native Postgres
- Demo
DevOps Days Boston 2017: Real-world Kubernetes for DevOps
DevOps Days Boston 2017
Microservices is an increasingly popular approach to building cloud-native applications. Dozens of new technologies that streamline adopting microservices development such as Docker, Kubernetes, and Envoy have been released over the past few years. But how do you actually use these technologies together to develop, deploy, and run microservices?
In this presentation, we’ll cover the nuances of deploying containerized applications on Kubernetes, including creating a Kubernetes manifest, debugging and logging, and how to build an automated continuous deployment pipeline. Then, we’ll do a brief tour of some of the advanced concepts related to microservices, including service mesh, canary deployments, resilience, and security.
MLOps implemented - how we combine the cloud & open-source to boost data scie...
Check out more about this presentation here: https://www.youtube.com/watch?v=nSsssYHiylQ&t=17s
Presentation from the performance given by our team during the NSML Summit.
Authors: Krzysztof Zarzycki, Marek Wiewiórka
Linkedin: https://www.linkedin.com/in/kzarzycki/
https://www.linkedin.com/in/marekwiewiorka/
___
Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets.
Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries.
https://getindata.com
Quebec - 16 November 2022 - Canada CNCF Meetups.pdf
This document appears to be notes from a Cloud Native Computing Foundation meetup in Quebec. The meetup featured presentations on topics like Istio Ambient Mesh, GitOps with OCI artifacts and Helm charts, and configuring a laptop for Kubernetes. It also included discussions on using GitOps and establishing a KCD chapter in Quebec. The agenda, speakers, and trends observed at KubeCon North America 2022 were recapped. Plans for upcoming meetups in Toronto, Montreal and Ottawa were announced.
Webinar: Building a multi-cloud Kubernetes storage on GitLab
In this webinar, we talk about how to set up redundant and highly available storage for your repos to make your key repositories easier to deploy, more reliable, and easier to back up or move to a different cloud. We reviewed the current practices for highly-available CI/CD and showcased how there’s a better way to do it with OpenEBS.
Cloud Native PostgreSQL
Cloud-Native PostgreSQL is a Kubernetes Operator for Postgres written by EDB entirely from scratch in the Go language and relying exclusively on the Kubernetes API.
This webinar covered:
- About DevOps & Cloud Native
- Overview of Cloud Native Postgres
- Storage for Postgres workloads in Kubernetes
- Start Using Cloud-Native Postgres
- Demo
Multi Master PostgreSQL Cluster on Kubernetes
The document discusses running PostgreSQL on Kubernetes using Crunchy PostgreSQL Containers. It describes how to run a simple PostgreSQL pod on OpenShift by writing a pod manifest file that defines the container image, ports, environment variables, volume mounts, and other settings. Pulling the PostgreSQL container image from DockerHub and defining the pod configuration in YAML allows running PostgreSQL as a stateful application on Kubernetes.
Recommended
Scalable Clusters On Demand
At Opendoor, we do a lot of big data processing, and use Spark and Dask clusters for the computations. Our machine learning platform is written in Dask and we are actively moving data ingestion pipelines and geo computations to PySpark. The biggest challenge is that jobs vary in memory, cpu needs, and the load in not evenly distributed over time, which causes our workers and clusters to be over-provisioned. In addition to this, we need to enable data scientists and engineers run their code without having to upgrade the cluster for every request and deal with the dependency hell.
To solve all of these problems, we introduce a lightweight integration across some popular tools like Kubernetes, Docker, Airflow and Spark. Using a combination of these tools, we are able to spin up on-demand Spark and Dask clusters for our computing jobs, bring down the cost using autoscaling and spot pricing, unify DAGs across many teams with different stacks on the single Airflow instance, and all of it at minimal cost.
Cloud Native PostgreSQL - APJ
EDB Cloud Native Postgres includes database container images and a Kubernetes Operator that manage the lifecycle of a database from deployment to operations. This Kubernetes Operator for Postgres is written by EDB entirely from scratch in the Go language and relies exclusively on the Kubernetes API.
Attend this webinar to learn about:
- DevOps & Cloud Native
- Overview of Cloud Native Postgres
- Storage for Postgres workloads in Kubernetes
- Using Cloud Native Postgres
- Demo
DevOps Days Boston 2017: Real-world Kubernetes for DevOps
DevOps Days Boston 2017
Microservices is an increasingly popular approach to building cloud-native applications. Dozens of new technologies that streamline adopting microservices development such as Docker, Kubernetes, and Envoy have been released over the past few years. But how do you actually use these technologies together to develop, deploy, and run microservices?
In this presentation, we’ll cover the nuances of deploying containerized applications on Kubernetes, including creating a Kubernetes manifest, debugging and logging, and how to build an automated continuous deployment pipeline. Then, we’ll do a brief tour of some of the advanced concepts related to microservices, including service mesh, canary deployments, resilience, and security.
MLOps implemented - how we combine the cloud & open-source to boost data scie...
Check out more about this presentation here: https://www.youtube.com/watch?v=nSsssYHiylQ&t=17s
Presentation from the performance given by our team during the NSML Summit.
Authors: Krzysztof Zarzycki, Marek Wiewiórka
Linkedin: https://www.linkedin.com/in/kzarzycki/
https://www.linkedin.com/in/marekwiewiorka/
___
Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets.
Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries.
https://getindata.com
Quebec - 16 November 2022 - Canada CNCF Meetups.pdf
This document appears to be notes from a Cloud Native Computing Foundation meetup in Quebec. The meetup featured presentations on topics like Istio Ambient Mesh, GitOps with OCI artifacts and Helm charts, and configuring a laptop for Kubernetes. It also included discussions on using GitOps and establishing a KCD chapter in Quebec. The agenda, speakers, and trends observed at KubeCon North America 2022 were recapped. Plans for upcoming meetups in Toronto, Montreal and Ottawa were announced.
Webinar: Building a multi-cloud Kubernetes storage on GitLab
In this webinar, we talk about how to set up redundant and highly available storage for your repos to make your key repositories easier to deploy, more reliable, and easier to back up or move to a different cloud. We reviewed the current practices for highly-available CI/CD and showcased how there’s a better way to do it with OpenEBS.
Cloud Native PostgreSQL
Cloud-Native PostgreSQL is a Kubernetes Operator for Postgres written by EDB entirely from scratch in the Go language and relying exclusively on the Kubernetes API.
This webinar covered:
- About DevOps & Cloud Native
- Overview of Cloud Native Postgres
- Storage for Postgres workloads in Kubernetes
- Start Using Cloud-Native Postgres
- Demo
Multi Master PostgreSQL Cluster on Kubernetes
The document discusses running PostgreSQL on Kubernetes using Crunchy PostgreSQL Containers. It describes how to run a simple PostgreSQL pod on OpenShift by writing a pod manifest file that defines the container image, ports, environment variables, volume mounts, and other settings. Pulling the PostgreSQL container image from DockerHub and defining the pod configuration in YAML allows running PostgreSQL as a stateful application on Kubernetes.
Which PostgreSQL is right for your multi cloud strategy? P2
The adoption of PostgreSQL in enterprises is becoming a strategic choice, more so with the adoption of Multi-Cloud now becoming a need for enterprise deployment. This availability creates multiple combinations of deployment options for you. So, it is important to identify the right strategy fitting into your organization’s needs.
Elephant Roads: PostgreSQL Patches and Variants
This document summarizes the history of PostgreSQL forks and variants over 17 years. It describes how Michael Stonebraker's POSTGRES database led to forks like Illustra and variants like PostgreSQL. It outlines four types of PostgreSQL variants and provides many examples of expired and active forks for various purposes like clustering, data warehousing, and exotic features. It encourages innovating with new PostgreSQL forks.
Elephant Roads: a tour of Postgres forks
Josh Berkus
Most users know that PostgreSQL has a 23-year development history. But did you know that Postgres code is used for over a dozen other database systems? Thanks to our liberal licensing, many companies and open source projects over the years have taken the Postgres or PostgreSQL code, changed it, added things to it, and/or merged it into something else. Illustra, Truviso, Aster, Greenplum, and others have seen the value of Postgres not just as a database but as some darned good code they could use. We'll explore the lineage of these forks, and go into the details of some of the more interesting ones.
Kubernetes Forum Seoul 2019: Re-architecting Data Platform with Kubernetes
Session Video: https://youtu.be/7MPH1mknIxE
In this talk, we share Devsisters' journey of migrating its internal data platform including Spark to Kubernetes, with its benefits and issues.
데브시스터즈에서 데이터플랫폼 컴포넌트를 쿠버네티스로 옮기면서 얻은 장점들과 이슈들에 대해 공유합니다.
Conference session page:
- English: https://sched.co/WIRK
- Korean: https://sched.co/WYRc
Heroku to Kubernetes & Gihub to Gitlab success story
A migration from github to gitlab, circle ci to gitlab-ci, heroku to kubernetes by using helm and vault (hashicorp)
Running PostgreSQL in Kubernetes: from day 0 to day 2 with CloudNativePG - Do...
Link: https://youtu.be/cegd3Exg05w
https://go.dok.community/slack
https://dok.community/
Gabriele Bartolini - Vice President/CTO of Cloud Native and Kubernetes, EDB
ABSTRACT OF THE TALK
Imagine this: you have a virtual infrastructure based on Kubernetes, made up of virtual data centers, possibly spread across multiple Kubernetes clusters and regions. Your infrastructure could even be hosted on premises or on different cloud service providers. Infrastructure as Code is a requirement. You’ve been tasked to run Postgres databases, alongside your applications.
The good news is that you can leverage a fully open source stack with Kubernetes, PostgreSQL and the CloudNativePG operator, and deploy your Postgres database in the same way you deploy applications.
Join me in this webinar to discover the key role that you have to make this succeed, starting from day 0 through day 2 operations.
I’ll share some examples and best practices for running Postgres databases in Kubernetes, before peeking at the new features we are developing for the months to come.
Multimedia support in WebKitGTK and WPE, current status and plans (GStreamer ...
By Philippe Normand.
This talk is about multimedia support in the WPE and GTK+ WebKit ports. I will give a status update about the HTML5 features currently supported by our GStreamer backend, such as WebRTC, MSE, MediaCapabilities support. The talk would also include a brief case study about using WPE and its Cog browser on IMX6 platforms.
(c) GStreamer Conference 2018
CC-BY-SA 3.0
https://gstreamer.freedesktop.org/conference/2018/
https://gstconf.ubicast.tv/videos/multimedia-support-in-webkitgtk-and-wpe-current-status-and-plans/
Kubernetes Basis: Pods, Deployments, and Services
Kubernetes is a container management platform and empowers the scalability to the container. In this repository, we address the issues of how to use Kubernetes with real cases. We start from the basic objects in Kubernetes, Pods, deployments, and Services. This repository is also a tutorial for those with advanced containerization skills trying to step into the Kubernetes. We also provide several YAML examples for those looking for quickly deploying services. Please enjoy it and let's start the journey to Kubernetes.
Enhance Your Kubernetes CI/CD Pipelines With GitLab & Open Source
Working within a heavily regulated environment brings a special set of challenges, including increased difficulty in application scaling. In this session you will learn how you can enhance your Kubernetes CI/CD pipelines with GitLab and other open source projects. We will demonstrate practices for deployments using newer GitLab features like the Web Application Firewall for Kubernetes Ingress, and managing serverless functions with Knative. The techniques covered in this session will give you new options to streamline your Kubernetes pipelines reliably and consistently.
Rejekts 24 EU No GitOps Pain, No Platform Gain
Presentation about challenges with building a GitOps managed kubernetes platform for fleets of clusters.
Kubernetes 1.12 Update and Container Security with Liz Rice
This document summarizes a meetup about Kubernetes and cloud native technologies in Montreal. It announces upcoming presentations on CNCF and Kubernetes 1.12 updates, and securing container deployments. It also provides information about getting involved with local user groups, upcoming hands-on workshops from CloudOps, the Rook and Cortex projects joining CNCF, and TiKV key-value store.
PySpark on Kubernetes @ Python Barcelona March Meetup
This talk explores what PySpark on Kubernetes looks like along with some of the areas of future development.
Introduction to kubernetes
This document provides an introduction to Kubernetes including:
- What Kubernetes is and what it does including abstracting infrastructure, providing self-healing capabilities, and providing a uniform interface across clouds.
- Key concepts including pods, services, labels, selectors, and namespaces. Pods are the atomic unit and services provide a unified access method. Labels and selectors are used to identify and group related objects.
- The Kubernetes architecture including control plane components like kube-apiserver, etcd, and kube-controller-manager. Node components include kubelet and kube-proxy. Optional services like cloud-controller-manager and cluster DNS are also described.
Benchmarking for postgresql workloads in kubernetes
ABSTRACT OF THE TALK
6 months have passed since our last DoK webinar about benchmarking PostgreSQL workloads in a Kubernetes environment. In the meantime, many things have happened at EDB, and we’re happy to share what we’ve learned in this timeframe. We’ll use cnp-bench and cnp-sandbox to help us describe some of the challenges we might face when running PostgreSQL workloads, how to spot them, and what actions to take to make your databases healthier and more longeve.
cnp-bench is a collection of Helm charts that help run storage and database benchmarks, using popular open source tools like fio, pgbench, and HammerDB. cnp-sandbox is a Helm chart that sets up a Prometheus/Grafana stack, including basic metrics and dashboards for Cloud Native PostgreSQL, the Kubernetes operator developed by EDB. Both cnp-sandbox and cnp-bench are open source and recommended for development, testing, and pre-production environments only.
BIO
A long time open-source programmer and entrepreneur, Gabriele has a degree in Statistics from the University of Florence. After having consistently contributed to the growth of 2ndQuadrant and its members through nurturing a lean and devops culture, he is now leading the Cloud Native initiative at EDB. Gabriele lives in Prato, a small but vibrant city located in the northern part of Tuscany, Italy - famous for having hosted the first European PostgreSQL conferences. His second home is Melbourne, Australia, where he studied at Monash University and worked in the ICT sector. He loves playing the Blues with his Fender Stratocaster, but his major passions are called Elisabeth and Charlotte!
KEY TAKE-AWAYS FROM THE TALK
- A methodology for benchmarking a PostgreSQL database in Kubernetes
- Open source set of tools for benchmarking a PostgreSQL database in Kubernetes
- Reasons why benchmarking both the storage and the database is important
https://github.com/EnterpriseDB/cnp-sandbox
https://github.com/EnterpriseDB/cnp-bench
Cross region replication in Percona Kubernetes operator
With Percona Kubernetes Operators you can migrate your data to Kubernetes or have a disaster recovery site to recover it in unfortunate times. I'll talk about the implementation, challenges, and caveats of such setups and what the future may bring.
Kubernetes & Cloud Native Indonesia X BukaMeetup - Feb 2023
The document summarizes an upcoming Kubernetes meetup organized by Bukalapak and BukaMeetup in February 2023. The agenda includes talks on Kubernetes community updates, secret management with Hashicorp Vault and ArgoCD, and a Q&A session. One talk will focus on managing secrets in Kubernetes using the ExternalSecretOperator and ArgoCD Vault plugin to pull secrets from Vault. The benefits and limitations of each approach are discussed. Attendees are encouraged to provide feedback to help improve future meetups.
Containerizing MongoDB with kubernetes
An overview of how to use MongoDB and Kubernetes (k8s) primitives to take advantage of all of the good things both have to offer.
SpringOne Platform 2018 Recap in 5 minutes
The document provides a recap of announcements and news from Spring One Platform 2018. Key points include:
- Buildpacks were contributed to CNCF and the buildpack ecosystem was unified with a well-defined platform to buildpack contract.
- PKS 1.2 reached general availability and can now deploy to AWS alongside other new features.
- The Pivotal Function Service was introduced, building on Knative to allow running functions anywhere and triggering via HTTP/message brokers.
- Several partners received awards for their work in areas like systems integration, independent software, and managed services on the Pivotal platform.
- New developments were announced for projects like Spring Framework, PAS, Concourse, and Kubernetes integration.
stackconf 2020 | The Path to OpenSource DBaaS with Kubernetes by Peter Zaitsev
DBaaS is the fastest growing way to deploy databases. It is fast and convenient and it helps to reduce toil a lot, yet it is typically done using proprietary software and tightly coupled to the cloud vendor. We believe Kubernetes finally allows us to build fully OpenSource DBaaS Solution capable to be deployed anywhere Kubernetes runs – on the Public Cloud or in your private data center.
In this presentation, we will describe the most important user requirements and typical problems you would encounter building DBaaS Solution and explain how you can solve them using Kubernetes Operator framework.
OpenEBS hangout #4
The OpenEBS Hangout #4 was held on 22nd December 2017 at 11:00 AM (IST and PST) where a live demo of cMotion was shown . Storage policies of OpenEBS 0.5 were also explained
Distributed Vector Databases - What, Why, and How
Distributed Vector Databases - What, Why, and How - Steve Pousty, VMware
In the last two years, AI machine learning has exploded in prominence. One of the key concepts used in the modeling and storage of AI is vectors. Feeling like you should learn more and how you would use them in your data work? Wondering how you would run this distributed on Kubernetes? Then have I got a talk for you! We will start by explaining the concept of (embedding) vectors and how they are used in the AI life cycle. From there we will go into putting them into a database. We will cover the use cases where this technology makes sense. As opposed to an RDBMS, vector databases are more tightly focused and optimized for particular use cases. To ground this discussion in something more concrete, there will be hands-on demos throughout the talk. You will see the advantages to running distributed vector databases on Kubernetes infrastructure. Bring your favorite Kube infrastructure and leave with hands-on experience running AI infrastructure on Kubernetes.
Is It Safe? Security Hardening for Databases Using Kubernetes Operators
Is It Safe? Security Hardening for Databases Using Kubernetes Operators - Robert Hodges, Altinity
Thanks to the Operator Pattern, Kubernetes is now an outstanding platform to run databases. But to quote Marathon Man, "is it safe?" This talk is a top-level review of the database security problem in Kubernetes, standard ways that operators can mitigate threats, and a wallet-sized checklist of security features you should look for in any operator you use. Our talk is practical and focused on needs of Kubernetes developers. Join us!
More Related Content
Similar to Why run Postgres in Kubernetes?
Which PostgreSQL is right for your multi cloud strategy? P2
The adoption of PostgreSQL in enterprises is becoming a strategic choice, more so with the adoption of Multi-Cloud now becoming a need for enterprise deployment. This availability creates multiple combinations of deployment options for you. So, it is important to identify the right strategy fitting into your organization’s needs.
Elephant Roads: PostgreSQL Patches and Variants
This document summarizes the history of PostgreSQL forks and variants over 17 years. It describes how Michael Stonebraker's POSTGRES database led to forks like Illustra and variants like PostgreSQL. It outlines four types of PostgreSQL variants and provides many examples of expired and active forks for various purposes like clustering, data warehousing, and exotic features. It encourages innovating with new PostgreSQL forks.
Elephant Roads: a tour of Postgres forks
Josh Berkus
Most users know that PostgreSQL has a 23-year development history. But did you know that Postgres code is used for over a dozen other database systems? Thanks to our liberal licensing, many companies and open source projects over the years have taken the Postgres or PostgreSQL code, changed it, added things to it, and/or merged it into something else. Illustra, Truviso, Aster, Greenplum, and others have seen the value of Postgres not just as a database but as some darned good code they could use. We'll explore the lineage of these forks, and go into the details of some of the more interesting ones.
Kubernetes Forum Seoul 2019: Re-architecting Data Platform with Kubernetes
Session Video: https://youtu.be/7MPH1mknIxE
In this talk, we share Devsisters' journey of migrating its internal data platform including Spark to Kubernetes, with its benefits and issues.
데브시스터즈에서 데이터플랫폼 컴포넌트를 쿠버네티스로 옮기면서 얻은 장점들과 이슈들에 대해 공유합니다.
Conference session page:
- English: https://sched.co/WIRK
- Korean: https://sched.co/WYRc
Heroku to Kubernetes & Gihub to Gitlab success story
A migration from github to gitlab, circle ci to gitlab-ci, heroku to kubernetes by using helm and vault (hashicorp)
Running PostgreSQL in Kubernetes: from day 0 to day 2 with CloudNativePG - Do...
Link: https://youtu.be/cegd3Exg05w
https://go.dok.community/slack
https://dok.community/
Gabriele Bartolini - Vice President/CTO of Cloud Native and Kubernetes, EDB
ABSTRACT OF THE TALK
Imagine this: you have a virtual infrastructure based on Kubernetes, made up of virtual data centers, possibly spread across multiple Kubernetes clusters and regions. Your infrastructure could even be hosted on premises or on different cloud service providers. Infrastructure as Code is a requirement. You’ve been tasked to run Postgres databases, alongside your applications.
The good news is that you can leverage a fully open source stack with Kubernetes, PostgreSQL and the CloudNativePG operator, and deploy your Postgres database in the same way you deploy applications.
Join me in this webinar to discover the key role that you have to make this succeed, starting from day 0 through day 2 operations.
I’ll share some examples and best practices for running Postgres databases in Kubernetes, before peeking at the new features we are developing for the months to come.
Multimedia support in WebKitGTK and WPE, current status and plans (GStreamer ...
By Philippe Normand.
This talk is about multimedia support in the WPE and GTK+ WebKit ports. I will give a status update about the HTML5 features currently supported by our GStreamer backend, such as WebRTC, MSE, MediaCapabilities support. The talk would also include a brief case study about using WPE and its Cog browser on IMX6 platforms.
(c) GStreamer Conference 2018
CC-BY-SA 3.0
https://gstreamer.freedesktop.org/conference/2018/
https://gstconf.ubicast.tv/videos/multimedia-support-in-webkitgtk-and-wpe-current-status-and-plans/
Kubernetes Basis: Pods, Deployments, and Services
Kubernetes is a container management platform and empowers the scalability to the container. In this repository, we address the issues of how to use Kubernetes with real cases. We start from the basic objects in Kubernetes, Pods, deployments, and Services. This repository is also a tutorial for those with advanced containerization skills trying to step into the Kubernetes. We also provide several YAML examples for those looking for quickly deploying services. Please enjoy it and let's start the journey to Kubernetes.
Enhance Your Kubernetes CI/CD Pipelines With GitLab & Open Source
Working within a heavily regulated environment brings a special set of challenges, including increased difficulty in application scaling. In this session you will learn how you can enhance your Kubernetes CI/CD pipelines with GitLab and other open source projects. We will demonstrate practices for deployments using newer GitLab features like the Web Application Firewall for Kubernetes Ingress, and managing serverless functions with Knative. The techniques covered in this session will give you new options to streamline your Kubernetes pipelines reliably and consistently.
Rejekts 24 EU No GitOps Pain, No Platform Gain
Presentation about challenges with building a GitOps managed kubernetes platform for fleets of clusters.
Kubernetes 1.12 Update and Container Security with Liz Rice
This document summarizes a meetup about Kubernetes and cloud native technologies in Montreal. It announces upcoming presentations on CNCF and Kubernetes 1.12 updates, and securing container deployments. It also provides information about getting involved with local user groups, upcoming hands-on workshops from CloudOps, the Rook and Cortex projects joining CNCF, and TiKV key-value store.
PySpark on Kubernetes @ Python Barcelona March Meetup
This talk explores what PySpark on Kubernetes looks like along with some of the areas of future development.
Introduction to kubernetes
This document provides an introduction to Kubernetes including:
- What Kubernetes is and what it does including abstracting infrastructure, providing self-healing capabilities, and providing a uniform interface across clouds.
- Key concepts including pods, services, labels, selectors, and namespaces. Pods are the atomic unit and services provide a unified access method. Labels and selectors are used to identify and group related objects.
- The Kubernetes architecture including control plane components like kube-apiserver, etcd, and kube-controller-manager. Node components include kubelet and kube-proxy. Optional services like cloud-controller-manager and cluster DNS are also described.
Benchmarking for postgresql workloads in kubernetes
ABSTRACT OF THE TALK
6 months have passed since our last DoK webinar about benchmarking PostgreSQL workloads in a Kubernetes environment. In the meantime, many things have happened at EDB, and we’re happy to share what we’ve learned in this timeframe. We’ll use cnp-bench and cnp-sandbox to help us describe some of the challenges we might face when running PostgreSQL workloads, how to spot them, and what actions to take to make your databases healthier and more longeve.
cnp-bench is a collection of Helm charts that help run storage and database benchmarks, using popular open source tools like fio, pgbench, and HammerDB. cnp-sandbox is a Helm chart that sets up a Prometheus/Grafana stack, including basic metrics and dashboards for Cloud Native PostgreSQL, the Kubernetes operator developed by EDB. Both cnp-sandbox and cnp-bench are open source and recommended for development, testing, and pre-production environments only.
BIO
A long time open-source programmer and entrepreneur, Gabriele has a degree in Statistics from the University of Florence. After having consistently contributed to the growth of 2ndQuadrant and its members through nurturing a lean and devops culture, he is now leading the Cloud Native initiative at EDB. Gabriele lives in Prato, a small but vibrant city located in the northern part of Tuscany, Italy - famous for having hosted the first European PostgreSQL conferences. His second home is Melbourne, Australia, where he studied at Monash University and worked in the ICT sector. He loves playing the Blues with his Fender Stratocaster, but his major passions are called Elisabeth and Charlotte!
KEY TAKE-AWAYS FROM THE TALK
- A methodology for benchmarking a PostgreSQL database in Kubernetes
- Open source set of tools for benchmarking a PostgreSQL database in Kubernetes
- Reasons why benchmarking both the storage and the database is important
https://github.com/EnterpriseDB/cnp-sandbox
https://github.com/EnterpriseDB/cnp-bench
Cross region replication in Percona Kubernetes operator
With Percona Kubernetes Operators you can migrate your data to Kubernetes or have a disaster recovery site to recover it in unfortunate times. I'll talk about the implementation, challenges, and caveats of such setups and what the future may bring.
Kubernetes & Cloud Native Indonesia X BukaMeetup - Feb 2023
The document summarizes an upcoming Kubernetes meetup organized by Bukalapak and BukaMeetup in February 2023. The agenda includes talks on Kubernetes community updates, secret management with Hashicorp Vault and ArgoCD, and a Q&A session. One talk will focus on managing secrets in Kubernetes using the ExternalSecretOperator and ArgoCD Vault plugin to pull secrets from Vault. The benefits and limitations of each approach are discussed. Attendees are encouraged to provide feedback to help improve future meetups.
Containerizing MongoDB with kubernetes
An overview of how to use MongoDB and Kubernetes (k8s) primitives to take advantage of all of the good things both have to offer.
SpringOne Platform 2018 Recap in 5 minutes
The document provides a recap of announcements and news from Spring One Platform 2018. Key points include:
- Buildpacks were contributed to CNCF and the buildpack ecosystem was unified with a well-defined platform to buildpack contract.
- PKS 1.2 reached general availability and can now deploy to AWS alongside other new features.
- The Pivotal Function Service was introduced, building on Knative to allow running functions anywhere and triggering via HTTP/message brokers.
- Several partners received awards for their work in areas like systems integration, independent software, and managed services on the Pivotal platform.
- New developments were announced for projects like Spring Framework, PAS, Concourse, and Kubernetes integration.
stackconf 2020 | The Path to OpenSource DBaaS with Kubernetes by Peter Zaitsev
DBaaS is the fastest growing way to deploy databases. It is fast and convenient and it helps to reduce toil a lot, yet it is typically done using proprietary software and tightly coupled to the cloud vendor. We believe Kubernetes finally allows us to build fully OpenSource DBaaS Solution capable to be deployed anywhere Kubernetes runs – on the Public Cloud or in your private data center.
In this presentation, we will describe the most important user requirements and typical problems you would encounter building DBaaS Solution and explain how you can solve them using Kubernetes Operator framework.
OpenEBS hangout #4
The OpenEBS Hangout #4 was held on 22nd December 2017 at 11:00 AM (IST and PST) where a live demo of cMotion was shown . Storage policies of OpenEBS 0.5 were also explained
Similar to Why run Postgres in Kubernetes? (20)
Which PostgreSQL is right for your multi cloud strategy? P2
Which PostgreSQL is right for your multi cloud strategy? P2
Kubernetes Forum Seoul 2019: Re-architecting Data Platform with Kubernetes
Kubernetes Forum Seoul 2019: Re-architecting Data Platform with Kubernetes
Heroku to Kubernetes & Gihub to Gitlab success story
Heroku to Kubernetes & Gihub to Gitlab success story
Running PostgreSQL in Kubernetes: from day 0 to day 2 with CloudNativePG - Do...
Running PostgreSQL in Kubernetes: from day 0 to day 2 with CloudNativePG - Do...
Multimedia support in WebKitGTK and WPE, current status and plans (GStreamer ...
Multimedia support in WebKitGTK and WPE, current status and plans (GStreamer ...
Enhance Your Kubernetes CI/CD Pipelines With GitLab & Open Source
Enhance Your Kubernetes CI/CD Pipelines With GitLab & Open Source
Kubernetes 1.12 Update and Container Security with Liz Rice
Kubernetes 1.12 Update and Container Security with Liz Rice
PySpark on Kubernetes @ Python Barcelona March Meetup
PySpark on Kubernetes @ Python Barcelona March Meetup
Benchmarking for postgresql workloads in kubernetes
Benchmarking for postgresql workloads in kubernetes
Cross region replication in Percona Kubernetes operator
Cross region replication in Percona Kubernetes operator
Kubernetes & Cloud Native Indonesia X BukaMeetup - Feb 2023
Kubernetes & Cloud Native Indonesia X BukaMeetup - Feb 2023
stackconf 2020 | The Path to OpenSource DBaaS with Kubernetes by Peter Zaitsev
stackconf 2020 | The Path to OpenSource DBaaS with Kubernetes by Peter Zaitsev
More from DoKC
Distributed Vector Databases - What, Why, and How
Distributed Vector Databases - What, Why, and How - Steve Pousty, VMware
In the last two years, AI machine learning has exploded in prominence. One of the key concepts used in the modeling and storage of AI is vectors. Feeling like you should learn more and how you would use them in your data work? Wondering how you would run this distributed on Kubernetes? Then have I got a talk for you! We will start by explaining the concept of (embedding) vectors and how they are used in the AI life cycle. From there we will go into putting them into a database. We will cover the use cases where this technology makes sense. As opposed to an RDBMS, vector databases are more tightly focused and optimized for particular use cases. To ground this discussion in something more concrete, there will be hands-on demos throughout the talk. You will see the advantages to running distributed vector databases on Kubernetes infrastructure. Bring your favorite Kube infrastructure and leave with hands-on experience running AI infrastructure on Kubernetes.
Is It Safe? Security Hardening for Databases Using Kubernetes Operators
Is It Safe? Security Hardening for Databases Using Kubernetes Operators - Robert Hodges, Altinity
Thanks to the Operator Pattern, Kubernetes is now an outstanding platform to run databases. But to quote Marathon Man, "is it safe?" This talk is a top-level review of the database security problem in Kubernetes, standard ways that operators can mitigate threats, and a wallet-sized checklist of security features you should look for in any operator you use. Our talk is practical and focused on needs of Kubernetes developers. Join us!
Stop Worrying and Keep Querying, Using Automated Multi-Region Disaster Recovery
Stop Worrying and Keep Querying, Using Automated Multi-Region Disaster Recovery - Shivani Gupta, Elotl & Sergey Pronin, Percona
Disaster Recovery(DR) is critical for business continuity in the face of widespread outages taking down entire data centers or cloud provider regions. DR relies on deployment to multiple locations, data replication, monitoring for failure and failover. The process is typically manual involving several moving parts, and, even in the best case, involves some downtime for end-users. A multi-cluster K8s control plane presents the opportunity to automate the DR setup as well as the failure detection and failover. Such automation can dramatically reduce RTO and improve availability for end-users. This talk (and demo) describes one such setup using the open source Percona Operator for PostgreSQL and a multi-cluster K8s orchestrator. The orchestrator will use policy driven placement to replicate the entire workload on multiple clusters (in different regions), detect failure using pluggable logic, and do failover processing by promoting the standby as well as redirecting application traffic
Transforming Data Processing with Kubernetes: Journey Towards a Self-Serve Da...
Transforming Data Processing with Kubernetes: Journey Towards a Self-Serve Data Mesh - Rakesh Subramanian Suresh & Jainik Vora, Intuit
This presentation explores how Intuit uses Kubernetes with Domain-Driven Design and Data Mesh principles to transform its data processing landscape, crucial for its AI-driven expert platform. We will discuss the importance of clean data in developing robust generative artificial intelligence and how Intuit is addressing this through the creation of paved paths for data platforms running on Kubernetes. We'll examine the challenges and solutions in managing 100,000 data pipelines and 1000+ engineers interacting with data, highlighting the need for scalable solutions. We'll also discuss how Intuit uses Kubernetes to build its batch and stream processing platform, overcoming hurdles in data pipeline deployment, scheduling, orchestration, and dependency management. We'll conclude by emphasizing how this transformation, based on treating data as a product, has improved decision-making speed and accuracy across the organization and fostered a more efficient, collaborative data culture.
The State of Stateful on Kubernetes
The State of Stateful on Kubernetes - Stateful Workloads in Kubernetes: A Deep Dive - Kaslin Fields & Michelle Au, Google
As a platform for distributed computing, Kubernetes enables users to run their workloads across machines. However data has gravity, and when workloads in Kubernetes have to share data with other applications, managing the application’s requirements can get more tricky. In this talk, we will explore what "Stateful" means from Kubernetes' perspective. We will discuss the different types of stateful workloads, and the challenges of deploying them on Kubernetes. We will also look at the features that exist in Kubernetes to support stateful workloads, as well as the features that are in the works. Key Takeaways: What is a stateful workload from Kubernetes’ perspective? What are the challenges of deploying stateful workloads on Kubernetes? What features exist in Kubernetes to support stateful workloads? What features are in the works to support stateful workloads better in the future?
Colocating Data Workloads and Web Services on Kubernetes to Improve Resource ...
Colocating Data Workloads and Web Services on Kubernetes to Improve Resource Utilization - He Cao, ByteDance
Recently, more and more data workloads are running on top of Kubernetes, such as ETL processes, Spark and Flink jobs, and more. These workloads typically exhibit high resource utilization and remain relatively stable over time. In contrast, web services often exhibit tidal patterns, characterized by significant fluctuations in resource utilization. The resource model of vanilla Kubernetes is static, which can lead to low resource utilization accumulated over 24 hours. In this talk, He will introduce how ByteDance uses Katalyst to colocate data workloads and online services on Kubernetes to improve resource utilization. In addition, He will explain how Katalyst ensures the QoS of these workloads through QoS-aware scheduling, service profiling, multi-dimensional resource isolation, real-time container resource adjustment, and more. In ByteDance, Katalyst has been deployed on 500,000+ nodes with tens of millions of cores, and has improved daily resource utilization from 20% to 60%.
Make Your Kafka Cluster Production-Ready
Make Your Kafka Cluster Production-Ready - Jakub Scholz, Red Hat
Kubernetes became the de-facto standard for running cloud-native applications. And more and more users turn to it also to run stateful applications such as Apache Kafka. While there are different tools such as Helm charts or operators which can get you quickly up and running, there is often still a long way to make sure the Kafka cluster is production-ready. This talk will take you through the main aspects you should consider for your Kafka cluster and will cover things such as resource management, storage, scheduling, rolling updates, or reliability. It will show you how to do it using the Strimzi operator, but the lessons learned will apply also to any other Kafka cluster. If you are interested in production-ready Apache Kafka on Kubernetes, this is a talk for you.
Dynamic Large Scale Spark on Kubernetes: Empowering the Community with Argo W...
Dynamic Large Scale Spark on Kubernetes: Empowering the Community with Argo Workflows and Argo Events - Ovidiu Valeanu, AWS & Vara Bonthu, Amazon
Are you eager to build and manage large-scale Spark clusters on Kubernetes for powerful data processing? Whether you are starting from scratch or considering migrating Spark workloads from existing Hadoop clusters to Kubernetes, the challenges of configuring storage, compute, networking, and optimizing job scheduling can be daunting. Join us as we unveil the best practices to construct a scalable Spark clusters on Kubernetes, with a special emphasis on leveraging Argo Workflows and Argo Events. In this talk, we will guide you through the journey of building highly scalable Spark clusters on Kubernetes, using the most popular open-source tools. We will showcase how to harness the potential of Argo Workflows and Argo Events for event-driven job scheduling, enabling efficient resource utilization and seamless scalability. By integrating these powerful tools, you will gain better control and flexibility for executing Spark jobs on Kubernetes.
Run PostgreSQL in Warp Speed Using NVMe/TCP in the Cloud
Run PostgreSQL in Warp Speed Using NVMe/TCP in the Cloud - Sagy Volkov, Lightbits
PostgreSQL as a SQL engine can accommodate a very high-transaction rate, but as your data grows and the number of connections and queries increases, there is a challenge for the storage to keep up with the SQL engine.
To the rescue comes NVMe over TCP (or NVMe/TCP). Developed by Lightbits Labs in 2016 and donated to the Linux community, it is the next evaluation of using NVMe based storage over TCP Fabric. NVMe/TCP simplifies how you interact with remote NVMe devices (targets) and allows your PostgreSQL storage to consume fast storage very easily.
In this session I will explain the core concept of the NVMe/TCP protocol, current storage providers that can use it, how you can consume it in Kubernetes (super easy), and discuss the possibilities of using NVMe/TCP in the cloud.
The session will also include a performance comparison of a few storage that are available in AWS and even a live demo of how PostgreSQL can run super fast - warp speed fast - in AWS.
The Kubernetes Native Database
Link: https://www.youtube.com/watch?v=D8kJCvsHD9Q&list=PLHgdNuGxrJt04Fwaip9aDYvXrbRSmc5HZ&index=12
https://go.dok.community/slack
https://dok.community/
From DoK Day NA 2022 (https://www.youtube.com/watch?v=YWTa-DiVljY&list=PLHgdNuGxrJt04Fwaip9aDYvXrbRSmc5HZ)
In the software industry we’re fond of terms that define major trends, like “cloud native”, “Kubernetes native” and “serverless”. As more and more organizations move stateful workloads to Kubernetes, we’ve started to see these terms applied to data infrastructure, where they can get overtaken by marketing hype unless we work to define them.
In this talk, we’ll examine two different databases, TiDB and Apache Cassandra, in order to identify what it means for a database to be Kubernetes native and why it matters. We’ll look at points including:
- The differences between cloud native, Kubernetes native, and serverless
- How databases become Kubernetes native
- Benefits of Kubernetes native databases
- How Kubernetes can better support databases
-----
Jeff has worked as a software engineer and architect in multiple industries and as a developer advocate helping engineers get up to speed on Apache Cassandra. He's involved in multiple open source projects in the Cassandra and Kubernetes ecosystems including Stargate and K8ssandra. Jeff is the author of the O’Reilly books “Cassandra: The Definitive Guide" and “Managing Cloud Native Data on Kubernetes".
ING Data Services hosted on ICHP DoK Amsterdam 2023
An explanation of how ING deals with local persistence at scale in secure and compliant manner for Elastic and Prometheus workloads today and other Data Services in the future.
In more detail we will elaborate on the following topics
How we solve local persistence
Type of workloads now and in the future
Typical requirements for a banking environment
Automation
Scale
Resilience
Security / Compliance
Service offering / demarcation
About Tor and Luuk:
Tor and Luuk are experienced engineers working at ING for over 10 years and working in the Kubernetes area for the last 5 years. They are specialized in and responsible for the Data Services OpenShift clusters in ING and have a strong focus on resilience, automation and security.
Implementing data and databases on K8s within the Dutch government
A small walkthrough of projects within the dutch government running Data(bases) on OpenShift. This talk shares success stories, provides a proven recipe to `get it done` and debunks some of the FUD.
About Sebastiaan:
I have always been a weird DBA, trying to combine Databases with out-of-the-box thinking and a DevOps mindset. Around 2016 I fell in love with both Postgres and Kubernetes, and I then committed my life to enabling Dutch organisations with running their Database workloads CloudNative.
Over the last few years I worked as a private contractor for 2 large government agencies doing exactly that, and I want to share my and others (success stories) hoping to enable and inspire Data on Kubernetes adoption.
StatefulSets in K8s - DoK Talks #154
https://go.dok.community/slack
https://dok.community/
Link: https://youtu.be/n_thXwyJNSU
ABSTRACT OF THE TALK
Deploying Stateless applications is easy but this is not the case for Stateful applications. StatefulSets are the K8s API object that helps to manage stateful application. Learn about what Stateful sets are, how to create, How it differs from Deployments.
KEY TAKE-AWAYS FROM THE TALK
This talk is focused on basics of StatefulSet, how StatefulSet differs from Deployments, How to manage Stateful app using StatefulSet
Analytics with Apache Superset and ClickHouse - DoK Talks #151
Link: https://youtu.be/Y-1uFVKDfgY
https://go.dok.community/slack
https://dok.community/
ABSTRACT OF THE TALK
This talk concerns performing analytical tasks with Apache Superset with ClickHouse as the data backend. ClickHouse is a super fast database for analytical tasks, and Apache Superset is an Apache Software foundation project meant for data visualization and exploration. Performing analytical tasks using this combo is super fast since both the software are designed to be scalable and capable of handling data of petabyte scale.
Overcoming challenges with protecting and migrating data in multi-cloud K8s e...
Link: https://youtu.be/EFaRyl4HmmE
https://go.dok.community/slack
https://dok.community/
ABSTRACT OF THE TALK
If you are running or planning a multi-cloud or even a multi-cluster environment, there are several considerations in implementing a data protection solution – especially if you plan on an organic home-grown, do-it-yourself option. This talk will highlight challenges and best practices around centralized management of configuration, credentials, compliance across multiple accounts, regions, providers etc. We will also highlight the deviations in CSI driver implementations of various storage vendors and cloud providers. Finally, we will cover the various recovery options available in the market today.
Kubernetes cloud services are popular since they mitigate, but do not eliminate, the difficulties of operating a Kubernetes environment. This is especially true for protecting the stateful configuration and data of your Kubernetes applications, where the inherent high-availability and infrastructure as code are not a substitute for have cloud-native backup and disaster recovery capabilities. Further, many companies now have multi-cloud strategies for their cloud-native applications. These challenges can be addressed with backup applications that are both Kubernetes managed service and multi-cloud aware in order to snapshot, copy, restore, and migrate Kubernetes workloads (resources and data) running on AKS, EKS and GKE. Capturing information from cloud accounts and how the cluster and storage resources are configured allows 1) centralized visibility into all cloud accounts and the clusters and resources in the accounts including for compliance; 2) cross-account, cross-cluster, and cross-region data restores; 3) automation of the cluster and data restores including for Dev, Test, and Production recovery use cases.
BIO
Sebastian Glab is a Cloud Architect for CloudCasa and he resides in Poland. He is responsible for integrating the different cloud providers with the CloudCasa service, and making sure that all clusters in the cloud service get discovered and protected. In his free time, he plays volleyball and develops his own projects.
Martin Phan is the Field CTO in North America for CloudCasa by Catalogic Software. With over 20+ years of experience in the software-industry, he takes pride in supporting, developing, implementing, and selling enterprise software and data protection solutions to help customer solve their backup and recovery challenges.
KEY TAKE-AWAYS FROM THE TALK
1) Challenges and best practices around centralized management of configuration, credentials, compliance across multiple accounts, regions, providers etc.
2) Advantages of cloud awareness and Kubernetes managed service awareness for application and data recovery and security
3) Examples of overcoming Container Storage Interface (CSI) deviations
4) Various recovery options available in the market today.
Evaluating Cloud Native Storage Vendors - DoK Talks #147
Link: https://youtu.be/YVXEpcSclwY
https://go.dok.community/slack
https://dok.community/
ABSTRACT OF THE TALK
In a continuation of a talk given at DoK day at KubeCon EU 2022, join Dinesh Majrekar, Civo's CTO as they walk through their evaluation process of the CNCF Storage market.
Civo offers managed Kubernetes clusters powered by K3s to customers around the world. We manage thousands of Virtual Machines and stateful customer data within multiple data centres across several continents.
In late 2021, Civo had the opportunity to evaluate the CNCF storage landscape to move to a new technology stack. During the migration project, Civo evaluated Mayastor, Ondat, Ceph and Longhorn against the following metrics:
Scalability
Performance
Ease of Support
Attendants will see practical examples on how they could carry out their own similar evaluation and see some of the results of the Civo research project.
BIO
Dinesh is CTO at Civo. Having worked in the hosting industry for many years, Dinesh has a passion for creating solutions that operate at scale. This not only applies to the technology stack, but for nurturing engineers through their career.
Kubernetes Cluster Upgrade Strategies and Data: Best Practices for your State...
Link: https://youtu.be/qUW8LkxYayc
https://go.dok.community/slack
https://dok.community/
ABSTRACT OF THE TALK
How do you make sure your Stateful Workloads remain available when your Kubernetes infrastructure updates? This talk will discuss different strategies of upgrading a Kubernetes cluster, and how you can manage risk for your workload. The talk will showcase demos of each upgrade strategy.
BIO
Peter is a Senior Software Engineer on GKE at Google. He works on improving Kubernetes for Stateful workloads. His main focus is on enhancing the Kubernetes ecosystem for high availability applications.
KEY TAKE-AWAYS FROM THE TALK
The mechanics of different upgrade strategies, when to apply a particular upgrade strategy depending on your Stateful workload and how to mitigate risk to your application’s availability.
We will Dok You! - The journey to adopt stateful workloads on k8s
Link: https://youtu.be/AjvwG53yLMY
https://go.dok.community/slack
https://dok.community/
ABSTRACT OF THE TALK
Stateful workloads are the heart of any application, yet they remain confusing and complicated even to daily K8s practitioners. That’s why many organizations shy away from migrating their data - their prized possession - to the unfamiliar stateful realm of Kubernetes.
After meeting with many organizations in the adoption phase, I discovered what works best, what to avoid, and how critical it is to gain confidence and the right knowledge in order to successfully adopt stateful workloads.
In this talk I will demonstrate how to optimally adopt Kubernetes and stateful workloads in a few steps, based on what I’ve learned from observing dozens of different adoption journeys. If you are taking your first steps in data on K8s or contemplating where to start - this talk is for you!
BIO
- A Developer turned Solution Architect.
- Working at Komodor, a startup building the first K8s-native troubleshooting platform.
- Love everything in infrastructure: storage, networks & security - from 70’s era mainframes to cloud-native.
- All about “plan well, sleep well”.
KEY TAKE-AWAYS FROM THE TALK
- Understand how critical stateful workloads are for any system, and that the key challenges to migrating it to Kubernetes are knowledge and confidence.
- How to build the foundational knowledge required to overcome adoption challenges by creating a learning path for individuals and teams.
- How to gain confidence to run stateful workloads on Kubernetes with support from the community (and yourself!)
Mastering MongoDB on Kubernetes, the power of operators
Link: https://youtu.be/Pi5ueyl_1jU
https://go.dok.community/slack
https://dok.community/
ABSTRACT OF THE TALK
During my first talk for DoK community I want to walk you through the world of NoSQL database MongoDB and Kubernetes Operators - Community Edition, Enterprise Edition (MongoDB and Ops Manager on K8s), and Atlas operator, highlight the most important capabilities, talk about use cases and challenges, the theory will be mixed with a live demos!
BIO
I'm a SRE / NoSQL / DevOps professional. I hold CKA, CKAD, CKS, also I’m MongoDB Certified DBA and MongoDB Champion. I have experience with multiple cloud providers, Kubernetes, different types of K8s operators (Strimzi, RabbitMQ Cluster Operator), but especially MongoDB K8s Operator. I also work with KEDA. Since 2017, I have been a speaker at MongoDB conferences all around the world (USA, China, Europe).
KEY TAKE-AWAYS FROM THE TALK
I would like to share the best practices of running NoSQL database - MongoDB on Kubernetes also I want to show how to manage Atlas (MongoDB cloud) via K8s operator
https://www.mongodb.com/developer/community-champions/arkadiusz-borucki/
Leveraging Running Stateful Workloads on Kubernetes for the Benefit of Develo...
Link: https://youtu.be/KUipuM3UJF4
https://go.dok.community/slack
https://dok.community/
DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)
Kubernetes comes with a lot of useful features like Volumes and StatefulSets, which make running stateful workloads simple. Interestingly, when combined with the right tools, these features can make Kubernetes very valuable for developers wanting to run massive production databases in development! This is exactly what was seen at "Extendi".
The developers at Extendi deal with a large amount of data in their production Kubernetes clusters. But when developing locally, they didn't have an easy way of replicating this data. This replication was needed because it allowed developers to test new features instantaneously without worrying if they would work as expected when pushed to production. But replicating a 100Gb+ production database for development wasn't turning out to be an easy task!
This is where leveraging Kubernetes + remote development environments came to the rescue. Running data on Kubernetes turned out to be way faster than any of the traditional approaches because of Kubernetes' ability to handle stateful workloads exceptionally well. And since Extendi already used Kubernetes in production - the setup process was fairly simple.
This talk will cover practical steps on how leveraging Kubernetes based development environments allowed dev teams at Extendi to run production data on Kubernetes during development using features like Volume Snapshots, having a huge positive impact on developer productivity.
-----
Arsh is a Developer Experience Engineer at Okteto. He is an active contributor to the upstream Kubernetes project and was awarded the Kubernetes Contributor Award for his contributions in 2021. Arsh has written blogs and spoken about different topics in the cloud-native ecosystem at various conferences before, including KubeCon + CloudNativeCon + Open Source Summit China 2021. He has also been on the Kubernetes Release Team since the 1.23 release. He also serves as the New Contributor Ambassador for the Documentation Special Interest Group of the Kubernetes project and continuously mentors new folks in the community. Previously, he worked at VMware and was an active contributor to other CNCF projects, including cert-manager and Kyverno.
-----
Lapo is a Software Engineer currently leading the development team of a Social Listening and Audience Intelligence platform. He started coding at the early age of 14 and since he turned his passion into a real job, he has always been looking for boosting his knowledge by constantly researching for newer and newer technologies.
Active on Ruby Open Source projects
-----
Ramiro Berrelleza is one of the founders of Okteto. He has spent most of his career (and his free time) building cloud services and developer tools. Before starting Okteto, Ramiro was an Architect at Atlassian and a Software Engineer at Microsoft Azure.
More from DoKC (20)
Is It Safe? Security Hardening for Databases Using Kubernetes Operators
Is It Safe? Security Hardening for Databases Using Kubernetes Operators
Stop Worrying and Keep Querying, Using Automated Multi-Region Disaster Recovery
Stop Worrying and Keep Querying, Using Automated Multi-Region Disaster Recovery
Transforming Data Processing with Kubernetes: Journey Towards a Self-Serve Da...
Transforming Data Processing with Kubernetes: Journey Towards a Self-Serve Da...
Colocating Data Workloads and Web Services on Kubernetes to Improve Resource ...
Colocating Data Workloads and Web Services on Kubernetes to Improve Resource ...
Dynamic Large Scale Spark on Kubernetes: Empowering the Community with Argo W...
Dynamic Large Scale Spark on Kubernetes: Empowering the Community with Argo W...
Run PostgreSQL in Warp Speed Using NVMe/TCP in the Cloud
Run PostgreSQL in Warp Speed Using NVMe/TCP in the Cloud
ING Data Services hosted on ICHP DoK Amsterdam 2023
ING Data Services hosted on ICHP DoK Amsterdam 2023
Implementing data and databases on K8s within the Dutch government
Implementing data and databases on K8s within the Dutch government
Analytics with Apache Superset and ClickHouse - DoK Talks #151
Analytics with Apache Superset and ClickHouse - DoK Talks #151
Overcoming challenges with protecting and migrating data in multi-cloud K8s e...
Overcoming challenges with protecting and migrating data in multi-cloud K8s e...
Evaluating Cloud Native Storage Vendors - DoK Talks #147
Evaluating Cloud Native Storage Vendors - DoK Talks #147
Kubernetes Cluster Upgrade Strategies and Data: Best Practices for your State...
Kubernetes Cluster Upgrade Strategies and Data: Best Practices for your State...
We will Dok You! - The journey to adopt stateful workloads on k8s
We will Dok You! - The journey to adopt stateful workloads on k8s
Mastering MongoDB on Kubernetes, the power of operators
Mastering MongoDB on Kubernetes, the power of operators
Leveraging Running Stateful Workloads on Kubernetes for the Benefit of Develo...
Leveraging Running Stateful Workloads on Kubernetes for the Benefit of Develo...
Recently uploaded
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Dmitrii Kamaev, PhD
Senior Product Owner - QIAGEN
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
The typical problem in product engineering is not bad strategy, so much as “no strategy”. This leads to confusion, lack of motivation, and incoherent action. The next time you look for a strategy and find an empty space, instead of waiting for it to be filled, I will show you how to fill it in yourself. If you’re wrong, it forces a correction. If you’re right, it helps create focus. I’ll share how I’ve approached this in the past, both what works and lessons for what didn’t work so well.
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
What is an RPA CoE? Session 1 – CoE Vision
In the first session, we will review the organization's vision and how this has an impact on the COE Structure.
Topics covered:
• The role of a steering committee
• How do the organization’s priorities determine CoE Structure?
Speaker:
Chris Bolin, Senior Intelligent Automation Architect Anika Systems
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Join us to learn how UiPath Apps can directly and easily interact with prebuilt connectors via Integration Service--including Salesforce, ServiceNow, Open GenAI, and more.
The best part is you can achieve this without building a custom workflow! Say goodbye to the hassle of using separate automations to call APIs. By seamlessly integrating within App Studio, you can now easily streamline your workflow, while gaining direct access to our Connector Catalog of popular applications.
We’ll discuss and demo the benefits of UiPath Apps and connectors including:
Creating a compelling user experience for any software, without the limitations of APIs.
Accelerating the app creation process, saving time and effort
Enjoying high-performance CRUD (create, read, update, delete) operations, for
seamless data management.
Speakers:
Russell Alfeche, Technology Leader, RPA at qBotic and UiPath MVP
Charlie Greenberg, host
"Choosing proper type of scaling", Olena Syrota
Imagine an IoT processing system that is already quite mature and production-ready and for which client coverage is growing and scaling and performance aspects are life and death questions. The system has Redis, MongoDB, and stream processing based on ksqldb. In this talk, firstly, we will analyze scaling approaches and then select the proper ones for our system.
Main news related to the CCS TSI 2023 (2023/1695)
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
Choosing The Best AWS Service For Your Website + API.pptx
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
The Microsoft 365 Migration Tutorial For Beginner.pptx
This presentation will help you understand the power of Microsoft 365. However, we have mentioned every productivity app included in Office 365. Additionally, we have suggested the migration situation related to Office 365 and how we can help you.
You can also read: https://www.systoolsgroup.com/updates/office-365-tenant-to-tenant-migration-step-by-step-complete-guide/
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Inconsistent user experience and siloed data, high costs, and changing customer expectations – Citizens Bank was experiencing these challenges while it was attempting to deliver a superior digital banking experience for its clients. Its core banking applications run on the mainframe and Citizens was using legacy utilities to get the critical mainframe data to feed customer-facing channels, like call centers, web, and mobile. Ultimately, this led to higher operating costs (MIPS), delayed response times, and longer time to market.
Ever-changing customer expectations demand more modern digital experiences, and the bank needed to find a solution that could provide real-time data to its customer channels with low latency and operating costs. Join this session to learn how Citizens is leveraging Precisely to replicate mainframe data to its customer channels and deliver on their “modern digital bank” experiences.
JavaLand 2024: Application Development Green Masterplan
My presentation slides I used at JavaLand 2024
Dandelion Hashtable: beyond billion requests per second on a commodity server
This slide deck presents DLHT, a concurrent in-memory hashtable. Despite efforts to optimize hashtables, that go as far as sacrificing core functionality, state-of-the-art designs still incur multiple memory accesses per request and block request processing in three cases. First, most hashtables block while waiting for data to be retrieved from memory. Second, open-addressing designs, which represent the current state-of-the-art, either cannot free index slots on deletes or must block all requests to do so. Third, index resizes block every request until all objects are copied to the new index. Defying folklore wisdom, DLHT forgoes open-addressing and adopts a fully-featured and memory-aware closed-addressing design based on bounded cache-line-chaining. This design offers lock-free index operations and deletes that free slots instantly, (2) completes most requests with a single memory access, (3) utilizes software prefetching to hide memory latencies, and (4) employs a novel non-blocking and parallel resizing. In a commodity server and a memory-resident workload, DLHT surpasses 1.6B requests per second and provides 3.5x (12x) the throughput of the state-of-the-art closed-addressing (open-addressing) resizable hashtable on Gets (Deletes).
FREE A4 Cyber Security Awareness Posters-Social Engineering part 3
Free A4 downloadable and printable Cyber Security, Social Engineering Safety and security Training Posters . Promote security awareness in the home or workplace. Lock them Out From training providers datahops.com
Skybuffer SAM4U tool for SAP license adoption
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
TrustArc Webinar - 2024 Global Privacy Survey
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
Presentation of the OECD Artificial Intelligence Review of Germany
Consult the full report at https://www.oecd.org/digital/oecd-artificial-intelligence-review-of-germany-609808d6-en.htm
Leveraging the Graph for Clinical Trials and Standards
Katja Glaß
OpenStudyBuilder Community Manager - Katja Glaß Consulting
Marius Conjeaud
Principal Consultant - Neo4j
Recently uploaded (20)
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
FREE A4 Cyber Security Awareness Posters-Social Engineering part 3
FREE A4 Cyber Security Awareness Posters-Social Engineering part 3
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
Why run Postgres in Kubernetes?
- 1. Why run Postgres in Kubernetes? Gabriele Bartolini, VP & CTO of Cloud Native at EDB DoK Day EU 2022 @ KubeCon - May 16 #DOKCDAY #CloudNativePG
- 2. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved About me Gabriele Bartolini ● VP and CTO of Cloud Native at EDB ○ Previously at 2ndQuadrant, from 2008 to 2020 ● PostgreSQL user since ~2000 ○ Community member since 2006 ○ Co-founder of PostgreSQL Europe ● DevOps evangelist ● Open source contributor ○ Barman ○ … 2 Twitter: @_GBartolini_ / @EDBPostgres
- 3. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved 3 Important announcement in this presentation!
- 4. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved What I won’t be covering in this presentation 4 ● Introduction to PostgreSQL ● Why Postgres is the most popular database in VM and bare metal contexts ● The rise of stateful workloads in Kubernetes ○ This is DoK Day after all! ● The operator pattern Due to time constraints
- 5. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved 5
- 6. “ PostgreSQL and Kubernetes complement each other, potentially bringing the best overall experience of a PostgreSQL database out there Gabriele Bartolini, VP Cloud Native at EDB
- 7. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved The need for a good Postgres operator Covering all aspects of the lifecycle of a database, including: 7 ● PostgreSQL native streaming replication ● Declarative configuration ● Container immutability ● Integration with the Kubernetes API server ○ Immediate detection of failures via liveness and readiness probes ○ Prompt reaction for self-healing, including automated failover ● Support for Continuous Backup and Point In Time Recovery (PITR) ○ Global RPO consistently below 5/10 minutes! ● Pervasive security ● Rolling updates ● Ensures seamless integration of PostgreSQL clusters in the Kubernetes infrastructure: ○ Monitoring, alerting, logging, audit
- 8. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved 8 Why run Postgres in Kubernetes?
- 9. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved 9
- 10. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved Announcing CloudNativePG 10 Originally created and sponsored by A new open source operator managing Postgres workloads ● Website and docs: cloudnative-pg.io ● Github repository: github.com/cloudnative-pg/cloudnative-pg ● License: Apache License 2.0 ● Owned by an openly governed and vendor-neutral community ○ Join now! ○ Enter the slack channel ○ Follow us on Twitter (@CloudNativePG) ● We have officially applied for the CNCF Sandbox ○ We need your help and support!
- 11. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved Declarative configuration 11 # Postgres cluster with 1 primary and 2 replicas # each with 10GB storage apiVersion: postgresql.cnpg.io/v1 kind: Cluster metadata: name: acdc spec: instances: 3 storage: size: 10Gi
- 12. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved Freedom of choice 12 Postgres in Kubernetes (with CloudNativePG) reduces risk of vendor lock-in ● Run Kubernetes everywhere you want ○ On premise, in the public cloud, on both (hybrid cloud), on different cloud providers (multi-cloud) ● Manage Kubernetes by yourself or let others do it for you ● Manage Postgres by yourself, or rely on a third party organization ● Adopt vanilla Kubernetes or a third party distribution ● Run Kubernetes on VM-based environments or bare metal ● Run Postgres in: ○ shared architectures ○ shared nothing architectures It’s your choice!
- 13. 2021 Copyright © EnterpriseDB Corporation All Rights Reserved Thank you! EDB: enterprisedb.com CloudNativePG: cloudnative-pg.io If you are in Valencia for KubeCon 2022, come and visit us at the EDB stand!