Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Operating PostgreSQL at Scale with Kubernetes


Published on

The maturation of containerization platforms has changed how people think about creating development environments and has eliminated many inefficiencies for deploying applications. These concept and technologies have made its way into the PostgreSQL ecosystem as well, and tools such as Docker and Kubernetes have enabled teams to run their own “database-as-a-service” on the infrastructure of their choosing.

All this sounds great, but if you are new to the world of containers, it can be very overwhelming to find a place to start. In this talk, which centers around demos, we will see how you can get PostgreSQL up and running in a containerized environment with some advanced sidecars in only a few steps! We will also see how it extends to a larger production environment with Kubernetes, and what the future holds for PostgreSQL in a containerized world.

We will cover the following:

* Why containers are important and what they mean for PostgreSQL
* Create a development environment with PostgreSQL, pgadmin4, monitoring, and more
* How to use Kubernetes to create your own "database-as-a-service"-like PostgreSQL environment
* Trends in the container world and how it will affect PostgreSQL

At the conclusion of the talk, you will understand the fundamentals of how to use container technologies with PostgreSQL and be on your way to running a containerized PostgreSQL environment at scale!

Published in: Technology
  • You might get some help from ⇒ ⇐ Success and best regards!
    Are you sure you want to  Yes  No
    Your message goes here
  • Have u ever tried external professional writing services like ⇒ ⇐ ? I did and I am more than satisfied.
    Are you sure you want to  Yes  No
    Your message goes here
  • Essay writing is really very hard for students and many students are really stressed to write it. So finding a best essay writing format is not easy. Anyway you can check any essay writing services and they can help you to find a good format to write the essay. I can recommend a site that has helped me. It's called Good luck!
    Are you sure you want to  Yes  No
    Your message goes here
  • great pres.
    Are you sure you want to  Yes  No
    Your message goes here

Operating PostgreSQL at Scale with Kubernetes

  1. 1. Operating PostgreSQL at Scale With Kubernetes JONATHAN S. KATZ MARCH 7, 2019 SCALE17X
  2. 2. • Director of Communications, Crunchy Data • Previously: Engineering leadership in startups • Longtime PostgreSQL community contributor • Advocacy & various committees for PGDG • @postgresql + .org content • Director, PgUS • Conference organization + speaking • @jkatz05 About Me 2
  3. 3. About Crunchy Data 3 Market Leading Data Security • Crunchy Certified PostgreSQL is open source and Common Criteria EAL 2+ Certified, with essential security enhancements for enterprise deployment • Author of the DISA Secure Technology Implementation Guide for PostgreSQL and co-author of CIS PostgreSQL Benchmark. Move ATO from weeks to days! Cloud Ready Data Management • Open source, Kubernetes-based solutions proven to scale to 1000s of database instances • Cloud-agnostic technology provide flexibility on how to deploy databases to public clouds, private clouds, or on-premise technology Leader in Open Source Enterprise PostgreSQL • Developer of essential open source tools for high availability, disaster recovery, and and monitoring for PostgreSQL • Leading contributor and sponsor of features that enhance stability, security, and performance of PostgreSQL
  4. 4. • Containers + PostgreSQL • Setting up PostgreSQL with Containers • Operating PostgreSQL at Scale With Kubernetes • Look Ahead: Trends in the Container World Outline 4
  5. 5. • Containers provide several advantages to running PostgreSQL: • Setup & distribution for developer environments • Ease of packaging extensions & minor upgrades • Separate out secondary applications (monitoring, administration) • Automation and scale for provisioning and creating replicas, backups Containers & PostgreSQL 5
  6. 6. • Containers also introduce several challenges: • Administrator needs to understand and select appropriate storage options • Configuration for individual database specifications and user access • Managing 100s - 1000s of containers requires appropriate orchestration (more on that later) • Still a database within the container; standard DBA tuning applies • However, these are challenges you will find in most database environments Containers & PostgreSQL 6
  7. 7. • We will use the Crunchy Container Suite • PostgreSQL (+ PostGIS): our favorite database; option to add our favorite geospatial extension • pgpool + pgbouncer: connection pooling, load balancing • pgBackRest: terabyte-scale disaster recovery management • Monitoring: pgmonitor • pgadmin4: UX-driven management • Open source! • Apache 2.0 license • Support for Docker 1.12+, Kubernetes 1.5+ • Actively maintained and updated Getting Started With Containers & PostgreSQL 7
  8. 8. Getting Started With Containers & PostgreSQL 8
  9. 9. Demo: Creating & Working With Containerized PostgreSQL 9 mkdir postgres && cd postgres docker volume create --driver local --name=pgvolume docker network create --driver bridge pgnetwork cat << EOF > pg-env.list PG_MODE=primary PG_PRIMARY_USER=postgres PG_PRIMARY_PASSWORD=password PG_DATABASE=whales PG_USER=jkatz PG_PASSWORD=password PG_ROOT_PASSWORD=password PG_PRIMARY_PORT=5432 PG_LOCALE=en_US.utf8 PGMONITOR_PASSWORD=monitorpassword EOF docker run --publish 5432:5432 --volume=pgvolume:/pgdata --env-file=pg-env.list --name="postgres" --hostname="postgres" --network="pgnetwork" --detach crunchydata/crunchy-postgres:centos7-11.2-2.3.1
  10. 10. Demo: Adding in pgadmin4 10 docker volume create --driver local --name=pga4volume cat << EOF > pgadmin4-env.list PGADMIN_SETUP_PASSWORD=securepassword SERVER_PORT=5050 EOF docker run --publish 5050:5050 --volume=pga4volume:/var/lib/pgadmin --env-file=pgadmin4-env.list --name="pgadmin4" --hostname="pgadmin4" --network="pgnetwork" --detach crunchydata/crunchy-pgadmin4:centos7-11.2-2.3.1
  11. 11. Demo: Adding Monitoring 11 cat << EOF > collect-env.list DATA_SOURCE_NAME=postgresql://ccp_monitoring:monitorpassword@postgres:5432/postgres?sslmode=disable EOF docker run --env-file=collect-env.list --network=pgnetwork --name=collect --hostname=collect --detach crunchydata/crunchy-collect:centos7-11.2-2.3.1 mkdir prometheus cat << EOF > prometheus-env.list COLLECT_HOST=collect SCRAPE_INTERVAL=5s SCRAPE_TIMEOUT=5s EOF docker run --publish 9090:9090 --env-file=prometheus-env.list --volume `pwd`/prometheus:/data --network=pgnetwork --name=prometheus --hostname=prometheus --detach crunchydata/crunchy-prometheus:centos7-11.2-2.3.1 mkdir grafana cat << EOF > grafana-env.list ADMIN_USER=jkatz ADMIN_PASS=password PROM_HOST=prometheus PROM_PORT=9090 EOF docker run --publish 3000:3000 --env-file=grafana-env.list --volume `pwd`/grafana:/data --network=pgnetwork --name=grafana --hostname=grafana --detach crunchydata/crunchy-grafana:centos7-11.2-2.3.1 1. Set up the metric collector 2. Set up prometheus to store metrics 3. Set up grafana to visualize
  12. 12. Running PostgreSQL on Kubernetes. At Scale.
  13. 13. • Value of Kubernetes increases exponentially as number of containers increases • Running databases on Kubernetes requires more specialized knowledge than running non- stateful applications • What happens to your data after a pod goes down? When to Use Kubernetes with PostgreSQL 13
  14. 14. • PostgreSQL Operator GA: March, 2017 • Allows an administrator to run PostgreSQL-specific commands to manage database clusters, including: • Creating / Deleting a cluster (your own DBaaS) • Scaling up / down replicas • High-Availability • Apply user policies to PostgreSQL instances • Managing backup intervals and policies • Define what container resources to use (RAM, CPU, etc.) • Upgrade management • Smart pod deployments to nodes • REST API Crunchy PostgreSQL Operator 14
  15. 15. • Utilizes Kubernetes Deployments: • Flexibility in storage classes • Flexibility in operating environments • Node affinity • Resource (CPU, RAM) configurations • Flexibility in database version runtimes Crunchy PostgreSQL Operator: Architecture 15
  16. 16. • Automation: Complex, multi-step DBA tasks reduced to one-line commands • Standardization: Many customizations, same workflow • Ease-of-Use: Simple CLI • Scale • Provision & manage clusters quickly amongst thousands of instances • Load balancing, disaster recovery, security policies, deployment specifications • Security: Sandboxed environments, RBAC, mass grant/revoke policies Why Use An Operator With PostgreSQL? 16
  17. 17. Why Use An Operator With PostgreSQL? 17
  18. 18. Demo: Provisioning a Cluster 18 pgo create cluster --autofail --pgbackrest --metrics --replica-count 1 scale17x pgo show cluster scale17x
  19. 19. Demo: Creating a User; Connectivity; Utilization 19 pgo create user jkatz scale17x --password password --managed --selector=name=scale17x pgo test scale17x pgo df scale17x
  20. 20. Demo: Running Some Tests; Utilization 20 # get the service forward command # run some pgbench pgbench -i -s 1 -h localhost -p 5434 userdb pgbench -c 2 -j 1 -t 128 --progress=1 -h localhost -p 5434 userdb pgbench -c 2 -j 1 -t 128 -S --progress=1 -h localhost -p 5434 userdb # Coming in 4.0: pgo benchmark! pgo df scale17x
  21. 21. Demo: Labels; Here is Where We Scale! 21 # labels pgo label scale17x --label=project=current pgo create cluster scale18x --labels project=future pgo create cluster scale19x --labels project=future pgo show cluster --selector=project=future pgo create user jkatz --password password --managed --selector=project=future pgo delete user jkatz --selector=project=future
  22. 22. Demo: High-Availability and Horizontal Scaling 22 # It's elastic! pgo scale scale17x --replica-count=1 # Run some queries on the replica # HA pgo failover scale17x --query pgo failover scale17x --autofail-replace-replica true --target <pod> pgo test scale17x
  23. 23. Demo: Setting Backup Policies 23 # backup policy pgo create schedule scale17x --schedule="0 0 * * *" --schedule-type=pgbackrest --pgbackrest-backup-type=full pgo create schedule scale17x --schedule="0 6,12,18 * * *" --schedule-type=pgbackrest --pgbackrest-backup-type=diff pgo show schedule scale17x
  24. 24. Demo: Disaster Strikes! 24 pgo backup scale17x --backup-type=pgbackrest # log in, do some stuff # oh no! restore # can choose to do point-in-time-recovery # pgo restore scale17x --backup-type=pgbackrest --pitr-target="2019-03-07 17:44:00" - backup-opts="--type=time" # or choose to back up up until the last archive # pgo restore scale17x --backup-type=pgbackrest
  25. 25. PostgreSQL & Containers: Looking Ahead
  26. 26. • Containers are no longer "new" - orchestration technologies have matured • Debate with containers + databases: storage & management • No different than virtual machines + databases • Databases are still databases: need expertise to manage • Stateful Sets vs. Deployments • Federation v2 API opens up new possibilities for high-availability • Database deployment automation flexibility • Deploy your architecture to any number of clouds • Monitoring: A new frontier Containerized PostgreSQL: Looking Ahead 26
  27. 27. Conclusion 27 • PostgreSQL + Containers + Kubernetes gives you: • Easy-to-setup development environments • Your own production database-as-a-service • Tools to automate management of over 1000s of instances in short-order
  28. 28. Jonathan S. Katz @jkatz05 Thank You!