Download to read offline
Uploaded byanynines GmbH
Automating the Entire PostgreSQL Lifecycle
This document discusses automating the lifecycle of PostgreSQL databases. It recommends using BOSH to automate provisioning, configuration, backups, upgrades, and other lifecycle tasks across infrastructure. Key points covered include: - BOSH allows automating the PostgreSQL lifecycle in a repeatable, scalable way across platforms. - It provisions VMs with persistent disks to decouple data from VM lifecycles. - Automation handles high availability clustering, failover, backups to object storage, and vertical/horizontal scaling. - The approach makes dedicated PostgreSQL instances on-demand and automates their full lifecycles.
More Related Content
Similar to Automating the Entire PostgreSQL Lifecycle
![20260201 [FOSDEM] gomodjail - library sandboxing for Go modules.pdf](https://cdn.slidesharecdn.com/ss_thumbnails/20260201fosdemgomodjail-librarysandboxingforgomodules-260201225659-76609ec4-thumbnail.jpg?width=640&height=640&fit=bounds)
Automating the Entire PostgreSQL Lifecycle
- 1.
- 2.
- 3. • Transform theway databases are being operated. • Looking at PostgreSQL and see what automation its lifecycle means. • Select a tooling to work with. • Draft an architectural outline. • Show use case examples for handling the PostgreSQL lifecycle in a fully automated way.
- 4.
- 7. –The anynines missionstatement “Automate the entire lifecycle at production grade of a growing number of data services across platforms, infrastructures at scale.”
- 8.
- 9. • Infrastructure • Platform •(Data) Service • (Data) Service Instance • Service Binding • Cluster = Streaming Replication PostreSQL Cluster
- 10.
- 11. “Database Administration is Subjectto Change.”
- 12. • Application developmentplatforms. CF, k8s, … • Microservices • More Apps • More Data Service Types • More Data Service Instances Change
- 13.
- 14.
- 15.
- 16.
- 17.
- 18. Automate the entirelifecycle of PostgreSQL to easily operate thousands of production grade DBs across …
- 19.
- 20. and integrate wellwith multiple platforms
- 21.
- 22.
- 23. • Provision databaseVM • Install database software • Configure database software • Setup replication & cluster mangement • Configure Monitoring • Configure availability monitoring • Configure alerting • Configure collection of logs • Configure collection of metrics • Create database • Create database users • Adapt configuration of individual database servers • e.g. enable extension • Adapt configuration of individual databases • Setup backup procedure • Perform backups regularly • Perform on-demand backups • Perform schema migrations • Query data • Determine performance bottleneck • Perform scale-out of database server • Recover from master db failure • Recover from standby failure • Recover from AZ failure • Recover from network partitioning (split brain) • Recover from PostgreSQL process failure • Perform upgrade of operating system • Restore data from backup • Apply patch-level upgrade • Apply major version upgrade • Perform data migration • Destroy data • Destroy database server PostgreSQL Lifecycle
- 24. PostgreSQL is not easyto automate.
- 25.
- 26.
- 27. Is there asingle strategy that works at production grade across data services at scale?
- 28.
- 29. Easy Deployment $> cfcreate-service postgresql single-small single-postgres-1
- 30. single-postgres-1 Postgresql VM#1 Service Instance #1 ServiceBroker PostgreSQL Automation
- 31. Easy Deployment $> cfcreate-service postgresql cluster-small postgres-cluster-2
- 32. single-postgres-1 Postgresql VM#1 Service Instance #1 ServiceBroker PostgreSQL Automation postgres-cluster-2 Postgresql VM#1 Postgresql VM#2 Postgresql VM#3 Service Instance #2
- 33. Data ServicesApplication Runtime AppApp App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App App
- 34. Data Service Automation Lifecycle vs. DataService Instance Lifecycle
- 35.
- 36. Test a9s Release Build Upstream Release Platform Environment #1 Platform Environment #2 Platform Environment#n Ship Automation Releases into Platform Environments Update Data Service Instances a9s PostgreSQL Release Repository Open Source PostgreSQL Building new PostgreSQL automation releases PostgreSQL CI/CD-Pipeline
- 37.
- 38.
- 39. Principles - FullLifecycle Management Iteratively increase the automation depth.
- 40. Create Modify Update Backup Recove ry Termina te Enabling/disabling data service plugins Scale-outScale down Add/remove a log/metric endpoint Major version update Misc. config changes Greenfield / Clone Single / clustered pre-provisioned / on-demand Trigger manual backup Scheduled backup Restore backup Hard reset of service instance Destroy service instance Minor version update Patchlevel version update Cluster Failure detection Cluster failover Self-healing by resurrection PostgreSQL Lifecycle - nth Iteration
- 41.
- 42. a9s Deployer Templates Deployments Bosh|| k8s a9s Service Broker my-3node-postgres-cluster-2 Postgresql VM#1 Postgresql VM#2 Postgresql VM#3 my-single-postgres-1 Postgresql VM#1 Middleware Adapter Open Service Broker API a9s PostgreSQL SPI Service InstanceService Instance my-3node-postgres-cluster-3 Postgresql VM#1 Postgresql VM#2 Postgresql VM#3 Service Instance … Cloud ControllerCF Client create service create service create deployment from template xy with attributes {…} deploy release abc & deployment manifest xyz Execute deployments create service specific credentials create binding
- 44. HTTP Verb Action ServiceCatalog GET /v2/catalog Deliver meta data about the data service. Create Service Instance PUT /v2/service_instances/:id Provision a VM, install and configure a data service VMs / Cluster representing a service instance. Create Service Binding PUT /v2/service_instances/:instance_id/service_bindings/:id Create a data service user and return credentials representing a service binding. Delete Service Binding DELETE /v2/service_instances/:instance_id/service_bindings/:id Remove credentials associated with the service binding. Delete Service Instance DELETE /v2/service_instances/:id Destroy the VMs and data associated with the service instance.
- 45.
- 46.
- 47.
- 48.
- 49.
- 50.
- 51.
- 52.
- 53.
- 54.
- 55. VMs Containers Strong isolation. Bad Neighborhood Protection. Faster dev cycles. Lessoverhead. Faster instance startup. Weak isolation.
- 56.
- 57.
- 58. “„BOSH let’s you orchestratethe lifecycle of large-scale deployments of stateful distributed systems to infrastructure.“”
- 59.
- 60. VMware BOSH BOSH CLI $> boshtarget http://bosh-on.vmware.com$> bosh deploy Some Service / App BOSH Agent VIRTUAL MACHINE Some Service / App BOSH Agent VIRTUAL MACHINE Some Service / App BOSH Agent VIRTUAL MACHINE Some Service / App BOSH Agent VIRTUAL MACHINE Some Service / App BOSH Agent VIRTUAL MACHINE Some Service / App BOSH Agent VIRTUAL MACHINE OpenStack BOSH AWS BOSH
- 61. BOSH CLI VMware AWSOpenStack $> bosh target http://bosh-on.aws.com$> bosh deploy Some Service / App BOSH Agent VIRTUAL MACHINE Some Service / App BOSH Agent VIRTUAL MACHINE Some Service / App BOSH Agent VIRTUAL MACHINE Some Service / App BOSH Agent VIRTUAL MACHINE Some Service / App BOSH Agent VIRTUAL MACHINE Some Service / App BOSH Agent VIRTUAL MACHINE BOSH BOSH BOSH
- 62.
- 63. Where to storestate?
- 64. Store state ona remotely attached block device = persistent disk. 🔑
- 65. Infrastructure as aService (IaaS), e.g. OpenStack VIRTUAL DATACENTER Router STORAGE Storage Node Storage Node Storage Node HDD HDD HDD HDD HDD HDD HDD HDD HDD HDD HDD HDD HDD HDD HDD HDD HDD HDD Storage Volume Operating System VIRTUAL MACHINE Infrasstructure API
- 66. The data lifecyclehas been decoupled from the VM lifecycle ⇒ The VM becomes disposable. 🔑
- 67.
- 68.
- 69. Horizontal Scaling Some Service BOSH Agent VIRTUALMACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE Some Service BOSH Agent VIRTUAL MACHINE
- 70.
- 71.
- 72.
- 73.
- 74.
- 75. • Applies toclustered Data Service Instances • Asynchronous streaming replication • Three (3) Nodes: • One (1) Master Node • Two (2) Standby Nodes • Replication slots configured to avoid early recycling of master WAL segments High Availability & Replication
- 76.
- 77. • Extends PostgreSQLstreaming replication • Manages replication • Performs failure detection • Assists during failover by triggering leader election • Facilitates monitoring of the replication health & performance Repmgr
- 78.
- 79. • Detect networkpartitioning & split brain situations • Periodically checks the repmgr database and verifies replication status and cluster status • Fires alarm if • master is not followed by a majority • standby is following the wrong master Custom PostgreSQL Automation
- 80.
- 81.
- 82.
- 83.
- 84.
- 85.
- 86.
- 87.
- 88.
- 89.
- 90.
- 91.
- 92.
- 93. ScalabilityVertical-Scale Out $> cfservice-update single-postgres-1 -p single-large
- 94. Single node PostgreSQLservice instance > turned into a large PostgreSQL server. single-postgres- 1 Postgresql VM#1
- 95. What has happened duringthe vertical scale- out?
- 96. 4GB RAM 1 vCPU 10GBpersistent disk BOSH Agent VIRTUAL MACHINE 4 GB RAM, 1 vCPU 10 GB Persistent Disk Data PostgreSQL
- 97. BOSH Agent PostgreSQL VIRTUAL MACHINE 8GB RAM, 2 vCPUs 10 GB Persistent Disk Data BOSH Agent VIRTUAL MACHINE 4 GB RAM, 1 vCPU PostgreSQL 10 GB Persistent Disk Data 20 GB Persistent Disk Data
- 98.
- 99.
- 100. Add DB Userand Grand Access
- 101. ScalabilityDB User &DB Access $> cf bind-service my-app single-postgres-1
- 102.
- 103. Backup & Restorefor Application Developers
- 104. Backup & Restoreof the user’s service instances
- 105. Backup & Restorefor Platform Operators
- 106. • Disaster RecoveryPlan for Data Services • Backup & Restore for platform operators • Backup & Restore of • All service instances • Individual service instances Backup & Restore for Platform Operators
- 107.
- 108.
- 109. BOSH Agent VIRTUAL MACHINE 4GB RAM, 1 vCPU 10 GB Persistent Disk Data PostgreSQL Replication Manager Log Agent my-3node-postgres-cluster-1 Postgresql VM#1 Postgresql VM#2 Postgresql VM#3 Backup Agent
- 110. Data Filter Chain a9sPostgreSQL Service Instance #32 Data Stream Reader Postgresql VM#2 Data Filter Data Stream Writer Data Filter Backup Agent Object Store, e.g. AWS S3 Database Object Store, e.g. AWS S3
- 111. my-3node-postgres-cluster-1 Postgresql VM#1 Postgresql VM#2 Postgresql VM#3 my-3node-postgres-cluster-2 Postgresql VM#1 Postgresql VM#2 Postgresql VM#3 Backup requested a9s BackupManager a9s Backup API ⏰ Amazon S3 Backup scheduled Tell backup agent to perform backup Store encrypted backup to storage Backup🔐
- 112.
- 113. • Dedicated serviceinstances are mandatory. • On-demand provisioning is essential. • Choosing the right automation technology is key.
- 114. Full PostgreSQL lifecycle automationis feasible…
- 115. … and itis already happening!
- 116.
- 117.
Editor's Notes
- #2 introduction into data service automation focus on postgresql focus on the postgresql lifecycle how the lifecycle of postgresql can be automated.
- #4 Why? Change > Adoption needed.
- #8 Why? Change > Adoption needed.
- #13 More database types and more database instances are to be operated than ever before. They are needed at short hand.
- #15 BOSH on-demand dedicated on-demand dedicated clustered, bosh self-healing, back up restore
- #22 Challange: complexity of the lifecycle.
- #24 VM lifecycle: VM provisioning, VM self-healing, VM scaling, Software management: package management, dependencies, etc. Process management: starting and monitoring of processes PostgreSQL lifecycle: configuration, databases, backup, restore, users, permissions Cluster lifecycle: replication, failure detection, failover, recovery from standard failure scenarios: master failure, standby failure, az failure, network partitioning
- #25 Things to worry about when automating.
- #26 replication but no cluster management ugly upgrade paths … assumed presence of human dba in design
- #35 Solo, clustered Different vertical scales.
- #39 Then Focus on Data Service Instance Lifecycle for the remainder of the talk!
- #50 VMs, Software, Processes, Self-Healing
- #51 Two deployments are absolutely identical. Convergence vs. declarative automation.
- #52 Automate once, deploy thousands of times. The infrastructure is the limit.
- #53 Portability
- #58 But k8s is conceptually similar.
- #60 large-scale: several hundred VMs distributed systems: such as CF. Dozenz of components. Deployment to infrastructure: prefer the resulting platform for anything that does not need to run directly on infrastructures.
- #76 Limited time. Exemplary lifecycle aspect.
- #78 Replication slots: WAL files with data that have not been sent to the standby are kept until the standby nodes have received them
- #85 Similar to app instances, backing services should be self-healing.
- #86 In this case a cluster node fails.
- #87 A clustered service detects such a failure and performs an automatic failover. The service instance remains available hickups are possible and depend on the backing service type. This this failover is much faster than re-building VMs and even Containers. However it is more costly, as additional VMs and additional storage is required to store data replicas.
- #88 Automatic recovery from degraded mode. Data is held in remote storage. Instance could also be recovered in case the remote storage fails.
- #90 Similar to app instances, backing services should be self-healing.
- #91 In this case a cluster node fails.
- #92 A clustered service detects such a failure and performs an automatic failover. The service instance remains available hickups are possible and depend on the backing service type. This this failover is much faster than re-building VMs and even Containers. However it is more costly, as additional VMs and additional storage is required to store data replicas.
- #93 Automatic recovery from degraded mode. Data is held in remote storage. Instance could also be recovered in case the remote storage fails.
- #94 Limited time. Exemplary lifecycle aspect.
- #97 Things change. Apps and services grow and need to be scaled out.
- #101 Limited time. Exemplary lifecycle aspect.
- #102 Limited time. Exemplary lifecycle aspect.
- #103 Limited time. Exemplary lifecycle aspect.
- #104 Limited time. Exemplary lifecycle aspect.
- #106 Limited time. Exemplary lifecycle aspect.
- #114 back plugins, filters, output plugins