Zero Downtime Postgres Upgrades
•
5 likes•2,822 views
On the surface, the tech behind a payments API may look like any other startup’s. You'll probably find some Rails apps, a database, and a bunch of stuff off to the sides to glue it together. GoCardless found it's mostly not the tech that differs, but the approach. Using their high-availability Postgres cluster as a running example, they explore how reliability became so important to them, and dive into the most recent feature they built into the cluster: zero-downtime patch upgrades.
More Related Content
What's hot
What's hot (11)
Viewers also liked
Viewers also liked (20)
Similar to Zero Downtime Postgres Upgrades
Similar to Zero Downtime Postgres Upgrades (20)
More from Outlyer
More from Outlyer (20)
Recently uploaded
Recently uploaded (20)
Zero Downtime Postgres Upgrades
- 1. Zero-downtime Postgres upgrades Restarting databases without the apps noticing @ChrisSinjo
- 2. GOCARDLESS
- 3. POST /cash/monies HTTP/1.1 { amount: 100 } 💰💰💰
- 5. Uptime is 🔑
- 10. –Postgres “Speak to this one node.”
- 11. Client Postgres
- 14. Wake a human up
- 20. You gotta perform: - Many steps - In the right order - Perfectly
- 21. Don’t make a tired SRE think
- 22. Add automation
- 25. How do we know a node has failed?
- 30. Client PostgresPostgresPostgres Repl Repl Pacemaker Pacemaker Pacemaker
- 31. Client PostgresPostgresPostgres Repl Repl Pacemaker Pacemaker Pacemaker VIP
- 32. Client PostgresPostgresPostgres Repl Repl Pacemaker Pacemaker Pacemaker VIP
- 38. $💯
- 40. It kinda is
- 41. You gotta know: - Postgres - Distributed systems - Pacemaker
- 42. Get someone else to run it for you
- 43. Client PostgresPostgresPostgres Repl Repl Pacemaker Pacemaker Pacemaker VIP
- 47. Every move means a connection reset
- 48. Every move means dropped requests
- 49. POST /cash/monies HTTP/1.1 { amount: 100 } 💰💰💰
- 50. POST /cash/monies HTTP/1.1 { amount: 100 } 500 Internal Server Error
- 51. What does this mean for upgrades?
- 53. Client PostgresPostgresPostgres Pacemaker Pacemaker Pacemaker 9.4.9 9.4.9 9.4.9 VIP
- 54. Client PostgresPostgresPostgres Pacemaker Pacemaker Pacemaker 9.4.9 9.4.9 9.4.9 Repl Repl VIP
- 55. Client PostgresPostgresPostgres Pacemaker Pacemaker Pacemaker 9.4.10 9.4.9 9.4.10 Repl Repl VIP
- 56. Client PostgresPostgresPostgres Repl Repl VIP Pacemaker Pacemaker Pacemaker 9.4.10 9.4.9 9.4.10
- 57. Every upgrade means a connection reset
- 58. Every upgrade means dropped requests
- 59. POST /cash/monies HTTP/1.1 { amount: 100 } 500 Internal Server Error
- 61. 🙄
- 62. Not upgrading is never an option
- 65. Solution: ???
- 66. 1thing missing
- 70. Client PostgresPostgresPostgres Pacemaker Pacemaker Pacemaker PgBouncerPgBouncer PgBouncer VIP VIP
- 71. PgBouncer has This One Weird Trick™
- 72. PAUSE;
- 73. Client PostgresPostgresPostgres Pacemaker Pacemaker Pacemaker PgBouncerPgBouncer PgBouncer VIP VIP
- 74. Client PostgresPostgresPostgres Pacemaker Pacemaker Pacemaker PgBouncerPgBouncer PgBouncer VIP VIP PAUSE;
- 75. Client PostgresPostgresPostgres Pacemaker Pacemaker Pacemaker PgBouncerPgBouncer PgBouncer VIP PAUSE; VIP
- 76. Client PostgresPostgresPostgres Pacemaker Pacemaker Pacemaker PgBouncerPgBouncer PgBouncer VIP PAUSE; VIP
- 77. So what does this mean for upgrades?
- 78. Client PostgresPostgresPostgres Pacemaker Pacemaker Pacemaker PgBouncerPgBouncer PgBouncer VIP VIP
- 82. Client PostgresPostgresPostgres PgBouncerPgBouncer PgBouncer VIP 9.4.10 9.4.9 9.4.10 VIP PAUSE;
- 83. Client PostgresPostgresPostgres PgBouncerPgBouncer PgBouncer VIP 9.4.10 9.4.9 9.4.10 VIP RESUME;
- 84. Client PostgresPostgresPostgres PgBouncerPgBouncer PgBouncer VIP 9.4.10 9.4.10 9.4.10 VIP RESUME;
- 85. $💯
- 86. Caveats
- 87. Minor versions
- 88. 9.4.9 → 9.4.10
- 89. pglogical
- 91. while(running_queries): if(now > timeout): abandon_migration else: sleep(0.1) promote_new_primary
- 93. 7-10s total
- 94. $💯