Postgres Scaling Opportunities and Options
•
2 likes•1,408 views
Review this presentation to discover the whens and hows of scaling at it provides and overview of multi-session, single-session, and multi-host scaling options and the workloads where these options are appropriate. Database scaling refers to increasing database throughput. Vertical scaling utilizes additional resources such as I/O, memory, CPU, while horizontal scaling uses additional computers. However, the high concurrency and write requirements of database servers make scaling a challenge. Sometimes scaling is only possible with multiple sessions, while other options require data model adjustments or server configuration changes. To learn more please visit www.EnterpriseDB.com or contact sales@enterprisedb.com.
More Related Content
What's hot
What's hot (20)
Similar to Postgres Scaling Opportunities and Options
Similar to Postgres Scaling Opportunities and Options (20)
More from EDB
More from EDB (20)
Recently uploaded
Recently uploaded (20)
Postgres Scaling Opportunities and Options
- 1. Postgres Scaling Opportunities BRUCE MOMJIAN December, 2014 Configuring Postgres for heavy workloads can take many forms. This talk explores available Postgres scaling options. Creative Commons Attribution License http://momjian.us/presentations 1 / 37
- 2. PostgreSQL the database… ◮ Open Source Object Relational DBMS since 1996 ◮ Distributed under the PostgreSQL License ◮ Similar technical heritage as Oracle, SQL Server & DB2 ◮ However, a strong adherence to standards (ANSI-SQL 2008) ◮ Highly extensible and adaptable design ◮ Languages, indexing, data types, etc. ◮ E.g. PostGIS, JSONB, SQL/MED ◮ Extensive use throughout the world for applications and organizations of all types ◮ Bundled into Red Hat Enterprise Linux, Ubuntu, CentOS and Amazon Linux Postgres Scaling Opportunities 2 / 37
- 3. PostgreSQL the community… ◮ Independent community led by a Core Team of six ◮ Large, active and vibrant community ◮ www.postgresql.org ◮ Downloads, Mailing lists, Documentation ◮ Sponsors sampler: ◮ Google, Red Hat, VMWare, Skype, Salesforce, HP and EnterpriseDB ◮ http://www.postgresql.org/community/ Postgres Scaling Opportunities 3 / 37
- 4. EnterpriseDB the company… ◮ The worldwide leader of Postgres based products and services founded in 2004 ◮ Customers include 50 of the Fortune 500 and 98 of the Forbes Global 2000 ◮ Enterprise offerings: ◮ PostgreSQL Support, Services and Training ◮ Postgres Plus Advanced Server with Oracle Compatibility ◮ Tools for Monitoring, Replication, HA, Backup & Recovery Community ◮ Citizenship ◮ Contributor of key features: Materialized Views, JSON, & more ◮ Nine community members on staff Postgres Scaling Opportunities 4 / 37
- 5. EnterpriseDB the company… Postgres Scaling Opportunities 5 / 37
- 6. Scaling Database scaling is the ability to increase database throughput by utilizing additional resources such as I/O, memory, CPU, or additional computers. However, the high concurrency and write requirements of database servers make scaling a challenge. Sometimes scaling is only possible with multiple sessions, while other options require data model adjustments or server configuration changes. Postgres Scaling Opportunities 6 / 37
- 7. Outline Postgres scaling opportunities: 1. Multi-session 2. Single-session 3. Multi-host Postgres Scaling Opportunities 7 / 37
- 9. Examples Vertical scaling examples: ◮ More and faster CPUs ◮ More memory ◮ More and faster storage ◮ Battery-backed cache (BBU) Horizontal scaling involves adding servers. Postgres Scaling Opportunities 9 / 37
- 10. Hardware Components I/O CPU Memory Postgres Scaling Opportunities 10 / 37
- 11. 1. Multi-Session Clients Database CPU I/O Memory Postgres Scaling Opportunities 11 / 37
- 12. I/O Spreading Using Tablespaces Clients Database CPU Memory Tablespaces Requires tables & indexes to be spread across tablespaces Tablespaces should be on different storage devices Postgres Scaling Opportunities 12 / 37
- 13. I/O Spreading Using Hash Partitions 0000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000 1111111111111111111111111111111111111111111111111111111111111 11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111 1111111111111111111111111111111111111111111111111111111111111 1111111111111111111111111111111111111111111111111111111111111 Clients Database CPU Memory Hash Partitions Requires rows to be hashed by key; one partition per device Postgres Scaling Opportunities 13 / 37
- 14. I/O Spreading Using RAID 0 0000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000 1111111111111111111111111111111111111111111111111111111111111 11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111 1111111111111111111111111111111111111111111111111111111111111 1111111111111111111111111111111111111111111111111111111111111 Clients Database CPU Memory RAID 0 Auto-hashed by storage block number Postgres Scaling Opportunities 14 / 37
- 15. Write Spreading Using WAL Relocation Clients Database CPU Memory WAL Tables and Indexes Separates WAL writes from table & index I/O Postgres Scaling Opportunities 15 / 37
- 16. Read Reduction via Increased Memory Clients Database CPU I/O Memory Additional memory caching reduces read requirements Postgres Scaling Opportunities 16 / 37
- 17. Scaling Connections Using a Pooler Clients Database Connection Pooler CPU I/O Memory Fewer idle connections reduces resource usage Postgres Scaling Opportunities 17 / 37
- 18. Multi-Session CPU Scaling Clients Database I/O Memory CPU Multiple sessions spread across available CPUs Postgres Scaling Opportunities 18 / 37
- 19. 2. Single-Session Client Database CPU I/O Memory Postgres Scaling Opportunities 19 / 37
- 20. Read Parallelism Using effective_io_concurrency Client Database CPU Memory Table Index Used during bitmap heap scans Assumes table is hashed across multiple devices Postgres Scaling Opportunities 20 / 37
- 21. I/O Scaling via Parallelism (not implemented) Client Database CPU Memory I/O Involves parallel index, heap, partition, and tablespace access Postgres Scaling Opportunities 21 / 37
- 22. CPU Scaling via Parallelism (not implemented) Database Client I/O Memory CPU Involves parallel sorts, joins, and function execution Postgres Scaling Opportunities 22 / 37
- 23. Sort I/O Reduction Using Work_mem Client Database CPU I/O Memory Sorts/Hashes Reduces temporary result reads & writes Postgres Scaling Opportunities 23 / 37
- 24. Logical Dump Parallelism CPU I/O Memory pg_dump Dumps tables using concurrent database connections Postgres Scaling Opportunities 24 / 37
- 25. Logical Restore Parallelism CPU I/O Memory pg_restore Loads tables and creates indexes using concurrent database connections Postgres Scaling Opportunities 25 / 37
- 27. Read Scaling Using Pgpool & Streaming Replication SELECTINSERT, UPDATE, DELETE to master host Slave SlaveMaster replication to any host replication streaming 000000000000000000000000 000000000000000000000000000000000000 000000000000000000000000000000000000 111111111111111111111111 111111111111111111111111111111111111 111111111111111111111111111111111111 00000000000000000000000000 000000000000000000000000000000000000000 00000000000000000000000000 11111111111111111111111111 111111111111111111111111111111111111111 11111111111111111111111111 pgpool A full copy of the data exists on every node. Postgres Scaling Opportunities 27 / 37
- 28. CPU/Memory Scaling With Asynchronous Multi-Master SQL Queries SQL Queries Asynchronous CPU Memory CPU Memory Tables and Indexes Tables and Indexes A full copy of the data exists on every node; requires conflict resolution. The asynchronous delay allows write-load buffering. Postgres Scaling Opportunities 28 / 37
- 29. Oracle Real Application Clusters (RAC) Synchronization SQL Queries SQL Queries CPU Memory CPU Memory CPU Memory WALWAL1 WAL2Tables and Indexes Tables and indexes on shared storage; inter-node synchronization required for cache consistency Postgres Scaling Opportunities 29 / 37
- 30. I/O Scaling with Sharding: Challenges ◮ Multi-host write queries require two-phase commit (except XC) ◮ Multi-host visibility snapshots are not supported (except XC) ◮ Sharding benefits are only possible with a shardable workload ◮ Changing the sharding layout can cause downtime ◮ Additional hosts reduce reliability; additional standby servers might be required Postgres Scaling Opportunities 30 / 37
- 31. Application-Based Sharding 000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 111111111111111111111111111111111111111111111111 111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111 0000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 1111111111111111111111111111111111111111111111 1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111 000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 111111111111111111111111111111111111111111111111 111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111 Applications SQL Queries Data Node Data Node Data Node Applications send queries based on the sharding layout. Postgres Scaling Opportunities 31 / 37
- 32. Sharding Using PL/Proxy 000000000000000000000000 000000000000000000000000 000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000 111111111111111111111111 111111111111111111111111 111111111111111111111111111111111111111111111111 111111111111111111111111111111111111111111111111 000000000000000000000000 000000000000000000000000 000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000 111111111111111111111111 111111111111111111111111 111111111111111111111111111111111111111111111111 111111111111111111111111111111111111111111111111 000000000000000000000000 000000000000000000000000 000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000 111111111111111111111111 111111111111111111111111 111111111111111111111111111111111111111111111111 111111111111111111111111111111111111111111111111 Function Calls SQL Queries PL/Proxy Data Node Data Node Data Node Requires rows to be hashed by key, supports parallel-node query execution Postgres Scaling Opportunities 32 / 37
- 33. Sharding Using Postgres-XC 00000000000000000000000 00000000000000000000000 0000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000 00000000000000000000000 11111111111111111111111 11111111111111111111111 1111111111111111111111111111111111111111111111 1111111111111111111111111111111111111111111111 11111111111111111111111 000000000000000000000000 000000000000000000000000 000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000 000000000000000000000000 111111111111111111111111 111111111111111111111111 111111111111111111111111111111111111111111111111 111111111111111111111111111111111111111111111111 111111111111111111111111 000000000000000000000000 000000000000000000000000 000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000 000000000000000000000000 111111111111111111111111 111111111111111111111111 111111111111111111111111111111111111111111111111 111111111111111111111111111111111111111111111111 111111111111111111111111 SQL Queries XC Coordinator SQL Queries Data NodeData Node Data Node Joins, sorts, aggregates Manager Global Transaction Enables hashing of large tables, replication of others; supports parallel-node consistent transactions and DDL Postgres Scaling Opportunities 33 / 37
- 34. Scaling Using Foreign Data Wrappers 000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000 000000000000000000000000 000000000000000000000000000000000000000000000000 111111111111111111111111111111111111111111111111 111111111111111111111111111111111111111111111111 111111111111111111111111 111111111111111111111111111111111111111111111111 0000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000 00000000000000000000000 0000000000000000000000000000000000000000000000 1111111111111111111111111111111111111111111111 1111111111111111111111111111111111111111111111 11111111111111111111111 1111111111111111111111111111111111111111111111 0000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000 00000000000000000000000 0000000000000000000000000000000000000000000000 1111111111111111111111111111111111111111111111 1111111111111111111111111111111111111111111111 11111111111111111111111 1111111111111111111111111111111111111111111111 SQL Queries Foreign Server Foreign Server Foreign Server SQL Queries PG FDW No joins, sorts, aggregates Requires rows to be hashed by key Postgres Scaling Opportunities 34 / 37
- 35. Bulk Data Scaling Using Hadoop 000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000 000000000000000000000000 000000000000000000000000 111111111111111111111111111111111111111111111111 111111111111111111111111111111111111111111111111 111111111111111111111111 111111111111111111111111 0000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000 00000000000000000000000 00000000000000000000000 1111111111111111111111111111111111111111111111 1111111111111111111111111111111111111111111111 11111111111111111111111 11111111111111111111111 0000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000 00000000000000000000000 00000000000000000000000 1111111111111111111111111111111111111111111111 1111111111111111111111111111111111111111111111 11111111111111111111111 11111111111111111111111 Map/Reduce Jobs Postgres Server Hadoop Postgres Server Postgres Server Postgres Scaling Opportunities 35 / 37
- 36. Additional Resources… ◮ Postgres Downloads: ◮ www.enterprisedb.com/downloads ◮ Product and Services information: ◮ info@enterprisedb.com Postgres Scaling Opportunities 36 / 37
- 37. Conclusion http://momjian.us/presentations thttps://www.flickr.com/photos/blieusong/ Postgres Scaling Opportunities 37 / 37