Scaling postgres

18,647 views

Published on

The slide deck describes options available in Postgre

Published in: Technology, News & Politics

Scaling postgres

  1. 1. / Scaling Postgres Denish Patel Database Architect https://twitter.com/DenishPatel Wednesday, September 18, 13
  2. 2. 1 Wednesday, September 18, 13
  3. 3. OmniTI 1 Wednesday, September 18, 13
  4. 4. OmniTI 1 • Helping customers navigate explosive growth with technology. Wednesday, September 18, 13
  5. 5. OmniTI 1 • Helping customers navigate explosive growth with technology. 100MM+ users $1B+ gross online sales Wednesday, September 18, 13
  6. 6. OmniTI 1 • Helping customers navigate explosive growth with technology. 100MM+ users $1B+ gross online sales Open and closed source thought leaders, experts and authors Wednesday, September 18, 13
  7. 7. OmniTI 1 • Helping customers navigate explosive growth with technology. 100MM+ users $1B+ gross online sales Open and closed source thought leaders, experts and authors Wednesday, September 18, 13
  8. 8. 2 Wednesday, September 18, 13
  9. 9. Talk Outline 2 Wednesday, September 18, 13
  10. 10. Talk Outline • Scalability • Database Scaling needs, costs, methods • Scaling Postgres • Vertically • Horizontally • Obstacles to Scalability • Beyond Postgres 2 Wednesday, September 18, 13
  11. 11. 3 Wednesday, September 18, 13
  12. 12. What is Scalability? 3 Wednesday, September 18, 13
  13. 13. What is Scalability? 3 A service is said to be scalable if when we increase the resources in a system, it results in increased performance in a manner proportional to resources added. Wednesday, September 18, 13
  14. 14. 4 Wednesday, September 18, 13
  15. 15. Why to scale databases? 4 Wednesday, September 18, 13
  16. 16. Why to scale databases? • Support a higher volume of users 4 Wednesday, September 18, 13
  17. 17. Why to scale databases? • Support a higher volume of users 4 Wednesday, September 18, 13
  18. 18. Why to scale databases? • Support a higher volume of users 4 Wednesday, September 18, 13
  19. 19. Why to scale databases? • Support a higher volume of users 4 Wednesday, September 18, 13
  20. 20. Why to scale databases? • Provide better performance for existing users • Support a higher volume of users 4 Wednesday, September 18, 13
  21. 21. Why to scale databases? • Provide better performance for existing users • Store a larger volume of data • Support a higher volume of users 4 Wednesday, September 18, 13
  22. 22. Why to scale databases? • Provide better performance for existing users • Store a larger volume of data • Improve system availability • Support a higher volume of users 4 Wednesday, September 18, 13
  23. 23. Why to scale databases? • Provide better performance for existing users • Store a larger volume of data • Improve system availability • Geographic dispersion • Support a higher volume of users 4 Wednesday, September 18, 13
  24. 24. 5 Wednesday, September 18, 13
  25. 25. Why is database scalability so hard? 5 Wednesday, September 18, 13
  26. 26. Why is database scalability so hard? 5 Wednesday, September 18, 13
  27. 27. Why is database scalability so hard? • Search 5 Wednesday, September 18, 13
  28. 28. Why is database scalability so hard? • Search • Concurrency 5 Wednesday, September 18, 13
  29. 29. Why is database scalability so hard? • Search • Concurrency • Consistency 5 Wednesday, September 18, 13
  30. 30. Why is database scalability so hard? • Search • Concurrency • Consistency • Speed 5 Wednesday, September 18, 13
  31. 31. Why is database scalability so hard? • Search • Concurrency • Consistency • Speed • Cost 5 Wednesday, September 18, 13
  32. 32. Why is database scalability so hard? • Search • Concurrency • Consistency • Speed • Cost • Cost of hardware 5 Wednesday, September 18, 13
  33. 33. Why is database scalability so hard? • Search • Concurrency • Consistency • Speed • Cost • Cost of hardware • Cost deployment effort 5 Wednesday, September 18, 13
  34. 34. Why is database scalability so hard? • Search • Concurrency • Consistency • Speed • Cost • Cost of hardware • Cost deployment effort • Cost Ongoing maintenance 5 Wednesday, September 18, 13
  35. 35. 6 Wednesday, September 18, 13
  36. 36. Scaling Needs & Methods 6 Wednesday, September 18, 13
  37. 37. Scaling Needs & Methods 6 Wednesday, September 18, 13
  38. 38. Scaling Needs & Methods • Data growth 6 Wednesday, September 18, 13
  39. 39. Scaling Needs & Methods • Data growth • Read requests 6 Wednesday, September 18, 13
  40. 40. Scaling Needs & Methods • Data growth • Read requests • Write requests 6 Wednesday, September 18, 13
  41. 41. Scaling Needs & Methods • Data growth • Read requests • Write requests • Vertical Scaling • Horizontal Scaling 6 Wednesday, September 18, 13
  42. 42. Scaling Needs & Methods • Data growth • Read requests • Write requests • Vertical Scaling • Horizontal Scaling 6 Wednesday, September 18, 13
  43. 43. Scaling Needs & Methods • Data growth • Read requests • Write requests • Vertical Scaling • Horizontal Scaling 6 Wednesday, September 18, 13
  44. 44. 7 Wednesday, September 18, 13
  45. 45. Vertical Scaling (Scale up) 7 Wednesday, September 18, 13
  46. 46. Vertical Scaling (Scale up) 7 Wednesday, September 18, 13
  47. 47. Vertical Scaling (Scale up) • Pros 7 Wednesday, September 18, 13
  48. 48. Vertical Scaling (Scale up) • Pros • Simple to implement 7 Wednesday, September 18, 13
  49. 49. Vertical Scaling (Scale up) • Pros • Simple to implement • Ease of maintenance 7 Wednesday, September 18, 13
  50. 50. Vertical Scaling (Scale up) • Pros • Simple to implement • Ease of maintenance • Cons 7 Wednesday, September 18, 13
  51. 51. Vertical Scaling (Scale up) • Pros • Simple to implement • Ease of maintenance • Cons • Cost of hardware 7 Wednesday, September 18, 13
  52. 52. Vertical Scaling (Scale up) • Pros • Simple to implement • Ease of maintenance • Cons • Cost of hardware • Sometimes SPOFs 7 Wednesday, September 18, 13
  53. 53. 8 Wednesday, September 18, 13
  54. 54. Horizontal Scaling (Scale Out) 8 Wednesday, September 18, 13
  55. 55. Horizontal Scaling (Scale Out) 8 Wednesday, September 18, 13
  56. 56. Horizontal Scaling (Scale Out) 8 Wednesday, September 18, 13
  57. 57. Horizontal Scaling (Scale Out) • Pros 8 Wednesday, September 18, 13
  58. 58. Horizontal Scaling (Scale Out) • Pros • Cheaper in hardware cost 8 Wednesday, September 18, 13
  59. 59. Horizontal Scaling (Scale Out) • Pros • Cheaper in hardware cost • Flexibility 8 Wednesday, September 18, 13
  60. 60. Horizontal Scaling (Scale Out) • Pros • Cheaper in hardware cost • Flexibility • Higher fault tolerance 8 Wednesday, September 18, 13
  61. 61. Horizontal Scaling (Scale Out) • Pros • Cheaper in hardware cost • Flexibility • Higher fault tolerance • Cons 8 Wednesday, September 18, 13
  62. 62. Horizontal Scaling (Scale Out) • Pros • Cheaper in hardware cost • Flexibility • Higher fault tolerance • Cons • Complex to implement 8 Wednesday, September 18, 13
  63. 63. Horizontal Scaling (Scale Out) • Pros • Cheaper in hardware cost • Flexibility • Higher fault tolerance • Cons • Complex to implement • Expensive to maintain 8 Wednesday, September 18, 13
  64. 64. Horizontal Scaling (Scale Out) • Pros • Cheaper in hardware cost • Flexibility • Higher fault tolerance • Cons • Complex to implement • Expensive to maintain • Bigger footprint in the Data Center 8 Wednesday, September 18, 13
  65. 65. Horizontal Scaling (Scale Out) • Pros • Cheaper in hardware cost • Flexibility • Higher fault tolerance • Cons • Complex to implement • Expensive to maintain • Bigger footprint in the Data Center • No built in support in databases 8 Wednesday, September 18, 13
  66. 66. 9 Wednesday, September 18, 13
  67. 67. Spec’ing Hardware 9 Wednesday, September 18, 13
  68. 68. Spec’ing Hardware • CPU 9 Wednesday, September 18, 13
  69. 69. Spec’ing Hardware • CPU • 8+ cores 9 Wednesday, September 18, 13
  70. 70. Spec’ing Hardware • CPU • 8+ cores • RAM 9 Wednesday, September 18, 13
  71. 71. Spec’ing Hardware • CPU • 8+ cores • RAM • 64GB+ 9 Wednesday, September 18, 13
  72. 72. Spec’ing Hardware • CPU • 8+ cores • RAM • 64GB+ • Disks 9 Wednesday, September 18, 13
  73. 73. Spec’ing Hardware • CPU • 8+ cores • RAM • 64GB+ • Disks • SSDs 9 Wednesday, September 18, 13
  74. 74. Spec’ing Hardware • CPU • 8+ cores • RAM • 64GB+ • Disks • SSDs • RAID 10 9 Wednesday, September 18, 13
  75. 75. Spec’ing Hardware • CPU • 8+ cores • RAM • 64GB+ • Disks • SSDs • RAID 10 • Network 9 Wednesday, September 18, 13
  76. 76. Spec’ing Hardware • CPU • 8+ cores • RAM • 64GB+ • Disks • SSDs • RAID 10 • Network • min Gigbit, 10Gigbit 9 Wednesday, September 18, 13
  77. 77. 10 Wednesday, September 18, 13
  78. 78. Tune Postgres/memory parameters 10 Wednesday, September 18, 13
  79. 79. Tune Postgres/memory parameters • shared_buffers 10 Wednesday, September 18, 13
  80. 80. Tune Postgres/memory parameters • shared_buffers • effective_cache_size 10 Wednesday, September 18, 13
  81. 81. Tune Postgres/memory parameters • shared_buffers • effective_cache_size • checkpoint_completion_target 10 Wednesday, September 18, 13
  82. 82. Tune Postgres/memory parameters • shared_buffers • effective_cache_size • checkpoint_completion_target • checkpoint_segments 10 Wednesday, September 18, 13
  83. 83. Tune Postgres/memory parameters • shared_buffers • effective_cache_size • checkpoint_completion_target • checkpoint_segments • max_connections 10 Wednesday, September 18, 13
  84. 84. Tune Postgres/memory parameters • shared_buffers • effective_cache_size • checkpoint_completion_target • checkpoint_segments • max_connections • work_mem 10 Wednesday, September 18, 13
  85. 85. Tune Postgres/memory parameters • shared_buffers • effective_cache_size • checkpoint_completion_target • checkpoint_segments • max_connections • work_mem • maintenance_work_mem 10 Wednesday, September 18, 13
  86. 86. Tune Postgres/memory parameters • shared_buffers • effective_cache_size • checkpoint_completion_target • checkpoint_segments • max_connections • work_mem • maintenance_work_mem http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server 10 Wednesday, September 18, 13
  87. 87. 11 Wednesday, September 18, 13
  88. 88. Tune Postgres/logging parameters 11 Wednesday, September 18, 13
  89. 89. Tune Postgres/logging parameters 11 Wednesday, September 18, 13
  90. 90. Tune Postgres/logging parameters 11 Wednesday, September 18, 13
  91. 91. Tune Postgres/logging parameters • logging_collector => 'on' 11 Wednesday, September 18, 13
  92. 92. Tune Postgres/logging parameters • logging_collector => 'on' • log_destination => 'stderr' 11 Wednesday, September 18, 13
  93. 93. Tune Postgres/logging parameters • logging_collector => 'on' • log_destination => 'stderr' • log_filename => 'postgresql-%Y-%m-%d_%H%M%S.log' 11 Wednesday, September 18, 13
  94. 94. Tune Postgres/logging parameters • logging_collector => 'on' • log_destination => 'stderr' • log_filename => 'postgresql-%Y-%m-%d_%H%M%S.log' • log_line_prefix => '%m [%r] [%p]: [%l-1] user=%u,db=%d,e=%e ' 11 Wednesday, September 18, 13
  95. 95. Tune Postgres/logging parameters • logging_collector => 'on' • log_destination => 'stderr' • log_filename => 'postgresql-%Y-%m-%d_%H%M%S.log' • log_line_prefix => '%m [%r] [%p]: [%l-1] user=%u,db=%d,e=%e ' • log_min_duration_statement => 1000ms 11 Wednesday, September 18, 13
  96. 96. Tune Postgres/logging parameters • logging_collector => 'on' • log_destination => 'stderr' • log_filename => 'postgresql-%Y-%m-%d_%H%M%S.log' • log_line_prefix => '%m [%r] [%p]: [%l-1] user=%u,db=%d,e=%e ' • log_min_duration_statement => 1000ms • log_autovacuum_min_duration => '0' 11 Wednesday, September 18, 13
  97. 97. 12 Wednesday, September 18, 13
  98. 98. Tune Postgres/logging parameters 12 Wednesday, September 18, 13
  99. 99. Tune Postgres/logging parameters 12 Wednesday, September 18, 13
  100. 100. Tune Postgres/logging parameters 12 Wednesday, September 18, 13
  101. 101. Tune Postgres/logging parameters • log_lock_waits => 'on' 12 Wednesday, September 18, 13
  102. 102. Tune Postgres/logging parameters • log_lock_waits => 'on' • log_temp_files => '0' 12 Wednesday, September 18, 13
  103. 103. Tune Postgres/logging parameters • log_lock_waits => 'on' • log_temp_files => '0' • log_checkpoints => 'on' 12 Wednesday, September 18, 13
  104. 104. Tune Postgres/logging parameters • log_lock_waits => 'on' • log_temp_files => '0' • log_checkpoints => 'on' • log_connections => 'on' 12 Wednesday, September 18, 13
  105. 105. Tune Postgres/logging parameters • log_lock_waits => 'on' • log_temp_files => '0' • log_checkpoints => 'on' • log_connections => 'on' • log_disconnections => 'on' 12 Wednesday, September 18, 13
  106. 106. Tune Postgres/logging parameters • log_lock_waits => 'on' • log_temp_files => '0' • log_checkpoints => 'on' • log_connections => 'on' • log_disconnections => 'on' • log_min_error_statement => 'warning' 12 Wednesday, September 18, 13
  107. 107. Tune Postgres/logging parameters • log_lock_waits => 'on' • log_temp_files => '0' • log_checkpoints => 'on' • log_connections => 'on' • log_disconnections => 'on' • log_min_error_statement => 'warning' • log_min_messages => 'warning' 12 Wednesday, September 18, 13
  108. 108. Tune Postgres/logging parameters • log_lock_waits => 'on' • log_temp_files => '0' • log_checkpoints => 'on' • log_connections => 'on' • log_disconnections => 'on' • log_min_error_statement => 'warning' • log_min_messages => 'warning' • log_statement => 'ddl' 12 Wednesday, September 18, 13
  109. 109. 13 Wednesday, September 18, 13
  110. 110. Optimize Queries/report slow queries 13 Wednesday, September 18, 13
  111. 111. Optimize Queries/report slow queries 13 Wednesday, September 18, 13
  112. 112. Optimize Queries/report slow queries 13 Wednesday, September 18, 13
  113. 113. 14 Wednesday, September 18, 13
  114. 114. Optimize Queries/Extensive monitoring 14 Wednesday, September 18, 13
  115. 115. Optimize Queries/Extensive monitoring 14 Wednesday, September 18, 13
  116. 116. 15 Wednesday, September 18, 13
  117. 117. Optimize Queries /Explain Analyze 15 Wednesday, September 18, 13
  118. 118. Optimize Queries /Explain Analyze explain (analyze,buffers) select col1,col2 from demo_ios where col2 between 0.01 and 0.02; QUERY PLAN ----------------------------------------------------------------------------------------------------------------------------------------------- Index Only Scan using idx_demo_ios on demo_ios (cost=0.00..35330.93 rows=993633 width=16) (actual time=58.100..3250.589 rows=1000392 loops=1) Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision)) Heap Fetches: 0 Buffers: shared hit=923073 read=3848 Total runtime: 4297.405 ms 15 Wednesday, September 18, 13
  119. 119. Optimize Queries /Explain Analyze explain (analyze,buffers) select col1,col2 from demo_ios where col2 between 0.01 and 0.02; QUERY PLAN ----------------------------------------------------------------------------------------------------------------------------------------------- Index Only Scan using idx_demo_ios on demo_ios (cost=0.00..35330.93 rows=993633 width=16) (actual time=58.100..3250.589 rows=1000392 loops=1) Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision)) Heap Fetches: 0 Buffers: shared hit=923073 read=3848 Total runtime: 4297.405 ms 15 Wednesday, September 18, 13
  120. 120. 16 Wednesday, September 18, 13
  121. 121. Optimize Queries /track functions 16 Wednesday, September 18, 13
  122. 122. Optimize Queries /track functions 16 Wednesday, September 18, 13
  123. 123. Optimize Queries /track functions • track_functions = pl # none, pl, all 16 Wednesday, September 18, 13
  124. 124. Optimize Queries /track functions • track_functions = pl # none, pl, all • reload online 16 Wednesday, September 18, 13
  125. 125. Optimize Queries /track functions • track_functions = pl # none, pl, all • reload online • select * from pg_stat_user_functions; 16 Wednesday, September 18, 13
  126. 126. Optimize Queries /track functions • track_functions = pl # none, pl, all • reload online • select * from pg_stat_user_functions; 16 Wednesday, September 18, 13
  127. 127. Optimize Queries /track functions • track_functions = pl # none, pl, all • reload online • select * from pg_stat_user_functions; funcid | schemaname | funcname | calls | total_time | self_time 16 Wednesday, September 18, 13
  128. 128. Optimize Queries /track functions • track_functions = pl # none, pl, all • reload online • select * from pg_stat_user_functions; funcid | schemaname | funcname | calls | total_time | self_time 16 Wednesday, September 18, 13
  129. 129. 17 Wednesday, September 18, 13
  130. 130. Partitioning 17 Wednesday, September 18, 13
  131. 131. Partitioning • As table size grows, queries eventually slows down, even with indexing 17 Wednesday, September 18, 13
  132. 132. Partitioning • As table size grows, queries eventually slows down, even with indexing • Allows data added, removed and queried fast 17 Wednesday, September 18, 13
  133. 133. Partitioning • As table size grows, queries eventually slows down, even with indexing • Allows data added, removed and queried fast • Partitioning pruning queries 17 Wednesday, September 18, 13
  134. 134. Partitioning • As table size grows, queries eventually slows down, even with indexing • Allows data added, removed and queried fast • Partitioning pruning queries • Manage partitions 17 Wednesday, September 18, 13
  135. 135. 18 Wednesday, September 18, 13
  136. 136. Partitioning /Postgres 18 Wednesday, September 18, 13
  137. 137. Partitioning /Postgres 18 Wednesday, September 18, 13
  138. 138. Partitioning /Postgres • Postgres partitioning 18 Wednesday, September 18, 13
  139. 139. Partitioning /Postgres • Postgres partitioning • Trigger based 18 Wednesday, September 18, 13
  140. 140. Partitioning /Postgres • Postgres partitioning • Trigger based • Rule based 18 Wednesday, September 18, 13
  141. 141. Partitioning /Postgres • Postgres partitioning • Trigger based • Rule based • Lack of built-in Postgres partition management 18 Wednesday, September 18, 13
  142. 142. Partitioning /Postgres • Postgres partitioning • Trigger based • Rule based • Lack of built-in Postgres partition management • Postgres partition management extension: pg_partman 18 Wednesday, September 18, 13
  143. 143. Partitioning /Postgres • Postgres partitioning • Trigger based • Rule based • Lack of built-in Postgres partition management • Postgres partition management extension: pg_partman • http://pgxn.org/dist/pg_partman/doc/pg_partman.html 18 Wednesday, September 18, 13
  144. 144. Partitioning /Postgres • Postgres partitioning • Trigger based • Rule based • Lack of built-in Postgres partition management • Postgres partition management extension: pg_partman • http://pgxn.org/dist/pg_partman/doc/pg_partman.html • Tomorrow’s session at 11:30AM (When Postgres can’t....) 18 Wednesday, September 18, 13
  145. 145. 19 Wednesday, September 18, 13
  146. 146. Partitioning /functional 19 Wednesday, September 18, 13
  147. 147. Partitioning /functional Configuration Data Transaction data Session data Configuration Tools Reporting Tools Monitoring Tools Web Applications Other Applications 19 Wednesday, September 18, 13
  148. 148. Partitioning /functional Configuration Data Transaction data Session data Configuration Tools Reporting Tools Monitoring Tools Web Applications Other Applications • Partition data based on functionality 19 Wednesday, September 18, 13
  149. 149. Partitioning /functional Configuration Data Transaction data Session data Configuration Tools Reporting Tools Monitoring Tools Web Applications Other Applications • Partition data based on functionality • Separate Postgres clusters 19 Wednesday, September 18, 13
  150. 150. Partitioning /functional Configuration Data Transaction data Session data Configuration Tools Reporting Tools Monitoring Tools Web Applications Other Applications • Partition data based on functionality • Separate Postgres clusters • Start with Separate schemas 19 Wednesday, September 18, 13
  151. 151. Partitioning /functional Configuration Data Transaction data Session data Configuration Tools Reporting Tools Monitoring Tools Web Applications Other Applications • Partition data based on functionality • Separate Postgres clusters • Start with Separate schemas • No relationship between data 19 Wednesday, September 18, 13
  152. 152. Partitioning /functional Configuration Data Transaction data Session data Configuration Tools Reporting Tools Monitoring Tools Web Applications Other Applications • Partition data based on functionality • Separate Postgres clusters • Start with Separate schemas • No relationship between data • Help to spread the load across server 19 Wednesday, September 18, 13
  153. 153. Partitioning /functional Configuration Data Transaction data Session data Configuration Tools Reporting Tools Monitoring Tools Web Applications Other Applications • Partition data based on functionality • Separate Postgres clusters • Start with Separate schemas • No relationship between data • Help to spread the load across server • Less complex compare to sharding! 19 Wednesday, September 18, 13
  154. 154. 20 Wednesday, September 18, 13
  155. 155. pgbouncer 20 Wednesday, September 18, 13
  156. 156. pgbouncer 20 Wednesday, September 18, 13
  157. 157. pgbouncer 20 Wednesday, September 18, 13
  158. 158. pgbouncer 20 Wednesday, September 18, 13
  159. 159. pgbouncer • A lightweight connection pooler 20 Wednesday, September 18, 13
  160. 160. pgbouncer • A lightweight connection pooler • Helps to reduce # of newly created connections on DB server 20 Wednesday, September 18, 13
  161. 161. pgbouncer • A lightweight connection pooler • Helps to reduce # of newly created connections on DB server • Abstracts DBs from App 20 Wednesday, September 18, 13
  162. 162. pgbouncer • A lightweight connection pooler • Helps to reduce # of newly created connections on DB server • Abstracts DBs from App • Helps to instrument smooth and easy failover 20 Wednesday, September 18, 13
  163. 163. pgbouncer • A lightweight connection pooler • Helps to reduce # of newly created connections on DB server • Abstracts DBs from App • Helps to instrument smooth and easy failover • Connection pooling Options 20 Wednesday, September 18, 13
  164. 164. pgbouncer • A lightweight connection pooler • Helps to reduce # of newly created connections on DB server • Abstracts DBs from App • Helps to instrument smooth and easy failover • Connection pooling Options • Session, Transaction, Statement pooling options 20 Wednesday, September 18, 13
  165. 165. pgbouncer • A lightweight connection pooler • Helps to reduce # of newly created connections on DB server • Abstracts DBs from App • Helps to instrument smooth and easy failover • Connection pooling Options • Session, Transaction, Statement pooling options • Beware! Transaction pooling doesn’t support prepared transactions 20 Wednesday, September 18, 13
  166. 166. 21 Wednesday, September 18, 13
  167. 167. Caching 21 Wednesday, September 18, 13
  168. 168. Caching 21 Wednesday, September 18, 13
  169. 169. Caching • Memcached 21 Wednesday, September 18, 13
  170. 170. Caching • Memcached • Open source, High-performance distributed memory object caching system 21 Wednesday, September 18, 13
  171. 171. Caching • Memcached • Open source, High-performance distributed memory object caching system • Speeds up dynamic web applications by alleviating database load. 21 Wednesday, September 18, 13
  172. 172. Caching • Memcached • Open source, High-performance distributed memory object caching system • Speeds up dynamic web applications by alleviating database load. • An in-memory key-value store for small chunks of arbitrary data 21 Wednesday, September 18, 13
  173. 173. Caching • Memcached • Open source, High-performance distributed memory object caching system • Speeds up dynamic web applications by alleviating database load. • An in-memory key-value store for small chunks of arbitrary data • Redis 21 Wednesday, September 18, 13
  174. 174. Caching • Memcached • Open source, High-performance distributed memory object caching system • Speeds up dynamic web applications by alleviating database load. • An in-memory key-value store for small chunks of arbitrary data • Redis • Open source, advanced key-value store. 21 Wednesday, September 18, 13
  175. 175. Caching • Memcached • Open source, High-performance distributed memory object caching system • Speeds up dynamic web applications by alleviating database load. • An in-memory key-value store for small chunks of arbitrary data • Redis • Open source, advanced key-value store. • Works with an in-memory & persistent dataset 21 Wednesday, September 18, 13
  176. 176. 22 Wednesday, September 18, 13
  177. 177. Replication /built-in 22 Wednesday, September 18, 13
  178. 178. Replication /built-in • Cluster Level Replication (Binary) 22 Wednesday, September 18, 13
  179. 179. Replication /built-in • Cluster Level Replication (Binary) • Streaming Replication 22 Wednesday, September 18, 13
  180. 180. Replication /built-in • Cluster Level Replication (Binary) • Streaming Replication • WAL-only replication 22 Wednesday, September 18, 13
  181. 181. Replication /built-in • Cluster Level Replication (Binary) • Streaming Replication • WAL-only replication • Hybrid replication 22 Wednesday, September 18, 13
  182. 182. Replication /built-in Master • Cluster Level Replication (Binary) • Streaming Replication • WAL-only replication • Hybrid replication 22 Wednesday, September 18, 13
  183. 183. Replication /built-in Master Failover• Cluster Level Replication (Binary) • Streaming Replication • WAL-only replication • Hybrid replication 22 Wednesday, September 18, 13
  184. 184. Replication /built-in Master Failover Read Salve 1 • Cluster Level Replication (Binary) • Streaming Replication • WAL-only replication • Hybrid replication 22 Wednesday, September 18, 13
  185. 185. Replication /built-in Master Failover Read Salve 1 • Cluster Level Replication (Binary) • Streaming Replication • WAL-only replication • Hybrid replication 22 Wednesday, September 18, 13
  186. 186. Replication /built-in Master Failover Read Salve 1 • Cluster Level Replication (Binary) • Streaming Replication • WAL-only replication • Hybrid replication 22 Wednesday, September 18, 13
  187. 187. Replication /built-in Master Failover Read Salve 1 Streaming Replication • Cluster Level Replication (Binary) • Streaming Replication • WAL-only replication • Hybrid replication 22 Wednesday, September 18, 13
  188. 188. Replication /built-in Master Failover Read Salve 1 PITR! Streaming Replication • Cluster Level Replication (Binary) • Streaming Replication • WAL-only replication • Hybrid replication 22 Wednesday, September 18, 13
  189. 189. Replication /built-in Master Failover Read Salve 1 PITR! PITR! Streaming Replication • Cluster Level Replication (Binary) • Streaming Replication • WAL-only replication • Hybrid replication 22 Wednesday, September 18, 13
  190. 190. Replication /built-in Master Failover Read Salve 1 PITR! PITR! PITR! Streaming Replication • Cluster Level Replication (Binary) • Streaming Replication • WAL-only replication • Hybrid replication 22 Wednesday, September 18, 13
  191. 191. Replication /built-in Master Failover Read Salve 1 PITR! PITR! PITR! Streaming Replication • Cluster Level Replication (Binary) • Streaming Replication • WAL-only replication • Hybrid replication 22 Wednesday, September 18, 13
  192. 192. Replication /built-in • Pros: Master Failover Read Salve 1 PITR! PITR! PITR! Streaming Replication • Cluster Level Replication (Binary) • Streaming Replication • WAL-only replication • Hybrid replication 22 Wednesday, September 18, 13
  193. 193. Replication /built-in • Pros: • Built-in Master Failover Read Salve 1 PITR! PITR! PITR! Streaming Replication • Cluster Level Replication (Binary) • Streaming Replication • WAL-only replication • Hybrid replication 22 Wednesday, September 18, 13
  194. 194. Replication /built-in • Pros: • Built-in • Allows to open replicated database in read-only mode Master Failover Read Salve 1 PITR! PITR! PITR! Streaming Replication • Cluster Level Replication (Binary) • Streaming Replication • WAL-only replication • Hybrid replication 22 Wednesday, September 18, 13
  195. 195. Replication /built-in • Pros: • Built-in • Allows to open replicated database in read-only mode • Cons: Master Failover Read Salve 1 PITR! PITR! PITR! Streaming Replication • Cluster Level Replication (Binary) • Streaming Replication • WAL-only replication • Hybrid replication 22 Wednesday, September 18, 13
  196. 196. Replication /built-in • Pros: • Built-in • Allows to open replicated database in read-only mode • Cons: • All or none Master Failover Read Salve 1 PITR! PITR! PITR! Streaming Replication • Cluster Level Replication (Binary) • Streaming Replication • WAL-only replication • Hybrid replication 22 Wednesday, September 18, 13
  197. 197. Replication /built-in • Pros: • Built-in • Allows to open replicated database in read-only mode • Cons: • All or none • Doesn’t allow write on replicated database Master Failover Read Salve 1 PITR! PITR! PITR! Streaming Replication • Cluster Level Replication (Binary) • Streaming Replication • WAL-only replication • Hybrid replication 22 Wednesday, September 18, 13
  198. 198. Replication /built-in • Pros: • Built-in • Allows to open replicated database in read-only mode • Cons: • All or none • Doesn’t allow write on replicated database • Doesn’t work across major version Master Failover Read Salve 1 PITR! PITR! PITR! Streaming Replication • Cluster Level Replication (Binary) • Streaming Replication • WAL-only replication • Hybrid replication 22 Wednesday, September 18, 13
  199. 199. Replication /built-in • Pros: • Built-in • Allows to open replicated database in read-only mode • Cons: • All or none • Doesn’t allow write on replicated database • Doesn’t work across major version • Postgres 9.2 (primary) does NOT replicate to 9.3 (secondary) Master Failover Read Salve 1 PITR! PITR! PITR! Streaming Replication • Cluster Level Replication (Binary) • Streaming Replication • WAL-only replication • Hybrid replication 22 Wednesday, September 18, 13
  200. 200. Replication /built-in • Pros: • Built-in • Allows to open replicated database in read-only mode • Cons: • All or none • Doesn’t allow write on replicated database • Doesn’t work across major version • Postgres 9.2 (primary) does NOT replicate to 9.3 (secondary) • Postgres 9.2.1 (primary) can replicate to 9.2.4 (secondary) Master Failover Read Salve 1 PITR! PITR! PITR! Streaming Replication • Cluster Level Replication (Binary) • Streaming Replication • WAL-only replication • Hybrid replication 22 Wednesday, September 18, 13
  201. 201. Replication /built-in • Pros: • Built-in • Allows to open replicated database in read-only mode • Cons: • All or none • Doesn’t allow write on replicated database • Doesn’t work across major version • Postgres 9.2 (primary) does NOT replicate to 9.3 (secondary) • Postgres 9.2.1 (primary) can replicate to 9.2.4 (secondary) Master Failover Read Salve 1 PITR! PITR! PITR! Streaming Replication https://wiki.postgresql.org/wiki/Binary_Replication_Tools • Cluster Level Replication (Binary) • Streaming Replication • WAL-only replication • Hybrid replication 22 Wednesday, September 18, 13
  202. 202. 23 Wednesday, September 18, 13
  203. 203. Replication /built-in 23 Wednesday, September 18, 13
  204. 204. Replication /built-in Read Salve 1 23 Wednesday, September 18, 13
  205. 205. Replication /built-in Read Salve 1 DW System 23 Wednesday, September 18, 13
  206. 206. Replication /built-in Read Salve 1 DW System 23 Wednesday, September 18, 13
  207. 207. Replication /built-in Read Salve 1 DW System postgres_fdw 23 Wednesday, September 18, 13
  208. 208. Replication /built-in Read Salve 1 DW System postgres_fdw 23 Wednesday, September 18, 13
  209. 209. Replication /built-in Read Salve 1 DW System postgres_fdw 23 Wednesday, September 18, 13
  210. 210. Replication /built-in • postgres_fdw Read Salve 1 DW System postgres_fdw 23 Wednesday, September 18, 13
  211. 211. Replication /built-in • postgres_fdw • Postgres 9.3 feature Read Salve 1 DW System postgres_fdw 23 Wednesday, September 18, 13
  212. 212. Replication /built-in • postgres_fdw • Postgres 9.3 feature • Allows to access data stored in external PostgreSQL servers Read Salve 1 DW System postgres_fdw 23 Wednesday, September 18, 13
  213. 213. Replication /built-in • postgres_fdw • Postgres 9.3 feature • Allows to access data stored in external PostgreSQL servers • cross version queries Read Salve 1 DW System postgres_fdw 23 Wednesday, September 18, 13
  214. 214. Replication /built-in • postgres_fdw • Postgres 9.3 feature • Allows to access data stored in external PostgreSQL servers • cross version queries • Postgres 9.3 could query Postgres 9.1 Read Salve 1 DW System postgres_fdw 23 Wednesday, September 18, 13
  215. 215. Replication /built-in • postgres_fdw • Postgres 9.3 feature • Allows to access data stored in external PostgreSQL servers • cross version queries • Postgres 9.3 could query Postgres 9.1 • Application Read Salve 1 DW System postgres_fdw 23 Wednesday, September 18, 13
  216. 216. Replication /built-in • postgres_fdw • Postgres 9.3 feature • Allows to access data stored in external PostgreSQL servers • cross version queries • Postgres 9.3 could query Postgres 9.1 • Application • Run query remotely on slave db Read Salve 1 DW System postgres_fdw 23 Wednesday, September 18, 13
  217. 217. Replication /built-in • postgres_fdw • Postgres 9.3 feature • Allows to access data stored in external PostgreSQL servers • cross version queries • Postgres 9.3 could query Postgres 9.1 • Application • Run query remotely on slave db • Data warehouse data refreshes Read Salve 1 DW System postgres_fdw 23 Wednesday, September 18, 13
  218. 218. 24 Wednesday, September 18, 13
  219. 219. Replication /third-party tools 24 Wednesday, September 18, 13
  220. 220. Replication /third-party tools 24 Wednesday, September 18, 13
  221. 221. Replication /third-party tools 24 Wednesday, September 18, 13
  222. 222. Replication /third-party tools • Table level Replication Tools (Trigger based) 24 Wednesday, September 18, 13
  223. 223. Replication /third-party tools • Table level Replication Tools (Trigger based) • Slony 24 Wednesday, September 18, 13
  224. 224. Replication /third-party tools • Table level Replication Tools (Trigger based) • Slony • Bucardo 24 Wednesday, September 18, 13
  225. 225. Replication /third-party tools • Table level Replication Tools (Trigger based) • Slony • Bucardo • Mimeo: http://pgxn.org/dist/mimeo/ 24 Wednesday, September 18, 13
  226. 226. Replication /third-party tools • Table level Replication Tools (Trigger based) • Slony • Bucardo • Mimeo: http://pgxn.org/dist/mimeo/ • Pros: 24 Wednesday, September 18, 13
  227. 227. Replication /third-party tools • Table level Replication Tools (Trigger based) • Slony • Bucardo • Mimeo: http://pgxn.org/dist/mimeo/ • Pros: • Allows to open replicated database in read- write mode 24 Wednesday, September 18, 13
  228. 228. Replication /third-party tools • Table level Replication Tools (Trigger based) • Slony • Bucardo • Mimeo: http://pgxn.org/dist/mimeo/ • Pros: • Allows to open replicated database in read- write mode • Allows table/database level replication 24 Wednesday, September 18, 13
  229. 229. Replication /third-party tools • Table level Replication Tools (Trigger based) • Slony • Bucardo • Mimeo: http://pgxn.org/dist/mimeo/ • Pros: • Allows to open replicated database in read- write mode • Allows table/database level replication • Allows rolling upgrade 24 Wednesday, September 18, 13
  230. 230. Replication /third-party tools • Table level Replication Tools (Trigger based) • Slony • Bucardo • Mimeo: http://pgxn.org/dist/mimeo/ • Pros: • Allows to open replicated database in read- write mode • Allows table/database level replication • Allows rolling upgrade • cross version replication is allowed 24 Wednesday, September 18, 13
  231. 231. Replication /third-party tools • Table level Replication Tools (Trigger based) • Slony • Bucardo • Mimeo: http://pgxn.org/dist/mimeo/ • Pros: • Allows to open replicated database in read- write mode • Allows table/database level replication • Allows rolling upgrade • cross version replication is allowed • Multi-master replication 24 Wednesday, September 18, 13
  232. 232. Replication /third-party tools • Table level Replication Tools (Trigger based) • Slony • Bucardo • Mimeo: http://pgxn.org/dist/mimeo/ • Pros: • Allows to open replicated database in read- write mode • Allows table/database level replication • Allows rolling upgrade • cross version replication is allowed • Multi-master replication • Cons: 24 Wednesday, September 18, 13
  233. 233. Replication /third-party tools • Table level Replication Tools (Trigger based) • Slony • Bucardo • Mimeo: http://pgxn.org/dist/mimeo/ • Pros: • Allows to open replicated database in read- write mode • Allows table/database level replication • Allows rolling upgrade • cross version replication is allowed • Multi-master replication • Cons: • Complicated to setup 24 Wednesday, September 18, 13
  234. 234. Replication /third-party tools • Table level Replication Tools (Trigger based) • Slony • Bucardo • Mimeo: http://pgxn.org/dist/mimeo/ • Pros: • Allows to open replicated database in read- write mode • Allows table/database level replication • Allows rolling upgrade • cross version replication is allowed • Multi-master replication • Cons: • Complicated to setup • Unknown territory 24 Wednesday, September 18, 13
  235. 235. 25 Wednesday, September 18, 13
  236. 236. Sharding 25 Wednesday, September 18, 13
  237. 237. Sharding 25 Wednesday, September 18, 13
  238. 238. Sharding 25 Wednesday, September 18, 13
  239. 239. Sharding • Sharding is the process of splitting up your data so it resides in different tables or often different physical databases. 25 Wednesday, September 18, 13
  240. 240. Sharding • Sharding is the process of splitting up your data so it resides in different tables or often different physical databases. • Application aware sharding 25 Wednesday, September 18, 13
  241. 241. Sharding • Sharding is the process of splitting up your data so it resides in different tables or often different physical databases. • Application aware sharding • Application transparent sharding 25 Wednesday, September 18, 13
  242. 242. 26 Wednesday, September 18, 13
  243. 243. Application aware sharding 26 Wednesday, September 18, 13
  244. 244. Application aware sharding 26 Wednesday, September 18, 13
  245. 245. Application aware sharding 26 Wednesday, September 18, 13
  246. 246. Application aware sharding • http://instagram-engineering.tumblr.com/ post/10853187575/sharding-ids-at- instagram 26 Wednesday, September 18, 13
  247. 247. Application aware sharding • http://instagram-engineering.tumblr.com/ post/10853187575/sharding-ids-at- instagram • Postgres allows “logical” shards through Schema 26 Wednesday, September 18, 13
  248. 248. Application aware sharding • http://instagram-engineering.tumblr.com/ post/10853187575/sharding-ids-at- instagram • Postgres allows “logical” shards through Schema • Easy to move to “physical” shard later 26 Wednesday, September 18, 13
  249. 249. 27 Wednesday, September 18, 13
  250. 250. Application transparent sharding 27 Wednesday, September 18, 13
  251. 251. Application transparent sharding 27 Wednesday, September 18, 13
  252. 252. 28 Wednesday, September 18, 13
  253. 253. Sharding Challenges 28 Wednesday, September 18, 13
  254. 254. Sharding Challenges • Reliability 28 Wednesday, September 18, 13
  255. 255. Sharding Challenges • Reliability • Distributed queries 28 Wednesday, September 18, 13
  256. 256. Sharding Challenges • Reliability • Distributed queries • Cross-shard join 28 Wednesday, September 18, 13
  257. 257. Sharding Challenges • Reliability • Distributed queries • Cross-shard join • Auto-increment key management 28 Wednesday, September 18, 13
  258. 258. Sharding Challenges • Reliability • Distributed queries • Cross-shard join • Auto-increment key management • Choosing shard key 28 Wednesday, September 18, 13
  259. 259. Sharding Challenges • Reliability • Distributed queries • Cross-shard join • Auto-increment key management • Choosing shard key • Shard schemes 28 Wednesday, September 18, 13
  260. 260. 29 Wednesday, September 18, 13
  261. 261. Obstacles for Scaling Postgres 29 Wednesday, September 18, 13
  262. 262. Obstacles for Scaling Postgres 29 Wednesday, September 18, 13
  263. 263. Obstacles for Scaling Postgres • Postgres table bloat 29 Wednesday, September 18, 13
  264. 264. Obstacles for Scaling Postgres • Postgres table bloat • FKs relationships 29 Wednesday, September 18, 13
  265. 265. Obstacles for Scaling Postgres • Postgres table bloat • FKs relationships • Insufficient logging 29 Wednesday, September 18, 13
  266. 266. Obstacles for Scaling Postgres • Postgres table bloat • FKs relationships • Insufficient logging • Insufficient Caching 29 Wednesday, September 18, 13
  267. 267. Obstacles for Scaling Postgres • Postgres table bloat • FKs relationships • Insufficient logging • Insufficient Caching • Insufficient Monitoring and Metrics 29 Wednesday, September 18, 13
  268. 268. Obstacles for Scaling Postgres • Postgres table bloat • FKs relationships • Insufficient logging • Insufficient Caching • Insufficient Monitoring and Metrics • ORMs 29 Wednesday, September 18, 13
  269. 269. Obstacles for Scaling Postgres • Postgres table bloat • FKs relationships • Insufficient logging • Insufficient Caching • Insufficient Monitoring and Metrics • ORMs • Single Point of Failure 29 Wednesday, September 18, 13
  270. 270. Obstacles for Scaling Postgres • Postgres table bloat • FKs relationships • Insufficient logging • Insufficient Caching • Insufficient Monitoring and Metrics • ORMs • Single Point of Failure • Lack of communications between teams 29 Wednesday, September 18, 13
  271. 271. 30 Wednesday, September 18, 13
  272. 272. Beyond Postgres 30 Wednesday, September 18, 13
  273. 273. Beyond Postgres 30 Wednesday, September 18, 13
  274. 274. Beyond Postgres • Avoid serialization in application code 30 Wednesday, September 18, 13
  275. 275. Beyond Postgres • Avoid serialization in application code • Feature Flags 30 Wednesday, September 18, 13
  276. 276. Beyond Postgres • Avoid serialization in application code • Feature Flags • Browse only mode (Read only mode) 30 Wednesday, September 18, 13
  277. 277. Beyond Postgres • Avoid serialization in application code • Feature Flags • Browse only mode (Read only mode) • Don’t use database for Queuing 30 Wednesday, September 18, 13
  278. 278. Beyond Postgres • Avoid serialization in application code • Feature Flags • Browse only mode (Read only mode) • Don’t use database for Queuing • RabbitMQ 30 Wednesday, September 18, 13
  279. 279. Beyond Postgres • Avoid serialization in application code • Feature Flags • Browse only mode (Read only mode) • Don’t use database for Queuing • RabbitMQ • Reconsider options for Full Text Search 30 Wednesday, September 18, 13
  280. 280. Beyond Postgres • Avoid serialization in application code • Feature Flags • Browse only mode (Read only mode) • Don’t use database for Queuing • RabbitMQ • Reconsider options for Full Text Search • tsearch provided by Postgres 30 Wednesday, September 18, 13
  281. 281. Beyond Postgres • Avoid serialization in application code • Feature Flags • Browse only mode (Read only mode) • Don’t use database for Queuing • RabbitMQ • Reconsider options for Full Text Search • tsearch provided by Postgres • Solr, Lucene 30 Wednesday, September 18, 13
  282. 282. 31 Wednesday, September 18, 13
  283. 283. Further reading . . . 31 Wednesday, September 18, 13
  284. 284. Further reading . . . • Scalable Internet Architectures - Theo Schlossnagle • Web Operations: Keeping the Data On Time - John Allspaw , Jesse Robbins • PostgreSQL 9.0 High Performance - Greg Smith 31 Wednesday, September 18, 13
  285. 285. 32 Wednesday, September 18, 13
  286. 286. References 32 Wednesday, September 18, 13
  287. 287. References 32 Wednesday, September 18, 13
  288. 288. References • http://www.postgresql.org/docs/ • http://www.circonus.com • http://pgbouncer.projects.pgfoundry.org/doc/usage.html • http://pgxn.org/dist/pg_partman/doc/pg_partman.html • http://dalibo.github.io/pgbadger/ • https://github.com/omniti-labs/omnipitr • http://instagram-engineering.tumblr.com/post/10853187575/ sharding-ids-at-instagram • http://memcached.org • http://redis.io • http://postgres-xc.sourceforge.net/docs/1_1/ 32 Wednesday, September 18, 13
  289. 289. Thanks • PostgresOpen Conference Committee • OmniTI • You!! 33 Wednesday, September 18, 13

×