Scaling postgres

/
Scaling Postgres
Denish Patel
Database Architect
https://twitter.com/DenishPatel
Wednesday, September 18, 13

OmniTI
1

OmniTI
1
• Helping customers
navigate explosive growth
with technology.

OmniTI
1
with technology.
100MM+ users
$1B+ gross online sales

OmniTI
1
with technology.
100MM+ users
$1B+ gross online sales
Open and closed source thought leaders,
experts and authors

Talk Outline
2

Talk Outline
• Scalability
• Database Scaling needs, costs, methods
• Scaling Postgres
• Vertically
• Horizontally
• Obstacles to Scalability
• Beyond Postgres
2

What is Scalability?
3

What is Scalability?
3
A service is said to be scalable if when we
increase the resources in a system, it results in
increased performance in a manner proportional
to resources added.

Why to scale databases?
4

• Support a higher volume of users
4

• Provide better performance for existing users
4

• Store a larger volume of data
4

• Improve system availability
4

• Improve system availability
• Geographic dispersion
4

Why is database scalability so hard?
5

• Search
5

• Search
• Concurrency
5

• Search
• Concurrency
• Consistency
5

• Search
• Concurrency
• Consistency
• Speed
5

• Search
• Concurrency
• Consistency
• Speed
• Cost
5

• Search
• Concurrency
• Consistency
• Speed
• Cost
• Cost of hardware
5

• Search
• Concurrency
• Consistency
• Speed
• Cost
• Cost deployment effort
5

• Search
• Concurrency
• Consistency
• Speed
• Cost
• Cost deployment effort
• Cost Ongoing maintenance
5

Scaling Needs & Methods
6

• Data growth
6

• Data growth
• Read requests
6

• Data growth
• Read requests
• Write requests
6

• Data growth
• Read requests
• Write requests
• Vertical Scaling
• Horizontal Scaling
6

Vertical Scaling (Scale up)
7

• Pros
7

• Pros
• Simple to implement
7

• Pros
• Ease of maintenance
7

• Pros
• Cons
7

• Pros
• Cons
• Sometimes SPOFs
7

Horizontal Scaling (Scale Out)
8

• Pros
8

• Pros
• Cheaper in hardware cost
8

• Pros
• Flexibility
8

• Pros
• Flexibility
• Higher fault tolerance
8

• Pros
• Flexibility
• Cons
8

• Pros
• Flexibility
• Cons
• Complex to implement
8

• Pros
• Flexibility
• Cons
• Expensive to maintain
8

• Pros
• Flexibility
• Cons
• Bigger footprint in the Data Center
8

• Pros
• Flexibility
• Cons
• Bigger footprint in the Data Center
• No built in support in databases
8

Spec’ing Hardware
9

Spec’ing Hardware
• CPU
9

Spec’ing Hardware
• CPU
• 8+ cores
9

Spec’ing Hardware
• CPU
• 8+ cores
• RAM
9

Spec’ing Hardware
• CPU
• 8+ cores
• RAM
• 64GB+
9

Spec’ing Hardware
• CPU
• 8+ cores
• RAM
• 64GB+
• Disks
9

Spec’ing Hardware
• CPU
• 8+ cores
• RAM
• 64GB+
• Disks
• SSDs
9

Spec’ing Hardware
• CPU
• 8+ cores
• RAM
• 64GB+
• Disks
• SSDs
• RAID 10
9

Spec’ing Hardware
• CPU
• 8+ cores
• RAM
• 64GB+
• Disks
• SSDs
• RAID 10
• Network
9

Spec’ing Hardware
• CPU
• 8+ cores
• RAM
• 64GB+
• Disks
• SSDs
• RAID 10
• Network
• min Gigbit, 10Gigbit
9

10

Tune Postgres/memory parameters
10

• shared_buffers
10

• shared_buffers
• effective_cache_size
10

• shared_buffers
• checkpoint_completion_target
10

• shared_buffers
• checkpoint_segments
10

• shared_buffers
• max_connections
10

• shared_buffers
• max_connections
• work_mem
10

• shared_buffers
• max_connections
• work_mem
• maintenance_work_mem
10

• shared_buffers
• max_connections
• work_mem
• maintenance_work_mem
http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server
10

11

Tune Postgres/logging parameters
11

• logging_collector => 'on'
11

• log_destination => 'stderr'
11

• log_filename => 'postgresql-%Y-%m-%d_%H%M%S.log'
11

• log_line_prefix => '%m [%r] [%p]: [%l-1] user=%u,db=%d,e=%e '
11

• log_min_duration_statement => 1000ms
11

• log_min_duration_statement => 1000ms
• log_autovacuum_min_duration => '0'
11

12

• log_lock_waits => 'on'
12

• log_temp_files => '0'
12

• log_checkpoints => 'on'
12

• log_connections => 'on'
12

• log_disconnections => 'on'
12

• log_min_error_statement => 'warning'
12

• log_min_messages => 'warning'
12

• log_min_messages => 'warning'
• log_statement => 'ddl'
12

13

Optimize Queries/report slow queries
13

14

Optimize Queries/Extensive monitoring
14

15

Optimize Queries /Explain Analyze
15

Optimize Queries /Explain Analyze
explain (analyze,buffers) select col1,col2 from demo_ios where col2 between 0.01 and 0.02;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------
Index Only Scan using idx_demo_ios on demo_ios (cost=0.00..35330.93 rows=993633 width=16) (actual time=58.100..3250.589
rows=1000392 loops=1)
Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))
Heap Fetches: 0
Buffers: shared hit=923073 read=3848
Total runtime: 4297.405 ms
15

16

Optimize Queries /track functions
16

• track_functions = pl # none, pl, all
16

• reload online
16

• reload online
• select * from pg_stat_user_functions;
16

17

Partitioning
17

Partitioning
• As table size grows, queries eventually slows down, even with indexing
17

Partitioning
• Allows data added, removed and queried fast
17

Partitioning
• Partitioning pruning queries
17

Partitioning
• Partitioning pruning queries
• Manage partitions
17

18

Partitioning /Postgres
18

• Postgres partitioning
18

• Trigger based
18

• Trigger based
• Rule based
18

• Trigger based
• Rule based
• Lack of built-in Postgres partition management
18

• Trigger based
• Rule based
• Postgres partition management extension: pg_partman
18

• Trigger based
• Rule based
• http://pgxn.org/dist/pg_partman/doc/pg_partman.html
18

• Trigger based
• Rule based
• Tomorrow’s session at 11:30AM (When Postgres can’t....)
18

19

Partitioning /functional
19

Configuration Data Transaction data Session data
Configuration Tools Reporting Tools Monitoring Tools
Web Applications Other Applications
19

• Partition data based on functionality
19

• Separate Postgres clusters
19

• Start with Separate schemas
19

• No relationship between data
19

• Help to spread the load across server
19

• Help to spread the load across server
• Less complex compare to sharding!
19

20

pgbouncer
20

pgbouncer
• A lightweight connection pooler
20

pgbouncer
• Helps to reduce # of newly created connections on DB server
20

pgbouncer
• Abstracts DBs from App
20

pgbouncer
• Helps to instrument smooth and easy failover
20

pgbouncer
• Connection pooling Options
20

pgbouncer
• Session, Transaction, Statement pooling options
20

pgbouncer
• Session, Transaction, Statement pooling options
• Beware! Transaction pooling doesn’t support prepared transactions
20

21

Caching
21

Caching
• Memcached
21

Caching
• Memcached
• Open source, High-performance distributed memory object
caching system
21

Caching
• Memcached
caching system
• Speeds up dynamic web applications by alleviating database load.
21

Caching
• Memcached
caching system
• An in-memory key-value store for small chunks of arbitrary data
21

Caching
• Memcached
caching system
• Redis
21

Caching
• Memcached
caching system
• Redis
• Open source, advanced key-value store.
21

Caching
• Memcached
caching system
• Redis
• Open source, advanced key-value store.
• Works with an in-memory & persistent dataset
21

22

Replication /built-in
22

• Cluster Level Replication (Binary)
22

• Streaming Replication
22

• WAL-only replication
22

• Hybrid replication
22

Master
22

Master
Failover• Cluster Level Replication (Binary)
22

Master
Failover
Read
Salve 1
22

Master
Failover
Read
Salve 1
Streaming Replication
22

Master
Failover
Read
Salve 1
PITR!
22

Master
Failover
Read
Salve 1
PITR!
PITR!
22

Master
Failover
Read
Salve 1
PITR!
PITR!
PITR!
22

• Pros:
Master
Failover
Read
Salve 1
PITR!
PITR!
PITR!
22

• Pros:
• Built-in
Master
Failover
Read
Salve 1
PITR!
PITR!
PITR!
22

• Pros:
• Built-in
• Allows to open replicated database in read-only mode
Master
Failover
Read
Salve 1
PITR!
PITR!
PITR!
22

• Pros:
• Built-in
• Cons:
Master
Failover
Read
Salve 1
PITR!
PITR!
PITR!
22

• Pros:
• Built-in
• Cons:
• All or none
Master
Failover
Read
Salve 1
PITR!
PITR!
PITR!
22

• Pros:
• Built-in
• Cons:
• All or none
• Doesn’t allow write on replicated database
Master
Failover
Read
Salve 1
PITR!
PITR!
PITR!
22

• Pros:
• Built-in
• Cons:
• All or none
• Doesn’t work across major version
Master
Failover
Read
Salve 1
PITR!
PITR!
PITR!
22

• Pros:
• Built-in
• Cons:
• All or none
• Postgres 9.2 (primary) does NOT replicate to 9.3 (secondary)
Master
Failover
Read
Salve 1
PITR!
PITR!
PITR!
22

• Pros:
• Built-in
• Cons:
• All or none
• Postgres 9.2.1 (primary) can replicate to 9.2.4 (secondary)
Master
Failover
Read
Salve 1
PITR!
PITR!
PITR!
22

• Pros:
• Built-in
• Cons:
• All or none
• Postgres 9.2.1 (primary) can replicate to 9.2.4 (secondary)
Master
Failover
Read
Salve 1
PITR!
PITR!
PITR!
https://wiki.postgresql.org/wiki/Binary_Replication_Tools
22

23

Read
Salve 1
23

Read
Salve 1
DW
System
23

Read
Salve 1
DW
System
postgres_fdw
23

• postgres_fdw
Read
Salve 1
DW
System
postgres_fdw
23

• postgres_fdw
• Postgres 9.3 feature
Read
Salve 1
DW
System
postgres_fdw
23

• postgres_fdw
• Allows to access data stored in external PostgreSQL
servers
Read
Salve 1
DW
System
postgres_fdw
23

• postgres_fdw
servers
• cross version queries
Read
Salve 1
DW
System
postgres_fdw
23

• postgres_fdw
servers
• Postgres 9.3 could query Postgres 9.1
Read
Salve 1
DW
System
postgres_fdw
23

• postgres_fdw
servers
• Application
Read
Salve 1
DW
System
postgres_fdw
23

• postgres_fdw
servers
• Application
• Run query remotely on slave db
Read
Salve 1
DW
System
postgres_fdw
23

• postgres_fdw
servers
• Application
• Run query remotely on slave db
• Data warehouse data refreshes
Read
Salve 1
DW
System
postgres_fdw
23

24

Replication /third-party tools
24

• Table level Replication Tools (Trigger based)
24

• Slony
24

• Slony
• Bucardo
24

• Slony
• Bucardo
• Mimeo: http://pgxn.org/dist/mimeo/
24

• Slony
• Bucardo
• Pros:
24

• Slony
• Bucardo
• Pros:
• Allows to open replicated database in read-
write mode
24

• Slony
• Bucardo
• Pros:
write mode
• Allows table/database level replication
24

• Slony
• Bucardo
• Pros:
write mode
• Allows rolling upgrade
24

• Slony
• Bucardo
• Pros:
write mode
• cross version replication is allowed
24

• Slony
• Bucardo
• Pros:
write mode
• Multi-master replication
24

• Slony
• Bucardo
• Pros:
write mode
• Cons:
24

• Slony
• Bucardo
• Pros:
write mode
• Cons:
• Complicated to setup
24

• Slony
• Bucardo
• Pros:
write mode
• Cons:
• Complicated to setup
• Unknown territory
24

25

Sharding
25

Sharding
• Sharding is the process of splitting up your data so it resides in
different tables or often different physical databases.
25

Sharding
• Application aware sharding
25

Sharding
• Application aware sharding
• Application transparent sharding
25

26

Application aware sharding
26

• http://instagram-engineering.tumblr.com/
post/10853187575/sharding-ids-at-
instagram
26

instagram
• Postgres allows “logical” shards through
Schema
26

instagram
• Postgres allows “logical” shards through
Schema
• Easy to move to “physical” shard later
26

27

Application transparent sharding
27

28

Sharding Challenges
28

Sharding Challenges
• Reliability
28

Sharding Challenges
• Reliability
• Distributed queries
28

Sharding Challenges
• Reliability
• Cross-shard join
28

Sharding Challenges
• Reliability
• Auto-increment key management
28

Sharding Challenges
• Reliability
• Choosing shard key
28

Sharding Challenges
• Reliability
• Choosing shard key
• Shard schemes
28

29

Obstacles for Scaling Postgres
29

• Postgres table bloat
29

• FKs relationships
29

• Insufficient logging
29

• Insufficient Caching
29

• Insufficient Monitoring and Metrics
29

• ORMs
29

• ORMs
• Single Point of Failure
29

• ORMs
• Single Point of Failure
• Lack of communications between teams
29

30

Beyond Postgres
30

Beyond Postgres
• Avoid serialization in application code
30

Beyond Postgres
• Feature Flags
30

Beyond Postgres
• Feature Flags
• Browse only mode (Read only mode)
30

Beyond Postgres
• Feature Flags
• Don’t use database for Queuing
30

Beyond Postgres
• Feature Flags
• RabbitMQ
30

Beyond Postgres
• Feature Flags
• RabbitMQ
• Reconsider options for Full Text Search
30

Beyond Postgres
• Feature Flags
• RabbitMQ
• tsearch provided by Postgres
30

Beyond Postgres
• Feature Flags
• RabbitMQ
• tsearch provided by Postgres
• Solr, Lucene
30

31

Scaling postgres

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (20)

Similar to Scaling postgres

Similar to Scaling postgres (20)

More from Denish Patel

More from Denish Patel (13)

Recently uploaded

Recently uploaded (20)

Scaling postgres