Postgres-BDR with Google Cloud Platform

2. Copyright ⓒ All right Reserved by 2016 Rockplace Inc. 윤성재 (공작명왕) gongjak@gmail.com https://gongjak.me • 리눅스 엔지니어 • 서비스 운영 Administrator • Database 운영 Administrator • 클라우드 솔루션 아키텍쳐 • 1984년 Apple II 와 만남 • 1994년 Linux 와의 첫 만남 • 2015년 서비스에 PostgreSQL 처음 적용 • 2016년 빠르게 훑어보는 구글 클라우드 플랫폼

4. 구글 클라우드 플랫폼 테스트 환경으로 선택한 이유

5. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Create Instance on Google Cloud Platform • Region 위치 : asia-east, eu-west, us-central • 각 Region 별로 3대씩, 총 9 대 생성 • Region 별 2 대는 BDR 설치, 1대는 pgbench 테스트용 • OS : Debian 8 Jessie

8. POSTGRES-BDR Installation & Setup

9. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Install Postgres-BDR BDR extension 이 반드시 설치되어야 한다. (package의 경우, postgresql-bdr-contrib) sudo sh -c 'echo "deb [arch=amd64] http://packages.2ndquadrant.com/bdr/apt/ jessie-2ndquadrant main" >> /etc/apt/sources.list.d/2ndquadrant.list' wget --quiet -O - http://packages.2ndquadrant.com/bdr/apt/AA7A6805.asc | sudo apt-key add - sudo apt-get update sudo apt-get -y install postgresql-bdr-9.4-bdr-plugin

10. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Initial PostgreSQL PostgreSQL 에 문제가 있다고 생각되어 초기화를 하고자 할 때 sudo service postgresql stop sudo rm -rf /var/lib/postgresql/9.4 sudo rm -rf /etc/postgresql/9.4/main sudo rm -rf /service/db/pgsql/9.4/main sudo chown -R postgres.postgres /var/lib/postgresql sudo pg_createcluster -d /var/lib/postgresql/9.4/main 9.4 main sudo service postgresql start

11. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Add DB account Username : dbadmin Password : dbadmin1234 sudo su - postgres psql -dpostgres -c "CREATE ROLE dbadmin LOGIN PASSWORD 'dbadmin1234' superuser;"

12. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved BDR을 위한 PostgreSQL 설정 /etc/postgresql/9.4/main/postgresql.conf • listen_addresses = '*' • shared_preload_libraries = 'bdr' • wal_level = 'logical' • track_commit_timestamp = on • max_connections = 100 • max_wal_senders = 10 • max_replication_slots = 10 • max_worker_processes = 10

13. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved BDR을 위한 PostgreSQL 설정 /etc/postgresql/9.4/main/pg_hba.conf all all 10.128.0.0/9 md5 # The standby server must connect with a user that has replication privileges. # for BDR setting echo 'host replication postgres 10.128.0.0/9 trust # 중간 IPv4부분을 아래와 같이 10.128.0.0/9 해 줘야 로컬에서도 접속이 됨 host all all 10.128.0.0/9 trust

14. POSTGRES-BDR Create Cluster

15. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Setup Postgres-BDR All nodes sudo su – postgres createdb testdb psql postgres=# connect testdb You are now connected to database "testdb" as user "postgres". testdb=# CREATE EXTENSION btree_gist; CREATE EXTENSION testdb=# CREATE EXTENSION bdr; CREATE EXTENSION

16. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Setup Postgres-BDR Instance 생성 순서 및 Cluster Join 2. bdr-asia-2 1. bdr-asia-1 3. bdr-us-1 4. bdr-us-2 5. bdr-eu-1 6. bdr-eu-2

18. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Setup Postgres-BDR bdr-asia-1 서버에서의 설정 (메인 노드 그룹 생성) testdb=# SELECT bdr.bdr_group_create( local_node_name := 'asia-node-001', node_external_dsn := 'host=bdr-asia-1 port=5432 dbname=testdb’ ); bdr_group_join ---------------- (1 row)

19. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Setup Postgres-BDR bdr-asia-1 서버에서의 설정 (확인) testdb=# SELECT bdr.bdr_node_join_wait_for_ready(); bdr_node_join_wait_for_ready ------------------------------ (1 row)

20. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Setup Postgres-BDR bdr-asia-2 서버와 그 외 서버에서의 설정 (메인 그룹 참여) testdb=# SELECT bdr.bdr_group_join( local_node_name := 'asia-node-002', node_external_dsn := 'host=bdr-asia-2 port=5432 dbname=testdb', join_using_dsn := 'host=bdr-asia-1 port=5432 dbname=testdb' ); bdr_group_join ---------------- (1 row)

21. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Setup Postgres-BDR bdr-asia-2 서버와 그 외 서버에서의 설정 (확인) testdb=# SELECT bdr.bdr_node_join_wait_for_ready(); bdr_node_join_wait_for_ready ------------------------------ (1 row)

22. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Add node • pg_basebackup 으로 SOURCE로부터 전체 백업 • bdr_init_copy 명령어를 이용 • 주요 Configuration Setting : 충분히 크게~~ max_wal_senders = 10 max_replication_slots = 10 max_worker_processes = 10 • bdr_init_copy_postgres.log 를 확인

23. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Remove node • 함수로 쉽게 노드를 제거 할 수 있다고 하나, 실제로는 함수 실행 후 확인해보면 노드 정보가 남아있다. (select * from bdr.bdr_nodes;) • node 하나만 제거 • 여러개의 노드를 한 번에 제거 • asia-node-003 제거 SELECT bdr.bdr_part_by_node_names(ARRAY['node-1']); SELECT bdr.bdr_part_by_node_names(ARRAY['node-1', 'node-2', 'node-3']); SELECT bdr.bdr_part_by_node_names(ARRAY['asia-node-003']);

24. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Monitoring • Monitoring nodes • Monitoring connected peers using pg_stat_replication • Monitoring replication slots • Monitoring global DDL locks SELECT * FROM bdr.bdr_nodes; SELECT * FROM pg_stat_replication; SELECT pg_xlog_location_diff(pg_current_xlog_insert_location(), flush_location) AS lag_bytes, pid, application_nameFROM pg_stat_replication; SELECT * FROM pg_replication_slots; SELECT slot_name, database, active, pg_xlog_location_diff(pg_current_xlog_insert_location(), restart_lsn) AS retained_bytesFROM pg_replication_slotsWHERE plugin = 'bdr'; select * from bdr.bdr_global_locks ;

26. PGBENCH TEST Test 실행

27. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved pgbench Test 실행 pgbench -U postgres -P 5432 -i testdb NOTICE: table "pgbench_history" does not exist, skipping NOTICE: table "pgbench_tellers" does not exist, skipping NOTICE: table "pgbench_accounts" does not exist, skipping NOTICE: table "pgbench_branches" does not exist, skipping creating tables...100000 of 100000 tuples (100%) done (elapsed 0.15 s, remaining 0.00 s). vacuum... set primary keys... done. -U postgres : postgres 유저로 접속 -P 5432 : 5432 포트로 접속 -i : 테스트준비 testdb : 디비로 testdb 를 이용하겠다.

28. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved pgbench Test 실행 pgbench -U postgres -p 5432 -S -c 10 -t 10000 testdb starting vacuum...end. transaction type: SELECT only scaling factor: 1 query mode: simple number of clients: 10 number of threads: 1 number of transactions per client: 10000 number of transactions actually processed: 100000/100000 latency average: 0.000 ms tps = 11496.597122 (including connections establishing) tps = 11520.436044 (excluding connections establishing) -U postgres : postgres 유저로 접속 -P 5432 : 5432 포트로 접속 -i : 테스트준비 testdb : 디비로 testdb 를 이용하겠다.

29. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Custom query 를 이용한 pgbench test • create table tbl_bdr(c1 int); • vi test.sql insert into tbl_bdr values (3); • pgbench -U postgres -n -S -T 60 -c 10 -f test.sql testdb transaction type: Custom query scaling factor: 1 query mode: simple number of clients: 10 number of threads: 1 duration: 60 s number of transactions actually processed: 37 9920 latency average: 1.579 ms tps = 6331.344284 (including connections est ablishing) tps = 6332.955115 (excluding connections est ablishing)

30. PGBENCH 결과

31. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved 각 Region 별 ping 테스트 결과 Count : 100 Ping 방향 결과 Asia  US 100 packets transmitted, 100 received, 0% packet loss, time 99131ms rtt min/avg/max/mdev = 153.147/153.280/154.927/0.547 ms US  Asia 100 packets transmitted, 100 received, 0% packet loss, time 99132ms rtt min/avg/max/mdev = 153.192/153.291/153.849/0.403 ms Asia  EU 100 packets transmitted, 100 received, 0% packet loss, time 99134ms rtt min/avg/max/mdev = 257.257/257.393/258.276/0.412 ms EU  Asia 100 packets transmitted, 100 received, 0% packet loss, time 99135ms rtt min/avg/max/mdev = 257.297/257.418/258.220/0.582 ms US  EU 100 packets transmitted, 100 received, 0% packet loss, time 99175ms rtt min/avg/max/mdev = 105.583/105.661/106.485/0.435 ms EU  US 100 packets transmitted, 100 received, 0% packet loss, time 99075ms rtt min/avg/max/mdev = 105.564/105.661/106.434/0.132 ms

32. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved case 1. 한 대의 서버에만 데이타 insert Number of Client Time (s) number of transactions actually processed latency average tps (a) tps (b) 1 600 608506 0.986 ms 1014.175220 1014.186607 5 600 2042736 1.469 ms 3404.536435 3404.757975 10 600 3928831 1.527 ms 6547.974117 6548.823522 * tps (a) : including connections establishing * tps (b) : excluding connections establishing

36. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved case 3-1. 모든 6대 서버의 동일한 table에 데이타 insert 1 Client * tps (a) : including connections establishing * tps (b) : excluding connections establishing Server number of transactions actually processed latency average tps (a) tps (b) asia-1 534279 1.123 ms 890.464274 890.475745 asia-2 502395 1.194 ms 837.324570 837.335128 us-1 599298 1.001 ms 998.827496 998.839511 us-1 579556 1.035 ms 965.924930 965.935587 eu-1 536775 1.118 ms 894.624563 894.635521 eu-2 491666 1.220 ms 819.441429 819.451690

38. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved case 2. 모든 6대 서버의 동일한 table에 데이타 insert 5 Client * tps (a) : including connections establishing * tps (b) : excluding connections establishing Server number of transactions actually processed latency average tps (a) tps (b) asia-1 1473954 2.035 ms 2456.558290 2456.709572 asia-2 1419295 2.114 ms 2365.475865 2365.627880 us-1 1885541 1.591 ms 3142.556035 3142.733625 us-1 1963248 1.528 ms 3272.055383 3272.237804 eu-1 1463636 2.050 ms 2439.364252 2439.493710 eu-2 1327172 2.260 ms 2211.920534 2212.054668

40. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved case 2-3. 모든 6대 서버의 동일한 table에 데이타 insert 10 Client * tps (a) : including connections establishing * tps (b) : excluding connections establishing Server number of transactions actually processed latency average tps (a) tps (b) asia-1 2463532 2.436 ms 4105.810928 4106.300082 asia-2 2331712 2.573 ms 3886.117520 3886.644086 us-1 3116742 1.925 ms 5194.523249 5195.134521 us-1 3368447 1.781 ms 5614.055016 5614.691748 eu-1 2276560 2.636 ms 3794.106359 3794.560195 eu-2 2085027 2.878 ms 3475.015874 3475.501669

42. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved case 3-1. 모든 6대 서버의 서로 다른 table에 데이타 insert 1 Client * tps (a) : including connections establishing * tps (b) : excluding connections establishing Server number of transactions actually processed latency average tps (a) tps (b) asia-1 529021 1.134 ms 881.701129 881.711758 asia-2 522471 1.148 ms 870.784077 870.795609 us-1 550149 1.091 ms 916.913836 916.924565 us-1 575519 1.043 ms 959.198086 959.209793 eu-1 532446 1.127 ms 887.408480 887.418299 eu-2 484087 1.239 ms 806.810680 806.820653

48. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved 마치며… • Support Multi-Master • Multi Region Replication • Effect Network latency • Community Support Email : bdr-list@2ndquadrant.com Forum : https://groups.google.com/a/2ndquadrant.com/forum/#!forum/bdr-list .

50. 감사합니다. http://gongjak.me gongjak@gmail.com

Postgres-BDR with Google Cloud Platform

SungJae Yun

Postgres-BDR with Google Cloud Platform