Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Postgres-BDR with Google Cloud Platform

922 views

Published on

구글 클라우드 플랫폼에서 Postgre-BDR 을 구성하고 테스트한 내용입니다.
보다 자세한 설정은 https://gongjak.me 에서 확인이 가능합니다.

Published in: Software
  • Be the first to comment

Postgres-BDR with Google Cloud Platform

  1. 1. POSTGRES-BDR WITH GOOGLE CLOUD PLATFORM ROCKPLACE Email: sjyun@rockplace.co.kr http://rockplace.co.kr Copyright ⓒ 2016 Rockplace Inc.. All right Reserved
  2. 2. Copyright ⓒ All right Reserved by 2016 Rockplace Inc. 윤성재 (공작명왕) gongjak@gmail.com https://gongjak.me • 리눅스 엔지니어 • 서비스 운영 Administrator • Database 운영 Administrator • 클라우드 솔루션 아키텍쳐 • 1984년 Apple II 와 만남 • 1994년 Linux 와의 첫 만남 • 2015년 서비스에 PostgreSQL 처음 적용 • 2016년 빠르게 훑어보는 구글 클라우드 플랫폼
  3. 3. Copyright ⓒ All right Reserved by 2016 Rockplace Inc. AGENDA 구글 클라우드 플랫폼 - 테스트 환경으로 선택한 이유 Postgres-BDR - Installation & Setup - Create Cluster pgbench Test - Test 실행 - 결과
  4. 4. 구글 클라우드 플랫폼 테스트 환경으로 선택한 이유
  5. 5. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Create Instance on Google Cloud Platform • Region 위치 : asia-east, eu-west, us-central • 각 Region 별로 3대씩, 총 9 대 생성 • Region 별 2 대는 BDR 설치, 1대는 pgbench 테스트용 • OS : Debian 8 Jessie
  6. 6. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved VPN 을 이용한 Network 구성 Separated Network Using VPN Setup Route table
  7. 7. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved GCP 의 Network 구성 Single Network No VPN No Setup Route table
  8. 8. POSTGRES-BDR Installation & Setup
  9. 9. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Install Postgres-BDR BDR extension 이 반드시 설치되어야 한다. (package의 경우, postgresql-bdr-contrib) sudo sh -c 'echo "deb [arch=amd64] http://packages.2ndquadrant.com/bdr/apt/ jessie-2ndquadrant main" >> /etc/apt/sources.list.d/2ndquadrant.list' wget --quiet -O - http://packages.2ndquadrant.com/bdr/apt/AA7A6805.asc | sudo apt-key add - sudo apt-get update sudo apt-get -y install postgresql-bdr-9.4-bdr-plugin
  10. 10. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Initial PostgreSQL PostgreSQL 에 문제가 있다고 생각되어 초기화를 하고자 할 때 sudo service postgresql stop sudo rm -rf /var/lib/postgresql/9.4 sudo rm -rf /etc/postgresql/9.4/main sudo rm -rf /service/db/pgsql/9.4/main sudo chown -R postgres.postgres /var/lib/postgresql sudo pg_createcluster -d /var/lib/postgresql/9.4/main 9.4 main sudo service postgresql start
  11. 11. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Add DB account Username : dbadmin Password : dbadmin1234 sudo su - postgres psql -dpostgres -c "CREATE ROLE dbadmin LOGIN PASSWORD 'dbadmin1234' superuser;"
  12. 12. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved BDR을 위한 PostgreSQL 설정 /etc/postgresql/9.4/main/postgresql.conf • listen_addresses = '*' • shared_preload_libraries = 'bdr' • wal_level = 'logical' • track_commit_timestamp = on • max_connections = 100 • max_wal_senders = 10 • max_replication_slots = 10 • max_worker_processes = 10
  13. 13. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved BDR을 위한 PostgreSQL 설정 /etc/postgresql/9.4/main/pg_hba.conf all all 10.128.0.0/9 md5 # The standby server must connect with a user that has replication privileges. # for BDR setting echo 'host replication postgres 10.128.0.0/9 trust # 중간 IPv4부분을 아래와 같이 10.128.0.0/9 해 줘야 로컬에서도 접속이 됨 host all all 10.128.0.0/9 trust
  14. 14. POSTGRES-BDR Create Cluster
  15. 15. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Setup Postgres-BDR All nodes sudo su – postgres createdb testdb psql postgres=# connect testdb You are now connected to database "testdb" as user "postgres". testdb=# CREATE EXTENSION btree_gist; CREATE EXTENSION testdb=# CREATE EXTENSION bdr; CREATE EXTENSION
  16. 16. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Setup Postgres-BDR Instance 생성 순서 및 Cluster Join 2. bdr-asia-2 1. bdr-asia-1 3. bdr-us-1 4. bdr-us-2 5. bdr-eu-1 6. bdr-eu-2
  17. 17. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Setup Postgres-BDR BDR Cluster 내부 구성 bdr-asia-2 bdr-asia-1 bdr-us-1 bdr-us-2 bdr-eu-1 bdr-eu-2
  18. 18. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Setup Postgres-BDR bdr-asia-1 서버에서의 설정 (메인 노드 그룹 생성) testdb=# SELECT bdr.bdr_group_create( local_node_name := 'asia-node-001', node_external_dsn := 'host=bdr-asia-1 port=5432 dbname=testdb’ ); bdr_group_join ---------------- (1 row)
  19. 19. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Setup Postgres-BDR bdr-asia-1 서버에서의 설정 (확인) testdb=# SELECT bdr.bdr_node_join_wait_for_ready(); bdr_node_join_wait_for_ready ------------------------------ (1 row)
  20. 20. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Setup Postgres-BDR bdr-asia-2 서버와 그 외 서버에서의 설정 (메인 그룹 참여) testdb=# SELECT bdr.bdr_group_join( local_node_name := 'asia-node-002', node_external_dsn := 'host=bdr-asia-2 port=5432 dbname=testdb', join_using_dsn := 'host=bdr-asia-1 port=5432 dbname=testdb' ); bdr_group_join ---------------- (1 row)
  21. 21. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Setup Postgres-BDR bdr-asia-2 서버와 그 외 서버에서의 설정 (확인) testdb=# SELECT bdr.bdr_node_join_wait_for_ready(); bdr_node_join_wait_for_ready ------------------------------ (1 row)
  22. 22. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Add node • pg_basebackup 으로 SOURCE로부터 전체 백업 • bdr_init_copy 명령어를 이용 • 주요 Configuration Setting : 충분히 크게~~ max_wal_senders = 10 max_replication_slots = 10 max_worker_processes = 10 • bdr_init_copy_postgres.log 를 확인
  23. 23. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Remove node • 함수로 쉽게 노드를 제거 할 수 있다고 하나, 실제로는 함수 실행 후 확인해보면 노드 정보가 남아있다. (select * from bdr.bdr_nodes;) • node 하나만 제거 • 여러개의 노드를 한 번에 제거 • asia-node-003 제거 SELECT bdr.bdr_part_by_node_names(ARRAY['node-1']); SELECT bdr.bdr_part_by_node_names(ARRAY['node-1', 'node-2', 'node-3']); SELECT bdr.bdr_part_by_node_names(ARRAY['asia-node-003']);
  24. 24. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Monitoring • Monitoring nodes • Monitoring connected peers using pg_stat_replication • Monitoring replication slots • Monitoring global DDL locks SELECT * FROM bdr.bdr_nodes; SELECT * FROM pg_stat_replication; SELECT pg_xlog_location_diff(pg_current_xlog_insert_location(), flush_location) AS lag_bytes, pid, application_nameFROM pg_stat_replication; SELECT * FROM pg_replication_slots; SELECT slot_name, database, active, pg_xlog_location_diff(pg_current_xlog_insert_location(), restart_lsn) AS retained_bytesFROM pg_replication_slotsWHERE plugin = 'bdr'; select * from bdr.bdr_global_locks ;
  25. 25. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Backup • 백업은 snapshot 또는 pg_dump, pg_basebackup 이용
  26. 26. PGBENCH TEST Test 실행
  27. 27. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved pgbench Test 실행 pgbench -U postgres -P 5432 -i testdb NOTICE: table "pgbench_history" does not exist, skipping NOTICE: table "pgbench_tellers" does not exist, skipping NOTICE: table "pgbench_accounts" does not exist, skipping NOTICE: table "pgbench_branches" does not exist, skipping creating tables...100000 of 100000 tuples (100%) done (elapsed 0.15 s, remaining 0.00 s). vacuum... set primary keys... done. -U postgres : postgres 유저로 접속 -P 5432 : 5432 포트로 접속 -i : 테스트준비 testdb : 디비로 testdb 를 이용하겠다.
  28. 28. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved pgbench Test 실행 pgbench -U postgres -p 5432 -S -c 10 -t 10000 testdb starting vacuum...end. transaction type: SELECT only scaling factor: 1 query mode: simple number of clients: 10 number of threads: 1 number of transactions per client: 10000 number of transactions actually processed: 100000/100000 latency average: 0.000 ms tps = 11496.597122 (including connections establishing) tps = 11520.436044 (excluding connections establishing) -U postgres : postgres 유저로 접속 -P 5432 : 5432 포트로 접속 -i : 테스트준비 testdb : 디비로 testdb 를 이용하겠다.
  29. 29. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Custom query 를 이용한 pgbench test • create table tbl_bdr(c1 int); • vi test.sql insert into tbl_bdr values (3); • pgbench -U postgres -n -S -T 60 -c 10 -f test.sql testdb transaction type: Custom query scaling factor: 1 query mode: simple number of clients: 10 number of threads: 1 duration: 60 s number of transactions actually processed: 37 9920 latency average: 1.579 ms tps = 6331.344284 (including connections est ablishing) tps = 6332.955115 (excluding connections est ablishing)
  30. 30. PGBENCH 결과
  31. 31. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved 각 Region 별 ping 테스트 결과 Count : 100 Ping 방향 결과 Asia  US 100 packets transmitted, 100 received, 0% packet loss, time 99131ms rtt min/avg/max/mdev = 153.147/153.280/154.927/0.547 ms US  Asia 100 packets transmitted, 100 received, 0% packet loss, time 99132ms rtt min/avg/max/mdev = 153.192/153.291/153.849/0.403 ms Asia  EU 100 packets transmitted, 100 received, 0% packet loss, time 99134ms rtt min/avg/max/mdev = 257.257/257.393/258.276/0.412 ms EU  Asia 100 packets transmitted, 100 received, 0% packet loss, time 99135ms rtt min/avg/max/mdev = 257.297/257.418/258.220/0.582 ms US  EU 100 packets transmitted, 100 received, 0% packet loss, time 99175ms rtt min/avg/max/mdev = 105.583/105.661/106.485/0.435 ms EU  US 100 packets transmitted, 100 received, 0% packet loss, time 99075ms rtt min/avg/max/mdev = 105.564/105.661/106.434/0.132 ms
  32. 32. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved case 1. 한 대의 서버에만 데이타 insert Number of Client Time (s) number of transactions actually processed latency average tps (a) tps (b) 1 600 608506 0.986 ms 1014.175220 1014.186607 5 600 2042736 1.469 ms 3404.536435 3404.757975 10 600 3928831 1.527 ms 6547.974117 6548.823522 * tps (a) : including connections establishing * tps (b) : excluding connections establishing
  33. 33. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Target : 1 Server / single table / 1 Client / Time 600s
  34. 34. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Target : 1 Server / single table / 5 Client / Time 600s
  35. 35. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Target : 1 Server / single table / 10 Client / Time 600s 8:38:43 8:45:08 8:45:37
  36. 36. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved case 3-1. 모든 6대 서버의 동일한 table에 데이타 insert 1 Client * tps (a) : including connections establishing * tps (b) : excluding connections establishing Server number of transactions actually processed latency average tps (a) tps (b) asia-1 534279 1.123 ms 890.464274 890.475745 asia-2 502395 1.194 ms 837.324570 837.335128 us-1 599298 1.001 ms 998.827496 998.839511 us-1 579556 1.035 ms 965.924930 965.935587 eu-1 536775 1.118 ms 894.624563 894.635521 eu-2 491666 1.220 ms 819.441429 819.451690
  37. 37. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Target : All Server / single table / 1 Client / Time 600s
  38. 38. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved case 2. 모든 6대 서버의 동일한 table에 데이타 insert 5 Client * tps (a) : including connections establishing * tps (b) : excluding connections establishing Server number of transactions actually processed latency average tps (a) tps (b) asia-1 1473954 2.035 ms 2456.558290 2456.709572 asia-2 1419295 2.114 ms 2365.475865 2365.627880 us-1 1885541 1.591 ms 3142.556035 3142.733625 us-1 1963248 1.528 ms 3272.055383 3272.237804 eu-1 1463636 2.050 ms 2439.364252 2439.493710 eu-2 1327172 2.260 ms 2211.920534 2212.054668
  39. 39. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Target : All Server / single table / 5 Client / Time 600s
  40. 40. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved case 2-3. 모든 6대 서버의 동일한 table에 데이타 insert 10 Client * tps (a) : including connections establishing * tps (b) : excluding connections establishing Server number of transactions actually processed latency average tps (a) tps (b) asia-1 2463532 2.436 ms 4105.810928 4106.300082 asia-2 2331712 2.573 ms 3886.117520 3886.644086 us-1 3116742 1.925 ms 5194.523249 5195.134521 us-1 3368447 1.781 ms 5614.055016 5614.691748 eu-1 2276560 2.636 ms 3794.106359 3794.560195 eu-2 2085027 2.878 ms 3475.015874 3475.501669
  41. 41. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Target : All Server / single table / 10 Client / Time 600s
  42. 42. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved case 3-1. 모든 6대 서버의 서로 다른 table에 데이타 insert 1 Client * tps (a) : including connections establishing * tps (b) : excluding connections establishing Server number of transactions actually processed latency average tps (a) tps (b) asia-1 529021 1.134 ms 881.701129 881.711758 asia-2 522471 1.148 ms 870.784077 870.795609 us-1 550149 1.091 ms 916.913836 916.924565 us-1 575519 1.043 ms 959.198086 959.209793 eu-1 532446 1.127 ms 887.408480 887.418299 eu-2 484087 1.239 ms 806.810680 806.820653
  43. 43. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Target : All Server / each table / 1 Client / Time 600s
  44. 44. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved case 3-2. 모든 6대 서버의 서로 다른 table에 데이타 insert 5 Client * tps (a) : including connections establishing * tps (b) : excluding connections establishing Server number of transactions actually processed latency average tps (a) tps (b) asia-1 1431422 2.096 ms 2385.673194 2385.813896 asia-2 1427147 2.102 ms 2378.573572 2378.737027 us-1 1859986 1.613 ms 3099.954957 3100.141470 us-1 1926080 1.558 ms 3210.126357 3210.306652 eu-1 1445888 2.075 ms 2409.783223 2409.921491 eu-2 1432652 2.094 ms 2387.676745 2387.819102
  45. 45. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Target : All Server / each table / 5 Client / Time 600s
  46. 46. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved case 3-3. 모든 6대 서버의 서로 다른 table에 데이타 insert 10 Client * tps (a) : including connections establishing * tps (b) : excluding connections establishing Server number of transactions actually processed latency average tps (a) tps (b) asia-1 2371002 2.531 ms 3951.535516 3952.012653 asia-2 2287929 2.622 ms 3813.095796 3813.547846 us-1 3132266 1.916 ms 5220.347514 5221.013372 us-1 3130898 1.916 ms 5218.109569 5218.779341 eu-1 2564901 2.339 ms 4274.759315 4275.233359 eu-2 2244939 2.673 ms 3741.478971 3741.955598
  47. 47. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Target : All Server / each table / 10 Client / Time 600s
  48. 48. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved 마치며… • Support Multi-Master • Multi Region Replication • Effect Network latency • Community Support Email : bdr-list@2ndquadrant.com Forum : https://groups.google.com/a/2ndquadrant.com/forum/#!forum/bdr-list .
  49. 49. OpenSource PLACE, ROCKPLACE Copyright ⓒ 2016 Rockplace Inc. All rights Reserved Q & A
  50. 50. 감사합니다. http://gongjak.me gongjak@gmail.com

×