Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
What to Upload to SlideShare
What to Upload to SlideShare
Loading in …3
×
1 of 34

patroni-based citrus high availability environment deployment

0

Share

Download to read offline

pgday.seoul 2021

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

patroni-based citrus high availability environment deployment

  1. 1. Patroni-based citus high availability environment deployment ( 基于Patroni的Citus高可用环境部署 ) PGDay.Seoul & PGConf.Asia 2021 Daniel Lee 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 1
  2. 2. About Me 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 2 • Dainel lee ( 李炯彩 ) • Community • MySQL Korea User Group • facebook.com/groups/mysqlko • PostgreSQL Korea • facebook.com/groups/130019777169719 • Github • github.com/dbdbdeep-git/pgday-seoul-2021-demo
  3. 3. Agenda 1. Service Discovery ( *Service Mesh ) • CoreOS’s ETCD • Hashicorp’s Consul 2. Dynamic Configuration Management • ConfD • Hashicorp’s Consul-Template 3. High-Availability • Zalando’s Patroni 4. Distributed PostgreSQL as an extension • Citusdata’s Citus 5. Demo 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 3
  4. 4. 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 4
  5. 5. 1. Service Discovery 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 5
  6. 6. What is … 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 6 Lock Service (chubby) Coordination Service (zookeeper,eureka,etcd) Service Discovery (surf,skydns,consul) Service Mesh (lstio,linkerd2)
  7. 7. What can you do with Consul? 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 7 Service Discovery for connectivity Service Registry enables services to register and discover each other. Service Segmentation for security Secure service-to-service communication with automatic TLS encryption and identity-based authorization. Service Configuration for runtime configuration Feature rich Key/Value store to easily configure services.
  8. 8. What can you do with Consul? 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 8 Consul-Kubernetes Deployments Use Consul service discovery and service mesh features with Kubernetes. Secure Service Communication Secure and observe communication between your services without modifying their code. Dynamic Load Balancing Automate load balancer configuration with Consul and HAProxy, Nginx, or F5.
  9. 9. 3 Marine Admirals 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 9 Properties Consul Etcd Zookeeper User Interface Available X X RPC Available Available Health Check HTTP API HTTP API TCP Key Value 3 Consistency modes (sync) Good Consistency (semi) Strong Consistency Token System Available X X Language Golang Golang Java Security ACL / HTTPS HTTPS ACL Spring Cloud O O O Multi Datacenter(gossip) O X X
  10. 10. 2. Dynamic Configuration 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 10
  11. 11. ConfD • http://www.confd.io • Manage local application configuration files using templates and data from etcd or consul • A lightweight configuration management tool focused on: • keeping local configuration files up-to-date using data stored in etcd, consul, dynamodb, redis, vault, zookeeper, aws ssm parameter store or env vars and processing template resources. • reloading applications to pick up new config file changes 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 11
  12. 12. ConfD or Consul-template Architecture 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 12 ETCD or Consul Server Confd or Consul-template Service Application (HAproxy, PostgreSQL) Configuration Template File Application Configuration File listen generate haproxy.cfg.tmpl haproxy.cfg reload check reload update
  13. 13. PostgreSQL Client authentication 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 13 Consul Server Consul-template Service Postgres Server Configuration Template File Application Configuration File listen generate pg_hba.conf.tmpl pg_hba.conf reload check reload update $ consul kv put service/database/primary/auth/client “host all haproxy 10.0.1.0/24 reject”
  14. 14. 3. Zalando’s Patroni 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 14
  15. 15. Zalando Company 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 15
  16. 16. ’s Patroni • Patroni is a template for you to create your own customized, high-availability solution using Python and - for maximum accessibility - a DCS (Distributed Configuration Store) like ZooKeeper, etcd, Consul or Kubernetes. • Currently supported PostgreSQL versions: 9.3 to 14. • Release lastest 2.1.1 ( 09.11.21 ) 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 16
  17. 17. ’s Spilo • Spilo is a Docker image that provides PostgreSQL and Patroni bundled together. Patroni is a template for PostgreSQL HA. Multiple Spilos can create a resilient High Available PostgreSQL cluster. For this, you'll need to start all participating Spilos with identical etcd addresses and cluster names. • https://github.com/zalando/spilo 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 17
  18. 18. 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 18
  19. 19. Raft : DC to DC 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 19
  20. 20. HA : Patroni vs pg_auto_failover 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 20
  21. 21. 4. Citusdata’s Citus 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 21
  22. 22. Hyperscale (Citus) on Azure Database for PostgreSQL 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 22
  23. 23. Citusとは? by Noriyoshi Shinoda • PostgreSQL でスケールアウト環境を実現 • 複数ノードにまたがったパラレル・クエリーとパーティショニング機能 • スループット拡大を目指す • PostgreSQL の拡張(EXTENSION)として実装 • PostgreSQL 本体の変更なし • Citus Data (https://www.citusdata.com/)が開発 • 2019年1月 Microsoft による買収 • Azure Database for PostgreSQL - Hyperscale (Citus) の中核技術となっている • オープンソース版も提供(https://github.com/citusdata/citus) • 以下の機能は含まない • 自動フェイルオーバー • 自動データ・リバランス • バックアップ等の運用機能 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 23 https://www.slideshare.net/noriyoshishinoda/postgresql-conference-japan-2021-b2-citus-10
  24. 24. Citus 란? • PostgreSQL로 스케일 아웃 환경 실현 • 다중 노드에 걸친 병렬 쿼리 및 파티셔닝 기능 • 처리량 확대를 목표로 • PostgreSQL 확장(EXTENSION)으로 구현 • PostgreSQL 본체 변경 없음 • Citus Data (https://www.citusdata.com/) 개발 • 2019년 1월 Microsoft에서 인수 • Azure Database for PostgreSQL - Hyperscale (Citus)의 핵심 기술입니다. • 오픈 소스 버전도 제공 (https://github.com/citusdata/citus) • 다음 기능은 포함되지 않음 • 자동 장애 조치 • 자동 데이터 재조정 • 백업 등의 운용 기능 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 24
  25. 25. What is Citus ? • Achieve a scale-out environment with PostgreSQL • Parallel query and partitioning capabilities across multiple nodes • Aim to increase throughput • Implemented as an extension of PostgreSQL (EXTENSION) • No changes to PostgreSQL itself • Developed by Citus Data (https://www.citusdata.com/) • Acquired by Microsoft in January 2019 • Azure Database for PostgreSQL-is the core technology of Hyperscale (Citus) • Open source version is also available (https://github.com/citusdata/citus) • The following functions are not included • Automatic failover • Automatic data rebalancing • Operational functions such as backup 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 25
  26. 26. What is Citus ? by Noriyoshi Shinoda Instance configuration • Coordinator Node • A PostgreSQL instance that accepts connections from clients • Manage metadata • Worker Node • PostgreSQL instance that actually stores the data • No communication between Worker Nodes • Install citus EXTENSION on all nodes 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 26
  27. 27. What is Citus ? by Noriyoshi Shinoda Instance configuration • Processing performed by Coordinator Node • Accepting connections from clients • Final sort (ORDER BY) • Sequence processing (including SERIAL column and GENERATED AS IDENTITY column) • Handling objects other than tables and indexes • Processing executed by Worker Node • Processing SQL statements requested by Coordinator Node • Execution of ANALYZE statement • Execution of VACUUM statement • Retaining table data 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 27
  28. 28. CREATE EXTENSION citus; • Add nodes: postgres=> SELECT master_add_node(‘worker-01_ip_or_dns’, 5432); Postgres=> SELECT master_add_node(’worker-02_ip_or_dns’, 5432); • Create distributed table: postgres=> CREATE TABLE events ( orders_id int, ... ); postgres=> SELECT create_distributed_table( ‘orders', ‘orders_id’ ); • Events is now distributed across shards on the nodes 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 28
  29. 29. if(kakao) 2021 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 29 https://if.kakao.com/session/59
  30. 30. Table structure by Noriyoshi Shinoda Distributed table • A table that distributes and stores data • Suitable for fact tables • Specify a column as the distribution key (default determines the distribution destination table by the hash value range) • The number of divisions can be specified • citus.shard_count (default value 32) • Replicas can be created on different Worker Nodes • citus.shard_replication_factor (default value 1 = no replica) 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 30
  31. 31. Table structure by Noriyoshi Shinoda Distributed table • Table created in worker Node • For the table name, ShardID is added to the original table name. • If a replica is specified, a table with the same name will be created in a different Worker Node. • Same data is stored in the table with the same name • Search the citus_shards catalog to see the table names created by the Worker Node postgres=> SET citus.shard_count = 6 ; postgres=> SET citus.shard_replication_factor = 2 ; 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 31 Coordinator Worker#1 Worker#2 Worker#3 Dist1 dist1_102046 dist1_102046 dist1_102047 dist1_102047 dist1_102048 dist1_102048 dist1_102049 dist1_102049 dist1_102050 dist1_102050 dist1_102051 dist1_102051
  32. 32. Demo | Skip 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 32
  33. 33. if(kakao) 2021 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 33
  34. 34. Q?!A 2021-11-24 PGDay.Seoul & PGConf.Asia 2021 34

×