In my talk, I will tell how we built a geographically distributed system of personal data storage based on Open Source software and PostgreSQL. The concept of the inCountry business is to provide customers with a ready-to-use infrastructure for personal data storage. Our business customers are ensured that their customer’s personal data is securely stored within their country’s borders. We wrote an API and SDK and built a variety of services. Our system complies with generally accepted security standards (SOC Type 1, Type 2, PCI DSS, etc.). We built our infrastructure with Consul, Nomad, and Vault, used PostgreSQL, ElasticSearch as a storage system, Nginx, Jenkins, Artifactory, other tools to automate management and deployment. We have assembled our development and management teams - DevOps, Security, Monitoring, and DBA. We use both cloud providers and bare-metal servers located in different regions of the world. Development of the system architecture and ensuring the stability of the infrastructure, consistent and secure operation of all its components is the main task facing our teams.

1. Pavel Konotopov - kakoka@gmail.com
Leoinid Albrecht - lalbrekht@gmail.com
DBA team
InCountry
Storing regulated data in 62 countries of the World

2. Idea
•Let's create and provide
the global business with an
infrastructure for the local
storage of regulated data
• Infrastructure must meet certain
requirements
2

3. PII Data
• PII Data Laws
Personal data: the ability to uniquely identify a person
"... any information relating directly or indirectly to a specific
or determinable natural person - the subject of personal data ..."
• PII data processing
set of actions performed with personal data, including collection, recording, systematization,
accumulation, storage, clarification, change, extraction, use, transfer, depersonalization, blocking,
deletion, destruction
3

4. Requirements
• Store data in the country of its origin;
• Geo-distributed infrastructure;
• At the same time, the data cannot cross the borders of the country of origin;
• Compliance with industry safety standards: SOC, PCI DSS, HIPAA, ISO ...;
• Compliance with the laws of the states: FZ152, GDPR, CCPA ...
• Ability to integrate into ready-made systems
• Encryption support
• Acceptable speed
• Cheap (we are a startup)
business
architect
4

5. Products
• REST API
- no extra effort
- modifies data inside an existing service
• SDK
- Java, PHP, Python, Node.js SDKs
- Self-service
- Client-side encryption
• Border proxy
5

6. Service Architecture
User App REST API
PoP InCountry
DB Cluster
6

7. PoP
FLIGHT RECORD
GDPR
PROFILE
PAYMENT
TRANSACTION
App
Your frontend web service
uses actual data in-country
Customer ID: 98765
First Name: John
Last Name: Smith
SS#: 123-45-6789
Phone: +1-415-555-1212
Your backend web
service uses redacted data
Customer ID: 98765
First Name: HashKey.fname
Last Name: HashKey.lname
SS#: HashKey.ss
Phone: HaskKey.phone
7

8. Infrastructure
•Points of presence - 62 countries
• 30% Cloud
• 70% BARE-METAL
• Immediately forget about the K8s
• 2 data centers in the country
• 4 to 8 servers per country
•Centralized deployment
infrastructure
•Centralized management
infrastructure
•Security
•Monitoring
8

9. Infrastructure
Nomad cluster
Consul cluster
Zabbix EFK Grafana
Jenkins
Regional HUB
EDGE
Application
Database
Backup
EDGE
Application
Database
Backup
Country of presence
DC1 DC2
Regional hub
9

10. Cheap vs expensive vs reasonable
•Use Open Source and make Enterprise
•But there is a problem
OpenSource + DevOps = T x $$$
Limited OpenSource + Devops = $ + T x $
Non-OpenSource + DevOps = $$$ + T x $
+ =
10

11. Tech Stack
• Terraform/Packer/Ansible/AWX
• Consul – discovery service
• Nomad – scheduler
• Vault – repository of secrets
• Docker – containerization
• Jenkins – CI/CD
• Nginx
• PostgreSQL/Pgbouncer/Patroni
• Zabbix/ElasticSearch/Fluentd/Grafana/Kibana
•JumpCloud – LDAP
•Cisco VPN
•OpsGenie – alerting
•JFROG artifactory – storage
of docker images
OpenSource Not OpenSource
11

12. ●Geo-distribution
●2 DCs in one country
●High Availability
●Sync Replica
●Auto Failover
●Protection from split brain. Fencing.
12
PostgreSQL Tasks

13. ●PostgreSQL
●pgbounсer - connection puller
●Patroni - a cluster management mechanism
●HashiCorp Consul:
● stores cluster status information
● stores the current cluster configuration
● Consul Templates
●HashiCorp Nomad:
● Automatically raises containers when they fail
●HashiCorp Vault - stores secrets, static and dynamic roles 13
PostgreSQL Tools

14. 14
PostgreSQL: Deployment
DB Server

15. ●As an out-of-the-box solution PostgreSQL does not have
automatic failover
●Daemon running next to PostgreSQL
●Interacts with DCS
●Patroni daemon decides on promotion / demotion
●Config in the form of YAML
●patronictl - convenient tool not only for DBA
15
PostgreSQL: Patroni

16. ●Cons
● You can’t just stop the container.
● In some “special” cases, the
container is cut down in an
unexpected place.
16
●Pros
● Centralized management location
● Declarative approach
● Convenient resource allocation for the
container
PostgreSQL: Nomad

17. update {
max_parallel = 1
min_healthy_time = "10s"
healthy_deadline = "3m"
progress_deadline = "10m"
auto_revert = false
canary = 0
}
…
resources {
cpu = ${NOMAD_RESOURCES_CPU}
memory = ${NOMAD_RESOURCES_MEM}
network {
mbits = 1
port "${NOMAD_POSTGRES_SERVICE_NAME}" {
static = “${NOMAD_POSTGRES_PORT}"…
17
Nomad.TemplateFile
port_map {
${NOMAD_PATRONI_SERVICE_NAME} = “$
{NOMAD_PATRONI_PORT}"
}
port_map {
${NOMAD_POSTGRES_SERVICE_NAME} = “$
{NOMAD_POSTGRES_PORT}”
}
mounts = [
{
type = "bind"
source = "/etc/patroni/patroni.yml"
target =
"/home/postgres/.config/patroni/patroni.yml"
readonly = true
},

18. 18
● Persistent Storage for database files
● Facilitate container with multi-stage build
● ~70-90 Mb image
● As a result, acceleration of delivery to the region
● It lives well if the container does not fall
● Virtually no overhead
PostgreSQL: Docker

19. 19
PostgreSQL: HashiCorp Vault
Vault
PostgreSQLUser/Application
● Keeps static secrets
● Allows serving static roles
● Comply with password update policy
● Can create dynamic roles
● Such roles are created for a given time when
querying Vault
● Have the necessary rights
● Deleted after Expiration
1. GET Creds 2. CREATE ROLE
3. SEND Creds
4. Login

20. 20
● Slots enabled by default. But don’t forget about wal_keep_segments
● Synchronous replication specificities:
● network is a bottleneck
● synchronous mode
● maximum_lag_on_failover: 1048576 (1M)
PostgreSQL: Replication

21. 21
●pg_probackup
●Validation
●Logging and Monitoring
●Customized retention policies for customers
●Specifics of using two DCs:
● Dual WAL archive
● Double backup archive
PostgreSQL: Backup

22. 22
PostgreSQL: Backup
DC1
DB Master
Backup
DC2
DB StandBy
Backup
WAL push
WAL
Archive
Backup
storage
WAL
Archive
Backup
storage
Backup Fetch

23. 23
• By default, a replica is created using the pg_basebackup utility.
• You can override this behavior with the ‘create_replica_methods’ parameter
postgresql:
create_replica_methods:
- probackup
- basebackup
probackup:
command: "ssh dbbackup@192.168.0.250 ‘bash /var/backup/pg_restore.sh'"
no_params: True
basebackup:
max-rate: '100M'
PostgreSQL: Backup

24. 24
● The ability to recover from backup to any point by:
● Time
● Transaction id (xid)
● Transactional Log Entry LSN
● A rare occurrence with a synchronous replica, but you need to prepare:
bootstrap:
method: probackup
probackup:
command: ssh dbbackup@backup 'bash /var/backup/pg_restore.sh'
keep_existing_recovery_conf: false
recovery_conf:
recovery_target_timeline: latest
restore_command: pg_probackup archive-get -B /var/backup --instance db-mt --remote-user=dbbackup
--wal-file-path %p --wal-file-name %f —remote-host=255.255.255.255
PostgreSQL: cluster recovery

25. 25
● Docker image for minimal launch
● Backup script from backup
● Minimum postgresql.conf to start
● Checking the success of the launch
● docker exec pgvalid pg_dump -h localhost -U postgres > /dev/null
● Service table for comparing data before and after backup
● amcheck
● CREATE EXTENSION amcheck
● pg_probackup checkdb —amcheck —heapallindexed
● We log and monitor every action and its success (failure).
PostgreSQL: Validation

26. ● Everything related to Consul:
● Network problems in one DC
● Not available from both DCs
● Replication and HA issues. Frequent failures during network
outages
● Unexpected container restart
Troubleshooting

27. Troubleshooting: Distances
27

28. Troubleshooting: Audit
• Personally identifiable information in the logs
• Pgaudit - missing
• DB Api - stored procedures
• Access limitation
• We give rights only to those who need
them
• Choosing data centers that are certified
(PCI DSS)

29. Project Team
• Architecture Team
• Developer Team
• DevOps
• CloudOps/DevOps
• SRE
• Monitoring Team
• DBA Team
• Security Team
• Application Security Team
• QA and Performance Testing Team
• NOC, SOC
Всё. Осталось вкрутить лампочку.
29

30. DBA role
•DBA role:
• Little Architect
• Little DevOps
• Little SecOps
• A bit of monitoring
• A bit of QA & Perf
• Little Developer
• Linux Administrator
30

31. inCountry team