PostgreSQL and
Linux Containers
Jignesh Shah
Founding Team @appOrbit
SF Bay Area PostgreSQL User Group – Jan 5th 2015

Disclaimer
The views expressed in this presentation are my own and do
not necessarily reflect the views of appOrbit.

About Jignesh (@jkshah)
• appOrbit
• My focus is on managing Application Data running in Containers
• VMware
• Lead and manage Postgres and Data Management teams at VMware for various products embedding PostgreSQL
running in virtualized embedded instances
• Sun Microsystems
• Team Member of first published SpecJAppServer 2004 benchmark with PostgreSQL
• Performance of PostgreSQL on Solaris/Sun Servers
• Working with PostgreSQL community since 2005
• http://jkshah.blogspot.com/2005/04/profiling-postgresql-using-dtrace-on_22.html
• Working with Container technologies (Solaris Zones) since 2004
• http://jkshah.blogspot.com/2004/08/db2-working-under-solaris-10-zones_30.html

Agenda
• Containers
• Definition
• Early examples
• Linux Containers
• Underlying Technologies
• LXC
• Systemd-nspawn
• Docker Containers
• Installation
• Images
• Volumes
• PostgreSQL in Docker Container
• What it means
• Best practices

What are Containers?
• OS Level virtualization where kernel allows for multiple isolated user-space instances
Operating System
Bare Metal Server
OS
Bare Metal Server
Hypervisor
OS
Operating System
Bare Metal Server
C C C C C OS
Bare Metal Server
Hypervisor
OS
C C C C

Advantages of Containers
• Lower footprint
• Very Quick Startup and Shutdown time
• Density
• Nesting

Disadvantages of Containers
• Same Kernel version
• Cannot run other OS natively
• Security (to be improved)

Where to use container?
• Recreate identical environment (cookie-cutter)
• Resource Grouping of specific processes in heavily loaded server
• Handling multiple versions of software applications
• Ephemeral application instances (Dev/Test)
• Many more

Implementations of Containers
• Chroot circa 1982
• FreeBSD Jails circa 2000
• Solaris Zones circa 2004
• Meiosys – MetaClusters with Checkpoint/Restore 2004-05
• Linux OpenVZ circa 2005 (not in mainstream Linux)
• AIX WPARs circa 2007
• LXC circa 2008
• Systemd-nspawn circa 2010-2013
• Docker circa 2013

What makes containers possible?
• Process Group Isolation
• Filesystem Isolation
• Network Isolation
• CPU Isolation
• Memory Isolation
• Example – Solaris Containers
• Base implementation provided Process, Filesystem and Network Isolation
• Resource pools consisting of CPU, memory was originally used
• Branded Zones – Userland Library Isolation

What makes Linux containers possible ?
• cgroups
• Allows limitation and prioritization of resources (CPU, memory, block I/O, network, etc.)
• Namespace isolation
• Mount namespace
• PID namespace
• Network namespace
• UTS (Allows changing hostname, domainname)
• IPC Namespace
• User namespace
• LXC
• Combines kernel’s cgroup and namespaces to provide an isolated environment

LXC
• CentOS 7 + EPEL Repository set
• Commands available
• Quick Guide to use an LXC based container of busybox
lxc-attach lxc-clone lxc-destroy lxc-ls lxc-stop lxc-usernsexec
lxc-autostart lxc-config lxc-execute lxc-monitor lxc-top lxc-wait
lxc-cgroup lxc-console lxc-freeze lxc-snapshot lxc-unfreeze
lxc-checkconfig lxc-create lxc-info lxc-start lxc-unshare
wget https://www.busybox.net/downloads/binaries/busybox-x86_64 -O busybox
chmod a+x busybox
PATH=$(pwd):$PATH lxc-create -t busybox -n mycontainer
lxc-start -d -n mycontainer
lxc-console –n mycontainer # (Use CTRL-A Q to exit console mode)
lxc-stop -n mycontainer
lxc-destroy -n mycontainer
yum install epel-release
yum install bridge-utils libvirt lxc lxc-templates

Systemd-nspawn
• Systemd
• Replacement of SysV init scripts
• Systemd-nspawn
• Used to run a command or OS in light weight namespace container
• Installed on most newer distros by default
• Commands available
systemd-analyze systemd-delta systemd-nspawn
systemd-ask-password systemd-detect-virt systemd-run
systemd-cat systemd-cgls systemd-loginctl
systemd-sysv-convert systemd-cgtop systemd-machine-id-setup
systemd-coredumpctl systemd-notify systemd-tty-ask-password-agent
systemd-inhibit systemd-stdio-bridge systemd-tmpfiles
systemctl machinectl hostnamectl journalctl
yum install systemd

Systemd-nspawn
• Quick guide to a container deployment using systemd-nspawn
• Useful tools
machinectl status mycontainer
systemd-cgls
systemd-cgtop
# Create an Image
yum -y --releasever=7 --nogpg --installroot=/mycontainers/centos7 install
systemd passwd yum fedora-release vim-minimal
# Change the root password in the image
systemd-nspawn -D /mycontainers/centos7
passwd
exit
# Start the container as if booting into the container image
systemd-nspawn -D /mycontainers/centos7 –M mycontainer –b
# Get into the container
nsenter -m -u -i -n -p –t $PID
machinectl login mycontainer

Trends of Container Technologies

Disruption of trends by Docker

Docker
• Installation
• Commands available with docker binary
• Quick Guide to use a docker based container
attach build commit cp create diff events
exec export history images import info inspect
kill load login logout logs pause port
ps pull push rename restart rm rmi
run save search start stats stop tag
top unpause version wait
docker run --name mycontainer -e POSTGRES_PASSWORD=mysecretpassword -d postgres
docker exec -ti mycontainer psql -U postgres
docker stop mycontainer
docker rm mycontainer
docker rmi postgres
yum install docker
systemctl start docker

Dockerfile – Custom Recipe
• Container images are created using “Dockerfile”
• Build an image using the recipe file
• Push to a public or private registry (hub account login or private registry needed)
FROM centos:centos7.0.1406
RUN yum install -y epel-release
RUN yum install -y nginx
EXPOSE 80
CMD ["bash", "-l", "-c", "/usr/sbin/nginx -g "daemon off;""]
docker build –t jkshah/nginx –t .
docker push jkshah/nginx:latest

Docker Images
• Docker Images are layered templates used by containers instances
• Container instance images are layered Copy on Write Images based on Docker images
• You can mutate your local container images (unless you use --read-only flag)
• Not good for fast and frequent changing data
• Can select a different underlying type using --storage-driver
docker push jkshah/nginx:latest
Do you really want to push to public registry? [y/n]: y
The push refers to a repository [docker.io/jkshah/nginx] (len: 1)
d498b5680966: Pushed
4b1d16518ce1: Pushed
3185bef36db4: Pushed
2f7013eef4b8: Pushed
539eca37bade: Pushed
f1b10cd84249: Pushed
latest: digest:
sha256:46208d1f0393946b33c2bdce498168de507b7186a897f332bab0cffc1ea601c7 size:
10608

Docker Volumes
• Persists beyond the life of a Docker container
• VOLUME command in Dockerfile or
• Using –v using docker run command
• Automatically created if not already present during docker run
• Not part of docker push/pull operations
• Can select a non-local directory using --volume-driver
• Third party components required to get multi-host support (NFS, etc )
• On CentOS with SELinux enabled need to set security context
• Different options using –v
• -v /hostsrc/data:/opt/data:ro # for read only volumes (default rw)
• -v /hostsrc/data:/opt/data:Z # Z – private volume, z – shared volume
• -v /etc/nginx.conf:/etc/nginx.conf # for mounting a single file only
• Volumes can be shared from another container using --volumes-from on same host
• Docker 1.9 gives first class status to Docker Volumes
chcon -Rt svirt_sandbox_file_t /hostpath/pgdata

PostgreSQL in Docker container
• Quick Deployment:
• Check Deployment:
• Use inspect command to get more information about the container
• To check PostgreSQL system logs
docker run --name mycontainer -v /hostpath/pgdata:/var/lib/postgresql/data -e
POSTGRES_PASSWORD=mysecretpassword -d postgres
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8d720b31c1fe postgres "/docker-entrypoint.s" 5 minutes ago Up 5 minutes 5432/tcp mycontainer
docker logs mycontainer
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
…
LOG: database system was shut down at 2016-01-04 22:58:40 UTC
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
docker inspect mycontainer

PostgreSQL in docker container
• Part of Postgres Dockerfile has
• Impacts
• Port is exposed only to other containers directly linking with this container
• Data is persistent only for the life of container (docker rm) but not easily accessible outside the container
• -v enables to access the data outside container
• Linking to PostgreSQL Server container
• Ports are exposed externally using –p option
• Verify using
docker run --name myapp
--link mycontainer:ds2db
-e POSTGRES_USER=postgres
-e POSTGRES_PASSWORD=mysecretpassword
-p 8080:80
-d jkshah/dvdstore2
EXPOSE 5432
VOLUME /var/lib/postgresql/data
docker port myapp
80/tcp -> 0.0.0.0:8080

PostgreSQL in docker container
• Check for statistics using
• Check for top processes in a container
docker stats mycontainer myapp
CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O
myapp 0.01% 51.51 MB/1.924 GB 2.68% 54.42 kB/7.576 MB
mycontainer 0.01% 133.3 MB/1.924 GB 6.93% 7.577 MB/54.42 kB
docker top mycontainer
UID PID PPID C STIME TTY TIME CMD
polkitd 25313 10698 0 16:58 ? 00:00:00 postgres
polkitd 25385 25313 0 16:58 ? 00:00:00 postgres: checkpointer process
polkitd 25386 25313 0 16:58 ? 00:00:00 postgres: writer process
polkitd 25387 25313 0 16:58 ? 00:00:00 postgres: wal writer process
polkitd 25388 25313 0 16:58 ? 00:00:00 postgres: autovacuum launcher process
polkitd 25389 25313 0 16:58 ? 00:00:00 postgres: stats collector process
docker top myapp
UID PID PPID C STIME TTY TIME CMD
root 30747 10698 0 18:56 ? 00:00:00 httpd -D FOREGROUND
48 30796 30747 0 18:56 ? 00:00:00 httpd -D FOREGROUND
48 30797 30747 0 18:56 ? 00:00:00 httpd -D FOREGROUND
48 30798 30747 0 18:56 ? 00:00:00 httpd -D FOREGROUND
48 30799 30747 0 18:56 ? 00:00:00 httpd -D FOREGROUND
48 30800 30747 0 18:56 ? 00:00:00 httpd -D FOREGROUND

PostgreSQL in docker container
• Alternate way to see Process Tree using systemd-cgls tool
systemd-cgls
├─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 21
├─user.slice
│ └─user-0.slice
│ └─session-34.scope
│ ├─25129 sshd: root@pts/0
│ ├─25131 -bash
│ └─31397 systemd-cgls
└─system.slice
├─docker-9457652d7f6ec24ebd95305e788fe39030b049deb22f240ee2b7383488d0c215.scope
│ ├─30747 httpd -D FOREGROUND
│ ├─30796 httpd -D FOREGROUND
│ ├─30797 httpd -D FOREGROUND
│ ├─30798 httpd -D FOREGROUND
│ ├─30799 httpd -D FOREGROUND
│ └─30800 httpd -D FOREGROUND
├─docker-8d720b31c1fe0de77d3cd89942c1a72902b67d466f821ce63ee1271561a36451.scope
│ ├─25313 postgres
│ ├─25385 postgres: checkpointer process
│ ├─25386 postgres: writer process
│ ├─25387 postgres: wal writer process
│ ├─25388 postgres: autovacuum launcher process
│ └─25389 postgres: stats collector process
├─docker.service
│ ├─10698 /usr/bin/docker daemon --selinux-enabled
│ └─30739 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8080 -contain...

PostgreSQL in Docker container
• Alternate way to see top containers using systemd-cgtop
systemd-cgtop
Path Tasks %CPU Memory Input/s Output/s
/ 100 0.9 1.5G - -
/user.slice 3 0.9 70.7M - -
/system.slice - 0.0 268.5M - -
/system.slice/docker.service 2 0.0 31.9M - -
/system.slice/nimbus.service 5 0.0 20.0K - -
/system.slice/tuned.service 1 0.0 36.0K - -
/system.slice/dock...49deb22f240ee2b7383488d0c215.scope 6 0.0 49.1M - -
/system.slice/auditd.service 1 - 11.9M - -
/system.slice/avahi-daemon.service 2 - 12.0K - -
/system.slice/crond.service 1 - 92.0K - -
/system.slice/dbus.service 1 - - - -
/system.slice/dock...7d466f821ce63ee1271561a36451.scope 6 - 127.3M - -
/system.slice/gssproxy.service 1 - - - -
/system.slice/libvirtd.service 3 - - - -
/system.slice/lvm2-lvmetad.service 1 - - - -
/system.slice/polkit.service 1 - 16.0K - -
/system.slice/postfix.service 3 - 3.5M - -
/system.slice/rsyslog.service 1 - 2.1M - -
/system.slice/sshd.service 1 - 3.0M - -
/system.slice/system-getty.slice/getty@tty1.service 1 - - - -
/system.slice/systemd-journald.service 1 - 16.0M - -
/system.slice/systemd-logind.service 1 - 20.0K - -
/system.slice/systemd-udevd.service 1 - 520.0K - -
/system.slice/xe-linux-distribution.service 2 - 1.0M - -
/user.slice/user-0.slice/session-34.scope 3 - - - -

PostgreSQL as a database server container
• Maybe you want a database server standalone
• Not all database clients will be on docker containers in the same host
• Need to limit memory usage
• Need different layout of how files are distributed (separage XLOG )
• Use the –p option to make the port available even to non containers clients
• Use –m to limit memory usage by the DB server (by default it can see and use all)
• Note this does not set shared buffers automatically with the library image
docker run --name mycontainer
-m 4g
-e POSTGRES_PASSWORD=mysecretpassword
-v /hostpath/pgdata:/var/lib/postgresql/data
-p 5432:5432 -d postgres

PostgreSQL in an enterprise environment
• However for a real production use case we would need
• Bigger shared memory configurations
• Need different layout of how files are distributed (separage XLOG )
• Ability to backup the database
• Ability to setup replication
• etc
• In short we need a more custom image of PostgreSQL

Best Practices for custom image
• For production install customize the docker image
• Allocate proper memory limits - example 8GB
• All pagecache usage shows up as docker container memory usage
• Bump up shared buffers and other parameters as required
• Hint: use PostgreSQL 9.3 or later otherwise have to privileged containers
• http://jkshah.blogspot.com/2015/09/is-it-privilege-to-run-container-in.html
• Support multiple volumes in your image
• Pg_xlog
• PITR archives
• Tablespaces as required
• Full Backup directory
• PostgreSQL Extensions
• Setup replication support
• Out of box replication setup
• Monitoring Tool
• Your favorite monitoring agent

References
• http://www.freedesktop.org/wiki/Software/systemd/
• https://linuxcontainers.org/
• http://www.haifux.org/lectures/299/netLec7.pdf
• http://haifux.org/lectures/320/netLec8_final.pdf

Revolutionary platform to encapsulate and manage
both legacy and new micro-services based applications
with data and configurations running on containers and
virtual machines in a private, public or hybrid cloud.
Put your DevOps on hyper-drive!
Efficiency and Velocity for Apps & Infrastructure
We are
HIRING !!!