1. Pluggable Infrastructure with
CI/CD and Docker.
Bob Killen
Systems Programmer/Analyst Senior
Department of Pathology
University of Michigan Health System
Jeffrey Sica
Business Systems Analyst Lead
Department of Pathology
University of Michigan Health System

2. Disclaimer
All opinions spoken by Bob or Jeff are their own and not that of the
University of Michigan or its affiliates.

3. A bit of background
Central IT vs Pathology Informatics
HIPAA Concerns
Our original infrastructure
● Before Docker, there were VMs
● Started on Kubernetes
● Migrated to Individual Hosts

4. Challenges we faced
● Managing multiple instances of the same infrastructure
○ Public applications must be firewalled from Internal applications (HIPAA)
● Rapid prototyping and demoing of applications
○ Victims of our own success
● Large infrastructure, small staff
○ 40+ independant in house developed applications or services
○ 1 Designer, 1.5 Developers, 1.5 Sysadmins

5. Workflow - Outside of Host Cluster
Commit and
Code Push
Source Control server notifies
Build Server of new commit.
Build Server schedules
build at new commit.
If build successful; tag image and
push to registry. Otherwise notify
group.
After successful build pushed to the registry,
trigger cluster update. Notify if problem arises
during update.Build

6. Workflow - In Cluster
Cluster is notified
of new task.
Cluster determines hosts with
appropriate resource offerings.
Hosts pull containers
associated with task.
Once containers pass healthcheck,
front end is notified of host:port
Front End updates configuration
and reloads with new services.
Service is now accessible to public network

7. The Pillars of a Container Centric Infrastructure

8. The path we have chosen.
Gitlab Jenkins DTR Mesos
Docker

9. Gitlab
● Open source
● Close-enough mirror to the familiar (GitHub)
● Low barrier to entry for non-developers
● Existing integration into other “pillars” (Jenkins)
● Now comes in a handy dandy container
○ “ docker pull gitlab/gitlab-ce ”
○ https://hub.docker.com/r/gitlab/gitlab-ce/

10. Jenkins
● Open Source
● Time tested solution
● Plug-ins galore
○ Docker
○ Mesos Cluster
○ Gitlab Webhooks
● No real constraints on architecture
○ No plugin? Use Bash!
● Blank slate for any project
● Windows Support

11. Docker Trusted Registry
● Security Requirements
○ 100% on premise.
○ LDAP backed.
○ RBAC - v0 api available, use and integration capability growing.
■ https://docs.docker.com/docker-trusted-registry/api/
○ One-click upgrade.
○ User friendly UI.
● Support Offerings
○ 24/7 support REQUIRED for use in a clinical care setting.
○ Guided install with staff on-site where needed

12. Mesos
● Generic abstraction of resources. Useful to combine varied host systems (both
physical and virtual).
● Caters well to an on prem solution.
● Slightly more mature in the cluster solution ecosystem.
○ Powered by Mesos Users - http://mesos.apache.org/documentation/latest/powered-by-mesos/
● Large framework ecosystem; with lots of active development.

13. Mesos (Cont.)
● https://open.mesosphere.com/frameworks/
Just to name a
few….

14. Mesos (Cont…future)
Mesos Windows Epic: https://issues.apache.org/jira/browse/MESOS-3094
Docker + Windows + Mesos = Match made in the clouds
Screencast Demo: https://www.youtube.com/watch?v=dC2vitIh84M

15. Docker (CE)
● Support Offerings
○ Touching on this again...24/7 support is REQUIRED when used in a clinical setting.
● Certified stable releases suitable for production use.
● Predictable release schedule for upgrade planning.
● Orchestration components are swappable...docker engine is not, and is the base
requirement for everything.

17. Final Form: Pluggable Infrastructure Pt. 1
● Application container function should be independent of infrastructure design.
○ Container init script + environment or cluster seed script.
■ container init script variables are static
■ seed script populates container init script with cluster specific information.
■ Build server can inject seed script for environment (local dev vs cluster deployment)

18. Example Init Script
$VAR1 and $VAR2 can be set at
container creation or ‘seeded’ via a
script sourced as
$ENVIRONMENT_INIT. The
supplied script could hit a variety
of endpoints for configuration
information (consul, etcd,
registrator, etc).
If nothing is provided, it assumes
the default value.
This allows for init script to remain
static, with only the sourced seed
script changing per environment.

19. Final Form: Pluggable Infrastructure Pt. 2
● Infrastructure components in source control; and if at all possible built as
containers themselves.
○ Apply standard ‘infrastructure as code’ methods.
■ Version Everything
■ Design tests and integrate with CI/CD

20. Final Form: Pluggable Infrastructure Pt. 2 (Cont.)
● Components that may modify host configuration require some level of clean-up. e.
g. anything that may require iptables.
○ trap SIGINT, SIGKILL etc. then perform action.
○ Does require some form of ‘init’ or process management. - supervisord, tini, monit etc
● Use of VRRP / Keepalived and deploying in pairs when services may be bound to
a single point of failure (no DNS load balancing etc).
○ Provides incredibly simple IP failover.
○ Prerequisites for use:
■ sysctl - net.ipv4.ip_nonlocal_bind=1
■ container run with host networking and cap NET_ADMIN
○ Added bonus of making it EXTREMELY easy to upgrade and switch services.
■ If upgrading: push new version to backup, then push to master.

28. Keepalived + HAproxy Demo

29. Example Keepalived Configuration
Master Backup Description
● state - MASTER or BACKUP - Defines server role in the group.
● interface - The interface that keepalived will use for VRRP traffic and
monitor by default.
● vrrp_unicast_bind - The ip in which vrrp should bind to.
● vrrp_unicast_peer - The bind ip of the other node in the group.
● virtual_router_id - Unique number between 1-255 that identifies the vrrp
group. Multiple instances of vrrp can run at the same time on the same servers.
● priority - In scenarios with more than 2 nodes, priority will dictate the
failover order with highest priority becoming the master.
● advert_int - Healthcheck interval (seconds)
● authentication - PASS (cleartext password) or AH (IPSEC
Authorization Header). Password and method used by the nodes
to validate each other.
● virtual_ipaddress - Virtual IP Address to be generated and interface it
should be bound to. IPs in this block WILL be tracked by keepalived. Used for
private network interface tracking.
● virtual_ipaddress_excluded - Virtual IP Address to be generated and
interface it should be bound to. IPs in this block will NOT be tracked, but
automatically failed over if an event would trigger a failure of IP(s) in the
virtual_ipaddress block. Used for public facing IPs.
● track_interface - Interface adapters that should have their link state
monitored for failover trigger.

31. Future State Challenges
● Better Secret Management
○ Consul / Vault injection at runtime and seeded via ENVIRONMENT_INIT.
○ Mount secure volume
● Profiling containers for lxc capabilities + apparmor profiles
○ https://github.com/jfrazelle/bane
● Improve logging with logstash framework
○ https://github.com/mesos/logstash
● Notary integration into build pipeline.

33. Questions?
Bob Killen
rkillen@med.umich.edu
@mrbobbytables
github.com/mrbobbytables
Jeffrey Sica
jsica@med.umich.edu
@jeefy
github.com/jeefy