Advertisement
Check these out next
ContainerDays Boston 2016: "Autopilot: Running Real-world Applications in Containers" (Tim Gross)
May. 26, 2016•0 likes•918 views
2 likes
Be the first to like this
Show More
views
Total views
0
On Slideshare
0
From embeds
0
Number of embeds
0
Report
Technology
Slides from Tim Gross's talk "Autopilot: Running Real-world Applications in Containers" at ContainerDays Boston 2016: http://dynamicinfradays.org/events/2016-boston/programme.html#autopilot
DynamicInfraDaysFollow
DynamicInfraDaysAdvertisement
Advertisement
Advertisement
Recommended
ContainerDays Boston 2016: "Docker For the Developer" (Borja Burgos)DynamicInfraDays
Docker for any type of workload and any IT InfrastructureDocker, Inc.
Docker Container As A Service - Mix-IT 2016Patrick Chanezon
Openbar 7 - Leuven - OpenShift - The Enterprise Container Platform - PirosOpenbar
A Dive Into Containers and DockerMatthew Farina
Automate App Container Delivery with CI/CD and DevOpsDaniel Oh
ContainerDays Boston 2016: "Autopilot: Running Real-world Applications in Containers" (Tim Gross)
- Applications on Autopilot Tim Gross @0x74696d (“tim”) github.com/autopilotpattern
- github.com/autopilotpattern What if your containers were self-aware and self-operating?
- github.com/autopilotpattern
- github.com/autopilotpattern How do we get from dev to prod? • Service Discovery • Load balancing • Automated-failover • Config changes • Monitoring
- github.com/autopilotpattern App
- github.com/autopilotpattern Nginx Consul MySQL Primary MySQL Replica ES Master ES Data Kibana Prometheus Sales Logstash Customers
- github.com/autopilotpattern Nginx Consul /custom ers /sales /sales/data /customers/data read/write read- only MySQL Primary MySQL Replica ES Master ES Data Kibana Prometheus Sales Logstash Customers Load Balancing
- github.com/autopilotpattern Nginx Consul /custom ers /sales /sales/data /customers/data read/write read- only async replication MySQL Primary MySQL Replica ES Master ES Data Kibana Prometheus Sales Logstash Customers Replication & Fail-over
- github.com/autopilotpattern Nginx Consul /custom ers /sales /sales/data /customers/data read/write read- only async replication MySQL Primary MySQL Replica ES Master ES Data Kibana Prometheus Sales Logstash Customers Service discovery
- github.com/autopilotpattern Nginx Consul /custom ers /sales /sales/data /customers/data read/write read- only async replication MySQL Primary MySQL Replica ES Master ES Data Kibana Prometheus Sales Logstash Customers Logging
- github.com/autopilotpattern Nginx Consul /custom ers /sales /sales/data /customers/data read/write read- only async replication MySQL Primary MySQL Replica ES Master ES Data Kibana Prometheus Sales Logstash Customers Monitoring
- github.com/autopilotpattern Problem: Service Discovery
- github.com/autopilotpattern App Application
- github.com/autopilotpattern Database App Application w/ database
- github.com/autopilotpattern Database App Application w/ database How does the app find the DB? Can we just use DNS?
- github.com/autopilotpattern Couchbase App CouchbaseCouchbaseCouchbase Application w/ Couchbase
- github.com/autopilotpattern Couchbase App CouchbaseCouchbaseCouchbase Application w/ Couchbase Nodes coordinate shards via IP address Can’t use an A-record
- github.com/autopilotpattern Couchbase Couchbase CouchbaseCouchbaseCouchbase App Application w/ Couchbase What happens when we add a node?
- github.com/autopilotpattern Couchbase Couchbase CouchbaseCouchbaseCouchbase App Application w/ Couchbase Does app respect DNS TTL?
- github.com/autopilotpattern Problem: Load Balancing
- github.com/autopilotpattern Nginx Customers Sales Microservices application /sales/data /customers/data /sales /custom ers
- github.com/autopilotpattern Nginx SalesCustomers /sales/data /customers/data /sales /custom ers Microservices application
- github.com/autopilotpattern Nginx SalesCustomers /sales/data /customers/data /sales /custom ers Microservices application How do apps update peers when we scale out?
- github.com/autopilotpattern Nginx SalesCustomers /sales/data /custom ers/data Microservices application Route everything thru Nginx (or LB)?
- github.com/autopilotpattern Nginx SalesCustomers /sales/data /custom ers/data Microservices application How do we update Nginx backends? Adds network path length and SPoF
- github.com/autopilotpattern Sales Sidecar/ Proxy Customers http://localhost http://192.168.1.1 ex. Bamboo Compute node
- github.com/autopilotpattern Sales Sidecar/ Proxy Customers http://localhost http://192.168.1.1 How do we update proxy config? Adds network path lengthCompute node
- github.com/autopilotpattern Nginx Sales Consul /custom ers /sales /sales/data /customers/data Customers Microservices application w/ discovery catalog
- github.com/autopilotpattern Nginx Sales Consul /custom ers /sales /sales/data /customers/data Customers How do we make existing applications use it? Microservices application w/ discovery catalog
- github.com/autopilotpattern Problem: Automated-Failover
- github.com/autopilotpattern read/write read-only async replication App Primary Replica MySQL with replication
- github.com/autopilotpattern read/write read-only async replication App Primary Replica MySQL with replication How does client find DB? How does replica find primary? How does primary tell replica where to start?
- github.com/autopilotpattern read/write read-only async replication App Primary Replica MySQL with replication How do we update client on failover? How do we promote a replica? How do we orchestrate backups?
- github.com/autopilotpattern Solutions that don’t work: Configuration Management (ex. Chef, Puppet, Ansible)
- github.com/autopilotpattern • No CM server in local development • No service discovery on change
- github.com/autopilotpattern Solutions that don’t work: *aaS (ex. PaaS, DBaaS)
- github.com/autopilotpattern • Vendor lock-in • Poor performance • Very expensive
- github.com/autopilotpattern Solutions that don’t work: Mega-orchestrator (ex. Kubernetes)
- github.com/autopilotpattern Shifts responsibility for app behavior away from app developers
- github.com/autopilotpattern Shifts responsibility for app behavior away from app developers
- github.com/autopilotpattern What if your containers were self-aware and self-operating?
- github.com/autopilotpattern
- github.com/autopilotpattern “[a] pattern where containers autonomously adapt to changes in their environment and coordinate their actions thru a globally shared state” Lukasz Guminski, Container Solutions http://container-solutions.com/containerpilot-on-mantl/
- github.com/autopilotpattern Make applications responsible for: Startup Shutdown Scaling Discovery Recovery Telemetry
- github.com/autopilotpattern Empower application development teams
- github.com/autopilotpattern 3 requirements
- github.com/autopilotpattern #1: Ability to provision containers across multiple compute nodes
- github.com/autopilotpattern VM or physical hardware VM or physical hardware VM or physical hardware Nginx Consul MySQL Primary ES Master Prometheus LogstashCustomers Nginx Consul MySQL Primary ES Master Prometheus Logstash Customers Customers Cluster management & provisioning
- github.com/autopilotpattern Options for cluster management and container placement:
- github.com/autopilotpattern #2: Network virtualization
- github.com/autopilotpattern IP inside the container IP outside the container ==
- github.com/autopilotpattern NAT Sales Customers 192.168.1.101Compute Node 172.17.0.2:80 192.168.1.100:32380 Docker bridge networking Consul
- github.com/autopilotpattern NAT Sales Customers Consul Compute Node 192.168.1.100:32380 Docker bridge networking “I’m listening on 172.17.0.2:80” 172.17.0.2:80
- github.com/autopilotpattern NAT Sales Customers Consul Compute Node 192.168.1.100:32380 Docker bridge networking “Where is Customers?” “172.17.0.2:80” 172.17.0.2:80
- github.com/autopilotpattern NAT Sales Customers Consul Compute Node 192.168.1.100:32380 Docker bridge networking WTF???!!! 172.17.0.2:80 172.17.0.2:80 No route to host!
- github.com/autopilotpattern Sales Customers Compute Node Docker host networking Consul 192.168.1.101 192.168.1.100:80
- github.com/autopilotpattern Sales Customers Compute Node Docker host networking Consul 192.168.1.101 192.168.1.100:80 “I’m listening on 192.168.1.100:80”
- github.com/autopilotpattern Sales Customers Compute Node Docker host networking Consul 192.168.1.101 192.168.1.100:80 “I’m listening on 192.168.1.100:80” Customers 192.168.1.100:80
- github.com/autopilotpattern Sales Customers Compute Node Docker host networking Consul 192.168.1.101 192.168.1.100:80 “I’m listening on 192.168.1.100:80” Customers 192.168.1.100:80 Port conflicts!
- github.com/autopilotpattern Sales Customers Compute Node Overlay networking Consul 192.168.1.101 192.168.1.100:80 “I’m listening on 192.168.1.100:80” Customers 192.168.1.102:80
- github.com/autopilotpattern Sales Customers Compute Node Overlay networking Consul 192.168.1.101 192.168.1.100:80 “I’m listening on 192.168.1.102:80” Customers 192.168.1.102:80
- github.com/autopilotpattern Options for overlay networking:
- github.com/autopilotpattern #3: Infrastructure-backed service discovery
- github.com/autopilotpattern Nginx Sales Consul Customers Microservices app
- github.com/autopilotpattern Nginx Sales Consul Customers Microservices app How do we bootstrap service catalog HA? How do services find service catalog?
- github.com/autopilotpattern Options to bootstrap service catalog: infrastructure-backed DNS * run on each node* Container Name Service (CNS)
- github.com/autopilotpattern #4: We might need some help
- github.com/autopilotpattern App-centric micro-orchestrator that runs inside the container. User-defined behaviors: • Lifecycle hooks (preStop, preStop, postStop) • Health checks w/ heart beats • Watch discovery catalog for changes • Update config on upstream changes • Gather performance metrics
- github.com/autopilotpattern Sales Container Pilot Application Application container http://localhost http://192.168.1.1 Side car?
- github.com/autopilotpattern Sales Container Pilot Application Application container http://localhost http://192.168.1.1 Not a side car!
- github.com/autopilotpattern Sales Container Pilot Application Consul Where is Sales? Application container
- github.com/autopilotpattern Sales Container Pilot Application Consul Where is Sales? 192.168.1.100 192.168.1.101 192.168.1.102 Application container
- github.com/autopilotpattern Sales Container Pilot Application Consul Where is Sales? 192.168.1.100 192.168.1.101 192.168.1.102 Application container onChange event
- github.com/autopilotpattern Sales Container Pilot Application http://192.168.1.100 Consul Where is Sales? 192.168.1.100 192.168.1.101 192.168.1.102 Application container onChange event
- github.com/autopilotpattern Application onChange event User-defined behavior hooks: • preStart • preStop • postStop • health • onChange • sensor • task Application container
- github.com/autopilotpattern read/write read-only async replication App Primary Replica Consul MySQL with replication
- github.com/autopilotpattern ~ $ git clone git@github.com:autopilotpattern/mysql.git ~ $ cd mysql ~/mysql $ tree --dirsfirst . ├── bin │ └── manage.py ├── etc │ ├── containerpilot.json │ └── my.cnf.tmpl ├── tests ├── _env ├── Dockerfile ├── docker-compose.yml ├── local-compose.yml └── setup.sh
- github.com/autopilotpattern ~/mysql/docker-compose.yml mysql: image: autopilotpattern/mysql:latest mem_limit: 4g restart: always # expose for linking, but each container gets a private IP for # internal use as well expose: - 3306 labels: - triton.cns.services=mysql env_file: _env environment: - CONTAINERPILOT=file:///etc/containerpilot.json
- github.com/autopilotpattern ~/mysql/docker-compose.yml mysql: image: autopilotpattern/mysql:latest mem_limit: 4g restart: always # expose for linking, but each container gets a private IP for # internal use as well expose: - 3306 labels: - triton.cns.services=mysql env_file: _env environment: - CONTAINERPILOT=file:///etc/containerpilot.json Infrastructure-backed service discovery requirement
- github.com/autopilotpattern ~/mysql/docker-compose.yml mysql: image: autopilotpattern/mysql:latest mem_limit: 4g restart: always # expose for linking, but each container gets a private IP for # internal use as well expose: - 3306 labels: - triton.cns.services=mysql env_file: _env environment: - CONTAINERPILOT=file:///etc/containerpilot.json Credentials from environment
- github.com/autopilotpattern ~/workshop/mysql $ ./setup.sh /path/to/private/key.pem ~/workshop/mysql $ emacs _env MYSQL_USER=me MYSQL_PASSWORD=password1 MYSQL_REPL_USER=repl MYSQL_REPL_PASSWORD=password2 MYSQL_DATABASE=mydb MANTA_BUCKET=/<username>/stor/triton-mysql MANTA_USER=<username> MANTA_SUBUSER= MANTA_ROLE= MANTA_URL=https://us-east.manta.joyent.com MANTA_KEY_ID=1a:b8:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx MANTA_PRIVATE_KEY=-----BEGIN RSA PRIVATE KEY——#… CONSUL=consul.svc.0f06a3e0-a0da-eb00-a7ae-989d4e44e2ad.us-east-1.cns.joyent.com
- github.com/autopilotpattern ~/mysql $ docker-compose -p my up -d Creating my_consul_1 Creating my_mysql_1 ~/mysql $ docker-compose -p my ps Name Command State Ports ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– my_consul_1 /bin/start -server -bootst... Up 53/tcp, 53/udp, 8300/tcp... my_mysql_1 containerpilot mysqld… Up 0.0.0.0:3600
- github.com/autopilotpattern ~/mysql $ docker-compose -p my scale mysql=2 Creating my_mysql_2 ~/mysql $ docker-compose -p my ps Name Command State Ports ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– my_consul_1 /bin/start -server -bootst... Up 53/tcp, 53/udp, 8300/tcp... my_mysql_1 containerpilot mysqld… Up 0.0.0.0:3600 my_mysql_2 containerpilot mysqld… Up 0.0.0.0:3600
- github.com/autopilotpattern FROM percona:5.6 RUN apt-get update && apt-get install -y python python-dev gcc curl percona-xtrabackup # get Python drivers MySQL, Consul, and Manta RUN curl -Ls -o get-pip.py https://bootstrap.pypa.io/get-pip.py && python get-pip.py && pip install PyMySQL==0.6.7 python-Consul==0.4.7 manta==2.5.0 mock==2.0.0 # get ContainerPilot release (see repo for checksum verification!) RUN curl -Lo /tmp/cp.tar.gz https://github.com/joyent/containerpilot/… tar -xz -f /tmp/cp.tar.gz && mv /containerpilot /usr/local/bin/ # configure ContainerPilot and MySQL COPY etc/* /etc/ COPY bin/* /usr/local/bin/ # override the parent entrypoint ENTRYPOINT [] # use --console to get error logs to stderr CMD [ “containerpilot", “mysqld”, "--console", "--log-bin=mysql-bin", "--log_slave_updates=ON", "--gtid-mode=ON", "--enforce-gtid-consistency=ON" ] ~/mysql/Dockerfile
- github.com/autopilotpattern FROM percona:5.6 RUN apt-get update && apt-get install -y python python-dev gcc curl percona-xtrabackup # get Python drivers MySQL, Consul, and Manta RUN curl -Ls -o get-pip.py https://bootstrap.pypa.io/get-pip.py && python get-pip.py && pip install PyMySQL==0.6.7 python-Consul==0.4.7 manta==2.5.0 mock==2.0.0 # get ContainerPilot release (see repo for checksum verification!) RUN curl -Lo /tmp/cp.tar.gz https://github.com/joyent/containerpilot/… tar -xz -f /tmp/cp.tar.gz && mv /containerpilot /usr/local/bin/ # configure ContainerPilot and MySQL COPY etc/* /etc/ COPY bin/* /usr/local/bin/ # override the parent entrypoint ENTRYPOINT [] # use --console to get error logs to stderr CMD [ “containerpilot", “mysqld”, "--console", "--log-bin=mysql-bin", "--log_slave_updates=ON", "--gtid-mode=ON", "--enforce-gtid-consistency=ON" ] ~/mysql/Dockerfile
- github.com/autopilotpattern FROM percona:5.6 RUN apt-get update && apt-get install -y python python-dev gcc curl percona-xtrabackup # get Python drivers MySQL, Consul, and Manta RUN curl -Ls -o get-pip.py https://bootstrap.pypa.io/get-pip.py && python get-pip.py && pip install PyMySQL==0.6.7 python-Consul==0.4.7 manta==2.5.0 mock==2.0.0 # get ContainerPilot release (see repo for checksum verification!) RUN curl -Lo /tmp/cp.tar.gz https://github.com/joyent/containerpilot/… tar -xz -f /tmp/cp.tar.gz && mv /containerpilot /usr/local/bin/ # configure ContainerPilot and MySQL COPY etc/* /etc/ COPY bin/* /usr/local/bin/ # override the parent entrypoint ENTRYPOINT [] # use --console to get error logs to stderr CMD [ “containerpilot", “mysqld”, "--console", "--log-bin=mysql-bin", "--log_slave_updates=ON", "--gtid-mode=ON", "--enforce-gtid-consistency=ON" ] ~/mysql/Dockerfile
- github.com/autopilotpattern FROM percona:5.6 RUN apt-get update && apt-get install -y python python-dev gcc curl percona-xtrabackup # get Python drivers MySQL, Consul, and Manta RUN curl -Ls -o get-pip.py https://bootstrap.pypa.io/get-pip.py && python get-pip.py && pip install PyMySQL==0.6.7 python-Consul==0.4.7 manta==2.5.0 mock==2.0.0 # get ContainerPilot release (see repo for checksum verification!) RUN curl -Lo /tmp/cp.tar.gz https://github.com/joyent/containerpilot/… tar -xz -f /tmp/cp.tar.gz && mv /containerpilot /usr/local/bin/ # configure ContainerPilot and MySQL COPY etc/* /etc/ COPY bin/* /usr/local/bin/ # override the parent entrypoint ENTRYPOINT [] # use --console to get error logs to stderr CMD [ “containerpilot", “mysqld”, "--console", "--log-bin=mysql-bin", "--log_slave_updates=ON", "--gtid-mode=ON", "--enforce-gtid-consistency=ON" ] ~/mysql/Dockerfile
- github.com/autopilotpattern ~/mysql/etc/containerpilot.json { "consul": "{{ .CONSUL }}:8500", "preStart": "python /usr/local/bin/manage.py", "services": [ { "name": "mysql", "port": 3306, "health": "python /usr/local/bin/manage.py health", "poll": 5, "ttl": 25 } ], "backends": [ { "name": "mysql-primary", "poll": 10, "onChange": "python /usr/local/bin/manage.py on_change" } ] }
- github.com/autopilotpattern ~/mysql/etc/containerpilot.json { "consul": "{{ .CONSUL }}:8500", "preStart": "python /usr/local/bin/manage.py", "services": [ { "name": "mysql", "port": 3306, "health": "python /usr/local/bin/manage.py health", "poll": 5, "ttl": 25 } ], "backends": [ { "name": "mysql-primary", "poll": 10, "onChange": "python /usr/local/bin/manage.py on_change" } ] } Environment variable interpolation
- github.com/autopilotpattern ~/mysql/etc/containerpilot.json { "consul": "{{ .CONSUL }}:8500", "preStart": "python /usr/local/bin/manage.py", "services": [ { "name": "mysql", "port": 3306, "health": "python /usr/local/bin/manage.py health", "poll": 5, "ttl": 25 } ], "backends": [ { "name": "mysql-primary", "poll": 10, "onChange": "python /usr/local/bin/manage.py on_change" } ] } Service definition
- github.com/autopilotpattern ~/mysql/etc/containerpilot.json { "consul": "{{ .CONSUL }}:8500", "preStart": "python /usr/local/bin/manage.py", "services": [ { "name": "mysql", "port": 3306, "health": "python /usr/local/bin/manage.py health", "poll": 5, "ttl": 25 } ], "backends": [ { "name": "mysql-primary", "poll": 10, "onChange": "python /usr/local/bin/manage.py on_change" } ] } Backend definition
- github.com/autopilotpattern ~/mysql/etc/containerpilot.json { "consul": "{{ .CONSUL }}:8500", "preStart": "python /usr/local/bin/manage.py", "services": [ { "name": "mysql", "port": 3306, "health": "python /usr/local/bin/manage.py health", "poll": 5, "ttl": 25 } ], "backends": [ { "name": "mysql-primary", "poll": 10, "onChange": "python /usr/local/bin/manage.py on_change" } ] } Huh? This isn’t in our docker-compose.yml
- github.com/autopilotpattern ~/mysql/etc/containerpilot.json { "consul": "{{ .CONSUL }}:8500", "preStart": "python /usr/local/bin/manage.py", "services": [ { "name": "mysql", "port": 3306, "health": "python /usr/local/bin/manage.py health", "poll": 5, "ttl": 25 } ], "backends": [ { "name": "mysql-primary", "poll": 10, "onChange": "python /usr/local/bin/manage.py on_change" } ] } Logic lives in manage.py
- github.com/autopilotpattern Container Pilot Consul Lifecycle: preStart MySQL container
- github.com/autopilotpattern Container Pilot Consul Lifecycle: preStart MySQL container PID1 Separate container
- github.com/autopilotpattern Container Pilot Consul Lifecycle: preStart Manta object store Store snapshots MySQL container
- github.com/autopilotpattern Container Pilot Consul preStart Lifecycle: preStart Manta object store MySQL container
- github.com/autopilotpattern Container Pilot Consul preStart Lifecycle: preStart Manta object store MySQL container Note: no main application running yet! If exit code of preStart != 0, ContainerPilot exits
- github.com/autopilotpattern Container Pilot Consul preStart Lifecycle: preStart Manta object store MySQL container “Has a snapshot been written to Manta?”
- github.com/autopilotpattern Container Pilot Consul MySQL container preStart Lifecycle: preStart Manta object store “Has a snapshot been written to Manta?” “Nope!”
- github.com/autopilotpattern Container Pilot Consul MySQL container preStart Lifecycle: preStart Manta object store “Has a snapshot been written to Manta?” “Nope!” initialize DB
- github.com/autopilotpattern ~/mysql/bin/manage.py def pre_start(): """ MySQL must be running in order to execute most of our setup behavior so we're just going to make sure the directory structures are in place and then let the first health check handler take it from there """ if not os.path.isdir(os.path.join(config.datadir, 'mysql')): last_backup = has_snapshot() if last_backup: get_snapshot(last_backup) restore_from_snapshot(last_backup) else: if not initialize_db(): log.info('Skipping database setup.') sys.exit(0)
- github.com/autopilotpattern ~/mysql/bin/manage.py def pre_start(): """ MySQL must be running in order to execute most of our setup behavior so we're just going to make sure the directory structures are in place and then let the first health check handler take it from there """ if not os.path.isdir(os.path.join(config.datadir, 'mysql')): last_backup = has_snapshot() if last_backup: get_snapshot(last_backup) restore_from_snapshot(last_backup) else: if not initialize_db(): log.info('Skipping database setup.') sys.exit(0) Check w/ Consul for snapshot
- github.com/autopilotpattern ~/mysql/bin/manage.py def pre_start(): """ MySQL must be running in order to execute most of our setup behavior so we're just going to make sure the directory structures are in place and then let the first health check handler take it from there """ if not os.path.isdir(os.path.join(config.datadir, 'mysql')): last_backup = has_snapshot() if last_backup: get_snapshot(last_backup) restore_from_snapshot(last_backup) else: if not initialize_db(): log.info('Skipping database setup.') sys.exit(0) calls /usr/bin/mysql_install_db
- github.com/autopilotpattern ~/mysql/bin/manage.py def pre_start(): """ MySQL must be running in order to execute most of our setup behavior so we're just going to make sure the directory structures are in place and then let the first health check handler take it from there """ if not os.path.isdir(os.path.join(config.datadir, 'mysql')): last_backup = has_snapshot() if last_backup: get_snapshot(last_backup) restore_from_snapshot(last_backup) else: if not initialize_db(): log.info('Skipping database setup.') sys.exit(0)
- github.com/autopilotpattern Container Pilot Consul Lifecycle: run Manta object store MySQL container
- github.com/autopilotpattern Container Pilot mysqld Consul Lifecycle: run • Attach to stdout/ stderr • Return exit code of application to Docker runtime MySQL container Manta object store
- github.com/autopilotpattern Container Pilot Consul Lifecycle: health mysqld health Manta object store MySQL container
- github.com/autopilotpattern Manta object store Container Pilot Consul Lifecycle: health User-defined health check inside the container. Runs every poll seconds. mysqld MySQL container health
- github.com/autopilotpattern mysqld MySQL container Container Pilot Consul health Lifecycle: health Manta object store first time? finish initialization
- github.com/autopilotpattern ~/mysql/bin/manage.pydef health(): """ Run a simple health check. Also acts as a check for whether the ContainerPilot configuration needs to be reloaded (if it's been changed externally), or if we need to make a backup because the backup TTL has expired. """ node = MySQLNode() cp = ContainerPilot(node) if cp.update(): cp.reload() return # Because we need MySQL up to finish initialization, we need to check # for each pass thru the health check that we've done so. The happy # path is to check a lock file against the node state (which has been # set above) and immediately return when we discover the lock exists. # Otherwise, we bootstrap the instance. was_ready = assert_initialized_for_state(node) ctx = dict(user=config.repl_user, password=config.repl_password, timeout=cp.config['services'][0]['ttl']) node.conn = wait_for_connection(**ctx) # Update our lock on being the primary/standby. if node.is_primary() or node.is_standby(): update_session_ttl() # Create a snapshot and send it to the object store if all((node.is_snapshot_node(), (not is_backup_running()), (is_binlog_stale(node.conn) or is_time_for_snapshot()))): write_snapshot(node.conn) mysql_query(node.conn, 'SELECT 1', ())
- github.com/autopilotpattern ~/mysql/bin/manage.pydef run_as_primary(node): """ The overall workflow here is ported and reworked from the Oracle-provided Docker image: https://github.com/mysql/mysql-docker/blob/mysql-server/5.7/docker-entrypoint.sh """ node.state = PRIMARY mark_as_primary(node) node.conn = wait_for_connection() if node.conn: # if we can make a connection w/o a password then this is the # first pass set_timezone_info() setup_root_user(node.conn) create_db(node.conn) create_default_user(node.conn) create_repl_user(node.conn) run_external_scripts('/etc/initdb.d') expire_root_password(node.conn) else: ctx = dict(user=config.repl_user, password=config.repl_password, database=config.mysql_db) node.conn = wait_for_connection(**ctx) stop_replication(node.conn) # in case this is a newly-promoted primary if USE_STANDBY: # if we're using a standby instance then we need to first # snapshot the primary so that we can bootstrap the standby. write_snapshot(node.conn) Set up DB, user, replication user, and expire password, etc.
- github.com/autopilotpattern ~/mysql/bin/manage.py def run_as_replica(node): try: ctx = dict(user=config.repl_user, password=config.repl_password, database=config.mysql_db) node.conn = wait_for_connection(**ctx) set_primary_for_replica(node.conn) except Exception as ex: log.exception(ex) def set_primary_for_replica(conn): """ Set up GTID-based replication to the primary; once this is set the replica will automatically try to catch up with the primary's last transactions. """ primary = get_primary_host() sql = ('CHANGE MASTER TO ' 'MASTER_HOST = %s, ' 'MASTER_USER = %s, ' 'MASTER_PASSWORD = %s, ' 'MASTER_PORT = 3306, ' 'MASTER_CONNECT_RETRY = 60, ' 'MASTER_AUTO_POSITION = 1, ' 'MASTER_SSL = 0; ' 'START SLAVE;') mysql_exec(conn, sql, (primary, config.repl_user, config.repl_password,))
- github.com/autopilotpattern ~/mysql/bin/manage.py def run_as_replica(node): try: ctx = dict(user=config.repl_user, password=config.repl_password, database=config.mysql_db) node.conn = wait_for_connection(**ctx) set_primary_for_replica(node.conn) except Exception as ex: log.exception(ex) def set_primary_for_replica(conn): """ Set up GTID-based replication to the primary; once this is set the replica will automatically try to catch up with the primary's last transactions. """ primary = get_primary_host() sql = ('CHANGE MASTER TO ' 'MASTER_HOST = %s, ' 'MASTER_USER = %s, ' 'MASTER_PASSWORD = %s, ' 'MASTER_PORT = 3306, ' 'MASTER_CONNECT_RETRY = 60, ' 'MASTER_AUTO_POSITION = 1, ' 'MASTER_SSL = 0; ' 'START SLAVE;') mysql_exec(conn, sql, (primary, config.repl_user, config.repl_password,)) gets from Consul
- github.com/autopilotpattern ~/mysql/bin/manage.py def run_as_replica(node): try: ctx = dict(user=config.repl_user, password=config.repl_password, database=config.mysql_db) node.conn = wait_for_connection(**ctx) set_primary_for_replica(node.conn) except Exception as ex: log.exception(ex) def set_primary_for_replica(conn): """ Set up GTID-based replication to the primary; once this is set the replica will automatically try to catch up with the primary's last transactions. """ primary = get_primary_host() sql = ('CHANGE MASTER TO ' 'MASTER_HOST = %s, ' 'MASTER_USER = %s, ' 'MASTER_PASSWORD = %s, ' 'MASTER_PORT = 3306, ' 'MASTER_CONNECT_RETRY = 60, ' 'MASTER_AUTO_POSITION = 1, ' 'MASTER_SSL = 0; ' 'START SLAVE;') mysql_exec(conn, sql, (primary, config.repl_user, config.repl_password,)) Remember our preStart downloaded the snapshot
- github.com/autopilotpattern Wait a sec. How do we know which instance is primary!?
- github.com/autopilotpattern Container Pilot Consul Lifecycle: health Exit! mysqld MySQL container health
- github.com/autopilotpattern Container Pilot mysqld Consul health Lifecycle: health Exit code is 0? “I am mysql-12345. I am available at 192.168.100.2:4000. I am healthy for the next 10 seconds.” MySQL container
- github.com/autopilotpattern Container Pilot mysqld Consul MySQL container health Lifecycle: health If exit code != 0, do nothing (TTL expires)
- github.com/autopilotpattern Ask Consul for Primary
- github.com/autopilotpattern I’m the primary! Ask Consul for Primary
- github.com/autopilotpattern I’m the primary! Ask Consul for Primary Update lock TTL w/ each health check
- github.com/autopilotpattern I’m the primary! Someone else is the primary! I’m a replica! Ask Consul for Primary
- github.com/autopilotpattern I’m the primary! Someone else is the primary! I’m a replica! Ask Consul for Primary Syncs up using snapshot and GTID
- github.com/autopilotpattern No Primary? I’m the Primary! I’m the primary! Someone else is the primary! I’m a replica! Ask Consul for Primary
- github.com/autopilotpattern No Primary? I’m the Primary? I’m the primary! Someone else is the primary! I’m a replica! Ask Consul for Primary Need to assert only 1 primary
- github.com/autopilotpattern No Primary? I’m the Primary? I’m the primary! Failed! Go back to start I’m the primary! Someone else is the primary! I’m a replica! Set lock in Consul w/ TTL Ask Consul for Primary
- github.com/autopilotpattern No Primary? I’m the Primary? I’m the primary! Failed! Go back to start I’m the primary! Someone else is the primary! I’m a replica! Set lock in Consul w/ TTL Ask Consul for Primary Update lock TTL w/ each health check. Rewrite ContainerPilot config and SIGHUP
- github.com/autopilotpattern ~/mysql/etc/containerpilot.json { "consul": "{{ .CONSUL }}:8500", "preStart": "python /usr/local/bin/manage.py", "services": [ { "name": "mysql", "port": 3306, "health": "python /usr/local/bin/manage.py health", "poll": 5, "ttl": 25 } ], "backends": [ { "name": "mysql-primary", "poll": 10, "onChange": "python /usr/local/bin/manage.py on_change" } ] }
- github.com/autopilotpattern ~/mysql/etc/containerpilot.json { "consul": "{{ .CONSUL }}:8500", "preStart": "python /usr/local/bin/manage.py", "services": [ { "name": “mysql-primary", "port": 3306, "health": "python /usr/local/bin/manage.py health", "poll": 5, "ttl": 25 } ], "backends": [ { "name": "mysql-primary", "poll": 10, "onChange": "python /usr/local/bin/manage.py on_change" } ] } Rewrite & reload config
- github.com/autopilotpattern ~/mysql/bin/manage.py def health(): """ Run a simple health check. Also acts as a check for whether the ContainerPilot configuration needs to be reloaded (if it's been changed externally), or if we need to make a backup because the backup TTL has expired. """ node = MySQLNode() cp = ContainerPilot(node) if cp.update(): cp.reload() return was_ready = assert_initialized_for_state(node) # cp.reload() will exit early so no need to setup # connection until this point ctx = dict(user=config.repl_user, password=config.repl_password, timeout=cp.config['services'][0]['ttl']) node.conn = wait_for_connection(**ctx) # Update our lock on being the primary/standby. # If this lock is allowed to expire and the health check for the primary # fails, the `onChange` handlers for the replicas will try to self-elect # as primary by obtaining the lock. # If this node can update the lock but the DB fails its health check, # then the operator will need to manually intervene if they want to # force a failover. This architecture is a result of Consul not # permitting us to acquire a new lock on a health-checked session if the # health check is *currently* failing, but has the happy side-effect of # reducing the risk of flapping on a transient health check failure. if node.is_primary() or node.is_standby(): update_session_ttl() # Create a snapshot and send it to the object store. if all((node.is_snapshot_node(), (not is_backup_running()), (is_binlog_stale(node.conn) or is_time_for_snapshot()))): write_snapshot(node.conn) mysql_query(node.conn, 'SELECT 1', ())
- github.com/autopilotpattern Wait a sec. How do we fail-over?
- github.com/autopilotpattern ~/mysql/etc/containerpilot.json { "consul": "{{ .CONSUL }}:8500", "preStart": "python /usr/local/bin/manage.py", "services": [ { "name": "mysql", "port": 3306, "health": "python /usr/local/bin/manage.py health", "poll": 5, "ttl": 25 } ], "backends": [ { "name": "mysql-primary", "poll": 10, "onChange": "python /usr/local/bin/manage.py on_change" } ] }
- github.com/autopilotpattern Container Pilot mysqld Consul Where is mysql-primary? 192.168.1.100 MySQL container Lifecycle: onChange
- github.com/autopilotpattern Container Pilot mysqld Consul Where is mysql-primary? 192.168.1.100 MySQL container Lifecycle: onChange Check Consul for services listed in backends. Runs every poll seconds.
- github.com/autopilotpattern replica primary Healthy! Healthy! Failed! Ask Consul for Primary no change Ask Consul for Primary no change Ask Consul for Primary fire onChange handler
- github.com/autopilotpattern ~/mysql/bin/manage.pydef on_change(): node = MySQLNode() ctx = dict(user=config.repl_user, password=config.repl_password, timeout=cp.config['services'][0]['ttl']) node.conn = wait_for_connection(**ctx) # need to stop replication whether we're the new primary or not stop_replication(node.conn) while True: try: # if there is no primary node, we'll try to obtain the lock. # if we get the lock we'll reload as the new primary, otherwise # someone else got the lock but we don't know who yet so loop primary = get_primary_node() if not primary: session_id = get_session(no_cache=True) if mark_with_session(PRIMARY_KEY, node.hostname, session_id): node.state = PRIMARY if cp.update(): cp.reload() return else: # we lost the race to lock the session for ourselves time.sleep(1) continue # we know who the primary is but not whether they're healthy. # if it's not healthy, we'll throw an exception and start over. ip = get_primary_host(primary=primary) if ip == node.ip: if cp.update(): cp.reload() return set_primary_for_replica(node.conn) return except Exception as ex: # This exception gets thrown if the session lock for `mysql-primary` # key has not expired yet (but there's no healthy primary either), # or if the replica's target primary isn't ready yet. log.debug(ex) time.sleep(1) # avoid hammering Consul continue
- github.com/autopilotpattern replica primary Healthy! Healthy! Failed! no change no change Ask Consul for Primary Ask Consul for Primary Ask Consul for Primary fire onChange handler
- github.com/autopilotpattern replica primary Healthy! Healthy! Failed! no change no change Ask Consul for Primary Ask Consul for Primary Ask Consul for Primary Ask Consul for Primary Ok, I’m primary Set lock in Consul Success! primary Healthy! fire onChange handler
- Applications on Autopilot Tim Gross @0x74696d (“tim”) github.com/autopilotpattern
Advertisement