This presentation covers adventures in research, design, implementation and migration of a private Docker registry, and a logging pipeline.

1. ╔══════════════════════════════════════════╗
║ CHOOSE YOUR OWN AWS ADVENTURE ║
║ ║
║ Noah Zoschke ║
║ noah@convox.com ║
║ @nzoschke ║
║ ║
║ 1/28/2016 ║
╚══════════════════════════════════════════╝
CONVOX

2. CONVOX
DEVOPS TOOLS AND SERVICES
Eliminate complex and boring parts of
architecting a system on AWS
Automate the hard parts of operating software
on AWS
Integrate the best pieces from AWS and Docker
ecosystem
Open source everything to share code and
expertise
Support and maintain for app and
infrastructure lifecycle

3. CONVOX
OPEN SOURCE PAAS ⟷ IAAS
Racks ⟷ ASG, CF, Dynamo, EC2, ECS, IAM, VPC
Apps ⟷ CF, ECS, ELB
Scale ⟷ ASG, CF, ECS
Environments ⟷ KMS, S3
Builds ⟷ S3, ECR
Logs ⟷ CloudWatch Logs, Kinesis, Lambda
Metrics ⟷ CloudWatch Metrics, Elasticsearch
Notifications ⟷ SNS

4. CHOOSE
YOUR OWN
ADVENTURE

5. BUILD AND PACKAGING
DECISION TREE
Build AMIs
What OS? ubuntu / redhat / systems / dockerd
Build packages
What format? RPMs / debs / tgzs
Push code
What tool? Ansible / Chef / CodeDeploy
Build Docker images
What registry? DockerHub / Quay / v1 / v2 / ECR

6. BUILD AND PACKAGING
ADVENTURE: DOCKER IMAGES / PRIVATE REGISTRY
┌─────────────────────────────────────────┐
│ convox API │
│ POST /apps/httpd/build │
┌──────────────────┐ │ │
│ │ │ ┌─────────────────────────────┐ │ ┌──────────────────┐
│ │ │ │ pull httpd-web:BLASTBUILD │◀───┼───────┤ │ ┌────────────────┐
│ │ │ └─────────────────────────────┘ │ │ │ │ │
│ │ │ ┌─────────────────────────────┐ │ │ │◀───────────┤ ECS Instance │
│ ├─────┼─────▶│ pull rabbitmq │ │ │ │ │ │
│ │ │ └─────────────────────────────┘ │ │ │ └────────────────┘
│ │ │ ┌─────────────────────────────┐ │ │ │
│ │ │ │tag httpd-rabbit:BQUWNCMIYZG │ │ │ │
│ │ │ └─────────────────────────────┘ │ │ │ ┌────────────────┐
│ DockerHub │ │ ┌─────────────────────────────┐ │ │ │ │ │
│ or │ │ │ push $REGISTRY_HOST ├────┼──────▶│ │◀───────────┤ ECS Instance │
│Private Registries│ │ └─────────────────────────────┘ │ │ Convox Registry │ │ │
│ │ │ ┌─────────────────────────────┐ │ │ │ └────────────────┘
│ ├─────┼─────▶│ pull debian │ │ │ │
│ │ │ └─────────────────────────────┘ │ │ │
│ │ │ ┌─────────────────────────────┐ │ │ │ ┌────────────────┐
│ │ │ │ build Dockerfile │ │ │ │ │ │
│ │ │ └─────────────────────────────┘ │ │ │◀───────────┤ ECS Instance │
│ │ │ ┌─────────────────────────────┐ │ │ │ │ │
│ │ │ │ tag httpd-web:BQUWNCMIYZG │ │ │ │ └────────────────┘
│ │ │ └─────────────────────────────┘ │ │ │
│ │ │ ┌─────────────────────────────┐ │ │ │
└──────────────────┘ │ │ push $REGISTRY_HOST ├────┼──────▶└───────┬──────────┘
│ └─────────────────────────────┘ │ │ ▲
│ ┌─────────────────────────────┐ │ │ │
│ │tag httpd-worker:BQUWNCMIYZG │ │ ▼ │
│ └─────────────────────────────┘ │ ┌──────────┴───────┐
│ ┌─────────────────────────────┐ │ │ │
│ │ ... │ │ │ S3 │
│ └─────────────────────────────┘ │ │ │
│ │ └──────────────────┘
│ │
└─────────────────────────────────────────┘

7. BUILD AND PACKAGING
R&D
Apr: AMIs are way too slow for CD workflow
May: DockerHub requirement doesn’t feel right
May: Running a Docker v1 registry with S3 store
works!
Jun: v1 registry is flaky and impossible to debug
and fix
Jun: v2 registry introduces lots of new stuff and
incompatibilities
Oct: AWS announces ECR at re:Invent. Wait…
Dec: ECR GA

8. ECR MIGRATION
ADVENTURE: CUSTOM RESOURCES
"RegistryRepository": {
"Type": "Custom::ECRRepository",
"Version": "1.0",
"Properties": {
"ServiceToken": { "Fn::GetAtt": [ "CustomTopic", "Arn" ] },
"Name": { "Ref": "AWS::StackName" }
}
},
"CustomTopic": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Code": {
"S3Bucket": { "Fn::Join": [ "-", [ "convox", { "Ref": "AWS::Region" } ] ] },
"S3Key": { "Fn::Join": [ "", [ "release/", { "Ref": "Version" }, "/formation.zip" ]
},
"Handler": "lambda.external",
"MemorySize": "128",
"Role": { "Fn::GetAtt": [ "CustomTopicRole", "Arn" ] },
"Runtime": "nodejs",
"Timeout": "30"
}
},

9. ECR MIGRATION
ADVENTURE: PROGRAMMABLE INFRASTRUCTURE
$ convox update
Next deploy creates RegistryRepository
Resource
Next deploy sees RegistryRepository
Output pushes to it
Keep v1 registry / data around for
rollbacks

10. ECR MIGRATION
ADVENTURE: GLOBAL INFRASTRUCTURE
+ "RegionHasRegistry": {
+ "Fn::Equals": [
+ {
+ "Ref": "AWS::Region"
+ },
+ "us-east-1"
+ ]
+ },

11. ECR MIGRATION
ADVENTURE: SUPPORT
watching for new regions
debugging ECR 500s on pushes
adding retry logic
adding error rate metrics
watching metrics around v1 usage

12. APP LOGS
DECISION TREE
Use Amazon
What service? Kinesis / CloudWatch
Use 3rd Party
What vendor? Papertrail / Loggly / Splunk
Build your own pipeline
What system? syslog / logspout / logstash
App interface
stdout/stderr, log files, log drivers

13. APP LOGS
ADVENTURE: AGENT, DOCKER APIS, KINESIS,
LAMBDA
┌──────────────────────────────────────────────────────────┐ ┌──────────────────┐
│ EC2 Instance in ECS Cluster │ │ app1 Kinesis │
│ │ │ ┌────────┐ │ ┌───────────────────────────────────────────┐
│ ┌──────────────┐ ┌──────────────────────────────────┐ │ ┌─┼───▶│shard 1 │ │──┐ │ Lambda w/ EventSourceMapping │
│ │ │ │ │ │ │ │ └────────┘ │ │ │ ┌──────────────────────────────────────┐ │
│ │ │ │ │ │ │ └──────────────────┘ │ │ │function(event, context) { │ │
│ │ app1 │ │ app2 │ │ │ │ │ │ event.records.forEach(function(r) { │ │
│ │ web.1 │ │ worker.1 │ │ │ │ │ │ winston.info(r.kinesis.data) │ │
│ │ │ │ │ │ │ └─┼▶│ }) │──┼────────▶┌───────────────┐
│ │ │ │ │ │ │ │ │ context.done() │ │ │ │
│ └──────────────┘ └──────────────────────────────────┘ │ │ ┌──────────────────┐ │ │} │ │ │ │
│ │ │ │ │ │ app2 Kinesis │ │ │ │ │ │ │
│ │ ┌─────────────────────┘ │ │ │ ┌────────┐ │ │ └──────────────────────────────────────┘ │ │ Syslog Server │
│ ▼ ▼ │ │ │ ┌─▶│shard 1 │ │ │ ┌────────────────────────────────┐ │ │ │
│ ┌────────────┐ ┌────────────┐─────────────┼───┘ │ │ └────────┘ │ │ │function(event, context) { ... }│──┼────────▶│ │
│ │ dockerd │◀─────────────│convox/agent│─────────────┼─────┼─┘ ┌────────┐ │ │ └────────────────────────────────┘ │ │ │
│ └────────────┘ └────────────┘─────────────┼─────┼───▶│shard 2 │ │ │ ┌────────────────────────────────┐ │ │ │
│ ▲ ┌────────────────────────────────────┐ │ │ └────────┘ │────┼─▶│function(event, context) { ... }│───────┼────────▶└───────────────┘
│ │ │GET docker /events (create) │ │ │ . │ │ └────────────────────────────────┘ │
│ ▼ │ GET ENV "Kinesis", "Process"│ │ │ . │ │ │
│ ┌────────────┐ │ GET Docker /logs?follow=1 │ │ │ . │ └───────────────────────────────────────────┘
│ │ ecs-agent │ │ PUT Kinesis /records │ │ │ ┌────────┐ │
│ └────────────┘ └────────────────────────────────────┘ │ │ │shard N │ │
│ │ │ └────────┘ │
└──────────────────────────────────────────────────────────┘ └──────────────────┘

14. APP LOGS
R&D
Apr: Kinesis is rad
Oct: Kinesis + Lambda is really rad.
Forward logs to Papertrail.
Nov: What about Kinesis to other
systems?
Dec: What about CloudWatch Logs?
Jan: docker awslogs driver is great
code

15. APP LOGS
ADVENTURE: DOCKER/DOCKER INTEGRATION

16. APP LOGS
ADVENTURE: INFRA AS JAVASCRIPT IN JAVASCRIPT

17. CONVOX
PHILOSOPHY
Integrate, don’t invent
All-in on Docker Container and Image workflow
All-in on pure AWS solutions (where available)
Stub in missing pieces where not available
Build and exercise automated infrastructure
migration paths
Remove custom pieces when no longer needed

18. noah@convox.com
@nzoschke
_ _ _ _
| |_| |__ __ _ _ __ | | _____| |
| __| '_ / _` | '_ | |/ / __| |
| |_| | | | (_| | | | | <__ _|
__|_| |_|__,_|_| |_|_|____(_)
I’m hiring...