Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Convox: Open Source Tooling for ECS

This presentation covers the benefits of deploying to ECS, as we explore techniques and open source tools to help manage multiple ECS clusters.

First we discuss why developing as containers and deploying to ECS offers a more cost-effective, faster, and safer workflow from dev to production than VMs on EC2.

Next we present techniques Convox uses to manage countless clusters and apps. We will see a CloudFormation template that reliably provisions, updates, and horizontally and vertically scales ECS clusters, and another template that configures application containers inside the cluster.

Finally we survey features that are "missing" on ECS and demonstrate simple tools and techniques Convox has developed to to fill in these gaps. For example:

* Running a private build and registry service inside the ECS cluster itself
* Splitting up an app's processes (i.e. web and worker) into their own task definitions for independent horizontal scaling
* Running an agent on every cluster instance for forwarding all container logs to Kinesis, Lambda and beyond
* Running an public API on a cluster that facilitates a `docker exec` debugging session inside application containers

  • Be the first to comment

Convox: Open Source Tooling for ECS

  1. 1. ╔══════════════════════════════════════════╗ ║ OPEN SOURCE TOOLING FOR ECS ║ ║ ║ ║ Noah Zoschke ║ ║ noah@convox.com ║ ║ @nzoschke ║ ║ ║ ║ 12/8/2015 ║ ╚══════════════════════════════════════════╝ CONVOX
  2. 2. EC2 CONTAINER SERVICE
 (BATTERIES NOT INCLUDED) API • Clusters • TaskDefinitions • Tasks and Services Bring Your Own • Instances • ecs-agent • Load Balancers • Logging • Builds / Images • Tools...
  3. 3. CONVOX OPEN SOURCE PAAS ⟷ IAAS Racks ⟷ ASG, CF, Dynamo, EC2, ECS, IAM, VPC Apps ⟷ CF, ECS, ELB, Kinesis Scale ⟷ ASG, CF, ECS Environments ⟷ KMS, S3 Builds ⟷ S3 Logs ⟷ Kinesis, Lambda Metrics ⟷ CloudWatch, Kinesis Notifications ⟷ SNS
  4. 4. CONVOX REST API $ convox api get /apps [ { "balancer": "myapp-1749418666.us-east-1.elb.amazonaws.com", "name": "myapp", "release": "REXIQURVKXE", "status": "running" }, { "balancer": "myapp2-689551992.us-east-1.elb.amazonaws.com", "name": "myapp2", "release": "RNEFHNIUSKF", "status": "running" }, { "balancer": "myapp3-435098803.us-east-1.elb.amazonaws.com", "name": "myapp3", "release": "RORJKBNVKDD", "status": "running" } ]
  5. 5. CONVOX REST API $ convox api get /apps/myapp/processes [ { "app": "myapp", "command": "bin/web", "cpu": 0.0329, "host": "10.0.3.135", "id": "13254981d20", "image": "registry.internal:5000/myapp-web:BHLRYHSMXNM", "memory": 0.2068, "name": "web", "ports": [ "80:3000", "443:3001" ], "release": "REXIQURVKXE", "started": "2015-01-01T00:00:00Z" } ]
  6. 6. BEHIND THE SCENES aws && docker || convox
  7. 7. RACK INSTALL CLOUDFORMATION PARAMETERIZED INFRASTRUCTURE → Ami ami-c5fa5aae → InstanceCount 3 → InstanceType t2.small → Password PuDpyqGTmxBN8ziGJ9UiMfrfGZfHDG → Tenancy default → Version 20151204013151 → VolumeSize 30 → VPCCIDR 10.0.0.0/16 ↑ Balancer convox AWS::ElasticLoadBalancing::LoadBalancer ↑ Cluster convox-Cluster-1JI343QBLSMYJ AWS::ECS::Cluster ↑ DynamoBuilds convox-builds AWS::DynamoDB::Table ↑ DynamoReleases convox-releases AWS::DynamoDB::Table ↑ EncryptionKey arn:aws:kms:...:key/d40c0153... Custom::KMSKey ↑ IamRole convox-IamRole-M1YZSNXNS1F7 AWS::IAM::Role ↑ Instances convox-Instances-PCWRQ6OWDWTT AWS::AutoScaling::AutoScalingGroup ↑ Kinesis convox-Kinesis-C09RDWFR8NOE AWS::Kinesis::Stream ↑ NotificationTopic arn:aws:sns:...:convox-notifications AWS::SNS::Topic ↑ Settings convox-settings-13c91daqrj90z AWS::S3::Bucket ↑ Vpc vpc-b27ff8d6 AWS::EC2::VPC ← Dashboard convox-820546104.us-east-1.elb.amazonaws.com ← Kinesis convox-Kinesis-C09RDWFR8NOE
  8. 8. CLOUDFORMATION CUSTOM RESOURCES ┌─────────────────────────────────────┐ │POST arn:aws:lambda:... │ │{ │ │ ResourceProperties: { │ │ Description: "Master Encryption",│ ┌─────────────────────────────────────┐ ┌───────────────────────────┐ │ KeyUsage: "ENCRYPT_DECRYPT" │ │aws kms create-key │ │200: OK │ │ } │ │ --description "Master Encryption" │ │400: LimitExceededException│ │} │ │ --key-usage ENCRYPT_DECRYPT │ │500: KMSInternalException │ └─────────────────────────────────────┘ └─────────────────────────────────────┘ └───────────────────────────┘ ┌────────────────┐ ┌──────────────┐──────────────────────▶┌───────────┐ │ CloudFormation │──────────────────────▶│ Lambda │ │AWS KMS API│ └────────────────┘ CREATE_IN_PROGRESS └──────────────┘◀──────────────────────└───────────┘ ▲ │ │ │ │ CREATE_COMPLETE │ │ OR ▼ │ CREATE_FAILED ┌─────────────┐ └────────────────────────────────│ S3 │ └─────────────┘
  9. 9. APP CREATE CLOUDFORMATION PARAMETERIZED CONTAINERS → Cluster convox-Cluster-1JI343QBLSMYJ → Cpu 200 → Environment https://httpd-settings-1e3ej4u01z4bv.s3.amazonaws.com/releases/RSAQCOYHGPV/env → Key arn:aws:kms:us-east-1:901416387788:key/d40c0153-4a57-4d50-9ca0-99a974daca11 → Release RSAQCOYHGPV → VPC vpc-b27ff8d6 → WebCommand → WebDesiredCount 1 → WebImage convox-820546104.us-east-1.elb.amazonaws.com:5000/httpd-web:BQIWNCMIYZG → WebMemory 256 → WebPort80Balancer 80 → WebPort80Certificate → WebPort80Host 42563 → WebPort80Secure No ↑ Balancer httpd AWS::ElasticLoadBalancing::LoadBalancer ↑ Kinesis httpd-Kinesis-FO32SUUFLX24 AWS::Kinesis::Stream ↑ LogsAccess AKIAIFI65IDSEURPK62Q AWS::IAM::AccessKey ↑ LogsUser httpd-LogsUser-96BAE2EL9TNL AWS::IAM::User ↑ ServiceRole httpd-ServiceRole-19LN8R18BIVRW AWS::IAM::Role ↑ Settings httpd-settings-1e3ej4u01z4bv AWS::S3::Bucket ↑ WebECSService arn:aws:ecs:...:service/httpd-web-SATOEEBOQNF Custom::ECSService ↑ WebECSTaskDefinition arn:aws:ecs:...:task-definition/httpd-web:6 Custom::ECSTaskDefinition ← BalancerWebHost httpd-908645489.us-east-1.elb.amazonaws.com ← Kinesis httpd-Kinesis-FO32SUUFLX24 ← Settings httpd-settings-1e3ej4u01z4bv ← WebPort80Balancer 80
  10. 10. APP DEPLOY DOCKER APPLICATION MANIFEST ⟷ AWS ┌──────────────────────────────────────────────────────────────────────────────────────────────────┐ │web: Task Definition httpd-web:6 │ │ command: bin/web Service httpd-web-SATOEEBOQNF │ │ build: . Docker Image httpd-web:BQIWNCMIYZG │ │ ports: │ │ - 80:80 ELB 80 : 52452 : 80 │ │ - 443:80 ELB (SSL) 443 : 52452 : 80 │ │ │ │worker: Task Definition httpd-worker:6 │ │ command: bin/worker Service httpd-worker-SHAOPEQONEF │ │ build: . Docker Image httpd-worker:BQIWNCMIYZG │ │ links: │ │ - redis REDIS_URL=rer45wxl0uj8jn6.1qae5u.ng.0001.usw2.cache.amazonaws.com:6379│ │ - rabbit RABBIT_URL=httpd-1222973998.us-west-2.elb.amazonaws.com:5672 │ │ │ │rabbit: Task Definition httpd-rabbit:6 │ │ command: rabbitmq-server Service httpd-rabbit-SPNFHGMWNUU │ │ image: rabbitmq Docker Image httpd-rabbit:BQUWNCMIYZG │ │ ports: │ │ - 5672 ELB (Internal) 5672 : 24324 : 5672 │ │ │ │redis: │ │ image: convox/redis AWS::ElastiCache::CacheCluster │ └──────────────────────────────────────────────────────────────────────────────────────────────────┘
  11. 11. APP BUILD PRIVATE BUILD AND REGISTRY SERVICE ┌─────────────────────────────────────────┐ │ convox API │ │ POST /apps/httpd/build │ ┌──────────────────┐ │ │ │ │ │ ┌─────────────────────────────┐ │ ┌──────────────────┐ │ │ │ │ pull httpd-web:BLASTBUILD │◀───┼───────┤ Convox Registry │ │ │ │ └─────────────────────────────┘ │ │ │ │ │ │ ┌─────────────────────────────┐ │ │ │ │ ├─────┼─────▶│ pull rabbitmq │ │ │ │ │ │ │ └─────────────────────────────┘ │ │ │ │ │ │ ┌─────────────────────────────┐ │ │ │ │ │ │ │tag httpd-rabbit:BQUWNCMIYZG │ │ │ │ │ │ │ └─────────────────────────────┘ │ │ │ ┌────────────────┐ │ DockerHub │ │ ┌─────────────────────────────┐ │ │ ├───────────▶│ S3 │ │ and │ │ │ push $REGISTRY_HOST ├────┼──────▶│ │◀───────────┤ RegistryBucket │ │Private Registries│ │ └─────────────────────────────┘ │ │ │ └────────────────┘ │ │ │ ┌─────────────────────────────┐ │ │ │ │ ├─────┼─────▶│ pull debian │ │ │ │ │ │ │ └─────────────────────────────┘ │ │ │ │ │ │ ┌─────────────────────────────┐ │ │ │ │ │ │ │ build Dockerfile │ │ │ │ │ │ │ └─────────────────────────────┘ │ │ │ │ │ │ ┌─────────────────────────────┐ │ │ │ │ │ │ │ tag httpd-web:BQUWNCMIYZG │ │ │ │ │ │ │ └─────────────────────────────┘ │ │ │ │ │ │ ┌─────────────────────────────┐ │ │ │ └──────────────────┘ │ │ push $REGISTRY_HOST ├────┼──────▶└──────────────────┘ │ └─────────────────────────────┘ │ │ ┌─────────────────────────────┐ │ │ │tag httpd-worker:BQUWNCMIYZG │ │ │ └─────────────────────────────┘ │ │ ┌─────────────────────────────┐ │ │ │ ... │ │ │ └─────────────────────────────┘ │ │ │ │ │ └─────────────────────────────────────────┘
  12. 12. …UNTIL EC2 CONTAINER REGISTRY IS GA
  13. 13. APP LOGS AGENT, DOCKER APIS, KINESIS, LAMBDA ┌──────────────────────────────────────────────────────────┐ ┌──────────────────┐ │ EC2 Instance in ECS Cluster │ │ app1 Kinesis │ │ │ │ ┌────────┐ │ ┌───────────────────────────────────────────┐ │ ┌──────────────┐ ┌──────────────────────────────────┐ │ ┌─┼───▶│shard 1 │ │──┐ │ Lambda w/ EventSourceMapping │ │ │ │ │ │ │ │ │ └────────┘ │ │ │ ┌──────────────────────────────────────┐ │ │ │ │ │ │ │ │ └──────────────────┘ │ │ │function(event, context) { │ │ │ │ app1 │ │ app2 │ │ │ │ │ │ event.records.forEach(function(r) { │ │ │ │ web.1 │ │ worker.1 │ │ │ │ │ │ winston.info(r.kinesis.data) │ │ │ │ │ │ │ │ │ └─┼▶│ }) │──┼────────▶┌───────────────┐ │ │ │ │ │ │ │ │ │ context.done() │ │ │ │ │ └──────────────┘ └──────────────────────────────────┘ │ │ ┌──────────────────┐ │ │} │ │ │ │ │ │ │ │ │ │ app2 Kinesis │ │ │ │ │ │ │ │ │ ┌─────────────────────┘ │ │ │ ┌────────┐ │ │ └──────────────────────────────────────┘ │ │ Syslog Server │ │ ▼ ▼ │ │ │ ┌─▶│shard 1 │ │ │ ┌────────────────────────────────┐ │ │ │ │ ┌────────────┐ ┌────────────┐─────────────┼───┘ │ │ └────────┘ │ │ │function(event, context) { ... }│──┼────────▶│ │ │ │ dockerd │◀─────────────│convox/agent│─────────────┼─────┼─┘ ┌────────┐ │ │ └────────────────────────────────┘ │ │ │ │ └────────────┘ └────────────┘─────────────┼─────┼───▶│shard 2 │ │ │ ┌────────────────────────────────┐ │ │ │ │ ▲ ┌────────────────────────────────────┐ │ │ └────────┘ │────┼─▶│function(event, context) { ... }│───────┼────────▶└───────────────┘ │ │ │GET docker /events (create) │ │ │ . │ │ └────────────────────────────────┘ │ │ ▼ │ GET ENV "Kinesis", "Process"│ │ │ . │ │ │ │ ┌────────────┐ │ GET Docker /logs?follow=1 │ │ │ . │ └───────────────────────────────────────────┘ │ │ ecs-agent │ │ PUT Kinesis /records │ │ │ ┌────────┐ │ │ └────────────┘ └────────────────────────────────────┘ │ │ │shard N │ │ │ │ │ └────────┘ │ └──────────────────────────────────────────────────────────┘ └──────────────────┘
  14. 14. DEBUGGING RUN, EXEC, SSH OVER WEB SOCKETS $ convox run web bash root@3e4160f0c4d0:/app# $ convox ps ID NAME RELEASE CPU MEM STARTED COMMAND 551967b75abd web RHQZEJZFCSD 0.39% 21.04% 2 hours ago rails server -b 0.0.0.0 f5ec95c38f58 worker RHQZEJZFCSD 0.00% 30.35% 2 hours ago sidekiq $ convox exec 551967b75abd bash root@281d0a9c33a:/app# $ convox exec 551967b75abd ps ax PID USER TIME COMMAND 1 root 0:00 sh -c bin/web 6 root 0:00 {web} /bin/sh bin/web 9 root 0:00 unicorn master -c unicorn.rb 11 root 0:00 unicorn worker[0] -c unicorn.rb
  15. 15. SCALING ONE APP ⟶ MANY SERVICES Service Name Task Definition Desired Running ═══════════════════════════════════════════════════════════════════════════ myapp-clock-SVQQEUPGZPS myapp-clock:106 1 1 myapp-scheduler-SSMOCJRAGOM myapp-scheduler:183 1 1 myapp-web-SLHARAVBAWZ myapp-web:119 2 2 myapp-runner-SEGBMHLWREH myapp-runner:163 4 4
  16. 16. CAPACITY PLANNING FITTING MANY APPS IN ONE CLUSTER | | | | | | | | | | | | | | | | | +---+ | | | . | | | | . | | | | . | | | | . | | | | . | | | | . | | | | . | | | +---+ | | | | | | | | | | | | | | | | | | | | ,---, +---+ | | | . | | . | | +---' . +---, ,---' . +---+---+--+ | . . . | . | | . . . | . . . | | + . +---+ . '---, | . ,---+ . . . | | | . | . | . . . | | . | | . . . | | +---+ . +---+ . | +---+---+---+---+ | | . . . | . | . | | . . . | . . . | | + . +---+ . +---+ | . . . + . . . | | | . | . . . . . | | . . . | . . . | | +---+---+---+---+---+---+---+---+---+--+ • rolling app deploys need slack capacity • process type with load balancer needs entire instance • scheduler can fail to place tasks behind the scenes...
  17. 17. noah@convox.com @nzoschke Discuss these techniques and get involved
 GitHub https://github.com/convox Slack http://invite.convox.com/ _ _ _ _ | |_| |__ __ _ _ __ | | _____| | | __| '_ / _` | '_ | |/ / __| | | |_| | | | (_| | | | | <__ _| __|_| |_|__,_|_| |_|_|____(_)

    Be the first to comment

    Login to see the comments

  • ilyadmitrichenko

    Dec. 10, 2015
  • WesleyCharlesBlake

    May. 7, 2016
  • schjus

    Sep. 6, 2017

This presentation covers the benefits of deploying to ECS, as we explore techniques and open source tools to help manage multiple ECS clusters. First we discuss why developing as containers and deploying to ECS offers a more cost-effective, faster, and safer workflow from dev to production than VMs on EC2. Next we present techniques Convox uses to manage countless clusters and apps. We will see a CloudFormation template that reliably provisions, updates, and horizontally and vertically scales ECS clusters, and another template that configures application containers inside the cluster. Finally we survey features that are "missing" on ECS and demonstrate simple tools and techniques Convox has developed to to fill in these gaps. For example: * Running a private build and registry service inside the ECS cluster itself * Splitting up an app's processes (i.e. web and worker) into their own task definitions for independent horizontal scaling * Running an agent on every cluster instance for forwarding all container logs to Kinesis, Lambda and beyond * Running an public API on a cluster that facilitates a `docker exec` debugging session inside application containers

Views

Total views

2,985

On Slideshare

0

From embeds

0

Number of embeds

1,989

Actions

Downloads

6

Shares

0

Comments

0

Likes

3

×