SlideShare a Scribd company logo
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Julien Simon
Principal Technical Evangelist, AWS
@julsimon
Advanced Task Scheduling
with Amazon ECS
Docker on Amazon Web Services
Amazon EC2 Container Service (ECS)
• https://aws.amazon.com/ecs/
• Launched in 04/2015
• No additional charge
Amazon EC2 Container Registry (ECR)
• https://aws.amazon.com/ecr/
• Launched in 12/2015
• Free tier: 500MB / month for a year
• $0.10 / GB / month + outgoing traffic
Both services are available in US, EU and APAC
ECS adoption
https://www.datadoghq.com/blog/3-clear-trends-in-ecs-adoption/
November 2016
Companies
Selected ECS customers
Container Partners
https://aws.amazon.com/containers/partners/
ECS Scheduling
The problem
Given a certain amount of
computing power and memory,
how can we best manage
an arbitrary number of apps
running in Docker containers?
http://tidalseven.com
Amazon ECS: Under the Hood
ALB ALB
AZ 1 AZ 2
user / scheduler
https://github.com/aws/amazon-ecs-agenthttp://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html
Scheduling on ECS: two options so far
1. Let ECS handle scheduling through Services
• Task Definition
• ECS transcription of the Docker Compose file
• Versioning
• cpu_shares, mem_limit
• Number of desired containers
2. Implement a custom scheduler with the ECS API
• Describe cluster state
• Select a specific ECS instance according to custom logic
• Run task on this instance
• Coursera use case: www.youtube.com/watch?v=a45J6xAGUvA
ECS Placement Engine
Placement Engine: giving developers more
control
ALB ALB
AZ 1 AZ 2
user / scheduler
Placement Engine
Constraints
& Strategies
Placement Constraints
Name Example
AMI ID attribute:ecs.ami-id == ami-eca289fb
Availability Zone attribute:ecs.availability-zone == us-east-1a
Instance Type attribute:ecs.instance-type == t2.small
Distinct Instances type=“distinctInstance”
Custom attribute:stack == prod
Example: Constraint on Instance Family/Type
Example: Constraint on Availability Zone
Example: Combining Multiple Constraints
Placement Strategies
Binpacking Spread Affinity Distinct Instance
Placement Strategy Chaining
Spread tasks across Zones
and Binpack within each Zone
Placing Tasks
Anatomy of Task Placement
Cluster Constraints
Custom Constraints
Placement Strategies
Apply Filter
Satisfy CPU, memory, and port requirements
Filter for location, instance-type, AMI, or custom
attribute constraints
Identify instances that meet spread or binpack
placement strategy
Select final container instances for placement
g2.2xlarge t2.small g2.2xlarge g2.2xlarge
Placement: Targeting Instance Type
g2.2xlarge t2.small t2.micro t2.medium
t2.medium t2.small g2.2xlarge
t2.small
t2.small t2.medium
us-east-1aus-east-1d
Placement: Targeting Instance Type & Zone
g2.2xlarge t2.small t2.micro t2.medium
t2.medium t2.small g2.2xlarge t2.small
us-east-1aus-east-1d
g2.2xlarge t2.medium
t2.micro t2.small
us-east-1c
Placement: Availability Zone Spread
g2.2xlarge t2.small t2.micro t2.medium
t2.medium t2.small g2.2xlarge t2.small
us-east-1aus-east-1d
g2.2xlarge t2.medium
t2.micro t2.small
us-east-1c
Placement: Spread across Zone and Binpack
g2.2xlarge t2.small t2.micro t2.medium
t2.medium t2.small g2.2xlarge t2.small
us-east-1aus-east-1d
g2.2xlarge t2.medium
t2.micro t2.small
us-east-1c
Placement: Affinity and Anti-Affinity
Running a Service
t2.medium t2.small t2.small
us-east-1aus-east-1d
t2.medium t2.micro t2.small
us-east-1c
Placement: Multiple Services on a Cluster
t2.medium g2.2xlarge t2.micro t2.small
t2.small t2.small g2.2xlarge t2.small
t2.small t2.small
g2.2xlarge t2.small
Placement: Services – Distinct Instances
Event Stream & Blox
Amazon ECS: Under the Hood
ALB ALB
AZ 1 AZ 2
user / scheduler
Placement Engine
Event Stream
Consuming Real-time Events
Handling ECS events with Blox
scheduler cluster state service
https://blox.github.io/
Amazon ECS: Under the Hood
ALB ALB
AZ 1 AZ 2
user / scheduler
Scheduler
Cluster State Service
Placement Engine
Event Stream
t2.small t2.small t2.small
Blox: Daemon Scheduler
t2.small t2.small t2.small
scheduler cluster state service
t2.small t2.small t2.small
Demo: Deploying Blox on AWS
Creating Clusters
Create an ECS cluster for Blox
CF template: https://github.com/blox/blox/blob/dev/deploy/aws/conf/cloudformation_template.json
à CloudWatch Event Rule + SQS queue
à 3 containers: Daemon Scheduler + Cluster State Service + etcd
à REST API exposing the Daemon Scheduler API
Create another ECS cluster managed by Blox
$ ecs-cli configure --cluster WebCluster
$ ecs-cli up --keypair admin --capability-iam --size 3 --instance-type t2.micro
Invoke the scheduler API
‘demo-cli’ tool: https://github.com/blox/blox/tree/dev/deploy/demo-cli
Listing Task Definitions
Grab the ARN for an nginx Task Definition, which the
Daemon Scheduler will manage on ‘WebCluster’.
$ ./list-task-definitions.py --region eu-central-1
== Blox Demo CLI - List Task Definitions ==
{
"taskDefinitionArns": [
"arn:aws:ecs:eu-central-1:ACCOUNT:task-definition/BloxFramework:2",
"arn:aws:ecs:eu-central-1:ACCOUNT:task-definition/nginx:1",
"arn:aws:ecs:eu-central-1:ACCOUNT:task-definition/nginx:2"
]
}
Creating an Environment
$ ./blox-create-environment.py --environment WebEnvironment --cluster
WebCluster --task-definition "arn:aws:ecs:eu-central-1:ACCOUNT:task-
definition/nginx:2" --stack Blox --apigateway --region eu-central-1
== Blox Demo CLI - Create Blox Environment ==
HTTP Response Code: 200
{
"taskDefinition": "arn:aws:ecs:eu-central-1:ACCOUNT:task-definition/nginx:2",
"deploymentToken": "17248257-08ec-4438-888f-e0ac28397653",
"health": "healthy",
"name": "WebEnvironment",
"instanceGroup": {
"cluster": "arn:aws:ecs:eu-central-1:ACCOUNT:cluster/WebCluster"
}
}
Listing Environments
$ ./blox-list-environments.py --stack Blox --apigateway --region eu-central-1
== Blox Demo CLI - List Blox Environments ==
HTTP Response Code: 200
{
"items": [
{
"taskDefinition": "arn:aws:ecs:eu-central-1:ACCOUNT:task-definition/nginx:2",
"deploymentToken": "17248257-08ec-4438-888f-e0ac28397653",
"health": "healthy",
"name": "WebEnvironment",
"instanceGroup": {
"cluster": "arn:aws:ecs:eu-central-1:ACCOUNT:cluster/WebCluster"
}
}
]
}
Creating a Deployment
$ ./blox-create-deployment.py --environment WebEnvironment
--deployment-token "17248257-08ec-4438-888f-e0ac28397653"
--stack Blox --apigateway --region eu-central-1
== Blox Demo CLI - Create Blox Deployment ==
HTTP Response Code: 200
{
"status": "pending",
"environmentName": "WebEnvironment",
"id": "7a05ea99-27a9-4339-a7a6-f4120065aea3",
"failedInstances": [],
"taskDefinition": "arn:aws:ecs:eu-central-1:613904931467:task-definition/nginx:2”
}
Listing Deployments
$ ./blox-list-deployments.py --environment WebEnvironment --stack Blox
--apigateway --region eu-central-1
== Blox Demo CLI - List Blox Deployments ==
HTTP Response Code: 200
{
"items": [
{
"status": "completed",
"environmentName": "WebEnvironment",
"id": "7a05ea99-27a9-4339-a7a6-f4120065aea3",
"failedInstances": [],
"taskDefinition": "arn:aws:ecs:eu-central-1:ACCOUNT:task-definition/nginx:2"
}
]
}
Scaling a Deployment
$ ecs-cli ps
Name State Ports TaskDefinition
26313cbe-d929-49de-9cc3-873bf5f32a91/nginx RUNNING nginx:2
98442432-fd5c-434d-b93c-0737bd06aaab/nginx RUNNING nginx:2
ce9bf217-4b34-4f31-9c7b-a8c3402f1ffd/nginx RUNNING nginx:2
$ ecs-cli scale --size 4 --capability-iam
$ ecs-cli ps
Name State Ports TaskDefinition
26313cbe-d929-49de-9cc3-873bf5f32a91/nginx RUNNING nginx:2
98442432-fd5c-434d-b93c-0737bd06aaab/nginx RUNNING nginx:2
c404ac9a-0948-4cc8-b5b0-2238ccdf4035/nginx RUNNING nginx:2
ce9bf217-4b34-4f31-9c7b-a8c3402f1ffd/nginx RUNNING nginx:2
Additional resources
ECS customers
https://www.mapbox.com/blog/switch-to-ecs/
https://segment.com/blog/rebuilding-our-infrastructure/
Tech articles by Werner Vogels, CTO, Amazon.com
http://www.allthingsdistributed.com/2014/11/amazon-ec2-container-service.html
http://www.allthingsdistributed.com/2015/04/state-management-and-scheduling-with-ecs.html
http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html
Blog articles & videos
https://aws.amazon.com/blogs/compute/amazon-ec2-container-service-at-aws-reinvent-2016-wrap-up/
https://aws.amazon.com/blogs/compute/continuous-deployment-to-amazon-ecs-using-aws-codepipeline-
aws-codebuild-amazon-ecr-and-aws-cloudformation/
Thank you!
Julien Simon
Principal Technical Evangelist, AWS
@julsimon

More Related Content

What's hot

From Docker to Production - ZendCon 2016
From Docker to Production - ZendCon 2016From Docker to Production - ZendCon 2016
From Docker to Production - ZendCon 2016
Chris Tankersley
 
Kubernetes Basis: Pods, Deployments, and Services
Kubernetes Basis: Pods, Deployments, and ServicesKubernetes Basis: Pods, Deployments, and Services
Kubernetes Basis: Pods, Deployments, and Services
Jian-Kai Wang
 
開放運算&GPU技術研究班
開放運算&GPU技術研究班開放運算&GPU技術研究班
開放運算&GPU技術研究班
Paul Chao
 
Kubernetes上で動作する機械学習モジュールの配信&管理基盤Rekcurd について
Kubernetes上で動作する機械学習モジュールの配信&管理基盤Rekcurd についてKubernetes上で動作する機械学習モジュールの配信&管理基盤Rekcurd について
Kubernetes上で動作する機械学習モジュールの配信&管理基盤Rekcurd について
LINE Corporation
 
KubeCon EU 2016: A Practical Guide to Container Scheduling
KubeCon EU 2016: A Practical Guide to Container SchedulingKubeCon EU 2016: A Practical Guide to Container Scheduling
KubeCon EU 2016: A Practical Guide to Container Scheduling
KubeAcademy
 
From Docker to Production - SunshinePHP 2017
From Docker to Production - SunshinePHP 2017From Docker to Production - SunshinePHP 2017
From Docker to Production - SunshinePHP 2017
Chris Tankersley
 
Using Nix and Docker as automated deployment solutions
Using Nix and Docker as automated deployment solutionsUsing Nix and Docker as automated deployment solutions
Using Nix and Docker as automated deployment solutions
Sander van der Burg
 
Dockerを利用したローカル環境から本番環境までの構築設計
Dockerを利用したローカル環境から本番環境までの構築設計Dockerを利用したローカル環境から本番環境までの構築設計
Dockerを利用したローカル環境から本番環境までの構築設計
Koichi Nagaoka
 
Monitoring, Logging and Tracing on Kubernetes
Monitoring, Logging and Tracing on KubernetesMonitoring, Logging and Tracing on Kubernetes
Monitoring, Logging and Tracing on Kubernetes
Martin Etmajer
 
nix-processmgmt: An experimental Nix-based process manager-agnostic framework
nix-processmgmt: An experimental Nix-based process manager-agnostic frameworknix-processmgmt: An experimental Nix-based process manager-agnostic framework
nix-processmgmt: An experimental Nix-based process manager-agnostic framework
Sander van der Burg
 
Docker Swarm 0.2.0
Docker Swarm 0.2.0Docker Swarm 0.2.0
Docker Swarm 0.2.0
Docker, Inc.
 
Deep Dive into Docker Swarm Mode
Deep Dive into Docker Swarm ModeDeep Dive into Docker Swarm Mode
Deep Dive into Docker Swarm Mode
Ajeet Singh Raina
 
Kubernetes for Java developers
Kubernetes for Java developersKubernetes for Java developers
Kubernetes for Java developers
Robert Barr
 
Automatically Renew Certificated In Your Kubernetes Cluster
Automatically Renew Certificated In Your Kubernetes ClusterAutomatically Renew Certificated In Your Kubernetes Cluster
Automatically Renew Certificated In Your Kubernetes Cluster
HungWei Chiu
 
KubeCon EU 2016: Kubernetes and the Potential for Higher Level Interfaces
KubeCon EU 2016: Kubernetes and the Potential for Higher Level InterfacesKubeCon EU 2016: Kubernetes and the Potential for Higher Level Interfaces
KubeCon EU 2016: Kubernetes and the Potential for Higher Level Interfaces
KubeAcademy
 
Docker meetup - PaaS interoperability
Docker meetup - PaaS interoperabilityDocker meetup - PaaS interoperability
Docker meetup - PaaS interoperability
Ludovic Piot
 
Dockerizing a multi-component Open Data app
Dockerizing a multi-component Open Data app  Dockerizing a multi-component Open Data app
Dockerizing a multi-component Open Data app
Stergios Tsiafoulis
 
Why Docker? Dayton PHP, April 2017
Why Docker? Dayton PHP, April 2017Why Docker? Dayton PHP, April 2017
Why Docker? Dayton PHP, April 2017
Chris Tankersley
 
Scaling docker with kubernetes
Scaling docker with kubernetesScaling docker with kubernetes
Scaling docker with kubernetes
Liran Cohen
 
[DockerCon 2019] Hardening Docker daemon with Rootless mode
[DockerCon 2019] Hardening Docker daemon with Rootless mode[DockerCon 2019] Hardening Docker daemon with Rootless mode
[DockerCon 2019] Hardening Docker daemon with Rootless mode
Akihiro Suda
 

What's hot (20)

From Docker to Production - ZendCon 2016
From Docker to Production - ZendCon 2016From Docker to Production - ZendCon 2016
From Docker to Production - ZendCon 2016
 
Kubernetes Basis: Pods, Deployments, and Services
Kubernetes Basis: Pods, Deployments, and ServicesKubernetes Basis: Pods, Deployments, and Services
Kubernetes Basis: Pods, Deployments, and Services
 
開放運算&GPU技術研究班
開放運算&GPU技術研究班開放運算&GPU技術研究班
開放運算&GPU技術研究班
 
Kubernetes上で動作する機械学習モジュールの配信&管理基盤Rekcurd について
Kubernetes上で動作する機械学習モジュールの配信&管理基盤Rekcurd についてKubernetes上で動作する機械学習モジュールの配信&管理基盤Rekcurd について
Kubernetes上で動作する機械学習モジュールの配信&管理基盤Rekcurd について
 
KubeCon EU 2016: A Practical Guide to Container Scheduling
KubeCon EU 2016: A Practical Guide to Container SchedulingKubeCon EU 2016: A Practical Guide to Container Scheduling
KubeCon EU 2016: A Practical Guide to Container Scheduling
 
From Docker to Production - SunshinePHP 2017
From Docker to Production - SunshinePHP 2017From Docker to Production - SunshinePHP 2017
From Docker to Production - SunshinePHP 2017
 
Using Nix and Docker as automated deployment solutions
Using Nix and Docker as automated deployment solutionsUsing Nix and Docker as automated deployment solutions
Using Nix and Docker as automated deployment solutions
 
Dockerを利用したローカル環境から本番環境までの構築設計
Dockerを利用したローカル環境から本番環境までの構築設計Dockerを利用したローカル環境から本番環境までの構築設計
Dockerを利用したローカル環境から本番環境までの構築設計
 
Monitoring, Logging and Tracing on Kubernetes
Monitoring, Logging and Tracing on KubernetesMonitoring, Logging and Tracing on Kubernetes
Monitoring, Logging and Tracing on Kubernetes
 
nix-processmgmt: An experimental Nix-based process manager-agnostic framework
nix-processmgmt: An experimental Nix-based process manager-agnostic frameworknix-processmgmt: An experimental Nix-based process manager-agnostic framework
nix-processmgmt: An experimental Nix-based process manager-agnostic framework
 
Docker Swarm 0.2.0
Docker Swarm 0.2.0Docker Swarm 0.2.0
Docker Swarm 0.2.0
 
Deep Dive into Docker Swarm Mode
Deep Dive into Docker Swarm ModeDeep Dive into Docker Swarm Mode
Deep Dive into Docker Swarm Mode
 
Kubernetes for Java developers
Kubernetes for Java developersKubernetes for Java developers
Kubernetes for Java developers
 
Automatically Renew Certificated In Your Kubernetes Cluster
Automatically Renew Certificated In Your Kubernetes ClusterAutomatically Renew Certificated In Your Kubernetes Cluster
Automatically Renew Certificated In Your Kubernetes Cluster
 
KubeCon EU 2016: Kubernetes and the Potential for Higher Level Interfaces
KubeCon EU 2016: Kubernetes and the Potential for Higher Level InterfacesKubeCon EU 2016: Kubernetes and the Potential for Higher Level Interfaces
KubeCon EU 2016: Kubernetes and the Potential for Higher Level Interfaces
 
Docker meetup - PaaS interoperability
Docker meetup - PaaS interoperabilityDocker meetup - PaaS interoperability
Docker meetup - PaaS interoperability
 
Dockerizing a multi-component Open Data app
Dockerizing a multi-component Open Data app  Dockerizing a multi-component Open Data app
Dockerizing a multi-component Open Data app
 
Why Docker? Dayton PHP, April 2017
Why Docker? Dayton PHP, April 2017Why Docker? Dayton PHP, April 2017
Why Docker? Dayton PHP, April 2017
 
Scaling docker with kubernetes
Scaling docker with kubernetesScaling docker with kubernetes
Scaling docker with kubernetes
 
[DockerCon 2019] Hardening Docker daemon with Rootless mode
[DockerCon 2019] Hardening Docker daemon with Rootless mode[DockerCon 2019] Hardening Docker daemon with Rootless mode
[DockerCon 2019] Hardening Docker daemon with Rootless mode
 

Viewers also liked

DevOps, NoOps, everything-as-code, commoditisation… Quel futur pour les ops ?
DevOps, NoOps, everything-as-code, commoditisation… Quel futur pour les ops ?DevOps, NoOps, everything-as-code, commoditisation… Quel futur pour les ops ?
DevOps, NoOps, everything-as-code, commoditisation… Quel futur pour les ops ?
Adrien Blind
 
When Docker Engine 1.12 features unleashes software architecture
When Docker Engine 1.12 features unleashes software architectureWhen Docker Engine 1.12 features unleashes software architecture
When Docker Engine 1.12 features unleashes software architecture
Adrien Blind
 
DevOps à l'échelle: ce que l'on a fait, ce que l'on a appris chez Societe Gen...
DevOps à l'échelle: ce que l'on a fait, ce que l'on a appris chez Societe Gen...DevOps à l'échelle: ce que l'on a fait, ce que l'on a appris chez Societe Gen...
DevOps à l'échelle: ce que l'on a fait, ce que l'on a appris chez Societe Gen...
Adrien Blind
 
Docker cloud hybridation & orchestration
Docker cloud hybridation & orchestrationDocker cloud hybridation & orchestration
Docker cloud hybridation & orchestration
Adrien Blind
 
Full stack automation - TIAD 2015
Full stack automation - TIAD 2015Full stack automation - TIAD 2015
Full stack automation - TIAD 2015
Adrien Blind
 
Docker, cornerstone of cloud hybridation ? [Cloud Expo Europe 2016]
Docker, cornerstone of cloud hybridation ? [Cloud Expo Europe 2016]Docker, cornerstone of cloud hybridation ? [Cloud Expo Europe 2016]
Docker, cornerstone of cloud hybridation ? [Cloud Expo Europe 2016]
Adrien Blind
 
Docker, cornerstone of an hybrid cloud?
Docker, cornerstone of an hybrid cloud?Docker, cornerstone of an hybrid cloud?
Docker, cornerstone of an hybrid cloud?
Adrien Blind
 
Docker, Pierre angulaire du continuous delivery ?
Docker, Pierre angulaire du continuous delivery ?Docker, Pierre angulaire du continuous delivery ?
Docker, Pierre angulaire du continuous delivery ?
Adrien Blind
 
The missing piece : when Docker networking and services finally unleashes so...
 The missing piece : when Docker networking and services finally unleashes so... The missing piece : when Docker networking and services finally unleashes so...
The missing piece : when Docker networking and services finally unleashes so...
Adrien Blind
 
Skynet vs planet of apes
Skynet vs planet of apesSkynet vs planet of apes
Skynet vs planet of apes
Adrien Blind
 
Identity & Access Management in the cloud
Identity & Access Management in the cloudIdentity & Access Management in the cloud
Identity & Access Management in the cloud
Adrien Blind
 
Dockercon Europe 2014 - Continuous Delivery leveraging on Docker CaaS
Dockercon Europe 2014 - Continuous Delivery leveraging on Docker CaaSDockercon Europe 2014 - Continuous Delivery leveraging on Docker CaaS
Dockercon Europe 2014 - Continuous Delivery leveraging on Docker CaaS
Adrien Blind
 
Monitoring de conteneurs en production - Jonathan Raffre & Jean-Pascal Thiery
Monitoring de conteneurs en production - Jonathan Raffre & Jean-Pascal ThieryMonitoring de conteneurs en production - Jonathan Raffre & Jean-Pascal Thiery
Monitoring de conteneurs en production - Jonathan Raffre & Jean-Pascal Thiery
Paris Container Day
 
Docker: Redistributing DevOps cards, on the way to PaaS
Docker: Redistributing DevOps cards, on the way to PaaSDocker: Redistributing DevOps cards, on the way to PaaS
Docker: Redistributing DevOps cards, on the way to PaaS
Adrien Blind
 
Petit déjeuner Octo - L'infra au service de ses projets
Petit déjeuner Octo - L'infra au service de ses projetsPetit déjeuner Octo - L'infra au service de ses projets
Petit déjeuner Octo - L'infra au service de ses projets
Adrien Blind
 
Continous delivery at docker age
Continous delivery at docker ageContinous delivery at docker age
Continous delivery at docker age
Adrien Blind
 
DevOps at scale: what we did, what we learned at Societe Generale
DevOps at scale: what we did, what we learned at Societe GeneraleDevOps at scale: what we did, what we learned at Societe Generale
DevOps at scale: what we did, what we learned at Societe Generale
Adrien Blind
 
Introduction to Unikernels at first Paris Unikernels meetup
Introduction to Unikernels at first Paris Unikernels meetupIntroduction to Unikernels at first Paris Unikernels meetup
Introduction to Unikernels at first Paris Unikernels meetup
Adrien Blind
 
Docker networking basics & coupling with Software Defined Networks
Docker networking basics & coupling with Software Defined NetworksDocker networking basics & coupling with Software Defined Networks
Docker networking basics & coupling with Software Defined Networks
Adrien Blind
 

Viewers also liked (19)

DevOps, NoOps, everything-as-code, commoditisation… Quel futur pour les ops ?
DevOps, NoOps, everything-as-code, commoditisation… Quel futur pour les ops ?DevOps, NoOps, everything-as-code, commoditisation… Quel futur pour les ops ?
DevOps, NoOps, everything-as-code, commoditisation… Quel futur pour les ops ?
 
When Docker Engine 1.12 features unleashes software architecture
When Docker Engine 1.12 features unleashes software architectureWhen Docker Engine 1.12 features unleashes software architecture
When Docker Engine 1.12 features unleashes software architecture
 
DevOps à l'échelle: ce que l'on a fait, ce que l'on a appris chez Societe Gen...
DevOps à l'échelle: ce que l'on a fait, ce que l'on a appris chez Societe Gen...DevOps à l'échelle: ce que l'on a fait, ce que l'on a appris chez Societe Gen...
DevOps à l'échelle: ce que l'on a fait, ce que l'on a appris chez Societe Gen...
 
Docker cloud hybridation & orchestration
Docker cloud hybridation & orchestrationDocker cloud hybridation & orchestration
Docker cloud hybridation & orchestration
 
Full stack automation - TIAD 2015
Full stack automation - TIAD 2015Full stack automation - TIAD 2015
Full stack automation - TIAD 2015
 
Docker, cornerstone of cloud hybridation ? [Cloud Expo Europe 2016]
Docker, cornerstone of cloud hybridation ? [Cloud Expo Europe 2016]Docker, cornerstone of cloud hybridation ? [Cloud Expo Europe 2016]
Docker, cornerstone of cloud hybridation ? [Cloud Expo Europe 2016]
 
Docker, cornerstone of an hybrid cloud?
Docker, cornerstone of an hybrid cloud?Docker, cornerstone of an hybrid cloud?
Docker, cornerstone of an hybrid cloud?
 
Docker, Pierre angulaire du continuous delivery ?
Docker, Pierre angulaire du continuous delivery ?Docker, Pierre angulaire du continuous delivery ?
Docker, Pierre angulaire du continuous delivery ?
 
The missing piece : when Docker networking and services finally unleashes so...
 The missing piece : when Docker networking and services finally unleashes so... The missing piece : when Docker networking and services finally unleashes so...
The missing piece : when Docker networking and services finally unleashes so...
 
Skynet vs planet of apes
Skynet vs planet of apesSkynet vs planet of apes
Skynet vs planet of apes
 
Identity & Access Management in the cloud
Identity & Access Management in the cloudIdentity & Access Management in the cloud
Identity & Access Management in the cloud
 
Dockercon Europe 2014 - Continuous Delivery leveraging on Docker CaaS
Dockercon Europe 2014 - Continuous Delivery leveraging on Docker CaaSDockercon Europe 2014 - Continuous Delivery leveraging on Docker CaaS
Dockercon Europe 2014 - Continuous Delivery leveraging on Docker CaaS
 
Monitoring de conteneurs en production - Jonathan Raffre & Jean-Pascal Thiery
Monitoring de conteneurs en production - Jonathan Raffre & Jean-Pascal ThieryMonitoring de conteneurs en production - Jonathan Raffre & Jean-Pascal Thiery
Monitoring de conteneurs en production - Jonathan Raffre & Jean-Pascal Thiery
 
Docker: Redistributing DevOps cards, on the way to PaaS
Docker: Redistributing DevOps cards, on the way to PaaSDocker: Redistributing DevOps cards, on the way to PaaS
Docker: Redistributing DevOps cards, on the way to PaaS
 
Petit déjeuner Octo - L'infra au service de ses projets
Petit déjeuner Octo - L'infra au service de ses projetsPetit déjeuner Octo - L'infra au service de ses projets
Petit déjeuner Octo - L'infra au service de ses projets
 
Continous delivery at docker age
Continous delivery at docker ageContinous delivery at docker age
Continous delivery at docker age
 
DevOps at scale: what we did, what we learned at Societe Generale
DevOps at scale: what we did, what we learned at Societe GeneraleDevOps at scale: what we did, what we learned at Societe Generale
DevOps at scale: what we did, what we learned at Societe Generale
 
Introduction to Unikernels at first Paris Unikernels meetup
Introduction to Unikernels at first Paris Unikernels meetupIntroduction to Unikernels at first Paris Unikernels meetup
Introduction to Unikernels at first Paris Unikernels meetup
 
Docker networking basics & coupling with Software Defined Networks
Docker networking basics & coupling with Software Defined NetworksDocker networking basics & coupling with Software Defined Networks
Docker networking basics & coupling with Software Defined Networks
 

Similar to Advanced Task Scheduling with Amazon ECS - Julien Simon

Advanced Task Scheduling with Amazon ECS (June 2017)
Advanced Task Scheduling with Amazon ECS (June 2017)Advanced Task Scheduling with Amazon ECS (June 2017)
Advanced Task Scheduling with Amazon ECS (June 2017)
Julien SIMON
 
Advanced Task Scheduling with Amazon ECS
Advanced Task Scheduling with Amazon ECSAdvanced Task Scheduling with Amazon ECS
Advanced Task Scheduling with Amazon ECS
Julien SIMON
 
Advanced Scheduling with Amazon ECS (September 2017)
Advanced Scheduling with Amazon ECS (September 2017)Advanced Scheduling with Amazon ECS (September 2017)
Advanced Scheduling with Amazon ECS (September 2017)
Julien SIMON
 
Advanced container management and scheduling
Advanced container management and schedulingAdvanced container management and scheduling
Advanced container management and scheduling
Amazon Web Services
 
Advanced Container Management and Scheduling
Advanced Container Management and SchedulingAdvanced Container Management and Scheduling
Advanced Container Management and Scheduling
Amazon Web Services
 
Advanced Container Management and Scheduling - DevDay Los Angeles 2017
Advanced Container Management and Scheduling - DevDay Los Angeles 2017Advanced Container Management and Scheduling - DevDay Los Angeles 2017
Advanced Container Management and Scheduling - DevDay Los Angeles 2017
Amazon Web Services
 
Advanced Container Management and Scheduling - DevDay Austin 2017
Advanced Container Management and Scheduling - DevDay Austin 2017Advanced Container Management and Scheduling - DevDay Austin 2017
Advanced Container Management and Scheduling - DevDay Austin 2017
Amazon Web Services
 
Monitoring Containers at Scale - September Webinar Series
Monitoring Containers at Scale - September Webinar SeriesMonitoring Containers at Scale - September Webinar Series
Monitoring Containers at Scale - September Webinar Series
Amazon Web Services
 
使用 Blox 實現容器任務調度與資源編排
使用 Blox 實現容器任務調度與資源編排使用 Blox 實現容器任務調度與資源編排
使用 Blox 實現容器任務調度與資源編排
Amazon Web Services
 
NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox
NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and BloxNEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox
NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox
Amazon Web Services
 
Announcing Blox - Open Source Projects for Customizing Scheduling on Amazon ECS
Announcing Blox - Open Source Projects for Customizing Scheduling on Amazon ECSAnnouncing Blox - Open Source Projects for Customizing Scheduling on Amazon ECS
Announcing Blox - Open Source Projects for Customizing Scheduling on Amazon ECS
Amazon Web Services
 
Weaveworks at AWS re:Invent 2016: Operations Management with Amazon ECS
Weaveworks at AWS re:Invent 2016: Operations Management with Amazon ECSWeaveworks at AWS re:Invent 2016: Operations Management with Amazon ECS
Weaveworks at AWS re:Invent 2016: Operations Management with Amazon ECS
Weaveworks
 
AWS reinvent 2019 recap - Riyadh - Containers and Serverless - Paul Maddox
AWS reinvent 2019 recap - Riyadh - Containers and Serverless - Paul MaddoxAWS reinvent 2019 recap - Riyadh - Containers and Serverless - Paul Maddox
AWS reinvent 2019 recap - Riyadh - Containers and Serverless - Paul Maddox
AWS Riyadh User Group
 
Docker on AWS with Amazon ECR & ECS - Pop-up Loft Tel Aviv
Docker on AWS with Amazon ECR & ECS - Pop-up Loft Tel Aviv Docker on AWS with Amazon ECR & ECS - Pop-up Loft Tel Aviv
Docker on AWS with Amazon ECR & ECS - Pop-up Loft Tel Aviv
Amazon Web Services
 
Docker clusters on AWS with Amazon ECS and Kubernetes
Docker clusters on AWS with Amazon ECS and KubernetesDocker clusters on AWS with Amazon ECS and Kubernetes
Docker clusters on AWS with Amazon ECS and Kubernetes
Julien SIMON
 
NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox
NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and BloxNEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox
NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox
Amazon Web Services
 
"AWS Fargate: Containerization meets Serverless" at AWS User Group Cologne 20...
"AWS Fargate: Containerization meets Serverless" at AWS User Group Cologne 20..."AWS Fargate: Containerization meets Serverless" at AWS User Group Cologne 20...
"AWS Fargate: Containerization meets Serverless" at AWS User Group Cologne 20...
Vadym Kazulkin
 
February 2016 Webinar Series - EC2 Container Service Deep Dive
February 2016 Webinar Series - EC2 Container Service Deep Dive February 2016 Webinar Series - EC2 Container Service Deep Dive
February 2016 Webinar Series - EC2 Container Service Deep Dive
Amazon Web Services
 
Running Docker Containers on AWS
Running Docker Containers on AWSRunning Docker Containers on AWS
Running Docker Containers on AWS
Vladimir Simek
 
A 60-minute tour of AWS Compute (November 2016)
A 60-minute tour of AWS Compute (November 2016)A 60-minute tour of AWS Compute (November 2016)
A 60-minute tour of AWS Compute (November 2016)
Julien SIMON
 

Similar to Advanced Task Scheduling with Amazon ECS - Julien Simon (20)

Advanced Task Scheduling with Amazon ECS (June 2017)
Advanced Task Scheduling with Amazon ECS (June 2017)Advanced Task Scheduling with Amazon ECS (June 2017)
Advanced Task Scheduling with Amazon ECS (June 2017)
 
Advanced Task Scheduling with Amazon ECS
Advanced Task Scheduling with Amazon ECSAdvanced Task Scheduling with Amazon ECS
Advanced Task Scheduling with Amazon ECS
 
Advanced Scheduling with Amazon ECS (September 2017)
Advanced Scheduling with Amazon ECS (September 2017)Advanced Scheduling with Amazon ECS (September 2017)
Advanced Scheduling with Amazon ECS (September 2017)
 
Advanced container management and scheduling
Advanced container management and schedulingAdvanced container management and scheduling
Advanced container management and scheduling
 
Advanced Container Management and Scheduling
Advanced Container Management and SchedulingAdvanced Container Management and Scheduling
Advanced Container Management and Scheduling
 
Advanced Container Management and Scheduling - DevDay Los Angeles 2017
Advanced Container Management and Scheduling - DevDay Los Angeles 2017Advanced Container Management and Scheduling - DevDay Los Angeles 2017
Advanced Container Management and Scheduling - DevDay Los Angeles 2017
 
Advanced Container Management and Scheduling - DevDay Austin 2017
Advanced Container Management and Scheduling - DevDay Austin 2017Advanced Container Management and Scheduling - DevDay Austin 2017
Advanced Container Management and Scheduling - DevDay Austin 2017
 
Monitoring Containers at Scale - September Webinar Series
Monitoring Containers at Scale - September Webinar SeriesMonitoring Containers at Scale - September Webinar Series
Monitoring Containers at Scale - September Webinar Series
 
使用 Blox 實現容器任務調度與資源編排
使用 Blox 實現容器任務調度與資源編排使用 Blox 實現容器任務調度與資源編排
使用 Blox 實現容器任務調度與資源編排
 
NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox
NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and BloxNEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox
NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox
 
Announcing Blox - Open Source Projects for Customizing Scheduling on Amazon ECS
Announcing Blox - Open Source Projects for Customizing Scheduling on Amazon ECSAnnouncing Blox - Open Source Projects for Customizing Scheduling on Amazon ECS
Announcing Blox - Open Source Projects for Customizing Scheduling on Amazon ECS
 
Weaveworks at AWS re:Invent 2016: Operations Management with Amazon ECS
Weaveworks at AWS re:Invent 2016: Operations Management with Amazon ECSWeaveworks at AWS re:Invent 2016: Operations Management with Amazon ECS
Weaveworks at AWS re:Invent 2016: Operations Management with Amazon ECS
 
AWS reinvent 2019 recap - Riyadh - Containers and Serverless - Paul Maddox
AWS reinvent 2019 recap - Riyadh - Containers and Serverless - Paul MaddoxAWS reinvent 2019 recap - Riyadh - Containers and Serverless - Paul Maddox
AWS reinvent 2019 recap - Riyadh - Containers and Serverless - Paul Maddox
 
Docker on AWS with Amazon ECR & ECS - Pop-up Loft Tel Aviv
Docker on AWS with Amazon ECR & ECS - Pop-up Loft Tel Aviv Docker on AWS with Amazon ECR & ECS - Pop-up Loft Tel Aviv
Docker on AWS with Amazon ECR & ECS - Pop-up Loft Tel Aviv
 
Docker clusters on AWS with Amazon ECS and Kubernetes
Docker clusters on AWS with Amazon ECS and KubernetesDocker clusters on AWS with Amazon ECS and Kubernetes
Docker clusters on AWS with Amazon ECS and Kubernetes
 
NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox
NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and BloxNEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox
NEW LAUNCH! Advanced Task Scheduling with Amazon ECS and Blox
 
"AWS Fargate: Containerization meets Serverless" at AWS User Group Cologne 20...
"AWS Fargate: Containerization meets Serverless" at AWS User Group Cologne 20..."AWS Fargate: Containerization meets Serverless" at AWS User Group Cologne 20...
"AWS Fargate: Containerization meets Serverless" at AWS User Group Cologne 20...
 
February 2016 Webinar Series - EC2 Container Service Deep Dive
February 2016 Webinar Series - EC2 Container Service Deep Dive February 2016 Webinar Series - EC2 Container Service Deep Dive
February 2016 Webinar Series - EC2 Container Service Deep Dive
 
Running Docker Containers on AWS
Running Docker Containers on AWSRunning Docker Containers on AWS
Running Docker Containers on AWS
 
A 60-minute tour of AWS Compute (November 2016)
A 60-minute tour of AWS Compute (November 2016)A 60-minute tour of AWS Compute (November 2016)
A 60-minute tour of AWS Compute (November 2016)
 

Recently uploaded

Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
AWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptxAWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptx
HarisZaheer8
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Zilliz
 
Trusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process MiningTrusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process Mining
LucaBarbaro3
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Zilliz
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
panagenda
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Alpen-Adria-Universität
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
Jakub Marek
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Zilliz
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Malak Abu Hammad
 
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdfNunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
flufftailshop
 
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Jeffrey Haguewood
 
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
System Design Case Study: Building a Scalable E-Commerce Platform - HiikeSystem Design Case Study: Building a Scalable E-Commerce Platform - Hiike
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
Hiike
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc
 
A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024
Intelisync
 
June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
Ivanti
 
UI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentationUI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentation
Wouter Lemaire
 
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
saastr
 
Operating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptxOperating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptx
Pravash Chandra Das
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
innovationoecd
 

Recently uploaded (20)

Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
AWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptxAWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptx
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
 
Trusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process MiningTrusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process Mining
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
 
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdfNunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
 
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
 
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
System Design Case Study: Building a Scalable E-Commerce Platform - HiikeSystem Design Case Study: Building a Scalable E-Commerce Platform - Hiike
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
 
A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024
 
June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
 
UI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentationUI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentation
 
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
 
Operating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptxOperating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptx
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
 

Advanced Task Scheduling with Amazon ECS - Julien Simon

  • 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Julien Simon Principal Technical Evangelist, AWS @julsimon Advanced Task Scheduling with Amazon ECS
  • 2. Docker on Amazon Web Services Amazon EC2 Container Service (ECS) • https://aws.amazon.com/ecs/ • Launched in 04/2015 • No additional charge Amazon EC2 Container Registry (ECR) • https://aws.amazon.com/ecr/ • Launched in 12/2015 • Free tier: 500MB / month for a year • $0.10 / GB / month + outgoing traffic Both services are available in US, EU and APAC
  • 7. The problem Given a certain amount of computing power and memory, how can we best manage an arbitrary number of apps running in Docker containers? http://tidalseven.com
  • 8. Amazon ECS: Under the Hood ALB ALB AZ 1 AZ 2 user / scheduler https://github.com/aws/amazon-ecs-agenthttp://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html
  • 9. Scheduling on ECS: two options so far 1. Let ECS handle scheduling through Services • Task Definition • ECS transcription of the Docker Compose file • Versioning • cpu_shares, mem_limit • Number of desired containers 2. Implement a custom scheduler with the ECS API • Describe cluster state • Select a specific ECS instance according to custom logic • Run task on this instance • Coursera use case: www.youtube.com/watch?v=a45J6xAGUvA
  • 11. Placement Engine: giving developers more control ALB ALB AZ 1 AZ 2 user / scheduler Placement Engine Constraints & Strategies
  • 12. Placement Constraints Name Example AMI ID attribute:ecs.ami-id == ami-eca289fb Availability Zone attribute:ecs.availability-zone == us-east-1a Instance Type attribute:ecs.instance-type == t2.small Distinct Instances type=“distinctInstance” Custom attribute:stack == prod
  • 13. Example: Constraint on Instance Family/Type
  • 14. Example: Constraint on Availability Zone
  • 16. Placement Strategies Binpacking Spread Affinity Distinct Instance
  • 17. Placement Strategy Chaining Spread tasks across Zones and Binpack within each Zone
  • 19. Anatomy of Task Placement Cluster Constraints Custom Constraints Placement Strategies Apply Filter Satisfy CPU, memory, and port requirements Filter for location, instance-type, AMI, or custom attribute constraints Identify instances that meet spread or binpack placement strategy Select final container instances for placement
  • 20. g2.2xlarge t2.small g2.2xlarge g2.2xlarge Placement: Targeting Instance Type
  • 21. g2.2xlarge t2.small t2.micro t2.medium t2.medium t2.small g2.2xlarge t2.small t2.small t2.medium us-east-1aus-east-1d Placement: Targeting Instance Type & Zone
  • 22. g2.2xlarge t2.small t2.micro t2.medium t2.medium t2.small g2.2xlarge t2.small us-east-1aus-east-1d g2.2xlarge t2.medium t2.micro t2.small us-east-1c Placement: Availability Zone Spread
  • 23. g2.2xlarge t2.small t2.micro t2.medium t2.medium t2.small g2.2xlarge t2.small us-east-1aus-east-1d g2.2xlarge t2.medium t2.micro t2.small us-east-1c Placement: Spread across Zone and Binpack
  • 24. g2.2xlarge t2.small t2.micro t2.medium t2.medium t2.small g2.2xlarge t2.small us-east-1aus-east-1d g2.2xlarge t2.medium t2.micro t2.small us-east-1c Placement: Affinity and Anti-Affinity
  • 26. t2.medium t2.small t2.small us-east-1aus-east-1d t2.medium t2.micro t2.small us-east-1c Placement: Multiple Services on a Cluster
  • 27. t2.medium g2.2xlarge t2.micro t2.small t2.small t2.small g2.2xlarge t2.small t2.small t2.small g2.2xlarge t2.small Placement: Services – Distinct Instances
  • 29. Amazon ECS: Under the Hood ALB ALB AZ 1 AZ 2 user / scheduler Placement Engine Event Stream
  • 31. Handling ECS events with Blox scheduler cluster state service https://blox.github.io/
  • 32. Amazon ECS: Under the Hood ALB ALB AZ 1 AZ 2 user / scheduler Scheduler Cluster State Service Placement Engine Event Stream
  • 33. t2.small t2.small t2.small Blox: Daemon Scheduler t2.small t2.small t2.small scheduler cluster state service t2.small t2.small t2.small
  • 35. Creating Clusters Create an ECS cluster for Blox CF template: https://github.com/blox/blox/blob/dev/deploy/aws/conf/cloudformation_template.json à CloudWatch Event Rule + SQS queue à 3 containers: Daemon Scheduler + Cluster State Service + etcd à REST API exposing the Daemon Scheduler API Create another ECS cluster managed by Blox $ ecs-cli configure --cluster WebCluster $ ecs-cli up --keypair admin --capability-iam --size 3 --instance-type t2.micro Invoke the scheduler API ‘demo-cli’ tool: https://github.com/blox/blox/tree/dev/deploy/demo-cli
  • 36. Listing Task Definitions Grab the ARN for an nginx Task Definition, which the Daemon Scheduler will manage on ‘WebCluster’. $ ./list-task-definitions.py --region eu-central-1 == Blox Demo CLI - List Task Definitions == { "taskDefinitionArns": [ "arn:aws:ecs:eu-central-1:ACCOUNT:task-definition/BloxFramework:2", "arn:aws:ecs:eu-central-1:ACCOUNT:task-definition/nginx:1", "arn:aws:ecs:eu-central-1:ACCOUNT:task-definition/nginx:2" ] }
  • 37. Creating an Environment $ ./blox-create-environment.py --environment WebEnvironment --cluster WebCluster --task-definition "arn:aws:ecs:eu-central-1:ACCOUNT:task- definition/nginx:2" --stack Blox --apigateway --region eu-central-1 == Blox Demo CLI - Create Blox Environment == HTTP Response Code: 200 { "taskDefinition": "arn:aws:ecs:eu-central-1:ACCOUNT:task-definition/nginx:2", "deploymentToken": "17248257-08ec-4438-888f-e0ac28397653", "health": "healthy", "name": "WebEnvironment", "instanceGroup": { "cluster": "arn:aws:ecs:eu-central-1:ACCOUNT:cluster/WebCluster" } }
  • 38. Listing Environments $ ./blox-list-environments.py --stack Blox --apigateway --region eu-central-1 == Blox Demo CLI - List Blox Environments == HTTP Response Code: 200 { "items": [ { "taskDefinition": "arn:aws:ecs:eu-central-1:ACCOUNT:task-definition/nginx:2", "deploymentToken": "17248257-08ec-4438-888f-e0ac28397653", "health": "healthy", "name": "WebEnvironment", "instanceGroup": { "cluster": "arn:aws:ecs:eu-central-1:ACCOUNT:cluster/WebCluster" } } ] }
  • 39. Creating a Deployment $ ./blox-create-deployment.py --environment WebEnvironment --deployment-token "17248257-08ec-4438-888f-e0ac28397653" --stack Blox --apigateway --region eu-central-1 == Blox Demo CLI - Create Blox Deployment == HTTP Response Code: 200 { "status": "pending", "environmentName": "WebEnvironment", "id": "7a05ea99-27a9-4339-a7a6-f4120065aea3", "failedInstances": [], "taskDefinition": "arn:aws:ecs:eu-central-1:613904931467:task-definition/nginx:2” }
  • 40. Listing Deployments $ ./blox-list-deployments.py --environment WebEnvironment --stack Blox --apigateway --region eu-central-1 == Blox Demo CLI - List Blox Deployments == HTTP Response Code: 200 { "items": [ { "status": "completed", "environmentName": "WebEnvironment", "id": "7a05ea99-27a9-4339-a7a6-f4120065aea3", "failedInstances": [], "taskDefinition": "arn:aws:ecs:eu-central-1:ACCOUNT:task-definition/nginx:2" } ] }
  • 41. Scaling a Deployment $ ecs-cli ps Name State Ports TaskDefinition 26313cbe-d929-49de-9cc3-873bf5f32a91/nginx RUNNING nginx:2 98442432-fd5c-434d-b93c-0737bd06aaab/nginx RUNNING nginx:2 ce9bf217-4b34-4f31-9c7b-a8c3402f1ffd/nginx RUNNING nginx:2 $ ecs-cli scale --size 4 --capability-iam $ ecs-cli ps Name State Ports TaskDefinition 26313cbe-d929-49de-9cc3-873bf5f32a91/nginx RUNNING nginx:2 98442432-fd5c-434d-b93c-0737bd06aaab/nginx RUNNING nginx:2 c404ac9a-0948-4cc8-b5b0-2238ccdf4035/nginx RUNNING nginx:2 ce9bf217-4b34-4f31-9c7b-a8c3402f1ffd/nginx RUNNING nginx:2
  • 42. Additional resources ECS customers https://www.mapbox.com/blog/switch-to-ecs/ https://segment.com/blog/rebuilding-our-infrastructure/ Tech articles by Werner Vogels, CTO, Amazon.com http://www.allthingsdistributed.com/2014/11/amazon-ec2-container-service.html http://www.allthingsdistributed.com/2015/04/state-management-and-scheduling-with-ecs.html http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html Blog articles & videos https://aws.amazon.com/blogs/compute/amazon-ec2-container-service-at-aws-reinvent-2016-wrap-up/ https://aws.amazon.com/blogs/compute/continuous-deployment-to-amazon-ecs-using-aws-codepipeline- aws-codebuild-amazon-ecr-and-aws-cloudformation/
  • 43. Thank you! Julien Simon Principal Technical Evangelist, AWS @julsimon