Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AWS ECS Meetup Talentica


Published on

This presentation is part of Pune Cloud Engineers meetup hosted by Talentica Software India Private Limited

Published in: Internet
  • Be the first to comment

AWS ECS Meetup Talentica

  1. 1. Container Cluster Management with AWS ECS Anshul Patel Talentica DevOps Engineering Team
  2. 2. What is Docker container ?  Open platform to build, ship and run distributed applications  Groups application with all the dependencies, shared libs  Compared to Traditional Virtualized system, which encapsulates OS + Kernel, Docker has same Kernel Space  Isolation between different containers is done through Namespaces, Cgroups, Network Interfaces  Current Docker Engine Stable version 1.11.2 (2016-05-31)  Default Engine Execution Driver : libcontainer  Docker uses libcontainer to manipulate Namespaces, Cgroups, Network Interfaces
  3. 3. Why Docker ?  Lightweight, Open and Secure  Portable and efficient in comparison to VM  Accelerates Developer onboarding  Empower Developer creativity  Eliminates Environmental Inconsistencies  Ability to scale quickly  Reduces time to market of your application
  4. 4. Why Docker ?
  5. 5. Why Container Cluster Management System ?  Provides clustering layer for controlling the deployment of your containers onto the underlying hosts  Manages container lifecycle within the cluster  Scheduling Containers across the cluster  Abstracting developers away from underlying machines  Scaling containers  Increases robustness and resilience of distributed containerized application
  6. 6. What is AWS ECS (EC2 Container Service) ?  Highly scalable, fast, container management service from Amazon Web Services.  Easily run, stop and manage Docker containers on cluster of Amazon EC2 instances  Schedules the placement of Docker containers across your cluster based on resource needs, availability and requirements  Eliminates need for you to operate your own cluster management systems or write wrappers/logic for scaling your underlying infrastructure
  7. 7. Components of ECS  Cluster - Logical group of container instances  Container Instance - Runs ECS agents and is registered to cluster  Task Definition - Description of application to be deployed  Scheduler - Method of placing task on container instance  Task - An instantiation of task definition running on container instance  Service - Runs and maintains predefined tasks simultaneously  Container - Docker Container created during task instantiation
  8. 8. ECS Architecture
  9. 9. ECS Architecture Key Components  Agent Communication Service - Gateway between ECS agents and ECS backend cluster management engine  API - Provides cluster state information  Cluster Management Engine - Provides cluster coordination and state management  Key/Value Store - It is used to storing cluster state information
  10. 10. ECS Architecture  It isn’t a blackbox, runs on your own EC2(Container) instances  ECS cluster is collection of EC2(Container) Instances  ECS agent is installed on each of EC2(Container) Instances  ECS agent registers instance to centralised ECS service  ECS agent handles incoming requests for container deployment  ECS agent handles the lifecycle of container
  11. 11. Creating ECS Cluster  Cluster can be created using AWS Console, alternatively you can create using aws ecs CLI  AWS Container Instance Requirement  Linux Kernel > = 3.10  Docker > = 1.5.0  ECS container agent  Nanny process to monitor ECS agent (For eg: ecs-init )  Amazon ECS optimized AMI is preconfigured with these requirements
  12. 12. Launching EC2 container Instances Points to remember  ec2InstanceRole must be created  If the EC2 instance is required to be part of pre-defined cluster, following should be added in user data #/bin/bash echo ECS_CLUSTER=your_cluster_name >> /etc/ecs/ecs.config  During the launch of AMI, it will consist of two EBS, one for OS and another for Docker’s use.
  13. 13. ECS Task Definition  Task definition is 1 or more container definitions  It defines  Docker Images to use  Port and Drive Volume Mapping  CPU and memory to use with container  Whether containers are linked  Environmental variable which is required to be passed to container  Whether task should continue if container finishes or fails
  14. 14. ECS Scheduler  By default, AWS supports two ways for scheduling  Running Tasks  Services  Apart from the above, StartTask API of ECS can be used to write custom scheduler. Currently Amazon has developed PoC scheduler for Mesos
  15. 15. ECS Running Tasks  Instantiation of Task Definition  Task can be either short lived or long lived. (For eg: event-driven- data task or batch jobs)  Architecturally all containers belonging to same task, have to run on the same machine. In order to run container on different machine, new task should be created.  Co-location of containers can be achieved by grouping them in the same task
  16. 16. ECS services  Allows you to run and maintain a specified/desired number of instances ( For eg : webservers )  If any tasks should fail or stop for any reason, ECS service scheduler launches another instance of your task definition to maintain desired count
  17. 17. Deploying ECS Cluster  Create a Load Balancer  Create a Launch configuration  Create a Autoscaling group, which specifies the desired number of instances  Create a task definition  Create a service
  18. 18. ECS Container Instance Autoscaling  ECS provides cluster-level parameters which can give the cluster utilization statistics  MemoryReservation - Current % of reserved memory by cluster  MemoryUtilization - Current % of utilized memory by cluster  CPUReservation - Current % of reserved CPU by cluster  CPUUtilization - Current % of utilized CPU by cluster  CloudWatch Alarms on the above parameters enables to Scale Up/Down the ECS cluster
  19. 19. ECS Task Autoscaling  ECS also provides the facility to scale up the tasks in the service.  Tasks can be autoscaled on following ECS service parameters  CPUUtilization - Current % CPU utilization by ECS service  MemoryUtilization - Current % Memory Utilization by ECS service
  20. 20. EC2 Container Registry  AWS managed Docker container registry  Stores and Manages Docker Images  Hosts images in a highly available and scalable architecture  It is integrated with ECS  No upfront fee, pay only for the data stored
  21. 21. Key Advantages of ECS Service  ECS is monitoring status of Docker containers, so if it goes down it will be deployed automatically  ASG is monitoring your EC2 instances, if one instance goes down, it will spawn new instance  ECS can do zero-downtime deployments of new version ( Blue - Green deployments )  Updation requires two things  Create new revision of task  Update the revision in the service
  22. 22. Key Challenges  No direct relation between EC2 Autoscaling and Task Autoscaling  ELB works with 1:1 port mapping, so if the number of tasks is equal to the number of instances, scheduler will not attempt to deploy another task  Service Discovery
  23. 23. ECS Limits Resource Default Limit Number of clusters per region, per account 1000 Number of container instances per cluster 1000 Number of services per cluster 500
  24. 24. ECS Limits Resource Default Limit Number of load balancers per service 1 Number of tasks per service (the desired count) 1000 Number of tasks launched (count) per run- task 10 Number of container instances per start- task 10 Throttle on container instance registration rate 1 per second / 60 max per minute Task definition size limit 32 KiB Task definition max containers 10 Throttle on task definition registration rate 1 per second / 60 max per minute
  25. 25. Thanks & Questions ?