Downloaded 36 times
![Avoid Relocating to Old Instance
• Problem:
• Tasks can get rescheduled to another old instance in the ASG that is about to be replaced - so
tasks can get bumped from instance to instance until all instances are replaced
• Solution:
• Deploy services with placement constraint
• This means that a service won’t be placed on an instance that has an attribute named “state” with
value “pre-drain”
• At cluster replacement time, we will stand up all new clusters, place old clusters into the “pre-
drain” state, and terminate the old instances in batches.
• Relocated tasks will only be placed on new instances, avoiding the default “thundering herd”
scenario
placement_constraints = [ {
type: 'memberOf',
expression: 'attribute:state !exists or attribute:state != pre-drain'
}]](https://image.slidesharecdn.com/con314-automatingzero-downtimeproductionclusterupgradesforamazonecs-171201205355/75/Automating-Zero-Downtime-Production-Cluster-Upgrades-for-Amazon-ECS-62-2048.jpg)
Uploaded byMatt Callanan
Automating Zero-Downtime Production Cluster Upgrades for Amazon ECS
The document discusses the strategies for automating zero-downtime production cluster upgrades for Amazon ECS, detailing the technical challenges faced during upgrades and the solutions implemented. It outlines the various approaches taken to ensure smooth transitions, including a custom rolling update automation and the development of a project called 'prism' which effectively minimized downtime during upgrades. Key aspects of the upgrade process include task relocation strategies, pre-flight checks, and the implementation of customized scaling and deployment strategies to avoid service outages.
More Related Content
Similar to Automating Zero-Downtime Production Cluster Upgrades for Amazon ECS
Automating Zero-Downtime Production Cluster Upgrades for Amazon ECS
- 1. Automating Zero-Downtime Production ClusterUpgrades for Amazon ECS
- 2. Matt Callanan Engineering Manager/TechLead “Cloud Acceleration Team” Expedia Brisbane, Australia • mcallanan@expedia.com • linkedin.com/in/matthewcallanan • @mcallana
- 3. Cluster Management • Howto upgrade the EC2 infrastructure underlying a live production Amazon ECS cluster without affecting service availability. • How to safely relocate hundreds of tasks onto new Amazon EC2 instances with zero-downtime to applications.
- 4. Region 1 Region 2Region 3 Region 4 Region 5 Expedia ECS Cluster Statistics 2,600 ECS Services (1,100 Applications) 13,000 Containers 860 EC2 Instances (13 ECS Clusters)
- 5. Region 1 Region 2Region 3 Region 4 Region 5 Expedia ECS Cluster Topology Production Cluster Test Environment Cluster 480 Services 230 Instances
- 6. Production Cluster Visualization 230Instances 480 Services 3,200 Containers c3vis Open Source: https://github.com/ExpediaDotCom/c3vis
- 7. ECS Cluster Creation Cloud Formation Stack EC2Instances Auto Scaling Group Amazon ECS Cluster
- 8. Immutable Servers Amazon-provided BaseAMI Standard Chef cookbook Custom setup baked into AMI ecs-optimized AMI Expedia standard image Docker Config Daemon containers Golden AMI docker ecs-agent
- 9. Immutable Servers ecs-optimized AMI Expediastandard image Docker Config Daemon containers Golden AMI docker ecs-agent ecs-optimized AMI Expedia standard image Docker Config Daemon containers Cluster Instance Custom bootstrap: • ECS Cluster Config • Start ECS Agent, Docker • Cron: Restart ECS agent • Cron: Custom Metrics docker ecs-agent
- 10.
- 11. First Approach • CloudFormationAuto-Scaling Rolling Update
- 12. Default Cloud FormationAuto-Scaling Rolling Update Old Instance New Instance Terminating Instance Active Task Relocated Task Stopped Task
- 13. Default Cloud FormationAuto-Scaling Rolling Update A A Old Instance New Instance Terminating Instance Active Task Relocated Task Stopped Task
- 14. Default Cloud FormationAuto-Scaling Rolling Update A A Old Instance New Instance Terminating Instance Active Task Relocated Task Stopped Task Problem: Tasks stop before they are relocated. Even with tasks on distinct instances, outage can happen if new instances are not pulling images fast enough
- 15. Default Cloud FormationAuto-Scaling Rolling Update A Old Instance New Instance Terminating Instance Active Task Relocated Task Stopped Task
- 16. Default Cloud FormationAuto-Scaling Rolling Update A A Problem: Tasks start on instances about to be terminated Old Instance New Instance Terminating Instance Active Task Relocated Task Stopped Task
- 17. Default Cloud FormationAuto-Scaling Rolling Update A A Old Instance New Instance Terminating Instance Active Task Relocated Task Stopped Task
- 18. Default Cloud FormationAuto-Scaling Rolling Update A A Old Instance New Instance Terminating Instance Active Task Relocated Task Stopped Task
- 19. Default Cloud FormationAuto-Scaling Rolling Update A A Old Instance New Instance Terminating Instance Active Task Relocated Task Stopped Task
- 20. Default Cloud FormationAuto-Scaling Rolling Update A A Old Instance New Instance Terminating Instance Active Task Relocated Task Stopped Task
- 21. Default Cloud FormationAuto-Scaling Rolling Update A A Old Instance New Instance Terminating Instance Active Task Relocated Task Stopped Task
- 22. Default Cloud FormationAuto-Scaling Rolling Update A A Old Instance New Instance Terminating Instance Active Task Relocated Task Stopped Task
- 23. Default Cloud FormationAuto-Scaling Rolling Update A Old Instance New Instance Terminating Instance Active Task Relocated Task Stopped Task
- 24. Default Cloud FormationAuto-Scaling Rolling Update A Old Instance New Instance Terminating Instance Active Task Relocated Task Stopped Task
- 25. Default Cloud FormationAuto-Scaling Rolling Update A Old Instance New Instance Terminating Instance Active Task Relocated Task Stopped Task
- 26. Default Cloud FormationAuto-Scaling Rolling Update A Old Instance New Instance Terminating Instance Active Task Relocated Task Stopped Task
- 27. Default Cloud FormationAuto-Scaling Rolling Update Old Instance New Instance Terminating Instance Active Task Relocated Task Stopped Task Problem: Service “A” experiences downtime
- 28. Default Cloud FormationAuto-Scaling Rolling Update A Old Instance New Instance Terminating Instance Active Task Relocated Task Stopped Task
- 29. Default Cloud FormationAuto-Scaling Rolling Update A Old Instance New Instance Terminating Instance Active Task Relocated Task Stopped Task
- 30. Default Cloud FormationAuto-Scaling Rolling Update A A Old Instance New Instance Terminating Instance Active Task Relocated Task Stopped Task
- 31. Default Cloud FormationAuto-Scaling Rolling Update A A Old Instance New Instance Terminating Instance Active Task Relocated Task Stopped Task
- 32. Default Cloud FormationAuto-Scaling Rolling Update A A Old Instance New Instance Terminating Instance Active Task Relocated Task Stopped Task Problem: Tasks bunch up on first few new instances
- 33. First Approach: ProblemSummary • CloudFormation AutoScaling Asynchrony • Rolling Update not granular enough • Outside our control • Rollbacks with no ability for manual intervention • Tasks stop before they are relocated • Service outage can happen if new instances are not pulling images fast enough • Tasks are not evenly spread across new instances CloudFormation Auto Scaling
- 34. Second Approach • CustomRolling Update automation • Programmatic Update Script using Ruby SDK • But still took 10 (nail-biting) hours to update a cluster with 100 instances • Any issues during the process required manual intervention Auto Scaling
- 35. Third Approach • “PRISM” •Project Replaced in Sixty Minutes
- 36. “PRISM” Goals • Zero-downtimefor applications as their workloads get relocated onto new instances Safety • Complete as fast as possible Speed • Quickly retreat back to known-good state if anything goes wrong Rollbackable • Resumeable if anything goes wrong Idempotent • Drain in batches to prevent burden on Docker registry and network • Avoid having tasks relocated to instances about to be drained Avoid “thundering herd” scenario
- 37. “PRISM” Phases Phase 1:Expand Phase 2: Relocate Tasks Phase 3: Clean Up
- 38. Zero-Downtime Cluster Updates Cloud Formation Stack EC2Instances Auto Scaling group Amazon ECS Cluster
- 39. Amazon ECS Cluster Cloud Formation Stack EC2 Instances Auto Scalinggroup Cloud Formation Stack EC2 Instances Auto Scaling group Zero-Downtime Cluster Updates Phase 1: Expand Cluster
- 40. Zero-Downtime Cluster Updates Phase2: Relocate Tasks Amazon ECS Cluster Cloud Formation Stack EC2 Instances Auto Scaling group Cloud Formation Stack EC2 Instances Auto Scaling group Draining…
- 41. Zero-Downtime Cluster Updates Phase3: Clean Up Amazon ECS Cluster Cloud Formation Stack EC2 Instances Auto Scaling group Cloud Formation Stack EC2 Instances Auto Scaling group
- 42. Zero-Downtime Cluster Updates Phase3: Clean Up Amazon ECS ClusterCloud Formation Stack EC2 Instances Auto Scaling group
- 43. Old Instance New Instance DrainingInstance A Active Task Relocated Task “DRAINING” Task A Blue Auto-Scaling Group Blue CFN Stack Cluster Update – Phase 1: Expand
- 44. AA Blue Auto-Scaling Group BlueCFN Stack Disable Auto-Scaling Processes Cluster Update – Phase 1: Expand Old Instance New Instance Draining Instance Active Task Relocated Task “DRAINING” Task
- 45. AA Blue Auto-Scaling Group BlueCFN Stack Green Auto-Scaling Group Green CFN Stack Disable Auto-Scaling Processes Disable Auto-Scaling Processes Cluster Update – Phase 1: Expand Old Instance New Instance Draining Instance Active Task Relocated Task “DRAINING” Task
- 46. AA Blue Auto-Scaling Group BlueCFN Stack Green Auto-Scaling Group Green CFN Stack state = ‘pre-drain’ Disable Auto-Scaling Processes Disable Auto-Scaling Processes Cluster Update – Phase 1: Expand Old Instance New Instance Draining Instance Active Task Relocated Task “DRAINING” Task
- 47. AA Blue Auto-Scaling Group BlueCFN Stack Green Auto-Scaling Group Green CFN Stack Disable Auto-Scaling Processes Cluster Update – Phase 2: Relocate Tasks (batches of 3 instances) Old Instance New Instance Draining Instance Active Task Relocated Task “DRAINING” Task Disable Auto-Scaling Processes state = ‘pre-drain’
- 48. Blue Auto-Scaling Group BlueCFN Stack Green Auto-Scaling Group Green CFN Stack Disable Auto-Scaling Processes Cluster Update – Phase 2: Relocate Tasks (batches of 3 instances) Old Instance New Instance Draining Instance Active Task Relocated Task “DRAINING” Task AA state = ‘pre-drain’ Disable Auto-Scaling Processes
- 49. Blue Auto-Scaling Group BlueCFN Stack Green Auto-Scaling Group Green CFN Stack Disable Auto-Scaling Processes Cluster Update – Phase 2: Relocate Tasks (batches of 3 instances) Old Instance New Instance Draining Instance Active Task Relocated Task “DRAINING” Task A AAA state = ‘pre-drain’ Disable Auto-Scaling Processes
- 50. Blue Auto-Scaling Group BlueCFN Stack A A Green Auto-Scaling Group Green CFN Stack Disable Auto-Scaling Processes Cluster Update – Phase 2: Relocate Tasks (batches of 3 instances) Old Instance New Instance Draining Instance Active Task Relocated Task “DRAINING” Task AA state = ‘pre-drain’ Disable Auto-Scaling Processes
- 51. Blue Auto-Scaling Group BlueCFN Stack A A Green Auto-Scaling Group Green CFN Stack Disable Auto-Scaling Processes Cluster Update – Phase 2: Relocate Tasks (batches of 3 instances) Old Instance New Instance Draining Instance Active Task Relocated Task “DRAINING” Task state = ‘pre-drain’ Disable Auto-Scaling Processes
- 52. Blue Auto-Scaling Group BlueCFN Stack A Green Auto-Scaling Group Green CFN Stack Disable Auto-Scaling Processes Cluster Update – Phase 2: Relocate Tasks (batches of 3 instances) Old Instance New Instance Draining Instance Active Task Relocated Task “DRAINING” Task A state = ‘pre-drain’ Disable Auto-Scaling Processes
- 53. Blue Auto-Scaling Group BlueCFN Stack A Green Auto-Scaling Group Green CFN Stack Disable Auto-Scaling Processes Cluster Update – Phase 2: Relocate Tasks (batches of 3 instances) Old Instance New Instance Draining Instance Active Task Relocated Task “DRAINING” Task A state = ‘pre-drain’ Disable Auto-Scaling Processes
- 54. Blue Auto-Scaling Group BlueCFN Stack A A Green Auto-Scaling Group Green CFN Stack Disable Auto-Scaling Processes Cluster Update – Phase 2: Relocate Tasks (batches of 3 instances) Old Instance New Instance Draining Instance Active Task Relocated Task “DRAINING” Task state = ‘pre-drain’ Disable Auto-Scaling Processes
- 55. Blue Auto-Scaling Group BlueCFN Stack A A Green Auto-Scaling Group Green CFN Stack Disable Auto-Scaling Processes Cluster Update – Phase 2: Relocate Tasks (batches of 3 instances) Old Instance New Instance Draining Instance Active Task Relocated Task “DRAINING” Task state = ‘pre-drain’ Disable Auto-Scaling Processes
- 56. A A Green Auto-Scaling Group GreenCFN Stack Disable Auto-Scaling Processes Cluster Update – Phase 3: Clean Up Old Instance New Instance Draining Instance Active Task Relocated Task “DRAINING” Task
- 57. A A Green Auto-Scaling Group GreenCFN Stack Resume Auto-Scaling Processes Cluster Update – Phase 3: Clean Up Old Instance New Instance Draining Instance Active Task Relocated Task “DRAINING” Task
- 58.
- 59. Pre-flight Checks • BeforeCluster Create/Update • Check # instances available of target instance type • Check IP addresses available in target subnets • Check EBS volume space available of target volume type
- 60. Drain on Scale-downwith Lifecycle Lambda • Lambda triggered by AutoScaling EC2_INSTANCE_TERMINATE SNS events • Updates instance state to “DRAINING” • ASG has 30-minute heartbeat to keep instance in Terminating:Wait state for 30mins • Allows ECS to safely relocate any tasks that are part of a service to another instance
- 61. AutoScaling Lifecycle Hook resource"ECSContainerInstanceTerminating", Type: "AWS::AutoScaling::LifecycleHook", Properties: { AutoScalingGroupName: ref("ECSAutoScalingGroup"), LifecycleTransition: "autoscaling:EC2_INSTANCE_TERMINATING", DefaultResult: "CONTINUE", RoleARN: ref("LifecycleHookRoleARN"), NotificationTargetARN: ref("ECSAutoscalingLifecycleTopic"), HeartbeatTimeout: 1800 # 30 minutes },
- 62. Avoid Relocating toOld Instance • Problem: • Tasks can get rescheduled to another old instance in the ASG that is about to be replaced - so tasks can get bumped from instance to instance until all instances are replaced • Solution: • Deploy services with placement constraint • This means that a service won’t be placed on an instance that has an attribute named “state” with value “pre-drain” • At cluster replacement time, we will stand up all new clusters, place old clusters into the “pre- drain” state, and terminate the old instances in batches. • Relocated tasks will only be placed on new instances, avoiding the default “thundering herd” scenario placement_constraints = [ { type: 'memberOf', expression: 'attribute:state !exists or attribute:state != pre-drain' }]
- 63. Task Definition PlacementConstraint
- 64. Task Definition PlacementConstraint • “state” is a custom ECS Instance Attribute • By default, the “state” attribute doesn’t exist on instances • Set only to “pre-drain” during cluster update • Prevents ECS scheduling tasks on instances that are about to be drained • Removed from instance only in case of rollback of cluster update
- 65. Instance Launch Considerations •Worked with AWS Auto-Scaling team to enable more appropriate Auto-Scaling “Launch Rate” • Start the ECS agent with exponential backoff • Throttle on container instance registration rate = 1 per second/60 max per minute
- 66. Gotchas • Found anumber of apps don't relocate easily • Slows down prod upgrades and old stack decommissions
- 67. Related AWS re:invent 2017:Going Big with Containers: Customer Case Studies of Large-Scale (ENT209) • https://www.youtube.com/watch?v=L3l_ZiYRrks • Covers full scope of Expedia’s ECS deployment automation platform