Container-based Microservices DevOps in AWS

Container-based Microservices DevOps in
AWS
How Perfecto Did it and What We Learned So Far
© 2018, Perfecto Mobile Ltd. All Rights Reserved.

About Perfecto
1/10/2018 2© 2018, Perfecto Mobile Ltd. All Rights Reserved.

How We Started
We started 11 years ago with developing monolith
servers in our own DCs
We were moving slowly…

But Then we Heard Some
Buzzwords
And we decided we want to
move faster and do more
impact on the company
business

Big Change
Waterfall
Monolith servers
Deployment in DC
Dependencies
Agile
Microservices
Cloud
Autonomous teams

The 3 Components of the Change
technology
methodologyculture

Autonomous Teams
DevOps
Dev
QA
Development
Continuous integration
Continuous deployment
Monitoring
Budget control

Technologies we Use (partial
list…)

Why we Chose ECS for Container
Orchestration
• We were new to the containers world,
but we understood container
orchestration is a key decision
• We looked at ECS, Kubernetes,
Swarm and other alternatives.
• ECS seemed best in terms on time to
value

Our First Microservice
• Deployed in ECS
• ELB + ECS tasks = ECS
service
• EC2 instances are
managed in an Auto
Scaling Group
• Service Discovery using
Route53
• Task per EC2 instance
(ELB static port
limitation)

Decisions we Took (1)
• Deploying in a single availability zone
• One of those decisions you regret -
overhead of changing it grows with
time

Decisions We took (2)
• Single VPC for all teams
• Seems natural – it’s network, right?
• Pros
• Less work for teams
• Simpler to move services between teams
• Cons
• Dependency between teams. Who owns the VPC?
• Simpler to take shortcuts (e.g. use VPN to DC)
• Budget control is more difficult - no option for account per team (need
to tag all services)

Decisions we took (3)
• ECS cluster per… what?
• Options
• One cluster to rule them all
• Cluster per service (group of microservices)
• Cluster per team
• We let our teams decide between the two last options
• No dependencies between teams
• Better budget control
• Reduce blast radius of ECS cluster issues (more on that soon)

Infrastructure as code is the only way
to go
• We (try to) do everything (except for very small
and initial POCs) with CloudFormation
• Every time you do a change in UI, CLI or API without
CloudFormation – think again
• CloudFormation templates stored in Git
• CloudFormation invoked by Jenkins
• We maintain shared CloudFormation templates
used by all teams to create ECS clusters, services
and more.

Working with CloudFormation
• There is a learning curve
• Templates can become long and unreadable
• Split to sub-templates
• Consider generating templates
• CloudFormation behavior can be surprising, but
it is consistent
• Practice in product-like environments (dev/staging)
• Using the UI is dangerous
• Automate all CloudFormation invocations
• Read-only access to UI
• Protect your stacks using stack policies

Moving to ALBs
• ALB – Application Load Balancer, can (should) replace ELB
• Why
• Cost - 1 ALB can replace X ELBs - Less expensive for clusters with
large number of services
• Dynamic port management – Allows deploying multiple services on one
EC2 instance, more flexibility
• ELB is (kind of) legacy – e.g. not supported in Fargate
• Routes requests to backend containers based on request path
rules
• Challenge with ALBs – no URL rewrite in rules. If if you have no
control on the request path in the deployed services, you will
need a reverse proxy.

ELB vs ALB

What about logs?
• We’re using CloudWatch logs
• Note perfect, but very simple to integrate with
anything in AWS
• Container logs
• Standard container logs can be sent to
CloudWatch – that is easy, supported natively in
Docker
• To take application log files to CloudWatch – we’re
using a ”satellite container” (AKA sidecar) per task
- https://github.com/moshebs/docker-awslogs
• ELB/ALB access logs:
• Sent to S3, natively supported by ELB/ALB
• CloudWatch event from S3  Lambda that parses
the logs and pushes them to CloudWatch logs

Logs

Monitoring with Prometheus

Dashboards with Grafana

Monitoring in Perfecto
• Each team owns their own monitoring system
• Deployment
• Maintenance
• Building dashboards
• Getting alerts, usually in Slack
• Deployed using CloudFormation
• All teams use the same templates
• Coniguration using sidecar containers

What we monitor
• EC2 instance metrics – by deploying Prometheus
node_exporter on the EC2 instances
• Application metrics
• If metrics are shared between the microservice nodes – scrape through
LB
• Otherwise – scrape each microservice tasks (how do you find them?
Next slide…)
• 3rd party (Mongo, RabbitMQ, Redis, e.g.) – standard open-
source exporters
• CloudWatch metrics – using cloudwatch_exporter (but be
careful, pulling metrics from CloudWatch is expensive!)

Monitoring Architecture

Scraping ECS Tasks
• The challenge:
• Prometheus needs to know where each task runs, and what port to use
for scraping
• But Prometheus supports filtering EC2 instances by tags only
• ECS decide which task goes where
• The solution – a container that dynamically tags EC2 instances
according to ECS tasks running on them

Scraping ECS Tasks

ECS Biggest Challenge
• The integration between ECS
and Auto Scaling Group is not
perfect
• ASG changes ignore ECS tasks
• Let’s look at 2 examples

Auto Scaling Group Downscale
VM1 VM2 VM3 VM4 VM5 VM6 VM7

Upgrade of ECS-Optimized AMI
VM1 VM2 VM3 VM4VM5
Auto Scaling Group

Simple Workaround
• You can control when the EC2 instance sends the “I’m ready”
signal to CloudFormation (in fact you must send it in the
userdata)
• Add a sleep, to allow the ECS task to start
• Helps with the AMI upgrade scenario only
• Upgrades are slower, but a bit safer
• In practice – this really helped us

Better Solution
• Auto Scaling Group has life cycle hooks
• We can add a hook to prevent VM shutdown until the task in the new VM is ready.
VM1 VM2 VM3 VM4VM5
Auto Scaling Group
Shutdown
Hook
SNS
ECS
Deregister
VM2
Wait
for task
Complete
lifecycle
action

But the truth is…
• We don’t want to manage VMs at all
• We just want to deploy containers over CPU and memory
• Enter Fargate – serverless containers
• We plan to try it soon, but we’re still missing
• Storage attachment
• Availability outside of us-east-1

Container-based Microservices DevOps in AWS

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Container-based Microservices DevOps in AWS

Similar to Container-based Microservices DevOps in AWS (20)

Recently uploaded

Recently uploaded (20)

Container-based Microservices DevOps in AWS

Editor's Notes