This document summarizes the key aspects of AWS orchestration for a company processing 6 billion requests per day across multiple services and regions. It discusses auto-scaling groups (ASGs) across 750 servers in 2 regions and 5 availability zones, with over 30 ASGs handling more than 20 API integrations. It also describes a homemade Redis autoscaling using master-slave replication across regions on spots and on-demand instances, handling 1.2 million operations per second. An event-driven architecture is implemented using a ØMQ mesh pipeline across ASGs for unidirectional data flow of 100k messages per second.

1. AWS guerrilla orchestration

2. KEY FIGURES
• 100k qps peak
• 6B requests/day
• 8 TB/day
• More than 20 services
• 750 servers peak
• 2 Regions (eu-west-1 and us-east-1)
• 5 AZs
• 30 ASGs
• Over 20 API integrations (Google, Twitter, Facebook, AppNexus, eBay, …)
What do we do (engineers perspective)

3. EC2 API based auto-discovery

4. WHY
• Cross region replication (EU and US)
• Cost (leverage spots)
• HyperLogLog
• Writable slaves (for set operations)
• Centralized monitoring and logging
Homemade Redis autoscaling

5. Our Redis structure
HOW
• Master EU
• Slaves EU (2-8)
• Replication over VPN
• Master US
• Slaves US (4-21)
• 1.2M ops peek

6. WE CARE ABOUT THE COSTS
• Two ASGs per region (on demand + spot)
• Slaves only in ASG
• Spot scales more aggressively
• All ASGs in one region behind same ELB
• ELB with TCP load balancing
• Jenkins job to monitor the crash of spot market
Deployment strategy

7. WHY NOT
• 1-2ms penalty per request
• Long lasting connections
• New machines don’t do anything
• Cross AZ requests add more latency
• Doesn’t consider replication
Going through ELB

8. DISTRIBUTED REDIS CONNECTION BALANCING
• If anything fails fall back to ELB
• Get the AZ for current host using AWS meta-data service
• Get Redis instances from the ELB
• Use instances from the same AZ and fall back to other AZ
• Use only running, healthy and replicated instances
• Check current number of clients connected and ops on each selected Redis
• Pick a Redis based on biased distribution and connect to it
Sneak behind ELB

9. Client Redis connection lifecycle

10. ØMQ mesh pipeline

11. OUR PIPELINE
• Unidirectional data flow
• Multiple ASG service layers
• Machines come and go all the time
• CPU based scaling
• 6 billion messages per day
• 100k messages per second peek time
Event driven architecture

12. ØMQ
WHY
• Connect your code in any language, on any platform
• Carries messages across inproc, IPC, TCP, TIPC, multicast
• Smart patterns like request-reply, pub-sub, push-pull
and router-dealer
• High-speed asynchronous I/O engines, in a tiny library
• Build any architecture: centralized, distributed,
small or large
• Smart handling of establishing connections
and reconnecting

13. HOW
• Define your network topology
using subnets in VPC
• Using subnets in ASGs you ensure
that you know where service will
potentially reside
• HINT: Don’t make a mess, there are
enough subnets to spare
Placement to the rescue

14. CONNECT TO WHERE THE SERVER WILL BE
Mesh (or was it mess) architecture

15. EC2 API based auto-discovery
• No maintenance
• Handles health checks
• Matches your deployment perfectly
• Adapts to changes fast
• Has unknown/unscalable API limit :(
ØMQ mesh pipeline
• Quick and dirty to setup
• Small and fast
• Queue is local to machines
• Limited scale (we tested up to 762 servers)

16. Thank you for your attention.
@utvara