Upcoming SlideShare
Advanced AWS techniques from the trenches of the Enterprise – Sourced Group
- 1. Advanced AWS Patterns from the trenches of the enterprise
- 2. John Painter Principal Consultant john.painter@sourcedgroup.com Brent Harrison Consultant brent.harrison@sourcedgroup.com OUR TEAM TODAY SYDNEY | TORONTO | VANCOUVER | KELOWNA
- 3. CONSULTING Banks Aviation Telecom FinTech Media Healthcare Smartphone Manufacturer Utilities
- 4. • Behind the firewall • BYO AWS Account • Guaranteed single tenancy • Multi-cloud options • Customer-controlled encryption • Customer retains custody of all data Over half a petabyte per year of Splunk throughput under management ENGINEERED SERVICES
- 5. PATTERN 1 – AUTO HEALING GEN2 AUTOMATED HEALING OF SINGLE INSTANCES WITH DEEP HEALTH CHECKING
- 6. THE BASIC AUTO-HEALING PATTERN 1. Create ASG with min:max of 1:1 2. Elastic Load Balancer (ELB) provides deep health checking != EC2 Auto Recovery Top Tip: Deep Health Check 1. Script that checks multiple variables (eg: process + disk space + memory) and opens/closes a port via Netcat 2. Set the ELB to “port” type check Auto Scaling Group min: 1, max: 1
- 7. There are strong fiscal motivators to reduce tier-1 operational costs via the use of automated healing actions
- 8. ASGS, ETH0, AND STATIC IPS • LOTS of “cloud” applications want static networks • ASG instances receive dynamic IPs • Add a secondary interface (Elastic Network Interface – ENI) which maintains a fixed network address Auto Scaling Group min: 1, max: 1 Other cluster members/users “Re-mappable” ENI
- 9. THE PREVIOUS APPROACH 1. Virtual Private Cloud (VPC) with large subnets (eg:>/24) 2. ASG with min:max of 1:1 3. Scripts call EC2 API on boot to “bring” a re-mappable interface to the instance Runs in the operating system, simple to pause apps for interface Lots of upstream/deployment co-ordination required Maintain support for multiple operating systems Incompatible with the increasing number of AWS Marketplace offerings Prone to failure if AWS API is under duress (which is also probably when you really want to be healing!)
- 10. ONE ALTERNATIVE Auto Scaling Group min: 1, max: 1 Other cluster members/users ASG Notifications SQS/SNS Lambda AWS CLI Slow Prone to AWS API/backplane duress Does not understand the state of the operating system
- 11. GOING BACK TO FIRST PRINCIPLES • What hands out IP addresses in AWS? –DHCP (VPC DHCP Options Group) • Where does the range of IPs come from? –Subnet size • Can we reduce the number of IPs available from DHCP to 1? –Provision an ENI in the subnet -> 1 less IP (for FREE!) –Provision lots of ENIs in a subnet and there will only be 1 IP left
- 12. AUTO-HEAL GEN2 1. Create a Subnet for the Auto-Heal node (at the moment /28 is the smallest) 2. Create enough ENIs to remove all but 1 IP from DHCP 3. Create the normal ASG with min:max 1:1 4. Create the ELB with deep health checking as per normal No scripts, co-ordination, or complexity inside the OS or the deployment framework Fully compatible with the wide range of black-box AMIs from AWS Marketplace ✕Wastes address space (which may not be an issue depending on your network design and integration points)
- 13. Same technique can be used for “fixed clusters”, sets of quorum servers, container systems
- 14. PATTERN 2 - ADVANCED PROXY A SCALABLE , HIGHLY AVAILABLE PROXY WITH ACTIVE DATA CONTROLS AND STATIC IP RANGES
- 15. EC2 with Outbound Internet Access MNAT EIP Public Subnet Private Subnet EC2EC2EC2 Uncontrolled access to the internet Reactive techniques such as VPC flow logs + Lambda are not capable of running in real-time * Diagram simplified for clarity, excludes multi availability zone elements
- 16. • Limitation of VPC: Routes can only reference single interface Active control of traffic HTTP/S inspection ? Non-trivial engineering required Not truly HA Relatively low and finite throughput Prone to EC2 backplane saturation 100s Mb/s per EIP ~HA Transparent Proxy Design Whitelist Blacklist IP List EIP Public Subnet Private Subnet PROXY PROXY EC2EC2EC2 * Diagram simplified for clarity, excludes multi availability zone elements ENI
- 17. Public SubnetPublic Subnet Availability Zone A Auto Scaling Proxy Availability Zone B ASG PROXYPROXY Active control of traffic Actively load balanced Truly HA ≈ Infinite bandwidth Variable public IPs Private Subnet Private Subnet EC2EC2EC2 EC2EC2EC2 “Auto Scaled” EIPs
- 18. Variable edge IPs are undesirable in the enterprise
- 19. Auto Scaling Proxies with Static IPs Active control of traffic Actively load balanced Static external IP addresses ? ≈ Infinite bandwidth requires co- ordination Private SubnetsPrivate Subnets Availability Zone A Availability Zone B ASG EC2EC2EC2 EC2EC2EC2 MNAT PROXY PROXY PROXY PROXY PROXY PROXY Public Subnets MNAT Public Subnets EIP EIP
- 20. Why? .....and hang on, I still see EIPs? Scaling Increments: 10GB @ $42/month/10GB 100’s of Mb/s @ ~$210/month/100’s Mb/s • Provision 50/100/200Gb/s upfront • If you move in increments of +/-100Gb/s, see pattern 3. • Simple, HA, static IP proxies for a relatively low uplift in cost Private SubnetsPrivate Subnets Availability Zone A Availability Zone B ASG EC2EC2EC2 EC2EC2EC2 MNAT PROXY PROXY PROXY PROXY PROXY PROXY Public Subnets MNAT Public Subnets EIP EIP
- 21. Complex Inspection Sandwich • Lots of vendor solutions can now support healing • Some even support scaling • Few support ENI/EIP handling Private SubnetsPrivate Subnets Availability Zone A Availability Zone B EC2EC2EC2 EC2EC2EC2 MNAT INSPECTION SANDWICH Public Subnets MNAT Public Subnets EIP EIP
- 22. PATTERN 3 - AUTO SCALING ANYTHING A TECHNIQUE LEVERAGING EXISTING SERVICES TO AUTOSCALE ALMOST ANYTHING
- 23. The fiscal and operational benefits of Auto Scaling are well understood. Auto Scaling is currently limited to scaling EC2 instances We want to apply scaling to entire solutions, not just EC2
- 24. SCALING CLUSTERS? SCALING CELLS? • Enterprises have many applications that cannot scale on compute alone – Sharded databases – Life Sciences Clusters – Simulation Clusters • Organisations are starting to adopt “Cell Architecture” to account for scale • Auto Scaling Auto Healing Client Example ~8000 instances connected in “rings” of 20 nodes via a cluster protocol + ~1500 Cassandra nodes. 50% variance in daily traffic volume. Ideal use-case for Auto Scale
- 25. THE GENERAL CASE - CELL / SHARD / CLUSTER EC2 Node1 EC2 Node2 EC2 Node-n… CloudFormation Stack Health Check
- 26. THE GENERAL CASE - CELL / SHARD / CLUSTER EC2 Node1 EC2 Node2 EC2 Node-n… CloudFormation Stack Health Check EC2 Node1 EC2 Node2 EC2 Node-n… CloudFormation Stack Health Check
- 27. STEP 1 – INSTRUMENT THE SCALING METRIC … … CloudWatch Custom Metric Number of Users CloudWatch Alarm ScaleUp CloudWatch Alarm ScaleDown
- 28. OPTION 1 – USE LAMBDA ScaleUp ScaleDown … … Number of Users SNS … SNS Build Lambda TeardownL ambda CloudFormation
- 29. WHY NOT LAMBDA? • Duplication of AWS engineering investment • Ongoing cost to maintain cadence with the growing features of Auto Scaling – Scheduled Scaling – Percentile Scaling – Machine Learning Scaling / Predictive Scaling • Lambda still needs a state machine • We don’t have healing
- 30. There are strong fiscal and complexity motivators to use native ASGs
- 31. STEP 2 – “SHADOW” ASG ScaleUp ScaleDown Number of Users … … … Shadow Shadow Shadow Shadow ASG
- 32. STEP 3 – ADD THE CFN LAMBDAS ScaleUp ScaleDown Number of Users … … … Shadow Shadow Shadow Auto Scaling SNS EC2_INSTANCE_LAUNCH Create Stack EC2_INSTANCE_TERMINATE Delete Stack Shadow ASG
- 33. $5.76 per month per stack (Unoptimized)
- 34. Auto Scale Auto Heal
- 35. STEP 4 – HEALTH CHECK THE CLUSTERS ScaleUp ScaleDown Number of Users … … … Shadow Shadow Shadow Auto Scaling SNS EC2_INSTANCE_LAUNCH Create Stack EC2_INSTANCE_TERMINATE Delete Stack Shadow ASG
- 36. HEALING SCENARIO 1 – CLUSTER FAILS ScaleUp ScaleDown Number of Users … … … Shadow Shadow Shadow Auto Scaling SNS EC2_INSTANCE_LAUNCH Create Stack EC2_INSTANCE_TERMINATE Delete Stack Shadow ASG
- 37. HEALING SCENARIO 1 – SHADOW TERMINATED ScaleUp ScaleDown Number of Users … … … Shadow Shadow Shadow Auto Scaling SNS EC2_INSTANCE_LAUNCH Create Stack EC2_INSTANCE_TERMINATE Delete Stack Shadow ASG
- 38. HEALING SCENARIO 1 – ASG IS IMPACTED ScaleUp ScaleDown Number of Users … … Shadow Shadow Shadow ASG Desired: 3 Actual: 2
- 39. HEALING SCENARIO 1 – CLUSTER RESTORED ScaleUp ScaleDown Number of Users … … … Shadow Shadow Shadow Auto Scaling SNS EC2_INSTANCE_LAUNCH Create Stack EC2_INSTANCE_TERMINATE Delete Stack Shadow ASG
- 40. Continuous Delivery for Clusters Blue/Green Updates for Clusters at Huge Scale
- 41. CONTINUOUS DELIVERY FOR CLUSTERS … … … … … … • Using nothing but the ASG capacity, blue/green roll clusters of almost any size • Increment ASG in V2.0, wait for health check, decrement ASG in V1.0 V1.0 V2.0
- 42. AUTO SCALE ANYTHING • Solution works with many non-scaling AWS services • CloudFormation can use Custom Resources to create almost anything • The “Shadow” system only needs the scaling alarms from any CloudWatch metric and a health check endpoint. Decoupled and does not interact with the system in any way.
- 43. Database Throughput ScaleUp Alarm SNS Lambda RDS Read Slave CloudFormation Shadow ASGRDS Read Slave RDS Read Slave ScaleDown Alarm CUSTOM SCALING EXAMPLES
- 44. Number of user sign-ups/logins ScaleUp Alarm SNS Lambda Application Shard CloudFormation Shadow ASGApplication Shard Application Shard ScaleDown Alarm CUSTOM SCALING EXAMPLES
- 45. CPU/Memory ScaleUp Alarm SNS Lambda VMWare Node CloudFormation Shadow ASGVMWare Node VMWare Node ScaleDown Alarm CUSTOM SCALING EXAMPLES
- 46. CPU/Memory ScaleUp Alarm SNS Lambda Other infrastructure platforms CloudFormation Shadow ASG Other infrastructure platforms Other infrastructure platforms ScaleDown Alarm CUSTOM SCALING EXAMPLES
- 47. Number of items in the queue ScaleUp Alarm SNS Lambda Life Sciences Application CloudFormation Shadow ASGLife Sciences Application Life Sciences Application ScaleDown Alarm CUSTOM SCALING EXAMPLES
- 48. Number of planes currently in the air ScaleUp Alarm SNS Lambda Flight Analysis Stack CloudFormation Shadow ASGFlight Analysis Stack Flight Analysis Stack ScaleDown Alarm CUSTOM SCALING EXAMPLES
- 49. Number of door entries ScaleUp Alarm SNS Lambda Trading Stack CloudFormation Shadow ASGTrading Stack Trading Stack ScaleDown Alarm CUSTOM SCALING EXAMPLES
- 50. Order Volume ScaleUp Alarm SNS Lambda Number of robots on station CloudFormation Shadow ASG Number of robots on station Number of robots on station ScaleDown Alarm CUSTOM SCALING EXAMPLES
- 51. Find Out MORE: Visit Us: At our booth or online – www.sourcedgroup.com Careers: www.sourcedgroup.com/careers In the news: • Computerworld (2016): • Foreign Exchange Service OFX Embarks on Cloud Migration • Connecting the Australian Channel (2015): • Meet the Partner who took Qantas to the AWS Cloud • The Australian Business Review (2015): • Greater Buying Power lets Aussie bank on Adobe Experience Manager Our Awards: • AWS – Sydney Partners Summit - Invent & Simplify (2015) • AWS – Global - Customer Obsessed Partner (2014)
- 52. Thank you!
Be the first to comment