1.
Advanced AWS Patterns from the
trenches of the enterprise

2.
John Painter
Principal Consultant
john.painter@sourcedgroup.com
Brent Harrison
Consultant
brent.harrison@sourcedgroup.com
OUR TEAM TODAY
SYDNEY | TORONTO | VANCOUVER | KELOWNA

3.
CONSULTING
Banks Aviation
Telecom
FinTech
Media
Healthcare
Smartphone
Manufacturer
Utilities

4.
• Behind the firewall
• BYO AWS Account
• Guaranteed single tenancy
• Multi-cloud options
• Customer-controlled encryption
• Customer retains custody of all data
Over half a petabyte per year of Splunk
throughput under management
ENGINEERED SERVICES

5.
PATTERN 1 – AUTO HEALING GEN2
AUTOMATED HEALING OF SINGLE INSTANCES WITH DEEP
HEALTH CHECKING

6.
THE BASIC AUTO-HEALING PATTERN
1. Create ASG with min:max of 1:1
2. Elastic Load Balancer (ELB) provides
deep health checking
!= EC2 Auto Recovery
Top Tip: Deep Health Check
1. Script that checks multiple variables (eg:
process + disk space + memory) and
opens/closes a port via Netcat
2. Set the ELB to “port” type check
Auto Scaling Group
min: 1, max: 1

7.
There are strong fiscal motivators to reduce tier-1 operational costs
via the use of automated healing actions

8.
ASGS, ETH0, AND STATIC IPS
• LOTS of “cloud” applications want static
networks
• ASG instances receive dynamic IPs
• Add a secondary interface (Elastic
Network Interface – ENI) which maintains
a fixed network address
Auto Scaling Group
min: 1, max: 1
Other cluster
members/users
“Re-mappable” ENI

9.
THE PREVIOUS APPROACH
1. Virtual Private Cloud (VPC) with large subnets (eg:>/24)
2. ASG with min:max of 1:1
3. Scripts call EC2 API on boot to “bring” a re-mappable interface to the instance
 Runs in the operating system, simple to pause apps for interface
Lots of upstream/deployment co-ordination required
Maintain support for multiple operating systems
Incompatible with the increasing number of AWS Marketplace offerings
Prone to failure if AWS API is under duress (which is also probably when you really want to
be healing!)

10.
ONE ALTERNATIVE
Auto Scaling Group
min: 1, max: 1
Other cluster
members/users
ASG Notifications SQS/SNS Lambda
AWS CLI
 Slow
 Prone to AWS API/backplane duress
 Does not understand the state of the operating system

11.
GOING BACK TO FIRST PRINCIPLES
• What hands out IP addresses in AWS?
–DHCP (VPC DHCP Options Group)
• Where does the range of IPs come from?
–Subnet size
• Can we reduce the number of IPs available from DHCP to 1?
–Provision an ENI in the subnet -> 1 less IP (for FREE!)
–Provision lots of ENIs in a subnet and there will only be 1 IP left

12.
AUTO-HEAL GEN2
1. Create a Subnet for the Auto-Heal node (at the moment /28 is the smallest)
2. Create enough ENIs to remove all but 1 IP from DHCP
3. Create the normal ASG with min:max 1:1
4. Create the ELB with deep health checking as per normal
No scripts, co-ordination, or complexity inside the OS or the deployment framework
Fully compatible with the wide range of black-box AMIs from AWS Marketplace
✕Wastes address space (which may not be an issue depending on your network design
and integration points)

13.
Same technique can be used for “fixed clusters”, sets of quorum
servers, container systems

14.
PATTERN 2 - ADVANCED PROXY
A SCALABLE , HIGHLY AVAILABLE PROXY WITH ACTIVE DATA
CONTROLS AND STATIC IP RANGES

15.
EC2 with Outbound Internet Access
MNAT
EIP
Public Subnet
Private Subnet
EC2EC2EC2
 Uncontrolled access to the internet
 Reactive techniques such as VPC flow
logs + Lambda are not capable of
running in real-time
* Diagram simplified for clarity, excludes multi availability zone elements

16.
• Limitation of VPC: Routes can only
reference single interface
 Active control of traffic
 HTTP/S inspection
? Non-trivial engineering required
 Not truly HA
 Relatively low and finite throughput
 Prone to EC2 backplane saturation
 100s Mb/s per EIP
~HA Transparent Proxy Design
Whitelist
Blacklist
IP List
EIP
Public Subnet
Private Subnet
PROXY PROXY
EC2EC2EC2
* Diagram simplified for clarity, excludes multi availability zone elements
ENI

17.
Public SubnetPublic Subnet
Availability Zone A
Auto Scaling Proxy
Availability Zone B
ASG
PROXYPROXY
 Active control of traffic
 Actively load balanced
 Truly HA
 ≈ Infinite bandwidth
 Variable public IPs
Private Subnet Private Subnet
EC2EC2EC2
EC2EC2EC2
“Auto Scaled” EIPs

18.
Variable edge IPs are undesirable in the enterprise

19.
Auto Scaling Proxies with Static IPs
 Active control of traffic
 Actively load balanced
 Static external IP addresses
? ≈ Infinite bandwidth requires co-
ordination
Private SubnetsPrivate Subnets
Availability Zone A Availability Zone B
ASG
EC2EC2EC2
EC2EC2EC2
MNAT
PROXY
PROXY
PROXY
PROXY
PROXY
PROXY
Public Subnets
MNAT
Public Subnets
EIP EIP

20.
Why? .....and hang on, I still see EIPs?
Scaling Increments:
10GB @ $42/month/10GB
100’s of Mb/s @ ~$210/month/100’s Mb/s
• Provision 50/100/200Gb/s upfront
• If you move in increments of
+/-100Gb/s, see pattern 3.
• Simple, HA, static IP proxies for a relatively
low uplift in cost
Private SubnetsPrivate Subnets
Availability Zone A Availability Zone B
ASG
EC2EC2EC2
EC2EC2EC2
MNAT
PROXY
PROXY
PROXY
PROXY
PROXY
PROXY
Public Subnets
MNAT
Public Subnets
EIP EIP

21.
Complex Inspection Sandwich
• Lots of vendor solutions can now support
healing
• Some even support scaling
• Few support ENI/EIP handling
Private SubnetsPrivate Subnets
Availability Zone A Availability Zone B
EC2EC2EC2
EC2EC2EC2
MNAT
INSPECTION SANDWICH
Public Subnets
MNAT
Public Subnets
EIP EIP

22.
PATTERN 3 - AUTO SCALING ANYTHING
A TECHNIQUE LEVERAGING EXISTING SERVICES TO AUTOSCALE ALMOST
ANYTHING

23.
The fiscal and operational benefits of Auto Scaling are well understood.
Auto Scaling is currently limited to scaling EC2 instances
We want to apply scaling to entire solutions, not just EC2

24.
SCALING CLUSTERS? SCALING CELLS?
• Enterprises have many applications that cannot scale on compute alone
– Sharded databases
– Life Sciences Clusters
– Simulation Clusters
• Organisations are starting to adopt “Cell Architecture” to account for scale
• Auto Scaling  Auto Healing
Client Example
~8000 instances connected in “rings” of 20 nodes via a cluster protocol + ~1500 Cassandra
nodes. 50% variance in daily traffic volume. Ideal use-case for Auto Scale

25.
THE GENERAL CASE - CELL / SHARD / CLUSTER
EC2
Node1
EC2
Node2
EC2
Node-n…
CloudFormation Stack
Health Check

26.
THE GENERAL CASE - CELL / SHARD / CLUSTER
EC2
Node1
EC2
Node2
EC2
Node-n…
CloudFormation Stack
Health Check
EC2
Node1
EC2
Node2
EC2
Node-n…
CloudFormation Stack
Health Check

27.
STEP 1 – INSTRUMENT THE SCALING METRIC
… …
CloudWatch Custom Metric
Number of Users CloudWatch Alarm
ScaleUp
CloudWatch Alarm
ScaleDown

28.
OPTION 1 – USE LAMBDA
ScaleUp ScaleDown
… …
Number of Users
SNS
…
SNS
Build
Lambda
TeardownL
ambda
CloudFormation

29.
WHY NOT LAMBDA?
• Duplication of AWS engineering investment
• Ongoing cost to maintain cadence with the growing features of Auto Scaling
– Scheduled Scaling
– Percentile Scaling
– Machine Learning Scaling / Predictive Scaling
• Lambda still needs a state machine
• We don’t have healing

30.
There are strong fiscal and complexity
motivators to use native ASGs

31.
STEP 2 – “SHADOW” ASG
ScaleUp
ScaleDown
Number of Users
… … …
Shadow Shadow Shadow
Shadow ASG

32.
STEP 3 – ADD THE CFN LAMBDAS
ScaleUp
ScaleDown
Number of Users
… … …
Shadow Shadow Shadow
Auto Scaling SNS
EC2_INSTANCE_LAUNCH
Create Stack
EC2_INSTANCE_TERMINATE
Delete Stack
Shadow ASG

33.
$5.76 per month per stack
(Unoptimized)

34.
Auto Scale  Auto Heal

35.
STEP 4 – HEALTH CHECK THE CLUSTERS
ScaleUp
ScaleDown
Number of Users
… … …
Shadow Shadow Shadow
Auto Scaling SNS
EC2_INSTANCE_LAUNCH
Create Stack
EC2_INSTANCE_TERMINATE
Delete Stack
Shadow ASG

36.
HEALING SCENARIO 1 – CLUSTER FAILS
ScaleUp
ScaleDown
Number of Users
… … …
Shadow Shadow Shadow
Auto Scaling SNS
EC2_INSTANCE_LAUNCH
Create Stack
EC2_INSTANCE_TERMINATE
Delete Stack
Shadow ASG

37.
HEALING SCENARIO 1 – SHADOW TERMINATED
ScaleUp
ScaleDown
Number of Users
… … …
Shadow Shadow Shadow
Auto Scaling SNS
EC2_INSTANCE_LAUNCH
Create Stack
EC2_INSTANCE_TERMINATE
Delete Stack
Shadow ASG

38.
HEALING SCENARIO 1 – ASG IS IMPACTED
ScaleUp
ScaleDown
Number of Users
… …
Shadow Shadow
Shadow ASG
Desired: 3
Actual: 2

39.
HEALING SCENARIO 1 – CLUSTER RESTORED
ScaleUp
ScaleDown
Number of Users
… … …
Shadow Shadow Shadow
Auto Scaling SNS
EC2_INSTANCE_LAUNCH
Create Stack
EC2_INSTANCE_TERMINATE
Delete Stack
Shadow ASG

40.
Continuous Delivery for Clusters
Blue/Green Updates for Clusters at Huge Scale

41.
CONTINUOUS DELIVERY FOR CLUSTERS
… … … … … …
• Using nothing but the ASG capacity, blue/green roll clusters of almost any size
• Increment ASG in V2.0, wait for health check, decrement ASG in V1.0
V1.0 V2.0

42.
AUTO SCALE ANYTHING
• Solution works with many non-scaling AWS services
• CloudFormation can use Custom Resources to create almost anything
• The “Shadow” system only needs the scaling alarms from any CloudWatch metric and a
health check endpoint. Decoupled and does not interact with the system in any way.

43.
Database Throughput ScaleUp Alarm
SNS
Lambda
RDS Read
Slave
CloudFormation
Shadow ASGRDS Read
Slave
RDS Read
Slave
ScaleDown Alarm
CUSTOM SCALING EXAMPLES

44.
Number of user sign-ups/logins ScaleUp Alarm
SNS
Lambda
Application
Shard
CloudFormation
Shadow ASGApplication
Shard
Application
Shard
ScaleDown Alarm
CUSTOM SCALING EXAMPLES

45.
CPU/Memory ScaleUp Alarm
SNS
Lambda
VMWare Node
CloudFormation
Shadow ASGVMWare Node VMWare Node
ScaleDown Alarm
CUSTOM SCALING EXAMPLES

46.
CPU/Memory ScaleUp Alarm
SNS
Lambda
Other
infrastructure
platforms
CloudFormation
Shadow ASG
Other
infrastructure
platforms
Other
infrastructure
platforms
ScaleDown Alarm
CUSTOM SCALING EXAMPLES

47.
Number of items in the queue ScaleUp Alarm
SNS
Lambda
Life Sciences
Application
CloudFormation
Shadow ASGLife Sciences
Application
Life Sciences
Application
ScaleDown Alarm
CUSTOM SCALING EXAMPLES

48.
Number of planes currently in the air ScaleUp Alarm
SNS
Lambda
Flight Analysis
Stack
CloudFormation
Shadow ASGFlight Analysis
Stack
Flight Analysis
Stack
ScaleDown Alarm
CUSTOM SCALING EXAMPLES

49.
Number of door entries ScaleUp Alarm
SNS
Lambda
Trading Stack
CloudFormation
Shadow ASGTrading Stack Trading Stack
ScaleDown Alarm
CUSTOM SCALING EXAMPLES

50.
Order Volume ScaleUp Alarm
SNS
Lambda
Number of
robots on
station
CloudFormation
Shadow ASG
Number of
robots on
station
Number of
robots on
station
ScaleDown Alarm
CUSTOM SCALING EXAMPLES

51.
Find Out MORE:
Visit Us: At our booth or online – www.sourcedgroup.com
Careers: www.sourcedgroup.com/careers
In the news:
• Computerworld (2016):
• Foreign Exchange Service OFX Embarks on Cloud Migration
• Connecting the Australian Channel (2015):
• Meet the Partner who took Qantas to the AWS Cloud
• The Australian Business Review (2015):
• Greater Buying Power lets Aussie bank on Adobe Experience Manager
Our Awards:
• AWS – Sydney Partners Summit - Invent & Simplify (2015)
• AWS – Global - Customer Obsessed Partner (2014)

52.
Thank you!