(ARC401) Black-Belt Networking for the Cloud Ninja | AWS re:Invent 2014

https://s3.amazonaws.com/reinvent-arc401/index.html

•Basic automation
–Automating network build
•Intermediate automation
–Going beyond initial network build
•Advanced automation
–Dynamic component configuration

aws ec2 create-vpc --cidr-block 10.0.0.0/16
aws ec2 replace-route --route-table-id $ROUTE_TABLE_ID
--destination-cidr-block 0.0.0.0/0
--instance-id $INSTANCE_ID
aws ec2 attach-network-interface --network-interface-id $ENI
--instance-id $INSTANCE_ID
--device-index 1
aws ec2 assign-private-ip-addresses --network-interface-id $ENI
--private-ip-addresses10.0.0.100
•AWS CLI

#!/bin/sh
export AWS_DEFAULT_REGION="us-east-1"
VPC_ID=`aws ec2 create-vpc--cidr-block 10.0.0.0/16 --output text | awk '{print $6;}'`
SUBNET_ID=`aws ec2 create-subnet--vpc-id $VPC_ID --cidr-block 10.0.1.0/24 --output text | awk '{print $6;}'`
echo "Created $VPC_ID &$SUBNET_ID"
#Clean up
aws ec2 delete-subnet--subnet-id $SUBNET_ID
aws ec2 delete-vpc--vpc-id $VPC_ID
•Custom scripts

#Powershell script
Initialize-AWSDefaults -Region 'us-east-1'
#Create new VPC
$vpc = New-EC2Vpc-CidrBlock '10.0.0.0/16'
$subnet = New-EC2Subnet-VpcId $vpc.VpcId -CidrBlock '10.0.1.0/24'
-AvailabilityZone 'us-east-1d'
Write-Host “Created VPC: " $vpc.VpcId " subnet: " $subnet.SubnetId
#Clean up VPC
Remove-EC2Subnet$subnet.subnetId -Force
Remove-EC2Vpc$vpc.VpcId -Force
•Amazon SDK

"Resources" : {
"VPC" : {
"Type" : "AWS::EC2::VPC",
"Properties" : {
"CidrBlock" : “10.0.0.0/16”,
"Tags" : [ { "Key" : “Name", "Value" : “VPCName“ } ]
}
},
"PublicSubnet" : {
"Type" : "AWS::EC2::Subnet",
"Properties" : {
"VpcId" : { "Ref" : "VPC" },
"CidrBlock" : “10.0.1.0/24”,
"Tags" : [ { "Key" : "Network", "Value" : "Public" } ]
}
}
•AWS CloudFormation

•Allows network build to be:
–Automated
–Tracked
–Version controlled
•A great start!
–Aspirational for many of my customers

•Control changes to the network
•Managing additional network components
–Peering or VPN connections
–NAT or VPN instances
•Automate application-specific network components
–EIPs, secondary IP assignments, routed VIPs

•Controlling network changes with CloudFormation
–Templates can be version controlled
–UpdateStack
•Add/Remove resources
•Modify security group rules
–Events are tracked by CloudFormation

•In-region network expansion with VPC peering
–Peering handshake can be scripted
–CloudFormation“AWS::EC2::VPCPeeringConnection” type
•Cross-region network expansion
–VPC, routes, VPN instances can be scripted
–Check out vpc2vpc as an example
https://github.com/vinayselvaraj/vpc2vpc
vpc2vpccreate 10.1.0.0/16 10.2.0.0/16 10.3.0.0/16

#!/bin/sh
NAT_ID=“i-12345”
NAT_RT_ID=“rtb-22574640”
REGION=“us-east-1”
…
# So we can monitor the other NAT instance
NAT_IP=`aws ec2 describe-instances--instance-id $NAT_ID --region $REGION | grep PrivateIpAddress -m 1 | awk '{print $2;}' | sed -re 's/[",]//g'`
…
aws ec2 replace-route--route-table-id $NAT_RT_ID --instance-id $Instance_ID
--destination-cidr-block 0.0.0.0/0 --region $REGION
•Manage networking components (for example, HA)
https://aws.amazon.com/articles/2781451301784570

•Allows networks and components to be:
–Automated
–Tracked
–Version controlled
•Not very dynamic
Instance_ID=“”
Route_Table_ID=“”
Virtual_IP=“”
EIP_Alloc_ID=“”

•Dynamic network automation scripts
•Automation that responds appropriately when network or applicationconditions change
–Without changing the scripts
•Examples
–VIP reassignment in response to Auto Scaling
–New subnets get appropriate dynamic routing rules
–Create VPN tunnels when new regions are brought online

•Dynamic network automation approaches
–Instance bootstrapping
–Store dynamic information in an external store
–Change polling, detection, and response
•Standard external stores
–Amazon S3, Amazon DynamoDB, Configuration Management Tool

•What about using resource tags?
–AWS resources can be tagged
•Tags are great for identifying resources
–In the console
–By project or environment
–For billing

•Network tagging scheme
–Route table tags
•NAT= true
•NATAZ= [any, us-east-1a]

Public Subnet 1
AWS Region
Availability Zone 1 Availability Zone 2
NAT
Public Subnet 2
Private Subnet 1 Private Subnet 2
TAG
NATAZ
any
Auto Scaling Group
TAG
NATAZ
This subnet any
needs NAT
This subnet
needs NAT

Public Subnet 1
AWS Region
NAT
Public Subnet 2
NAT
This subnet
needs AZ-specific
NAT
This subnet
needs AZ-specific
NAT
TAG
NATAZ
AZ1
TAG
NATAZ
AZ2

--filters "Name=tag-key,Values=NAT"
--filters "Name=tag:NATAZ,Values=any,us-east-1a"
•Filtering AWS resources by tag
--filters are awesome!

#!/bin/bash
INSTANCE_ID=`curl --silent http://169.254.169.254/latest/meta-data/instance-id`
AZ=`curl --silent http://169.254.169.254/latest/meta-data/placement/availability-zone`
REGION="${AZ%?}"
MAC=`curl --silent http://169.254.169.254/latest/meta-data/network/interfaces/macs/`
VPC_ID=`curl --silent http://169.254.169.254/latest/meta-data/network/interfaces/macs/$MAC/vpc-id`

#!/bin/bash
INSTANCE_ID=`/usr/bin/curl --silent http://169.254.169.254/latest/meta-data/instance-id`
AZ=`/usr/bin/curl --silent http://169.254.169.254/latest/meta-data/placement/availability-zone`
REGION="${AZ%?}"
ROUTE_TABLES=`aws ec2 describe-route-tables--region $REGION --output text
--filters "Name=tag:NATAZ,Values=any,$AZ" |
grep ROUTETABLES | awk '{print $2}'`

#!/bin/bash
INSTANCE_ID=`/usr/bin/curl --silent http://169.254.169.254/latest/meta-data/instance-id`
AZ=`/usr/bin/curl --silent http://169.254.169.254/latest/meta-data/placement/availability-zone`
REGION="${AZ%?}"
ROUTE_TABLES=`aws ec2 describe-route-tables --region $REGION --output text
--filters "Name=tag:NATAZ,Values=any,$AZ" | grep ROUTETABLES | awk '{print $2}'`
# Parse through RouteTables that need to be modified
forMY_RT_IDin $ROUTE_TABLES;do
aws ec2 replace-route --route-table-id $MY_RT_ID --destination-cidr-block 0.0.0.0/0
--instance-id $INSTANCE_ID` --region $REGION
done

Public Subnet 1
US-West-2
Public Subnet 2
US-East-1
TAG
VPN
EIP
TAG
VPN
true
TAG
VPN
true
TAG
VPN
EIP

•Network automation doesn’t require hard-core development
•Simple scripts can be very powerful
–Create dynamic, resilient networks and network components
–Respond to application or business requirements

•Network automation
•Manipulating IPs
•Network monitoring

•Virtual IP addresses
–Support less cloud-friendly use cases
•Multicast
–Support legacy multicast use cases
•Floating networks
–Support overlapping network use cases

• Elastic IP
10.0.0.55
72.44.63.250
10.0.1.79
AWS Region
aws ec2 associate-address --private-ip-address 10.0.1.79
--allocation-id [EIP Allocation ID] --allow-reassociation
Availability Zone Availability Zone

• Secondary private IP
10.0.0.55
72.44.63.250
10.0.0.79
AWS Region
aws ec2 assign-private-ip-addresses --private-ip-addresses 10.0.0.10
--network-interface-id eni-123abcde --allow-reassociation
10.0.0.10
Availability Zone

• Routed virtual IP
10.0.0.55
192.168.0.10
10.0.1.79
AWS Region
#ifconfig eth0:1 192.168.0.10/32 up
aws ec2 replace-route --route-table-id [Route Table ID]
--instance-id [Instance ID]
Availability Zone Availability Zone

•Configure your instance OS with another IP
•Disable SRC/DST checking
•Use a replace-route API call to direct traffic
# ifconfig eth0:1 192.168.0.10/32 up
aws ec2 replace-route--route-table-id [Route Table ID]
--instance-id [Instance ID]
aws ec2 modify-instance-attribute--instance-id [Instance ID] –no-source-dest-check

Approach
Pros
Cons
EIP
Multi-AZ
Public IP only
(split-brained DNS within VPC for privateIP)
Secondary IP
Public and/or private IP
Single AZ
Routed VIP
Multi-AZ
Private IP only
Onlyaccessible from instances within the VPC

•Each VIP option supports application-specific requirements
•Embrace automation
–Build this into application deployment
•Build your network as an integral part of the application it supports

•By and large, a disappointment
•Some legacy apps/app servers require multicast
–Node discovery
–Session management
–Automated failover

• Not directly supported
• Can be implemented with an overlay network
• GRE or L2TP tunnels, Ntop’s N2N
10.0.0.54
10.0.0.79
10.0.1.132
Subnet 10.0.0.0/24 Subnet 10.0.1.0/24
10.0.1.183
10.0.0.41

•This technically isn’t multicast
–Multicast packets are wrapped in unicast packets
•Not a solution for unicast scaling
–Additional packet overhead
•GRE approach adds 38 bytes (MTU of 1500 will effectively be 1462)
•Unicast traffic can also traverse the overlay network
–Not subject to security group filtering
•Decent option for small, legacy clusters
–App server node discover
–Low traffic volumes

• GRE configuration can be automated
– Multicast configuration stored in tags
• Periodically check for new members (60 seconds)
172.31.16.124
172.31.28.164
172.31.47.71
Subnet 172.31.16.0/20 Subnet 172.31.32.0/20
TAG: multicast
App1,192.168.0.12/24
TAG: multicast
App1,192.168.0.11/24
TAG: multicast
App1,192.168.0.10/24
192.168.0.0/24 Overlay
Community: App1

•Trend: We have less control over the network ranges our networks must interact with
–SaaS
–Mergers
–DevOps

Customer ‘n’
192.168.0.0/16
Customer 3
10.0.0.0/16
10.0.0.0/16
Customer 2
10.0.0.0/16
Customer 1
172.16.0.0/16
172.16.0.268
Service 1
Service 2
Service EIPs
Amazon Route 53
Service ‘n’
.
.
.
Service load balancers

•Most straightforward approach
•Great when your services do not need to initiate connections

10.0.0.0/22
Customer 2
10.0.0.0/16
10.0.0.58
10.0.0.105
192.168.0.0/22
192.168.0.124
Customer 1
172.16.0.0/16
172.16.0.268

•Fairly straightforward
•Viable option with AWS automation
•Typically best for single tenants
–Shared systems can be challenging

10.0.0.0/16
10.0.0.0/16
192.168.0.0/16
10.0.0.58
10.0.0.105
SRC: 10.0.0.58
DST: 192.168.0.105
SRC: 192.168.0.58
DST: 10.0.0.105

192.168.0.0/16
10.0.0.0/16
10.0.0.58 10.0.0.0/16
10.0.0.105
SNAT: 192.168.0.0/16
DNAT: 10.0.0.0/16
# iptables -t nat -A PREROUTING -d 192.168.0.0/16 -j NETMAP --to 10.0.0.0/16
# iptables -t nat -A POSTROUTING -s 10.0.0.0/16 -j NETMAP --to 192.168.0.0/16

Customer #
Customer #
10.0.0.0/16
Customer #
Customer #

•Slightly more complicated
•Consider a shared service, HA VPN tier
•Viable option with VPN appliance automation
•May be better for multitenant solutions

•Monitoring has traditionally been siloed
–Application
–Server
–Network
•Limited access to networking devices
–NAT/VPN instances
–No access to cloud switches/routers
•Need to think about monitoring differently

•Lorien Wood School
–OpenDNS, SNMP (cacti)
•New network monitoring requirements
–More visibility into student activity

•Urlsnarf
•Logrotated
•Cron
•SCP
•Python script
•PHP/MySQL app
t2.micro

•Focus on requirements
–“Network monitoring” can mean just about anything
–Plan for scale
•Find the right tool to meet the requirements
–Gateway-based vs. host-based
–Simple scripts
–OS tools (job schedulers, SSH/WMI, logging capabilities, and log file rotators)
–AWS partner solutions
•Leverage reusable architectural patterns

AWS_Regions
CloudWatch_Region
cw ec2.cloudwatch.connect_to_regionregionAWS_Regions
vpcconn vpc.connect_to_region
vpns vpcconn.get_all_vpn_connectionsvpn vpnsvpn.stateavailable
active_tunnelsvpn.tunnels[0].status UP
active_tunnelsvpn.tunnels[1].status UP
active_tunnels
cw.put_metric_dataVPNStatusvpn.idVGWvpn.vpn_gateway_idCGWvpn.customer_gateway_id

•Log network events locally and ship them to:
–CloudWatch logs
–Syslog / Windows logs
–AlertLogic, Cloudlytics, Logentries, Loggly, Splunk, SumoLogic…
•Leverage existing log monitoring infrastructure
–Should already be built for scale
•Add network requirements and admins to log monitoring infrastructure development

Orchestrator
Configure
Monitoring
Distributed
Monitoring
Store Results
Analyze and Report

Monitoring Orchestrator
SCP
SSH
tcpdump-G 60
pcap collector
pcap analyzer
curl

•Network monitoring gateways
–Barracuda Networks, Checkpoint, Palo Alto Networks, Sophos
•Host-based network intrusion detection
–AlertLogic, TrendMicro, ThreatStack
•Network traffic analysis
–Boundary, CloudShark
•Application network performance
–Compuware

•Focus on requirements
•There are actually a lot of tools out there to help
–Some you might not have thought of before
•Leverage automation to create application-specific network monitoring solutions
•Plan for scale from the start
•Don’t fear doing things a little differently

Please give us your feedback on this session.
Complete session evaluations and earn re:Invent swag.
http://bit.ly/awsevals
https://s3.amazonaws.com/reinvent-arc401/index.html

(ARC401) Black-Belt Networking for the Cloud Ninja | AWS re:Invent 2014

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to (ARC401) Black-Belt Networking for the Cloud Ninja | AWS re:Invent 2014

Similar to (ARC401) Black-Belt Networking for the Cloud Ninja | AWS re:Invent 2014 (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Recently uploaded

Recently uploaded (20)

(ARC401) Black-Belt Networking for the Cloud Ninja | AWS re:Invent 2014