This document discusses using IP multicast and layer 2 networking techniques in AWS VPC to enable features like VRRP and software load balancers. It describes how to implement pseudo broadcast/multicast in VPC using IP multicast and packet duplication. Examples are provided showing how to configure LVS and Keepalived for VRRP heartbeat and failover. The document concludes that VPC provides more flexibility and options for architecting compared to EC2-Classic.

1. IP Multicast on EC2
-- Who said VRRP doesn’t work on AWS?--
Kenta Yasukawa
Solutions Architect
Amazon Data Services Japan

2. What this presentation is about
My experiences on low layer networking stuff on EC2
• Around L3 or L2
• No L1 stuff
We can’t reach L1 of AWS

3. Do you use VPC?
EC2-Classic
Amazon Virtual Private
Cloud (VPC)
Each user gets private
sections to deploy
instances

4. VPC is not only for private networks
What EC2-VPC can do whereas EC2-Classic cannot
• Static private IP address allocation
• Multiple IP addresses allocation
• Multiple network interfaces
• Dynamic security group membership configuration
• Outbound packet filtering by security group
• NACL
• …

5. L2 looks different from an EC2 instance
EC2-Classic
• Same answer for any ARP request
EC2-VPC
• Familiar output!

6. It means L2 tricks may work in VPC
L2 NAT (e.g. ebtables, tc)
Software which requires L2 addressing (e.g. LVS)
DSR (Direct Server Return) Load balancer
Pseudo IP broadcast/multicast
Will explain this in more detail

7. How Pseudo L2 Broadcast works
VPC Subnet
IP Multicast
Dst: Ethernet Broadcast address
Src: Sender’s MAC address
Duplicate
Unicast-ify
Note: Disable Src/Dest on ENI
Fine. Let’s use
unicast then.
IP Multicast packet is sent over
Ethernet broadcast frame which
just goes into black hole on EC2
Applications bind to IP
address, not MAC
address. So most of
them just work fine.

8. How to implement
Script below is based on packet capturing (Just a Proof of Concept)
• Easy to understand
• It performs ok for an ENI which does not get much traffic
• Daemonizing is necessary (e.g. supervisord)
https://gist.github.com/kntyskw/5231182

9. How to implement
tc mirred + pedit + csum
The sample shell script configures tc to perform the job
https://gist.github.com/kntyskw/5633755
Much better both from performance and maintenance perspective
A little trickier to understand, though
Requires two ENIs for a subnet because tc mirred does not like to send
loop for copied packets
# ec2_multicast.sh
<interface to grab multicast packets from>
<interface to send modified packets to>
[target MAC address 1] [target MAC address 2] ...
# ec2_multicast.sh eth0 eth1 00:11:22:33:44:55 66:77:88:99:aa:bb
Example:
Usage:

10. Determine neighbors’ MAC addresses
Use AWS API
• Example：
https://gist.github.com/kntyskw/5413698

11. Is it useful?
It has quite some overhead
since a packet is duplicated
and sent multiple times
But maybe ok to use for
infrequent messaging?
Sure. Don’t
use it for
sreaming
HD movies
Right.
So, what about using it for
node discovery and
heartbeating?
Ask yourself.
Yeah, like
VRRP?

12. Let’s now try to make VRRP work
Virtual Router Redundancy Protocol (VRRP)
• Gives a virtual IP address to a group of nodes (VRRP group)
which forms a fault tolerant cluster
• Nodes in a VRRP group communite with IP multicast

13. Verification with LVS + Keepalived
VPC Subnet
LVS +
keepalived
LVS +
keepalived
WebWebWeb
172.31.0.0/20
Elastic IP
Secondary:
172.31.24.1
Primary:
172.31.24.100
Primary:
172.31.24.101
VRRP
Normal state Request routing
path

14. Verification with LVS + Keepalived
VPC Subnet
LVS +
keepalived
LVS +
keepalived
WebWebWeb
172.31.0.0/20
Elastic IP
Secondary:
172.31.24.1
Primary:
172.31.24.100
Primary:
172.31.24.101
VRRP
Upon failover Request routing
path

15. How to take over an IP address
VPC is not using ARP to determine dest MAC address
• Needs to provision the mapping b/w IP address and ENI
Call AWS API to reassign VIP from Master to backup upon failover
https://gist.github.com/kntyskw/5417140

16. Keepalived configuration
Execute the script when promoted to the master
vrrp_instance VI_1 {
state BACKUP
interface eth1
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.31.24.1 dev eth0
}
notify_master /etc/keepalived/assign_vip.sh
}

17. Advantages of using LVS
UDP load balancing!
• Neither ELB nor HAProxy can do
LVS does not need to terminate TCP
• Lower overhead
No need to involve user land process
• Lower latency
• Higher throughput

18. Quick benchmarking results sss
LVS v.s. HAProxy
• Both run on a m1.small instance
• Tested to fetch a 4KB HTML file with BeesWithMachineGuns
• Each of 8 bees will fire 2000 rounds, 10 at a time.
0
500
1000
1500
2000
2500
3000
3500
4000
4500
LVS HAProxy
Requestperseconds
Throughput
0
10
20
30
40
50
60
Average tp50 tp90
ResponseTimein[ms]
Response Time
LVS
HAProxy

19. Things to note when you depend on L2
VPC Subnet cannot span across multiple AZs
ENI cannot reach a subnet in a different AZ
Availability Zone Availability Zone
VPC Subnet
Availability Zone Availability Zone

20. Building Multi-AZ architecture with LVS
VPC Subnet
LVS +
keepalived
LVS +
keepalived
WebWebWeb
172.31.0.0/20
Elatic IP
Secondary:
172.31.24.1
Primary:
172.31.24.100
Primary:
172.31.24.1
01
VRRP
If you can use multiple IP addresses
• Use DNS RR as well as ELB does
• Route53 Health Check can be used to detect failure
Request flow
VPC Subnet
LVS +
keepalived
LVS +
keepalived
WebWebWeb
172.31.64.0/20
Secondary:
172.31.78.1
Primary:
172.31.78.
100
Primary:
172.31.78.101
VRRP
Availability Zone Availability Zone
Amazon Route
53

21. Building Multi-AZ architecture with LVS
VPC Subnet
LVS +
keepalived
LVS +
keepalived
WebWebWeb
172.31.0.0/20
Elatic IP
Secondary:
172.31.24.1
Primary:
172.31.24.100
Primary:
172.31.24.1
01
VRRP
If you have to stick to one single IP address
(e.g. due to NAT traversal issue for UDP based services)
• Perform Heartbeating b/w VRRP master nodes and move EIP upon failover
Request flow
Availability Zone Availability Zone
VPC Subnet
LVS +
keepalived
WebWebWeb
172.31.64.0/20
Primary:
172.31.78.101
VRRP
LVS +
keepalived
Secondary:
172.31.78.1
Primary:
172.31.78.
100

22. Concluding remarks
VPC gives you freedom on L2 which allows:
• L2 NAT
• L2 addressing
• Pseudo Broadcast/Multicast
Pseudo IP Multicast enables most applications which
require IP multicast
• Confirmed LVS + Keepalived works well
LVS has advantages over other LBs depending on use
cases
• Don’t forget to build Multi-AZ architecture
Use VPC and get more options for architecting!