Many customers are using Amazon EC2 instances to run applications with high performance networking requirements. In this session, we provide an overview of Amazon EC2 network performance features (enhanced networking, ENA, placement groups, etc.), and discuss how we are innovating on behalf of our customers to improve networking performance in a scalable and cost-efficient manner. We share best practices and performance tips for getting the best networking performance out of your Amazon EC2 instances.

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Brian Barrett, Principal Software Engineer, AWS
Nick Matthews, Partner Solutions Architect, AWS
November 2016
CMP315
Optimizing Network Performance
for Amazon EC2 Instances

2. What to Expect from the Session
• High Performance networking applications
• Enhanced networking
• Deep dive
• Performance
• What next?
• Elastic Network Adapter
• What is ENA?
• Driver details
• Customer benefits

3. Networking on AWS
AWS proprietary networking
• Highest performance in largest EC2 instance sizes
• Full bi-section bandwidth in placement groups, with no
network oversubscription
Enhanced Networking
• Over 1 MPPS performance, reduced instance-to-instance
latencies
• Supporting Intel SR-IOV for consistent network
performance in a virtualized environment
• Amazon Elastic Network Adaptor – new for 2016

4. Traditional EC2 Networking
Instance
Virtualization Layer
XenPV
XenPV
eth0 eth1

5. Traditional EC2 Networking
Instance
Virtualization Layer
XenPV
XenPV
eth0 eth1

6. Enhanced Networking (Intel 82599)
Instance
Virtualization Layer
PCI VFeth0
eth1 PCI VF
Intel
82599

7. Enhanced Networking (Intel 82599)
Instance
Virtualization Layer
PCI VFeth0
eth1 PCI VF
Intel
82599

8. Elastic Network Adapter (ENA)
• New PCI network device developed for EC2
• Supported on X1, P2, and m4.16xlarge instance types
• Ability to scale across variety of bandwidths
• 10 and 20 Gbps instance types today
• Device interface supports up to 400 Gbps networking
• Increased number of request queues per device
• Improved debugging features

9. Enhanced Networking Latency Improvements
0
50
100
150
200
250
300
tp50 tp99
cc2.8xlarge c3.8xlarge c4.8xlarge m4.16xlarge

10. TCP Bandwidth
0
5000
10000
15000
20000
25000
64 128 256 512 1024 2048 4096 8192
Mbps
TCP Packet Size
cc2.8xlarge c3.8xlarge m4 m4.16xlarge

11. TCP Bandwidth
0
5000
10000
15000
20000
25000
64 128 256 512 1024 2048 4096 8192
Mbps
TCP Packet Size
cc2.8xlarge c3.8xlarge m4 m4.16xlarge m4.16xlarge Multistream

12. ENA Driver Support
• Supported in Amazon Linux out of the box
• Linux and Windows drivers available
• Linux: https://github.com/amzn/amzn-drivers
• Windows Server 2008 R2, 2012, 2012 R2 support
• Part of Linux 4.9
• DPDK support as of 16.04, recommend 16.11
• FreeBSD driver coming soon

13. High Performance Network Application
Communications
• Global communications company providing real time
communications to 1M developers
• Routing calls between AWS regions is done internally
and requires low latency

14. How does Twilio benefit?
• HVM instances that allows them to have access to
physical network cards getting 10 Gbps
• Enhanced networking allows them to achieve high PPS
that helps them in getting less variance in packet
processing times
• Quicker call routing and happy customers

15. • Distributed application with thousands of TCP connections between
instances
• Migrated Clash of Clans from EC2 Classic to VPC
• 100% on Enhanced Networking instances
Internet
Front-end
Proxy Layer
Application
Service Layer
Storage Layer
Supercell

16. How did Supercell benefit?
• In EC2 Classic, hundreds of servers each with thousands
of TCP connections took 30 minutes to register, discover
services, and establish TCP connections
• With Enhanced Networking on VPC, it takes less than 1
minute
• TCP reliability
• Monitoring
• Database connections
• Message queue depths dramatically smaller

17. Getting players back in the game
Network maintenance and
update downtimes are 20
minutes instead 1 hour

18. Cisco Cloud Services Router
• Same router experience on AWS or on-premises
• Advanced features for VPN, routing, and security
AWS Marketplace offering

19. Cisco Transit VPC
• Connect multiple VPCs
• Centralize connectivity
• Routers terminate VPN and
pass all traffic
• Fully automated using tags
See a demo at booth #200
https://aws.amazon.com/answers/transit-vpc/

20. Cisco benefits from Enhanced Networking
CEF Forwarding: 100%+ increase to 4.6 Gbps
IPsec forwarding: 100%+ increase to 4.2 Gbps

21. Enhanced Networking Requirements
• Supported instance types:
• ixgbe: C3, C4, D2, I2, M4 (except m4.16xlarge), R3
• ena: m4.16xlarge, P2, X1
• AMI requires either ixgbevf or ena driver
• ixgbevf version 2.14.2 or later
• Any ENA release
• AMI tag to enable Enhanced Networking

22. Instance Enablement
% aws ec2 describe-instance-attribute
--instance-id i-0ec179afafe026710
--attribute sriovNetSupport
{
"InstanceId": "i-01286b75ff50326ea",
"SriovNetSupport": {}
}
Not yet!

23. Instance Enablement
% aws ec2 describe-instance-attribute
--instance-id i-07312ca8e93d69514
--attribute sriovNetSupport
{
"InstanceId": "i-07312ca8e93d69514",
"SriovNetSupport": {
"Value": "simple”
}
}
82599
Support!

24. Instance Enablement
% aws ec2 describe-instances
--instance-id i-07a94b1806d6cd309
--query "Reservations[].Instances[].EnaSupport”
[
true
] ENA
Support!

25. AMI Check
• 82599 Enhanced Networking:
% aws ec2 describe-image-attribute --image-id ami_id
--attribute sriovNetSupport
• ENA Enhanced Networking:
% aws ec2 describe-image-attribute --image-id ami_id
--attribute enaSupport

26. Instance Enablement
1. Start an instance without Enhanced Networking
support
2. Build the ENA kernel driver
3. Build a new AMI or modify the instance
# aws ec2 modify-instance-attribute
--instance-id instance_id --ena-support
Or
# aws ec2 register-image --ena-support ...
4. Restart / relaunch and enjoy!
More info: https://docs.aws.amazon.com

27. Yes!
[ec2-user ~]$ ethtool -i eth0
driver: vif
version:
firmware-version:
bus-info: vif-0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
Is this thing on (ixgbe)?
No
[ec2-user ~]$ ethtool -i eth0
driver: ixgbevf
version: 2.14.2
firmware-version: N/A
bus-info: 0000:00:03.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

28. Yes!
[ec2-user ~]$ ethtool -i eth0
driver: vif
version:
firmware-version:
bus-info: vif-0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
Is this thing on (ena)?
No
[ec2-user ~]$ ethtool -i eth0
driver: ena
version: 0.6.6
firmware-version:
bus-info: 0000:00:03.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

29. Thank you!

30. Remember to complete
your evaluations!