Twilio provides a communications API that enables voice, VoIP, and messaging capabilities for web and mobile apps. They migrated their infrastructure from the isolated EC2-Classic platform to EC2-VPC to enable global routing between regions and services. This reduced complexity, improved performance and latency, and allowed for more frequent and less risky deployments. The migration required bridging traffic between EC2-Classic and EC2-VPC instances and using software routers and service discovery for peering between regions. The new global VPC infrastructure improved customer experience and satisfaction.

1. © 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in partwithout the express consent of Amazon.com, Inc.
November 12, 2014 | Las Vegas
SDD302A Tale of One Thousand Instances -Migrating from EC2-Classic to EC2-VPC
Donald Sumbry, Twilio
Jonas Borjesson, Twilio

2. Hi, I’m Jonas@borjessonjonas
Tech Lead of the SIP Connectivity team at Twilio

3. Hi, I’m Sumbry@sumbry
Director of Cloud Services at Twilio

4. Background

5. We provide a communications API that enables phones, VoIP, and messaging to be embedded into web, desktop, and mobile software.
twilio

6. twilio

7. How does it work?
A user calls your
number
Twilioreceives the call
Your app responds

8. twilio
6 years ago, things were simple
•Few machines on EC2-Classic
•Single service -PSTN to HTTP
•US-based customers
•Single AWSregion

9. ProcessingNode
SIP
RTP
ProcessingNode
ProcessingNode
Load Balancer
Load Balancer
Database

10. twilio
Now, not so simple anymore
•Thousands of instances
•Many services
•Global customers
•Resources in every AWSregion

11. SIP
RTP
Virginia
Ireland
Brazil

12. SIP
RTP
Signalling

13. Issues
All regions completely separated
–Traffic has to go through known endpoints
•Un-necessary hops
•Complicates deployments
•More difficult to debug
•Easier to create routing bugs

14. Forcing us to…
•Open up firewalls
•Secure traffic between regions using our own “VPN”
•Traffic has to go through known endpoints
•Known endpoints assigned EIPs

15. Which means…
•Un-necessary hops
•Complicates deployments
•More difficult to debug
•Easier to introduce bugs
•Cannot deploy nodes behind EIPs without affecting traffic

16. That translates to…
•Fewer deploys
•Riskier deploys
•Harder to nail down bugs
•Takes longer to get fixes out
•Less happy customers!

17. Why VPC?

18. “EC2 2.0”(aka EC2-VPC)
•Global routing tables
•Enhanced Networking with SR-IOV
•Elastic network interfaces
•Software defined network
•Hardware security manager
Twilioconsiders VPC an evolutionary step or upgrade of the Amazon EC2 platform.

19. Global routingtables
•Per subnet or per VPC routing tables
•Route traffic to instances
•Tunnel traffic between regions
Routing traffic to instances enables the easy creation of things like loadbalancers, tunnels, or even VPCs inside of VPCs.

20. HVM and SR-IOV
•HVM images with Enhanced Networking
•PCI Express speeds to network adapter
•Low-latency access to network adapter
•Up to 10gb network speeds
Enhanced Networking with SR-IOV means fast performance even under virtualized hardware.

21. Elastic networkinterfaces
•Multiple EIPs and multiple private IPs
•Multiple ENIs per instance
•Security groups follow an ENI
•ENI has a MAC address
ENIs are more like network cards that you can move around and attach to different instances.

22. Software definednetwork
•Control over my instances’ routes
•Number my own network
•Network ACLs
•Data-in-transit protected by more than just a security group
•Provision networks like virtual machines
Use of a software defined network solves the data-in-transit issue that many certifications require.

23. Hardware securitymanager
•Easily integrates with IAM policies
•Centralized management of keys and certificates
•Easily and quickly encrypt customer data
Use of the HSM solves the data-at-rest issue that many certifications require.

24. TwilioCloudRequirements

25. TwilioCloudRequirements
•Services can be deployed anywhere
•Services can communicate anywhere
•Services can be discovered anywhere
Solving the issue of global service discovery is easy once the underlying cloud infrastructure is in place.

26. US1
US2
BR1
AU1
JP1
SG1
IE1
DE1

27. EC2-VPC Building Blocks
•Global routing tables
•HVM and SR-IOV
•Elastic Network Interfaces
•Software Defined Network
•Hardware Security Manager

28. Region-to-regionconnectivity
Performing routing among multiple VPCs in different regions is a bit more complicated and necessitates the use of a routing protocol.
router
router
us-east-1 / 10.1.0.0
us-west-2 / 10.2.0.0
vpc-abcdef
vpc-zyxwv
IPSEC tunnel
host
host
host
host

29. US1
US2
BR1
AU1
JP1
SG1
IE1
DE1

30. VPC-enabledinfrastructure

31. SIP
RTP
Virginia
Ireland
Brazil

32. Which may look insignificant,but...
•A single global network
–Global servicediscovery
•Much easier call flow
–Easier to debug
–Less risk to deploy
–More frequent deploys
–Call setup latency down 25%
•Less infrastructure and complexity

33. Also…
•Blocking firewall rules
–Important for stopping attacks
•ENI
–Aid us in deploying new edge services
–Improved network performance
–Better audio quality

34. Happier customers!

35. Migrating from EC2-Classic to EC2-VPC

36. Migration Requirements
•Equivalent to moving a datacenter
–Zero downtime
–Bridge traffic between services in a region
–Easily discover services in EC2-Classic or EC2-VPC

37. Peering vs bridging
Peering is two VPCs talking in different regions.
Bridging is EC2-Classic and EC2-VPC in the same account talking in the same region.
vpc-bbb
vpc-aaa
vpc-aaa
classic
us-east-1
us-west-2
us-east-1

38. Migrating from EC2-Classic to EC2-VPC
•Use IP Tunnel Manager for bridging traffic
•Use software routers for peering traffic
•Use Service Discovery for discovering new services as they move
Make sure any services you want to move from EC2-Classic to EC2-VPC share the same AWS account and are in the same region!

39. Conclusion

40. •Services can be deployed globally
•Services can communicate globally
•Services can be discovered globally
•New VoIP infrastructure deployed in:
–all regions around the world
–taking live traffic for new products
–existing carrier traffic is being migrated
Where we are today

41. How could this have been easier?
•Feature to bridgeEC2-Classic and EC2-VPC
•Feature to connect VPCs in different regions
Are you listening,AWS? Maybe. :-)

42. http://bit.ly/awsevals