Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

(NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC


Published on

"Amazon Virtual Private Cloud (Amazon VPC) has many obvious benefits. For example, you can use Amazon VPC to define a virtual network in your own logically isolated area within the AWS cloud, and launch your EC2 instances into a VPC. But how can you take advantage of the EC2-VPC platform if your services and infrastructure are already deployed in the EC2-Classic platform? In this deep-dive session, learn how to safely and reliably migrate from EC2-Classic to EC2-VPC with zero downtime. We show you how Twilio approached the problem of a VPC migration, or what we internally called the “Moving Datacenters Project.” We discuss the technologies and tools (both internal and external) we used to complete the migration, the infrastructure we built along the way, and the lessons we learned. 

Session sponsored by Twilio."

Published in: Technology
  • Be the first to comment

(NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC

  1. 1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. @Sumbry Director of Cloud Services October, 2015 NET409 Movin' On Up to the VPC How Twilio Migrated its Infrastructure from EC2-Classic to EC2-VPC
  2. 2. Purpose of this talk - Learn about Twilio - Review legacy infrastructure - Why EC2-VPC? - How we built the Twilio Cloud - How we migrated - Internal tools developed - Lessons learned
  3. 3. What Is a Twilio? - A global communications company - A real-time communications API - Used by over 500,000 developers - Requires low-latency resilient infrastructure - Has lots of infrastructure on EC2-Classic
  4. 4. Who are Twilio customers?
  5. 5. Legacy Twilio
  6. 6. What did Twilio look like yesterday? - Twilio has used AWS since 2008 - Three products - All infrastructure located in us-east-1 - Hundreds of instances - 10/8 shared private network - Non-consecutive EIPs
  7. 7. Before global
  8. 8. What is going global? - Launched outside US - Global provisioning - Route traffic between regions - Low-latency communications - Global service discovery
  9. 9. The network after global
  10. 10. Problems with going global - Overlapping 10/8 networks - Proxies not ideal, point-to-point - Routing around failovers - Need low latency connectivity
  11. 11. Why EC2-VPC?
  12. 12. What is EC2-VPC? EC2-VPC is the next major revision of the EC2 platform: - Software Defined Network - Elastic Network Interfaces - HVM and SR-IOV
  13. 13. What is a software defined network? - Define your own network - VPC and subnet routing tables - Network Access Control Lists - Provision networks like virtual machines - Protects data-in-transit
  14. 14. What are elastic network interfaces? - Public and Private EIPs - Multiple Private EIPs per interface - Multiple ENIs per instance - Security groups follow an ENI - ENI has a MAC address
  15. 15. What are HVM instances? - Hardware Virtualized Machine instances - PCI Express speeds to network adapter - Low-latency access to network adapter - Up to 10 GB network speeds
  16. 16. Why move to EC2-VPC? - SDN solves overlapping 10/8 networks - Route tables eliminates proxies - Routing around failovers is an API call - HVM solves low latency connectivity problem
  17. 17. The Twilio Cloud
  18. 18. What is the Twilio Cloud? - Iteration 2.0 of our infrastructure - Addresses many EC2-Classic limitations - Connectivity between data centers - Automatic failover and redundancy - Provider agnostic
  19. 19. What does the Twilio Cloud look like?
  20. 20. What about routing?
  21. 21. We built it, did they come? We solved all previous issues but no one used it: - Twilio Cloud was isolated from EC2-Classic - Existing services had no migration path
  22. 22. Data center migration
  23. 23. Why is a migration like moving data centers? - Separate infrastructure from EC2-Classic - Need to migrate all your compute - Zero downtime
  24. 24. The networks
  25. 25. What problems do we need to solve? - Move an instance from Classic to VPC - Network connectivity - Instance discoverability - No service interruptions
  26. 26. Classic deploy
  27. 27. VPC deploy
  28. 28. Kill Classic
  29. 29. Steps to migrate a service
  30. 30. Wait - you just invented a bunch of stuff … - Bridge EC2-Classic and VPC? - Global Service Discovery? - Multiple Service Deployments? - WTF!
  31. 31. Migration tools
  32. 32. What are the tools for migrating to EC2-VPC? We modified existing internal tools: - IP Tunnel Manager / ClassicLink - Global Service Discovery - HAProxy Distributed Load-Balancing - Config-Renderer
  33. 33. What is IP Tunnel Manager ClassicLink? ClassicLink allows you to link your EC2-Classic instance to a VPC in your same account in the same region. It provides network connectivity between EC2- Classic and EC2-VPC instances.
  34. 34. What is Global Service Discovery? GSD stores IP addresses for any service in the cluster and serves them on-demand.
  35. 35. What is distributed load balancing? Every instance in the cluster runs its own instance of HAProxy. It load balances requests to any downstream services.
  36. 36. What is Config-Renderer? Config-Renderer renders configuration files filled with data from Global Service Discovery, like HAProxy Configs!
  37. 37. What about deploying services? Our internal provisioning tool called BoxConfig lets us deploy services with the click of a button.
  38. 38. How does it all work?
  39. 39. Unix philosophy We use lots of small tools and combine them: - Twilio Cloud to route - ClassicLink to bridge - HAProxy for distributed load-balancing - Global Service Discovery for IP info - Config-Renderer to write HAProxy configs - BoxConfig to deploy
  40. 40. In conclusion
  41. 41. Where are you today? - The Twilio Cloud is live today - Routes traffic through nine virtual data centers - Over 100 IPSEC Mesh links - Automatic region failover thanks to EIGRP - 35% of Twilio infrastructure is in EC2-VPC - We can complete the migration in 2015
  42. 42. What are some lessons learned? - Properly subnet your VPC. You have one shot. - No need to do a giant migration all at once. - Tools need to work both ways in case you screw up. - Less complexity always wins.
  43. 43. Thank you!
  44. 44. Remember to complete your evaluations!
  45. 45. Related Sessions