Downloaded 14 times
Uploaded bySoftLayer Technologies
Softlayer Technologies texasipv6taskforce
SoftLayer has over 32,000 physical servers under management across 4,000 server pods clustered in regional network cores located in datacenters in Dallas, Seattle, and Washington DC. SoftLayer began looking at IPv6 in 2008 and launched support for it in January 2009, but faced challenges with network hardware support, provider support for their allocation scheme, tracking applications, operations support, and customer adoption. They addressed these challenges by using tunnels where hardware didn't support IPv6, negotiating with providers, designing their own allocation scheme, adapting their tracking applications, automating operations, and educating champion customers.
More Related Content
Similar to Softlayer Technologies texasipv6taskforce
![IPv6 Deployment Planning Tutorial, by Philip Smith [APNIC 38]](https://cdn.slidesharecdn.com/ss_thumbnails/apnic38-ipv6-deployment-planning1410239786-140916190200-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
Softlayer Technologies texasipv6taskforce
- 1. Matt ChilekVP, SoftwareDevelopmentSoftLayer Technologies, Incmchilek@softlayer.comAS363512607:f0d0::
- 2. BackgroundSoftLayer has around32,000 physical servers under managementThese are clustered into 4,000 server ‘pods’A group of pods is supported by a regional network coreNetwork cores are interconnected, and supported by a network of POPsDatacenters are located in Dallas, Seattle, Washington, DCCurrent cumulative sustained IPv4 traffic of 222GbpsCurrent IPv6 traffic: 50Mbps base, peak 200Mbps2SoftLayer Technologies, Inc10/12/2010
- 3.
- 4. IPv6 initiativeBegan lookingat IPv6 in 2008Received ARIN allocation of /32IPv6 initiative driven by long term planning, SoftLayer is an early adopterBeta tested IPv6 customers in fall 2008, plus hardware upgradesJanuary 21, 2009 - GA launchsoftlayer.com is currently a top 50 IPv6 website by both host and raw domainhttp://bgp.he.net/ipv6-progress-report.cgi4SoftLayer Technologies, Inc10/12/2010
- 5. Implementation ProblemsNetwork hardwareProvidersupportAllocation schemeTracking applicationOperations supportCustomer adoption5SoftLayer Technologies, Inc10/12/2010
- 6. P1. Network hardwareSomeequipment did not support IPv6, even in dual stack.
- 7. Some equipment didnot support a dual stack without firmware updates.
- 8. Some equipment supportedIPv6, but had limited implementation feature sets.6SoftLayer Technologies, Inc10/12/2010
- 9. S1. Network hardwareNoIPv6? Solution: TunnelsFirmware or software upgrades? Solution: Staggered maintenanceLimited feature sets?Solution: Use what you have to get it done.7SoftLayer Technologies, Inc10/12/2010
- 10. P2. Provider supportInitialtrouble finding provider to support our implementation schemeAllocated a /32 from ARINProvider accepts /32 or larger3 datacenters, but don’t want to anycastAsk ARIN for 2 more /32s, not going to happen**Before discrete network rules amendmentEnded up anycast anywaysProvider finally allowed us to broadcast /368SoftLayer Technologies, Inc10/12/2010
- 11. S2. Provider support9SoftLayerTechnologies, Inc10/12/2010LOTS OF PEERING!+ others
- 12. P3. Allocation schemeAlmostno published “best” practices No best practices for our organizational complexity10SoftLayer Technologies, Inc10/12/2010
- 13. S3. Allocation schemeBestpractices?!!Solution: Went with our own…/40: region/48: router/64: host11SoftLayer Technologies, Inc10/12/2010
- 14. P4. Tracking applicationSoftLayertracks and routes about 91,000 distinct IPv4 subnets assigned to customer hosts in the SoftLayer datacenter.Used “dense tree” tree for IPv4 tracking.12SoftLayer Technologies, Inc10/12/2010
- 15.
- 16. S4. Tracking applicationUsed“sparse tree” tree for IPv6 trackingFit into our existing data model with minor updatesUse existing search algorithmsFeels the same to the end user14SoftLayer Technologies, Inc10/12/2010
- 17. “Sparse tree” tracking15/32/40/40/48/48/48/64/48/64/64/64/64/64/64/64/64/64/64/64/64SoftLayerTechnologies, Inc10/12/2010
- 18. P5. Operations support16SoftLayerTechnologies, Inc10/12/2010“Uh, that’s a lot of numbers and letters and stuff” – anonymous support technicianS5. Operations support17SoftLayer Technologies, Inc10/12/2010Make it look like your IPv4 tools!Feels the same, looks the same, acts the same, therefore must be the same.It largely is the same.Automate as much as possible. Human brains can only carry so many numbers, and 32 is too many.
- 19. P6. Customer adoptionWhyshould customers start adopting? Our external customer typically doesn’t understand the size of the IPv6 address space18SoftLayer Technologies, Inc10/12/2010
- 20. S6. Customer adoptionApplicationwise, same as operationsWe make it look like our IPv4 tools.We found a champion who loves to talk with and educate customers.Awesome! Users that are ordering it know how to use it! Very few escalations.We made it easy to adoptLow price (free)Automated allocation and routing, no wait!19SoftLayer Technologies, Inc10/12/2010
- 21.
Editor's Notes
- #5 Rankings courtesy of Hurricane Electric
- #6 The capability of providing support for IPv6 is becoming increasingly urgent for ISPs and datacenters as the end of the IPv4 allocation pool lurks nearer.Infrastructure providers consume a lot of IP space due to the volume of machines, and the rate of consumption is only increasing with virtualization!These are some of the problems that our organization had when attempting to roll out an IPv6 implementation.
- #7 Before we started, we determined we had enough memory and CPU resources to run most of our hardware in dual stack.SoftLayer aggressive on technology adoption, so we were fortunate to already have dual stack support in the majority of our network hardware. Even so, we had several places that needed some adjustment.Some equipment did not support IPv6.We had a number of routers that had been in service for quite a few years, and needed firmware updates to support dual stack.Don’t go out and buy that new router just yet, check if there is a new firmware release to handle IPv6!Some equipment supported dual stack, but had limited IPv6 feature implementation.
- #8 For equipment that just could not support IPv6 and that we could not immediately replace, we created tunnels to jump through.Made sure to add it as a required feature set for all future purchases.For those that required firmware upgrades, most of these were due for upgrade anyways, so we bit the bullet and scheduled these out over several months.We schedule this in advance to minimize impact for customers.We presented this as a network upgrade and most customers appreciated it.In regards to the limited feature set, we used the tools available. For instance, we use IPv4 MPLS to transport IPv6 traffic using 6PE.
- #9 When we first attempted to get this on-line, we had some trouble finding providers that would support our implementation. We received an initial allocation of /32 from ARIN. The provider we were working with would only accept routes of /32 or larger. CONCERN! The problem was that we didn’t want to anycast the /32 from our 3 datacenters. If we did and lost transit between them we would have some stranded traffic. At first, we tried to ask ARIN for 2 more /32s, but this was before the discrete network rules amendment.Make the leap: We anycast the /32 subnet to startProvider relented a few weeks later and allowed us to broadcast /36. Eventually they went as small as /48
- #10 As I mentioned, there were very few providers offering native IPv6 support when we started in 2008, so our choice was limited.To improve connectivity we also peered with HE.Peering still remains a strong part of the mix for providing IPv6 connectivity.Currently we have uplinks through NTT, Comcast, and Global Crossing; L3 is on the way. We have LOTS of peering, including HE.We are now seeing more significantly more native support, though there are still a couple of stubborn holdouts.
- #11 Another thing we learned as early adopters, is that no one had put together a best practices summary yet. The couple of people that *had* published some information about their allocation scheme were all over the board. They really weren’t comparable to our organizational complexity.
- #12 We attempted a wait and see approach for about 6 months. During that time we researched and watched for someone to take the lead, but there wasn’t too much movement, so we just went ahead and did it. After a considerable amount of wrangling we finally sorted it down to a simple format, /40 region /48 router /64 host. We built in a large buffer for migration to a more virtual, “cloud” oriented infrastructure.
- #13 Once you figure out how to implement the IPv6 technology in your network, how are you going to manage it going forward?How many of you use a spread sheet to track subnets? SoftLayer tracks and routes about 91,000 distinct IPv4 subnets assigned to customer hosts in the SoftLayer datacenters.Obviously manual tracking wasn’t going to work for us from the start, so we implemented what I will refer to as a “dense tree model” to make sure nothing got lost.
- #14 A dense tree model means that all nodes are populated to the end leaves, with every node being a CIDR prefix and having 2 children of half the size. The parent node continues to exist both as a place holder and to maintain relationships between the nodes. You can see that every leaf node had a fully populated path back to the root. A problem with this can be that recursive searches can start to be rather inefficient if the tree gets too tall.To address this issue as we started moving into larger chunks of IPv4 allocations, we broke up the roots of our tree to limit the overall height and therefore our resource utilization. By spreading it out to many roots, usually around the size of a /19 or /20, we were able to achieve some horizontal scalability as we continue to consume space. To put some numbers on this, using this method, we have a maximum depth of 14 and a typical search depth of 11 when searching for user assigned subnets. This system works great for IPv4 management. Because of the dense tree model, we efficiently track every assigned subnet block, and we are assured of timely coalescence of unused space through automated routing management. There is however a problem with using this model for tracking IPv6 address space. The space is so huge that creating a node for every CIDR would result in far more overhead in regards to the parent nodes to be tracked. The search depth balloons to something between 32 and 96 depending on your allocation scheme, which would make it nearly unusable in many database environments.
- #15 Used “sparse tree” tree for IPv6 tracking.Keeping a tree model fit into our existing data model with minor updates.Using the existing model allowed us to re-use many existing algorithms for subnet management.Most of all, it simplified the work required to make the application feel the same to the user.
- #16 A sparse tree model means that we only populate nodes that matter. This means that at each level in our allocation scheme we populate nodes, but we skip all the white space in between, because it really doesn’t do anything for us. Its like Highway 40 between Albuquerque and Flagstaff, a lot empty space. We just fold that up to keep it simplified, and keep tabs on the important places in between. The downside is that this does somewhat reduce flexibility in automated systems, because we have locked ourselves into channels at the application layer. The goods news is that the model will support any allocation we need, we just have to tweak the application as needed.
- #17 Lots of resistance here, just because its different.Mileage may vary due to the nature of who the customers are, whether internal or external, and as either an end user or a more technical middle tier consumer.
- #18 Our solution was to make it look like your IPv4 tools! This helps the support staff become comfortable quickly, lowering the barrier to entry when acquiring new knowledge.Teach them how to read notation and you are done!When support uses the application, it feels the same, looks the same (kind of), acts the same therefore it must be the same.The truth is it largely is the same.We made the user experience the same, so we had less resistance to change/implementationAutomate as much as possible. Human brains can only carry so many numbers and 32 is too many. Even in short notation, when we get in the weeds the going gets tough. Transcription errors, and tracking in large data sets can be difficult if done manually.
- #19 Why should customers start adopting? The industry software they use largely doesn’t support it. In the hosting industry, the mainstream control panels have been very slow to integrate this technology, if they are even making the effort. And why should they? As middleware providers, they consider demand both from the end user and the provider. The user in rural Wyoming, or South America, or Eastern Europe won’t see native IPv6 to his home for years. Fortunately most of these vendors are starting to implement IPv6, after years of prodding (by Stan Barber). The feature set will grow as the user base grows.Often they will request multiple subnets per host, not knowing that “/64” means more addresses than they will use in their lifetime, and they may ask for more than one per host.
- #20 Application wise, this was easy. We did it the same as operations, we make IPv6 application look like our IPv4 tools!We found a champion to build a knowledge base and document.We found someone in our organization that is a hobbyist, that geeked out on the new technology. Every crusade needs a champion that can rally the troops. One thing we have found thus far is that users that are ordering it know how to use it. This results in very few tickets and escalations.We made it easy to adopt by making it free. We automated the allocation, so there was no wait! If ordered with a compute instance or server, we will set it up and bind it right to the host.Ultimately, by frontloading the development required for the implementation process, SoftLayer was able to streamline user adoption.