This document discusses IPv6-only data centers at Yandex. It notes that they transitioned to IPv6-only internally due to running out of public IPv4 addresses. Their WAN, firewalls, load balancers, and services have all been optimized for and transitioned to IPv6. They discussed initial designs for IPv6-only data center networks including separate backbone and fastbone networks. Some challenges discussed include lack of checks inside IPv6 tunnels, lack of support for 6to4 in load balancers, and needing to decapsulate traffic. They also discussed solutions for services that still need IPv4 connectivity like downloading content or using public DNS resolvers.

2. IPv6 Only Data Centers
in Yandex
Nikita V. Shirokov, network engineer, AS13238

3. Before we start
«How we launched IPv6 in Yandex» by Vladimir Ivanov
Two networks:
〉 Backbone (user's traffic handling)
〉 Technological network (so called «Fastbone»; bulk traffic)
!
https://tech.yandex.ru/events/yac/2012/talks/380/
3

4. Update on external IPv6 connectivity
mx.yandex.ru
4
mc.yandex.ru

5. Why we decided to build IPv6 only DC
〉 We ran out of public v4 addresses (and almost ran out of 1918)
〉 We think that public v6 is better that CGN for InterDC
connectivity
5

6. Points of interest
〉 WAN
〉 Firewalls and SLBs
〉 Intra DC Network (rack to rack etc)
〉 Services
6

7. WAN
IPv6 Ready
〉 MPLS 6VPE for Intra Project traffic
〉 Native IPv6 for the rest
7

8. Firewalls
8
Current: optimized IPv6 kernel code and fw!
Future: Netmap based solution!

9. SLBs
9
WAN
L2 DC Fabric

10. SLBs(cont)
10
Issues:
No checks inside tunnels
No 6over4 in LVS
Someone needs do
decapsulateL3 DC Fabric
WAN

11. Issues:
– No checks inside tunnels
– No 6over4 in LVS
– Someone needs do
– decapsulate
11
WAN
L3 DC Fabric

12. Our initial design for IPv6 only DC
12
Backbone
Fastbone

13. Services
Most of the services IPv6 ready since 2011+ (or so)
But:
〉 It’s easy to forget about monitoring
〉 or cluster’s management
〉 or replication’s transport
〉 or bootstrapping
13

14. Deploy. Phase 1
14
Backbone
Fastbone# show ipv6 bgp summary vrf …
BGP summary information for VRF …, address family IPv6 Unicast
BGP router identifier …, local AS number 65400
BGP table version is 288725, IPv6 Unicast config peers 210, capable peers 210
370 network entries and 370 paths using 45880 bytes of memory
BGP attribute entries [9/1224], BGP AS path entries [2/16]
BGP community entries [2/64], BGP clusterlist entries [0/0]

15. Not only IPv6 DC
RND: (or «I need something from github!111»)
〉 6to4 Nat. No need for high performance
Non Realtime servers needs to dl something from internet
(such as webrobots/spiders etc)
〉 4over6 tunneling or Dedicated racks with /26 IPv4 subnets
〉 Future: high performance 6to4 Netmap bassed NAT/FW
15

16. Not only IPv6 DC
!
Realtime servers needs to dl/query something from internet
(such as public dns resolvers etc)
〉 4over6 tunneling
〉 Future: high performance 6to4 Netmap bassed NAT/FW
16

17. Questions?

18. 18
Nikita V. Shirokov
Network engineer, AS13238
Contacts
tehnerd@yandex-team.ru

19. Additional Slides:
19
WAN
L2 DC Fabric
SLB check:
http get from 10.x.x.x to
10.x.x.y
external IP to VIP
external IP to 10.x.x.y
PBR: from 10.x.x.y thru 10.x.x.x

20. Additional Slides:
20
WAN
L3 DC Fabric
external IP to VIP
external IP to VIP
encapsulated into SLB to
Server
we cant http get on VIP @ SLB (will go
to local IP @ loopback). therefore cant
check if this address exist @ Server