ScaleDay conference

2. Delbecq Adrien
Network SRE
IP Fabric :
How to design a scalable, multi-
services datacenter network ?

3. • ~ 100K physical servers
• 7 Tbps of Internet capacity
• 5 datacenters
• Several platforms, 3 – 4 network
architectures
• Thousands of switches
• Vendor based
• Homemade
Our Infrastructure
3

4. Summary
• Need
• Legacy topology
• From SP to DC
• Overlay
• Underlay
• What’s next ?
IP fabric
4

5. Scalable network
IP Fabric : Need
5

6. Multi-service network : isn’t it VPC ?
IP Fabric : Need
6

7. • Resilient
• Scalable• Multi Service
• Multi Service
• Fast to design and deploy
IP Fabric : Target
7

8. Physical topology : 3 tiers design
IP Fabric : legacy solution
8

9. Multi-service – l2 isolation – dot1q, q-in-q
IP Fabric : legacy solution
9

10. Multi-service – l2 loop prevention & link usage
IP Fabric : legacy solution
10

11. Multi-service – l3 isolation - vrf-lite
IP Fabric : legacy solution
• Provide layer 3 isolation
• Hard to propagate
11

12. What’s Old is New Again
• Clos topology since 50’s
IP Fabric : from SP to DC
12

13. What’s Old is New Again
• VPN services since 2000’s
• L2VPN :
• Pseudo wire
• Multi-point L2VPN
• L3VPN
IP Fabric : from SP to DC
13

14. Do it, Strate it, abstract it !
• Layered network
• Underlay : SP/DC support network
• Overlay : customers, services network
IP fabric : from SP to DC
14

15. Do it, Strate it, abstract it !
IP fabric : from SP to DC
15

16. Do it, Strate it, abstract it !
IP fabric : from SP to DC
16

17. Do it, Strate it, abstract it !
IP fabric : from SP to DC
17

18. Layer1 : Remember Clos
IP fabric : Underlay
18

19. Clos scale : more ingress/egress ?
IP fabric : Underlay
19

20. Clos scale : more bandwith
IP fabric : Underlay
20

21. Grow more with Clos
IP fabric : Underlay
21

22. External connectivity ?
IP fabric : Underlay
22

23. Data Plane : IPv4
IP fabric : Underlay
• No extended broadcast domain
• DC shelves not really ready for MPLS
underlay
• IPv6 not ready with chosen solution
• L3 sub-interface everywhere
• Efficient loop prevention
• ECMP : 100% bandwith used
23

24. Control Plane : eBGP
IP fabric : Underlay
• No link-state protocol
• No OSPF
• No IS-IS
• iBGP isn’t really good as IGP
• eBGP just fits
• RFC7938 – draft Lapukhov
24

25. Control Plane : eBGP
IP fabric : Underlay
25

26. Addressing plan
IP fabric : Underlay
• Internet-like addressing plan
• Use next available prefix
• No waste
• Topology –Driven Addressing
• IP address = function ( topology )
• Human-friendly
26

27. Addressing plan : Topology-Driven Addressing
IP fabric : Underlay
27

28. Data Plane : VXLAN Encapsulation
IP fabric : Overlay
• Virtual eXtensible Local Area Network
• Standard
• No vendor lock-in
• Available on standard DC shelf
• Native Hardware support
28

29. Data Plane : VXLAN Encapsulation
• Important concepts :
• VXLAN ID : VNI , up to 16M ID
• L2 over UDP
• Device performing (des)encaps : VTEP
IP fabric : Overlay
29

30. Data Plane : VXLAN Encapsulation
IP fabric : Overlay
30

31. Control Plane : MP-BGP EVPN
IP fabric : Overlay
• Already used with MPLS
• Basic BGP :
– iBGP in our use case
– Specific NLRI
» type2 : hosts advertisement
» type5 : prefixes advertisement
– Next-HOP are VTEPs
31

32. VXLAN with BGP-EVPN, using RR
IP fabric : Overlay
32
• Using route-reflection
64 leaf, 2 rr
128 iBGP sessions
• Using full-mesh :
2016 iBGP sessions

33. End Device redundancy
• we choose MC-LAG
• No ESI
• No pure L3
IP fabric : what else ?
33

34. IP fabric journey continue
• Software VTEP
• Standalone device under the leaf
• More fabric
• Bigger scale
• Sharding
• Datacenter Interconnection
IP fabric : what’s next?
34

35. Follow our news, exclusive tutorials and cloud infos on Twitter and
LinkedIn @Scaleway
Thank you
And follow me on twitter @Adelbecq
35