Downloaded 44 times
Uploaded byShawn Zandi
LinkedIn's Approach to Programmable Data Center
LinkedIn's approach to its infrastructure focuses on a programmable data center architecture that relies on robust, scalable control protocols and efficient traffic distribution methods. The system prioritizes a unified architecture using a single SKU for hardware and software across all switches, while removing complexity through a direct management of the end-to-end stack. Key projects like 'Project Falco' and 'Open19' aim to enhance data center networks by decoupling hardware from software and promoting a simplified infrastructure for better control.
More Related Content
Similar to LinkedIn's Approach to Programmable Data Center
LinkedIn's Approach to Programmable Data Center
- 1. LinkedIn’s Approach toProgrammable Data Center Shawn Zandi Principal Architect Infrastructure Engineering
- 2. LinkedIn Infrastructure • Infrastructurearchitecture based on application’s behavior & requirements • Pre-planned static topology • Single operator • Single tenant with many applications • As oppose to multi-tenant with different (or unknown) needs • 34% infrastructure growth on annual basis and close to half a billion users
- 3. Edge Network toEyeballs Backbone Network Bare Metal Operating System Container Application End to End Network Design Data Center Network End to end control enables us to tackle problems at different parts of the stack From application code, os, network or client software to solve by architecture… Bare Metal Operating System Container Application
- 4. Traffic Demands • Highintra and inter-DC bandwidth demand due to organic growth • Every single byte of member activity, creates thousands bytes of east-west traffic inside data center: • Application Call Graph • Metrics, Analytics and Tracking via Kafka • Hadoop and Offline Jobs • Machine Learning • Data Replications • Search and Indexing • Ads, recruiting solutions, etc.
- 5. Scaling Out DataCenters Network - Hardware • White-box Switches (ODM) • Vendor Switches Based (OEM) • Based on Merchant Silicon • Big Chassis Switches • Designed around robustness (NSR, ISSU, etc.) • Feature-rich but mostly irrelevant to LinkedIn needs Project Falco
- 6. Data center weredesigned by redundant chassis at core controlling and forwarding east-west and north-south traffic
- 7. PodW SpineSpineSpine LeafLeafLeafLeaf Spine PodX SpineSpineSpine LeafLeafLeafLeaf Spine PodY SpineSpineSpine LeafLeafLeafLeaf Spine PodZ SpineSpineSpine LeafLeafLeafLeaf Spine Fabric 2 Spine SpineSpine Fabric 4 Spine Spine Spine Fabric 1 Spine Spine Spine 4,096 x100G ports Non-Blocking Scale-out Spine Spine Spine Fabric 3 Chassis Free Data Center
- 8. Why No Chassis? •Robust-yet-Fragile • Complex due to NSR, ISSU, feature-sets, etc. • Larger fault domain, Fail-over/Fail-back • Indeterministic boot up process and long upgrade procedures • Moved complexity from big boxes to our advantage, where we can manage and control! • Better control and visibility to internals by removing black-box abstraction! • Same Switch SKU on ToR, Leaf and Spine (Entire DC) • Single chipset uniform IO design (same bandwidth, latency and buffering) • True 5-Staged Clos Topology! with deterministic latency • Dedicated control plane, OAM and CPU for each ASIC
- 9. W X YZ W X Y Z W X Y Z Distributed Control Plane Complexity Pod 1 2 32…1 Pod 11 322 352…321 Pod 21 642 672…641 Pod 31 962 992…961 W X Y Z 2171217021692168213121302129212820912090208920882051205020492048 2339233823372336 2368 2369 2370 2371 2400 2401 2402 24032307230623052304
- 10. “Fabric wide visibilityand telemetry” The wider the fabric, flow tracking and fault isolation becomes more difficult Problem 1
- 11. “Fabric wide trafficdistribution and packet scheduling!” Forwarding is different than routing, and out of scope for routing protocols. Problem 2 We need a robust and scalable control protocol designed for a data center fabric
- 12. Control Plane ::Routing • Routing protocols provide destination-based reachability information • Routing protocols are not traffic aware. • Best path selection is elementary. • Network graph is built based on series of ECMP groups, “Routing protocols are more about the destination than the journey”
- 13. ECMP forwarding simplydoes not cut it! Problem #3
- 14. ECMP is notreally equal! • Elephants and mice issue • ECMP Hashing is not bandwidth aware. Devices use an algorithm to distribute traffic amongst links regardless of load. • Traffic is routed using shortest path, not all the available paths, hence not maximizing all the available capacity. Some links may suffer while the other may be underutilized. • Flows stick to a certain path, as hashing is performed per flow. An established socket cannot be moved to a different path easily!
- 15. “We need arobust and scalable fabric-wide forwarding policy” Problem #4
- 16. Lack of CentralizedPolicy and Control • The more parallel links you add, forwarding decision becomes more random. • Devices were configured and maintained individually • Routing/Forwarding policy management tasks are performed individually and hop by hop. • Know when/where to centralize or distribute to scale out!
- 17. “End to EndPath Selection & Control” No application, protocol or packet can dictate a path Centralized flow based routing does not scale! Problem #5
- 18. “Using the samefamiliar, robust and well-known solutions brings along the same restrictions when they were originally designed” Problem #6
- 19. Hardware Network Transport Application BGP (1990s) Clos Topology(1950s) Ethernet & IP (1980s)
- 20. IP Routing History •IP routing is defined hop-by-hop • BGP is “the” IDR designed to work between different autonomous system, to provide policy and control between different routing domains to select a best path. • True: BGP can scale and is extensible. BGP has many policy knobs. • A datacenter fabric operated under a single administrative domain instead of series of individual routers with different policies and decision process.
- 21. Forwarding traffic basedon demands & patterns: • Application • Latency • Loss • Bandwidth (Throughput) Programmable Data Center A data center fabric that distributes traffic amongst all available links efficiently and effectively while maintaining lowest latency and providing the most possible bandwidth to different applications based on different needs and priorities.
- 22. Program forwarding tablesindividually on all switches from a centralized location Approach #1
- 23. Flow x >Port 1 Flow x > Port 3 Flow x > Port 2 Forwarding and Control Element Separation
- 24. Encode path informationinto packet header Approach #2
- 25. Distributed control planefor topology discovery and reachability information + Use a controller software for forwarding policy and optimizations Approach #3
- 26. Scale: No stateor flow information required to be stored on every box Network can choose and move flows dynamically Application can choose and move flows dynamically Works with existing data plane (merchant silicon support) Supports ECMP with fallback to IP routing Automatic Local Repair / LFA
- 27. Hardware Routing Policy Applications Link Selection andScheduling Topology Discovery and Network Graph Control Telemetry/Visibility, Machine Learning, Prediction Engine, Self Healing, etc. Forwarding Merchant Silicon Rethinking The Network Stack
- 28. Network Element Management Plane SNMP, Syslog,etc. System & Environmental Data Packet & Flow Data Network Operating System Kafka Network Agent ASIC System Drivers Reducing Protocols
- 29. Network ElementNetwork ElementNetworkElementNetwork Element Management Plane SNMP, Syslog, etc. System & Environmental Data Packet & Flow Data Network Operating System Kafka Agent Monitoring and Management System Kafka Broker Machine Learning & Data Processing Alert Processor Log Retention Data Store Event Correlation Kafka Pub/Sub Pipeline Record, Process and Replay Network State
- 30. Open19 OpenFabric ASIC ODM RIB /Forwarding Abstraction Layer FALCO Apps Linux OS Hardware Physical Layer Hardware Abstraction Layer Metrics & Analytics Machine Learning Self Healing etc. (API to Infrastructure) Policy & Control Operating System Base Networking LinkedIn Infrastructure Strategy
- 31. • Unified Architecture •We used a single SKU (hardware and software) for all switches while procuring hardware from multiple ODM channels (multi-homing) • One Software: Base Networking on Merchant Silicon with minimum req. features • No Overlay - For the infrastructure, the application is stateless • No Middle-box (Firewall, Load-balancer, etc.) Moved to application • Network is only a set of intermediate boxes running linux Simplified Infrastructure to Own
- 32. • To controland own your architecture: • End to end stack (app, operating system, network and architecture.) • Ultimate sophistication: Simplicity • In house support as far as possible • Move complexity to your comfort zone! Stay in Control
- 33. • SDN isnor a protocol or a tool or product off the shelf • SDN is the whole network stack and architecture that enables applications to meet and interact with infrastructure to: SDN for LinkedIn • Discover and Learn • Provision • Manage • Control • Monitor
- 34. Project Altair: TheEvolution of LinkedIn’s Data Center Network Project Falco: Decoupling Switching Hardware and Software Open19: A New Vision for the Data Center
Editor's Notes
- #4 This is the luxury that public cloud doesn’t offer: solving scale or performance issues, many different possible ways as we own the end to end stack from the application serving in the DC to the app on clients phone! Scale Example: one possible approach is to throw more machines at it We believe if application performance and user experience is our primary mission, owning end to end infrastructure is not optional. That is why we do not have any plan to move to any infrastructure outside LinkedIn’s control from top to bottom.
- #5 The infrastructure growth more than anything was on data center networking as traffic demands.
- #6 and we chose to build our network on top of merchant silicon. a very common strategy for mega scaled data center.
- #7 Let’s look at the history of data center network: DCN were built by same building blocks that constructs campus or enterprise networks, chassis architecture and hierarchical access, distribution and core model of deployment Vendors: same set of software features set, protocols and architecture was being used for campus, enterprise and data center networks.
- #8 Making a big switch out of pizza box switches that can scale horizontally. In this model each plane can have up to 32 switches, with 4 planes we can have 128 switches in the fabric. That is 4,096 x100 Gig ports. You cannot buy a chassis switch with 4096 ports in the market.
- #9 No control over what line cards boot first, and black holing traffic until software is fully loaded and control plane in sync Chassis switches simplify management and control plane of multiple line cards (modules) by managing multiple chipsets under the same roof, as the code abstracts the complexity inside the chassis from the rest of the network however it translates into code complexity where we did not have control over. Our strategy is to simplify what we don’t have control over, and move complexity to where we can manage and control. We can manage multiple pizza-box switches via distributed control plane, software automation and zero touch provisioning… In our case, we used the same switch code and chipset as the building block for the entire data center where operation staff no longer need to focus on hardware variables and can shift their attention to the software and automation pieces.
- #10 As we grow there will be more switches and links to manage, the more the feel and need to enhance programmability into the network. This wide ecmp structure creates complexity
- #12 chassis switches provide cell switching, dbd, virtual output queue (head of the line blocking) across backplane, since there’s no big enough switch to serve the whole data center, this task should be performed across the fabric!
- #13 non of the routing protocols were designed (originally) for a fully mesh, ecmp network, extensions to support multi path and features such as mesh group added later, but it does not really solve this use case, routing protocols says: here is the destination prefix, and here is the next hop, good luck with that
- #15 Folded CLOS is not perfect, once that network utilizes more bandwidth we will see some links carry more traffic than the others because ECMP algorithm or equal cost multi path (sharing bandwidth across multiple links) is based on a hashing mechanism.
- #17 Once that a path (a series of links) instead of series of nodes is determined, apply and enforce this in an end to end . Distributed architectures are usually faster to compute, but depending on the case, there are examples that centralize compute can be faster than overall distributed form, due to eliminating duplicated tasks and event propagation…
- #19 BGP in DC not the complete answer. It works for what is expected for BGP, not necessarily for what expected in a data center. BGP does a perfect job for the intent behind its design, the intent was to have a good enough, path-vector protocol to scale to internet size with policy control…
- #20 Although protocols will change over time as they gain new functionality, the foundation and fundamental thought process behind them when there were designed does not or cannot be changed.
- #21 While we are looking for fabric based forwarding, higher-level intent based forwarding, operation is bgp brings an illusion of control in the dc, ask those operator if they really control whats happening there?
- #24 ForCES and OpenFlow, providing flow based routing We already know that this does not scale!
- #25 Examples, Segment Routing, MPLS, or Cisco ACI by utilizing VXLAN
- #31 OpenFabric is open and extensible software code that extends to run on different silicons and operating systems. Open19 provides passive backplane for data without a need of wiring and power rails to fit servers, storage and switches in standard 19 inch cabinets.
- #32 Simplified architecture is desirable to simplify ownership
- #33 We believe it is important as a content provider company to control our own destiny, own to customize based on your needs, and only your use cases Priority #1 is to stick to and own our architecture, the simple architecture that is driven by our application. in order to control this we need to own several pieces of this puzzle.