Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

PLNOG 17 - Nicolai van der Smagt - Building and connecting the eBay Classifieds cloud


Published on

It seems everybody is talking about SDN. But where are the implementations? Nicolai talks about the intricacies of a successful cloud building project, at eBay Classifieds, and will discuss the implementation of the underlay network, virtualized overlay, hybrid cloud and MPLS integration.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

PLNOG 17 - Nicolai van der Smagt - Building and connecting the eBay Classifieds cloud

  1. 1. Building and connecting the eBay Classifieds cloud Nicolai van der Smagt Cloud Solutions Architect
  2. 2. We are Infradata We are backed by a multi-billion euro private equity group We value innovation and best-of-breed We are headquartered in the Netherlands We are a high growth SP integrator We support our customers around the globe We are highly skilled people
  3. 3. Meet eBay Classifieds ˥ Twelve brands – 1000 cities – global reach ˥ Separate division of eBay Inc. ˥ Offices in Netherlands, Germany and Canada ˥ Data centers in Netherlands, Germany, US ˥ Distributed network presence in those countries
  4. 4. eBay Classifieds in Poland is the 27th most popular website in Poland – more popular than Twitter, Linkedin and Bing
  5. 5. ECGLegacyarchitecture
  6. 6. ECG legacy infrastructure Chassis-basedeverything redundant uplinks bonding interfaces
  7. 7. eBay Classifieds needed cloud Self-service / Time-To-Market Automation / Lower Opex Lineair scalability Fault tolerance / Reliability
  8. 8. Cloud requirements No monoculture but ecosystem Generic rack layout for scalability Built in 6 months Integrate with legacy (hybrid cloud) DEV, QA, PROD IaaS functionality Multiple regions
  9. 9. Deploy more regions and features Learn and build experience Launch cloud with limited feature set Build first region (AMS) Multi-rack validation test Single rack, experimental deployment / PoC Approach 3-way partnership: eBay Classifieds, Infradata, Juniper Networks 3-way partnership: eBay Classifieds, Infradata, Juniper Networks
  10. 10. ECGCloudarchitecture Underlay network Compute and storage infrastructure Overlay network and orchestration Overlay network and orchestration Cloud interconnection network
  11. 11. Cloud orchestration Openstack Kilo architecture
  12. 12. Cloud orchestration ˥ Endpoint for compute, network and storage abstraction layers ˥ GUI and CLI clients, API ˥ Scheduling ˥ Exposes infrastructure to administrators for operational purposes ˥ Exposes infrastructure to end users (ie. eBay Classifieds developers)for self-service ˥ Developers use infrastructure-as-code tooling (such as Terraform)to quickly deploy resources in cloud
  13. 13. Underlay fabric ⌉ Built on proven tech (mini version of Internet) ⌉ L3 (IP) only ⌉ Ultra simple ⌉ Load-balancing via ECMP ⌉ Fine-grained policies ⌉ Linear scaling ⌉ Fault tolerance ⌉ Commoditization COMPUTE SPINE LEAF
  14. 14. Underlay fabric @ eBay Classifieds 12 Spine switches: 32x40G 32 Leaf switches: 24x40G We use 12 * 40G for uplink from leaf to spine (1x40G to each spine) We use 12 * 40G for downlink from leaf, each of them split into 4, allowing for 48 10G connections in the rack Juniper QFX-5100-24Q – 24 ports of 40G (or 32 ports with breakout module) Topology independent ISSU (saves leaf layer cost!) Each switch his own BGP ASN in IP fabric
  15. 15. IP fabric – single routing protocol – eBGP ˥ Leafs peer with all spines, using p2p interface addresses ˥ Leafs only advertise connected subnets (ie. compute/storage nodes) S1 – AS 65001 S2-ASN 65002 S3-ASN 65003 S12- ASN65012 L1 – ASN 65101 L2 – ASN65102 L3 – ASN65103 L32 – ASN 65132 12 Spine switches: 32x40G 32 Leaf switches: 24x40G
  16. 16. Leaf network – Top of Rack switches ˥ A single QFX5100-24Q per rack ˥ ISSU for software upgrades ˥ Upstream connectivity to 12 spines with 40GE ˥ Connecting 30 compute nodes and a storage node using 10GE ˥ Configured by ZTP
  17. 17. Zero Touch Provisioning 1. Switch boots, requests IP 2. Switch receives IP, requests config script 3. Script investigates location 4. Script upgrades Junos 5. Script creates config based on location (loopback/link addressing and BGP peerings) 6. We’re live
  18. 18. Spine network – External backplane ˥ 12x QFX5100-24Q per data center ˥ Each spine connects to all leafs (40GE) ˥ External backplane ˥ BGP sessions to all leafs (10 BGP sessions per spine) ˥ Each switch in the fabric is an EBGP AS ˥ No oversubscription anywhere in the fabric
  19. 19. L3 Network L3 Network Physical Topology Logical Topologies ⌉ Enables multi-tenancy ⌉ Distributed routing and security ⌉ “Glue” between Openstack and underlay network ⌉ Software-defined ⌉ Central control ⌉ Signalled with XMPP and BGP Router/ Firewall L3 Network L3 Network Overlay network – SDN
  20. 20. Juniper Contrail Architecture Physical Network (no changes) Analytics OPENCONTRAIL CONTROLLER ControlConfiguration Physical Host with Hypervisor vRouter VM VM VM VM Physical Host with Hypervisor vRouter VM VM VM VM WAN, Internet Gateway Assigns networkandsecurity based on orchestrator requests for VM creation Real-timeanalytics engine collects, stores and analyzes networkand security telemetryConfigures physical and virtual networkelements, keepsstate (control plane) vRouter: Virtualized routing element handles localized control planeand forwarding plane workon the compute node Gateway: Any carrier-grade routercan serve as gateway eliminatingneedfor SW gateway, improving scale and performance BGP API VXLAN or GRE or UDP
  21. 21. Opencontrail open source approach OpenContrail Advisory B oard (OCAB)  Industry veterans and key project users/adopters  Governance, Evangelism, Roadmap, Operational efficiency Continuous Integration/Development Features & Bug fixes Single Github Source Code Repository OpenContrail Community Release Community Support (Email, IRC, Forums) Juniper Contrail Releases  Hardened for Production  Licensed Software  24x7 JTAC & Engineering B ug Fix Release Launchpad Ope n Source (Use rs, Devs) Bugs, Design Blueprints Customers xx OpenContrail Developer Community  Majority Juniper, Some External developers  Proposing features & Contribute Code  Participate in Code review process Community Release
  22. 22. Contrail key features Routing & Switching (IPv4, v6) Network Services (IPAM, DNS, DHCP SNAT, FIP, QoS) Load Balancing (customizable ECMP) Security Policy Enf., Distributed FW 3rd Party Netw. Svc. Gateway Services (L2, L3 GW) Rich Analytics, Overlay-Underlay Correlation Service Chaining (PNF, VNF, etc.) High Availability API Services (multi-vendor Orch.)
  23. 23. Distributed security policies in Contrail Legacy Data Center security Overlay Data Center security
  24. 24. Ebay Classifieds MPLS backbone ˥ AS41552 ˥ Juniper MX edge routing ˥ Legacy Cisco core ˥ Connectivity to AMSIX, LINX, DECIX, NLIX and loads of transits for best connectivity to end users
  25. 25. Contrail L3VPN gateway functionality Contrail Controller VM R1 VM R2 BGP + Netconf L3VPN Overlay Tunnel MPLS/GRE Route Reflector LSP (RSVP, LDP) BGP Gateway Router (PE Router) Physical L3VPN Red Virtual Network VM VM VM VM
  26. 26. L3VPN for Cloud Interconnect Data Center 1 VM G1 VM G2 VM G3 Green Virtual Network VM R1 VM R2 VM R3 Red Virtual Network L3VPN L3VPN Data Center 2 VM G4 VM G5 VM G6 Green Virtual Network VM R4 VM R5 VM R6 Red Virtual Network
  27. 27. eBay Classifieds hybrid cloud Core Ne twork Core router Core router Juniper MX80 Juniper MX80 Legacy infrastructure Contrail control Contrail control Contrail control L3 VPN Gateway functionality C loud Infra BGP session XMPP session MPLSoGRE flow
  28. 28. It works! Cloud in production since 2016 (300 nodes in Amsterdam) Expansion to Düsseldorf ongoing Developers strongly prefer cloud over legacy infra All managed by 4-person cloud team without headaches
  29. 29. Up next… Upgrade Openstack(Kilo to Newton) andContrail(2.2 to 3.0) •Still not trivial Load Balancingas a Service (LBaaS) v2 ˥ Contrail now supports lots of LBaaS providers (including Haproxy, F5 Networks, A10 Networksand AviNetworks) ˥ Automatically create load-balancers fromTerraform – Infrastructure-as-code Platform as a Service/Containers(PaaS/CaaS)? ˥ Developers implement things like Kubernetes today, themselves ˥ Better to providesupported option fromthe cloud DNSaaS(OpenstackDesignate)
  30. 30. @NicolaivdSmagt