NEUTRON HYBRID MODE
Vinay Bannai
SDN Architect, Nov 8 2013
ABOUT PAYPAL
PayPal offers flexible and innovative
payment solutions for consumers and
merchants of all sizes.

• 137 Mill...
INTRODUCTION
• Data Center Architecture
• Neutron Basics

• Overlays vs Physical Networks
• Use Cases
• Problem Definition...
DATA CENTER ARCHITECTURE
Internet
Core

Data Center

Layer-3 router
Bisection BW

Aggregation

Layer-3 switch
Bisection BW...
NEW DATACENTER ARCHITECTURE
Internet

Data Center

Layer-3 router

Core

Bisection BW

Aggregation

Layer-3 switch
Bisecti...
DATACENTER WITH VSWITCHES
Data Center

Layer-3 switch

Access
VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

...
NEUTRON BASICS

7

Confidential and Proprietary
OVERLAY NETWORKS
• Overlays provide connectivity between VMs and Network
Devices using tunnels
• The physical core network...
PHYSICAL NETWORKS
• Physical Networks connect VM’s and Network Devices using
provider network
• VM’s are first class citiz...
PHYSICAL VS OVERLAY
Tenant on
Physical
Network

VM

VM

VM

VM

VM

VM

VM

VM

VM

L2
L3

L2

L2

Network Virtualization ...
PROS & CONS
Function

Bridged VMs
(VLAN)

Tunneled
VM’s

Throughput

Best

Better

Worse

Latency

Best

Better

Worse

Fl...
USE CASES
• Production Environment
− Production website across multiple data centers
− Low latency and high throughput
− B...
PROBLEM STATEMENT
• Support flexibility, low latency, high throughput and
overlapping address space all at the same time
•...
TYPICAL VSWITCH
VM
Ta

VM
Tb

VM
Tc

 HYBRID VSWITCH

VLAN 200
Hypervisor

br-int

Bridged
Traffic
Overlay
Traffic
br-tun...
CONFIGURATION OF HYBRID MODE
• Create the neutron networks
− Flat Network

− neutron net-create bridged-flat --provider:ne...
CONTD.
• Neutron networks (contd.)
− Overlay Network
− neutron net-create overylay-net
− neutron subnet-create --allocatio...
PERFORMANCE DATA
• To measure latency and throughput, we ran following tests
• Within a rack (L2 switching)
− Bare metal t...
HYPERVISOR, VM AND OS DETAILS
• Compute Hypervisors
− 2 sockets, 16 cores/socket SandyBridge @ 2.6GHz (32 Hyper Threaded)
...
TEST SETUP

Half rack with
Two Fault Zones

X.X.X.X/23

19

L3 Gateways
For Overlays

Y.Y.Y.Y/23

Confidential and Proprie...
TESTING METHODOLOGY
• Tunneling VM uses STT (OVS)
• Bridged VM uses Flat Network (OVS)
• Used nttcp 1.47 for throughput
• ...
TEST SETUP FOR SAME RACK

21

Confidential and Proprietary
WITHIN A RACK (L2 SWITCHING)
THROUGHPUT

22

Confidential and Proprietary
WITHIN A RACK (L2 SWITCHING)
PING LATENCY

23

Confidential and Proprietary
ANALYSIS
• Observations
• Results for buffer size < MTU size
− Tunneled VM’s tend to have best overall throughput

− Bridg...
TEST SETUP ACROSS RACKS

25

Confidential and Proprietary
ACROSS RACKS (L3 SWITCHING)
THROUGHPUT

26

Confidential and Proprietary
ACROSS R3ACKS (L SWITCHING)
PING LATENCY

27

Confidential and Proprietary
ANALYSIS
• No bridged VM’s in the tests (setup problem)
• Results for buffer size < MTU size
− tunneled VM’s tend to have ...
TEST SETUP ACROSS L3 GATEWAY

29

Confidential and Proprietary
ACROSS NETWORK GATEWAY
THROUGHPUT

30

Confidential and Proprietary
ACROSS NETWORK GATEWAY
PING LATENCY

31

Confidential and Proprietary
ANALYSIS
• tunneled VM’s tend to have similar if not better throughput as
bare metal or bridged VM
• tunneled VM’s have a ...
CONCLUSION & FUTURE WORK
• Understand your network requirements
− Latency, bandwidth throughput, flexibility

• Overlay Vs...
THANK YOU
vbannai@paypal.com
Upcoming SlideShare
Loading in …5
×

Neutron hybrid openstack hk

1,615 views
1,377 views

Published on

Deploying both overlay and bridged networks on the same hypervisor in Openstack

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,615
On SlideShare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
78
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • Paypal is a Payments companyBetter for merchantsEasier for youCan be considered your digital walletWe’re not just on the internet, but we are also in store in places like Home Depot and GNCScale is importantShopping habits importantPaypal scales to meet needs worldwide and can be deployed in-country depending on local laws.
  • Neutron hybrid openstack hk

    1. 1. NEUTRON HYBRID MODE Vinay Bannai SDN Architect, Nov 8 2013
    2. 2. ABOUT PAYPAL PayPal offers flexible and innovative payment solutions for consumers and merchants of all sizes. • 137 Million Active Users • $300,000 Payments processed by PayPal each minute • 193 markets / 26 currencies • PayPal is the World’s Most Widely Used Digital Wallet 2 Confidential and Proprietary
    3. 3. INTRODUCTION • Data Center Architecture • Neutron Basics • Overlays vs Physical Networks • Use Cases • Problem Definition • Hybrid Solution • Performance Data • Analysis • Q&A 3 Confidential and Proprietary
    4. 4. DATA CENTER ARCHITECTURE Internet Core Data Center Layer-3 router Bisection BW Aggregation Layer-3 switch Bisection BW Access Layer-3 switch Bisection BW Racks 4 Confidential and Proprietary
    5. 5. NEW DATACENTER ARCHITECTURE Internet Data Center Layer-3 router Core Bisection BW Aggregation Layer-3 switch Bisection BW Layer-3 switch Access Bisection BW Edge Layer VM 5 vswitches VM Confidential and Proprietary VM VM VM VM VM VM VM VM
    6. 6. DATACENTER WITH VSWITCHES Data Center Layer-3 switch Access VM VM VM VM VM VM VM VM VM VM VM VM VM VM VM Racks 6 Confidential and Proprietary Racks Racks
    7. 7. NEUTRON BASICS 7 Confidential and Proprietary
    8. 8. OVERLAY NETWORKS • Overlays provide connectivity between VMs and Network Devices using tunnels • The physical core network does not need to be re-provisioned constantly • The tunneling encap/decap is done at the edge in the virtual switch • Decouples the tenant network address from the physical Data Center network address • Easy to support overlapping address • Tunneling techniques in vogue − VXLAN − STT − NVGRE 8 Confidential and Proprietary
    9. 9. PHYSICAL NETWORKS • Physical Networks connect VM’s and Network Devices using provider network • VM’s are first class citizens with the hypervisor and the networking devices • No tunneling protocols used • Tenant separation is achieved by using VLANs or IP subnetting • Hard to achieve overlapping address spaces • Underlying network needs to be provisioned with VLANs 9 Confidential and Proprietary
    10. 10. PHYSICAL VS OVERLAY Tenant on Physical Network VM VM VM VM VM VM VM VM VM L2 L3 L2 L2 Network Virtualization Layer VM Tenant on Overlay Network
    11. 11. PROS & CONS Function Bridged VMs (VLAN) Tunneled VM’s Throughput Best Better Worse Latency Best Better Worse Flexibility Worse Better Best Overlapping IP addresses Worse Worse Best Operational Dependency 11 Hypervisor Worse Better Best Confidential and Proprietary
    12. 12. USE CASES • Production Environment − Production website across multiple data centers − Low latency and high throughput − Bridged Mode • Mergers & Acquisitions Private Community Cloud − Private Community Cloud − Needs address isolation and overlapping − Address isolation, Flexibility, low latency and high throughput − Overlay Mode • Development & QA Environment − Production development, QA & Staging − Flexibility, high throughput but can tolerate higher latency − Bridged and Overlay Mode 12 Confidential and Proprietary
    13. 13. PROBLEM STATEMENT • Support flexibility, low latency, high throughput and overlapping address space all at the same time • Support both bridged and overlay networks • VM’s on a hypervisor should be able to choose networks • Need a consistent deployment pattern • Configurable by automation tools (puppet, chef, salt etc) 13 Confidential and Proprietary
    14. 14. TYPICAL VSWITCH VM Ta VM Tb VM Tc  HYBRID VSWITCH VLAN 200 Hypervisor br-int Bridged Traffic Overlay Traffic br-tun br-bond Bond Intf Prod Interface Mgmt Interface 14 Confidential and Proprietary IP Interface
    15. 15. CONFIGURATION OF HYBRID MODE • Create the neutron networks − Flat Network − neutron net-create bridged-flat --provider:network_type=flat --provider: physical_network=<Physnet> − neutron subnet-create --allocation-pool start=10.x.x.100, end=10.x.x.200 bridged-flat --gateway 10.x.x.1 10.0.0.0/23 --name bridged-flat-subnet -enable_dhcp=False − VLAN Network − neutron net-create bridged-vlan --provider:network_type=vlan --provider: physical_network=<Physnet> --provider:segmentation_id=<vlan-id> − neutron subnet-create --allocation-pool start=10.x.x.100, end=10.x.x.200 bridged-vlan 10.x.x.1 10.0.0.0/23 --name bridged-vlan-subnet 15 Confidential and Proprietary
    16. 16. CONTD. • Neutron networks (contd.) − Overlay Network − neutron net-create overylay-net − neutron subnet-create --allocation-pool start=10.x.x.100, end=10.x.x.200 overlay-net --gateway 10.x.x.1 10.0.0.0/23 --name overlay-net-subnet • On the compute node − Configure the bond − ovs-vsctl add-br br-bond0 − Configure the OVS − ovs-vsctl br-set-external-id br-bond0 bridgeid br-bond0 − ovs-vsctl set Bridge br-bond0 fail-mode=standalone − ovs-vsctl add-port br-bond0 eth0 eth1 16 Confidential and Proprietary
    17. 17. PERFORMANCE DATA • To measure latency and throughput, we ran following tests • Within a rack (L2 switching) − Bare metal to Bare metal − Bridged VM to Bridged VM − Tunneled VM to Tunneled VM • Across racks (L3 switching) − Bare metal to Bare metal − Bridged VM to Bridged VM − tunneled VM to tunneled VM • Across the Network Gateway − Bare metal to Bare metal (outside the cloud) − Bridged VM to Bare metal (outside the cloud) 17 − tunneled VM to Bare metal (outside the cloud) Confidential and Proprietary
    18. 18. HYPERVISOR, VM AND OS DETAILS • Compute Hypervisors − 2 sockets, 16 cores/socket SandyBridge @ 2.6GHz (32 Hyper Threaded) − 2 x 10G ports (Intel PCIe) − RAM : 256GB − Disk: 4 x 600GB in RAID-10 − RHEL 6.4 running OVS • VM − vCPUs: 2 − RAM: 8GB − Disk: 20GB − RHEL 6.4 18 Confidential and Proprietary
    19. 19. TEST SETUP Half rack with Two Fault Zones X.X.X.X/23 19 L3 Gateways For Overlays Y.Y.Y.Y/23 Confidential and Proprietary X.X.X.X/23 Y.Y.Y.Y/23 X.X.X.X/23 Y.Y.Y.Y/23
    20. 20. TESTING METHODOLOGY • Tunneling VM uses STT (OVS) • Bridged VM uses Flat Network (OVS) • Used nttcp 1.47 for throughput • Bi-directional TCP with varying buffer size • Buffer size in bytes : [64,… 65536] • MTU size : 1500 Bytes (on both bare metal and VM’s) • Used ping for latency measurement (60 samples) • Used python scripts and paramiko to run the tests • Tests done with other traffic (Dev/QA) − Around 470+ active VM’s − Around 100 Hypervisors − Multiple half racks 20 Confidential and Proprietary
    21. 21. TEST SETUP FOR SAME RACK 21 Confidential and Proprietary
    22. 22. WITHIN A RACK (L2 SWITCHING) THROUGHPUT 22 Confidential and Proprietary
    23. 23. WITHIN A RACK (L2 SWITCHING) PING LATENCY 23 Confidential and Proprietary
    24. 24. ANALYSIS • Observations • Results for buffer size < MTU size − Tunneled VM’s tend to have best overall throughput − Bridged VM’s tend to better than bare metal − OVS and tunnel optimizations at play • Results for buffer size > MTU size − Tunneled VM’s and bare metal performance about the same − Bridged VM’s bests both bare-metal and tunneled VMs (??) • OVS and tunnel optimizations apply for buffer sizes smaller than MTU • OVS optimization apply for buffer sizes greater than MTU • Tunneled and Bridged VM’s have a slightly higher latency than bare metal 24 Confidential and Proprietary
    25. 25. TEST SETUP ACROSS RACKS 25 Confidential and Proprietary
    26. 26. ACROSS RACKS (L3 SWITCHING) THROUGHPUT 26 Confidential and Proprietary
    27. 27. ACROSS R3ACKS (L SWITCHING) PING LATENCY 27 Confidential and Proprietary
    28. 28. ANALYSIS • No bridged VM’s in the tests (setup problem) • Results for buffer size < MTU size − tunneled VM’s tend to have best overall throughput − OVS and tunnel optimizations at play • Results for buffer size > MTU size − tunneled VM’s and bare metal performance about the same • OVS and tunnel optimizations apply for buffer sizes smaller than MTU • tunneled and Bridged VM’s have a slightly higher latency than bare metal 28 Confidential and Proprietary
    29. 29. TEST SETUP ACROSS L3 GATEWAY 29 Confidential and Proprietary
    30. 30. ACROSS NETWORK GATEWAY THROUGHPUT 30 Confidential and Proprietary
    31. 31. ACROSS NETWORK GATEWAY PING LATENCY 31 Confidential and Proprietary
    32. 32. ANALYSIS • tunneled VM’s tend to have similar if not better throughput as bare metal or bridged VM • tunneled VM’s have a slightly higher latency • Bridged VM’s tend to have same overall throughput as the hypervisor • Bridged VM’s tend to have same latency as the hypervisor • Latency from a tunneled VM across L3 gateway is higher than Physical VMs due to extra hops, but need to re-run the tests 32 Confidential and Proprietary
    33. 33. CONCLUSION & FUTURE WORK • Understand your network requirements − Latency, bandwidth throughput, flexibility • Overlay Vs Physical • Hybrid Mode • Performance Analysis • Make your deployment patterns simple and repeatable • Future work − Additional performance tests − VXLAN, NVGRE − Varying MTU size − Setup without background traffic • Let me know if you are interested to collaborate 33 Confidential and Proprietary
    34. 34. THANK YOU vbannai@paypal.com

    ×