How Quantum configures Virtual Networks under the Hood?
Upcoming SlideShare
Loading in...5
×
 

How Quantum configures Virtual Networks under the Hood?

on

  • 9,497 views

 

Statistics

Views

Total Views
9,497
Views on SlideShare
8,522
Embed Views
975

Actions

Likes
35
Downloads
502
Comments
0

35 Embeds 975

http://packetpushers.net 686
https://twitter.com 49
https://t.co 48
http://www.gossamer-threads.com 25
http://www.scoop.it 25
http://chavezy.wordpress.com 23
https://www.evernote.com 16
https://mail.google.com 14
http://wiki.anutanetworks.com 11
http://webchat.freenode.net 8
http://lists.openstack.org 8
https://wiki.csc.fi 7
https://si0.twimg.com 5
https://www.facebook.com 5
http://moderation.local 5
http://216.220.39.53 4
https://kippt.com 4
https://chavezy.wordpress.com 4
http://m1-wiki.us.dell.com 3
http://getpocket.com 3
http://192.168.73.2 3
http://www.readability.com 3
http://weibo.com 3
http://t.co 2
http://uk-mg42.mail.yahoo.com 1
https://www.google.fr 1
https://192.168.73.2 1
https://abs.twimg.com 1
https://twimg0-a.akamaihd.net 1
https://mail.ilimit.net 1
https://todoist.com 1
https://arista.app.box.com 1
http://hootsuite.com 1
https://delicious.com 1
https://cms.prod.bloomberg.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

How Quantum configures Virtual Networks under the Hood? How Quantum configures Virtual Networks under the Hood? Presentation Transcript

  • Quantum Under the Hood Open Cloud Campus How Quantum Configures Virtual Networks Under the Hood? 2012/12/10 ver1.2 E.Nakai (Twitter @enakai00)
  • Quantum Under the Hood Physical and Logical Configuration Open Cloud Campus2
  • Quantum Under the Hood Physical Network Connection and Agent Placement Public Network Management Network Instances are launched インスタンスが Virtual Routers on these nodes 起動するノード are configured here eth0 eth2 eth0 eth0 Network Node Quantum Service Compute Node Compute Node L2 Agent L2 Agent L2 Agent DHCP Agent L3 Agent Quantum Service can be eth1 eth1 eth1 placed on anther node Private Network Open Cloud Campus3
  • Quantum Under the Hood Logical Configuration of the Virtual Network - Case1 Some network can be dedicated to a specific tenant Each instance can choose connected private network.Connections from/to Public Network (Multiple choice is allowed.) are NAT-ed here NAT Router and Firewall Public Network Packets from different subnet are filtered with the “Security Group.” net01 net02 Private Network Private Network (Subnet01) (Subnet02) Packets between same subnet are not filtered. Open Cloud Campus4
  • Quantum Under the Hood Logical Configuration of the Virtual Network - Case2  By assigning a dedicated router for each tenant, each tenant can create its own private networks freely, even overlapping subnets can be used by multiple tenants. Public Network Connections from/to other Tenants or Public Network are NAT-ed here Router for Router for TenantA TenantB Subnet01 Subnet02 Subnet03 Subnet04 192.168.1.0/24 192.168.2.0/24 192.168.1.0/24 192.168.2.0/24 net01 net02 net03 net04 Open Cloud Campus5
  • Quantum Under the Hood Notes on the Configuration  There are various plugins for L2 Agent. In this document, we describe “LinuxBridge plugin” and “Open vSwitch plugin” which provides basic bridge functionality using Linux bridge / Open vSwitch.  The single Network Node could be a potential bottle neck and a single point of failure. There is a lot of design discussion about it in the upstream.  Currently dnsmasq is used by DHCP agent as in the following charts, but different DHCP options are possible by design. Other options may be added in the future.  Network Namespace support is _not_ mandatory for Case1 configuration. If you want to use multiple routers (and optionally the overlapping IP address feature), however, you have to use Case2 design and Network Namespace support is required there.  It depends on Linux distributions whether you can use the Network Namespace feature. For example, RHEL6.3/6.4 doesnt support it. Fedora17/18 supports it. Open Cloud Campus6
  • Quantum Under the Hood Network Components configured by LinuxBridge Plugin using VLAN separation - Case1 Configuration - NAT Router and Firewall Public Network net01 net02 Private Network Private Network (Subnet01) (Subnet02) Model Network of this Section vm01 vm02 vm03 Open Cloud Campus7
  • Quantum Under the Hood Network Components on Compute Nodes TAP device Configured by Nova Compute VLAN device vm01 vm02 vm03 Linux Bridge IP IP IP IP eth0 eth0 eth1 eth0 IP is assigned via dnsmasq running on Network Node tap01 tap02 tap03 tap04 brqXXXX brqYYYY net01 net02 VLAN device is created for eth1.101 eth1.102 each private network Configured by L2 Agent eth1 VLAN101 Physical L2 Switch for Private Network VLAN102 Open Cloud Campus8
  • Quantum Under the Hood Network Components on Network Nodes To Public Network Configured by L3 Agent eth0 veth pair Linux Bridge tapVVV brqxxxx VLAN device NAT with iptables IP qg-VVV dnsmasq dnsmasq dnsmasq is assigned to each subnet IP IP IP IP ns-XXX qr-YYY qr-ZZZ ns-WWW tapXXX tapYYY tapZZZ tapWWW Configured by DHCP Agent brqxxxx brqxxxx Suffix “xxxx” is derived from net01 net02 the head of network UUID. eth1.101 eth1.102 Configured by L2 Agent eth1 To Private Network Open Cloud Campus9
  • Quantum Under the Hood Network Namespace Separation To Public Network eth0 Suffix “bbbb” corresponds Network to router UUID. Namespace tapVVV brqxxxx qrouter-bbbb IP qdhcp-cccc qdhcp-aaaa qg-VVV Suffix “aaaa” corresponds to network UUID. dnsmasq dnsmasq IP IP IP IP ns-XXX qr-YYY qr-ZZZ ns-WWW tapXXX tapYYY tapZZZ tapWWW brqxxxx brqxxxx net01 net02 eth1.101 eth1.102 eth1 To Private Network Open Cloud Campus10
  • Quantum Under the Hood Network Components configured by LinuxBridge Plugin Plugin using VLAN separation - Case2 Configuration - Public Network Router for Router for TenantA TenantB Model Network of this Section vm01 vm02 vm03 vm04 Open Cloud Campus11
  • Quantum Under the Hood Network Components on Compute Nodes TAP device Configured by Nova Compute VLAN device vm01 vm02 vm02 vm03 Linux Bridge IP IP IP IP eth0 eth0 eth0 eth0 IP is assigned via dnsmasq running on Network Node tap01 tap02 tap03 tap04 brqXXXX brqYYYY net01 net02 VLAN device is created for eth1.101 eth1.102 each private network Configured by L2 Agent eth1 VLAN101 Physical L2 Switch for Private Network VLAN102 Open Cloud Campus12
  • Quantum Under the Hood Network Components on Network Nodes To Public Network Configured by L3 Agent brqxxxx eth0 veth pair Linux Bridge tapZZZ tapCCC NAT with iptables VLAN device IP IP qg-ZZZ qg-CCC dnsmasq dnsmasq dnsmasq is assigned to each subnet IP IP IP IP ns-XXX qr-YYY qr-BBB ns-AAA tapXXX tapYYY tapBBB tapAAA Configured by DHCP Agent brqxxxx brqxxxx Suffix “xxxx” is derived from net01 net02 the head of network UUID. eth1.101 eth1.102 Configured by L2 Agent eth1 To Private Network Open Cloud Campus13
  • Quantum Under the Hood Network Components on Network Nodes To Public Network brqxxxx eth0 Network Suffix “bbbb” corresponds Namespace to router UUID. tapZZZ tapCCC IP qrouter-bbbb IP qrouter-cccc qdhcp-aaaa qg-ZZZ qg-CCC qdhcp-dddd Suffix “aaaa” corresponds to network UUID. dnsmasq dnsmasq IP IP IP IP ns-XXX qr-YYY qr-BBB ns-AAA tapXXX tapYYY tapBBB tapAAA brqxxxx brqxxxx net01 net02 eth1.101 eth1.102 eth1 To Private Network Open Cloud Campus14
  • Quantum Under the Hood Network Components configured by Open vSwitch Plugin using VLAN separation - Case1 Configuration - NAT Router and Firewall Public Network net01 net02 Private Network Private Network (Subnet01) (Subnet02) Model Network of this Section vm01 vm02 vm03 Open Cloud Campus15
  • Quantum Under the Hood Network Components on Compute Nodes Configured by Nova Compute TAP device vm01 vm02 vm03 IP IP IP IP eth0 eth0 eth1 eth0 veth pair Linux Bridge vnet0 vnet1 vnet2 vnet3 qbrXXX qbrYYY qbrZZZ qbrWWW Open vSwitch qvbXXX qvbYYY qvbZZZ qvbWWW qvoXXX qvoYYY qvoZZZ qvoWWW Port VLAN tag:1 Port VLAN tag:2 Tenant flows are separated br-int by internally assigned VLAN ID int-br-eth1 VLAN ID is converted with flow table Configured by L2 Agent dl_vlan=101 ⇒ mod_vlan_vid:1 phy-br-eth1 dl_vlan=102 ⇒ mod_vlan_vid:2 br-eth1 Tenant flows are separated by user defined VLAN ID eth1 VLAN ID is converted with flow table dl_vlan=1 ⇒ mod_vlan_vid:101 dl_vlan=2 ⇒ mod_vlan_vid:102 VLAN101 Physical L2 Switch for Private Network VLAN102 Open Cloud Campus16
  • Quantum Under the Hood Network Components on Network Node To Public Network Internal port eth1 veth pair br-ex phy-br-ex NAT with iptables qg-VVV Open vSwitch IP dnsmasq Configured by L3 Agent dnsmasq dnsmasq is assigned to each subnet IP IP IP IP tapXXX qr-YYY qr-ZZZ tapWWW Configured by DHCP Agent Port VLAN tag:1 Port VLAN tag:2 br-int int-br-ex int-br-eth1 VLAN ID is converted with flow table Configured by L2 Agent dl_vlan=101 ⇒ mod_vlan_vid:1 dl_vlan=102 ⇒ mod_vlan_vid:2 phy-br-eth1 br-eth1 VLAN ID is converted with flow table dl_vlan=1 ⇒ mod_vlan_vid:101 eth1 dl_vlan=2 ⇒ mod_vlan_vid:102 To Private Network Open Cloud Campus17
  • Quantum Under the Hood Network Namespace Separation To Public Network Network eth1 Namespace br-ex phy-br-ex qg-VVV qrouter-bbbb IPSuffix “bbbb” corresponds to router UUID. dnsmasq dnsmasq qdhcp-aaaa qdhcp-cccc IP IP IP IP Suffix “aaaa” corresponds tapXXX qr-YYY qr-ZZZ tapWWW to network UUID. Port VLAN tag:1 Port VLAN tag:2 br-int int-br-ex int-br-eth1 phy-br-eth1 br-eth1 eth1 To Private Network Open Cloud Campus18
  • Quantum Under the Hood Network Components configured by Open vSwitch Plugin using VLAN separation - Case2 Configuration - Public Network Router for Router for TenantA TenantB Model Network of this Section vm01 vm02 vm03 vm04 Open Cloud Campus19
  • Quantum Under the Hood Network Components on Compute Nodes Configured by Nova Compute vm01 vm02 vm03 vm04 TAP device IP IP IP IP eth0 eth0 eth0 eth0 veth pair Linux Bridge vnet0 vnet1 vnet2 vnet3 qbrXXX qbrYYY qbrZZZ qbrWWW Open vSwitch qvbXXX qvbYYY qvbZZZ qvbWWW qvoXXX qvoYYY qvoZZZ qvoWWW Port VLAN tag:1 Port VLAN tag:2 Tenant flows are separated br-int by internally assigned VLAN ID int-br-eth1 VLAN ID is converted with flow table Configured by L2 Agent dl_vlan=101 ⇒ mod_vlan_vid:1 phy-br-eth1 dl_vlan=102 ⇒ mod_vlan_vid:2 br-eth1 Tenant flows are separated by user defined VLAN ID eth1 VLAN ID is converted with flow table dl_vlan=1 ⇒ mod_vlan_vid:101 dl_vlan=2 ⇒ mod_vlan_vid:102 VLAN101 Physical L2 Switch for Private Network VLAN102 Open Cloud Campus20
  • Quantum Under the Hood Network Components on Network Node To Public Network Internal port eth1 veth pair br-ex phy-br-ex NAT with iptables qg-ZZZ qg-CCC Open vSwitch IP IP dnsmasq Configured by L3 Agent dnsmasq dnsmasq is assigned to each subnet IP IP IP IP tapXXX qr-YYY qr-BBB tapAAA Configured by DHCP Agent Port VLAN tag:1 Port VLAN tag:2 br-int int-br-ex int-br-eth1 VLAN ID is converted with flow table Configured by L2 Agent dl_vlan=101 ⇒ mod_vlan_vid:1 dl_vlan=102 ⇒ mod_vlan_vid:2 phy-br-eth1 br-eth1 VLAN ID is converted with flow table dl_vlan=1 ⇒ mod_vlan_vid:101 eth1 dl_vlan=2 ⇒ mod_vlan_vid:102 To Private Network Open Cloud Campus21
  • Quantum Under the Hood Network Namespace Separation To Public Network Network eth1 Namespace br-ex phy-br-ex qg-ZZZ qg-CCC qrouter-bbbb qrouter-ccccSuffix “bbbb” corresponds IP IP to router UUID. dnsmasq dnsmasq qdhcp-aaaa qdhcp-dddd IP IP IP IPSuffix “aaaa” corresponds tapXXX qr-YYY qr-BBB tapAAA to network UUID. Port VLAN tag:1 Port VLAN tag:2 br-int int-br-ex int-br-eth1 phy-br-eth1 br-eth1 eth1 To Private Network Open Cloud Campus22
  • Quantum Under the Hood Notes on the Configuration  On Compute Nodes, the use of Linux Bridge between the integration switch (br-int) and VM tap devices may look redundant. But Its required for Novas security group feature to work. They are configured with Novas “LibvirtHybridOVSBridgeDriver”.  You can choose another driver if you dont need the security group functionality. Then, the configuration will be different.  The security group feature will be integrated with Quantum in the future, and this would be more simplified. Open Cloud Campus23
  • Quantum Under the Hood Example of Configuration Steps Open Cloud Campus24
  • Quantum Under the Hood Example Configuration Steps for Case1 (1/2)  Under the “service” tenant, create the shared router, define the public network, and set it as a default gateway of the router. # tenant=$(keystone tenant-list | awk /service/ {print $2}) # quantum router-create router01 # quantum net-create --tenant-id $tenant public01 --provider:network_type flat --provider:physical_network physnet1 --router:external=True # quantum subnet-create --tenant-id $tenant --name public01_subnet01 --gateway 10.64.201.254 public01 10.64.201.0/24 --enable_dhcp False # quantum router-gateway-set router01 public01  Under the user tenant “redhat”, create the private network “net01” and its subnet, and connect it to the router. # tenant=$(keystone tenant-list|awk /redhat/ {print $2}) # quantum net-create --tenant-id $tenant net01 --provider:network_type vlan --provider:physical_network physnet2 --provider:segmentation_id 101 # quantum subnet-create --tenant-id $tenant --name net01_subnet01 net01 192.168.101.0/24 # quantum router-interface-add router01 net01_subnet01 Open Cloud Campus25
  • Quantum Under the Hood Example Configuration Steps for Case1 (2/2)  Another network “net02” can be added in the same way. # tenant=$(keystone tenant-list|awk /redhat/ {print $2}) # quantum net-create --tenant-id $tenant net02 --provider:network_type vlan --provider:physical_network physnet2 --provider:segmentation_id 102 # quantum subnet-create --tenant-id $tenant --name net02_subnet01 net02 192.168.102.0/24 # quantum router-interface-add router01 net02_subnet01 Open Cloud Campus26
  • Quantum Under the Hood Example Configuration Steps for Case2 (1/2)  Under the “service” tenant, define the public network. # tenant=$(keystone tenant-list | awk /service/ {print $2}) # quantum net-create --tenant-id $tenant public01 --provider:network_type flat --provider:physical_network physnet1 --router:external=True # quantum subnet-create --tenant-id $tenant --name public01_subnet01 --gateway 10.64.201.254 public01 10.64.201.0/24 --enable_dhcp False  Under the user tenant “redhat”, create the tenant router and set its gateway for the public network. # tenant=$(keystone tenant-list|awk /redhat/ {print $2}) # quantum router-create --tenant-id $tenant router01 # quantum router-gateway-set router01 public01  Then, define private network “net01” and its subnet, and connect it to the router. # quantum net-create --tenant-id $tenant net01 --provider:network_type vlan --provider:physical_network physnet2 --provider:segmentation_id 101 # quantum subnet-create --tenant-id $tenant --name net01_subnet01 net01 192.168.101.0/24 # quantum router-interface-add router01 net01_subnet01 Open Cloud Campus27
  • Quantum Under the Hood Example Configuration Steps for Case2 (2/2)  Additonal router and private networks for another tenant “redhat2” can be added in the same way. # tenant=$(keystone tenant-list|awk /redhat2/ {print $2}) # quantum router-create --tenant-id $tenant router02 # quantum router-gateway-set router02 public01 # quantum net-create --tenant-id $tenant net02 --provider:network_type vlan --provider:physical_network physnet2 --provider:segmentation_id 102 # quantum subnet-create --tenant-id $tenant --name net02_subnet01 net01 192.168.101.0/24 # quantum router-interface-add router02 net02_subnet01 Open Cloud Campus28
  • Quantum Under the Hood iptable chains Open Cloud Campus29
  • Quantum Under the Hood Packet filtering chains on Compute Nodes filter table Packet filtering for instances are done on Compute Nodes FORWARD nova-filter-top Filtering chain is created for nova-compute-local each instance. nova-compute-inst-xx Packets from the same Subnet are not filtered. nova-compute-provider Security Group is Packets from the same subnet ACCEPT applied here Filtering by Security Group ACCEPT nova-compute-sg-fallback DROP nova-compute-FORWARD *These are configured by Nova Compute ACCEPT (“Security Group” is a part of Nova functions.) Open Cloud Campus30
  • Quantum Under the Hood NAT chains on Network Node (1/2) Packet filtering for instances nat table POSTROUTING are done on Compute Nodes quantum-l3-agent-POSTROUTING Packets _not_ on Public NW port ACCEPT quantum-postrouting-bottom NAT is applied only at the edge of Public Network quantum-l3-agent-snat SNAT from Private IP quantum-l3-agent-float-snat to the associated Floating IP Packets from Private IP SNAT associated with Floating IP Packets from Private IP SNAT nova-api-POSTROUTING SNAT from Private IP to Routers Public IP nova-api-postrouting-bottom *These are mainly configured by L3 Agent. ACCEPT Open Cloud Campus31
  • Quantum Under the Hood NAT chains on Network Node (2/2) nat table PREROUTING quantum-l3-agent-PREROUTING Packet to Floating IP DNAT nova-api-PREROUTING DNAT from Floating IP to the corresponding Private IP ACCEPT *These are mainly configured by L3 Agent. Open Cloud Campus32
  • Quantum Under the Hood References Open Cloud Campus33
  • Quantum Under the Hood References  OpenStack Network (Quantum) Administration Guide – http://docs.openstack.org/trunk/openstack-network/admin/content/index.html  Quantum L2 Linux Bridge Plugin – http://wiki.openstack.org/Quantum-Linux-Bridge-Plugin  QuickStart with RHOS(Red Hat OpenStack) Folsom Preview – http://d.hatena.ne.jp/enakai00/20121118/1353226066 Open Cloud Campus34
  • Quantum Under the Hood Open Cloud Campus Enjoy the Life with OpenStack! Etsuji Nakai Twitter @enakai00