Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Giuliano Santandrea – CIRI ICT University of Bologna
● OpenStack description 
● Openstack components and allocation 
● Neutron abstractions 
● The virtual network infrastructu...
 vif: virtual network interface 
 VR: virtual router 
 VNI: virtual network infrastructure 
 OVS: Open vSwitch© virtua...
● OpenStack is a cloud platform management software 
● A cloud platform is a cluster of machines that host 
some servers (...
OS is composed of the following components: 
● Web dashboard (Horizon) 
● Compute (Nova): it manages instances lifecycle 
...
External net 
Controller Network node CPU node 1 CPU node 2 CPU node 3 
Management net 
Data net (flat) 
Internet
● Data net 
● Mgmt net 
● External/API net 
These networks are implemented as physical 
separated networks. 
Cesena cluste...
External net 
Controller Network node CPU node 1 CPU node 2 CPU node 3 
Management net 
Data net (flat) 
Internet 
Allows ...
External net 
Controller Network node CPU node 1 CPU node 2 CPU node 3 
Management net 
Data net (flat) 
Internet 
Used fo...
External net 
Controller Network node CPU node 1 CPU node 2 CPU node 3 
Management net 
Data net (flat) 
Internet 
It allo...
External net 
●Keystone 
●Nova 
Controller Network node CPU node 1 CPU node 2 CPU node 3 
Management net 
Data net (flat) ...
External net 
Controller Network node CPU node 1 CPU node 2 CPU node 3 
Management net 
Data net (flat) 
Internet 
Neutron...
External net 
Controller Network node CPU node 1 CPU node 2 CPU node 3 
Management net 
Data net (flat) 
Internet 
In each...
● User send REST API calls to the service enpoints, 
using: 
– the web dashboard 
– CLI clients 
● OS components communica...
 Sometimes the external network is not 
directly connected the internet, but there is a 
datacenter gateway allowing the ...
Public net 
gateway 
External net 
Controller CPU node 1 CPU node 2 CPU node 3 
Mgmt net 
data net 
nova api 
Network node...
Public net 
gateway 
External net 
Internet 
REST API 
User 
port forwarding 
Controller Network node CPU node 1 CPU node ...
Public net 
gateway 
External net 
Controller Network node CPU node 1 CPU node 2 CPU node 3 
rete dati (flat) 
REST API 
U...
Public net 
gateway 
External net 
Internet 
REST API 
Controller Network node CPU node 1 CPU node 2 CPU node 3 
nova 
com...
Public net 
gateway 
External net 
Internet 
REST API 
Controller Network node CPU node 1 CPU node 2 CPU node 3 
nova 
com...
Public net 
gateway 
External net 
Internet 
REST API 
Controller Network node CPU node 1 CPU node 2 CPU node 3 
Mgmt net ...
Public net 
gateway 
External net 
Internet 
REST API 
Controller Network node CPU node 1 CPU node 2 CPU node 3 
Mgmt net ...
Public net 
gateway 
External net 
Internet 
REST API 
NAT/ 
port forwarding 
Controller Network node CPU node 1 CPU node ...
Public net 
gateway 
External net 
Internet 
REST API 
Controller Network node CPU node 1 CPU node 2 CPU node 3 
Mgmt net ...
Public net 
gateway 
External net 
Internet 
REST API 
Controller Network node CPU node 1 CPU node 2 CPU node 3 
data net ...
Neutron defines these network abstractions: 
 Network – an isolated L2 network segment 
 Subnet – an IP address block on...
 Tenant network: a network created by a cloud 
user. The VM takes a fixed IP from this net (not 
modifiable after the VM ...
VIRTUAL NETWORK 
Managed by Neutron 
Physical 
interface of the 
network node
 They are “leaky abstractions”! For example: 
◦ Net creation is limited to the effective VLAN ID 
availability for the ph...
 server (REST API endpoint): receives API requests, saves all 
the network info in a database, instructs the agents 
 pl...
● Create a L2 network 
– neutron net-create net1 
● Associate a L3 subnet to the network 
– neutron subnet-create net1 10....
A user that wants to create a VM: 
1. Sends a REST API (via CLI client or web dashboartd) to the Keystone REST 
endpoint (...
source: http://goo.gl/n3Bb5s
 Network namespaces is a technology that allows 
to separate/isolate multiple network domains inside a single 
host by re...
Network 
namespace 1 
process 
process 
Linux kernel 
global namespace 
(PID 1) 
Hardware 
Network 
namespace N 
Network 
...
VIF VIF 
Network 
namespace 1 
Linux kernel 
global namespace 
(PID 1) 
Hardware 
Network 
namespace N 
Network 
namespace...
 The virtual bridges can be connected to (physical or virtual) interfaces that 
reside in different namespaces: the virtu...
Physical host 
process process 
process process 
They are 
completely 
isolated!
 In each node there is: 
◦ An integration bridge 
◦ A bridge for each physical network, connected to 
 The integration b...
public net 
gateway 
External net 
Mgmt net 
Controller Network node CPU node 1 
Data net 
br-int 
br-data 
VM 
linux 
bri...
Internet 
gateway 
External net 
Controller Network node CPU node 1 
br-int 
br-data 
VM 
linux 
bridge 
br-int 
br-data 
...
Internet 
gateway 
Controller Network node CPU node 1 
br-int 
br-data 
VM 
linux 
bridge 
br-int 
br-data 
br-ex 
br-int ...
 Advantages: namespaces allow to manage multiple L3 
functions at the host level in the same node. 
 This is a key enabl...
 In the compute node: 
◦ nova-compute makes a REST API call to neutron-server 
asking for a port allocation (and a fixed ...
 In the network node: 
◦ neutron-dhcp 
 creates a netns ("qdhcp-…") 
 creates a vif inside that netns 
 spawns a dnsma...
 In the configuration files Neutron associates 
each “physical network” to a virtual bridge. 
For example: 
◦ Physnet1: b...
 Use case: a user 
◦ creates a private user network 
( 10.0.0.0/24) 
◦ boots a VM on this network 
source ~/devstack/open...
 The admin can even specify the VLAN Id used on the physical data 
network (even outside the VLAN ID pool specified in th...
PIF on the data 
network 
Veth pair: a pair of 
vif that act as a 
pipe (everything 
entering from one 
exits from the 
ot...
VM 
eth0 
LB managment interface 
Tap interface (host-level 
view of the VM 
interface) 
Veth pair che 
collega br-int al ...
specific 
routing 
tables 
dhcp server 
dhcp server 
Network 
namespaces 
No traffic here
VM 
eth0 eth0
Provider network (external) 
NB it’s the physical external network of 
the cluster 
User network (internal) 
The VMs will ...
Openstack Networking Internals - first part
Openstack Networking Internals - first part
Openstack Networking Internals - first part
Openstack Networking Internals - first part
Openstack Networking Internals - first part
Openstack Networking Internals - first part
Openstack Networking Internals - first part
Openstack Networking Internals - first part
Openstack Networking Internals - first part
Openstack Networking Internals - first part
Upcoming SlideShare
Loading in …5
×

Openstack Networking Internals - first part

Openstack Networking Internals - first part
Description of the Virtual Network Infrastructure inside an OpenStack cluster
The pictures of the VNI were taken with the "Show my network state" tool
https://sites.google.com/site/showmynetworkstate/

Openstack Networking Internals - first part

  1. 1. Giuliano Santandrea – CIRI ICT University of Bologna
  2. 2. ● OpenStack description ● Openstack components and allocation ● Neutron abstractions ● The virtual network infrastructure
  3. 3.  vif: virtual network interface  VR: virtual router  VNI: virtual network infrastructure  OVS: Open vSwitch© virtual bridge  LB: Linux bridge
  4. 4. ● OpenStack is a cloud platform management software ● A cloud platform is a cluster of machines that host some servers (instances): the servers are offered to the users as as a “service”. The user is able to create a “virtual infrastructure” composed of servers and network appliances (firewalls, routers, ...) ● The servers can be implemented as: – VM (KVM,VMWare,..) – light container (LXC-Docker,..) – bare metal (PXE boot,…)
  5. 5. OS is composed of the following components: ● Web dashboard (Horizon) ● Compute (Nova): it manages instances lifecycle ● Keystone: credentials, service catalog of all the OS services (list of REST service endpoints) ● Glance: image management. An image is a blob file containing a file system with a “pre-cooked” VM: it can be used by hypervisors to boot a new instance! ● Networking(Neutron): network management ● Block storage (Cinder): persistent storage (volumes) ● Other services: – Object storage (Swift): distributed storage for non-structured data
  6. 6. External net Controller Network node CPU node 1 CPU node 2 CPU node 3 Management net Data net (flat) Internet
  7. 7. ● Data net ● Mgmt net ● External/API net These networks are implemented as physical separated networks. Cesena cluster: I configured a switch with “port-based, access mode” VLANs
  8. 8. External net Controller Network node CPU node 1 CPU node 2 CPU node 3 Management net Data net (flat) Internet Allows the admin to access the cluster nodes and it is used for inter-service communication EVERY NODE IS ON THIS NET
  9. 9. External net Controller Network node CPU node 1 CPU node 2 CPU node 3 Management net Data net (flat) Internet Used for inter-VM communication. Depending on the chosen network virtualization mechanism, the packets will be VLAN tagged packets, or encapsulated packets (VXLAN, GRE)
  10. 10. External net Controller Network node CPU node 1 CPU node 2 CPU node 3 Management net Data net (flat) Internet It allows the VMs to access the Internet, the user to access the VMs
  11. 11. External net ●Keystone ●Nova Controller Network node CPU node 1 CPU node 2 CPU node 3 Management net Data net (flat) Internet ●API: REST endpoint, receives user requests ●Scheduler: chooses a compute node ●Glance: ●API ●Registry ●Neutron ●Server: REST API endpoint ●plugin: implements the VNI ●Cinder: ●API,… ●message queue: middleware for inter-service communication
  12. 12. External net Controller Network node CPU node 1 CPU node 2 CPU node 3 Management net Data net (flat) Internet Neutron • plugin: implements the VNI • L3: virtual routers creation • dhcp • metadata
  13. 13. External net Controller Network node CPU node 1 CPU node 2 CPU node 3 Management net Data net (flat) Internet In each Compute node: ●Neutron ●plugin: implements VNI ●Nova ●compute: manages the hypervisor
  14. 14. ● User send REST API calls to the service enpoints, using: – the web dashboard – CLI clients ● OS components communicate between each other using: – Message passing (an AMQP server resides in the controller node) – rest API calls ● Some components (neutron-server, keystone, etc.) access directly to a DB to save/modify their state
  15. 15.  Sometimes the external network is not directly connected the internet, but there is a datacenter gateway allowing the access to the internet Public net gateway Internet REST API External net Private cloud
  16. 16. Public net gateway External net Controller CPU node 1 CPU node 2 CPU node 3 Mgmt net data net nova api Network node REST API User port forwarding Internet REST protocol
  17. 17. Public net gateway External net Internet REST API User port forwarding Controller Network node CPU node 1 CPU node 2 CPU node 3 Horizon (Web server apache2) HTTP request Mgmt net data net
  18. 18. Public net gateway External net Controller Network node CPU node 1 CPU node 2 CPU node 3 rete dati (flat) REST API User port forwarding Internet xvncproxy VNC protocol Port forwarding on 6080 port!! hypervisor The hypervisor pipes the VM video output through the network sudo iptables -t nat -I PREROUTING -i eth1 -p tcp --dport 80 -j DNAT --to-destination 10.250.0.1:80 VM Mgmt net
  19. 19. Public net gateway External net Internet REST API Controller Network node CPU node 1 CPU node 2 CPU node 3 nova compute nova api message queue server Mgmt net data net
  20. 20. Public net gateway External net Internet REST API Controller Network node CPU node 1 CPU node 2 CPU node 3 nova compute Glance API REST endpoint REST API call Mgmt net data net
  21. 21. Public net gateway External net Internet REST API Controller Network node CPU node 1 CPU node 2 CPU node 3 Mgmt net data net VM VM
  22. 22. Public net gateway External net Internet REST API Controller Network node CPU node 1 CPU node 2 CPU node 3 Mgmt net data net Virtual router VM VM
  23. 23. Public net gateway External net Internet REST API NAT/ port forwarding Controller Network node CPU node 1 CPU node 2 CPU node 3 Mgmt net data net Virtual router VM User
  24. 24. Public net gateway External net Internet REST API Controller Network node CPU node 1 CPU node 2 CPU node 3 Mgmt net data net VM Virtual router (NAT/ Floating IP) NAT/ port forwarding User The VM has: • a fixed private IP on a private tenant net • an optional floating IP on the external network The VR does: • The NAT for private IPs • DNAT of the floating IPs
  25. 25. Public net gateway External net Internet REST API Controller Network node CPU node 1 CPU node 2 CPU node 3 data net Mgmt net VM hypervisor storage The VM sees the disk (block device) as a local device! Network file system protocol (NFS, iSCSI, …) Storage node
  26. 26. Neutron defines these network abstractions:  Network – an isolated L2 network segment  Subnet – an IP address block on a certain network  Router – a gateway between subnets  Fixed IP – an IP on a tenant network  Floating IP – a mapping between IP of external networks and a private fixed IP  Port – an attachment point to a network. Users only see these abstractions!!!  Neutron implements these abstraction inside the VNI in the cluster nodes (i.e. at the host level) so that VMs (guest level) can see the virtual networks
  27. 27.  Tenant network: a network created by a cloud user. The VM takes a fixed IP from this net (not modifiable after the VM creation) ◦ implementation detail: the VM receives the IP from a DHCP server configured to give always the same fixed IP to that VM!  Provider network: network external to the cluster, allows outside connectivity, passing through the network node. A VM can allocate a floating IP to gain external visibility (OpenStack maps each floating IP to the related fixed IP). Floating Ips are deallocable.
  28. 28. VIRTUAL NETWORK Managed by Neutron Physical interface of the network node
  29. 29.  They are “leaky abstractions”! For example: ◦ Net creation is limited to the effective VLAN ID availability for the physical network! ◦ During the network creation you could specify low level implementation details ( such as the VLAN ID) or leave OpenStack decide them for you
  30. 30.  server (REST API endpoint): receives API requests, saves all the network info in a database, instructs the agents  plugin agent: implements the VNI inside the cluster node, using the technology specified (GRE tunnel, VLAN, VXLAN, …).  dhcp: implements the dhcp servers  L3: implements the virtual routers  Metadata: the VMs contact the metadata service at the creation
  31. 31. ● Create a L2 network – neutron net-create net1 ● Associate a L3 subnet to the network – neutron subnet-create net1 10.0.0.0/24 --name subnet1 ● Boot a VM on that subnet – nova boot --image img --flavor flv --nic net-id=uuid vm_name
  32. 32. A user that wants to create a VM: 1. Sends a REST API (via CLI client or web dashboartd) to the Keystone REST endpoint (request+auth) and Nova endpoint 2. “nova-scheduler”, internally, chooses the best suitable compute (CPU) node that will host the VM 3. In that CPU node, the “nova-compute” component does the following things: 1. prepares the hypervisor 2. asks Glance the VM image 3. asks Neutron components to allocate the VNI 4. asks Cinder to allocate the persistent block storage (volumes) for the VM
  33. 33. source: http://goo.gl/n3Bb5s
  34. 34.  Network namespaces is a technology that allows to separate/isolate multiple network domains inside a single host by replicating the network software stack  A process executed in a namespace sees only specific: ◦ physical/virtual network interfaces ◦ routing/arp tables ◦ firewall/NAT rules  You can: ◦ Create a netns ◦ Create a process/virtual network component inside that netns
  35. 35. Network namespace 1 process process Linux kernel global namespace (PID 1) Hardware Network namespace N Network namespace 2 ... A global namespace created at boot time. Processes usually reside here You can: • Create a netns • create a vif inside the netns • start a linux process VIF process process
  36. 36. VIF VIF Network namespace 1 Linux kernel global namespace (PID 1) Hardware Network namespace N Network namespace 2 ... Namespaces guarantee L3 isolation, so the interfaces can have overlapping IP addresses!
  37. 37.  The virtual bridges can be connected to (physical or virtual) interfaces that reside in different namespaces: the virtual bridges act as bridges between the namespaces Physical host Network namespace 1 Network namespace 2 VIF packet PIF Virtual bridge process
  38. 38. Physical host process process process process They are completely isolated!
  39. 39.  In each node there is: ◦ An integration bridge ◦ A bridge for each physical network, connected to  The integration bridge  The physical network interface (pif)
  40. 40. public net gateway External net Mgmt net Controller Network node CPU node 1 Data net br-int br-data VM linux bridge br-int br-data br-ex br-int br-data br-data: connected to the data net Internet
  41. 41. Internet gateway External net Controller Network node CPU node 1 br-int br-data VM linux bridge br-int br-data br-ex br-int br-data br-int: intermediate bridge, act as a hub of a star network Mgmt net Data net public net
  42. 42. Internet gateway Controller Network node CPU node 1 br-int br-data VM linux bridge br-int br-data br-ex br-int br-data br-ex: connected to the external network (only present only in the network node!!) External net Mgmt net Data net public net
  43. 43.  Advantages: namespaces allow to manage multiple L3 functions at the host level in the same node.  This is a key enabler is implementing VNIs with advanced functionalities: the cloud users to create overlapping virtual L3 networks!!! ◦ Two tenants can create isolated L3 networks with the same IP addresses  The namespaces are used only in the network node (because L3 functionalities at the host level are present only inside the network node)  A new network namespace is created by OpenStack when you create… ◦ … a new virtual L2 network ◦ … a new virtual router
  44. 44.  In the compute node: ◦ nova-compute makes a REST API call to neutron-server asking for a port allocation (and a fixed IP for the VM) ◦ neutron-agent configures the virtual bridges (br-data, br-int) via OpenFlow protocol ◦ nova-compute boots the VM
  45. 45.  In the network node: ◦ neutron-dhcp  creates a netns ("qdhcp-…")  creates a vif inside that netns  spawns a dnsmasq process (DHCP server) using that vif ◦ neutron-l3  creates a netns for the virtual router ("qrouter-…")  creates a vif inside that netns  configure the routing tables inside that netns
  46. 46.  In the configuration files Neutron associates each “physical network” to a virtual bridge. For example: ◦ Physnet1: br-data ◦ Extphysnet: br-ex  The admin, during the creation of a provider network, must specify the associated physical network
  47. 47.  Use case: a user ◦ creates a private user network ( 10.0.0.0/24) ◦ boots a VM on this network source ~/devstack/openrc demo demo pass && nova boot --key-name hc01 --image cirros-0.3.1-x86_64- disk --flavor 2 --nic net-id=61821a27-69b8- 43c2-afa8-633304d8be50,v4-fixed-ip=10.0.0.66 myserver
  48. 48.  The admin can even specify the VLAN Id used on the physical data network (even outside the VLAN ID pool specified in the configuration file)  The user cloud is not allowed to do this (OpenStack autonomously picks a VLAN from the VLAN pool available in the data network) source ~/devstack/openrc admin admin pass && neutron net-create net2 --provider:network_type vlan --provider:physical_network physnet1 --provider:segmentation_id 1000 +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | True | | id | 61821a27-69b8-43c2-afa8-633304d8be50 | | name | private | | provider:network_type | vlan | | provider:physical_network | physnet1 | | provider:segmentation_id | 1000 | | router:external | False | | shared | False | | status | ACTIVE | | subnets | bbc97757-f297-4c7b-b032-e70768fe8485 | | tenant_id | a370af83e43a432abb3adfbf976d1cf8 | +---------------------------+--------------------------------------+ NB this VLAN is that used on the physical network!
  49. 49. PIF on the data network Veth pair: a pair of vif that act as a pipe (everything entering from one exits from the other)
  50. 50. VM eth0 LB managment interface Tap interface (host-level view of the VM interface) Veth pair che collega br-int al linux bridge interfaces connecting the bridges
  51. 51. specific routing tables dhcp server dhcp server Network namespaces No traffic here
  52. 52. VM eth0 eth0
  53. 53. Provider network (external) NB it’s the physical external network of the cluster User network (internal) The VMs will receive a fixed IP on this net.

×