Ref: https://www.youtube.com/watch?v=KX0LvsIXsU4

1. Netforce: Extending neutron to support
routed networks at scale in eBay.
Aliasgar Ginwala
Andrew Qu

2. Subnet on-boarding without Netforce
• Reliability engineer monitors the dashboard for IPs available for a vpc in
an AZ.
• If they find IP counts dropping, they will file a network ticket to get new
block of subnet.
• Network ticket takes few days of time to resolve since releasing a new
subnet would require certain approvals and validations before
configuring on the device.
• This makes it tough to follow up and avoid missing the daily checks of
available counts of IPs.
• Hence, on-demand subnet creation was much needed.

3. VRF BM on-boarding without Netforce
• Onboarding team has to provision the BM using ironic.
• File a network ticket to flip the port to particular vlan in mode access.
• Network ticket takes few days of time to resolve since flipping these
assets to access would require certain approvals and validations before
configuring on the device.
• This impacted VRF on-demand provisioning via ironic.
• Hence we leveraged routed network in neutron.

4. Netforce Use cases:
1. On-demand subnet provisioning for OpenStack VMs/BMs.
2. VRF BM on-boarding for a vpc.
3. On-demand subnet provisioning for Kubernetes pods.
4. Subnet recycling.
5. Support network configuration changes using cli.

5. Netforce Framework, deployment and driver support
• Uses Neutron’s framework.
• Uses keystone for authentication.
• Uses NAPALM (Network Automation and Programmability Abstraction Layer with Multivendor
support) library that implements a set of functions to interact with different vendor devices using a
unified API by Spotify [1].
• Current driver support for Cisco (Nxos), Juniper (Junos) and arista (Eos).
• Deployed in production, running on Kubernetes as a global service.
• Netforce cli to interface with devices.
• 1: https://github.com/napalm-automation/napalm

6. BGP enabled VRF
Bubble 1 Bubble2

7. Netforce data-model

8. Netforce data-model definitions
• Pod: A logical grouping of devices that are connected to the same core switches.
• Bubble: A logical grouping of devices that are connected to the same distribution switches.
• BridgeGroup: A group of network devices that share a common L2 environment.
• VLAN: VLAN (Virtual Local Area Network) is any broadcast domain that is partitioned and isolated in a computer network at L2
layer. VLAN's with the same number on the same bridegroup are connected to each other.
• Subnet: A logical sub-division of an IP network.
• Devices: Any kind of network device, which includes and not limited to, routers, switches, etc. There are various types of
devices depending on their location in the network topology. The types of devices are:
– ToRS (Top of the Rack switch)
– LB switch
– Distribution
– Core
– Border
– Backbone
– Fabric
– Aggregation
• Port: These are physical ports that are available on the devices.
• VRF: VRF, stands for Virtual routing and forwarding, is a technology that allows you to create multiple instances of the routing
table to an existing router of switch and provides isolation at L3 layer. The routing table includes subnet information, routing
protocol, route distinguisher and other parameters that governs how data packets are routed. In this context, we will have one VRF
config for each VPC.

9. Neutron Routed networks
• It is implemented before the same feature was available in upstream [1], so that we could deploy for VRF bubbles for
customer needs.
• With routed network, user sees only one network per VPC in a given AZ, without worrying about per sub-rack network
selection. .
• Main data entities:
– Similar to upstream with some changes in data models since ebay specific routed network was live before upstream.
– We have physicalnetworks (No TOR object modelling in upstream ) as an object which is attribute in the upstream
– We have switch port mode (access/trunk)
– Upstream have host-segment binding and no host-physical network binding
– Access to access port vlan flipping is not allowed.
• Physicalnetworks (tenant_id, uuid, name)
• segments: (id, network_id, network_type, physical_network, segmentation_id)
• Hostphysicalnetworkbindings: (uuid, host_id, physical_net, externel_id, vlan_mode, mac_address)
» E.g. (fake-uuid, fake_host, fake_uuid, asset_fake, trunk, fa:ke:ma:cf:ak:ee)
• Hostsegmentbindings: (uuid, host_binding_id, segement_id, is_native_vlan)
• 1: https://review.openstack.org/#/c/225384/22/specs/newton/routed-networks.rst

10. On-demand Subnet on-boarding
Neutron
Network Engineer
Cloud
IPAM
Create subnet 10.x.x.x/x Create subnet 10.x.x.x/xCreate subnet 10.x.x.x/x
Createsubnet10.x.x.x/x
Getsubnet
Use10.x.x.x/x
Success
Success
Success
Success
Get subnet
Use 10.x.x.x/x
switch
Global
IPAM
Get subnet
Use 10.x.x.x/x
Note:
Cloud IPAM is a subset of Global IPAM
Kubernetes

11. Adding a subnet to a VLAN
Is subnet in
production
status in
IPAM?
Do not proceed
Yes
Update / allocate
subnet
No
Is the
subnet part
of the block
allocated to
this
bubble?
No
Does
subnet
overlap with
existing
subnet?
Yes
Yes
Does trace
route stop at
the bubble
where we are
adding it?
No
No
Do any DNS
records exist
for this
subnet?
Yes
Yes
Authorization
(see details
in notes)
NoNo
Interface/IRB
is in correct
VRF
No
Vlan/
interface is
within limit of
subnets that
can be
added
Yes
Yes
No
Add subnet
Yes
Route for subnet
is present on
upstream switch
and points to
correct switch
Rollback change No
Trace route
ends at the
correct switch
Yes
No
Reserve first 10 IP’s in
subnet in DNS, see
notes for details
Yes
Create subnet + validations
curl -s -X POST http://$HOST:9696/v2.0/subnets.json
-H "User-Agent: python-neutronclient" -H "Content-Type: application/json"
-H "Accept: application/json" -H "X-Auth-Token: $TOKEN"
-d '{"subnet": {
"name": "prod-routed-subnet-10-x-x-0_23 ",
"cidr": "10.x.x.0/23",
"gateway_ip": "10.x.x.1",
"broadcast_ip": "10.x.x.255 ",
"netmask": "255.255.254.0",
"vlan_id": "fake-uuid",
"reserve_ip_count": 10
}}'
Added secondary subnet for Juniper
show configuration interfaces irb.0
description fake-interface-1;
family inet {
address 10.x.x.1/22;
address 10.x.x.1/23;
}
family inet6 {
address 2655:0:555:5555:0:0:0:1/64;
}
neutron subnet-list --network-id=fa6984a3-0fe0-4ff7-8317-e7a9a961d465
+--------------------------------------+----------------------------------------+-----------------+-----------------------------------------------------+
| id | name | cidr | allocation_pools |
+--------------------------------------+----------------------------------------+-----------------+-----------------------------------------------------+
| 65d9afd1-b5fc-468c-b09b-4bc93b75eae2 | prod-routed-subnet-10-x-x-0_22 | 10.x.x.0/22 |
{"start": "10.x.x.11", "end": "10.x.x.254"} |
| 75d9afd1-b5fc-468c-b09b-4bc93b75eae2 | prod-routed-subnet-10-x-x-0_23 | 10.x.x.0/23 |
{"start": "10.x.x.11", "end": "10.x.x.254"} |
| +--------------------------------------+----------------------------------------+-----------------+-----------------------------------------------------+

12. Ironic BM onboarding for a VPC in VRF
• Goal is to set Access mode VLAN-ID for ports on TORs for Bare-metals
• When a BM is provisioned for end-user (not as a hypervisor) for a dedicated VPC-A, instead of allowing multiple trunk
VLANs to the BM, Ironic will trigger neutron driver to call netforce API to set a single access VLAN to the BM during BM
provisioning.
• Nova/Ironic:
– Boot a BM on the routed network for VPC-A
– Scheduler pick up the host, and request neutron port creation on the routed network for the VPC and the host
• This triggers the trunk VLAN for VPC-A added for the host on the connected switch port, and subnet/IP allocation,
just the same as the use case for VM boot
– Ironic boot the BM and communicate the metadata (including new IP/subnet config) to the BM via native VLAN
– After network configuration is done on the BM, Ironic call Neutron API again to delete the native host-segment
binding, and update the VPC host-segment binding to "access" mode.
• This triggers the VLAN flipping on the switch, and SPN access is disabled for the BM from now on, and when new
IP configuration is applied to the BM it will be able to access the VPC network.

13. Port vlan-update/labelling for BMs
Ironic
Neutron
Success
get/update port-vlanget/update host-segment
get/update port-vlan
SuccessSuccess
Success
Network Engineer
Portoperations
Switch

14. Flip between Access and Trunk Mode
Port has
link
Make
Change
Stop
Don’t
make
change
No
Does MAC
match device
being
changed?
Yes
What state
is server in
CMS
Production
Not in Production
Traffic on
port >
1kbps
Unknown
Yes
No
YesNo
Update host-segment binding +
validations
curl -s -X PUT http://$HOST:9696/v2.0/host_bindings/$HOST_BINDING
-H "User-Agent: python-neutronclient" -H "Content-Type: application/json"
-H "Accept: application/json" -H "X-Auth-Token: $TMP_TOKEN"
-d '{"host_binding":{
"hostname": "'"$HOST_NAME"'",
"vlan:mode": "access",
"segment_bindings": [
{"segment_id": "'"$SEGMENT_ID_DEV"'", "vlan:is_native": true}
]}}'
| python -m json.tool

15. Netforce-CLI – Update port-vlan/label association
+-----------+-------------+------------------+-----------+----------------+-------------+--------+---------------+--------------------------+
| Port | Device Name | Switch Port Mode | Vlan Tags | Native Vlan ID | Description | status | Ticket# |
+-----------+-------------+------------------+-----------+----------------+-------------+--------+---------------+--------------------------+
| Ethernet1 | arista | trunk | 3,1 | 1 | test1 | ACTIVE| fake_change1234
|
+-----------+-------------+------------------+-----------+----------------+-------------+--------+---------------+ -------------------------+
Post change device interface config
#show running-config interfaces et1
interface Ethernet1
description test1
switchport trunk native vlan 1
switchport trunk allowed vlan 1,3
switchport mode trunk
Current device interface config
#show running-config interfaces et1
interface Ethernet1
description test1
switchport access vlan 1
netforce list-devices
+-------------+--------------+---------+
| Device Name | Device IP | OS Type |
+-------------+--------------+---------+
| arista | 10.x.x.x | eos |
+-------------+--------------+---------+
netforce update-port --vlans 1,3 --port Ethernet1 --device-name arista --native-vlan 1 --mode trunk --check-mac fa:ke:ma:cf:ak:ee
current port status is :
+-----------+-------------+------------------+----------------+-------------+--------+---------------+
| Port | Device Name | Switch Port Mode | Native Vlan ID | Description | status |
+-----------+-------------+------------------+----------------+-------------+--------+---------------+
| Ethernet1 | arista | access | 1 | test1 | ACTIVE |
+-----------+-------------+------------------+----------------+-------------+--------+---------------+

16. Pre-provision subnets dynamically
• A back-ground script is deployed to check the available Ips for the VM/BMs in neutron.
• We have set default_ip_threshold = 64 for the background job.
• If available Ips reach default_ip_threshold or less, back ground script will trigger a new subnet creation workflow to
add more IPs for VMs/BM.

17. Subnet recycling
• When all workloads of a subnet/segment are deleted, the subnet/segment will be recycled and the subnet will be
returned to cloud IPAM pools.
• To avoid flapping, the recycling can be implemented with a delayed deletion mechanism, e.g. periodical cleanups.

18. Netforce
• Network infrastructure automation service to support routed networks at scale.
• Can manage and execute all network-related configurations, which are performed manually by the Networking Team via
ticketing process.
• Helps reduce operational cost and increased efficiency, compared to networking ticket based approach.
• Helps achieve end to end data center automation by supporting device drivers for different vendors.
• Programming network infrastructure through Neutron had a lot of challenges in terms of inter-operability and reliability which
have been addressed.

19. Thank You
Aliasgar Ginwala
aginwala@ebay.com
Andrew Qu
jxiaobin@ebay.com