NFV infrastructure will be distributed across multiple geo-locations, to place workload close to the end user for better user experience, and introduce some cross -site requirements like service chaining, high availability and disaster recovery. There are gaps in OpenStack in such geo-distributed clouds such as: Support VNF high availability across OpenStack Support Multisite VNF geo site-disaster recovery Single unified view for operators Multi-site quota management Sync tenant resources like images, ssh-keys, security groups Policy driven VNF placements Optimized resource utilization Dynamic Service Functions Chaining across sites Several projects like OPNFV Multisite, Tacker, Kingbird and Tricircle projects have collaborated to address the requirements for NFV. In this session, come learn about: Need for multisite in NFV Gaps in OpenStack Existing projects and solutions Gaps in these solutions and what different teams are doing to fix it

1. Multisite OpenStack for NFV
Bridging The Gap

2. ● Gaps in multi-site OpenStack for NFV
● Tacker
● Kingbird
● Tricircle
● Putting it all together
Agenda

3. Gaps in Multi-site OpenStack for NFV
OpenStack VIM OpenStack VIM
VNF1
vRouter 1
(IPv6)
VNF2
vRouter 2
(IPv6)
OpenStack
VNF2
vRouter 2
(IPv6)
? Disaster recovery of application…
● App level consistency snapshot
● Volume level cross site replication
NFV Orchestrator

4. Gaps in Multi-site OpenStack for NFV
OpenStack VIM OpenStack VIM
VNF1
vRouter 1
(IPv6)
VNF2
vRouter 2
(IPv6)
NFV Orchestrator ? Tenant level image, SSH key,
security group, IP/mac, metering,
monitoring...
? Tenant level L2/L3 networking for E-W traffic
? Tenant level quota control and resources
view
? Tenant level image, SSH key, security group,
IP/mac, metering, monitoring...

5. OpenStack OpenStack OpenStack
Gaps in Multi-site OpenStack for NFV
Distributed Session DB
Stateless Session Process Unit
Distributed LB
Distributed VNF
? Tenant level L2/L3 networking for E-W traffic
? Tenant level quota control and resources view
? Tenant level image, SSH key, security group,
IP/mac, metering, monitoring...
NFV Orchestrator

6. OPNFV Multisite Use Cases
Use case 1 Multisite identity service
management
● Bug reported to KeyStone (Liberty)
Use case 2 VNF high availability across VIM ● Cross OpenStack L2 networking requirements to
L2GW (Newton)
Use case 3 Multisite VNF Geo-Site disaster
recovery
● Group VMs consistency snapshot via
quiesce/unquiesce API (Newton)
● Volume level replication
Use case 4 Resources management and
synchronization. ● Kingbird
● Tricircle
( for different deployment scenario )
Use case 5 Centralized monitoring service.

7. VNF1 VNF2 VNF2
VNF3
VNF5
Service Function Chaining across sitesSite 1
Site 2
Site 3
Monitor and heal service chain in case of VNF failures
Modify/update service chains
Resilient to WAN bandwidth and delay
NFV Orchestrator
End-to-end Service Orchestration
traffic

8. Tacker Project Overview
● Open NFV Orchestrator
● VNF life cycle management
○ monitoring framework
○ configuration framework
● VNF catalog management
○ TOSCA template support
● EPA support
○ CPU-Pinning, Huge Page, NUMA awareness, SR-IOV
● Auto resource creation
○ Flavor, Network and Image creation

9. Multisite VIM Support
● Unified view of VIM management using a single dashboard
● Ease of deployment for operators to integrate Tacker with their existing
OpenStack installations with minimal overhead
● Fulfillment of a certain level of RO functionality, by combining necessary
existing OpenStack modules such as heat and keystone

10. Unified View of VIM Management
Tacker
VIM 0
(local OpenStack
Controller)
Liberty
Tacker can instantiate VNFs only
on same OpenStack controller
Mitaka
Tacker
Site 1 - VIM
(default – local
OpenStack
Controller)
Site 2 -VIM
(OpenStack
Controller)
Site 3 - VIM
(OpenStack
Controller)
VNF placement on specific target OpenStack VIM
Explicit Region support

11. Multiple OpenStack Versions
Tacker
Site 1 –
(OpenStack
controller)
Site 2 -
(OpenStack
Controller)
Site 3 -
(OpenStack
Controller)
• Operators have multiple OpenStack VIMs with different OpenStack versions
• Detect KeyStone and HOT template version and translate accordingly
• Gracefully downgrade resource requests on older VIMs
Kilo Liberty Mitaka

12. Tacker Multisite Architecture
● Multisite VIM module part of
orchestrator component
● Pluggable driver framework
○ (aka Bring your own VIM)
● VIMs shared across tenants
● Horizon dashboard and CLI support
Tacker API
NFV Catalog
NFVO
Multsite VIM
VNFM
Custom Driver
OpenStack VIM driver

13. Tacker Resources
● https://wiki.openstack.org/wiki/Tacker
● https://github.com/openstack/tacker
● https://github.com/openstack/tacker-specs/blob/master/specs/mitaka/multi-
site-feature.rst

14. Kingbird Project
Resource synchronization and management for
multi-region OpenStack deployments.
Based on OPNFV Multisite UC5, UC6.
Aggregated view of distributed resources.
Synchronization (ssh-keys, security groups,
images, flavors, etc.)
Centralized quota management.

15. Quotas in OpenStack
● Defined on a per-region basis.
● Fragmented quota management in nova, neutron,
cinder.
● Assigned per region.
○ E.g. max 50 instances in R1, max 30 instances in R2
● No process for synchronizing allocated quotas
across regions.
R1
Neutron
QM
Nova
QM
R2
Neutron
QM
Nova
QM
Cinder
QM
Cinder
QM

16. Kingbird Quota Management
● New centralized quota management function.
● Global quota limits across multiple regions.
● Minimal impact on the existing OpenStack services.
R1
Neutron Nova Cinder
QMQM QM
R2
Neutron
QM
Nova
QM
Cinder
QM
Kingbird
Keystone

17. Kingbird Quota Management
● Use existing APIs to dynamically balance quota values.
● Calculate resource usage upon synchronization.
● Store the default/tenant quota limits in Kingbird.
○ Provide CRUD operations for the known quota limits
○ Expose similar APIs (os-quota-set, os-quota-class-sets)

18. Kingbird Architecture
kingbird-api
Provides an API for managing global quota
limits and on-demand quota synchronization.
kingbird-engine
Responsible for communicating with OpenStack
services in regions, fetching tenant resource
usages, periodically rebalancing quota limits.
kingbird-api
rabbit-mq
kingbird-engine database
python-kingbirdclient
Horizon
Mistral
Tacker
OSS/BSS
...

19. Kingbird Resources
● Project status, bugs, and blueprints are tracked on Launchpad
https://launchpad.net/kingbird
● Source code on Github
https://github.com/openstack/kingbird/

20. Tricircle
Site 1 -VIM
(OpenStack
Controller)
Site 2 -VIM
(OpenStack
Controller)
Site 3 - VIM
(OpenStack
Controller)
OpenStack API Gateway:
● Reserve all API ecosystem like
Tacker, Heat, Magnum, Murano,
SDK, CLI…
● Cross OpenStack L2/L3 networking
automation and data movement
OpenStack API
OpenStack APIs OpenStack APIsOpenStack APIs
Tricircle
Tricircle is OpenStack API gateway with added value like cross OpenStack L2/L3 networking,
volume/VM movement, image distribution, global resource view, distributed quota management …
This makes massive distributed edge clouds work like one inter-connected cloud, one OpenStack

21. Tricircle
Site 2 -VIM
(OpenStack
Controller)
Site 2 -VIM
(OpenStack
Controller)
Site 3 - VIM
(OpenStack
Controller)
VM1 vRouter 1 VM2vRouter 2 VM3
OpenStack API Gateway:
● Tenant level L2/L3 networking
automation for E-W traffic
isolation
OpenStack API
Tricircle
L2GW1 L2GW1

22. Tricircle
22
Neutron API
Tricircle Plugin
L2GW Driver
bottom OpenStack bottom OpenStack
Network1-1
Nova
API-GW
VLAN1
VM1 VM2
Network1-2
VxLAN2
XJob
11. Start async job for L2 Networking
for (Network1-1, Network1-2)
12. Create L2GW local connection
13. Create L2GW remote connection
14. Populate remote mac/IP info
12. Create L2GW local connection
13. Create L2GW remote connection
14. Populate remote mac/IP info
L2 Networking
(EVPN)
Tricircle-extend the network to where the VM is, automatically
Cinder
API-GW
L2GW1 L2GW2

23. Tricircle
Tricircle
Site 2 -VIM
(OpenStack
Controller)
Site 2 -VIM
(OpenStack
Controller)
Site 3 - VIM
(OpenStack
Controller)
VM1
(Trans
Tool)
VM2
(Trans
Tool)
OpenStack API Gateway:
● Move tenant’s data (VM,Volume,
Image,etc) across site leverage the
cross site tenant L2/L3 networking
OpenStack APIs
volume volume
Create VM with transportation tool, and attach the volume( data to be moved) to the VM, move the data
across OpenStack through tenant level L2/L3 networking.
*Conveyor, a project built above Tricircle will help to do this:https://launchpad.net/conveyor

24. Tricircle Resources
● Project status, bugs, and blueprints are tracked on Launchpad
https://launchpad.net/tricircle
● Source code on Github
https://github.com/openstack/tricircle

25. Putting it all together
Kingbird
Tacker
Site 1 - VIM
(OpenStack
Controller)
Site 2 -VIM
(OpenStack
Controller)
Tricircle
Site 4 - VIM
(OpenStack
Controller)
Site 5 -VIM
(OpenStack
Controller)
Site 6 - VIM
(OpenStack
Controller)
Multi-site NFV orchestration
Tenant level cross site L2/L3 networking
automation, service chaining, data movement

26. Thanks, Q&A

27. Tacker Multisite Horizon Workflow

28. Tacker Multisite Horizon Workflow

29. Tacker Multisite Horizon Workflow

30. Tacker Multisite Horizon Workflow

31. Tacker Multisite Horizon Workflow

32. Need and Gap for Multi-site OpenStack in NFV (3)
A new work item ETSI NFV GS IFA-020 was approved on Apr 7th to provide a report on the NFVO Split,
where a first time ever try for a holistic view on multiple site NFVO architecture will be discussed and
introduced
❏ The NFVO split architecture options should be
designed to be complementary and not contradict
each other
❏ Reference points should be reused as much as
possible
❏ A potential option is shown on the left
❏ In this context, OpenStack would be expected to
provide certain level of RO functionality
❏ Could be a combination of modules
❏ Could be a not full-fledged RO entity

33. Gaps in these solutions and what different teams are doing to fix it
Kingbird
Tacker
Site 1 - VIM
(OpenStack
Controller)
Site 2 -VIM
(OpenStack
Controller)
Tricircle
Site 4 - VIM
(OpenStack
Controller)
Site 5 -VIM
(OpenStack
Controller)
Site 6 - VIM
(OpenStack
Controller)
Landscape for projects working together

34. Gaps in these solutions and what different teams are doing to fix it
Kingbird
Tacker
Site 1 - VIM
(OpenStack
Controller)
Site 2 -VIM
(OpenStack
Controller)
Tricircle
Site 4 - VIM
(OpenStack
Controller)
Site 5 -VIM
(OpenStack
Controller)
Site 6 - VIM
(OpenStack
Controller)
If tenant level cross site L2/L3 networking
automation, service chaining, data
movement are needed

35. Gaps in these solutions and what different teams are doing to fix it
Kingbird
Tacker
Site 1 - VIM
(OpenStack
Controller)
Site 2 -VIM
(OpenStack
Controller)
Tricircle
Site 4 - VIM
(OpenStack
Controller)
Site 5 -VIM
(OpenStack
Controller)
Site 6 - VIM
(OpenStack
Controller)
If tenant level quota control, resource sync..etc are required for multi-region deployment

36. Gaps in these solutions and what different teams are doing to fix it
Kingbird
Site 1 - VIM
(OpenStack
Controller)
Site 2 -VIM
(OpenStack
Controller)
Tricircle
Site 4 - VIM
(OpenStack
Controller)
Site 5 -VIM
(OpenStack
Controller)
Site 6 - VIM
(OpenStack
Controller)
If tenant level quota control, resource sync..etc are required for multi-region deployment
Region1 Tacker Region2 Tacker Region3 Tacker

37. Gaps in these solutions and what different teams are doing to fix it
Tacker
Site 1 - VIM
(OpenStack
Controller)
Site 2 -VIM
(OpenStack
Controller)
Tricircle
Site 4 - VIM
(OpenStack
Controller)
Site 5 -VIM
(OpenStack
Controller)
Site 6 - VIM
(OpenStack
Controller)
If tenant level cross site L2/L3 networking automation, service chaining, data
movement in all sites are needed

38. Existing projects and solutions
● Multisite
● Tacker
● Kingbird
● Tricircle

39. Tricircle
Tricircle
Site 1 -VIM
(OpenStack
Controller)
Site 2 -VIM
(OpenStack
Controller)
Site 3 - VIM
(OpenStack
Controller)
OpenStack API Gateway:
● Tenant level quota control and resources
view
● Tenant level image, SSH key, security
group, IP/mac, metering, monitoring...
OpenStack API
OpenStack API OpenStack APIOpenStack API

40. Tricircle
40
Neutron API
Tricircle Plugin
L2GW Driver
bottom OpenStack bottom OpenStack
L2 NetworkingVM1 VM2
Network1-2Network1-1
1,2
Nova
API-GW
3,4
5 5
VLAN1 VxLAN2
L2GW1 L2GW2
acb
Tricircle

41. Quota sync algorithm
S0: Initial state.
>+-----+----+----+----+----+
>| | A | B | C | Kb |
>+-----+----+----+----+----+
>| S0 | | | | 10 |
>+-----+----+----+----+----+
S1: Kingbird syncs the limit in all regions:
>+-----+----+----+----+----+
>| | A | B | C | Kb |
>+-----+----+----+----+----+
>| S0 | | | | 10 |
>+-----+----+----+----+----+
>| S1 | 10 | 10 | 10 | 10 |
>+-----+----+----+----+----+
Three regions A, B, C and the global quota limit is vcpus=10 set in Kingbird

42. Quota sync algorithm
S2: Allocate 2 vcpus in A and 1 in C.
>+-----+----+----+----+----+----+----+----+----+
>| | A | B | C | Kb | Au | Bu | Cu | Su |
>+-----+----+----+----+----+----+----+----+----+
>| S0 | | | | 10 | 0 | 0 | 0 | 0 |
>+-----+----+----+----+----+----+----+----+----+
>| S1 | 10 | 10 | 10 | 10 | 0 | 0 | 0 | 0 |
>+-----+----+----+----+----+----+----+----+----+
>| S2 | 9 | 7 | 8 | 10 | 2 | 0 | 1 | 3 |
>+-----+----+----+----+----+----+----+----+----+
S3: Tenant allocates 2 more in A, 2 in B, and
2 more in C: so now the usage will be 4 in A,
2 in B, 3 in C (total 9).
>+-----+----+----+----+----+----+----+----+----+
>| | A | B | C | Kb | Au | Bu | Cu | Su |
>+-----+----+----+----+----+----+----+----+----+
>| S0 | | | | 10 | 0 | 0 | 0 | 0 |
>+-----+----+----+----+----+----+----+----+----+
>| S1 | 10 | 10 | 10 | 10 | 0 | 0 | 0 | 0 |
>+-----+----+----+----+----+----+----+----+----+
>| S2 | 9 | 7 | 8 | 10 | 2 | 0 | 1 | 3 |
>+-----+----+----+----+----+----+----+----+----+
>| S3 | 5 | 3 | 4 | 10 | 4 | 2 | 3 | 9 |
>+-----+----+----+----+----+----+----+----+----+
Global_remaining_limit = Kingbird_global_limit - Su(sum of all usages)
Region_new_limit = Global_remaining_limit + resource_usage_in_that_region.

43. Tricircle
43
Neutron API
Tricircle Plugin
L2GW Driver
bottom OpenStack bottom OpenStack
Network1-1
3 Create Network1-1
Nova
API-GW
1 Create Network1
2 Create VM1(Network1, AZ1)
VLAN1
L2GW1 L2GW2
5. Create Port1 for VM1
VM1
6 Create VM1(Port1, Network1-1)
4. update Network1( segment1 = Network1-1@ AZ1)
*support
from
Networking
L2GW
project
Tricircle

44. Tricircle
44
Neutron API
Tricircle Plugin
L2GW Driver
bottom OpenStack bottom OpenStack
Network1-1
8 Create Network1-2
Nova
API-GW
7 Create VM2(Network1, AZ2)
VLAN1
L2GW1
10. Create Port2 for VM2
VM1
11 Create VM2(Port2, Network1-2)
9. update Network1( segment2 = Network1-2 @ AZ2)
VM2
Network1-2
VxLAN2
L2GW2
Tricircle