This document discusses issues with running OpenStack in a multi-region mode and proposes Tricircle as a solution. It notes that in a multi-region OpenStack deployment, each region runs independently with separate instances of services like Nova, Cinder, Neutron, etc. Tricircle aims to integrate multiple OpenStack regions into a unified cloud by acting as a central API gateway and providing global views and replication of resources, tenants, and metering data across regions. It discusses how Tricircle could address issues around networking, quotas, resource utilization monitoring and more in a multi-region OpenStack deployment.

1. HUAWEI TECHNOLOGIES CO., LTD. Page 1Huawei Confidential
Issues of
OpenStack multi-region mode
Chaoyi Huang ( joehuang@huawei.com )
Last edited May 28, 2015
Last update Jan.12, 2016

2. HUAWEI TECHNOLOGIES CO., LTD. Page 2Huawei Confidential
Nova
Cinder
OpenStack
Region1
Glance
Neutron
Ceilometer
KeyStone
Horizon
Nova
Cinder
OpenStack
Region2
Glance
Neutron
Ceilometer
API Req.
API Req.
Token Req.
OpenStack mutli-region mode:
each site will be deployed with independent OpenStack instance (including Nova,Cinder,Neutron, Ceilometer, Glance,
Glance sometimes will be shared for multi-site). KeyStone service is shared for same credential to access multi-region,
and Horizon functions as the unified portal for the multi-region cloud.
Site1 Site2

3. HUAWEI TECHNOLOGIES CO., LTD. Page 3Huawei Confidential
 Neutron:
 No global view for tenant level IP address space management
 Issue of clone security groups across regions
 Issue of establishing tenant level L2/L3 networking across region for application high availability across OpenStack
instances
Issues from production multi-site / multi-region cloud:
Vancouver session moderated by Comcast: https://etherpad.openstack.org/p/YVR-ops-multi-site
OPNFV multisite project: https://etherpad.opnfv.org/p/multisite_usecase_collection
Neutron vApp
Standby
Neutron
Network CIDR… Network CIDR…
Security Group… Security Group…
APP networking across OpenStack instance…
For heartbeat, states replication….vApp
master

4. HUAWEI TECHNOLOGIES CO., LTD. Page 4Huawei Confidential
 Quotas, tenants: Missing global view across regions
 Global quota issue including quotas for Nova, Cinder, Neutron...
 Issue for soft/hard quotas based on current usage for all region for this tenant
 Issue for each of the individual projects managing quotas provide a global view
Issues from production multi-site / multi-region cloud:
Vancouver session moderated by Comcast: https://etherpad.openstack.org/p/YVR-ops-multi-site
OPNFV multisite project: https://etherpad.opnfv.org/p/multisite_usecase_collection
vApp 5
vApp 1
vApp 2
vApp 6
OpenStack
vApp 8
vApp 4vApp 3
vApp 7
OpenStack
Nova Quota Cinder Quota Neutron Quota Nova Quota Cinder Quota Neutron Quota
Issue: discrete quota control

5. HUAWEI TECHNOLOGIES CO., LTD. Page 5Huawei Confidential
 Resource utilization across regions: Missing global view across regions
 Issue for a user being able to management all his virtual resources from one centralized management interface, at least
to have a summarized view of the total resource capacity and the live utilization of his virtual resources, for example
Issues from production multi-site / multi-region cloud:
Vancouver session moderated by Comcast: https://etherpad.openstack.org/p/YVR-ops-multi-site
OPNFV multisite project: https://etherpad.opnfv.org/p/multisite_usecase_collection
vApp 5
vApp 1
vApp 2
vApp 6
OpenStack
vApp 8
vApp 4vApp 3
vApp 7
OpenStack
VM… Volume… Network, Port… VM… Volume… Network, Port…
Issue: discrete resource utilization

6. HUAWEI TECHNOLOGIES CO., LTD. Page 6Huawei Confidential
 Missing global view for tenant level metering data.
 Issue for a user being able to access all the logs and indicators produced by multiple OpenStack
instances, in a centralized way
Issues from production multi-site / multi-region cloud:
Vancouver session moderated by Comcast: https://etherpad.openstack.org/p/YVR-ops-multi-site
OPNFV multisite project: https://etherpad.opnfv.org/p/multisite_usecase_collection
Ceilometer Ceilometer
Metering…
Sample…
Issue: discrete metering data
Metering…
Sample…

7. HUAWEI TECHNOLOGIES CO., LTD. Page 7Huawei Confidential
 Nova:
 Lack of clone ssh keys from one region to another
Issues from production multi-site / multi-region cloud:
Vancouver session moderated by Comcast: https://etherpad.openstack.org/p/YVR-ops-multi-site
OPNFV multisite project: https://etherpad.opnfv.org/p/multisite_usecase_collection
Nova Nova
KeyPair… KeyPair…

8. HUAWEI TECHNOLOGIES CO., LTD. Page 8Huawei Confidential
 Glance:
 Lack of replicate images across regions
Issues from production multi-site / multi-region cloud:
Vancouver session moderated by Comcast: https://etherpad.openstack.org/p/YVR-ops-multi-site
OPNFV multisite project: https://etherpad.opnfv.org/p/multisite_usecase_collection
Glance Glance
Upload image… Upload Imgae…

9. HUAWEI TECHNOLOGIES CO., LTD. Page 9Huawei Confidential
 Cinder:
 Lack the function like volume replication across regions for Geo-site
Redundancy/Disaster Recovery
Issues from production multi-site cloud:
Vancouver session moderated by Comcast: https://etherpad.openstack.org/p/YVR-ops-multi-site
OPNFV multisite project: https://etherpad.opnfv.org/p/multisite_usecase_collection
Cinder CinderVolume
Volume replication for Geo-site
Redundancy/Disaster Recovery ….
Volume

10. HUAWEI TECHNOLOGIES CO., LTD. Page 10Huawei Confidential
Tricircle
…
Tenant 1
Virtual Resources
Tenant 2
Virtual Resources
Tenant x
Virtual Resources
Cascaded OpenStack 2Cascaded OpenStack 1 Cascaded OpenStack y
OpenStack API OpenStack API OpenStack API
Tricircle – best matched multisite solution:
Integrate multiple OpenStack instances from multi-site into one unified cloud for tenant’s resources distribution
https://wiki.openstack.org/wiki/Tricircle
Tenant 2
Tenant 1
Tenant x
OpenStack API
•L2/L3 networking across regions
•clone security groups across regions
•global view of tenant quota control
•global view of tenant resource utilization
•global view of tenant metering data
•tenant level global IP address space
management
•KeyPair replication
•Replicate images
•Volume replication
•…..
Service
Orchestrator x
Service
Management
Service
Orchestrator 1
…

11. HUAWEI TECHNOLOGIES CO., LTD. Page 11Huawei Confidential
Tricircle architecture, stateless design:
Tricircle essentially serves as the central OpenStack API calls gateway to other OpenStack instances that are called upon
https://wiki.openstack.org/wiki/Tricircle
https://github.com/openstack/tricircle

12. HUAWEI TECHNOLOGIES CO., LTD. Page 12Huawei Confidential
Tricircle
…
Tenant 1
Virtual Resources
Tenant 2
Virtual Resources
Tenant x
Virtual Resources
Cascaded OpenStack 2Cascaded OpenStack 1 Cascaded OpenStack y
OpenStack API OpenStack API
OpenStack API
Service
Orchestrator x
Service
Management
Service
Orchestrator 1
OpenStack API OpenStack API
OpenStack API
… …
Cloud over clouds by OpenStack cascading: the cascaded OpenStack instance even can be running inside Amazon AWS or
Windows Azure, and then being integrated into a cloud through the cascading OpenStack. This is cloud over clouds.
More thinking

13. HUAWEI TECHNOLOGIES CO., LTD. Page 13Huawei Confidential
Service
Orchestrator x
Service
Management
Service
Orchestrator 1
Tricircle
(Tenant 1) …
Tricircle
(Tenant 2)
Tricircle
(Tenant x)
…
Tenant 1
Virtual Resources
Tenant 2
Virtual Resources
Tenant x
Virtual Resources
Cascaded OpenStack 2Cascaded OpenStack 1 Cascaded OpenStack y
https://tenant1.OpenStack/ https://tenant2.OpenStack/ https://tenantx.OpenStack/
OpenStack API
OpenStack API
OpenStack API
OpenStack API
OpenStack API OpenStack API OpenStack API
OpenStack API
OpenStack API
Fully distributed cascading layer: cascading OpenStack layer can be fully distributed if each small piece of cascading
OpenStack only serve limited tenants. Each tenant can still has globe view for his distributed resources. Unlimited scalability can be
achieved through this horizontal split.
… …More thinking

14. HUAWEI TECHNOLOGIES CO., LTD. Page 14Huawei Confidential
Also solve OpenStack scalability at least 10 times level
OpenStack …
…
1 2 100
1 2 1000
OpenStack
…
1 2 1000
OpenStack
…
1 2 1000
Tricircle
Max. 100 data centers
Max. 100k physical server nodes
Max. 1 million VMs
OpenStack API OpenStack API OpenStack API
OpenStack API
Scalability inside one data center, multi-
data centers or multi-sites
Test report: http://www.slideshare.net/JoeHuang7/test-report-for-open-stack-cascading-solution-to-support-1-million-v-ms-in-100-data-centers

15. HUAWEI TECHNOLOGIES CO., LTD. Page 15Huawei Confidential
More information:
Wiki: https://wiki.openstack.org/wiki/OpenStack_cascading_solution
Wiki: https://wiki.openstack.org/wiki/Tricircle
Source Code: https://github.com/openstack/tricircle