The document discusses using cells in OpenStack to scale cloud infrastructure across multiple geographic locations. Key points include using cells to distribute OpenStack compute services around Australia, with over 6000 users, 700 hypervisors, and 30,000 cores spread across 8 sites and 14 cells. It also discusses strategies for operating, upgrading, and scheduling across multiple cells.

1. Tips, Tricks and Tactics with Cells
and Scaling OpenStack
●
●
●

2. OpenStack Summit - Paris 2014
Multi-Cell Openstack: How to Evolve your Cloud to Scale
https://www.openstack.org/summit/openstack-paris-summit-2014/session-videos/presentation/multi-cell-
openstack-how-to-evolve-your-cloud-to-scale

3. Sam Morrison
sam.morrison@unimelb.edu.au

4. Australian Research Cloud
● Started in 2011
● Funded by the Australian Government
● 8 institutions around the country
● Production early 2012 -
Openstack Diablo
● Now running a mix of Juno
and Icehouse
● Use Ubuntu 14.04 and KVM
● 100Gbps network connecting most sites (AARNET)

5. Reasons for using cells
Single API endpoint, compute cells dispersed
around australia.
Simpler from users perspective.
Single set of security groups, keypairs etc.
Less openstack expertise needed as only one
version of some core openstack services.

6. Size
● 8 sites
● 14 cells
● ~6000 users registered
● ~700 hypervisors
● 30,000+ cores
People
● Core team of 3 devops
● 1-2 operators per site
http://status.rc.nectar.org.au/growth/infrastructure/

7. Interaction with other services
Each cell also has one or more:
● cinder-volume host, using Ceph, LVM and NetApp
backends
● A globally replicated swift region per site
● Glance-api pointing to local swift proxy for images
● Ceilometer collectors at each cell push up to a central
mongo
● Private L3 network spanning all cells - will be useful for
neutron

8. Cells Infrastructure
Each compute cell has:
● MariaDB Galera cluster
● RabbitMQ cluster
● nova scheduler/cells/vnc/compute/network/api-metadata
● glance-api
● swift region - proxy and storage nodes
● ceilometer-collectors to forward up to a global collector
● cinder-volume
API cell:
● nova-api, nova-cells
● keystone
● glance-registry
● cinder-api, scheduler
● heat, designate, ceilometer-api

9. Scheduling
● Have some “private” cells only available to certain tenants. This is usually
determined by funding source.
● Global flavours and cell local flavours
● Cell level aggregates for intra cell scheduling
● Some sites have GPUs, fast IO for their own use.
○ Introduced compute and ram optimised flavours
○ Not all cells support all flavours
● Each cell advertises 1 or more availability zones to use in scheduling.
○ Ties in with cinder availability zones

10. Bringing on new cells
Want to test in production before opening to public
Don’t want to flood brand new cells
Scheduler filters
● Role based access to cell. Cells advertise what roles can schedule to them
● Direct only - Allow public to select that cell directly but global scheduler
doesn’t count it

11. Operating cells
Have a small openstack cluster to manage all global infrastructure
Standard environment - use puppet
Upgrade cells one at a time - live upgrades
● upgrade compute conductors
● upgrade API cell
● upgrade compute cells
● upgrade compute nodes
Read access to compute cells RabbitMQs for troubleshooting and monitoring.
Only real interface into each of the cells.
Console-log is a good test of cells functionality - have one in each cell and
monitor

12. Future plans
Move to Neutron - in planning and testing stage
● Currently have a single public network per cell, want to provide tenant
networks and higher level services
Start off with a global neutron and simple shared flat provider networks per cell.
All hypervisors talking to the same rabbit - scale issues?
Also looking at other higher level openstack services (which there are many!)

13. Belmiro Moreira
belmiro.moreira@cern.ch
@belmiromoreira

14. CERN - Large Hadron Collider

15. CERN - Cloud Infrastructure
●
●
○
●
●
○
●
○
○
○
○

16. CERN - Cloud Infrastructure
●
○
○
○
○
■
●
○

17. CERN - Prune Nova DBs
●
●
●
●
○
○
○

18. CERN - Cells scheduling
●
○
●
○
■
■
■
■

19. CERN - Flavors management
●
○
●
○
■
●
○
○

20. CERN - Testing Cells
●
○
○
●
○
○
○ …

21. CERN - Testing Cells

22. CERN - Testing Cells

23. Matt Van Winkle
mvanwink@rackspace.com
@mvanwink
www.rackspace.com
Cells at Rackspace

24. • Managed Cloud company offering a suite of dedicated and cloud hosting products
• Founded in 1998 in San Antonio, TX
• Home of Fanatical Support
• More than 200,000 customers in 120 countries
Rackspace
24www.rackspace.com

25. • In production since August 2012
– Currently running: Nova; Glance; Neutron; Ironic; Swift; Cinder
• Regular upgrades from trunk
– Package built on trunk pull from mid March in testing now
• Compute nodes are Debian based
– Run as VMs on hypervisors and manage via XAPI
• 6 Geographic regions around the globe
– DFW; ORD; IAD; LON; SYD; HKG
• Numbers
– 10’s of 1000’s of hypervisors (Over 340,000 Cores, Just over 1.2 Petabytes of RAM)
• All XenServer
– Over 170,000 virtual machines
– API per region with multiple Compute cells (3 – 35+) each
Rackspace – Cloud Infrastructure
25www.rackspace.com

26. • Cells Infrastructure
– Size between ~100 and ~600 hosts per cell
– Different Flavor Types (General Purpose, HIgh I/O, Compute Optimized, etc)
– Working on exposing maintenance zones or near/far scheduling (host, shared IP space, network aggregation)
– Separate DB cluster for each cell
• Run our Cells infrastructure in cells
– Control Plane exists as instances in small OpenStack deployment
– Multiple Hardware types
– Separate tenants – Control plane instances from other internal users
Rackspace – Cloud Infrastructure - Cells
26www.rackspace.com

27. • Multiple cells within each flavor class
– Hardware Profile
• Additionally, we group by vendor
• Live migration needs matching CPUs
– Range of flavor size within each cell (eg. General Purpose 1, 2, 4 and 8 Gig)
• Tenant Scheduling
– Custom filter schedules by Flavor class first
• All General Purpose cells, for example
– Scheduled by available RAM afterwards
• Enhancements for spreading out tenant load and max IOPs per host
– In some cases, filters can bind a cell to specific tenants (testing and internal use)
• Work in Cells V2 to enhance scheduling
– https://review.openstack.org/#/c/141486/ as one example
27
Cell Scheduling
www.rackspace.com

28. • Common control plane nodes deployed by ansible play book
– DB Pair
– Cells service
– Scheduler
– Rabbit
• Playbook Populates flavor info based on hardware type
• Hypervisors bootstrapped once CP exists
– Create Compute Node VM
– Deploy Code and configure
– Update routes, etc
• Provision IP blocks
• Test
• Link via playbook
28
Deploying a Cell
www.rackspace.com

29. • Larger region has run rate
around 50,000 VMs
• 1000’s of VMs created/deleted
per hour in busiest regions
• Downstream BI and Revenue
assurance teams require deleted
instance records be kept for 90
days
• Current deleted instance counts
range between 132,000 and
900,000
29
Rackspace – Purge Nova DBs
www.rackspace.com

30. 30
Rackspace – Purge Nova DBs
www.rackspace.com

31. • By Pass URL prior to linking a cell up
– Test API endpoint: http://nova-admin-api01.memory1-0002.XXXX.XXXXXX.XXXX:8774/v2
• Full set of tests
– Instance creates, deletes, resizes
– Overlay network creation
– Volume provisioning
– Integration with other RS products
• Trickier to test hosts being added to an existing cell
– Hosts are either enabled or disabled
– Targeting helps
• --hint target_cell=’<cellname>’
• --hint 0z0ne_target_host=<host_name>
31
Testing Cells
www.rackspace.com

32. • No formal way of disabling a cell
• Weighting helps – but is not absolute
– Weighting cell can still “win” scheduler calculation based on available RAM
• Solution: custom filter uses specific weight offset value to avoid scheduling (- 42)
32
Managing Cells – “Disable”
www.rackspace.com
class DisableCellFilter(filters.BaseCellFilter):
"""Disable cell filter. Drop cell if weight is -42.
"""
def filter_all(self, cells, filter_properties):
"""Override filter_all() which operates on the full list
of cells...
"""
output_cells = []
for cell in cells:
if cell.db_info.get('weight_offset', 0) == -42:
LOG.debug("cell disabled: %s" % cell)
else:
output_cells.append(cell)
return output_cells

33. 33
Managing Cells – “Disable”
www.rackspace.com

34. • Rackspace uses Quark Plugin
– https://github.com/rackerlabs/quark
• Borrowed old idea from Quantum/Melange days
– Default tenant for each cell
– Each cell is a segment
– Provider subnets are scoped to a segment
– Nova requests ports on provider network for the segment
• Public
• Private
• MAC addresses too
34
Neutron and Cells
www.rackspace.com

35. ?
●
●
●