The document outlines a testbed configuration for a 13-node OpenStack cluster using Mirantis Fuel 6.0 and various hardware specifications. It details performance testing for network and disk operations using iperf and bonnie++, revealing performance ranges and issues, particularly under different MTU settings and configurations. The conclusion notes that while network performance was solid, it could be improved with better hardware and a more optimized Ceph setup.

1. M2 Reference Architecture Performance
(Mirantis, Openstack, and Commodity Hardware)
Ubuntu 12.04
Fuel 6.0
Openstack Juno
Myricom 10G Ethernet
Ceph RBD Glance/Cinder
8 Disk Local Raid5 Ephemeral/Root

2. Ryan Aydelott - CELS
Introduction
The purpose of this exercise is to create a small testbed using the Mirantis
“Fuel” deployment tool (version 6.0) to build a small 13 node cluster.
This deployment consisted of: 3 Controller nodes, 10 compute nodes, and 13
Ceph nodes
Each node class (controller/compute) was configured identically with 2TB of
local disk for ephemeral storage and 1TB of local disk for Ceph per node, the
remaining being used for local OS storage. The array holding this data
consisted of a single 8 disk Raid5 array made up of 500GB drives. Additionally
a small Raid1 array consisting of two 50GB SSD drives was used for Ceph
journal storage.
In testing, RBD, ephemeral volume, and inter-VM network performance will be
measured.
2

3. Ryan Aydelott - CELS
Testbed Hardware
‣ IBM System x3650 M3
‣ 12 Cores @2.67Ghz
‣ x12 Dimms for 48GB @1.33Ghz
‣ ServeRAID M5015 x2 (x2 SSD for Ceph Journal, x8 Magnetic for Instance/
Ceph)
‣ x1 1G ethernet for provisioning, x1 10G ethernet for storage/vm tenant
networking (MYRICOM Inc. Myri-10G Dual-Protocol NIC (rev 01))
‣ Juniper ex4500-40f
3

4. Ryan Aydelott - CELS
Testbed Software
‣ Deployment Node: Fuel 6.0
‣ Controller Nodes: Ubuntu 12.04
‣ Compute Nodes: Ubuntu 12.04
‣ Openstack Release Juno
‣ Ceph version 0.80.7 (Firefly) with replica/3
4

5. Ryan Aydelott - CELS
Testbed Configuration
‣ A flavor on Openstack was created that consumed an entire HV
‣ 10 total instances were created, each on it’s own dedicated hypervisor
‣ The guest OS was Ubuntu 14.04 for each instance
‣ iperf version 2.0.5 (08 Jul 2010) pthreads (networking)
‣ bonnie++ v1.97 (disk)
‣ Ephemeral storage ran on local hardware raid5
‣ Ceph OSD ran on top of local hardware raid5 (not ideal, but simplest
deployment with Fuel given hardware contraints)
5

6. Ryan Aydelott - CELS
Local Network Performance
‣ iperf was utilized to test intervm network performance
‣ By default the HV and all backend network devices had an MTU set at
9000, the MTU on the default ubuntu 14.04 cloud image is set to 1500, we
ran tests using both 1500 byte and 9000 byte MTU’s on the the guest.
‣ 1500 Byte MTU: 5Gigabit per instance/HV (25% of theoretical max)
‣ 9000 Byte MTU: 16Gigabit per instance/HV (80% of theoretical max)
‣ Latency ~.5ms average guest<—>guest
‣ Traffic was appropriately divided by node count, whereas one node would
use 80% of the theoretical max, 10 nodes would each use 8% of the
theoretical max, 5 nodes @ 16%, etc.
6

7. Ryan Aydelott - CELS
Local Disk (ephemeral) Performance
‣ bonnie++ 1.97 was used to benchmark local disk
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
ryantest-640 80000M 550 99 282290 80 134720 21 1272 98 152762 43 189.3 11
Latency 16469us 1550ms 2709ms 36363us 1195ms 1820ms
Version 1.97 ------Sequential Create------ --------Random Create--------
ryantest-64011122-4 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 13526 64 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 166ms 526us 461us 58307us 45us 132us
7

8. Ryan Aydelott - CELS
Ceph RBD (Cinder) Performance
‣ bonnie++ 1.97 was used to benchmark Ceph RBD
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
ryantest-5b7 80000M 593 99 95170 27 45953 10 956 75 124535 19 4283 125
Latency 15259us 6451ms 1568ms 820ms 999ms 31255us
Version 1.97 ------Sequential Create------ --------Random Create--------
ryantest-5b7edbb6-2 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 10107 80 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 110ms 597us 468us 11835us 581us 105us
8

9. Ryan Aydelott - CELS
Conclusion
While not amazing, a solid network performance using nova-network was
achieved. Running under Neutron/OVS with VLAN segmentation was less then
ideal and only achieved 10% of theoretical network performance under the best
of circumstances.
Furthering this, Neutron OVS also had varying latency from sub millisecond to
upwards of 50ms for interVM latency.
Local disk was as expected ~130-280mb/s with Ceph RBD fluttering
~50-125mb/s depending on the type of workload.
Disk performance could certainly be improved with better local disks and a
more optimal Ceph configuration, however as a generic deployment this
configuration would work for most basic use cases.
9