The OpenStack Innovation Center (OSIC), established by Intel and Rackspace, is created to accelerate adoption of open source cloud operating system while supporting open source principles. OSIC provides ready-to-use data center facilities to the OpenStack community for development and test. This case study presentation highlights a scale-out test performed within a 3 week period using OpenStack Ansible Community based on Liberty with an SDN overlay network connecting 131 nodes running over 1,000 VMs. Tempest and Rally tests were conducted to validate functions including high availability failure scenarios. Join this session to find out more about OSIC and the SDN scale-out test configuration, scenarios, and results.

1. OpenStack Summit | Barcelona, Spain
SDN Scale-out Testing at
OpenStack Innovation Center (OSIC)

2. Copyright © PLUMgrid, Inc. 2011-2016Copyright © PLUMgrid, Inc. 2011-2016
Introduction
Speakers
2
Title/Designation
Hillsman
Melvin
Solutions Architect
PLUMgrid
Arif
Jamal
Product Marketing
Engineer
Intel Corporation
Broughton
Travis

3. Copyright © PLUMgrid, Inc. 2011-2016Copyright © PLUMgrid, Inc. 2011-2016
What is the OpenStack Innovation Center?
OSIC’s Mission:
Making OpenStack* easy to deploy, operate and use, with all of the features of an
enterprise-class computing platform
OSIC has three main tenets:
1) Grow the number of OpenStack contributors
2) Increase upstream OpenStack contributions
3) Enable OpenStack innovation at Unmatched scale
3 *Other names and brands may be
claimed as the property of others.

4. Copyright © PLUMgrid, Inc. 2011-2016Copyright © PLUMgrid, Inc. 2011-2016
Grow the number of OpenStack* Contributors
4 *Other names and brands may be
claimed as the property of others.

5. Copyright © PLUMgrid, Inc. 2011-2016Copyright © PLUMgrid, Inc. 2011-2016
Increase Upstream OpenStack* Contributions
5 *Other names and brands may be
claimed as the property of others.

6. Copyright © PLUMgrid, Inc. 2011-2016Copyright © PLUMgrid, Inc. 2011-2016
Enable OpenStack* Innovation at Unmatched Scale
6
To support advanced, community-wide,
large-scale testing of OpenStack.
*Other names and brands may be
claimed as the property of others.

7. PLUMgrid Scale-Out Testing at OSIC:
Infrastructure Setup
7

8. Copyright © PLUMgrid, Inc. 2011-2016Copyright © PLUMgrid, Inc. 2011-2016
OSIC Test Bed (April 2016)
8
• 131 Physical Node Cluster
• Three Weeks
• Installation
• OpenStack Ansible Community (OSA) Version 12 (Liberty)
• PLUMgrid OpenStack Networking Suite (ONS) Version 4 (Neutron Plugin)
• PLUMgrid CloudApex Version 1 (Cloud Visualization and Operation Tool)
• Scale:
• 80 Tenants (Virtual Domains) 1600 Virtual Machines

9. Copyright © PLUMgrid, Inc. 2011-2016Copyright © PLUMgrid, Inc. 2011-2016
OSIC Test Bed (Oct 2016)
9
• 131 Physical Node Cluster
• Three Weeks
• Installation
• OpenStack Ansible Community (OSA) Version 13 (Mitaka)
• PLUMgrid OpenStack Networking Suite (ONS) Version 6 (Neutron Plugin)
• PLUMgrid CloudApex Version 1.2 (Cloud Visualization and Operation Tool)
• Scale:
• 140 Tenants (Virtual Domains) 4000 Virtual Machines

10. Copyright © PLUMgrid, Inc. 2011-2016Copyright © PLUMgrid, Inc. 2011-2016
What is PLUMgrid ONS?
Extensive software-only networking solution for OpenStack® Clouds
10
Top Security & Compliance Support with
built-in isolation, micro-segmentation via
Virtual Domains & BYO service
Unmatched Operational tools with proactive
visibility & analytics (CloudApex)
Virtual networks provisioned by users
Scalable, distributed & highly available
architecture enables Production
Deployments

11. Copyright © PLUMgrid, Inc. 2011-2016Copyright © PLUMgrid, Inc. 2011-2016
PLUMgrid ONS Overview
Full Hybrid Solution for Private and Hybrid Cloud
11
Internet
IO Visor Gateway
IO Visor Edges (Compute Nodes)
PLUMgrid Directors
VXLAN-based
Overlay
PLUMgrid CloudApex & OpsVM
PLUMgrid IO Visor based Edges
PLUMgrid Director
PLUMgrid Gateways

12. Copyright © PLUMgrid, Inc. 2011-2016Copyright © PLUMgrid, Inc. 2011-2016
OSIC Test Bed Setup
12
Internet
External Servers x 2
Gateway Servers x 4
Director /
OpenStack Controllers x 3
Infra Host
Compute Nodes x 121
OPS LCM BOOT

13. Copyright © PLUMgrid, Inc. 2011-2016Copyright © PLUMgrid, Inc. 2011-2016
OpenStack Dashboard + PLUMgrid Console
13

14. Copyright © PLUMgrid, Inc. 2011-2016Copyright © PLUMgrid, Inc. 2011-2016
CloudApex (Cloud Visualization and Orchestration Tool) (1)
14

15. Copyright © PLUMgrid, Inc. 2011-2016Copyright © PLUMgrid, Inc. 2011-2016
CloudApex (Cloud Visualization and Orchestration Tool) (2)
15

16. PLUMgrid Scale-Out Testing
16

17. Copyright © PLUMgrid, Inc. 2011-2016Copyright © PLUMgrid, Inc. 2011-2016
PLUMgrid Scale-Out Testing
17
• OpenStack Functional Validation with Tempest Suite
• Cluster Scale testing using Heat templates
• PLUMgrid High-Availability Testing at Scale
• PLUMgrid Scale Out Testing with Rally Tests

18. Copyright © PLUMgrid, Inc. 2011-2016Copyright © PLUMgrid, Inc. 2011-2016
OpenStack Functional/Api Validation
18
There were NO applicable deviations
observed
Tempest tests to cover the OpenStack
functional/api validation (neutron, image,
identity, compute)

19. Copyright © PLUMgrid, Inc. 2011-2016Copyright © PLUMgrid, Inc. 2011-2016
PLUMgrid Scale using Heat Templates
19
Service Virtual
Domain For External
Connectivity
Tenant Virtual Domains Network Topology
Each Tenant with distributed virtual
network functions (Distributed Bridge,
Distribted Router, Distributed NAT,
Distributed Security Policy framework)
Heat templates to reach 140 tenants
and 4000 VMs

20. Copyright © PLUMgrid, Inc. 2011-2016Copyright © PLUMgrid, Inc. 2011-2016
Is it Highly Available at Scale?
20
PLUMgrid Single Director Failure Test
• Successfully verified Connectivity of
VMs
3 Controllers ( 3 PG Directors)), 121
Compute Nodes (PG Edges, 4 PG
Gateway Servers, 140 tenants, 4000 VMs
PLUMgrid Director Cluster Failure
(Control Plane Failure)
• Successfully verified Connectivity of
VMs in Headless Mode

21. Copyright © PLUMgrid, Inc. 2011-2016Copyright © PLUMgrid, Inc. 2011-2016
OpenStack/PLUMgrid Rally Testing
21
Tests at 80 Tenants (Virtual Domains)
160 VMs
Tests at No Load (Bench Marking)
80 Tenants (Virtual Domains) 1600
VMs
• Neutron Networks
• neutron.create_network
• Neutron.list_networks
• Neutron Routers
• neutron.create_network
• neutron.create_subnet
• neutron.create_router
• neutron.add_interface_router
• neutron.list_routers
• Neutron Security Groups
• neutron.create_security_group
• neutron.list_security_groups
• Neutron Subnets
• neutron.create_network
• neutron.create_subnet
• neutron.list_subnets
• Neutron VMs
• neutron.boot_servers
• neutron.delete_servers

22. Copyright © PLUMgrid, Inc. 2011-2016Copyright © PLUMgrid, Inc. 2011-2016
Rally Bench Marking with NO Load
22

23. Copyright © PLUMgrid, Inc. 2011-2016Copyright © PLUMgrid, Inc. 2011-2016
80 Tenants, 160 VMs
23

24. Copyright © PLUMgrid, Inc. 2011-2016Copyright © PLUMgrid, Inc. 2011-2016
80 Tenants, 1600 VMs
24

25. Copyright © PLUMgrid, Inc. 2011-2016Copyright © PLUMgrid, Inc. 2011-2016
What did we Learn?(1)
25
• Host inventory playbook issues
• Tweaked the SSH settings (timeout/retries) to reduce ‘unreachable’
errors
• Time – 2 days (48 hours)
Installation

26. Copyright © PLUMgrid, Inc. 2011-2016Copyright © PLUMgrid, Inc. 2011-2016
What did we Learn?(2)
26
• RabbitMQ
• Optimization patches
• Increasing the number of file descriptors
• Overrides for RPC thread pool sizes for nova & neutron services
• Horizon dashboard slow response at Scale
• Average network request service times showed an incremental
increase with scale
Scale Out Testing

27. Visit Us @Booth B37
THANK YOU!
www.plumgrid.com
Visit Us @Booth B37
THANK YOU!
www.plumgrid.com