After constructing your OpenStack cloud, it can be difficult to determine whether you've actually configured all the components correctly. OpenStack Rally and Tempest have been created to help run verification and benchmarking tests for you, but they themselves are difficult to configure and use. This session will explore creating an easy and repeatable process verification and benchmarking process for your OpenStack cloud. Drawing on experience from numerous customer installations, it will delve into the benefits and pitfalls of using specific tools and technologies to achieve your testing goals. This presentation was given at the 2014 Fall OpenStack Summit. A recording of the presentation is available at https://www.openstack.org/summit/openstack-paris-summit-2014/session-videos/presentation/are-we-done-yet-testing-openstack-deployments .

1. Accelerating the adoption of Cloud Computing
Are We Done Yet ?
Testing Your OpenStack Deployment
OpenStack Summit, Paris, France
November 4, 2014

2. © Solinea, Inc. 2
Who Am I ?
Ken Pepple is the Chief Technology Officer of
Solinea
Prior to founding Solinea, he led the introduction of
Internap's OpenStack-based public cloud services
while serving as their Director of Cloud
Development
Author of O'Reilly "Deploying OpenStack” book
OpenStack code contributor since Bexar release

3. © Solinea, Inc.
!
Who we are
3
Solinea is the leading software and services company that delivers integrated
OpenStack™ solutions, enabling enterprise adoption of Cloud Computing
Cloud is the only domain we focus on, with vertical industry
and horizontal solutions specialization
Purpose-built
for cloud
Track record of success architecting, building and operating
production clouds – private and public – world-wide
Proven Delivery
Success
We understand cloud adoption challenges of global enterprise
companies
Enterprise IT
Experience
Pre-configured management and monitoring platform for
OpenStack™ and other Open Infrastructure components
Differentiating
IP
Built the first OpenStack production clouds and
contributors to the platform since its inception
OpenStack™
Experience

4. © Solinea, Inc.
Agenda
! Problem statement
! Types of testing
! Testing tools
! Future improvements
4

5. © Solinea, Inc.
Problem
! Many components, even more connections
! No two clouds are identical
– Hypervisor choices, network choices, storage choices
! Many rarely used code paths (backups ?)
! Operational and integration concerns
! Tools for operators
! Requires skills across all the services
5

6. © Solinea, Inc.
In a nutshell … distributed systems
OpenStack Object Store
OpenStack
Image Service OpenStack Compute
OpenStack
Dashboard
OpenStack
Identity
Service
OpenStack Compute
API /
Admin API
keystone
(service & admin APIs)
nova-api
(OS, EC2, Metadata, Admin)
nova-consoleauth
nova-cert/
objectstore
nova-console
nova-*proxy
VNC/Spice/RDP
OpenStack
Object API
http://www.solinea.com
OpenStack
Data Processing
sahara-api
sahara-engine sahara
database
Queue sahara-conductor
OpenStack
Data Processing API
(8386)
OpenStack
Identity
API
OpenStack
Data Processing API
Queue
nova-compute
nova-scheduler
nova
database
OpenStack
Compute API
(8774, 8775, 8776)
OpenStack
Image API
Horizon
OpenStack
Image API
(9292)
identity
backend
swift-proxy
objectcontaineraccount
object
store
account
DB
container
DB
OpenStack
Object API
HTTP(S)
OpenStack
Object API
OpenStack
Identity API
OpenStack
Identity
API
OpenStack
Identity
API
OpenStack
Image API
OpenStack
Identity
API
OpenStack
Image
API
catalog
backend
token backend
OpenStack
Identity
API
(5000)
hypervisor
libvirt, XenAPI, etc.
HTTP(S)
Amazon
Web Services
EC2 API
Internet / Enterprise Network
OpenStack
Network Service
glance-api
glance
database
OpenStack
Block Storage
OpenStack
Block Storage API
cinder-api
cinder-volume
neutron-server
neutron
plugin(s)
OpenStack
Identity
API
cinder-scheduler
cinder
database
OpenStack
Network API
(9696)
network
provider
OpenStack
Block
Storage
API
OpenStack
Network API
policy
backend
Queue
OpenStack
Network API
neutron
database
neutron
agent(s)
nova-conductor
memcached
OpenStack
Identity
API
⁃ OpenStack Command Line Tools (nova-client, swift-client, etc.)
⁃ Cloud Management Tools (CloudForms, Enstratius, etc.)
⁃ GUI tools (Web Browsers, Cyberduck, iPhone client, etc.)
volume provider
cinder-backup
OpenStack Object API
OpenStack
Identity
API
Queue
OpenStack
Block Storage API
OpenStack
Orchestration
heat-api
heat-engine
heat
database
Queue
cloudwatch-api
OpenStack
Orchestration API
OpenStack
Identity
API
OpenStack Compute
API
OpenStack Bock Storage
API
OpenStack Network API
glance-registry
OpenStack
Metering
Service
ceilometer-api
ceilometer
database
ceilometer-collector
OpenStack
Database
trove-api
trove-taskmgr
trove
database
trove-conductor
OpenStack
Database API
(8779)
OpenStack Block Storage API
OpenStack Orchestration API
OpenStack Compute API
Database
(mysql, redis, mongo,
cassandra, etc.)
Queue
trove-guestagent
OpenStack
Identity
API
6

7. © Solinea, Inc.
What Should We Be Testing?
! Functionality
– User scenarios
– Not code quality … unless running trunk
! Non-functional requirements
– Scalability
– Performance
– Security
! Operations
– High availability and failover
7

8. © Solinea, Inc.
How Should We Test ?
! Automated
– Repeatability
– Multiple checks
! Continual
! Ops not specialists
– Should need coders or QA people
– Testing should be an easy command
! Integrated into operational procedures
– “Let me send a canary in …”
8

9. © Solinea, Inc.
Types of Testing
! Unit testing
! Integration
! User Acceptance Tests
! Operational
9

10. © Solinea, Inc.
Functional Areas
! Networking
! Storage
! Compute
! User functional process
! OUTSIDE applications …
– Billing/metering, user signup, audit/compliance …
10

11. © Solinea, Inc.
OpenStack Testing Options
! Tempest
! Rally
! Manual
11

12. © Solinea, Inc.
Tempest
! OpenStack testing project
! Runs hundreds of tests
– Unit
– Scenario
– Stress
– API
– CLI
! Complex to configure
! Not portable to every cloud deployment configuration
! Requires significant investment in time
12

13. © Solinea, Inc.
An OpenStack project to
create a framework for
validating, performance
testing and benchmarking
OpenStack at scale with
Tempest
https://wiki.openstack.org/
wiki/Rally
13
Rally

14. © Solinea, Inc.
Rally Scenario
$ rally task start summit/create-and-list-users.json
================================================================================
Task 57bc6d22-76c2-4450-b834-daa48c41d6e0 is started
--------------------------------------------------------------------------------
================================================================================
Task 57bc6d22-76c2-4450-b834-daa48c41d6e0 is finished.
--------------------------------------------------------------------------------
test scenario KeystoneBasic.create_and_list_users
args position 0
args values:
{u'args': {u'name_length': 10},
u'runner': {u'concurrency': 10, u'times': 100, u'type': u'constant'}}
+----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
| action | min (sec) | avg (sec) | max (sec) | 90 percentile | 95 percentile | success | count |
+----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
| keystone.create_user | 1.767 | 3.478 | 5.944 | 4.565 | 4.694 | 100.0% | 100 |
| keystone.list_users | 0.077 | 0.464 | 1.516 | 1.01 | 1.143 | 100.0% | 100 |
| total | 1.982 | 3.942 | 6.218 | 4.956 | 5.214 | 100.0% | 100 |
+----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
Whole scenario time without context preparation: 40.6584420204
14

15. © Solinea, Inc.
Rally Database
! Rally stores all scenario/task runs
! Can be used to trend and benchmark results
$ rally task list
+--------------------------------------+----------------------------+----------+--------+-----+
| uuid | created_at | status | failed | tag |
+--------------------------------------+----------------------------+----------+--------+-----+
| 57bc6d22-76c2-4450-b834-daa48c41d6e0 | 2014-11-05 12:10:38.235211 | finished | False | |
| 12d34393-0520-4830-9516-e7404935c0e9 | 2014-11-05 12:32:39.632810 | finished | False | |
| 3b160652-ed3b-4e2b-b09c-14b6d900e18e | 2014-11-05 12:40:21.544483 | running | False | |
+--------------------------------------+----------------------------+----------+--------+-----+
15

16. © Solinea, Inc.
Rally Scenarios
{
"KeystoneBasic.create_delete_user": [
{
"args": {
"name_length": 10
},
"runner": {
"type": "constant",
"times": 100,
"concurrency": 10
}
}
]
}
16
Method
Customization
Concurrency

17. © Solinea, Inc.
Rally Scenarios
! Authenticate
! Ceilometer
! Cinder
! Designate
! Glance
! Heat
! Keystone
! Neutron
! Nova
! quotas
! Requests
! Sahara
! Tempest
! VM
! Zaqar 17

18. © Solinea, Inc.
Rally Visualization
18

19. RALLY DEMO
19

20. © Solinea, Inc.
Operational HA Tests
! Kill process / hung process
! Lost interface
! Lost machine
! Lost rack
20

21. © Solinea, Inc.
Continual Testing
! Testing should be part of monitoring and
management
! “Order Canary”
– Chao monkey may not be appropriate but
“happy path” test should be
– Define rally scenarios or custom scripts to feed
monitoring tools
! Subset of tests that affect business critical
functions
21

22. © Solinea, Inc. 22
Putting it all together
OpenStack CloudRally
[os-admin]
OS API commands
Monitoring
[goldstone]

23. © Solinea, Inc.
Best Practices
! ABT (Always be testing)
! Manual testing " operational procedure
– Failover regularly as part of normal operations
– HA is an event, it is SOP
! Testing is part of the deployment, monitoring and
upgrade process
23

24. © Solinea, Inc.
Goldstone Log Monitoring

25. © Solinea, Inc.
Future
! /info APIs for all services
– Describe deployment beyond services
! Simplified DSL to configure tempest
! Monitoring product incorporate testing
25

26. Ken Pepple, CTO
ken@solinea.com
http://www.solinea.com
Thank You