2. Agenda
One
• Introduction
• OpenStack & DevStack
Two
• Tempest & Architecture
• DevStack config and H/w required
Three
• Various tests & Test Results
• Key Findings
Four
• Summary & Conclusion
• Tech-Scenarios & Citations
3. To demonstrate
starting/running
of OpenStack
services
Specifically for
development
work on
OpenStack
We can
download the
latest version of
DevStack from
”git” repo.
Cloud operating
system/platform
that controls large
pools of compute,
storage, and
networking
resources
Used for
developing private
clouds
• Keystone
• Cinder
• Glance
• Neutron
• Horizon
• Nova
OpenStack & DevStack
OpenStack
DevStack
4. OpenStack Testing ??
OpenStack
integration Test
Suite
• Tempest Test Suite
• Ready to deploy
Unit Tests in each
project
• Nova, Cinder, Swift
• Neutron etc
• OpenStack Code
base is written in
Python
• So how to
use/write tests?
• Isolate the
working
Environment
• Quick testing
• Automated
configuration
• Ease of use
5. Tempest
• Tempest is a
set of
integration
tests
• To be run
against any
live
OpenStack/
Dev Stack
cluster
F
e
a
t
u
r
e
s
Official OpenStack
Integration Test Suite
Interacts with majority
ReSTful API’s
Around 3000+ tests
covering all projects
Used as gating tests for all
new projects
Runs on all OpenStack
releases
Self Testing/Self Cleaning
Uses a unified tempest
REST client
Latest version of “Tempest
12.1.1.dev37”
A
t
t
r
i
b
u
t
e
s
Scenario based Tests
CLI Tests Stress tests
Ensure that OpenStack cloud
works with the OpenStack API
6. Basic Tempest Reference Architecture
OpenStack/ DevStack
deployment
Service
Clients
Python
Clients Boto Tests
Rest Client
API Tests
Scenario
Tests
Third party
tests
CLI based
Tests
Library CLI
OpenStack ReST API’s EC2 API’s
OpenStack ReST API’s
1
7. Basic Tempest Reference Architecture
Compute
Node
Compute
Node
Controller
Node
Test Server
(Tempest Node)
HTTP Request HTTP Response
OpenStack configuration
{Tempest tests executed
from controller node}
2
26. Key Findings-Failed Tests analysis
OS/Tests toxefull toxesmoke run_tempest
CentOS 7.0 Most of the Tempest test case
failures are
• Volume API’s
• Third party boto (this is related
to Elastic Compute (EC2) and
Object storage-S3 features of
AWS) ReST API failures.
Most of the Tempest test case
failures are
• Volume API’s related
Most of the Tempest test case failures are
• Scenario tests, snapshot pattern related,
Volume API’s and third party boto(this is
related to Elastic Compute (EC2) and
Object storage-S3 features of AWS)
ReST API failures.
Ubuntu 14.04 Most of the Tempest test case
failures are
• Compute, scenario and Cinder
Volume related
• Third party boto (this is related to
Elastic Compute (EC2) and
Object storage-S3 features of
AWS) ReST API failures.
Most of the Tempest test case
failures are
• Compute ,scenario and
Volume Rest API failures.
Most of the Tempest test case failures are
• Compute, scenario, Volume and third
party boto (this is related to Elastic
Compute (EC2) and Object storage-S3
features of AWS) ReST API failures.
Fedora 21 Most of the Tempest test case
failures are
• Volume API’s
• Third party boto (this is related
to Elastic Compute (EC2) and
Object storage-S3 features of
AWS) ReST API failures.
• Only one of the test case
failed here and is related to
Volume ReST API failure.
Most of the Tempest test case failures are
• Volume API’s and third party boto (this is
related to Elastic Compute (EC2) and
Object storage-S3 features of AWS)
ReST API failures.
Kilo
27. Key Findings-Failed Tests analysis
OS/Tests toxefull toxesmoke run_tempest
CentOS 7.0 Most of the Tempest test case
failures are
• Volume API’s
• Third party boto (this is related to
Elastic Compute (EC2) and
Object storage-S3 features of
AWS)
• Server multi-node ReST API
failures.
Tempest test case failure is
related to following
• Volume API’s
Most of the Tempest test case failures are
• Compute, Volume API’s, basic-Ops test
server, multi-node
• third party boto(this is related to Elastic
Compute (EC2) and Object storage-S3
features of AWS) ReST API failures.
Ubuntu 14.04
Most of the Tempest test case
failures are
• third party boto” (this is related to
Elastic Compute (EC2) and
Object storage-S3 features of
AWS) ReST API failures.
No failures found. Most of the Tempest test case failures are
• Compute, scenario, Volume and third party
boto (this is related to Elastic Compute
(EC2) and Object storage-S3 features of
AWS) ReST API failures.
Fedora 21 Most of the Tempest test case
failures are
• scenario tests, compute,
snapshot deleting, Volume API’s
and third party boto (this is
related to Elastic Compute (EC2)
and Object storage-S3 features
of AWS) ReST API failures.
Found two failures which are
• Volume ReST API failure.
Most of the Tempest test case failures are
• Volume API’s and third party boto (this is
related to Elastic Compute (EC2) and
Object storage-S3 features of AWS) ReST
API failures.
Liberty
28. Key Findings-Skipped Tests analysis
Some of the major reasons for “Skipped” tests are,
Pending bugs need to be resolved
Live block migration not supported
Instance validation and multiple instance support not available
SSH & VNC support not available in some cases
Some of the features not included in Dev Stack
Ephemeral disk verification could not be done
Multiple policy feature not supported
Instance validation and multiple instance support not available
Note:- These findings are across OpenStack releases and OS flavors
Liberty
Kilo
30. Conclusion
• Tempest test results across different OpenStack releases and OS flavors is distinct
• One of the primary reasons of this ambiguity is each OS flavor having different
kernel versions.
• The more latest the Kernel version used, the more stable are the test results
• The other reason is because of different OpenStack releases on different hardware
platforms
• Tempest execution on Liberty release on various Linux OS flavors is yielding more
stable test results as compared to Kilo release.
• The more latest the OpenStack version, the more stable are the test results.
• These test results can be used for future reference (based on h/w architecture used)
31. Troubleshooting…..
1. We have observed some times that when “localrc” file from root user and try to run
stack.sh command from the stack user, following error will be encountered.
Solution:- When such error is encountered, change the ownership of devstack
directory to stack user using the following command
32. Troubleshooting…..
2. If you are using proxy, While running stack.sh command you may get the
following error
Solution:- Best practice is to have internet connection without proxy, which will
avoid majority of the issues.
3. When we clone “nova” manually to /opt/stack/nova from git repository and try to
execute the command “stack.sh”, following exception will be encountered.
Solution:- To avoid the above exception, copy .git directory from cloned devstack
directory to /opt/stack/nova directory. This will help us to overcome the above
mentioned exception.
34. Tech-Scenario-I Validation of OpenStack workload management tool
Context
Validation and functionality test of
OpenStack & management tool together
was required
Validation of RESTful API’s for supported
OpenStack releases was required
Certification testing on different OS
platforms supporting OpenStack was
required.
Solution/approach used
Used Tempest test suite (along with some
customization where required) for the
validating stability of ReSTful API’s
Using Tempest test suite and some home
grown test cases, we were able to verify the
management tool and various OpenStack
features
Triaging with multiple combinatorial tests
and identifying gaps and helping respective
customer to fix the issues on time.
Outcomes & Learning
Since OpenStack was niche technology,
understanding the requirement clearly was a
challenge
State of the art certification of RHEL and
Ubuntu on OpenStack was a challenge.
Support for Continuous OpenStack testing
for supported releases was a challenge
Technical Challenges
Technology Ubuntu, SuSE, RHEL, Cent OS , OpenStack releases Havana, Icehouse and Juno
Test Suite used Tempest
Validation of complete workload management
tool on OpenStack with various test scenarios
Validating Nova, Keystone, AMQP message
bus protocol and Horizon interfaces.
Evaluation of the management tool for
robustness and stability of usage.
Triaging issues that are related to OpenStack
product
Certification of various OpenStack releases
done for RHEL and Ubuntu.
Ready test suite for future releases of
OpenStack was evolved
35. Tech-Scenario-II: Validation of OpenStack API’s using Tempest suite
Context
Complete coverage of OpenStack API
testing required
Reliability and Serviceability of OpenStack
on a specified hardware configuration was
required
Solution/approach used
Validation of complete ReST API suite for
various OpenStack releases and for specified
h/w platforms.
Complete test failure analyses with log
reporting on time.
Parallel Tempest configurations kick started
with test executions
Outcomes & Learning
Running Tempest test suite on multiple
Linux platforms
Debugging and finding failures and causes
Rally configuration and execution
Technical Challenges
Technology Ubuntu, RHEL, Cent OS , OpenStack releases Juno and Kilo,
Test Suite used Tempest
Tools Rally
Tempest configuration on multiple
platforms and OpenStack releases.
Validated complete basic OpenStack API
suite and reported failures
Suggested fixes where possible
39. What is Tox?
tox is a generic virtual environment management and test command line tool
that we can use for the following purposes.
We can check our package installed correctly with different Python
versions and interpreters or not.
Running our tests in each of the environments, configuring our test tool
of choice is possible with using tox.
tox acts as a frontend to Continuous Integration servers.
tox will come default along with DevStack package. But if you want to
manually install and configure it, install tox with pip install tox. Then put
basic information about the test environments you want to carry out
into a tox.ini file and follow the tox execution sequences as explained
earlier.
