Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

OpenStack Tempest and REST API testing

2,372 views

Published on

by Shashidhar Soppin, WIPRO

Published in: Technology

OpenStack Tempest and REST API testing

  1. 1. OpenStack Tempest and REST API Testing
  2. 2. Agenda One • Introduction • OpenStack & DevStack Two • Tempest & Architecture • DevStack config and H/w required Three • Various tests & Test Results • Key Findings Four • Summary & Conclusion • Tech-Scenarios & Citations
  3. 3.  To demonstrate starting/running of OpenStack services  Specifically for development work on OpenStack  We can download the latest version of DevStack from ”git” repo.  Cloud operating system/platform that controls large pools of compute, storage, and networking resources  Used for developing private clouds • Keystone • Cinder • Glance • Neutron • Horizon • Nova OpenStack & DevStack OpenStack DevStack
  4. 4. OpenStack Testing ?? OpenStack integration Test Suite • Tempest Test Suite • Ready to deploy Unit Tests in each project • Nova, Cinder, Swift • Neutron etc • OpenStack Code base is written in Python • So how to use/write tests? • Isolate the working Environment • Quick testing • Automated configuration • Ease of use
  5. 5. Tempest • Tempest is a set of integration tests • To be run against any live OpenStack/ Dev Stack cluster F e a t u r e s Official OpenStack Integration Test Suite Interacts with majority ReSTful API’s Around 3000+ tests covering all projects Used as gating tests for all new projects Runs on all OpenStack releases Self Testing/Self Cleaning Uses a unified tempest REST client Latest version of “Tempest 12.1.1.dev37” A t t r i b u t e s Scenario based Tests CLI Tests Stress tests Ensure that OpenStack cloud works with the OpenStack API
  6. 6. Basic Tempest Reference Architecture OpenStack/ DevStack deployment Service Clients Python Clients Boto Tests Rest Client API Tests Scenario Tests Third party tests CLI based Tests Library CLI OpenStack ReST API’s EC2 API’s OpenStack ReST API’s 1
  7. 7. Basic Tempest Reference Architecture Compute Node Compute Node Controller Node Test Server (Tempest Node) HTTP Request HTTP Response OpenStack configuration {Tempest tests executed from controller node} 2
  8. 8. Tempest in a “Nutshell” 1
  9. 9. Tempest in a “Nutshell” 2
  10. 10. Hardware configuration required and OS used 1
  11. 11. Hardware configuration required and OS used 2
  12. 12. Tempest Execution….steps
  13. 13. Tempest Execution….steps
  14. 14. Tempest Execution….steps
  15. 15. Tempest Execution….steps
  16. 16. Tempest Execution….steps
  17. 17. Tempest Execution….steps
  18. 18. Tempest Execution….steps
  19. 19. Tempest Execution….steps
  20. 20. Test Results-Kilo 968 724 962 22 76 13 141 126 141 1131 926 1116 0 200 400 600 800 1000 1200 CentOS Ubuntu Fedora toxefull Passed Failed Skipped Total TC
  21. 21. Test Results-Liberty 999 1023 944 10 5 24 136 136 138 1145 1164 1106 0 200 400 600 800 1000 1200 CentOS Ubuntu Fedora toxefull Passed Failed skipped Total TC
  22. 22. Test Results-Kilo 41 32 43 3 9 1 39 38 39 83 79 83 0 10 20 30 40 50 60 70 80 90 CentOS Ubuntu Fedora toxesmoke Passed Failed Skipped Total TC
  23. 23. Test Results-Liberty 44 45 43 1 0 2 39 39 39 83 84 84 0 10 20 30 40 50 60 70 80 90 CentOS Ubuntu Fedora toxesmoke Passed Failed skipped Total TC
  24. 24. Test Results-Kilo 867 700 1015 55 102 15 102 96 107 1024 898 1137 0 200 400 600 800 1000 1200 CentOS Ubuntu Fedora run_tempest Passed Failed Skipped Total TC
  25. 25. Test Results-Liberty 894 955 984 51 26 15 138 139 139 1024 898 1137 0 200 400 600 800 1000 1200 CentOS Ubuntu Fedora run_tempest Passed Failed skipped Total TC
  26. 26. Key Findings-Failed Tests analysis OS/Tests toxefull toxesmoke run_tempest CentOS 7.0 Most of the Tempest test case failures are • Volume API’s • Third party boto (this is related to Elastic Compute (EC2) and Object storage-S3 features of AWS) ReST API failures. Most of the Tempest test case failures are • Volume API’s related Most of the Tempest test case failures are • Scenario tests, snapshot pattern related, Volume API’s and third party boto(this is related to Elastic Compute (EC2) and Object storage-S3 features of AWS) ReST API failures. Ubuntu 14.04 Most of the Tempest test case failures are • Compute, scenario and Cinder Volume related • Third party boto (this is related to Elastic Compute (EC2) and Object storage-S3 features of AWS) ReST API failures. Most of the Tempest test case failures are • Compute ,scenario and Volume Rest API failures. Most of the Tempest test case failures are • Compute, scenario, Volume and third party boto (this is related to Elastic Compute (EC2) and Object storage-S3 features of AWS) ReST API failures. Fedora 21 Most of the Tempest test case failures are • Volume API’s • Third party boto (this is related to Elastic Compute (EC2) and Object storage-S3 features of AWS) ReST API failures. • Only one of the test case failed here and is related to Volume ReST API failure. Most of the Tempest test case failures are • Volume API’s and third party boto (this is related to Elastic Compute (EC2) and Object storage-S3 features of AWS) ReST API failures. Kilo
  27. 27. Key Findings-Failed Tests analysis OS/Tests toxefull toxesmoke run_tempest CentOS 7.0 Most of the Tempest test case failures are • Volume API’s • Third party boto (this is related to Elastic Compute (EC2) and Object storage-S3 features of AWS) • Server multi-node ReST API failures. Tempest test case failure is related to following • Volume API’s Most of the Tempest test case failures are • Compute, Volume API’s, basic-Ops test server, multi-node • third party boto(this is related to Elastic Compute (EC2) and Object storage-S3 features of AWS) ReST API failures. Ubuntu 14.04 Most of the Tempest test case failures are • third party boto” (this is related to Elastic Compute (EC2) and Object storage-S3 features of AWS) ReST API failures. No failures found. Most of the Tempest test case failures are • Compute, scenario, Volume and third party boto (this is related to Elastic Compute (EC2) and Object storage-S3 features of AWS) ReST API failures. Fedora 21 Most of the Tempest test case failures are • scenario tests, compute, snapshot deleting, Volume API’s and third party boto (this is related to Elastic Compute (EC2) and Object storage-S3 features of AWS) ReST API failures. Found two failures which are • Volume ReST API failure. Most of the Tempest test case failures are • Volume API’s and third party boto (this is related to Elastic Compute (EC2) and Object storage-S3 features of AWS) ReST API failures. Liberty
  28. 28. Key Findings-Skipped Tests analysis Some of the major reasons for “Skipped” tests are,  Pending bugs need to be resolved  Live block migration not supported  Instance validation and multiple instance support not available  SSH & VNC support not available in some cases  Some of the features not included in Dev Stack  Ephemeral disk verification could not be done  Multiple policy feature not supported  Instance validation and multiple instance support not available Note:- These findings are across OpenStack releases and OS flavors Liberty Kilo
  29. 29. Key Findings-Summary KILO LIBERTY toxefull toxefull Passed Failed Skipped Total TC Passed Failed skipped Total TC Centos 968 22 141 1131 CentOS 999 10 136 1145 Ubuntu 724 76 126 926 Ubuntu 1023 5 136 1164 Fedora 962 13 141 1116 Fedora 944 24 138 1106 toxesmoke toxesmoke Passed Failed Skipped Total TC Passed Failed skipped Total TC Centos 41 3 39 83 CentOS 44 1 39 83 Ubuntu 32 9 38 79 Ubuntu 45 0 39 84 Fedora 43 1 39 83 Fedora 43 2 39 84 run_tempest run_tempest Passed Failed Skipped Total TC Passed Failed skipped Total TC Centos 867 55 102 1024 CentOS 894 51 138 1024 Ubuntu 700 102 96 898 Ubuntu 955 26 139 898 Fedora 1015 15 107 1137 Fedora 984 15 139 1137
  30. 30. Conclusion • Tempest test results across different OpenStack releases and OS flavors is distinct • One of the primary reasons of this ambiguity is each OS flavor having different kernel versions. • The more latest the Kernel version used, the more stable are the test results • The other reason is because of different OpenStack releases on different hardware platforms • Tempest execution on Liberty release on various Linux OS flavors is yielding more stable test results as compared to Kilo release. • The more latest the OpenStack version, the more stable are the test results. • These test results can be used for future reference (based on h/w architecture used)
  31. 31. Troubleshooting….. 1. We have observed some times that when “localrc” file from root user and try to run stack.sh command from the stack user, following error will be encountered. Solution:- When such error is encountered, change the ownership of devstack directory to stack user using the following command
  32. 32. Troubleshooting….. 2. If you are using proxy, While running stack.sh command you may get the following error Solution:- Best practice is to have internet connection without proxy, which will avoid majority of the issues. 3. When we clone “nova” manually to /opt/stack/nova from git repository and try to execute the command “stack.sh”, following exception will be encountered. Solution:- To avoid the above exception, copy .git directory from cloned devstack directory to /opt/stack/nova directory. This will help us to overcome the above mentioned exception.
  33. 33. Tech-Scenarios
  34. 34. Tech-Scenario-I Validation of OpenStack workload management tool Context  Validation and functionality test of OpenStack & management tool together was required  Validation of RESTful API’s for supported OpenStack releases was required  Certification testing on different OS platforms supporting OpenStack was required. Solution/approach used  Used Tempest test suite (along with some customization where required) for the validating stability of ReSTful API’s  Using Tempest test suite and some home grown test cases, we were able to verify the management tool and various OpenStack features  Triaging with multiple combinatorial tests and identifying gaps and helping respective customer to fix the issues on time. Outcomes & Learning  Since OpenStack was niche technology, understanding the requirement clearly was a challenge  State of the art certification of RHEL and Ubuntu on OpenStack was a challenge.  Support for Continuous OpenStack testing for supported releases was a challenge Technical Challenges Technology Ubuntu, SuSE, RHEL, Cent OS , OpenStack releases Havana, Icehouse and Juno Test Suite used Tempest  Validation of complete workload management tool on OpenStack with various test scenarios  Validating Nova, Keystone, AMQP message bus protocol and Horizon interfaces.  Evaluation of the management tool for robustness and stability of usage.  Triaging issues that are related to OpenStack product  Certification of various OpenStack releases done for RHEL and Ubuntu.  Ready test suite for future releases of OpenStack was evolved
  35. 35. Tech-Scenario-II: Validation of OpenStack API’s using Tempest suite Context  Complete coverage of OpenStack API testing required  Reliability and Serviceability of OpenStack on a specified hardware configuration was required Solution/approach used  Validation of complete ReST API suite for various OpenStack releases and for specified h/w platforms.  Complete test failure analyses with log reporting on time.  Parallel Tempest configurations kick started with test executions Outcomes & Learning  Running Tempest test suite on multiple Linux platforms  Debugging and finding failures and causes  Rally configuration and execution Technical Challenges Technology Ubuntu, RHEL, Cent OS , OpenStack releases Juno and Kilo, Test Suite used Tempest Tools Rally  Tempest configuration on multiple platforms and OpenStack releases.  Validated complete basic OpenStack API suite and reported failures  Suggested fixes where possible
  36. 36. Citations http://docs.openstack.org/ https://github.com/openstack/ http://docs.openstack.org/developer/tempest/field_guide/ https://wiki.openstack.org https://wiki.openstack.org/wiki/devstack http://docs.openstack.org/developer/devstack/overview.html http://devstack.org http://devstack.org/faq.html https://github.com/openstack-dev/devstack http://docs.openstack.org/developer/tempest/configuration.html
  37. 37. Thank You Shashidhar Soppin Senior Architect Reachable @ “shashidhar.soppin@wipro.com”
  38. 38. Back-up Slides
  39. 39. What is Tox? tox is a generic virtual environment management and test command line tool that we can use for the following purposes.  We can check our package installed correctly with different Python versions and interpreters or not.  Running our tests in each of the environments, configuring our test tool of choice is possible with using tox.  tox acts as a frontend to Continuous Integration servers. tox will come default along with DevStack package. But if you want to manually install and configure it, install tox with pip install tox. Then put basic information about the test environments you want to carry out into a tox.ini file and follow the tox execution sequences as explained earlier.
  40. 40. How to download latest Devstack?

×