The document outlines the QA strategy for Apache CloudStack releases. It discusses the goals of continuous integration and validation through automated testing. The QA process includes installation and upgrade validation, new feature testing, regression testing, performance and scalability testing, and security reviews. Future plans include increasing test automation, implementing continuous integration, and standardizing certifications. Community participation is encouraged in QA meetings, IRC sessions, and the developer mailing list.

1. ASF CloudStack – QA Strategy
Apache Cloudstack Collaboration Conference
Dec 1, 2012
Sudha R Ponnaganti,
Sr Manager, SQA Engineering, Citrix Systems Inc.,
Email:sudha.ponnaganti@citrix.com

2. QA Strategy - Introduction
 Need for QA strategy
 Goals for current presentation
 Transparency
 Expected outcome
 Community participation
 Superior Quality Releases
 Expected release cycles

3. Agenda
 Releases and QA Overview
 QA strategy
 Future
 Community participation
 Q & A and Feedback

4. Releases and QA Overview

5. Releases and QA overview
QA Activity Major Releases ASF CS Minor Releases ASF CS
Installation and Upgrade validation    
New feature validation  
Regression validation    Automated
Integration validation 
Performance and scalability testing 
Solution validation 
Security review  
Recovery validation 
Usability validation 
L10N validation 
Defect validation    
Documentation review    RN only

6. Continuous Release Cycles
Currently we are doing minor and major releases
 Cycles are longer
 Time based vs. feature based release discussion
There is a need for
 Continuous integration
 Continuous validation

7. Continuous Integration
 Every check-in gets tested as and when it is submitted
 Builds upfront quality faster
 Should take less than 5 minutes to complete
Below is the process that is being worked up on currently
Jenkins kicks off
a job after each
checkin
That job spins up
a devcloud
appliance
devcloud checks
out the commit
that kicked off
the job;
devcloud builds
cloudstack
devcloud adds
itself as a host
devcloud runs
the smoke test

8. Continuous Validation
 Run all the tests on nightly builds
 With current structure, below are the sequence of tests
that run nightly
RAT (
release
audit tool)
Compile +
junit
Docs +
apidocs
Package
building
Install tests Smoke tests
Regression
Tests

9. QA Strategy

10. Installation and Upgrade Validation
 Compatibility Matrix
 Manual Validation
Priority Type TypeofDist Base UpgradeV OS Networking NetworkDevices Storage GuestVMs Browsers FeaturestoCover
Xen KVM Vmware OVM BareMetal
P1 OpenSource ASFCS4.1 CentOS6.3 6.0.2+HF,6.1 RHEL6.2 5.1 2.2.2 N/A Advanced/Basic
VR,F5,SRX,
NetscalerVPX ISCSI,NFS,Local
Template: CentOS5.6PV64bit,
CentOS5.5PV64bit, IE8,FF10,Chrome,SafariBVTs
P1 ASFCS4.1 Ubuntu12.04 6.0.2+HF,6.1 RHEL6.2 5.1 2.2.2 N/A Advanced/Basic
VR,F5,SRX,
NetscalerVPX ISCSI,NFS,Local
Template:OVMSystem.img
ISO: OVMSystem.img IE8,FF10,Chrome,SafariBVTs
Multinode install
P1 Multiple MS ASFCS4.1 CentOS6.3 6.0.2+HF,6.1 RHEL6.2 5.1 2.2.2 N/A Advanced/Basic
VR,F5,SRX,
NetscalerVPX
(NS10,build54.7.nc) ISCSI,NFS,Local
Template: CentOS5.6PV64bit,
CentOS5.5PV64bit,
Windows2008R264bit, IE8,FF10,Chrome,SafariBVTS
P1 ASFCS 4.0 ASFCS4.1 CentOS6.3 6.0.2+HF,6.1 RHEL6.2 5.1 2.2.2 N/A Advanced/Basic
VR,F5,SRX,
NetscalerVPX ISCSI,NFS,Local
Template: CentOS5.6PV64bit,
CentOS5.5PV64bit, IE8,FF10,Chrome,SafariBVTs
P1 ASFCS 4.0 ASFCS4.1 Ubuntu12.04 6.0.2+HF,6.1 RHEL6.2 5.1 2.2.2 N/A Advanced/Basic
VR,F5,SRX,
NetscalerVPX ISCSI,NFS,Local
Template: CentOS5.6PV64bit,
CentOS5.5PV64bit, IE8,FF10,Chrome,SafariBVTs
Compatibility MatrixASF4.1- Supported VsTested
Upgrade
Fresh
Hypervisors

11. Installation and Upgrade Validation
Install and Deployment validation Specifics:
 Install and deployment Validation need to be done in
context of release artifacts to be in alignment with
voting process.
 Step by step instructions are posted here.
 https://cwiki.apache.org/CLOUDSTACK/cloudstack-40-
test-procedure.html

12. New Feature and Regression Validation
New Feature Validation Regression Validation
• QA team writes test cases for new
features based on FS.
• QA posts test plans to community to get
feedback
• QA automates and executes new test
cases for current release. Adds TCs to
regression for future releases
• We can provide template if you need to
add test cases.
• You can find test suites on wiki
• There are around 3000 + test cases in
the regression suite currently.
• This suite includes Automated and
Manual tests
• QA reviews overall feature list and
identifies regression coverage based on
impact for each release – not sustainable
• To be successful in open source env,
need to have 100% automated coverage

13. New Features and Regression Validation
 Various Frameworks are available based on Junit, Python,
Selenium to automate test cases
 A combination of all of these provide comprehensive
coverage
Unit Tests BVT Tests Regression Suite
Other Integration
Suites ( EC2 and
S3)
UI Automation

14. New Features and Regression Validation
• Unit Test framework ( JUnit )
• Provides early feedback and fixes issues close to source
• How to Write Unit tests - https://cwiki.apache.org/CLOUDSTACK/unit-
testing-101.html
• Currently all unit tests running with 100% pass rate
• Runs on every build
• Code coverage results are posted here

15. New feature and Regression Validation
 BVTs and Functional Automation - Marvin
 600+ test cases running on multiple configurations and covering
multiple hypervisors
 How to write functional automated TC :
https://cwiki.apache.org/CLOUDSTACK/testing-with-python.html
BVT
Hypervisor Configuration
Xen ( multiple versions) Advanced, Basic
VM ware Advanced, Basic
KVM Advanced, Basic
OVM Planning
Functional Regression
Hypervisor Configuration
Xen ( multiple versions)
Advanced, Advanced - Netscaler, EIP/ELB Rules, Basic with and
without SG
VM ware Advanced, Advanced - Netscaler, EIP/ELB Rules, Basic w/o SG
KVM
Advanced, Advanced - Netscaler, EIP/ELB Rules, Basic with and
w/o SG
OVM Planning

16. New features and Regression Validation
 Multiple test pods are being setup to run suites in parallel
– around 15 test pods are required
 Porting to ASF CS env has been in progress. BVTs are
done.
 Prasanna and Edison are covering more detailed talk on
Automation on Sunday morning.

17. New Features and Regression Validation
 Integration frameworks
 Currently we are in the process of submitting test suite that
covers EC2 Query API validation – JCloud framework
 Around 40 APIs are automated
 In the process of covering all APIs
 Failed test cases are being fixed
 Need to automate complex scenarios.
 Details on test implementation are posted below
 https://cwiki.apache.org/confluence/display/CLOUDSTACK/EC2+Qu
ery+API+Test+Suite+Automation

18. New Features and Regression Validation
UI Automation
 Selenium
 Faster and Easy
 Compatible with python backend
 Plan to Deliver Smoke Tests by – Early FY 13
 POC and initial submits – coming soon

19. Performance and Scalability Testing
 Native Tool - Simulator
 Simulator - mock hypervisor and will react to
every command sent to it by the MS similar to a
regular hypervisor - easy to setup and run
 Simulator and Perf resources:
http://wiki.cloudstack.org/pages/viewpage.action?pageId=1344953
 2 papers have been published Covering
 Basic Zone
 Advanced Zone

20. Performance Use cases
• Time taken to propagate security group rules in basic zone
• Deploy VM time, CPU, load averages and # of DB connections
• Steady state CPU, load average and # of DB Connections
• Restart MS measures: Time to stop MS, Start MS and rebalance
with 30K Hosts across 4 MS (agent load size to 500 and 1000)
• Restart MS measures : with Host in maintenance mode
• Bandwidth usage collection frequency for routers
• Router Status poll metrics in RVR env

21. Performance Testing
What Next
 Holistic Coverage
 Router
 Storage
 Network
 Realistic Scale Testing

22. Security Validation
 Security validation has been important aspect of ASF CS releases
 All FS should include security considerations so that QA can identify
and cover these areas
 Security Coverage done included the following along with manual
validation and other security reviews

23. Solution Validation
o Current solution Validation use cases include
o reference architecture deployment and
o additional complex deployment scenarios.
o Test plans are posted on wiki
o There is no need to adopt these right away in to Cloudstack but will
monitor if there is a need in future

24. Recovery, Usability and L10N
 Recovery scenarios, Usability Reviews/ L10N - Not prioritized
 Jessica Wang has been doing L10N review along with few other
contributors
 If you have expertise in these areas, do participate in QA forums
 Documentation review
 Focused validation at the end of the cycles
 Watch for QA requests for review

25. Defect Triage and Validation
 You can participate by testing product logging defects here (
ASF JIRA)
 Always log defects when you start discussion on mailing lists
 Add automation for defects that you validate. Tests can be
checked in to appropriate suites based on type of test –
unit/smoke/regression suites
 Participate in Bug fest activities

26. Future

27. Future - Status
Some improvements that QA has been doing Q3/ Q4 FY ‘12
 Automation 
 Functional – BVTs and Regression (Python framework) - goal
700+ TCs
 Unit Test Framework ( Junit ) – Add TCs for ongoing releases +
improve legacy by adding TC for patches
 Integration automation frameworks (ec2 and s3) –
implementation and make it part of regression suite ( Jcloud
Test framework)

28. Future - status
• TCM 
Implementation and loading test cases –
Address Test Case gaps ( total inventory) – Currently 3000 test
cases exist. Goal is to get to 5000
• Performance 
White paper – Use cases collected and release V3 with known use
cases – backlog has been built.
• Tools 
Static code analyzer – In progress
Code coverage – Cobertura
Code inspection -

29. Future - status
 Infrastructure improvements 
 Test Env improvements ( Fresh Install Automation for Functional testing)
 Acquire external devices and standardize test env
 Build infrastructure improvements
 Continuous integration implementation 
 Continuous validation 
 Standardized certifications 
 New versions of HV, NS and other integrations
 Standardized templates 
 Test plan/Test case / Defect writing Guidelines, Test case execution
reports, baseline metrics

30. Future - status
Initiatives for FY ‘13
 Move towards enterprise quality standards 
 Unit Test Coverage + Automation – 70% + Code coverage
 Regression coverage – 90% + coverage
 New features – 100% coverage
 Defects regressed through automation
 Install/Upgrade Automation
 UI automation - selenium
 Standardize QA coverage for enterprise readiness
 Regression
 New functionality
 Automation
 Install/upgrade ( compatibility matrix automation)
 Usability
 Performance ( tools and use case expansion)
 Reliability ( longevity under load)
 L10N ( internal or vendor based)

31. Future - status
 508 ( Accessibility)
 Integration
 Security
 Real world use cases
 Install Upgrade automation 
 Infrastructure improvements to run automation across
multiple configurations 

32. Community Participation

33. Community Participation
 Scheduled QA Meetings on weekly basis – Will be publishing meeting details for
CS 4.1 and expect it to be starting soon. Everyone is welcomed to join and
participate in QA Activities for ASF CS 4.1 release
 David has planned IRC Sessions to drive Quality as a continuous process. Please do
participate and help to drive this continuous test process - timings 1800 UTC in
#cloudstack-dev on Wednesdays
 ASF Cloudstack Dev mailing list is great way to share your ideas and contributions
 cloudstack-dev@incubator.apache.org
 All Dev, Build and QA documents are posted here
 Defects can be logged here

34. Q & A

35. ASF CloudStack – QA Strategy
Apache Cloudstack Collaboration Conference
Dec 1, 2012
Sudha R Ponnaganti,
Sr Manager, SQA Engineering, Citrix Systems Inc.,
Email:sudha.ponnaganti@citrix.com