Turbocharge Your Automation Framework to Shorten Regression Execution Time

T3

Test
Automation

5/5/16
9:45

Turbocharge
Your
Automation

Framework
to
Shorten
Regression

Execution
Time

Presented
by:

Asad
Faruqui
Moni Mau
Symantec

Brought
to
you
by:

350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073

888-268-8770 ···· 904-278-0524 --- info@techwell.com --- http://www.stareast.techwell.com/

Asad
Faruqui
Symantec

QA
director
for
trust
services
business
unit,
Asad
Faruqui
is
responsible
for

leading
the
quality
organization
for
market8 leading
SSL,
malware/vulnerability

scanning,
and
authentication
solutions
under
the
Symantec,
Thawte,
Geotrust,

and
Rapid
SSL
brands.
Prior
to
Symantec,
Asad
worked
at
Cisco
leading
their

routing
protocol
test
teams
under
their
central
OS
group.
In
all
his
roles,
Asad
has

focused
on
increasing
test
automation,
increasing
test
coverage,
and
measuring

QA
metrics
around
test
efficiency.

Moni Mau
Symantec
Moni Mau, a senior principal QA engineer for trust services business unit, focuses
mainly on designing and improving the test automation framework, integrating
automation with Jenkins to support continuous integration testing, and executing
performance testing. With more than fifteen years of software testing and
development experience, he has led his automation team in developing an
automation solution to significantly reduce regression time. Prior to Symantec, he
worked at VeriSign and IBM as a software developer.

Turbocharge Your Automation Framework
to Shorten Regression Execution Time
Asad Faruqui (Director QA)
Moni Mau (Sr. Principal QA)

About Us
• Part of Website Security BU at Symantec
• SSL product line: Verisign, Thawte, Geotrust, Rapid
• What we are responsible for:
– Web portal testing for customer life cycle management (e.g New, Renew, Replace, Revoke etc)
– Test Internal Applications for Customer Authentication
– Test Infrastructural Core Services (e.g OCSP, CRL, Seal)
– Test API’s (e.g Partner API’s, Encryption Everywhere)
– Test Conformance to standards (CA Browser Forum, EV guidelines)
– Performance/Scale testing
2
Copyright © 2016 Symantec Corporation

Agenda
3
1 Problem we’re solving for?
2 Selenium Grid Exploration
3 Our framework
4 High Level Design
5 How can this help you?
5 What’s next
6 Q &A

Problem we’re solving for?
• 17 total products for all brands
• Support different locales (13), and all browsers
• We have 100’s of testcases per product
• Testcases require end-to-end testing
– Execution time varies between 5-10 mins
• Let’s assume a release touching 5 products:
– 5 * 13 * 4 * 100 * 7 min ~ 126 days
• Challenge: Release products quicker with high quality!
4

Selenium Grid Exploration
• https://code.google.com/p/selenium/wiki/Grid2
• Each test case be executed parallel with different browsers
• Pros of using grid:
– Without grid, test suite with 50 test cases and each testcase takes 5 minutes to execute. Test for 3
main browsers (IE, FF, Chrome), then it takes 50 * 5 * 3 = 750 minutes
– Using grid, executing each test case in parallel for different browsers, it takes 50 * 5 = 250 minutes
• Even using grid, it takes more than 4 hours just to run 50 test cases!
• Can we shorten execution time more
5

Requirements for new framework
• Easy to be leveraged by existing automation;
– Existing automation will be running in a virtual image independently;
– Existing automation will be invoked remotely;
– Minimize code changes to existing automation. The required changes to existing automation should
be made only in configuration files;
• The test automation’s job must be executed from a central location;
• Provide ability to create multiple images to run automation in parallel;
• Provide ability to divide test cases from a test suite into small test groups and send to
run in the images;
• Provide a web portal to search and view test results
6

High Level Design
• Jenkins as central location to invoke tests
• Web2PY to build web portal application to search and view test results
• Build java application tool to make API calls to VMWare’s vCloud Director to create/delete
images;
• Build an automation engine with the following capabilities when it is invoked:
– divide the test cases in a test suite into small test groups;
– create test job for each test group;
– send test groups to be executed in parallel in the images that are just created;
– must have ability to queue up of the test jobs if there are not enough images;
– combine all the test results from different jobs to a single report;
– copy test report to a central test result location;
– save test results (test job/passed/failed) into database;
7

Central Image
Hub Image
Database Image
Node Image
Jenkins
Web Portal <Web2Py>
Automation
Test Code
Test Results
MySql
PhpMyAdmin
Controller
Automation
AgentStart
Hub Image
Controller
Node Image
Automation
AgentStart
Node Image
Automation
AgentStart
Node Image
Automation
AgentStart
Node Image
Automation
AgentStart
Node Image
Automation
AgentStart

Build Automation Source Code
9
Central
Image
Jenkins
Web Portal
<Web2Py>
Automation
Test Code
Test Results
Build script is invoked by a cron job twice a day (6 am & 6 pm)
 Check if there is any changed in automation test code.
• No changes – exit build script
• Yes, do following:
 Extract the test codes from source control
 Build automation test codes into a jar file
Invoke a build job from Jenkins if there is any changes in
automation test codes after cron job already ran

Test Jobs in Jenkins
10
Central
Image
Jenkins
Web Portal
<Web2Py>
Automation
Test Code
Test Results

Test Jobs Diagram
11
Product Folder
US AU FR ......DE CH SE ZS TW
Downstream Jobs

Web portal for searching results
12
Central Image
Jenkins
Web Portal
<Web2Py>
Automation
Test Code
Test Results
Date User Product Test Level
Build
Numbers
Passed Failed Results
02/13/2016 moni_mau demo regression 01_demo 100 2 View Results
02/14/2016 moni_mau demo regression 02_demo 100 0 View Results

Hub – Create Images (Nodes)
13
Hub Image Images
Creator
Controller
Test Jobs
Creator
Test Jobs
Processor
Test Result
Merger
Images
Remover
 Invoked remotely by Jenkins
 Take command line arguments
 controller [numbers of image, image
template, product, testbed url, locale, ,..]
 Invoke utility programs to execute the test
 Invoke by Controller as: CreateImage 10 templateName
 Call vCloud Director API to create images
 Succeeded: returns a list of newly created images’ IPs
 Failed: returns errors to controller

Hub – Create Test Jobs
14
Hub Image Images
Creator
Controller
Test Jobs
Creator
Test Jobs
Processor
Test Result
Merger
Images
Remover
Invoke by Controller as: CreateJobs product testBedUrl locale data
1. Update automation config file with new testbed url and locale
2. Read test class name and number of test cases to be divided in each test class
3. Access data directory (xml, excel) or database to divide test cases to small groups
4. Create testng.xml files for each test group
5. Create folders to contain: config file, testng file, test data file.
6. Zip test folders and name each zip folder with a sequential number as test job
Test Class Name Execute Number
DemoClass1 Yes 10
DemoClass2 Yes 5
TC# Desc Execute usr pwd Expect
1..10 a yes aa $a1 Passed
11..20 b yes bb Failed
21..30 c yes cc $c1 Passed
31..40 d yes dd ??? Failed
TC# Desc Execute Enroll Renew Expect
1..5 a yes data data Passed
6..10 b yes data data Passed
11..15 c yes data data Passed
16..20 d yes data data Failed
21..25 e yes data data Passed
26..30 f yes data data Passed
<classes>
<class name="demo.DemoClass[1/2]"/>
</classes>
result/send/
Job1
Job10
Job..
config.txt
estng.xml
data.[xls/
sql/xml]
Job

Hub – Process Test Jobs
15
Hub Image Images
Creator
Controller
Test Jobs
Creator
Test Jobs
Processor
Test Result
Merger
Images
Remover
 Create an array of Nodes’ IP and an array of Ports (start with 6001) assigned to Nodes
 Connect to Nodes using (nodeIP, 65535) and send command to start Agent in Nodes
 Create an array of thread jobs: Thread[] jobs = new Thread[N] where N is number of Nodes
that Controller in Hub can communicate with Nodes
 Create a stack of jobIdStack and put “result/send/job[n].zip” to this stack
Send jobs to execute:
while (! jobIdStack .isEmpty()) {
for(int i = 0; i < N; i++) {
if(! jobIdStack .isEmpty() && jobs[i].getState()==Thread.State.TERMINATED) {
jobs[i] = new Thread(new ManageJob(nodeIP[i], portNum[i], jobIdStack .pop(), product));
jobs[c].start();
}
}
Wait for all the job to finish:
while(jFinished < N) {
if(jobs[jFinished].getState()==Thread.State.TERMINATED) { jFinished ++; waitTime=0;
}else{
if(waitTime > WAIT_TIME){ KSFinished++; waitTime=0;continue;}
Thread.sleep(10000); waitTime++;
}
}
Communicating between ManageJob process and Agent process:
ServerSocket(port)
Write data
Socket(hubIP, port)
Read data
ServerSocket(port)
Write data
Socket(nodeIP, port)
Read data

Hub – Merge Test Results
16
Hub Image Images
Creator
Controller
Test Jobs
Creator
Test Jobs
Processor
Test Result
Merger
Images
Remover
 Invoked by Controller as: reportMerger resultDir
 Test result directory locates at: results/receive/job[N]
 Final report locates at: results/jenkinsJob
 Invoked by Controller as: deleteImage imageNames

Node – Client Image
17
Node
Image
Agent
Start <script>
 Node Image is created by createImage from a template image
 After the image is created and started successfully, the start script is
automatically invoked to do the following:
• Start Agent
• Connect to Central Image, transfer to update automation jar file.
Automation
 Invoked by start script
 Start running with configurable port (ex: default 65535)
 Controller in Hub connects to Node via above port to start agentProc
process
 Start processing job sending from Controller by agentProc
 Invoking automation and wait for automaton to finish
 Send results back to Hub
 Wait for next job to process

Results Achieved
• We told you that if we do serial execution it would take: ~ 126 days
• Now running same automation with 100 VM images: 31 hours ~1.26 days!!
18

How this will help you?
• Have an idea how to divide test cases in each test class into small groups
• Have better idea of how the test data and config files are built into jobs
• Sample code to build the core engine of this framework
• We’re moving to open source report, called ExtentReport
http://extentreports.relevantcodes.com/java/version2/docs.html
– Has merge report tool which can be used with
framework: http://extentreports.relevantcodes.com/extentmerge/docs.html
– This report also allows us to save the report in DB http://extentx.herokuapp.com/
• If you don’t have vCloud Director, you can use Docker
• Freshly created images from templates: helps reduce test enrollment issues due to old config from previous
executions
• Extend this framework usage beyond Regression to Build Verification Test and Feature Testing and more!
19

Thank you!
Copyright © 2014 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be
trademarks of their respective owners.
This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by
law. The information in this document is subject to change without notice.
Asad Faruqui/Moni Mau
asad_faruqui@symantec.com/moni_mau@symantec.com
21

Turbocharge Your Automation Framework to Shorten Regression Execution Time

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Turbocharge Your Automation Framework to Shorten Regression Execution Time

Similar to Turbocharge Your Automation Framework to Shorten Regression Execution Time (20)

More from Josiah Renaudin

More from Josiah Renaudin (20)

Recently uploaded

Recently uploaded (20)

Turbocharge Your Automation Framework to Shorten Regression Execution Time