Vijayan Reddy discusses building a scalable test infrastructure by integrating various testing tools and tasks. He recommends taking a blue-print approach and using available commercial and open source tools. A meta controller can help orchestrate test execution across platforms and provide metrics collection, defect analysis tracing, and intelligent reporting. Building interfaces between source control, test case management, bug tracking, and test results can help scale the infrastructure. Automating tasks like builds, BVT, and reporting can improve efficiency.

1. Vijayan Reddy
Engineering Manager
Adobe Systems

2. Why Test Infrastructure
How much of your test project Projected Time slices
is manual efforts
How much of your test project
is test execution
Manual Testing
Automated Testing
Manual Testing
Test Setup
Automated Testing
Reporting
Test Planning
Where are we missing? How can we save?

3. In Scope
 Approach from a typical test
lifecycle – End to End tasks
for shipping a project
 A blue-print for scalable test
infrastructure
 Giving tips on cross-linking of
the tools
This is no Rocket Science

4. Test Infrastructure
 All Material Used Tools,
Methods, Scripts
 Platform that hosts &
integrates tools and scripts
 Commercial tools don’t offer
complete support of all tasks
An interface/glue that connects every single task and
provides plug-ability for new tasks

5. How To Build
 Available off-the shelf? Probably not
 Buy/Build/Customize is the mantra
 great commercial software
 great open source frameworks
 great engineers in your team 

6. Top Opportunities
 Test Execution – Biggest
opportunity
 Build Systems
 BVT
 PIT
 Defect Analysis
 Linked Test Cases creation &
Maintenance
 Test Environments
Maintenance
 Setup / clean up after
executions
 Test Reporting / Archiving /
Data Mining

7. Scale with Meta Controller
Test
Repository
Test Test
Execution Archive
Controller
Metrics Test
Collection Reporting
PIT BVT

8. Meta Controller - Requirements
Able to work p2p, or client – server
Able to work across platforms
Able to invoke, monitor tasks, Remotely – Trigger Execution, eg.
Able to transfer files
Builds, Installers
Test Results, Logs, Thread dumps
Able to parallelize across nodes
http://staf.sourceforge.net

9. Optimization @ Test Setup
 Time on Test Setup, Test Clean up
 Clean Systems
 Imaging / ghosting can help
 Across different platforms?
 Build the software stack -
○ Eg: Application Server, Java/.NET run time
○ Database
○ Deployment of your applications
○ Deployment of test suites & Setup
 Multiply savings by the number of builds, and
Machines

10. Meta Controller
 Control the test automation framework(s)
 Glue the controller and tool(s), Script(s)
 APIs exposed
 Command line options / Batch files
 Parameterize the launch:
○ Platform / machine agnostic scripting
○ specific details of each test machine abstracted in
configuration files
 Enable Metrics Collection – Code Coverage,
Profiling

11. Scale with Helpful Interfacing
 Test cases managed under a system – (Manual &
Automated tests) (Eg. http://testlink.sourceforge.net)
 Why do we need a system, not DOC, PDF, HTML
 To assure test coverage / traceability
 Cross linking with Feature Tracking
 Cross linking with Bug tracker
 Cross linking with Source Control
 Help track estimates as well, against test cases/cycles
 Interface SCM and Bug Tracker

12. Scale in Defect Analysis: Facilitators
Traceability Linking
 Feature -> Test Case ->
Changed Files, versions ->
Bug -> dependent bugs

13. SCM – Bug tracker Linking
 One – spot update of bugs & SCM
 When fix is checked in
 Use a template
 Fill description of fix
 Fill ‘areas affected’
 Fill ‘Recommended tests’
 Auto - Update Bug tracker
 ‘Areas Affected’
 ‘Files Affected’ and versions of these files
 Useful in Regression testing & Injections check
 Search up other bugs which had fix in same files and
retest.

14. Scale with Early Warnings
Static Code Analyzers
 Catch bugs even before
the compilation is done, let
alone built and tested 
 Standard Code violations,
Uncaught exceptions, Un-
freed memory etc.
 Code Security audits for
script injections

15. Automated Builds
 Check-in Triggered
Builds, Scheduled Builds
 Build System to access
builds, logs, changed
files, changed versions
 Promotion to QA on
Success, Auto Deploy
 Tagging / Merging on
Success on SCM

16. Scale with BVT
Catch the Bugs early with BVT
Automate triggering of BVT on
builds
Use your Automated Sanity /
Acceptance Tests
Link with Build Setup to ‘Bless’
builds for QA promotion –
Saves QA cycles
Automate failure mails – Send
with exact test log, pick the
code changes from the build
Auto-log a bug

17. Pre Integration Test Setup
PIT saves pitfalls 
 Huge development teams, central builds
 Check-in to a PIT branch (that is Sync with Trunk)
 Kick off
 Auto – Build
 BVT
 If Success, auto-Merge change into Trunk
 If not, follow BVT failure handling and auto- revert from
PIT Branch

18. Infrastructure for Reports
 Common results repository
 Different test cases / suites
/ tools output differently
Manual
 Build interfaces to import, Automated
tool
Test
Results
export
Test Scripts
 Define common results
format
 Preferable format
Common Results
 Database
 XML

19. How to Export
 Commercial tools export in XMLs
 If that helps, standardize other tools / scripts for
that
 Use XSLT to transform to common format
 Define Performance results schema

20. Scale through intelligent reporting
 Easier access for data mining
 Easier reporting, trends along builds, along
features, along platforms, locales
 Drill down ability on any individual test case, and
see history along builds to isolate injections
 Easier collation of all test results
 Charting for Non functional parameters
improvements
 Cross Link to Feature / Test Case tracker

21. Take Away
 Identify these bridges that
work for you – No “one
size fits All theory ”
 Create an integrated
platform
 Build
 Buy
 Customize
 Available at
http://qualinfra.blogspot.com
Vijayan.reddy@gmail.com

22. Questions