Dolphin is a highly available, reliable and scalable cloud based testing infrastructure. Dolphin, conceived in 2010, delivered its promised results by providing regression running capabilities with record-breaking performance. The entire automated test suite for Dragon 1.0 formal system testing was executed in a matter of few days using Dolphin. Simplicity and efficiency are among top focuses of Dolphin. Dolphin simplifies testers' work by providing them with a well configured and ready to use virtual test station. The burden of managing and configuring test station and test execution has been shifted from each individual tester to Dolphin. As to Dolphin, the service provider itself, there are many unique features specifically designed for its robustness and efficiency. We will focus on revealing those features in details in this presentation.

1. Inside Dolphin
Test Infrastructure for Dragon-Jaguar
Tao Jiang
http://www.linkedin.com/in/taojiang2000/

2. Outline
• Test Automation and Execution
• Dolphin as a Test Execution Infrastructure
• Results of using Dolphin in Dragon-Jaguar 1.0
2

3. Challenges in Automated Test
Execution
• Lots of automated tests (1,169 in Dragon-Jaguar 1.0)
• Each test may require a unique set of resources:
– Hardware devices and equipment
– Supporting software and test libraries
– Test environment
• Want to run tests non stop, unattended
• Run tests on distributed test stations in parallel
• Dealing with failure
– Hardware/OS Failure
– Misconfiguration in Software / Libraries
– Misbehaving Test
3

4. Dolphin: Testing Infrastructure
• Automatic scheduling and distribution of tests
• Automatic configuration of test resources:
• Hardware setup
• Software and test library configuration
• Deploy and configure test environment
• Provide a clean slate to run each test
• Fault-tolerance
• Run-time Scalability
• Status and Monitoring
• Low Maintenance
4

5. Dolphin and fortyTwo Collaboration
5

6. Dolphin Rack in Cluster
6

7. Test Station Layout
7

8. Dolphin Cluster
• Each cluster has one controller as master node
• Runs controller service program for job scheduling
• File server for storing test scripts and test logs
• Master location for all test libraries
• Multiple test stations acting as worker nodes
• a “Dolphinized” test station
• Running with Dolphin Agent program
• Has fixed set of HW, including at least:
• One type of communicator
• Single PG board / PG Simulator
• Controller and Workers are talking via RESTful web
service
8

9. Work Order Scheduling by Controller
• Periodic check each worker for HW/SW info
– Assign work order only to worker in healthy condition
• Assign work order to worker by matching HW/SW
– Worker Info is based on last scanned result
– Worker double checks HW/SW matching before execution
– Scheduling based on stale info will be returned for re-execution
• Redeploy test environment as needed
• Scheduling goal: keep every worker busy
• Result: Dramatic shortening job completion time
9

10. Dealing with failure
Worker :
– Failure Prevention
• Detect failure via periodic Self Test
• Auto garbage collection after each test run
• Enforce timeout limit on each test run
– Failure Recovery
• Reboot unresponsive devices, e.g. TTM
• Load PG and Communicator FW when failure
• Send work order back to Controller for re-execution
• Offline itself when failure cannot be recovered by itself
Controller Failure:
– Unlikely and didn’t happen in Dragon 1.0
– Auto-recover from connection lose, even power outage
10

11. Simple and Maintainable
• Test station (worker) baseline (rarely)
• Test libraries upgrade
– Install libraries on controller only
• PG and Communicator FW upgrade
– Just copy new FW to specific folder on controller
– New FW will be loaded into PG/Communicator automatically
• Hot swap failed hardware
• Offline individual station for trouble shooting
• Reconfigure a test station (worker) to different set of HW
• RESTful interface for monitoring cluster status
11

12. Performance: Results from Dragon 1.0
Regression/Test Management
# of test stations “Dolphinized” 19 + 1 offshore
# of dolphin workorders
completed
26,209
# of regression batches
submitted
581
# of tests run on the 8 fulltime
dolphin stations
17,778
Passing test rate on the 8
fulltime dolphin stations
53% (overall project) with
branded new test suite, test
libraries, com FW, Latitude.
# of hours of run time on the
fulltime dolphin stations
2946.88 (= 123 days, equivalent
to running non-stop from
September 1st until 2012)
Wall time to complete Formal
DVT Regressions (4 Rounds)
5 days 23 hours (including idle
time)
Passing test rate during first
Formal DVT Batch
78%
# of Automated Tests 1,169
12

13. Conclusions
• Dolphin has proven to be useful and effective
• Gaining popularity across functional departments
• Greatly simplifies large-scale test executions
• Dolphin makes running daily regression possible
• Fun to use: focus on testing, let infrastructure deal with
messy details
13