Big Button 2.0 Alexey Zverev, CEO, Exactpro 11 November 2015 Trading Technology Trends & Quality Assurance Conference in St. Petersburg

1. Big Button 2.0Big Button 2.0
Quality Assurance & Related Software Development for Innovative Trading Systems
Tel: +7 495 640 2460, +1 415 830 38 49
www.exactpro.com
A new approach to automated regressionA new approach to automated regression
Alexey Zverev, CEO ExactproAlexey Zverev, CEO Exactpro
11 November 201511 November 2015

2. 2
Goal: to execute a test regression and make it:
•as extensive as possible (A)
•as efficient as possible (B)
How do people usually achieve that?
•They develop software which performs the tests and ensure that it requires
an adequate amount of maintenance when the software under test changes
Test Automation
Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com

3. 3
Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
exactpro.com
Test Automation Classics
1. Design Test Library 2. Develop Test Harness
• Script execution framework
• GUI injectors
• DPI injectors
• Upstream / Downstream
systems stubs
3. Develop Test Case 4. Develop Many Test Cases
5. Team Of QA Engineers Runs Regression
Login A
Login B
A send Order (…)
B Send Order (…)
A verify Ask (…)
B verify Trade (…)
A verify Trade (…)
Login A
Login B
A send Order (…)
B Send Order (…)
A verify Ask (…)
B verify Trade (…)
A verify Trade (…)

4. 4
Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
exactpro.com
The Job is not finished
The next task is to make test execution more efficient:
•We need a “Big Button”, i.e. software allowing executing a large number of
automated test scripts with minimum human attendance
•When successfully implemented it can reduce up to 80% effort (in man
days) spent on regression

5. 5
Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
exactpro.com
Big Button 1.0 concept
1. We have many automated
test cases
2. Prepare multiple
test environment
3. Prepare reference data 4. Redesign test library multiple threads
5. Develop orchestrating
test harness
Login A
Login B
A send Order (…)
B Send Order (…)
A verify Ask (…)
B verify Trade (…)
A verify Trade (…)
6. Run regression
- SHELL
- CRON
- ...
Login A Login B Login C

6. 6
Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
exactpro.com
Limitations of Big Button 1.0
• Complete control over test environments is required
• Automated GUI tests are not stable
• Dependent test cases are not stable
• Independent test cases require overhead
• A lot of copy/paste code
• Executing tests consecutively, one by one, takes significant time
• Big Button conditions are not similar to production (discussed later)

7. 7
Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
exactpro.com
Alternative approach - “Load with Trades”
• This was a success
• We have learned that it is called HiVAT (High volume automated
testing)
• Unfortunately it is still not a regression

8. 8
Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
exactpro.com
Meanwhile - Robots!
• The Trading Industry lives and breathes High Transaction Volumes
• High Volume is created by Trading Algos and we have created a test
tool called Minirobots to simulate that
• The tool good for simulation of production conditions
• Classical test scenarios are still not validated

9. 9
Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
exactpro.com
Big Button 2.0
• A set of Minirobots Algorithms allowing to perform a complete test
regression of a trading system within a few hours
• All Robots work concurrently against the test environment
• Robots are independent from each other, and their actions are based on
current conditions on the order books
• Robots use an abstract Test Library and their main goal is to cover all
test cases within the library

10. 10
Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
exactpro.com
How it will work
1. Random load “Pace Maker” 2. Variety of Passive Liquidity
3. “Agressors” try to interact
with the market to increase
number of test cases
happening in the environment
4. Analyze situation in retrospective
Make sure that what happened is correct
Tick checkboxes against test that was
actually executed
Updated
test
library

11. 11
Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
exactpro.com
Example
1. Read book 2. Search for appropriate test case
3. Try to test the scenario
4. Wait for full trade
or auto cancel
5. Iterate through actual event
orders:
6. Conclusion
•We hit only 2 orders
•Another robot cancelled 3rd
order
•Ticking box against this test case
IOC hits 3 passive orders at exact price
PFILL PFILL
CANCEL ASK

12. 12
Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
exactpro.com
Discussion
+ It can work really fast
+ Maintenance is limited (with adequate robot design)
+ A challenging and motivating task for QA
- May be too complex to implement
- It is not clear how to incorporate DLC related scenarios and manual
tests

13. 13
Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
exactpro.com
Q&A
Thank You!