Iosif Itkin, Exactpro Co-Founder and CEO 9 April 2016, Ekaterinburg AIST is a scientific conference on Analysis of Images, Social Networks, and Texts. The conference is intended for computer scientists and practitioners whose research interests involve Internet mathematics and other related fields of data science. Similar to the previous year, the conference will be focused on applications of data mining and machine learning techniques to various problem domains: image processing, analysis of social networks, and natural language processing. We hope that the participants will benefit from the interdisciplinary nature of the conference and exchange experience.

1. Machine Learning and Testing in
London Stock Exchange Group
Iosif Itkin, Co-Founder and CEO, Exactpro
9 April 2016, AIST Conference
Quality Assurance & Related Software Development for Innovative Trading Systems
Tel: +7 495 640 2460, +1 415 830 38 49
www.exactpro.com

2. 2
Exactpro Systems Company Overview
Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com
Exactpro is: Clients Worldwide:
• A specialist firm focused on functional and non functional
testing of systems that process wholesale financial products,
particularly market infrastructure
• A UK company with operations in the US and four QA &
software development centres in Eastern Europe
• Part of London Stock Exchange Group as of May 29, 2015
• Incorporated in 2009 with 10 people, our company has
experienced significant growth as satisfied clients require
more services;

3. 3
London Stock Exchange Group
Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com
• A leading diversified international exchange and infrastructure group with assets across the entire exchange value chain
• The Group can trace its history back to 1698
• Headquartered in the United Kingdom, with significant global operations, the Group employs approximately 5,500 people
• Post trade and risk management services are a significant part of the Group’s business operations
• Through FTSE Russell, the Group is a global leader in financial indexing, benchmarking and analytic services with
approximately $10 trillion benchmarked to its indexes

4. 4
Tools & Methods of Program Analysis
Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com
Sailfish:
Automated testing
Compares intended and received results
ClearTH:
• Automated end 2 end clearing system testing
Dolphin:
Market surveillance testing
Shsha:
• Automated monitoring, analysis and
reporting
Load Injector:
• 75K messages / second from a single CPU core
• Measures latencies in microsecond range
Minirobots:
• A variety of algoes simulating end-clients
will run and see how system performs
Quality Assurance:
test planning and
managementProcess audit and test
coverage analysis
Intelligent Management
of Large Data Sets
Test data management
Protocol level testing using FIX/FAST, SOAP, HTTP,
ITCH, SWIFT, MQ, SQL, proprietary binary and text
based data formats, etc.
Automated regression testing
Gathering requirements and test scenario creation
(human, message & reporting interfaces)
Creating and productizing
state-of-the-art
test harnesses
Test automation
Intelligent functional and
exploratory testing
Latency & capacity
testing

5. 5
Quality Assurance Trends
Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com
FrAgile Process
Crowd-sourced Testing
Formal Verification Methodologies
Cognitive Technology

6. 6
FrAgile Process
Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com
Test is Dead
Ideas before Products
Happy Software Developers
Software Engineer in Test

7. 7
Crowd-Sourced Testing
Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com
Immediate Feedback from
Diverse User Base
Instrumentation is Required
to Process Data

8. 8
Formal Methods
Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com
Model Checking
Theorem Proving
Static Analysis
Verification

9. 9
What’s Next
Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com
Can We Learn from Other Industries?
Be Both Faster and Safe
Who Had the Most Severe Problems with Technology?

10. 10
Proposed Approach
Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com
Build Software to Test Software
Test Tool Sophistication should
Match Platform’s Complexity
A Good Robot is Required to
Survive in Robot Apocalypse

11. 11
Trading Systems
Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com
Testing Instruments, Risk Control and Monitoring Tools should have the
Same Priority as the Trading Platform
Testability is Required for Agile Process and Continuous Integration
SET = Building Software to Test Software
No Formal Artefacts from Agile Process. Parallel Stream for Test Harness
Formal Verification can Improve Quality
The Software will Break Anyway
The Absence of Adequate Monitoring and Kill Switches is What Turns a
Problem into a Disaster

12. 12
Passive Testing
Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com

13. 13
Test Tools for Client Onboarding and Certification
Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com

14. Regular Test Automation
14
Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com
Execute a test regression and make it:
as extensive as possible
as efficient as possible

15. 15
Minirobots
Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com
A set of Minirobots Algorithms allowing to perform a complete test
regression of a trading system within a few hours
All Robots work concurrently against the test environment
Robots are independent from each other, and their actions are based
on current conditions on the order books
Robots use an abstract Test Library and their main goal is to cover all
test cases within the library

16. Testing Algo-Trading System
16
Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
exactpro.com
Deploy one or several scalable matching engines to serve as market simulators.
Repeat your tests multiple times to assess the quality and efficiency of your software

17. Machine Learning
17
Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
exactpro.com
Select efficient limited subset of tests from an infinite number of possibilities
Identify test cases that were actually executed and their status
Teach robots to adhere to our values:
Integrity, Excellence, Partnership and Innovation

18. Human Learning
18
Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
exactpro.com
https://ru.linkedin.com/in/iosifitkin http://extentconf.com