29 November 2016 Harvard Club, New York, QA-Financial Forum Exactpro is the specialist testing subsidiary of the London Stock Exchange Group. In this presentation, Iosif Itkin will cover the wide range of testing requirements faced by broker-dealers, banks and exchanges beyond the requirements of functional testing. Plus: In Conversation with Yann L’Huillier, CIO, Tradition and Iosif Itkin, CEO, Exactpro - Iosif Itkin: Exactpro CEO – The London Stock Exchance Group - Yann L’Huillier: Group Chief Information Officer – Tradition

1. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com1
Non-Functional Testing at
London Stock Exchange
29 November 2016
Iosif Itkin, Exactpro CEO & Co-Founder
London Stock Exchange Group
Open Access Quality Assurance & Related Software Development for Financial Markets
Tel: +7 495 640 2460, +1 415 830 38 49
www.exactpro.com

2. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com2
 London Stock Exchange Group
 Exactpro - Open Access Software Testing
 Testing Execution Venues
 Efficient Load Generation
 Latency Measurement
 Non-Functional Testing Principles
 Other Ways to Produce Load
 Approach to Test Automation
 Failover Testing Approach
 Passive Software Testing
 Research in Tools and Methods of Program Analysis
ContentsContents

3. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com3
London Stock Exchange Group
• A leading diversified international exchange and infrastructure group with assets across the entire exchange value chain
• The Group can trace its history back to 1698
• Headquartered in the United Kingdom, with significant global operations, the Group employs approximately 3,500
people
• Post trade and risk management services are a significant part of the Group’s business operations
• Through FTSE Russell, the Group is a global leader in financial indexing, benchmarking and analytic services with
approximately $10 trillion benchmarked to its indexes

4. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com4
Open Access Software Testing
Exactpro is:
• A specialist firm operating an open access model with focus
on functional and operational testing of securities data
distribution, trading systems, risk management, market
surveillance and post-trade infrastructures.
• A UK company with operations in the US and four QA &
software development centres in Russia
• Part of London Stock Exchange Group as of May 29, 2015
• Incorporated in 2009 with 10 people, our company has
experienced significant growth as satisfied clients require
more services; now employing over 420 specialists
Clients Worldwide:

5. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com5
Open Access Software Testing
Toronto
New York
Tysons Corner
Montreal
Portland
Seattle
Dubai
Colombo
Johannesburg
Sydney
Singapore
Hong Kong
Moscow
Oslo
Paris
Milan
Athens
Vienna
London Amsterdam
A specialist firm operating an open access model
with focus on functional and operational testing for global financial markets worldwide.
A UK company with operations in the US
and four QA & software development
centres in Russia
Part of LSEG Since May 29, 2015
Incorporated in 2009 with 10 people, our
company has experienced significant
growth as satisfied clients require more
services;
Now employing over 400 specialists

6. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com6
Types of Trading Systems
Algo Trading
HFT
Brokerage
and IDB
Quality Assurance:
test planning and
management
Process audit and test
coverage analysis
Intelligent
Management
of Large Data Sets
Test data management
Protocol level testing using
FIX/FAST, SOAP, HTTP, ITCH,
SWIFT, MQ, SQL, proprietary
binary and text based
data formats, etc.
Automated regression
testing
Gathering requirements
and test scenario creation
(human, message & reporting
interfaces)
Creating and productizing
state-of-the-art
test harnesses
Test automation
Intelligent functional and
exploratory testing
Latency & capacity
testing
Market
Infrastructures

7. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com7
Execution Venues Testing
Our experience in market
infrastructures testing
embraces all components
and operational processes.
Unique tools have been
used to test a wide variety
of vendor platforms. A
holistic integrated
automated test solution for
trading, clearing,
settlement and other
systems

8. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com8
Performance Testing Cost Structure
System Under
Test
Production
Connectivity
Test
Tools
Infrastructure:
≈ 10 - 100 servers
Infrastructure:
thousands of servers
Infrastructure:
1 - 4 servers

9. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com9
The Idea Behind Efficient Load Generation

10. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com10
The Idea Behind Efficient Load Generation

11. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com11
Latency Measurement

12. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com12
Latency Distribution

13. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com13
Question Everything
Do Not Trust:
• Test Tools
• Average Metrics and Percentiles
• That Load Shape is Right
• System Requirements
• Anything

14. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com14
Keep Calm
Have No Fear:
• Annihilate the System Under Test
• Repeat the Same Tests Many Times
• Load Levels that Exceed Presumed Production
• Make no Assumptions About
Extrapolating Test Results into
More Scalable Environments

15. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com15
Be Confident
Do Not Beg For:
• Test Tools
• Confirmation that PROD is Configured as TEST
• Max Load Figure
• Load Scenarios for Existing Production System
• Constant Support

16. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com16
Two Load Generator Types
1.
2.  Simulate real trading
participants
 Adopt to changing market
conditions
 Check data consistency from
member’s point of view

17. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com17
Achieving Diverse and Relevant Load
1. Random load “Pace Maker” 2. Variety of Passive Liquidity
3. “Aggressors” try to interact with
the market to increase the number
of test cases happening in the
environment
4. Analyze the situation in retrospect. Make
sure that what happened is correct. Tick
checkboxes against tests that were actually
executed
Updated
test
library

18. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com18
Types of Non-Functional Tests

19. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com19
Failover Tests and Catching In-Flight Transactions

20. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com20
Use Passive Testing Tools to Collect Audit Trails

21. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com21
Questions & Answers
• Competing conference papers go through a review by a number
of leading software development and analysis experts and
scientists
• The best papers are recommended for their inclusion in Russia’s
leading professional and scientific publications
The Fourth TMPA Conference Find out more: www.tmpaconf.org
Renowned experts in program engineering speaking on:
• software test automation
• static program analysis
• verification
• dynamic methods of program analysis
• analysis of parallel and distributed systems
• testing and analysis of high-availability systems
• analysis of hardware and software systems
• methods of building quality software
• tools for software analysis, testing and verification

22. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com22
Questions & Answers
https://ru.linkedin.com/in/iosifitkin
The seventh EXTENT conference will take place in
London, UK.
Find out more:
www.extentconf.com
– LSEG Technology and Quality Assurance
– Risk controls and FPGA
– Market Surveillance systems
– Blockchain and trading technology trends
– Artificial intelligence