EXTENT-2017: Software Testing & Trading Technology Trends Conference 29 June, 2017, 10 Paternoster Square, London Climbing Out of the Stability Sinkhole - Survivor’s Guide Sergei Poliakoff, CIO, Moscow Exchange Would like to know more? Visit our website: extentconf.com Follow us: https://www.linkedin.com/company/exactpro-systems-llc?trk=biz-companies-cym https://twitter.com/exactpro #extentconf #exactpro

1. CLIMBING OUT OF THE STABILITY
SINKHOLE

2. BASIC REAL-TIME POST-TRADE
REAL-TIME ORDER
PRE-CLEARING
TRADES
END-OF-DAY, INTRA-DAY
MARGIN CALL
END-OF-DAY
OR MINI BATCH
ORDERS
~400
µs
RISK AND MARGIN
CALCULATED
IN BATCHES
RISK AND MARGIN
CALCULATED
POST-TRADE IN REAL-TIME
PROJECTED RISK
CALCULATED
BEFORE ACCEPTING THE ORDER
IF PROJECTED MARGIN
REQUIREMENT BREACHES
AVAILABLE MARGIN,
ORDER IS NOT ADMITTED TO
MATCHBOOK
ORDERS
TRADES
ORDERS ORDERS
REAL-TIME POST-TRADE
RISK CHECK
AND
MARGIN CALL
TRADES
ORDERS ORDERS
~12000 orders/second
HYPOTHETICAL TRADE
RISK CALCULATION
~400
µsACCOUNT
SHUTOFF
UNIQUE REAL-TIME RISK MANAGEMENT = STABILITY CHALLENGE

3. INCIDENT DATE SOLUTION
HARDWARE FAULT
DISRUPTING
THE BACKUP SCHEME
August, 12
September, 1
September, 8
 Hardware replacement and upgrade (< 3 years)
 Migration to «flat» network topology
 Network segregation
 Human resource development in operation and maintenance department
 New Tier III data center
CLEARING MODULE
FAULTS
January, 12
March, 5
June, 15
 Segregation of Trading and Clearing modules
 Emergency limit check scheme
 Orders risk check model update
 Development process improvement
 Software Development Life Cycle practices implementation
 Introduction of “destructive testing”
 Testing cycle extension
TRADE ENGINE FAULTS September, 21
 Trade engine cloning (as a part of trading and clearing modules
segregation programme)
 Common development process improvement
THE INGLORIOUS 2015
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
CM CM CM H H
H
TE
HARDWARE FAULTSH
CLEARING MODULE
FAULTS
CM TRADE ENGINE FAULTSTE
CRITICAL FAULTS IN 2015 TIMESCALE

4. 4
OLD SOFTWARE DEV PRACTICES COULDN’T COPE WITH COMPLEXITY
=> NEW DEV PROCESS
Unit-like Testing Practice
Auto tests Coverage Extending
Changes reference to projects, tasks and issues
Regular Code Review Practices
Static and Dynamic Code Analysis
Continuous Integration (CI)
Auto deployment
Software
Development
Life Cycle
(SDLC)
SOFTWARE DEVELOPMENT LIFE CYCLE
QUALITY ASSURANCE PRACTICES
2017 – NEW & IMPROVED PRACTICES2014
PREVIOUS IMPROVED NEW
Unit Tests
UAT (external)
Acceptance testing
Integrational testing
(cross-system)
Simulation testing
Destructive testing
Manual functional
testing
Manual regression
testing
Automatic functional
testing
Automatic regression
testing
DEVELOPMENT
Testing metrics
Static code review
TESTING
IMPLEMENTATION Testing metrics
TOOLS
Auto deploymentClarive + Ansible
Coverity Static code analysis
GitLab, Crucible Code review
Continuous integrationJenkins
Serena, Jira Bug Tracking, Task Tracking
HP ALM Testing lifecycle management
Valgrind Dynamic code analysis (Spectra)
Regression testingPyTests
Undefined Sanitizer
Address Sanitizer
Dynamic code analysis (ASTS)
Doxygen Auto documentation
FUNCTIONS
AUTODEPLOYMENT COVERAGE: 100% for real-time systems
OTHER TOOLS COVERAGE: ~80% of current releases

5. AFTER THE ACTIVE PHASE OF DEVELOPMENT, DEVELOPERS
 PARTICIPATE IN BUG CORRECTION
 START WORKING ON THE TASK OF THE NEXT RELEASE
 IMPROVE METHODS AND TOOLS FOR TESTING
 WORK ON APPROVED NON-RELEASE TASKS
 WORK ON OPTIMIZATION AND TECHNOLOGICAL DEVELOPMENT
RELEASE CYCLE
FUNCTIONAL TASK APPROVAL
5 weeks
DEVELOPMENT
13 weeks
TESTING
14 weeks
PREPARING
5 weeks
DEVELOPMENT
13 weeks
TESTING
14 weeks
5
RELEASE
1
RELEASE
2
3 WEEKS 2 WEEKS 13 WEEKS 12 WEEKS
APPROVAL OF THE
RELEASE COMPOSITION,
DEPLOYMENT PLAN
AND ACCEPTANCE TESTING
WORKING GROUP
WEEKOF
SILENCE
FUNCTIONAL TASK
APPROVAL
TERMS OF
REFERENCE AND
DESIGN
DEVELOPMENT
BUGFIXING
REFINED
FUNCTIONAL
REQUIREMENTS
START OF
DEVELOPMENT
OF RELEASE
TASKS
END OF
DEVELOPMENT OF
RELEASE TASKS
RELEASE
TESTINGRELEASE TASKS DEVELOPMENT AND TESTING
NEW DEVELOPMENT
PRACTICES
NEW TESTING
PRACTICES
CAN’T SPEND 100% TIME TESTING, NEED TO DELIVER!

6. 2014 - 20152013 2016-2017
HIERARCHICAL NETWORK
SPANNING TREE
DEVELOPMENT
TESTING
GAME STAND
OPERATIONAL SYSTEMS
OPTIMIZED FLAT
NETWORK
SPINE&LEAF
SEGMENTED NETWORK
SPINE&LEAF
OPERATIONAL
SYSTEM
DEVELOPMENT
TESTING
GAME SEGMENT
OPERATIONAL SYSTEMS
TEST & GAME
SYSTEMS
OFFICE DEVELOPMENT
MAIN LOAD ON THE
ROOT DEVICES
NETWORK STORMS
NON-PERSISTENT
INCREASED
RESISTANCE TO
NETWORK STORMS
CONTAINMENT OF NETWORK
DAMAGE IN ONE SEGMENT
SERVERS
NETWORK
EXTERNAL
ACCESS
EXTERNAL
ACCESS
EXTERNAL
ACCESS
EXTERNAL
ACCESS
EXTERNAL
ACCESS
Spine & Leaf «flat» network topology implementation significantly decrease expectation of repeating serious consequences in case
of network storm
6
THE GREAT NETWORK MELTDOWN OF AUGUST, AND WHAT WE DID ABOUT IT

7. SYSTEM CRITICALITY CLASS EQUIPMENT UPDATE PERIOD
TRADING SYSTEM ENGINE (REAL-TIME) 1A 3 YEARS
MAIN PRODUCTION SYSTEMS 2A, 3А 4 YEARS
RESPONSIBLE SYSTEMS B 4 YEARS
NON-CRITICAL SYSTEMS C 5 YEARS
NETWORK --- 5 YEARS
TECHNICAL POLICY REQUIREMENTS (INTRODUCED IN 2014)
58
30
22 19
29
26
16 33
24
13 66
16
54
44
57 58
34
0
10
20
30
40
50
60
70
80
90
100
1A 2A 3A B C Network
Percentageofhardwarecomposition(%)
CRITICALITY CLASS
2015 STATE
Less than 3 years
2013 - 2015
3 - 5 years
2011 - 2012
Older than 5 years
2003 - 2010
100
88
70 76 83
12
30 24 17
96
4
0
10
20
30
40
50
60
70
80
90
100
1A 2A 3A B C Network
Percentageofhardwarecomposition(%)
CRITICALITY CLASS
TARGET STATE
Less than 3 years
2013 - 2015
3 - 5 years
2011 - 2012
Older than 5 years
2003 - 2010
7
HARDWARE PARK BY AGE
HARDWARE: NEWER IS BETTER

8. 8
 Tier-3 certified data center delivers 99,98% availability
 Compliance with safety requirements of Payment Card
Industry Data Security Standard (PCI DSS) v.3 to ensure the
security of customer information
 High level of safety and resistance to adverse external
influences
RELIABILITY
SECURITY
 Stand-alone building
 Security service
 Administered surrounding territory, guarded area
 Access control system
 CCTV monitoring
FURTHER DEVELOPMENT CAPACITY
NEW DATA CENTER

9. ICING ON A RELIABILITY CAKE: BETTER PERFORMANCE
Results below from annual joint Exchange/brokers stress tests of core infrastructure , fall 2016
LOAD TESTING RESULTS
ASTS+ SPECTRA
Average response time: 230 µs
90% responses < 270 µs
99% responses (transaction frequency less than 50 per
second)
< 400 µs
99% responses (transaction frequency higher than 500 per
second)
1500 µs
99.9% responses (typical real market frequency of
transactions)
< 600 µs
Average response time:1 250 µs
Under load up to 50 000 Tr/sec < 250 µs
99% responses < 1000 µs
1 For forecasted within the next year peak frequencies of 20
000 - 30 000 transactions per second
TWIME AND CGATE COMPARISON
transactions Clients’ transaction
TIME

10. 99.98 99.98 99.98
99.95
99.98
99.98
99.91
99.92
99.93
99.94
99.95
99.96
99.97
99.98
99.99
2012 2013 2014 2015 2016 Q1-2 2017
2016 – RETURN TO STABILITY, HOPEFULLY LASTING
Real-time systems availability, %
Target value Average value, 2012-2017