SlideShare a Scribd company logo
Build Software to Test Software
exactpro.com
User-Assisted Log Analysis for Quality Control
of Distributed Fintech Applications
Iosif Itkin1 , Anna Gromova1 , Anton Sitnikov1 , Elena Treshcheva1 , Rostislav
Yavorskiy3, Evgenii Tsymbalov2, Andrey Novikov3 and Kirill Rudakov3
1 Exactpro, UK, Georgia, USA, Russia
2 Skolkovo Institute of Science and Technology, Russia
3 Higher School of Economics, Russia
2 Build Software to Test Software
exactpro.com
Exactpro overview
● A specialist firm focused on functional and non-functional testing of
exchanges, clearing houses, depositories, trade repositories and other
financial market infrastructures.
● We provide software testing services for mission critical technology that
underpins global financial markets. Our clients are regulated by FCA, Bank
of England and their counterparts from other countries.
● We were part of the London Stock Exchange Group (LSEG) from May
2015 till January 2018. Exactpro management buyout from LSEG was
successfully completed in January 2018. We are headquartered in the UK
and have operations in the US, Georgia and Russia.
● Incorporated in 2009 with 10 people, our company has
experienced significant growth and is now employing
over 550 specialists.
3 Build Software to Test Software
exactpro.com
We have a global Software Quality Assurance client network
4 Build Software to Test Software
exactpro.com
Build Software to Test Software
5 Build Software to Test Software
exactpro.com
SUT: Clearing and Settlement Platform
6 Build Software to Test Software
exactpro.com
Log usage in passive testing
- Receive new error alerts
- See alerts on re-appeared old bugs
- Know if application behaves differently
7 Build Software to Test Software
exactpro.com
User-Assisted Log Analysis: Challenges
- Massive logs (105 error messages / day is normal)
- Log messages for same error types are different as strings.
Need to present huge volumes as a human-perceivable collection of error
messages
8 Build Software to Test Software
exactpro.com
Automated Clustering with User Correction
9 Build Software to Test Software
exactpro.com
- Case is built on 3.2M records volume
- Message length – up to 15,000 characters
The challenge: Logs classification
10 Build Software to Test Software
exactpro.com
Message structure
11 Build Software to Test Software
exactpro.com
Message text
12 Build Software to Test Software
exactpro.com
Approach 1: Template-Based
- For short messages only (no long stack traces, ~80% of messages)
- Messages considered equal if differ by < 20% content (3+ char substrings)
- Greedy algorithm: start with seed, compare each new message to classes
already found
13 Build Software to Test Software
exactpro.com
- 109 classes identified
Approach 1: Template-Based
14 Build Software to Test Software
exactpro.com
Approach 2: NLP-based
- All messages, including long ones
- Tokenize, leave only alphabet chars
- TF/IDF to identify significant feature set – dimensions for
vectorization
- Document-term sparse matrix (6,776-dimensions)
- Truncate dimensions to 25 (SVD)
- K-means assessed with Silhouette Coefficient
15 Build Software to Test Software
exactpro.com
Approach 2: NLP-based
16 Build Software to Test Software
exactpro.com
Approaches comparison
- Each returned around 100 clusters
- Clustering correspond to each other, Rand index around 0,9
- Naïve algorithm is “good enough” for initial classification.
17 Build Software to Test Software
exactpro.com
P.S. Latest Results: Beyond K-Means
Problems of “naïve” approach:
- 3.2M records is too much, need preliminary tokenization – from
record to “signature”
- Loose clusters in K-means output – not single error type in human
understanding. Need more dense clustering.
18 Build Software to Test Software
exactpro.com
Iterate k-means?
- Idea – to run k-means again on loose clusters
- Did not work. We have loose clusters, and singles on next step.
19 Build Software to Test Software
exactpro.com
Pipeline: Initial Classification
3.2M
records
20 Build Software to Test Software
exactpro.com
Pipeline: New Logs Processing
21 Build Software to Test Software
exactpro.com
Improvement Gained
- Working with signatures simplifies the process (but requires tough
tokenization)
- Agglomerative (vs k-means) naturally produces dense clusters
- No dimension reduction – better intuition in clustering
Questions?
Thank You

More Related Content

What's hot

Istqb advanced practioner_level_practice_exam_1
Istqb advanced practioner_level_practice_exam_1Istqb advanced practioner_level_practice_exam_1
Istqb advanced practioner_level_practice_exam_1fkhry
 
The Road Not Taken: Estimating Path Execution Frequency Statically
The Road Not Taken: Estimating Path Execution Frequency StaticallyThe Road Not Taken: Estimating Path Execution Frequency Statically
The Road Not Taken: Estimating Path Execution Frequency StaticallyRay Buse
 
WHITE BOX & BLACK BOX TESTING IN DATABASE
WHITE BOX & BLACK BOXTESTING IN DATABASEWHITE BOX & BLACK BOXTESTING IN DATABASE
WHITE BOX & BLACK BOX TESTING IN DATABASE
Salman Memon
 
White Box testing by Pankaj Thakur, NITTTR Chandigarh
White Box testing by Pankaj Thakur, NITTTR ChandigarhWhite Box testing by Pankaj Thakur, NITTTR Chandigarh
White Box testing by Pankaj Thakur, NITTTR Chandigarh
Pankaj Thakur
 
Software testing lab manual
Software testing lab manualSoftware testing lab manual
Software testing lab manual
Tanzeem Syed
 
Testing, black ,white and gray box testing
Testing, black ,white and gray box testingTesting, black ,white and gray box testing
Testing, black ,white and gray box testing
Aamir Shakir
 
Software Testing & Debugging
Software Testing & DebuggingSoftware Testing & Debugging
Software Testing & Debugging
Computing Cage
 
Chapter 8 software testing
Chapter 8 software testingChapter 8 software testing
Chapter 8 software testing
despicable me
 
5 black box and grey box testing
5   black box and grey box testing5   black box and grey box testing
5 black box and grey box testing
Yisal Khan
 
Black box testing or behavioral testing
Black box testing or behavioral testingBlack box testing or behavioral testing
Black box testing or behavioral testingSlideshare
 
Black & White Box testing
Black & White Box testingBlack & White Box testing
Software Testing Tecniques
Software Testing TecniquesSoftware Testing Tecniques
Software Testing Tecniquesersanbilik
 
Software Quality Testing
Software Quality TestingSoftware Quality Testing
Software Quality Testing
Kiran Kumar
 
Strategies to Avoid Test Fixture Smells durin Software Evolution
Strategies to Avoid Test Fixture Smells durin Software EvolutionStrategies to Avoid Test Fixture Smells durin Software Evolution
Strategies to Avoid Test Fixture Smells durin Software Evolution
Michaela Greiler
 
Testing Fundamentals
Testing FundamentalsTesting Fundamentals
Testing Fundamentals
Kiran Kumar
 
White boxvsblackbox
White boxvsblackboxWhite boxvsblackbox
White boxvsblackboxsanerjjd
 
Software Testing - Day One
Software Testing - Day OneSoftware Testing - Day One
Software Testing - Day OneGovardhan Reddy
 

What's hot (19)

Istqb advanced practioner_level_practice_exam_1
Istqb advanced practioner_level_practice_exam_1Istqb advanced practioner_level_practice_exam_1
Istqb advanced practioner_level_practice_exam_1
 
The Road Not Taken: Estimating Path Execution Frequency Statically
The Road Not Taken: Estimating Path Execution Frequency StaticallyThe Road Not Taken: Estimating Path Execution Frequency Statically
The Road Not Taken: Estimating Path Execution Frequency Statically
 
WHITE BOX & BLACK BOX TESTING IN DATABASE
WHITE BOX & BLACK BOXTESTING IN DATABASEWHITE BOX & BLACK BOXTESTING IN DATABASE
WHITE BOX & BLACK BOX TESTING IN DATABASE
 
White Box testing by Pankaj Thakur, NITTTR Chandigarh
White Box testing by Pankaj Thakur, NITTTR ChandigarhWhite Box testing by Pankaj Thakur, NITTTR Chandigarh
White Box testing by Pankaj Thakur, NITTTR Chandigarh
 
Software testing lab manual
Software testing lab manualSoftware testing lab manual
Software testing lab manual
 
Testing, black ,white and gray box testing
Testing, black ,white and gray box testingTesting, black ,white and gray box testing
Testing, black ,white and gray box testing
 
Software Testing & Debugging
Software Testing & DebuggingSoftware Testing & Debugging
Software Testing & Debugging
 
Testing techniques
Testing techniquesTesting techniques
Testing techniques
 
Chapter 8 software testing
Chapter 8 software testingChapter 8 software testing
Chapter 8 software testing
 
5 black box and grey box testing
5   black box and grey box testing5   black box and grey box testing
5 black box and grey box testing
 
Black box testing or behavioral testing
Black box testing or behavioral testingBlack box testing or behavioral testing
Black box testing or behavioral testing
 
Black & White Box testing
Black & White Box testingBlack & White Box testing
Black & White Box testing
 
Software Testing Tecniques
Software Testing TecniquesSoftware Testing Tecniques
Software Testing Tecniques
 
Software Quality Testing
Software Quality TestingSoftware Quality Testing
Software Quality Testing
 
Strategies to Avoid Test Fixture Smells durin Software Evolution
Strategies to Avoid Test Fixture Smells durin Software EvolutionStrategies to Avoid Test Fixture Smells durin Software Evolution
Strategies to Avoid Test Fixture Smells durin Software Evolution
 
Software Testing 1/5
Software Testing 1/5Software Testing 1/5
Software Testing 1/5
 
Testing Fundamentals
Testing FundamentalsTesting Fundamentals
Testing Fundamentals
 
White boxvsblackbox
White boxvsblackboxWhite boxvsblackbox
White boxvsblackbox
 
Software Testing - Day One
Software Testing - Day OneSoftware Testing - Day One
Software Testing - Day One
 

Similar to User-Assisted Log Analysis for Quality Control of Distributed Fintech Applications

Defects mining in exchanges - medvedev, klimakov, yamkovi
Defects mining in exchanges - medvedev, klimakov, yamkoviDefects mining in exchanges - medvedev, klimakov, yamkovi
Defects mining in exchanges - medvedev, klimakov, yamkovi
DataFest Tbilisi
 
Machine programming
Machine programmingMachine programming
Machine programming
DESMOND YUEN
 
Effort Used to Create Domain-Specific Modeling Languages
Effort Used to Create Domain-Specific Modeling LanguagesEffort Used to Create Domain-Specific Modeling Languages
Effort Used to Create Domain-Specific Modeling Languages
Juha-Pekka Tolvanen
 
Internship msc cs
Internship msc csInternship msc cs
Internship msc cs
Pooja Bhojwani
 
Path to continuous delivery
Path to continuous deliveryPath to continuous delivery
Path to continuous delivery
Anirudh Bhatnagar
 
Three Interviews About Static Code Analyzers
Three Interviews About Static Code AnalyzersThree Interviews About Static Code Analyzers
Three Interviews About Static Code Analyzers
Andrey Karpov
 
QA Financial Singapore: Deliberate Practice of Software Testing in Agile World
QA Financial Singapore: Deliberate Practice of  Software Testing in Agile WorldQA Financial Singapore: Deliberate Practice of  Software Testing in Agile World
QA Financial Singapore: Deliberate Practice of Software Testing in Agile World
Iosif Itkin
 
Finding Zero-Days Before The Attackers: A Fortune 500 Red Team Case Study
Finding Zero-Days Before The Attackers: A Fortune 500 Red Team Case StudyFinding Zero-Days Before The Attackers: A Fortune 500 Red Team Case Study
Finding Zero-Days Before The Attackers: A Fortune 500 Red Team Case Study
DevOps.com
 
API automation with JMeter + Bamboo CI
API automation with JMeter + Bamboo CIAPI automation with JMeter + Bamboo CI
API automation with JMeter + Bamboo CI
Mykola Kovsh
 
Jun 08 - PMWT Featured Paper -Tarabykin - XP PAPER - FINAL
Jun 08 - PMWT Featured Paper -Tarabykin - XP PAPER - FINALJun 08 - PMWT Featured Paper -Tarabykin - XP PAPER - FINAL
Jun 08 - PMWT Featured Paper -Tarabykin - XP PAPER - FINALAlex Tarra
 
ICIC 2013 New Product Introductions CEPT
ICIC 2013 New Product Introductions CEPTICIC 2013 New Product Introductions CEPT
ICIC 2013 New Product Introductions CEPTDr. Haxel Consult
 
Use Case Patterns for LLM Applications (1).pdf
Use Case Patterns for LLM Applications (1).pdfUse Case Patterns for LLM Applications (1).pdf
Use Case Patterns for LLM Applications (1).pdf
M Waleed Kadous
 
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
Brian Brazil
 
How to teach your data scientist to leverage an analytics cluster with Presto...
How to teach your data scientist to leverage an analytics cluster with Presto...How to teach your data scientist to leverage an analytics cluster with Presto...
How to teach your data scientist to leverage an analytics cluster with Presto...
Alluxio, Inc.
 
10 years in Network Protocol testing L2 L3 L4-L7 Tcl Python Manual and Automa...
10 years in Network Protocol testing L2 L3 L4-L7 Tcl Python Manual and Automa...10 years in Network Protocol testing L2 L3 L4-L7 Tcl Python Manual and Automa...
10 years in Network Protocol testing L2 L3 L4-L7 Tcl Python Manual and Automa...Mullaiselvan Mohan
 
Shriraam-performance test engineer 5.4 years
Shriraam-performance test engineer 5.4 yearsShriraam-performance test engineer 5.4 years
Shriraam-performance test engineer 5.4 years
shriraam ms
 

Similar to User-Assisted Log Analysis for Quality Control of Distributed Fintech Applications (20)

Defects mining in exchanges - medvedev, klimakov, yamkovi
Defects mining in exchanges - medvedev, klimakov, yamkoviDefects mining in exchanges - medvedev, klimakov, yamkovi
Defects mining in exchanges - medvedev, klimakov, yamkovi
 
Machine programming
Machine programmingMachine programming
Machine programming
 
Effort Used to Create Domain-Specific Modeling Languages
Effort Used to Create Domain-Specific Modeling LanguagesEffort Used to Create Domain-Specific Modeling Languages
Effort Used to Create Domain-Specific Modeling Languages
 
Amit Bhandari
Amit BhandariAmit Bhandari
Amit Bhandari
 
Neha_Maggu
Neha_MagguNeha_Maggu
Neha_Maggu
 
Internship msc cs
Internship msc csInternship msc cs
Internship msc cs
 
Path to continuous delivery
Path to continuous deliveryPath to continuous delivery
Path to continuous delivery
 
Three Interviews About Static Code Analyzers
Three Interviews About Static Code AnalyzersThree Interviews About Static Code Analyzers
Three Interviews About Static Code Analyzers
 
QA Financial Singapore: Deliberate Practice of Software Testing in Agile World
QA Financial Singapore: Deliberate Practice of  Software Testing in Agile WorldQA Financial Singapore: Deliberate Practice of  Software Testing in Agile World
QA Financial Singapore: Deliberate Practice of Software Testing in Agile World
 
Updated resume
Updated resumeUpdated resume
Updated resume
 
Finding Zero-Days Before The Attackers: A Fortune 500 Red Team Case Study
Finding Zero-Days Before The Attackers: A Fortune 500 Red Team Case StudyFinding Zero-Days Before The Attackers: A Fortune 500 Red Team Case Study
Finding Zero-Days Before The Attackers: A Fortune 500 Red Team Case Study
 
API automation with JMeter + Bamboo CI
API automation with JMeter + Bamboo CIAPI automation with JMeter + Bamboo CI
API automation with JMeter + Bamboo CI
 
Jun 08 - PMWT Featured Paper -Tarabykin - XP PAPER - FINAL
Jun 08 - PMWT Featured Paper -Tarabykin - XP PAPER - FINALJun 08 - PMWT Featured Paper -Tarabykin - XP PAPER - FINAL
Jun 08 - PMWT Featured Paper -Tarabykin - XP PAPER - FINAL
 
ICIC 2013 New Product Introductions CEPT
ICIC 2013 New Product Introductions CEPTICIC 2013 New Product Introductions CEPT
ICIC 2013 New Product Introductions CEPT
 
Use Case Patterns for LLM Applications (1).pdf
Use Case Patterns for LLM Applications (1).pdfUse Case Patterns for LLM Applications (1).pdf
Use Case Patterns for LLM Applications (1).pdf
 
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
 
How to teach your data scientist to leverage an analytics cluster with Presto...
How to teach your data scientist to leverage an analytics cluster with Presto...How to teach your data scientist to leverage an analytics cluster with Presto...
How to teach your data scientist to leverage an analytics cluster with Presto...
 
10 years in Network Protocol testing L2 L3 L4-L7 Tcl Python Manual and Automa...
10 years in Network Protocol testing L2 L3 L4-L7 Tcl Python Manual and Automa...10 years in Network Protocol testing L2 L3 L4-L7 Tcl Python Manual and Automa...
10 years in Network Protocol testing L2 L3 L4-L7 Tcl Python Manual and Automa...
 
Updated resume
Updated resumeUpdated resume
Updated resume
 
Shriraam-performance test engineer 5.4 years
Shriraam-performance test engineer 5.4 yearsShriraam-performance test engineer 5.4 years
Shriraam-performance test engineer 5.4 years
 

More from Iosif Itkin

Foundations of Software Testing Lecture 4
Foundations of Software Testing Lecture 4Foundations of Software Testing Lecture 4
Foundations of Software Testing Lecture 4
Iosif Itkin
 
QA Financial Forum London 2021 - Automation in Software Testing. Humans and C...
QA Financial Forum London 2021 - Automation in Software Testing. Humans and C...QA Financial Forum London 2021 - Automation in Software Testing. Humans and C...
QA Financial Forum London 2021 - Automation in Software Testing. Humans and C...
Iosif Itkin
 
Exactpro FinTech Webinar - Global Exchanges FIX Protocol
Exactpro FinTech Webinar - Global Exchanges FIX ProtocolExactpro FinTech Webinar - Global Exchanges FIX Protocol
Exactpro FinTech Webinar - Global Exchanges FIX Protocol
Iosif Itkin
 
Operational Resilience in Financial Market Infrastructures
Operational Resilience in Financial Market InfrastructuresOperational Resilience in Financial Market Infrastructures
Operational Resilience in Financial Market Infrastructures
Iosif Itkin
 
20 Simple Questions from Exactpro for Your Enjoyment This Holiday Season
20 Simple Questions from Exactpro for Your Enjoyment This Holiday Season20 Simple Questions from Exactpro for Your Enjoyment This Holiday Season
20 Simple Questions from Exactpro for Your Enjoyment This Holiday Season
Iosif Itkin
 
Testing the Intelligence of your AI
Testing the Intelligence of your AITesting the Intelligence of your AI
Testing the Intelligence of your AI
Iosif Itkin
 
EXTENT 2019: Exactpro Quality Assurance for Financial Market Infrastructures
EXTENT 2019: Exactpro Quality Assurance for Financial Market InfrastructuresEXTENT 2019: Exactpro Quality Assurance for Financial Market Infrastructures
EXTENT 2019: Exactpro Quality Assurance for Financial Market Infrastructures
Iosif Itkin
 
EXTENT Talks 2019 Tbilisi: Failover and Recovery Test Automation - Ivan Shamrai
EXTENT Talks 2019 Tbilisi: Failover and Recovery Test Automation - Ivan ShamraiEXTENT Talks 2019 Tbilisi: Failover and Recovery Test Automation - Ivan Shamrai
EXTENT Talks 2019 Tbilisi: Failover and Recovery Test Automation - Ivan Shamrai
Iosif Itkin
 
EXTENT Talks QA Community Tbilisi 20 April 2019 - Conference Open
EXTENT Talks QA Community Tbilisi 20 April 2019 - Conference OpenEXTENT Talks QA Community Tbilisi 20 April 2019 - Conference Open
EXTENT Talks QA Community Tbilisi 20 April 2019 - Conference Open
Iosif Itkin
 
QAFF Chicago 2019 - Complex Post-Trade Systems, Requirements Traceability and...
QAFF Chicago 2019 - Complex Post-Trade Systems, Requirements Traceability and...QAFF Chicago 2019 - Complex Post-Trade Systems, Requirements Traceability and...
QAFF Chicago 2019 - Complex Post-Trade Systems, Requirements Traceability and...
Iosif Itkin
 
QA Community Saratov: Past, Present, Future (2019-02-08)
QA Community Saratov: Past, Present, Future (2019-02-08)QA Community Saratov: Past, Present, Future (2019-02-08)
QA Community Saratov: Past, Present, Future (2019-02-08)
Iosif Itkin
 
Machine Learning and RoboCop Testing
Machine Learning and RoboCop TestingMachine Learning and RoboCop Testing
Machine Learning and RoboCop Testing
Iosif Itkin
 
Behaviour Driven Development: Oltre i limiti del possibile
Behaviour Driven Development: Oltre i limiti del possibileBehaviour Driven Development: Oltre i limiti del possibile
Behaviour Driven Development: Oltre i limiti del possibile
Iosif Itkin
 
2018 - Exactpro Year in Review
2018 - Exactpro Year in Review2018 - Exactpro Year in Review
2018 - Exactpro Year in Review
Iosif Itkin
 
Exactpro Discussion about Joy and Strategy
Exactpro Discussion about Joy and StrategyExactpro Discussion about Joy and Strategy
Exactpro Discussion about Joy and Strategy
Iosif Itkin
 
FIX EMEA Conference 2018 - Post Trade Software Testing Challenges
FIX EMEA Conference 2018 - Post Trade Software Testing ChallengesFIX EMEA Conference 2018 - Post Trade Software Testing Challenges
FIX EMEA Conference 2018 - Post Trade Software Testing Challenges
Iosif Itkin
 
BDD. The Outer Limits. Iosif Itkin at Youcon (in Russian)
BDD. The Outer Limits. Iosif Itkin at Youcon (in Russian)BDD. The Outer Limits. Iosif Itkin at Youcon (in Russian)
BDD. The Outer Limits. Iosif Itkin at Youcon (in Russian)
Iosif Itkin
 
Sibos 2017: Disruptive functional testing - the next frontier in post-trade s...
Sibos 2017: Disruptive functional testing - the next frontier in post-trade s...Sibos 2017: Disruptive functional testing - the next frontier in post-trade s...
Sibos 2017: Disruptive functional testing - the next frontier in post-trade s...
Iosif Itkin
 
Using Cluster Analysis for Characteristics Detection in Software Defect Reports
Using Cluster Analysis for Characteristics Detection in Software Defect ReportsUsing Cluster Analysis for Characteristics Detection in Software Defect Reports
Using Cluster Analysis for Characteristics Detection in Software Defect Reports
Iosif Itkin
 
EXTENT-2017: Testing in Distributed Ledger Systems
EXTENT-2017: Testing in Distributed Ledger SystemsEXTENT-2017: Testing in Distributed Ledger Systems
EXTENT-2017: Testing in Distributed Ledger Systems
Iosif Itkin
 

More from Iosif Itkin (20)

Foundations of Software Testing Lecture 4
Foundations of Software Testing Lecture 4Foundations of Software Testing Lecture 4
Foundations of Software Testing Lecture 4
 
QA Financial Forum London 2021 - Automation in Software Testing. Humans and C...
QA Financial Forum London 2021 - Automation in Software Testing. Humans and C...QA Financial Forum London 2021 - Automation in Software Testing. Humans and C...
QA Financial Forum London 2021 - Automation in Software Testing. Humans and C...
 
Exactpro FinTech Webinar - Global Exchanges FIX Protocol
Exactpro FinTech Webinar - Global Exchanges FIX ProtocolExactpro FinTech Webinar - Global Exchanges FIX Protocol
Exactpro FinTech Webinar - Global Exchanges FIX Protocol
 
Operational Resilience in Financial Market Infrastructures
Operational Resilience in Financial Market InfrastructuresOperational Resilience in Financial Market Infrastructures
Operational Resilience in Financial Market Infrastructures
 
20 Simple Questions from Exactpro for Your Enjoyment This Holiday Season
20 Simple Questions from Exactpro for Your Enjoyment This Holiday Season20 Simple Questions from Exactpro for Your Enjoyment This Holiday Season
20 Simple Questions from Exactpro for Your Enjoyment This Holiday Season
 
Testing the Intelligence of your AI
Testing the Intelligence of your AITesting the Intelligence of your AI
Testing the Intelligence of your AI
 
EXTENT 2019: Exactpro Quality Assurance for Financial Market Infrastructures
EXTENT 2019: Exactpro Quality Assurance for Financial Market InfrastructuresEXTENT 2019: Exactpro Quality Assurance for Financial Market Infrastructures
EXTENT 2019: Exactpro Quality Assurance for Financial Market Infrastructures
 
EXTENT Talks 2019 Tbilisi: Failover and Recovery Test Automation - Ivan Shamrai
EXTENT Talks 2019 Tbilisi: Failover and Recovery Test Automation - Ivan ShamraiEXTENT Talks 2019 Tbilisi: Failover and Recovery Test Automation - Ivan Shamrai
EXTENT Talks 2019 Tbilisi: Failover and Recovery Test Automation - Ivan Shamrai
 
EXTENT Talks QA Community Tbilisi 20 April 2019 - Conference Open
EXTENT Talks QA Community Tbilisi 20 April 2019 - Conference OpenEXTENT Talks QA Community Tbilisi 20 April 2019 - Conference Open
EXTENT Talks QA Community Tbilisi 20 April 2019 - Conference Open
 
QAFF Chicago 2019 - Complex Post-Trade Systems, Requirements Traceability and...
QAFF Chicago 2019 - Complex Post-Trade Systems, Requirements Traceability and...QAFF Chicago 2019 - Complex Post-Trade Systems, Requirements Traceability and...
QAFF Chicago 2019 - Complex Post-Trade Systems, Requirements Traceability and...
 
QA Community Saratov: Past, Present, Future (2019-02-08)
QA Community Saratov: Past, Present, Future (2019-02-08)QA Community Saratov: Past, Present, Future (2019-02-08)
QA Community Saratov: Past, Present, Future (2019-02-08)
 
Machine Learning and RoboCop Testing
Machine Learning and RoboCop TestingMachine Learning and RoboCop Testing
Machine Learning and RoboCop Testing
 
Behaviour Driven Development: Oltre i limiti del possibile
Behaviour Driven Development: Oltre i limiti del possibileBehaviour Driven Development: Oltre i limiti del possibile
Behaviour Driven Development: Oltre i limiti del possibile
 
2018 - Exactpro Year in Review
2018 - Exactpro Year in Review2018 - Exactpro Year in Review
2018 - Exactpro Year in Review
 
Exactpro Discussion about Joy and Strategy
Exactpro Discussion about Joy and StrategyExactpro Discussion about Joy and Strategy
Exactpro Discussion about Joy and Strategy
 
FIX EMEA Conference 2018 - Post Trade Software Testing Challenges
FIX EMEA Conference 2018 - Post Trade Software Testing ChallengesFIX EMEA Conference 2018 - Post Trade Software Testing Challenges
FIX EMEA Conference 2018 - Post Trade Software Testing Challenges
 
BDD. The Outer Limits. Iosif Itkin at Youcon (in Russian)
BDD. The Outer Limits. Iosif Itkin at Youcon (in Russian)BDD. The Outer Limits. Iosif Itkin at Youcon (in Russian)
BDD. The Outer Limits. Iosif Itkin at Youcon (in Russian)
 
Sibos 2017: Disruptive functional testing - the next frontier in post-trade s...
Sibos 2017: Disruptive functional testing - the next frontier in post-trade s...Sibos 2017: Disruptive functional testing - the next frontier in post-trade s...
Sibos 2017: Disruptive functional testing - the next frontier in post-trade s...
 
Using Cluster Analysis for Characteristics Detection in Software Defect Reports
Using Cluster Analysis for Characteristics Detection in Software Defect ReportsUsing Cluster Analysis for Characteristics Detection in Software Defect Reports
Using Cluster Analysis for Characteristics Detection in Software Defect Reports
 
EXTENT-2017: Testing in Distributed Ledger Systems
EXTENT-2017: Testing in Distributed Ledger SystemsEXTENT-2017: Testing in Distributed Ledger Systems
EXTENT-2017: Testing in Distributed Ledger Systems
 

Recently uploaded

State of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 previewState of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
Prayukth K V
 
Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........
Alison B. Lowndes
 
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdfFIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance
 
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
Product School
 
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Ramesh Iyer
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
Product School
 
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
Product School
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Inflectra
 
Accelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish CachingAccelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish Caching
Thijs Feryn
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Product School
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
Safe Software
 
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
James Anderson
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
RTTS
 
UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4
DianaGray10
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Jeffrey Haguewood
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Tobias Schneck
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Product School
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
Product School
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
Fwdays
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance
 

Recently uploaded (20)

State of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 previewState of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
 
Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........
 
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdfFIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
 
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
 
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
 
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
 
Accelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish CachingAccelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish Caching
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
 
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
 
UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
 

User-Assisted Log Analysis for Quality Control of Distributed Fintech Applications

  • 1. Build Software to Test Software exactpro.com User-Assisted Log Analysis for Quality Control of Distributed Fintech Applications Iosif Itkin1 , Anna Gromova1 , Anton Sitnikov1 , Elena Treshcheva1 , Rostislav Yavorskiy3, Evgenii Tsymbalov2, Andrey Novikov3 and Kirill Rudakov3 1 Exactpro, UK, Georgia, USA, Russia 2 Skolkovo Institute of Science and Technology, Russia 3 Higher School of Economics, Russia
  • 2. 2 Build Software to Test Software exactpro.com Exactpro overview ● A specialist firm focused on functional and non-functional testing of exchanges, clearing houses, depositories, trade repositories and other financial market infrastructures. ● We provide software testing services for mission critical technology that underpins global financial markets. Our clients are regulated by FCA, Bank of England and their counterparts from other countries. ● We were part of the London Stock Exchange Group (LSEG) from May 2015 till January 2018. Exactpro management buyout from LSEG was successfully completed in January 2018. We are headquartered in the UK and have operations in the US, Georgia and Russia. ● Incorporated in 2009 with 10 people, our company has experienced significant growth and is now employing over 550 specialists.
  • 3. 3 Build Software to Test Software exactpro.com We have a global Software Quality Assurance client network
  • 4. 4 Build Software to Test Software exactpro.com Build Software to Test Software
  • 5. 5 Build Software to Test Software exactpro.com SUT: Clearing and Settlement Platform
  • 6. 6 Build Software to Test Software exactpro.com Log usage in passive testing - Receive new error alerts - See alerts on re-appeared old bugs - Know if application behaves differently
  • 7. 7 Build Software to Test Software exactpro.com User-Assisted Log Analysis: Challenges - Massive logs (105 error messages / day is normal) - Log messages for same error types are different as strings. Need to present huge volumes as a human-perceivable collection of error messages
  • 8. 8 Build Software to Test Software exactpro.com Automated Clustering with User Correction
  • 9. 9 Build Software to Test Software exactpro.com - Case is built on 3.2M records volume - Message length – up to 15,000 characters The challenge: Logs classification
  • 10. 10 Build Software to Test Software exactpro.com Message structure
  • 11. 11 Build Software to Test Software exactpro.com Message text
  • 12. 12 Build Software to Test Software exactpro.com Approach 1: Template-Based - For short messages only (no long stack traces, ~80% of messages) - Messages considered equal if differ by < 20% content (3+ char substrings) - Greedy algorithm: start with seed, compare each new message to classes already found
  • 13. 13 Build Software to Test Software exactpro.com - 109 classes identified Approach 1: Template-Based
  • 14. 14 Build Software to Test Software exactpro.com Approach 2: NLP-based - All messages, including long ones - Tokenize, leave only alphabet chars - TF/IDF to identify significant feature set – dimensions for vectorization - Document-term sparse matrix (6,776-dimensions) - Truncate dimensions to 25 (SVD) - K-means assessed with Silhouette Coefficient
  • 15. 15 Build Software to Test Software exactpro.com Approach 2: NLP-based
  • 16. 16 Build Software to Test Software exactpro.com Approaches comparison - Each returned around 100 clusters - Clustering correspond to each other, Rand index around 0,9 - Naïve algorithm is “good enough” for initial classification.
  • 17. 17 Build Software to Test Software exactpro.com P.S. Latest Results: Beyond K-Means Problems of “naïve” approach: - 3.2M records is too much, need preliminary tokenization – from record to “signature” - Loose clusters in K-means output – not single error type in human understanding. Need more dense clustering.
  • 18. 18 Build Software to Test Software exactpro.com Iterate k-means? - Idea – to run k-means again on loose clusters - Did not work. We have loose clusters, and singles on next step.
  • 19. 19 Build Software to Test Software exactpro.com Pipeline: Initial Classification 3.2M records
  • 20. 20 Build Software to Test Software exactpro.com Pipeline: New Logs Processing
  • 21. 21 Build Software to Test Software exactpro.com Improvement Gained - Working with signatures simplifies the process (but requires tough tokenization) - Agglomerative (vs k-means) naturally produces dense clusters - No dimension reduction – better intuition in clustering