This document summarizes a thesis on automating test routine creation through natural language processing. The author proposes using word embeddings and recommender systems to automatically generate test cases from requirements documents and link them together. The methodology involves representing text as word vectors, calculating similarity between requirements and test blocks, and applying association rule mining on test block sequences. An experiment on a space operations dataset showed the approach improved productivity in test creation and requirements tracing over manual methods. Future work could explore using deep learning models and collecting additional evaluation metrics from users.

1. Test Routine Automation through Natural
Language Processing Techniques
Voulivasi Triantafyllia
Guided by:
A. Symeonidis, Aristotle University of Thessaloniki
E. Gómez, European Space Operations Centre
11 July 2017
ARISTOTLE UNIVERSITY OF THESSALONIKI
Faculty of Engineering
Department of Electrical and Computer Engineering
Information Processing Laboratory

2. 1. Introduction
2. Outcomes
3. Background
4. Methodology
5. Experiments
6. Conclusion & Future Work
AGENDA

3. 1. Introduction
2. Outcomes
3. Background
4. Methodology
5. Experiments
6. Conclusion & Future Work
AGENDA

4. Introduction
● Ground Segment is a SoS
(System of Systems)
● Testing is complicated
● Requires well-established testing
infrastructure

5. Objectives
1. Automation in the creation of tests
2. Requirements Tracing

6. Objectives
1. Automation in the creation of tests
● Time-consuming procedure: Senior SE → Tester
● Difficult to work with for new testers
2. Requirements Tracing

7. Objectives
1. Automation in the creation of tests
● Time-consuming procedure: Senior SE → Tester
● Difficult to work with for new testers
2. Requirements Tracing
● Tests originate from SPRs (Software Problem Report) instead of
requirements
● Lack of consistency in evaluation of software requirements

8. Ground Segment Testing Infrastructure

9. Ground Segment Testing Infrastructure

10. Ground Segment Testing Infrastructure

11. Ground Segment Testing Infrastructure

12. Test Scenario

13. Test Scenario
Natural Language

14. Automated test
Sequence of test building blocks

15. Automated test
Sequence of test building blocks
Test Block Attributes:
● Name
● Description
● Parameters
● Precondition
● Postcondition

16. Automated test
Sequence of test building blocks
Test Block Attributes:
● Name
● Description
● Parameters
● Precondition
● Postcondition
Natural Language

17. Software Requirement
“The SCOS-2000 environment should warn for
any parameters with Out-Of-Limit values.”

18. Software Requirement
“The SCOS-2000 environment should warn for
any parameters with Out-Of-Limit values.”
Natural Language

19. 1. Introduction
2. Outcomes
3. Background
4. Methodology
5. Experiments
6. Conclusion & Future Work
AGENDA

22. High-Level System Design

23. High-Level System Design
Information Retrieval
Natural Language Processing
Word Embeddings
Recommender Systems
Association Rules

24. 1. Introduction
2. Outcomes
3. Background
4. Methodology
5. Experiments
6. Conclusion & Future Work
AGENDA

25. Natural Language Representation
?Natural
Language
Machine
Representation

26. Natural Language Representation
Vector Space
Model
Natural
Language
Machine
Representation

27. Natural Language Representation
Vector Space
Model
Natural
Language
Machine
Representation
capital 195

28. Natural Language Representation
Vector Space
Model
Natural
Language
Machine
Representation
One-hot encoding
● Does not capture semantics
● Huge Length
-- equal to the size of the total unique
vocabulary in the corpora
capital 195

29. Natural Language Representation
Vector Space
Model
Natural
Language
Machine
Representation
One-hot encoding
● Does not capture semantics
● Huge Length
-- equal to the size of the total unique
vocabulary in the corpora
capital 195

30. Word Embeddings
● state-of-the-art word embedding methods: Word2Vec, Glove and FastText changed
completely NLP
● reduce dimensionality
● capture semantics
Word2Vec vector:
[0.12, 0.23, 056]

31. Word2vec: simplified structure
● Shallow feed-forward neural network with one hidden layer and linear activation function
● Input and output are hot-encoded vectors of pairs of words: drink | juice, New | York
● The word vectors are referring to the first (left) weight matrix

32. Word2vec: simplified structure
● Shallow feed-forward neural network with one hidden layer and linear activation function
● Input and output are hot-encoded vectors of pairs of words: drink | juice, New | York
● The word vectors are referring to the first (left) weight matrix

33. Word2Vec architectures
Skip-gram CBOW

34. Word2Vec architectures

35. Word2Vec Parameters
● size A value of 100 - 1000 for the dimension of the hidden layer
● window The maximum distance between the target word and a
neighbor word
● min_count Minimum frequency count of words
● workers How many threads to use behind the scenes?
● sg Whether to use skip-gram or CBOW architecture
● negative Whether to use negative sampling
● corpus relevant documents

36. Word2Vec Parameters
● size A value of 100 - 1000 for the dimension of the hidden layer
● window The maximum distance between the target word and a
neighbor word
● min_count Minimum frequency count of words
● workers How many threads to use behind the scenes?
● sg Whether to use skip-gram or CBOW architecture
● negative Whether to use negative sampling
● corpus relevant documents

37. Word2Vec Parameters
● size A value of 100 - 1000 for the dimension of the hidden layer
● window The maximum distance between the target word and a
neighbor word
● min_count Minimum frequency count of words
● workers How many threads to use behind the scenes?
● sg Whether to use skip-gram or CBOW architecture
● negative Whether to use negative sampling
● corpus relevant documents

38. Word2Vec Corpus: Software Documentation
● Glossary ● Software Problem Report
● Technical Notes ● Software Design Document
● Software Development Plan ● Kick Off Meeting Minutes
● Software User Manual ● Final Report
● Software Requirements Specification ● Software Validation Specification
● Software Unit and Integration Test
Plan
● Configuration and Installation Guide

39. Text Similarity
Consider sentences A, B:
A contains words:
B contains words:

40. Text Similarity
sentence representations:
Consider sentences A, B:
A contains words:
B contains words:

41. Text Similarity
sentence representations:
Consider sentences A, B:
A contains words:
B contains words:

42. Text Similarity: other approaches
Jaccard Index
TF-IDF
LSI

43. 1. Introduction
2. Outcomes
3. Background
4. Methodology
5. Experiments
6. Conclusion & Future Work
AGENDA

44. High-Level System Design

47. Spell Checker

48. Spell Checker
Levenshtein distance (LD)
s = "test"
t = "test" → LD(s,t) = 0
no transformations are needed
s = "test"
t = "tent" → LD(s,t) = 1
one substitution transforms s into t

49. Presenter
● Present the processed information to the
UI
● Communicate with the other components
to trigger data processing in the system
Presenter & Parser
Parser
● Identify how many blocks match to a
sentence
● Identify a test step’s category
1. Informative
“Open the Manual Stack and disable
dynamic PTV checks.” → 2 test blocks
2. Repetitive
“Repeat steps 1 to 4.” → 4 test blocks

50. NLP Filter

51. Recommender
1. Compute similarities:
2. Improve recommendations from user feedback:

52. Recommender - Keywords
step keywords:
block keywords:

53. Recommender - Parameters
Step Parameters Block
Parameters
Parameter
Score
a, b, c a, b, c 1.0
- a, b, c 0.0
a, b, c - 0.0
a, b, c a, c 0.6667

54. Recommender - Association Rules Mining
● Re-ranking based on Association Rules Mining
Itemsets of interest: the previous blocks and the
block in question
● Calculate all those itemsets and the number of their
occurences together σ: support count
● Calculate the confidence scores c of X → Y
○ X : the previous blocks
○ Y : each test block in question
ID Items
1 {1, 2}
2 {1, 2, 3, 4}
3 {1, 2, 3 5}

55. Recommender - Association Rules Mining
● Re-ranking based on Association Rules Mining
Itemsets of interest: the previous blocks and the
block in question
● Calculate all those itemsets and the number of their
occurences together σ: support count
● Calculate the confidence scores c of X → Y
○ X : the previous blocks
○ Y : each test block in question
ID Items
1 {1, 2}
2 {1, 2, 3, 4}
3 {1, 2, 3 5}

56. Flow Checker & Data Container
Flow Checker Data Container
● data import
● data export
● data distribution to other components
(i.e. Repetitive category test step)

57. 1. Introduction
2. Outcomes
3. Background
4. Methodology
5. Experiments
6. Conclusion & Future Work
AGENDA

58. Dataset
Word2Vec Training Corpus
ESA Mission Control System Infrastructure
● Files: ~200
● Requirements: 5569
● Test Scenarios: 5040
Word counts: 3.006.330
Word embeddings: 31580
Testing Dataset
● Requirements: 5569
● Test Blocks
○ Libraries: 21
○ Extracted Test Blocks: 2160
○ Filtered - High Level Test Blocks: 685
● Test Scenarios
○ Automated by a test engineer: 8
○ Total Test Steps: 181
○ Associated Test Blocks: 260
○ Linked Requirements: 36

59. Evaluation Measures - Precision And Recall

60. Evaluation Measures

61. Word2Vec Models

62. Vector Space Models
Test Blocks Requirements

63. User Feedback

64. Efficiency and Productivity
78.8%
Automated Test
Creation
57.4%
Requirements
Tracing
<0.1 sec
Automated Test
Creation
<1 sec
Requirements
Tracing

65. 1. Introduction
2. Outcomes
3. Background
4. Methodology
5. Experiments
6. Conclusion & Future Work
AGENDA

66. ● This work is the first step towards AI of SWE data in ESOC
● Retrieve software documentation information
● Increase productivity of the Testing team
● Hidden purpose: gather labeled data
Conclusion

67. ● This work is the first step towards AI of SWE data in ESOC
● Retrieve software documentation information
● Increase productivity of the Testing team
● Hidden purpose: gather labeled data
Future Work
● Use of a Deep Learning Model for recommendations
● Embed pre-trained Word vectors
● On-site experiments in time and effort
● Incorporate Software Testing metrics
Conclusion

68. Thanks to:
● Assoc. Professor Andreas Symeonidis
● Eduardo Gómez
● ESOC Data Analytics team
● ISSEL Labgroup

69. Thank you for your attention.
Questions?