Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Triantafyllia Voulibasi

79 views

Published on

Test Routine Automation through Natural Language Processing Techniques

Published in: Education
  • Be the first to comment

  • Be the first to like this

Triantafyllia Voulibasi

  1. 1. Test Routine Automation through Natural Language Processing Techniques Voulivasi Triantafyllia Guided by: A. Symeonidis, Aristotle University of Thessaloniki E. Gómez, European Space Operations Centre 11 July 2017 ARISTOTLE UNIVERSITY OF THESSALONIKI Faculty of Engineering Department of Electrical and Computer Engineering Information Processing Laboratory
  2. 2. 1. Introduction 2. Outcomes 3. Background 4. Methodology 5. Experiments 6. Conclusion & Future Work AGENDA
  3. 3. 1. Introduction 2. Outcomes 3. Background 4. Methodology 5. Experiments 6. Conclusion & Future Work AGENDA
  4. 4. Introduction ● Ground Segment is a SoS (System of Systems) ● Testing is complicated ● Requires well-established testing infrastructure
  5. 5. Objectives 1. Automation in the creation of tests 2. Requirements Tracing
  6. 6. Objectives 1. Automation in the creation of tests ● Time-consuming procedure: Senior SE → Tester ● Difficult to work with for new testers 2. Requirements Tracing
  7. 7. Objectives 1. Automation in the creation of tests ● Time-consuming procedure: Senior SE → Tester ● Difficult to work with for new testers 2. Requirements Tracing ● Tests originate from SPRs (Software Problem Report) instead of requirements ● Lack of consistency in evaluation of software requirements
  8. 8. Ground Segment Testing Infrastructure
  9. 9. Ground Segment Testing Infrastructure
  10. 10. Ground Segment Testing Infrastructure
  11. 11. Ground Segment Testing Infrastructure
  12. 12. Test Scenario
  13. 13. Test Scenario Natural Language
  14. 14. Automated test Sequence of test building blocks
  15. 15. Automated test Sequence of test building blocks Test Block Attributes: ● Name ● Description ● Parameters ● Precondition ● Postcondition
  16. 16. Automated test Sequence of test building blocks Test Block Attributes: ● Name ● Description ● Parameters ● Precondition ● Postcondition Natural Language
  17. 17. Software Requirement “The SCOS-2000 environment should warn for any parameters with Out-Of-Limit values.”
  18. 18. Software Requirement “The SCOS-2000 environment should warn for any parameters with Out-Of-Limit values.” Natural Language
  19. 19. 1. Introduction 2. Outcomes 3. Background 4. Methodology 5. Experiments 6. Conclusion & Future Work AGENDA
  20. 20. High-Level System Design
  21. 21. High-Level System Design Information Retrieval Natural Language Processing Word Embeddings Recommender Systems Association Rules
  22. 22. 1. Introduction 2. Outcomes 3. Background 4. Methodology 5. Experiments 6. Conclusion & Future Work AGENDA
  23. 23. Natural Language Representation ?Natural Language Machine Representation
  24. 24. Natural Language Representation Vector Space Model Natural Language Machine Representation
  25. 25. Natural Language Representation Vector Space Model Natural Language Machine Representation capital 195
  26. 26. Natural Language Representation Vector Space Model Natural Language Machine Representation One-hot encoding ● Does not capture semantics ● Huge Length -- equal to the size of the total unique vocabulary in the corpora capital 195
  27. 27. Natural Language Representation Vector Space Model Natural Language Machine Representation One-hot encoding ● Does not capture semantics ● Huge Length -- equal to the size of the total unique vocabulary in the corpora capital 195
  28. 28. Word Embeddings ● state-of-the-art word embedding methods: Word2Vec, Glove and FastText changed completely NLP ● reduce dimensionality ● capture semantics Word2Vec vector: [0.12, 0.23, 056]
  29. 29. Word2vec: simplified structure ● Shallow feed-forward neural network with one hidden layer and linear activation function ● Input and output are hot-encoded vectors of pairs of words: drink | juice, New | York ● The word vectors are referring to the first (left) weight matrix
  30. 30. Word2vec: simplified structure ● Shallow feed-forward neural network with one hidden layer and linear activation function ● Input and output are hot-encoded vectors of pairs of words: drink | juice, New | York ● The word vectors are referring to the first (left) weight matrix
  31. 31. Word2Vec architectures Skip-gram CBOW
  32. 32. Word2Vec architectures
  33. 33. Word2Vec Parameters ● size A value of 100 - 1000 for the dimension of the hidden layer ● window The maximum distance between the target word and a neighbor word ● min_count Minimum frequency count of words ● workers How many threads to use behind the scenes? ● sg Whether to use skip-gram or CBOW architecture ● negative Whether to use negative sampling ● corpus relevant documents
  34. 34. Word2Vec Parameters ● size A value of 100 - 1000 for the dimension of the hidden layer ● window The maximum distance between the target word and a neighbor word ● min_count Minimum frequency count of words ● workers How many threads to use behind the scenes? ● sg Whether to use skip-gram or CBOW architecture ● negative Whether to use negative sampling ● corpus relevant documents
  35. 35. Word2Vec Parameters ● size A value of 100 - 1000 for the dimension of the hidden layer ● window The maximum distance between the target word and a neighbor word ● min_count Minimum frequency count of words ● workers How many threads to use behind the scenes? ● sg Whether to use skip-gram or CBOW architecture ● negative Whether to use negative sampling ● corpus relevant documents
  36. 36. Word2Vec Corpus: Software Documentation ● Glossary ● Software Problem Report ● Technical Notes ● Software Design Document ● Software Development Plan ● Kick Off Meeting Minutes ● Software User Manual ● Final Report ● Software Requirements Specification ● Software Validation Specification ● Software Unit and Integration Test Plan ● Configuration and Installation Guide
  37. 37. Text Similarity Consider sentences A, B: A contains words: B contains words:
  38. 38. Text Similarity sentence representations: Consider sentences A, B: A contains words: B contains words:
  39. 39. Text Similarity sentence representations: Consider sentences A, B: A contains words: B contains words:
  40. 40. Text Similarity: other approaches Jaccard Index TF-IDF LSI
  41. 41. 1. Introduction 2. Outcomes 3. Background 4. Methodology 5. Experiments 6. Conclusion & Future Work AGENDA
  42. 42. High-Level System Design
  43. 43. Spell Checker
  44. 44. Spell Checker Levenshtein distance (LD) s = "test" t = "test" → LD(s,t) = 0 no transformations are needed s = "test" t = "tent" → LD(s,t) = 1 one substitution transforms s into t
  45. 45. Presenter ● Present the processed information to the UI ● Communicate with the other components to trigger data processing in the system Presenter & Parser Parser ● Identify how many blocks match to a sentence ● Identify a test step’s category 1. Informative “Open the Manual Stack and disable dynamic PTV checks.” → 2 test blocks 2. Repetitive “Repeat steps 1 to 4.” → 4 test blocks
  46. 46. NLP Filter
  47. 47. Recommender 1. Compute similarities: 2. Improve recommendations from user feedback:
  48. 48. Recommender - Keywords step keywords: block keywords:
  49. 49. Recommender - Parameters Step Parameters Block Parameters Parameter Score a, b, c a, b, c 1.0 - a, b, c 0.0 a, b, c - 0.0 a, b, c a, c 0.6667
  50. 50. Recommender - Association Rules Mining ● Re-ranking based on Association Rules Mining Itemsets of interest: the previous blocks and the block in question ● Calculate all those itemsets and the number of their occurences together σ: support count ● Calculate the confidence scores c of X → Y ○ X : the previous blocks ○ Y : each test block in question ID Items 1 {1, 2} 2 {1, 2, 3, 4} 3 {1, 2, 3 5}
  51. 51. Recommender - Association Rules Mining ● Re-ranking based on Association Rules Mining Itemsets of interest: the previous blocks and the block in question ● Calculate all those itemsets and the number of their occurences together σ: support count ● Calculate the confidence scores c of X → Y ○ X : the previous blocks ○ Y : each test block in question ID Items 1 {1, 2} 2 {1, 2, 3, 4} 3 {1, 2, 3 5}
  52. 52. Flow Checker & Data Container Flow Checker Data Container ● data import ● data export ● data distribution to other components (i.e. Repetitive category test step)
  53. 53. 1. Introduction 2. Outcomes 3. Background 4. Methodology 5. Experiments 6. Conclusion & Future Work AGENDA
  54. 54. Dataset Word2Vec Training Corpus ESA Mission Control System Infrastructure ● Files: ~200 ● Requirements: 5569 ● Test Scenarios: 5040 Word counts: 3.006.330 Word embeddings: 31580 Testing Dataset ● Requirements: 5569 ● Test Blocks ○ Libraries: 21 ○ Extracted Test Blocks: 2160 ○ Filtered - High Level Test Blocks: 685 ● Test Scenarios ○ Automated by a test engineer: 8 ○ Total Test Steps: 181 ○ Associated Test Blocks: 260 ○ Linked Requirements: 36
  55. 55. Evaluation Measures - Precision And Recall
  56. 56. Evaluation Measures
  57. 57. Word2Vec Models
  58. 58. Vector Space Models Test Blocks Requirements
  59. 59. User Feedback
  60. 60. Efficiency and Productivity 78.8% Automated Test Creation 57.4% Requirements Tracing <0.1 sec Automated Test Creation <1 sec Requirements Tracing
  61. 61. 1. Introduction 2. Outcomes 3. Background 4. Methodology 5. Experiments 6. Conclusion & Future Work AGENDA
  62. 62. ● This work is the first step towards AI of SWE data in ESOC ● Retrieve software documentation information ● Increase productivity of the Testing team ● Hidden purpose: gather labeled data Conclusion
  63. 63. ● This work is the first step towards AI of SWE data in ESOC ● Retrieve software documentation information ● Increase productivity of the Testing team ● Hidden purpose: gather labeled data Future Work ● Use of a Deep Learning Model for recommendations ● Embed pre-trained Word vectors ● On-site experiments in time and effort ● Incorporate Software Testing metrics Conclusion
  64. 64. Thanks to: ● Assoc. Professor Andreas Symeonidis ● Eduardo Gómez ● ESOC Data Analytics team ● ISSEL Labgroup
  65. 65. Thank you for your attention. Questions?

×