Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Automated and Scalable Solutions for Software Testing: The Essential Role of Model-Driven Engineering


Published on

MODELS 2018 - invited talk at Industry Day

Published in: Software
  • Be the first to comment

  • Be the first to like this

Automated and Scalable Solutions for Software Testing: The Essential Role of Model-Driven Engineering

  1. 1. .lusoftware verification & validation VVS Automated and Scalable Solutions for Software Testing: The Essential Role of Model-Driven Engineering Lionel C. Briand MODELS 2018, Industry Day
  2. 2. Objectives 2 • Not a scientific presentation, but an experience report • Reflecting on more than 20 years of collaborative research with industry at the intersection of test automation and model-driven engineering • Domains: Automotive, satellite, aerospace, energy, finance, government … • Main message: Modeling is an essential component of many test automation solutions in numerous industrial contexts
  3. 3. Software Testing 3 SW Representation (e.g., specifications) SW Code Derive Test cases Execute Test cases Compare Expected Results or properties Get Test Results Test Oracle [Test Result==Oracle][Test Result!=Oracle] • Automation! • Context • Trade-off • White-box vs. black-box • Sampling strategy • Constraint solving • Expected results • General properties • Observability • Trace analysis • Repeatability • Controllability • Early testing
  4. 4. Research Project Examples
  5. 5. Cyber-Physical Systems (CPSs) 5
  6. 6. Testing Cyber-physical Systems: Challenges 6 complex systems interacting with complex environment hybrid behavior intertwining of software and hardware complex requirements uncertainty
  7. 7. CPS Development Process 7
  8. 8. Simulink vs. SysML • Two commonly used, industry-strength and complementary modeling languages for CPS • Simulink • Functional and algorithmic computations (MiL-level) • E.g., controllers, plant and environment models • SysML • Software architecture modeling (SiL-level) plus aspects of hardware • E.g., State machines integrating different controllers, real-time behaviors 8
  9. 9. Model Testing
  10. 10. Context 10 Attitude Determination and Control System (ADCS) Earth
  11. 11. CPS Development Process 11
  12. 12. Model Testing 12 • Testing the implemented system on the deployment platform is necessarily limited • Shift the bulk of testing from implemented systems to models of such systems and their environments: • Enables early testing of CPSs (in the design space) • Is Scalable • Can execute a large number of test scenarios (simulations) • Efficiently explore the design space • Guides the testing process at later stages by identifying high-risk test scenarios
  13. 13. Requirements • Efficient model execution • To execute thousands of test cases within practical time • No user intervention • Co-Simulation of heterogeneous models • Software (SysML) and function (Simulink) models • Controllability • To emulate, as precisely as possible, the expected runtime behavior of the future deployed system and its environment • Observability • To enable automated failure detection based on temporal and timing properties 13
  14. 14. Our Approach • A modeling methodology • To specify testable models of a CPS (relying on SysML) • To integrate software models with function (Simulink) models capturing hardware and environment • A co-simulation framework • To execute software and function models in a synchronized way • To enable the generation of adequate execution traces (oracles, guiding test generation) • Entirely generated from models 14
  15. 15. CPS Simulation & Test 15 Test inputs CPS SysML Models CPS Function Models Integrates Modeling Co-Simulation Matlab Runtime C Code Code Generation Executes Execution Traces Calls Data Test input: - initial state of satellite (speed, attitude, orbit, date), - a sequence of time-stamped external events (tele-commands)
  16. 16. 16 Requirements-Driven Testing
  17. 17. Context International Electronics & Engineering (IEE) IEE develops real-time embedded systems: • Automotive safety sensing systems, e.g., seat occupancy status • Automotive comfort & convenience systems, 17
  18. 18. Traceability 18 • In many domains, various types of traceability are required. • For example, in automotive (ISO 26262), traceability between requirements and system tests: requirements-driven testing. • Many requirements, many tests, therefore many traces … • Automation is required.
  19. 19. Objectives • Support generation test cases from requirements • Capture and create traceability information between test cases and requirements • Requirements are captured through use cases • Use cases are used to communicate with customers and the system test team • Complete and precise behavioral models are not an option: too difficult and expensive (no model-based testing) 19
  20. 20. Strategy • Analyzable use case specifications • Automatically extract test model from the use case specifications using Natural Language Processing • Minimize modeling, domain modeling only • No behavioral modeling 20
  21. 21. Errors.size() == 0 Status != null t > 0 && t < 50 Constraints Domain Model Test Cases Test Scenarios 21 THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND Use Cases UMTG Based on Natural Language Processing Traceability
  22. 22. • RUCM is based on a (1) template, (2) restriction rules, and (3) specific keywords constraining the use of natural language in use case specifications • RUCM reduces ambiguity and facilitates automated analysis of use cases • Conformance is supported by a tool based on NLP • Trade-off between analyzability and natural language Restricted Use Case Modeling: RUCM 22
  23. 23. RUCM Use Case Name: Identify Occupancy Status Actors: AirbagControlUnit Precondition: The system has been initialized . . . Basic Flow 1. The seat SENDS occupancy status TO the system. 2. INCLUDE USE CASE Classify occupancy status. 3. The system VALIDATES THAT the occupant class for airbag control is valid. 4. The system SENDS the occupant class for airbag control TO AirbagControlUnit. Specific Alternative Flow RFS 3 1. IF the occupant class for airbag control is not valid THEN Postcondition: The occupant class for airbag control has been sent. Postcondition: The previous occupant class for airbag control has been sent. [Yue et al. TOSEM’13]
  24. 24. 24 Basic Flow 1. The seat SENDS occupancy status TO the system. 2. INCLUDE USE CASE Classify occupancy status. 3. The system VALIDATES THAT the occupant class for airbag control is valid and the occupant class for seat belt reminder is valid. 4. The system SENDS the occupant class for airbag control TO AirbagControlUnit. 5. The system SENDS the occupant class for seat belt reminder TO SeatBeltControlUnit. 6. The System Waits for next execution cycle. Postcondition: The occupant class for airbag control and the occupant class for seat belt reminder have been sent. INPUT STEP INCLUDE STEP CONDITIONAL STEP OUTPUT STEP OUTPUT STEP INTERNAL STEP POSTCONDITION DOMAIN ENTITY CONSTRAINT CONSTRAINT DOMAIN ENTITY DOMAIN ENTITY
  25. 25. Case Study • BodySense, embedded system for detecting occupancy status in a car • Evaluation: • Cost of additional modelling (Constraints) • Effectiveness in terms of covered scenarios compared to current practice at IEE • Keep in mind changes and repeated testing 25
  26. 26. Costs of Additional Modeling 26 Use Case Steps Use Case Flows OCL Constraints UC1 50 8 9 UC2 44 13 7 UC3 35 8 8 UC4 59 11 12 UC5 30 8 5 UC6 25 6 12 5 to 10 minutes to write each constraints => A maximum of 10 hours in total
  27. 27. Effectiveness: scenarios covered 27 0 5 10 15 20 25 30 35 40 UC1 UC2 UC3 UC4 UC5 UC6 Scenarios Covered By Engineer Scenarios Covered By UMTG 100% 100% 100% 100% 100% 100% 81% 77% 100% 86% 50% 67% It is hard for engineers to capture all the possible scenarios involving error conditions.
  28. 28. Generating OCL Constraints 28 • Constraints may be a challenge in practice • NLP: Semantic Role Labeling • Determine the role of words in a sentence (e.g., affected actor) • Match words with corresponding concepts in the domain model • Generate an OCL formula based on patterns
  29. 29. Semantic Role Labeling (SRL) “no error has been detected” Error.allInstances()->forAll( i | i.isDetected = false) A1 “The system detects temperature errors TemperatureError.allInstances()->forAll( i | i.isDetected = true) A1 A1A0 A1: actor that is affected by the activity described in a sentence A0: actor that performs an activity A1 verb verb verb verb
  30. 30. Empirical Evaluation • Case study: BodySense, embedded system for detecting occupancy status in a car • Evaluation: • Automatically generate the OCL constraints required to automatically derive executable test cases • Automatically generate executable test cases 30
  31. 31. OCL generation: Precision and Recall • 88 OCL constraints to be generated • OCLGen generates 69 constraints • 66 correct, only 3 incorrect • Very high precision • High Recall 31 # Correctly generated constraints # Generated constraints =precision = = 0.97 66 69 # Correctly generated constraints # Constraints to be generated =recall = = 0.7566 88
  32. 32. Results: Limiting Factors • Imprecise specifications • “The system VALIDATES THAT the temperature is valid“ • Inconsistent terminology • “The system VALIDATES THAT the occupancy status is valid“ BodySense.allInstances()->forAll( i | i.temperature < 200 ) BodySense.allInstances()->forAll( i | i.occupancyStatus <> Empty ) 32
  33. 33. Schedulability Analysis and Stress Testing 33
  34. 34. Case Study 34 Drivers (Software-Hardware Interface) Control Modules Alarm Devices (Hardware) Multicore Architecture Real-Time Operating System System monitors gas leaks and fire in oil extraction platforms
  35. 35. Problem and Context • Schedulability analysis encompasses techniques that try to predict whether (critical) tasks are schedulable, i.e., meet their deadlines • Stress testing runs carefully selected test cases that have a high probability of leading to deadline misses • Stress testing is complementary to schedulability analysis • Testing is typically expensive, e.g., hardware in the loop • Finding stress test cases is difficult 35
  36. 36. Finding Stress Test Cases is Hard 36 0 1 2 3 4 5 6 7 8 9 j0, j1 , j2 arrive at at0 , at1 , at2 and must finish before dl0 , dl1 , dl2 J1 can miss its deadline dl1 depending on when at2 occurs! 0 1 2 3 4 5 6 7 8 9 j0 j1 j2 j0 j1 j2 at0 dl0 dl1 at1 dl2 at2 T T at0 dl0 dl1 at1 at2 dl2
  37. 37. Challenges and Solutions • Ranges for arrival times of all tasks form a very large input space • Task interdependencies and properties constrain what parts of the space are feasible • Task architecture captured by UML MARTE profile. • Solution: We re-expressed the problem as a constraint optimization problem and used a combination of constraint programming (IBM CPLEX) and meta-heuristic search (GA) 37
  38. 38. Constraint Optimization 38 Constraint Optimization Problem Static Properties of Tasks (Constants) Dynamic Properties of Tasks (Variables) Performance Requirement (Objective Function) OS Scheduler Behaviour (Constraints)
  39. 39. Solution Overview 39 UML Modeling (e.g., MARTE) Constraint Optimization Optimization Problem (Find arrival times that maximize the chance of deadline misses) System Platform Solutions (Task arrival times likely to lead to deadline misses) Deadline Misses Analysis System Design Design Model (Time and Concurrency Information) INPUT OUTPUT Stress Test Cases Constraint Prog. and Genetic Algorithms
  40. 40. Combining CP and GA 40 A:12 S. Di Alesio et al. Fig. 3: Overview of GA+CP: the solutions x , y and z in the initial population of GA evolve into
  41. 41. Summary • We provided a solution for generating stress test cases by combining meta-heuristic search and constraint programming • Meta-heuristic search (GA) identifies high risk regions in the input space • Constraint programming (CP) finds provably worst-case schedules within these (limited) regions • Achieve (nearly) GA efficiency and CP effectiveness • Our approach can be used both for stress testing and schedulability analysis (assumption free) 41
  42. 42. Beyond CP Systems • Models play a role in test automation in other domains than cyber-physical systems 42 Bridging the gap between requirements models (business processes) and Behavior-Driven Development
  43. 43. Conclusions 43
  44. 44. The Role of Models • No effective and scalable test automation is possible, in many contexts, without models: Guiding test generation, generating oracles • Requirements (e.g., use case specifications) • Architecture (e.g., task properties and dependencies) • Behavior of system and environment (e.g., state and timing properties) • Business processes • Diverse domains: Cyber-physical systems, finance, e-government … 44
  45. 45. Models are not Enough • Models are not enough for test automation • Optimization and metaheuristic search • Natural language processing • Simulation • Constraint solving • … • Test automation solutions are necessarily multidisciplinary 45
  46. 46. Modeling is a Trade-off • Modeling is costly and a challenge to many • Effective tool support, e.g., QA, is important • Upfront investment in tooling and training • Benefits can mostly be obtained from exploiting models for automation of costly and error-prone tasks, e.g., testing in the large • Scope and depth of modeling is a trade-off • This trade-off is driven by contextual factors 46
  47. 47. Open Problems • Testing cannot be exhaustive: Strategies to explore the test input space • Metaheuristic search, efficient and effective constraint solving … • Early testing: How to find significant problems in early artifacts, e.g., requirements, architecture • Executable models, co-simulation … • Oracles: Striking the right balance between cost and effectiveness • Scalable trace analysis, DSL for expressing trace properties, probabilistic oracles for non-deterministic systems … 47
  48. 48. Industry-Academia Collaborations • Crucial to address relevant and open problems, develop scalable and applicable solutions. 48 Adapted from [Gorschek et al. 2006] Problem Formulation Problem Identification State of the Art Review Candidate Solution(s) Initial Validation Training Realistic Validation Industry Partners Research Groups 1 2 3 4 5 7 Solution Release 8 6
  49. 49. 49
  50. 50. Acknowledgments • Shiva Nejati • Fabrizio Pastore • Chunhui Wang • Stefano Di Alesio • … 50
  51. 51. Selected References • L. Briand et al. “Testing the untestable: Model testing of complex software-intensive systems”, IEEE/ACM ICSE 2016, V2025 • C. González et al., “Model Testing of Cyber-Physical Systems”, ACM/IEEE MODELS 2018 • C. Wang et al., “Automatic Generation of System Test Cases from Use Case Specifications”, ACM ISSTA 2017 • C. Wang et al., “Automated Generation of Constraints from Use Case Specifications to Support System Testing”, IEEE ICST 2018 • S. Di Alesio et al. “Combining genetic algorithms and constraint programming to support stress testing of task deadlines”, ACM Transactions on Software Engineering and Methodology (TOSEM), 2015 • More on: 51
  52. 52. .lusoftware verification & validation VVS Automated and Scalable Solutions for Software Testing: The Essential Role of Model-Driven Engineering Lionel C. Briand MODELS 2018