Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Automating Test Maintenance as Code and Requirements Change
Similar to Automating Test Maintenance as Code and Requirements Change (20)
Recently uploaded
Recently uploaded (20)
Automating Test Maintenance as Code and Requirements Change
- 1. 0Copyright 2018 – QA Systems GmbH www.qa-systems.com Automating Test Maintenance as Code and Requirements Change Requirements Tests Code
- 2. 1Copyright 2018 – QA Systems GmbH www.qa-systems.com The Test Maintenance Burden Why Maintain Tests? Regression Tests act as safety net against regression errors Return of Investment of tests - increasing marginal returns Structured repeatable quality gate retained Which Dependencies Affect Test Maintenance? Requirements Code Test Environment Quality Policy
- 3. 2Copyright 2018 – QA Systems GmbH www.qa-systems.com Depth of Maintenance Dependencies Code Level of complexity in the code Level of test scenario System – Integration – Unit test Level of test abstraction from the code Black-Box – White-box Test management of function calls Requirements Level of detail in the requirements Level of complexity in the requirements Level of detail in the test cases Common Factors: Granularity of traceability to tests Scope of changes Frequency of changes Requirements Code
- 4. 3Copyright 2018 – QA Systems GmbH www.qa-systems.com Test Dependency Traceability Test Results 2 Test Results 1 Test Results 3 Code Code Code 1.A 1.B 1.C 2.B 2.A 2.C 3.A 3.B 3.C High Level Requirement 1 High Level Requirement 2 Low Level Req 1 Low Level Req 2 Low Level Req 3 Test Script 3 Test Script 2 Test Script 2 Test Script 1
- 5. 4Copyright 2018 – QA Systems GmbH www.qa-systems.com Why use Traceability? Standards Compliance – the 100% Picture Bi-directional requirements traceability All executable code is justified tested Evidence of success is: passing tests + traceability Helps Ensure Completeness Changed requirements capture & validation Just enough code changes Test case design updates RBT processes can be most effective when iterative Code TestReq SOFTWARE CONSIDERATIONS IN AIRBORNE SYSTEMS AND EQUIPMENT CERTIFICAION RTCA DO-178B SOFTWARE CONSIDERATIONS IN AIRBORNE SYSTEMS AND EQUIPMENT CERTIFICAION RTCA
- 6. 5Copyright 2018 – QA Systems GmbH www.qa-systems.com Automating Test Maintenance Manual Test Maintenance is Hard Work ! Insufficiently validated requirements changes High-reliance on code coverage & reverse engineering Inefficiently traced requirements changes What requirements are already traced to existing tests What changes in requirements should trigger assessing existing tests Visibility in both requirements management & test tools Understanding scope of impact of code changes on existing tests Resolving iterative build / run errors Reducing the Test Maintenance Burden through automation Dependency analysis - identification and details of traceability to tests Updating traceability and tests
- 7. 6Copyright 2018 – QA Systems GmbH www.qa-systems.com C O D E Code Change Analysis C O D E T E S T T E S T CODE Easier Test Maintenance Stores last tested state of the SUT Compares to current state of SUT Change impacted test scripts identified Identifies specific code changes in SUT Changes hyper-linked to code editor Suggests options for ‘updating’ the test Displays accepted state of suggestions Implements accepted change options Keep Code & Tests Synchronised New baseline established
- 8. 7Copyright 2018 – QA Systems GmbH www.qa-systems.com Icon Code Item Type of Changes Detected Source Files added | removed | renamed Header Files added | removed | renamed Functions & Methods added | removed signature changed (scope, return type, name & parameters) call changed - which may affect stubs & wrappers body added | removed body changed - which could affect the code coverage Global & Static variables added | removed changed (scope, type & name etc) Changes in Code Detected Configurable Dependency Analysis Automatically on (test script data loading | saved source code change) By user selection on (project | individual test script) Code Changes Detected
- 9. 8Copyright 2018 – QA Systems GmbH www.qa-systems.com Guided Automatic Test Updates Scope of Impacts on Tests Identified Which test scripts are affected Which source files changed Which code items in source changed Suggested Updates Suggestions to update the test Guidance on the effects of accepting the suggestion Test scripts automatically refactored throughout to implement the accepted test updates
- 10. 9Copyright 2018 – QA Systems GmbH www.qa-systems.com Requirements Change Analysis T E S T T E S T REQ R E Q R E QEasier Test Maintenance Stores last imported set of requirements Stores last traced state of requirements Stores last status of tests Presents delta of changes in requirements Automatic copying of traceability for changed requirements sets Requirements detail visible alongside test detail and status Changed data updated & exported Keep Requirements & Tests Synchronised New traced baseline established
- 11. 10Copyright 2018 – QA Systems GmbH www.qa-systems.com How Requirements - Test Traceability Works Bi-Directional Requirements Traceability Imports requirements data to Cantata server Text, images & hyperlinks Drag & drop assignment Controlled export with results status and coverage Integrated with Requirements Tools © All Copyright and Trademarks of their respective owners are acknowledged Requirements Management Tools
- 12. 11Copyright 2018 – QA Systems GmbH www.qa-systems.com Managing Requirements Changes
- 13. 12Copyright 2018 – QA Systems GmbH www.qa-systems.com Selecting Requirements Changes
- 14. 13Copyright 2018 – QA Systems GmbH www.qa-systems.com Updating Tests for Requirements Changes
- 15. 14Copyright 2018 – QA Systems GmbH www.qa-systems.com Automated CI & Regression Testing Continuous Integration Regression Test Automated CI build and test runs via a CLI CI tool integrations Summaries in Jenkins / SonarQube Hyperlinks to more detailed test reporting
- 16. 15Copyright 2018 – QA Systems GmbH www.qa-systems.com Automated Test Maintenance T E S T CODE T E S T REQ As Requirements & Code Change Analyse Dependencies Update Traceability and Tests Make DevOps Efficient
- 17. 16Copyright 2018 – QA Systems GmbH www.qa-systems.com Automated Test Maintenance Automated Unit & Integration Testing Come and see more
Editor's Notes
- Requirements changes Code changes – more code dependent Environment changes (compilers etc) FREQUENCY for req & code Test strategy changes (regulatory, quality policy, automation etc)
- Managers need centralised reporting and individuals may need to collaborate together. [NEXT] Cantata Team Reporting is an add-on to the Cantata tool for Managers, providing a structured folder hierarchy reflecting code function, team structure, projects etc. Data is aggregated for each folder, and its sub-folders, showing current code test status and code coverage metrics with history build by build, to identify trends and plan appropriate actions. Managers get both a complete picture across all test runs, and advanced filtering between test runs. [NEXT] Anyone can access all the summaries and drill-down to details over both the web interface or via a full REST AP.I so test data can be accessed from 3rd party interfaces {Deep dive = picture}
- Managers need centralised reporting and individuals may need to collaborate together. [NEXT] Cantata Team Reporting is an add-on to the Cantata tool for Managers, providing a structured folder hierarchy reflecting code function, team structure, projects etc. Data is aggregated for each folder, and its sub-folders, showing current code test status and code coverage metrics with history build by build, to identify trends and plan appropriate actions. Managers get both a complete picture across all test runs, and advanced filtering between test runs. [NEXT] Anyone can access all the summaries and drill-down to details over both the web interface or via a full REST AP.I so test data can be accessed from 3rd party interfaces {Deep dive = picture}
- Bi-directional requirements traceability is required by all the main safety standards, and for testing this means using your existing tool for everything – NOT a separate solution just for tests. [NEXT – diagram animates in stages] [NEXT] Cantata Trace provides a full featured import of requirements into the Cantata IDE where testers use the data, make and view associations and control export round trip to the requirements tool. It supports all the standard requirements tools, and further tools can be configured. {Deep dive = tool logos}
- Cantata allows users to accelerate their testing to achieve standards compliance. It is independently certified as suitable for use to achieve the highest SIL for all the main functional safety standards by an accredited 3rd party SGS-TÜV GmbH.