Automated DevOps Dynamic Testing for CI and CD

0Copyright 2019 – QA Systems GmbH www.qa-systems.cn
Automated DevOps
Dynamic Testing for CI & CD

Presenter
Matt Davis
Managing Director
QA Systems Ltd

DRIVING EMBEDDED SOFTWARE QUALITYDRIVING EMBEDDED SOFTWARE QUALITY
QA SYSTEMS

QA Systems Company History
1996
Company founded by
Andreas Sczepansky in
Stuttgart
2011
Subsidiary QA Systems Ltd
(Bath, UK) founded
2012
Purchase of Cantata and AdaTEST from IPL
Information Processing Ltd and transfer of
complete engineering team to QA Systems Ltd.
2013
QA Systems France
(Paris) founded
2014
Focus of business on
Static and Dynamic
Testing
2015
QA Systems Italy
(Milan) founded
2018
Founded QA
Systems North
America Inc.
(Boston, MA)

Paris Munich
StuttgartBath
Stuttgart
•Headquarters
Munich
•Office
Bath
•Engineering Centre
Paris
•Office
Milan
•Office
Boston, MA
•Office
Milan
QA Systems Company Locations
Boston,
MA

India
TECHTRUST & Meteonic
Scandinavia
Nohau Solutions
China
Watertek & Hirain Technologies
Japan
Toyo Corp.
APAC / Korea
EACT Technologies / UPS Technologies
Russia
AVD Systems
QA Systems Reseller Network
Taiwan
GCOM Technologies
Tukey
TEKTRONIK

Accelerated Compliance Testing
Certified Unit & Integration Testing

System
Requirements
SW System
Requirements
SW Unit
Requirements
SW Item
Requirements
Code
SW System
Test
System
Test
SW Unit
Test
SW Integration
Test
Code
Verification Order
1 Does code meet the quality standard?
 Static Analysis
2 Does the code do what it should?
Functional Requirements Testing
Non-functional Requirements Testing
3 Does the code not do what it should?
Robustness Testing
4 Has the code been tested enough?
Structural Coverage Testing
Testing in the Software Development Lifecycle
Static
Analysis

Whole Development Organization Benefits
Efficient automation to
“Test Early, Test Often”
Increased confidence in
collaboration
Increased control over
delivery quality and time
Certification ready
proof of quality
Cost of
Quality
Cost of
Quality

Selection of Customers

AutoTest
What is DevOps ?

What is Continuous Integration / Delivery ?

Continuous Integration Flow

Continuous Delivery Flow

Classic CI / CD
Continuous Integration &
Regression Testing
Automated CI build and test runs via a CLI
Cantata make files for CLI test build & run
Classic Groovy scripting integrations for:
Hyperlinks to Cantata Team Reporting

Continuous Integration & Regression Testing
Make files
Works with existing make file systems
Used with existing code tree – avoids
sandboxing
Hierarchy of gmake make files
automatically maintained
Separate target build files
Build & run regression tests on CLI
Customisable with pre-build and post-
build commands for tool extensions e.g.
Source Code Management tools
Helix QAC
Bug Tracking tools

End

DeployStart

Static Analysis

System Test

Unit Test
on host


on target X
on target Y

Integration Test


on host
on target X
on target Y

Build
Modern CI / CD with Pipelines
Pipelines for Continuous Integration & Continuous Delivery
Declarative Scripting for full pipeline CLI automation

Steps in Pipeline Stages
Each Pipeline Stage can automate various Steps
with declarative scripting via CLI
Stage Triggers
Independent or Dependent Steps
Quality Gates
Resulting actions

Declarative & Scripted
Guided scripting for ease of use
Full powerful groovy scripting syntax
for power users
Declarative Pipeline Scripting

Unit Test Pipeline
Cantata static metrics measure non-functional qualities of code.

The Same for Integration Tests

Cantata Reports Linked from Pipelines
Monitoring test progress
Test execution on different variants (HW & SW)
Results Filtering – Additional Data
Trending history displays and Filters

Accelerated Compliance Testing
Certified Unit & Integration Testing

Technical Testing Challenges
CODE
COVERAGETOOL
INTEGRATION
AUTOMATION
CALL
INTERFACE
CONTROL
TRACING
REQUIREMENTS
TO TESTS
TEST
EVIDENCE
REPORT
Certified Solutions
CODE
CHANGES

Target Deployment
Unlimited Target Deployment Capability
Control of on-target Configurations
Synchronise Static analysis & Dynamic testing
DevOps: CI / CD, SCM and Bug Tracking
IDE Integrations
Tool Integrations
Deployment to embedded targets should be straight forward
Integrating into a tool chain should not be a challenge
Integrated with your environment
© All Copyright and Trademarks of their respective owners are acknowledged

Structured Flexible Tests
Unit / Integration
Black-Box / White-Box
Procedural / Object Oriented
Powerful Test Editing
Custom Code Injection
AutoTest
Regression Test Runs
Automatic Test Generation
Flexible Automation
Automatic test framework for unit & integration test
Automatic Test Case generation
Flexible testing styles

AutoTest Generation
Flexible application
 GUI or CLI invocation
 Complete suite of passing unit tests
 Additional test cases to fill gaps
 Black-box cluster integration test through public functions
 White-box unit isolation test of static functions
 Uses Cantata workspace preferences
Test cases exercise all paths through the code
 Entry-Point
 Statement
 Decision
 MC/DC (unique cause)
Test Cases are complete & maintainable for full control
 All required inputs: parameters + accessible data
 All expected outputs: parameters + accessed data + call-order
 Each test case path solving purpose explained

Build RunTest
Exe
Instruments
AutoTest
Makefiles
Tests
Code
AutoTest Process
Code Copy
Generation
Report
Test Results
Automatic Test Generation
Automatic Test Execution

Example AutoTest Exercise
• 541 Source Files
• 807 C Functions
• 55,151 Executable LoC
• 4,901 McCabe total complexity
Source
Files
• 93% Fully executed
• 95% Fully executed
• 95%+ Fully executed
• 5,035 Total test cases
• 336,355 Total checks
Tested
Source
Files
Execution
36 minutes
Generation
2.03 hours

Control of Interfaces
Isolation of units from system
Multiple behaviors for each call
Control to Simulate & Intercept
Automatic generation
Simulated calls (Stubs and Isolates)
Intercept & use called object (Wrappers)
Resolved Link dependencies
Programmable instances for each call
Automated checks on parameters
Flexible call sequence control / test case
Intercept
Simulate

Interface Testing by Type of Test
Integration Test‘Isolation’ Unit Test

Coverage target integrated in tests
Diagnostics over stages / test runs
Coverage redundancy optimization
Simple Standards Compliance
Rule-Sets for safety standards and SILs
Metrics:
Standalone or integrated with Cantata tests
Powerful drill-down views, filters and reports
Automatic test case coverage optimization
Pin-point Analysis
Code Coverage
● Entry- Points ● Statements
● Decisions ● Call Returns
● Relational Operators ● Loops
● Conditions (inc. masking + unique cause MC/DC)

Requirements Traceability
Bi-Directional Requirements Traceability
Imports requirements data to Cantata server
Text, images & hyperlinks
Drag & drop assignment
Controlled export with results
status and coverage
Integrated with Requirements / ALM Tools
Bi-directional requirements traceability
Integrate with existing Requirements Management Tools
Cantata Trace
LLT
HLR
LLR
Code
HL
T
Requirements
Management Tools

Easy Linking in Cantata Trace
Bi-directional drag and drop interface, immediately creates links on a server
Whole Test Scripts linked to Requirements
Individual TestCases linked to Requirements

Evidence Reporting & Collaboration
Management View on Testing Status
Data centrally grouped & stored
Aggregates Test & Coverage Results
History and trends
Advanced filtering
Differentiates tests between runs
Timeframe selection
Team Collaboration
Avoids duplicating over workspaces
Reports & drills-down to details for internal / external sharing
Managers need centralized reporting for maximum control
Teams need to work together
Team Reporting (Add-on)
Web Interface
REST Query

Code Change Analysis
Analysis of code changes and their impact on tests
Automation of changes to code dependent tests
Easier Test Maintenance
Stores last tested state of the SUT
Compares to current state of SUT
Change impacted test scripts identified
in Test Explorer
Identifies specific code changes in SUT
Changes linked to CDT editor
Suggests options for ‘updating’ the test
Displays accepted changes
Implements accepted change options
Keeps Code & Tests Synchronised

Build
Cantata
Library
Run
Source
Code
Called
Objects Other
Libraries
Copy of
Source
Code
Copy of
Source
Code
Test
Script(s)
Options
Test
Exe
Makefiles
Test Execution
Target
Independent
Target
Specific
Instrumented

Connection
• USB
• JTAG
• Serial
• LAN
• Debugger
Pass/Fail
OS code
Host
File
Output
.CTR .CTG .COV
Run
Test
Exe
Target
Console
Summary ASCII
Results
Graphic
Results
Graphic
Coverage
De-Multiplex
Multiplex
Memory Buffer
No File
Output
Multiplex
Test Execution on Target

3 Part Automation
1 Automatic Test Vector Generation
Test case vectors from code exercising all paths (up to MC/DC coverage)
Sets input parameters & data throughout test execution
Checks expected vs actual data, input & output parameters and call order
3 Automated Traceability & Coverage Data Production
Complete Requirements imported/exported for testing
AutoTest cases generated with traceable descriptions
Test status, Requirements traceability & Structural coverage evidence
2 Automated Test Execution
Continuous integration build, run and reporting

DevOps Datasheet
Download our DevOps Datasheet and find
out how Cantata fits into a DevOps
Pipeline
Click the image to go to our website
https://www.qa-
systems.com/resources/?eID=tx_download_ajax&did=203

Initial Passing Pipeline Run

Now Failing Pipeline Run

Re-passing Pipeline Run

Pipeline Logs & Reporting

Learn More - Explore Cantata @ Website

Automated DevOps Dynamic Testing for CI and CD

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Automated DevOps Dynamic Testing for CI and CD

Similar to Automated DevOps Dynamic Testing for CI and CD (20)

Recently uploaded

Recently uploaded (20)

Automated DevOps Dynamic Testing for CI and CD