nterest in Continuous Testing has been growing for 5 years now—yet the more we talk about it, the more polarized the discussion becomes. Complicating the conversation is the fact that Agile and DevOps are both driving the need for Continuous Testing, but both require distinctly different things from a quality perspective.
Join me for a lively discussion on what’s really required for Continuous Testing in the context of Agile and DevOps. Join Eran Kinsbruner, author of Continuous Testing for DevOps Professionals, as he explores:
How DevOps and Agile change the game for testing
Which elements of Continuous Testing are absolutely essential for Agile and DevOps
The top myths, misconceptions, and mistakes surrounding Continuous Testing
Strategies for measuring Continuous Testing progress and ROI
6. There Are Patterns for “Unstable” Test Automation
80% of issues have a pattern52% success rate
10% of devices,
causing 80% of lab
issues
Lab
25%
Orches
tration
25%
Scripts
& FW
50%
FAILURE REASON
Objects Codding Time Other
Scripts & FW issues
Device in use
No Device
Orchestration issues
Networking Stability Lock
Other
Lab issues
What’s
wrong
With my
Scripts
What’s wrong
With my Lab
What’s wrong
With my
Executions
10. 1. What’s the test engineer’s gut feeling 😊
2. Risk calculated as probability to occur and
impact to customers
3. Value – does the test provide new
information and, if failed, how much time to
fix?
4. Cost efficiency to develop – how long does
it take to develop and how easy is it to
script?
5. History of test – volume of historical
failures in related areas and frequency of
breaks
Source: Angie Jones
11. Insights into the CI
Pipeline
Risk/Focus Area
Mapping
Summary Report
List
Single Test Report
Visual Validations
Noise reduction through
error/failure-classification
12. • Pairing / Coaching
• Use the right object identification strategy
• Use the right test framework to work with
• Measure test efficiency within the CI
• Risk-based approach to test automation
• Continuous test data analysis and improvement
13. • How fast are testing activities moving, and what is slowing down these activities?
• Test flakiness
• Test duration
• % of automated vs. manual tests
• Application quality measurements
• # of escaped defects and in which areas
• MTTD – mean time to detection of defect
• Build quality
• Pipeline efficiency measurements
• # of user stories implemented per iteration
• Test automation as part of DoD across iterations
• Broken builds with categories
• CI length trending
• Lab availability and utilization
• Quality costs measurements
• Operational costs, lab availability issues
• Cost of hardware/software
• Costs of defects by severity and stage