The document discusses continuous testing as a means to automate tests in the software delivery pipeline for fast feedback on business risks. It highlights common issues in test automation, such as instability and failure patterns, and emphasizes the need for a risk-based approach to improve testing efficiency. Additionally, it outlines the role of machine learning in test automation, including use cases for recognizing objects and making quality-related decisions based on data.

2. •
•
•
•
•
•
Twitter: @ek121268 (https://twitter.com/ek121268)
Blog: http://continuoustesting.blog
LinkedIn: https://www.linkedin.com/in/eran-kinsbruner-4b47a81/

3. 5/24/2018 3© 2018, Perfecto Mobile Ltd. All Rights Reserved.
Continuous Testing is the process of executing
automated tests as part of the software
delivery pipeline in order to obtain feedback on
the business risks associated with a software
release candidate as rapidly as possible.

4. State of DevOps Report, 2018 - DORA

5. State of DevOps Report, 2018 - DORA

6. There Are Patterns for “Unstable” Test Automation
80% of issues have a pattern52% success rate
10% of devices,
causing 80% of lab
issues
Lab
25%
Orches
tration
25%
Scripts
& FW
50%
FAILURE REASON
Objects Codding Time Other
Scripts & FW issues
Device in use
No Device
Orchestration issues
Networking Stability Lock
Other
Lab issues
What’s
wrong
With my
Scripts
What’s wrong
With my Lab
What’s wrong
With my
Executions

7. 5/24/2018 7© 2018, Perfecto Mobile Ltd. All Rights Reserved.
3
2
1
3
2
●
●
●
●
●
● ●
●
●

10. 1. What’s the test engineer’s gut feeling 😊
2. Risk calculated as probability to occur and
impact to customers
3. Value – does the test provide new
information and, if failed, how much time to
fix?
4. Cost efficiency to develop – how long does
it take to develop and how easy is it to
script?
5. History of test – volume of historical
failures in related areas and frequency of
breaks
Source: Angie Jones

11. Insights into the CI
Pipeline
Risk/Focus Area
Mapping
Summary Report
List
Single Test Report
Visual Validations
Noise reduction through
error/failure-classification

12. • Pairing / Coaching
• Use the right object identification strategy
• Use the right test framework to work with
• Measure test efficiency within the CI
• Risk-based approach to test automation
• Continuous test data analysis and improvement

13. • How fast are testing activities moving, and what is slowing down these activities?
• Test flakiness
• Test duration
• % of automated vs. manual tests
• Application quality measurements
• # of escaped defects and in which areas
• MTTD – mean time to detection of defect
• Build quality
• Pipeline efficiency measurements
• # of user stories implemented per iteration
• Test automation as part of DoD across iterations
• Broken builds with categories
• CI length trending
• Lab availability and utilization
• Quality costs measurements
• Operational costs, lab availability issues
• Cost of hardware/software
• Costs of defects by severity and stage

14. Key ML Use Cases In Test Automation
• Recognize objects
• Transcribe speech – Accessibility
• Make quality related decisions based on data
• Identify Trends and/or Patterns
• Security use cases – Identify signatures e.g.
11/20/2018 14© 2015, Perfecto Mobile Ltd. All Rights Reserved.

15. Mindset and Workflow Changes in Test Automation
11/20/2018 15© 2015, Perfecto Mobile Ltd. All Rights Reserved.
# Category
1 Test Authoring
2 Test Maintenance
3 Test Execution
4 Test Analysis
5 Tools Maturity
6 Available Integrations
7 Req. Test/Dev Skillset
8 Testing Env.
9 Testing Types
10 Supported App Types
Test Engineers/Developers Manual Testers (Developers?)
Traditional Test Automation
Define Manual Flows, BDD Style, etc.
Test Code & Reusable Functions (Java,
JS, etc.)
Define Objects, POM, use Object Spy
Structured Page Based Test Steps
Define Visual Validations and Assertions
Longer Time To Develop, Complex
Changes required pro-actively, SCM
Supported
Configure Env. (TestNG Data Provider),
Execute Locally/CI/Cloud
Flexible, OSS, Commercial
High, Including Guidelines, Doc’s
Plenty, OSS, Defect Management, etc.
Medium-High
IDE
API, Load, Functional
All Types
ML Based Test Automation
Record Test Flows (No Coding – in most cases)
Groups Reusable Functions
Objects Generated On-The-Fly (transparent to user)
Flow Based Scenarios
Part of Test Authoring Recording/Authoring
Shorter Time, Advanced Capabilities
Self-Healing/Correction Automatically handled/Object
Scoring/Local SCM Controlled
Configure Env., Execution Management Built-In,
Locally/CI/Cloud
Relies on ML/AI Tool Vendor in Most Cases
Emerging Technology, Web More Advanced Than
Mobile
Most capabilities built-in, some exists
Low-Medium
ML UI
Mostly Functional (E2E) & API
Mostly Web