This document outlines 10 best practices for quality engineering to achieve quality at speed, including establishing an "automate first" mindset, impact-based testing, eliminating dependencies through virtualization and containerization, enabling continuous integration/delivery, improving security with DevSecOps, continuous performance testing, leveraging cloud capabilities to scale automation, shifting to intelligent DevOps, behavior driven development, and leveraging artificial intelligence. It provides examples and strategies for implementing each best practice.

1. #ATAGTR2019
TOP 10 QUALITY ENGINEERING BEST PRACTICES TO ACHIEVE QUALITY-AT-SPEED
By :
Vinod Sundararaju Antony, Senthilkumar Thirumalaisamy, Santhosh Kumar Vasudevan &
Ameet Pradhan
14th 15th Dec 2019

2. #ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
2
TOP 10 QUALITY ENGINEERING BEST PRACTICES TO ACHIEVE
QUALITY-AT-SPEED
The Digital era will see a proliferation of intelligent products, transaction channels and influencers
available at the fingertips of a digital customer.
• Need to deliver improved customer experience, at speed and scale with higher efficiency
• Quality, Customer Experience & Trust (Data privacy and Security) are therefore becoming more
important than ever
• It has become paramount for enterprises to assure quality across the application lifecycle with better
test processes, tools, people and governance.
This paper provides :
• A guideline to transform traditional testing approaches into best-in-class QA for the Digital ICE (intelligent, continuous
and early) age.
• Describes business case studies with best practices that illustrate testing strategies for functional, automation teams,
DevOps engineers, production support teams and non-functional teams to deliver quality at speed.

3. #ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
3
Establish an “Automate First” Mind Set
Strategy
• Prioritize Automation – Risk
based approach
• Invert Automation Pyramid:
focus on unit level
automation
• Early Automation : write
components during Dev
• Use test design pattern and
principles : Page Object,
Screenplay, SOLID,GRASP
• Static code analysis
• Build reusable components
• Test design automation
Framework
• Build a Robust
Automation Framework
• Data driven
• Page Object Model
• Reporting
• API Libraries
• Integrates with Test
management ,Version
control and CI tools
• Supports Web,
Mobile, API, SAP,
Mainframe
application, etc.
Collaboration
• Automation - A team
effort
• Developer to follow
coding standards in UI
• Support from Dev for
Automated build and
deployments - Enables
faster QA
Infrastructure
• Scalable: Identify un-
interrupted automation
execution platform on
VMs - Dockers /
Kubernetes
• AWS – Helps configure
test run machines based
on the requirements ,
full admin access
Reporting
• Intelligent ROI capture :
Automated measure or
capture of ROI
• Dashboard - Create
online real time QA
reports and dashboard
01 02 03 04 05
1

4. #ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
4
Automated Impact Based Testing
Traditional Approach
Run entire tests for even “one-
line change” of code
Change Based Testing
• Analyze build dependencies: Baseline
code Vs new build code
• Identifying set of files/code impacted
and their respective tests to be
executed
Inside-Out Approach
• Begin with details about the
situation and identify risks
associated with them
• Study a product and
repeatedly check “What can
go wrong here?”
Outside-In Approach
• Identify a predefined list of
risks and determine whether
they apply to the situation
• Organize RBT based on Risk
watch list, risk / task matrix
and component risk matrix.
a) Change based testing (CBT) b) Heuristic Risk-Based Testing
Module-based Approach
Manual Identification of tests based
on impacted modules
Next-Gen CBT
• Predictive test selection - model
predicts probability of each test failing
for a newly proposed code change
• ML Model leverages Gradient-boosted
decision-tree algorithm
• Jira to catalogue and prioritize risks
• Jubula to mitigate them
• Mylyn to mix everything (Risk Based Interface)Tools
• VectorCAST
• Parasoft DTP
• Coverity Test Advisor etc.Tools
2

5. #ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
5
Eliminate dependencies through Virtualization and Containerization
01
02
05
04
03
When to Adapt?
• Components are not available or are restricted
• Critical component testing impacts business
Benefits : Enables TDD , “Shift Left” QA: replicate/fix
defects faster, continuous testing , enables parallel
testing: test the pre-prod/UAT in parallel
Containerization
• AWS - Amazon ECS, Amazon EKS (Amazon
ECS for Kubernetes) and AWS Fargate
• Docker Swarm
Return on Investment
• Dev/QA Staff Productivity savings,
• Reduced costs through shifting defects left,
• Reduced Infrastructure costs and
• Time to market savings (days saved)
• Capital and Operational expenditure reduction
Ways of creating/generating SV
• Recording Traffic : Record  Process  Model
• Ground up from WSDL, REST
• Using Request/Response (RR) pairs for Web
Services
Microservices Testing
• Require heavy orchestration for
service virtualization
• “Mountebank” approach : Open
source, lightweight tool , provides
cross-platform multi-protocol test
doubles
3
Virtualize anything, test anything
• Mainframe , Platforms (Oracle, Siebel, SAP)
• SOA protocols (SOAP, HTTP, REST, XML)
• Banking message format (SWIFT)
• Teleconference & SMS gateway providers (WebEx,
Skype), Card processing services (PayPal)

6. #ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
6
Enable Continuous Everything
DevOps Continuous Integration (CI) Continuous Delivery (CD) Continuous Deployment (CI/CD)
Best Practices :
• Gated check-in build system
• Setting up a change control board
• Automatically identify issues in every commit
and pull requests
• Speed up the test automation phase
Manage Source Code Repository for ease of
Release Management Workflow
• Centralized Workflow
• Feature Branch Workflow
• GitFlow Workflow
Implement SCM in Test for
better CI
Collaboration through
ChatOps
Continuous Monitoring in
DevOps
Continuous Inspection
Infrastructure as Code (IaC) –
Runtime Execution
Infrastructure Monitoring
• Visualize events and get alerts in real time
• Monitor compute resources, storage, network and
measure inventory usage, and health of an
infrastructure's resources
Application Performance Monitoring (APM)
• Helps find performance bottlenecks
• Provides usage and performance of application
dependency systems like web services, databases,
caching, etc.
• Code Quality Check using Static Code
Analysis
• Management of code quality of new,
changed and added code before
anything else – Water leak paradigm
• Infrastructure via source control, thus providing a detailed
audit trail for changes
• Enables collaboration around infrastructure configuration
and provisioning, most notably between dev and ops
• Connects people, process and
automation in a transparent flow
• Helps collaborate and control
DevOps pipeline through BOTs
4

7. #ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
7
Continuous Everything: Improve Security through DevSecOps
DevSecOps automatically detect alerts on vulnerable components, outdated libraries, and license compliance issues in code as defined by
OWASP guidelines. It enables
Some of the best practices to be improve security in DevOps model is represented below:
Detection and solution for vulnerable
open source components
Generate comprehensive open source
inventory reports per project or build
Enforcing open source license
compliance
Identification outdated open source
libraries with recommendations
Embrace a DevSecOps model
Embedded Governance and Cybersecurity
functions:
• Identity and access management (IAM),
privilege management
• Firewalling / unified threat
management, code review
• configuration management, and
vulnerability management
Control, monitor, and audit access with privileged access management
Deploy a secured jump server
• Multi-factor authentication
• Adaptive access authorization
• Use session monitoring to provide
oversight
4

8. #ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
8
Continuous Everything : Continuous Performance Test
Continuous performance testing involves:
 Developing and deploying low volume load tests to capture performance metrics of critical application user flows, API end points and
database queries of the systems without applying stress.
 Key focus is to capture performance metrics such as latency, throughput, bandwidth, Time to First Byte (TTFB), application http response
codes (404, 500), CPU, Memory, Disk I/O and others
• Benefits continuous Performance
Testing in CI/CD
• Early detection of performance issues or
degradation trends.
• Quick feedback to developers on check-in –
resolution of performance issues or bottlenecks in
the code early in SDLC
• Help identify latency issues, over the time
performance degradations, location specific latency
issues
• Very helpful for cloud hosted applications and
services to measure long term stability and site
performance on regular basis.
Pre requisites
• List of application’s critical pages, end to end
user flow scenarios, API end points and
database queries
• Measure and analyze on regular basis as
automated performance test
• Establish baselines performance
• Execute performance tests on daily basis in
production environment along with real
users’ traffic to capture performance metrics.
4

9. #ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
9
Scale Automation : Leverage the Power of Cloud
Identify un-interrupted execution platform for running automation test scripts in multiple Virtual Machines (on-demand cloud
based infrastructure)
5
Planned In-Progress Completed
GIT
Repository
Continuous
Integration
Automation
Reports
Defects
Test Planning & Feasibility
Requirement
Refinement
Failure
Analysis
Execution
Environments
Test Data
(Application
URLs / Sites)
Parallel execution across selected
markets
Automation Framework
JVM Libraries /
Reusable
methods
Automation
Tests / Reusable
Tests
APP
Select
combination of
sites/markets &
QA checks
Docker is lightweight container
helps in running tests distributed
with NO minimal costs
compared to traditional
distributed execution platforms
AWS : Helps to configure test run
machines based on the requirements.
Full administrator access with any
software, browsers, tools and access
with different operating systems

10. #ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
10
Shift from DevOps to Intelligent DevOps
Intelligent DevOps to make data-driven decisions that help them improve software delivery
Automation Analysis
• Employ algorithms and intelligence to extract meaning from data, execute judgment-based activities using diagnostic analytics,
and make recommendations based on patterns leveraging advanced automation systems
Automation that learns and evolves
• Adaptive continuous delivery pipeline. E.g. - Correlate code quality and defect detection and slippage patterns to dynamically
decide what tests to be run
• DevOps process optimization based on insight across the lifecycle. E.g. - Determine the level of failure risk by correlating
production log data with past code change data
• Usage of parallel pipeline to prevent disruption , Release Orchestration
InteliDevOps - Intelligent Development Operations
• InteliDevOps is an intelligent build & release management system designed to fail fast and fail cheap by prioritizing tests &
injecting them as dynamic stages
1
2
3
Generation of automated test cases
( from manual) using NLP
Self-healing
automation
Automated generation of test scenarios
based on analytics on production logs
Automated verification of system
requirements based on CX analytics
6

11. #ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
11
Behavior Driven Development
WHY
Less code, less maintenance, more
acceptance testing, readability,
standardization
WHEN
Early automation, business driven
model
WHAT
Business value, collaboration,
traceability, coverage, metrics
Testing APIs, UI and non-UI components of any application by Domain-Specific Language (DSL)
Collaboration Strategy
Imperative vs
Declarative
Gherkin Tag
Feature Mapping
Design Pattern
Structured using “Context-
Action-Outcome” pattern Aligned Features
• For identification of different
release, version, type, etc.
• Traceability - Map with
requirements
• Helps team write great
Executable Specifications
Natural Language Processing (NLP) testing platform
to enhance QE automation and improve quality
Page Object Pattern aims to move logic out of UI
elements
BDD
Frameworks
7

12. #ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
12
Test SMART : Leverage the power of Artificial Intelligence (AI)
Create Automation
using AI
• AI-driven Web & Mobile App Test : User
interaction /behaviour is recorded by AI
to auto generates test scripts
• Smart API Test Generator
• AI based Visual Test Validation across
OS, browsers and screen resolution
Maintenance through
Self Healing
AI engine auto updated ‘dynamic locator’
to self heal automation scripts
Automated Failure
Analysis
• Traditional failure analysis by manual
investigation and categorization of
automation error logs, screenshots, etc.. and
provide meaningful interpretation for test
reports
• AI-driven Application Failure Log Analysis -
Automating and building predictive model
that continuously learn from error logs such
as system logs, application logs, database log
tables, etc.. (pattern based) to identify failure
root cause
Robotic Process Automation
Automation across large scale standard
systems like SAP, Microsoft Dynamics 365,
Salesforce etc.
Leveraging types of AI
Automation
• AI-driven automation test creation
• AI which monitors HTTP(S) traffics to
generate API scenarios
• Visual Validation Automation Test
• Self-healing systems
• AI for automation failure analysis
8

13. #ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
13
Extend Application Testing in Production - Shift-Right QA
• Canary Release - push of programming code changes
to a small group of end users
• A/B Testing - compares two versions of a product
hosted in production
• Blue-green deployment
• Split testing
• Automated Roll back strategy
• Visual Testing across OS, browsers and screen
resolution. Tools like Applitools helps in pixel-to-pixel
testing.
• New Relic Synthetic User Testing – a suite of
automated, scriptable tools to monitor your websites,
critical business transactions, and API endpoints.
• Infrastructure Integration Testing (IIT)
• Disaster Recovery (DR) Testing
Testing Web Services with live production traffic
• Live production traffic helps to test new version of a
web service and to determine if unpredictable
changes in API usage cause unexpected behavior,
such as slower response times or deviations in the
CPU consumption.
Helps reveal
• Potential weak points and causes of
failure
• How efficiently servers can be restarted
• How fast the app can switch to backup
systems and
• How resilient the software is?
• How long can entire application run on
an average, in practice, before anything
slows down, goes down, crashes, or
freezes?
Adapt
deployment
strategies
Web Services
Testing
Test
Methods
APM in Real
time
Shift Right
QA
Shift Right QA : Brings in the business perspective in testing for improved quality through end to end business component testing and defect
prediction/trend analytics, prevent the performance bottlenecks and increase system performance. Some of the best practices are:
9

14. #ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
14
Shift from traditional Testing to Quality Engineering metrics
‘Single View of Quality’ @Speed and Scale : Establish a ‘Metrics Driven’ QE management framework aimed to measure, manage and optimize
‘Business Outcome Aligned Metrics’ and plan for progressive continuous improvements
10
BUSINESS OUTCOMES DELIVERY OUTCOMES KPIs
Faster Time To Market
Cycle Time Reduction QA Vs ADM Cycle Time
Automation Coverage % Regression Automation Coverage
Automation effectiveness % Defects identified through automation
Reduce Total Cost of Ownership COQ Reduction
QA vs. AD spend
QA spend on Run the Business - High Maturity/
Stable applications
Improved User experience Advocacy / Brand Value
End user satisfaction score
% Application Availability