Better Software East 2016: Evolving Automated to Continuous

Parasoft Copyright © 2016 1
2016-11-23
Evolving from Automated to
Continuous Testing
Better Software 2016 - Orlando

Intro
Arthur Hicken is Chief Evangelist at Parasoft
where he has been involved in automating
various software development and testing
practices for over 20 years. He has worked
on projects including cybersecurity,
database development, the software
development lifecycle, web publishing and
monitoring, and integration with legacy
systems and maintains the IoT Hall-of-
Shame http://bit.ly/iotshame
Follow him @codecurmudgeon
Blog: http://codecurmudgeon.com
Web: http://parasoft.com

Agenda
Today’s Challenges
What IS Continuous?
Barriers
Best practices
Role of automation
Role of static analysis!?

Normal Situation
 Pressure to deliver software quickly
 Pressure to reduce defect rate
 Late stage “bug-hunting”
 No time for error prevention
 Quality can be rushed
 Security is forgotten

How to you know a build is successful?
 When it compiles?
 When all unit-tests have run?
 When the right coverage goal is achieved?
 When the right % of test failures occurs?
 When it’s deployed?

Continuous
 Build, Integration, Testing, Release, Delivery,
Deployment
 Builds on and enables DevOps
 High dependency on automation:
 processes
 assessment
 decisions

Feature
Release
Business Stakeholder Developer Customer Support
Continuous Testing is Beyond Automation
Software
Feature
Continuous measurements mean continuous refinement of the process
Real-time feedback from
Objective assessment and go/no go
Defects are eliminated at the point
that they are easiest to fix
Business Stakeholder Developer Customer Support
Quality gates: Organizations can automatically promote software
through quality gates when business expectations have been met

Challenges of Continuous
 Tests must produce binary decision go/no-go
 Reuse unit test and functional tests from dev
to QA
 High level of automation
 Requires disciplined mature process
 Testing must automatically answer
 Is it stable
 Will it do what it’s supposed to do

Importance of Testing
 How do you know when you're done?
 How do you know if a fix for a minor bug
broke a major function of the system?
 How can the system evolve into something
more than is currently envisioned?
 Testing, both unit and functional, needs to be
an integrated part of the development
process.

Continuous Delivery
Continuous Integration
Delivery
Team
Version
Control
Build &
Unit Test
Automated
Acceptance
Tests
User
Acceptance
Tests
Release
Check In Trigger
Trigger Trigger
TriggerTrigger
Approval Approval
Check In
Check In
Feedback
Feedback
Continuous Deployment
Production
Business
Continuous *

Elements of Continuous Testing
Continuous Testing re-positions the question from “are you done testing?”
to “is the level of risk understood and accepted?”
Continuous
Testing
Risk
Assessment
Policy
Analysis
Requirements
Traceability
Test
Environment
Access
Test
Optimization
Advanced
Analysis
Ensure access to complete
test environments
Automate defect prevention and
Policy measurement
Expand test coverage and
measure test effectiveness
Define actionable
practices
Connect functional with
non-functional requirements
Process improvement
opportunities

Prerequisites
Tools
• Version control
• Build server
• Deployment server
• CI tools
• Automation tools (test, etc)
Process
• Commit/update often (each change)
• Always create tests (pass and fail)
• Test regularly
• Run regularly

Continuous Testing
 Tests are logically
componentized
 Tests correlated with
business requirements
 Tests are incremental
 Tests are repeatable
 Tests are deterministic
 Tests are maintainable
in a process
 A process that is
prescriptive based on
test results

Penetration of Automated Testing Practices
Aggregated from analyst research: Gartner, Forrester, voke, IDC, Ovum for enterprise IT (embedded market eliminated)
Practice Penetration Opportunities
Static Analysis 16 –38% • Business/Risk driven models
• Process enabled
Unit Testing 15 – 32% • Workflow
• Test optimization
• Deterministic tests
Peer Review ~ 26% • Integrated results
• Contextual review
• Compliance triggers
API Testing 12 – 21% • 2nd lowest penetration
• Regression models
Load Testing 32 – 49% • Smoke testing
• Continuous
Service Virtualization 9 - 16% • Green field
• Process enabler
• ROI is a NO-Brainer!

Continuous needs
Proper infrastructure
A binary definition of “done”
Reliable quality gates
Extreme automation

Infrastructure
 Real requirements system
 Flexible developer level project/scrum
management system
 Automated build/testing system
 Data collection that can answer
 Is it done?
 When will it BE done?
 Is it good enough to release

Defining Done
 Are requirements coded?
 Do tests exist for the requirements and code?
 Are the tests passing
 To your satisfaction (often not 100%)
 Will the tests be enough?

Delivery is Part of Software
 Delivery is an important part of software too
 Development and Test need access to
production-like systems
 Deployment has to be repeatable and reliable
 Quality must be validated (measured)

Defining Quality
• Consistent
• Repeatable
• Meaningful
Automated objective measure
• Unit tests
• Functional tests
• Regressions tests
• Static analysis
Made up of

Effective Continuous Testing
 Avoid manual end-to-end testing
 Invest more into automated tests at
component
 Follow testing pyramid rules
 Measure:
 test results
 test effectiveness
 Determine:
 Risk

How to do it better ?
 Give absolute priority to automated tests
 Invest time into designing interfaces (API)
 Use API (service) testing tools to cover
interfaces
 Measure the quality of the test

How Can Static Analysis Help?
Coding standards
Quality gate (bug detection)
Prevention

Bug Gates
 Binary Decisions
 No “bugs”
 No “critical” static analysis findings

Quotable Quotes
 An ounce of prevention is worth a pound of
cure. – Benjamin Franklin
 Cease dependence on inspection to achieve
quality. Eliminate the need for inspection on a
mass basis by building quality into the product
in the first place. – Deming
 Simply finding bugs with static analysis isn’t
enough.
 Avoid risky behavior

Fix or Prevent

Static Analysis Continuous Feedback Loop
Identify
Error
Isolate
Root Cause
Fix
New Rule
to Prevent
Monitor
 Code review
 Regression
 QA
 Field bugs

Choosing rules
Things
happening in
the field
Things you
worry will
happen
Things
happening in
the news
Standards you
must comply
with

Main Points for Today
 We must define “all” quality expectations upfront
 We must change the primary goal of testing
 We must accept that testing is only a task in the
quality process
 We must be able to measure trust of the process
 We must automate breaking the application

Re-Inventing the SDLC
The evolution of the software development lifecycle over the
past 5 years is forcing the transformation of software testing
Transformation Business Impact
• “Digital Industrial Revolution”
makes all a software company
• Trend toward Dev/Test insourcing as
competitive advantage is via software
• Strong adoption of “agile”
development practices
• Forces more technical testing, early.
Agile is forcing “shift-left”
• DevOps is applying lean practices
to SDLC
• DevOps is exposing 20 years of process
barriers protected by silos and culture
• Cloud has disrupted traditional
SDLC and is driving down costs
• Switching costs associated with
applications at an all time low
• Business risk associated with
software failure at an all time high
• Public companies lose an average of
$2.5 Bn in market cap on the day of
the announcement

DevOps is Continuous Improvement
Speeding up the conveyor belt does not yield better results…
Modern DevOps must embrace systematic process improvement—
Focused on testing

How to Achieve Quality @ Speed
 Business risks drive quality
activities
 Teams collaborate on risk
definition
 All team members trained on
risks
 Acceptance criteria visible and
measured
 DevTest team activities
prioritized per risk definition
 Results of quality activities are
visible and translated for all
levels of business

Survey on Non-Functional Requirements
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Waterfall
Hybrid
Iterative
Agile
Agile-ish
“Yes” My Team Monitors Compliance to
NON-Functional Requirements (NFRs)
34%
58%

From Automated to Continuous

Continuous Testing Mitigates Business Risk
Why Continuous
Testing?
Systematic, centralized
decision making to factor
business risk into SDLC
Safety net to allow
developers to bring
innovations to market faster
…and automation to break
the application
A feedback system for better trade-off
decisions between release scope,
time and quality

Enabling Technologies
 Stubs
 Service Virtualization
 API testing
 Test data management
 Environment management
 Self-service test environments

How to do it better ?
 Use service virtualization to improve testing
automation
 Isolate at the message layer
 Simulate functional scenarios and performance
conditions

The Test Environment Challenge
Test environment access is outside the control of development and test leaving gaps in the
process
IT Operations
Parallel development
delays… Need simple,
realistic access to
dependent components…
Too much time
waiting for access…Need
reliable test data
Need a realistic test
environment easy to
maintain
Scheduling
Configuration
Access Limits
Dependent
Applications
Staged Assets
3rd Party Assets Virtual
Environments
Hyper Visor
App App App

The Test Environment Challenge
Test environment access is outside the control of development and test leaving gaps in the
process
IT Operations
Scheduling
Configuration
Access Limits
Dependent
Applications
Staged Assets
3rd Party Assets Virtual
Environments
Hyper Visor
App App App
Create, Manage, Provision

 The Challenge
 Multiple teams using the same test database
 Teams not respecting data integrity & others test data records
 Regression tests consistently failing. Takes >1 hour to determine that it
was due to “data changes”.
 “Real problems” were getting lost in the noise
Shared
Database
Eliminated 83% of configuration time for a
major telecom company
Test data management for complex transactions
✔
✖
✖

 The Solution
 Setup Virtual Assets to model the SQL queries and use API testing tool
to manage automated nightly regressions against both virtual assets
and live systems
 The Business Benefit
 Test teams able to focus on ‘real regressions’ and separate out data
integrity issues from functional test failure
Virtual
Asset
Virtual
Asset
Virtual
Asset
✔
✔
✔
Eliminated 83% of configuration time for a
major telecom company

Reduced wait time for test team by 60% for a
major media conglomerate
 The Challenge
 Large agile development effort to adopt Service Oriented Architecture (SOA)
 High risk project but the Test team “stuck waiting for the first build”
 Development of functionality was not easy to coordinate as different teams
had different schedules; not all finished at the same time
Agile/Parallel development limited by system dependencies
Iteration
Iteration
Iteration
Team A
Team B
Team C
Iteration Iteration
Iteration Iteration Iteration
Iteration Iteration
Current
Development/Testing
dependencies

 The Solution
 Use descriptions of the new services (WSDL, XSD, example JSON payloads) to
build Virtual assets prototyping the new functionality.
 Test team builds tests with against the prototypes with API testing tool and the
independent development tests use the prototypes to perform early stage
Integration Testing
 The Business Benefits
 Met business goals and timelines, were able to test functionality “as soon as”
it was available. Practiced TDD against prototypes to get a head-start on ‘full
system testing’
Iteration
Iteration
Iteration
Team A
Team B
Team C
Iteration Iteration
Iteration Iteration Iteration
Iteration Iteration
✔✖✔✖✔✖
Reduced wait time for test team by 60% for a
major media conglomerate

Summary
 Continuous * requires mature infrastructure
and process
 Quality gates must produce a binary answer
 Automation is everything
 Policy is critical, because automation is critical
 Static analysis gets you to the gate

 Web
 http://www.parasoft.com
 IoT Hall-of-Shame
 http://bit.ly/iotshame
 Blog
 http://alm.parasoft.com
 http://codecurmudgeon.com
 Book:
 http://alm.parasoft.com/continuoustestingbook

Better Software East 2016: Evolving Automated to Continuous

More Related Content

What's hot

Viewers also liked

Similar to Better Software East 2016: Evolving Automated to Continuous

More from Parasoft

Recently uploaded

Better Software East 2016: Evolving Automated to Continuous