Testing in a Continuous Delivery Pipeline: Faster, Better, Cheaper

W5

DevOps
&
Testing

10/5/16
11:30

Testing
in
a
Continuous
Delivery

Pipeline:
Faster,
Better,
Cheaper

Presented
by:

Gene
Gotimer

Coveros,
Inc.

Brought
to
you
by:

350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073

888-‐-‐-‐268-‐-‐-‐8770
·∙·∙
904-‐-‐-‐278-‐-‐-‐0524
-‐
info@techwell.com
-‐
http://www.starwest.techwell.com/

Gene
Gotimer

Gene
Gotimer
is
a
senior
architect
at
Coveros,
Inc.,
a
software
company
that
uses

agile
methods
to
accelerate
the
delivery
of
secure,
reliable
software.
As
a
consultant,

Gene
works
with
his
customers
build
software
better,
faster,
and
more
securely
by

introducing
agile
development
and
DevOps
practices.
He
has
many
years
of

experience
in
web-‐based
enterprise
application
design,
and
extensive
experience

establishing
and
using
development
ecosystems
such
as
continuous
integration,

continuous
delivery,
DevOps,
secure
software
development,
source
code
control,

build
management,
release
management,
issue
tracking,
project
planning
and

tracking,
and
a
variety
of
software
assurance
tools
and
supporting
processes.
Gene

feels
strongly
the
repeatability,
quality,
and
security
are
all
strongly
intertwined;

each
of
them
is
dependent
on
the
other
two,
which
just
makes
DevOps
that
much

more
crucial
to
software
development.

9/21/16
1
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 1 @CoverosGene
Agility. Security. Delivered.
Gene Go'mer
Senior Architect
About Coveros
•  Coveros builds security-criKcal applicaKons using agile methods.
•  Coveros Services
•  Agile transformaKons
•  Agile development and tesKng
•  DevOps and conKnuous integraKon
•  ApplicaKon security analysis
•  Agile & Security training
•  Government qualiﬁcaKons
•  DCAA approved rates and accounKng
•  TS facility clearance
Areas of Exper8se

9/21/16
2
Select Clients
Delivery Pipeline
Process of taking a code change
from developers and geVng it deployed
into producKon or delivered to the customer
•  Stages along the way
•  Later stages lead
•  to higher conﬁdence
•  closer to producKon

9/21/16
3
Delivery Pipeline
Do we have a
viable candidate for producKon?

Delivery Pipeline
Requirement
Code
Check-in
Unit Tests
Deploy to Test
Functional Tests
Deploy to Staging
Acceptance Tests
Deploy to Pre-Prod
Quality Gate
Trigger
Performance Tests
Security Tests Deploy to Prod
More expensive quality gates
Rapid Feedback
No surprises

9/21/16
4
Goal is to Balance
Early
Rapid
Feedback
No Late
Surprises
Everything Can’t Be First
Do just enough
of each type of testing
early in the pipeline
to determine if
further testing is justified.

9/21/16
5
Value Stream
•  List out steps from developer to producKon
•  That is the delivery pipeline
•  whether manual or automated
•  IdenKfy Kme for each step
•  execuKon Kme
•  wait Kme
•  Helps show
•  where bo_lenecks are
•  what should be automated

9/21/16
6
Pipeline Stages
•  Not hard-and-fast stages
•  Gradual change in focus
Commit Stage
Commit Stage
Requirement
Code
Check-in
Unit Tests
Deploy to Test
Functional Tests
Deploy to Staging
Acceptance Tests
Deploy to Pre-Prod
Performance Tests
•  Code-focused
•  Rapid feedback
•  10 minutes maximum
•  Developers are waiKng

9/21/16
7
Acceptance Stage
Acceptance Stage
Requirement
Code
Check-in
Unit Tests
Deploy to Test
Functional Tests
Deploy to Staging
Acceptance Tests
Deploy to Pre-Prod
Performance Tests
•  Quality-focused
•  Is this is a viable candidate for producKon?
End Game
End Game
Requirement
Code
Check-in
Unit Tests
Deploy to Test
Functional Tests
Deploy to Staging
Acceptance Tests
Deploy to Pre-Prod
Performance Tests
•  Delivery-focused
•  Steps that only get done when
we are releasing
•  Does not begin unKl you are conﬁdent
there will be no surprises

9/21/16
8
Pipeline Steps
Commit Stage
•  Compile
•  Unit tests
•  StaKc analysis
Acceptance Stage
•  FuncKonal tests
•  Regression tests
•  Acceptance tests
•  System integraKon
•  Security tesKng
•  Performance tesKng
•  Exploratory tesKng
•  Usability tesKng
End Game
•  Exploratory tesKng
•  Packaging
•  Printed documentaKon
•  Release announcement
Pipeline Steps
Commit Stage
•  Compile
•  Unit tests
Acceptance Stage
•  Some security tesKng
•  Performance trend
•  Early exploratory
tesKng
•  Basic usability tesKng
End Game
•  Mandated security test
•  Full load and
performance test
•  ConKnuing exploratory
tesKng
•  Focus group usability
tesKng
•  Packaging
Do just enough tes'ng to determine if further tes'ng is jus'ﬁed.

9/21/16
9
Example: Performance TesEng
•  Short JMeter test
•  On development system, no isolaKon
•  10 concurrent users for 10,000 requests
•  Track the trend
•  Answers: “Are we geVng slower or faster?”
•  Full load and performance test
•  Dedicated environment, no other traﬃc
•  ProducKon-sized servers
•  1,000 concurrent users for 4 hours
•  Answers: “What is the sustained capacity and throughput?”
Example: Security TesEng
•  FuncKonal tests run through
OWASP ZAP proxy
•  During early tesKng
•  Piggy-back on exisKng tesKng
•  Answers: “Do we have any XSS
vulnerabiliKes?”
•  OpenVAS system scanning
•  Weekly in test environment
•  Looks for open network ports
•  Looks for sonware with CVEs
•  Answers: “Is Nessus likely to ﬁnd
anything?”
•  HP WebInspect applicaKon
security scanning
•  By corporate security group
•  Looks for black-box web
vulnerabiliKes
•  Answers: “Do we have any XSS
vulnerabiliKes?”
•  Nessus system scanning
•  By corporate security group
•  Looks for open network ports
•  Looks for sonware with CVEs
•  Answers: “Is system compliant?”

9/21/16
10
Advantages of Earlier TesEng
•  Quicker feedback cycle
•  Easier to ﬁx problems
that are found
•  Developer sKll has
context of changes
•  Less rework on
defecKve product
•  ProacKve response,
not reacKve
Code-focused

9/21/16
11
TesEng in the Commit Stage
•  Code-focused
•  Developer-centric
•  Rapid feedback
•  Developer waits unKl complete
•  10 minutes maximum
Types of AcKviKes
•  ConKnuous integraKon
•  Compile
•  Unit tests
•  Dependency analysis
Unit TesEng
•  Unit tes'ng is not QA!
•  Developer tool
•  Early conﬁrmaKon of code behavior
•  Executable documentaKon
•  Fearless refactoring

9/21/16
12
Code Coverage
•  A tool, not a target
•  Measures code executed while unit tests running
•  NOT amount of code tested
•  Not covered = not tested
•  Covered = possibly tested
MutaEon TesEng
•  Reruns unit tests against modiﬁed versions of your code
•  If tests sKll pass, code isn’t tested
•  Tests quality of tests
public int foo(int i) {
i--;
return i;
}
public int foo(int i) {
i++;
return i;
}

9/21/16
13
StaEc Analysis
•  Early detecKon of coding issues
•  style issues
•  duplicate code blocks
•  declared but unused variables
•  confusing code
•  race condiKons
•  SQL injecKon
•  resource leaks
Third-party Components
OWASP Top 10 2013:
A9-Using Components with
Known VulnerabiliKes

Scan your third-party libraries

Update proacKvely,
not reacKvely

9/21/16
14
Quality-focused
TesEng in the Acceptance Stage
•  Quality-focused
•  Bulk of the pipeline
•  UnKl conﬁdent that you have a
viable candidate for producKon
Types of AcKviKes
•  Some security tesKng
•  Performance trend
•  Early exploratory tesKng
•  Basic usability tesKng

9/21/16
15
Automated Deployment
•  Repeatable, reliable deployments
•  Test that through pracKce
•  Same deploy process everywhere
•  You will ﬁnd more reasons to deploy
Smoke TesEng
•  Aner every deployment
•  Must be quick
•  Test the deployment,
not the funcKonality
•  Focus on
•  basic signs of life
•  interfaces between systems
•  conﬁguraKon seVngs

9/21/16
16
Delivery-focused
TesEng in the End Game
•  Delivery-focused
•  Steps that only get done when
we are releasing
•  Tests that are too expensive to
do every build
•  Kme
•  resources
•  eﬀort
•  Don’t start the End Game unKl
you are conﬁdent you won’t be
surprised
Types of AcKviKes
•  Non-funcKonal tests
•  Mandated security tesKng
•  Full load and performance test
•  ConKnuing exploratory tesKng
•  Focus group usability tesKng
•  Packaging

9/21/16
17
Non-funcEonal TesEng
•  Availability tesKng
•  Accessibility tesKng
•  Baseline tesKng
•  CompaKbility tesKng
•  Compliance tesKng
•  ConﬁguraKon tesKng
•  DocumentaKon tesKng
•  Endurance tesKng
•  Ergonomics tesKng
•  Interoperability tesKng
•  InstallaKon tesKng
•  InternaKonalizaKon tesKng
•  Load tesKng
•  LocalizaKon tesKng
•  Maintainability tesKng
•  OperaKonal readiness tesKng
•  Portability tesKng
•  Recovery tesKng
•  Reliability tesKng
•  Resilience tesKng
•  Scalability tesKng
•  Stability tesKng
•  Stress tesKng
•  Supportability tesKng
•  Testability tesKng
•  Volume tesKng
Image by Andrew Stellman via h_p://www.stellman-greene.com/2010/02/17/nonfuncKonal-requirements-qa/
Parallel TesEng
•  Conduct long-running tests in parallel
•  Upside: less elapsed Kme
•  Downside: no feedback between tests
•  Should already be an expectaKon
that these tests will pass

9/21/16
18
Summary
•  Early rapid feedback vs. no late surprises
•  Do just enough of each type of tesKng early in the pipeline to
determine if further tesKng is jusKﬁed

9/21/16
19
QuesEons?

Gene Go'mer
gene.goKmer@coveros.com
@CoverosGene

Testing in a Continuous Delivery Pipeline: Faster, Better, Cheaper

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Testing in a Continuous Delivery Pipeline: Faster, Better, Cheaper

Similar to Testing in a Continuous Delivery Pipeline: Faster, Better, Cheaper (20)

More from TechWell

More from TechWell (20)

Recently uploaded

Recently uploaded (20)

Testing in a Continuous Delivery Pipeline: Faster, Better, Cheaper