The software delivery pipeline is the process of taking new or changed features from developers and getting them quickly delivered to the customers by getting the feature deployed into production. Testing within continuous delivery pipelines should be designed so the earliest tests are the quickest and easiest to run, giving developers the fastest feedback. Successive rounds of testing lead to increased confidence that the code is a viable candidate for production and that more expensive tests—be it time, effort, cost—are justified. Manual testing is performed toward the end of the pipeline, leaving computers to do as much work as possible before people get involved. Although it is tempting to arrange the delivery pipeline in phases (e.g., functional tests, then acceptance tests, then load and performance tests, then security tests), this can lead to serious problems progressing far down the pipeline before they are caught. Be prepared to discuss your pipeline, automated or not, and talk about what you think is slowing you down and what is keeping you up at night. In this interactive workshop, we will discuss how to arrange your tests so each round provides just enough testing to give you confidence that the next set of tests is worth the investment. We'll explore how to get the right types of testing into your pipeline at the right points so that you can determine quickly which builds are viable candidates for production.

1. 10/22/2017
1
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 1@CoverosGene
Agility. Security. Delivered.
Gene Gotimer
@CoverosGene
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 2@CoverosGene
About Coveros
• Coveros builds security-critical applications using agile methods.
• Coveros Services
• Agile transformations
• Agile development and testing
• DevOps and continuous integration
• Application security analysis
• Agile & Security training
• Government qualifications
• DCAA approved rates and accounting
• TS facility clearance
Areas of Expertise

2. 10/22/2017
2
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 3@CoverosGene
Select Clients
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 4@CoverosGene

3. 10/22/2017
3
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 5@CoverosGene
Delivery Pipeline
Process of taking a code change
from developers and getting it deployed
into production or delivered to the customer
• Stages along the way
• Later stages lead
• to higher confidence
• closer to production
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 6@CoverosGene
Delivery Pipeline
Do we have a
viable candidate for production?

4. 10/22/2017
4
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 7@CoverosGene
Delivery Pipeline
Requirement
Code
Check-in
Unit Tests
Deploy to Test
Functional Tests
Deploy to Staging
Acceptance Tests
Deploy to Pre-Prod
Quality Gate
Trigger
Performance Tests
Security Tests Deploy to Prod
Rapid Feedback
No surprises
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 8@CoverosGene
Goal is to Balance
Early
Rapid
Feedback
No Late
Surprises

5. 10/22/2017
5
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 9@CoverosGene
Everything Can’t Be First
Do just enough
of each type of testing
early in the pipeline
to determine if
further testing is justified.
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 10@CoverosGene

6. 10/22/2017
6
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 11@CoverosGene
Value Stream Exercise
• List out steps from developer to production
• That is the delivery pipeline
• whether manual or automated
• Identify time for each task
• work time
• wait time
• Take about 10 minutes
• feel free to discuss with your neighbors
Task
Wait
Work
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 12@CoverosGene
Value Stream
• Helps show
• where bottlenecks are
• what to automate
• where time is being
spent and/or wasted

7. 10/22/2017
7
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 13@CoverosGene
Pipeline Stages
• Not hard-and-fast stages
• Gradual change in focus
minutes hours to daysbulk of the process
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 14@CoverosGene
Commit Stage
Commit Stage
Requirement
Code
Check-in
Unit Tests
Deploy to Test
Functional Tests
Deploy to Staging
Acceptance Tests
Deploy to Pre-Prod
Performance Tests
Security Tests Deploy to Prod
Code-focused
• Rapid feedback
• CI cycle: 10 minutes maximum
• Developers are waiting

8. 10/22/2017
8
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 15@CoverosGene
Acceptance Stage
Acceptance Stage
Requirement
Code
Check-in
Unit Tests
Deploy to Test
Functional Tests
Deploy to Staging
Acceptance Tests
Deploy to Pre-Prod
Performance Tests
Security Tests Deploy to Prod
Quality-focused
• Is this is a viable candidate for production?
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 16@CoverosGene
End Game
End Game
Requirement
Code
Check-in
Unit Tests
Deploy to Test
Functional Tests
Deploy to Staging
Acceptance Tests
Deploy to Pre-Prod
Performance Tests
Security Tests Deploy to Prod
Delivery-focused
• Steps that only get done when
we are releasing
• Does not begin until you are confident
there will be no surprises

9. 10/22/2017
9
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 17@CoverosGene
Back to the Value Stream
• Break into roughly three stages
• won’t be hard-and-fast boundaries
• some overlap is expected
• look for wildly out-of-place tasks and
consider reordering
• Take about 5 minutes
Task
Wait
Work
Code Focused
Quality Focused
End Game
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 18@CoverosGene

10. 10/22/2017
10
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 19@CoverosGene
Pipeline Steps
Commit StageCommit Stage
• Compile
• Unit tests
• Static analysis
Acceptance StageAcceptance Stage
• Functional tests
• Regression tests
• Acceptance tests
• System integration
• Security testing
• Performance testing
• Exploratory testing
• Usability testing
End GameEnd Game
• Security testing
• Performance testing
• Exploratory testing
• Usability testing
• Packaging
• Printed documentation
• Release announcement
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 20@CoverosGene
Pipeline Steps
Commit StageCommit Stage
• Compile
• Unit tests
• Static analysis
Acceptance StageAcceptance Stage
• Functional tests
• Regression tests
• Acceptance tests
• System integration
• Some security testing
• Performance trend
• Early exploratory testing
• Basic usability testing
End GameEnd Game
• Mandated security test
• Full load and
performance test
• Continuing exploratory
testing
• Focus group usability
testing
• Packaging
• Printed documentation
• Release announcement
Do just enough testing to determine if further testing is justified.

11. 10/22/2017
11
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 21@CoverosGene
Example: Performance Testing
• Short JMeter test
• On development system, no isolation
• 10 concurrent users for 10,000 requests
• Track the trend
• Answers: “Are we getting slower or faster?”
• Full load and performance test
• Dedicated environment, no other traffic
• Production-sized servers
• 1,000 concurrent users for 4 hours
• Answers: “What is the sustained capacity and throughput?”
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 22@CoverosGene
Example: Security Testing
• Functional tests run through
OWASP ZAP proxy
• During early testing
• Piggy-back on existing testing
• Answers: “Do we have any XSS
vulnerabilities?”
• OpenVAS system scanning
• Weekly in test environment
• Looks for open network ports
• Looks for software with CVEs
• Answers: “Is Nessus likely to find
anything?”
• HP WebInspect application
security scanning
• By corporate security group
• Looks for black-box web
vulnerabilities
• Answers: “Do we have any XSS
vulnerabilities?”
• Nessus system scanning
• By corporate security group
• Looks for open network ports
• Looks for software with CVEs
• Answers: “Is system compliant?”

12. 10/22/2017
12
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 23@CoverosGene
Advantages of Earlier Testing
• Quicker feedback cycle
• Easier to fix problems
that are found
• Developers still have
context of changes
• Less rework on
defective product
• Proactive response,
not reactive
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 24@CoverosGene
Back to the Value Stream Again
• When you find a problem, how much does
it disrupt the pipeline?
• time/effort expended on this build
• time/effort expended on next build in flight
• Does chaos ensue when problems are
identified late in the pipeline?
• surprise
• commitment to release already made
• will they ship it anyway?
• Take about 5 minutes
Task
Wait
Work
Code Focused
Quality Focused
End Game

13. 10/22/2017
13
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 25@CoverosGene
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 26@CoverosGene
Automation isn’t Free
• Committed Management
• Budgeted Cost
• time, effort, tools, training
• Process
• Dedicated Resources
• not everyone can automate
• learning curve
• Realistic Expectations
• 100% automated is unreasonable
• benefits may not be immediately obvious

14. 10/22/2017
14
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 27@CoverosGene
Automated Deployment
• Repeatable, reliable deployments
• Test that through practice
• Same deploy process everywhere
• You will find more reasons to deploy
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 28@CoverosGene
Smoke Testing
• After every deployment
• Must be quick
• Test the deployment,
not the functionality
• Focus on
• basic signs of life
• interfaces between systems
• configuration settings

15. 10/22/2017
15
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 29@CoverosGene
One last time with the Value Stream
• If it can be automated,
• what would it take to automate it?
• would it remove bottlenecks?
• could it be moved earlier?
• would it reduce late surprises?
• Where are the manual tests?
• What steps are not adding value?
• Strongly consider wait times
• Take about 5 minutes
Task
Wait
Work
Code Focused
Quality Focused
End Game
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 30@CoverosGene

16. 10/22/2017
16
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 31@CoverosGene
#Coveros5
• Balance early rapid feedback with no late surprises
• Do just enough of each type of testing early in the pipeline to
determine if further testing is justified
• Your pipeline should generally progress from
code-focused
to quality-focused
to delivery-focused
• Consider automation to help remove bottlenecks
• Use value-stream mapping to help you understand your process
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 32@CoverosGene
Value Stream Back at Home
• Per story/feature/task
• Record each step as you do it
• Record actual time
• work time
• wait time
• Once a pattern emerges, make it visible
• post it in the team room, online
• keep it up-to-date
• include it in retrospectives
Task
Wait
Work

17. 10/22/2017
17
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 33@CoverosGene
Reading List
• Learning to See: Value-Stream Mapping to Create Value and Eliminate
MUDA, by Mike Rother and John Shook. ISBN-13: 978-0966784305
• The Principles of Product Development Flow: Second Generation Lean
Product Development, by Donald G. Reinertsen.
ISBN-13: 978-1935401001
• Implementing Lean Software Development: From Concept to Cash, by
Mary and Tom Poppendieck. ISBN-13: 978-0321437389
• Toyota Kata: Managing People for Continuous Improvement,
Adaptiveness, and Superior Results, by Mike Rother.
ISBN-13: 978-0071635233
• The Art of Business Value, by Mark Schwartz.
ISBN-13: 978-1942788041
© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 34@CoverosGene
Questions?
Gene Gotimer
gene.gotimer@coveros.com
@CoverosGene
Come see me at 2pm in Wekiwa 7&8 for
Which Development Metrics Should I Watch?