This document discusses continuous testing in the context of DevOps. It defines continuous testing as including automated testing, managing production and non-production environments, application monitoring, and evaluating business objectives. Continuous testing relies on DevOps activities like automating builds, infrastructure, deployments, and monitoring. It advocates for smaller, more frequent deliveries through practices like test automation, infrastructure as code, and treating testing as integral to the software delivery process. The conclusion emphasizes automation, configuration management, and obtaining frequent feedback to enable continuous testing.

1. - Technical Architect @Planday
- Lead consultant & trainer at @
LeanSoftwareServices
- Based in London & Madrid
Peter Marshall
MAIL
info@leansoftwareservices.com
TWITTER
@petemar5hall
LINKEDIN
https://www.linkedin.com/in/petedmarshall

2. Training
Coaching
Consulting
Madrid
Barcelona
London
- Testing
- DevOps
- Continuous
Delivery
- Automation
- Virtualisation
TWITTER
@petemar5hall

3. Testing software in high
frequency delivery
environments
DevOps &
Testing
Understanding the
Foundations
@petemar5hall

4. Gain insight into how devops and organisational change are
required for continuous testing
Understand how to leverage devops & continuous testing practices to
gain more time testing
Objectives

5. Creating
environments
deploying to
environments
testing
environments
looking at/creating
requirements
creating
test data
writing
automated
checks
running through
regression scripts
what else does a
tester do?
bug triage
mitigating
delays
Testing Tasks?

6. “A survey by IBM and uTest showed
that almost 40% of testers spent
between 50% and 100% on non
testing activities per week.”

7. Sources: Applied Software Measurement: Global Analysis of
Productivity and Quality by Capers Jones, & IBM The Future of Testing
Most forms of testing find fewer than 30% of all bugs
almost 40% of testers have seen an increase in the frequency of release
Productivity and quality are directly coupled: projects with
high quality have high productivity and vice versa

8. How can
continuous
testing help?

9. Agile gave us..

10. What does
this really
mean???

11. the test and QA
headache is
compounded
further by the need
to release faster
and more often
Amazon is on record as making changes
to production every 11.6 seconds on
average in May of 2011.
Facebook releases to production twice a
day.
Many Google services see releases
multiple times a week, and almost
everything in Google is developed on
mainline.
Etsy push unto 70 changes a day to its
monolithic software!

12. Key Concepts
>>> continuous
testing >>>

13. Continuous Testing
…not just about test
automation! (ignore
wikipedia)
Actually includes, but not limited to:
Automated testing
Automated management of production and non
production environments
Automated application feedback through
monitoring
Automated evaluation of business objectives
Evaluating product concepts
Testing
engaging in XP practices
organisational change

14. The DevOps movement is built
around a group of people who
believe that the application of a
combination of appropriate
technology and attitude can
revolutionize the world of software
development and delivery.
Stephen Nelson-Smith - 2010
its about:
Automating building, testing, and
deployment
Bringing consistency to the way we
use and create technical assets
Providing teams with the tools
required to achieve technical
excellence
Continuous
testing is
dependent on
DevOps
Activities

15. Build Agent & Devel
The continuous test flow
Pre commit tests
Code review
PO/UX review
Branch / Merge
Build
Unit tests
Contract tests
Component tests
Coverage
Static analysis
Component
performance
End to end
component tests
NFR tests
Exploratory Test
performance testing
Story sign off /
Demos
NFRs
Analytics
Monitoring
Production tests
canary or blue /
green release
CI Environments
Developer
Local
Pre-Production Production
Virtualised Full stack environments
Commit Auto-Deploy Promote Release
Build Pipeline:
There can be multiple pipelines, usually per system, should execute in minutes
Commit:
The commit can include changes to the infrastructure, database, or application code.
Outputs:
Software, infrastructure and configuration
Environment manifests
Release notes

16. the cycle
begins on a
developers
machine
pair programming
pair testing

17. automate the
build
Reduce general coding errors, create a single source of
truth, ensure basic component integration

18. vs.
script & test your
configuration!
say NO to hand crafted configuration files!
remove configuration issues by scripting your configuration.

19. automate & test
the infrastructure
Remove configuration defects by having a repeatable
mechanism for creating environments.
Over 70% of testers are delayed
on a regular basis by environment
issues

20. automate & test
deployments
say NO to sneaker net
say NO to manually to copying files
say NO to manually pulling and pushing servers in and out of load balancers
deploying software is one of the
most mission critical aspects of
what we do, let a machine do it!

21. vs.automate the testing
continuous testing does away
with the concept of
regression testing.
..an effective software delivery process
requires no regression testing stages. It
becomes integral to everything we do..
write tests for:
Configuration
Functionality
Deployments
Infrastructure

22. monitoring &
Alerting
One of the greatest and
overlooked tools in the arsenal of
the tester is monitoring.
Realtime access to
application logs
Realtime access to
error logs
Realtime access to
infrastructure logs
Testing also
occurs in
production.

23. “A good, easily accessible
event & error monitoring
system is essential for anyone
releasing software. Build an
early warning system into your
environment!”

24. “Outputting more logs and
events is as good as having a
houseful of automated
checks!”

25. If your system is not producing any or enough
data, ask your developers to change that
Log technical and business events
Log successes as well as failures
Use tools to capture logs that allow
visualisation

26. “Understanding your systems
steady state allows you to react
quickly to any deviation”
Remember:

27. Make sure you get notified of unusual
behaviour
Act on unusual behaviour
Share the data

28. Create a
single view
of quality

29. smaller,
more
frequent
deliveries

30. Conclusion
Small batches, lots of feedback, automation of complex tasks, configuration
management, automated deployments, automate your build

31. Thank
you!
TWITTER
@petemar5hall
Agile Consultant & Technical
Architect @planday
Peter Marshall
fiveMAIL
info@leansoftwareservices.com
LINKEDIN
https://www.linkedin.com/in/
petedmarshall