Integrating Automated Testing into DevOps

TE
AM Tutorial
10/14/2014 8:30:00 AM
"Integrating Automated Testing into
DevOps"
Presented by:
Jeff Payne
Coveros, Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Jeff Payne
Coveros, Inc.
Jeff Payne is CEO and founder of Coveros, Inc., a software company that builds secure
software applications using agile methods. Since its inception in 2008, Coveros has become a
market leader in secure agile principles and was recognized by Inc. magazine as one of the
fastest growing private US companies. Prior to founding Coveros, Jeff was chairman of the
board, CEO, and cofounder of Cigital, Inc., a market leader in software security consulting. Jeff
has published more than thirty papers on software development and testing, and testified before
Congress on issues of national importance, including intellectual property rights, cyber
terrorism, and software quality. Follow Jeff on Twitter @jefferyepayne.

1© Copyright 2014 Coveros Corporation. All rights reserved.
Integrating Automated Testing
into DevOps
Jeffery Payne
@jefferyepayne

2© Copyright 2014 Coveros, Inc.. All rights reserved.
Bio
Jeffery Payne
jeff.payne@coveros.com
@jefferyepayne
Jeffery Payne is CEO and founder of Coveros, Inc., a software company that
helps organizations accelerate the delivery of secure, reliable software. Coveros
uses agile development methods and a proven software assurance framework to
build security and quality into software from the ground up. Prior to founding
Coveros, Jeffery was Chairman of the Board, CEO, and co-founder of Cigital, Inc.
Under his direction, Cigital became a leader in software security and software
quality solutions, helping clients mitigate the risk of software failure. Jeffery is a
recognized software expert and popular speaker at both business and technology
conferences on a variety of software quality, security, and agile development
topics. He has also testified before Congress on issues of national importance,
including intellectual property rights, cyber-terrorism, software research funding,
and software quality.

 Coveros helps organizations accelerate the delivery of secure, reliable
software
 Our consulting services:
– Agile software development
– Agile testing & automation
– Application security
– DevOps automation
– Agile training & coaching
 Our key markets:
– Financial services
– Healthcare
– Defense
– Critical Infrastructure
Areas of Expertise
About Coveros

Agenda
 What is DevOps?
– Definition(s)
– Common terminology
– Common architecture
 Automated CI Testing
 Automated CD Testing
 Demos of various types of automated testing

What is DevOps?

What is DevOps?
Common Definition
 DevOps is a software development method that stresses
communication, collaboration and integration between
software developers, QA, and information technology (IT)
professionals. DevOps is a response to the
interdependence of software development and IT
operations. It aims to help an organization rapidly produce
software products and services.
 DevOps is not a tool
 DevOps is not a methodology
 DevOps is a philosophy for how people in the software
development supply chain interact

What is DevOps?
DevOps practices include processes and tools that streamline
software delivery by improving feedback from production to
development and automating delivery activities.
○ Processes
■ Alignment of responsibilities with Artifacts
■ Well defined (and automated!) pipeline from code development to
production environments (ideally)
■ Implementation within agile iterations/sprints further support agile
principles
○ Tools
■ Tools to automate and manage builds
■ Tools to automate testing and results reporting
■ Tools to provision environments (physical and virtual, privide and cloud)
■ Tools to orchestrate continuous delivery of applications

What is DevOps?
Development vs. Operations - The Essential Conflict
● Development
○ Need for Change - Desire for rapid rollout to provide value to Customers.
Customers often demand change, not weighing risk
■ New Features
■ Feature Changes
■ Bug Fixes
● Operations
○ Fear of Change - Desire for stable conditions for production systems
■ Server Uptime, Reliability, Availability,
■ Capacity, response times
■ Security
■ SLAs (contracts)
■ This conflict results in quality and agility problems

 It compiles, therefore it works!
 Big bang integration
 Throwing software over the fence to late lifecycle QA
 “It works on my machine”
 “We’ll get that QA environment stood up for you by … 2016”
DevOps helps solve common quality/agility problems
What is DevOps?

What is DevOps?
DevOps Terminology
 Build automation – Automating the compilation, linking, and
packaging of software applications
 Continuous integration – Leveraging build automation
capabilities to integrate and test software on a regular basis
to thwart integration and quality issues earlier in the process
 Check-in builds – Builds performed whenever code is
changed. Often includes unit tests and code quality checks.
 Nightly builds – Integration builds performed nightly to
assure nothing changed during the day breaks existing
functionality. Regression testing included. Platform TBD.

What is DevOps?
DevOps Terminology
 Continuous delivery – A series of practices to assure code
can be rapidly and safely deployed to downstream
environments (QA, staging, production, etc.) by delivering
every change to downstream environments while ensuring
functionality works as expected through automated testing.
 Continuous deployment – A series of practices to assure
code can be rapidly and safely deployed to production by
delivering every change to a production (or production-like)
environment while ensuring functionality works as expected
through automated testing.

When to Test
● Testing practices should be broken up into different
environments
○ Development env
■ Unit Testing
■ Acceptance Tests
■ Sanity Testing
○ Continuous Integration env
■ Unit Testing
■ Acceptance Tests
■ Incremental integration testing
○ QA env
■ Smoke testing
■ Integration testing
■ Functional Testing
■ Usability testing
■ Compatibility testing
■ Install/uninstall testing
■ Regression testing
○ Staging env
■ Performance Testing
■ Stress Testing
■ Load testing
■ End-to-end testing
■ System Testing

Environments
● While continuously testing and building software is
important for a robust final product, if it difficult to
continuously release software
● Having multiple environments allows for continuous
development and low level testing to continue, while
having more static versions of the software available for
testers
● Ideally, a set of 4 or 5 environments will exist
○ Development
○ Dev Integration (CI)
○ QA
○ Staging
○ Production

Environments (cont.)
● Development
○ Should be updated continuously, always with the most up to date code
that passed all static analysis, unit tests, and coverage bars
● QA
○ Should be updated nightly, always with the last code to pass all bars
○ Updating more frequently will cause manual and longer regression testing
issues
● Staging
○ Somewhere with stable code, mimicking production
○ Used for end-to-end testing and non-functional testing that must be
performed on production-like environments
● Production
○ Should be updated with releases only, always with stable fully testing
code

What is DevOps?
Simple Example DevOps architecture
Dev Int QA Staging Production

Automating CI Tests

Continuous Integration
● Continuous Integration will enable the integration of
multiple software, data, and tests from multiple teams
into one robust, cohesive application
● CI combines the use of automated build tools in
conjunction with automated testing and deployment tools.
● The continuous integration effort will promote and
coordinate functionality between developers rapidly,
while also delivering an effective automated defect
identification and resolution.

Continuous Integration (cont.)
● Each level of testing will be dependent on previous levels
succeeding to keep out bad code, and new successful
builds being processed at a high rate.
● All users involved in this effort, should be able to see the
iterative functionality, giving a window into development
processes and quick knowledge on the results of each
piece of functionality.
● Monitoring build will provide quick and easy access to
failures, which will shorten the timeline to fix issues and
increase overall quality.

 Developer commits code to a branch
 CI server detects changes and fires off a build request
– Static analysis is performed on code
– Unit tests are execute
– Code coverage is calculated on units
 If build/testing is successful, branch merged to master
 Appropriate applications are built and put into repository
– Smoke testing is performed on new version in QA if possible
 When the build breaks (either code does not complete or
does not pass testing/quality checks), team is obligated to
FIX THE BUILD.
 Do nightly builds go here?
Typical CI Process
Automating CI Tests

20© Copyright 2014 Coveros, Inc.. All rights reserved.© Copyright 2014 Coveros, Inc. All rights reserved.
Common Tools

Source Control
● Source control and tight integration with build automation
tools is a key element of a successful CI implementation
● The ability to use source/version control technology is
fundamental to
○ Ability for teams to work on a code base with out clobbering each others
changes
○ Meet rapid build and deployment objectives
○ Assure that identified defect identification are tied to correct code
version
○ Proper rollbacks when mistakes are made and problems cannot be
debugged quickly
● Code should be checked in regularly for a CI process to
be most effective

● Git
○ Decentralized, and aims to be
fast, flexible, and robust
● Subversion
○ Open source versioning control
system inspired by CVS
● CVS
○ Built on RCS, with emphasis put
on security and source code
correctness
● Vesta
○ Build system with a versioning
file system and support for
distributed repositories
Source Control - Examples
Git Flow Example within CI

Build Automation
● Build Automation is the scripting of common tasks
performed by the software developers for:
○ Extracting the appropriate version of the source code from a source-
code repository;
○ Assuring the necessary compilation dependencies are in place;
○ Compiling source code into binary objects while referencing the
compilation dependencies;
○ Packaging binary objects;
○ Assuring the necessary unit testing dependencies are in place;
○ Executing tests while referencing the testing dependencies;
○ Managing the artifacts created during the process.
● All software developers perform these operations to
some degree or as part of their daily tasks for the
modules to which s/he is contributing.
● Tools: Ant, Anthill, Maven, make

CI Servers
● CI Servers have become increasingly popular to help
manage the continuous integration process
● They provide a central location for all code to be
examined, compiled, and executed
● Most of these tools provide a web interface for
○ Viewing the status of any of the modules on the project
○ Maintaining a history of each build outcome
○ A detailed account of what happened: These details usually lead to the
cause of the failure.
● While these tools provide information about the health of
the code, they are still no replacement for developer run
acceptance tests

Build Tools - Examples
● Jenkins / (fka Hudson)
○ Open source continuous integration
tool written in Java
● CruiseControl
○ Open source continuous integration
tool written for Java-based
frameworks
● Bamboo
○ Bamboo is a continuous integration
server from Atlassian
● TeamCity
○ continuous integration and build
management

Test Tools
● The goal of test automation within CI is to identify coding
and subsystem errors as close to their introduction as
possible
● Supports the notion that code is not “done” until it is
tested
● The amount (and levels) of testing performed during CI
depends upon the environments you have control over
and the amount of time it takes to build and run your
application.

Test Tools - Static Analysis
● Test automation begins with static analysis
● Before any code is compiled or tested, it needs to meet
specific requirements
● Each time code is checked in, a style checker should run
over the code to checking for
○ layout issues
○ comments
○ documentation
● A rating will be assigned to each piece of new code, and
code failing to meet a certain bar should be rejected

Test Tools - Static Analysis (cont.)
● Again, before the code is even compiled, a code checker
should be run over the source code to check for common
programming flaws.
● This can catch things such as
○ Unused variables
○ Empty catch blocks
○ Unnecessary object creation
○ Duplicate code

Test Tools - Static Analysis (cont.)
● A bug checker should be run over the un-compiled code
to check for common bugs
○ This will provide an extremely quick check for potential issues and most
tools rank them in severity
○ If no level 1, 2 or 3 level issues are found, the application and test code
should finally be compiled.
● After the code is compiled, design quality metric tools can
traverse the compiled directories to generate design
quality metrics
● Depending on the tool, reports can display
○ extensibility, reusability, and maintainability
○ dependencies, and cycles for each analyzed package

Static Analysis Tools - Examples
● Ruby Stack
○ rubocop
○ Rails best practices
○ Cane
○ Reek
● Java Stack
○ Checkstyle
○ PMD
○ Findbugs
○ JDepend
○ .Net Stack
○ StyleCop
○ SourceMonitor
○ Clone Detective
○ NDepend

Test Tools - Unit Tests
● Unit tests are simple and each should
○ Look at each singular piece of functionality
○ Have a fixed set of datum
○ Expect deterministic results
● A unit test provides a strict, written contract that the piece
of code must satisfy.
● All developers should be responsible to provide unit tests
to test and prove the completeness and soundness of
their checked-in code
● All unit tests should be expected to pass for the
application to move onto the next phase in testing

Test Tools - Unit Tests (cont.)
● The goal of unit testing is to isolate each part of the
program and show that the individual parts are correct
● Unit Testing affords several benefits
○ Finds problems early
○ Facilitates change
○ Simplifies integration
○ Provides documentation
● Unit testing only tests the functionality of the units
themselves, and will not catch integration errors or
broader system-level errors.

Unit Testing Tools - Examples
● JUnit
○ A popular framework for writing unit tests in Java/J2EE environment
● TestNG
○ Similar to jUnit, but provides dependencies and expanded annotations
● NUnit
○ includes GUI, command line, integrates into Visual Studio
● MSTest
○ A command-line tool for executing Visual Studio created unit tests
outside of the Visual Studio IDE
○ Not really a testing framework as it is a part of the Visual Studio Unit
Testing Framework
● RSpec
○ Behaviour-driven development framework

CI Test Tools - Code Coverage
● The last stage of testing at the code level will be to run a
coverage tools to examine the previous tests run
● The amount of code accessed by the unit tests will be
calculated, and based on the result, the completeness of
the unit tests will be determined
● Code lacking proper coverage will fail this test, and the
code be sent back for further development
○ Usually this means developers will need to add more unit tests, not fix
their code
○ Code Coverage Tools: Cobertura (Java), NCover (.NET),
Simplecov (Ruby)

CI Testing Tools - Reporting
● Multiple tools exist outside of build tools to provide further
analysis on the application
● Many of these tools, such as SonarQube or HP Quality
Center, allow the centralization of all metrics into one
area

Demo
Demo of CI Environment

CI Demo

Automating CD Tests

Automating CD Tests
Tests
 Provisioning
 Deployment tests
 Integration smoke tests
 Security tests
 Others?

Continuous delivery
● Continuous Delivery enables integration and testing on downstream
environments on a continuous basis to assure software works
properly in these environments
● CD is typically orchestrated by a CI Server and a Deployment
Manager
● The continuous delivery effort will promote and coordinate
functionality between developers, QA, and operations rapidly to make
sure the software works in all environments
● Also supports automated setup, teardown, and use of testing
environments if possible

 New build has been promoted to repository after having
passed smoke tests
 Application is deployed onto stable QA environment for
testing. If possible, QA environment is automatically
provisioned either locally or in the cloud
– Automated and manual functional testing is performed
– Regression run as well to assure quality
– Non-functional testing is performed as appropriate
– At appropriate times based upon environment, test environments
are torn down
 Application is deployed onto production-like Staging
environment. If possible Stating is automatically
provisioned either locally or in the cloud
– Tests are performed that need to be run on a production-like env
 Application is provisioned and deployed in production
Typical CD Process
Automating CD Tests

Common Tools

Artifact Repository’s and Repository Manager’s
● Artifact Repository’s provide a structured destination for
artifact’s produce during the software build process
● Repository Manager’s provide a centralized mechanism
for accessing artifacts within both public and private
repository’s, increasing speed and efficiency of building
and accessing artifacts
● Artifact Repository’s – Maven, Alfresco, Yum
● Repository Manager’s – Apache Archive, Artifactory,
Sonatype Nexus

CI Servers in CD
● CI Servers typically control CD activities now as well
● Server Scripts are written to push new releases into
downstream environments
● These scripts can be launched manually with a “button
push” or tied together into an automated delivery pipeline
that moves new versions of applications through
environments automatically as tests are passed
● Without effective testing, Continuous Delivery (and also
DevOps in general) cannot be optimized!
● Bad automated tests used to result in bad test results faster
● Not bad automated tests result in bad code being deployed!

 IT Automation Tools allow teams to define environment,
system, and application configurations and parameters such
that downstream environments can be automatically
provisions and any conflicts between expected application
parameters can be identified
 Significantly reduces the amount of operational IT support
needed for environment setup, application
installation/configuration, and teardown necessary to move
applications from development to production
 80-90% decrease in ops support with accelerated delivery
 Common Tools: Puppet, Chef, Ansible
IT Automation Tools

Test Tools for CD
● Test tools used within CD are typically the transitional
subsystem, integration, and system test tools used in
traditional processes.
● Difference is that these tests are run much more
frequently than in the past
● There is also more emphasis on testing of functionality
below the UI so integration tests can be run earlier in the
development process and are less brittle
● Common Tools: Selenium, FIT & FITness, Cucumber,
QTP

Demo
Demo of CD Environment

Wrap-Up

Questions?
Contact Information:
jeff.payne@coveros.com
703.431.2920
@jefferyepayne
http://www.coveros.com

Integrating Automated Testing into DevOps

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Integrating Automated Testing into DevOps

Similar to Integrating Automated Testing into DevOps (20)

More from TechWell

More from TechWell (20)

Recently uploaded

Recently uploaded (20)

Integrating Automated Testing into DevOps