This is more of Day 5 covering buld and test

1. Architecting for the Cloud
Len and Matt Bass
Build/Test

2. Overview
• What is the problem?
• Deployment pipeline
• Integration testing
• Staging

3. Goal of Developers
• Get code into production with minimum errors and
maximum speed
• In this lecture we are concerned with
– Errors caused by working in a team
– Errors caused by having multiple versions of a system
simultaneously
– Automating the testing process
• Errors equate to delays since it takes time to fix them.
• Between team coordination is another source of
delays. We have already seen a coordination-free
architectural style.

4. Errors caused by working in a team
• Consider the following scenarios
– A team member suddenly leaves.
– One team member overwrites the code of
another team member
– Code written by one team member must be
integrated with code written by another
• Preventing these errors is the purpose of a
configuration management tool.

5. Errors caused by having multiple
versions of a system simultaneously
• Consider the following scenarios
– A version of the system is in production and a new
version is being tested
– A problem develops in the production version and
• The problem can be fixed without affecting either the
production version or any development versions
– Two different members of the team are working
on different versions of the system
• Preventing these errors is the purpose of a
version control system

6. Branches
• Version control systems
– Contain a main “trunk”
– Support the creation of branches for particular sets of
modifications.
• Branches allow for independent evolution of different
streams of work.
– Might have production version, version being UAT tested,
version with new features, version with bug fix being tested
• Once a branch has been released into production, it should
be merged back into the trunk
– May be time consuming depending on how trunk has evolved
since the creation of the branch (delay).
• Best practice is to restrict the number of branches.

7. Overview
• What is the problem?
• Deployment pipeline
• Integration testing
• Staging

8. The Deployment Pipeline
Version control and configuration management are combined into a
continuous integration server.
• Build and integration tests are performed as a portion of
continuous integration
• User Acceptance Testing and performance testing are performed as
the last stage before deploying to production. We will call these
steps staging.
• A developer commits to a branch and the remainder of the
deployment pipeline is for that branch.

9. Environments
• Before we worry about delivery,
we need to worry about the
steps preceding it.
• An environment is a set of
software running on a collection
of virtual machines with defined
external connections.
– Entry is always through a load
balancer
– External references are through
URLs
An environment

10. Changing environments
• Changing the external
connections changes
the behavior of the
system.
• The difference
between using a
production database
and a test database is
the URL of the
database.
Production Database Test Database

11. • Developer tests locally and commits latest version of their module to version
control system
• Build is triggered that draws latest version of each module used from version
control system
• If build is successful a machine image is stored in the build environment.
• If build unsuccessful, a message is sent to the development team.
• If developer’s module does not compile then not only that build will break but
builds by any team member whose module depends on the newly committed
module will also break. (delay)
What happens when a developer
commits a change to a branch?
Build environment
Success
Failure
Pre-committests(local)
VersionControlSystemPre-committests(local)
...
CompileandBuild
Commit

12. Overview
• What is the problem?
• Deployment pipeline
• Integration testing
• Staging

13. Integration Tests
• If the newly committed module
does not pass testing then testing
any commit by other team
members will also fail.
• Continuous integration servers can
be used to test a build prior to
committing.
• Fixing a broken build or broken
integration test is high priority.
• Tests can use copy of production
environment for services not a
portion of newly built process.
• URLs used to specify location of
services
– build environment
– copy of production environment Build
Environment
Test Harness
Test Input
Test output
Copy of
Production
Environment

14. Managing the database for integration
testing
• Integration testing will read and write a
database.
• Use a separate test database to prevent
impacting the production database
• The test database is preloaded with test data.
• Executing the tests will produce writes to the
database.
• Check the state of the database as a portion of
verifying the tests.

15. Complications with test database
• The initial state of the database may not be the
same from one test sequence to another because
tests write to the database
• Reinitialize every time a test sequence is run
• The sequence of tests is important. Some tests
may be dependent on the results of other tests
• Create account must be done before
• Add money to account
• The correctness of the database after tests are
run must be tested by a special utility designed
for that purpose.

16. Feature Toggles (switches/flags)
• A feature toggle is an alternative to creating a new
branch
• If “toggle is on” then
execute new code
else execute old code
• Widely used in internet companies such as Facebook,
Google, Netflix.
• Requires feature toggle manager to coordinate the
setting of global values for feature toggles.
– Archaius is open source system released by Netflix that,
among other things, acts as a feature toggle manager.

17. Pluses and Minuses of Feature Toggles
• Pluses
– Allows uncompleted code to be checked in. Toggle feature
off and it will not be executed during test and, hence, will
not break the build
– Allows for flexible deployment process. We will see this
later.
• Minuses
– Clutters code.
– Misuse of feature toggle can have serious consequences.
Knight Capital lost (US) $440 Million in 45 minutes partially
as a result of a repurposed feature toggle.
• Best practice: Remove feature toggles once the code
has been committed to production.

18. Overview
• What is the problem?
• Deployment pipeline
• Integration testing
• Staging

19. Staging Testing
• After successful integration
testing
• Machine image does not
change from successful
integration testing
• Mirror input from
production system to
create an input stream for
performance testing
• Production system input
stream will not test new
features. Additional tests
are required for new
features. Staging
Environment
Production
Environment

20. Staging Testing Database
• Make copy of production database.
– Assumes reasonable size production database
– Some cloud vendors make hourly back ups of
production database, can use one of these as test.
• Ensure writes to database do not affect
production database
• After test completes you will need to check
test database for correctness.

21. Where do tests come from?
• Test Driven Development advocates writing
tests before developing a feature.
– Results in a collection of tests
• Regression tests – past problems with the
system result in a collection of test cases
• Edge/boundary cases – how well are edge
cases handled?
• Error cases – what happens when erroneous
input occurs?

22. Additional tests
• Static analyzers
– Analyze code for known problems. E.g. SQL
injection, buffer overflow
– Detect some problems at code level with
modularity
• Dynamic analyzers
– Mainly for security
– Penetration testing. Looks for back door ways into
a system.

23. Continuous Deployment/Delivery
• Continuous deployment and continuous delivery are
two terms meaning almost the same thing.
• Both terms imply automated movement between
stages of the deployment pipeline.
• Continuous delivery: a developer’s commit triggers a
sequence that automatically results either in failure to
pass tests or in a machine image prepared for
production, i.e. in UAT environment.
• Continuous deployment: a developer’s commit triggers
a sequence that automatically results either in failure
to pass tests or in a machine image being placed into
production, i.e. moved from UAT into production.

24. Continuous Delivery
• Not all organizations can adopt continuous delivery.
– Banks, for example, require that a human certify any release
– A human examines the output of the AUT/Performance testing
to ensure compliance with various financial regulations
• Continuous delivery assumes a good test suite.
• Some organizations will use continuous delivery with quick
roll backs in case of errors (live testing).
• Continuous delivery
– Virtues – reduces manual intervention and speeds up the
delivery process.
– Drawbacks - implies adherence to a number of constraints that
we will discuss.

25. Configuration Parameters
• All external references from the environment are specified by URLs
– These constitute the difference between operating in a build
environment, the UAT environment, or the production
environment
• URLs for the different environments can be specified in a database.
• Other configuration parameters can also be specified in the database.
These may vary based on environment or circumstances. E.g.
– Feature toggle values
– Maximum number of virtual machines in autoscaling group.
• At start up, the system is informed of the location of its configuration
parameters, it reads them in, and uses them to perform its duties.

26. Tools and Scripts
• Specialized tools exist for the different stages
of the deployment pipeline
– Continuous integration tools
– Environment specification and migration tools
• Programs for these tools are called “scripts”
• One (and only one) script should exist that
invokes a system and tells it where to find its
environmental parameters.

27. Traceability
• Everything to goes into a component in
production should be traceable
– What version of which modules are included in the
component
– What version of the database is being accessed
– What version of dependent systems are being used
– What configuration parameters were set
– What script was used to build the environment and
place the system into production.
– What values of the features toggles were used.
– …

28. Achieving traceability
• All of the information could be available from a version
control system
• This requires that everything be version controlled –
code, scripts, and dependent systems.
• During operational problems, there is a tendency to
modify the production system via the console. This is
not traceable.
• Best practice is to lock down the environment creation
and modification process to authorized users and
require everyone to go through the process of
modifying a script and checking it in before using it.

29. Architectural implications of build and
test
• Use feature toggles
• Interact with external world only through
URLs
• Small modules are easier to test.

30. Summary
• The deployment pipeline describes the steps and the processes to
move code from development to production
– Build
– Staging
– Production
• Associating an environment with each step and restricting
interaction from within the environment to outside the
environment to URLs allows simpler movement through the steps
• Continuous deployment has movement through all of the steps
automated
• Everything should be configuration controlled and access to the
production system should be restricted to configuration controlled
items.