Architecting for the cloud storage build test


Published on

This is more of Day 5 covering buld and test

Published in: Software
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Architecting for the cloud storage build test

  1. 1. Architecting for the Cloud Len and Matt Bass Build/Test
  2. 2. Overview • What is the problem? • Deployment pipeline • Integration testing • Staging
  3. 3. Goal of Developers • Get code into production with minimum errors and maximum speed • In this lecture we are concerned with – Errors caused by working in a team – Errors caused by having multiple versions of a system simultaneously – Automating the testing process • Errors equate to delays since it takes time to fix them. • Between team coordination is another source of delays. We have already seen a coordination-free architectural style.
  4. 4. Errors caused by working in a team • Consider the following scenarios – A team member suddenly leaves. – One team member overwrites the code of another team member – Code written by one team member must be integrated with code written by another • Preventing these errors is the purpose of a configuration management tool.
  5. 5. Errors caused by having multiple versions of a system simultaneously • Consider the following scenarios – A version of the system is in production and a new version is being tested – A problem develops in the production version and • The problem can be fixed without affecting either the production version or any development versions – Two different members of the team are working on different versions of the system • Preventing these errors is the purpose of a version control system
  6. 6. Branches • Version control systems – Contain a main “trunk” – Support the creation of branches for particular sets of modifications. • Branches allow for independent evolution of different streams of work. – Might have production version, version being UAT tested, version with new features, version with bug fix being tested • Once a branch has been released into production, it should be merged back into the trunk – May be time consuming depending on how trunk has evolved since the creation of the branch (delay). • Best practice is to restrict the number of branches.
  7. 7. Overview • What is the problem? • Deployment pipeline • Integration testing • Staging
  8. 8. The Deployment Pipeline Version control and configuration management are combined into a continuous integration server. • Build and integration tests are performed as a portion of continuous integration • User Acceptance Testing and performance testing are performed as the last stage before deploying to production. We will call these steps staging. • A developer commits to a branch and the remainder of the deployment pipeline is for that branch.
  9. 9. Environments • Before we worry about delivery, we need to worry about the steps preceding it. • An environment is a set of software running on a collection of virtual machines with defined external connections. – Entry is always through a load balancer – External references are through URLs An environment
  10. 10. Changing environments • Changing the external connections changes the behavior of the system. • The difference between using a production database and a test database is the URL of the database. Production Database Test Database
  11. 11. • Developer tests locally and commits latest version of their module to version control system • Build is triggered that draws latest version of each module used from version control system • If build is successful a machine image is stored in the build environment. • If build unsuccessful, a message is sent to the development team. • If developer’s module does not compile then not only that build will break but builds by any team member whose module depends on the newly committed module will also break. (delay) What happens when a developer commits a change to a branch? Build environment Success Failure Pre-committests(local) VersionControlSystemPre-committests(local) ... CompileandBuild Commit
  12. 12. Overview • What is the problem? • Deployment pipeline • Integration testing • Staging
  13. 13. Integration Tests • If the newly committed module does not pass testing then testing any commit by other team members will also fail. • Continuous integration servers can be used to test a build prior to committing. • Fixing a broken build or broken integration test is high priority. • Tests can use copy of production environment for services not a portion of newly built process. • URLs used to specify location of services – build environment – copy of production environment Build Environment Test Harness Test Input Test output Copy of Production Environment
  14. 14. Managing the database for integration testing • Integration testing will read and write a database. • Use a separate test database to prevent impacting the production database • The test database is preloaded with test data. • Executing the tests will produce writes to the database. • Check the state of the database as a portion of verifying the tests.
  15. 15. Complications with test database • The initial state of the database may not be the same from one test sequence to another because tests write to the database • Reinitialize every time a test sequence is run • The sequence of tests is important. Some tests may be dependent on the results of other tests • Create account must be done before • Add money to account • The correctness of the database after tests are run must be tested by a special utility designed for that purpose.
  16. 16. Feature Toggles (switches/flags) • A feature toggle is an alternative to creating a new branch • If “toggle is on” then execute new code else execute old code • Widely used in internet companies such as Facebook, Google, Netflix. • Requires feature toggle manager to coordinate the setting of global values for feature toggles. – Archaius is open source system released by Netflix that, among other things, acts as a feature toggle manager.
  17. 17. Pluses and Minuses of Feature Toggles • Pluses – Allows uncompleted code to be checked in. Toggle feature off and it will not be executed during test and, hence, will not break the build – Allows for flexible deployment process. We will see this later. • Minuses – Clutters code. – Misuse of feature toggle can have serious consequences. Knight Capital lost (US) $440 Million in 45 minutes partially as a result of a repurposed feature toggle. • Best practice: Remove feature toggles once the code has been committed to production.
  18. 18. Overview • What is the problem? • Deployment pipeline • Integration testing • Staging
  19. 19. Staging Testing • After successful integration testing • Machine image does not change from successful integration testing • Mirror input from production system to create an input stream for performance testing • Production system input stream will not test new features. Additional tests are required for new features. Staging Environment Production Environment
  20. 20. Staging Testing Database • Make copy of production database. – Assumes reasonable size production database – Some cloud vendors make hourly back ups of production database, can use one of these as test. • Ensure writes to database do not affect production database • After test completes you will need to check test database for correctness.
  21. 21. Where do tests come from? • Test Driven Development advocates writing tests before developing a feature. – Results in a collection of tests • Regression tests – past problems with the system result in a collection of test cases • Edge/boundary cases – how well are edge cases handled? • Error cases – what happens when erroneous input occurs?
  22. 22. Additional tests • Static analyzers – Analyze code for known problems. E.g. SQL injection, buffer overflow – Detect some problems at code level with modularity • Dynamic analyzers – Mainly for security – Penetration testing. Looks for back door ways into a system.
  23. 23. Continuous Deployment/Delivery • Continuous deployment and continuous delivery are two terms meaning almost the same thing. • Both terms imply automated movement between stages of the deployment pipeline. • Continuous delivery: a developer’s commit triggers a sequence that automatically results either in failure to pass tests or in a machine image prepared for production, i.e. in UAT environment. • Continuous deployment: a developer’s commit triggers a sequence that automatically results either in failure to pass tests or in a machine image being placed into production, i.e. moved from UAT into production.
  24. 24. Continuous Delivery • Not all organizations can adopt continuous delivery. – Banks, for example, require that a human certify any release – A human examines the output of the AUT/Performance testing to ensure compliance with various financial regulations • Continuous delivery assumes a good test suite. • Some organizations will use continuous delivery with quick roll backs in case of errors (live testing). • Continuous delivery – Virtues – reduces manual intervention and speeds up the delivery process. – Drawbacks - implies adherence to a number of constraints that we will discuss.
  25. 25. Configuration Parameters • All external references from the environment are specified by URLs – These constitute the difference between operating in a build environment, the UAT environment, or the production environment • URLs for the different environments can be specified in a database. • Other configuration parameters can also be specified in the database. These may vary based on environment or circumstances. E.g. – Feature toggle values – Maximum number of virtual machines in autoscaling group. • At start up, the system is informed of the location of its configuration parameters, it reads them in, and uses them to perform its duties.
  26. 26. Tools and Scripts • Specialized tools exist for the different stages of the deployment pipeline – Continuous integration tools – Environment specification and migration tools • Programs for these tools are called “scripts” • One (and only one) script should exist that invokes a system and tells it where to find its environmental parameters.
  27. 27. Traceability • Everything to goes into a component in production should be traceable – What version of which modules are included in the component – What version of the database is being accessed – What version of dependent systems are being used – What configuration parameters were set – What script was used to build the environment and place the system into production. – What values of the features toggles were used. – …
  28. 28. Achieving traceability • All of the information could be available from a version control system • This requires that everything be version controlled – code, scripts, and dependent systems. • During operational problems, there is a tendency to modify the production system via the console. This is not traceable. • Best practice is to lock down the environment creation and modification process to authorized users and require everyone to go through the process of modifying a script and checking it in before using it.
  29. 29. Architectural implications of build and test • Use feature toggles • Interact with external world only through URLs • Small modules are easier to test.
  30. 30. Summary • The deployment pipeline describes the steps and the processes to move code from development to production – Build – Staging – Production • Associating an environment with each step and restricting interaction from within the environment to outside the environment to URLs allows simpler movement through the steps • Continuous deployment has movement through all of the steps automated • Everything should be configuration controlled and access to the production system should be restricted to configuration controlled items.