What can be done, when the test automation coverage is poor and the project wants to move into continuous delivery mode. Speech at the Ohjelmistotestaus2016 conference, 24th of May 2016, Crowne Plaza Helsinki, Finland

1. CONTINUOUS DELIVERY AND
THE CHALLENGES FOR THE
TEST AUTOMATION
DISA KAKKO (MAY 2016)
DISA.KAKKO@BASWARE.COM
HTTPS://WWW.LINKEDIN.COM/IN/DISAKAKKO

3. CONTINUOUS DELIVERY- IDEAL SITUATION
• Continuous integration against continuous test automation.
• Continuous automatic builds and release packages.
• Automatic deployment to the QA and production environments (includes
automatic verification of the deployment successfulness).
• QA personnel is focused on testing the new functionality with exploratory
testing. Test automation covers smoke and regression testing of the
existing functionality on all test automation levels.
• Monitoring data from the production environment is gathered and the
information is utilized to develop each phase preceding the production
deployment.

4. REALITY?
• Some of the teams in the same SW program/project are doing test automation
for all levels of sw (unit/API/system), but not all teams. In some cases some test
automation level is skipped completely (e.g. unit tests).
• There is no pressure to do test automation since the QA personnel will manually
test all new features and the existing functionality for every code change.
• DoD does not exist or it is not followed.
• Release packaging and deployment are done manually. Deployment verification
is done by performing manual tests.
• Monitoring of the production environment does not exist or the monitoring data
is not utilized by the R&D.

5. WHAT KIND OF TEST AUTOMATION
COVERAGE CONTINUOUS DELIVERY
REQUIRES

6. CHALLENGES IN THE SOFTWARE
DEVELOPMENT- CONSEQUENCES TO THE
TEST AUTOMATION?

7. WHAT CAUSES THE AVALANCHE?
• ”We’re too busy” is often the excuse for dismissing the needed internal QA
actions for R&D (refactoring, test automation, static analysis, etc). It is more
important to get features released, than to do them properly the first time
(which would mean test automation and non-functional aspects taken into
account for the new functionality).
• QA actions for the R&D (such as test automation or refactoring) is put to the
same backlog with the new features. This way the responsibility of doing
quality is outsourced to the product management.
• If there is a pressure from the sales promises for the new features, the
internal QA actions for the R&D (such as test automation work) is moved
down in the backlog. Eventually test automation does not get done. -> test
automation is not seen as part of the new feature development.

8. WHAT CAUSES THE AVALANCHE?
• There is no advocate for the test automation work within a
team (e.g. the lead developer).
• A viewpoint that the QA personnel will take care of the quality
(and therefore all leaked bugs are seen as the QA personnel’s
fault, not as the team’s fault).
• Test automation framework that would suite the team’s needs
is missing.
• No-one is following the DoD.

9. WHAT CAN BE DONE WHEN THE TEST
AUTOMATION COVERAGE IS POOR?

10. HOW TO CHANGE THE ORGANIZATION
CULTURE TO BE ”PRO TEST AUTOMATION”
• Velocity allocation within each team for team internal QA
development work -> Effort estimates for the new features should
include test automation work.
• A test automation advocate for each team.
• TDD/BDD
• Test automation creation based on the leaked bugs.
• Top-down approach when starting to create the missing test
automation coverage (unit test level is a ”lost case” when there hasn’t
been any test automation done).
• Production monitoring data utilization (e.g. estimates how data

11. HOW TO CHANGE THE ORGANIZATION
CULTURE TO BE ”PRO TEST AUTOMATION”
• Test automation framework taken into use that addresses the needs
of the team.
• Automatic tests as part of the CI system and the radiator.
• Team’s commitment to the test automation work:
• Responsibility agreements (developers focusing on the unit/integration test
automation and QA personnel to the UI/system test automation).
• Test follow-up responsibilities within the team (test status from the CI and
updating the tests when needed).
• DoD dicipline (minimum set that the team will commit to).
• Test automation work done as part of the feature branch work (or
behind the same feature flag)

12. CASE: BASWARE – ACTIONS TAKEN
• Started to automate the different phases of the deployment and the
checking of the deployment successfulness.
• Setting up a system for generating the test environments automatically.
• Automatic deployment of builds into the automatically generated
environments.
• System level smoke and regression test automation. Also all APIs covered
with test automation.
• Test data tool development based on the production monitoring data
(possibility to generate average data amounts or amounts greater than the
known maximum amounts).
• Rotating test automation upgrade responsibility within each team (to avoid
test automation to be commented out instead of fixing the outdated tests).

13. THANK YOU!
• Thoughts that came to Your mind based on this speech?
• Questions?