This document discusses automated integration regression testing. It describes the different layers of testing including using Gherkin and the behave library to write test cases in a readable format, mapping test steps to Python functions using the Germanium library, and using Selenium in the backend to interact with browsers. Jenkins is used to run the test suites in Docker containers on a nightly basis against deployed applications. Typical tests involve a series of user actions and verifications like logging in, completing tasks, and logging out. Challenges include test maintenance, failures in the Jenkins environment that are hard to reproduce locally, and dependencies on constantly changing environments and application states.

1. Automated
Integration
Regression Testing
Michal Oravec

2. 2
 Testing Layers
 Test Projects
 Jenkins/Docker Runs
 Typical Test
 Challenges
Content

3. 3
Testing Layers
 Gherkin meta-language (behave Python library)
 Python code using (Germanium API Python library)
 Germanium itself handling browsers (includes Selenium
Python library and web-drivers)

4. 4
Testing Layers
 Gherkin meta-language (behave Python library)
‒ Basically almost English free text test case executable by human.
‒ Simple text files are used called features. The feature can contain one
or more scenarios (test cases) and each scenario may be executed one
or more times if there are more test data sets available.
‒ The scenarios are self-contained integration “tests”.
‒ behave: http://pythonhosted.org/behave/
Single test step (sentence)
Single Python
function gets
called

5. 5
Testing Layers
 Gherkin meta-language (behave Python library)
‒ We tend to have one scenario per one feature (best practice).
‒ We store the feature files in repository along the Python testing code.
‒ The sentences (test steps) are read by a Python library called “behave”
that links them with their implementation and executes them one by
one.

6. 6
Testing Layers
 Gherkin meta-language (behave Python library)
‒ There is very little structure – feature name and scenario name +
keywords Given / When / Then.
‒ “When” marks action and “Then” marks assertion (not necessary – just
best practice)

7. 7
Testing Layers
 Gherkin meta-language (behave Python library)
‒ Note convention for providing data used for reusing steps (=
sentences) in more tests.
‒ Same convention enables the same test to run with multiple data sets
– note part “Examples”. Each line represents a test run.

8. 8
Testing Layers
 Python code using (Germanium API Python library)
‒ Each sentence (test step) is mapped to one Python function.
‒ These may use other functions, mappings of elements (selectors of
buttons, dialogs, parts of text…) and generally anything doable in
Python and/or Germanium API.
‒ This layer is simply code in Python.
‒ We use and depend fully on the API
https://germaniumhq.com/documentation/

9. 9
Testing Layers
 Python code using (Germanium API Python library)
‒ Example:

10. 10
Testing Layers
 Germanium itself handling browsers
(includes/integrates Selenium Python library and
web-drivers)
‒ Basically that’s it because we do not directly use Selenium.
‒ We write / customize stuff only for Gherkin/behave and
Germanium/Python, this layer just does the job in background if
all goes well.

11. 11
Test Projects

12. 12
 Example test projects
for our apps called
Workplace and
Administrator
Test Projects
12

13. 13
Jenkins/Docker Runs

14. 14
Jenkins/Docker Runs
‒ Jenkins part of running tests is more technical and/or complicated.
‒ In our case there are two Jenkins jobs that run the whole tests suite
for the two tested apps against whatever is currently deployed on test
machine.
‒ These jobs are triggered at certain time generally every night
regardless of other jobs/builds/deployments.
‒ Typically this time is when we expect or suppose that the build and
deployment to test machine was already done.

15. 15
Jenkins/Docker Runs
‒ The how/where of the test run is complicated/technical. Somehow a
virtual environment (Docker) is created with the specified browser, all
the tests are fetched from repository and executed against .90 test
server running the recently built and deployed web-application.
‒ There is very little configuration in the Jenkins GUI – most is hidden in
the linked Jenkinsfile stored in the repository

16. 16
Jenkins/Docker Runs
‒ More necessary details how to create the environment from which the
tests will be run are hidden inside the “Dockerfile” (in repository):

17. 17
Jenkins/Docker Runs
‒ The output (log) is available in simple text form (very messy, not
structured) in the Jenkins job output to potentially analyze by a
person.
‒ One way to analyze failures is to rerun failed tests locally (from Idea)
and/or manually (by hand).

18. 18
Typical Test

19. 19
Typical Test
‒ The test cases are generally rather short and simple.
‒ Generally it is a series of actions (click link/button) and assertions
(expect certain text is on the page / in a dialog or expect certain page
to load or dialog to show up)
‒ Example for our Workplace application: Login -> check status and
version if it is even reasonable to continue -> run Business Case ->
check if Human Task opened correctly and if it has all the expected
content -> continue to Debug Human Task and check content ->
finish Business Case -> Logout.
‒ Challenge: It is difficult, error-prone and time consuming to check that
certain actions cause a new page to load with a random content when
it takes a random relatively long time.

20. 20
Typical Test
‒ Having short and simple and static TCs seems to be best practice -
one would argue that this is good / sufficient approach.
‒ Best practice: One should expect fixed / static texts and values and
never compute expected values (never replicate features of the tested
app).
‒ Automated integration regression testing should be like a sanity check
that really basic feature still work and that stuff generally opens and
loads as expected every day after nightly deployment.
‒ You don’t do it to find bugs – more like to:
 avoid testing the same basic stuff every day
 have some confidence in the morning that generally stuff works
 free real people to do more investigative testing

21. 21
Challenges

22. 22
Challenges
1. Maintenance.
2. Random technical failures in Jenkins/Docker run not replicable
locally.
3. Jenkins run output is not structured – it is very difficult to analyze
and offers almost no help in determining which tests failed and why.
4. Running the same tests on other browsers (Firefox, recent IE)
5. Marking tests that fail because known real failures - bugs.
6. Found bugs are ignored long-term – ”fix” the tests?
7. Tests are dependent on constant state of “public” and constantly
changing environment (application settings, content/resource
storage)
8. Tests too slow. Faster execution causes random failures.
9. Difficult/complicated to configure Jenkins job/run to execute only a
subset of all tests in a project.

23. 23
Conclusion
‒ Integration level of automated regression testing in web-frontend
area is useful but benefits depend on many factors
‒ It is always an open and valid question how to automate, how much
to automate, who should automate and also if to automate at all
Thank you for your attention.