The document discusses continuous testing, detailing its role in providing immediate feedback on business risks within the software delivery pipeline. It covers best practices, tools like Docker and AWS, and emphasizes version control and test plan ownership by the QA team. Key tips include running real browsers in headless mode and leveraging AWS for efficient test environments.

1. Continuous testing:
What exactly is it and why use it?
Oleksandr Metelytsia
Software Testing
IT Weekend Lviv 2016
1

2. About me
Test Automation Engineer in SoftServe
6+ years experience in software testing on different projects
2

3. Agenda
● What is Continuous Testing?
● Build, Test, Deploy!
● Divide and Conquer allthetests
● Docker and AWS for the win
● Tips and Tricks
● Q&A
3

4. What’s the problem?
4

5. Continuous Testing
5

6. Process of executing automated tests as part
of the software delivery pipeline to obtain
immediate feedback on the business risks
associated with a software release candidate.
Continuous Testing
6

7. Be the change
7

8. When Continuous Testing Done
8

9. Sample project from my experience
On-premises server version of a product
shipped as virtual image (OVA or AMI)
● Server with CLI
● Admin web panel
● API
● Clients: web, mac, win, linux, ios, android
9

10. Build Test Deploy
10

11. Simplified Pipeline
11

12. Our example pipeline
● Merge PR into component repo
● Trigger component build
● Run component unit tests and build .deb file
● Trigger system build
● Build product and run “test_build” tests
● Trigger all CI plans (acceptance, CLI, perf, etc)
● Press “Deploy” button
12

13. Version control
● All tests code under version control (Git, svn, etc)
○ No excuse not to use
● Distributed version control (GitHub, Bitbucket)
● Same PRs flow
○ Don’t hesitate to add devs as reviewers
● Store tests with production code (Unit, Integration)
○ Easy to get older version of the software with related tests
● Separate repo for Acceptance test, Perf, etc
13

14. Testing plans
● Basic regression (selenium, nose, python)
● CLI (bash, selenium)
● AD/LDAP (another flow of users setup)
● Integrations (special pre conditions and selenium)
● Web client (nodejs, karma)
● Perf (locust, new relic)
● Export -> Import
● Upgrade plans (RC, dev -1, with reboot, etc)
● Component integration tests
● ...
14

15. Basic rules
● Test plans owned by QA team
● Fight tests flakiness
● Plans should be green
● Test team checks all “reds”
● Jira tickets linked to every valid test fail
● Dev team checks labeled CI tickets
● Skip test if needed (@skip(“JIRA-ID”))
● Execution time for plan < 1 hour (*)
15

16. Practical tip: parallel jobs with nose
16

17. Tip: parallel jobs
17

18. Practical tip: headless mode
18

19. Tip: headless mode
19

20. Do not use PhantomJS (*)
● not a real browser
● additional setup
● limitations (e.g. incorrect alerts handing)
● “own” bugs (e.g. drag&drop)
● not stable on “big” runs (*IMHO)
● Xvfb + FF or Chrome are faster
* all above valid for version 1.x
Lesson learned: Run Real Browsers
Headless.
20

21. Tip: Automatic screenshot on a fail is
awesome
21

22. Tip: two browsers on one screen
1. Pass parameter num_of_browser_instances
2. Calculate driver screen resolution:
screen_width() / num_of_browser_instances
1. Launch browser at desired position
driver.set_window_position(width_coordinate, 0)
22

23. ●Long running test environment
vs
●Launch new fresh env on each CI run
Test environment
23

24. AWS
EC2 AMIs For The Win:
● production like environment
● fast deployment
● “super” speed between S3 buckets
● secure isolated environment (VPC)
● pay as you go
24

25. ● AWS CLI for your CI
● Route53 for DNS
● AWS Lambda - serverless computing
● Additional servers for infrastructure stuff like
AD, mail or proxy server
● Different instance types
● CI agents in EC2
● and many more...
and more AWS
25

26. Typical test development env problems
● difficult to set up and maintain
● hard to track dependencies
- python versions
- browser versions
- selenium and other packages versions
- other tools and software
● managing several environments
● OS differences between CI and local env
● inconsistency of executions
26

27. Docker
Docker containers wrap a piece of software in a
complete filesystem that contains everything
needed to run: code, runtime, system tools,
system libraries – anything that can be installed
on a server. This guarantees that the software
will always run the same, regardless of its
environment.
27

28. Docker diagram
28

29. Dockerfile
29

30. Docker
Build container:
$ docker build -t firefox:42 .
Push to registry:
$ docker push your_registry/firefox:42
Pull from registry:
$ docker pull your_registry/firefox:42
Run container:
$ docker run -t -i -v “/app:/app” firefox:42
30

31. ● Docker is not a virtual machine solution
● thousands images in docker hub
● own private docker registry
● run the same container on linux, mac, win
● lightweight to run
● lightweight to store
● lightweight to change
Docker
31

32. Docker in testing CI
Simplified pattern:
32

33. Practical tip: CI your images
1. Put Dockerfiles under version control
2. Create CI plans to build and push images
3. Trigger plans on repo commit
4. Profit
33

34. Testing is not a destination
but a journey
34

35. Questions?
Oleksandr Metelytsia
ometel@softserveinc.com
metelitsa.alexandr@gmail.com
35