Software Testing IT Weekend Lviv 2016 conference. How to achieve "real-time" testing and integrate it into software development life cycle.

1. Continuous testing:
What exactly is it and why use it?
Oleksandr Metelytsia
Software Testing
IT Weekend Lviv 2016
1

2. About me
Test Automation Engineer in SoftServe
6+ years experience in software testing on different projects
2

3. Agenda
● What is Continuous Testing?
● Build, Test, Deploy!
● Divide and Conquer allthetests
● Docker and AWS for the win
● Tips and Tricks
● Q&A
3

4. What’s the problem?
4

5. Continuous Testing
5

6. Process of executing automated tests as part
of the software delivery pipeline to obtain
immediate feedback on the business risks
associated with a software release candidate.
Continuous Testing
6

7. Be the change
7

8. When Continuous Testing Done
8

9. Sample project from my experience
On-premises server version of a product
shipped as virtual image (OVA or AMI)
● Server with CLI
● Admin web panel
● API
● Clients: web, mac, win, linux, ios, android
9

10. Build Test Deploy
10

11. Simplified Pipeline
11

12. Our example pipeline
● Merge PR into component repo
● Trigger component build
● Run component unit tests and build .deb file
● Trigger system build
● Build product and run “test_build” tests
● Trigger all CI plans (acceptance, CLI, perf, etc)
● Press “Deploy” button
12

13. Version control
● All tests code under version control (Git, svn, etc)
○ No excuse not to use
● Distributed version control (GitHub, Bitbucket)
● Same PRs flow
○ Don’t hesitate to add devs as reviewers
● Store tests with production code (Unit, Integration)
○ Easy to get older version of the software with related tests
● Separate repo for Acceptance test, Perf, etc
13

14. Testing plans
● Basic regression (selenium, nose, python)
● CLI (bash, selenium)
● AD/LDAP (another flow of users setup)
● Integrations (special pre conditions and selenium)
● Web client (nodejs, karma)
● Perf (locust, new relic)
● Export -> Import
● Upgrade plans (RC, dev -1, with reboot, etc)
● Component integration tests
● ...
14

15. Basic rules
● Test plans owned by QA team
● Fight tests flakiness
● Plans should be green
● Test team checks all “reds”
● Jira tickets linked to every valid test fail
● Dev team checks labeled CI tickets
● Skip test if needed (@skip(“JIRA-ID”))
● Execution time for plan < 1 hour (*)
15

16. Practical tip: parallel jobs with nose
16

17. Tip: parallel jobs
17

18. Practical tip: headless mode
18

19. Tip: headless mode
19

20. Do not use PhantomJS (*)
● not a real browser
● additional setup
● limitations (e.g. incorrect alerts handing)
● “own” bugs (e.g. drag&drop)
● not stable on “big” runs (*IMHO)
● Xvfb + FF or Chrome are faster
* all above valid for version 1.x
Lesson learned: Run Real Browsers
Headless.
20

21. Tip: Automatic screenshot on a fail is
awesome
21

22. Tip: two browsers on one screen
1. Pass parameter num_of_browser_instances
2. Calculate driver screen resolution:
screen_width() / num_of_browser_instances
1. Launch browser at desired position
driver.set_window_position(width_coordinate, 0)
22

23. ●Long running test environment
vs
●Launch new fresh env on each CI run
Test environment
23

24. AWS
EC2 AMIs For The Win:
● production like environment
● fast deployment
● “super” speed between S3 buckets
● secure isolated environment (VPC)
● pay as you go
24

25. ● AWS CLI for your CI
● Route53 for DNS
● AWS Lambda - serverless computing
● Additional servers for infrastructure stuff like
AD, mail or proxy server
● Different instance types
● CI agents in EC2
● and many more...
and more AWS
25

26. Typical test development env problems
● difficult to set up and maintain
● hard to track dependencies
- python versions
- browser versions
- selenium and other packages versions
- other tools and software
● managing several environments
● OS differences between CI and local env
● inconsistency of executions
26

27. Docker
Docker containers wrap a piece of software in a
complete filesystem that contains everything
needed to run: code, runtime, system tools,
system libraries – anything that can be installed
on a server. This guarantees that the software
will always run the same, regardless of its
environment.
27

28. Docker diagram
28

29. Dockerfile
29

30. Docker
Build container:
$ docker build -t firefox:42 .
Push to registry:
$ docker push your_registry/firefox:42
Pull from registry:
$ docker pull your_registry/firefox:42
Run container:
$ docker run -t -i -v “/app:/app” firefox:42
30

31. ● Docker is not a virtual machine solution
● thousands images in docker hub
● own private docker registry
● run the same container on linux, mac, win
● lightweight to run
● lightweight to store
● lightweight to change
Docker
31

32. Docker in testing CI
Simplified pattern:
32

33. Practical tip: CI your images
1. Put Dockerfiles under version control
2. Create CI plans to build and push images
3. Trigger plans on repo commit
4. Profit
33

34. Testing is not a destination
but a journey
34

35. Questions?
Oleksandr Metelytsia
ometel@softserveinc.com
metelitsa.alexandr@gmail.com
35