code https://github.com/xebia/scalable-qa-with-docker Stop wasting time with manual configuration and maintenance Are you tired of maintaining all your Jenkins slaves and the inconsistency of the setup? We need to become aware that it’s inefficient to keep all the environments, browsers and dependencies up to date manually. During this session the attendees will be introduced to setting up scalable and solid test environments with practical examples using selenium, mesos and docker. Key takeaways How to build and scale the Selenium Grid(s) using Docker and Mesos? How to setup a scalable cluster to run specific test environments in no-time? How to leverage the mesos cluster to test with other frameworks than selenium?

1. Software Development Done Right
Scalable QA
with Docker
1
Maarten van den Ende – @mjvdende
Adé Mochtar – amochtar@xebia.com

2. Introduction
Maarten van den Ende
Test Automation Architect
@mjvdende
Adé Mochtar
Data Center Automator
amochtar@xebia.com
2

3. Slides
http://bit.ly/tw-scale
3

4. Code Examples
4
$ git clone https://github.com/xebia/scalable-qa-with-docker.git
$ cd scalable-qa-with-docker/
$ vagrant up
Download data.zip from http://bit.ly/tw-scale-data
and unzip in scalable-qa-with-docker/ dir
1
2
3

5. 5
Scaling tests is hard

6. 6
Test environment hard to
maintain

7. 7
Adding resources takes time

8. Selenium Grid to
the rescue
Use Grid to run all your tests
One interface for testing all the browsers
Fixed set of browsers nodes are at your
disposal, ready for more
8

9. 9
Selenium Grid Over Time

10. Selenium Grid Over Time
10
Not enough
nodes
configured
- For example:
growing need to
regression test
each branch
after each
commit
So you really need to take care of
your selenium grid...
Selenium Grid
hard to
maintain
- Browser updates
- Selenium
updates
Selenium Grid
stability
- Selenium nodes
go out of
memory
- Browser timeouts

11. 11

12. 12
How To Scale Properly

13. How to scale properly
Start test
- Scale up
- Start nodes when
needed
- Start different flavor
End test
- Scale down
- Remove nodes
Ready for More!
13
Server X
Source
Repository
Continuous
Integration
Datacenter Y
Testrunner
Chrome node
Testrunner
Chrome node
TestrunnerSystem
Under Test

14. Next 60 minutes
1. Setup environment
2. Selenium job
3. Setup selenium grid
4. Add node to grid
5. Scale the grid
14

15. 15
1. SET UP ENVIRONMENT

16. Set up
Start environment
16
Host OS
Virtualbox - Virtualisatie
Infrastructure
Docker
Guest OS
CoreOS
App 1 App 2
Guest OS
CoreOS
Docker
App 3 App 4
$ vagrant up
Bringing machine 'core-01' up with 'virtualbox'
provider...
Bringing machine 'core-02' up with 'virtualbox'
provider...
==> core-01: Importing base box
'coreos_766.3.0.box'...
==> core-01: Matching MAC address for NAT
networking...
==> core-01: Setting the name of the VM:
scalable-qa-with-docker_core-
01_1443034087957_90607
==> core-01: Clearing any previously set network
interfaces...
==> core-01: Preparing network interfaces based
on configuration...
1. SET UP

17. What are Vagrant & Virtualbox
Virtualbox
- Runs virtual machines on your machine
- Cross platform (OS X, Windows, Linux)
- Open source
Vagrant
- Utility for building development environments
- Utility for distributing these environments
- Vagrant cloud
- Open source
Vagrant + Virtualbox provide reproducible
environments, ideal for dev and test
17
1. SET UP

18. What is Docker
Platform for building, shipping and running
applications
Standardized container format
- Application packaging
- Isolation
- Distribution
Runs any app, everywhere
18
1. SET UP

19. What is Docker
Lightweight
- Share system kernel and
common files
Open
- Runs on Linux (and
Windows)
Secure
- Isolate applications
Share and collaborate
- Store, distribute and
manage Docker images in
docker registry
Scale
- Containers spin up in
seconds
Accelerate developer
- Copy production code and
run locally
Empower developer
- Free to use the best
language for the job
without causing conflict
issues
Eliminate environment
inconsistencies
- Shipping the application
with configs and
dependencies as a
container
19
1. SET UP

20. What is CoreOS
Bare Linux OS, designed to
run Linux Containers like
Docker
Designed for security,
consistency, and reliability
No package manager , have
to run everything in a
container
Clustering is first class citizen
Automatic updates
20
1. SET UP

21. CoreOS components
Docker - Container runtime
Etcd - Distributed K/V store, basis for
clustering
Fleet - Basic scheduler
Systemd - Service manager
Flannel - Networking between containers
21
1. SET UP

22. CoreOS Unit Files
22
[Unit]
Description=petclinic
After=docker.service
Requires=docker.service
[Service]
Restart=always
RestartSec=5
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker pull docker-registry:5000/petclinic
ExecStartPre=-/usr/bin/docker kill petclinic
ExecStartPre=-/usr/bin/docker rm petclinic
ExecStart=/usr/bin/docker run
-p 8282:8080
--name="petclinic"
docker-registry:5000/petclinic
ExecStop=/usr/bin/docker stop petclinic
1. SET UP

23. Up and Running
System under test
- Petclinic
Source Repository
- Gitbucket
Continuous Integration
- Jenkins
23
1. SET UP
core-01
Source
Repository
Continuous
Integration
System
Under Test
core-02

24. 24
2. SELENIUM JOB

25. Run test job on Jenkins
Create test
Commit & Push
Run test
25
Core 01
GitBucket Jenkins
Testrunner
2. SELENIUM JOB

26. Does it scale?
No, Why not?
Test fixed to specific selenium node
What if
- Different browser tests
- +100 tests?
- Selenium upgrade
26
2. SELENIUM JOB

27. 27
3. SETUP SELENIUM GRID

28. Core 01
Node
Setup Selenium Grid
Start Selenium Grid
- Add Hub
- Add node: chrome
- Add node: Firefox
28
3. SETUP SELENIUM GRID
GitBucket Jenkins
NodeHub

29. Does it Scale?
Yes we can test multiple browsers!
But…!?
What if?
- +100 tests?
- Selenium upgrades?
- Grid Stability?
29
3. SETUP SELENIUM GRID

30. 30
4. ADD NODE TO GRID

31. Add Core to Grid
Start Selenium Grid
Add Selenium node
on new core
Test with new node
31
4. ADD NODE TO GRID
Core 01
GitBucket Jenkins
TestrunnerTestrunner
Core 02
Testrunner

32. Does it scale?
Yes we can 100+ tests!!!
But what if
- Selenium upgrade?
- Grid Stability?
32
4. ADD NODE TO GRID

33. 33
5. SCALE THE GRID

34. Really scale up the grid!
Manually scaling grids does not scale
Add scheduling component: Mesos
34
5. SCALE THE GRID

35. Mesos to rule them all
Mesos groups separate machines into one
big resource pool
35
5. SCALE THE GRID

36. What is Mesos?
Mesosphere provides features such as
application scheduling, scaling, fault-
tolerance, and self-healing.
It also provides application service discovery,
port unification, and end-point elasticity.
36
5. SCALE THE GRID

37. Mesos Architecture
37
5. SCALE THE GRID

38. Scale: Manage all the resources
Mesos master
Mesos slave
Mesos plugin for Jenkins
- Registers framework to
start Jenkins slaves on
Mesos slaves
38
5. SCALE THE GRID
Core 01
GitBucket Jenkins
Mesos Master
Core 02
Mesos slave

39. Bonus: Other test tool
39
ADDED BONUS
SCALE ALL THE THINGS

40. All the things?
Mesos is a generic scheduler
Can run any job, not just selenium
Use your mesos cluster to run any test
- Integration testing
- Security testing
- Performance testing
- ...
40
ADDED BONUS: SCALEALL THE THINGS

41. Questions?
Scalable QA With Docker