1. PyCon Ireland 2016

2. Containerised Testing at Demonware

3. “Quality is free, but
only to those willing
to pay heavily for
it.”
T. DeMarco and T. Lister

4. Who are Demonware?
Demonware provide online services and infrastructure for some of the
world’s most popular video game franchises.

5. Bio
Thomas is a Build Engineer at Demonware and works closely with the
development teams to optimise their CI/CD pipelines. His background is
in QA and other roles include Configuration Management and Test Tools
automation.
James is a Django developer on a Demonware internal product. Before
Demonware he worked at a handful of startups, largely in web
development.

6. Pets vs Cattle
CaaS
4 Stages of
Evolution
TaaS
Continuous
Delivery
Agenda
The future

7. Continuous Delivery
Define Develop Build Integrate Test Release Deploy
Continuous Delivery
Continuous Integration

8. Why Continuous Delivery ?
● Deliver new services, features and updates rapidly
● Reduce manual intervention
● Shorter feedback loop
● Reduce cost of deployments
● Reduce risk

9. Quality is no longer important ...
It’s
C R I T I C A L

10. Continuous Delivery without Quality
Best case scenario Worst case scenario

11. Time to deliver
● The Continuous Delivery pipeline is only as fast as the slowest stage
● The Build and Deploy stages took minutes
● The Test stage took 6+ hours for a full set of testsuites to complete
● The Test stage was a blockage in our CD pipeline

12. Development teams began optimising
their tests.
• Split across containers
• Measure execution time to
manage slices of tests
• Quality metrics like static analysis
and coverage reports
The Build Engineering team began
optimising the test environments.
• Test infrastructure as code
• Immutable and platform agnostic
test environments
• Eliminate “Dependency Hell”
Optimising the Test stage

13. Tests are run in fresh containers;
new test run, new container.
Container images are built from
code.
Developers control the test
environment dependencies.
Test environments are easily
reproducible anywhere.
Test environments are manually
configured. Setup and tear down
happens with each test run.
Each test environment is a
snowflake.
Everyone has root access.
Environments are never updated
for fear of breaking tests.
Pets Cattle

14. Baremetal Vagrant VMs Docker
Containers
./setup
./start tests
./teardown
./setup
./start
./teardown
./setup
./start
./teardown
./setup
./start
./teardown
./setup
./start
./teardown
Moving from Pets towards Cattle
2012 2013 2014 - Present

15. 4 Stages of Evolution
1 : Fat containers
2 : Containers wrapped in Bash
3 : Containers defined in yaml
4 : Containers as a service

16. Fat containers
What is a “Fat” container and how was it created?
In this context it is a container with multiple services installed.
We tarred up a CentOS vagrant vm and imported into a docker image.
Pros :
1. It helped get the ball rolling
2. Everything needed to run tests was included
3. Image caching reduced the time required to build new images
Cons :
1. Large base container image. Approximately 3gig
2. The container ran multiple supporting services such as MySQL, RabbitMQ, Apache
3. It took almost as long as a VM for all services to become available

17. Containers wrapped in Bash
As we began splitting services out into their own containers we needed a way to co-ordinate them.
In particular we needed to link containers and wait for services to become available. We used a
high level bash wrapper to start, link and wait for containers to become ready.
Pros :
1. The script was initially quite simple
2. It filled a gap in tooling around multi container deployments (June 2014)
Cons :
1. The script was duplicated across multiple projects and became unwieldy
2. A lot of logic required to check a service health/status
3. Fragile

18. Containers defined in yaml
We began using docker-compose in August 2015. Compose allowed developers to define
complex container orchestration in YAML.
Example :
testsuite:
build: unittest
command: python testrunner.py
ports:
- "80:80"
volumes:
- unittests:/unittests
links:
- percona
percona:
image: percona

19. Containers defined in yaml
Pros :
1. Compose is easy to use
2. Portable
3. Repeatable
4. Container configuration defined in code
5. Container orchestration requirements defined in code
Cons :
1. Runs on a single host. Not fully cluster aware. This will change very soon.

20. Test Containers
Limitation of first 3 stages
● Single host, fixed resources
● Fixed number of Test Containers per host
● Fixed number of Tests per Container
● Local debugging of failures is difficult
● Re-running failed/flaky tests is expensive

21. Containers as a service
We are currently in this stage. With the release of Docker 1.12 we started to look at how and
where tests are being run.
2 key features in 1.12 :
● Swarm Mode
○ Natively manage a cluster of Docker Engines called a swarm. Use the Docker CLI to
create a swarm, deploy application services to a swarm, and manage swarm behavior.
● Services
○ Docker services enables the use to start a replicated, distributed, load balanced
service on a swarm of Engines.

22. Containers as a service
Docker Swarm mode and Services enable us to provide a much more flexible yet robust and
scalable test environment.

23. Tests as a service
Like any Swarm service, the number of containers is scalable based on the resources available in
the Test Cluster.
We need a dynamic way of splitting tests at execution time to maximise the usefulness of this.
Populate a Redis service with test case names.
Each container grabs a chunk of tests from the Redis service. Tests are run and results stored in
a shared mount point across all nodes.

24. What does the test cluster look like?
Shared volume mounted using GlusterFS. Used for results, logs etc
Manager Manager Manager WorkerWorkerWorker Worker Worker
Team A Unit
Tests
Team B Unit
Tests
Docker Registry
(Global)
Redis (Global)

25. Tests as a service
How do we create a test service?
● docker service create --name unittests --replicas 1 <image_name>
How do we scale up the test tasks as resources become available ?
● docker service scale unittests=50
How do we scale down when tests are finished ?
● docker service scale unittests=0

26. What does this look like ?
Check
Cluster
Resources
Create
overlay
network
Start Redis
Population
service
Create Test
Service
Cleanup
Services
Execute
Tests
Stop Redis
Population
service
Create Redis
Global
service
Create
shared work
directory

27. The code
Create overlay network :
docker network create -d overlay --subnet 10.0.9.0/24 network_unittests_129
Create global Redis service :
docker service create --name 129_redis --network network_unittests_129 --mount
type=bind,src=/home/test_cluster/unittest/129,dst=/data -p 6379:6379 redis
Populate Redis service :
docker service create --name 129_redis_populate --network network_unittests_129 --mount
type=bind,src=/home/test_cluster/unittest/129,dst=/tmp --replicas=1
127.0.0.1:5000/populate_redis:latest -c 'cat /tmp/testlist |redis-cli -h 129_redis -p 6379'
Start Test Service :
docker service create --name 129_tests --env REDIS_SERVICE=129_redis --network
network_unittests_129 --mount type=bind,src=/home/test_cluster/results/129,dst=/results
--replicas=12 127.0.0.1:5000/unittest:129

28. What other benefits does Docker
Swarm provide ?
● Easy to setup
○ docker swarm init
○ docker swarm join --token <swarm_token> <ip of manager>
● Secure by default. Uses TLS
● Service discovery
● Load Balancing
● Scaling
● Desired State reconciliation
● Multi-host networking
● Rolling updates

29. Test Containers
What Docker Swarm solves
● Single host, fixed resources
● Fixed number of Test Containers per host
● Fixed number of Tests per Container
● Local debugging of failures is difficult
● Re-running failed/flaky tests is expensive
SOLVED
SOLVED
SOLVED
SOLVED
SOLVED

30. Summary
● We increase our resource use, but not by 'throwing resources at it'
- instead we optimise for parallel execution
● With a relatively small time investment we were able to innovate
● The tools used to create and manage the test cluster are open source
■ Docker : https://www.docker.com/
■ Ansible : https://www.ansible.com/
■ GlusterFS : https://www.gluster.org/
■ Swarm Visualizer :
https://github.com/ManoMarks/docker-swarm-visualizer
● Our containerised tests can scale seamlessly across 3 different
platforms

31. We are hiring:
Full details at demonware.net
Or email : jobs@demonware.net
Internships available.
Drop by our stand for more details.

32. Contact details :
thshaw@demonware.net / @tomwillfixit
jheslin@demonware.net / @PROGRAM_IX
Follow us @demonware for engineering
related content.
Meetup : meetup.com/Docker-Dublin/

33. Upcoming Event