Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Containerised Testing at Demonware : PyCon Ireland 2016

A journey through the history of containerised testing and how we use tools such as Docker Compose and Docker Swarm to continuously deliver great gaming experiences for our customers.

  • Login to see the comments

  • Be the first to like this

Containerised Testing at Demonware : PyCon Ireland 2016

  1. 1. PyCon Ireland 2016
  2. 2. Containerised Testing at Demonware
  3. 3. “Quality is free, but only to those willing to pay heavily for it.” T. DeMarco and T. Lister
  4. 4. Who are Demonware? Demonware provide online services and infrastructure for some of the world’s most popular video game franchises.
  5. 5. Bio Thomas is a Build Engineer at Demonware and works closely with the development teams to optimise their CI/CD pipelines. His background is in QA and other roles include Configuration Management and Test Tools automation. James is a Django developer on a Demonware internal product. Before Demonware he worked at a handful of startups, largely in web development.
  6. 6. Pets vs Cattle CaaS 4 Stages of Evolution TaaS Continuous Delivery Agenda The future
  7. 7. Continuous Delivery Define Develop Build Integrate Test Release Deploy Continuous Delivery Continuous Integration
  8. 8. Why Continuous Delivery ? ● Deliver new services, features and updates rapidly ● Reduce manual intervention ● Shorter feedback loop ● Reduce cost of deployments ● Reduce risk
  9. 9. Quality is no longer important ... It’s C R I T I C A L
  10. 10. Continuous Delivery without Quality Best case scenario Worst case scenario
  11. 11. Time to deliver ● The Continuous Delivery pipeline is only as fast as the slowest stage ● The Build and Deploy stages took minutes ● The Test stage took 6+ hours for a full set of testsuites to complete ● The Test stage was a blockage in our CD pipeline
  12. 12. Development teams began optimising their tests. • Split across containers • Measure execution time to manage slices of tests • Quality metrics like static analysis and coverage reports The Build Engineering team began optimising the test environments. • Test infrastructure as code • Immutable and platform agnostic test environments • Eliminate “Dependency Hell” Optimising the Test stage
  13. 13. Tests are run in fresh containers; new test run, new container. Container images are built from code. Developers control the test environment dependencies. Test environments are easily reproducible anywhere. Test environments are manually configured. Setup and tear down happens with each test run. Each test environment is a snowflake. Everyone has root access. Environments are never updated for fear of breaking tests. Pets Cattle
  14. 14. Baremetal Vagrant VMs Docker Containers ./setup ./start tests ./teardown ./setup ./start ./teardown ./setup ./start ./teardown ./setup ./start ./teardown ./setup ./start ./teardown Moving from Pets towards Cattle 2012 2013 2014 - Present
  15. 15. 4 Stages of Evolution 1 : Fat containers 2 : Containers wrapped in Bash 3 : Containers defined in yaml 4 : Containers as a service
  16. 16. Fat containers What is a “Fat” container and how was it created? In this context it is a container with multiple services installed. We tarred up a CentOS vagrant vm and imported into a docker image. Pros : 1. It helped get the ball rolling 2. Everything needed to run tests was included 3. Image caching reduced the time required to build new images Cons : 1. Large base container image. Approximately 3gig 2. The container ran multiple supporting services such as MySQL, RabbitMQ, Apache 3. It took almost as long as a VM for all services to become available
  17. 17. Containers wrapped in Bash As we began splitting services out into their own containers we needed a way to co-ordinate them. In particular we needed to link containers and wait for services to become available. We used a high level bash wrapper to start, link and wait for containers to become ready. Pros : 1. The script was initially quite simple 2. It filled a gap in tooling around multi container deployments (June 2014) Cons : 1. The script was duplicated across multiple projects and became unwieldy 2. A lot of logic required to check a service health/status 3. Fragile
  18. 18. Containers defined in yaml We began using docker-compose in August 2015. Compose allowed developers to define complex container orchestration in YAML. Example : testsuite: build: unittest command: python ports: - "80:80" volumes: - unittests:/unittests links: - percona percona: image: percona
  19. 19. Containers defined in yaml Pros : 1. Compose is easy to use 2. Portable 3. Repeatable 4. Container configuration defined in code 5. Container orchestration requirements defined in code Cons : 1. Runs on a single host. Not fully cluster aware. This will change very soon.
  20. 20. Test Containers Limitation of first 3 stages ● Single host, fixed resources ● Fixed number of Test Containers per host ● Fixed number of Tests per Container ● Local debugging of failures is difficult ● Re-running failed/flaky tests is expensive
  21. 21. Containers as a service We are currently in this stage. With the release of Docker 1.12 we started to look at how and where tests are being run. 2 key features in 1.12 : ● Swarm Mode ○ Natively manage a cluster of Docker Engines called a swarm. Use the Docker CLI to create a swarm, deploy application services to a swarm, and manage swarm behavior. ● Services ○ Docker services enables the use to start a replicated, distributed, load balanced service on a swarm of Engines.
  22. 22. Containers as a service Docker Swarm mode and Services enable us to provide a much more flexible yet robust and scalable test environment.
  23. 23. Tests as a service Like any Swarm service, the number of containers is scalable based on the resources available in the Test Cluster. We need a dynamic way of splitting tests at execution time to maximise the usefulness of this. Populate a Redis service with test case names. Each container grabs a chunk of tests from the Redis service. Tests are run and results stored in a shared mount point across all nodes.
  24. 24. What does the test cluster look like? Shared volume mounted using GlusterFS. Used for results, logs etc Manager Manager Manager WorkerWorkerWorker Worker Worker Team A Unit Tests Team B Unit Tests Docker Registry (Global) Redis (Global)
  25. 25. Tests as a service How do we create a test service? ● docker service create --name unittests --replicas 1 <image_name> How do we scale up the test tasks as resources become available ? ● docker service scale unittests=50 How do we scale down when tests are finished ? ● docker service scale unittests=0
  26. 26. What does this look like ? Check Cluster Resources Create overlay network Start Redis Population service Create Test Service Cleanup Services Execute Tests Stop Redis Population service Create Redis Global service Create shared work directory
  27. 27. The code Create overlay network : docker network create -d overlay --subnet network_unittests_129 Create global Redis service : docker service create --name 129_redis --network network_unittests_129 --mount type=bind,src=/home/test_cluster/unittest/129,dst=/data -p 6379:6379 redis Populate Redis service : docker service create --name 129_redis_populate --network network_unittests_129 --mount type=bind,src=/home/test_cluster/unittest/129,dst=/tmp --replicas=1 -c 'cat /tmp/testlist |redis-cli -h 129_redis -p 6379' Start Test Service : docker service create --name 129_tests --env REDIS_SERVICE=129_redis --network network_unittests_129 --mount type=bind,src=/home/test_cluster/results/129,dst=/results --replicas=12
  28. 28. What other benefits does Docker Swarm provide ? ● Easy to setup ○ docker swarm init ○ docker swarm join --token <swarm_token> <ip of manager> ● Secure by default. Uses TLS ● Service discovery ● Load Balancing ● Scaling ● Desired State reconciliation ● Multi-host networking ● Rolling updates
  29. 29. Test Containers What Docker Swarm solves ● Single host, fixed resources ● Fixed number of Test Containers per host ● Fixed number of Tests per Container ● Local debugging of failures is difficult ● Re-running failed/flaky tests is expensive SOLVED SOLVED SOLVED SOLVED SOLVED
  30. 30. Summary ● We increase our resource use, but not by 'throwing resources at it' - instead we optimise for parallel execution ● With a relatively small time investment we were able to innovate ● The tools used to create and manage the test cluster are open source ■ Docker : ■ Ansible : ■ GlusterFS : ■ Swarm Visualizer : ● Our containerised tests can scale seamlessly across 3 different platforms
  31. 31. We are hiring: Full details at Or email : Internships available. Drop by our stand for more details.
  32. 32. Contact details : / @tomwillfixit / @PROGRAM_IX Follow us @demonware for engineering related content. Meetup :
  33. 33. Upcoming Event