The document discusses SpilGames' use of Docker containers and Puppet for infrastructure configuration management and testing. Key points: 1) SpilGames uses Docker containers to run functional tests for Puppet modules in parallel, providing isolation and speed compared to virtual machines. A baseline Docker image is tested and snapshots are used to spawn containers for testing other modules. 2) Puppet is used to configure all of SpilGames' servers through roles and classes. Code is verified using Gerrit and functional tests are run inside Docker containers using an orchestration script. 3) SpilGames is evaluating using Docker and the open source PaaS Stackato to further migrate production systems to their private OpenStack cloud for an

2. Me and lxc-containers
2011 Hyves (now part of TMG)
chat service - python service, single threaded (GIL)
Solution:
lxc-containers (since 2010!!!): 100+ instances
AUFS: advanced multi layered unification
filesystem
Gentoo: patched kernel

3. 2013 SpilGames
a global network of online gaming platforms
(180 million active users each month) and
growing
Technology used:
Production: PHP, Erlang
Internally: Python
Platform: Scientific Linux
Private cloud: Openstack

4. Openstack at SpilGames
Swift in production since Diablo 2011
Swift - highly available, distributed, eventually
consistent object/blob store
Compute in production since Folsom end of 2012
Compute - cloud computing fabric controller

5. Goal: IaaS, migrate most of production
systems to private cloud
Ultimate Goal: PaaS, Autoscaling

6. SpilGames infrastructure today
Two locations: Amsterdam, Toronto
Two openstack availability zones
Both sites serve live traffic

7. Puppet Intro
$puppet apply test.pp
Manifest file test.pp:
file {'testfile':
path => '/tmp/testfile',
ensure => present,
mode => 0640,
content => "I'm a test file.",
}
service { 'ntp':
ensure => running,
enable => true,
}

8. Puppet in SpilGames
Every server has puppet role registered and assigned in
CMDB (Configuration Management Databse):
hadoop_datanode
gerrit_standalone
During puppet run, role gets resolved and mapped to the
puppet class with the same name:
class roles::hadoop_datanode
class roles::gerrit_standalone

9. Role classes are used as containers for the rest of puppet
classes which provide actual configuration:
class roles::gerrit_standalone inherits roles::spil_base {
class{ '::nginx':
port => 80,
}
class{ '::gerrit':
mysql_host
mysql_user
mysql_password
mysql_db
}
=> '127.0.0.1',
=> 'gerrit2',
=> 'gerrit2',
=> 'reviewdb',
class { '::mysql::server':}
mysql::db { 'reviewdb':}
Class['::nginx'] → Mysql::Db['reviewdb'] → Class['::Gerrit']
}

10. Role classes include "roles::spil_base" class
class roles::gerrit_standalone inherits
roles::spil_base {
...
}
This "spil_base" class provide minimal necessary
baseline configuration:
ntp, accounts, sysctl, security etc

11. Code Verification
We use Gerrit to run: syntax check (puppetlint), parser validation (puppet parser)
Syntax checks, code verification is very fast
(< 10 seconds) and initiated by Gerrit commit
hooks

12. Functional testing
Launch host -> Confgure host by applying specific puppet
module -> Verify result
Functional testing to be useful require:
1) Every single test has to be run inside isolated virtual
environments
2) Each module has to be tested on freshly provisioned host
3) Time to run tests should be minimal: < 5-7 min
4) Everything should be automated
Ideally we'd like to test modified puppet modules during Gerrit
verification.

13. Available tools
rspec-system - framework for creating system tests using the rspec
testing library
launch tests nodes
● copy test content and execute commands on such nodes to be tested
with standard rspec assertions within the standard rspec test format
●
Supports: Vagrant + VirtualBox/VSphere, Openstack
Problems:
●
both VirtualBox/VSphere and Openstack are slow and require
resource intensive virtaulziation
●
functionality is limited

14. Why Docker
lxc-containers – light, fast, resource efficient, possible to
bind external directories, support snapshots, good python
library
Puppet roles share same baseline configuration!
Main benefits:
●
●
Having SNAPSHOT of baseline system we can use it later
to spawn containers and use them to test the rest of the
modules.
Because of low resource overhead containers with tests
can be run in parallel

15. Prepare
Initial image
(ssh, puppet)
Test Baseline image
(roles::spil_base)
Snapshot
Baseline image
Test Module
(mysql_master)
Test Module
(nginx)
Analyze results
Test Module
(hadoop)

16. Jenkins build results

17. https://github.com/lruslan/puppet_test
Orchestration script has the following features:
●
●
●
●
●
●
‘full’ mode: find puppet modules, build base docker image
using base module and run tests for the rest of the modules
‘quick’ mode: reuse previously created base image and run
tests for the modules
parallel mode: it’s possible to specify the number of workers
(so multiple tests run in a parallel)
Jenkins integration: detect which Puppet modules have
been changed since last jenkins build
Results publishing: generates html report with results and
ability to see details (stdout/stderr) of every test
ability to set timer and stop containers if the test takes
longer than expected

18. Future plans
Our orchestration script is still proof of concept but already
serves us very well.
Next step would be bring similar functionality to one of
existing rspec tools used by community.
Ooops … rspec-system retired 10 days ago :(
Now official tool for the rspec tool is:
Beaker (beaker-rspec) https://github.com/puppetlabs/beaker

19. Docker + Stackato = PaaS ?
●
Stackato started in 2012
●
Stackato 3.0 switched to Docker containers
●
Uses Cloud Foundry v2 - open source
PaaS, developed by Vmware
Evaluating … to be continued

20. Spil Games are hiring