2. Me and lxc-containers
2011 Hyves (now part of TMG)
chat service - python service, single threaded (GIL)
Solution:
lxc-containers (since 2010!!!): 100+ instances
AUFS: advanced multi layered unification
filesystem
Gentoo: patched kernel
3. 2013 SpilGames
a global network of online gaming platforms
(180 million active users each month) and
growing
Technology used:
Production: PHP, Erlang
Internally: Python
Platform: Scientific Linux
Private cloud: Openstack
4. Openstack at SpilGames
Swift in production since Diablo 2011
Swift - highly available, distributed, eventually
consistent object/blob store
Compute in production since Folsom end of 2012
Compute - cloud computing fabric controller
5. Goal: IaaS, migrate most of production
systems to private cloud
Ultimate Goal: PaaS, Autoscaling
8. Puppet in SpilGames
Every server has puppet role registered and assigned in
CMDB (Configuration Management Databse):
hadoop_datanode
gerrit_standalone
During puppet run, role gets resolved and mapped to the
puppet class with the same name:
class roles::hadoop_datanode
class roles::gerrit_standalone
9. Role classes are used as containers for the rest of puppet
classes which provide actual configuration:
class roles::gerrit_standalone inherits roles::spil_base {
class{ '::nginx':
port => 80,
}
class{ '::gerrit':
mysql_host
mysql_user
mysql_password
mysql_db
}
=> '127.0.0.1',
=> 'gerrit2',
=> 'gerrit2',
=> 'reviewdb',
class { '::mysql::server':}
mysql::db { 'reviewdb':}
Class['::nginx'] → Mysql::Db['reviewdb'] → Class['::Gerrit']
}
10. Role classes include "roles::spil_base" class
class roles::gerrit_standalone inherits
roles::spil_base {
...
}
This "spil_base" class provide minimal necessary
baseline configuration:
ntp, accounts, sysctl, security etc
11. Code Verification
We use Gerrit to run: syntax check (puppetlint), parser validation (puppet parser)
Syntax checks, code verification is very fast
(< 10 seconds) and initiated by Gerrit commit
hooks
12. Functional testing
Launch host -> Confgure host by applying specific puppet
module -> Verify result
Functional testing to be useful require:
1) Every single test has to be run inside isolated virtual
environments
2) Each module has to be tested on freshly provisioned host
3) Time to run tests should be minimal: < 5-7 min
4) Everything should be automated
Ideally we'd like to test modified puppet modules during Gerrit
verification.
13. Available tools
rspec-system - framework for creating system tests using the rspec
testing library
launch tests nodes
● copy test content and execute commands on such nodes to be tested
with standard rspec assertions within the standard rspec test format
●
Supports: Vagrant + VirtualBox/VSphere, Openstack
Problems:
●
both VirtualBox/VSphere and Openstack are slow and require
resource intensive virtaulziation
●
functionality is limited
14. Why Docker
lxc-containers – light, fast, resource efficient, possible to
bind external directories, support snapshots, good python
library
Puppet roles share same baseline configuration!
Main benefits:
●
●
Having SNAPSHOT of baseline system we can use it later
to spawn containers and use them to test the rest of the
modules.
Because of low resource overhead containers with tests
can be run in parallel
15. Prepare
Initial image
(ssh, puppet)
Test Baseline image
(roles::spil_base)
Snapshot
Baseline image
Test Module
(mysql_master)
Test Module
(nginx)
Analyze results
Test Module
(hadoop)
17. https://github.com/lruslan/puppet_test
Orchestration script has the following features:
●
●
●
●
●
●
‘full’ mode: find puppet modules, build base docker image
using base module and run tests for the rest of the modules
‘quick’ mode: reuse previously created base image and run
tests for the modules
parallel mode: it’s possible to specify the number of workers
(so multiple tests run in a parallel)
Jenkins integration: detect which Puppet modules have
been changed since last jenkins build
Results publishing: generates html report with results and
ability to see details (stdout/stderr) of every test
ability to set timer and stop containers if the test takes
longer than expected
18. Future plans
Our orchestration script is still proof of concept but already
serves us very well.
Next step would be bring similar functionality to one of
existing rspec tools used by community.
Ooops … rspec-system retired 10 days ago :(
Now official tool for the rspec tool is:
Beaker (beaker-rspec) https://github.com/puppetlabs/beaker
19. Docker + Stackato = PaaS ?
●
Stackato started in 2012
●
Stackato 3.0 switched to Docker containers
●
Uses Cloud Foundry v2 - open source
PaaS, developed by Vmware
Evaluating … to be continued
