This document discusses how to use open source tools like Puppet, Jenkins, and Vagrant to create development and testing environments that closely mimic production. It recommends configuring Puppet to provision development environments locally in Vagrant VMs as well as provisioning test VMs in Jenkins. It also provides guidance on setting up a local Debian package repository to control software versions and deploy applications via packages.

1. ENTER
How to use Open Source Software to have near
production development and testing
environments
23.04.15 Matthias Klein

2. IntroductionIntroduction11
PuppetPuppet22
Package ManagementPackage Management33
JenkinsJenkins44
VagrantVagrant55
23.04.15 Matthias Klein 2

3. 23.04.15 Matthias Klein 3
●
Since 6 years SysAdmin in browser gaming companies
●
Since 2 years responsible for internal systems @InnoGames,
including SysAdmin internals and centralised services for our games
●
InnoGames is a developer and provider of online games with 150
million registered players, located in Hamburg
About me

4. 23.04.15 Matthias Klein 4
●
We are operating around 8000 servers, 90% are VMs
●
We are using Debian on almost all machines
●
For management, deployment and monitoring we use Puppet,
Jenkins, dpkg-deb and Icinga (+ mod_gearman)
●
We have our self written admintool (inventory system) which can
pass information to puppet
●
We rewrote apt-mirror to have a better handling of our repositories
About the infrastructure

5. 23.04.15 Matthias Klein 5
You may know
new fancy code
Server
deployment
BOOM

6. 23.04.15 Matthias Klein 6
●
Different software versions on dev/testing/production servers
●
Missing software on production machines
●
Deployment has also build steps
●
Configurations on dev/testing/production don't match
Things that cause trouble

7. 23.04.15 Matthias Klein 7
How it could be
new fancy code
Server
deployment
Surrrr

8. 23.04.15 Matthias Klein 8
●
Developers test locally in a vagrant machine which is provisioned
by the same puppet files as the production servers (WIP)
●
Jenkins is a job starter, tests are executed on VMs, also using the
puppet files from production
●
Jenkins builds a .deb package on demand, which then is made
available at the update server and tested on staging systems
●
The tested package is deployed to the production system
How did we achieve this?

9. IntroductionIntroduction11
PuppetPuppet22
Package ManagementPackage Management33
JenkinsJenkins44
VagrantVagrant55
23.04.15 Matthias Klein 9

10. 23.04.15 Matthias Klein 10
●
Puppet is a configuration management system that allows you to
define the state of your IT infrastructure, then automatically
enforces the correct state. (https://puppetlabs.com)
●
Environments: have multiple Puppet masters glued into one with
possible interaction
●
external_nodes: instead of searching in site.pp, Puppet executes a
script which returns the classes to apply, the environment and other
useful variables defined by you
●
Hiera: define additional variables to use in different scopes and/or
environments
Puppet and extensions

11. 23.04.15 Matthias Klein 11
One possibility
nginx-config.server1
…
server_name a.example.com
...
nginx-config.server2
…
server_name b.example.com
...
if ($::fqdn == server1) {
file { 'nginx-config':
source => 'nginx-config.server1'
}
}
elsif ($::fqdn == server2) {
file { 'nginx-config':
source => 'nginx-config.server2'
}
}

12. 23.04.15 Matthias Klein 12
Using Hiera and environments
…
"project::servername": "a.example.com"
...
…
"project::servername": "b.example.com"
...
class project($servername)
…
file {'nginx-config':
template => 'nginx-config.erb'
...
nginx-config.erb
…
server_name <%= @servername %>;
...

13. 23.04.15 Matthias Klein
IntroductionIntroduction11
PuppetPuppet22
Package ManagementPackage Management33
JenkinsJenkins44
VagrantVagrant55
13

14. 23.04.15 Matthias Klein 14
●
Control which packages and versions are available
●
Use puppets 'latest' functionality
●
Make testing easier with dedicated repositories
●
Deploy your application with a package via 'apt-get install'
Why you need your own package server

15. 23.04.15 Matthias Klein 15
●
Use apt-mirror to mirror one/multiple repositories to your server
●
Put the packages you need to a new folder (to save disk space use
hardlinks, don't forget the dependencies) => this will become your
new repository
●
Use apt-ftparchive to generate signatures and Packages files
●
Make sure you enabled the possibility to download the packages
(http/ftp)
●
Soon to be available: ig.mirror / ig.package => a more comfortable
way to handle your repositories
How to get your own package server

16. 23.04.15 Matthias Klein 16
●
You should not allow other package sources except yours
(especially when using puppets 'latest' functionality)
●
You should check if every package installed on the machine is
available in your repository (you may miss updates)
●
You should check if the packages on your machine are the same
version as in your repositories
●
You should check if your repositories are up to date with the
upstream
●
You may have to explain to your Devs why they can't install most
packages anymore
Caveats

17. 23.04.15 Matthias Klein
IntroductionIntroduction11
PuppetPuppet22
Package ManagementPackage Management33
JenkinsJenkins44
VagrantVagrant55
17

18. 23.04.15 Matthias Klein 18
●
Tests aren't done on the Jenkins server anymore, Jenkins starts
them on dedicated VMs (easy to create thanks to Puppet)
●
We use XEN VMs because we had severe problems with vagrant for
testing:
– Start/Stop multiple vagrant boxes at the same time is a bad idea, at least with
VirtualBox
– Provisioning takes a long time (but can be reduced by suspending boxes)
– Overall perfomance is very bad (around 1/10th compared to XEN)
Jenkins does new things

19. 23.04.15 Matthias Klein 19
●
You need a build trigger in Jenkins (we use the Tag event in gitlab)
●
You need your application built and put to the desired path
●
You need a control file in the DEBIAN folder, containing information
like name, version, dependencies, conflicts
●
You can add scripts to the DEBIAN/(pre|post)(inst|rm|upgrade) files
to do stuff your package needs to work (e.g. starting/stopping
daemons)
●
To build: dpkg-deb -b <SOURCE_DIR> <PACKAGE_DIR>
●
Push / pull the package to your repository and regenerate Packages
files
How to build a Debian package

20. 23.04.15 Matthias Klein
IntroductionIntroduction11
PuppetPuppet22
Package ManagementPackage Management33
JenkinsJenkins44
VagrantVagrant55
20

21. 23.04.15 Matthias Klein 21
●
Use the puppet_server provisioner, make sure you set up an
environment for vagrant => to not conflict with production use a
dedicated server
●
Teach your Devs how to configure their stuff in puppet, let them put
it to a VCS, check these definitions against the ones you are using
in production and merge/reject changes
●
You will have classes which you only use for the vagrant boxes, but
check them also and have them in production, because important
settings may hide there
Vagrant is for your Devs

22. 23.04.15 Matthias Klein 22
Thanks for listening
Questions?
For additional information, availability of the repository scripts and
some nagios checks contact matthias.klein@innogames.com
The End