Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Delivering	Premium	Quality	Puppet	Modules	
René	Last	&	Dimitri	Tischenko	
Puppetconf	2016	
Using Beaker and VMpooler for M...
Who	are	we?
Agenda	
	
About	KPN	
Our	problem	
Our	soluBon
About	KPN
Our	problem	
KPN	decided	to	manage	10K	
servers	for	various	clients	
with	Puppet…
Our	problem	
…	most	of	them	Windows.
Our	problem	–	consequences	(1)	
Need	to	
support	Windows
Our	problem	–	consequences	(2)	
Need	to	guarantee		
stable	and	predictable	
infrastructure	state
Our	problem	–	consequences	(3)	
Need	to	guarantee		
code	quality
Our	soluEon	(1)	
A	dedicated	team	
to	develop	modules
Our	soluEon	(2)	
An	automated	
tesBng	factory
TesEng	Factory	Requirements	
Automated	pipeline	configuraBon	
Puppet	3.x	tests	
Puppet	4.x	tests	
Unit	tests	on	Linux	and	W...
Module	CI/CD	Pipeline	Overview	
syntax	
check	
Auto	
puppet-
lint	
rspec	
test	
acceptance	
test	
Release	
to	TEST	
Auto	 ...
Puppet	CI	Components	
Jenkins	Master	
Jenkins	
Linux	Slave	
Jenkins	
Windows	Slave	
vmpooler	
Github	Enterprise	
vcenter	
...
Jenkins	master	and	build	slaves	
rtyler/jenkins	puppet	module
Ruby	
	
	
	
maestrodev/rvm	
PIK	or	uru	!=	rvm	L
Jenkins	jobs	
	
Jenkins	job	builder	
stankevich/python
PuppeEzing	Jenkins	Jobs	
puppet	 Yaml	file	
Jenkins	job	
builder	
xml	 Jenkins	job
How	to	add	a	module	
1.  	kpn_jenkins::pipeline::module	{	'kpn-puppet-forge/puppet-kpn-hosts':	
2.  		ensure														...
How	to	add	a	module
Jenkins	Master	
Jenkins	
Linux	Slave	
Jenkins	
Windows	Slave	
vmpooler	
Github	Enterprise	
vcenter	
Create		VMs	
Windows	p...
Beaker	
A	test	harness	focused	on	acceptance	tesBng	
using		(virtual)	machines
Beaker	
Runs	tests	on	“real”	servers
Beaker	
Supports	vm	providers	like	Vagrant,	vSphere,	
docker,	AWS	or	vmpooler
Beaker	InstallaEon	
	
#	gem	install	beaker	beaker-rspec	beaker-puppet_install_helper
Beaker	ConfiguraEon	
spec/acceptance/nodesets/*.yml
1.  HOSTS:	
2.  		centos-7-x64-pe4:	
3.  				roles:	
4.  						-	agent	...
Prepare	a	System	For	TesEng	
spec/spec_helper_acceptance.rb
1.  require	'beaker-rspec/spec_helper'	
2.  require	'beaker-rs...
Example	Test	
spec/acceptance/*_spec.rb
1.  require	'spec_helper_acceptance'	
2.  		
3.  describe	'kpn-centerity_agent	mod...
TesEng	for	Installed	Packages	
1.  context	'Test	installed	software'	do	
2.  		case	fact('osfamily')	
3.  				when	/RedHat...
Run,	beaker,	Run!	
#	export	BEAKER_setfile=~/nodesets/${platform}.yml	
#	rake	beaker
Rules	for	WriEng	Acceptance	Tests	(1)	
Only	test	your	puppet	code,	
not	the	supplier’s	socware
Rules	for	WriEng	Acceptance	Tests	(2)	
A	beaker	test	should	be	self-contained
Rules	for	WriEng	Acceptance	Tests	(3)	
If	you	can	rspec	it,	don’t	beaker	it
Improving	Beaker	DocumentaEon	
	
	
	
	
	
	
	
heps://github.com/RARYates/beaker
Jenkins	Master	
Jenkins	
Linux	Slave	
Jenkins	
Windows	Slave	
Github	Enterprise	
vcenter	
Create		VMs	
Windows	pipeline	
M...
vmpooler	
	
Vmpooler	pools	VMs	
	
Provides	configurable	pools	of	instantly-available	
(running)	virtual	machines
vmpooler	
	
Needs	its	own	DHCP	zone		
and	properly	configured	DDNS
vmpooler	
	
System	running	beaker	must	use	vmpooler’s	
nameserver
vmpooler	
vmpooler	has	a	strange	logo
vmpooler	installaEon	
include	‘vmpooler’
vmpooler	
	
	
❝	
Template	set-up	is	lec	as	an	exercise	to	the	reader.	Somehow,	either	via	PXE,	embedded	
bootstrap	scripts...
vmpooler	host	naming	puzzle	
vm	name	!=	host	name	
need	to	set	host	name	automaBcally	to	vm	name	
don’t	want	to	give	each	...
Just	5	lines	of	code	(2	months	later)	
								#	Annotate	with	creation	time,	origin	template,	etc.	
								#	Add	extraco...
Seang	the	hostname	
Linux:	
#	hostname=$(vmtoolsd	--cmd	"info-get	guestinfo.hostname")	
	
Windows:	
>	$hostname	=	cmd.exe	...
vmpooler	lessons	learned	
• Can	use	more	logging	(PRs	forthcoming)	
• Pool	name	needs	to	equal	template	name	
• Docs	refer...
Conclusions	
Automated	pipeline	configuraBon		
Puppet	3.x	tests	
Puppet	4.x	tests	
Unit	tests	on	Linux	and	Windows	
Accepta...
What	did	we	learn?	
+ 	Regression	test	
+ 	MulB	plaQorm	
+ 	Catch	errors	early
What	did	we	learn?
Future	Improvements	
Speed	of	tests
Future	Improvements	
Stability	of	vmpooler	
Windows	support
And	now	…	for	some	numbers
Modules	per	OperaEng	System	
Windows	
32	
Linux	
43	
Linux	and	
Windows	
16
Test	Types	
102	 94	
79	
0	
20	
40	
60	
80	
100	
120	
Syntax	 Rspec	(unit)	 Acceptance	
#	modules
Test	Runs	
Flow	
8%	
Init	
14%	
Syntax	
21%	
Unit	
18%	
Acceptance	
39%
Top	10	Modules	with	Failed	Tests
Vmpooler:	Average	Time	to	Ready
Q&A
Windows	Modules	
• kpn-ad_ds	
• kpn-dotnet	
• kpn-rdp	
• kpn-kms	
• kpn-chocolatey_client	
• kpn-chocolatey_server	
• kpn-...
PuppetConf 2016: Delivering Premium Quality Modules: Using Beaker and VMpooler for Multi-Platform Testing – Rene Last, KPN...
PuppetConf 2016: Delivering Premium Quality Modules: Using Beaker and VMpooler for Multi-Platform Testing – Rene Last, KPN...
PuppetConf 2016: Delivering Premium Quality Modules: Using Beaker and VMpooler for Multi-Platform Testing – Rene Last, KPN...
PuppetConf 2016: Delivering Premium Quality Modules: Using Beaker and VMpooler for Multi-Platform Testing – Rene Last, KPN...
Upcoming SlideShare
Loading in …5
×

PuppetConf 2016: Delivering Premium Quality Modules: Using Beaker and VMpooler for Multi-Platform Testing – Rene Last, KPN B.V & Dimitri Tischenko, Myndrik

403 views

Published on

Here are the slides from Rene Last & Dimitri Tischenko's PuppetConf 2016 presentation called Delivering Premium Quality Modules: Using Beaker and VMpooler for Multi-Platform Testing. Watch the videos at https://www.youtube.com/playlist?list=PLV86BgbREluVjwwt-9UL8u2Uy8xnzpIqa

Published in: Technology
  • Be the first to comment

  • Be the first to like this

PuppetConf 2016: Delivering Premium Quality Modules: Using Beaker and VMpooler for Multi-Platform Testing – Rene Last, KPN B.V & Dimitri Tischenko, Myndrik

  1. 1. Delivering Premium Quality Puppet Modules René Last & Dimitri Tischenko Puppetconf 2016 Using Beaker and VMpooler for Mul4-Pla7orm Tes4ng
  2. 2. Who are we?
  3. 3. Agenda About KPN Our problem Our soluBon
  4. 4. About KPN
  5. 5. Our problem KPN decided to manage 10K servers for various clients with Puppet…
  6. 6. Our problem … most of them Windows.
  7. 7. Our problem – consequences (1) Need to support Windows
  8. 8. Our problem – consequences (2) Need to guarantee stable and predictable infrastructure state
  9. 9. Our problem – consequences (3) Need to guarantee code quality
  10. 10. Our soluEon (1) A dedicated team to develop modules
  11. 11. Our soluEon (2) An automated tesBng factory
  12. 12. TesEng Factory Requirements Automated pipeline configuraBon Puppet 3.x tests Puppet 4.x tests Unit tests on Linux and Windows Acceptance tests on all plaQorms (currently 8) Run tests in parallel
  13. 13. Module CI/CD Pipeline Overview syntax check Auto puppet- lint rspec test acceptance test Release to TEST Auto Auto Manual Release to PROD Manual Change in git Periodic trigger NoEfy by mail & Hipchat Fail Fail Fail Fail
  14. 14. Puppet CI Components Jenkins Master Jenkins Linux Slave Jenkins Windows Slave vmpooler Github Enterprise vcenter beaker Create VMs Linux pipeline Windows pipeline MulE-PlaXorm Acceptance Tests Syntax, lint & rspec Syntax, lint & rspec Systems Under Test
  15. 15. Jenkins master and build slaves rtyler/jenkins puppet module
  16. 16. Ruby maestrodev/rvm PIK or uru != rvm L
  17. 17. Jenkins jobs Jenkins job builder stankevich/python
  18. 18. PuppeEzing Jenkins Jobs puppet Yaml file Jenkins job builder xml Jenkins job
  19. 19. How to add a module 1.  kpn_jenkins::pipeline::module { 'kpn-puppet-forge/puppet-kpn-hosts': 2.  ensure => present, 3.  test_platform => ['linux', 'windows'], 4.  pe3_syntax => false, 5.  pe4_syntax => true, 6.  pe4_unit => true, 7.  pe3_unit => false, 8.  acceptance_test => true, 9.  acceptance_platform => [ 'centos-7-x64-pe4', 10.  'windows-2008r2-x64', 11.  'windows-2012r2-x64', 12.  'windows-2008r2-x64-pe4', 13.  'windows-2012r2-x64-pe4', 14.  'windows-2016-x64-pe4', 15.  ], 16.  hipchat_notification => true, 17.  hipchat_room => 'Module team', 18.  email => ‘puppetteam@kpn.com', 19.  }
  20. 20. How to add a module
  21. 21. Jenkins Master Jenkins Linux Slave Jenkins Windows Slave vmpooler Github Enterprise vcenter Create VMs Windows pipeline MulE-PlaXorm Acceptance Tests Syntax, lint & rspec Syntax, lint & rspec Systems Under Test Beaker beaker Linux pipeline
  22. 22. Beaker A test harness focused on acceptance tesBng using (virtual) machines
  23. 23. Beaker Runs tests on “real” servers
  24. 24. Beaker Supports vm providers like Vagrant, vSphere, docker, AWS or vmpooler
  25. 25. Beaker InstallaEon # gem install beaker beaker-rspec beaker-puppet_install_helper
  26. 26. Beaker ConfiguraEon spec/acceptance/nodesets/*.yml 1.  HOSTS: 2.  centos-7-x64-pe4: 3.  roles: 4.  - agent 5.  platform: el-7-x86_64 6.  hypervisor: vmpooler 7.  template: centos-7-x64 8.  CONFIG: 9.  masterless: true 10.  pe_ver: '2016.2.0' 11.  pe_promoted_builds_url: 'http://repo.kpn.com/pe' 12.  pooling_api: 'http://vmpooler.kpn.com:4567' 13.  ssh: 14.  auth_methods: 15.  - password 16.  user: root 17.  password: XXXXXXXX
  27. 27. Prepare a System For TesEng spec/spec_helper_acceptance.rb 1.  require 'beaker-rspec/spec_helper' 2.  require 'beaker-rspec/helpers/serverspec' 3.  require 'beaker/puppet_install_helper' 4.  5.  # Install puppet agent on a node 6.  run_puppet_install_helper 7.  8.  # Copy our module dependencies to the SUT 9.  system('git clone git@github.kpn.com/stdlib spec/fixtures/modules/stdlib') 10.  11. copy_module_to(hosts, :source => 'spec/fixtures/modules/stdlib', 12.  :module => 'stdlib') 13.  14. # Or if you have internet access 15. install_puppet_module_via_pmt_on(host, :module_name => 'stdlib')
  28. 28. Example Test spec/acceptance/*_spec.rb 1.  require 'spec_helper_acceptance' 2.  3.  describe 'kpn-centerity_agent module' do 4.  context 'install centerity_agent software' do 5.  pp = <<-EOS 6.  class { 'centerity_agent': } 7.  EOS 8.  it '1st apply should run without errors' do 9.  apply_manifest(pp, :catch_failures => true) 10.  end 11.  it '2nd apply should run without changes' do 12.  apply_manifest(pp, :catch_changes => true) 13.  end 14.  end 15. end
  29. 29. TesEng for Installed Packages 1.  context 'Test installed software' do 2.  case fact('osfamily') 3.  when /RedHat/ 4.  describe package('sshd') do 5.  it { should be_installed } 6.  end 7.  when /windows/ 8.  describe command('C:ProgramsChocolateybinclist -l') do 9.  its(:stdout) { should match /sshd/ } 10.  end 11.  end 12.  end 13. end
  30. 30. Run, beaker, Run! # export BEAKER_setfile=~/nodesets/${platform}.yml # rake beaker
  31. 31. Rules for WriEng Acceptance Tests (1) Only test your puppet code, not the supplier’s socware
  32. 32. Rules for WriEng Acceptance Tests (2) A beaker test should be self-contained
  33. 33. Rules for WriEng Acceptance Tests (3) If you can rspec it, don’t beaker it
  34. 34. Improving Beaker DocumentaEon heps://github.com/RARYates/beaker
  35. 35. Jenkins Master Jenkins Linux Slave Jenkins Windows Slave Github Enterprise vcenter Create VMs Windows pipeline MulE-PlaXorm Acceptance Tests Syntax, lint & rspec Syntax, lint & rspec Systems Under Test beaker vmpooler vmpooler Linux pipeline
  36. 36. vmpooler Vmpooler pools VMs Provides configurable pools of instantly-available (running) virtual machines
  37. 37. vmpooler Needs its own DHCP zone and properly configured DDNS
  38. 38. vmpooler System running beaker must use vmpooler’s nameserver
  39. 39. vmpooler vmpooler has a strange logo
  40. 40. vmpooler installaEon include ‘vmpooler’
  41. 41. vmpooler ❝ Template set-up is lec as an exercise to the reader. Somehow, either via PXE, embedded bootstrap scripts, or some other method -- clones of VM templates need to be able to set their hostname, register themselves in your DNS, and be resolvable by the vmpooler applicaBon acer compleBng the clone task and booBng up. ❞
  42. 42. vmpooler host naming puzzle vm name != host name need to set host name automaBcally to vm name don’t want to give each vm vsphere credenBals so - how to do it?
  43. 43. Just 5 lines of code (2 months later) # Annotate with creation time, origin template, etc. # Add extraconfig options that can be queried by vmtools configSpec = RbVmomi::VIM.VirtualMachineConfigSpec( annotation: JSON.pretty_generate( name: vm['hostname'], created_by: $config[:vsphere]['username'], base_template: vm['template'], creation_timestamp: Time.now.utc ), extraConfig: [ { key: 'guestinfo.hostname', value: vm['hostname'] } ])
  44. 44. Seang the hostname Linux: # hostname=$(vmtoolsd --cmd "info-get guestinfo.hostname") Windows: > $hostname = cmd.exe /c 'C:Progra~1VMwarevmtoolsd.exe --cmd "info-get guestinfo.hostname"'
  45. 45. vmpooler lessons learned • Can use more logging (PRs forthcoming) • Pool name needs to equal template name • Docs refer to vm templates, need to be vms • Will not install if ESX host is not “green” • Can use beeer excepBon handling
  46. 46. Conclusions Automated pipeline configuraBon Puppet 3.x tests Puppet 4.x tests Unit tests on Linux and Windows Acceptance tests on all plaQorms (currently 8) Run tests in parallel P P P P P P
  47. 47. What did we learn? +  Regression test +  MulB plaQorm +  Catch errors early
  48. 48. What did we learn?
  49. 49. Future Improvements Speed of tests
  50. 50. Future Improvements Stability of vmpooler Windows support
  51. 51. And now … for some numbers
  52. 52. Modules per OperaEng System Windows 32 Linux 43 Linux and Windows 16
  53. 53. Test Types 102 94 79 0 20 40 60 80 100 120 Syntax Rspec (unit) Acceptance # modules
  54. 54. Test Runs Flow 8% Init 14% Syntax 21% Unit 18% Acceptance 39%
  55. 55. Top 10 Modules with Failed Tests
  56. 56. Vmpooler: Average Time to Ready
  57. 57. Q&A
  58. 58. Windows Modules • kpn-ad_ds • kpn-dotnet • kpn-rdp • kpn-kms • kpn-chocolatey_client • kpn-chocolatey_server • kpn-local_security_policy

×