The document discusses challenges with developing infrastructure using Puppet in production environments. It describes how editing and committing code directly to production leads to chaos. To address this, it recommends using branches to represent environments, committing code changes to branches, and then using Mcollective to update Puppet on nodes in that environment. This allows for independent development without impacting production. It also acknowledges the reality that hardcoded values and site-specific code and data can still lead to slow development cycles.

1. Test driven
Infrastructure
development
Tomas (t0m) Doran
<tomas.doran@timgroup.com>
@bobtfish
https://github.com/bobtfish
https://github.com/youdevise
Thursday, 14 March 13

2. ‘Real men’ develop in
production!
Thursday, 14 March 13
Repeat again and again. Development cycle SLOOOW.

3. ‘Real men’ develop in
production!
• Edit / Commit / Push
Thursday, 14 March 13
Repeat again and again. Development cycle SLOOOW.

4. ‘Real men’ develop in
production!
• Edit / Commit / Push
• Update puppetmaster
Thursday, 14 March 13
Repeat again and again. Development cycle SLOOOW.

5. ‘Real men’ develop in
production!
• Edit / Commit / Push
• Update puppetmaster
• puppet agent -t
Thursday, 14 March 13
Repeat again and again. Development cycle SLOOOW.

6. ‘Real men’ develop in
production!
• Edit / Commit / Push
• Update puppetmaster
• puppet agent -t
• Repeat
Thursday, 14 March 13
Repeat again and again. Development cycle SLOOOW.

7. This is insane!
Thursday, 14 March 13
CHOAS and FAIL result when you break each other. Or, MORE likely (this happens twice a
day!)

8. This is insane!
• Try it on an 8 person team.
Thursday, 14 March 13
CHOAS and FAIL result when you break each other. Or, MORE likely (this happens twice a
day!)

9. This is insane!
• Try it on an 8 person team.
Thursday, 14 March 13
CHOAS and FAIL result when you break each other. Or, MORE likely (this happens twice a
day!)

10. This is insane!
• Try it on an 8 person team.
• ‘LOL - I broke puppet’
Thursday, 14 March 13
CHOAS and FAIL result when you break each other. Or, MORE likely (this happens twice a
day!)

11. OI!!!
Thursday, 14 March 13

12. OI!!!
OI t0m!!!!
Thursday, 14 March 13

13. OI!!!
OI t0m!!!!
You broke
puppet!
Thursday, 14 March 13

14. OI!!!
OI t0m!!!!
You broke
puppet!
AARRRGGH!!!
Thursday, 14 March 13

15. Lets fix this!
• First, a glossary:
Thursday, 14 March 13

16. Lets fix this!
• First, a glossary:
• mco - mcollective
Thursday, 14 March 13

17. Lets fix this!
• First, a glossary:
• mco - mcollective
• ENC - External node classifier
Thursday, 14 March 13

18. We can do better
Thursday, 14 March 13
This at least lets you develop things independently. Everyone can do dev in their own branch
and merge once they have something that doesn’t break _everything_. You can also rebase -i
(squash) all the ARGH PUPPET SYNTAX commits.

19. We can do better
• Branch == environment
Thursday, 14 March 13
This at least lets you develop things independently. Everyone can do dev in their own branch
and merge once they have something that doesn’t break _everything_. You can also rebase -i
(squash) all the ARGH PUPPET SYNTAX commits.

20. We can do better
• Branch == environment
• Branch / Commit / Push
Thursday, 14 March 13
This at least lets you develop things independently. Everyone can do dev in their own branch
and merge once they have something that doesn’t break _everything_. You can also rebase -i
(squash) all the ARGH PUPPET SYNTAX commits.

21. We can do better
• Branch == environment
• Branch / Commit / Push
• mco puppetupdate
Thursday, 14 March 13
This at least lets you develop things independently. Everyone can do dev in their own branch
and merge once they have something that doesn’t break _everything_. You can also rebase -i
(squash) all the ARGH PUPPET SYNTAX commits.

22. We can do better
• Branch == environment
• Branch / Commit / Push
• mco puppetupdate
• puppet agent -t
--environment xxx
Thursday, 14 March 13
This at least lets you develop things independently. Everyone can do dev in their own branch
and merge once they have something that doesn’t break _everything_. You can also rebase -i
(squash) all the ARGH PUPPET SYNTAX commits.

23. Sounds good?
•Then you’ll be wanting:
•https://github.com/
youdevise/puppetupdate
Thursday, 14 March 13
It’s a bit basic, but then I ripped it out of work internal code at 8am ;)

24. So we fixed it?
Thursday, 14 March 13

25. So we fixed it?
Thursday, 14 March 13

26. Refactoring
Thursday, 14 March 13
Sorry Chris, but when you say ‘refactoring’ - it’s not refactoring unless you have tests.
The problem is that you can’t always remember to run the right branch on all the right nodes.
Or rather, how do you even know what all the right nodes are? And if you’re hacking on
custom functions, or anything using exported resource - WOE

27. Refactoring
• We change things to be consistent across
codebase:
• Why did puppet just delete all the
firewall rules on the production
database?
Thursday, 14 March 13
Sorry Chris, but when you say ‘refactoring’ - it’s not refactoring unless you have tests.
The problem is that you can’t always remember to run the right branch on all the right nodes.
Or rather, how do you even know what all the right nodes are? And if you’re hacking on
custom functions, or anything using exported resource - WOE

28. Refactoring
• We change things to be consistent across
codebase:
• Why did puppet just delete all the
firewall rules on the production
database?
• We don’t refactor:
• Add bugs all the time due to
inconsistency
Thursday, 14 March 13
Sorry Chris, but when you say ‘refactoring’ - it’s not refactoring unless you have tests.
The problem is that you can’t always remember to run the right branch on all the right nodes.
Or rather, how do you even know what all the right nodes are? And if you’re hacking on
custom functions, or anything using exported resource - WOE

29. Unfortunate reality:
• Hard coded IPs in 10 places
Thursday, 14 March 13
So, despite our best efforts, our puppet code was SHIIIIT.
Exported resources IS NOT a good fit for non-trivial things (like generating load balancer
configs). Ergo lots of hard coded IPs in multiple places. Ergo puppet code per site.

30. Unfortunate reality:
• Hard coded IPs in 10 places
• role::oy_lb
Thursday, 14 March 13
So, despite our best efforts, our puppet code was SHIIIIT.
Exported resources IS NOT a good fit for non-trivial things (like generating load balancer
configs). Ergo lots of hard coded IPs in multiple places. Ergo puppet code per site.

31. Unfortunate reality:
• Hard coded IPs in 10 places
• role::oy_lb
• hiera data split by domain (colo)
Thursday, 14 March 13
So, despite our best efforts, our puppet code was SHIIIIT.
Exported resources IS NOT a good fit for non-trivial things (like generating load balancer
configs). Ergo lots of hard coded IPs in multiple places. Ergo puppet code per site.

32. Unfortunate reality:
• Hard coded IPs in 10 places
• role::oy_lb
• hiera data split by domain (colo)
• mco puppet
Thursday, 14 March 13
So, despite our best efforts, our puppet code was SHIIIIT.
Exported resources IS NOT a good fit for non-trivial things (like generating load balancer
configs). Ergo lots of hard coded IPs in multiple places. Ergo puppet code per site.

33. Unfortunate reality:
• Hard coded IPs in 10 places
• role::oy_lb
• hiera data split by domain (colo)
• mco puppet
• 4 weeks per app per environment
Thursday, 14 March 13
So, despite our best efforts, our puppet code was SHIIIIT.
Exported resources IS NOT a good fit for non-trivial things (like generating load balancer
configs). Ergo lots of hard coded IPs in multiple places. Ergo puppet code per site.

34. The state of the art
Thursday, 14 March 13

35. The state of the art
• It’s certainly in a state
Thursday, 14 March 13
Nobody does automatic runs
Puppet becomes an auditing tool (automatic noop runs + reports)

36. The state of the art
• It’s certainly in a state
• Automatic runs dangerous
Thursday, 14 March 13
Nobody does automatic runs
Puppet becomes an auditing tool (automatic noop runs + reports)

37. The state of the art
• It’s certainly in a state
• Automatic runs dangerous
• cron --noop runs
Thursday, 14 March 13
Nobody does automatic runs
Puppet becomes an auditing tool (automatic noop runs + reports)

38. The state of the art
• It’s certainly in a state
• Automatic runs dangerous
• cron --noop runs
• puppet becomes an auditing system
Thursday, 14 March 13
Nobody does automatic runs
Puppet becomes an auditing tool (automatic noop runs + reports)

39. The state of the art
• It’s certainly in a state
• Automatic runs dangerous
• cron --noop runs
• puppet becomes an auditing system
• This isn’t what I signed up for!
Thursday, 14 March 13
Nobody does automatic runs
Puppet becomes an auditing tool (automatic noop runs + reports)

40. Business says no!
Thursday, 14 March 13

41. Business says no!
• Launching new products has a long lead
time
• This is unhelpful if your company is trying
to branch out into new markets
Thursday, 14 March 13

42. Business says no!
• Launching new products has a long lead
time
• This is unhelpful if your company is trying
to branch out into new markets
• CI / stage environments unlike prod
• Issues when new functionality goes live
• Developers think you’re incompetent
Thursday, 14 March 13

43. What is wrong
with this picture?
Thursday, 14 March 13
You just don’t know the answer to any of these questions in any reliable way...
But, generally, the answers are NO, YES, NO, NO

44. What is wrong
with this picture?
• Did you run it everywhere?
Thursday, 14 March 13
You just don’t know the answer to any of these questions in any reliable way...
But, generally, the answers are NO, YES, NO, NO

45. What is wrong
with this picture?
• Did you run it everywhere?
• Does it affect anything you’re
not expecting?
Thursday, 14 March 13
You just don’t know the answer to any of these questions in any reliable way...
But, generally, the answers are NO, YES, NO, NO

46. What is wrong
with this picture?
• Did you run it everywhere?
• Does it affect anything you’re
not expecting?
• Can you rebuild cleanly?
Thursday, 14 March 13
You just don’t know the answer to any of these questions in any reliable way...
But, generally, the answers are NO, YES, NO, NO

47. What is wrong
with this picture?
• Did you run it everywhere?
• Does it affect anything you’re
not expecting?
• Can you rebuild cleanly?
• Does the code even make things
reflect current state?
Thursday, 14 March 13
You just don’t know the answer to any of these questions in any reliable way...
But, generally, the answers are NO, YES, NO, NO

48. ‘We use puppet’
Thursday, 14 March 13
Hint - you don’t!

49. ‘We use puppet’
• Means nothing
Thursday, 14 March 13
Hint - you don’t!

50. ‘We use puppet’
• Means nothing
• State of your system is
the sum of all changes
Thursday, 14 March 13
Hint - you don’t!

51. ‘We use puppet’
• Means nothing
• State of your system is
the sum of all changes
• How do you know your
code can rebuild things?
Thursday, 14 March 13
Hint - you don’t!

52. It’s all mierda
Thursday, 14 March 13
We need to grow up, and raise the level of the conversation..

53. It’s all mierda
• Development communities are 10
years ahead
Thursday, 14 March 13
We need to grow up, and raise the level of the conversation..

54. It’s all mierda
• Development communities are 10
years ahead
• We don’t integration test
• (repeatably)
Thursday, 14 March 13
We need to grow up, and raise the level of the conversation..

55. It’s all mierda
• Development communities are 10
years ahead
• We don’t integration test
• (repeatably)
• We can’t build / rebuild
• (reliably)
Thursday, 14 March 13
We need to grow up, and raise the level of the conversation..

56. Infra is hard
Thursday, 14 March 13
Sure - it’s much much harder to get a standalone testable system in infra than it is in
development.

57. Infra is hard
• Infrastructure is inherently more complex
Thursday, 14 March 13
Sure - it’s much much harder to get a standalone testable system in infra than it is in
development.

58. Infra is hard
• Infrastructure is inherently more complex
• Less control
Thursday, 14 March 13
Sure - it’s much much harder to get a standalone testable system in infra than it is in
development.

59. Infra is hard
• Infrastructure is inherently more complex
• Less control
• More moving parts
Thursday, 14 March 13
Sure - it’s much much harder to get a standalone testable system in infra than it is in
development.

60. Infra is hard
• Infrastructure is inherently more complex
• Less control
• More moving parts
• ‘End to end’ testing
Thursday, 14 March 13
Sure - it’s much much harder to get a standalone testable system in infra than it is in
development.

61. Infra is hard
• Infrastructure is inherently more complex
• Less control
• More moving parts
• ‘End to end’ testing
• Persistent data
Thursday, 14 March 13
Sure - it’s much much harder to get a standalone testable system in infra than it is in
development.

62. No excuses:
Scientific method
Thursday, 14 March 13
I do not consider this an excuse to abandon sanity.

63. The solution?
Thursday, 14 March 13

64. The solution?
• Re-provision everything in tests
• N.B. Not perfect (but better!)
Thursday, 14 March 13

65. The solution?
• Re-provision everything in tests
• N.B. Not perfect (but better!)
Thursday, 14 March 13

66. The solution?
• Re-provision everything in tests
• N.B. Not perfect (but better!)
• Proper software engineering
• Unit and integration tests
• Build pipeline + promotion
Thursday, 14 March 13

67. Openstack
• Our tests spinning up 12 machines => VMs
Thursday, 14 March 13
So, we should use openstack, right? As of December, when we looked - 2 networks max,
inflexible. lvs not possible.

68. Openstack
• Our tests spinning up 12 machines => VMs
• Openstack going to be awesome, right now:
Thursday, 14 March 13
So, we should use openstack, right? As of December, when we looked - 2 networks max,
inflexible. lvs not possible.

69. Openstack
• Our tests spinning up 12 machines => VMs
• Openstack going to be awesome, right now:
• Networking sucks
Thursday, 14 March 13
So, we should use openstack, right? As of December, when we looked - 2 networks max,
inflexible. lvs not possible.

70. Openstack
• Our tests spinning up 12 machines => VMs
• Openstack going to be awesome, right now:
• Networking sucks
• Load balancing is a shambles
Thursday, 14 March 13
So, we should use openstack, right? As of December, when we looked - 2 networks max,
inflexible. lvs not possible.

71. Openstack
• Our tests spinning up 12 machines => VMs
• Openstack going to be awesome, right now:
• Networking sucks
• Load balancing is a shambles
• lvs / vlans / metal / bonding - nope
Thursday, 14 March 13
So, we should use openstack, right? As of December, when we looked - 2 networks max,
inflexible. lvs not possible.

72. My desires:
Thursday, 14 March 13

73. My desires:
• Reuse as much code as possible! (e.g. load
balancers)
Thursday, 14 March 13

74. My desires:
• Reuse as much code as possible! (e.g. load
balancers)
• No per colo/environment puppet code
Thursday, 14 March 13

75. My desires:
• Reuse as much code as possible! (e.g. load
balancers)
• No per colo/environment puppet code
• No IPs anywhere
Thursday, 14 March 13

76. My desires:
• Reuse as much code as possible! (e.g. load
balancers)
• No per colo/environment puppet code
• No IPs anywhere
• ‘DRY’
Thursday, 14 March 13

77. My desires:
• Reuse as much code as possible! (e.g. load
balancers)
• No per colo/environment puppet code
• No IPs anywhere
• ‘DRY’
• CI pipeline to promote to production
Thursday, 14 March 13

78. My desires:
• Reuse as much code as possible! (e.g. load
balancers)
• No per colo/environment puppet code
• No IPs anywhere
• ‘DRY’
• CI pipeline to promote to production
• 1 puppet run from provisioned to working
Thursday, 14 March 13

79. My desires:
• Reuse as much code as possible! (e.g. load
balancers)
• No per colo/environment puppet code
• No IPs anywhere
• ‘DRY’
• CI pipeline to promote to production
• 1 puppet run from provisioned to working
• Repeatable and testable!
Thursday, 14 March 13

80. Orc
Thursday, 14 March 13

81. Orc
• Continuous (zero downtime) deployment
Thursday, 14 March 13

82. Orc
• Continuous (zero downtime) deployment
• Development / infrastructure application
contract
Thursday, 14 March 13

83. Orc
• Continuous (zero downtime) deployment
• Development / infrastructure application
contract
• Model driven
Thursday, 14 March 13

84. Orc
• Continuous (zero downtime) deployment
• Development / infrastructure application
contract
• Model driven
• https://github.com/youdevise/orc/
Thursday, 14 March 13

85. Puppetroll
Thursday, 14 March 13

86. Puppetroll
• Rolls out a consistent sha1 from the
puppetmaster to an entire environment
Thursday, 14 March 13

87. Puppetroll
• Rolls out a consistent sha1 from the
puppetmaster to an entire environment
• Fails if any puppet run fails
Thursday, 14 March 13

88. Puppetroll
• Rolls out a consistent sha1 from the
puppetmaster to an entire environment
• Fails if any puppet run fails
• https://github.com/youdevise/puppetroll
Thursday, 14 March 13

89. Provisioning tools
Thursday, 14 March 13

90. Provisioning tools
• debootstrap custom gold images
Thursday, 14 March 13

91. Provisioning tools
• debootstrap custom gold images
• mcollective ‘computenode’ agent for kvm
Thursday, 14 March 13

92. Provisioning tools
• debootstrap custom gold images
• mcollective ‘computenode’ agent for kvm
• ‘provision me a machine called X, on
networks Y and Z’
Thursday, 14 March 13

93. Provisioning tools
• debootstrap custom gold images
• mcollective ‘computenode’ agent for kvm
• ‘provision me a machine called X, on
networks Y and Z’
• Dynamic IP allocation (dnsmasq locally,
DDNS for real)
Thursday, 14 March 13

94. stacks
Thursday, 14 March 13

95. stacks
• Model driven deployment
Thursday, 14 March 13

96. stacks
• Model driven deployment
• DSL for describing groups of systems +
dependencies
Thursday, 14 March 13

97. stacks
• Model driven deployment
• DSL for describing groups of systems +
dependencies
• rake tasks to provision / test / clean up
stack + deps
Thursday, 14 March 13

98. stacks
• Model driven deployment
• DSL for describing groups of systems +
dependencies
• rake tasks to provision / test / clean up
stack + deps
• Can provision a full environment, run E2E
tests, tear it down - in CI.
Thursday, 14 March 13

99. Thursday, 14 March 13

100. Thursday, 14 March 13

101. I want to hack on load
balancers
= 4 new, independent machines
Thursday, 14 March 13

102. How it works?
Thursday, 14 March 13

103. How it works?
• DSL creates model of systems
Thursday, 14 March 13

104. How it works?
• DSL creates model of systems
• rake task ‘launch’:
Thursday, 14 March 13

105. How it works?
• DSL creates model of systems
• rake task ‘launch’:
• mco provisions boxes on compute nodes
Thursday, 14 March 13

106. How it works?
• DSL creates model of systems
• rake task ‘launch’:
• mco provisions boxes on compute nodes
• each box runs puppet --waitforcert
Thursday, 14 March 13

107. How it works?
• DSL creates model of systems
• rake task ‘launch’:
• mco provisions boxes on compute nodes
• each box runs puppet --waitforcert
• mco signs cert
Thursday, 14 March 13

108. How it works?
• DSL creates model of systems
• rake task ‘launch’:
• mco provisions boxes on compute nodes
• each box runs puppet --waitforcert
• mco signs cert
• puppet runs for each box
Thursday, 14 March 13

109. mco computenode
Thursday, 14 March 13

110. Puppetmaster
Thursday, 14 March 13

111. Puppetmaster
• Uses the same model
Thursday, 14 March 13

112. Puppetmaster
• Uses the same model
• Generates an ENC for each node
Thursday, 14 March 13

113. Puppetmaster
• Uses the same model
• Generates an ENC for each node
• Puppet code:
Thursday, 14 March 13

114. Puppetmaster
• Uses the same model
• Generates an ENC for each node
• Puppet code:
• Just installs things / starts services
Thursday, 14 March 13

115. Puppetmaster
• Uses the same model
• Generates an ENC for each node
• Puppet code:
• Just installs things / starts services
• I.E. what it’s good at!
Thursday, 14 March 13

116. External node classifier
Thursday, 14 March 13

117. Putting it together
Thursday, 14 March 13
So, what do we have? Well - everything I showed you already...
Building proxy server layer (by refactoring puppet code) right now. Databases to follow!

118. Putting it together
• Still ongoing - live production apps ETA two
weeks.
Thursday, 14 March 13
So, what do we have? Well - everything I showed you already...
Building proxy server layer (by refactoring puppet code) right now. Databases to follow!

119. Putting it together
• Still ongoing - live production apps ETA two
weeks.
• Still haven’t solved re-provisioning problem
for live environments!
Thursday, 14 March 13
So, what do we have? Well - everything I showed you already...
Building proxy server layer (by refactoring puppet code) right now. Databases to follow!

120. Putting it together
• Still ongoing - live production apps ETA two
weeks.
• Still haven’t solved re-provisioning problem
for live environments!
• Do have repeatable and testable / tested
infrastructure building in CI!
Thursday, 14 March 13
So, what do we have? Well - everything I showed you already...
Building proxy server layer (by refactoring puppet code) right now. Databases to follow!

121. Thursday, 14 March 13

122. Thursday, 14 March 13
The top table is our test overview - we have two types of tests, those which are for a specific
machine (i.e. a VM) and those which are for a virtual service (backed by multiple machines)
‘behaves like’ is an rspec thing we haven’t overridden.
For each machine, we test that it’s pingable, then run every nrpe (nagios) agent and check

123. In the (near) future?
Thursday, 14 March 13

124. In the (near) future?
• Live application stack in production
Thursday, 14 March 13

125. In the (near) future?
• Live application stack in production
• Automated ‘promotion’ of good changes to
production
Thursday, 14 March 13

126. In the (near) future?
• Live application stack in production
• Automated ‘promotion’ of good changes to
production
• Integrated environment support for dev
stacks on dev branches/environments
Thursday, 14 March 13

127. In the (near) future?
• Live application stack in production
• Automated ‘promotion’ of good changes to
production
• Integrated environment support for dev
stacks on dev branches/environments
• Open source all the things!
Thursday, 14 March 13

128. Thanks!
Thursday, 14 March 13

129. Thanks!
• puppet is an awesome tool.
• It doesn’t solve higher level system
modeling problems
• It shouldn’t try to!
Thursday, 14 March 13

130. Thanks!
• puppet is an awesome tool.
• It doesn’t solve higher level system
modeling problems
• It shouldn’t try to!
• sysadmins need to level up
• It’s not done till you can test it still works
Thursday, 14 March 13

131. Photo Credits
• Escher's "Relativity" in LEGO - Andrew Lipson (http://www.andrewlipson.com)
• Manure - Flickr - chesbayprogram
• Provisions - Flickr - quinn.anya
• Stacked - Flickr - andrewrennie
• Dilbert - Flickr - osde-info
• Stacking wood - fickr - arthuserea
• Square wheels - Flickr - vrogy
• Puppets - Flickr - SkipSteuart
• Light bulb - Flickr - bazik
• This-is-not-art - Wikimedia commons - Loran Davis
• Danger of death - Flickr - zigazou76
• Bob the Builder - Flickr - jamesclay
• Swiss roll - Flickr - add1sun
• Orc - Flickr - photo_munki
• Danger! Danger - Flickr - donsolo
• Cow of the future - Flickr - thewamphyri
• SCIENCE - Flickr - chasblackman
Thursday, 14 March 13

132. Links!
• http://github.com/youdevise
• http://github.com/bobtfish
• https://devblog.timgroup.com/
• (Yes, we are hiring)
Thursday, 14 March 13