SlideShare a Scribd company logo
1 of 57
Download to read offline
Sock Puppets: Growing your
puppet codebase

Fabio Lessa & Andrew Myers
Part 1
Empty server room
Project teams




          teams
OPS team for app
    support
The
                                   codebase
                                   we've
                                   started with
Something that represents a mess
Standard
Puppet Master setup
Code was split by environments
Workflow:

1. Push change

2. Wait for it to propagate to that
environment

3. Check it worked

If you need that applied to another
environment, go to 1
The real Workflow:
1. Push change

1.1. Stop the puppet daemon in all the critical nodes for that
environment

1.2. Run puppetd manually in one host
to ensure it works

1.3. Start the daemon again (on all the hosts you remember)

2. Wait for it to propagate to that environment

3. Check it worked (meh)

If you need that applied to another environment, go to 1 (if you
remember, that is)
Production




UAT
                   System Testing




           CI
Making changes
was painful
Part 2
"don't duplicate your manifests in your tests" Focus
  on the catalog problems he described earlier and
 test your logic. Don't test if puppet is doing it's job,
         test that your logic it's doing it's job.

                                        - Nikolay Sturm
The goal...
Before          After
So how do we test/refactor
Compile Puppet Catalogues
Compare Versions
Application specific tests
Tying it all together
Part 3
Continuous Delivery
We were moving application deploys, why
not do the same with infrastructure?
MCollective
The new setup
Picture of the pipelines
info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/
puppet/modules/hostinfo/lib/facter/hostinfo.rb
info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/
puppet/modules/concat/lib/facter/concat_basedir.rb
info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb
info: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rb
info: Loading facts in /var/lib/puppet/facts/zone.rb
info: Loading facts in /var/lib/puppet/facts/read_facts.rb
info: Loading facts in /var/lib/puppet/facts/stomp_host.rb
info: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rb
info: Caching catalog for somehost.nbndc.local
err: Failed to apply catalog: Could not find dependency File[/root/.hgrc] for
Mercurial::Repo::Clone[proddc2-repo] at /usr/share/puppet-recipes/noop/puppet-
environments/puppet/modules/puppetmaster/manifests/repos.pp:36
info: Retrieving plugin
info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb
info: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rb
info: Loading facts in /var/lib/puppet/facts/zone.rb
info: Loading facts in /var/lib/puppet/facts/read_facts.rb
info: Loading facts in /var/lib/puppet/facts/stomp_host.rb
info: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rb
info: Caching catalog for somehost.nbndc.local
err: Failed to apply catalog: You cannot specify more than one of
content, source, target at /usr/share/puppet-recipes/noop/puppet-
environments/puppet/modules/sbs/manifests/init.pp:73
notice: Class[Hostinfo]: Would have triggered 'refresh' from 1 events
notice: /Stage[main]/Java::Sun/Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]/
Java::Jdk::Default[1.6.0.31-sun]/Notify[java::jdk::default:sun]/message:
current_value absent, should be Setting java the following java as default:
1.6.0.31 sun 1.6.0 (noop)
notice: Java::Jdk::Default[1.6.0.31-sun]: Would have triggered 'refresh'
from 1 events
notice: Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]: Would have triggered
'refresh' from 1 events
notice: Class[Java::Sun]: Would have triggered 'refresh' from 1 events
notice: /Stage[main]/Greenmail/File[/etc/greenmail]/ensure: current_value
absent, should be directory (noop)
err: /Stage[main]/Greenmail/File[/etc/greenmail/greenmail.conf]: Could not
evaluate: Could not retrieve information from environment noop source(s)
puppet://puppet/modules/greenmail/etc/greenmail/greenmail.conf at /usr/
share/puppet-recipes/noop/puppet-environments/puppet/modules/greenmail/
manifests/init.pp:30
notice: Class[Puppetagent]: Would have triggered 'refresh' from 1 events
notice: /Stage[main]/Greenmail/File[/usr/greenmail]/ensure: current_value
directory, should be absent (noop)
notice: /Stage[main]/Greenmail/File[/usr/greenmail/startgm.sh]/ensure:
current_value file, should be absent (noop)
We were moving application deploys, why
not do the same with infrastructure?
Our MCollective agent
The new action
At every push, all nodes connect to the noop
     environment and run in noop mode
Picture of a report
Change
management
process
  ?
What changed?
What did you test?
Who will do it?
Part 4
New data centre is ready!
Should be easy to migrate....
Okay so our manifests are incomplete,
but we need to be able to rebuild machines...




        Virtualisation to the rescue...
Magnets?
What happens
Wrap Up
120 pushes to
production in the
last 8 months: 2
failures
MTTR FTW!



       5 minutes!
Thank you!

More Related Content

Similar to Sock puppets

Scalable Systems Management with Puppet
Scalable Systems Management with PuppetScalable Systems Management with Puppet
Scalable Systems Management with PuppetPuppet
 
Scalable systems management with puppet
Scalable systems management with puppetScalable systems management with puppet
Scalable systems management with puppetPuppet
 
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...Puppet
 
Testing your infallibleness
Testing your infalliblenessTesting your infallibleness
Testing your infalliblenessCorey Osman
 
Puppet Systems Infrastructure Construction Kit
Puppet Systems Infrastructure Construction KitPuppet Systems Infrastructure Construction Kit
Puppet Systems Infrastructure Construction KitAlessandro Franceschi
 
A Fabric/Puppet Build/Deploy System
A Fabric/Puppet Build/Deploy SystemA Fabric/Puppet Build/Deploy System
A Fabric/Puppet Build/Deploy Systemadrian_nye
 
CNS440 – Lab Assignment Week 2 Complete Lab 1 Perfo.docx
CNS440 – Lab Assignment Week 2   Complete Lab 1 Perfo.docxCNS440 – Lab Assignment Week 2   Complete Lab 1 Perfo.docx
CNS440 – Lab Assignment Week 2 Complete Lab 1 Perfo.docxmary772
 
INTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCE
INTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCEINTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCE
INTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCEIPutuAdiPratama
 
How to Make a Honeypot Stickier (SSH*)
How to Make a Honeypot Stickier (SSH*)How to Make a Honeypot Stickier (SSH*)
How to Make a Honeypot Stickier (SSH*)Jose Hernandez
 
K8s Quick Start
K8s Quick StartK8s Quick Start
K8s Quick StartGanesh Pol
 
Do you know all of Puppet?
Do you know all of Puppet?Do you know all of Puppet?
Do you know all of Puppet?Julien Pivotto
 
Puppet slides for intelligrape
Puppet slides for intelligrapePuppet slides for intelligrape
Puppet slides for intelligrapeSharad Aggarwal
 
From SaltStack to Puppet and beyond...
From SaltStack to Puppet and beyond...From SaltStack to Puppet and beyond...
From SaltStack to Puppet and beyond...Yury Bushmelev
 
Managing-Splunk-with-Puppet 31-January-2022.pdf
Managing-Splunk-with-Puppet 31-January-2022.pdfManaging-Splunk-with-Puppet 31-January-2022.pdf
Managing-Splunk-with-Puppet 31-January-2022.pdfssusera181ef
 
Bamboo Hands on training 2016
Bamboo Hands on training 2016Bamboo Hands on training 2016
Bamboo Hands on training 2016Takahiro Yamaki
 
Artem zhurbila the story of rebuilding puppet (devops meetup 29.10.2015)
Artem zhurbila  the story of rebuilding puppet (devops meetup 29.10.2015)Artem zhurbila  the story of rebuilding puppet (devops meetup 29.10.2015)
Artem zhurbila the story of rebuilding puppet (devops meetup 29.10.2015)Artem Zhurbila
 
PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...
PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...
PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...Puppet
 
Creating a Scheduled Backup and Replicating System Folders Introduct.docx
Creating a Scheduled Backup and Replicating System Folders Introduct.docxCreating a Scheduled Backup and Replicating System Folders Introduct.docx
Creating a Scheduled Backup and Replicating System Folders Introduct.docxwilliejgrant41084
 
When Good Code Goes Bad: Tools and Techniques for Troubleshooting Plone
When Good Code Goes Bad: Tools and Techniques for Troubleshooting PloneWhen Good Code Goes Bad: Tools and Techniques for Troubleshooting Plone
When Good Code Goes Bad: Tools and Techniques for Troubleshooting PloneDavid Glick
 

Similar to Sock puppets (20)

Scalable Systems Management with Puppet
Scalable Systems Management with PuppetScalable Systems Management with Puppet
Scalable Systems Management with Puppet
 
Scalable systems management with puppet
Scalable systems management with puppetScalable systems management with puppet
Scalable systems management with puppet
 
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...
 
Puppet quick start guide
Puppet quick start guidePuppet quick start guide
Puppet quick start guide
 
Testing your infallibleness
Testing your infalliblenessTesting your infallibleness
Testing your infallibleness
 
Puppet Systems Infrastructure Construction Kit
Puppet Systems Infrastructure Construction KitPuppet Systems Infrastructure Construction Kit
Puppet Systems Infrastructure Construction Kit
 
A Fabric/Puppet Build/Deploy System
A Fabric/Puppet Build/Deploy SystemA Fabric/Puppet Build/Deploy System
A Fabric/Puppet Build/Deploy System
 
CNS440 – Lab Assignment Week 2 Complete Lab 1 Perfo.docx
CNS440 – Lab Assignment Week 2   Complete Lab 1 Perfo.docxCNS440 – Lab Assignment Week 2   Complete Lab 1 Perfo.docx
CNS440 – Lab Assignment Week 2 Complete Lab 1 Perfo.docx
 
INTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCE
INTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCEINTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCE
INTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCE
 
How to Make a Honeypot Stickier (SSH*)
How to Make a Honeypot Stickier (SSH*)How to Make a Honeypot Stickier (SSH*)
How to Make a Honeypot Stickier (SSH*)
 
K8s Quick Start
K8s Quick StartK8s Quick Start
K8s Quick Start
 
Do you know all of Puppet?
Do you know all of Puppet?Do you know all of Puppet?
Do you know all of Puppet?
 
Puppet slides for intelligrape
Puppet slides for intelligrapePuppet slides for intelligrape
Puppet slides for intelligrape
 
From SaltStack to Puppet and beyond...
From SaltStack to Puppet and beyond...From SaltStack to Puppet and beyond...
From SaltStack to Puppet and beyond...
 
Managing-Splunk-with-Puppet 31-January-2022.pdf
Managing-Splunk-with-Puppet 31-January-2022.pdfManaging-Splunk-with-Puppet 31-January-2022.pdf
Managing-Splunk-with-Puppet 31-January-2022.pdf
 
Bamboo Hands on training 2016
Bamboo Hands on training 2016Bamboo Hands on training 2016
Bamboo Hands on training 2016
 
Artem zhurbila the story of rebuilding puppet (devops meetup 29.10.2015)
Artem zhurbila  the story of rebuilding puppet (devops meetup 29.10.2015)Artem zhurbila  the story of rebuilding puppet (devops meetup 29.10.2015)
Artem zhurbila the story of rebuilding puppet (devops meetup 29.10.2015)
 
PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...
PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...
PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...
 
Creating a Scheduled Backup and Replicating System Folders Introduct.docx
Creating a Scheduled Backup and Replicating System Folders Introduct.docxCreating a Scheduled Backup and Replicating System Folders Introduct.docx
Creating a Scheduled Backup and Replicating System Folders Introduct.docx
 
When Good Code Goes Bad: Tools and Techniques for Troubleshooting Plone
When Good Code Goes Bad: Tools and Techniques for Troubleshooting PloneWhen Good Code Goes Bad: Tools and Techniques for Troubleshooting Plone
When Good Code Goes Bad: Tools and Techniques for Troubleshooting Plone
 

Recently uploaded

Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machinePadma Pradeep
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software
 
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationRidwan Fadjar
 
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr LapshynFwdays
 
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Enterprise Knowledge
 
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...shyamraj55
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
costume and set research powerpoint presentation
costume and set research powerpoint presentationcostume and set research powerpoint presentation
costume and set research powerpoint presentationphoebematthew05
 
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks..."LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...Fwdays
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):comworks
 
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Patryk Bandurski
 
SAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxSAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxNavinnSomaal
 
My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024The Digital Insurer
 
Story boards and shot lists for my a level piece
Story boards and shot lists for my a level pieceStory boards and shot lists for my a level piece
Story boards and shot lists for my a level piececharlottematthew16
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed
 
Artificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxArtificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxhariprasad279825
 
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Wonjun Hwang
 
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsMemoori
 

Recently uploaded (20)

Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machine
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
 
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 Presentation
 
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
 
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024
 
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
costume and set research powerpoint presentation
costume and set research powerpoint presentationcostume and set research powerpoint presentation
costume and set research powerpoint presentation
 
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks..."LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR Systems
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):
 
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
 
SAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxSAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptx
 
My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024
 
Story boards and shot lists for my a level piece
Story boards and shot lists for my a level pieceStory boards and shot lists for my a level piece
Story boards and shot lists for my a level piece
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL Certs
 
Artificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxArtificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptx
 
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
 
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial Buildings
 

Sock puppets

  • 1. Sock Puppets: Growing your puppet codebase Fabio Lessa & Andrew Myers
  • 2.
  • 6.
  • 7. OPS team for app support
  • 8. The codebase we've started with Something that represents a mess
  • 10. Code was split by environments
  • 11.
  • 12. Workflow: 1. Push change 2. Wait for it to propagate to that environment 3. Check it worked If you need that applied to another environment, go to 1
  • 13. The real Workflow: 1. Push change 1.1. Stop the puppet daemon in all the critical nodes for that environment 1.2. Run puppetd manually in one host to ensure it works 1.3. Start the daemon again (on all the hosts you remember) 2. Wait for it to propagate to that environment 3. Check it worked (meh) If you need that applied to another environment, go to 1 (if you remember, that is)
  • 14.
  • 15. Production UAT System Testing CI
  • 18.
  • 19. "don't duplicate your manifests in your tests" Focus on the catalog problems he described earlier and test your logic. Don't test if puppet is doing it's job, test that your logic it's doing it's job. - Nikolay Sturm
  • 21. So how do we test/refactor
  • 24.
  • 26. Tying it all together
  • 29. We were moving application deploys, why not do the same with infrastructure?
  • 32. Picture of the pipelines
  • 33. info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/ puppet/modules/hostinfo/lib/facter/hostinfo.rb info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/ puppet/modules/concat/lib/facter/concat_basedir.rb info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb info: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rb info: Loading facts in /var/lib/puppet/facts/zone.rb info: Loading facts in /var/lib/puppet/facts/read_facts.rb info: Loading facts in /var/lib/puppet/facts/stomp_host.rb info: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rb info: Caching catalog for somehost.nbndc.local err: Failed to apply catalog: Could not find dependency File[/root/.hgrc] for Mercurial::Repo::Clone[proddc2-repo] at /usr/share/puppet-recipes/noop/puppet- environments/puppet/modules/puppetmaster/manifests/repos.pp:36
  • 34. info: Retrieving plugin info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb info: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rb info: Loading facts in /var/lib/puppet/facts/zone.rb info: Loading facts in /var/lib/puppet/facts/read_facts.rb info: Loading facts in /var/lib/puppet/facts/stomp_host.rb info: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rb info: Caching catalog for somehost.nbndc.local err: Failed to apply catalog: You cannot specify more than one of content, source, target at /usr/share/puppet-recipes/noop/puppet- environments/puppet/modules/sbs/manifests/init.pp:73
  • 35. notice: Class[Hostinfo]: Would have triggered 'refresh' from 1 events notice: /Stage[main]/Java::Sun/Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]/ Java::Jdk::Default[1.6.0.31-sun]/Notify[java::jdk::default:sun]/message: current_value absent, should be Setting java the following java as default: 1.6.0.31 sun 1.6.0 (noop) notice: Java::Jdk::Default[1.6.0.31-sun]: Would have triggered 'refresh' from 1 events notice: Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]: Would have triggered 'refresh' from 1 events notice: Class[Java::Sun]: Would have triggered 'refresh' from 1 events notice: /Stage[main]/Greenmail/File[/etc/greenmail]/ensure: current_value absent, should be directory (noop) err: /Stage[main]/Greenmail/File[/etc/greenmail/greenmail.conf]: Could not evaluate: Could not retrieve information from environment noop source(s) puppet://puppet/modules/greenmail/etc/greenmail/greenmail.conf at /usr/ share/puppet-recipes/noop/puppet-environments/puppet/modules/greenmail/ manifests/init.pp:30 notice: Class[Puppetagent]: Would have triggered 'refresh' from 1 events notice: /Stage[main]/Greenmail/File[/usr/greenmail]/ensure: current_value directory, should be absent (noop) notice: /Stage[main]/Greenmail/File[/usr/greenmail/startgm.sh]/ensure: current_value file, should be absent (noop)
  • 36. We were moving application deploys, why not do the same with infrastructure?
  • 37.
  • 40. At every push, all nodes connect to the noop environment and run in noop mode
  • 41.
  • 42. Picture of a report
  • 45. What did you test?
  • 46. Who will do it?
  • 48. New data centre is ready!
  • 49. Should be easy to migrate....
  • 50. Okay so our manifests are incomplete, but we need to be able to rebuild machines... Virtualisation to the rescue...
  • 54. 120 pushes to production in the last 8 months: 2 failures
  • 55.
  • 56. MTTR FTW! 5 minutes!