SlideShare a Scribd company logo
1 of 57
Download to read offline
Sock Puppets: Growing your
puppet codebase

Fabio Lessa & Andrew Myers
Part 1
Empty server room
Project teams




          teams
OPS team for app
    support
The
                                   codebase
                                   we've
                                   started with
Something that represents a mess
Standard
Puppet Master setup
Code was split by environments
Workflow:

1. Push change

2. Wait for it to propagate to that
environment

3. Check it worked

If you need that applied to another
environment, go to 1
The real Workflow:
1. Push change

1.1. Stop the puppet daemon in all the critical nodes for that
environment

1.2. Run puppetd manually in one host
to ensure it works

1.3. Start the daemon again (on all the hosts you remember)

2. Wait for it to propagate to that environment

3. Check it worked (meh)

If you need that applied to another environment, go to 1 (if you
remember, that is)
Production




UAT
                   System Testing




           CI
Making changes
was painful
Part 2
"don't duplicate your manifests in your tests" Focus
  on the catalog problems he described earlier and
 test your logic. Don't test if puppet is doing it's job,
         test that your logic it's doing it's job.

                                        - Nikolay Sturm
The goal...
Before          After
So how do we test/refactor
Compile Puppet Catalogues
Compare Versions
Application specific tests
Tying it all together
Part 3
Continuous Delivery
We were moving application deploys, why
not do the same with infrastructure?
MCollective
The new setup
Picture of the pipelines
info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/
puppet/modules/hostinfo/lib/facter/hostinfo.rb
info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/
puppet/modules/concat/lib/facter/concat_basedir.rb
info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb
info: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rb
info: Loading facts in /var/lib/puppet/facts/zone.rb
info: Loading facts in /var/lib/puppet/facts/read_facts.rb
info: Loading facts in /var/lib/puppet/facts/stomp_host.rb
info: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rb
info: Caching catalog for somehost.nbndc.local
err: Failed to apply catalog: Could not find dependency File[/root/.hgrc] for
Mercurial::Repo::Clone[proddc2-repo] at /usr/share/puppet-recipes/noop/puppet-
environments/puppet/modules/puppetmaster/manifests/repos.pp:36
info: Retrieving plugin
info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb
info: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rb
info: Loading facts in /var/lib/puppet/facts/zone.rb
info: Loading facts in /var/lib/puppet/facts/read_facts.rb
info: Loading facts in /var/lib/puppet/facts/stomp_host.rb
info: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rb
info: Caching catalog for somehost.nbndc.local
err: Failed to apply catalog: You cannot specify more than one of
content, source, target at /usr/share/puppet-recipes/noop/puppet-
environments/puppet/modules/sbs/manifests/init.pp:73
notice: Class[Hostinfo]: Would have triggered 'refresh' from 1 events
notice: /Stage[main]/Java::Sun/Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]/
Java::Jdk::Default[1.6.0.31-sun]/Notify[java::jdk::default:sun]/message:
current_value absent, should be Setting java the following java as default:
1.6.0.31 sun 1.6.0 (noop)
notice: Java::Jdk::Default[1.6.0.31-sun]: Would have triggered 'refresh'
from 1 events
notice: Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]: Would have triggered
'refresh' from 1 events
notice: Class[Java::Sun]: Would have triggered 'refresh' from 1 events
notice: /Stage[main]/Greenmail/File[/etc/greenmail]/ensure: current_value
absent, should be directory (noop)
err: /Stage[main]/Greenmail/File[/etc/greenmail/greenmail.conf]: Could not
evaluate: Could not retrieve information from environment noop source(s)
puppet://puppet/modules/greenmail/etc/greenmail/greenmail.conf at /usr/
share/puppet-recipes/noop/puppet-environments/puppet/modules/greenmail/
manifests/init.pp:30
notice: Class[Puppetagent]: Would have triggered 'refresh' from 1 events
notice: /Stage[main]/Greenmail/File[/usr/greenmail]/ensure: current_value
directory, should be absent (noop)
notice: /Stage[main]/Greenmail/File[/usr/greenmail/startgm.sh]/ensure:
current_value file, should be absent (noop)
We were moving application deploys, why
not do the same with infrastructure?
Our MCollective agent
The new action
At every push, all nodes connect to the noop
     environment and run in noop mode
Picture of a report
Change
management
process
  ?
What changed?
What did you test?
Who will do it?
Part 4
New data centre is ready!
Should be easy to migrate....
Okay so our manifests are incomplete,
but we need to be able to rebuild machines...




        Virtualisation to the rescue...
Magnets?
What happens
Wrap Up
120 pushes to
production in the
last 8 months: 2
failures
MTTR FTW!



       5 minutes!
Thank you!

More Related Content

Similar to Sock puppets

Scalable Systems Management with Puppet
Scalable Systems Management with PuppetScalable Systems Management with Puppet
Scalable Systems Management with PuppetPuppet
 
Scalable systems management with puppet
Scalable systems management with puppetScalable systems management with puppet
Scalable systems management with puppetPuppet
 
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...Puppet
 
Testing your infallibleness
Testing your infalliblenessTesting your infallibleness
Testing your infalliblenessCorey Osman
 
Puppet Systems Infrastructure Construction Kit
Puppet Systems Infrastructure Construction KitPuppet Systems Infrastructure Construction Kit
Puppet Systems Infrastructure Construction KitAlessandro Franceschi
 
A Fabric/Puppet Build/Deploy System
A Fabric/Puppet Build/Deploy SystemA Fabric/Puppet Build/Deploy System
A Fabric/Puppet Build/Deploy Systemadrian_nye
 
CNS440 – Lab Assignment Week 2 Complete Lab 1 Perfo.docx
CNS440 – Lab Assignment Week 2   Complete Lab 1 Perfo.docxCNS440 – Lab Assignment Week 2   Complete Lab 1 Perfo.docx
CNS440 – Lab Assignment Week 2 Complete Lab 1 Perfo.docxmary772
 
INTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCE
INTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCEINTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCE
INTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCEIPutuAdiPratama
 
How to Make a Honeypot Stickier (SSH*)
How to Make a Honeypot Stickier (SSH*)How to Make a Honeypot Stickier (SSH*)
How to Make a Honeypot Stickier (SSH*)Jose Hernandez
 
K8s Quick Start
K8s Quick StartK8s Quick Start
K8s Quick StartGanesh Pol
 
Do you know all of Puppet?
Do you know all of Puppet?Do you know all of Puppet?
Do you know all of Puppet?Julien Pivotto
 
Puppet slides for intelligrape
Puppet slides for intelligrapePuppet slides for intelligrape
Puppet slides for intelligrapeSharad Aggarwal
 
From SaltStack to Puppet and beyond...
From SaltStack to Puppet and beyond...From SaltStack to Puppet and beyond...
From SaltStack to Puppet and beyond...Yury Bushmelev
 
Managing-Splunk-with-Puppet 31-January-2022.pdf
Managing-Splunk-with-Puppet 31-January-2022.pdfManaging-Splunk-with-Puppet 31-January-2022.pdf
Managing-Splunk-with-Puppet 31-January-2022.pdfssusera181ef
 
Bamboo Hands on training 2016
Bamboo Hands on training 2016Bamboo Hands on training 2016
Bamboo Hands on training 2016Takahiro Yamaki
 
Artem zhurbila the story of rebuilding puppet (devops meetup 29.10.2015)
Artem zhurbila  the story of rebuilding puppet (devops meetup 29.10.2015)Artem zhurbila  the story of rebuilding puppet (devops meetup 29.10.2015)
Artem zhurbila the story of rebuilding puppet (devops meetup 29.10.2015)Artem Zhurbila
 
PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...
PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...
PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...Puppet
 
Creating a Scheduled Backup and Replicating System Folders Introduct.docx
Creating a Scheduled Backup and Replicating System Folders Introduct.docxCreating a Scheduled Backup and Replicating System Folders Introduct.docx
Creating a Scheduled Backup and Replicating System Folders Introduct.docxwilliejgrant41084
 
When Good Code Goes Bad: Tools and Techniques for Troubleshooting Plone
When Good Code Goes Bad: Tools and Techniques for Troubleshooting PloneWhen Good Code Goes Bad: Tools and Techniques for Troubleshooting Plone
When Good Code Goes Bad: Tools and Techniques for Troubleshooting PloneDavid Glick
 

Similar to Sock puppets (20)

Scalable Systems Management with Puppet
Scalable Systems Management with PuppetScalable Systems Management with Puppet
Scalable Systems Management with Puppet
 
Scalable systems management with puppet
Scalable systems management with puppetScalable systems management with puppet
Scalable systems management with puppet
 
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...
 
Puppet quick start guide
Puppet quick start guidePuppet quick start guide
Puppet quick start guide
 
Testing your infallibleness
Testing your infalliblenessTesting your infallibleness
Testing your infallibleness
 
Puppet Systems Infrastructure Construction Kit
Puppet Systems Infrastructure Construction KitPuppet Systems Infrastructure Construction Kit
Puppet Systems Infrastructure Construction Kit
 
A Fabric/Puppet Build/Deploy System
A Fabric/Puppet Build/Deploy SystemA Fabric/Puppet Build/Deploy System
A Fabric/Puppet Build/Deploy System
 
CNS440 – Lab Assignment Week 2 Complete Lab 1 Perfo.docx
CNS440 – Lab Assignment Week 2   Complete Lab 1 Perfo.docxCNS440 – Lab Assignment Week 2   Complete Lab 1 Perfo.docx
CNS440 – Lab Assignment Week 2 Complete Lab 1 Perfo.docx
 
INTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCE
INTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCEINTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCE
INTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCE
 
How to Make a Honeypot Stickier (SSH*)
How to Make a Honeypot Stickier (SSH*)How to Make a Honeypot Stickier (SSH*)
How to Make a Honeypot Stickier (SSH*)
 
K8s Quick Start
K8s Quick StartK8s Quick Start
K8s Quick Start
 
Do you know all of Puppet?
Do you know all of Puppet?Do you know all of Puppet?
Do you know all of Puppet?
 
Puppet slides for intelligrape
Puppet slides for intelligrapePuppet slides for intelligrape
Puppet slides for intelligrape
 
From SaltStack to Puppet and beyond...
From SaltStack to Puppet and beyond...From SaltStack to Puppet and beyond...
From SaltStack to Puppet and beyond...
 
Managing-Splunk-with-Puppet 31-January-2022.pdf
Managing-Splunk-with-Puppet 31-January-2022.pdfManaging-Splunk-with-Puppet 31-January-2022.pdf
Managing-Splunk-with-Puppet 31-January-2022.pdf
 
Bamboo Hands on training 2016
Bamboo Hands on training 2016Bamboo Hands on training 2016
Bamboo Hands on training 2016
 
Artem zhurbila the story of rebuilding puppet (devops meetup 29.10.2015)
Artem zhurbila  the story of rebuilding puppet (devops meetup 29.10.2015)Artem zhurbila  the story of rebuilding puppet (devops meetup 29.10.2015)
Artem zhurbila the story of rebuilding puppet (devops meetup 29.10.2015)
 
PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...
PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...
PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...
 
Creating a Scheduled Backup and Replicating System Folders Introduct.docx
Creating a Scheduled Backup and Replicating System Folders Introduct.docxCreating a Scheduled Backup and Replicating System Folders Introduct.docx
Creating a Scheduled Backup and Replicating System Folders Introduct.docx
 
When Good Code Goes Bad: Tools and Techniques for Troubleshooting Plone
When Good Code Goes Bad: Tools and Techniques for Troubleshooting PloneWhen Good Code Goes Bad: Tools and Techniques for Troubleshooting Plone
When Good Code Goes Bad: Tools and Techniques for Troubleshooting Plone
 

Recently uploaded

A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?Igalia
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processorsdebabhi2
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Igalia
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slidevu2urc
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls
 
Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slidespraypatel2
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationMichael W. Hawkins
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityPrincipled Technologies
 
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CVKhem
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEarley Information Science
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsJoaquim Jorge
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024Rafal Los
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Servicegiselly40
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)Gabriella Davis
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUK Journal
 

Recently uploaded (20)

A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
 
Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slides
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
 

Sock puppets

  • 1. Sock Puppets: Growing your puppet codebase Fabio Lessa & Andrew Myers
  • 2.
  • 6.
  • 7. OPS team for app support
  • 8. The codebase we've started with Something that represents a mess
  • 10. Code was split by environments
  • 11.
  • 12. Workflow: 1. Push change 2. Wait for it to propagate to that environment 3. Check it worked If you need that applied to another environment, go to 1
  • 13. The real Workflow: 1. Push change 1.1. Stop the puppet daemon in all the critical nodes for that environment 1.2. Run puppetd manually in one host to ensure it works 1.3. Start the daemon again (on all the hosts you remember) 2. Wait for it to propagate to that environment 3. Check it worked (meh) If you need that applied to another environment, go to 1 (if you remember, that is)
  • 14.
  • 15. Production UAT System Testing CI
  • 18.
  • 19. "don't duplicate your manifests in your tests" Focus on the catalog problems he described earlier and test your logic. Don't test if puppet is doing it's job, test that your logic it's doing it's job. - Nikolay Sturm
  • 21. So how do we test/refactor
  • 24.
  • 26. Tying it all together
  • 29. We were moving application deploys, why not do the same with infrastructure?
  • 32. Picture of the pipelines
  • 33. info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/ puppet/modules/hostinfo/lib/facter/hostinfo.rb info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/ puppet/modules/concat/lib/facter/concat_basedir.rb info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb info: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rb info: Loading facts in /var/lib/puppet/facts/zone.rb info: Loading facts in /var/lib/puppet/facts/read_facts.rb info: Loading facts in /var/lib/puppet/facts/stomp_host.rb info: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rb info: Caching catalog for somehost.nbndc.local err: Failed to apply catalog: Could not find dependency File[/root/.hgrc] for Mercurial::Repo::Clone[proddc2-repo] at /usr/share/puppet-recipes/noop/puppet- environments/puppet/modules/puppetmaster/manifests/repos.pp:36
  • 34. info: Retrieving plugin info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb info: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rb info: Loading facts in /var/lib/puppet/facts/zone.rb info: Loading facts in /var/lib/puppet/facts/read_facts.rb info: Loading facts in /var/lib/puppet/facts/stomp_host.rb info: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rb info: Caching catalog for somehost.nbndc.local err: Failed to apply catalog: You cannot specify more than one of content, source, target at /usr/share/puppet-recipes/noop/puppet- environments/puppet/modules/sbs/manifests/init.pp:73
  • 35. notice: Class[Hostinfo]: Would have triggered 'refresh' from 1 events notice: /Stage[main]/Java::Sun/Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]/ Java::Jdk::Default[1.6.0.31-sun]/Notify[java::jdk::default:sun]/message: current_value absent, should be Setting java the following java as default: 1.6.0.31 sun 1.6.0 (noop) notice: Java::Jdk::Default[1.6.0.31-sun]: Would have triggered 'refresh' from 1 events notice: Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]: Would have triggered 'refresh' from 1 events notice: Class[Java::Sun]: Would have triggered 'refresh' from 1 events notice: /Stage[main]/Greenmail/File[/etc/greenmail]/ensure: current_value absent, should be directory (noop) err: /Stage[main]/Greenmail/File[/etc/greenmail/greenmail.conf]: Could not evaluate: Could not retrieve information from environment noop source(s) puppet://puppet/modules/greenmail/etc/greenmail/greenmail.conf at /usr/ share/puppet-recipes/noop/puppet-environments/puppet/modules/greenmail/ manifests/init.pp:30 notice: Class[Puppetagent]: Would have triggered 'refresh' from 1 events notice: /Stage[main]/Greenmail/File[/usr/greenmail]/ensure: current_value directory, should be absent (noop) notice: /Stage[main]/Greenmail/File[/usr/greenmail/startgm.sh]/ensure: current_value file, should be absent (noop)
  • 36. We were moving application deploys, why not do the same with infrastructure?
  • 37.
  • 40. At every push, all nodes connect to the noop environment and run in noop mode
  • 41.
  • 42. Picture of a report
  • 45. What did you test?
  • 46. Who will do it?
  • 48. New data centre is ready!
  • 49. Should be easy to migrate....
  • 50. Okay so our manifests are incomplete, but we need to be able to rebuild machines... Virtualisation to the rescue...
  • 54. 120 pushes to production in the last 8 months: 2 failures
  • 55.
  • 56. MTTR FTW! 5 minutes!