Iteratively introducing Puppet technologies in the brownfield; Jeffrey Miller
ORNL is managed by UT-Battelle, LLC
for the US Department of Energy
Iteratively introducing Puppet
technologies in the brownfield
HPC Linux Systems Engineer
HPC Core Ops Group
National Center for Computational Sciences
22 Open slide master to edit
Citation and Disclaimer
This work was supported by the Oak Ridge Leadership
Computing Facility (OLCF) and the Computer and Data
Environment for Science (CADES) at Oak Ridge National
Laboratory (ORNL) for the Department of Energy (DOE)
under Prime Contract Number DE-AC05-00OR-22725
This presentation does not contain any proprietary or
33 Open slide master to edit
Greg Shutt, CADES Task Lead
Cory Stargel, HPC Infrastructure Task Lead
Larry Orcutt, HPC Linux Systems Engineer
Michael Shute, HPC Linux Systems Engineer
James “Jake” Wynne, III, HPC Linux Systems Engineer
44 Open slide master to edit
55 Open slide master to edit
What We Do:
Infrastructure Team – HPC Core Ops Group
As part of the National Center for Computational
Sciences (NCCS), the HPC Core Ops group provides all
the necessary infrastructure services, networking
support, security oversight, and monitoring analytics
required to keep the OLCF leadership supercomputing
The Infrastructure Team provides necessary external
services for use by the OLCF HPC resources as well as
other programs and projects supported by NCCS.
CADES provides a compute and data infrastructure
environment to enable the scientific discovery process
for researchers at ORNL and their collaborators.
66 Open slide master to edit
Wouldn’t this be great?
From Deer Standing, by Petr Kratochvil. Retrieved from
77 Open slide master to edit
• Preexisting environment
providing production services
• Inventory? What inventory?
• Conglomeration of
• Disaster recovery? Backups?
From Dry Agricultural Brown Soil, by George Hodan.
88 Open slide master to edit
Don’t Touch Anything
From Fire in Dumpster, by Ben Watts, 2009. Retrieved from
99 Open slide master to edit
Where to start???
Retrieved from https://www.reddit.com/r/factorio/comments/982cw3/spaghetti_spaghetti/
1010 Open slide master to edit
Bolt – What is this?
“An open source orchestration
tool that automates the manual
work it takes to maintain your
- Works against local or remote
- Run scripts or commands
- Organized in tasks and plans
- Plans can be written in
Puppet or yaml
- Connects to remote targets
over SSH or WinRM
1111 Open slide master to edit
Bolt – What can you do?
You probably have a desktop system and/or a
• Install bolt and start writing a “laptop_config” plan
– Install git and other tools
– Run .dotfiles setup script
• Keep your code in git and commit often
• Start a habit of automate first
• Read Ben Ford’s April 2, 2020 blog post :
1212 Open slide master to edit
Automate All the Repos
Consider setting up a GitLab instance using bolt and the
Vox Pupuli GitLab puppet module on a system if you
don’t have an instance already.
GitLab and GitLab runners can enable:
- Code review process
- Infrastructure code deployment to a management
- Puppet code repositories validation testing and
deployment to Puppet Servers
1313 Open slide master to edit
application that can deploy
both bare metal and virtual
systems” by Puppet
- PXE boot management
- Hypervisor deployment
- Automate the hand off to
“a tool for for building,
changing, and versioning
infrastructure safely and
efficiently” by HashiCorp
- Infrastructure as Code
- Terraform creates the VM and
1414 Open slide master to edit
Puppet Agent and Facter
• Facter: Puppet’s system profiling library that is included
with the Puppet Agent package
– Bolt leverages facter to retrieve node facts
– But, for Bolt to use facter, the Puppet agent needs to be
• Enter apply_prep
– Built in Bolt function like run_command, run_script, etc.
– Installs the Puppet agent package if it isn’t already installed
– Collects facts from the node into the running inventory
– This can be an expensive operation…
1515 Open slide master to edit
• Using Bolt:
– Install the Puppet Agent on several new VMs
– Install and configure a PuppetCA (and optionally catalog
– Deploy PuppetDB and backend PostgreSQL database
– Reconfigure Puppet Server systems to use PuppetDB
• Using Puppet or Bolt
– Deploy Puppetboard (or alternative) dashboard to PuppetDB
1616 Open slide master to edit
Puppet Agent Rollout
• Using Bolt:
– Install the Puppet agent on each system (hardware or VM)
– Configure the Puppet agent to register with the Puppet Server
– Then, (this is key) have the Puppet agent configure absolutely
Yes… Absolutely nothing. Null. Zero. Zilch.
What you have now is a growing inventory that furthers
systems and services discovery.
1717 Open slide master to edit
--noop or noop()
When Puppet is run locally on
the system, a “--noop” flag may
be passed to report what
would change but not actually
Similar result with the “noop”
parameter set puppet.conf.
Function in the trlinkin-noop
module that sets a scope to
For examples of this working see: Puppet noop, no-noop, and the path
to safe Puppet Deployments, Alessandro Franceschi, 2017.
1818 Open slide master to edit
Systems Configuration – Iterative Style
• If you haven’t already, `git init`
• What do you want to configure today?
• Leverage a git workflow for puppet code use
– Create branch
– Add puppet code
– Deploy branch to Puppet Servers
• Use bolt to test canary systems using noop
• Find edge cases on the other nodes using noop
• Merge and enforce (i.e. no-noop)
1919 Open slide master to edit
- Dedicate Puppet Servers to serve out test code
- Test VMs for types of nodes you support in production
- Consider using a unique environment for testing
instead of a branch against production
- Code dev -> squash -> merge -> cherry pick
2020 Open slide master to edit
Even better better…
Multi-node acceptance tests for fun and profit. Trevor
2121 Open slide master to edit
Implementation takes a willing team.
Learn to trust the process and the tooling.
Trust, but verify.
Be flexible. There will be landmines in the brownfield.
Focus on what can be done rather than shortcomings.
Patience – the iterative process may take a long time…
Notice I didn’t mention Ruby skillz… oops. #facepalm
The Puppet Community.