- Puppet is used to manage 43 physical servers and 97 EC2 instances at Thumbtack, with roughly half the EC2 instances used for staging and research. - A custom AMI and shell script were created to automate server provisioning and distribute configuration in a standardized way across environments. - A development workflow was established using additional Puppet masters so developers can test changes locally or on staging instances before pushing to the main Puppet master.

1. Server Management with
Puppet on AWS
Marco Almeida - Site Reliability Engineer

2. About us
Thumbtack helps you
accomplish the personal
projects that are central to
your life.
Whether you need to paint
your home, learn a new
language, or plan your
daughter's birthday party,
Thumbtack is the easiest
and most dependable way
to hire the right
professional for your
projects.

3. Our infrastructure

4. Our infrastructure
● 43 physical servers
● 97 EC2 instances
○ roughly half for staging/research purposes
● Everything managed with Puppet

5. Scaling the infrastructure
● Goals:
○ completely automate server provisioning
○ proper development environment for the
whole team
○ good way of distributing sensitive
information
● Two components:
○ Custom AMI + a shell script
○ Puppet

6. Scaling the infrastructure
● Debian 7 (Wheezy) or 8 (Jessie)
● Simple, i.e., minimal base system
○ include the contrib and non-free areas
○ don't install suggested packages by default
The custom AMI

7. Scaling the infrastructure
● Set the hostname
● Update DNS (forward and reverse)
● Run the Puppet agent
● Notify Slack
● Uninstall itself
● Reboot
● Log everything
The shell script

8. Configuration Management
Bootstrapping servers
1. Generate new SSL key and request a
new certificate
2. Run policy-based autosign script
3. Compile catalog
4. Download catalog
5. Apply the catalog
1. Basic checks: certname was provided, no
certificate with the same name has already been
signed, etc.
2. Verify that an instance with the exact same
name exists
3. ...
1
puppet.internal
bender.internal
2, 34
5

9. Configuration Management
Development environment
In the old days...
● Git repo
● Single puppet master
● No staging environment

10. Configuration Management
Development environment
In the old days...
● Write code
● Commit
● Push
● Update master
● Dry run on some node
Syntax error at 'FOO'; expected '}' at BAR

11. Configuration Management
Development environment
In the old days...
● Write code
● Commit
● Push
● Update master
● Dry run on some node
Could not find class FOO

12. Configuration Management
Development environment
In the old days...
● Write code
● Commit
● Push
● Update master
● Dry run on some node
Duplicate declaration: Package[FOO] is
already declared

13. Configuration Management
Development environment
In the old days...
● Write code
● Commit
● Push
● Update master
● Dry run on some node
Could not apply complete catalog: Found 1
dependency cycle

14. Configuration Management
Development environment
puppet.internal
bender.internalGit
fry.internal
.
.
.
dev-1.internal
dev-2.internal
dev-3.internal
Puppet masters

15. Configuration Management
Development environment
● A few extra configuration options
○ ca = false
○ dns_alt_names = dev-1, dev-1.internal,
puppet, puppet.internal
● Test changes locally or from a staging box
○ puppetd --test --server dev-1.internal

16. Configuration Management
Development environment
● Write code
● Test changes
● Commit
● Code review
● Push
● Update master
● Let the agent do its thing

17. Configuration Management
Sensitive data
● Problem
○ Licence keys
○ Passwords
○ Anything that shouldn’t be on Github
● The solution we want
○ Easy to use with Puppet code, e.g.,
templates
○ Shouldn’t require another password
○ Simple and easy to understand

18. Configuration Management
Sensitive data
● Easy to use
○ Variables
● Don’t require another password
○ IAM role
● Simple and easy to understand
○ S3 or DynamoDB

19. Configuration Management
Sensitive data
● Store the data on an S3 Bucket (or DynamoDB)
● Create an IAM role with read-only permissions
● Assign the role to all puppet master instances
● Use an ENC
○ On every run Puppet gets all the sensitive
data relevant to that node
○ Data is made available through top-scope
variables

20. Configuration Management
Sensitive data
class { 'datadog':
stage => monitoring,
api_key => $::datadog_api_key,
}
production:
host: puppetdashboard.foo.rds.amazonaws.com
database: puppetdashboard
password: <%= @puppetdashboard_db_password %>

21. Configuration Management
Speed things up
● Re-deploy the provisioning script
● Create an AMI

22. Configuration Management
Conclusion
● Each developer has his/hers own puppet
master
● Changes can be easily tested locally or on
staging instances
○ just point the agent to your puppet master
● Development can happen in parallel
● No need to babysit agent runs
● All standard tools, didn’t reinvent the wheel

23. Questions?