This talk will begin by explaining what a package manager is and how package managers work, at a high level. Next, we'll observe the common patterns seen on the internet of compiling software in a Puppet manifest and discuss why this not ideal. This talk will conclude by showing how you can add package repositories to your infrastructure using Puppet and what settings are important for ensuring secure access to remote package repositories.

1. Package Managers
and Puppet
Joe Damato
packagecloud.io

2. slides available at:
blog.packagecloud.io

3. hi, i’m joe
i like computers
i once had a blog
called timetobleed.com
@joedamato

4. packagecloud.io
@packagecloudio

5. marc falardeau, https://flic.kr/p/8gKeGS

6. Wade M, https://flic.kr/p/5aghr9

7. Why?
• Central to maintaining, building, and testing
infrastructure.
• Packages are a primitive in Puppet.
• Understanding where packages come from, and how
to store them properly is a requirement for
infrastructure of any size.
• Packages and packaging are much trickier than they
seem!

8. Overview
• what is a package?
• what is a package manager?
• ./configure && make && make install pattern
• open source tools for package repositories
• HOWTO manage repos in your infra with puppet

9. What is a package?
Beck Gusler, https://flic.kr/p/4A15jm

13. Flavors
• packages come in many flavors
• 2 relatively important flavors
are…

14. RPM deb

16. RPM Internals
• Used on CentOS, RHEL, Scientific Linux, Fedora,
…
• files typically have the “.rpm” file extension
• can be inspected, installed, and removed with rpm
• are actually a:
• header structure (binary data)
• CPIO archive

17. sounds simple
but it barely works

18. • librpm CPIO reader lolz
• GPG signing/verifying RPMs
story time

19. gpg v3 signatures
%__gpg_sign_cmd %{__gpg}
gpg --force-v3-sigs --digest-
algo=sha1 --batch --no-verbose --no-
armor --passphrase-fd 3 --no-secmem-
warning -u "%{_gpg_name}" -sbo %
{__signature_filename} %
{__plaintext_filename}

20. (hi)

21. man 8 rpm
for more lies info:

23. deb internals
• Deb packages:
• Used on Ubuntu, Debian, Knoppix, …
• files typically have the “.deb” file extension
• can be inspected, installed, and removed with
dpkg

24. deb internals
• Deb packages:
• are actually an AR archive with:
• version file: the debian format version
• data.tar.gz: the actual files to write to the
filesystem
• control.tar.gz: the package metadata

25. sounds simple
but it barely works

26. • debsigs vs dpkg-sig
• gpg signing is pointless
• XML policy documents
story time

27. /etc/debsig/policies/
DDDF2F4CE732A79A/hi.pol
<?xml version="1.0"?>
<!DOCTYPE Policy SYSTEM "http://www.debian.org/debsig/1.0/policy.dtd">
<Policy xmlns="http://www.debian.org/debsig/1.0/">
!
<Origin Name="test" id="DDDF2F4CE732A79A" Description="Test package"/>
!
<Selection>
<Required Type="origin" File="debsig.gpg" id="DDDF2F4CE732A79A"/>
</Selection>
!
<Verification MinOptional="0">
<Required Type="origin" File="debsig.gpg" id="DDDF2F4CE732A79A"/>
</Verification>
</Policy>

28. (hi)

29. man 1 dpkg
for more lies info:

30. anw

31. Lots of flavors
• There are lots more! (ruby gems, npm, java,
python, …)
• Some packaging systems also have source
packages.

32. What is a source package?
• A source package consists of:
• metadata (version, architecture(s), build deps,
etc).
• source files (C source, C++ source, py scripts,
etc).
• Allows you to rebuild a binary package easily.

33. Install packages with puppet
Use the resource type ‘package’:
package { 'pygpgme':
ensure => latest,
}

34. Install packages with puppet
package { 'pygpgme':
ensure => ‘0.3-11’,
}
Specify the version you want:

35. Summary
• Packages are a collection of files with metadata.
• The metadata usually has info like:
• architecture
• version
• dependency info
• and more.
• Installation is easy if you don’t have dependencies.

36. Dependencies
Nick Sieger, https://flic.kr/p/qQu1e

37. Dependencies
• Installing 1 package is as easy as:
• dpkg -i filename.deb
• rpm -ivh filename.rpm
• Of course, you should use puppet instead :D
• But what if your program needs other programs?
• For example: nginx depends on libssl, zlib, …

38. r-hol, https://flic.kr/p/6UZb98

39. So, what’s a package
manager?

40. Package manager
• A package manager is a collection of software
that allows you to:
• install, upgrade, remove packages
• query package info from local system or repos
• Some tools include more advanced features like
mirroring or more advanced caching features.

41. Common package managers
http://en.wikipedia.org/wiki/
Yellowdog_Updater,_Modified#mediaviewer/File:Yum.png

42. • yum (Yellowdog Updater, Modified)
• Common on RHEL, CentOS, Fedora, …
• Used for installing, removing, configuring, and
querying RPM packages and dependencies.
Common package managers

43. Common package managers
APT

44. Common package managers
• APT (Advanced Package Tool)
• Common on Debian, Ubuntu, KNOPPIX, …
• Used for installing, removing, configuring, and
querying Debian packages and dependencies.

45. Install packages with puppet
• puppet automatically detects which package
manager to use.
• Don’t need to worry about which command to
run, or what options to pass; puppet will take
care of that for you!

46. Summary
• package managers help you install software and
associated dependencies
• easily remove, upgrade, and query packages
• Puppet will automatically detect the system’s
package manager when you install a package.

47. almost forgot…

48. neither actually work

49. • https
• more GPG madness
story time

50. Kellie Parker, https://flic.kr/p/mtNMb

51. A problem
• You run Ubuntu 10.04 LTS
• You want to install redis
• Ubuntu 10.04 comes with redis-server 1.2.0-1
• That’s too old! You need 2.8.19!
• So, now what?

52. Common (not great) solution
• A common solution to this sort of problem is
building redis (or ruby, or …) from source in your
puppet manifest
• Like this….

53. exec { 'install-redis':
command => "make && make install PREFIX=${redis_bin_dir}",
cwd => $redis_src_dir,
path => '/bin:/usr/bin',
unless => "test $(${redis_bin_dir}/bin/redis-server --version | cut -d
' ' -f 1) = ‘Redis'",
require => [ Exec['unpack-redis'], Class['gcc'] ],
}
Common (not great) solution

54. Why?
• It’s easy!
• ./configure && make && make install
• It works!
• I’m using puppet so it’s reproducible!

55. But…
• What happens if you need to:
• completely remove Redis?
• install a security update?
• install a new version?
• install the same exact Redis on 200 machines?

56. The not-so great side
• Not all Makefiles have uninstall targets, so you
have to remove files manually
• Leaving artifacts on the filesystem can cause
really, really hard to debug problems later
• If the build process changes version to version,
it can be painful to rollback

57. The not-so great side
• Rebuilding the same source does not necessarily
get you the same byte-for-byte binary
• If the binaries aren’t identical, you can end up
with bugs in some of the compiled binaries but
not others
• Painful to recreate source builds inside of puppet
• Makes writing tests for manifests painful

58. Make a package
• Install the same binary on every machine
• When the package is removed, all installed files are
removed
• Versioning of build process built in (with most tools)
• Keep your puppet manifests about config
management
• Your build steps are “factored out” into the package

59. Your new puppet manifest
package { 'redis':
ensure => latest,
}

60. Your package
• Your build steps get encapsulated in the package
itself
• Makes iterating on the build more straight forward
• Don’t need to apply (potentially) a bunch of
manifests to a machine every time you do a build

61. Duncan Hull, https://flic.kr/p/iVLZt

62. “How do I make a package?”

63. OZinOH, https://flic.kr/p/bRHn2v

64. Use tools!
• debbuild
• rpmbuild
• fpm
• mock and pbuilder (more advanced)

65. Tradeoffs
• Takes time to learn new tools
• Takes time to understand packaging
• No one ever has enough time

66. BUT…

67. Tradeoffs
• Once you learn how to make packages you can
build reproducible infrastructure much more
easily
• You can use your prod environment in dev and
test
• You can more easily build tests for your
infrastructure with beaker/kitchen.ci

68. Duncan Hull, https://flic.kr/p/iVLZt

69. “How do I store and
organize my packages?”

70. Package repositories
• Major linux distributions keep repositories of
packages for users:
• EPEL
• Ubuntu / Debian official repositories
• You can store a package and its dependencies to
make it easy to install them all on your infrastructure

71. OZinOH, https://flic.kr/p/bRHn2v

72. Package repositories
• createrepo: creates yum repositories
• reprepro: creates apt repositories
• Many other free tools available!
• Read the documentation carefully. Lots of tricky
options.
• I’ll show some examples to get you started!

73. createrepo
http://en.wikipedia.org/wiki/
Yellowdog_Updater,_Modified#mediaviewer/File:Yum.png

74. createrepo
• mkdir /var/www/myrepo
• cp /path/to/rpms/*.rpm /var/www/myrepo
• createrepo /var/www/myrepo
• gpg --detach-sign --armor /var/www/my/repo/repomd.xml

75. GPG is important
• Using GPG to sign the generated repository
guarantees that you generated the repository.
• This is important.
• This means that no one else modified, removed, or
inserted a package other than you.
• GPG signing the repository is not a very well known
security measure, but it is incredibly important!
• This is NOT the same as using rpmsign/rpm --sign.

76. Secure YUM repos
• Sign repository metadata with GPG
• Sign packages with GPG (use rpmsign)
• Serve repositories over SSL
• Enable all the right options for SSL verification,
repository GPG checking, AND package GPG
checking.

77. Wouldn’t it be cool to do all
that with Puppet instead?
Good news: you can!

78. createrepo via puppet
Puppet can create YUM repositories for you!
$ puppet module install palli-createrepo

79. createrepo { 'yumrepo':
repository_dir => '/var/yumrepos/yumrepo',
repo_cache_dir => '/var/cache/yumrepos/yumrepo'
}
createrepo via puppet

80. You still need to GPG sign
the repository yourself :(
exec { “gpg_sign_yumrepo”:
command => “gpg --detach-sign --armor
/var/yumrepos/yumrepo/repodata/repomd.xml“,
}

81. Once the repository is created, it must
be added to the client machines.

82. Add YUM repos with puppet
yumrepo { 'my_repo':
baseurl => "http://myurl.com/repo",
gpgcheck => 1,
repo_gpgcheck => 1,
gpgkey => “http://myurl.com/gpg.pub.key”,
sslverify => 1,
sslcacert => “/etc/pki/tls/certs/ca-bundle.crt”,
enabled => 1,
}
most people never turn on repo_gpgcheck or
sslverify, or set the ssl certificate path, but you
should!!

83. But that’s not all!
• You MUST have the ‘pygpgme’ package
installed on the system that will verify the
signatures.
• Without pygpgme, yum will not be able to verify
signatures!
• Some versions of CentOS / RHEL do not
automatically install pygpgme with yum!!

84. Make sure to install pygpgme
package { 'pygpgme':
ensure => latest,
}

85. OR
just use packagecloud.io
instead

86. reprepro
APT

87. reprepro
• mkdir /var/www/myrepo
• mkdir /var/www/myrepo/conf
• Create a file named “distributions” in the conf
directory

88. reprepro
Codename: precise
Components: main
Architectures: i386 amd64
SignWith: 7ABDB001
/var/www/myrepo/conf/distributions:

89. reprepro
• You can add more sections if you need more code
names (lucid, trusty, etc).
• SignWith specifies which GPG key to use for signing
repository metadata
• You can get your gpg key ID by looking at the output
of gpg —list-keys
• This is not the same as using debsigs/debsign !!!

90. reprepro
import your Ubuntu 12.04 packages:
reprepro -b /var/www/myrepo/ includedeb precise filename.deb

91. Wouldn’t it be cool to do all
that with Puppet instead?
Good news: you can!

92. reprepro via puppet
Puppet can create APT repositories for you!
$ puppet module install jtopjian-reprepro

93. # Base Directory shortcut
$basedir = '/var/lib/apt/repo'
!
# Main reprepro class
class { 'reprepro':
basedir => $basedir,
}
!
# Set up a repository
reprepro::repository { 'localpkgs':
ensure => present,
basedir => $basedir,
options => ['basedir .'],
}
reprepro via puppet

94. # Create a distribution within that repository
reprepro::distribution { 'precise':
basedir => $basedir,
repository => 'localpkgs',
origin => 'Foobar',
label => 'Foobar',
suite => 'precise',
architectures => 'amd64 i386',
components => 'main contrib non-free',
description => ‘My repo',
sign_with => 'F4D5DAA8',
not_automatic => 'No',
}
reprepro via puppet

95. Add APT repos with puppet
apt::source { 'myrepo':
location => ‘http://myurl.com/repo',
release => 'precise',
repos => 'main',
key => '7ABDB001',
key_source => ‘http://myurl.com/gpg.pub.key',
include_src => true,
}
$ puppet module install puppetlabs-apt

96. But that’s not all!
• You MUST have the ‘apt-transport-https’ package
installed on the system if your repository is served
over HTTPS!
• Without apt-transport-https, you can’t install
packages over HTTPS.
• You definitely want this.

97. Make sure to install apt-transport-https
package { ‘apt-transport-https‘:
ensure => latest,
}

98. OR
just use packagecloud.io
instead

99. Alosh Bennett, https://flic.kr/p/WJ7rE

100. Success
• You can now use beaker/kitchen.ci/etc to test your
infrastructure.
• Determine if the packages you need are actually
installed after your manifests are applied.
• Determine if the repositories you added are
actually added after your manifests are applied.
• Don’t need to wait forever for Ruby, redis, et al to
build during a test run.

101. BEST OF ALL !!!!
• You can now run Puppet on your development
VM using the same manifests you use in
production
• The manifests are applied and you are running
the same exact binaries you run in production
• Won’t catch ALL production bugs, but getting
closer to production during development is
super useful

102. Summary
• Creating package repositories can be tricky. Make
sure to GPG sign repository metadata.
• 99% of package repositories get this wrong.
• Carefully read the documentation of createrepo and
reprepro.
• Make sure to install necessary libraries for verifying
signatures and accessing repositories via HTTPS.
• Always serve up your repositories over HTTPS.

103. Use puppet to automate this.

104. ?@packagecloudio
https://packagecloud.io
joe@packagecloud.io