Download to read offline
![exec { 'install-redis':
command => "make && make install PREFIX=${redis_bin_dir}",
cwd => $redis_src_dir,
path => '/bin:/usr/bin',
unless => "test $(${redis_bin_dir}/bin/redis-server --version | cut -d
' ' -f 1) = ‘Redis'",
require => [ Exec['unpack-redis'], Class['gcc'] ],
}
Common (not great) solution](https://image.slidesharecdn.com/puppetconf-2015-151009210509-lva1-app6891/75/Package-manages-and-Puppet-PuppetConf-2015-53-2048.jpg)
![# Base Directory shortcut
$basedir = '/var/lib/apt/repo'
!
# Main reprepro class
class { 'reprepro':
basedir => $basedir,
}
!
# Set up a repository
reprepro::repository { 'localpkgs':
ensure => present,
basedir => $basedir,
options => ['basedir .'],
}
reprepro via puppet](https://image.slidesharecdn.com/puppetconf-2015-151009210509-lva1-app6891/75/Package-manages-and-Puppet-PuppetConf-2015-93-2048.jpg)
More Related Content
Similar to Package manages and Puppet - PuppetConf 2015
Package manages and Puppet - PuppetConf 2015
- 1. Package Managers and Puppet JoeDamato packagecloud.io
- 2.
- 3. hi, i’m joe ilike computers i once had a blog called timetobleed.com @joedamato
- 4.
- 5.
- 6.
- 7. Why? • Central tomaintaining, building, and testing infrastructure. • Packages are a primitive in Puppet. • Understanding where packages come from, and how to store them properly is a requirement for infrastructure of any size. • Packages and packaging are much trickier than they seem!
- 8. Overview • what isa package? • what is a package manager? • ./configure && make && make install pattern • open source tools for package repositories • HOWTO manage repos in your infra with puppet
- 9. What is apackage? Beck Gusler, https://flic.kr/p/4A15jm
- 13. Flavors • packages comein many flavors • 2 relatively important flavors are…
- 14.
- 16. RPM Internals • Usedon CentOS, RHEL, Scientific Linux, Fedora, … • files typically have the “.rpm” file extension • can be inspected, installed, and removed with rpm • are actually a: • header structure (binary data) • CPIO archive
- 17. sounds simple but itbarely works
- 18. • librpm CPIOreader lolz • GPG signing/verifying RPMs story time
- 19. gpg v3 signatures %__gpg_sign_cmd%{__gpg} gpg --force-v3-sigs --digest- algo=sha1 --batch --no-verbose --no- armor --passphrase-fd 3 --no-secmem- warning -u "%{_gpg_name}" -sbo % {__signature_filename} % {__plaintext_filename}
- 20.
- 21. man 8 rpm formore lies info:
- 23. deb internals • Debpackages: • Used on Ubuntu, Debian, Knoppix, … • files typically have the “.deb” file extension • can be inspected, installed, and removed with dpkg
- 24. deb internals • Debpackages: • are actually an AR archive with: • version file: the debian format version • data.tar.gz: the actual files to write to the filesystem • control.tar.gz: the package metadata
- 25. sounds simple but itbarely works
- 26. • debsigs vsdpkg-sig • gpg signing is pointless • XML policy documents story time
- 27. /etc/debsig/policies/ DDDF2F4CE732A79A/hi.pol <?xml version="1.0"?> <!DOCTYPE PolicySYSTEM "http://www.debian.org/debsig/1.0/policy.dtd"> <Policy xmlns="http://www.debian.org/debsig/1.0/"> ! <Origin Name="test" id="DDDF2F4CE732A79A" Description="Test package"/> ! <Selection> <Required Type="origin" File="debsig.gpg" id="DDDF2F4CE732A79A"/> </Selection> ! <Verification MinOptional="0"> <Required Type="origin" File="debsig.gpg" id="DDDF2F4CE732A79A"/> </Verification> </Policy>
- 28.
- 29. man 1 dpkg formore lies info:
- 30.
- 31. Lots of flavors •There are lots more! (ruby gems, npm, java, python, …) • Some packaging systems also have source packages.
- 32. What is asource package? • A source package consists of: • metadata (version, architecture(s), build deps, etc). • source files (C source, C++ source, py scripts, etc). • Allows you to rebuild a binary package easily.
- 33. Install packages withpuppet Use the resource type ‘package’: package { 'pygpgme': ensure => latest, }
- 34. Install packages withpuppet package { 'pygpgme': ensure => ‘0.3-11’, } Specify the version you want:
- 35. Summary • Packages area collection of files with metadata. • The metadata usually has info like: • architecture • version • dependency info • and more. • Installation is easy if you don’t have dependencies.
- 36.
- 37. Dependencies • Installing 1package is as easy as: • dpkg -i filename.deb • rpm -ivh filename.rpm • Of course, you should use puppet instead :D • But what if your program needs other programs? • For example: nginx depends on libssl, zlib, …
- 38.
- 39. So, what’s apackage manager?
- 40. Package manager • Apackage manager is a collection of software that allows you to: • install, upgrade, remove packages • query package info from local system or repos • Some tools include more advanced features like mirroring or more advanced caching features.
- 41.
- 42. • yum (YellowdogUpdater, Modified) • Common on RHEL, CentOS, Fedora, … • Used for installing, removing, configuring, and querying RPM packages and dependencies. Common package managers
- 43.
- 44. Common package managers •APT (Advanced Package Tool) • Common on Debian, Ubuntu, KNOPPIX, … • Used for installing, removing, configuring, and querying Debian packages and dependencies.
- 45. Install packages withpuppet • puppet automatically detects which package manager to use. • Don’t need to worry about which command to run, or what options to pass; puppet will take care of that for you!
- 46. Summary • package managershelp you install software and associated dependencies • easily remove, upgrade, and query packages • Puppet will automatically detect the system’s package manager when you install a package.
- 47.
- 48.
- 49. • https • moreGPG madness story time
- 50.
- 51. A problem • Yourun Ubuntu 10.04 LTS • You want to install redis • Ubuntu 10.04 comes with redis-server 1.2.0-1 • That’s too old! You need 2.8.19! • So, now what?
- 52. Common (not great)solution • A common solution to this sort of problem is building redis (or ruby, or …) from source in your puppet manifest • Like this….
- 53. exec { 'install-redis': command=> "make && make install PREFIX=${redis_bin_dir}", cwd => $redis_src_dir, path => '/bin:/usr/bin', unless => "test $(${redis_bin_dir}/bin/redis-server --version | cut -d ' ' -f 1) = ‘Redis'", require => [ Exec['unpack-redis'], Class['gcc'] ], } Common (not great) solution
- 54. Why? • It’s easy! •./configure && make && make install • It works! • I’m using puppet so it’s reproducible!
- 55. But… • What happensif you need to: • completely remove Redis? • install a security update? • install a new version? • install the same exact Redis on 200 machines?
- 56. The not-so greatside • Not all Makefiles have uninstall targets, so you have to remove files manually • Leaving artifacts on the filesystem can cause really, really hard to debug problems later • If the build process changes version to version, it can be painful to rollback
- 57. The not-so greatside • Rebuilding the same source does not necessarily get you the same byte-for-byte binary • If the binaries aren’t identical, you can end up with bugs in some of the compiled binaries but not others • Painful to recreate source builds inside of puppet • Makes writing tests for manifests painful
- 58. Make a package •Install the same binary on every machine • When the package is removed, all installed files are removed • Versioning of build process built in (with most tools) • Keep your puppet manifests about config management • Your build steps are “factored out” into the package
- 59. Your new puppetmanifest package { 'redis': ensure => latest, }
- 60. Your package • Yourbuild steps get encapsulated in the package itself • Makes iterating on the build more straight forward • Don’t need to apply (potentially) a bunch of manifests to a machine every time you do a build
- 61.
- 62. “How do Imake a package?”
- 63.
- 64. Use tools! • debbuild •rpmbuild • fpm • mock and pbuilder (more advanced)
- 65. Tradeoffs • Takes timeto learn new tools • Takes time to understand packaging • No one ever has enough time
- 66.
- 67. Tradeoffs • Once youlearn how to make packages you can build reproducible infrastructure much more easily • You can use your prod environment in dev and test • You can more easily build tests for your infrastructure with beaker/kitchen.ci
- 68.
- 69. “How do Istore and organize my packages?”
- 70. Package repositories • Majorlinux distributions keep repositories of packages for users: • EPEL • Ubuntu / Debian official repositories • You can store a package and its dependencies to make it easy to install them all on your infrastructure
- 71.
- 72. Package repositories • createrepo:creates yum repositories • reprepro: creates apt repositories • Many other free tools available! • Read the documentation carefully. Lots of tricky options. • I’ll show some examples to get you started!
- 73.
- 74. createrepo • mkdir /var/www/myrepo •cp /path/to/rpms/*.rpm /var/www/myrepo • createrepo /var/www/myrepo • gpg --detach-sign --armor /var/www/my/repo/repomd.xml
- 75. GPG is important •Using GPG to sign the generated repository guarantees that you generated the repository. • This is important. • This means that no one else modified, removed, or inserted a package other than you. • GPG signing the repository is not a very well known security measure, but it is incredibly important! • This is NOT the same as using rpmsign/rpm --sign.
- 76. Secure YUM repos •Sign repository metadata with GPG • Sign packages with GPG (use rpmsign) • Serve repositories over SSL • Enable all the right options for SSL verification, repository GPG checking, AND package GPG checking.
- 77. Wouldn’t it becool to do all that with Puppet instead? Good news: you can!
- 78. createrepo via puppet Puppetcan create YUM repositories for you! $ puppet module install palli-createrepo
- 79. createrepo { 'yumrepo': repository_dir=> '/var/yumrepos/yumrepo', repo_cache_dir => '/var/cache/yumrepos/yumrepo' } createrepo via puppet
- 80. You still needto GPG sign the repository yourself :( exec { “gpg_sign_yumrepo”: command => “gpg --detach-sign --armor /var/yumrepos/yumrepo/repodata/repomd.xml“, }
- 81. Once the repositoryis created, it must be added to the client machines.
- 82. Add YUM reposwith puppet yumrepo { 'my_repo': baseurl => "http://myurl.com/repo", gpgcheck => 1, repo_gpgcheck => 1, gpgkey => “http://myurl.com/gpg.pub.key”, sslverify => 1, sslcacert => “/etc/pki/tls/certs/ca-bundle.crt”, enabled => 1, } most people never turn on repo_gpgcheck or sslverify, or set the ssl certificate path, but you should!!
- 83. But that’s notall! • You MUST have the ‘pygpgme’ package installed on the system that will verify the signatures. • Without pygpgme, yum will not be able to verify signatures! • Some versions of CentOS / RHEL do not automatically install pygpgme with yum!!
- 84. Make sure toinstall pygpgme package { 'pygpgme': ensure => latest, }
- 85.
- 86.
- 87. reprepro • mkdir /var/www/myrepo •mkdir /var/www/myrepo/conf • Create a file named “distributions” in the conf directory
- 88. reprepro Codename: precise Components: main Architectures:i386 amd64 SignWith: 7ABDB001 /var/www/myrepo/conf/distributions:
- 89. reprepro • You canadd more sections if you need more code names (lucid, trusty, etc). • SignWith specifies which GPG key to use for signing repository metadata • You can get your gpg key ID by looking at the output of gpg —list-keys • This is not the same as using debsigs/debsign !!!
- 90. reprepro import your Ubuntu12.04 packages: reprepro -b /var/www/myrepo/ includedeb precise filename.deb
- 91. Wouldn’t it becool to do all that with Puppet instead? Good news: you can!
- 92. reprepro via puppet Puppetcan create APT repositories for you! $ puppet module install jtopjian-reprepro
- 93. # Base Directoryshortcut $basedir = '/var/lib/apt/repo' ! # Main reprepro class class { 'reprepro': basedir => $basedir, } ! # Set up a repository reprepro::repository { 'localpkgs': ensure => present, basedir => $basedir, options => ['basedir .'], } reprepro via puppet
- 94. # Create adistribution within that repository reprepro::distribution { 'precise': basedir => $basedir, repository => 'localpkgs', origin => 'Foobar', label => 'Foobar', suite => 'precise', architectures => 'amd64 i386', components => 'main contrib non-free', description => ‘My repo', sign_with => 'F4D5DAA8', not_automatic => 'No', } reprepro via puppet
- 95. Add APT reposwith puppet apt::source { 'myrepo': location => ‘http://myurl.com/repo', release => 'precise', repos => 'main', key => '7ABDB001', key_source => ‘http://myurl.com/gpg.pub.key', include_src => true, } $ puppet module install puppetlabs-apt
- 96. But that’s notall! • You MUST have the ‘apt-transport-https’ package installed on the system if your repository is served over HTTPS! • Without apt-transport-https, you can’t install packages over HTTPS. • You definitely want this.
- 97. Make sure toinstall apt-transport-https package { ‘apt-transport-https‘: ensure => latest, }
- 98.
- 99.
- 100. Success • You cannow use beaker/kitchen.ci/etc to test your infrastructure. • Determine if the packages you need are actually installed after your manifests are applied. • Determine if the repositories you added are actually added after your manifests are applied. • Don’t need to wait forever for Ruby, redis, et al to build during a test run.
- 101. BEST OF ALL!!!! • You can now run Puppet on your development VM using the same manifests you use in production • The manifests are applied and you are running the same exact binaries you run in production • Won’t catch ALL production bugs, but getting closer to production during development is super useful
- 102. Summary • Creating packagerepositories can be tricky. Make sure to GPG sign repository metadata. • 99% of package repositories get this wrong. • Carefully read the documentation of createrepo and reprepro. • Make sure to install necessary libraries for verifying signatures and accessing repositories via HTTPS. • Always serve up your repositories over HTTPS.
- 103. Use puppet toautomate this.
- 104.