This document discusses using Pulp to manage software patch repositories and automate Linux patch management. Currently, patches are downloaded individually to each system, requiring logging into each one. Pulp would allow creating different repositories for live, development, and production environments, and automating the promotion of patches from one environment to the next after testing. It describes using Pulp to synchronize repositories from upstream, create repository snapshots to pin package versions, and rollback changes if needed. Clients would simply point to the appropriate repository rather than each managing its own patches. Salt could be used to remotely install patches and restart services on schedule.

1. http://slides.com/roblyon/wsu-system-http://slides.com/roblyon/wsu-system-
updates-with-pulp/liveupdates-with-pulp/live

2. Linux PatchLinux Patch
ManagementManagement

3. Current ModelCurrent Model
Patches are downloaded to each system every second
Tuesday and a local repo is created.
When patch time has arrived, all repos except the
local repo are disabled and yum update is run.
This is currently done for Dev on Tuesday and
prod on Thursday.
Requires logging into each system.
Systems are rebooted when there is a kernel update.
Some systems have specific problems that need to be
tested.
Issues can throw off the schedule significantly.

4. IssuesIssues
There can still be package drift in the repos.
Waste of disk space and bandwidth.
Does not gracefully handle the situation where a
single server may require patch exceptions.
Doesn't lock the entire repository.
If a package is installed outside of the patching
cycle, it may pull in additional updates that you
were not expecting or testing for.

5. OptionsOptions

6. SameSame

7. SpacewalkSpacewalk

8. KatelloKatello

9. KatelloKatello Repository Management
Subscription/Entitlement
Management
Content Management
Scheduled remote tasks
SLA groups
Activation groups
Protected Repositories
etc....

10. Three reasons Katello wasThree reasons Katello was
not chosennot chosen
1. Postgres
2. Mongo
3. ElasticSearch

11. They are not bad separately, butThey are not bad separately, but
they fight for resources andthey fight for resources and
together add complexity.together add complexity.

12. PulpPulp

13. PulpPulp Content Mangement
Repo Synchronization
Manage 'environments'

14. What about SAM?What about SAM?
We are, for the most part, currently a RHEL shop

16. WorkflowWorkflow

17. Live Repo
Dev/Test
Repo
Production
Repo
# pulp-admin rpm repo create
--repo-id=centos-6-updates-x86_64-live
--display-name="CentOS 6 Updates x86_64 (Live)"
--description="CentOS 6 Updates x86_64 (Live)"
--feed="http://mirror.centos.org/centos-6/6/updates/x86_64/"
--serve-http=True
--serve-https=False
--relative-url=/centos/6/updates/x86_64/live
--gpg-key=/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6

18. Live Repo
Dev/Test
Repo
Production
Repo
# pulp-admin rpm repo create
--repo-id=centos-6-updates-x86_64-dev
--display-name="CentOS 6 Updates x86_64 (Dev)"
--description="CentOS 6 Updates x86_64 (Dev)"
--serve-http=True
--serve-https=False
--relative-url=/centos/6/updates/x86_64/dev
--gpg-key=/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6

19. Live Repo
Dev/Test
Repo
Production
Repo
Same as Dev

20. Live Repo
Dev/Test
Repo
Production
Repo
Synchronized Daily

21. Snap
Live Repo
Dev/Test
Repo
Production
Repo
Patch day has arrived!!!
Automatically promote 'Live' to 'Dev',
snapshotting 'Dev' in the process.
Pins RPM versions at that current state.
Doesn't actually copy files.

22. Live Repo
Dev/Test
Repo
Production
Repo
Dev systems have passed!!!
Manually promote 'Dev' to 'Prod'.
Pins RPM versions at that current state
of 'Dev'.
Promote script auto snapshots Prod.
Again, it doesn't actually copy files.
# repo-snap-promote
--parent=centos-6-updates-x86_64
--from=dev
--to=prod
Snap

23. Oh Noes!!!Oh Noes!!!

24. Live Repo
Dev/Test
Repo
Roll back the changes.
Don't even think about touching
Prod.
Branch off another rollback repo.
Point the client to the new repo.
Repatch and test.
# repo-rollback --parent=centos-6-updates-x86_64
--snapshot=20141101
--match='name=^php.*$'
custom-repo
Production
Repo
Snap
Custom Rollback
Repo

25. Client SideClient Side

26. ClientClient Disable or remove standard
repos in yum.repos.d.
Add your own repo file that
points back to the server.
That's it, nothing special.

27. ClientClient Disable or remove standard
repos in yum.repos.d .
Add your own repo file that
points back to the server.
That's it, nothing special.
[centos-6-updates-x86_64]
name = CentOS 6 Updates (x86_64)
enabled = 1
gpgcheck = 1
baseurl = http://sys-dev-repo.sys-sandbox.local/pulp/repos/centos/6/updates/x86_64/$pkgenv

28. ClientClient Disable or remove standard
repos in yum.repos.d .
Add your own repo file that
points back to the server.
That's it, nothing special.
[centos-6-updates-x86_64]
name = CentOS 6 Updates (x86_64)
enabled = 1
gpgcheck = 1
baseurl = http://sys-dev-repo.sys-sandbox.local/pulp/repos/centos/6/updates/x86_64/$pkgenv
# cat /etc/yum/vars/pkgenv
prod

29. RedhatRedhat
(Specific)
We won't be using opensource
entitlement management.
Use subscription-manager to
register the system to RHN.
Once the system is registered,
use repo-override to turn off the
all enabled repos.
Add the repos like we did for
CentOS.
subscription-manager repo-override --repo=rhel-5-server-rpms --add=enabled:0

30. AutomationAutomation

31. SaltSalt Great remote execution tool.
Easy to install and configure.
Has plugins to initiate yum
updates.
Has plugins to restart and test
services.

32. Paired with cron or salt'sPaired with cron or salt's
scheduler you have automaticscheduler you have automatic
patching for non criticalpatching for non critical
systems.systems.

33. Questions?Questions?