Packages , Repositories,Packages , Repositories,
Pipelines & PromotionsPipelines & Promotions
Kris Buytaert
@krisbuytaert

Repositories as CodeRepositories as Code
Kris Buytaert
@krisbuytaert

KrisKris BuytaertBuytaert
● I used to be a Dev,I used to be a Dev,
● Then Became an OpThen Became an Op
● Chief Trolling Officer and Open SourceChief Trolling Officer and Open Source
Consultant @inuits.euConsultant @inuits.eu
● Everything is an effing DNS ProblemEverything is an effing DNS Problem
● Organising too many confs , #devopsdays,Organising too many confs , #devopsdays,
#loadays, ...#loadays, ...
● Evangelizing devopsEvangelizing devops

Why talk about RepositoryWhy talk about Repository
management ?management ?

devops =~ clamsdevops =~ clams
● CultureCulture
● (Lean)(Lean)
● Automate all the things ...Automate all the things ...
– Build AutomationBuild Automation
– Package all the thingsPackage all the things
– Test AutomationTest Automation
– IACIAC
● Monitoring , Metrics ...Monitoring , Metrics ...
● SharingSharing

Let's talk about PackagingLet's talk about Packaging
● Do you package ?Do you package ?
– Packaging software in a distro ?Packaging software in a distro ?
– Packaging languages ?Packaging languages ?
– Packaging in an enterprisePackaging in an enterprise

Software Delivery AdoptionSoftware Delivery Adoption
● Level 0Level 0
– curlcurl http://somenaughtysite.io/random.shhttp://somenaughtysite.io/random.sh| sh| sh
● Level 1Level 1
– curl -sSL https://get.rvm.io | bashcurl -sSL https://get.rvm.io | bash

Solution :Solution :
Package all the thingsPackage all the things

Why ops like to packageWhy ops like to package
● Packages give you featuresPackages give you features
•Consistency, security, dependenciesConsistency, security, dependencies
● Uniquely identify where files come fromUniquely identify where files come from
•Package or cfg-mgmtPackage or cfg-mgmt
● Source repo not always availableSource repo not always available
•Firewall / Cloud etc ..Firewall / Cloud etc ..
● Weird deployment locations , no easy accessWeird deployment locations , no easy access
● Little overhead when you automateLittle overhead when you automate
● CONFIG does not belong in a packageCONFIG does not belong in a package

In Continuous DeliveryIn Continuous Delivery
● Unmodified , Tested artifacts go trough aUnmodified , Tested artifacts go trough a
pipeline.pipeline.
application code,application code,
Infra codeInfra code
metadatametadata
teststests
● We need to package these so they becomeWe need to package these so they become
immutableimmutable

#devopsdays 2010 Open#devopsdays 2010 Open
Space ConclusionsSpace Conclusions
● Always package software YOU deployAlways package software YOU deploy
– Exceptions: code that changes faster thanExceptions: code that changes faster than
you can package it. (Very rare)you can package it. (Very rare)
● Do NOT package Config FILES ,Do NOT package Config FILES ,
– Use a cfgmgmt tool for thisUse a cfgmgmt tool for this
● Languages are still reinventing the wheel :(Languages are still reinventing the wheel :(

NotNot allall packagespackages areare equalequal

From
#packagingsucks
To
I love fpm

So we 'solved' packaging, now how to shipSo we 'solved' packaging, now how to ship
packages ?packages ?

Level 2: Random yumLevel 2: Random yum
repo’srepo’s
● Enable repoEnable repo
● dnf/yum install packagednf/yum install package
● 3 weeks later package has been3 weeks later package has been
– RenamedRenamed
– UpgradedUpgraded
– MovedMoved
● Random upgrades / Differend versionsRandom upgrades / Differend versions
deployeddeployed
● Conflicting Dependencies from different repos.Conflicting Dependencies from different repos.

Level 3: Local mirrorsLevel 3: Local mirrors
● FixesFixes
– Upstream changesUpstream changes
– Upstream dissapearingUpstream dissapearing
● Doesn’t fix duplicate / conflicting dependenciesDoesn’t fix duplicate / conflicting dependencies

The ChallengeThe Challenge
● 20 + customer platforms/stacks20 + customer platforms/stacks
● 1 distro1 distro
● 2 majrel2 majrel
● 3 fte3 fte
● 2-3 environments (dev/uat/prod/...) per2-3 environments (dev/uat/prod/...) per
customer platformcustomer platform
● Evolving at different speedEvolving at different speed

Environment Based RepositoriesEnvironment Based Repositories
● Dedicated Set of repositories per EnvironmentDedicated Set of repositories per Environment
● {Stack/Customer} / uat /prod /dev{Stack/Customer} / uat /prod /dev
– Consistent stable repositoriesConsistent stable repositories
– Repoducable platformsRepoducable platforms
– Different versions stacks per customerDifferent versions stacks per customer
– Based on local mirrorsBased on local mirrors

3 types of packages3 types of packages
● UpstreamUpstream
– Standard ReposStandard Repos
– Extended ReposExtended Repos
– Community build reposCommunity build repos

3 types of packages3 types of packages
● UpstreamUpstream
● Custom/Build SoftwareCustom/Build Software
– Upstream Doesn’t packageUpstream Doesn’t package
– Upstream has broken packagesUpstream has broken packages
– Patched UpstreamPatched Upstream

3 types of packages3 types of packages
● UpstreamUpstream
● Custom Build SoftwareCustom Build Software
● Own SoftwareOwn Software

Repository ManagementRepository Management

Early PulpEarly Pulp
● Redhat CommunityRedhat Community
● Redhat Emerging TechnologyRedhat Emerging Technology
● Part of Katello / Foreman .. EcosystemPart of Katello / Foreman .. Ecosystem

Pulp and puppetPulp and puppet
● Upstream katello-pulp moduleUpstream katello-pulp module
● To be pulp_apiTo be pulp_api
– Includes types and providers for repositoriesIncludes types and providers for repositories
● Pulp repos now configured from hieraPulp repos now configured from hiera
● https://github.com/SimonPe/puppet-pulpapihttps://github.com/SimonPe/puppet-pulpapi

Pulp hiera (mirrors)Pulp hiera (mirrors)
•
xx
profile_pulp::purge_repos: true
profile_pulp::mirrors:
mirrors/centos/7/os/x86_64/:
url: http://mirror.centos.org/centos/7/os/x86_64/
mirrors/centos/7/updates/x86_64/:
url: http://mirror.centos.org/centos/7/updates/x86_64/
mirrors/centos/7/extras/x86_64/:
url: http://mirror.centos.org/centos/7/extras/x86_64/
mirrors/centos/7/sclo/x86_64/rh/:
url: http://mirror.centos.org/centos/7/sclo/x86_64/rh/
mirrors/centos/7/sclo/x86_64/sclo/:
url: http://mirror.centos.org/centos/7/sclo/x86_64/sclo/
mirrors/epel/7/x86_64/:
url: https://dl.fedoraproject.org/pub/epel/7/x86_64/
mirrors/puppetlabs/el/7/products/x86_64/:
url: http://yum.puppetlabs.com/el/7/products/x86_64/
mirrors/puppetlabs/el/7/dependencies/x86_64/:
url: http://yum.puppetlabs.com/el/7/dependencies/x86_64/
mirrors/puppetlabs/puppet/el/7/x86_64:
url: http://yum.puppetlabs.com/puppet/el/7/x86_64/
mirrors/passenger/el/7/x86_64/:
url: https://oss-binaries.phusionpassenger.com/yum/passenger/el/7/x86_64
mirrors/theforeman/latest/el7/x86_64/:
url: http://yum.theforeman.org/releases/latest/el7/x86_64/
mirrors/theforeman/plugins/latest/el7/x86_64/:
url: http://yum.theforeman.org/plugins/latest/el7/x86_64/
mirrors/pulp/stable/2/7/x86_64:
url: https://repos.fedorapeople.org/pulp/pulp/stable/2/7/x86_64/

Pulp hiera (defaults)Pulp hiera (defaults)
profile_pulp::promotion_defaults:profile_pulp::promotion_defaults:
repositories:repositories:
centos_base:centos_base:
upstream: /pub/mirrors/centos/7/os/x86_64/upstream: /pub/mirrors/centos/7/os/x86_64/
centos_updates:centos_updates:
upstream: /pub/mirrors/centos/7/updates/x86_64/upstream: /pub/mirrors/centos/7/updates/x86_64/
centos_extras:centos_extras:
upstream: /pub/mirrors/centos/7/extras/x86_64/upstream: /pub/mirrors/centos/7/extras/x86_64/
upstream:upstream:
allow_upload_from: [jenkins]allow_upload_from: [jenkins]
retain_old_count: 5retain_old_count: 5
internal:internal:
allow_upload_from: [jenkins]allow_upload_from: [jenkins]
retain_old_count: 10retain_old_count: 10
unpromotable: trueunpromotable: true
custom_build:custom_build:
allow_upload_from: [jenkins]allow_upload_from: [jenkins]
retain_old_count: 10retain_old_count: 10

Pulp hiera (promotions)Pulp hiera (promotions)
profile_pulp::yum_promotion_trees:profile_pulp::yum_promotion_trees:
hakka:hakka:
first_target:first_target:
- hakkadev- hakkadev
targets:targets:
hakkadev:hakkadev:
next_targets:next_targets:
- hakkauat- hakkauat
hakkauat:hakkauat:
next_targets:next_targets:
- hakkaprod- hakkaprod
hakkaprod:hakkaprod:
archive: truearchive: true
Generates all repositories andGenerates all repositories and
promotion scripts :promotion scripts :
promote-hakka-hakkadevpromote-hakka-hakkadev
promote-hakka-hakkaprodpromote-hakka-hakkaprod
promote-hakka-hakkaprod-archivepromote-hakka-hakkaprod-archive
promote-hakka-hakkauatpromote-hakka-hakkauat

Build the Upstream repoBuild the Upstream repo
● Initially :Initially :
– Manually pulp-admin upload filesManually pulp-admin upload files
– Pulp-admin copy files to other repoPulp-admin copy files to other repo

Build the Upstream repoBuild the Upstream repo
● Step 1Step 1
– Manually pulp-admin upload filesManually pulp-admin upload files
– Pulp-admin copy files to other repoPulp-admin copy files to other repo

Build the Upstream repoBuild the Upstream repo
● Yaml file that list files (+Yaml file that list files (+
versions) from sourceversions) from source
● Jenkins builds repo onJenkins builds repo on
commitcommit
- desc: epel tools + certbot (Let's Encrypt)- desc: epel tools + certbot (Let's Encrypt)
from: mirrors/epel/7/x86_64from: mirrors/epel/7/x86_64
pkgs:pkgs:
- htop- htop
- iftop- iftop
- mytop- mytop
- jq- jq
- ncdu- ncdu
- rkhunter- rkhunter
- certbot- certbot
- ngrep- ngrep
- desc: gluster- desc: gluster
from:from:
- mirrors/gluster41/- mirrors/gluster41/
pkgs:pkgs:
- glusterfs- glusterfs
- glusterfs-api- glusterfs-api
- glusterfs-cli- glusterfs-cli
- glusterfs-client-xlators- glusterfs-client-xlators
- glusterfs-fuse- glusterfs-fuse
- glusterfs-libs- glusterfs-libs
- glusterfs-server- glusterfs-server
- userspace-rcu- userspace-rcu

Promoting your own packagesPromoting your own packages
● Application specific Jenkins Pipeline uploadsApplication specific Jenkins Pipeline uploads
promoted individual package to $environmentpromoted individual package to $environment
repositoryrepository
● Deploy & upload 2 repoDeploy & upload 2 repo
● (rebootstrappable but no delay in deployment)(rebootstrappable but no delay in deployment)

ConclusionConclusion
● Scalable approach to managing mirrorsScalable approach to managing mirrors
● Scalable approach to managing upstreamScalable approach to managing upstream
packagespackages
● Reproducable repositoriesReproducable repositories
● We’re all Yaml engineers now.We’re all Yaml engineers now.

One more thingOne more thing
Config Management Camp 2019Config Management Camp 2019
4-6 february 20194-6 february 2019
Gent , BelgiumGent , Belgium
https://cfp.cfgmgmtcamp.be/https://cfp.cfgmgmtcamp.be/

ContactContact
Kris Buytaert kris.buytaert@inuits.euKris Buytaert kris.buytaert@inuits.eu
Further ReadingFurther Reading
@krisbuytaert@krisbuytaert
http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/
http://inuits.eu/http://inuits.eu/
Find Inuits inFind Inuits in
Antwerpen,Ghent,Antwerpen,Ghent,
Rotterdam,Prague,KieRotterdam,Prague,Kie
vv