Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Repositories as Code


Published on

Packages, Repositories , Pipelines, Promotions,
Managing Repositories in a MultiClient environmment
My talk for the 2018 #centosdojo @ Cern.

Published in: Technology
  • Be the first to comment

Repositories as Code

  1. 1. Packages , Repositories,Packages , Repositories, Pipelines & PromotionsPipelines & Promotions Kris Buytaert @krisbuytaert
  2. 2. Repositories as CodeRepositories as Code Kris Buytaert @krisbuytaert
  3. 3. KrisKris BuytaertBuytaert ● I used to be a Dev,I used to be a Dev, ● Then Became an OpThen Became an Op ● Chief Trolling Officer and Open SourceChief Trolling Officer and Open Source Consultant @inuits.euConsultant ● Everything is an effing DNS ProblemEverything is an effing DNS Problem ● Organising too many confs , #devopsdays,Organising too many confs , #devopsdays, #loadays, ...#loadays, ... ● Evangelizing devopsEvangelizing devops
  4. 4. Why talk about RepositoryWhy talk about Repository management ?management ?
  5. 5. devops =~ clamsdevops =~ clams ● CultureCulture ● (Lean)(Lean) ● Automate all the things ...Automate all the things ... – Build AutomationBuild Automation – Package all the thingsPackage all the things – Test AutomationTest Automation – IACIAC ● Monitoring , Metrics ...Monitoring , Metrics ... ● SharingSharing
  6. 6. Let's talk about PackagingLet's talk about Packaging ● Do you package ?Do you package ? – Packaging software in a distro ?Packaging software in a distro ? – Packaging languages ?Packaging languages ? – Packaging in an enterprisePackaging in an enterprise
  7. 7. Software Delivery AdoptionSoftware Delivery Adoption ● Level 0Level 0 – curlcurl| sh| sh ● Level 1Level 1 – curl -sSL | bashcurl -sSL | bash
  8. 8. Solution :Solution : Package all the thingsPackage all the things
  9. 9. Why ops like to packageWhy ops like to package ● Packages give you featuresPackages give you features •Consistency, security, dependenciesConsistency, security, dependencies ● Uniquely identify where files come fromUniquely identify where files come from •Package or cfg-mgmtPackage or cfg-mgmt ● Source repo not always availableSource repo not always available •Firewall / Cloud etc ..Firewall / Cloud etc .. ● Weird deployment locations , no easy accessWeird deployment locations , no easy access ● Little overhead when you automateLittle overhead when you automate ● CONFIG does not belong in a packageCONFIG does not belong in a package
  10. 10. In Continuous DeliveryIn Continuous Delivery ● Unmodified , Tested artifacts go trough aUnmodified , Tested artifacts go trough a pipeline.pipeline. application code,application code, Infra codeInfra code metadatametadata teststests ● We need to package these so they becomeWe need to package these so they become immutableimmutable
  11. 11. #devopsdays 2010 Open#devopsdays 2010 Open Space ConclusionsSpace Conclusions ● Always package software YOU deployAlways package software YOU deploy – Exceptions: code that changes faster thanExceptions: code that changes faster than you can package it. (Very rare)you can package it. (Very rare) ● Do NOT package Config FILES ,Do NOT package Config FILES , – Use a cfgmgmt tool for thisUse a cfgmgmt tool for this ● Languages are still reinventing the wheel :(Languages are still reinventing the wheel :(
  12. 12. NotNot allall packagespackages areare equalequal
  13. 13. From #packagingsucks To I love fpm
  14. 14. So we 'solved' packaging, now how to shipSo we 'solved' packaging, now how to ship packages ?packages ?
  15. 15. Level 2: Random yumLevel 2: Random yum repo’srepo’s ● Enable repoEnable repo ● dnf/yum install packagednf/yum install package ● 3 weeks later package has been3 weeks later package has been – RenamedRenamed – UpgradedUpgraded – MovedMoved ● Random upgrades / Differend versionsRandom upgrades / Differend versions deployeddeployed ● Conflicting Dependencies from different repos.Conflicting Dependencies from different repos.
  16. 16. Level 3: Local mirrorsLevel 3: Local mirrors ● FixesFixes – Upstream changesUpstream changes – Upstream dissapearingUpstream dissapearing ● Doesn’t fix duplicate / conflicting dependenciesDoesn’t fix duplicate / conflicting dependencies
  17. 17. The ChallengeThe Challenge ● 20 + customer platforms/stacks20 + customer platforms/stacks ● 1 distro1 distro ● 2 majrel2 majrel ● 3 fte3 fte ● 2-3 environments (dev/uat/prod/...) per2-3 environments (dev/uat/prod/...) per customer platformcustomer platform ● Evolving at different speedEvolving at different speed
  18. 18. Environment Based RepositoriesEnvironment Based Repositories ● Dedicated Set of repositories per EnvironmentDedicated Set of repositories per Environment ● {Stack/Customer} / uat /prod /dev{Stack/Customer} / uat /prod /dev – Consistent stable repositoriesConsistent stable repositories – Repoducable platformsRepoducable platforms – Different versions stacks per customerDifferent versions stacks per customer – Based on local mirrorsBased on local mirrors
  19. 19. 3 types of packages3 types of packages ● UpstreamUpstream – Standard ReposStandard Repos – Extended ReposExtended Repos – Community build reposCommunity build repos
  20. 20. 3 types of packages3 types of packages ● UpstreamUpstream ● Custom/Build SoftwareCustom/Build Software – Upstream Doesn’t packageUpstream Doesn’t package – Upstream has broken packagesUpstream has broken packages – Patched UpstreamPatched Upstream
  21. 21. 3 types of packages3 types of packages ● UpstreamUpstream ● Custom Build SoftwareCustom Build Software ● Own SoftwareOwn Software
  22. 22. Repository ManagementRepository Management
  23. 23. Early PulpEarly Pulp ● Redhat CommunityRedhat Community ● Redhat Emerging TechnologyRedhat Emerging Technology ● Part of Katello / Foreman .. EcosystemPart of Katello / Foreman .. Ecosystem
  24. 24. Pulp and puppetPulp and puppet ● Upstream katello-pulp moduleUpstream katello-pulp module ● To be pulp_apiTo be pulp_api – Includes types and providers for repositoriesIncludes types and providers for repositories ● Pulp repos now configured from hieraPulp repos now configured from hiera ●
  25. 25. Pulp hiera (mirrors)Pulp hiera (mirrors) • xx profile_pulp::purge_repos: true profile_pulp::mirrors: mirrors/centos/7/os/x86_64/: url: mirrors/centos/7/updates/x86_64/: url: mirrors/centos/7/extras/x86_64/: url: mirrors/centos/7/sclo/x86_64/rh/: url: mirrors/centos/7/sclo/x86_64/sclo/: url: mirrors/epel/7/x86_64/: url: mirrors/puppetlabs/el/7/products/x86_64/: url: mirrors/puppetlabs/el/7/dependencies/x86_64/: url: mirrors/puppetlabs/puppet/el/7/x86_64: url: mirrors/passenger/el/7/x86_64/: url: mirrors/theforeman/latest/el7/x86_64/: url: mirrors/theforeman/plugins/latest/el7/x86_64/: url: mirrors/pulp/stable/2/7/x86_64: url:
  26. 26. Pulp hiera (defaults)Pulp hiera (defaults) profile_pulp::promotion_defaults:profile_pulp::promotion_defaults: repositories:repositories: centos_base:centos_base: upstream: /pub/mirrors/centos/7/os/x86_64/upstream: /pub/mirrors/centos/7/os/x86_64/ centos_updates:centos_updates: upstream: /pub/mirrors/centos/7/updates/x86_64/upstream: /pub/mirrors/centos/7/updates/x86_64/ centos_extras:centos_extras: upstream: /pub/mirrors/centos/7/extras/x86_64/upstream: /pub/mirrors/centos/7/extras/x86_64/ upstream:upstream: allow_upload_from: [jenkins]allow_upload_from: [jenkins] retain_old_count: 5retain_old_count: 5 internal:internal: allow_upload_from: [jenkins]allow_upload_from: [jenkins] retain_old_count: 10retain_old_count: 10 unpromotable: trueunpromotable: true custom_build:custom_build: allow_upload_from: [jenkins]allow_upload_from: [jenkins] retain_old_count: 10retain_old_count: 10
  27. 27. Pulp hiera (promotions)Pulp hiera (promotions) profile_pulp::yum_promotion_trees:profile_pulp::yum_promotion_trees: hakka:hakka: first_target:first_target: - hakkadev- hakkadev targets:targets: hakkadev:hakkadev: next_targets:next_targets: - hakkauat- hakkauat hakkauat:hakkauat: next_targets:next_targets: - hakkaprod- hakkaprod hakkaprod:hakkaprod: archive: truearchive: true Generates all repositories andGenerates all repositories and promotion scripts :promotion scripts : promote-hakka-hakkadevpromote-hakka-hakkadev promote-hakka-hakkaprodpromote-hakka-hakkaprod promote-hakka-hakkaprod-archivepromote-hakka-hakkaprod-archive promote-hakka-hakkauatpromote-hakka-hakkauat
  28. 28. Build the Upstream repoBuild the Upstream repo ● Initially :Initially : – Manually pulp-admin upload filesManually pulp-admin upload files – Pulp-admin copy files to other repoPulp-admin copy files to other repo
  29. 29. Build the Upstream repoBuild the Upstream repo ● Step 1Step 1 – Manually pulp-admin upload filesManually pulp-admin upload files – Pulp-admin copy files to other repoPulp-admin copy files to other repo
  30. 30. Build the Upstream repoBuild the Upstream repo ● Yaml file that list files (+Yaml file that list files (+ versions) from sourceversions) from source ● Jenkins builds repo onJenkins builds repo on commitcommit - desc: epel tools + certbot (Let's Encrypt)- desc: epel tools + certbot (Let's Encrypt) from: mirrors/epel/7/x86_64from: mirrors/epel/7/x86_64 pkgs:pkgs: - htop- htop - iftop- iftop - mytop- mytop - jq- jq - ncdu- ncdu - rkhunter- rkhunter - certbot- certbot - ngrep- ngrep - desc: gluster- desc: gluster from:from: - mirrors/gluster41/- mirrors/gluster41/ pkgs:pkgs: - glusterfs- glusterfs - glusterfs-api- glusterfs-api - glusterfs-cli- glusterfs-cli - glusterfs-client-xlators- glusterfs-client-xlators - glusterfs-fuse- glusterfs-fuse - glusterfs-libs- glusterfs-libs - glusterfs-server- glusterfs-server - userspace-rcu- userspace-rcu
  31. 31. Promoting your own packagesPromoting your own packages ● Application specific Jenkins Pipeline uploadsApplication specific Jenkins Pipeline uploads promoted individual package to $environmentpromoted individual package to $environment repositoryrepository ● Deploy & upload 2 repoDeploy & upload 2 repo ● (rebootstrappable but no delay in deployment)(rebootstrappable but no delay in deployment)
  32. 32. ConclusionConclusion ● Scalable approach to managing mirrorsScalable approach to managing mirrors ● Scalable approach to managing upstreamScalable approach to managing upstream packagespackages ● Reproducable repositoriesReproducable repositories ● We’re all Yaml engineers now.We’re all Yaml engineers now.
  33. 33. One more thingOne more thing Config Management Camp 2019Config Management Camp 2019 4-6 february 20194-6 february 2019 Gent , BelgiumGent , Belgium
  34. 34. ContactContact Kris Buytaert kris.buytaert@inuits.euKris Buytaert Further ReadingFurther Reading @krisbuytaert@krisbuytaert Find Inuits inFind Inuits in Antwerpen,Ghent,Antwerpen,Ghent, Rotterdam,Prague,KieRotterdam,Prague,Kie vv