Kris Buytaert discusses the gap between software developers (devs) and operations teams (ops) when using Linux distributions. There is often a lack of communication and different priorities, with devs focused on getting the latest code quickly and ops concerned with stability, security and deployability at scale. Buytaert provides examples of issues, such as packages containing PHP code directly in /etc, and distributions packaging outdated or broken upstream software. He advocates for better collaboration between distributions, upstream projects and power users through activities like meetups, user advisory boards, and tools to more easily build and test packages. The goal is to improve culture, automation, measurement and sharing between all parties.

1. Closing the gapClosing the gap
between Distros (devs)between Distros (devs)
and their Users (ops)and their Users (ops)
Kris Buytaert
@krisbuytaert

2. Kris BuytaertKris Buytaert
● 20+ years of Linux System Administration20+ years of Linux System Administration
● Release manager of upstream open sourceRelease manager of upstream open source
software (back in 2001)software (back in 2001)
● Assisting organisations in deploying software ,Assisting organisations in deploying software ,
better, faster, stronger,better, faster, stronger,
● Slackware, SLS, RedHat, Debian, Suse, Fedora,Slackware, SLS, RedHat, Debian, Suse, Fedora,
RHEL, CentOS, SLES, UbuntuRHEL, CentOS, SLES, Ubuntu
● Busy building stufBusy building stuf

3. devops =~C(L)AMSdevops =~C(L)AMS
● CultureCulture
● (Lean)(Lean)
● AutomationAutomation
•
Build, Deploy, Package,ConfigureBuild, Deploy, Package,Configure
● MeasurementMeasurement
● SharingSharing
Damon Edwards and John WillisDamon Edwards and John Willis

4. Culture,Culture,
automation,automation,
Measturement,Measturement,
sharingsharing

5. How I illustrate dev-oopsHow I illustrate dev-oops
● ““Put this Code Live, here's a tarball” NOW!Put this Code Live, here's a tarball” NOW!
● What dependencies ?What dependencies ?
● No machines available ?No machines available ?
● What database ?What database ?
● Security ?Security ?
● High Availability ?High Availability ?
● Scalability ?Scalability ?
● My computer can't install this ?My computer can't install this ?

6. How I illustrate distro-How I illustrate distro-
oopsoops
● We've introduced “XYZ” in the releaseWe've introduced “XYZ” in the release
● What where ..What where ..
•
Shit just broke ..Shit just broke ..
•
All tests gone redAll tests gone red
•
That's a major refactorThat's a major refactor
•
6 renamed packages and diferent defaults6 renamed packages and diferent defaults
•
Where's the disable buttonWhere's the disable button

7. Root CauseRoot Cause
● Ops are to busy fixing …Ops are to busy fixing …
● Someone else will react to this .. this is “trivial”Someone else will react to this .. this is “trivial”
● They can't be this …They can't be this …
● No trivial feedback loopNo trivial feedback loop
● Ops Ops folks are not involved in the communitiesOps Ops folks are not involved in the communities
● Is your community, your userbase ?Is your community, your userbase ?

8. Your Team
Your community :
Power users, evangelists
contributors
Upstream
Users Users active in other Open Source Communities
Your Customers ?

9. Your Team
Your active vocal users
aka community
Upstream
Users Users active in other Open Source Communities
D
iferent
U
se
Cases,
D
iferent
N
eeds

10. Culture,Culture,
Automation,Automation,
Measurement,Measurement,
SharingSharing

11. Launch all deamonsLaunch all deamons
● Because in a clustered setup I want theBecause in a clustered setup I want the
daemon running only on the node my CRM tellsdaemon running only on the node my CRM tells
it to.it to.
● Because I first want to reconfigure that serviceBecause I first want to reconfigure that service
etc before you launch itetc before you launch it
● Or I`m building a containerOr I`m building a container
● Mostly aimed at single server setups :(Mostly aimed at single server setups :(

12. IdeaIdea
● apt-get install vs apt-get runapt-get install vs apt-get run
● yum install vs yum runyum install vs yum run

13. #packagingsucks#packagingsucks
• Really ..Really ..
●
Missing upstreamMissing upstream
●
Ancient upstreamAncient upstream
●
Unneeded dependenciesUnneeded dependencies
●
Broken upstreamBroken upstream
●
Distro PoliciesDistro Policies
● Packaging is neededPackaging is needed
●
Dependencies, tooling, repositoriesDependencies, tooling, repositories

14. Languages and PackagingLanguages and Packaging

18. CPAN,RubyGems,PIP,...CPAN,RubyGems,PIP,...
Every single language tries to package theirEvery single language tries to package their
libraries,libraries,
Ops people hate those packagesOps people hate those packages
And repackage themAnd repackage them
fpm -s gem -t rpm fpmfpm -s gem -t rpm fpm

19. Devs vs opsDevs vs ops
● Getting the latestGetting the latest
versions working onversions working on
their dev setuptheir dev setup
● Get that from theGet that from the
internet fastinternet fast
● curl http://shaky.li.nk |curl http://shaky.li.nk |
sudo /bin/bashsudo /bin/bash
● Works on my machineWorks on my machine
● A reproducableA reproducable
● ShippableShippable
● ArtifactArtifact
● MaintainabilityMaintainability

20. What ops do ..What ops do ..
● Deploy ruby in jrubyDeploy ruby in jruby
● Software Collections (newer versions)Software Collections (newer versions)
●

22. What ops do ..What ops do ..
● Build a full stack in a chrootBuild a full stack in a chroot
•
Virtualenv, rbenv, rvmVirtualenv, rbenv, rvm
•
Build in containersBuild in containers

23. Distros and PackagingDistros and Packaging

24. Some DistrosSome Distros
● Package unexisting releasesPackage unexisting releases
● Change version numbersChange version numbers
● Make developers hate those distrosMake developers hate those distros
● The distro that once shipped an 2.6 KernelThe distro that once shipped an 2.6 Kernel
version of openMosix knows I`m talking aboutversion of openMosix knows I`m talking about
them.them.

25. Contributing to Distros ?Contributing to Distros ?
Distribution packaging policies are not designedDistribution packaging policies are not designed
for people who package softwarefor people who package software
John Vincent, on his blog in 2013John Vincent, on his blog in 2013

26. Should this even be in a distro ?Should this even be in a distro ?
● ““What do you mean there is php code in /etc ?”What do you mean there is php code in /etc ?”
● /etc/drupal ?/etc/drupal ?
● Are you kidding me ?Are you kidding me ?
● Just kill that package, don't ever use it againJust kill that package, don't ever use it again
and let our jenkins roll our own.and let our jenkins roll our own.

27. Hey Distro:Hey Distro:
● There is no way you can keep up with theThere is no way you can keep up with the
speed of some webapps releasing newspeed of some webapps releasing new
versions.versions.
● Maybe you shouldn't even tryMaybe you shouldn't even try

28. Culture,Culture,
Automation,Automation,
Measurement,Measurement,
SharingSharing

29. Distro's do great workDistro's do great work
•
OBSOBS
•
CBSCBS
● Not all of it is knownNot all of it is known

30. CI for UpstreamCI for Upstream
● Engage with upstreamEngage with upstream
● Provide tools for them to build packages easyProvide tools for them to build packages easy

31. Who is your user ?Who is your user ?
● Junior Software developer who needsJunior Software developer who needs
•
Database, webserver etcDatabase, webserver etc
•
Running triviallyRunning trivially
● Ops guy who needs to deploy , manage andOps guy who needs to deploy , manage and
monitor large inframonitor large infra

32. Get your power users involvedGet your power users involved
● More MeetupsMore Meetups
● Lugs are dissapearingLugs are dissapearing
● User Advisory boardUser Advisory board
● Engage them, before they complain, or switchEngage them, before they complain, or switch
to another distroto another distro

34. ContactContact
Kris Buytaert kris.buytaert@inuits.euKris Buytaert kris.buytaert@inuits.eu
Further ReadingFurther Reading
@krisbuytaert@krisbuytaert
http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/
http://www.inuits.eu/http://www.inuits.eu/
Find Inuits inFind Inuits in
Brasschaat,Ghent,Brasschaat,Ghent,
Rotterdam,Prague,Rotterdam,Prague,
Kiev,BrnoKiev,Brno