WebLion Hosting: Leveraging Laziness, Impatience, and Hubris
by Erik Rose
- 1,991 views
Behind the scenes of WebLion's Plone hosting service, which uses Debian packages and a custom repository to deliver reliable, unattended updates to a cluster of heterogeneous departmental virtual ...
Behind the scenes of WebLion's Plone hosting service, which uses Debian packages and a custom repository to deliver reliable, unattended updates to a cluster of heterogeneous departmental virtual servers. And it's all available for your own use for free.
Accessibility
Categories
Upload Details
Uploaded via SlideShare as Apple Keynote
Usage Rights
© All Rights Reserved
Statistics
- Likes
- 2
- Downloads
- 18
- Comments
- 0
- Embed Views
- Views on SlideShare
- 1,971
- Total Views
- 1,991
Leonardo J. Caballero G.
You can think of WL Hosting as…
came out of 2 realizations: lots more to a Plone deployment than Zope & Plone. \\ Python, Apache, Squid, cron jobs for DB maint & backups, SNMP for remote monitoring, …. Then kernel, libs, etc.
2nd thing: I realized there’s a strangeness in WebLion’s business model…
Didn’t realize: Plone apparently hard to sysadmin
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
However, as we accumulated more and more partners, this didn’t scale. We don’t have any dedicated sysadmins on the team, and I was spending a huge chunk of my time teaching and debugging people’s setups and not enough coding. I’m really a programmer, after all, and only a sysadmin by necessity. So, bringing a programmer’s point of view to the problem, I thought “How I do all this stuff once instead of repeating it for every partner?”
It was evident we’d have to change our don’t-do-anything business model, but how to do it? Well, in an ideal world, everybody’d have…
But that ain’t gonna happen. They’re gonna need different… 1 2 (and versions of products) 3 4 5.
So the question became not “How do we build a gigantic megaserver that can take care of everybody?” but “How do we deploy a bunch of similar-but-not-identical servers”
But that ain’t gonna happen. They’re gonna need different… 1 2 (and versions of products) 3 4 5.
So the question became not “How do we build a gigantic megaserver that can take care of everybody?” but “How do we deploy a bunch of similar-but-not-identical servers”
But that ain’t gonna happen. They’re gonna need different… 1 2 (and versions of products) 3 4 5.
So the question became not “How do we build a gigantic megaserver that can take care of everybody?” but “How do we deploy a bunch of similar-but-not-identical servers”
But that ain’t gonna happen. They’re gonna need different… 1 2 (and versions of products) 3 4 5.
So the question became not “How do we build a gigantic megaserver that can take care of everybody?” but “How do we deploy a bunch of similar-but-not-identical servers”
But that ain’t gonna happen. They’re gonna need different… 1 2 (and versions of products) 3 4 5.
So the question became not “How do we build a gigantic megaserver that can take care of everybody?” but “How do we deploy a bunch of similar-but-not-identical servers”
But that ain’t gonna happen. They’re gonna need different… 1 2 (and versions of products) 3 4 5.
So the question became not “How do we build a gigantic megaserver that can take care of everybody?” but “How do we deploy a bunch of similar-but-not-identical servers”
But that ain’t gonna happen. They’re gonna need different… 1 2 (and versions of products) 3 4 5.
So the question became not “How do we build a gigantic megaserver that can take care of everybody?” but “How do we deploy a bunch of similar-but-not-identical servers”
Command-&-control philosophy. assume every machine updates in lockstep: cluster-oriented. \\ I want the option of telling control freak sysadmins “Sure, you can use our stuff. Just set up your own box, and hit ‘update’ manually when you see fit.” without running into situations where a config file assumes a certain version of the software and is surprised.
Cross-OS abstraction: major feature: manage Windows & UNIXes & Mac from 1 conf file \\ invent own language \\ we’ll pick 1 OS \\ Don’t need cross-OS abstraction. \\ Don’t pay for another language in learning time. \\ Want people to hack on this system as easily as possible.
Non-concurrent: updates to config not synced with updates to the software it configures, which could conceivably cause problems, for example if a new version of a package changes the meaning of a config directive.
Command-&-control philosophy. assume every machine updates in lockstep: cluster-oriented. \\ I want the option of telling control freak sysadmins “Sure, you can use our stuff. Just set up your own box, and hit ‘update’ manually when you see fit.” without running into situations where a config file assumes a certain version of the software and is surprised.
Cross-OS abstraction: major feature: manage Windows & UNIXes & Mac from 1 conf file \\ invent own language \\ we’ll pick 1 OS \\ Don’t need cross-OS abstraction. \\ Don’t pay for another language in learning time. \\ Want people to hack on this system as easily as possible.
Non-concurrent: updates to config not synced with updates to the software it configures, which could conceivably cause problems, for example if a new version of a package changes the meaning of a config directive.
Command-&-control philosophy. assume every machine updates in lockstep: cluster-oriented. \\ I want the option of telling control freak sysadmins “Sure, you can use our stuff. Just set up your own box, and hit ‘update’ manually when you see fit.” without running into situations where a config file assumes a certain version of the software and is surprised.
Cross-OS abstraction: major feature: manage Windows & UNIXes & Mac from 1 conf file \\ invent own language \\ we’ll pick 1 OS \\ Don’t need cross-OS abstraction. \\ Don’t pay for another language in learning time. \\ Want people to hack on this system as easily as possible.
Non-concurrent: updates to config not synced with updates to the software it configures, which could conceivably cause problems, for example if a new version of a package changes the meaning of a config directive.
Command-&-control philosophy. assume every machine updates in lockstep: cluster-oriented. \\ I want the option of telling control freak sysadmins “Sure, you can use our stuff. Just set up your own box, and hit ‘update’ manually when you see fit.” without running into situations where a config file assumes a certain version of the software and is surprised.
Cross-OS abstraction: major feature: manage Windows & UNIXes & Mac from 1 conf file \\ invent own language \\ we’ll pick 1 OS \\ Don’t need cross-OS abstraction. \\ Don’t pay for another language in learning time. \\ Want people to hack on this system as easily as possible.
Non-concurrent: updates to config not synced with updates to the software it configures, which could conceivably cause problems, for example if a new version of a package changes the meaning of a config directive.
buildout’s a fine development tool. I use it myself all the time. But it doesn’t work in my mass-deployment situation.
Redoes existing work. There are already excellent packages of these, QA’d by thousands of Debian users. And wherever you stop, you’re going to have some kind of dependency impedence mismatch—are you going to repackage the kernel?
At least 3 network points of failure for a default Plone buildout. About half a dozen times a week, I rescue some poor user who can’t run buildout because PyPI is down, plone.org is down, zope.org is down, or PSC is broken. You can mirror it all yourself, but geez.
On failure, breaks the site. If any of the above—or any other kind of error—happens after buildout’s begun to change things, there’s no turning back. You can’t let local admins write to buildout.cfg, because they can make it run arbitrary, crashing code during nightly unattended updates.
Package QA is lacking. There’s no vetting process for putting up new versions either; all the QA is the developer’s responsibility. Martin Aspeli recognizes this problem, saying “publishing known good sets of versions is quite painful”. (Ironically, he solved this problem by introducing yet another network service, good-py, which went down several days later.)
Not truly repeatable. Seen people put up new versions on PyPI with the same version numbers as old. So even if you pin your versions, you’re hosed.
So buildout wasn’t really suitable for unattended deployment. But what about…
buildout’s a fine development tool. I use it myself all the time. But it doesn’t work in my mass-deployment situation.
Redoes existing work. There are already excellent packages of these, QA’d by thousands of Debian users. And wherever you stop, you’re going to have some kind of dependency impedence mismatch—are you going to repackage the kernel?
At least 3 network points of failure for a default Plone buildout. About half a dozen times a week, I rescue some poor user who can’t run buildout because PyPI is down, plone.org is down, zope.org is down, or PSC is broken. You can mirror it all yourself, but geez.
On failure, breaks the site. If any of the above—or any other kind of error—happens after buildout’s begun to change things, there’s no turning back. You can’t let local admins write to buildout.cfg, because they can make it run arbitrary, crashing code during nightly unattended updates.
Package QA is lacking. There’s no vetting process for putting up new versions either; all the QA is the developer’s responsibility. Martin Aspeli recognizes this problem, saying “publishing known good sets of versions is quite painful”. (Ironically, he solved this problem by introducing yet another network service, good-py, which went down several days later.)
Not truly repeatable. Seen people put up new versions on PyPI with the same version numbers as old. So even if you pin your versions, you’re hosed.
So buildout wasn’t really suitable for unattended deployment. But what about…
buildout’s a fine development tool. I use it myself all the time. But it doesn’t work in my mass-deployment situation.
Redoes existing work. There are already excellent packages of these, QA’d by thousands of Debian users. And wherever you stop, you’re going to have some kind of dependency impedence mismatch—are you going to repackage the kernel?
At least 3 network points of failure for a default Plone buildout. About half a dozen times a week, I rescue some poor user who can’t run buildout because PyPI is down, plone.org is down, zope.org is down, or PSC is broken. You can mirror it all yourself, but geez.
On failure, breaks the site. If any of the above—or any other kind of error—happens after buildout’s begun to change things, there’s no turning back. You can’t let local admins write to buildout.cfg, because they can make it run arbitrary, crashing code during nightly unattended updates.
Package QA is lacking. There’s no vetting process for putting up new versions either; all the QA is the developer’s responsibility. Martin Aspeli recognizes this problem, saying “publishing known good sets of versions is quite painful”. (Ironically, he solved this problem by introducing yet another network service, good-py, which went down several days later.)
Not truly repeatable. Seen people put up new versions on PyPI with the same version numbers as old. So even if you pin your versions, you’re hosed.
So buildout wasn’t really suitable for unattended deployment. But what about…
buildout’s a fine development tool. I use it myself all the time. But it doesn’t work in my mass-deployment situation.
Redoes existing work. There are already excellent packages of these, QA’d by thousands of Debian users. And wherever you stop, you’re going to have some kind of dependency impedence mismatch—are you going to repackage the kernel?
At least 3 network points of failure for a default Plone buildout. About half a dozen times a week, I rescue some poor user who can’t run buildout because PyPI is down, plone.org is down, zope.org is down, or PSC is broken. You can mirror it all yourself, but geez.
On failure, breaks the site. If any of the above—or any other kind of error—happens after buildout’s begun to change things, there’s no turning back. You can’t let local admins write to buildout.cfg, because they can make it run arbitrary, crashing code during nightly unattended updates.
Package QA is lacking. There’s no vetting process for putting up new versions either; all the QA is the developer’s responsibility. Martin Aspeli recognizes this problem, saying “publishing known good sets of versions is quite painful”. (Ironically, he solved this problem by introducing yet another network service, good-py, which went down several days later.)
Not truly repeatable. Seen people put up new versions on PyPI with the same version numbers as old. So even if you pin your versions, you’re hosed.
So buildout wasn’t really suitable for unattended deployment. But what about…
buildout’s a fine development tool. I use it myself all the time. But it doesn’t work in my mass-deployment situation.
Redoes existing work. There are already excellent packages of these, QA’d by thousands of Debian users. And wherever you stop, you’re going to have some kind of dependency impedence mismatch—are you going to repackage the kernel?
At least 3 network points of failure for a default Plone buildout. About half a dozen times a week, I rescue some poor user who can’t run buildout because PyPI is down, plone.org is down, zope.org is down, or PSC is broken. You can mirror it all yourself, but geez.
On failure, breaks the site. If any of the above—or any other kind of error—happens after buildout’s begun to change things, there’s no turning back. You can’t let local admins write to buildout.cfg, because they can make it run arbitrary, crashing code during nightly unattended updates.
Package QA is lacking. There’s no vetting process for putting up new versions either; all the QA is the developer’s responsibility. Martin Aspeli recognizes this problem, saying “publishing known good sets of versions is quite painful”. (Ironically, he solved this problem by introducing yet another network service, good-py, which went down several days later.)
Not truly repeatable. Seen people put up new versions on PyPI with the same version numbers as old. So even if you pin your versions, you’re hosed.
So buildout wasn’t really suitable for unattended deployment. But what about…
buildout’s a fine development tool. I use it myself all the time. But it doesn’t work in my mass-deployment situation.
Redoes existing work. There are already excellent packages of these, QA’d by thousands of Debian users. And wherever you stop, you’re going to have some kind of dependency impedence mismatch—are you going to repackage the kernel?
At least 3 network points of failure for a default Plone buildout. About half a dozen times a week, I rescue some poor user who can’t run buildout because PyPI is down, plone.org is down, zope.org is down, or PSC is broken. You can mirror it all yourself, but geez.
On failure, breaks the site. If any of the above—or any other kind of error—happens after buildout’s begun to change things, there’s no turning back. You can’t let local admins write to buildout.cfg, because they can make it run arbitrary, crashing code during nightly unattended updates.
Package QA is lacking. There’s no vetting process for putting up new versions either; all the QA is the developer’s responsibility. Martin Aspeli recognizes this problem, saying “publishing known good sets of versions is quite painful”. (Ironically, he solved this problem by introducing yet another network service, good-py, which went down several days later.)
Not truly repeatable. Seen people put up new versions on PyPI with the same version numbers as old. So even if you pin your versions, you’re hosed.
So buildout wasn’t really suitable for unattended deployment. But what about…
buildout’s a fine development tool. I use it myself all the time. But it doesn’t work in my mass-deployment situation.
Redoes existing work. There are already excellent packages of these, QA’d by thousands of Debian users. And wherever you stop, you’re going to have some kind of dependency impedence mismatch—are you going to repackage the kernel?
At least 3 network points of failure for a default Plone buildout. About half a dozen times a week, I rescue some poor user who can’t run buildout because PyPI is down, plone.org is down, zope.org is down, or PSC is broken. You can mirror it all yourself, but geez.
On failure, breaks the site. If any of the above—or any other kind of error—happens after buildout’s begun to change things, there’s no turning back. You can’t let local admins write to buildout.cfg, because they can make it run arbitrary, crashing code during nightly unattended updates.
Package QA is lacking. There’s no vetting process for putting up new versions either; all the QA is the developer’s responsibility. Martin Aspeli recognizes this problem, saying “publishing known good sets of versions is quite painful”. (Ironically, he solved this problem by introducing yet another network service, good-py, which went down several days later.)
Not truly repeatable. Seen people put up new versions on PyPI with the same version numbers as old. So even if you pin your versions, you’re hosed.
So buildout wasn’t really suitable for unattended deployment. But what about…
We need them anyway to manage the kernel, libraries, and basic services.
Unbeatable QA. Just outrageous. Debian has 3 QA tiers: unstable, testing, stable. Immediate, 10 days, once every year and a half. We run stable. Actually, we’re one behind, but we still get another year of full security support.
Nearly atomic. High-level stuff like Apache gets updated at darn close to the same time as low-level stuff like the libraries it depends on, making for fewer states. And fewer states means fewer unexpected behaviors.
Tolerance of local changes. APT has been around since 1998 and is very mature. It has this sophisticated framework for tolerating local config changes during upgrades. No paving \\ asks
Reliable. Downloads everything before changing anything. If something’s unreachable, the stuff that depends on it doesn’t happen. And if anything unexpected happens during installation…
We need them anyway to manage the kernel, libraries, and basic services.
Unbeatable QA. Just outrageous. Debian has 3 QA tiers: unstable, testing, stable. Immediate, 10 days, once every year and a half. We run stable. Actually, we’re one behind, but we still get another year of full security support.
Nearly atomic. High-level stuff like Apache gets updated at darn close to the same time as low-level stuff like the libraries it depends on, making for fewer states. And fewer states means fewer unexpected behaviors.
Tolerance of local changes. APT has been around since 1998 and is very mature. It has this sophisticated framework for tolerating local config changes during upgrades. No paving \\ asks
Reliable. Downloads everything before changing anything. If something’s unreachable, the stuff that depends on it doesn’t happen. And if anything unexpected happens during installation…
We need them anyway to manage the kernel, libraries, and basic services.
Unbeatable QA. Just outrageous. Debian has 3 QA tiers: unstable, testing, stable. Immediate, 10 days, once every year and a half. We run stable. Actually, we’re one behind, but we still get another year of full security support.
Nearly atomic. High-level stuff like Apache gets updated at darn close to the same time as low-level stuff like the libraries it depends on, making for fewer states. And fewer states means fewer unexpected behaviors.
Tolerance of local changes. APT has been around since 1998 and is very mature. It has this sophisticated framework for tolerating local config changes during upgrades. No paving \\ asks
Reliable. Downloads everything before changing anything. If something’s unreachable, the stuff that depends on it doesn’t happen. And if anything unexpected happens during installation…
We need them anyway to manage the kernel, libraries, and basic services.
Unbeatable QA. Just outrageous. Debian has 3 QA tiers: unstable, testing, stable. Immediate, 10 days, once every year and a half. We run stable. Actually, we’re one behind, but we still get another year of full security support.
Nearly atomic. High-level stuff like Apache gets updated at darn close to the same time as low-level stuff like the libraries it depends on, making for fewer states. And fewer states means fewer unexpected behaviors.
Tolerance of local changes. APT has been around since 1998 and is very mature. It has this sophisticated framework for tolerating local config changes during upgrades. No paving \\ asks
Reliable. Downloads everything before changing anything. If something’s unreachable, the stuff that depends on it doesn’t happen. And if anything unexpected happens during installation…
We need them anyway to manage the kernel, libraries, and basic services.
Unbeatable QA. Just outrageous. Debian has 3 QA tiers: unstable, testing, stable. Immediate, 10 days, once every year and a half. We run stable. Actually, we’re one behind, but we still get another year of full security support.
Nearly atomic. High-level stuff like Apache gets updated at darn close to the same time as low-level stuff like the libraries it depends on, making for fewer states. And fewer states means fewer unexpected behaviors.
Tolerance of local changes. APT has been around since 1998 and is very mature. It has this sophisticated framework for tolerating local config changes during upgrades. No paving \\ asks
Reliable. Downloads everything before changing anything. If something’s unreachable, the stuff that depends on it doesn’t happen. And if anything unexpected happens during installation…
We need them anyway to manage the kernel, libraries, and basic services.
Unbeatable QA. Just outrageous. Debian has 3 QA tiers: unstable, testing, stable. Immediate, 10 days, once every year and a half. We run stable. Actually, we’re one behind, but we still get another year of full security support.
Nearly atomic. High-level stuff like Apache gets updated at darn close to the same time as low-level stuff like the libraries it depends on, making for fewer states. And fewer states means fewer unexpected behaviors.
Tolerance of local changes. APT has been around since 1998 and is very mature. It has this sophisticated framework for tolerating local config changes during upgrades. No paving \\ asks
Reliable. Downloads everything before changing anything. If something’s unreachable, the stuff that depends on it doesn’t happen. And if anything unexpected happens during installation…
We need them anyway to manage the kernel, libraries, and basic services.
Unbeatable QA. Just outrageous. Debian has 3 QA tiers: unstable, testing, stable. Immediate, 10 days, once every year and a half. We run stable. Actually, we’re one behind, but we still get another year of full security support.
Nearly atomic. High-level stuff like Apache gets updated at darn close to the same time as low-level stuff like the libraries it depends on, making for fewer states. And fewer states means fewer unexpected behaviors.
Tolerance of local changes. APT has been around since 1998 and is very mature. It has this sophisticated framework for tolerating local config changes during upgrades. No paving \\ asks
Reliable. Downloads everything before changing anything. If something’s unreachable, the stuff that depends on it doesn’t happen. And if anything unexpected happens during installation…
This is a breakdown of how the APT system installs or upgrades a package. Each smiley face marks a point where something might go wrong, and there’s a remediation step to return things to a working state.
And it’s not until way down here at this big red line that you’re committed to the upgrade; it can roll back at any point before that.
Imagine if buildout did this! Imagine how many fewer people we’d have showing up in the #plone channel screaming about how it broke their install!
hosting-node: Everything that should be on every box \\ Kerberos, ssh, ntp, kernel upgrader, sudo, snmpd \\ Want something on all the boxes? Add it to this thing’s dependencies.
auto-update: Don’t want it? Don’t install it. Nightly automatics 4-5am.
plone-3.1-stack: All the rest \\ Packaged Plone.
center: “config packages” \\ shiny new way to package config using framework \\ Tim Abbott @ MIT
massdeploy \\ I mean…
config-package-dev
0:20
hosting-node: Everything that should be on every box \\ Kerberos, ssh, ntp, kernel upgrader, sudo, snmpd \\ Want something on all the boxes? Add it to this thing’s dependencies.
auto-update: Don’t want it? Don’t install it. Nightly automatics 4-5am.
plone-3.1-stack: All the rest \\ Packaged Plone.
center: “config packages” \\ shiny new way to package config using framework \\ Tim Abbott @ MIT
massdeploy \\ I mean…
config-package-dev
0:20
hosting-node: Everything that should be on every box \\ Kerberos, ssh, ntp, kernel upgrader, sudo, snmpd \\ Want something on all the boxes? Add it to this thing’s dependencies.
auto-update: Don’t want it? Don’t install it. Nightly automatics 4-5am.
plone-3.1-stack: All the rest \\ Packaged Plone.
center: “config packages” \\ shiny new way to package config using framework \\ Tim Abbott @ MIT
massdeploy \\ I mean…
config-package-dev
0:20
hosting-node: Everything that should be on every box \\ Kerberos, ssh, ntp, kernel upgrader, sudo, snmpd \\ Want something on all the boxes? Add it to this thing’s dependencies.
auto-update: Don’t want it? Don’t install it. Nightly automatics 4-5am.
plone-3.1-stack: All the rest \\ Packaged Plone.
center: “config packages” \\ shiny new way to package config using framework \\ Tim Abbott @ MIT
massdeploy \\ I mean…
config-package-dev
0:20
hosting-node: Everything that should be on every box \\ Kerberos, ssh, ntp, kernel upgrader, sudo, snmpd \\ Want something on all the boxes? Add it to this thing’s dependencies.
auto-update: Don’t want it? Don’t install it. Nightly automatics 4-5am.
plone-3.1-stack: All the rest \\ Packaged Plone.
center: “config packages” \\ shiny new way to package config using framework \\ Tim Abbott @ MIT
massdeploy \\ I mean…
config-package-dev
0:20
hosting-node: Everything that should be on every box \\ Kerberos, ssh, ntp, kernel upgrader, sudo, snmpd \\ Want something on all the boxes? Add it to this thing’s dependencies.
auto-update: Don’t want it? Don’t install it. Nightly automatics 4-5am.
plone-3.1-stack: All the rest \\ Packaged Plone.
center: “config packages” \\ shiny new way to package config using framework \\ Tim Abbott @ MIT
massdeploy \\ I mean…
config-package-dev
0:20
hosting-node: Everything that should be on every box \\ Kerberos, ssh, ntp, kernel upgrader, sudo, snmpd \\ Want something on all the boxes? Add it to this thing’s dependencies.
auto-update: Don’t want it? Don’t install it. Nightly automatics 4-5am.
plone-3.1-stack: All the rest \\ Packaged Plone.
center: “config packages” \\ shiny new way to package config using framework \\ Tim Abbott @ MIT
massdeploy \\ I mean…
config-package-dev
0:20
hosting-node: Everything that should be on every box \\ Kerberos, ssh, ntp, kernel upgrader, sudo, snmpd \\ Want something on all the boxes? Add it to this thing’s dependencies.
auto-update: Don’t want it? Don’t install it. Nightly automatics 4-5am.
plone-3.1-stack: All the rest \\ Packaged Plone.
center: “config packages” \\ shiny new way to package config using framework \\ Tim Abbott @ MIT
massdeploy \\ I mean…
config-package-dev
0:20
hosting-node: Everything that should be on every box \\ Kerberos, ssh, ntp, kernel upgrader, sudo, snmpd \\ Want something on all the boxes? Add it to this thing’s dependencies.
auto-update: Don’t want it? Don’t install it. Nightly automatics 4-5am.
plone-3.1-stack: All the rest \\ Packaged Plone.
center: “config packages” \\ shiny new way to package config using framework \\ Tim Abbott @ MIT
massdeploy \\ I mean…
config-package-dev
0:20
hosting-node: Everything that should be on every box \\ Kerberos, ssh, ntp, kernel upgrader, sudo, snmpd \\ Want something on all the boxes? Add it to this thing’s dependencies.
auto-update: Don’t want it? Don’t install it. Nightly automatics 4-5am.
plone-3.1-stack: All the rest \\ Packaged Plone.
center: “config packages” \\ shiny new way to package config using framework \\ Tim Abbott @ MIT
massdeploy \\ I mean…
config-package-dev
0:20
hosting-node: Everything that should be on every box \\ Kerberos, ssh, ntp, kernel upgrader, sudo, snmpd \\ Want something on all the boxes? Add it to this thing’s dependencies.
auto-update: Don’t want it? Don’t install it. Nightly automatics 4-5am.
plone-3.1-stack: All the rest \\ Packaged Plone.
center: “config packages” \\ shiny new way to package config using framework \\ Tim Abbott @ MIT
massdeploy \\ I mean…
config-package-dev
0:20
hosting-node: Everything that should be on every box \\ Kerberos, ssh, ntp, kernel upgrader, sudo, snmpd \\ Want something on all the boxes? Add it to this thing’s dependencies.
auto-update: Don’t want it? Don’t install it. Nightly automatics 4-5am.
plone-3.1-stack: All the rest \\ Packaged Plone.
center: “config packages” \\ shiny new way to package config using framework \\ Tim Abbott @ MIT
massdeploy \\ I mean…
config-package-dev
0:20
wanted common caretaking system for Plone, Apache, Squid; kernel, libraries\\ buildout power users worked toward but couldn’t take the whole way \\ config-package-dev brings final piece
Frankly, starting with Debian \\ already packages everything \\ adding Plone \\ easier than starting with buildout \\ packages Plone \\ trying to add everything else in the world.
wanted common caretaking system for Plone, Apache, Squid; kernel, libraries\\ buildout power users worked toward but couldn’t take the whole way \\ config-package-dev brings final piece
Frankly, starting with Debian \\ already packages everything \\ adding Plone \\ easier than starting with buildout \\ packages Plone \\ trying to add everything else in the world.
wanted common caretaking system for Plone, Apache, Squid; kernel, libraries\\ buildout power users worked toward but couldn’t take the whole way \\ config-package-dev brings final piece
Frankly, starting with Debian \\ already packages everything \\ adding Plone \\ easier than starting with buildout \\ packages Plone \\ trying to add everything else in the world.
wanted common caretaking system for Plone, Apache, Squid; kernel, libraries\\ buildout power users worked toward but couldn’t take the whole way \\ config-package-dev brings final piece
Frankly, starting with Debian \\ already packages everything \\ adding Plone \\ easier than starting with buildout \\ packages Plone \\ trying to add everything else in the world.
wanted common caretaking system for Plone, Apache, Squid; kernel, libraries\\ buildout power users worked toward but couldn’t take the whole way \\ config-package-dev brings final piece
Frankly, starting with Debian \\ already packages everything \\ adding Plone \\ easier than starting with buildout \\ packages Plone \\ trying to add everything else in the world.
auto-update: screwing with cron-apt
squid-config: one conf file to rule them all
plone-site-config: listen on localhost, hook up to ZEO, restart leaky Zope, pack DB
Not on this diagram:
weblion-krb5-config
weblion-snmpd-config
weblion-ssh-server-config
Crown jewel: apache-config
“primary” vhost \\ full of includes \\ fill out tiny conffiles included by the vhost \\ contracts \\ all made out of includes
Example fixes so far: HTTP_REMOTE_USER hole, route auth’d stuff through Squid. \\ pattern worked really well. recommend.
additional vhosts \\ alias vhosts
“primary” vhost \\ full of includes \\ fill out tiny conffiles included by the vhost \\ contracts \\ all made out of includes
Example fixes so far: HTTP_REMOTE_USER hole, route auth’d stuff through Squid. \\ pattern worked really well. recommend.
additional vhosts \\ alias vhosts
“primary” vhost \\ full of includes \\ fill out tiny conffiles included by the vhost \\ contracts \\ all made out of includes
Example fixes so far: HTTP_REMOTE_USER hole, route auth’d stuff through Squid. \\ pattern worked really well. recommend.
additional vhosts \\ alias vhosts
“primary” vhost \\ full of includes \\ fill out tiny conffiles included by the vhost \\ contracts \\ all made out of includes
Example fixes so far: HTTP_REMOTE_USER hole, route auth’d stuff through Squid. \\ pattern worked really well. recommend.
additional vhosts \\ alias vhosts
new stuff enters @ unstable after as much testing as possible \\ test for clean upgrade \\ move to testing \\ testing moves as whole to stable
When we get to lenny \\ etch -> lenny-unstable \\ work its way up
new stuff enters @ unstable after as much testing as possible \\ test for clean upgrade \\ move to testing \\ testing moves as whole to stable
When we get to lenny \\ etch -> lenny-unstable \\ work its way up
new stuff enters @ unstable after as much testing as possible \\ test for clean upgrade \\ move to testing \\ testing moves as whole to stable
When we get to lenny \\ etch -> lenny-unstable \\ work its way up
new stuff enters @ unstable after as much testing as possible \\ test for clean upgrade \\ move to testing \\ testing moves as whole to stable
When we get to lenny \\ etch -> lenny-unstable \\ work its way up
new stuff enters @ unstable after as much testing as possible \\ test for clean upgrade \\ move to testing \\ testing moves as whole to stable
When we get to lenny \\ etch -> lenny-unstable \\ work its way up
new stuff enters @ unstable after as much testing as possible \\ test for clean upgrade \\ move to testing \\ testing moves as whole to stable
When we get to lenny \\ etch -> lenny-unstable \\ work its way up