The document describes the problems with traditional ad hoc systems administration and software deployment methods. It proposes implementing three main components to standardize the process: 1. A package repository to store all software packages in a centralized, version controlled location. 2. A recipe store containing scripts to automate software installation from the package repository in a standardized way. 3. A meta-data repository to describe the infrastructure and direct automated installs by specifying which recipes and package versions to use for each host. This approach aims to eliminate inconsistencies between systems and human errors by fully automating software deployment and configuration management from authoritative central sources.

1. The Problem

2. Our intrepid systems administrator installs the
operating system from disk

3. From other disks, the anti-virus, office suite,and
other vendor-provided applications the system
needs are loaded.

4. Patches for the operating system and applications
are downloaded, either directly or indirectly, and
are applied to the system.

5. What's wrong whit that?
● The CDROMs may be different versions on
different installs
● The patches from external sources will amost
certainly be different on different installs
● The systems administrator may not put systems
together in an identical manner
● Basically no two systems will be alike
● This will result in two “identical” systems with
different behavior

6. The system is released to developers, who write
and compile code, and copy it to the system.

7. I said “developers.”
(you could add DBAs and application owners to
this list, but you see where I'm going)

8. What's wrong with that?
● It almost guarantees that different code will be
deployed to different “identical” systems
● If a system “blows up” and needs to be rebuilt,
you have to throw operations resources and
development resources at it to get it running
again.
● If you need to re-create a system “as-built” you
have to find all the sources again (this just
won't happen).

9. For the next system, wash, rinse repeat.

10. And so on, and so on, until the work not only fills
all the time of all the bodies, but additional bodies
are needed to keep up with the work.

11. Welcome to “meat cloud” computing.
(also, you'll need managers, “senior” managers to
manage them, and “Directors” to manage them.)

12. All the while, no one will really have any idea
what's going on, (beyond a holistic level)

13. But what can you do?

14. Identify your “authoritative” sources of information.

15. Found 'em! Now what?

16. Umm, I think I may see the problem...

17. Let's start over...

18. What if, instead of installing operating systems
and applications striaght from media, the
sysadmin stored them in a “repository”?

19. And then, any time a system was built, a script
would grab those packages, and install the
operating system, and the applications?

20. One small script (or recipe) for each application,
describing how it should be installed.

21. Now the application (package) repository will
always be changing, as will the recipies that install
the applications (packages)...

22. A “version control system” (vcs) shall be
maintained for both the package repositories and
the installation scripts.

23. A host “meta-data” repository will instruct from
which repository to pull packages, and which
scripts to use to install and configure them.

24. Patches for the Operating Systems and
applications could be staged in the appropriate
repositories, and scripts written to install them.

25. Developers would need to package their
applications for installation, and provide
automated scripts for installation.

26. Now when boxes need built or rebuilt, no human
resources are needed other than to initiate the
process.

27. Now identify your “authoritative” sources of
information...

28. Found them!

29. Now that's more like it.

30. There are 3 major components
● The package repository
● The recipe store
● The meta-data repository

31. The Package Repository
● Nothing gets installed to a system, by hand or
in an auotmated fashion without the installer
being copied to the package repository first.
● The repository will be version controlled by
cloning with hard-links, and then and only then
adding new content.
● The repositories versions will be given unique
IDs that can be linked to hosts in the meta-data
repository
● A host shall only pull from one repository, even
if a new one needs to be cloned to satisfy this.

32. The Recipe Store
● The recipe store will contain recipes for
installing any and everything we install.
● Every component will have a unique ID that can
be tied to a host using the meta-data repository
● All recipies will be version controlled.
● No items unique to a host will go in the recipe
store.
● Items unique to a host will be pulled from the
meta-data repository and “merged” with the
recipe to complete the configuration

33. The meta-data repository
● The meta-data repository will describe the
infrastructure at the highest level
● The meta-data repository must be able to be
queried by any script or application, or by hand
● It must conform to an open standard
● It must be network facing (no application tiers)
● It must be secure (SSL/TLS)
● It must handle authentication and authorization
internally

34. Additional components
● Monitoring rules may be added to the recipe
store such that hosts tied to the monitoring
rules via the meta-data repository will be
monitored in the manner described in the store.
● Back-up rules may be added to the recipie
store such that hosts tied to the back-up rules
via the meta-data repository will back up
accordingly
● Any other rules may be set up in the same
manner...

35. Process flow

36. The players

37. On the “Install Inititiator”, the command is given to deploy the
“Target Host.” The initiator queries the meta-data repository for the
specifics of the install

38. The meta-data repository replies with, what to install, where, and
the personalization information specific to the host.

39. The install initiator (using the meta-data) instructs the target host
which recipies to use to install which packages from a specific
repository.

40. The target host pulls the appropriate packages from the package
repository, and the installation recipies and does the work needed
to install itself.

41. Once installation is complete the target host routinely queries the
meta-data-repository and the recipie store for changes, and does
any work defined therein. It is now in “configuration management”
mode.

42. If the recipe store or the meta-data repository indicate that the
target host should point to a new repository, then the target host
will update it's packages accordingly. This is how patches are
applied.

43. Design Terminology
● The “package repository” may be an apt
repository, yum repository, a CIFS share, or a
combination of them, depending on the
operating system(s) you need to install
● The “recipe store” may be version-controlled
shell scripts, batch files, cfengine policies,
puppet manifests, or whatever you need to get
the job done.
● The “meta-data repository” can be an LDAP
server (or a set of them) or a bunch of YAML
files in git/svn, again, whatever you need...

44. Questions?