Asset and Conﬁgura/on Tracking: the most boring topic in the Universe Greg Lindahl CTO, blekko
The problem(s)• 100s of servers, 1000s of disks, 100s of desktops, 100s of laptops• Some of the servers are leased. Disks move around constantly. Your ﬁnancial people want to do an audit. Your startup ﬁnally is old enough to need asset tags. blah blah blah…• How many hosts got that new kernel, and when? What soOware is installed on that host that’s down? What’s the up/me history of the en/re cluster?• Did my integrator screw up and return broken parts instead of sending them for repair? 1/3
System requirements• Based on tools I already have• Dead simple• Support date searches, showing when a string appears and disappears from the database• I know what you’re thinking… I mean something simpler than that.
Global part• DVCS -‐-‐ mercurial, in my case• Collect text ﬁles for every host in a separate directory• Check in ﬁles daily• hg log -‐p to see changes, global or per-‐host• hg grep -‐-‐all -‐f -‐d 6QG06ZQN• (put the repo on an SSD…) 1/3
Local part: every server• Create a script to generate text ﬁles containing: – MAC addrs, running kernel version, rpms installed, disk serial #s, dmidecode, installed sysctl, installed ﬁrewall conﬁg, ipmi info, vendor info (hpdiscovery)• Important: these ﬁles should only change if there’s a change. Don’t store up/me, store the boot /me.• Commit these ﬁles locally 1/3
Tie them together• Copy the local ﬁles to a central server, check in there. Push to several backup repos.• See, now isn’t this simpler than what you were thinking? NoSQL.
What about laptops?• Macs are Unix, done.• mercurial runs under Windoze, too• They need their own data collectors.• The only trick is delivering ﬁles to the mothership on a regular basis. 1/2
slash the web! • yes, we’re hiring • vote for us in the Crunchies• check out: /linux, /python, /ruby, /…