• Like
  • Save
DST @ Yahoo!
Upcoming SlideShare
Loading in...5
×
 

DST @ Yahoo!

on

  • 553 views

A presentation on how changes in Daylight Saving Time were handled at Yahoo!. Originally given at BayLISA in May 2007. Slides are available here:

A presentation on how changes in Daylight Saving Time were handled at Yahoo!. Originally given at BayLISA in May 2007. Slides are available here:

netmeister.org/misc/dst_yahoo.pdf

Statistics

Views

Total Views
553
Views on SlideShare
532
Embed Views
21

Actions

Likes
0
Downloads
1
Comments
0

2 Embeds 21

http://www.linkedin.com 12
http://coderwall.com 9

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    DST @ Yahoo! DST @ Yahoo! Presentation Transcript

    • DST ’07 @ Yahoo! Slide 1 DST ’07 @ Yahoo! Jan Schaumann jschauma@netmeister.org PGP: 136D 027F DC29 8402 7B42 47D6 7C5B 64AF AF22 6A4CJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 2whoami$ ME=$(id -un)$ grep ${ME} /etc/passwd | cut -d: -f5Jan Schaumann$Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 3whoami$ ME=$(id -un)$ grep ${ME} /etc/passwd | cut -d: -f5Jan Schaumann$ groups ${ME}netbsd sa yahoo$Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 4whoami$ ME=$(id -un)$ grep ${ME} /etc/passwd | cut -d: -f5Jan Schaumann$ groups ${ME}netbsd sa yahoo$Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 5whoami$ ME=$(id -un)$ grep ${ME} /etc/passwd | cut -d: -f5Jan Schaumann$ groups ${ME}netbsd sa yahoo$Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 6whoami$ ME=$(id -un)$ grep ${ME} /etc/passwd | cut -d: -f5Jan Schaumann$ groups ${ME}netbsd sa yahoo$Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 7whoami$ ME=$(id -un)$ grep ${ME} /etc/passwd | cut -d: -f5Jan Schaumann$ groups ${ME}netbsd sa yahoo$Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 8whoami$ ME=$(id -un)$ awk -F: -v ME=${ME} ’$0 ~ ME { print $5 }’ /etc/passwdJan Schaumann$ groups ${ME}netbsd sa yahoo$Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 9whoami$ ME=$(id -un)$ grep ${ME} /etc/passwd | cut -d: -f5Jan Schaumann$ groups ${ME}netbsd sa yahoo$Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 10whoami$ ME=$(id -un)$ grep ${ME} /etc/passwd | cut -d: -f5Jan Schaumann$ groups ${ME}netbsd sa yahoo$Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 11 DST ’07 @ Yahoo! – Saving the living daylight out of you! or How the YSA team stopped worrying and learned to love UTC!Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 12In the beginning......the universe was created.Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 13In the beginning......the universe was created.“This has made a lot of people very angry and been widely regarded as a bad move.”Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 14Daylight Saving Time for DummiesJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 15Daylight Saving Time for DummiesJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 16Daylight Saving Time for DummiesJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 17Daylight Saving Time for DummiesJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 18Daylight Saving Time for DummiesJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 19Daylight Saving Time for DummiesJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 20Daylight Saving Time for DummiesJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 21Fast forward to 2005...Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 22Fast forward to 2005...Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 23Fast forward to 2005...Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 24March 11th, 2007 In 2007, Daylight Saving Time will start on March 11th.Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 25March 11thJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 26March 11thJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 27August 8th, 2005Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 28One day later... Yahoo! decides to take immediate action!Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 29August, 29th, 2005Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 30December, 22nd, 2005Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 31August 8th, 2005Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 32One day later... Yahoo! decides to take immediate action!Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 33Alas...Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 34“Persistence of Memory”Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 35Fast forward to January 2007... “Hey, we need to know which hosts can’t deal with the new DST.”Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 36Script 0 See remote0.sh (and remote0.5.sh).Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 37January 2007 need to know hosts that can’t deal with new DSTJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 38January 2007 need to know hosts that can’t deal with new DST need to run a short command on each hostJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 39January 2007 need to know hosts that can’t deal with new DST need to run a short command on each hostSo: write a script that runs a single command on hosts given as input.Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 40January 2007 need to know hosts that can’t deal with new DST need to run a short command on each hostSo: write a script that runs a single command on hosts given as input.$ cat >checkhosts.sh <<EOS#!/bin/shwhile read host; do ssh $host "single-command"done < $1EOS$Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 41“single command”TIMTOWTDI: TZ=PST8PDT date -j 200703110200Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 42“single command”TIMTOWTDI (or is there?): TZ=PST8PDT date -j 200703110200 – conflicting flags and return values on different platformsJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 43“single command”TIMTOWTDI (or is there?): TZ=PST8PDT date -j 200703110200 – conflicting flags and return values on different platforms zdump -v PST8PDT | grep "2007 UTC = Sun Mar 11"Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 44“single command”TIMTOWTDI (or is there?): TZ=PST8PDT date -j 200703110200 – conflicting flags and return values on different platforms zdump -v PST8PDT | grep "2007 UTC = Sun Mar 11" – triggers a bug in FreeBSD/amd64 and zdump spins out of controlJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 45“single command”TIMTOWTDI (or is there?): TZ=PST8PDT date -j 200703110200 – conflicting flags and return values on different platforms zdump -v PST8PDT | grep "2007 UTC = Sun Mar 11" – triggers a bug in FreeBSD/amd64 and zdump spins out of control env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 46“single command”TIMTOWTDI (or is there?): TZ=PST8PDT date -j 200703110200 – conflicting flags and return values on different platforms zdump -v PST8PDT | grep "2007 UTC = Sun Mar 11" – triggers a bug in FreeBSD/amd64 and zdump spins out of control env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’ – Hooray!Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 47“single command”TIMTOWTDI (or is there?): TZ=PST8PDT date -j 200703110200 – conflicting flags and return values on different platforms zdump -v PST8PDT | grep "2007 UTC = Sun Mar 11" – triggers a bug in FreeBSD/amd64 and zdump spins out of control env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’ – Hooray!Yes, we happen to have perl installed on every machine.Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 48Script I See remote1.sh.Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 49So far, so good!We now have a list of hostnames, sorted by: operating system need to patchJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 50So far, so good!We now have a list of hostnames, sorted by: operating system need to patchSurely, our work here is done.Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 51So far, so good!We now have a list of hostnames, sorted by: operating system need to patchSurely, our work here is done.Well, not entirely...Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 52Forward to middle of February 2007...Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 53Forward to middle of February 2007... detailed communication with properties about impact gather information from vendors select patching full-time focus on data collectionJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 54Data Collection need to know hosts that can’t deal with new DST need to run a short command on each host env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 55Data Collection need to know hosts that can’t deal with new DST need to run a short command on each host env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’ need to know what packages are installedJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 56Data Collection need to know hosts that can’t deal with new DST need to run a short command on each host env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’ need to know what packages are installed need to know timestamp of a certain fileJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 57Data Collection need to know hosts that can’t deal with new DST need to run a short command on each host env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’ need to know what packages are installed need to know timestamp of a certain file need to know if client is running oracleJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 58Data Collection need to know hosts that can’t deal with new DST need to run a short command on each host env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’ need to know what packages are installed need to know timestamp of a certain file need to know if client is running oracleSo: revamp tool to not run hard-coded command, but any given script.Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 59Script II See remote2.sh and remote3.sh.Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 60Data Collection: Input input from central databaseJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 61Data Collection: Input input from central database input from DNS zone filesJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 62Data Collection: Input input from central database input from DNS zone files prepare exclusion lists exclude windows hosts exclude routers, switches etc. exclude NetApp filers exclude hosts in corp exclude DNS rotations, decomissioned hosts, placeholder DNS entries etc. ...Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 63Data Collection: Output a lot of information is collected → output per host fairly large sort output by patch-needed sort output by OS sort output by propertyJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 64Patch actions FreeBSD Linux SolarisJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 65Patch actions FreeBSD all versions prior to a certain 4.11 based image require patch Linux SolarisJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 66Patch actions FreeBSD all versions prior to a certain 4.11 based image require patch some versions higher than that (including some 6.x based images) may require a patch Linux SolarisJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 67Patch actions FreeBSD all versions prior to a certain 4.11 based image require patch some versions higher than that (including some 6.x based images) may require a patch upgrade to latest 6.x based version recommended Linux SolarisJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 68Patch actions FreeBSD all versions prior to a certain 4.11 based image require patch some versions higher than that (including some 6.x based images) may require a patch upgrade to latest 6.x based version recommended patch installation innocuous → no reboot required Linux SolarisJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 69Patch actions FreeBSD all versions prior to a certain 4.11 based image require patch some versions higher than that (including some 6.x based images) may require a patch upgrade to latest 6.x based version recommended patch installation innocuous → no reboot required Linux no change necessary for versions >= RHEL3u7, RHEL4u3 SolarisJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 70Patch actions FreeBSD all versions prior to a certain 4.11 based image require patch some versions higher than that (including some 6.x based images) may require a patch upgrade to latest 6.x based version recommended patch installation innocuous → no reboot required Linux no change necessary for versions >= RHEL3u7, RHEL4u3 installation of rpm tzdata for other versions RHEL3 and RHEL 4 SolarisJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 71Patch actions FreeBSD all versions prior to a certain 4.11 based image require patch some versions higher than that (including some 6.x based images) may require a patch upgrade to latest 6.x based version recommended patch installation innocuous → no reboot required Linux no change necessary for versions >= RHEL3u7, RHEL4u3 installation of rpm tzdata for other versions RHEL3 and RHEL 4 upgrade to latest RHEL release for all other rpm based Linux versions recommend SolarisJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 72Patch actions FreeBSD all versions prior to a certain 4.11 based image require patch some versions higher than that (including some 6.x based images) may require a patch upgrade to latest 6.x based version recommended patch installation innocuous → no reboot required Linux no change necessary for versions >= RHEL3u7, RHEL4u3 installation of rpm tzdata for other versions RHEL3 and RHEL 4 upgrade to latest RHEL release for all other rpm based Linux versions recommend forced (ugh!) installation of tzdata rpm possible SolarisJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 73Patch actions FreeBSD all versions prior to a certain 4.11 based image require patch some versions higher than that (including some 6.x based images) may require a patch upgrade to latest 6.x based version recommended patch installation innocuous → no reboot required Linux no change necessary for versions >= RHEL3u7, RHEL4u3 installation of rpm tzdata for other versions RHEL3 and RHEL 4 upgrade to latest RHEL release for all other rpm based Linux versions recommend forced (ugh!) installation of tzdata rpm possible Solaris Oy vey!Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 74TZDATA everywhere! ICU (Internationalization Components for Unicode) JVM MySQL Oracle Perl DateTimeJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 75TZDATA everywhere! ICU (Internationalization Components for Unicode) need to update ICU to >= 3.6 need to recompile apps linked against libicu∗ JVM MySQL Oracle Perl DateTimeJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 76TZDATA everywhere! ICU (Internationalization Components for Unicode) need to update ICU to >= 3.6 need to recompile apps linked against libicu∗ JVM need to update to recent version March 8th (!): new bug in JDK’s found (for EST, MST, and HST zones); no vendor patch but workaround solution MySQL Oracle Perl DateTimeJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 77TZDATA everywhere! ICU (Internationalization Components for Unicode) need to update ICU to >= 3.6 need to recompile apps linked against libicu∗ JVM need to update to recent version March 8th (!): new bug in JDK’s found (for EST, MST, and HST zones); no vendor patch but workaround solution MySQL restart only for MySQL 4.0 upgrade “mysql tzdata” package for MySQL 4.1Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 78TZDATA everywhere! Oracle patch following instructions provided by vendor March 3rd: new patch → repatch! Perl DateTimeJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 79TZDATA everywhere! Oracle patch following instructions provided by vendor March 3rd: new patch → repatch! Perl DateTime update to >= 0.38 total of < 100 hosts using this packageJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 80Post-patch actions if possible, reboot verify that host displays correct date adjust /etc/localtime if not a symlink patch virtual hosts / jails on given host ensure all services are restarted (mysql, apache, etc.) per-application testsJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 81Data Collection need to know hosts that can’t deal with new DST need to run a short command on each host env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’ need to know what packages are installed need to know timestamp of a certain file need to know if client is running oracle check /etc/localtimeJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 82Data Collection need to know hosts that can’t deal with new DST need to run a short command on each host env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’ need to know what packages are installed need to know timestamp of a certain file need to know if client is running oracle check /etc/localtime check for virtual hosts / jailsJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 83Data Collection need to know hosts that can’t deal with new DST need to run a short command on each host env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’ need to know what packages are installed need to know timestamp of a certain file need to know if client is running oracle check /etc/localtime check for virtual hosts / jails gather output of ps auwwxJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 84Data Collection need to know hosts that can’t deal with new DST need to run a short command on each host env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’ need to know what packages are installed need to know timestamp of a certain file need to know if client is running oracle check /etc/localtime check for virtual hosts / jails gather output of ps auwwx oh, and also gather firmware information, why notJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 85Script N See remote.final.sh.Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 86How to run the script remotely?while read host; do scp remote.sh $host:/tmp/ ssh $host "/tmp/remote.sh"done < $1Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 87How to run the script remotely?while read host; do scp remote.sh $host:/tmp/ ssh $host "/tmp/remote.sh"done < $1 But: scp(1) followed by ssh(1) is expensiveJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 88How to run the script remotely?while read host; do cat remote.sh | ssh $host "cat >/tmp/remote.sh; /tmp/remote.sh"done < $1Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 89How to run the script remotely?while read host; do cat remote.sh | ssh $host "cat >/tmp/remote.sh; /tmp/remote.sh"done < $1 But: /bin/sh vs. Solaris anybody?Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 90How to run the script remotely?while read host; do cat remote.sh | ssh $host "cat >/tmp/remote.sh; /usr/local/bin/bash /tmp/remote.sh"done < $1Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 91How to run the script remotely?while read host; do cat remote.sh | ssh $host "cat >/tmp/remote.sh; /usr/local/bin/bash /tmp/remote.sh"done < $1 But: contrary to common assumption, /usr/local/bin/bash does not exist on all hostsJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 92How to run the script remotely?while read host; do cat remote.sh | ssh $host "cat >/tmp/remote.sh && bash /tmp/remote.sh"done < $1Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 93How to run the script remotely?while read host; do cat remote.sh | ssh $host "cat >/tmp/remote.sh && bash /tmp/remote.sh"done < $1 appears to work, even if it relies on $PATHJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 94Performing data collection originally done on a single host, sequentiallyJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 95Performing data collection originally done on a single host, sequentially (full scan took almost a week)Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 96Performing data collection originally done on a single host, sequentially (full scan took almost a week) then done in parallel on single hostJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 97Performing data collection originally done on a single host, sequentially (full scan took almost a week) then done in parallel on single host (try to get work done on a machine with a load >100)Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 98Performing data collection originally done on a single host, sequentially (full scan took almost a week) then done in parallel on single host (try to get work done on a machine with a load >100) then done in parallel on multiple hostsJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 99Performing data collection originally done on a single host, sequentially (full scan took almost a week) then done in parallel on single host (try to get work done on a machine with a load >100) then done in parallel on multiple hosts (managed to fill up all local diskspace when uploading results)Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 100Performing data collection originally done on a single host, sequentially (full scan took almost a week) then done in parallel on single host (try to get work done on a machine with a load >100) then done in parallel on multiple hosts (managed to fill up all local diskspace when uploading results) finally done massively parallel and unattended on many dedicated hosts with plenty of diskspaceJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 101Performing data collection originally done on a single host, sequentially (full scan took almost a week) then done in parallel on single host (try to get work done on a machine with a load >100) then done in parallel on multiple hosts (managed to fill up all local diskspace when uploading results) finally done massively parallel and unattended on many dedicated hosts with plenty of diskspace (full scan took about 50 minutes)Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 102“Parallelization” in shell - *cough* split input into N files farm out to N hosts (slaves) on each slave split input into M files run M jobs in parallel run all jobs in screen(1) session, then detach on master, wait until N slaves report done perform post-processing on master, dump data into http dir and send mailJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 103Actual patching mostly done by property “leftovers” done by YSA team FreeBSD patches trivial (thanks to internal kick-ass package manager) RHEL patching less trivial Solaris patching least trivialJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 104March 11th cometh! SAs everywhere frantically exchange batteries in fire alarms.Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 105March 11th goes!Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 106March 12th comes! SAs return to their daily routine.Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 107DST ’07 - Bottom line a non-eventJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 108DST ’07 - Bottom line a non-event (excellent!)Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 109DST ’07 - Bottom line a non-event (excellent!) we now have a bunch of useful tools to do massive scansJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 110DST ’07 - Bottom line a non-event (excellent!) we now have a bunch of useful tools to do massive scans (though they should be rewritten in perl, python, C, anything!)Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 111DST ’07 - Bottom line a non-event (excellent!) we now have a bunch of useful tools to do massive scans (though they should be rewritten in perl, python, C, anything!) not all of our machines are entirely idleJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 112DST ’07 - Bottom line a non-event (excellent!) we now have a bunch of useful tools to do massive scans (though they should be rewritten in perl, python, C, anything!) not all of our machines are entirely idle: up 72 days, 23:01, 0 users, load averages: 1021.89, 974.45, 881.28Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 113DST ’07 - Bottom line a non-event (excellent!) we now have a bunch of useful tools to do massive scans (though they should be rewritten in perl, python, C, anything!) not all of our machines are entirely idle: up 72 days, 23:01, 0 users, load averages: 1021.89, 974.45, 881.28 up 235 days, 11:39, 0 users, load averages: 305.07, 344.82, 338.55Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 114DST ’07 - Bottom line a non-event (excellent!) we now have a bunch of useful tools to do massive scans (though they should be rewritten in perl, python, C, anything!) not all of our machines are entirely idle: up 72 days, 23:01, 0 users, load averages: 1021.89, 974.45, 881.28 up 235 days, 11:39, 0 users, load averages: 305.07, 344.82, 338.55 up 1885 days, 12 mins, 0 users, load averages: 33.36, 31.59, 29.82Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 115DST ’07 - Bottom line a non-event (excellent!) we now have a bunch of useful tools to do massive scans (though they should be rewritten in perl, python, C, anything!) not all of our machines are entirely idle: up 72 days, 23:01, 0 users, load averages: 1021.89, 974.45, 881.28 up 235 days, 11:39, 0 users, load averages: 305.07, 344.82, 338.55 up 1885 days, 12 mins, 0 users, load averages: 33.36, 31.59, 29.82 we continue to do periodical scans and gather the same dataJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 116DST ’07 - Bottom line a non-event (excellent!) we now have a bunch of useful tools to do massive scans (though they should be rewritten in perl, python, C, anything!) not all of our machines are entirely idle: up 72 days, 23:01, 0 users, load averages: 1021.89, 974.45, 881.28 up 235 days, 11:39, 0 users, load averages: 305.07, 344.82, 338.55 up 1885 days, 12 mins, 0 users, load averages: 33.36, 31.59, 29.82 we continue to do periodical scans and gather the same data we still will be on alert on October 28th (old switchback date)Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 117DST ’07 - Bottom lineJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 118DST ’07 - Bottom line a non-event (excellent!) we now have a bunch of useful tools to do massive scans (though they should be rewritten in perl, python, C, anything!) not all of our machines are entirely idle: up 1885 days, 12 mins, 0 users, load averages: 33.36, 31.59, 29.82 up 72 days, 23:01, 0 users, load averages: 1021.89, 974.45, 881.28 up 235 days, 11:39, 0 users, load averages: 305.07, 344.82, 338.55 we continue to do periodical scans and gather the same data we still will be on alert on October 28th (old switchback date) and November 4th (new switchback date)Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 119DST ’07 - Bottom lineJan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 120DST ’07 - Bottom line a non-event (excellent!) we now have a bunch of useful tools to do massive scans (though they should be rewritten in perl, python, C, anything!) not all of our machines are entirely idle: up 1885 days, 12 mins, 0 users, load averages: 33.36, 31.59, 29.82 up 72 days, 23:01, 0 users, load averages: 1021.89, 974.45, 881.28 up 235 days, 11:39, 0 users, load averages: 305.07, 344.82, 338.55 we continue to do periodical scans and gather the same data we still will be on alert on October 28th (old switchback date) and November 4th (new switchback date) we’ll be rather busy in the Fall of 2037...Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 121Y2K38 time t stored as a signed 32bit int latest time possible: 03:14:07 UTC on Tuesday, January 19, 2038Suggested solution: using a signed 64bit value.Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 122Y2K38 time t stored as a signed 32bit int latest time possible: 03:14:07 UTC on Tuesday, January 19, 2038Suggested solution: using a signed 64bit value.New wraparound date on Sunday, December 4, 292,277,026,596 (ie in about 290billion years).Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 123Y2K38 time t stored as a signed 32bit int latest time possible: 03:14:07 UTC on Tuesday, January 19, 2038Suggested solution: using a signed 64bit value.New wraparound date on Sunday, December 4, 292,277,026,596 (ie in about 290billion years).“This is not widely regarded as a pressing issue.”Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 124To summarize: ∆t′ = γ(∆t − v∆x ) c2 v∆x′ ∆t = γ(∆t′ + 2 ) c ∆t′ = γ∆tTime is relative...Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 125To summarize: ∆t′ = γ(∆t − v∆x ) c2 v∆x′ ∆t = γ(∆t′ + 2 ) c ∆t′ = γ∆tTime is relative (and an illusion)...Jan Schaumann BayLISA, May 17, 2007
    • DST ’07 @ Yahoo! Slide 126To summarize:...and Daylight Saving Time doubly so.Jan Schaumann BayLISA, May 17, 2007