DST @ Yahoo!

787 views
660 views

Published on

A presentation on how changes in Daylight Saving Time were handled at Yahoo!. Originally given at BayLISA in May 2007. Slides are available here:

netmeister.org/misc/dst_yahoo.pdf

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
787
On SlideShare
0
From Embeds
0
Number of Embeds
46
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

DST @ Yahoo!

  1. 1. DST ’07 @ Yahoo! Slide 1 DST ’07 @ Yahoo! Jan Schaumann jschauma@netmeister.org PGP: 136D 027F DC29 8402 7B42 47D6 7C5B 64AF AF22 6A4CJan Schaumann BayLISA, May 17, 2007
  2. 2. DST ’07 @ Yahoo! Slide 2whoami$ ME=$(id -un)$ grep ${ME} /etc/passwd | cut -d: -f5Jan Schaumann$Jan Schaumann BayLISA, May 17, 2007
  3. 3. DST ’07 @ Yahoo! Slide 3whoami$ ME=$(id -un)$ grep ${ME} /etc/passwd | cut -d: -f5Jan Schaumann$ groups ${ME}netbsd sa yahoo$Jan Schaumann BayLISA, May 17, 2007
  4. 4. DST ’07 @ Yahoo! Slide 4whoami$ ME=$(id -un)$ grep ${ME} /etc/passwd | cut -d: -f5Jan Schaumann$ groups ${ME}netbsd sa yahoo$Jan Schaumann BayLISA, May 17, 2007
  5. 5. DST ’07 @ Yahoo! Slide 5whoami$ ME=$(id -un)$ grep ${ME} /etc/passwd | cut -d: -f5Jan Schaumann$ groups ${ME}netbsd sa yahoo$Jan Schaumann BayLISA, May 17, 2007
  6. 6. DST ’07 @ Yahoo! Slide 6whoami$ ME=$(id -un)$ grep ${ME} /etc/passwd | cut -d: -f5Jan Schaumann$ groups ${ME}netbsd sa yahoo$Jan Schaumann BayLISA, May 17, 2007
  7. 7. DST ’07 @ Yahoo! Slide 7whoami$ ME=$(id -un)$ grep ${ME} /etc/passwd | cut -d: -f5Jan Schaumann$ groups ${ME}netbsd sa yahoo$Jan Schaumann BayLISA, May 17, 2007
  8. 8. DST ’07 @ Yahoo! Slide 8whoami$ ME=$(id -un)$ awk -F: -v ME=${ME} ’$0 ~ ME { print $5 }’ /etc/passwdJan Schaumann$ groups ${ME}netbsd sa yahoo$Jan Schaumann BayLISA, May 17, 2007
  9. 9. DST ’07 @ Yahoo! Slide 9whoami$ ME=$(id -un)$ grep ${ME} /etc/passwd | cut -d: -f5Jan Schaumann$ groups ${ME}netbsd sa yahoo$Jan Schaumann BayLISA, May 17, 2007
  10. 10. DST ’07 @ Yahoo! Slide 10whoami$ ME=$(id -un)$ grep ${ME} /etc/passwd | cut -d: -f5Jan Schaumann$ groups ${ME}netbsd sa yahoo$Jan Schaumann BayLISA, May 17, 2007
  11. 11. DST ’07 @ Yahoo! Slide 11 DST ’07 @ Yahoo! – Saving the living daylight out of you! or How the YSA team stopped worrying and learned to love UTC!Jan Schaumann BayLISA, May 17, 2007
  12. 12. DST ’07 @ Yahoo! Slide 12In the beginning......the universe was created.Jan Schaumann BayLISA, May 17, 2007
  13. 13. DST ’07 @ Yahoo! Slide 13In the beginning......the universe was created.“This has made a lot of people very angry and been widely regarded as a bad move.”Jan Schaumann BayLISA, May 17, 2007
  14. 14. DST ’07 @ Yahoo! Slide 14Daylight Saving Time for DummiesJan Schaumann BayLISA, May 17, 2007
  15. 15. DST ’07 @ Yahoo! Slide 15Daylight Saving Time for DummiesJan Schaumann BayLISA, May 17, 2007
  16. 16. DST ’07 @ Yahoo! Slide 16Daylight Saving Time for DummiesJan Schaumann BayLISA, May 17, 2007
  17. 17. DST ’07 @ Yahoo! Slide 17Daylight Saving Time for DummiesJan Schaumann BayLISA, May 17, 2007
  18. 18. DST ’07 @ Yahoo! Slide 18Daylight Saving Time for DummiesJan Schaumann BayLISA, May 17, 2007
  19. 19. DST ’07 @ Yahoo! Slide 19Daylight Saving Time for DummiesJan Schaumann BayLISA, May 17, 2007
  20. 20. DST ’07 @ Yahoo! Slide 20Daylight Saving Time for DummiesJan Schaumann BayLISA, May 17, 2007
  21. 21. DST ’07 @ Yahoo! Slide 21Fast forward to 2005...Jan Schaumann BayLISA, May 17, 2007
  22. 22. DST ’07 @ Yahoo! Slide 22Fast forward to 2005...Jan Schaumann BayLISA, May 17, 2007
  23. 23. DST ’07 @ Yahoo! Slide 23Fast forward to 2005...Jan Schaumann BayLISA, May 17, 2007
  24. 24. DST ’07 @ Yahoo! Slide 24March 11th, 2007 In 2007, Daylight Saving Time will start on March 11th.Jan Schaumann BayLISA, May 17, 2007
  25. 25. DST ’07 @ Yahoo! Slide 25March 11thJan Schaumann BayLISA, May 17, 2007
  26. 26. DST ’07 @ Yahoo! Slide 26March 11thJan Schaumann BayLISA, May 17, 2007
  27. 27. DST ’07 @ Yahoo! Slide 27August 8th, 2005Jan Schaumann BayLISA, May 17, 2007
  28. 28. DST ’07 @ Yahoo! Slide 28One day later... Yahoo! decides to take immediate action!Jan Schaumann BayLISA, May 17, 2007
  29. 29. DST ’07 @ Yahoo! Slide 29August, 29th, 2005Jan Schaumann BayLISA, May 17, 2007
  30. 30. DST ’07 @ Yahoo! Slide 30December, 22nd, 2005Jan Schaumann BayLISA, May 17, 2007
  31. 31. DST ’07 @ Yahoo! Slide 31August 8th, 2005Jan Schaumann BayLISA, May 17, 2007
  32. 32. DST ’07 @ Yahoo! Slide 32One day later... Yahoo! decides to take immediate action!Jan Schaumann BayLISA, May 17, 2007
  33. 33. DST ’07 @ Yahoo! Slide 33Alas...Jan Schaumann BayLISA, May 17, 2007
  34. 34. DST ’07 @ Yahoo! Slide 34“Persistence of Memory”Jan Schaumann BayLISA, May 17, 2007
  35. 35. DST ’07 @ Yahoo! Slide 35Fast forward to January 2007... “Hey, we need to know which hosts can’t deal with the new DST.”Jan Schaumann BayLISA, May 17, 2007
  36. 36. DST ’07 @ Yahoo! Slide 36Script 0 See remote0.sh (and remote0.5.sh).Jan Schaumann BayLISA, May 17, 2007
  37. 37. DST ’07 @ Yahoo! Slide 37January 2007 need to know hosts that can’t deal with new DSTJan Schaumann BayLISA, May 17, 2007
  38. 38. DST ’07 @ Yahoo! Slide 38January 2007 need to know hosts that can’t deal with new DST need to run a short command on each hostJan Schaumann BayLISA, May 17, 2007
  39. 39. DST ’07 @ Yahoo! Slide 39January 2007 need to know hosts that can’t deal with new DST need to run a short command on each hostSo: write a script that runs a single command on hosts given as input.Jan Schaumann BayLISA, May 17, 2007
  40. 40. DST ’07 @ Yahoo! Slide 40January 2007 need to know hosts that can’t deal with new DST need to run a short command on each hostSo: write a script that runs a single command on hosts given as input.$ cat >checkhosts.sh <<EOS#!/bin/shwhile read host; do ssh $host "single-command"done < $1EOS$Jan Schaumann BayLISA, May 17, 2007
  41. 41. DST ’07 @ Yahoo! Slide 41“single command”TIMTOWTDI: TZ=PST8PDT date -j 200703110200Jan Schaumann BayLISA, May 17, 2007
  42. 42. DST ’07 @ Yahoo! Slide 42“single command”TIMTOWTDI (or is there?): TZ=PST8PDT date -j 200703110200 – conflicting flags and return values on different platformsJan Schaumann BayLISA, May 17, 2007
  43. 43. DST ’07 @ Yahoo! Slide 43“single command”TIMTOWTDI (or is there?): TZ=PST8PDT date -j 200703110200 – conflicting flags and return values on different platforms zdump -v PST8PDT | grep "2007 UTC = Sun Mar 11"Jan Schaumann BayLISA, May 17, 2007
  44. 44. DST ’07 @ Yahoo! Slide 44“single command”TIMTOWTDI (or is there?): TZ=PST8PDT date -j 200703110200 – conflicting flags and return values on different platforms zdump -v PST8PDT | grep "2007 UTC = Sun Mar 11" – triggers a bug in FreeBSD/amd64 and zdump spins out of controlJan Schaumann BayLISA, May 17, 2007
  45. 45. DST ’07 @ Yahoo! Slide 45“single command”TIMTOWTDI (or is there?): TZ=PST8PDT date -j 200703110200 – conflicting flags and return values on different platforms zdump -v PST8PDT | grep "2007 UTC = Sun Mar 11" – triggers a bug in FreeBSD/amd64 and zdump spins out of control env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’Jan Schaumann BayLISA, May 17, 2007
  46. 46. DST ’07 @ Yahoo! Slide 46“single command”TIMTOWTDI (or is there?): TZ=PST8PDT date -j 200703110200 – conflicting flags and return values on different platforms zdump -v PST8PDT | grep "2007 UTC = Sun Mar 11" – triggers a bug in FreeBSD/amd64 and zdump spins out of control env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’ – Hooray!Jan Schaumann BayLISA, May 17, 2007
  47. 47. DST ’07 @ Yahoo! Slide 47“single command”TIMTOWTDI (or is there?): TZ=PST8PDT date -j 200703110200 – conflicting flags and return values on different platforms zdump -v PST8PDT | grep "2007 UTC = Sun Mar 11" – triggers a bug in FreeBSD/amd64 and zdump spins out of control env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’ – Hooray!Yes, we happen to have perl installed on every machine.Jan Schaumann BayLISA, May 17, 2007
  48. 48. DST ’07 @ Yahoo! Slide 48Script I See remote1.sh.Jan Schaumann BayLISA, May 17, 2007
  49. 49. DST ’07 @ Yahoo! Slide 49So far, so good!We now have a list of hostnames, sorted by: operating system need to patchJan Schaumann BayLISA, May 17, 2007
  50. 50. DST ’07 @ Yahoo! Slide 50So far, so good!We now have a list of hostnames, sorted by: operating system need to patchSurely, our work here is done.Jan Schaumann BayLISA, May 17, 2007
  51. 51. DST ’07 @ Yahoo! Slide 51So far, so good!We now have a list of hostnames, sorted by: operating system need to patchSurely, our work here is done.Well, not entirely...Jan Schaumann BayLISA, May 17, 2007
  52. 52. DST ’07 @ Yahoo! Slide 52Forward to middle of February 2007...Jan Schaumann BayLISA, May 17, 2007
  53. 53. DST ’07 @ Yahoo! Slide 53Forward to middle of February 2007... detailed communication with properties about impact gather information from vendors select patching full-time focus on data collectionJan Schaumann BayLISA, May 17, 2007
  54. 54. DST ’07 @ Yahoo! Slide 54Data Collection need to know hosts that can’t deal with new DST need to run a short command on each host env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’Jan Schaumann BayLISA, May 17, 2007
  55. 55. DST ’07 @ Yahoo! Slide 55Data Collection need to know hosts that can’t deal with new DST need to run a short command on each host env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’ need to know what packages are installedJan Schaumann BayLISA, May 17, 2007
  56. 56. DST ’07 @ Yahoo! Slide 56Data Collection need to know hosts that can’t deal with new DST need to run a short command on each host env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’ need to know what packages are installed need to know timestamp of a certain fileJan Schaumann BayLISA, May 17, 2007
  57. 57. DST ’07 @ Yahoo! Slide 57Data Collection need to know hosts that can’t deal with new DST need to run a short command on each host env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’ need to know what packages are installed need to know timestamp of a certain file need to know if client is running oracleJan Schaumann BayLISA, May 17, 2007
  58. 58. DST ’07 @ Yahoo! Slide 58Data Collection need to know hosts that can’t deal with new DST need to run a short command on each host env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’ need to know what packages are installed need to know timestamp of a certain file need to know if client is running oracleSo: revamp tool to not run hard-coded command, but any given script.Jan Schaumann BayLISA, May 17, 2007
  59. 59. DST ’07 @ Yahoo! Slide 59Script II See remote2.sh and remote3.sh.Jan Schaumann BayLISA, May 17, 2007
  60. 60. DST ’07 @ Yahoo! Slide 60Data Collection: Input input from central databaseJan Schaumann BayLISA, May 17, 2007
  61. 61. DST ’07 @ Yahoo! Slide 61Data Collection: Input input from central database input from DNS zone filesJan Schaumann BayLISA, May 17, 2007
  62. 62. DST ’07 @ Yahoo! Slide 62Data Collection: Input input from central database input from DNS zone files prepare exclusion lists exclude windows hosts exclude routers, switches etc. exclude NetApp filers exclude hosts in corp exclude DNS rotations, decomissioned hosts, placeholder DNS entries etc. ...Jan Schaumann BayLISA, May 17, 2007
  63. 63. DST ’07 @ Yahoo! Slide 63Data Collection: Output a lot of information is collected → output per host fairly large sort output by patch-needed sort output by OS sort output by propertyJan Schaumann BayLISA, May 17, 2007
  64. 64. DST ’07 @ Yahoo! Slide 64Patch actions FreeBSD Linux SolarisJan Schaumann BayLISA, May 17, 2007
  65. 65. DST ’07 @ Yahoo! Slide 65Patch actions FreeBSD all versions prior to a certain 4.11 based image require patch Linux SolarisJan Schaumann BayLISA, May 17, 2007
  66. 66. DST ’07 @ Yahoo! Slide 66Patch actions FreeBSD all versions prior to a certain 4.11 based image require patch some versions higher than that (including some 6.x based images) may require a patch Linux SolarisJan Schaumann BayLISA, May 17, 2007
  67. 67. DST ’07 @ Yahoo! Slide 67Patch actions FreeBSD all versions prior to a certain 4.11 based image require patch some versions higher than that (including some 6.x based images) may require a patch upgrade to latest 6.x based version recommended Linux SolarisJan Schaumann BayLISA, May 17, 2007
  68. 68. DST ’07 @ Yahoo! Slide 68Patch actions FreeBSD all versions prior to a certain 4.11 based image require patch some versions higher than that (including some 6.x based images) may require a patch upgrade to latest 6.x based version recommended patch installation innocuous → no reboot required Linux SolarisJan Schaumann BayLISA, May 17, 2007
  69. 69. DST ’07 @ Yahoo! Slide 69Patch actions FreeBSD all versions prior to a certain 4.11 based image require patch some versions higher than that (including some 6.x based images) may require a patch upgrade to latest 6.x based version recommended patch installation innocuous → no reboot required Linux no change necessary for versions >= RHEL3u7, RHEL4u3 SolarisJan Schaumann BayLISA, May 17, 2007
  70. 70. DST ’07 @ Yahoo! Slide 70Patch actions FreeBSD all versions prior to a certain 4.11 based image require patch some versions higher than that (including some 6.x based images) may require a patch upgrade to latest 6.x based version recommended patch installation innocuous → no reboot required Linux no change necessary for versions >= RHEL3u7, RHEL4u3 installation of rpm tzdata for other versions RHEL3 and RHEL 4 SolarisJan Schaumann BayLISA, May 17, 2007
  71. 71. DST ’07 @ Yahoo! Slide 71Patch actions FreeBSD all versions prior to a certain 4.11 based image require patch some versions higher than that (including some 6.x based images) may require a patch upgrade to latest 6.x based version recommended patch installation innocuous → no reboot required Linux no change necessary for versions >= RHEL3u7, RHEL4u3 installation of rpm tzdata for other versions RHEL3 and RHEL 4 upgrade to latest RHEL release for all other rpm based Linux versions recommend SolarisJan Schaumann BayLISA, May 17, 2007
  72. 72. DST ’07 @ Yahoo! Slide 72Patch actions FreeBSD all versions prior to a certain 4.11 based image require patch some versions higher than that (including some 6.x based images) may require a patch upgrade to latest 6.x based version recommended patch installation innocuous → no reboot required Linux no change necessary for versions >= RHEL3u7, RHEL4u3 installation of rpm tzdata for other versions RHEL3 and RHEL 4 upgrade to latest RHEL release for all other rpm based Linux versions recommend forced (ugh!) installation of tzdata rpm possible SolarisJan Schaumann BayLISA, May 17, 2007
  73. 73. DST ’07 @ Yahoo! Slide 73Patch actions FreeBSD all versions prior to a certain 4.11 based image require patch some versions higher than that (including some 6.x based images) may require a patch upgrade to latest 6.x based version recommended patch installation innocuous → no reboot required Linux no change necessary for versions >= RHEL3u7, RHEL4u3 installation of rpm tzdata for other versions RHEL3 and RHEL 4 upgrade to latest RHEL release for all other rpm based Linux versions recommend forced (ugh!) installation of tzdata rpm possible Solaris Oy vey!Jan Schaumann BayLISA, May 17, 2007
  74. 74. DST ’07 @ Yahoo! Slide 74TZDATA everywhere! ICU (Internationalization Components for Unicode) JVM MySQL Oracle Perl DateTimeJan Schaumann BayLISA, May 17, 2007
  75. 75. DST ’07 @ Yahoo! Slide 75TZDATA everywhere! ICU (Internationalization Components for Unicode) need to update ICU to >= 3.6 need to recompile apps linked against libicu∗ JVM MySQL Oracle Perl DateTimeJan Schaumann BayLISA, May 17, 2007
  76. 76. DST ’07 @ Yahoo! Slide 76TZDATA everywhere! ICU (Internationalization Components for Unicode) need to update ICU to >= 3.6 need to recompile apps linked against libicu∗ JVM need to update to recent version March 8th (!): new bug in JDK’s found (for EST, MST, and HST zones); no vendor patch but workaround solution MySQL Oracle Perl DateTimeJan Schaumann BayLISA, May 17, 2007
  77. 77. DST ’07 @ Yahoo! Slide 77TZDATA everywhere! ICU (Internationalization Components for Unicode) need to update ICU to >= 3.6 need to recompile apps linked against libicu∗ JVM need to update to recent version March 8th (!): new bug in JDK’s found (for EST, MST, and HST zones); no vendor patch but workaround solution MySQL restart only for MySQL 4.0 upgrade “mysql tzdata” package for MySQL 4.1Jan Schaumann BayLISA, May 17, 2007
  78. 78. DST ’07 @ Yahoo! Slide 78TZDATA everywhere! Oracle patch following instructions provided by vendor March 3rd: new patch → repatch! Perl DateTimeJan Schaumann BayLISA, May 17, 2007
  79. 79. DST ’07 @ Yahoo! Slide 79TZDATA everywhere! Oracle patch following instructions provided by vendor March 3rd: new patch → repatch! Perl DateTime update to >= 0.38 total of < 100 hosts using this packageJan Schaumann BayLISA, May 17, 2007
  80. 80. DST ’07 @ Yahoo! Slide 80Post-patch actions if possible, reboot verify that host displays correct date adjust /etc/localtime if not a symlink patch virtual hosts / jails on given host ensure all services are restarted (mysql, apache, etc.) per-application testsJan Schaumann BayLISA, May 17, 2007
  81. 81. DST ’07 @ Yahoo! Slide 81Data Collection need to know hosts that can’t deal with new DST need to run a short command on each host env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’ need to know what packages are installed need to know timestamp of a certain file need to know if client is running oracle check /etc/localtimeJan Schaumann BayLISA, May 17, 2007
  82. 82. DST ’07 @ Yahoo! Slide 82Data Collection need to know hosts that can’t deal with new DST need to run a short command on each host env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’ need to know what packages are installed need to know timestamp of a certain file need to know if client is running oracle check /etc/localtime check for virtual hosts / jailsJan Schaumann BayLISA, May 17, 2007
  83. 83. DST ’07 @ Yahoo! Slide 83Data Collection need to know hosts that can’t deal with new DST need to run a short command on each host env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’ need to know what packages are installed need to know timestamp of a certain file need to know if client is running oracle check /etc/localtime check for virtual hosts / jails gather output of ps auwwxJan Schaumann BayLISA, May 17, 2007
  84. 84. DST ’07 @ Yahoo! Slide 84Data Collection need to know hosts that can’t deal with new DST need to run a short command on each host env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’ need to know what packages are installed need to know timestamp of a certain file need to know if client is running oracle check /etc/localtime check for virtual hosts / jails gather output of ps auwwx oh, and also gather firmware information, why notJan Schaumann BayLISA, May 17, 2007
  85. 85. DST ’07 @ Yahoo! Slide 85Script N See remote.final.sh.Jan Schaumann BayLISA, May 17, 2007
  86. 86. DST ’07 @ Yahoo! Slide 86How to run the script remotely?while read host; do scp remote.sh $host:/tmp/ ssh $host "/tmp/remote.sh"done < $1Jan Schaumann BayLISA, May 17, 2007
  87. 87. DST ’07 @ Yahoo! Slide 87How to run the script remotely?while read host; do scp remote.sh $host:/tmp/ ssh $host "/tmp/remote.sh"done < $1 But: scp(1) followed by ssh(1) is expensiveJan Schaumann BayLISA, May 17, 2007
  88. 88. DST ’07 @ Yahoo! Slide 88How to run the script remotely?while read host; do cat remote.sh | ssh $host "cat >/tmp/remote.sh; /tmp/remote.sh"done < $1Jan Schaumann BayLISA, May 17, 2007
  89. 89. DST ’07 @ Yahoo! Slide 89How to run the script remotely?while read host; do cat remote.sh | ssh $host "cat >/tmp/remote.sh; /tmp/remote.sh"done < $1 But: /bin/sh vs. Solaris anybody?Jan Schaumann BayLISA, May 17, 2007
  90. 90. DST ’07 @ Yahoo! Slide 90How to run the script remotely?while read host; do cat remote.sh | ssh $host "cat >/tmp/remote.sh; /usr/local/bin/bash /tmp/remote.sh"done < $1Jan Schaumann BayLISA, May 17, 2007
  91. 91. DST ’07 @ Yahoo! Slide 91How to run the script remotely?while read host; do cat remote.sh | ssh $host "cat >/tmp/remote.sh; /usr/local/bin/bash /tmp/remote.sh"done < $1 But: contrary to common assumption, /usr/local/bin/bash does not exist on all hostsJan Schaumann BayLISA, May 17, 2007
  92. 92. DST ’07 @ Yahoo! Slide 92How to run the script remotely?while read host; do cat remote.sh | ssh $host "cat >/tmp/remote.sh && bash /tmp/remote.sh"done < $1Jan Schaumann BayLISA, May 17, 2007
  93. 93. DST ’07 @ Yahoo! Slide 93How to run the script remotely?while read host; do cat remote.sh | ssh $host "cat >/tmp/remote.sh && bash /tmp/remote.sh"done < $1 appears to work, even if it relies on $PATHJan Schaumann BayLISA, May 17, 2007
  94. 94. DST ’07 @ Yahoo! Slide 94Performing data collection originally done on a single host, sequentiallyJan Schaumann BayLISA, May 17, 2007
  95. 95. DST ’07 @ Yahoo! Slide 95Performing data collection originally done on a single host, sequentially (full scan took almost a week)Jan Schaumann BayLISA, May 17, 2007
  96. 96. DST ’07 @ Yahoo! Slide 96Performing data collection originally done on a single host, sequentially (full scan took almost a week) then done in parallel on single hostJan Schaumann BayLISA, May 17, 2007
  97. 97. DST ’07 @ Yahoo! Slide 97Performing data collection originally done on a single host, sequentially (full scan took almost a week) then done in parallel on single host (try to get work done on a machine with a load >100)Jan Schaumann BayLISA, May 17, 2007
  98. 98. DST ’07 @ Yahoo! Slide 98Performing data collection originally done on a single host, sequentially (full scan took almost a week) then done in parallel on single host (try to get work done on a machine with a load >100) then done in parallel on multiple hostsJan Schaumann BayLISA, May 17, 2007
  99. 99. DST ’07 @ Yahoo! Slide 99Performing data collection originally done on a single host, sequentially (full scan took almost a week) then done in parallel on single host (try to get work done on a machine with a load >100) then done in parallel on multiple hosts (managed to fill up all local diskspace when uploading results)Jan Schaumann BayLISA, May 17, 2007
  100. 100. DST ’07 @ Yahoo! Slide 100Performing data collection originally done on a single host, sequentially (full scan took almost a week) then done in parallel on single host (try to get work done on a machine with a load >100) then done in parallel on multiple hosts (managed to fill up all local diskspace when uploading results) finally done massively parallel and unattended on many dedicated hosts with plenty of diskspaceJan Schaumann BayLISA, May 17, 2007
  101. 101. DST ’07 @ Yahoo! Slide 101Performing data collection originally done on a single host, sequentially (full scan took almost a week) then done in parallel on single host (try to get work done on a machine with a load >100) then done in parallel on multiple hosts (managed to fill up all local diskspace when uploading results) finally done massively parallel and unattended on many dedicated hosts with plenty of diskspace (full scan took about 50 minutes)Jan Schaumann BayLISA, May 17, 2007
  102. 102. DST ’07 @ Yahoo! Slide 102“Parallelization” in shell - *cough* split input into N files farm out to N hosts (slaves) on each slave split input into M files run M jobs in parallel run all jobs in screen(1) session, then detach on master, wait until N slaves report done perform post-processing on master, dump data into http dir and send mailJan Schaumann BayLISA, May 17, 2007
  103. 103. DST ’07 @ Yahoo! Slide 103Actual patching mostly done by property “leftovers” done by YSA team FreeBSD patches trivial (thanks to internal kick-ass package manager) RHEL patching less trivial Solaris patching least trivialJan Schaumann BayLISA, May 17, 2007
  104. 104. DST ’07 @ Yahoo! Slide 104March 11th cometh! SAs everywhere frantically exchange batteries in fire alarms.Jan Schaumann BayLISA, May 17, 2007
  105. 105. DST ’07 @ Yahoo! Slide 105March 11th goes!Jan Schaumann BayLISA, May 17, 2007
  106. 106. DST ’07 @ Yahoo! Slide 106March 12th comes! SAs return to their daily routine.Jan Schaumann BayLISA, May 17, 2007
  107. 107. DST ’07 @ Yahoo! Slide 107DST ’07 - Bottom line a non-eventJan Schaumann BayLISA, May 17, 2007
  108. 108. DST ’07 @ Yahoo! Slide 108DST ’07 - Bottom line a non-event (excellent!)Jan Schaumann BayLISA, May 17, 2007
  109. 109. DST ’07 @ Yahoo! Slide 109DST ’07 - Bottom line a non-event (excellent!) we now have a bunch of useful tools to do massive scansJan Schaumann BayLISA, May 17, 2007
  110. 110. DST ’07 @ Yahoo! Slide 110DST ’07 - Bottom line a non-event (excellent!) we now have a bunch of useful tools to do massive scans (though they should be rewritten in perl, python, C, anything!)Jan Schaumann BayLISA, May 17, 2007
  111. 111. DST ’07 @ Yahoo! Slide 111DST ’07 - Bottom line a non-event (excellent!) we now have a bunch of useful tools to do massive scans (though they should be rewritten in perl, python, C, anything!) not all of our machines are entirely idleJan Schaumann BayLISA, May 17, 2007
  112. 112. DST ’07 @ Yahoo! Slide 112DST ’07 - Bottom line a non-event (excellent!) we now have a bunch of useful tools to do massive scans (though they should be rewritten in perl, python, C, anything!) not all of our machines are entirely idle: up 72 days, 23:01, 0 users, load averages: 1021.89, 974.45, 881.28Jan Schaumann BayLISA, May 17, 2007
  113. 113. DST ’07 @ Yahoo! Slide 113DST ’07 - Bottom line a non-event (excellent!) we now have a bunch of useful tools to do massive scans (though they should be rewritten in perl, python, C, anything!) not all of our machines are entirely idle: up 72 days, 23:01, 0 users, load averages: 1021.89, 974.45, 881.28 up 235 days, 11:39, 0 users, load averages: 305.07, 344.82, 338.55Jan Schaumann BayLISA, May 17, 2007
  114. 114. DST ’07 @ Yahoo! Slide 114DST ’07 - Bottom line a non-event (excellent!) we now have a bunch of useful tools to do massive scans (though they should be rewritten in perl, python, C, anything!) not all of our machines are entirely idle: up 72 days, 23:01, 0 users, load averages: 1021.89, 974.45, 881.28 up 235 days, 11:39, 0 users, load averages: 305.07, 344.82, 338.55 up 1885 days, 12 mins, 0 users, load averages: 33.36, 31.59, 29.82Jan Schaumann BayLISA, May 17, 2007
  115. 115. DST ’07 @ Yahoo! Slide 115DST ’07 - Bottom line a non-event (excellent!) we now have a bunch of useful tools to do massive scans (though they should be rewritten in perl, python, C, anything!) not all of our machines are entirely idle: up 72 days, 23:01, 0 users, load averages: 1021.89, 974.45, 881.28 up 235 days, 11:39, 0 users, load averages: 305.07, 344.82, 338.55 up 1885 days, 12 mins, 0 users, load averages: 33.36, 31.59, 29.82 we continue to do periodical scans and gather the same dataJan Schaumann BayLISA, May 17, 2007
  116. 116. DST ’07 @ Yahoo! Slide 116DST ’07 - Bottom line a non-event (excellent!) we now have a bunch of useful tools to do massive scans (though they should be rewritten in perl, python, C, anything!) not all of our machines are entirely idle: up 72 days, 23:01, 0 users, load averages: 1021.89, 974.45, 881.28 up 235 days, 11:39, 0 users, load averages: 305.07, 344.82, 338.55 up 1885 days, 12 mins, 0 users, load averages: 33.36, 31.59, 29.82 we continue to do periodical scans and gather the same data we still will be on alert on October 28th (old switchback date)Jan Schaumann BayLISA, May 17, 2007
  117. 117. DST ’07 @ Yahoo! Slide 117DST ’07 - Bottom lineJan Schaumann BayLISA, May 17, 2007
  118. 118. DST ’07 @ Yahoo! Slide 118DST ’07 - Bottom line a non-event (excellent!) we now have a bunch of useful tools to do massive scans (though they should be rewritten in perl, python, C, anything!) not all of our machines are entirely idle: up 1885 days, 12 mins, 0 users, load averages: 33.36, 31.59, 29.82 up 72 days, 23:01, 0 users, load averages: 1021.89, 974.45, 881.28 up 235 days, 11:39, 0 users, load averages: 305.07, 344.82, 338.55 we continue to do periodical scans and gather the same data we still will be on alert on October 28th (old switchback date) and November 4th (new switchback date)Jan Schaumann BayLISA, May 17, 2007
  119. 119. DST ’07 @ Yahoo! Slide 119DST ’07 - Bottom lineJan Schaumann BayLISA, May 17, 2007
  120. 120. DST ’07 @ Yahoo! Slide 120DST ’07 - Bottom line a non-event (excellent!) we now have a bunch of useful tools to do massive scans (though they should be rewritten in perl, python, C, anything!) not all of our machines are entirely idle: up 1885 days, 12 mins, 0 users, load averages: 33.36, 31.59, 29.82 up 72 days, 23:01, 0 users, load averages: 1021.89, 974.45, 881.28 up 235 days, 11:39, 0 users, load averages: 305.07, 344.82, 338.55 we continue to do periodical scans and gather the same data we still will be on alert on October 28th (old switchback date) and November 4th (new switchback date) we’ll be rather busy in the Fall of 2037...Jan Schaumann BayLISA, May 17, 2007
  121. 121. DST ’07 @ Yahoo! Slide 121Y2K38 time t stored as a signed 32bit int latest time possible: 03:14:07 UTC on Tuesday, January 19, 2038Suggested solution: using a signed 64bit value.Jan Schaumann BayLISA, May 17, 2007
  122. 122. DST ’07 @ Yahoo! Slide 122Y2K38 time t stored as a signed 32bit int latest time possible: 03:14:07 UTC on Tuesday, January 19, 2038Suggested solution: using a signed 64bit value.New wraparound date on Sunday, December 4, 292,277,026,596 (ie in about 290billion years).Jan Schaumann BayLISA, May 17, 2007
  123. 123. DST ’07 @ Yahoo! Slide 123Y2K38 time t stored as a signed 32bit int latest time possible: 03:14:07 UTC on Tuesday, January 19, 2038Suggested solution: using a signed 64bit value.New wraparound date on Sunday, December 4, 292,277,026,596 (ie in about 290billion years).“This is not widely regarded as a pressing issue.”Jan Schaumann BayLISA, May 17, 2007
  124. 124. DST ’07 @ Yahoo! Slide 124To summarize: ∆t′ = γ(∆t − v∆x ) c2 v∆x′ ∆t = γ(∆t′ + 2 ) c ∆t′ = γ∆tTime is relative...Jan Schaumann BayLISA, May 17, 2007
  125. 125. DST ’07 @ Yahoo! Slide 125To summarize: ∆t′ = γ(∆t − v∆x ) c2 v∆x′ ∆t = γ(∆t′ + 2 ) c ∆t′ = γ∆tTime is relative (and an illusion)...Jan Schaumann BayLISA, May 17, 2007
  126. 126. DST ’07 @ Yahoo! Slide 126To summarize:...and Daylight Saving Time doubly so.Jan Schaumann BayLISA, May 17, 2007

×