Headless Host Scanning

983 views

Published on

A talk I gave at Twitter in April 2011 about scanmaster and sigsh and how to scan hundreds of thousands of hosts in minutes.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
983
On SlideShare
0
From Embeds
0
Number of Embeds
31
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Headless Host Scanning

  1. 1. Headless Host Scanning Jan Schaumann <jschauma@netmeister.org>99CE 1DC7 770A C5A8 09A6 0DCD 66CE 4FE9 6F6B D3D7 @jschauma
  2. 2. Headless Host Scanning -- @jschauma 2 11/19/12
  3. 3. Life Lessons: Pen testing cant solve everything.Headless Host Scanning -- @jschauma 3 11/19/12
  4. 4. Inside-OutHeadless Host Scanning -- @jschauma 4 11/19/12
  5. 5. Distance to Yahoo! ad in feet: 404 Construction cost: $357 million Capacity: 41,915Headless Host Scanning -- @jschauma 5 11/19/12
  6. 6. Ballpark numbers Distance to Yahoo! ad in feet: 30927 Construction cost: $1.5 billion Capacity: 52,325 Headless Host Scanning -- @jschauma 6 11/19/12
  7. 7. Life Lessons: Size mattersHeadless Host Scanning -- @jschauma 7 11/19/12
  8. 8. It all began some time in January of 2007...Yahoo! Presentation; Confidential 11/19/12
  9. 9. Initial Problem Statement“Hey, we need to know which hosts can’t deal with the new Daylight Saving Time.” Headless Host Scanning -- @jschauma 9 11/19/12
  10. 10. A Systems Administrators ToolboxHeadless Host Scanning -- @jschauma 10 11/19/12
  11. 11. A simple tool perl python /bin/shHeadless Host Scanning -- @jschauma 11 11/19/12
  12. 12. Initial "Solution"#!/bin/shwhile read host; do ssh $host "env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’"doneHeadless Host Scanning -- @jschauma 12 11/19/12
  13. 13. Problem Statement “Please run this script on these hosts.”Headless Host Scanning -- @jschauma 13 11/19/12
  14. 14. 2nd Pass#!/bin/shwhile read host; do scp script.sh $host:/tmp/ ssh $host "/tmp/script.sh"doneHeadless Host Scanning -- @jschauma 14 11/19/12
  15. 15. Mensch, nutz doch ein Konfigurationsmanagementtool!Headless Host Scanning -- @jschauma 15 11/19/12
  16. 16. Configuration ManagementHeadless Host Scanning -- @jschauma 16 11/19/12
  17. 17. Configuration ManagementGood at:§  keeping state§  ensuring flat text file propertiesNot so good at:§  deploying applications§  executing (one-time) scriptsTerrible at:§  reporting output of individual commands or scriptsHeadless Host Scanning -- @jschauma 17 11/19/12
  18. 18. Life Lessons: Sturgeons Law (1st adage)"Nothing is always absolutely so."Headless Host Scanning -- @jschauma 18 11/19/12
  19. 19. Mensch, nutz doch Deine Logaggregationsinfrastruktur!Headless Host Scanning -- @jschauma 19 11/19/12
  20. 20. Log Aggregation§  makes assumptions about message format§  frequently unreliable and/or asynchronous§  adds additional complexityHeadless Host Scanning -- @jschauma 20 11/19/12
  21. 21. 3rd Pass#!/bin/shwhile read host; do cat script.sh | ssh $host "bash –s"doneHeadless Host Scanning -- @jschauma 21 11/19/12
  22. 22. 3rd Pass#!/bin/shwhile read host; do ssh $host "bash –s" < script.shdoneHeadless Host Scanning -- @jschauma 22 11/19/12
  23. 23. Problem Refinement “Run this set of arbitrary commands on our hosts. Yes, on all of them.”Headless Host Scanning -- @jschauma 23 11/19/12
  24. 24. Wait a second – all hosts?$ wtf hostwtf: I dont know what host means!$§  a computer connected to a network§  a virtual machine§  a container §  jails, zones, ... §  chrootHeadless Host Scanning -- @jschauma 24 11/19/12
  25. 25. Sturgeons Law (1st adage) "Hosts" on the network"Hosts" in asset database "Hosts" in DNS Yahoo! Presentation; Confidential 11/19/12
  26. 26. 3rd Pass#!/bin/shwhile read host; do ssh $host "bash –s" < script.shdoneHeadless Host Scanning -- @jschauma 26 11/19/12
  27. 27. 4th PassHeadless Host Scanning -- @jschauma 27 11/19/12
  28. 28. Headless Host Scanning -- @jschauma 28 11/19/12
  29. 29. Life Lessons: Causality (Duh!)$ w10:45AM up 51 days, 22:19, 11 users, load averages: 976.23, 618.17, 1020.25$ time ls 13.354 real 0.001 user 0.002 sys$ Headless Host Scanning -- @jschauma 29 11/19/12
  30. 30. Horizontal Scalability Scanmaster Scanslaves Scanslaves TargethostsHeadless Host Scanning -- @jschauma 30 11/19/12
  31. 31. SSH Complications: assword prompts Interactive connections? Password: $USER@hostnames password:Headless Host Scanning -- @jschauma 31 11/19/12
  32. 32. SSH Complications: assword prompts Interactive connections begat password injectors.Headless Host Scanning -- @jschauma 32 11/19/12
  33. 33. SSH Complications: assword prompts proc respond { pw } { global pws if {![info exists pws($pw)]} { send_user " (autopw) " stty -echo expect_user -re "(.+)n" stty echo set pws($pw) [exec perl -ne "print pack(q{u}, $_)" << $expect_out(1,string)] } else { send_user " (autopwed)" } log_user 0 send -- "[exec perl -ne "print unpack(q{u}, $_)" << $pws($pw)]n"; log_user 1 } eval spawn -noecho $argv expect { "assword:" { respond system exp_continue }Yahoo! Presentation; Confidential 11/19/12
  34. 34. SSH Complications: assword prompts Key Authentication to the rescue!Yahoo! Presentation; Confidential 11/19/12
  35. 35. Horizontal Scalability Scanmaster Scanslaves Scanslaves TargethostsYahoo! Presentation; Confidential 11/19/12
  36. 36. SSH Complications: assword prompts Key Authentication to the rescue?§  private keys need to be present on all hosts connecting to targets§  you need to provide passphrase for private keys at scan startEnter passphrase for /home/${USER}/.ssh/id_rsa:Enter passphrase for /home/${USER}/.ssh/id_dsa:Enter passphrase for /home/${USER}/.ssh/scankey:Yahoo! Presentation; Confidential 11/19/12
  37. 37. SSH Complications: assword prompts Key Authentication also uses password injectors.Headless Host Scanning -- @jschauma 37 11/19/12
  38. 38. SSH Complications: assword prompts SSH Agent Forwarding to the rescue!§  private keys stored in agent on single "safe" machine§  agent forwarding enabled to scan slaves§  everythings peachy!Headless Host Scanning -- @jschauma 38 11/19/12
  39. 39. SSH Agent Forwarding proxy agent Slave Targethosts MasterHeadless Host Scanning -- @jschauma 39 11/19/12
  40. 40. SSH Agent Forwarding proxy agent Slave Targethosts MasterHeadless Host Scanning -- @jschauma 40 11/19/12
  41. 41. SSH Agent Forwarding MasterHeadless Host Scanning -- @jschauma 41 11/19/12
  42. 42. SSH Agent ForwardingHeadless Host Scanning -- @jschauma 42 11/19/12
  43. 43. Scalability issues Scanmaster Scanslaves Scanslaves TargethostsHeadless Host Scanning -- @jschauma 43 11/19/12
  44. 44. Miscellaneous Scalability Issues§  doing hostname lookups for each connection is expensive §  use IP addresses§  parsing known_hosts with several 100K entries is expensive §  use UserKnownHostsFile=/dev/null §  use GlobalKnownHostsFile=/dev/null§  hosts change ssh keys "frequently" §  we need StrictHostKeyChecking=no :-(§  collecting even only a few lines from every single host can fill up your disk quickly§  ssh connections can hang "indefinitely" (=> tkill(1) cron scheduled killing of processes)Headless Host Scanning -- @jschauma 44 11/19/12
  45. 45. Yay, Open Source!Headless Host Scanning -- @jschauma 45 11/19/12
  46. 46. Scanmaster Scripts§  checkhosts(1) §  run given script on all hosts in input via ssh(1) (in background)§  scanhosts(1) §  invoke checkhosts(1) against split input (again in background) §  defaults to 150 invocations of checkhosts(1) in parallel§  scanslave(1) §  starts ssh-agent(1) and adds keys §  adds tkill(1) crontab §  invoke scanhosts(1) against split input§  scanmaster(1) §  splits input into number of scanslaves chunks §  ssh to each scanslave and kicks off scanslave(1) in screen session §  aggregates and post-processes resultsHeadless Host Scanning -- @jschauma 46 11/19/12
  47. 47. Scanmaster Scripts$ wc -l * 330 checkhosts 259 scanhosts 325 scanmaster 179 scanslave 1093 total$ file *checkhosts: Bourne shell script text executablescanhosts: Bourne shell script text executablescanmaster: Bourne shell script text executablescanslave: Bourne shell script text executable$ Headless Host Scanning -- @jschauma 47 11/19/12
  48. 48. "Every item on the menu at Taco Bell is justa different configuration of roughly eightingredients. With this simple periodic tableof meat and produce, the company pulleddown $1.9 billion last year."http://teddziuba.com/2010/10/taco-bell-programming.htmlHeadless Host Scanning -- @jschauma 48 11/19/12
  49. 49. Miscellaneous Security Issues§  ssh-agent exposes our keys to target hosts §  use ForwardAgent=no, ClearAllForwardings=yes§  ssh-agent on scanslaves exposes our privkeys to anybody with super-user privileges on the scanslaves §  use a scan-only key with from= and various no- restrictions§  password injector will respond to anything asking for your assword §  uhm... dont use password auth?§  use of a "real" user is stupidHeadless Host Scanning -- @jschauma 49 11/19/12
  50. 50. Going topless. T.O.P.L.E.S.S.Headless User Requirement #1: §  selective access (via ssh keys)Headless User Requirement #2: §  a restricted shellHeadless User Requirement #3: §  an un-restricted shellHeadless User Requirement #2 and #3: §  allow arbitrary commands, provided they were sanctioned by somebody we trustHeadless Host Scanning -- @jschauma 50 11/19/12
  51. 51. Windows PowerShellHeadless Host Scanning -- @jschauma 51 11/19/12
  52. 52. Windows PowerShell ExecutionPolicy§  Restricted §  No scripts can be run. Windows PowerShell can be used only in interactive mode.§  Unrestricted §  No restrictions; all Windows PowerShell scripts can be run.§  RemoteSigned §  Downloaded scripts must be signed by a trusted publisher before they can be run.§  AllSigned §  Only scripts signed by a trusted publisher can be run.Headless Host Scanning -- @jschauma 52 11/19/12
  53. 53. The (TacoBell) SigshNAME sigsh -- a signature verifying shellSYNOPSIS sigsh [-c certs] [-x] [-p prog]DESCRIPTION sigsh is a non-interactive, signature requiring and verifying command interpreter. More accurately, it is a signature verification wrapper around a given shell. It reads input in PKCS#7 format from standard in, verifies the signature and, if the signature matches, pipes the decoded input into the command interpreter.Headless Host Scanning -- @jschauma 53 11/19/12
  54. 54. Yay, Open Source!Headless Host Scanning -- @jschauma 54 11/19/12
  55. 55. The (TacoBell) Sigsh$ find . ( -type f -a ! -path *.svn/* ) | xargs wc -l 50 ./test/sigsh.test.pl 36 ./certs/Makefile 5 ./certs/README 121 ./doc/sigsh.1 17 ./doc/Makefile 176 ./src/sigsh.sh 12 ./README 29 ./LICENSE 446 total$Headless Host Scanning -- @jschauma 55 11/19/12
  56. 56. Yay, Open Source!Headless Host Scanning -- @jschauma 56 11/19/12
  57. 57. The (TacoBell) Sigsh$ egrep -v "^[ ]*#" sigsh.sh | wc -l 93$Headless Host Scanning -- @jschauma 57 11/19/12
  58. 58. The (TacoBell) SigshEXAMPLES The following examples illustrate possible usage of this tool. To execute the commands in the file script.bash: openssl smime -sign -nodetach -signer mycert.pem -inkey mykey.pem -in script.bash -outform pem | sigsh To execute the perl code contained in the signed PKCS#7 file code.pem: sigsh -p /usr/bin/perl <code.pem Headless Host Scanning -- @jschauma 58 11/19/12
  59. 59. The Good, the Bad and the Ugly
  60. 60. The Good§  private cert need not be present on scanslave hosts§  signing party need not be executing party§  if you grant passwordless sudo to the headless user, you can do *§  certs can expire => we have builtin time-bomb powers!§  multiple people can have signing-power independent of each other§  we can scan over XXXK "hosts" in approximately 15 minutes§  we have the flexibility to collect all sorts of interesting dataHeadless Host Scanning -- @jschauma 60 11/19/12
  61. 61. Scanning over 400 hosts/second[...]16:09:14: 0 / 30 done - waiting...16:10:16: 0 / 30 done - waiting...16:11:18: 1 / 30 done - waiting...16:12:20: 6 / 30 done - waiting...16:13:22: 20 / 30 done - waiting...16:14:24: 27 / 30 done - waiting...16:15:25: 29 / 30 done - waiting...16:16:27: 29 / 30 done - waiting...16:17:30: 30 / 30 done16:17:30: Scanned XXXXXX hosts in XX minutes and 32 seconds.Aggregating data...Yahoo! Presentation; Confidential 11/19/12
  62. 62. The Bad§  this is still a TacoBell sigsh, not a powershell§  we have not solved the ssh-key access problem (though a leaked ssh-key cannot allow arbitrary command execution, only execution of previously approved commands)§  multiple people can have signing-power independent of each otherHeadless Host Scanning -- @jschauma 62 11/19/12
  63. 63. Life Lessons: Paretos PrincipleThe "trivial many" The "vital few" Headless Host Scanning -- @jschauma 63 11/19/12
  64. 64. Life Lessons: Milton Friedman was right.Theres nothing so permanent asa temporary government program solution.Yahoo! Presentation; Confidential 11/19/12
  65. 65. The Ugly§  Only ~80% of our hosts have sigsh => we need to combine headless with "headful" scans. Thus, we currently: §  scan headlessly (80%, in <15 minutes) §  scan with regular user and dedicated scankey (11%) §  scan with regular user and password, twice (7%) §  scan with regular user and old password §  scan with regular user and older password §  scan with regular user and even older password§  We still (have to) trust the binaries on the remote host. §  no veriexec, TPE, ...Headless Host Scanning -- @jschauma 65 11/19/12
  66. 66. Life Lessons§  Pen testing cant solve everything.§  Size matters.§  "Nothing is always absolutely so." (Sturgeons Law) ›  "90% of everything is crud." (2nd adage; freebie)§  Causality (Duh!)§  Mmm, mmm TacoBell!§  80% of the effects come from 20% of the causes. (Pareto Principle)§  Theres nothing so permanent as a temporary solution.§  The worst form of on-demand data collection except all the others that have been tried.Headless Host Scanning -- @jschauma 66 11/19/12
  67. 67. Information§  http://www.netmeister.org/apps/scanmaster/§  http://www.netmeister.org/apps/sigsh/§  jschauma@netmeister.org§  @jschaumaThis talk in 140 chars or less:Scanning 100s of K hosts headlessly using a signature verifyingshell is awful. Still better than most alternatives. That is all.Headless Host Scanning -- @jschauma 67 11/19/12

×