Headless Host Scanning
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Headless Host Scanning

on

  • 718 views

A talk I gave at Twitter in April 2011 about scanmaster and sigsh and how to scan hundreds of thousands of hosts in minutes.

A talk I gave at Twitter in April 2011 about scanmaster and sigsh and how to scan hundreds of thousands of hosts in minutes.

Statistics

Views

Total Views
718
Views on SlideShare
713
Embed Views
5

Actions

Likes
0
Downloads
4
Comments
0

3 Embeds 5

https://www.linkedin.com 3
https://twitter.com 1
http://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Headless Host Scanning Presentation Transcript

  • 1. Headless Host Scanning Jan Schaumann <jschauma@netmeister.org>99CE 1DC7 770A C5A8 09A6 0DCD 66CE 4FE9 6F6B D3D7 @jschauma
  • 2. Headless Host Scanning -- @jschauma 2 11/19/12
  • 3. Life Lessons: Pen testing cant solve everything.Headless Host Scanning -- @jschauma 3 11/19/12
  • 4. Inside-OutHeadless Host Scanning -- @jschauma 4 11/19/12
  • 5. Distance to Yahoo! ad in feet: 404 Construction cost: $357 million Capacity: 41,915Headless Host Scanning -- @jschauma 5 11/19/12
  • 6. Ballpark numbers Distance to Yahoo! ad in feet: 30927 Construction cost: $1.5 billion Capacity: 52,325 Headless Host Scanning -- @jschauma 6 11/19/12
  • 7. Life Lessons: Size mattersHeadless Host Scanning -- @jschauma 7 11/19/12
  • 8. It all began some time in January of 2007...Yahoo! Presentation; Confidential 11/19/12
  • 9. Initial Problem Statement“Hey, we need to know which hosts can’t deal with the new Daylight Saving Time.” Headless Host Scanning -- @jschauma 9 11/19/12
  • 10. A Systems Administrators ToolboxHeadless Host Scanning -- @jschauma 10 11/19/12
  • 11. A simple tool perl python /bin/shHeadless Host Scanning -- @jschauma 11 11/19/12
  • 12. Initial "Solution"#!/bin/shwhile read host; do ssh $host "env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’"doneHeadless Host Scanning -- @jschauma 12 11/19/12
  • 13. Problem Statement “Please run this script on these hosts.”Headless Host Scanning -- @jschauma 13 11/19/12
  • 14. 2nd Pass#!/bin/shwhile read host; do scp script.sh $host:/tmp/ ssh $host "/tmp/script.sh"doneHeadless Host Scanning -- @jschauma 14 11/19/12
  • 15. Mensch, nutz doch ein Konfigurationsmanagementtool!Headless Host Scanning -- @jschauma 15 11/19/12
  • 16. Configuration ManagementHeadless Host Scanning -- @jschauma 16 11/19/12
  • 17. Configuration ManagementGood at:§  keeping state§  ensuring flat text file propertiesNot so good at:§  deploying applications§  executing (one-time) scriptsTerrible at:§  reporting output of individual commands or scriptsHeadless Host Scanning -- @jschauma 17 11/19/12
  • 18. Life Lessons: Sturgeons Law (1st adage)"Nothing is always absolutely so."Headless Host Scanning -- @jschauma 18 11/19/12
  • 19. Mensch, nutz doch Deine Logaggregationsinfrastruktur!Headless Host Scanning -- @jschauma 19 11/19/12
  • 20. Log Aggregation§  makes assumptions about message format§  frequently unreliable and/or asynchronous§  adds additional complexityHeadless Host Scanning -- @jschauma 20 11/19/12
  • 21. 3rd Pass#!/bin/shwhile read host; do cat script.sh | ssh $host "bash –s"doneHeadless Host Scanning -- @jschauma 21 11/19/12
  • 22. 3rd Pass#!/bin/shwhile read host; do ssh $host "bash –s" < script.shdoneHeadless Host Scanning -- @jschauma 22 11/19/12
  • 23. Problem Refinement “Run this set of arbitrary commands on our hosts. Yes, on all of them.”Headless Host Scanning -- @jschauma 23 11/19/12
  • 24. Wait a second – all hosts?$ wtf hostwtf: I dont know what host means!$§  a computer connected to a network§  a virtual machine§  a container §  jails, zones, ... §  chrootHeadless Host Scanning -- @jschauma 24 11/19/12
  • 25. Sturgeons Law (1st adage) "Hosts" on the network"Hosts" in asset database "Hosts" in DNS Yahoo! Presentation; Confidential 11/19/12
  • 26. 3rd Pass#!/bin/shwhile read host; do ssh $host "bash –s" < script.shdoneHeadless Host Scanning -- @jschauma 26 11/19/12
  • 27. 4th PassHeadless Host Scanning -- @jschauma 27 11/19/12
  • 28. Headless Host Scanning -- @jschauma 28 11/19/12
  • 29. Life Lessons: Causality (Duh!)$ w10:45AM up 51 days, 22:19, 11 users, load averages: 976.23, 618.17, 1020.25$ time ls 13.354 real 0.001 user 0.002 sys$ Headless Host Scanning -- @jschauma 29 11/19/12
  • 30. Horizontal Scalability Scanmaster Scanslaves Scanslaves TargethostsHeadless Host Scanning -- @jschauma 30 11/19/12
  • 31. SSH Complications: assword prompts Interactive connections? Password: $USER@hostnames password:Headless Host Scanning -- @jschauma 31 11/19/12
  • 32. SSH Complications: assword prompts Interactive connections begat password injectors.Headless Host Scanning -- @jschauma 32 11/19/12
  • 33. SSH Complications: assword prompts proc respond { pw } { global pws if {![info exists pws($pw)]} { send_user " (autopw) " stty -echo expect_user -re "(.+)n" stty echo set pws($pw) [exec perl -ne "print pack(q{u}, $_)" << $expect_out(1,string)] } else { send_user " (autopwed)" } log_user 0 send -- "[exec perl -ne "print unpack(q{u}, $_)" << $pws($pw)]n"; log_user 1 } eval spawn -noecho $argv expect { "assword:" { respond system exp_continue }Yahoo! Presentation; Confidential 11/19/12
  • 34. SSH Complications: assword prompts Key Authentication to the rescue!Yahoo! Presentation; Confidential 11/19/12
  • 35. Horizontal Scalability Scanmaster Scanslaves Scanslaves TargethostsYahoo! Presentation; Confidential 11/19/12
  • 36. SSH Complications: assword prompts Key Authentication to the rescue?§  private keys need to be present on all hosts connecting to targets§  you need to provide passphrase for private keys at scan startEnter passphrase for /home/${USER}/.ssh/id_rsa:Enter passphrase for /home/${USER}/.ssh/id_dsa:Enter passphrase for /home/${USER}/.ssh/scankey:Yahoo! Presentation; Confidential 11/19/12
  • 37. SSH Complications: assword prompts Key Authentication also uses password injectors.Headless Host Scanning -- @jschauma 37 11/19/12
  • 38. SSH Complications: assword prompts SSH Agent Forwarding to the rescue!§  private keys stored in agent on single "safe" machine§  agent forwarding enabled to scan slaves§  everythings peachy!Headless Host Scanning -- @jschauma 38 11/19/12
  • 39. SSH Agent Forwarding proxy agent Slave Targethosts MasterHeadless Host Scanning -- @jschauma 39 11/19/12
  • 40. SSH Agent Forwarding proxy agent Slave Targethosts MasterHeadless Host Scanning -- @jschauma 40 11/19/12
  • 41. SSH Agent Forwarding MasterHeadless Host Scanning -- @jschauma 41 11/19/12
  • 42. SSH Agent ForwardingHeadless Host Scanning -- @jschauma 42 11/19/12
  • 43. Scalability issues Scanmaster Scanslaves Scanslaves TargethostsHeadless Host Scanning -- @jschauma 43 11/19/12
  • 44. Miscellaneous Scalability Issues§  doing hostname lookups for each connection is expensive §  use IP addresses§  parsing known_hosts with several 100K entries is expensive §  use UserKnownHostsFile=/dev/null §  use GlobalKnownHostsFile=/dev/null§  hosts change ssh keys "frequently" §  we need StrictHostKeyChecking=no :-(§  collecting even only a few lines from every single host can fill up your disk quickly§  ssh connections can hang "indefinitely" (=> tkill(1) cron scheduled killing of processes)Headless Host Scanning -- @jschauma 44 11/19/12
  • 45. Yay, Open Source!Headless Host Scanning -- @jschauma 45 11/19/12
  • 46. Scanmaster Scripts§  checkhosts(1) §  run given script on all hosts in input via ssh(1) (in background)§  scanhosts(1) §  invoke checkhosts(1) against split input (again in background) §  defaults to 150 invocations of checkhosts(1) in parallel§  scanslave(1) §  starts ssh-agent(1) and adds keys §  adds tkill(1) crontab §  invoke scanhosts(1) against split input§  scanmaster(1) §  splits input into number of scanslaves chunks §  ssh to each scanslave and kicks off scanslave(1) in screen session §  aggregates and post-processes resultsHeadless Host Scanning -- @jschauma 46 11/19/12
  • 47. Scanmaster Scripts$ wc -l * 330 checkhosts 259 scanhosts 325 scanmaster 179 scanslave 1093 total$ file *checkhosts: Bourne shell script text executablescanhosts: Bourne shell script text executablescanmaster: Bourne shell script text executablescanslave: Bourne shell script text executable$ Headless Host Scanning -- @jschauma 47 11/19/12
  • 48. "Every item on the menu at Taco Bell is justa different configuration of roughly eightingredients. With this simple periodic tableof meat and produce, the company pulleddown $1.9 billion last year."http://teddziuba.com/2010/10/taco-bell-programming.htmlHeadless Host Scanning -- @jschauma 48 11/19/12
  • 49. Miscellaneous Security Issues§  ssh-agent exposes our keys to target hosts §  use ForwardAgent=no, ClearAllForwardings=yes§  ssh-agent on scanslaves exposes our privkeys to anybody with super-user privileges on the scanslaves §  use a scan-only key with from= and various no- restrictions§  password injector will respond to anything asking for your assword §  uhm... dont use password auth?§  use of a "real" user is stupidHeadless Host Scanning -- @jschauma 49 11/19/12
  • 50. Going topless. T.O.P.L.E.S.S.Headless User Requirement #1: §  selective access (via ssh keys)Headless User Requirement #2: §  a restricted shellHeadless User Requirement #3: §  an un-restricted shellHeadless User Requirement #2 and #3: §  allow arbitrary commands, provided they were sanctioned by somebody we trustHeadless Host Scanning -- @jschauma 50 11/19/12
  • 51. Windows PowerShellHeadless Host Scanning -- @jschauma 51 11/19/12
  • 52. Windows PowerShell ExecutionPolicy§  Restricted §  No scripts can be run. Windows PowerShell can be used only in interactive mode.§  Unrestricted §  No restrictions; all Windows PowerShell scripts can be run.§  RemoteSigned §  Downloaded scripts must be signed by a trusted publisher before they can be run.§  AllSigned §  Only scripts signed by a trusted publisher can be run.Headless Host Scanning -- @jschauma 52 11/19/12
  • 53. The (TacoBell) SigshNAME sigsh -- a signature verifying shellSYNOPSIS sigsh [-c certs] [-x] [-p prog]DESCRIPTION sigsh is a non-interactive, signature requiring and verifying command interpreter. More accurately, it is a signature verification wrapper around a given shell. It reads input in PKCS#7 format from standard in, verifies the signature and, if the signature matches, pipes the decoded input into the command interpreter.Headless Host Scanning -- @jschauma 53 11/19/12
  • 54. Yay, Open Source!Headless Host Scanning -- @jschauma 54 11/19/12
  • 55. The (TacoBell) Sigsh$ find . ( -type f -a ! -path *.svn/* ) | xargs wc -l 50 ./test/sigsh.test.pl 36 ./certs/Makefile 5 ./certs/README 121 ./doc/sigsh.1 17 ./doc/Makefile 176 ./src/sigsh.sh 12 ./README 29 ./LICENSE 446 total$Headless Host Scanning -- @jschauma 55 11/19/12
  • 56. Yay, Open Source!Headless Host Scanning -- @jschauma 56 11/19/12
  • 57. The (TacoBell) Sigsh$ egrep -v "^[ ]*#" sigsh.sh | wc -l 93$Headless Host Scanning -- @jschauma 57 11/19/12
  • 58. The (TacoBell) SigshEXAMPLES The following examples illustrate possible usage of this tool. To execute the commands in the file script.bash: openssl smime -sign -nodetach -signer mycert.pem -inkey mykey.pem -in script.bash -outform pem | sigsh To execute the perl code contained in the signed PKCS#7 file code.pem: sigsh -p /usr/bin/perl <code.pem Headless Host Scanning -- @jschauma 58 11/19/12
  • 59. The Good, the Bad and the Ugly
  • 60. The Good§  private cert need not be present on scanslave hosts§  signing party need not be executing party§  if you grant passwordless sudo to the headless user, you can do *§  certs can expire => we have builtin time-bomb powers!§  multiple people can have signing-power independent of each other§  we can scan over XXXK "hosts" in approximately 15 minutes§  we have the flexibility to collect all sorts of interesting dataHeadless Host Scanning -- @jschauma 60 11/19/12
  • 61. Scanning over 400 hosts/second[...]16:09:14: 0 / 30 done - waiting...16:10:16: 0 / 30 done - waiting...16:11:18: 1 / 30 done - waiting...16:12:20: 6 / 30 done - waiting...16:13:22: 20 / 30 done - waiting...16:14:24: 27 / 30 done - waiting...16:15:25: 29 / 30 done - waiting...16:16:27: 29 / 30 done - waiting...16:17:30: 30 / 30 done16:17:30: Scanned XXXXXX hosts in XX minutes and 32 seconds.Aggregating data...Yahoo! Presentation; Confidential 11/19/12
  • 62. The Bad§  this is still a TacoBell sigsh, not a powershell§  we have not solved the ssh-key access problem (though a leaked ssh-key cannot allow arbitrary command execution, only execution of previously approved commands)§  multiple people can have signing-power independent of each otherHeadless Host Scanning -- @jschauma 62 11/19/12
  • 63. Life Lessons: Paretos PrincipleThe "trivial many" The "vital few" Headless Host Scanning -- @jschauma 63 11/19/12
  • 64. Life Lessons: Milton Friedman was right.Theres nothing so permanent asa temporary government program solution.Yahoo! Presentation; Confidential 11/19/12
  • 65. The Ugly§  Only ~80% of our hosts have sigsh => we need to combine headless with "headful" scans. Thus, we currently: §  scan headlessly (80%, in <15 minutes) §  scan with regular user and dedicated scankey (11%) §  scan with regular user and password, twice (7%) §  scan with regular user and old password §  scan with regular user and older password §  scan with regular user and even older password§  We still (have to) trust the binaries on the remote host. §  no veriexec, TPE, ...Headless Host Scanning -- @jschauma 65 11/19/12
  • 66. Life Lessons§  Pen testing cant solve everything.§  Size matters.§  "Nothing is always absolutely so." (Sturgeons Law) ›  "90% of everything is crud." (2nd adage; freebie)§  Causality (Duh!)§  Mmm, mmm TacoBell!§  80% of the effects come from 20% of the causes. (Pareto Principle)§  Theres nothing so permanent as a temporary solution.§  The worst form of on-demand data collection except all the others that have been tried.Headless Host Scanning -- @jschauma 66 11/19/12
  • 67. Information§  http://www.netmeister.org/apps/scanmaster/§  http://www.netmeister.org/apps/sigsh/§  jschauma@netmeister.org§  @jschaumaThis talk in 140 chars or less:Scanning 100s of K hosts headlessly using a signature verifyingshell is awful. Still better than most alternatives. That is all.Headless Host Scanning -- @jschauma 67 11/19/12