Addmi 13-discovery overview


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • The next four slides of animation show the basic approach to discovery alongside the nodes that are built in the model. The emphasis is that everything we do is recorded
  • On the first scan the only likely cause is “Excluded”. Very rarely you can get “OptAlreadyProcessing” if the same endpoint is injected while one is still in the queue - see later slide.
  • Pinging before scan allows us to optimise our detection of real device that will respond to discovery as opposed to dark space. Advanced Use “ Ping hosts before scanning” can be disabled globally for environments that suppress ICMP, but at the expense of slower performance in dark space. Consider the use of TCP ACK or TCP SYN ping to replace the standard ICMP ping if environment allows (“Use TCP ACK ping before scanning”, “Use TCP SYN ping before scanning”) or use “Exclude ranges from ping” if only a small area of the environment is an issue (maybe a DMZ)
  • If the endpoint responds to ping then discovery goes on to look for open ports. If the estate is hardened, discovery can have difficulty detecting open ports. In these situations consider modifying the discovery configuration setting “Valid Port States”. Contact support for advice before making modifications
  • It’s important that the appliance can see these ports open (or regarded as valid if you read the notes on the ports slide), otherwise discovery will not proceed. This list of ports has been aggressively honed from experience to focus only on regular stable service ports that are minimum risk whilst still allowing for effective discovery. Attempting to use fewer ports will reduce the quality and stability of discovery.
  • Depending on Dark Space settings we may or may not retain DiscoveryAccess nodes marked as NoResponse
  • UNIX methods will only be tried if the appliance can detect an open UNIX port (22 SSH, 23 telnet, 513 rlogin) at the end point and there is a credential for that endpoint *and* port in the vault.
  • If the slaves are restricted then only the ones valid for that endpoint. It is a common source of confusion, but vital to understand, that the slave is only a proxy and not a distributed discovery agent. If the appliance cannot detect that port 135 (Windows RPC) is open on the endpoint then discovery will not attempt to use ay windows slave. This can often be an issue with clients deploying Windows Slaves in protected areas of the network in the assumption this will allow scanning, it will not, and in this situation using multiple appliances and consolidation is the correct deployment. Advanced Use If there is no option but to have the appliance in a situation where it cannot detect port 135 on the endpoint then “Check port 135 before using Windows access methods” can be set to “no”. In this situation the appliance will direct all discovery requests that do not respond to a UNIX method via all registered slaves in sequence, this will cause discovery to take significantly longer per endpoint and noticeably degrade performance.
  • The SNMP discovery methods are more limited and should be regarded as fallback methods as they provide only basic information. No access to files or running of commands will be possible. The SNMP port is 161(UDP) OS currently supported in this fashion are IBM I (formerly OS/400), Netware, OpenVMS, z/OS (formerly OS/390). Netware is only available via SNMP v1
  • If the access methods have failed so far, then discovery will attempt the following methods to try to identify the device. If the device has a SNMP port 161 open, discovery will try to recover basic system information with a public community string. IP Stack Fingerprinting exploits the fact that is a close relation between an IP Stack and an OS, as each OS normally has a dedicated IP Stack; it is often possible to determine the OS quite accurately. But for IP Stack Fingerprinting to work well it needs to investigate closed as well as open ports. We use port 4 for the closed port. For the open ports we only use the ports used for our access methods. If the device has the telnet port 23 open than frequently the banner is presented before the login prompt and this will provide information about the device and its OS. Similarly a simple HTTP GET is used if port 80 is open. The results will often contain information about the device and its OS. All these methods are required for credentialess scanning. Disabling or modifying them is not recommended as without them identifying Hosts that need credentials to be deployed is very inefficient. Advanced Use IP Fingerprinting can be turned off with the “ Use IP Fingerprinting to Identify OS” option set to “no”, or the list of ports used for fingerprinting can be altered. Neither are recommended. Telnet banner sampling can be turned off using the “ Use Telnet Banner to Identify OS” option. SNMP SysDescr can be turned off by using the “ Use SNMP SysDescr to Identify OS” option. HTTP HEAD can be turned off by using the “ Use HTTP HEAD Request to Identify OS” option Contact support before attempt to change these settings.
  • At this stage we have already got a successful getDeviceInfo as we have an active session. In later modules we will refer back to the fact that these three methods need to succeed in order to creat/update a Host node.
  • Without success in completing DeviceInfo, HostInfo and InterfaceList we do not have enough information to feed the Host Identification algorithm. The system *can* cope with partial results in those methods, although the identity of the Host will be less stable the less properties it has to work on. Common reasons to not complete: Credential permissions Poor edits to scripts with uncaught stderr or other script termination issues. Login Timeout – check for timeout Script Timeout – check for timeout ScriptFailure related to the method. Increase the credential timeout to 180 seconds Parse failure (or incomplete DeviceInfo) – check for parsefailure ScriptFailure related to the method, check for scrambled session output. Turn on session logging and check for out of sequence characters. Consider increasing Session Line Delay,
  • The Host Algorithm uses a weighting technique to try and compute a key. The weighting compares the current properties with those from existing candidate Host nodes. If there is a difference and it is significant a new Host.key is generated, otherwise it uses the closest match. This allows a certain amount of change (such as upgrading an OS or changing a NIC) without forcing a new identity. We cannot compare every existing Host so we pre select candidates. These include the Host that this endpoint was associated with last time, Hosts with interfaces on the same IP as the current endpoint as well as Hosts that have the same serial number as that of the current properties.
  • end_state only relates to establishing a good quality session to the endpoint and relating it to an existing node.
  • On the first scan the only likely cause is “Excluded”. Very rarely you can get “OptAlreadyProcessing” if the same endpoint is injected while one is still in the queue - see later slide. Later on we can get OptNotBestIP and OptRemote from the optimization systems – these are described next OptNotBestIP – we know this endpoint was optimized last time so we assume it will be this time and do not contact it OptRemote – only seen on a Consolidation Appliance. Means that the endpoint was optimised on the Scanning Appliance. Full details of state will be on the Scanning Appliance.
  • Why do we still go to the OS/Device classifier, rather than further down? Because if we are using widely deployed credentials they may well work on another Host, and we still have to check if it is the same Host and not another one that the credentials works on that has moved to this IP since we last scanned.
  • Under some conditions the same IP can be requested while another scan of the same IP is still in progress. To prevent collisions if a duplicate is detected then one of the endpoints is skipped.
  • In general the level of access to the OSI over each interface is the same. There is no point scanning over the same endpoint several times in a range (or indeed across ranges) so we should only scan over one of the interfaces
  • Essentially the first endpoint that provides the GoodAccess end_state is the one we will attempt to use. Note hat as we have recovered up to date DeviceInfo, HostInfo and InterfaceList these properties of the Host node are updated. This is fine detail and probably only confuses the issue in an overview, but is included in the notes for completeness. Sometimes we will talk about the “BestIP”, this is an internal name for the system that picks the highest quality endpoint and is sometimes used to refer to the endpoint that is picked.
  • By default we will do this every 7 days. Advanced Use The setting is controlled by the value of “Scan optimization timeout” and this is a Model Maintenance setting rather than a discovery one. We don’t advise changing this value without advice.
  • It’s highly unlikely that you will get an early error state as that suggests a fundamental error in the core system and these are picked up in internal testing if they occur. More likely is an error from amended discovery scripts.
  • Pattern success or failure does not alter the summary states that track session establishment. These have their own tracking methods that will be described later. Note also that not all the standard discovery scripts may have completed successfully. Again further tracking methods will be described later to allow any issues to be understood.
  • This is subtle change but a Sweep Scan scan level never intends to get beyond a DeviceIndentified vs NoRepsonse state as it is intended for surveys of the estate during roll out and sizing of the project. Other scan levels are not included in the chart as these are the two that should be used during normal use; other scan levels should be used under guidance.
  • You may wish to download the state charts that were used during this module. Please download the chart zip file that should be available where you accessed this module.
  • Addmi 13-discovery overview

    1. 1. Discovery Overview Getting Data from the Estate
    2. 2. Outline <ul><li>The Basic Philosophy </li></ul><ul><li>First Scan </li></ul><ul><li>Second Scan </li></ul><ul><li>Optimization </li></ul><ul><li>Bringing It All Together </li></ul><ul><ul><li>Completed State Charts </li></ul></ul><ul><ul><li>Summary </li></ul></ul>
    3. 3. The Basic Philosophy
    4. 4. Basic Discovery Sequence (1) <ul><li>We start a scan of a collection of IPs </li></ul><ul><li>We try to contact each one – we record success/failure </li></ul><ul><li>We try to establish what sort of device is on the IP from a combination of heuristics and direct access and record that </li></ul>Discovery Run Discovery Access Device Info Discovery Run Discovery Run Discovery Access
    5. 5. Basic Discovery Sequence (2) <ul><li>If the device is a Host and we were able to log on we try and get more information about it and it’s list of interfaces </li></ul><ul><li>If we get the Host Information and Interface List then we have enough to try to infer a host, and we carry on with our standard set of discovery </li></ul>Discovery Run Discovery Access Device Info Host Info Interface List Discovery Run Discovery Access Device Info Host Info Interface List Host Processes
    6. 6. Basic Discovery Sequence (3) <ul><li>Once we have finished standard discovery we can infer other hardware items like Network Interfaces and link the Host to global nodes like subnet </li></ul><ul><li>At this stage in the sequence we will start inferring software products, correlating virtual hosts with their physical hosts etc </li></ul>Discovery Run Discovery Access Device Info Host Info Interface List Process List Host Discovered Network Interfaces NIC Subnet
    7. 7. Basic Discovery Sequence (4) <ul><li>Although our standard discovery has finished, we still have the ability to request more information if we need it </li></ul><ul><li>This could be the pattern to create software looking for further data or a platform related pattern looking for additional information to do with the Host </li></ul><ul><li>Additional discovery will be done via commands, files, registry queries, WMI queries, SNMP queries, SQL queries… </li></ul>Discovery Run Discovery Access Device Information Host Information Interface List Process List Host Oracle Discovered File
    8. 8. First Scan Details
    9. 9. <ul><li>We will build up the following Discovery Access state chart: </li></ul><ul><ul><li>The DiscoveryAccess node contains 3 key summary attributes to record what happened during session establishment: </li></ul></ul>Discovery Walk Through result [ Success | NoAccess | Skipped | NoResponse ] end_state [ From state diagram ] reason [ Free text summary reason for lack of success ]
    10. 10. Is Access Allowed? <ul><li>Check to see if we are allowed to access the endpoint </li></ul><ul><li>result = Skipped </li></ul><ul><li>end_state = Excluded </li></ul>
    11. 11. Ping Response? <ul><li>Pings the endpoint to see if anything responds </li></ul>
    12. 12. Check For Open Ports <ul><li>If something responds then we see if any of the ports we can use are open </li></ul>
    13. 13. Ports <ul><li>Ports we can use </li></ul><ul><li>UNIX </li></ul><ul><ul><li>22 SSH </li></ul></ul><ul><ul><li>23 Telnet </li></ul></ul><ul><ul><li>513 rlogin </li></ul></ul><ul><li>Windows </li></ul><ul><ul><li>135 RPC </li></ul></ul><ul><li>SNMP </li></ul><ul><ul><li>161 (UDP) SNMP </li></ul></ul><ul><li>External OS Detection Only </li></ul><ul><ul><li>4 (closed port for IP fingerprint) </li></ul></ul><ul><ul><li>80 HTTP </li></ul></ul>
    14. 14. Dark Space <ul><li>If we have not got a response at this point we regard this endpoint as Dark Space </li></ul><ul><li>end_state = NoResponse </li></ul><ul><li>result = NoResponse </li></ul>
    15. 15. Credential Vault <ul><li>We look in the vault for a credential that matches: </li></ul><ul><ul><li>The IP of the endpoint </li></ul></ul><ul><ul><li>The service ports seen open on the device (SSH, Telnet..) </li></ul></ul>
    16. 16. Credential Selection <ul><li>We try to establish a session with each one of the credentials that match </li></ul><ul><li>Used in the order they are defined in the vault (as seen on the credential page) </li></ul>
    17. 17. UNIX Access <ul><li>We try the UNIX access methods first </li></ul><ul><li>Only tried if we found a UNIX port open </li></ul><ul><li>If we get a response we ask the device what it is </li></ul>
    18. 18. Windows Access <ul><li>If the UNIX methods don’t get a result then we try the Windows methods </li></ul><ul><li>Only tried if we found a Windows port open </li></ul><ul><li>Slaves are used in the order defined </li></ul><ul><li>If we get a response we ask the device what it is </li></ul>
    19. 19. SNMP Access <ul><li>If the UNIX and Windows methods don’t get a result then we try the SNMP methods </li></ul><ul><li>Only tried if we found the SNMP port open </li></ul><ul><li>If we get a response we ask the device what it is </li></ul>
    20. 20. Other Attempts <ul><li>If no access methods have worked we try to determine what the device might be from external evidence </li></ul><ul><ul><li>SNMP SysDescr </li></ul></ul><ul><ul><li>IP Stack Fingerprinting </li></ul></ul><ul><ul><li>Telnet Banner </li></ul></ul><ul><ul><li>HTTP HEAD </li></ul></ul>
    21. 21. Host Classification <ul><li>All the results so far go through the OS/Device classifier </li></ul><ul><li>If the device is a “Host” we will continue </li></ul><ul><li>Otherwise we skip this endpoint </li></ul><ul><li>end_state = UnsupportedDevice </li></ul><ul><li>result = skipped </li></ul>
    22. 22. Once We Determine a Host… <ul><li>Now we know it is a Host we return to the session and ask for </li></ul><ul><ul><li>getHostInfo </li></ul></ul><ul><ul><li>getInterfaceList </li></ul></ul><ul><li>Both these are critical for running the Host Identification algorithm </li></ul>
    23. 23. Access Failure <ul><li>If these methods fail to complete then discovery stops here </li></ul><ul><li>end_state = NoAccess </li></ul><ul><li>result = NoAccess </li></ul>
    24. 24. Host Identity Algorithm <ul><li>Host Algorithm uses strong identity properties to compute a Host.key </li></ul><ul><ul><li>OS, Kernel, MAC, IP, Serial,… </li></ul></ul><ul><li>A new Host is created </li></ul><ul><li>end_state = GoodAccess </li></ul><ul><li>result = Success </li></ul>
    25. 25. Credential Caching <ul><li>We cache the successfully used credential and slave for use next time </li></ul>
    26. 26. Further Discovery <ul><li>Standard discovery continues collecting </li></ul><ul><ul><li>Processes </li></ul></ul><ul><ul><li>Packages </li></ul></ul><ul><ul><li>Etc </li></ul></ul><ul><li>TPL based discovery starts after Standard discovery </li></ul><ul><li>No further change to end_state </li></ul>
    27. 27. Second Scan Details
    28. 28. Is Access Allowed? <ul><li>Check to see if we allowed to access the endpoint </li></ul><ul><li>result = Skipped </li></ul><ul><li>end_state = Excluded, end_state = OptNotBestIP or end_state = OptRemote </li></ul>
    29. 29. Cached Credential <ul><li>We check to see if there are cached results from the previous access to this endpoint </li></ul>
    30. 30. Use Last Slave <ul><li>We try the previous credential/slave to see if we make contact </li></ul><ul><li>This shortcuts establishing a session </li></ul>
    31. 31. Does This Cached Attempt Succeed? <ul><li>If we do not re-establish a session we have to go back to the full analysis </li></ul><ul><li>If we do succeed we start at the OS / Device classifier </li></ul>
    32. 32. Back to the Standard Tasks… <ul><li>OS/Device classifier </li></ul><ul><li>HostInfo/InterfaceList </li></ul><ul><li>Host Algorithm </li></ul>
    33. 33. Optimization Details
    34. 34. Optimize – Skipped Endpoints <ul><li>In order to maximise throughput and reduce load on the targets there are a series of optimisations </li></ul><ul><li>These can result in skipped endpoints </li></ul>
    35. 35. Duplicate IP in Progress <ul><li>To prevent collisions if an IP is already in progress duplicates are dropped </li></ul><ul><li>result = Skipped </li></ul><ul><li>end_state = OptAlreadyProcessing </li></ul>
    36. 36. Best IP <ul><li>Many Hosts have more than one active interface </li></ul><ul><li>Many endpoints in a range that relate to the same Host </li></ul><ul><li>No point scanning the same Host 12 times in one range </li></ul>
    37. 37. Best IP - Aims <ul><li>Scan over a single endpoint </li></ul><ul><li>Try and keep the single endpoint chosen stable over time </li></ul><ul><li>Minimize network access </li></ul>
    38. 38. Best IP – 1 st Scan Optimization <ul><li>Kicks in when we try to update an existing Host </li></ul><ul><li>If the Host has already been updated by an endpoint reaching the GoodAccess end_state discovery stops </li></ul><ul><li>result = Skipped </li></ul><ul><li>end_state = Opt1stScan </li></ul>
    39. 39. Multiple IP – 2 nd Scan Optimization <ul><li>Kicks in when we scan the same endpoint again </li></ul><ul><li>We check to see if it was optimised last time, if it was we assume it will be again </li></ul><ul><li>Discovery stops here </li></ul><ul><li>result = Skipped </li></ul><ul><li>end_state = OptNotBestIP </li></ul>
    40. 40. Multiple IP – 2 nd Scan Optimization <ul><li>We don’t assume the optimisation is correct forever </li></ul><ul><li>Every so often we will contact the Host again to confirm </li></ul><ul><ul><li>If it turns out the IP is new Host full discovery will occur </li></ul></ul><ul><ul><li>Otherwise you will get 1 st Scan Optimization again </li></ul></ul>
    41. 41. Bringing It All Together
    42. 42. Errors <ul><li>If an error occurs the result will be set Error </li></ul><ul><li>The end_state will be set to the last state reached, or Error if it occurs before any other state </li></ul>
    43. 43. Complete State Chart <ul><li>Bringing together all the states allows us to draw a complete state chart for standard discovery </li></ul>
    44. 44. Complete State Chart – Additional Discovery <ul><li>Discovery doesn’t stop once a session is established </li></ul><ul><li>Patterns will cause further discovery as required </li></ul>
    45. 45. Complete State Chart – Sweep Scan <ul><li>If the scan level is restricted to Sweep Scan then the DeviceIndentified state is regarded as a success </li></ul><ul><li>result = Success </li></ul><ul><li>end_state = DeviceIdentified </li></ul>
    46. 46. Summary <ul><li>Atrium Discovery can be restricted from scanning sensitive/high risk endpoints </li></ul><ul><li>Atrium Discovery needs to be able to see network ports on the target to pick the right access methods and credentials </li></ul><ul><li>Atrium Discovery does full discovery of Host devices and basic discovery of other devices </li></ul><ul><li>Atrium Discovery caches successful access methods and credentials for faster future session establishment </li></ul><ul><li>Atrium Discovery optimizes it’s access to ensure that Hosts with multiple IPs are not repeatedly scanned </li></ul><ul><li>Atrium Discovery needs to succeed with getDeviceInfo , getHostInfo and getInterfaceList in order to create/update a Host node </li></ul>
    47. 47. <ul><li>Online Documentation: </li></ul><ul><ul><li> </li></ul></ul>Further Resources Tideway Foundation Version 7.2 Documentation Title
    48. 48. Discovery Overview State Charts