networking
- 1. Rob Thomas 60 Daysof Basic Naughtiness Probes and Attacks Endured by an Active Web Site 16 March 2001
- 2. Rob Thomas 60 Daysof Basic Naughtiness • Statistical analysis of log and IDS files. • Statistical analysis of a two-day DDoS attack. • Methods of mitigation. • Questions.
- 3. Rob Thomas About theSite • Production site for several (> 4) years. • Largely static content. • No e-commerce. • Layers of defense – more on that later!
- 4. Rob Thomas About theData • Data from router logs. • Data from IDS logs. • Snapshot taken from 60 days of combined data. • Data processed by several home-brew tools (mostly Perl and awk).
- 5. Rob Thomas Definition of“Naughty” • Any traffic that is logged by a specific “deny” ACL. • Any traffic that presents a pattern detected by the IDS software. • The two log sources are not necessarily synchronized.
- 6. Rob Thomas Daily Probesand Attacks • TCP and UDP Probes and Attacks – ICMP not counted. • Average – 529.00 • Standard deviation – 644.10! • 60 Day Low – 83.00 • 60 Day High – 4355.00
- 7. Rob Thomas Daily Probesand Attacks Daily Probes and Attacks 0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000 11/17/00 11/22/00 11/27/00 12/2/00 12/7/00 12/12/00 12/17/00 12/22/00 12/27/00 1/1/01 1/6/01 1/11/01 Day Hits TCP UDP
- 8. Rob Thomas Weekly Probesand Attacks • There is no steady-state. • Attacks come in waves, generally on the heels of a new exploit and scan. • Certain types of scans (e.g. Netbios) tend to run 24x7x365. • Proactive monitoring, based on underground and public alerts, will result in significant data capture.
- 9. Rob Thomas Weekly Probesand Attacks Trend Analysis Weekly Probes and Attacks 0 1000 2000 3000 4000 5000 6000 7000 8000 11/12 - 11/18 11/19 - 11/25 11/26 - 12/02 12/03 - 12/09 12/10 - 12/16 12/17 - 12/23 12/24 - 12/30 12/31 - 01/06 01/07 - 01/13 01/14 - 01/20 Week Hits Hits
- 10. Rob Thomas Hourly Probesand Attacks • Myth: “Most attacks occur at night.” • An attacker’s evening may be a victim’s day – the nature of a global network. • Truth: Don’t plan based on the clock.
- 11. Rob Thomas Hourly Probesand Attacks Trend Analysis Hourly Probes and Attacks 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 24 Hour Clock Hits
- 12. Rob Thomas UDP Probesand Attacks Top Five Destination Ports • First – 137 NETBIOS • Second – 53 DNS • Third – 27960 • Fourth – 500 ISAKMP • Fifth – 33480 (likely UNIX traceroute)
- 13. Rob Thomas UDP Probesand Attacks Trend Analysis UDP Probes and Attacks 0 50 100 150 200 250 300 350 11/17/00 11/24/00 12/1/00 12/8/00 12/15/00 12/22/00 12/29/00 1/5/01 1/12/01 Day NumberofHits Port 137 Hits Port 53 Hits
- 14. Rob Thomas TCP Probesand Attacks Top Five Destination Ports • First – 3663 (DDoS Attack) • Second – 0 Reserved (DDoS Attack) • Third – 6667 IRC (DDoS Attack) • Fourth – 81 (DDoS Attack) • Fifth – 21 FTP-control
- 15. Rob Thomas TCP Probesand Attacks Trend Analysis TCP Probes and Attacks 0 20 40 60 80 100 120 11/17/00 11/24/00 12/1/00 12/8/00 12/15/00 12/22/00 12/29/00 1/5/01 1/12/01 Date Hits Port 0 Hits Port 21 Hits
- 16. Rob Thomas Source Addressof Probes and Attacks Classful Sources of Probes and Attacks 0 500 1000 1500 2000 2500 3000 3500 A B C D E IP Netblock Class NumberofUniqueIPAddressesSeen Source Address Class Percentage 20% 7% 20% 26% 27% A B C D E
- 17. Rob Thomas Source Addressof Probes and Attacks Bogon Source Percentages 2346 803 2275 1128 167 270 0 500 1000 1500 2000 2500 3000 3500 4000 A B C IP Netblock Class UniqueIPAddresses Bogon Addresses Total Addresses
- 18. Rob Thomas Source Addressof Probes and Attacks • Bogon source attacks still common. • Of all source addresses, 53.39% were in the Class D and Class E space. • Percentage of bogons, all classes – 66.85%! • This is good news – prefix-list, ACL defense, and uRPF will block 66.85% of these nasties!
- 19. Rob Thomas Source Regionof the Naughty A dangerously misleading slide RIR for Source Addresses 58% 37% 5% ARIN RIPE APNIC
- 20. Rob Thomas Intrusion (attempt)Detection • IDS is not foolproof! • Incorrect fingerprinting does occur. • You can not identify that which you can not see.
- 21. Rob Thomas Top FiveIDS Detected Probes IDS Detected Probes 0 200 400 600 800 1000 1200 1400 NetBus Backorifice TFTP IDENT Deep Throat Type Hits
- 22. Rob Thomas Top FiveDetected IDS Probes IDS Detected Probes - Trend Analysis 0 20 40 60 80 100 120 140 160 180 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 Date Hits NetBus Backorifice TFTP IDENT Deep Throat
- 23. Rob Thomas Top FiveIDS Detected Attacks IDS Detected Attacks 0 50 100 150 200 250 300 350 400 450 500 TCP Port 0 FIN flood Fragments ICMP flood RST flood Type Hits Number
- 24. Rob Thomas Top FiveIDS Detected Sources IDS Detected Source Netblocks 0 20 40 60 80 100 120 140 160 180 200 Azerbaijan USA 01 South Korea USA 02 Canada Netblock Location Hits Count
- 25. Rob Thomas Top FiveIDS Detected Sources IDS Detected Attacks - Trend Analysis 0 20 40 60 80 100 120 140 160 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 Day Hits A B C D E
- 26. Rob Thomas Match aSource with a Scan Source to Hit Matching 0 20 40 60 80 100 120 140 160 1 2 3 4 5 6 7 Day Hits B NetBus Backorifice TFTP IDENT Deep Throat
- 27. Rob Thomas Two Daysof DDoS • Attack that resulted in 10295 hits on day one and 77466 hits on day two. • Attack lasted 25 hours, 25 minutes, and 44 seconds. • Quasi-random UDP high ports (source and destination), small packets.
- 28. Rob Thomas Two Daysof DDoS • Perhaps as many as 2000 hosts used by the attackers. • 23 unique organizations. • 9 different nations located in the Americas, Europe, and Asia. • Source netblocks all legitimate.
- 29. Rob Thomas Two Daysof DDoS Packets per minute 0 10 20 30 40 50 60 70 24:21:13 24:22:03 24:22:53 24:23:46 25:00:36 25:01:26 25:02:16 25:03:06 25:03:56 25:04:46 25:05:36 25:06:26 25:07:16 25:08:06 25:08:56 25:09:46 25:10:36 25:11:26 25:12:16 25:13:06 25:13:56 25:14:46 25:15:36 25:16:26 25:17:16 25:18:06 25:18:57 25:19:48 25:20:39 25:21:37 25:22:29 DATE:HOUR:MINUTE Packets
- 30. Rob Thomas Two Daysof DDoS DDoS Sources 0 500 1000 1500 2000 2500 3000 3500 4000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 Hour Packets
- 31. Rob Thomas Site Defenseand Attack Mitigation • While you can not prevent an attack, you can choose how to react to an attack. • Layers of defense that use multiple tools. • Layers of monitoring and alert mechanisms. • Know how to respond before the attack begins.
- 32. Rob Thomas Site Defenseand Attack Mitigation • Border router – Protocol shaping and filtering. – Anti-bogon and anti-spoofing defense (uRPF), ingress and egress filtering. – NetFlow. • IDS device(s) – Attack and probe signatures. – Alerts.
- 33. Rob Thomas Site Defenseand Attack Mitigation • Border firewall – Port filtering. – Logging. – Some IDS capability. • End systems – Tuned kernel. – TCP wrappers, disable services, etc. – Crunchy through and through!
- 34. Rob Thomas Site Defenseand Attack Mitigation • Don’t panic! • Collect data! • The good news - you can survive!
- 35. Rob Thomas References andshameless self advertisements • RFC 2267 - http://rfc.net/rfc2267.html • Secure IOS Template – http://www.cymru.com/~robt/Docs/Articles/secure-ios- template.html • Secure BGP Template – http://www.cymru.com/~robt/Docs/Articles/secure-bgp- template.html • UNIX IP Stack Tuning Guide – http://www.cymru.com/~robt/Docs/Articles/ip-stack- tuning.html
- 36.
- 37. Rob Thomas Thank youfor your time! • Thanks to Jan, Luuk, and Jacques for inviting me to speak with you today. • Thanks to Surfnet/CERT-NL for picking up the travel. • Thanks for all of the coffee!
Editor's Notes
- #2 This is part of a larger project – to analyze and track attack and probe methods and sources. A holistic view of site probes and attacks. To create an early warning and verification/validation system/site for others to use. To track particularly popular source netblocks and assist the netblock owners with proper defense and mitigation.
- #3 The analysis was performed using copious amounts of Perl, sed, awk, and coffee. The methods are still being honed – they are far from perfect. Started with a series of simple questions raised by a manager : “When did this attack start?” “Did it slowly gain speed? Should we have seen it coming?” “When will it end? Can we predict the end of the attack?”
- #4 Agreed to keep the site name and related data confidential. I am always looking for more log files! Please feel free to share anything you have. Send it to my e-mail address.
- #5 The router and IDS logs are not in sync, unfortunately. This is the first significant effort with copious amounts (942MB) of log data. Previous efforts utilized a significantly smaller sample size.
- #6 ACLs – deny and log RPC, bogon source addresses, etc. IDS – detect SYN floods, Backorifice probes, etc. The data files do not overlap perfectly. This would be ideal for a completely holistic view of site probes and attacks. Remember – NTP is your friend!
- #7 The standard deviation is higher than the mean! In other words, the ebb and flow of probes and attacks is of a greatly variable nature, and therefore difficult to model. Note the 60 day low figure of 83.00. This means that there is never a completely “safe” day!
- #8 Note that never a day goes by without at least 83 probes and/or attacks! The miscreants hit the site 24 x 7 x 365. Why so many? Because the miscreants do NOT share data! Elucidate this point (IRC wars).
- #9 There is no “steady-state,” and there is no “normal” week or day. The vulnerability du jour (e.g. BIND hole, sendmail hole) tends to wane over a two to three month period. When an alert comes out – particularly one that lists a port or ports – fire up an ACL and watch the log entries grow.
- #11 We are now part of a global network – a network that NEVER sleeps. The average DDoS attack lasts between 24 and 72 hours – the zombies don’t need a break. Take a guess – what hour of the day will be the most utilized by the miscreants to hit this site?
- #15 With the exception of TCP 21, all other listed ports were part of a DDoS attack attempt.
- #16 TCP port zero remains very popular with those who target this site. TCP port 21 (FTP) probes have been supplanted by other probes, such as DNS.
- #17 Note that Multicast and experimental source address packets account for 53% of the naughty packets. Thus 53% of the nasty packets are blocked at our border! What about the other classes? Are the sources therein all legitimate?
- #18 Class A bogon source percentage – 48.08%! Class B bogon source percentage – 20.80%. Class C bogon source percentage – 11.87%. Class D and E bogon source percentage – 100.00%.
- #19 Block bogons, both inbound and outbound, at your border! Add anti-spoofing at your egress points so that only source addresses within your allocated netblocks may pass. Three attack types: Spoof the source using bogon addresses. Spoof the source using legitimate addresses. No spoofing. If all networks performed these basic steps, how successful would spoof attacks be?
- #20 The location of the miscreant is difficult to determine because of: Spoof attacks. Zombies. Legacy netblock assignments. Key point – the RIR or netblock does not necessarily indicate the location of the miscreant(s)! Statistics can be misleading. Proper analysis requires a certain degree of “clue.” Latest groups are from Brasil and Romania.
- #21 IDS is really a misnomer – it should be Intrusion Attempt Detection System, or IADS. Intrusion Detection is easy – you will know your hull has been breached when your web page reads “Hackers love Ramen!” IDS can be overwhelmed. Stick is nothing new – we have likely all seen it before. Keep your IDS database current! Place your IDS tool where it will provide the best detection and analysis.
- #22 NetBus – TCP 12345 Backorifice – UDP 31337 TFTP – UDP 70 IDENT – TCP 113 Deep Throat – TCP/UDP 2140
- #23 Notice how the probes converge around day 13 and particularly days 26 and 27. This is not likely to be a coincidence!
- #25 Azerbaijan – ISP USA 01 – Consulting Company South Korea – ISP USA 02 – ISP dialup pool Canada – Cable modem
- #26 Notice how netblock B (USA 01) trends sharply upward around days 24 through 25. In the raw data spreadsheet, the sudden appearance of netblock B coincides with the sharp increase in probe activity as seen in the Top Five Probes slide.
- #28 This was a relatively mild attack, and this makes for simple analysis. However, the analysis steps are the same regardless of attack intensity.
- #29 The use of legitimate source addresses makes defense that much more difficult. This was an extremely mild DDoS – perhaps a test of things to come.
- #30 As with all DDoS attacks, the ramp up time was quite fast – on the order of several seconds. The end came just as suddenly.
- #31 Note the rise and fall of myriad sources. This is not uncommon during even the most intense DDoS attacks, and can be due to several factors, such as: Congestion at the attacker site(s) Reactionary filtering (local and intermediate) during the attacks Zombie control issues Insert the IRC “out of control zombie story” here.
- #32 Each layer should protect the layer beyond. Each layer should integrate with the layers on either side. Each layer should be untrusted by the next site-facing layer. Funnel the traffic, filtering at an ever more granular level of detail.
- #33 Block bogons and prevent spoofing! The presence of an IDS device does not obviate the need to have the other filtering devices logging (ACL logs, rule base logs, NetFlow, etc.). Compare log entries often – remember, NTP is your friend.