Velocity 2011 - Our first DDoS attack

2,030 views

Published on

Your website just went down. As you try to understand what has gone wrong, you quickly realize something is different this time. There’s no clear reason why your site should be down, but indeed it is.

This talk is about the story of our team’s first unprepared fight against a DDoS attack.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,030
On SlideShare
0
From Embeds
0
Number of Embeds
59
Actions
Shares
0
Downloads
30
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Velocity 2011 - Our first DDoS attack

  1. 1. Our My first DDoS attack Velocity Europe 2011 – Berlin Cosimo Streppone Operations Lead
  2. 2. <video of Mr. Wolf going to Jimmys house in Pulp Fiction> this couldnt fit in the PDF... sorry.http://www.youtube.com/watch?v=hsKv5d0sIlU
  3. 3. my.opera.com/Ao-Trang-Oi/blog/
  4. 4. nginx – secret sauces?# Pavels secret gzip tuning saucegzip on;gzip_disable msie6;gzip_min_length 1100;gzip_buffers 16 8k;gzip_comp_level 3;gzip_types text/plain application/xml application/x-javascript text/css;
  5. 5. nginx – secret sauces?# Michaels secret file cache sauceopen_file_cache max=1000 inactive=20s;open_file_cache_valid 30s;open_file_cache_min_uses 2;open_file_cache_errors on;
  6. 6. nginx – antidos.conf# More on https://calomel.org/nginx.htmlclient_header_timeout 5;client_body_timeout 10;ignore_invalid_headers on;send_timeout 10;# To limit slowloris-like attacksclient_header_buffer_size 4k;large_client_header_buffers 4 4k;
  7. 7. nginx – drop client connections# Cut abusive established connections,# forcing clients to reconnectlocation ~ ^/Ao-Trang-Oi/blog/ { return 444;}
  8. 8. nginx – varnish caching nginx varnish backends
  9. 9. iptraf
  10. 10. tcpdump of anomalous trafficGET /Ao-Trang-Oi/blog/show.dml/14715682 HTTP/1.1 User-Agent: 1.{RND 10}.{RND 10} Referrer: http://my.opera.com/Ao-Trang-Oi/ Cache-Control: no-cache Cookie: __utma=218314117.745395330 […] __utmz=218314117.1286774593. […] utmcsr=google|utmccn= […] utmctr=cach%20de%20hoc%20mon […] <... random high speed junk follows ...>
  11. 11. tcpdump of anomalous trafficGET /Ao-Trang-Oi/blog/?startidx=1295 HTTP/1.1 User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;) Gecko/20030624 Netscape/7.1 (ax) Accept: Accept=text/html,application/xhtml+xml,... Accept-Language: Accept-Language=en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: Accept-Charset=ISO-8859-1,... Referer: http://my.opera.com/Ao-Trang-Oi/blog/ Pragma: no-cache Keep-Alive: 300 ua-cpu: x86 Connection: close
  12. 12. #nginx, 14th October 2010 cosimo: were seeing a pretty "interesting" problem within our nginx fronts BLAH BLAH BLAH cosimo: theres a few hosts sending a legitimate HTTP GET request BLAH BLAH BL cosimo: followed by a binary stream of random bytes that never ends BLAH BLAH BLAH cosimo: this is just 1 request going on and on cosimo: is there some way to alter the nginx config to shut down these client connections? OMGWTFBBQ!!!!11111 cosimo: the client is sending something like: cosimo: GET /blah HTTP/1.1 “this is nkiller2” cosimo: Host: ... cosimo: Etc: etc... cosimo: and then random bullshit vr: :) vr: this is nkiller2 vr: haproxy can fight this vr: you can set a timeout http-request vr: dont know if nginx can do this cosimo: cool
  13. 13. PHRACK#66
  14. 14. tcp window zero?
  15. 15. iptables -A -m u32 --u32 “6&0xFF=0x6 && 4&0x1FFF=0 && 0>>22&0x3C () 12&0xFFFF=0x0000” -j ZERO_WINDOW_RECENT
  16. 16. u32 zero window filter6 &0xFF =0x6
  17. 17. u32 zero window filter4 &0x1FFF =0x0
  18. 18. u32 zero window filter0>>22 &0x3C ()12 &0xFFFF=0x0
  19. 19. u32 zero window filter0>>22 &0x3C ()12 &0xFFFF =0x0 ??
  20. 20. 0>>22&0...@12&0xFFFF=0x0000
  21. 21. 0>>22&0x3C@12&0xFFFF=0x0000
  22. 22. 0>>22& [EMAIL PROTECTED] &0xFFFF=0x0000
  23. 23. 0>>22&0x3C@12&0xFFFF=0x0000
  24. 24. u32 zero window filter0>>22 &0x3C @12 &0xFFFF=0x0
  25. 25. iptables rules - logging$ipt -N ZERO_WINDOW_RECENT$ipt -A INPUT -m u32 --u32 "6&0xFF=0x6 && 4&0x1FFF=0 && 0>>22&0x3C@12&0xFFFF=0x0000" -j ZERO_WINDOW_RECENT$ipt -A ZERO_WINDOW_RECENT -m recent --set --nameZERO_WINDOW$ipt -A ZERO_WINDOW_RECENT -m recent --update --seconds 60 --hitcount 20 --name ZERO_WINDOW -j LOG --log-level info --log-prefix "ZeroWindow"
  26. 26. ~18k distinct IPs
  27. 27. iptables rules - blocking$ipt -N ZERO_WINDOW_RECENT$ipt -A INPUT -m u32 --u32 "6&0xFF=0x6 && 4&0x1FFF=0 && 0>>22&0x3C@12&0xFFFF=0x0000" -j ZERO_WINDOW_RECENT$ipt -A ZERO_WINDOW_RECENT -m recent –set --name ZERO_WINDOW$ipt -A ZERO_WINDOW_RECENT -m recent –update --seconds 60 --hitcount 20 --name ZERO_WINDOW -j DROP
  28. 28. shields-up.vcl cacheable content nginx varnish non-cacheable content backends
  29. 29. shields-up.vcl all HTTP content varnish nginx HTTPS-only traffic backends
  30. 30. nginx feels better
  31. 31. Pingdom response time 20s 10s 0s
  32. 32. End 29-Oct-2010
  33. 33. Packets/s seen by firewallStart 13-Oct-2010 End 29-Oct-2010
  34. 34. ¿Questions?
  35. 35. What can we, as Ops, do better? ● Embrace failures and learn from them ● Be fast (no panic/blame, think Mr. Wolf) ● Coordinate (#ops, war rooms, ...) ● Take notes ● Learn TCP/IP ● Know your tools (tcpdump, tcpflow, strace, nc, iptraf, …)
  36. 36. my base_packages puppet module class base_packages { $packagelist = [ "ack-grep", "colordiff", "curl", "facter", "git-core", "htop", "iftop", "iptraf", "jed", "joe", "libwww-perl", "logrotate", "lsof", "make", "mc", "oprofile", "psmisc", "rsync", "screen", "svn", "sysstat", "tcpdump", "tcpflow", "telnet", "unzip", "vim", "zip" ] package { $packagelist: ensure => "installed", } }
  37. 37. Thanks to... ● ithilgore (sock-raw.org) for writing nkiller2 ● @vr in #nginx for pointing us at nkiller2 ● David Falloon for his great “untested” idea ● marc.info for correctly handling “@” in ml ● SANS Institute for the TCP/IP references ● My team at Opera
  38. 38. Danke!

×