Velocity 2011 - Our first DDoS attack
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Velocity 2011 - Our first DDoS attack

  • 1,965 views
Uploaded on

Your website just went down. As you try to understand what has gone wrong, you quickly realize something is different this time. There’s no clear reason why your site should be down, but indeed it......

Your website just went down. As you try to understand what has gone wrong, you quickly realize something is different this time. There’s no clear reason why your site should be down, but indeed it is.

This talk is about the story of our team’s first unprepared fight against a DDoS attack.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,965
On Slideshare
1,910
From Embeds
55
Number of Embeds
3

Actions

Shares
Downloads
26
Comments
0
Likes
1

Embeds 55

http://velocityconf.com 49
https://www.linkedin.com 5
http://www.linkedin.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Our My first DDoS attack Velocity Europe 2011 – Berlin Cosimo Streppone Operations Lead
  • 2. <video of Mr. Wolf going to Jimmys house in Pulp Fiction> this couldnt fit in the PDF... sorry.http://www.youtube.com/watch?v=hsKv5d0sIlU
  • 3. my.opera.com/Ao-Trang-Oi/blog/
  • 4. nginx – secret sauces?# Pavels secret gzip tuning saucegzip on;gzip_disable msie6;gzip_min_length 1100;gzip_buffers 16 8k;gzip_comp_level 3;gzip_types text/plain application/xml application/x-javascript text/css;
  • 5. nginx – secret sauces?# Michaels secret file cache sauceopen_file_cache max=1000 inactive=20s;open_file_cache_valid 30s;open_file_cache_min_uses 2;open_file_cache_errors on;
  • 6. nginx – antidos.conf# More on https://calomel.org/nginx.htmlclient_header_timeout 5;client_body_timeout 10;ignore_invalid_headers on;send_timeout 10;# To limit slowloris-like attacksclient_header_buffer_size 4k;large_client_header_buffers 4 4k;
  • 7. nginx – drop client connections# Cut abusive established connections,# forcing clients to reconnectlocation ~ ^/Ao-Trang-Oi/blog/ { return 444;}
  • 8. nginx – varnish caching nginx varnish backends
  • 9. iptraf
  • 10. tcpdump of anomalous trafficGET /Ao-Trang-Oi/blog/show.dml/14715682 HTTP/1.1 User-Agent: 1.{RND 10}.{RND 10} Referrer: http://my.opera.com/Ao-Trang-Oi/ Cache-Control: no-cache Cookie: __utma=218314117.745395330 […] __utmz=218314117.1286774593. […] utmcsr=google|utmccn= […] utmctr=cach%20de%20hoc%20mon […] <... random high speed junk follows ...>
  • 11. tcpdump of anomalous trafficGET /Ao-Trang-Oi/blog/?startidx=1295 HTTP/1.1 User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;) Gecko/20030624 Netscape/7.1 (ax) Accept: Accept=text/html,application/xhtml+xml,... Accept-Language: Accept-Language=en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: Accept-Charset=ISO-8859-1,... Referer: http://my.opera.com/Ao-Trang-Oi/blog/ Pragma: no-cache Keep-Alive: 300 ua-cpu: x86 Connection: close
  • 12. #nginx, 14th October 2010 cosimo: were seeing a pretty "interesting" problem within our nginx fronts BLAH BLAH BLAH cosimo: theres a few hosts sending a legitimate HTTP GET request BLAH BLAH BL cosimo: followed by a binary stream of random bytes that never ends BLAH BLAH BLAH cosimo: this is just 1 request going on and on cosimo: is there some way to alter the nginx config to shut down these client connections? OMGWTFBBQ!!!!11111 cosimo: the client is sending something like: cosimo: GET /blah HTTP/1.1 “this is nkiller2” cosimo: Host: ... cosimo: Etc: etc... cosimo: and then random bullshit vr: :) vr: this is nkiller2 vr: haproxy can fight this vr: you can set a timeout http-request vr: dont know if nginx can do this cosimo: cool
  • 13. PHRACK#66
  • 14. tcp window zero?
  • 15. iptables -A -m u32 --u32 “6&0xFF=0x6 && 4&0x1FFF=0 && 0>>22&0x3C () 12&0xFFFF=0x0000” -j ZERO_WINDOW_RECENT
  • 16. u32 zero window filter6 &0xFF =0x6
  • 17. u32 zero window filter4 &0x1FFF =0x0
  • 18. u32 zero window filter0>>22 &0x3C ()12 &0xFFFF=0x0
  • 19. u32 zero window filter0>>22 &0x3C ()12 &0xFFFF =0x0 ??
  • 20. 0>>22&0...@12&0xFFFF=0x0000
  • 21. 0>>22&0x3C@12&0xFFFF=0x0000
  • 22. 0>>22& [EMAIL PROTECTED] &0xFFFF=0x0000
  • 23. 0>>22&0x3C@12&0xFFFF=0x0000
  • 24. u32 zero window filter0>>22 &0x3C @12 &0xFFFF=0x0
  • 25. iptables rules - logging$ipt -N ZERO_WINDOW_RECENT$ipt -A INPUT -m u32 --u32 "6&0xFF=0x6 && 4&0x1FFF=0 && 0>>22&0x3C@12&0xFFFF=0x0000" -j ZERO_WINDOW_RECENT$ipt -A ZERO_WINDOW_RECENT -m recent --set --nameZERO_WINDOW$ipt -A ZERO_WINDOW_RECENT -m recent --update --seconds 60 --hitcount 20 --name ZERO_WINDOW -j LOG --log-level info --log-prefix "ZeroWindow"
  • 26. ~18k distinct IPs
  • 27. iptables rules - blocking$ipt -N ZERO_WINDOW_RECENT$ipt -A INPUT -m u32 --u32 "6&0xFF=0x6 && 4&0x1FFF=0 && 0>>22&0x3C@12&0xFFFF=0x0000" -j ZERO_WINDOW_RECENT$ipt -A ZERO_WINDOW_RECENT -m recent –set --name ZERO_WINDOW$ipt -A ZERO_WINDOW_RECENT -m recent –update --seconds 60 --hitcount 20 --name ZERO_WINDOW -j DROP
  • 28. shields-up.vcl cacheable content nginx varnish non-cacheable content backends
  • 29. shields-up.vcl all HTTP content varnish nginx HTTPS-only traffic backends
  • 30. nginx feels better
  • 31. Pingdom response time 20s 10s 0s
  • 32. End 29-Oct-2010
  • 33. Packets/s seen by firewallStart 13-Oct-2010 End 29-Oct-2010
  • 34. ¿Questions?
  • 35. What can we, as Ops, do better? ● Embrace failures and learn from them ● Be fast (no panic/blame, think Mr. Wolf) ● Coordinate (#ops, war rooms, ...) ● Take notes ● Learn TCP/IP ● Know your tools (tcpdump, tcpflow, strace, nc, iptraf, …)
  • 36. my base_packages puppet module class base_packages { $packagelist = [ "ack-grep", "colordiff", "curl", "facter", "git-core", "htop", "iftop", "iptraf", "jed", "joe", "libwww-perl", "logrotate", "lsof", "make", "mc", "oprofile", "psmisc", "rsync", "screen", "svn", "sysstat", "tcpdump", "tcpflow", "telnet", "unzip", "vim", "zip" ] package { $packagelist: ensure => "installed", } }
  • 37. Thanks to... ● ithilgore (sock-raw.org) for writing nkiller2 ● @vr in #nginx for pointing us at nkiller2 ● David Falloon for his great “untested” idea ● marc.info for correctly handling “@” in ml ● SANS Institute for the TCP/IP references ● My team at Opera
  • 38. Danke!