T.Pollak y C.Yaconi - Prey

2,650 views
2,521 views

Published on

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,650
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
13
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

T.Pollak y C.Yaconi - Prey

  1. 1. Post mortem.Or how me managed to handle72 DDoSs a day. Using Rails.
  2. 2. Hello there.
  3. 3. 200 OK404 Not Found
  4. 4. Bash Objective-CPython JavaRuby C / C++NodeJS C# (.NET)
  5. 5. !Bash Objective-CPython JavaRuby C / C++NodeJS C# (.NET)
  6. 6. The problem.
  7. 7. 1.5M+tracked devices
  8. 8. 25M+reports received
  9. 9. */20 * interval * execution **
  10. 10. 250Krequests per minute
  11. 11. 90%secondthe very same
  12. 12. +
  13. 13. =
  14. 14. We did whatthe book says.
  15. 15. Master/slave
  16. 16. Load distribution
  17. 17. Memcached
  18. 18. Static cache
  19. 19. Click click!
  20. 20. [error] upstream prematurelyclosed connection while readingresponse header from upstream,client: 24.26.30.50, server:*.preyproject.com
  21. 21. TCP: Possible SYN flooding onport 443. Sending cookies.
  22. 22. nf_conntrack: table full,dropping packet
  23. 23. nf_conntrack: table full,dropping packet(…)net_ratelimit: 8130 callbackssuppressed
  24. 24. !
  25. 25. Time to rollup our sleeves.
  26. 26. Anhour in the life
  27. 27. #%@!
  28. 28. So wheres the bottleneck?
  29. 29. Nginx Stub Status./configure--with-http_stub_status_moduleGET /nginx_statusActive connections: 2076server accepts handled requests 16630948 16630948 31070465Reading: 720 Writing: 379 Waiting: 981
  30. 30. Raindrops for Unicornconfig.ruuse Raindrops::MiddlewareGET /_raindropscalling: 254writing: 00.0.0.0:8080 active: 480.0.0.0:8080 queued: 0
  31. 31. Tuning backlogsUnicorn, config.rblisten shared/sockets/unicorn.sock,:backlog => 4096Nginx Vhost confserver { listen 80 backlog=16384; ...}
  32. 32. Linux TCP/IP Stack TuningConnection count by status$ netstat -an | awk /tcp/ {print $6}| sort | uniq -c 30 CLOSE_WAIT 2234 ESTABLISHED 4 FIN_WAIT1 14 LISTEN 6 SYN_RECV 3222 TIME_WAIT
  33. 33. Linux TCP/IP Stack Tuning$ sysctl -a# max sockets, connectionsnet.core.somaxconn = 131072net.core.netdev_max_backlog = 131072net.ipv4.tcp_max_syn_backlog = 35536# reuse & recycle TCP socketsnet.ipv4.tcp_tw_reuse = 1net.ipv4.tcp_tw_recycle = 1
  34. 34. Linux TCP/IP Stack Tuning# disable syncookiesnet.ipv4.tcp_syncookies = 0# timeouts & retriesnet.ipv4.tcp_orphan_retries = 3net.ipv4.tcp_fin_timeout = 2net.ipv4.tcp_synack_retries = 2net.ipv4.tcp_syn_retries = 2# sysctl -p # reloads settings
  35. 35. nf_conntrack_max - limited by kernel memory - decreased tcp_timeout_time_wait - decreased tcp_fin_timeout - no workie - we disabled connection trackingaltogether, and it worked!iptables -A PREROUTING -p tcp --dport80 -j NOTRACK
  36. 36. The workers!
  37. 37. Decreasememory usage.
  38. 38. config/environment.rbconfig.frameworks -= [:action_view]
  39. 39. Speedthem up.
  40. 40. Benchmark!
  41. 41. Imlib2 >ImageMagick
  42. 42. PaperclipProcessor
  43. 43. Throttle (only)when needed.
  44. 44. Angry Boss!
  45. 45. Angry Boss!- 11 jobs pending. 6 workers running. On loop 75036!- Spawning worker #7...- [Worker #6] Report.process! completed after 2.1819- [Worker #2] Report.process! completed after 2.6006- [Worker #1] Notifier.deliver_report_notificationcompleted after 0.1067- 8 jobs pending. 7 workers running. On loop 75036!- Spawning worker #8...
  46. 46. Imlib2 + Angry Boss!
  47. 47. What welearned.
  48. 48. Rails can scale.
  49. 49. But it needsa bit of help.
  50. 50. All appsare different.
  51. 51. Dont justfollow the book.
  52. 52. Whats next.
  53. 53. Thats it.
  54. 54. Drew, Tom, Yehuda, Bennythe guys that make it possible
  55. 55. <a href>Git reposgithub.com/preyAngry Boss, Resizer, etcgithub.com/tomasHomepreyproject.com
  56. 56. Thanks!Carlos Yaconi@cyaconiTomás Pollak@tomaspollak forkhq.com

×