• Like
  • Save
T.Pollak y C.Yaconi - Prey
Upcoming SlideShare
Loading in...5
×
 

T.Pollak y C.Yaconi - Prey

on

  • 2,486 views

 

Statistics

Views

Total Views
2,486
Views on SlideShare
2,486
Embed Views
0

Actions

Likes
0
Downloads
11
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    T.Pollak y C.Yaconi - Prey T.Pollak y C.Yaconi - Prey Presentation Transcript

    • Post mortem.Or how me managed to handle72 DDoSs a day. Using Rails.
    • Hello there.
    • 200 OK404 Not Found
    • Bash Objective-CPython JavaRuby C / C++NodeJS C# (.NET)
    • !Bash Objective-CPython JavaRuby C / C++NodeJS C# (.NET)
    • The problem.
    • 1.5M+tracked devices
    • 25M+reports received
    • */20 * interval * execution **
    • 250Krequests per minute
    • 90%secondthe very same
    • +
    • =
    • We did whatthe book says.
    • Master/slave
    • Load distribution
    • Memcached
    • Static cache
    • Click click!
    • [error] upstream prematurelyclosed connection while readingresponse header from upstream,client: 24.26.30.50, server:*.preyproject.com
    • TCP: Possible SYN flooding onport 443. Sending cookies.
    • nf_conntrack: table full,dropping packet
    • nf_conntrack: table full,dropping packet(…)net_ratelimit: 8130 callbackssuppressed
    • !
    • Time to rollup our sleeves.
    • Anhour in the life
    • #%@!
    • So wheres the bottleneck?
    • Nginx Stub Status./configure--with-http_stub_status_moduleGET /nginx_statusActive connections: 2076server accepts handled requests 16630948 16630948 31070465Reading: 720 Writing: 379 Waiting: 981
    • Raindrops for Unicornconfig.ruuse Raindrops::MiddlewareGET /_raindropscalling: 254writing: 00.0.0.0:8080 active: 480.0.0.0:8080 queued: 0
    • Tuning backlogsUnicorn, config.rblisten shared/sockets/unicorn.sock,:backlog => 4096Nginx Vhost confserver { listen 80 backlog=16384; ...}
    • Linux TCP/IP Stack TuningConnection count by status$ netstat -an | awk /tcp/ {print $6}| sort | uniq -c 30 CLOSE_WAIT 2234 ESTABLISHED 4 FIN_WAIT1 14 LISTEN 6 SYN_RECV 3222 TIME_WAIT
    • Linux TCP/IP Stack Tuning$ sysctl -a# max sockets, connectionsnet.core.somaxconn = 131072net.core.netdev_max_backlog = 131072net.ipv4.tcp_max_syn_backlog = 35536# reuse & recycle TCP socketsnet.ipv4.tcp_tw_reuse = 1net.ipv4.tcp_tw_recycle = 1
    • Linux TCP/IP Stack Tuning# disable syncookiesnet.ipv4.tcp_syncookies = 0# timeouts & retriesnet.ipv4.tcp_orphan_retries = 3net.ipv4.tcp_fin_timeout = 2net.ipv4.tcp_synack_retries = 2net.ipv4.tcp_syn_retries = 2# sysctl -p # reloads settings
    • nf_conntrack_max - limited by kernel memory - decreased tcp_timeout_time_wait - decreased tcp_fin_timeout - no workie - we disabled connection trackingaltogether, and it worked!iptables -A PREROUTING -p tcp --dport80 -j NOTRACK
    • The workers!
    • Decreasememory usage.
    • config/environment.rbconfig.frameworks -= [:action_view]
    • Speedthem up.
    • Benchmark!
    • Imlib2 >ImageMagick
    • PaperclipProcessor
    • Throttle (only)when needed.
    • Angry Boss!
    • Angry Boss!- 11 jobs pending. 6 workers running. On loop 75036!- Spawning worker #7...- [Worker #6] Report.process! completed after 2.1819- [Worker #2] Report.process! completed after 2.6006- [Worker #1] Notifier.deliver_report_notificationcompleted after 0.1067- 8 jobs pending. 7 workers running. On loop 75036!- Spawning worker #8...
    • Imlib2 + Angry Boss!
    • What welearned.
    • Rails can scale.
    • But it needsa bit of help.
    • All appsare different.
    • Dont justfollow the book.
    • Whats next.
    • Thats it.
    • Drew, Tom, Yehuda, Bennythe guys that make it possible
    • <a href>Git reposgithub.com/preyAngry Boss, Resizer, etcgithub.com/tomasHomepreyproject.com
    • Thanks!Carlos Yaconi@cyaconiTomás Pollak@tomaspollak forkhq.com