Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Border Patrol - Count, throttle, kick & ban in perl

1,213 views

Published on

Presentation of the Schedule::AdaptiveThrottler perl module, under its original name. Made for the Portuguese Perl Workshop in Porto in 2010. The module is available on https://metacpan.org/pod/Schedule::AdaptiveThrottler and https://github.com/dmorel/Schedule-AdaptiveThrottler

Published in: Internet
  • Be the first to comment

  • Be the first to like this

Border Patrol - Count, throttle, kick & ban in perl

  1. 1. throttle count kick ban limit protect Border Patrol david.morel@booking.com
  2. 2. Common problems • Prevent DoS • Prevent brute-force attacks • Throttle • Kick • Ban • Limit *anything*
  3. 3. Rely on memcached • Because it is easy • Lightweight, very fast • Use LRU and TTL for auto cleanup
  4. 4. Dead-simple principle • Store TTL epochs only (time the event happened => counter was incremented) • Number of timestamps *is* the counter • memcached key name indicates what is counted
  5. 5. No, really, it *is* simple! • Just do a grep to eliminate expired timestamps • Is the remaining number of timestamps above threshold? => return 0 / 1 • Use built-in LRU for ‘block’ records
  6. 6. Throttle to 100 requests per minute for a single IP + User-Agent combination Use case #1
  7. 7. my ( $status, $messages ) = BorderPatrol->authorize( memcached_client => Cache::Memcached::Fast->new(...), all => { ip_ua => { max => 100, ttl => 60, message => 'This IP+UA combination ran more ' .'than 100 queries in the last minute', value => "${client_ip}_${user_agent}", }, }, identifier => 'robot_connect', ); CODE
  8. 8. robot_connect#ip_ua#92.243.24.26_Googlebot => [ 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731713, 1275731714, 1275731714, ... ] RECORD
  9. 9. Prevent access to an authentication form for 5 minutes if: • more than 5 login attempts for a single username • OR more than 50 login attempts from a single IP Use case #2
  10. 10. use BorderPatrol; my ( $status, $messages ) = BorderPatrol->authorize( memcached_client => Cache::Memcached::Fast->new(...), either => { ip => { max => 50, ttl => 300, message => 'More than 50 login tries from this IP ' .'in the last 5 minutes', value => $client_ip, }, login => { max => 5, ttl => 60, message => 'More than 5 login tries for this ' .'username in the last minute', value => $username, }, }, lockout => 600, identifier => 'user_login_tries', );
  11. 11. user_login_tries#ip#92.243.24.26 => [ 1275731713, 1275731713, 1275731713, 1275731714, 1275731714, ] user_login_tries#username#dmorel => [ 1275731713, 1275731713, 1275731714, ] 2 RECORDS
  12. 12. IMPLEMENTATION DETAILS
  13. 13. $frozen_time = time; # $record is an arrayref of timestamps # - contains one timestamp per previous access # - timestamps can be duplicated many times # if many accesses in the same second... # it doesn't matter! if ( ref $record eq 'ARRAY' ) { # purge the expired timestamps # the magic happens *HERE* @$record = grep { $_ > $frozen_time } @$record;
  14. 14. # Since we are about to add a record: # - if we already have the max number of allowed records, # set to blocked # - if no lockout time specified, use a bucket, # so return false, but do not touch the record if ( @$record >= $condition->{max} ) { if ($lockout) { $memcached_client->set( $memcached_key, 'block', $lockout ); } push @$messages_notok, $condition->{message}; } # if under threshold, push the current timestamp + TTL to # the record, and set the record expiration in memcached # at TTL seconds from now else { push @$record, $frozen_time + $condition->{ttl}; $memcached_client->set( $memcached_key, $record, $condition->{ttl} ); $conditions_ok++; }
  15. 15. # Do we have a 'block' value in the record? # if so, simple return the message. # The 'block' record will be automatically # removed from storage at the object's # eviction time, so just let memcached do # its job. elsif ( $record eq 'block' ) { push @$messages_notok, $condition->{message}; }
  16. 16. # Check the conditions stack and determine # the final answer # If logic was 'either', one 'notok' (or more) # should block. If logic was 'all', we should # have no 'ok' at all, or else we allow. if ( $condition_type eq 'either' ) { return ( @$messages_notok > 0 ) ? ( AC_BLOCKED, $messages_notok ) : ( AC_AUTHORIZED, undef ); } else { # condition is 'all' return ( $conditions_ok == 0 ) ? ( AC_BLOCKED, $messages_notok ) : ( AC_AUTHORIZED, undef ); }
  17. 17. MANY OTHER USE CASES The same sliding time window approach can be used in many ways. It really is just a generic event hit rate counter
  18. 18. ON YOUR FAVORITE MIRROR SOON
  19. 19. (hopefully) © Booking.com 2010 Thanks for listening! Special thanks to Ruud, Kris, Philippe, Dennis and Liz

×