DDoS mitigation in the
real world
19th May 2014
Vienna DevOps/Security Meetup
Michael Renner
@terrorobe
My name is Michael Renner
In IT since 2001, focus on system administrations and web operations.
...
Company I'm freelancing for.
Viennese company, founded around 2010 - our offices are on Mariahilfer
Strasse
Operations, De...
DDoS?
In computing, a denial-of-service (DoS) or
distributed denial-of-service (DDoS) attack is an
attempt to make a machi...
Why people DDoS
...which leads us to the topic of our today's talk.
DDoS is something that has gotten more and more news c...
...to extort money
This one is easy.
Was very prevalent in the gambling and sports betting area. Back at Inode
in 2003 peo...
...to underline a
political agenda
Operation Payback in 2010 - done by Anonymous - targeted against
recording industry pro...
...to escalate issues in
customer care
Has happened in the past - if you annoy the wrong customer he might
escalate things...
...because somebody
pissed them off on IRC
User chats from your server/network, annoys the wrong people and they
show them...
...to prove a point
...or just to prove a point, whatever that might be.
The more high profile/recognized your site is the ...
DDoS attacks have
asymmetrical costs
The problem with DDoS attacks is that they have highly asymmetrical
costs.
The attack...
Attacks are cheap
"Prices per hour may depend on your target. Price starts 10$ per hour to
100$ per hour depending on how ...
Attacks happen when
people have free time
• Weekends, holidays, bridging days
• In their respective timezone!
• More often...
Attacks are hard to
identify
• "The DB server has too many connections
open"
• "I'm seeing packet loss and a large amount
...
Mitigating is hard without
proper infrastructure
• "Let's see if I can whip up an u32 classifier
to filter out this pattern ...
Hard to find the
sources of attacks
• Most attacks use spoofed source addresses
• If it's a reflection or amplification attac...
In the end...
• ...it's you and your customers who're
unhappy
so it's best to try not to get knocked off the net in the fir...
DDoS war stories
To give the users who haven't experienced a thorough attack yet a
glimpse of what it may look like I've c...
So you run a website...
Most of you operate a web/online service I guess, which generates you
fame, fortune or both.
And a...
Somebody does a slow-
read attack
...open up hundreds of connections, read a few bytes at a time...
basically exhausting o...
But they still know
your server's IP address
Then the attacker goes on netcraft.com, checks the former IP address of
your ...
Suddenly you get a
ERR_NAME_NOT_RESOLVED
when visiting your site
After a few hours the monitoring starts to page you. You ...
You notice that you're
sending mails
[..]
Received: from web01.yourcoolapp.com ([198.51.100.42])
Subject: WELCOME TO YOURC...
...so you think you're
good for now.
And then your ISP goes
offline
By then you think you've seen it all and you think you'...
Mitigating attacks...
in Theory!
...so, this should give you a nice overview what can happen in the real
world.
So what ca...
Increase your attack
surface
• Geo-distribution is the way to go
• either via DNS or IP anycast
• -> more traffic sinks!
Fi...
150 Gbit/sec sounds
scary
• Less scary when it's:
• 40 Gbit Hongkong
• 20 Gbit Frankfurt
• 35 Gbit London
• 25 Gbit US Wes...
Scrubbing the traffic
• In general: you want to filter out all
"DDoSy" or anomalous requests from your
traffic
• Statistical ...
Don't build your own
scrubbing setup unless
you've got good reasons
In most cases it's much cheaper to just lease scrubbin...
Mitigating attacks...
...in Practice!
...so much for basics - what can we actually do to improve our security?
Don't do DDoS mitigation
as an afterthought
First and foremost...
it's much nicer to talk and discuss about these things d...
Decide on a scope
• Threat modeling
• Build playbooks
• Prepare & implement low-hanging
fruits right now
You need to do th...
Have good OpSec
One example for a low-hanging fruit: Operational Security
Make sure you don't leak IP addresses you don't ...
Choose an approach
And as with every idea - there's more than one way to do it!
Proxies
• The easy/cheap solution
• Protocol proxies (HTTP, SMTP, etc.)
• Geo-distributed
The cheap approach would be a pr...
Proxy Caveats
• Few control over proxy servers
• Can hurt troubleshooting times
• Applications don't see end-user IPs
anym...
Routed solution
• You need a /24 network prefix
• IP tunnel to scrubbing center
• All scrubbing locations announce the prefi...
Routing caveats
• Takes time to implement & test properly
• More expensive to setup and maintain
• Routing & BGP can be sc...
Appliances
There's still appliance vendors in the DDoS market, e.g. Arbor or
Radware.
Was a bit surprised by that, but it ...
Ask your ISP
something you shouldn't neglect is that your ISP may already have DDoS
infrastructure set up - it doesn't hur...
Common caveats
• Your applications latency is most likely
going to increase
• There can be false positives & overfiltering
...
Interested?
Talk to us!
info@ixolit.com
m.renner@ixolit.com
I hope that gave you a bit of an overview of how the current s...
Questions?
Do you have any questions?
Thanks!
Upcoming SlideShare
Loading in …5
×

DDoS mitigation in the real world

549
-1

Published on

With the proliferation of cheap bandwidth and vulnerable systems the DDoS attack volume has increased tremendously over the last years. The talk will cover current threat models and possible countermeasures to mitigate the attacks when they should happen

Published in: Technology, News & Politics
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
549
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
31
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

DDoS mitigation in the real world

  1. 1. DDoS mitigation in the real world 19th May 2014 Vienna DevOps/Security Meetup
  2. 2. Michael Renner @terrorobe My name is Michael Renner In IT since 2001, focus on system administrations and web operations. Freelancing since 2010 for multiple companies doing a hodge-podge of tasks - from building data centers, DBA work, troubleshooting, performance optimization... If you've got questions, just ask!
  3. 3. Company I'm freelancing for. Viennese company, founded around 2010 - our offices are on Mariahilfer Strasse Operations, Development, Customer Care, payment solutions, SMS delivery, you name it ;) And since a year ago also DDoS protection
  4. 4. DDoS? In computing, a denial-of-service (DoS) or distributed denial-of-service (DDoS) attack is an attempt to make a machine or network resource unavailable to its intended users. Source: http://en.wikipedia.org/wiki/Denial-of-service_attack
  5. 5. Why people DDoS ...which leads us to the topic of our today's talk. DDoS is something that has gotten more and more news coverage in the last few years. Chances are high that you already tried using a service while it was being DDoSed or that you run a service on your own which was the target of an attack So why do people DDoS?
  6. 6. ...to extort money This one is easy. Was very prevalent in the gambling and sports betting area. Back at Inode in 2003 people where already DDoSing gambling companies! "Comes with the business"
  7. 7. ...to underline a political agenda Operation Payback in 2010 - done by Anonymous - targeted against recording industry prosecuting piracy as well as finance companies denying service to wikileaks
  8. 8. ...to escalate issues in customer care Has happened in the past - if you annoy the wrong customer he might escalate things by his own means
  9. 9. ...because somebody pissed them off on IRC User chats from your server/network, annoys the wrong people and they show them that they mean business
  10. 10. ...to prove a point ...or just to prove a point, whatever that might be. The more high profile/recognized your site is the higher the chance that somebody will attack it just for the fun of it.
  11. 11. DDoS attacks have asymmetrical costs The problem with DDoS attacks is that they have highly asymmetrical costs. The attack itself can be quite cheap whereas the lost revenue as well as time and money to bandaid-patch an attack can be quite expensive
  12. 12. Attacks are cheap "Prices per hour may depend on your target. Price starts 10$ per hour to 100$ per hour depending on how Huge or Protected the target is." When we're talking money one must know that attacks are quite cheap. This is an example from end-customer site, completely with youtube video and everything! Depending on how smart the attackers are and how good you're connected attacks can be significantly cheaper.
  13. 13. Attacks happen when people have free time • Weekends, holidays, bridging days • In their respective timezone! • More often than not: • the time when you'd like to have free time as well Another problem is that the attacks will happen when people have free time.
  14. 14. Attacks are hard to identify • "The DB server has too many connections open" • "I'm seeing packet loss and a large amount of softinterrupts on CPU0" • "The server is down" DDoS attacks are very disorienting Depending on how much instrumentation you have for your infrastructure you might not even be able to tell that you're getting DDoSed "Newrelic might not be enough to spot it" Depending on attack you might need OS, application level even network packet statistics.
  15. 15. Mitigating is hard without proper infrastructure • "Let's see if I can whip up an u32 classifier to filter out this pattern I'm seeing" And when the attack is already running you'll often have a hard time jury- rigging yourself out of the attack.
  16. 16. Hard to find the sources of attacks • Most attacks use spoofed source addresses • If it's a reflection or amplification attack you won't even get those. • And even if you'v got real sources, it's a hacked server in nearly all of the cases More often than not management will be interested on who's doing the attack and why. This is nigh impossible if you're not the NSA, having a big picture of all the traffic going on. And even if you've got "real" source addresses nearly all of them will be hacked servers connected to an anonymous C&C servers
  17. 17. In the end... • ...it's you and your customers who're unhappy so it's best to try not to get knocked off the net in the first place!
  18. 18. DDoS war stories To give the users who haven't experienced a thorough attack yet a glimpse of what it may look like I've collected a few things we faced in the past.
  19. 19. So you run a website... Most of you operate a web/online service I guess, which generates you fame, fortune or both. And along the way you picked up a few people who don't have the best interests regarding your site, and so they decide to...
  20. 20. Somebody does a slow- read attack ...open up hundreds of connections, read a few bytes at a time... basically exhausting open workers, tcp connections, file descriptors, memory, whatever comes first. so you get a proxy setup like CloudFlare up and running, all HTTP requests proxied through They detect the botnet IPs either through pattern matching or existing botnet IP lists and start filtering those out or presenting them with captchas. Attack solved.
  21. 21. But they still know your server's IP address Then the attacker goes on netcraft.com, checks the former IP address of your site ...and decides to send a few hundred megabits worth of UDP packets your way. You ask your ISP for a new subnet, renumber your servers, add the new IP addresses to your proxy setup and hope for the best Things are quiet for now.
  22. 22. Suddenly you get a ERR_NAME_NOT_RESOLVED when visiting your site After a few hours the monitoring starts to page you. You check your site and you get a NAME NOT RESOLVED. ...turns out the cheap registrar where you bought your domain ...who still does your DNS ...isn't quite DDoS resilient. So you shop around, see companies like dyn.com or Amazon route 53, move domains over, wait for cached entries to expire and after a few hours you should be back online!
  23. 23. You notice that you're sending mails [..] Received: from web01.yourcoolapp.com ([198.51.100.42]) Subject: WELCOME TO YOURCOOLAPP! From: noreply@yourcoolapp.com To: evilattacker@example.com But then you notice that you're sending emails! And your attacker noticed that the email address contain interesting information in their headers. ...and the DDoS attack on your server resumes. ...so you configure your mail gateway to not include servers IP addresses ...get new IP addresses ...hope for the best
  24. 24. ...so you think you're good for now. And then your ISP goes offline By then you think you've seen it all and you think you're good to go. ...most smaller ISPs are not equipped to sustain attacks even 10 to 50 gigabits in size. If regular traffic is in the single-gigabit per second range it's easy to knock them offline. Usually they then just filter the destination IPs at their carriers and be done with it.
  25. 25. Mitigating attacks... in Theory! ...so, this should give you a nice overview what can happen in the real world. So what can we do about these attacks? In the end all of the DDoS Mitigation services out there are doing the same thing.
  26. 26. Increase your attack surface • Geo-distribution is the way to go • either via DNS or IP anycast • -> more traffic sinks! First part is increasing the attack surface Most customers have just a single datacenter, amazon availability zone and so on. Very easy to concentrate your attack on. If you operate multiple Datacenters it's much harder to knock all of them offline - loadbalancing done either via DNS & geoip or IP anycast setups - same prefix announced on multiple locations This all boils down to have more traffic sinks to filter the traffic out on
  27. 27. 150 Gbit/sec sounds scary • Less scary when it's: • 40 Gbit Hongkong • 20 Gbit Frankfurt • 35 Gbit London • 25 Gbit US West • 30 Gbit US East I'm just pulling a few numbers out of my hat to give you an example. Suppose you've got a 150gbit attack running - which to this day is still a very large number when it comes to DDoS attacks - this gets far more manageable when you operate multiple sites where the traffic has to go to And even if they manage to knock one of the sites off the net all the others are still online and answering requests.
  28. 28. Scrubbing the traffic • In general: you want to filter out all "DDoSy" or anomalous requests from your traffic • Statistical models, signature-based filters, hand-written rules, etc. • Specialized hardware with in-ASIC rule processing • Beyond the scope of this talk The other big part is to scrub the traffic and remove all packets which belong to "probably unwanted" traffic. The further down the traffic goes through your stack the more expensive is the processing of a malicious request. Compare the time needed to throw away a request on the firewall vs. throwing it away in the web application. Big Companies use specialized hardware with network chips, being capable of doing complex rule processing at wirespeed of the interface. They often use statistical models to detect network anomalies, have rules for known attack patterns (e.g. NTP reflection attacks) and can also deploy custom rules specific to attacks currently going on.
  29. 29. Don't build your own scrubbing setup unless you've got good reasons In most cases it's much cheaper to just lease scrubbing services from one of the established players instead of building your own infrastructure.
  30. 30. Mitigating attacks... ...in Practice! ...so much for basics - what can we actually do to improve our security?
  31. 31. Don't do DDoS mitigation as an afterthought First and foremost... it's much nicer to talk and discuss about these things during office hours with sober colleagues than on a saturday night at 4 am after 8 beers
  32. 32. Decide on a scope • Threat modeling • Build playbooks • Prepare & implement low-hanging fruits right now You need to do threat modeling - which attacks do you expect, how you're going to manage them. Prepare contacts up front even if you don't want to buy something right now. Have a playbook describing how you're going to react Implement low hanging fruits right now!
  33. 33. Have good OpSec One example for a low-hanging fruit: Operational Security Make sure you don't leak IP addresses you don't want to It makes sense to have two different network segments - one for public- facing IPs, another one for the actual servers. Actual server IPs don't even have to be public ones if you can avoid it. Things to check: Mail Headers Test systems like beta.yourcoolapp.com MX records!
  34. 34. Choose an approach And as with every idea - there's more than one way to do it!
  35. 35. Proxies • The easy/cheap solution • Protocol proxies (HTTP, SMTP, etc.) • Geo-distributed The cheap approach would be a proxy setup. You get a list of protocol- level proxies where you point your DNS records to - you configure the destination for the proxies and all the traffic hits these servers first. Well-known company is CloudFlare, offering this service even for free Good if you need a fast solution
  36. 36. Proxy Caveats • Few control over proxy servers • Can hurt troubleshooting times • Applications don't see end-user IPs anymore • Fix for HTTP: X-Forward-For • If your real IPs are leaked it's game over again • Not feasible for all protocols This comes not without draw-backs though. If you need to troubleshoot things you might not have the level of control you'd like, since somebody else is already terminating your HTTP requests for you. Your applications don't see IPs anymore If you should leak your IPs it's easy for an attacker to just go for your real servers. And it's not feasible for all protocols, VoIP, UDP-based gameserver protocols, etc.
  37. 37. Routed solution • You need a /24 network prefix • IP tunnel to scrubbing center • All scrubbing locations announce the prefix • Traffic gets sent to closest location, scrubbed and forwarded to you Those get easier to manage if you've got a routed setup Routed approaches are more expensive and complex to maintain Pro: Protocol- and application agnostic
  38. 38. Routing caveats • Takes time to implement & test properly • More expensive to setup and maintain • Routing & BGP can be scary It's not without it's drawbacks though:
  39. 39. Appliances There's still appliance vendors in the DDoS market, e.g. Arbor or Radware. Was a bit surprised by that, but it absolutely makes sense Appliances targeted at end users should always come with a traffic scrubbing offer, otherwise they're snakeoil - it's easy to DDoS a single device running behind even a 10Gbit link. They can make live easier though since they automate tasks like route changes and provide additional information for the scrubbing center
  40. 40. Ask your ISP something you shouldn't neglect is that your ISP may already have DDoS infrastructure set up - it doesn't hurt to ask them what it is they're running and if they can make you an offer
  41. 41. Common caveats • Your applications latency is most likely going to increase • There can be false positives & overfiltering • Your system is becoming more complex Something to consider with any choice you're going to make is Latency is going to increase false positives and overfiltering - how are you going to spot this? your system is getting more complex
  42. 42. Interested? Talk to us! info@ixolit.com m.renner@ixolit.com I hope that gave you a bit of an overview of how the current situation looks like when dealing with DDoS If you're interested - please talk to us Ixolit is both a Prolexic and a Dyn Partner Prolexic is one of the major traffic scrubbing operators in the world. Got recently acquired by Akamai. Dyn.com offers anycasted DNS setups which are DDoS resilient Already have a few production setups running for _huge_ sites which solved all our DDoS issues to this day
  43. 43. Questions? Do you have any questions? Thanks!

×