ASM DDoS profile - This session provides an overview on how to configure the ASM DoS profile to detect and mitigate denial of service (DoS) attacks at layer 7 of the OSI model.
This training was created by Lior Rotkovitch
4. 2
3
4
1
Servers
Human Bot
DoS Profile
Amazoogle
CAPTCHA Challenge | Block Suspicious Browsers
Rate Limit, Client-Side Identity Defense, CAPTCHA, Block All
Rate limit requests from IP or based on server health,
Detect and block known bots, DoS tools, other agents
6. Hacktivism
Google Web Bot
Unidentified User
User
Users Or Bots?
Web Bot
Source IP
addresses
IP:X.X.X.X
Servers Database
Web Site
IP:X.Y.Z.A
IP:A.B.C.D
30 requests
second
25 requests
second
200 requests
second
12. Hacktivism
Google Web Bot
Unidentified User
User
Users Or Bots?
Web Bot
Servers Database
Web Site
• Measuring increases in requests from a Device IDID: LK142
ID:LQ87A
ID:PN76A
ID:Z18B3
ID:B58B3
21. App URLs &
objects
• Measuring requests increase on both IP’s and URL’s
Hacktivism
Google Web Bot
Unidentified User
User
Users Or Bots?
Web Bot
Source IP
addresses
IP:X.X.X.X
Servers
Web Site
IP:X.Y.Z.A
IP:A.B.C.D
30 requests
second
25 requests
second
25 requests
second
23. Site Wide
By Geolocation
By Source IP
By Device ID
By Source IP
• a
• Request block
Prevention Policy
1. Client Side Integrity Defense
2. CAPTCHA
3. Request blocking
I. Rate limit
II. Request block
Client Side Integrity Defense
24. Client: Hey server, can I get the web page ?
ASM: Can you prove you are a browser?
User
If a browser:
Yes, I’m a browser
ASM: Ok, you are allowed. Here is the
web page you asked for.
ASM: Bye Bye – Blocked
Server
If a bot:
*^lkjdfg@#$
25. Client Side Integrity Defense - Flow
User Browser DoS Profile App
First main page access HTTP Request (no cookie)
Computational challenge
Solve challenge/
set cookie with time stamp
HTTP Request (cookie) Reconstruct request
Original HTTP Request
HTTP Response (main page)HTTP Response (main page)
More object requests (cookie)
Validate cookie: format & time stamp
More object requests
More responses
More responsesDeliver page
Valid source:
• The client supports JavaScript
• The client supports HTTP cookies
• The client calculates a JS challenge
Not valid:
• Didn’t pass the above
• Cookie is wrong format – Block (RST)
• Time stamp expired – Block (RST)
Send JS test
26. Site Wide
By Geolocation
By Source IP
By Device ID
By Source IP
• a
• Request block
Prevention Policy
1. Client Side Integrity Defense
2. Completely Automatic Public
Test for Telling Computers and
Humans Apart
3. Request blocking
I. Rate limit
II. Request block
27. User
Client: Hey server, can I get the web page ?
ASM: No. First prove you are a human by
solving this CAPTCHA puzzle.If a human:
Here is what I see…
If not human:
*^lkjdfg@#$
ASM: Ok, you are allowed. Here is the
web page you asked for.
ASM: Bye Bye – Block him dude!
Server
Web Bot
28. User Browser DoS Profile App
Request login.php GET / mypage.php (no cookie)
CAPTCHA HTML +JS response
Cookie with time stamp
Solve CAPTCHA
CAPTCHA rendered
Submit CAPTCHA
solution
GET /mypage.php + CAPTCHA
cookie
Verify CAPTCHA solution
Validate cookie
GET /mypage.php
HTML of mypage.phpHTML of mypage.phpmypage.php
rendered
Send
CAPTCHA
• While the system is still in a
state of attack the offending
source will be presented with
another CAPTCHA every 5
minutes
• Same as CSID, request is
held by ASM until CAPTCHA
is solved
29. Site Wide
By Geolocation
By Source IP
By Device ID
By Source IP
• a
• Request block
Prevention Policy
1. Client Side Integrity Defense
2. CAPTCHA
3. Request blocking
I. Rate limit
II. Block all
30. Client: Hey server, can I get the web page ?
ASM: (a) no, I’m limiting your requests sending rate
CSID and CAPTCHA try to understand the offending source
Request blocking limits / blocks the offending sources.
Server
Just 1 request per minute ?
ASM: (b) no, I’m totally blocking your
35. Pros Cons
No mitigation Collect information and present
it in AVR
• Provides visibility into DoS
attack.
• Can switch to any of the
protection mode immediately
No mitigation
Standard Protect the server availability at
all cost
Protects server availability at all
cost
Might drop legitimate users
Conservative More accurate mitigation and
only bad actors will be blocked
Blocks only bad actors Will not protect server
availability at all cost
Aggressive Effective way to mitigate bad
actors and drastically slow
down requests from an
anomalous IP address
Blocks any suspicious IP
aggressively and will protect
server availability at all cost
Might block good actors
36. Process name admd
Description Anomaly detection daemon of BADoS
Module (realted) Behavioral DoS
Process name admd
Does what ? Aggregates behavioral statistics, looks for anomalies, detects
bad actors
Communication with other process Communicates with tmm, mcp
Runs When? When ASM is provisioned
Log file /var/log/adm/admd.log
License required ASM
File system Location /var/run/adm/
Configuration File Part of DoS profile in bigip.conf
How to enable Debug mode Enable debug logging
modify sys db adm.logs.level value debug_lite
Watchdog process ADMD.Publisher
38. • curl
• wget
• ab
I’m a simple Bot
ASM: Yes, I have your signature.
Sorry mate you are blocked.
Server
ab -c 1 -n 100 -r -H "User-Agent: Amazoogle" http://10.10.X.101/
42. DNS Server
Gohogle
I’m a GoogleBot
ha ha ha
Gohogle
Bummer
ASM: You are not google bot.
Bye Bye! Block this creature !
ASM: let’s see if you are. I’m doing
Reverse DNS lookup.
ASM: Hey DNS, who’s this guy ?
DNS: No one important
Server
DNS Server
Googlebot/2.1 (+http://www.google.com/bot.html)
Google
I’m a google Bot
43.
44. Bummer
Capabilities ?
CAPTCHA ?
ASM: OK, what are your capabilities ? If you answer
wrongly, you will have to answer a CAPTCHA
or / and be blocked
I’m a Bot
pretending to be a
browser
Server
You are not human! Bye! Block this unhuman !
45.
46.
47.
48. • If Block Suspicious Browsers is unchecked send CS Challenge
• If Block Suspicious Browsers is checked and CAPTCHA is checked send Client Capabilities
challenge and give it a score: If score in doubt send a CAPTCHA for human verification
• If Block Suspicious Browsers is checked but CAPTCHA Challenge is unchecked do not
send CAPTCHA and only block if the score is more than a human
49. User Browser DoS Profile App
First request GET /sell.php GET /sell.php (no cookie)
Client Capabilities Challenge response
Return Client Capabilities
verification
Reconstruct request
HTTP Response (cookie)HTTP Response
GET /img.png (cookie)
Blank page & Set cookie
Original HTTP Request + cookie
Authenticate and decrypted JS results,
Compute browser score based on result
Determine an action based on score
GET /img.png (cookie)
Validate cookie:
format & time stamp
50.
51.
52. # EXAMPLE: enable client-side challenges on a specific URL
when BOTDEFENSE_REQUEST {
if {[HTTP::uri] eq "/search.php"} {
BOTDEFENSE::cs_allowed true
}
}
https://devcentral.f5.com/wiki/iRules.BOTDEFENSE.ashx
# EXAMPLE: allow CSID actions on URLs with the .html extension
when BOTDEFENSE_REQUEST {
if {[HTTP::uri] ends_with ".html"} {
BOTDEFENSE::cs_allowed true
}
}
53. • Allows access to the web site
regardless of geolocation detection
criteria thresholds only i.e. other
thresholds still apply
• Specifies the countries that the system always
blocks whenever the system is in a state of DoS
detection.
• Done regardless of the thresholds set in the DoS
profile
Welcome to ASM Layer 7 DoS Mitigations.
This session provides an overview on how to configure the ASM DoS profile to detect and mitigate denial of service (DoS) attacks at layer 7 of the OSI model.
This training was created by Lior Rotkovitch and produced by Erik Novak.
This training is provided by the F5 Security Incident Response Team (SIRT)
To learn more about the F5 SIRT, <CLICK> visit go/f5sirt.
Please E-mail any security related inquiries to <CLICK> f5sirt@f5.com
ASM can detect and mitigate various DoS attacks against a web application by a subjecting HTTP request traffic to security checks configured in the DoS profile.
This training focuses on the following configuration settings in the DoS profile:
TPS-based detection and its prevention options
Stress-based detection and behavioral analysis
Bot detection and defense mechanism
A few iRule examples for DoS mitigation
It is assumed that you are familiar with ASM and the basic concept of a brute force attack against a web application.
The DoS profile includes four major detection methods.
Each has a different approach for identifying a DoS attack.
Let’s start with TPS-Based detection.
which identifies anomalies, typically increases in Requests Per Second at the source OR destination.
Behavioral and stress-based detection identifies latency in the backend servers first, and only then measures the increase in requests per second.
Proactive Bot defense can classify the attack agent as a human, a browser, or a bot.
Bot signatures identify a pattern of the exploit or the attack agent in the payload, and work with or without proactive bot defense.
All detection options can use a different prevention policy to mitigate attacks.
Let’s start with the TPS-based attack detection. TPS-based protection detects anomalies in how many requests a client is sending every second. If the rate of requests exceeds a defined threshold, then a DoS attack is underway.
There are five TPS-based detection types and four prevention options let’s review each one of them, beginning with By Source IP.
With By Source IP detection, ASM measures increases in requests per second from a Source IP address arriving to the application. The idea is to identify a source IP that is exceeding a maximum threshold of requests.
This detection method is useful when the DoS attack is executed from multiple IPs and sends massive traffic that exceeds the thresholds specified in the DoS profile.
In the DoS profile, click Edit in the By Source IP row to navigate to threshold configuration sections.
The GUI for By Source IP is divided into Detection Threshold and Prevention Policy sections.
There are two types of TPS thresholds that can be triggered: One is a Relative threshold which is ratio-based, and the other is a fixed rate or Absolute threshold.
If the Relative threshold is reached, or the Absolute threshold is reached, the configured prevention policy will apply.
The prevention policy is the same for all detection methods and is covered later in this lesson.
It is important to understand how the detection thresholds are calculated. Relative thresholds are determined by keeping a request rate history in ASM memory. The ratio of requests per second in a long history interval is compared with the ratio of requests per second in a short, or current interval. For example if the historic request per second rate was 50 and the current rate is 370, then that is an increase of 640 percent.
Since this exceeds 500% the configured prevention policy will be applied on the specific source IP.
Take a minute to examine this page, click next when you are ready.
The Absolute Threshold is a fixed number of requests per second. If a source IP exceeds the threshold, it is considered to be an attacker.
Absolute Threshold is a quicker way to activate the prevention policy than using a relative threshold, because there is no need to do any calculation.
Any source IP that exceeds the Absolute Threshold is consider to be an attacker and the prevention policy action will apply to it. If all the options in the prevention policy are checked, then the order in which they apply is top down as ordered in the GUI.
The second type of TPS-based Detection is by Device ID. The same prevention policies apply.
An attacker can use different usernames, and actual source IP addresses can be hidden by NAT, so more accurate protection can be configured using the Device ID method.
A Device ID can uniquely identify each source by recording hundreds of attributes, such has browser settings, add-ons, and other elements. To create a Device ID, ASM returns a response that includes a proprietary JavaScript. ASM then labels the device based on fingerprint data that the JavaScript collected and stores it in a hash.
After the Device ID is assigned, ASM measures increases in requests per second from a specific source, identified by its ID.
The By Device ID configuration uses the same relative and absolute threshold concepts as By Source IP, which if reached, activate the prevention policy.
As with By Source IP, there are pros and cons to each detection method. Using the Relative Threshold is good for mitigating a source that historically sent a consistent rate of requests per second, and is currently increasing the rate of requests.
Using the Absolute Threshold is better for quickly defining the limit of acceptable requests per second by a given source.
If a source reaches the threshold, ASM considers it an attacker.
The third type of TPS-based Detection is By Geolocation. The same prevention policies apply.
By Geolocation measures requests per second from a specific country.
If requests from a specific country exceed a defined threshold, the prevention policy will be activated.
Geolocation thresholds are calculated in a different way than the other detections we have seen so far.
DoS detection is done by tracking an increase in overall traffic share, meaning total requests from a specific country, and then comparing it with a minimum percentage of traffic share from that country.
In this example, the traffic share from a specific location should increase by 500% AND represent at least 10% of traffic to the entire site.
The fourth type of TPS-based Detection is By URL. This detection method is useful when the DoS attack is targeting specific URLs.
The same prevention policies apply.
By URL-based protection is based on requests per second for a specific URL. Note that this is different then measuring by Source IP only, because ASM measures requests per second for the URLs arriving from multiple source IP addresses.
If thresholds are reached, a URL is considered under attack and the prevention policy will be applied.
The detection threshold configuration for By URL is similar to By Source IP and By Device ID, and includes the relative threshold based on an increase in the ratio of current and historical requests per second, or the fixed absolute threshold.
When the requests threshold for URLs are exceeded, ANY source IP sending a request to this URL will receive the configured prevention policy.
The fifth type of TPS-based Detection is Site Wide which should be used when other detection thresholds are not exceeded but the site is still experiencing heavy load.
The same prevention policies apply.
Sometimes DoS attacks evade detection thresholds but the site still experiences stress.
Site-wide detection monitors both requests per second from source IPs and requests for URLs which makes it more effective in detecting low and slow DoS attacks.
The values for Site Wide detection thresholds are greater than those of other detection methods because the measurement applies to all traffic on a virtual server.
Relative thresholds will apply historical and current TPS metrics to the entire site so any traffic increase will be detected.
The Absolute Threshold is the fixed number of requests allowable for the entire site. If thresholds are reached, the Prevention policy is applied.
As mentioned earlier, each detection method has three types of prevention policy. The first one is Client side integrity defense also known as CSID.
CSID inspects the source and tries to identify it is a browser or a bot.
A source sends a request. ASM sends a CSID JavaScript to the source and waits for a reply which indicates the nature of the source.
If the source passes the CSID test, it will get 101 cookie and be exempt from other checks. If the source fails the test, or refuses to respond to the JavaScript challenge, it will be blocked.
CSID is transparent to the user and is executed in the browser, without needing any user intervention.
Here is the CSID Event Flow.
A user send a request to the web application. Since it is the first request it has no ASM cookies in it. The request arrives at the ASM DoS profile, which holds it and sends an empty HTML page with the CSID JavaScript back to the browser. The JavaScript must be interpreted by the browser and applies the following computational checks:
Does the source support JavaScript?
Does the source support HTTP cookies?
Is the source capable of calculating a challenge inside the JavaScript?
A source that passes the JavaScript challenge is considered a legitimate browser.
In this case ASM will add a 101 cookie indicating that this source is legitimate and from now on will be exempt from additional checks for the next 5 minutes.
ASM then reconstructs the original request and sends it to the application.
Requests containing valid cookie, format, and time stamp are legitimate.
If the checks failed then the source will not get access to the web application.
In addition if the ASM cookie is in the wrong format or its time stamp has expired the request will be blocked. Take a minute to examine this slide and click next when you are ready.
The second prevention policy option is the CAPTCHA challenge for telling humans and bots apart.
The CAPTCHA challenge is more intrusive than the CSID since it requires human intervention for both interpretation and action. If the CAPTCHA is answered correctly, the request will be allowed to pass to the application.
If a source keeps on sending requests without answering the CAPTCHA it will be blocked.
This illustration is the sequence of events from the initial user request through successful completion of the CAPTCHA challenge. Once the CAPTCHA challenge is solved, the client will have a cookie that validates each request for format and time stamp.
After 5 minutes, ASM will issue a new CAPTCHA to this client.
The CAPTCHA response can be configured and supports HTML and JavaScript.
The last prevention policy option is Request blocking which has two options: Rate limit and Block all.
The request blocking prevention policy can be used in two ways:
One is to block all traffic from the source IP, the other is to rate limit the amount of traffic the source IP can send.
This approach is useful when we know that a source is executing an attack and should be blocked or rate limited to reduce the number of requests it can send.
While CSID and CAPTCHA try to understand who is the offending source (bots or browsers) request blocking is indifferent to the “identity” and simply rate limits or blocks the offending sources.
Behavioral and Stress-based Detection methods can be configured to work with TPS-based Detection or independently.
The main difference between them is that stress-based detection will trigger the prevention policy only if the backend latency increases and the TPS thresholds have been reached.
The section for Behavioral and Stress-based Detection contains two major detection types and mitigations.
The first section is Stress-based Detection and Mitigation which is similar to the TPS-based configuration options we examined earlier.
However these thresholds will only be activated when the behavioral analysis DoS daemon working in the background detects stress on the backend server—typically increased response latency.
The second section is behavior detection and mitigation which is a dedicated engine with it own detection and prevention mechanism.
Both types can work together and the first one to get triggered will be the active mitigation.
Since the first type of detection is similar to TPS-based, we will focus on the Behavioral Detection and Mitigation section.
Behavioral DoS is an automatic detection method based on analysis of behavioral data which the behavioral engine collects.
The data is characterized without the need to define thresholds and once Behavioral anomalies occur the offending sources are automatically mitigated.
The engine is self-adjusting and adaptive to changes in traffic.
Selecting the checkbox for Bad Actors behavior detection enables the engine for attack detection.
There are four mitigation modes:
No mitigation allows Behavioral DoS to collect information and present it in AVR to provide visibility into DoS attacks. This mode also builds a history and makes it easy to switch to another of the mitigation modes immediately.
Standard protection is the default mitigation and will protect the server availability at all cost however might terminate connections for legitimate source IPs.
Conservative protection allows more accurate mitigation and only bad actors will be blocked. This will not protect server availability at all cost.
Aggressive protection is a highly effective way to mitigate bad actors and drastically slow down requests from an anomalous IP address. However it might also block legitimated source IP addresses.
This table summarizes mitigation options.
This table provides details on the Behavioral DoS daemon and log. Behavioral DoS is enabled when ASM is provisioned and licensed on the BIG-IP. The main BADos process is ADMD which is implemented inside the hud chain.
Take a minute to review the table. Click next when you are ready.
Now let’s look at detecting and mitigating bots.
Typical we classify bots in 3 main categories:
Simple bots – are command line bots such as wget or Curl
Impersonating Bots – are malicious bots that pretend to be a legitimate bot
Bots acting as a browser - are bots with advanced client-side capabilities
Simple bots are also known as command line bots and typically include a known user agent string in their requests. The DoS profile has a bot signature mechanism which can identify those types of bots.
Note that bot signature is independent of the ASM attack signatures and work only when a DoS profile is assigned to the virtual server.
The bot signature tab in the DoS profile has three main sections:
The first section is the malicious category which includes known DoS tools, Spam Bots, etc.
The following actions can be taken if a bot from a known category is detected.
None: no action will be taken.
Report: ASM will report the category on the bot defense reports page.
Block: ASM will report the category and will block the request
The second section is benign bots—those signatures that are allowable on the web site. However, a detected benign bot can also be subjected to reporting or blocking action.
The final section in the bot signature page is the bot signature list where a specific signature can be disabled and excluded from the configured action.
Bot signatures are updated via the standard process for attack signature updates. Custom bot signatures can be added from the options tab.
A Simple Edit Mode allows creation of a bot signature for the the DoS profile.
The DoS profile has a verification process to distinguish between legitimate and impersonating bots.
Legitimate bots declare who they are with a domain name in the user agent header. ASM then takes this domain name and executes a reverse DNS lookup. If the outcome is similar to the ASM-expected IP address, then the request is excluded from the DoS profile tests.
Bots can easily impersonate a legitimate bot, such as googlebot, and include a google user agent in their request. However, this technique will fail the DNS lookup done by ASM.
ASM ships with a list of known bots that include their domain name.
The list is on the Search Engines page. Additional domain name can be defined by clicking create and then providing the details.
It is important to configure DNS for this system to work. (system ›› configuration ›› device ›› DNS.)
Some bots are capable of simulating a browser, and can pass CSID challenges.
Proactive bot defense is a mechanism designed to detect and prevent advanced bots from accessing the site by sending client side challenges to ALL sources all of the time.
These challenges include browser-specific evaluations. If ASM cannot determine if a client is legitimate or a bot masquerading as a browser, then a CAPTCHA is sent.
Before looking at configuration options, note that Bot Signatures are enabled automatically when Proactive Bot Defense is selected. Proactive bot defense runs in one of three Operation Modes.
It can be off, in which case no challenge is sent.
It can run During attack, which means another mitigation is in place, and the Proactive Bot Defense challenge is a secondary mitigation.
It can Always run, and send CSID all the time.
The grace period of indicates that a successful client will not get challenged again for five minutes.
ASM Proactive bot defense requires a DNS resolver configuration to fully operate.
A red message is displayed if the DNS is not configured. Clicking on the link will navigate to the DoS profile DNS resolver.
Proactive bot defense has an option to Block requests from suspicious browsers. Click the Edit button to fine-tine settings. The two checkboxes in the GUI allow you to modify the level of defense and accuracy by immediately blocking highly suspect browsers and sending a CAPTCHA to browsers which are suspect but not yet known as malicious.
Proactive bot defense has two scripts that challenge bots.
If the Block Suspicious Browsers checkbox is not selected, then proactive bot defense sends the client side integrity defense to the client and if it is a qualified browser it is allowed to access the server.
If Block Suspicious Browsers is selected, then an additional script called the Client Capabilities Challenge is sent to the client. This script inspect the browser and tries to find discrepancies between what the browser says it is to what it actually sent. For example if the client has a screen resolution of 800 X 1200, but the user agent indicates an iPhone, then this request is most likely not from a browser, and CAPTCHA will be sent to the user to verify it is a human.
Take a minute to review this slide and then click next when you are ready.
The first request arrives at the DoS profile with no cookie. Proactive bot defense holds the request and sends the Client Capabilities script back to the browser. Note that the request doesn’t arrive at the server at this point.
The JavaScript then interrogates the browser and returns results to the proactive bot defense .
Proactive Bot Defense then authenticates and decrypts the JavaScript. The browser score is computed based on the two database comparisons, and finally the action that will be taken is determined.
The score can indicate that the source is a browser.
Or the score can indicate that the source is unknown and a CAPTCHA is sent
The score can indicate a problem, and the request is blocked with connection reset.
If the score is a browser or the CAPTCHA is correctly answered then the original request is sent to the web server.
The entire flow described here is transparent to the user and causes only a minor delay from the web server.
ASM has a good bot reporting page. The detailed log provides the following main details:
The Request URI displays the URI that the bot accessed.
Request status displays if the request was legal or got challenged.
Action indicates if the request was accepted or subjected to browser challenge .
Reason describes the rationale for the action taken.
Bot signature displays the bot signature name that was triggered inside the request .
Click next when you are ready.
Bot defense reporting is not enabled by default and requires additional configuration. First, create new logging profile.
In the new logging profile, click the checkbox for Bot Defense. Make sure all the relevant options are turned on under the bot defense tab.
Bot defense functionality can also be invoked from an iRule. For example, invoking a client side integrity defense challenge is a useful technique to mitigate DoS if it needs to be done on a specific URL. This allows flexibility when inspecting sources during a DoS attack.
A full description of the bot defense iRule is available on F5 Dev Central.
On the DoS profile main page, there is an additional mitigation option which allows creation of a Geolocation blacklist. Blacklisted countries will be blocked whenever the system is in a state of DoS detection.
The white listing of geolocations will exempt the TPS or stress based geolocation detection criteria thresholds while other mitigations still apply.
In summary, TPS and stress based detection methods seek anomalies in the rate of HTTP requests, and are ideal for detecting application DoS.
TPS-based detection is quick and easy to define, and stress based has the advantage of the behavioral DoS engine.
Bot signatures are an additional layer of defense to the ASM attack signatures. Bot signatures easily detect known bots and can be customized using Simple Edit Mode.
Protective bot defense can mitigate attacks by advanced bots
acting as browsers.
All ASM DoS profile activity can be viewed on the bot defense reporting page. Note that it is not enabled by default and a logging profile should be created.
Finally, iRules, the F5 swiss army knife, can invoke the bot defense script for specific use cases.
Thank you for listening!
For any question or feedback please send an email to lior@f5.com or the f5sirt@f5.com