-
1.
© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Securing Dispatcher + CDN and Client-side caching
Andrew Khoury – Senior Customer Satisfaction Engineer
1
-
2.
© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Prerequisite Knowledge
Before watching this, you need to know:
Basics of HTTP protocol
Apache HTTP Server configurations
AEM Dispatcher
Which requests get cached
Familiarity with dispatcher.any configurations
For review, refer to this past Dispatcher
webinar:https://github.com/cqsupport/webinar-dispatchercache
2
-
3.
© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Securing Apache HTTP Server
Keep Apache HTTP Server binaries up to date - http://httpd.apache.org/
Be aware of the latest Apache security reports:
http://httpd.apache.org/security_report.html
Limit Apache user access –http://httpd.apache.org/docs/2.2/misc/security_tips.html
Disable .htaccess files:
AllowOverride None
If using SSI, set:
Options +IncludesNOEXEC
not:
Options +Includes
Disable UserDir or don’t load mod_userdir
UserDir disabled
Disable directory listing in Apache
Options -Indexes
Disable Apache modules you are not using
Consider using a Web Application Firewall (WAF) such as mod_security or using a WAF appliance
(not covered in this presentation)
3
-
4.
© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Firewall Rules
4
TCP 4503
TCP
80/443
Load
Balancer
Dispatchers Publish Instances Author
Instance(s)
TCP
80/443
TCP
4503
TCP
80/443
TCP
4503
TCP
80/443
TCP
4503
TCP
80/443
External
Networks
*Allow all
outbound
TCP/IP
for Link
Checker
and cloud
services.
*Allow all outbound
TCP/IP
for Link Checker
Internal
Network
Author
Dispatcher
TCP 4502
TCP
80/443
-
5.
© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Firewall Rules
If you are using the Link Checker then allow all outbound TCP/IP connections
If you are not using the Link Checker, but plan to use some Cloud Services then
implement outbound firewall rules as mentioned here:
http://helpx.adobe.com/analytics/kb/adobe-ip-addresses.html
5
-
6.
© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Keep bad traffic out!
Leverage dispatcher to keep bad traffic out
Web Server + Dispatcher
Last line of defense before AEM
Prevent extra load by
Blocking bad requests
Caching valid requests (whenever possible)
6
-
7.
© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Dispatcher Security
Keep dispatcher up to date
Bug Fix Listing:
http://www.aemstuff.com/tools/dispatcheronlinetracker.html
Latest Dispatcher Download:
https://www.adobeaemcloud.com/content/companies/public/adobe/dispatcher/di
spatcher.html
7
-
8.
© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Dispatcher Security – Implementing /filter in dispatcher.any
Keeping bad traffic out using /filter rules
Use the new dispatcher rule format
(covered earlier)
Use a whitelist style /filter section
Deny everything first
Then only allow what you need
For allow rules, be specific
For example, specify the “method”
(“GET”, “POST”, “HEAD”, etc.)
For deny rules, don’t be specific
Use the new vanity URL feature
(After dispatcher 4.1.9 is released)
8
-
9.
© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Dispatcher Security
If your site doesn’t allow user logins then
Block HTTP basic auth
List all allowed headers in /clientheaders in dispatcher.any
Omit header “Authorization”
Block AEM token authentication (/filter section)
/0091 { /type ”deny” /url ”*/j_security_check” }
Block unused request methods (Apache httpd.conf)
<LimitExcept HEAD GET POST>
deny from all
</LimitExcept>
9
-
10.
© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Dispatcher Security – Cache Flooding and Flushing
Error pages
4xx and 5xx responses
Set correct HTTP status (in response from publish)
Cache custom error pages
use DispatcherPassError (httpd.conf)
Return 403 or 404 for bad requests
Block unused selectors (in AEM publish)
Block unused querystrings (in AEM publish)
How?
Use cq-urlfilter –
https://github.com/justinedelson/cq-urlfilter
Or implement a solution (javax.servlet.Filter) in your
application
Set /serveStaleOnError “1”
Block unwanted cache flushes
/allowedClients - restrict which hosts can flush the
cache
10
-
11.
© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Dispatcher Security – Preventing Against DoS Attacks
Implement a periodic refreshing script to cache expensive requests.
RSS feed
Site map
Sample script:
11
#!/bin/bash
#recache_file.sh usage: recache_file.sh /content/geometrixx/en.html
PUBLISH_SERVER=http://host:4503
CACHE_ROOT=/var/www/html
filename=$(basename "$1")
tmpfilepath=/tmp/tmp_cache_$filename
if [ -f $tmpfilepath$1 ] ; then
echo "Not running recache_file.sh - File exists: $tmpfilepath$1"
exit 0
fi
status=`curl -o $tmpfilepath --silent --write-out '%{http_code}n'
$PUBLISH_SERVER$1`
if [ $status -eq 200 ]; then
mv $tmpfilepath $CACHE_ROOT$1;
#chown apache:apache $CACHE_ROOT$1;
Else
rm -f $tmpfilepath
fi
-
12.
© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Dispatcher Security – Protect Against DoS Attacks
Configure /ignoreUrlParams
Allowing requests with querystrings to get cached
Allow rules to “ignore” querystring parameters
Set request timeout per AEM instance (in /renders section)
Set /timeout so that you don’t run out of threads in a apache when the back
end is unresponsive.
5-10 minutes is usually long enough.
12
-
13.
© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
DEMO
13
-
14.
© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Using a CDN
Use a CDN
A content delivery network (CDN) is a large distributed system of cache servers that
optimize content delivery using geographical proximity
CDNs leverage the Cache-Control header or manually configured TTL values to decide
when the cached item is stale.
Some CDNs support purge requests where you can flush items from the cache on
demand.
Popular CDN providers – Akamai, Amazon, Rackspace, etc…
14
Browser
Cache
CDN Dispatcher
AEM Publish
Instances
-
15.
© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Using a CDN
Use a CDN
Another way of reducing traffic from
reaching the back-end
Multiple Deployment Options
1. Use short TTLs and serve all URLs
through CDN
2. Serve assets (images, videos,
etc.), clientlibs and static resources
such as js, css, etc. only through
CDN
3. Implement a custom flush agent
that can purge the CDN and use
long TTLs (serve everything
through CDN) 15
-
16.
© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Using a CDN
1. Whole site through CDN
Pros
Html pages benefit from CDN edge cache performance.
Optimal page load performance
Cons
Potentially expensive as you are serving everything out of the CDN
User waits for TTL expiration before receiving latest content
2. Assets, clientlibs and static resources in CDN
Pros
Save money on CDN charges
Immediate content updates
Cons
More traffic going to Dispatcher servers
16
-
17.
© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Using a CDN
3. Custom flush agent + long TTLs
Pros
Gives ability to deliver content on-demand and cache for long periods
Reduces dispatcher traffic
Cons
In real practice not effective or worth the effort:
CDN purges are generally slow anyway (so I have been told)
Development and maintenance costs on a custom flush agent
Possibly makes sense if you have a very large high traffic site.
17
-
18.
© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Using a CDN
For non-cacheable URLs (with querystring, POST requests, etc) Dispatcher will not
process the response through the Apache handler.
So the headers returned will match those coming from AEM.
Some CDNs (e.g. Cloudfront) will cache responses that don’t have a “Cache-
Control: max-age” set.
Solutions
Set headers to tell the CDN and browser not to cache:
Cache-Control: no-cache
Pragma: no-cache
Or if relevant, set a short expiration like 30 seconds, for example:
Cache-Control: max-age=30
Or allow the browser to cache, but not the CDN, for example:
Cache-Control: private, max-age=30
18
-
19.
© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Using a CDN
For all approaches it would be nice to:
Cache js, css and other “static” files for a
very long time
Be able to use domain sharding
How do we do that?
Use solution developed by Adobe
Consulting
http://adobe-consulting-
services.github.io/acs-aem-
commons/#features
Versioned Clientlibs – adds md5 hash to
clientlib urls
Static Reference Rewriter – rewrites the
domain of resources included in the page.
Used to point clientlibs and other static
resources to the CDN
and handles domain sharding
19
-
20.
© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Using a CDN
Tips for using a CDN
Use mod_deflate in apache to save
money on CDN charges.
If your site has personalization then
consider leveraging ESI
20
-
21.
© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Client-side Browser Caching
Using the Client-side Browser Cache
Often overlooked
Easy to implement
Saves you money on CDN charges
Use mod_expires
Leverage Etags and Last Modified Since headers
When not using a CDN, use Sticky Sessions on your load balancer
21
-
22.
© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Client-side Browser Caching
If using SSI in Apache
Last Modified Since will not be sent
Use mod_expires to set an expiration on those html files
***No cache related headers are sent for files not cached by dispatcher
Note: If user clicks refresh it will re-request the URL (bypassing cache)
22
-
23.
© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
DEMO
-
24.
© 2014 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Thanks Dominique, Hi my name is Andrew Khoury and today I’ll be covering
some basic tips on how to secure your dispatcher
and how to leverage a CDN and client-side browser caches to improve your site beyond what dispatcher provides.
Before watching this presentation, you should already have a basic understanding of
the HTTP protocol,
Apache HTTP Server configurations
and an understanding of what the AEM dispatcher is and how to use it.
Before securing dispatcher, here are some things you can do to make apache http server a little more secure:
First of all, keep your Apache server binaries up to date as security patches are released all the time.
be aware of the latest apache security reports
limit the files and directories that the apache user has access to.
if you are not using htaccess files then disable them,
If you are using SSI then use IncludesNOEXEC instead of Includes to make it so SSI calls cannot execute commands on the Operating system shell.
Disable user directories as this sometimes can expose information that we don’t intend to share.
Block directory listing in apache to prevent users from exploring the server
Disable any apache modules
Use mod_security or some other intrusion detection and prevention system.
This diagram shows a basic AEM architecture.
You can use this diagram as a reference for how you would configure your firewalls.
The idea here is to only allow traffic to flow in the direction it needs to and over the ports it needs to.
When configuring your firewalls, keep in mind that
if you are not disabling the link checker then you will need to allow all
outbound tcp/ip connections from author and publish instances.
If you plan on disabling the link checker, but need to integrate with Adobe’s
Cloud services then you can refer to the Adobe knowledge base for a list of ip
addresses to allow outbound connections to.
If you are familiar with basic AEM architecture
then you know that your web server and dispatcher
are your last line of defense before a request can
reach the publish instances.
Due to this, it is important to lock down your dispatcher and block as much unwanted traffic as possible before it reaches the publish instances.
As a first step in locking down your dispatcher’s security you should always keep the dispatcher binary up to date with the latest security fixes.
The next important thing is to
Create a strong set of filter rules in your dispatcher.any file. Filter rules will help
You keep bad traffic from reaching your publish instances.
When implementing the filter rules, it is best to use a whitelist.
This means that you deny all first, then
Only allow the requests you need for your site to function properly.
When creating allow rules, be specific as to request methods and URL patterns you want to allow.
For deny rules, be as general as possible to block all variations of bad requests.
Also, If you use the vanity URL feature in AEM then in order to implement a white list you will need to leverage the new dispatcher feature that Dominique covered earlier.
(Show dispatcher.any file) Now I’ll quickly show you my dispatcher.any filter rules.
After configuring filter rules, the next thing to consider is authentication.
If your site doesn’t allow users to log in then block users from authenticating against experience manager.
To do this, Block HTTP basic auth by listing all request headers in the /clientheaders and omit the authorization header.
Then block AEM token auth by filtering out all requests for j_security_check
For additional security, you could also block any request methods that are not supported by the site at the apache level in your httpd.conf using the LimitExcept directive.
Even with the best Filter rules you cannot filter out all invalid request patterns.
So to protect the dispatcher further there are a few things that can help.
Make sure that your error responses such as 404 not found return the correct error codes and don’t return status 200.
Cache your custom error pages in the dispatcher cache by configuring the DispatcherPassError feature.
Return 403 or 404 for bad querystring or selectors in URLs
This can be done by using the open source cq-urlfilter tool or by implementing your own javax.servlet.Filter that blocks the unwanted traffic.
One other setting that can help protect your publish instances is to set the serveStaleOnError flag. This flag tells the dispatcher to serve whatever cached files it has in case all publish instances are inaccessible.
Additionally, to protect against false dispatcher flushes we should always set /allowedClients with IP addresses of the publish instances to restrict which servers can perform dispatcher flushes.
If your site has any expensive requests such as RSS feeds or large site maps then it might make sense to exclude those requests from the dispatcher cache rules and use a periodic script to cache those files instead.
To do this, you would block the url in the /filter section of dispatcher.any and use a script like the one on this slide to handle re-requesting and re-caching the file.
If you don’t use querystrings in your site then set/ignoreUrlParams to allow requests with querystrings to get cached. This feature basically lets you specify rules for which querystrings you want to remove from the url before forwarding the request to the publish instance.
Finally, one thing you can do to prevent running out of apache threads is to set the connection /timeout in the /renders section of dispatcher.any in case requests are hanging, waiting on the publish instances.
The next step in keeping your site running smoothly is to leverage other upstream caches such as browser caches and CDNs.
First I’ll start with CDNs.
If you are not familiar with CDNs, they are large distributed networks of cache servers that optimize content delivery using geographical proximity.
To manage your cached content within a CDN, most of the time people rely on Cache-Control headers or manually configured TTLs to control the freshness of the cache.
Some CDN providers such as Akamai support on demand flush requests.
A CDN in our case is yet another way of reducing the amount of traffic that reaches the back-end.
When integrating a CDN with AEM and dispatcher there are multiple options.
We can…
(read numbers)
Here are the pros and cons
One issue that can come up when integrating a CDN into your AEM architecture is that non-cacheable requests respond with the headers set by AEM, not apache. This presents an issue when the response is served without a Cache-Control header set as some CDNs cache these responses.
The solution is to set cache-control headers at the AEM level so that if a file is non-cacheable it will still have the correct headers.
When integrating a CDN with your AEM instances it is nice to be able to
Cache js, css and other static files for a very long time by using a unique URL per version.
And to be able to implement domain sharding
It’s basically where you use multiple subdomains pointing to your CDN to serve resources such as images, js and css. By using multiple domains the browser is able to download files in parallel and the page will load faster.
To do this, the Adobe consulting team has implemented two tools
The first is called versioned clientlibs which adds a unique identifier to clientlib urls
The second is called Static reference rewriter which rewrites certain urls to point to a different domain. It also supports domain sharding.