Apache Web
Performance
Leveraging Apache to make your
site fly
Friday, March 16, 12 1

Who am I?
&
Why would you listen to me?
Friday, March 16, 12 2

Devon Hillard
Web Application Architect
Built 10MinuteMail
Run Spark::red, an enterprise
eCommerce Hosting company
Complex Web Apps with extremely
high traffic and critical performance
needs
devon@sparkred.com
Friday, March 16, 12 3
Been doing this for 14 years.
10MinuteMail handled a Slashdotting, front page listing on Yahoo
Tech, and front page Digg at the same time.
Worked on sites such as AT&T.com, People’s Choice Awards, Payless
Shoes, Vermont Country Store.

What is Performance?
Friday, March 16, 12 4

Server-Side
Performance
• Traffic Capacity
• Request handling speed
• Request handling
throughput
• Lower CPU/memory/
bandwidth usage
• Scalability
Friday, March 16, 12 5

Client-Side
Performance
• Page Load Time
• First draw
• Inter-actable
• Complete
• Page Interaction
Responsiveness
• Time to Complete Use Case
Friday, March 16, 12 6
When a user sees the page start loading, to when they can do stuff,
to when the browser’s loading animation is done.
Use Ajax to pre-populate fields, type-ahead, real-time field
validation, reduce the number of full page submits/requests.
Solving for client side performance can have a dramatic impact on
the server side performance. Cached assets draw pages faster AND
reduce requests the server has to handle.

These two are TIED
• Browser caching of static assets reduces page load
time AND reduces the number of requests the server
has to handle for the same number of page views
• AJAX requests can typically be handled with far fewer
resources than full page requests
• Reduced asset sizes means less bandwidth used and
shorter request response sending times
• Solving for the Client brings Server performance gains!
Friday, March 16, 12 7
In engine building they say “Build for torque and the horsepower will
follow”

Why Should I Care?
Friday, March 16, 12 8

Everyone Wins!
• Client-Side Performance means
• happier users
• increased conversions
• increased SEO ranking
• Server-Side Performance means
• more capacity on same hardware
• saves money
• scaling is easier
Friday, March 16, 12 9

Increased Conversions
• +100 ms of page load time = 1% drop in sales
• +500 ms of page load time = 20% drop in searches
• +400 ms of page load time = 5-9% increase in clicking
“back” button before the page finished loading
Friday, March 16, 12 10
We’ve all seen these stats, but they’re worth repeating. This isn’t
ivory tower stuff, this is simple business.

Why Apache?
Friday, March 16, 12 11

Why I Use Apache
• Popular web server - it’s everywhere
• Easy to install, troubleshoot, and find information on
• Mature and stable
• Lots of extensions
• Enterprise support requirements
• It Is fast enough to max out your hardware!
Friday, March 16, 12 12
Why not nginx (engine-X), Lighttpd, etc...?
Extensions like mod_pagespeed, mod_cluster, and more.

CPU Util @ 930 Mbit/sec
http://www.webperformance.com/load-testing/blog/2011/11/what-is-the-fastest-webserver/
Friday, March 16, 12 13

iftop output - 613 Mbit/sec
PCA Awards
• Saw 4x the planned for
traffic during spikes
• 140,000+ pages/minute -
2,333 pps (2012)
• 3,000+ Mbit/sec (2011)
• 1,200+ Mbit/sec PLUS
Akamai CDN offloaded traffic
(2012)
• The site stayed up and was
quick to load and responsive
to interact with the whole
time
Friday, March 16, 12 14

Puppies
Friday, March 16, 12 15

Basic Apache Configs
Friday, March 16, 12 16
The problem with Apache configs are that the defaults are typically
extremely out of date and based on much older hardware, network
bandwidth, browser abilities, and application complexity

Every App
is
Different
Friday, March 16, 12 17

Which MPM?
• Worker MPM scales for high traffic without running out
of memory better
• Less time spent tuning worker configs
• Unless you’re still using non-threadsafe Apache code
(some PHP, etc..)
• New in Apache 2.4 is the Event MPM
Friday, March 16, 12 18
Limited testing I did with Event MPM shows it’s ~5% faster than
Worker, without real tuning.

MPM Worker Configs
Default Mine
<IfModule worker.c> <IfModule worker.c>
StartServers 2 ThreadLimit 100
MaxClients 150 StartServers 5
MinSpareThreads 25 MaxClients 5500
MaxSpareThreads 75 ServerLimit 200
ThreadsPerChild 25 MinSpareThreads 100
MaxRequestsPerChild 0 MaxSpareThreads 1000
</IfModule> ThreadsPerChild 100
MaxRequestsPerChild 0
</IfModule>
Friday, March 16, 12 19
Just set it and forget it.
If you have memory leak issues, set MaxRequestsPerChild to
something like 5,000.

ab with small file
Version Apache 2.2.3 MPM Worker
Document Size 119 bytes
Concurrency Level 1,000
Total Requests 100,000
Requests Per Second 20,790 r/s
Throughput 8,077 kb/s
hex core X5675
Friday, March 16, 12 20

ab with large file
Version Apache 2.2.3 MPM Worker
Document Size 95,002 bytes
Concurrency Level 1,000
Total Requests 100,000
Requests Per Second 1,258 r/s
Throughput 119,966 kb/s
hex core X5675
Friday, March 16, 12 21

Linux Kernel Configs
net.ipv4.tcp_max_syn_backlog=30000
net.core.netdev_max_backlog=1200
net.ipv4.tcp_timestamps=0
net.ipv4.tcp_sack=0
net.ipv4.tcp_window_scaling=0
net.ipv4.tcp_max_tw_buckets=2000000
net.ipv4.tcp_mem=100000000 100000000 100000000
net.ipv4.tcp_wmem=100000000 100000000 100000000
net.ipv4.tcp_rmem=30000000 30000000 30000000
net.ipv4.ip_conntrack_max = 231072
Friday, March 16, 12 22
This last setting (or larger) will keep your kernel from dropping
incoming connections under high load

Keepalive - On or Off?
• reduces overhead of establishing new connections for
each request from the browser
• can waste memory and other resources if left open too
long
• Common practice is to disable them
• I turn keepalive on, set to 6 seconds or 500 requests
• For CDNs like Akamai, you’ll want to turn up the time to
more like 120 seconds
Friday, March 16, 12 23
Defaults are On, 15 seconds (too long), 100 requests (too small)

GZip - mod_deflate
• gzip compressing text resources - html, js, xml, css
dramatically decreases the size of the response for those
assets: often up to 90% reduction in size
• reduces transfer time, especially for clients with slower
connections (or big files)
• This decreases page load time (Client) and reduces the
time the server thread is sending the response (Server)
# Removing Hosts vary
Header unset Vary
<IfModule mod_deflate.c>
AddOutputFilterByType DEFLATE text/html text/plain text/xml
text/css application/x-javascript
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4.0[678] no-gzip
BrowserMatch bMSIE !no-gzip !gzip-only-text/html
</IfModule>
Friday, March 16, 12 24

Browser Caching
• Tell the browser to cache all static assets, for as long as
you can bear it
• Set the ETag
• Set cache control and expiration response headers
<LocationMatch '^/pca/images/.*'>
FileETag MTime Size
Header set Cache-Control "max-age=2764800, public"
<IfModule mod_expires.c>
ExpiresActive On
ExpiresByType image/jpg A2764800
ExpiresByType image/jpeg A2764800
ExpiresByType image/png A2764800
</IfModule>
</LocationMatch>
Friday, March 16, 12 25
This is HUGE! You can reduce the number of HTTP requests you
have to handle by an order of magnitude or two by setting good
cache headers.

Disk Caching
• Why not use mod_mem_cache?
• Transparent web server caching of assets reduces load
without deployment and development complexities
• Some warnings!
• Doesn’t auto clean itself
• Can grow to be huge if you don’t check the
headers being cached
• Can grow to be huge if you have dynamic URI
content - blah.jpg;jsessionid=foo
• Another place to purge cache for changed files
Friday, March 16, 12 26
Mod_mem_cache has some bad bugs. mod_disk_cache lets the
kernel’s in memory file caching algorithm do all the heavy lifting
without tuning or exhausting available memory.
For 3 tier environments, it’s common to have static assets that get
deployed to the web servers separately from the application itself.
This makes deployments more complex and error prone and makes
local dev environments much harder to keep up to date.
Seam and richfaces image tags insert jsessionid to URI, etc...

Disk Caching - Config
<IfModule mod_disk_cache.c>
CacheRoot /var/cache/mod_disk_cache
CacheDirLevels 2
CacheDirLength 1
CacheEnable disk /pca/img
CacheEnable disk /pca/flash
CacheEnable disk /pca/css
CacheEnable disk /pca/js
CacheEnable disk /pca/images
CacheMaxFileSize 200715200
CacheDefaultExpire 3600
CacheIgnoreHeaders Set-Cookie
</IfModule>
Friday, March 16, 12 27

Mod_PageSpeed
• Open Source Apache Module
• Lots and Lots of filters
• image compression
• combine js/css
• sprite images
• domain sharding
• etc...
Friday, March 16, 12 28
I know a previous presentation was done on mod_pagespeed, so I
won’t dive too deeply into all of it’s features and configs. Trying to
optimize for performance can make dev more complicated.
packtag. pagespeed.

Three Tier
• SSL Termination
• Load Balancing
• Mod_proxy/mod_cluster
• Mod_Security
Friday, March 16, 12 29
Apache is key for 3-tier architectures. It provides significant
performance, security, and PCI compliance wins.
Big improvements in Mod_proxy_balancer with 2.4 (dynamic cluster,
session bleedoff, etc)

Building
Scalable
Clusters w/
Apache
• VIP
• heartbeat
• haproxy
• DNS LB
• simple round robin
• smart
Friday, March 16, 12 30

Questions?
Friday, March 16, 12 31

Thank you!
devon@sparkred.com
Friday, March 16, 12 32