Site Studio Performance Tuning Brian “Bex” Huff Chief Software Architect

Agenda General web-site tuning tips The complex basics High-performance architecture Tune your system, network, and servers Tuning Site Studio pages Oracle UCM specific tips

General web-site tuning tips

The complex basics

High-performance architecture

Tune your system, network, and servers

Tuning Site Studio pages

Oracle UCM specific tips

Philosophy of Performance Tuning “ We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil ” -- Donald Knuth Need a holistic view of your systems and processes Use data-flow analysis and diagrams

“ We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil ”

-- Donald Knuth

Need a holistic view of your systems and processes

Use data-flow analysis and diagrams

Why Are Network Applications Slow? Not enough CPU Needed for text processing, number crunching, cryptography Not enough memory Needed to cache temporary data for faster processing Not enough bandwidth Needed to support large data transfers, large user bases Too much latency Multiple round-trips to servers makes any single request slow Most web applications are slow because of bandwidth/latency extra CPU and memory help, but usually are not enough

Not enough CPU

Needed for text processing, number crunching, cryptography

Not enough memory

Needed to cache temporary data for faster processing

Not enough bandwidth

Needed to support large data transfers, large user bases

Too much latency

Multiple round-trips to servers makes any single request slow

Most web applications are slow because of bandwidth/latency

extra CPU and memory help, but usually are not enough

Bandwidth and Latency in Real Life 48 inches wide Bandwidth: 2,000,000 barrels per day! 800 miles long Latency: 15.8 days!

48 inches wide

Bandwidth:

2,000,000 barrels per day!

800 miles long

Latency:

15.8 days!

Bandwidth and Latency On The Web Bandwidth: Number Of Simultaneous Requests You Can Support Latency: Time Added To Each Request Due To Network and Distance A web page with 50 images means 51 HTTP requests ! High bandwidth and high latency? Pages “feel” slow 80-90% of response time spent in downloading web resources Images, CSS files, JS files, AJAX requests Source: http:// developer.yahoo.com/performance/rules.html

Bandwidth:

Number Of Simultaneous Requests You Can Support

Latency:

Time Added To Each Request Due To Network and Distance

A web page with 50 images means 51 HTTP requests !

High bandwidth and high latency? Pages “feel” slow

80-90% of response time spent in downloading web resources

Images, CSS files, JS files, AJAX requests

Source: http:// developer.yahoo.com/performance/rules.html

Analyze The Flow Of Data How does the data get to the client? How many applications are involved? Where is the bottleneck? Latency: ping , traceroute , or tracert Bandwidth: mrtg, wget , or ask your ISP/IT department CPU and Memory: top or taskmanager On the web, latency is usually a bigger problem than bandwidth Distance and network health between web client and web server Assume New York to San Francisco latency is ~100 ms if your page has 100 images, that's 10 seconds for just latency! In practice, browser caching and multiple downloads help

How does the data get to the client?

How many applications are involved?

Where is the bottleneck?

Latency: ping , traceroute , or tracert

Bandwidth: mrtg, wget , or ask your ISP/IT department

CPU and Memory: top or taskmanager

On the web, latency is usually a bigger problem than bandwidth

Distance and network health between web client and web server

Assume New York to San Francisco latency is ~100 ms

if your page has 100 images, that's 10 seconds for just latency!

In practice, browser caching and multiple downloads help

UCM Data Flow Diagram How do the actual bytes flow from client, to server, and back?

How do the actual bytes flow from client, to server, and back?

Reduce CPU Needs of Web Pages General tip: simplify your web pages so they render faster Use fewer DOM elements tables within tables within tables Put CSS at the top, and JS at the bottom CSS is rendered immediately JS must be 100% loaded before it executes Avoid complex CSS AlphaImageLoader filter CSS expressions

General tip: simplify your web pages so they render faster

Use fewer DOM elements

tables within tables within tables

Put CSS at the top, and JS at the bottom

CSS is rendered immediately

JS must be 100% loaded before it executes

Avoid complex CSS

AlphaImageLoader filter

CSS expressions

Reduce Bandwidth Needs of Web Pages General tip: use smaller web resources Compress your HTML, CSS, and JS files minimize their size in general inline GZIP compression works well Use small, web-optimized images avoid scaling them in HTML Use GET for AJAX Reduce use and complexity of cookies cookie state is sent with every request

General tip: use smaller web resources

Compress your HTML, CSS, and JS files

minimize their size in general

inline GZIP compression works well

Use small, web-optimized images

avoid scaling them in HTML

Use GET for AJAX

Reduce use and complexity of cookies

cookie state is sent with every request

Reduce Latency Of Web Pages General tip: use fewer web resources eliminate images when CSS works just as well consolidate CSS and JS into fewer files correct any 404 error for missing images Use CSS SPRITES Like Google does

General tip: use fewer web resources

eliminate images when CSS works just as well

consolidate CSS and JS into fewer files

correct any 404 error for missing images

Use CSS SPRITES

Like Google does

Reduce Latency Of Web Pages (cont.) Make fewer network requests Avoid multiple DNS lookups on the same page Minimize URL redirects Serve static files from multiple servers Browsers download 2 web resources at a time from one domain So, make more domains: img1 .company.com, img2 .company.com Use content delivery network Amazon Cloudfront : cheap, but mainly North American Akamai : expensive, but global Make content “ cacheable ”

Make fewer network requests

Avoid multiple DNS lookups on the same page

Minimize URL redirects

Serve static files from multiple servers

Browsers download 2 web resources at a time from one domain

So, make more domains: img1 .company.com, img2 .company.com

Use content delivery network

Amazon Cloudfront : cheap, but mainly North American

Akamai : expensive, but global

Make content “ cacheable ”

High-Performance Architecture Web Page Cache / Reverse Proxy Cache rendered web pages to take load off of Site Studio make sure it understands cookies and GZIP compression Contribution / consumption environment Optimize contribution for read/write Optimize consumption for read (mostly) Use a dedicated server for each Oracle UCM instance Do not share with other applications Schedule resource-intensive behavior in the off-hours Virus scanners and web spiders Backups, archives, batchloads

Web Page Cache / Reverse Proxy

Cache rendered web pages to take load off of Site Studio

make sure it understands cookies and GZIP compression

Contribution / consumption environment

Optimize contribution for read/write

Optimize consumption for read (mostly)

Use a dedicated server for each Oracle UCM instance

Do not share with other applications

Schedule resource-intensive behavior in the off-hours

Virus scanners and web spiders

Backups, archives, batchloads

High-Performance Architecture (cont.) Create a cluster in most cases with 10gr3, significantly improves performance general best practice for failover, high reliability, and uptime Minimize latency in local network keep database, LDAP, and other resources on same subnet keep filesystem speedy: attached storage, SAN, NAS Monitor CPU and Memory usage bandwidth, and latency are usually the culprits CPU and memory can be a bottleneck for heavy consumption servers If you use UCM 7.5 with Verity search, CPU can be an issue if you use database search, database CPU may be an issue

Create a cluster

in most cases with 10gr3, significantly improves performance

general best practice for failover, high reliability, and uptime

Minimize latency in local network

keep database, LDAP, and other resources on same subnet

keep filesystem speedy: attached storage, SAN, NAS

Monitor CPU and Memory usage

bandwidth, and latency are usually the culprits

CPU and memory can be a bottleneck for heavy consumption servers

If you use UCM 7.5 with Verity search, CPU can be an issue

if you use database search, database CPU may be an issue

Sample High-Performance Architecture

Make Static Content Cacheable Web caches will store static content (JS, CSS, images) Modest performance gains by keeping them in-memory Caching dynamic content is better, but harder… Add Expires or MaxAge HTTP header to content allows a cache to know how long to store content for static content, set this in the web server for dynamic content, set it in Site Studio Properly configure ETag HTTP headers on clusters If you're not using them, just remove them

Web caches will store static content (JS, CSS, images)

Modest performance gains by keeping them in-memory

Caching dynamic content is better, but harder…

Add Expires or MaxAge HTTP header to content

allows a cache to know how long to store content

for static content, set this in the web server

for dynamic content, set it in Site Studio

Properly configure ETag HTTP headers on clusters

If you're not using them, just remove them

Make Dynamic Content Cacheable Dynamic HTML pages might be different per user Multi-step forms and personalization are cache-killers Different users see different things at the same URL! Not all caches work when cookies are present Need smart caches, and smart personalization Drive customization with static CSS and JS files Read cookies from JS, use AJAX to alter pages Configure cache to ignore cookies, or Configure web server to omit cookie header Security checks can be slow How far away is your user database? Is it cached locally, or do we need to check ActiveDirectory or LDAP?

Dynamic HTML pages might be different per user

Multi-step forms and personalization are cache-killers

Different users see different things at the same URL!

Not all caches work when cookies are present

Need smart caches, and smart personalization

Drive customization with static CSS and JS files

Read cookies from JS, use AJAX to alter pages

Configure cache to ignore cookies, or

Configure web server to omit cookie header

Security checks can be slow

How far away is your user database?

Is it cached locally, or do we need to check ActiveDirectory or LDAP?

Site Studio Specifics UCM performance rule-of-thumb 20 read-only service calls per second per GHz CPU 5 read/write service calls per second per GHz CPU Problem: some Site Studio pages are multiple service calls! Each calls at least the SS_GET_PAGE service Many fragments and templates call GET_SEARCH_RESULTS Many also call DOC_INFO Not uncommon for one page to have 4 UCM service calls! Reduces performance to 5 page views per second per GHz CPU

UCM performance rule-of-thumb

20 read-only service calls per second per GHz CPU

5 read/write service calls per second per GHz CPU

Problem: some Site Studio pages are multiple service calls!

Each calls at least the SS_GET_PAGE service

Many fragments and templates call GET_SEARCH_RESULTS

Many also call DOC_INFO

Not uncommon for one page to have 4 UCM service calls!

Reduces performance to 5 page views per second per GHz CPU

UCM Specifics Minimize number of queries per page Use static lists instead of dynamic lists when possible Optimize search queries Use <matches> or <contains> instead of <substring> Note: <contains> only works in &quot;zone indexed&quot; metadata fields Avoid time stamps in queries Tune Java VM to have as much memory as you can give it JAVA_MAX_HEAP_SIZE=1024 Optimize the Oracle Database Install the OracleQueryOptimizer component Cursor_Sharing=SIMILAR optimizer_mode=first_rows

Minimize number of queries per page

Use static lists instead of dynamic lists when possible

Optimize search queries

Use <matches> or <contains> instead of <substring>

Note: <contains> only works in &quot;zone indexed&quot; metadata fields

Avoid time stamps in queries

Tune Java VM to have as much memory as you can give it

JAVA_MAX_HEAP_SIZE=1024

Optimize the Oracle Database

Install the OracleQueryOptimizer component

Cursor_Sharing=SIMILAR

optimizer_mode=first_rows

Finding Slow Pages: Passive Configure web server logs to measure page creation time Apache: use mod_log_config and the LogFormat directive IIS: use W3C extended log file format, and the &quot;Time Taken&quot; field Enable UCM System Audit Information flags requestaudit pagecreation searchquery systemdatabase Parse log files to find slow pages and slow queries

Configure web server logs to measure page creation time

Apache: use mod_log_config and the LogFormat directive

IIS: use W3C extended log file format, and the &quot;Time Taken&quot; field

Enable UCM System Audit Information flags

requestaudit

pagecreation

searchquery

systemdatabase

Parse log files to find slow pages and slow queries

Finding Slow Pages: Active Run a web spider on your system Caution: this can crash your server! Use web logs or spider logs to find slow pages Test slow pages once with auditing turned on You may be surprised how many database calls there are! Test slow pages with a load tester How many users can do this simultaneously?

Run a web spider on your system

Caution: this can crash your server!

Use web logs or spider logs to find slow pages

Test slow pages once with auditing turned on

You may be surprised how many database calls there are!

Test slow pages with a load tester

How many users can do this simultaneously?

Finding Slow Pages: Active (cont.) Bandwidth: How fast can you download a static 1MB HTML file? How many users can do this simultaneously? Latency: How fast can you download a 1k HTML file with a hundred 10k JS files? How many users can do this simultaneously? Keep excellent notes! Only way to measure performance improvement

Bandwidth:

How fast can you download a static 1MB HTML file?

How many users can do this simultaneously?

Latency:

How fast can you download a 1k HTML file with a hundred 10k JS files?

How many users can do this simultaneously?

Keep excellent notes!

Only way to measure performance improvement

Making Pages Faster Reduce number of service calls Reduce number of database calls Optimize IdocScript Minimize eval and executeService calls Instead use setValue , setResourceInclude Eliminate all IdocScript errors: these are performance drags Use cacheInclude for slow sections of pages Optimize JSP/ASP/Java code In general, same rules apply Reduce complexity, reduce number of service calls

Reduce number of service calls

Reduce number of database calls

Optimize IdocScript

Minimize eval and executeService calls

Instead use setValue , setResourceInclude

Eliminate all IdocScript errors: these are performance drags

Use cacheInclude for slow sections of pages

Optimize JSP/ASP/Java code

In general, same rules apply

Reduce complexity, reduce number of service calls

Making Pages Faster (cont.) Consolidate JS and CSS files Place them into a global fragment, or global region template Standard best-practice for maintainable code Set flags to cache pages Built-in with the custom section property &quot;ssMaxAge&quot; Passes the cache control header Watch out for cookies! Disable unnecessary JavaScript files Requires custom component On consumption environment, disable contributor JS files Most sites don't need the JS files that contain navigation

Consolidate JS and CSS files

Place them into a global fragment, or global region template

Standard best-practice for maintainable code

Set flags to cache pages

Built-in with the custom section property &quot;ssMaxAge&quot;

Passes the cache control header

Watch out for cookies!

Disable unnecessary JavaScript files

Requires custom component

On consumption environment, disable contributor JS files

Most sites don't need the JS files that contain navigation

Review General web site performance tuning is critical Data flow diagrams can highlight bottlenecks Some architectures scale better than others Size CPU and Memory appropriately Cluster your servers Include a reverse proxy Keep network resources close, and fast! Understand why specific Site Studio pages are slow Reduce number of service requests per page Focus on making “cacheable” content

General web site performance tuning is critical

Data flow diagrams can highlight bottlenecks

Some architectures scale better than others

Size CPU and Memory appropriately

Cluster your servers

Include a reverse proxy

Keep network resources close, and fast!

Understand why specific Site Studio pages are slow

Reduce number of service requests per page

Focus on making “cacheable” content

Useful Tools and References Yahoo YSlow plug-in to Firefox: a must have! http:// developer.yahoo.com/yslow / JMeter load tester http:// jakarta.apache.org/jmeter / Heritrix web crawler http:// crawler.archive.org / Varnish HTTP Accelerator http:// varnish.projects.linpro.no / Yahoo web site performance tuning best practices http:// developer.yahoo.com/performance/rules.html Oracle UCM performance tuning guide: http://download.oracle.com/docs/cd/E10316_01/cs/cs_doc_10/documentation/admin/performance_tuning_10en.pdf

Yahoo YSlow plug-in to Firefox: a must have!

http:// developer.yahoo.com/yslow /

JMeter load tester

http:// jakarta.apache.org/jmeter /

Heritrix web crawler

http:// crawler.archive.org /

Varnish HTTP Accelerator

http:// varnish.projects.linpro.no /

Yahoo web site performance tuning best practices

http:// developer.yahoo.com/performance/rules.html

Oracle UCM performance tuning guide:

http://download.oracle.com/docs/cd/E10316_01/cs/cs_doc_10/documentation/admin/performance_tuning_10en.pdf

Now Available…

My Company: http:// bezzotech.com My Blog: http:// bexhuff.com My Self: [email_address] Questions?

My Company: http:// bezzotech.com

My Blog: http:// bexhuff.com

My Self: [email_address]