Portlet Performance Tuning Presented By: Jason Shao CAMPUSEAI CONSORTIUM

Presented By: Jason Shao

Approach Systems Architecture Tuning: Browsers, Portlets, DB, JVM Testing: Load & QA Monitoring Discussion Agenda

Approach

Systems Architecture

Tuning: Browsers, Portlets, DB, JVM

Testing: Load & QA

Monitoring

Discussion

Requirements Measure Performance Identify Bottlenecks Fix Bottlenecks Wash, Rinse, Repeat (2-4) Approach The Approach

Requirements

Measure Performance

Identify Bottlenecks

Fix Bottlenecks

Wash, Rinse, Repeat (2-4)

Tools & Services Architecture Service Mix Usage Patterns Scenarios (e.g. use cases) Traffic Patterns Approach – Requirements Define Requiremens – Check Variables

Tools & Services

Architecture

Service Mix

Usage Patterns

Scenarios (e.g. use cases)

Traffic Patterns

Page Load Time Responsiveness Memory Usage CPU Utilization Scalability Approach - Requirements What Are Our Targets?

Page Load Time

Responsiveness

Memory Usage

CPU Utilization

Scalability

Process Environment Tools Approach – Measurement Measurement

Process

Environment

Tools

Approach – Measure Performance “ In this business… Perception is Everything”

Mostly browser-side Client-side Bandwidth Latency Loading Resources (Images, CSS) Building Browser Model DOM, CSS, JS Approach – Measurement Perception is a Client Issue

Mostly browser-side

Client-side Bandwidth

Latency

Loading Resources (Images, CSS)

Building Browser Model

DOM, CSS, JS

Caching Connections Rendering Approach – Measurement Understand Browser Performance

Caching

Connections

Rendering

80/20 Rule Avoid premature optimization Keep a checklist DB: Indexes, Queries, Dataset Retrieval Caching Understanding Libraries & Algorithms Approach – Identify Bottlenecks Find & Fix the Bottlenecks

80/20 Rule

Avoid premature optimization

Keep a checklist

DB: Indexes, Queries, Dataset Retrieval

Caching

Understanding Libraries & Algorithms

Approach – Measure Performance “ SYSTEM ARCHITECTURE… sounds serious”

Availability Capacity Complexity Systems Architecture Balance Key Concerns

Availability

Capacity

Complexity

Systems Architecture

Where does my business logic live? What persistence strategy do I use? Are their opportunities for caching? Who does the heavy lifting? Portal? Appserver? DB? Webservice? Systems Architecture Portlet Architecture

Where does my business logic live?

What persistence strategy do I use?

Are their opportunities for caching?

Who does the heavy lifting?

Portal? Appserver? DB? Webservice?

Approach – Measure Performance “ 5. Tuning… the repeating part of wash, rinse, repeat”

Caching Connections Rendering Tuning – Browsers Browser Tuning

Caching

Connections

Rendering

Approach – Measure Performance “ Broadband and caching fix everything, right?”

Approach – Measure Performance “ How often have you been on a poor wifi connection?”

Approach – Measure Performance

Changes default browser caching Less info cached to disk Cached data not saved between sessions May not be getting caching benefit you think you are Tuning – Browsers HTTPS – We’re Safe, Right?

Changes default browser caching

Less info cached to disk

Cached data not saved between sessions

May not be getting caching benefit you think you are

Bad, Bad, Bad Typically based off i-node/file handle Apache: inode-size-timestamp IIS: Filetimestamp:ChangeNumber Kill caching in clustered environment Kill caching of dynamic content Tuning – Browsers E-Tags

Bad, Bad, Bad

Typically based off i-node/file handle

Apache: inode-size-timestamp

IIS: Filetimestamp:ChangeNumber

Kill caching in clustered environment

Kill caching of dynamic content

Good, Good, Good Some odd combinations/trickery required to work in all browsers Prevent round trips to the server Tuning – Browsers Expires

Good, Good, Good

Some odd combinations/trickery required to work in all browsers

Prevent round trips to the server

Most browsers use 2 threads per host to retrieve resources Most servers throttle max connections from a client IP Minimize separate server round-trips Tuning – Browsers Connections

Most browsers use 2 threads per host to retrieve resources

Most servers throttle max connections from a client IP

Minimize separate server round-trips

3xx status codes – mostly 301 and 302 HTTP/1.1 301 Moved Permanently add Expires headers to cache redirects Watch for trailing slashes (esp. w/Apache) Tuning – Browsers Redirects

3xx status codes – mostly 301 and 302

HTTP/1.1 301 Moved Permanently

add Expires headers to cache redirects

Watch for trailing slashes (esp. w/Apache)

Tuning – Browsers CSS Sprites

Approach – Measure Performance “ How does the browser put the page together?”

Styles at the Top CSS blocks rendering in IE Scripts at the Bottom Scripts block rendering and downloads Scripts can force BOM rebuilds Tuning – Browsers Rendering Tricks

Styles at the Top

CSS blocks rendering in IE

Scripts at the Bottom

Scripts block rendering and downloads

Scripts can force BOM rebuilds

Firebug https://addons.mozilla.org/en-US/firefox/addon/1843 Yslow https://addons.mozilla.org/en-US/firefox/addon/5369 Tuning – Browsers Handy Tools

Firebug

https://addons.mozilla.org/en-US/firefox/addon/1843

Yslow

https://addons.mozilla.org/en-US/firefox/addon/5369

Approach – Measure Performance “ Don’t forget: Portlet s ”

Portlet Caching JSR-168 specification Action vs. Render Requests Tuning – Portlets Portlets

Portlet Caching

JSR-168 specification

Action vs. Render Requests

Initialization Data Lookups Object Creation Shared State Tuning – Portlets Portlets – What Gets Cached?

Initialization

Data Lookups

Object Creation

Shared State

Initialization Data can be cached: Globally Per-User Per-Login Per-Session Per-Transaction Tuning – Portlets Initialization

Initialization Data can be cached:

Globally

Per-User

Per-Login

Per-Session

Per-Transaction

Approach – Measure Performance “ Tuning is the DBA’s Job”

Approach – Measure Performance

Indexing Caching Connection Pooling Table Size Query Tuning Tuning – Database Be Kind to your Database

Indexing

Caching

Connection Pooling

Table Size

Query Tuning

Hibernate, Toplink, ORM bugs indexes not created Indexes poorly used Differences between databases Remember: Primary/Foreign keys Tuning – Database Index, Index, Index

Hibernate, Toplink, ORM bugs

indexes not created

Indexes poorly used

Differences between databases

Remember: Primary/Foreign keys

Application Caching ORM Caching Database-side caching & tuning e.g. MySQL QueryCaching Tuning – Database Caching

Application Caching

ORM Caching

Database-side caching & tuning

e.g. MySQL QueryCaching

Audit Your Configuration MySQL: url@javax.sql.BaseDataSource=jdbc:mysql:// localhost:3306/sakai? useUnicode=true&characterEncoding=UTF -8&useServerPrepStmts=false&cachePrepSt mts=true&prepStmtCacheSize=4096&prep StmtCacheSqlLimit=4096 Tuning – Database Configuration

Audit Your Configuration

MySQL:

url@javax.sql.BaseDataSource=jdbc:mysql:// localhost:3306/sakai? useUnicode=true&characterEncoding=UTF -8&useServerPrepStmts=false&cachePrepSt mts=true&prepStmtCacheSize=4096&prep StmtCacheSqlLimit=4096

Approach – Measure Performance “ JVM Tuning ~ Black Magic”

Approach – Measure Performance

JVM Memory Model JVM Versions Heap Settings GC Tuning Tuning – JVM Key Factors

JVM Memory Model

JVM Versions

Heap Settings

GC Tuning

JDK 1.4 (Hotspot) JDK 1.5 JDK 1.5 (64-bit) JDK 6 Tuning – JVM Sun JVM Versions

JDK 1.4 (Hotspot)

JDK 1.5

JDK 1.5 (64-bit)

JDK 6

Tuning – JVM Memory Model

-d64 –Xmx3000m –Xms3000m –Xmn1g -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=15 -XX:MaxPermSize=256m -XX:PermSize=64m Tuning – JVM JVM Options: Heap & GC

-d64 –Xmx3000m –Xms3000m –Xmn1g -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=15 -XX:MaxPermSize=256m -XX:PermSize=64m

http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp http://java.sun.com/performance/reference/whitepapers/tuning.html http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html Tuning – JVM Resources

http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp

http://java.sun.com/performance/reference/whitepapers/tuning.html

http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html

-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintAdaptiveSizePolicy -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime Tuning – JVM Verbose GC

-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintAdaptiveSizePolicy -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime

Profilers Yourkit GC Visualizers http://java.sun.com/performance/jvmstat/visualgc.html Tuning – JVM More Handy Tools

Profilers

Yourkit

GC Visualizers

http://java.sun.com/performance/jvmstat/visualgc.html

Approach – Measure Performance “ Is this a test?”

Load Testing Soak Testing Capacity Testing Throughput Testing QA Testing Testing

Load Testing

Soak Testing

Capacity Testing

Throughput Testing

QA

Script Environment Conditions Reports Testing Load Testing

Script

Environment

Conditions

Reports

Develop a comparison point Base workload Typical workload Minimum workload Testing Baselining

Develop a comparison point

Base workload

Typical workload

Minimum workload

Soak: What happens over time? Capacity: What happens with xxx users? Or xxxx users? Or xxxxx users? Throughput: How much can I push? Testing Probe Seams

Soak: What happens over time?

Capacity: What happens with xxx users?

Or xxxx users?

Or xxxxx users?

Throughput: How much can I push?

What’s next year’s anticipated enrollment? Usage? Scheduling? Have we had problems in the past? Testing Refine Test Scenarios

What’s next year’s anticipated enrollment?

Usage?

Scheduling?

Have we had problems in the past?

What do you need to setup? What do you need to teardown? How do you start? Stop? Watch? Testing Automate, Automate, Automate

What do you need to setup?

What do you need to teardown?

How do you start?

Stop?

Watch?

Small changes can have large impacts If you thought it would cause problems, would you ever put it in? Fixing problems before they become problems helps us sleep at night Testing Test Regularly

Small changes can have large impacts

If you thought it would cause problems, would you ever put it in?

Fixing problems before they become problems helps us sleep at night

Load Generators Jmeter - http://jakarta.apache.org/jmeter/ Mercury Loadrunner Functional Test Tools Wiki & Documentation Selenium - http://selenium.openqa.org/ Testing Tools

Load Generators

Jmeter - http://jakarta.apache.org/jmeter/

Mercury Loadrunner

Functional Test Tools

Wiki & Documentation

Selenium - http://selenium.openqa.org/

Approach – Measure Performance “ I’m done, right?”

Portals are dynamic environments Content is always changing Users can modify the environment Lots of interactions Monitoring Monitoring – Know When It Breaks

Portals are dynamic environments

Content is always changing

Users can modify the environment

Lots of interactions

When writing portlets, think about errors Logging is good: Never eat an exception Catch (Exception e) {} Sometimes calling home is better Monitoring Error Reporting

When writing portlets, think about errors

Logging is good: Never eat an exception

Catch (Exception e) {}

Sometimes calling home is better

Approach – Measure Performance Automate, Automate, Automate

Approach – Measure Performance

Notification Nagios Log Analysis Splunk grep Monitoring Monitoring Tools

Notification

Nagios

Log Analysis

Splunk

grep

Ken & Erin for assistance & review Dan Mindler - everything I know about the JVM I learned from Dan Chuck Hedrick - lots of experimentation at Rutgers with tweaking & tuning Steve Souders @ Yahoo – client tuning Monitoring Acknowledgements

Ken & Erin for assistance & review

Dan Mindler - everything I know about the JVM I learned from Dan

Chuck Hedrick - lots of experimentation at Rutgers with tweaking & tuning

Steve Souders @ Yahoo – client tuning

Approach – Measure Performance Let’s Talk