Taking the high road with Enterprise Java Performance - for a helicopter view Performance of Enterprise Java Applications is a requirement and usually a challenge. Business requirements on systems can be stiff, successful systems can easily be overloaded and complex application architectures can add a burden too. Improving performance by tuning the application after it has been built seldomly renders huge improvements. By taking a step back - or even two - and regarding the application and the performance from a distance, it becomes possible to really design and architect for performance according to the ISYITF-method: it is staring you in the face. Order of magnitude improvements are attainable through logical reasoning and careful application of multi-tier architecture principles and JEE platform facilities.

1. Lucas Jellema (AMIS, The Netherlands)
THINKING THROUGH
JAVA ENTERPRISE PERFORMANCE
JavaOne 2012, San Francisco

2. OVERVIEW
• What is performance?
• Where is performance established?
• Advanced tuning methods
• ISYITF Method for Performance Improvement:
– Do not do it …
• Architecting Enterprise Java applications for improved
performance

3. PERFORMANCE DEGRADATION
65 %

4. PERFORMANCE DEGRADATION
Response
time
+ 65 %

5. PERFORMANCE
Who determines in what way the performance
Expectations
Measure objectively
Business Owner
Process Duration Business Objectives
Wait time
SLAs
Availability ≈ Performance
Disappearance Hourglass
Response time
Meaningful response

6. THE THEATER MANAGER

7. TYPICAL LAYOUT OF
ENTERPRISE JAVA APPLICATIONS
Performance ≈ Wait for Response
Web Browser
JEE Application Server
RDBMS

8. PERFORMANCE CONTRIBUTORS IN
ENTERPRISE JAVA APPLICATIONS
Performance ≈ Wait for Response
Response = Wait + Processing
Web Browser Wait = Network 1 + Response
AppServer
1
JEE Application Server Response = Wait + Processing
Wait = Network 2 + Response
Database
2
Response = Processing
RDBMS Processing = internal wait (I/O) + CPU

9. ADVANCED TUNING METHODS
• Use StringBuffer rather than plain String concatenation
• Use SAX for XML parsing instead of DOM
• Carefully select the collection class to use
– optimize hashcode() in custom Map implementations
• Use profiling tools to identify hotspots in the Java code
• Remove Assertions from production code
• Find optimal JVM switches through trial-and-error
– Focus on GC, Heap size, thread pools
• Pool resources and reuse objects rather than recreate
• Leverage concurrency features in Java to
– speed up time-to-completion through parallel execution
– prevent underuse of CPU during I/O operations
• Optimize algorithms for sorting, pattern matching, iteration,
serialization, …

10. ISYITF METHOD
FOR PERFORMANCE IMPROVEMENT
The fastest way to perform a task:
DO NOT DO IT

11. PREVENT UNNEEDED PROCESSING
if ( expensiveEvaluation & someFlag) {
...
}
if ( someFlag && expensiveEvaluation) {
...
}

12. PREVENT UNNEEDED PROCESSING
log.debug ( “Outcome step 2: ” +
resultOfExpensiveProcessing );
if (log.doDebug)
log.debug ( “Outcome step 2: ” +
resultOfExpensiveProcessing );

13. THE SHOPPING ALGORITHM

14. THE SHOPPING ALGORITHM
• shopForItem Item ( String itemName) {
driveToShop;
Item item = buyItemAtShop ( itemName);
driveHomeFromShop;
return item;
}

15. GET THIS WEEK’S GROCERIES
getGroceries Item[] ( String[] shoppingList) {
Item[] items = new Item[ shoppingList.length];
for (int i=0; i < shoppingList.length; i++) {
items[i] = shopForItem (shoppingList[i]);
}
return items;
}

16. ISYITF METHOD
FOR PERFORMANCE IMPROVEMENT
AT ALL
MORE OFTEN THAN REQUIRED
DO NOT DO IT

17. SWEET MEMORIES

18. STOCK MANAGEMENT

19. STOCK MANAGEMENT

20. DO NOT DO IT…
MORE OFTEN THAN REQUIRED
• If it has been produced before…
– Reuse before re-produce!
• If it has been shipped before…
– Reuse instead of re-ship Web Browser
• … provided it is still fresh
JEE Application Server
RDBMS

21. DO NOT DO IT…
MORE OFTEN THAN REQUIRED
• Save on network trips, context
switches and tiers to cross
• Save on ‘reproducing’ same results
-JS data (memory)
-Cookies Web Browser
- HTML 5 db
Edge Cache
Cache JEE Application Server
Cluster Fail-Over
(Session State)
Result Store
Write Behind
Client Result
Cache
Result Cache
RDBMS
Materialized
View

22. MORE PERFORMANCE REQUIRES
PARALLEL

23. MORE PERFORMANCE REQUIRES
PARALLEL

24. ISYITF METHOD
FOR PERFORMANCE IMPROVEMENT
DO NOT DO IT …
AT ALL
MORE OFTEN THAN REQUIRED
ON YOUR OWN

25. MOORE’S LAW REVISED: CORES LAW

26. DO NOT DO IT…
ON YOUR OWN
• Parallel means: multiple resources contributing to a
task at the same time
• Leverage multi-core CPU
• Achieve scalability and performance with a Cluster
• Introduce parallellism into your application
– Java: Concurrency (ThreadPools), WorkManager
– Database: parallel query and DML ,
dbms_parallel_execute, dbms_job, parallel table
functions
– Multi threading: on every tier and even across tiers
• Engage compute grid: multi node processing unit
– For example: make Hadoop make those nodes work
for you (divvy up the work and merge the results)

27. BALANCE RESOURCES TO PREVENT
CLOGGING

28. HILTON UNION SQUARE – LAST FRIDAY

29. THE POORLY PERFORMING
BUSINESS PROCESS LOAN REQUEST
Reject
request
Check Identity
Loan Evaluate
request
with Federal Fraud Analysis
Request
Service
Transfer
Money
Max 1 day Max 2 days Max 3 days Max 3 mins
98% is OK 99.99% is OK
Max 6 days + 3 mins

30. THAT DINNER IS TAKING MIGHTY
LONG…
Dinner is
Served
All are
satisfied
Max 3 hours + 3 mins

31. THE DINNER PROCESS
Dinner is John eats Mary eats Daisy Marty
Served soup soup eats soup eats soup
Daisy Marty
John eats Mary eats
eats eats
main main
main main
Daisy
John eats All are
eats satisfied
desert
desert
Max 3 hours + 3 mins

32. THE SAME DINNER PROCESS
John eats John eats
soup main
John eats
Mary eats Mary eats desert
Dinner is
soup main All are
Served satisfied
Daisy Daisy
Daisy eats
eats eats
soup
main desert
Marty
Marty
eats
eats soup
main
Max 3 hours + 3 mins

33. THE OPTIMIZED DINNER PROCESS
John eats John eats
soup main
John eats
Mary eats Mary eats desert
soup main
Dinner is All are
Served satisfied
Daisy Daisy
Daisy eats
eats eats
soup
main desert
Marty
Marty
eats
eats soup
main
Max 1 hours + 3 mins

34. FURTHER OPTIMIZATIONS
(IF NOT REFINEMENT)
John eats John eats
soup main
John eats
Mary eats Mary eats desert
soup main
Dinner is All are
Served satisfied
Daisy Daisy
Daisy eats
eats eats
soup
main desert
Marty
Marty
eats
eats soup
main
Max 53 mins

35. THE POORLY PERFORMING
BUSINESS PROCESS LOAN REQUEST
Reject
request
Check Identity
Loan Evaluate
request
with Federal Fraud Analysis
Request
Service
Transfer
Money
Max 1 day Max 2 days Max 3 days Max 3 mins
98% is OK 99.99% is OK
Max 6 days + 3 mins

36. SPEEDING UP THE BUSINESS
PROCESS LOAN REQUEST
Check Identity
with Federal
Service Reject
request
Loan
request
Fraud Analysis
Transfer
Evaluate Money
Request
Max 1 day Max 3 mins
Max 2 days
Max 3 days
Max 3 days + 3 mins

37. PANCAKE PARTY

38. BETTER PERFORMING PANCAKE
PARTY

39. PIPELINED PANCAKE PARTY: BEST
PERFORMANCE

40. PIPELINING ACROSS THE TIERS
• Database:
– Pipelined Table Functions
– Pipes and Queues
• Middle tier: Web Browser
– Multiple threads
– Queues (JMS)
• Client tier:
– AJAX “channels”
– WebSockets
RDBMS

41. THE PINNACLE OF UN-PERFORMANCE

42. FIRE AND FORGET

43. ISYITF METHOD
FOR PERFORMANCE IMPROVEMENT
DO NOT DO IT …
AT ALL
MORE OFTEN THAN REQUIRED
ON YOUR OWN
IMMEDIATELY

44. FIRE AND FORGET IN THE REAL WORLD

45. DO NOT DO IT…
IMMEDIATELY (OR SYNCHRONOUSLY)
• Submit information, file a complaint or request, start a
process, trigger interaction
– No immediate response is required!
• Asynchronous
– Start batch job (db) or worker-thread (java)
• Or fire event
– Write behind (from grid) (NO SQL)
– DML Error log

46. DO NOT DO IT…
IN BATCH (UN-IMMEDIATELY)
• Batch jobs can put peak load on a system – choking
on line applications
– Monthly reporting, quarterly prolongation, yearly
calculation,
• Batch jobs are increasingly unwanted in 24/7
– When is the “nightly” batch window?
– Data not current (enough) by today’s standards: “batch
has not run yet”
• Batch jobs used to be desirable or needed as a result
of technical limitations – that may not apply anymore
• Continuous, near real-time operations – leveraging
events, decoupling and integration architectures – are
a serious alternative

47. DON’T CALL US … WE’LL CALL YOU

48. ISYITF METHOD
FOR PERFORMANCE IMPROVEMENT
DO NOT DO IT …
AT ALL
MORE OFTEN THAN REQUIRED
ON YOUR OWN
IMMEDIATELY
AS PER REQUEST

49. DO NOT DO IT…
AS PER REQUEST
• Push has two key advantages over poll
– Most up to date information
– Reduction in network traffic and load on server
• Push is available in several flavors
– Middleware to Browser: comet, ADF Active Data
Service, WebLogic Server HTTP Channels, long poll,
WebSockets in HTML 5
– Database to Middleware: JMS/AQ, Database Query
Result Change Notification, Table Triggers, utl_http
push to servlet
– “piggy backing” – adding subscribed-to information in
regular requests
• Event driven architecture is
based on push (of events) to
mediating event handling
infrastructure

50. BITE OFF MORE THAN YOU CAN HAVE TO
CHEW

51. ISYITF METHOD
FOR PERFORMANCE IMPROVEMENT
DO NOT DO IT …
AT ALL
MORE OFTEN THAN REQUIRED
ON YOUR OWN
IMMEDIATELY
AS PER REQUEST
IN TOO BIG OR TOO SMALL STEPS

52. DO NOT DO IT…
IN TOO BIG STEPS
• Performance perception is often: time until page is
displayed and accessible (hourglass disappears)
• Web pages frequently contain much more than is
initially visible or even required
– Tree nodes, inactive tabs, invisible popups, unopened
dropdown lists
• Performance perception can be enhanced by not
initially loading what is not required
• Use AJAX based post-loading to (lazy-)fetch content in
subsequent, background round-trips
• /*+ First Rows */ vs. /*+ All Rows */

53. PENSION FUND – SEPTEMBER 2012
Employer < >
Participants
Job & Benefits

54. FETCHING THE DATA OF THE PENSION
FUND FOR THE WEB APPLICATION
select *
1 record
< >
from employers
where id = < 324>
select * 100s records
from participants
where employer_id = < 324>
select * 10s records
from benefits
where participant_id = <#>

55. REPORTING ON MANY EMPLOYERS
select *
100s records
from employers 1 query
select * 10k records
from participants 100s queries
where employer_id = <#>
select * 100k records
from benefits 10k queries
where participant_id = <#>

56. SINGLE BULK RETRIEVE REPLACING
MULTIPLE QUERIES
• Have the database bulk up the data retrieval
• Return Ref Cursor, Types and Collections or
JSON/XML
Benefits Package
select *
from employers
where id in <some set> select *
from participants
where employer_id in <some set>
select b.*
from benefits b join participants p
on (p.id = b.participant_id)
where p.employer_id in <some set>

57. DO NOT DO IT…
IN TOO BIG OR TOO SMALL STEPS
• Every network round-trip and context-switch adds
overhead
– Compare dialing the operator for every digit in the
telephone number you want to learn about
• Bundling up information to reduce the number of
round trips can be advantageous for performance
– Bring all items from the shop in one round trip
– Leverage collections and types, XML or JSON to
retrieve complex, structured object graphs from DB
– Zipping up multiple web resources in single archive
– Mashing up icons or images into a single big picture
– Piggy-back information onto requests

58. THE HARD WAY

59. A CONVOLUTED WAY

60. ISYITF METHOD
FOR PERFORMANCE IMPROVEMENT
DO NOT DO IT …
AT ALL
MORE OFTEN THAN REQUIRED
ON YOUR OWN
IMMEDIATELY
AS PER REQUEST
IN TOO BIG OR TOO SMALL STEPS
IN A CONVOLUTED WAY

61. DO NOT DO IT…
IN A CONVOLUTED WAY
• Pragmatism can overcome theoretical purity (or old
dogs’ tricks)
– With good reason and well documented
• Have client side Java Script directly access Google
Maps – by-passing the application server
• Have client directly access Database services
• Use RESTful (JSON, CSV) rather than WS* and XML
between browser client and application server
• Use POJOs (Entities) throughout the application, from
JPA to Web Tier – rather than copying/transferring
• When that suffices, use simple substring i/o parsing
big xml in DOM
• Pass plain CSV/JSON/XML from DB through Java
middle tier to Client when that is appropriate

62. BOTTLENECK / CRITICAL CHAIN

63. ISYITF METHOD
FOR PERFORMANCE IMPROVEMENT
DO NOT DO IT …
AT ALL
MORE OFTEN THAN REQUIRED
ON YOUR OWN
IMMEDIATELY
AS PER REQUEST
IN TOO BIG OR TOO SMALL STEPS
IN A CONVOLUTED WAY
IN A SUBOPTIMAL PLACE

64. BOTTLENECK / CRITICAL CHAIN
• Make sure that the bottleneck resource in your
enterprise application is not used (at peak times) for
work that is not critical or that can be outsourced
– Use auxiliary resources – outside critical chain
– Engage specialized servers, optimized for specific
tasks
– Manage resources in a strict manner
• Resource manager (DB) or Work manager (WLS)

65. DO NOT DO IT… LIVE EVENT PROCESSING
IN A SUBOPTIMAL PLACE
• The league of real time events
– Continuous stream of a multitude of tiny events with
hardly any payload, to analyze & aggregate
– Sent from physical sensors (temperature, pressure,
RFID, security gates), process sensors, Twitter,
manufacturing equipment, database triggers, web
servers, ESBs, stock trade tickers, sport statistics,
RSS, network switches, …

66. DO NOT DO IT… HTML RENDERING
IN A SUBOPTIMAL PLACE
• (X)HTML is not very compact
– Information density of HTML is very low
• DHTML, JavaScript &
AJAX allow for Web Browser
– Dynamic HTML
rendering in browser
– Dynamic, Partial
Page Refresh
• Most HTML presented JEE Application Server
by application is pre-
defined
– Dynamic data content
fetched from RDBMS
or other services is
small fraction RDBMS

67. DO NOT DO IT…
IN A SUBOPTIMAL PLACE
• Do not perform a task in a resource that is not ideally
suited for that task
– If it directly contributes to overall performance

68. DO NOT DO IT…
IN A SUBOPTIMAL PLACE
• Leverage database for what it’s good at
– Data Integrity – Primary Key /Unique Key /Foreign Key
– Aggregation
– Sorting
– Data Rule enforcement
– Bulk DML and Copy data
– Analytical Functions, Model clause, Rollup
• Specialized engines for
– Imaging and Document Processing
– Match and Search
– Speech Recognition
– Cryptography
– 3D
– Real time Data Processing
– ….

69. ISYITF METHOD
FOR PERFORMANCE IMPROVEMENT
DO NOT DO IT …
AT ALL
MORE OFTEN THAN REQUIRED
ON YOUR OWN
IMMEDIATELY
AS PER REQUEST
IN TOO BIG OR TOO SMALL STEPS
IN A CONVOLUTED WAY
IN A SUBOPTIMAL PLACE

70. ARCHITECT FOR PERFORMANCE
Web Browser
JEE Application Server
RDBMS

71. ARCHITECT(URE) FOR PERFORMANCE
Services ( Google Maps,
Translation, Conversion,
Data Feeds
-JS data (memory) Web Browser
-Cookies HTML rendering
- HTML 5 local db Validation, Calculation, Parsing
“Processing” (image, cryption, compression, SETI)
Fire&Forget
piggy- Post load
back (AJAX)
by-pass
JEE Application Server

72. ARCHITECT(URE) FOR PERFORMANCE
Services ( Google Maps,
Translation, Conversion,
Data Feeds
-JS data (memory) Web Browser
-Cookies HTML rendering
- HTML 5 local db Validation, Calculation, Parsing
“Processing” (image, cryption, compression, SETI)
Fire&Forget
push piggy- Post load
back (AJAX) by-pass
Search
Edge Cache Load balancer &
Match
Sticky ip sessions, Throttling CEP
Cache
CMS
Cluster Fail-Over JEE JEE Compute
WorkManager
(Session State) App App Parallel Threads Grid
Result Store Server Server JMS Crypto
Write Behind
Node Node Node Image
Print
Server

73. ARCHITECT(URE) FOR PERFORMANCE
by-pass
Search
Edge Cache Load balancer &
Match
Sticky ip sessions, Throttling CEP
Cache
CMS
Cluster Fail-Over JEE JEE Compute
WorkManager
(Session State) App App Parallel Threads Grid
Result Store Server Server JMS Crypto
Write Behind
Node Node Node Image
push Print
Server
Fire&Forget
Client Result Post
Cache load
AQ/JMS
HTTP Push
DB QRCN
Result Cache
Aggregation
Resource Mgt
Materialized Filter & Sort
Jobs
Data Integrity
View Bulk DML Pipelining
Parallel Processing
CBO RDBMS

74. Services ( Google Maps,
Translation, Conversion,
Data Feeds
-JS data
(memory)
Web Browser
HTML rendering
-Cookies Validation, Calculation, Parsing
- HTML 5 local db “Processing” (image, cryption, compression, SETI)
Post load
Fire&Forget
push piggy-
back (AJAX)
by-pass
Edge
Cache Load balancer
Sticky ip sessions, Throttling
CEP
Cache JEE JEE CMS
Cluster Fail-Over Compute
(Session State) App App WorkManager
Grid
Parallel Threads
Result Store Serv Serv JMS Cryp
Write Behind erNo erNo to
Node
de de Image
push Print
Server
Fire&Forget
Client Result Post
Cache load
AQ/JMS
HTTP Push
DB QRCN
Result
Cache Aggregation Resource Mgt
Materialized Filter & Sort Jobs
View Data Integrity Pipelining
Bulk DML Parallel Processing
CBO
RDBMS

75. SUMMARY
• Performance requirements are derived from
measurable and meaningful business objectives
• Unavailability equals Zero Performance
– Treat Performance and Availability elements in the
same equation
• Performance should [also] be addressed in a top-down
approach, across all tiers and constituent parts
• Some ISYITF guidelines:
– Do not do it … [AT ALL | MORE OFTEN THAN
REQUIRED | ON YOUR OWN | IMMEDIATELY | AS
PER REQUEST | IN TOO BIG OR TOO SMALL
STEPS | IN A CONVOLUTED WAY | IN A
SUBOPTIMAL PLACE ]