7560 stress testing your people soft 8 and 8.4 online applications

357 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
357
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

7560 stress testing your people soft 8 and 8.4 online applications

  1. 1. 7560 Stress Testing Peoplesoft Online Applications Sridhar Srinivasan PeopleSoft, Inc. Leo Cai Technology Alliances Manager Mercury Interactive
  2. 2. Page 2 Agenda Stress Testing Process Flow Capacity Planning For Your Stress Test Stress Testing Techniques Performance Monitoring Stress Testing PeopleSoft Applications using Load Runner: Live Demo Questions ?
  3. 3. Page 3 Stress Testing Process Flow Identify Transactions /User Breakdown Volumetric Analysis/ Service Levels Iterative Single Slice Tests Load Runner Scripts Performance Monitoring Steps 6 - 9 Iterative Multi- Server Tests Server Sizing & Network Sizing Capacity Plan Physical Layout 1 2 3 4 5a 6 5b 8 97 Baseline Results Final Results
  4. 4. Page 4 Agenda Stress Testing Process Flow Capacity Planning For Your Stress Test Stress Testing Techniques Performance Monitoring Stress Testing PeopleSoft Applications using Load Runner: Live Demo Questions ?
  5. 5. Page 5 Capacity Planning For Your Stress Test Capacity Planning: Application Side Transaction Identification Most Used User Function Breakdown Who does What? How many times? Volumetric Analysis Peak Load Scenarios, Present and Future Response Time Requirements under Load
  6. 6. Page 6 Capacity Planning For Your Stress Test Identify Transactions /User Breakdown Volumetric Analysis/ Service Levels Iterative Single Slice Tests Load Runner Scripts Performance Monitoring Steps 6 - 9 Iterative Multi- Server Tests Server Sizing & Network Sizing Capacity Plan Physical Layout 1 2 3 4 5a 6 5b 8 97 Baseline Results Final Results
  7. 7. Page 7 Capacity Planning For Your Stress Test Transaction Identification It is key to stress test transactions that have the following attributes: Critical to the business Most Used Resource Intensive High Throughput Transactions “Use transactions that are a barometer of your business, present and future”
  8. 8. Page 8 Capacity Planning For Your Stress Test User Workload Breakdown Business process frequencies vary and so do the underlying transactions. The following attributes define User Breakdown How many transactions ? User Type Definition: Self Service Vs Manager Which User Performs what? How Often? “Size transaction Load for Peaks and not for Averages”
  9. 9. Page 9 Capacity Planning For Your Stress Test User Workload Breakdown (Contd…)  Pace your transactions for the real-world 0 20 40 60 80 100 120 1:20:00 1:24:00 1:28:00 1:32:00 1:36:00 1:40:00 1:44:00 1:48:00 1:52:00 1:56:00 2:00:00 2:04:00 2:08:00 2:12:00 2:16:00 100 User Load – No Pacing Paced Load (100 User)
  10. 10. Page 10 Capacity Planning For Your Stress Test Volumetric Analysis Based on the transactions identified and workload characteristics recorded, the following metrics will be derived: Daily transaction Volumes Average Workload per hour Load Factor Transaction Growth rate for the future “ Well-defined Metrics help build better predictive models for real-world scenarios”
  11. 11. Page 11 Capacity Planning For Your Stress Test Capacity Plan Identify Transactions /User Breakdown Volumetric Analysis/ Service Levels 1 2 “The 80/20 Rule: >80% of your volume will involve less than 20% of your transactions”
  12. 12. Page 12 Capacity Planning For Your Stress Test Identify Transactions /User Breakdown Volumetric Analysis/ Service Levels Iterative Single Slice Tests Load Runner Scripts Performance Monitoring Steps 6 - 9 Iterative Multi- Server Tests Server Sizing & Network Sizing Capacity Plan Physical Layout 1 2 3 4 5a 6 5b 8 97 Baseline Results Final Results
  13. 13. Capacity Planning For Your Stress Test Capacity Planning : Server Sizing How many JVMs?  Average JVM Supports about between 200 to 500 users based on memory footprint. How many Application Server Domains?  JOLT Handler Sizing  Per PSAPPSRV supports between 20 to 60 users based on application type  CRM – 50 Users (Average Memory 100 Mb)  HRMS – 40 Users (Average Memory 150 Mb)  FDM – 20 Users ( Average Memory 300 Mb) Page 13
  14. 14. Capacity Planning For Your Stress Test Capacity Planning : Network Sizing Define per Transaction Footprint  Browser <-> Web Server  Web Server <-> App Server  App Server <-> DB Server Record Transaction Duration Average out to find the Average Network Utilization per transaction Use Load Factor of 2.5 to 4 to define peak Network Utilization Calculate per segment Network requirements and Size Accordingly Page 14
  15. 15. Capacity Planning For Your Stress Test Capacity Planning : Size for Peaks Page 15
  16. 16. Capacity Planning For Your Stress Test Capacity Planning : Defined Layout Page 16 Web Server Farm Application Servers Web Server 1 App Server 1 App Server 2 Web Server 2 Web Server n App Server n Database Load Runner Driver Volumetric Analysis/ Service Levels Server Sizing & Network Sizing 3 4
  17. 17. Agenda Capacity Planning For Your Stress Test Stress Testing Techniques Performance Monitoring Stress Testing PeopleSoft Applications using Load Runner: Live Demo Questions ? Page 17
  18. 18. Page 18 Stress Testing Process Flow Identify Transactions /User Breakdown Volumetric Analysis/ Service Levels Iterative Single Slice Tests Load Runner Scripts Performance Monitoring Steps 6 - 9 Iterative Multi- Server Tests Server Sizing & Network Sizing Capacity Plan Physical Layout 1 2 3 4 5a 6 5b 8 97 Baseline Results Final Results
  19. 19. Stress Testing Techniques Stress Testing: Methodology Configure environments based on best practices Single Slice Testing Multiple Server Testing Page 19
  20. 20. Stress Testing Techniques Stress Testing: Configure Environments Web Server Tuning App Server Tuning Caching Network Testing for Performance Page 20
  21. 21. Page 21 Stress Testing: Web Server Tuning Web Server JOLT Application Server T U X E D O RDBMS Server SQL O D B C Services App Messaging Processor Query Processor Component Processor Bus Interlink Processor User Interface Generator Process scheduler Portal Processor Security Manager Application Engine RDBMS Instance Application Data Tables Application Meta data Portal Registry Servlet Engine PeopleSoft Servlets
  22. 22. Page 22 Stress Testing: Web Server Tuning ThreadCount Tuning N = (½ * concurrent PIA users/pagelets requests) + (½ * concurrent Portal users) + (½ * concurrent dynamic pagelet requests). WebLogic Change the entry in %WLS_HOME%/<peoplesoft web domain>/config.xml as <Server ListenPort="80" Name="PIA" Notes="The PIA server is the default server for PeopleSoft 8.40 Internet Architecture." TransactionLogFilePrefix="C:Appsbeawlserver6.1/config/peoplesoft/logs /" XMLEntityCache="XMLCacheMBean" XMLRegistry="PeopleSoft IG XML Registry"> <ExecuteQueue Name="default" ThreadCount="N" /> ... </Server>
  23. 23. Page 23 Stress Testing: Web Server Tuning ThreadCount Tuning WebSphere  Set MinimumSize and MaximumSize to be the same In %WAS_HOME%/config/server-cfg.xml, locate the following line: <webContainer xmi:id="WebContainer_1" installedWebModules="WebModuleRef_1 WebModuleRef_2 WebModuleRef_3 WebModuleRef_9 WebModuleRef_10 WebModuleRef_11"> <threadPool xmi:id="ThreadPool_1" minimumSize="n" maximumSize="N" inactivityTimeout="100" isGrowable="false"/> </webContainer>
  24. 24. Page 24 Stress Testing: Web Server Tuning JVM Heap Sizing  Set MinimumHeapSize and MaximumHeapSize the same Improved Garbage Collection Lesser Resource Allocation during run-time  Set HeapSize to minimum of 256Mb  With JDK 1.3 onwards: Garbage Collection is multi-threaded Higher JVM sizes, upto 1024M are common  JDK 1.3 Hotspot has many performance knobs  (Refer to http://java.sun.com/hotspot/performance.html)  Try –noclassgc – It might help  Make Sure Heap Size does not exceed Real Memory
  25. 25. Page 25 Stress Testing: Web Server Tuning File Descriptors Set OS FileDescriptor to 100 * ThreadCount  For Unix Systems Ulimit –n 32767 For setting it as a hardlimit set it in a startup file like rc2.d script  Windows Systems There is not an explicit parameter for the number of file descriptors. It is implicitly limited by hardware resources, mainly system memory. Make Sure you do not exceed real memory
  26. 26. Page 26 Stress Testing: Web Server Tuning TCP Timeout Settings  PeopleSoft is Socket Intensive  Once a session is completed, a Socket is marked for closure  Socket disconnects only when Socket Timeout Expires  Defaults range from 4 minutes to 11 minutes  Reduce related OS TCP Time Wait Parameter to between 30 seconds and 1 minute NT Registry Setting TcpTimedWaitDelay AIX Tcp_timewait Solaris Tcp_time_wait_interval HP Tcp_time_wait_interval
  27. 27. Page 27 Stress Testing: Web Server Tuning Servlet Reload  Defines interval for JVM to automatically reload servlets  In production environments, Servlets do not change often  Set this parameter off, so that reload does not occur WEBLOGIC In %WLS_HOME%/<peoplesoft web domain>/config.xml there is a parameter – ServletReloadCheckSecs – that dictates how often WebLogic checks whether a servlet has been modified, and if so reloads it: <WebAppComponent Name="PORTAL" ServletReloadCheckSecs="-1" Targets="PIA" URI="PORTAL" /> WebSphere %WAS_HOME%/<peoplesoft web domain>/[PORTAL/PSIGW/PSINTERLINKS]/WEB-INF/ibm-web-ext-xmi (usually this is the first line): <webappext:WebAppExtension xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:webappext="webappext.xmi" xmlns:webapplication="webapplication.xmi" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmi:id="WebAppExtension_1" reloadInterval="0" reloadingEnabled="false" fileServingEnabled="true" directoryBrowsingEnabled="true" serveServletsByClassnameEnabled="false">
  28. 28. Page 28 Stress Testing: Web Server Tuning Compression CompressResponse=true PortalCompressResponse=true PortalAcceptCompressedEncodings=true Enable Compression (Gzip, compress) for communication between browser and web server Sends compressed HTML to the compression- enabled (HTTP 1.1 enabled) browser, such as Microsoft IE 5 or above. Modify Configuration.properties
  29. 29. Page 29 Stress Testing: App Server Tuning Web Server JOLT Application Server T U X E D O RDBMS Server SQL O D B C Services App Messaging Processor Query Processor Component Processor Bus Interlink Processor User Interface Generator Process scheduler Portal Processor Security Manager Application Engine RDBMS Instance Application Data Tables Application Meta data Portal Registry Servlet Engine PeopleSoft Servlets
  30. 30. Page 30 Stress Testing: App Server Tuning Cache Setting Set server cache initially to 2 In PSAPPSRV.CFG ;------------------------------------------------------- ; EnableServerCaching - ; 0 Server File caching disabled ; 1 Server File caching limited to most used classes ; 2 Server File caching for all types EnableServerCaching=2 ;--------------------------------------------------------
  31. 31. Page 31 Stress Testing: App Server Tuning Jolt Handlers  Size based on number of users per domain  Optimally run 20 users per Handler  Per Handler Memory Footprint is about 2M In PSAPPSRV.CFG, see below: [JOLT Listener] ;====================================================================== ; Settings for JOLT Listener ;====================================================================== ;Address Note: Can be either Machine Name or IP address. ;Address Note: %PS_MACH% will be replaced with THIS machine's name Address=%PS_MACH% Port=9000 Encryption=0 Min Handlers=60 Max Handlers=80 Max Clients per Handler=20 Client Cleanup Timeout=10 Init Timeout=5 Client Connection Mode=ANY Jolt Compression Threshold=1000000
  32. 32. Page 32 Stress Testing: App Server Tuning Recycle Count  Set Recycle count between 1000 and 3000  Helps with optimal memory usage on App Server JOLT Compression  Large HTML Pages  Set Threshold down to 100000 [JOLT Listener] ;====================================================================== ; Settings for JOLT Listener ;====================================================================== ;Address Note: Can be either Machine Name or IP address. ;Address Note: %PS_MACH% will be replaced with THIS machine's name Address=%PS_MACH% Port=9000 Encryption=0 Min Handlers=40 Max Handlers=60 Max Clients per Handler=20 Client Cleanup Timeout=10 Init Timeout=5 Client Connection Mode=ANY Jolt Compression Threshold=100000
  33. 33. Page 33 Stress Testing: App Server Tuning PSAPPSRV Sizing Use # of CPUs to size # of PSAPPSRV’s n Times # of CPUs where “n” is 1,2,3.. Use pq and psr commands to size the domain. Queueing on the domain is OK as long as it clears up quickly. Do not forget PUB/SUB, PSSAMSRV, PSQRYSRV processes
  34. 34. Page 34 Stress Testing: Caching PeopleSoft Apps rely heavily on cache both at the web server and app server layer Execute couple of runs These runs will build a representative cache at the web, app server and database layer Ignore the first few runs as they are predominantly cache building exercises
  35. 35. Page 35 Stress Testing: Network Tests FTP Large files among the servers Average speeds on fast Ethernet should be around 2000KBytes/Sec Make sure adapters use the same Media Speed on either end That is, 100_Auto_neg to 100_Auto_neg or 100_full_duplex to 100_full_duplex.  Mixing of these causes network errors and hence slowdown Use Ping between servers to check for dropouts Usage : ping –l <packet size> -n <# of times> server name Time on the ping output should be <10ms (windows) or 0 ms(UNIX) Use normal packet sizes of 1544 bytes for this testing
  36. 36. Page 36 Stress Testing: Single Slice Testing Run one representative web server , app server and database server Advantages  Define baseline application characteristics – usage, scalability  Iterative runs quantify improvements in a smaller manageable environment  Refine Layout and Capacity plan with results from single slice runs  Identify all configuration issues in one set of servers and easily propagate to a multi-server environment App Server 1 Web Server 1 Database Load Runner Driver
  37. 37. Page 37 Stress Testing: Multiple Server Testing  Run tests on planned layout having multiple servers at the web and app server layer.  Identify Scalability Characteristics  Iterative runs leading to optimal performance Web Server Farm Application Servers Web Server 1 App Server 1 App Server 2 Web Server 2 Web Server n App Server n Database Load Runner Driver
  38. 38. Agenda Capacity Planning For Your Stress Test Stress Testing Techniques Performance Monitoring Stress Testing PeopleSoft Applications using Load Runner: Live Demo Questions ? Page 38
  39. 39. Page 39 Stress Testing Process Flow Identify Transactions /User Breakdown Volumetric Analysis/ Service Levels Iterative Single Slice Tests Load Runner Scripts Performance Monitoring Steps 6 - 9 Iterative Multi- Server Tests Server Sizing & Network Sizing Capacity Plan Physical Layout 1 2 3 4 5a 6 5b 8 97 Baseline Results Final Results
  40. 40. Page 40 Performance Monitoring Diagram goes here. Infrastructure Response Time Network Components Memory Application CPU Context User Time Resource Load Storage
  41. 41. Page 41 Performance Monitoring Windows 2000  Use Perflog not PerfMon – Perfmon is resource intensive  PerfLog is part of the Administrative Tools->Performance
  42. 42. Page 42 Performance Monitoring Windows 2000  Define Counters at a high level – mainly CPU, memory, Disk & Network
  43. 43. Page 43 Performance Monitoring Windows NT  Use Perflog not PerfMon – Perfmon is resource intensive  Perflog is available as part of the Resource Kit under I386logtools folder
  44. 44. Page 44 Performance Monitoring UNIX  CPU Use vmstat or Sar
  45. 45. Page 45 Performance Monitoring UNIX  Memory Use Posix PS , which allows definition of custom format e.g ps –A –o “user pid comm vsz time thcount”
  46. 46. Page 46 Performance Monitoring UNIX  Network Use netstat at the ethernet adapter level repetitively Netstat – I hme0 Connect Usage Summary
  47. 47. Page 47 Performance Monitoring: Application Use Web Server Access Logs  Extended logging provides hits, time taken per hit and bytes into and out of the web server  Overhead is minimal, as the web server performs buffered writes to the log file Sample Output
  48. 48. Page 48 Performance Monitoring: Application Web Server Access Logs: WebLogic 6.1 Example
  49. 49. Page 49 Performance Monitoring: Application Tuxedo Monitoring  Queuing What does this provide? export TUXCONFIG=/pt818/appserv/domain/PSTUXCFG export TUXDIR=/products/tuxedo/6.5-j1.2 export TUXOFFSET=/pt818/appserv/domain while [ "$i" -lt "$t2" ] do tmadmin -r < test sleep $t1 i=`expr $i + 1` done > Prog Name Queue Name # Serve Wk Queued # Queued Ave. Len Machine JSL 0095.00200 1 - 0 - bench22 PSAPPSRV APPQ 24 - 0 - bench22 WSL 00001.00020 1 - 0 - bench22 BBL 34817 1 - 0 - bench22 PSQRYSRV QRYQ 8 - 0 - bench22
  50. 50. Page 50 Performance Monitoring: Application Tuxedo Monitoring  Average Response Time of Application Server Requests Output Example SVCNAME 9a-10a 10a-11a 11a-12n 13p-14p 14p-15p TOTALS Num/Avg Num/Avg Num/Avg Num/Avg Num/Avg Num/Avg ICScript 13798/0.18 15675/0.14 27754/0.18 60/0.15 28744/0.16 86031/0.20 GetCertificate 2335/0.37 2643/0.28 4711/0.38 16/0.19 5030/0.59 14735/0.44 ICPanel 2275/4.25 2603/2.32 4637/3.31 5/2.07 4781/4.34 14301/3.61 TOTALS 18408/0.71 20921/0.43 37102/0.60 81/0.28 38555/0.81 115067/0.66 Add this to the PSAPPSRV entry in psappsrv.ubx: CLOPT="-r -e tux.out {$PSAPPSRVSpawn Server} -s@..{FS}psappsrv.lst -s@..{FS}psqcksrv.lst -sICQuery - sSqlQuery:SqlRequest -- -C {CFGFILE} -D {$Domain SettingsDomain ID} -S PSAPPSRV" This will create a tux.out in the domain dir. To generate the report: Shell out of psadmin and execute: Txrpt<tux.out
  51. 51. Page 51 Performance Monitoring: Application Response Time Monitoring  Load Runner Provides transaction boundaries  Response time samples for transactions at P&B are recorded using QuickTest Database  Use the tools that the database provides TKPROF for Oracle Event Monitor / Snapshots for DB2 Profiler for SQL Server  Do not have the database tracing on for your final stress test runs
  52. 52. Page 52 Agenda Capacity Planning For Your Stress Test Stress Testing Techniques Performance Monitoring Stress Testing PeopleSoft Applications using Load Runner: Live Demo Questions ?
  53. 53. Page 53 Stress Testing Process Flow Identify Transactions /User Breakdown Volumetric Analysis/ Service Levels Iterative Single Slice Tests Load Runner Scripts Performance Monitoring Steps 6 - 9 Iterative Multi- Server Tests Server Sizing & Network Sizing Capacity Plan Physical Layout 1 2 3 4 5a 6 5b 8 97 Baseline Results Final Results
  54. 54. Page 54 Related Sessions Tuesday: Aug 26th 3:00 pm PeopleSoft Experience with Production Environment Performance and Sizing Online Performance Tuning on PeopleSoft 8 and Beyond
  55. 55. Agenda Capacity Planning For Your Stress Test Stress Testing Techniques Performance Monitoring Stress Testing PeopleSoft Applications using Load Runner: Live Demo Questions ? Page 55
  56. 56. Page 56 Questions ?

×