• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Planning Optimal Lotus Quickr services for Portal (J2EE) Deployments
 

Planning Optimal Lotus Quickr services for Portal (J2EE) Deployments

on

  • 6,189 views

As per the Quickr Wiki ( http://www-10.lotus.com/ldd/lqwiki.nsf/dx/20052009045545WEBCGW.htm ):...

As per the Quickr Wiki ( http://www-10.lotus.com/ldd/lqwiki.nsf/dx/20052009045545WEBCGW.htm ):

"This document contains the presentation from Quickr masterclass covering planning optimal deployments – crawl/walk/run.

Discussing simplistic deployment architectures which can be linearily scaled over time (e.g. from POC to simple-non-clustered to clustered)

Sharing of key tips/recommendations from SVT and Perf - so as to help avoid expensive crit-sits in the field

Tuning for performance, stability and reliability"

Please note, I do not claim any ownership of this presentation, just am uploading to allow sharing via the Quickr Blog. Any questions/comments/issues, just let me know!

Statistics

Views

Total Views
6,189
Views on SlideShare
5,980
Embed Views
209

Actions

Likes
3
Downloads
207
Comments
0

8 Embeds 209

http://quickrblog.com 125
http://www.slideshare.net 36
http://planetlotus.org 24
http://www.filug.com 14
http://www.quickrblog.com 7
resource://brief-content 1
http://feeds.feedburner.com 1
https://www.filug.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as OpenOffice

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Planning Optimal Lotus Quickr services for Portal (J2EE) Deployments Planning Optimal Lotus Quickr services for Portal (J2EE) Deployments Presentation Transcript

    • Planning Optimal Quickr Deployments Presented by: Norman McBride SVT Reliability, Dublin Software Lab
    • Agenda For This Session
      • Planning and Preparing for Your Deployment.
      • Installation Options and Deployment Topologies
      • Performance and Tuning Considerations.
      • The Basis for Our Recommendations?
      • Basic Troubleshooting Techniques.
    • Planning and Preparing for your Deployment.
    • Planning is an essential step: If you Fail to Plan then Plan to Fail !!
    • Planning should be based on the intended short term use of the deployment and the possible future long term use of the system to avoid future performance and reliability issues and possible re-deployment at a later stage.
    • Some items to consider during Planning
      • The purpose of the system?
      • Hardware and Software considerations.
      • What is the intended user population size? How many concurrent users will be 'on' the system?
      • Consider what the intended user population will primarily use the system for, will the primary use be document centric or Wiki Centric or a mix.
      • What are the criteria for availability?
      • What is the criteria for user response times?
      • The Number of “TeamPlaces” that will potentially be on the system.
      • Place Design and Template design.
      • Version 8.1.1 has new functionality such as ECM and new connectors integration. Plan to upgrade from previous versions if doing a new deployment. 8.1.1 is the current recommended level.
    • Machine Allocation for Your Deployment.
      • In an ideal situation each component/node in a deployment should reside on its own machine.
      • During SVT Reliability testing we have each component installed on a single high spec machine.
      • If co-location of some components is unavoidable it is preferable that at a minimum the Database server should be installed on separate hardware.
      • Co-location of components, whilst feasible is not advisable in a production environment.
      • Separate systems helps to avoid resource contention.
      • Co-located servers will compete for resources.
    • Minimum Hardware requirements
      • We would recommend at a minimum hardware with the following specifications:
        • Absolute Minimum 4 GB free disk space for installation for Lotus Quickr.
        • Under minimal load, Lotus Quickr can function with 2GB of RAM. However, 4GB is an optimal starting point for RAM in a production environment.
        • CD-ROM/DVD-ROM drive
        • Processor: CPU speeds of late-model, mid-range to high-end servers are recommended. Pentium 1 GHz or equivalent at a minimum. Production environments should consider the Pentium 4 processor at 2.5 GHz or higher. SVT test Systems used Dual-Core Intel Xeon processors.
        • Quickr can take advantage of multi-CPU systems therefore if possible it should be deployed on systems with these processors.
    • Optimal Hardware as used by SVT in our Test Environment
    • Software Requirements
      • The following Operating Systems are supported:
        • Microsoft Windows 2003 Enterprise Edition and Standard Edition with SP2
        • Microsoft Windows Server 2003 R2 on 32-bit only
        • Red Hat Enterprise Linux (RHEL) Enterprise Server (ES), Advanced Server (AS).
        • We would advise where possible that a 'dedicated' OS is provided for each Lotus Quickr component.
    • Detailed Hardware and Software requirements can be found at the url below, it is advisable to check these in detail during the planning stage: http://www-01.ibm.com/support/docview.wss?rs=3264&uid=swg27014510
    • Planning Considerations
      • Considerations for building your environment
      • Web server considerations
      • Database considerations
      • Security considerations
      • Clustering considerations
      • Other considerations
    • Considerations when building your Environment You should consider the following when building a staging/integration or a production environment
        • Planning for integration and staging environments
                    • Environment should mimic the intended production environment.
        • Planning for a production environment:
                    • Environment should ideally be designed to provide for redundancy and failover.
                    • Should be designed with criteria considerations in mind.
    • Web server considerations
      • Do you have an existing Web server that you want to use?
      Yes: Ensure existing HTTP server is at supported level. No: A Web server is not required by Lotus Quickr, However an external web server is recommended.
    • Database Planning
      • When installing Lotus Quickr, you can choose one of the following general approaches for your database software:
          • Install and use Lotus Quickr with the default OOTB DB2 database:
              • This is not recommended for a high load production environment.
          • Install with the default DB and then transfer the database information to a remote DB:
              • Allows you to begin working with Lotus Quickr without additional DB deployment or if you wish to wait before integrating the remote DB into your environment.
      • By default, IBM Lotus Quickr 8.1 installs and uses IBM DB2 Universal Database Enterprise Server Edition version 9.1fp4a. Installing with DB2 lets you quickly get Lotus Quickr installed and running in a production-level environment.
          • When you plan to transfer data to a remote database, perform the database transfer before you use Quickr extensively.
          • Large amounts of data in the databases can cause the database transfer to fail if your Java heap size is not large enough.
          • Apart from the potential for transfer failure, large databases can take a considerable amount of time to transfer.
          • Perform the database transfer as soon as it is practical to avoid bottleneck problems in a production environment.
    • SVT – DB Experiences
      • During SVT Reliability testing we have always installed our database software on a remote system and performed our reliability testing on deployments with remote databases. We have found this to be the optimal deployment method.
    • LDAP/User Registry considerations
      • It is recommended to always have security enabled otherwise certain applications may not function properly.
      • After a user has been authenticated, the system can determine if that user is authorised to access the resources that are requested. Security is resource driven and users need to be allowed permission to access specific resources on the system.
      • A user registry holds user account information, such as a user ID and password that can be accessed during authentication. IBM WebSphere Application Server and IBM Lotus Quickr support multiple types of user registries, the most commonly used being:
                      • Lightweight Directory Access Protocol (LDAP) user registry.
    • Other Common Security Considerations
      • Do you intend to use SSO(single sign-on)/why would I want to use it?
              • Single sign-on is used to provide a secure method of authenticating a user one time within an environment and using that single authentication (for the duration of the session) as a basis for access to other applications and systems. For example if Lotus Quickr is to be integrated with an ECM backend system SSO can be used to provide a simpler user experience removing the necessity for multiple log on on each system.
      • Do you intend to use SSL?(secure socket layer)/Why would I want to use it?
              • Configuring Lotus Quickr for SSL adds security to the client-portal exchange. It encrypts all traffic between the client browser and the server, so that no one can "eavesdrop" on the information that is exchanged over the network between the client browser and Lotus Quickr.
    • General Clustering Considerations
      • Clustering is the concept of distributing the workload across multiple machines.
      • Why use clustering? - For redundancy, scalability, performance, failover, and simpler maintenance.
      • A Lotus Quickr cluster is essentially a collection of multiple Lotus Quickr servers that are identically configured.
      • The deployment manager node should be installed separately before the cluster is configured.
      • You must install and configure Lotus Quickr separately on each node. In addition, there can only be one Lotus Quickr cluster per cell.
      • Do not use spaces in the cluster name or the cluster member name.
      • For the deployment manager and each Lotus Quickr node to be in the cluster, verify that each system clock is set to within 5 minutes of the others or the addNode command will fail.
      • In a clustered environment the 'Search Indexes' need to be located in a shared location that is used by each of the nodes.
    • Operating System Preparation
      • Windows
      • Verify the Operating System & service pack version and upgrade as necessary.
      • Check for a configured, fully-qualified host name.
      • Each System in the deployment should have valid hostname that can resolve to a valid static IP address.
      • Before installing Lotus Quickr, check that the system logon user ID you will use during installation has the necessary permissions and rights.
      • Verify that there is adequate space on the destination drive.
      • Quickr creates a DB user during install, this users password must be consistent with the security policies that are in place on the machine where installation is taking place. .
      • Disable any firewall products.
      • Ensure that the time and date settings on each system that will be used in the deployment are correct and consistent across all machines in the deployment. Also ensure that the regional settings are the same.
      • To avoid problems related to excessively long paths names, consider the following recommendations when installing -
            • Use a short installation path. For example, use C:Quickr instead of C:Program FilesIBMQuickr.
            • Keep node names as short as possible.
    • Operating System Preparation
      • Linux
      • Verify the Operating System & service pack version and upgrade as necessary.
      • Check for a configured, fully-qualified host name.
      • Each System in the deployment should have valid hostname that can resolve to a valid static IP address.
      • Before installing Lotus Quickr, check that the user ID you will use during installation has the necessary permissions and rights.
      • Verify that there is adequate space on the destination drive.
      • Quickr creates a DB user during install, this users password must be consistent with the security policies that are in place on the machine where installation is taking place.
      • Disable any firewall products.
      • Ensure that the time and date settings on each system that will be used in the deployment are correct and consistent across all machines in the deployment. Also ensure that the regional settings are the same.
    • Operating System Preparation Linux cont'd .
      • Planning Linux disk space:
      • You should carefully plan in advance for the size of your file system in order to avoid related problems. The following disk space is the minimum required for each directory:
              • / Must have 1.5 GB or more available.
              • /opt Must have 4 GB or more available.
              • /home Must have 700 MB or more available.
              • /tmp Must have 600 MB or more available.
      • The default install directory for Lotus Quickr on Linux is '/opt'
      • The above are minimum recommendations, depending on your particular scenario you may require more space. Carefully consider the space allocations on your DB2 server as the database size will grow as content is added.
      • Setting the File Descriptor Limit:
      • Prior to installing Lotus Quickr, set the file descriptor limit to 40000. You can set this by issuing the command 'ulimit -n 40000'. You can check this setting by issuing the command 'ulimit -a'.
      • Installation program considerations:
      • To use the graphical user interface that is provided by the installation program you will need to install and configure X server on Linux (for example X-Windows or GNOME or KDE).
    • Operating System Preparation Linux cont'd.
      • Enabling Document Conversion Services:
      • To use document conversion services make sure that you have the appropriate package installed for your system. Enabling document conversion requires some extra steps on the Linux Platform.
      • For more detailed information on enabling Document conversion services see the infocenter at the following url:
              • http://publib.boulder.ibm.com/infocenter/lqkrhelp/v8r0/index.jsp?topic=/com.ibm.lotus.quickr.admin.wpv81.doc/wpf/dcs_info.html
    • Section Summary Planning and Preparing for your Deployment. - What have we covered in this Section?
        • The importance of Planning BEFORE beginning to deploy.
        • Items to consider during Planning.
        • Machine Allocation considerations.
        • Minimum Hardware requirements.
        • Optimal Hardware as used by SVT in our Test Environment.
        • Software Requirements.
        • Specific Planning considerations:
                        • The type of environment: Staging or Production
                        • Web Server considerations.
                        • Database considerations
                        • Security Considerations
                        • Clustering Considerations.
        • Operating System Preparation:
                        • Windows & Linux.
    • Installation Options and Deployment Topologies
    • Installation Options
    • The Express Install
        • The installation process places all components on one node. With previous products, had to install individual components and then the 'main' product.
        • This method of install has the fastest install path.
        • This install type is suitable for evaluation or demonstration purposes or light departmental usage.
        • Websphere Application Server and WebSphere Portal server and DB2 database server are installed on the same system.
    • Advanced Single Server
      • This is referred to as the 'Typical' install option.
      • Use this option if you plan to expand the deployment into a cluster at some point.
      • Use this option if you want to configure the DB user ID.
      • Use this option to install a Single Server configuration with remote DB and remote Web Server.
      • This install is similar to the express install in that all the components are installed on the same machine.
    • Single Server With Remote HTTP/DB
      • Install using the 'typical' option.
      • Provides greater capacity and performance as the DB and web server are remotely located.
      • Quickr uses jdbc type 4 drivers, These are direct to database drivers that allow the local db2 to be disabled after db transfer to a remote system.
      • Can be used where there is a plan to expand into a cluster at a later point.
      • May be suitable for applications where failover and redundancy are not a requirement.
      • May be suitable for small to mid range deployments where clustering is not required yet.
    • Typical Two Node Cluster Configuration with remote HTTP and Remote DB
      • A Clustered configuration is recommended for production environments.
      • Install using the “Advanced Enterprise Cluster” option
      • A Cluster provides Load Balancing between the nodes.
      • Fail-over is provided for. If one server fails during a user session then the user session is not lost but 'fails-over' to the other node.
      • Redundancy is also provided in a clustered topology. This allows one node to be stopped for maintenance while the other node continues to provide service.
      • Performance is increased and the system can handle large loads better.
    • Cluster Deployment - Typical SVT Process
      • The typical (SVT reliability) process for installing a two node cluster with a remote DB2 server and remote web server is as follows:
      • Install and prepare the deployment manager.
      • Install the primary node.
      • Install the secondary node.
      • Install the DB server on a remote machine.
      • Transfer the Database to the remote Database server.
      • Enable Security against an external Directory.
      • Install and configure the remote http server and generate plugin file for http server.
      • Configure Search for the cluster.
      • System Tuning.
      • System Backup
    • Additional Deployment Options
      • IBM Lotus Quickr supports integration with other offerings within the Lotus Portfolio - such as Sametime and Connections.
      • IBM Lotus Quickr also supports a number of Third Party options such as:
              • Netegrity SiteMinder 5.5 and 6.0
              • Tivoli Access Manager (TAM) 5.1 and 6.0
              • Forward and reverse proxy servers via:
                      • Apache HTTP Server 2.0.54
              • WebSphere Edge Server 6.0
    • Section Summary Install Options and Deployment Topologies - What have we covered in this Section?
        • Deployment topologies:
                        • Express install.
                        • Advanced Single Server(Typical)
                        • Advanced Single Server with remote HTTP and remote DB.
                        • Typical SVT cluster deployment Process.
                        • Advanced Enterprise Cluster with remote HTTP and remote DB.
                        • Additional Deployment Options.
    • Performance and Tuning Considerations
    • General Performance Recommendations
      • For optimal performance a Cluster Configuration is recommended:
            • A cluster will provides Load Balancing.
            • There is redundancy in a Clustered configuration.
            • Fail over is catered for allowing for session preservation in the event of a node failing.
      • In Terms of Hardware configuration:
            • The ideal configuration for Lotus Quickr is at least Dual Processor systems.
            • The Database server should have a highly efficient Disk System.
            • This should be a dedicated internal disk array or a properly configured external disk array ideally connected via fiber.
      • Platform Selection:
            • Memory management is generally superior on Linux.
                • This allows for a larger Java heap size thus supporting more concurrent users.
            • I/O management is more efficient on Linux.
    • Performance Considerations - Installation
      • It is recommended to upgrade to the 8.1.1 build as significant performance improvements have been made over previous releases:
              • Reduction in the load on the database and directory servers.
              • Improved response times for specific operations.
              • Better Cluster scalability.
              • ECM integration.
              • Better connections integration.
      • Ensure that all applicable Quickr fixes available on fix central are applied. (See the references section at the end of the presentation for location information)
      • It is recommended to 'break out' the Lotus Quickr DB domains into separate databases. Four separate databases is considered to be optimum for basic Quickr deployments running under production level user loads:
    • Performance Recommendation for a Clustered Configuration Lotus Quickr Servers 2 Processor/4 GB Memory Database Server 4 processor 8 GB Memory Dedicated Storage or disk array Directory Server 2 processor 4 GB Memory Disk array HTTP Servers 2 Processor 4 GB Memory Load Balancer 2 Processor 4 GB Memory Cell Deployment Mgr 2 Processor 4 GB Memory
    • System Specifications in SVT Reliability
    • Performance Considerations
      • It is essential to perform regular maintenance on databases to ensure that they perform well.
              • Use the automatic maintenance function in DB2 to perform regular maintenance.
      • The key operations to automate in DB2 are:
              • Defragmentation (REORG)
              • Optimizing Access (RUNSTATS)
    • Performance Considerations (cont'd.)
      • In a cluster the Search Indexes are stored in a share and mounted as a mapped drive on each node.
      • The search indexes should ideally be placed on High speed disk such as FAStT storage server.
      • Text Indexing should be run frequently
      • The default indexing time is every 30 minutes:
                • 'jcr.textsearch.indexmaintenance.interval = 30'
      • This interval can be modified by altering the property shown above in the file icm.properties in the directory QuickrPortalServerjcrlibcomibmicm
      • The default time should be acceptable for most installations.
      • In a cluster environment only one node needs to run indexing.
      • The indexer cannot be scheduled to run at specific times, only at intervals.
    • Usability Performance Tips
      • You should encourage users to exploit the “Favourites” feature for quick access to Teamspaces.
      • Encourage users to bookmark their most frequently used pages.
      • Manage your Teamspaces:
        • A lot of Public Teamspaces can make “My Places” awkward to use.
        • Does a Teamspace have to be made Public?
        • Ensure there is a policy for inactive places.
        • Ensure there is a monitoring policy in place for Place sizes.
        • Do not create a Place for everything, consider who should be allowed to create places.
      • Manage content distribution efficiently, consider the folder structure that you will use in libraries. For example on your OS file system would you place 15,000 documents into a single folder?
      • Consider the impact of theme customisation: seemingly small theme changes can have a significant performance impact.
      • Create templates for your users based on requirements.
      • Consider page design when creating custom places or templates.
        • You should ideally place one component on each page:
              • So for example have one page for a library, another for a wiki.
              • Do not place a group of components on the same page, for example a library, a wiki, a calendar and a blog all on the same page.
              • This is an important consideration for the most commonly 'hit' pages.
              • It is preferable to have more pages in a place than to have multiple components on each page.
      • There is a nightly sensor task that runs to check the size of each Teamspace.
      • Upgrade to the latest applicable WAS FP level, check here for more information on this:
      • http://www-01.ibm.com/support/docview.wss?rs=3264&context=SSVGKL&context=SSKU2L&uid=swg27013353&loc=en_US&cs=UTF-8&lang=en
      Usability Performance Tips
    • Tuning
      • Much of the tuning that was developed and refined during testing is now implemented by default.
      • Our tuning is based upon measurements collected during testing in our controlled lab environment.
      • Customers should firstly apply the tuning steps as outlined in the Tuning Guide which is available from the following url: http://www-01.ibm.com/support/docview.wss?rs=3264&uid=swg27010632
      • Our SVT tuning is particular to our environment and may or may not be suitable for yours.
      • It is important to remember that tuning and capacity are affected by a wide variety of factors including the workload on the system and the overall environment.
      • Our tuning steps may not be applicable in your environment and we are not recommending them. The purpose is to make users aware of the parameters used in our configuration.
      • When tuning your systems it is important to establish a baseline and monitor performance metrics to determine if any parameters need to be changed. When a change is made performance should be monitored to establish the effectiveness of the change.
      • SVT apply the following tuning to our systems before each test run:
          • Portal and Application Server Tuning.
          • DB Tuning.
          • HTTP server Tuning.
          • Operating System and Network Tuning
    • Tuning (cont'd.)
      • Adjusting the JVM Heap Size
      • The value of the JVM heap size is directly related to the amount of Physical memory on the system. The JVM heap size should never be set higher than the amount of physical memory on the system.
      • When setting the JVM heap size for a server the following points should be kept in mind:
        • Ensure that the system has adequate main memory for all the processes running on the system and for the Operating system.
        • After making any changes to the Heap size it is advisable to monitor the system to see that paging is not occurring.
        • 32 bit Operating Systems have an address space limit of 4Gbytes.
        • In addition to this limit some Operating Systems restrict the size of processes to be even less than the limit. Greater detail on how this applies to Windows systems can be found here: http://support.microsoft.com/default.aspx?scid=kb;en-us;555223
        • On our SVT deployments we have enabled the '3G' switch to circumvent this problem. This switch can be enabled by the addition of the /3G flag in the boot.ini on a Windows system.
        • See here for more info on the 3G switch: http://technet.microsoft.com/en-us/library/bb124810.aspx
        • If the JVM process grows to a size larger than that allowed by the Operating System then the process can terminate suddenly with unexpected consequences.
    • Tuning (cont'd.)
      • Adjusting the JVM Heap Size - How to set?
      • On a Single server, login @ https://host.domain:10039/admin
      • On a Cluster, login to the deployment manager admin console @ https://dmgrhostname.domain:9043/ibm/console/logon.jsp
        • Select Servers, Application Servers on the left hand side and then select WebSphere_Portal.
        • Server Infrastructure select Java and Process Management and then select Process Definition
        • Now under the section heading "Additional Properties" click on " Java Virtual Machine".
        • Modify the value "Maximum Heap Size" as required based on the table below:
    • Tuning (cont'd.)
      • Thread Pool Tuning
      • The values that we use here are those that we have used in our reliability test runs. We have achieved consistent results with these values in our environment within the constraints of our particular scenario.
      • Monitor the results of any adjustments to these values and adjust the values based on observations such as increasing WebContainer values if the servlet threads are busy most of the time.
      • Adjusting Thread Pool sizes - How to set?
      • On a Cluster, login to the deployment manager admin console @ https://dmgrhostname.domain:9043/ibm/console/logon.jsp
        • Select Servers, Application Servers on the left hand side and then select WebSphere_Portal.
        • Under the section titled "Additional Properties" select the section "Thread Pools".
        • Modify each of the threadpool settings so that the maximum value = Minimum value as shown in the screenshot below. If there is no "Thread inactivity timeout" a value will be required, use 3500 for this value.
    • Tuning (cont'd.)
      • Transport Buffer Size Tuning
      • Adjusting the Transport buffer sizes can lead to improved data replication performance in a clustered environment.
      • Adjusting Transport Buffer sizes - How to set?
      • On a Cluster, login to the deployment manager admin console @ https://dmgrhostname.domain:9043/ibm/console/logon.jsp
      • 1. For each Portal server:
        • Select Servers, Application Servers on the left hand side and then select WebSphere_Portal.
        • Under Additional Properties select Core Group Service and then Transport Buffer Size.
        • 2. For each Node agent:
        • Navigate to System administration > Node agents > node_agent_name > Core group service
        • 3. For the Deployment Manager:
        • Navigate to System administration > Deployment Manager > Core group service
    • Tuning (cont'd.)
      • DataSource Connection Pool Tuning
      • The Lotus Quickr Database is used to hold information for the various applications in Lotus Quickr. In our SVT Reliability environment we have found the settings presented here to provide optimum results based on our scenario in our environment.
      • Adjusting Connection Pool Settings - How to set?
      • On a Cluster, login to the deployment manager admin console @ https://dmgrhostname.domain:9043/ibm/console/logon.jsp
        • Select Servers, Application Servers on the left hand side and then select WebSphere_Portal.
        • Select Resources, JDBC providers.
        • Now select wpdbJDBC_db2 from the resulting screen.
        • Now select Data sources from the resulting screen.
        • Now select wpdbDS from the resulting screen.
        • Now select Connection pool properties from the resulting screen.
        • On this screen you need to change the Maximum connections to 120.
        • Select OK and then save the changes.
    • Tuning (cont'd.)
      • Database Tuning
      • Lotus Quickr uses databases for its core functionality. In our controlled environment we used a single database for all content.
      • As mentioned earlier we would recommend using a remote database server and also to 'break out' the Lotus Quickr DB domains into separate databases in a production environment.
      • We used the configuration specified in the Infocenter for database setup @ http://publib.boulder.ibm.com/infocenter/lqkrhelp/v8r0/topic/com.ibm.lotus.quickr.admin.wpv81.doc/wpf/setup_db2.html
      • While the Quickr database is configured for high capacity performance it is possible that various tuning adjustments will be necessary from time to time. These tuning adjustments are based on the amount of database traffic and the size of the table populations
      • In addition to the to the recommendations in the InfoCenter we adjusted the following settings. These settings can all be applied from a DB2 CLP:
          • db2set DB2_ASYNC_IO_MAXFILOP=512 ( to set the DB2 registry variable).
          • db2set DB2_USE_ALTERNATE_PAGE_CLEANING=on
          • db2set DB2_INLIST_TO_NLJN=YES
          • db2set DB2_EVALUNCOMMITTED=YES
          • db2 "update db cfg for wpsdb using MAXFILOP 512" ( This adjustment allows for additional database files to be allowed opened)
          • db2 "update dbm cfg using JAVA_HEAP_SZ 4096" ( Adjust the heap size for the DB2 JVM)
    • Tuning (cont'd.)
      • Database Tuning
      • We also made the following Bufferpool adjustments, again please note these are entirely subjective and dependent on the volume of data in the database and the workload on the server, these are the settings that we used in our environment.
      • As the volume of content in the database increases it may be generally be necessary to increase the size of the bufferpools.
      • We also created the following index - Note this is now included by default in 8.1.1, via JCR PK60132
      • DB2 CREATE INDEX JCR.ICMSTJCRLINKS_IXQ81 ON JCR.ICMSTJCRLINKS (TVID ASC, TIID ASC) ALLOW REVERSE SCANS
    • Tuning (cont'd.)
      • Recommended Database Maintenance
      • There are two database attributes that DB2 relies upon to perform optimally. These attributes are the database catalog statistics and the physical organisation of the data in the tables.
      • Catalog statistics should be recomputed periodically.
      • It is recommended that such maintenance is performed during periods of low demand or off peak hours or if possible when the system is offline.
      • The DB2 'runstats' command is used to count and record the statistical details about tables, indexes and columns.
      • The format we use on our database is of the form:
      • db2 -x -r "runstats.db2" "select rtrim(concat('runstats on table ', concat(rtrim(tabSchema), concat('.', concat(rtrim(tabname),' on all columns with distribution and detailed indexes all allow write access'))))) from syscat.tables where type='T'"
      • The command above generates a script 'runstats.db2' which contains the computed statistics on all the database tables, then execute that on that script as per the command below.
      • db2 -v -f "runstats.db2"
    • Tuning (cont'd.)
      • Recommended Database Maintenance
      • To determine which tables in the DB might benefit from re-organisation (re-org) we can use the command:
            • db2 reorgchk current statistics on table all > "reorgchk.txt"
      • For those tables that require re-organisation you can use the following command to re-organise the table based on its primary key.
            • db2 reorg table tableschema.tablename
      • To perform a re-org on all tables you can use the command:
            • db2 reorgchk update statistics on table all
      • Database performance is critically important for obtaining good performance from Lotus Quickr.
      • We have found that the maintenance practices that we have outlined here have been crucial to the performance of Lotus Quickr in our environment. Additional database maintenance may be required in your production environment.
      • It is possible to schedule these maintenance tasks such as table re-organisation.
      • Additional information on DB2 administration, tuning and monitoring can be found in the infocenter @ http://publib.boulder.ibm.com/infocenter/db2luw/v9/index.jsp
    • Tuning (cont'd.)
      • Web Server Tuning
      • Our SVT Reliability configurations were tested using IBM HTTP Server 6.0.2.29. It is advisable to have the HTTP server remotely located to avoid resource contention with other servers.
      • In our test environment we had a dedicated LAN with clients and servers located on the same segment. This meant that we had high bandwidth and low latency.
      • In real work networks bandwidth is limited and latency can be significant.
      • There are some steps that can help reduce transaction times:
            • Compress content on the http server using gzip compression.
            • Allow client side caching of images, javascript files and stylesheets.
      • Much of the content that is server by Quickr can be compressed to reduce transmission time and save network bandwidth.
      • IBM HTTP server supports compression through the mod_deflate module. When enabled, the http server will check the 'Accept-Encoding' header sent by the browser to see if the browser can accept a compressed version of the content. If it can then the http server will compress the content before sending it to the browser.
      • Whilst the reduction in network traffic from enabling compression can be significant there is a trade off as compression on the http server does result in increased processor utilisation on the http server.
      • The process for enabling gzip compression can be viewed at this link: http://www-01.ibm.com/support/docview.wss?rs=3264&uid=swg21289555
    • Tuning (cont'd.)
      • Web Server Tuning
      • Another method of improving the efficiency of network transactions is to enable client side caching.
      • It is possible to add Cache Control headers to the content that we wish to make cacheable. Lotus Quickr includes these headers in the stylesheets that it uses by default, allowing the content to be cacheable at a client for 432,000 seconds (5 days)
      • We can use mod_headers in IBM HTTP server to add the same headers to images and JavaScript files by adding the following lines to the httpd.conf file:
      • LoadModule headers_module modules/mod_headers.so
          • <Location ~ &quot;.(js|gif|jpg|jpeg|png)$&quot;>
          • Header add Cache-Control &quot;public, max-age=432000, post-check=172000&quot;
          • </Location>
      • See this technote for more information on http response cache headers: http://www-01.ibm.com/support/docview.wss?rs=899&uid=swg21289550
    • Web Server TuningTuning (cont'd.)
      • I
      • In addition to gzip compression and cache headers we also tune the following parameters in our environment:
      • You can find detailed tuning information on IBM HTTP server at the following url:
      • http://publib.boulder.ibm.com/httpserv/ihsdiag/ihs_performance.html
    • Tuning (cont'd.)
      • Web Server Tuning
    • Network Tuning on Windows
      • Windows Server 2003 is designed to be largely self-tuning operating system. A standard, plain installation of the operating system yields reasonably good performance results for most purposes.
      • In some instances, specific server settings and parameters can be tuned to optimize the performance of the Windows server operating system even further.
      • We apply several tuning steps to our network parameters on the Windows OS to fully exploit the network bandwidth that is available .
      • You should optimise your network card settings for your particular network environment. Important parameters to consider include the speed and duplex settings of your network cards.
      • Consult the Lotus Quickr tuning Guide for information on basic network tuning steps for Windows.
      • http://www-01.ibm.com/support/docview.wss?rs=3264&uid=swg27010632
      • There is also an excellent and indepth guide to tuning the Windows 2003 OS available as a red paper from this url: http://www.redbooks.ibm.com/abstracts/redp3943.html
    • Network Tuning on Linux
      • It is also important to consider the speed and duplex settings on Linux systems and ensure that they are compatible with your network environment as incorrect settings can cause a significant degradation in overall performance.
      • Consult the Lotus Quickr tuning Guide for information on basic network tuning steps for Linux.
      • http://www-01.ibm.com/support/docview.wss?rs=3264&uid=swg27010632
      • There is an excellent tuning guide for Red Hat available as a red book at the following url: http://www.redbooks.ibm.com/abstracts/redp3861.html?Open
    • Monitoring and Ongoing Tuning
      • You should make use of monitoring tools to identify bottlenecks and areas for potential additional tuning in your environment.
      • You can use readily available tools like 'Perfmon' on windows and nmon on Linux to monitor resources and detect bottlenecks.
        • http://www.ibm.com/developerworks/aix/library/au-analyze_aix/
        • http://technet.microsoft.com/en-us/library/bb490957.aspx
      • You can monitor servlet threads, and other application data using PMI.
      • Carefully monitor the database system for potential bottlenecks.
      • There is no definitive 'one size fits all' tuning that can be applied.
      • Performance and Tuning are dependent on multiple factors and performance should be carefully monitored as the system scales and tuning adjusted based on the data collected.
      • It is recommended that you regularly schedule data collection from your monitoring tools and review the data collected.
    • Section Summary Performance and Tuning considerations - What have we covered in this Section?
        • General Performance Recommendations.
        • Performance considerations relating to installation.
        • System specifications used in SVT and recommended Cluster configuration.
        • Performance considerations:
                        • DB maintenance.
                        • Search optimisation.
        • Tuning:
                        • Portal/App server tuning.
                        • DB tuning.
                        • HTTP server Tuning.
                        • OS and Network related tuning.
        • Usability Performance Tips.
        • Monitoring and ongoing Tuning.
    • What do We base Our Recommendations on?
    • Where do our recommendations come from? (Based on various experiences in our labs and our internal deployments)
    • Statistics for Internal Sample Deployment
      • Synthetic site built for performance benchmarking
      • Model Topology: Single Server
      • Directory of 100,000 users
      • 10,000 users have access to Lotus Quickr
      • 512 teamspaces containing about 145,000 pieces of content
        • 100,000 Documents
        • 45,000 “other” content items (blog posts, wiki pages)
      • Database Size: 89.5 GB
    • Statistics for Internal Sample Deployment
      • Internal deployment for reliability testing
      • Run at 75% capacity for 5 days with acceptable response times
      • Approximates a 6 month old customer deployment (at start)
      • Model Topology: Single Server and Clustered Server
      • Directory of 10,000 users
      • 100 Teamspaces containing about 304,000 pieces of content
        • 100,000 Documents (average size: 500KB)
        • 22,000 Wiki pages (average size: 41KB)
        • 22,000 Blog entries (average size: 4KB)
        • 160,000 List entries
      • Database size: 55 GB
      • Roughly .5M pieces of content at the end of a 5 day run
    • Example Deployment: IBM Internal Site
      • IBM Technology Adoption Program (TAP) :
      • Number of Quickr Users = 18,500+
      • Number of Quickr Places = 6,000+
      • Total size Database = 430+ GB
      • IBM Lotus Greenhouse
      • Number of places: 4,499 J2EE places
      • Number of registered users: 15,227
      HTTP Server DB2 with FastT storage Lotus Quickr Service for WebSphere Portal) Servers Deployment Manager
      • Basic Troubleshooting
    • Basic Troubleshooting
      • The Troubleshooting process:
          • There is no magic solution, troubleshooting requires data collection and analysis performed in an iterative fashion.
      • What type of issue are we seeing? Is it a functional failure or performance related?
      • Known problems are documented in the form of individual technotes in the Support knowledge base at http://www-01.ibm.com/software/lotus/products/quickr/support/. As problems are discovered and resolved, the IBM Support team updates the knowledge base. By searching the knowledge base, you can quickly find workarounds or solutions to problems.
      • Some basic info to consider:
              • Is Lotus Quickr at the latest level, currently 8.1.1
              • CPU utilisation for Portal and for the other components.
              • Are the minimum H/W & S/W requirements met?
              • Has the system a remote DB? Were there any issues during DB transfer
              • What is the system configuration (Single server/Cluster) and what was the install/upgrade path.
              • What is the search index size, find the location of the index in the icm.properties file.
      • A good place to start is the SystemOut/SystemErr.logs.
      • Verify that the installed PK's match the recommended list.
      • You can find a list of the Portal PK's in SystemOut.log or Version.log
      • SystemOut.log and SystemErr.log are located in the QuickrPortalServerlog dir on a Single server or node 1 of a cluster and in the dir Quickrwp_profilelogsWebSphere_Portal_name2 on node 2 in a cluster.
    • Basic Troubleshooting cont'd.
      • For database issues, examine the db2diag.log file from the DB2 server.
      • Use event monitoring and snapshot collection to gather data:
        • http://publib.boulder.ibm.com/infocenter/db2luw/v8/index.jsp?topic=/com.ibm.db2.udb.doc/admin/r0005991.htm
        • http://publib.boulder.ibm.com/infocenter/db2luw/v8/index.jsp?topic=/com.ibm.db2.udb.doc/admin/c0006003.htm
      • When was the last runstats/re-org performed on the database.
      • What is the total size of the DB, is there enough free space on the db server?
      • Some useful commands to retrieve configuration information from db2 include:
          • db2 &quot;get db cfg for wpsdb&quot; > dbcfg.txt
          • db2 &quot;get dbm cfg&quot; > dbmcfg.txt
          • db2set -all > db2set.txt
          • db2 &quot;select * from syscat.bufferpools&quot; > dbbuffpools.txt
      • Some useful queries that can be used to retrieve the amount of Documents in a db and the amount of webcontent are:
                      • SELECT COUNT(*) FROM DB2USER.ICMSTJCRWSNODES WHERE CTID IN (select nodetypecomptypeid from db2user.icmstjcrnodetypes where nodetypename='clbDocument')
                      • SELECT COUNT(*) FROM DB2USER.ICMSTJCRWSNODES WHERE CTID IN (select nodetypecomptypeid from db2user.icmstjcrnodetypes where nodetypename='webContent')
    • Basic Troubleshooting cont'd.
      • When troubleshooting a specific issue it can be helpful to increase the log file sizes and the backups of these files. This is especially true if applying tracing.
      • To increase the log file sizes on a Cluster, login to the deployment manager admin console @ https://dmgrhostname.domain:9043/ibm/console/logon.jsp
                • Select Servers, Application Servers on the left hand side and then select WebSphere_Portal.
                • Now select Logging and Tracing from the resulting screen.
                • Now select JVM Logs from the resulting screen.
                • On this screen you need to change the maximum size of SystemOut and SystemErr to 50 MB and the Maximum Number of Historical Log Files to 300.
                • Select OK and then save the changes.
      • In certain circumstances it can be helpful to apply tracing to try to identify the issue.
      • If applying tracing you should minimise the activity on the system during this period.
      • To set tracing on a Cluster, login to the deployment manager admin console @ https://dmgrhostname.domain:9043/ibm/console/logon.jsp
                • Select Servers, Application Servers on the left hand side and then select WebSphere_Portal
                • Select Change Log Detail levels(under troubleshooting, near the bottom)
                • Select the runtime tab and set the trace string
                • Click apply at the bottom of the page, the change occurs without a restart.
                • Some common strings are shown on the next slide:
      • JCR Trace String
                      • com.ibm.icm.jcr.NodeImpl=finest:
                      • com.ibm.icm.jcr.WorkspaceImpl=finest:
                      • com.ibm.icm.jcr.PropertyImpl=finest:
                      • com.ibm.icm.jcr.query.*=finest:
                      • com.ibm.icm.da.portable.common.sql.PPreparedStatement=finest
      • For Debugging Searches add
                      • com.ibm.icm.ts.*=finest:
      • Wikis/Blogs add this trace on top of JCR
                      • com.ibm.workplace.wcm.teamspace.*=finest:
                      • com.ibm.workplace.wcm.services.repository.*=finest:
      • Team Calendar add this trace on top of JCR
                      • com.ibm.workplace.team.calendar.*=all:
                      • com.ibm.workplace.teamcalendar.*=all:
                      • com.ibm.workplace.wcm.teamspace.*=all:
                      • com.ibm.wps.portlets.Tcalendar.api.TCalendarAPI.*=finest:
                      • com.ibm.wkplc.team.calendar.*=finest:
      Basic Troubleshooting cont'd.
      • If there is a reproducible problem or failure, the process we have found most suitable for collecting focussed logs relating to the problem is:
          • Set up the trace and log file options
          • Login to Quickr
          • Navigate to right before the point where the problem occurs.
          • Record the start time from the trace log
          • Recreate the problem as normal.
          • Record the stop time from the trace log
          • Collect logs and examine in the period where the issue has occurred.
          • You can also collect DB2 snapshot and event monitor information as required.
      Basic Troubleshooting cont'd.
      • For any OOM issues:
        • Verify that the correct WAS PK's are installed as recommended.
        • Verify the version of WAS JVM is correct: For 811 we tested with SR11.
        • Run <Appserver>javain>java -version to get this information
        • Follow the process detailed here for OOM's: http://www-1.ibm.com/support/docview.wss?uid=swg21111364
      Basic Troubleshooting cont'd.
    • Rebuilding Juru Indexes (Troubleshooting)
      • If we need to cleanly rebuild our JURU search indexes for any reason we use the following process:
          • Stop all servers.
          • Delete index from the search directory (the &quot;1&quot; directory within the Search directory)
          • On your DB server run the following commands:
                • delete from jcr.icmstjcrtspending
                • delete from jcr.icmstjcrtspathinfo
          • Restart your servers.
          • The indexes will start building again around 30 minutes after the start of Portal server if the default interval value is left unchanged in the icm.properties file.
          • You can monitor the SystemOut log to see the index activity.
          • The search directory will also keep increasing in size until indexing is complete. You can also run the query db2 select count(*) from jcr.icmstjcrtspathinfo during indexing, this value will keep increasing during the indexing build process.
          • We have found it optimal to backup our databases and search indexes in sync. This way if a database needs to be restored we can restore the associated index as well removing the need for a resynchronisation between the content and the indexes.
    • Tips and Tricks
      • Before making any major modifications to the system, take a full system backup including indexes and databases.
      • Backup any property files before modifying them.
      • When backing up your database it is best to backup your search indexes as well to keep the data in sync.
      • We have found significant space savings by compressing our database backups both on Windows and Linux. There is a consideration when doing this however as depending on the size of the backup/machine specifications it can take a number of hours to compress/decompress the backups.
      • It is helpful to 'tail' Logs when you are troubleshooting, on Linux this can be done with the tail command, On windows there are several alternative 'tail' programs that can be used.
      • Find WAS information – run the script historyInfo.bat from the dir QuickrAppServerin, this will generate a report called versionReport.html in the same directory with detailed information on the level and patches applied.
      • Find Portal information – Run the script genVersionReport.bat from the directory: QuickrPortalServerin This will create a detailed report called: versionReport.html in the same directory. You can create a detailed history report by running the script: genHistoryReport.bat from the PortalServerin directory, this will generate a report called historyReport.html in the same directory.
      • List patches installed – Check the version.log file, to review the history review the contents of the directory: QuickrPortalServerversionhistory
    • Section Summary Basic Troubleshooting - What have we covered in this Section?
        • Basic Troubleshooting:
                    • The process.
                    • Items to consider.
                    • Log Locations.
                    • Database issues.
                    • Tracing
                    • Process for MustGather information for OOM's.
        • Tips and Tricks.
        • How we rebuild our search indexes.
        • Useful links/Resources.
    • Useful Links/Resources
      • IBM Lotus Quickr Version 8.1.1 Information Center @ http://publib.boulder.ibm.com/infocenter/lqkrhelp/v8r0/index.jsp
      • The hardware and software requirements for Lotus Quickr 8.1.1 can be found in detail @ http://www-01.ibm.com/support/docview.wss?rs=3264&uid=swg27009740
      • 8.1.1: Readme for upgrading IBM Quickr 8.1 services for IBM WebSphere Portal -- Clustered Configuration @ http://www.ibm.com/support/docview.wss?rs=3264&uid=swg27013257
      • Required post-install fixes for Lotus Quickr 8.1.1 services for WebSphere Portal @ http://www-01.ibm.com/support/docview.wss?rs=3264&uid=swg21330316
      • Recommended 8.1.1 fixes @ http://www.ibm.com/eserver/support/fixes/fixcentral/swg/quickorder?brandid=2&productid=Lotus+Quickr&searchtype=recommended&vrmf=8.1.1
      • To view all technotes for Lotus Quickr go to the Lotus Quickr Support page @ http://www-306.ibm.com/software/lotus/products/quickr/support/
      • Lotus Quickr: Product documentation, white papers, Redbooks, and more - http://www.ibm.com/developerworks/lotus/documentation/quickr/
      • Lotus Quickr Support page - http://www-306.ibm.com/software/lotus/products/quickr/support/
      • Lotus Quickr product home page on ibm.com - http://www-306.ibm.com/software/lotus/products/quickr/
      • IBM DB2 Database for Linux, UNIX, and Windows Information Center - http://publib.boulder.ibm.com/infocenter/db2luw/v9//index.jsp
      • WebSphere Portal and Workplace Web Content Management product documentation - http://www.ibm.com/developerworks/websphere/zones/portal/proddoc.html
      • A comprehensive list of recommended, generally available (GA) fixes for IBM® Lotus® Quickr™ releases. @ http://www-01.ibm.com/support/docview.wss?rs=3264&context=SSVGKL&dc=DA400&uid=swg27010121&loc=en_US&cs=UTF-8&lang=en&rss=ct3264lotus
      • Portal Tuning Guide @ http://www-1.ibm.com/support/docview.wss?rs=688&uid=swg27008511
      • Lotus Quickr Tuning Guide @ http://www-1.ibm.com/support/docview.wss?rs=3264&uid=swg27010632
      • Diagnosing Performance Problems for WebSphere Portal 5.1 (though this document was written for WebSphere Portal 5.1, the lessons can apply to Lotus Quickr as well): http://www.ibm.com/support/docview.wss?uid=swg27007059.
    • © IBM Corporation 2009. All Rights Reserved. The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. IBM, the IBM logo, DB2, DB2 Universal Database, Lotus, Domino, Quickr, Redbooks and WebSphere are trademarks of International Business Machines Corporation in the United States, other countries, or both. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others.