© 2004 Hewlett-Packard Development Company, L.P.


Published on

Published in: Technology, Business
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • This presentation is a collection of best practices for Oracle on HPUX servers from HP’s Labs, IT and Performance teams as well as independent sources. There is more material than can be thoroughly covered in the alloted time, so as I go thru each section, I’lll hit the high points and take questions. However I’ll leave a copy of this slide set for you to use as a reference and a set of guidelines. Oracle is a big product. As such, any presentation of this scope will still miss many important topics. E.G. New features of DataGuard In Depth discussion of OEM Oracle parameter settings Etc. I chose to focus on the HPUX setup and tuning for Oracle as well as Best Practices for Oracle from within HP.
  • This is well documented in the Oracle Unix Install Guide, but it’s easy to miss the most important steps and the HPUX unique settings.
  • I’ll mention SAM throughout this presentation. SAM is the HPUX GUI tool for System Administration.
  • If you choose to use your Window’s desktop for doing Unix work, there is X environment software such as ReflectionsX, Hummingbird’s Exceed or RealVNC. You can also use Putty for terminal window emulation but you’ll need an X environment for the Oracle installation. If you have a very large memory, it may not be necessary to create swap space that’s twice the size of physical memory, although this is Oracle’s recommendation. Since disk storage is relatively cheap compared to memory, the suggestion is to create large swap space. The disk space is the amount needed for the Oracle binaries.
  • One of the most important steps BEFORE installing Oracle is to update HPUX with the required patches. To do this: locate the list of recommended HPUX patches for your version of the OS in the Oracle Install Guide. Research these patches – Check the HP support web site for each patch to find any superceded patches or dependent patches When you have the latest list of patches, check your system to see if any are installed. The swlist commands on this slide can be used to list all installed patches. Download and install all required patches Depending on the customer’s HPUX support contract, they may be able to have HP support do the research and provide the newest patches In the lab, we’ve found that we need to increase system tmp space to at least 2GB or the Oracle installation may fail. Also before installing Oracle, modify the kernel parameters. Oracle gives a recommended list of MINIMUM HPUX kernel settings in its Install Guide. However, this presentation will go thru these parameters explaining their use by the HPUX kernel.
  • The SCHED_NOAGE policy gives processes holding a latch a fixed priority and makes them nonpreemptable during this time. This causes less latch waits and latch sleeps, hence higher throughput. To enable the SCHED_NOAGE policy for Oracle, the hpux_sched_noage init.ora parameter needs to be set: 154 to 255 (for HP/UX 11.0) 178 to 255 (for HP/UX 11i) For example: hpux_sched_noage=178 In addition, the system privileges RTPRIO and RTSCHED need to be added to the DBA group as root in /etc/privgroup: dba RTPRIO RTSCHED
  • Determine Whether the Oracle Inventory Group Exists When you install Oracle software on the system for the first time, the Installer creates the oraInst.loc file. This file identifies the name of the Oracle Inventory group and the path of the Oracle Inventory directory. To determine whether the Oracle Inventory group exists, enter the following command: # more /var/opt/oracle/oraInst.loc If the oraInst.loc file exists, the output from this command is similar to the following: inventory_loc=/u01/app/oracle/oraInventory inst_group=oinstall The inst_group init.ora parameter shows the name of the Oracle Inventory group (oinstall). If you’re installing Oracle10g, you may need to create a “nobody” user (named extjob) for the oracle extjob. This is documented in the Oracle10g installation manual. We have successfully installed Oracle10g without creating this user or running this set of steps following the installation. However, it is called out in the manual specifically for HPUX so I wanted to point it out.
  • Few Important Environment variables For installation of oracle as well as to use it you need to set following environment variables. ORACLE_HOME This holds the base directory for oracle installation. e.g. export ORACLE_HOME=/oracle/9i_64bit ORACLE_SID This is used as the system (oracle instance) identifier. Needs to start with alphabet and can contain only alphanumeric values. e.g. export ORACLE_SID=oratest PATH To use oracle you need to set PATH variable such that it includes $ORACLE_HOME/bin e.g. export PATH=$ORACLE_HOME/bin:$PATH SHLIB_PATH/LD_LIBRARY_PATH Used by shared library loader at run-time. e.g. export LD_LIBRARY_PATH=$ LD_LIBRARY_PATH :$ORACLE_HOME/lib64.
  • Later in this presentation, I’ll review a recommendation from HP’s IT for a simplified OFA (Optimal Flexible Architecture), the directory naming standard created by Cary Milsap of Oracle in the late 1990’s. This simplified naming structure is OSA (Optimal Simplified Architecture).
  • I will compare the use of raw vs. filesystem with HPUX later in this presentation. Note that you need to perform these steps ONLY IF using raw. max_async_ports limits the total number of open ports to the ansynchronous disk-I/O driver that processes on the system can have at any given time (this has nothing to do with any RS-232 asynchronous data-communications interfaces). The system allocates an array of port structures for each port when it is opened that is used for all communication between the process and the asynchronous disk driver. The number of asynchronous ports required by a given application is usually specified in the documentation for that application (such as database applications software, video management software, etc.).
  • The kernel parameter page in SAM with max_asynch_ports highlighted.
  • This section is deliberately detailed. Since our time is limited, I’ll describe the types of kernel parameters and discuss of few of the most important. However, the notes have more details about each of the parameters.
  • Note: If the current value for any kernel parameter (on your system) is higher than the value listed in this table do not change the value of that parameter ( except for the attributes whose recommended value is 0).
  • bufpages To enable dynamic buffer cache allocation, set bufpages to zero. This is an old parameter which was left in for backward compatibility dbc_max_pct defines the maximum percentage of memory to be used by dynamic buffer cache dbc_min_pct defines the minimum percentage of memory to be used by dynamic buffer cache maxfiles specifies the system default soft limit for the number of files a process is allowed to have open at any given time. It is possible for a process to increase its soft limit and therefore open more than maxfiles files. Non-superuser processes can increase their soft limit until they reach the hard limit ( maxfiles_lim) maxfiles_lim sets the hard limit for the number of files a process is allowed to have open simultaneously nfile defines the maximum number of files that can be open simultaneously, system-wide, at any given time nflocks defines the maximum combined total number of file locks that are available system-wide to all processes at any given time ninode defines the number of slots in the inode table, and thus the maximum number of open inodes that can be in memory at any given time. The inode table is used as a cache memory. For efficiency reasons, the most recent Ninode (number of) open inodes is kept in main memory. The table is hashed. (? Define an inode ?)
  • maxvgs specifies the maximum number of volume groups on the system maxswapchunks sets the maximum amount of swap space that can be configured, system-wide (? space in Kbytes, bytes ?) swapmem_on enables or disables the reservation of pseudo-swap, which is space in system memory considered as available virtual memory space in addition to device swap space on disk. By default, pseudo-swap is enabled. Virtual memory (swap) space is normally allocated from the device swap area on system disks. However, on systems that have massive amounts of installed RAM and large disks or disk arrays, there may be situations where it would be advantageous to not be restricted to the allocated device swap space. swchunk specifies the chunk size to be used for swap vps_ceiling specifies the maximum page size (in Kbytes) that the kernel can select when it chooses a page size based on system configuration and object size
  • maxdsiz and maxdsiz_64bit define the maximum size of the static data storage segment for executing 32-bit processes and 64-bit processes on 32-bit and 64-bit processors, respectively. This segment contains fixed data storage such as globals, arrays, statics, locals to main(), strings, space allocated using sbrk() and malloc(), and such. maxssiz and maxssiz_64bit define, for 32-bit and 64-bit processors respectively, the maximum size of the dynamic storage segment (DSS), also called the user-stack segment, or an executing process's run-time stack. This segment contains stack and register storage space, and such. maxtsiz and maxtsiz_64bit define, for 32-bit and 64-bit processors respectively, the maximum size of the shared text segment (program storage space) of an executing process. Program executable object code is stored as read-only, and thus can be shared by multiple processes if two or more processes are executing the same program simultaneously, for example. maxuprc establishes the maximum number of simultaneous processes available to each user on the system. A user is identified by the user ID number, not by the number of login instances. max_thread_proc limits the number of threads a single process is allowed to create. This protects the system from excessive use of system resources if a run-away process creates more threads than it should in normal operation. nkthread limits the combined total number of threads that can be running on the system at any given time from all processes on the system. nproc specifies the maximum number of processes that can exist simultaneously on the system at any given time (limited by memory). May need to be increased if the server runs both the application and the database.
  • msgmap Message queues are implemented as linked lists in shared memory, each message consisting of one or more contiguous slots in the message queue. As messages are allocated and deallocated, the shared memory area reserved for messages may become fragmented. msgmap specifies the size of a resource map used for allocating space for new messages. This map shows the free holes in the shared memory message space used by all message queues. Each entry in the map contains a pointer to a corresponding set of contiguous unallocated slots, and includes a pointer to the set plus the size of (number of segments in) the set. msgmni specifies the maximum number of message queues that can exist simultaneously on the system. msgseg specifies the system-wide maximum total number of message segments that can exist in all message queues at any given time. msgtql specifies the maximum number of messages that are allowed to exist on the system at any given time. semmap Each set of semaphores allocated per identifier occupies 1 or more contiguous slots in the sem array. As semaphores are allocated and deallocated, the sem array might become fragmented. semmap dimensions the resource map which shows the free holes in the sem array. An entry in this map is used to point to each set of contiguous unallocated slots; the entry consists of a pointer to the set, plus the size of the set. semmni specifies the maximum number of sets of IPC semaphores that can exist simultaneously on the system. semmns defines the system-wide maximum total number of individual semaphores that can be made available to system users. semmni less then or equal to semmns semmns less then or equal to (semmni * semmsl) semmsl is the value of the maximum number of semaphores that can be associated with a semaphore ID. The value of semmsl is set at 500 and is not configurable. semmnu defines the maximum number of processes that can have undo operations pending on any given IPC semaphore on the system. semvmx specifies the maximum value a semaphore can have. This limit must not exceed the largest number that can be stored in a 16-bit unsigned integer (65535) or undetectable semaphore overflows can occur.
  • shmmax defines the system-wide maximum allowable shared memory segment size in bytes. Oracle recommends that this be large enough to hold the entire SGA in one shared memory segment shmmni specifies system-wide maximum allowable number of shared memory segments (by limiting the number of segment identifiers). shmseg defines the maximum number of shared memory segments that can be simultaneously attached to a single process. max_async_ports specifies the system-wide maximum number of ports to the asynchronous disk I/O driver that processes can have open at any given time. maxusers limits system resource allocation, not the actual number of users on the system. maxusers does not itself determine the size of any structures in the system; instead, the default value of other global system parameters depend on the value of maxusers. When other configurable parameter values are defined in terms of maxusers, the kernel is made smaller and more efficient by minimizing wasted space due to improperly balanced resource allocations.
  • This slide lists the commands to use when updating kernel parameters. However, it is recommended that you use SAM for this process.
  • If your application is consolidated on a single system, with both the application and the database on the same system, you may want to partition the system to accommodate the right system version and patch level and setting of kernel parameters. Combining the automatic resource provisioning of WLM (Workload Manager) with Virtual Partitions maximizes usage of system resources and guarantees that the highest priority jobs are given the most resources.
  • This slide is a build. In this case there are three applications served out of the server resource pool: Oracle Apps, Oracle reports, and a security application. Oracle Apps requires more server resources to address the pre-set service level objective (SLO), for example of 1 second response time. The policy engine measures on a continuous base the response time and discovers that the Oracle application requires more server resources to meet the SLO of 1 sec response time. Accordingly it allocates immediately, on-the-fly more resources to the Oracle application. In this scenario there were sufficient underutilized server resources, which accordingly could have been allocated to the Oracle application. Depending on how the priorities are set, the resources for the Oracle reports can be reduced and given to the higher prioritized Oracle application.
  • Recommendations for most environments: (Note, this finding pertained to Oracle8i. Oracle9i may not have same behavior.) To gain the most performance from the JFS file system, the Online JFS product should also be implemented. This will allow the use of enhanced mount options that optimize the interaction of Oracle and HP-UX. These options include: delaylog – this option allows the file system to delay the writing on non-critical information to the JFS intent log. This improves the performance of the file system by allowing some I/O operations to return before this information is put into the intent log on the disk. An example would be when an I/O operation only entails changing the timestamp on the file. In this case, the contents and structure of the file would not be compromised if a crash were to occur before the intent log information made it to the disk. nodatainlog – Oracle always uses O_SYNC writes, this option will avoid JFS writing the data to the intent log as well as the file. mincache=direct – The default read operation for JFS copies data from disk to the HP-UX buffer cache, and then copies data to the Oracle SGA. Setting this mount options causes the data to be moved directly into the Oracle SGA; this may provide a minor improvement in the performance for non-sequential read operations. In 8.x versions of Oracle, this mount option will cause unnecessary physical I/O for sequential I/Os. This mount option should NOT BE USED with Oracle 8.x tablespace files , however it is recommended for Oracle 8.x redo and archive file systems. convosync=direct – This option changes the behavior of files opened with the Osync flag enable, which Oracle always uses. This will enable Osync I/O operations to operate the same as non-osync file operations and thus use the mincache=direct mount option. In 8.x versions of Oracle, this mount option will cause unnecessary physical I/O for sequential I/Os. This mount option should NOT BE USED with Oracle 8.x tablespace files , however it is recommended for Oracle 8.x redo and archive file systems.
  • The design of the LVM code that HP acquired from OSF assumes an 8K blocksize. Matching Oracle’s blocksize to LVM’s 8K blocksize (or a multiple thereof) makes for more efficient use of resources.
  • Oracle databases can be implemented either in raw space or on top of UNIX file systems. Raw is the better choice for performance, but many customers prefer filesystem because they think it’s easier to manage. We don’t need to go too deep into the performance advantages of raw - modern file systems, especially the Advanced File System available for Tru64 - have come within 5% of the performance of raw, though the filesystems available for HP-UX generally are 25-50% worse performers than raw. For clusters, though, file systems would have to be made “cluster-aware” in order to be used in a cluster environment - that’s because file systems have their own memory-based cache, and the file-system caches on the different nodes would have to be synchronized using a scheme much like Oracle 9iRAC cache fusion. So you couldn’t just mount a standard, non-cluster-aware file system from multiple nodes and expect it to work. HP-UX does not yet support a clustered file system. We could have assigned lab resources to supporting a CFS like our competitors have, but we didn’t think it would add much value compared to having the same resources work on incorporating leading-edge Tru-Cluster features into HP-UX. Other presentations discuss the advantages of Tru-Clustering but here the remarks will be focused on raw devices and why they’re not as scary as some people make them out to be...
  • Let’s look at how Oracle sees the storage, both file-system and raw.. Oracle objects (data, indexes, etc) are allocated to “tablespaces” which are assigned to one or more “Oracle data files”. These Oracle files map one-on-one to either the individual files in a UNIX filesystem, or to single RAW logical volumes. A file-system is simply a structure that is built on top of a logical volume using the Unix command newfs. There is a bunch of structure that is added to this to allow you to keep track of the individual files in the file system. Note that a “raw” logical volume is simply a LV which does not have a file-system built on top of it. (The diagram shows how filesystems are built within LVM logical volumes.) (For completeness, the LV’s are then assigned space within various Physical Volumes - PVs - which are either individual disks or LUNs in a disk array.) In an Oracle database environment since a logical volume is normally dedicated to store data files you only need to know what tablespace it is attached to, how large it is and how much free capacity is available. OEM will give you that for RAW.
  • To monitor disk space, OEM will show the internal contents of each Oracle datafile. The UNIX “bdf” command will show the space available for more files within a file system, but with raw, this whole concept is moot. All the space inside a raw LV belongs to - and is managed wholly by - Oracle. To add more space for the database to use, in a raw environment that isn’t clustered, we’d just create a new LV if there’s room for one in an existing volume group, or we’d create a new volume group if we needed to. In a clustered environment if we do not use Veritas CVM, we’d have to deactivate the shared volume group in order to alter its structure (such as adding a new LV would require). To avoid deactivating the shared VG (which would require the database to be brought down), we can make use of pre-existing empty LVs that we’ve created for just such a purpose. Or, we can create and activate a brand-new volume group, containing as many LVs as we like. Many people think it’s easier to destroy or damage the database when it’s implemented in raw, but there are actually more ways to accidentally erase data with file systems! (And even ‘lvremove’ will not let you remove a logical volume that’s open - such as when the database is up and running!) Even if you delete the /dev/vgora/* files, the data is still on the disk and has not been destroyed. You can recreate the special device file names that again point to the data on the disks without destroying any of your data.
  • Oracle Recovery Manager - RMAN - is the best approach to backing up an Oracle database, because RMAN guarantees the integrity of the backup. Backing up the individual data files (or raw logical volumes) using tar (or dd ) is technically feasible, but leaves the database vulnerable - it’s easy to forget to update the backup script whenever a DBA adds a new datafile to the database. RMAN ensures that all the database files get backed up. RMAN just backs up the data, and does not care (probably cannot even tell) if the data is raw or filesystem-based. You can even use RMAN to achieve the move from filesystem files to raw devices.
  • Tru64’s Advanced File System (clustered) comes within 5% of the performance of raw, but AFS is not available on HP-UX yet. For now, the best we can do is with JFS (with some non-default mount options), which is about 25% slower than raw (thus, raw is about 33% better than JFS). HFS is about 50% slower than raw (raw is 100% better than HFS, in terms of throughput). Reminder: HP-UX will incorporate Tru-Cluster features in the future. Note: In the lab, we’ve seen huge performance increases by using raw. However, our Performance Consultants tell us that with real customers, this is not necessarily the case.
  • The evolution of disk technology from small, independent disks (JBODs – Just a Bunch of Disks) to large RAID arrays has had a dramatic impact on the way we define Oracle objects on storage.
  • I kept an Oracle White Paper for quite awhile which recommended no less than 20 different data sets, each on separate disks, to store Oracle data (e.g. indexes, data files, logs, temp space, etc.). As storage technologies have changed and we’re no longer defining individual disks, translating the Oracle data definition to the actual mapping to storage has become a complicated process.
  • Juan Loaiza’s paper first introduced a simplified layout, called SAME (Stripe and Mirror Everything). In essence, it documented what our performance lab engineers had been doing for some time with Oracle on large storage arrays.
  • The reasons why SAME works.
  • Our performance engineers have modified the original recommendations as storage arrays have increased in size and capacity.
  • The next few slides are here to show the mapping between the Oracle storage definition, the OS storage definition and the Array storage definition.
  • You may think you’re putting data on a separate device, but with the complex structures of large storage mechanisms, you may even be putting this on the same device. We’re moving away to more layers of abstraction and more complexity.
  • The architecture of your storage system will determine the best choice for creating the stripeset. Here the example is for HP’s XP storage which uses array groups as the unit of storage.
  • Details about extent striping: lvcreate - create logical volume in LVM volume group /usr/sbin/lvcreate [-A autobackup ] [-c mirror_consistency ] [-C contiguous ] [-d schedule ] [-D distributed ] [-i stripes -I stripe_size ] [-l le_number | -L lv_size ] [-m mirror_copies ] [-M mirror_write_cache ] [-n lv_name ] [-p permission ] [-r relocate ] [-s strict ] vg_name -D distributed  Set the distributed allocation policy. distributed can have one of the following values: y Turn on distributed allocation.n Turn off distributed allocation. This is the default. When the distributed allocation policy is turned on, only one free extent is allocated from the first available physical volume. The next free extent is allocated from the next available physical volume. Allocation of free extents proceeds in round-robin order on the list of available physical volumes. When the distributed allocation policy is turned off, all available free extents are allocated from each available physical volume before proceeding to the next available physical volume. This is the default. The distributed allocation policy REQUIRES the PVG-strict allocation policy ( -s g ) to ensure that mirrors of distributed extents do not overlap (for maximum availability). lvcreate (1M) will obtain the list of available physical volumes from /etc/lvmpvg. See vgextend (1M) for more information on physical volume groups and /etc/lvmpvg. When a logical volume with distributed extents is mirrored, the resulting layout is commonly referred to as EXTENT-BASED MIRRORED STRIPES. Note that EXTENT-BASED MIRRORED STRIPES can be created without the distributed allocation policy by adding one extent at a time to the desired physical volumes through lvextend (1M) . The distributed allocation policy is incompatible with the striped scheduling policy ( -i stripes ) and the contiguous allocation policy ( -C y ). The lvchange (1M) command can be used to assign the distributed allocation policy to an existing logical volume.
  • Oracle’s online redo logs are of critical importance to update/modify throughput and play a key role in database recovery operations. Failure of an online log group can lead to the most painful of all Oracle database recovery scenarios. For these reasons, online logs deserve very special attention. We regard as Best Practice that Oracle’s log multiplexing be used in favour of, or in addition to, RAID features. For optimal availability, members of multiplexed log groups should be located such that they share no common points of failure at the disk, channel, or board level. Finally, where update/modify performance is important, logs should be located on dedicated (or otherwise quiet) disk spindles. These practices have two known downside effects. First, one must typically configure an apparent excess of disk capacity to meet all these constraints. With per disk capacities soaring, this is increasingly difficult to justify. Second, since Oracle must do each log write twice to keep both log group members synchronized, peak throughput is slightly reduced. Still, this proves a small price to pay for providing optimal protection to these most critical files.
  • Problem statement Many Oracle DBA’s regard the example of the OFA standard in Oracle training as the only way that an Oracle database should be installed. Actually, the OFA defines a set of characteristics. The textbook example given in training is not the only way. Internally, HP’s Oracle IT group has developed the Optimal Simplified Architecture because it is much more appropriate for Oracle installations on UNIX servers. It is much easier to support without sacrificing database performance or reliability. The example (from HP/Oracle IT internal training) suggests features such as having multiple disk mount points so that disk I/O can be isolated. It also recommends mirroring redo logs, control files, and other database files. These are great ideas in general, but implementing these features in the current HPUX environment only decreases performance and provides very little additional reliability. The disk subsystems on HPUX today usually take care of disk mirroring on a hardware level. Distributing data files to independent disk spindles actually yields worse performance than putting all data files on a single volume that is distributed-striped across at least four drives. We contend that our solution, the Optimal Simplified Architecture for Oracle on UNIX, is more compliant to the OFA standard than the example given in Oracle training.
  • Performance benchmark Disk systems today are so big and so fast that tuning individual disks just doesn’t make sense. I/O is faster when configuring disks in a distributed stripe instead of mounting disks individually on a single mount point. Here are standard Oracle BSTAT/ESTAT benchmark results comparing the difference between configuring a database on individual spindles (BEFORE) versus distributed disk striping on the same hardware (AFTER):
  • Also: scalable from a single server to the whole enterprise. Integrations across multiple applications, operating systems & storage architectures
  •  Adds point and click GUI for the restore of Oracle 8/9i databases: administrators can select all or individual RMAN restore options from the Graphical User Interface freeing them from dealing with the Oracle RMAN Command Line Interface.  Offers simplified SAN auto-configuration wizard: automatically detects and configures the backup drives in the SAN on all desired SAN attached systems. The new "group by devices" and "group by hosts" views allow optimized shared device access. Increased resilience with recovery from any disruption, from instant recovery to site or system disaster recovery    Eliminates backup windows for HP StorageWorks EVA & XP customers with the industry's first fully-integrated out-of-the-box Zero-Downtime Backup solution for the HP StorageWorks EVA array    Eliminates recovery windows for EVA & XP customers by integrating its Instant Recovery capability to enable the recovery of even terabytes of application data in minutes, not the hours it would take to restore this data from tape Increased extensibility which scales from single server to enterprise    Incorporates comprehensive protection of emerging IA-64 platforms (local, network and SAN-based): Windows Server 2003 on IA-64; HP-UX 11.23 on IA-64); Linux on IA-64 (Redhat, SuSE, Debian - network-based protection)           Adds support for applications, operating and storage environments: DB2 online backup for IBM AIX and HP-UX; OpenVMS backup; Windows Server 2003 online backup of Oracle 9i, SQL Server 2000; ZDB Backup on Windows Server 2003 for Oracle, Exchange, SQL, and SAP    
  • IMAGE 1: 11:55am INCIDENT.. ...and the most recent tape backup is from midnight last night - 12 hours ago. IMAGE 2: INSTANT RECOVERY.. ...offers continuous protection, allowing fine granularity of recovery images.. Periodic and frequent zero-downtime backups are performed to disk.. IMAGE 3: DATA PROTECTOR AUTOMATES SCHEDULING.. ...and replication of images -- multiple disk mirror images are round-robined automatically and efficiently.
  • (? Check Oracle manuals for the stmt: “No extra redo generated during online backup” – How has this changed in RMAN ?)
  • Offline backup - this is done with the database shutdown so that no activity is occuring against the DB and the state is consistent. Either Oracle or the OS (dd) can be used. Online backup - Requires application to issue BEGIN/END BACKUP statements for an OS backup. Not needed for RMAN. Corrupt block - Oracle writes to V$DATABASE_CORRUPTION Automatic backup - RMAN coordinates - specifies names and locations of all data to be backed up (whole database, tablespace, datafile (raw or fs) or control file). Backup catalogs - RMAN records backup info in the recovery catalog and in the control file.
  • IF USING TAPE BACKUP/RESTORE With the increasing use of RMAN/Data Protector, increased testing has been occurring to backup and recover complete Oracle databases. We have noted that the recovery of a database took significantly longer than the backup of that same database when using a large number of RMAN channels to a small number of tape drives. In one instance, a recover took 3 to 4 times longer than the backup when using 5 RMAN channels to one tape drive for both the database backup and recovery. To understand RMAN/Data Protector performance, we created a 200GB test database and performed test backups and recoveries using different numbers of tape drives and RMAN channels. The following conclusions were reached as a result of our tests. We obtained the best database restore performance by using one channel per tape drive during the backup and using double the number of channels during the restore, than during the backup. For example, with a database backup using 3 tape drives, using 3 RMAN channels (1 per tape drive), then using 6 RMAN channels for the data recover yielded the best overall recovery time. In fact, our tests yielded recovery times equal to the original backup time using the above settings. We used different test settings using 1 to 3 tape drives and one to two channels per tape drives during the backups and all recoveries were done with double the number of channels from each backup. The results of all of these tests showed that recovery times were less than 10% greater than the backup times. The reason for the database recovery needing double the number of RMAN channels from the backup is that this insures that when 1 datafile restore completes, the next datafile to begin restoring will already be queued in Data Protector so that the restore starts as the tape is being read serially and no tape repositioning will occur. Tape repositioning is the main reason for recoveries taking longer than backups. In summary, allocate 1 or 2 RMAN channels per tape drive to be used for the backup, i.e. if using 8 tape drives, allocate 8-16 RMAN channels. For the database recovery, allocate twice as many RMAN channels for the recovery, if 8 RMAN channels were used for database backup, use 16 RMAN channels f allocate 1 or 2 RMAN channels per tape drive to be used for the backup or the database recovery.
  • This is a recommended set of responsibilities from HP’s Oracle IT team.
  • This diagram is from a sizing for Oracle Applications with the Oracle database for a customer. Notice the addition of Development, Test/Integration and Backup servers.
  • When one considers the word “disaster”, it often conjures up images of life-threatening events. But if your business relies upon Information Systems, if you haven’t protected your data, something as simple as a power outage or human error can be a disaster. Of course, events such as September 11 th , are considered disasters. It is important, and perhaps comforting to note, that the methods one uses to protect data from some of the smaller events can also protect them from something as horrific as the events of Sept. 11 th . However, some of the most common disasters aren’t always planned for. These are the ones which can catch you by surprise. A disaster?: any situation which separates users from their information and affects revenue and profit customer satisfaction publicity productivity competitiveness What Is Downtime? The definition of downtime is very easy to understand. It is the amount of time that a customer’s IT applications, databases or infrastructure are unavailable to employees or customers. Downtime is typically measured in the number of minutes/year an IT environment is not operating Some business critical or mission critical companies measure downtime in seconds/year. Acceptable downtime duration is often set by agencies and governing bodies Downtime goals can be set by management or in some instances by agencies or governing bodies. Downtime has a cost that can be calculated Typically this cost is calculated into $ of lost business due to a disruption of processing orders, loosing contact with customers, or back office process and procedure disruption. Note how commonly human errors cause data loss. Aside from providing better procedures, training, and security, point-in-time copies of data ensure that the data can be restored back to a known state. Software malfunctions can be prevented in many cases by running tests on a point-in-time copy that has been split off from the live data. If all else fails, the prior working release of software and data can be recovered from a tape. The key to avoiding impact of hardware malfunctions is to eliminate all single points of failure. RAID arrays are inherently redundant, but must also include redundant components such as networking and power supplies. Host clustering allows applications to fail to another host, which might be running applications of its own. Multi-path networking between hosts and storage can improve performance, and eliminate network failures. Local mirrors can provide known data states to continue from after a hardware failure. Remote replication or offsite archiving are the only options to protect against a site or natural disaster. Traditional methods of backup/restore provide the most common means of recovery. More advanced methods such as mirroring and clustering to a remote site provide faster recovery options, but cost more. Oracle approach: Our approach started with looking initially at the causes of downtime, the outages we wanted to handle effectively and then build an architecture with these in mind. Oracle Support recently reviewed 761 recovery tars (User Managed Backup and Recovery Issue Analysis Report For December 2001). The results showed that 46% of the issues were due to data failures or corruptions, most of which were caused by hardware or software failures. 32% of the issues were due to user errors, such as accidentally removing data files or dropping a table. The final 22% required support assistant for resolution. This review indicates that data failures and user errors are likely and you need an architecture to deal with these outages. Automation and simplicity are important components that help achieve service levels and target MTTR. After 9/11, customers are raising the priority for disaster recovery and protection from site disasters or sabotage. We looked at both scheduled outages, used for maintenance purposes, and unscheduled outages. Unscheduled outages fall into three main areas: system failures, such as a system crash; data failures and disasters, where the data is either inaccessible or is unusable in its current state; and human error, which are unavoidable.
  • Fast Restart: New with Oracle9i the DBA can specify time limit for recovery process in seconds with the init.ora parameter fast_start_MTTR_target Oracle9 i dramatically reduces recovery times required for roll forward and makes it bounded and predictable Flashback Query: Feature for recovering from human errors This feature allows users to see a consistent view of the database at a point in the past Oracle Flashback leverages the Undo Management functionality Undo information is kept for a specified retention interval at system level (UNDO_RETENTION initialization parameter) Oracle LogMiner Feature for recovering from human errors Tool that allows log files to be read, analyzed and interpreted by the administrator using SQL GUI and Command Line Interface Online Index Operations like Index rebuild Table Reorganisation Parameters Have Been Made Dynamically Configurable in Oracle9 i They can be Modified with Alter System Commands and Their New Value is Retained in the System Parameter File Shared_Pool_Size Large_Pool_Size (9.2 feature) Open_Cursors Session_Cached_Cursors CPU_Count Log_Checkpoints_To_Alert Local_Listener, Remote_Listener (9.2)
  • © 2004 Hewlett-Packard Development Company, L.P.

    1. 1. Sandy Gruver Senior Technical Consultant HP/Oracle Advanced Technology Center [email_address]   Best Practices for Oracle on HPUX
    2. 2. Topics <ul><li>Installation setup for Oracle on HPUX </li></ul><ul><li>Recommended Kernel parameter settings for Oracle </li></ul><ul><li>Using WLM with Oracle and HPUX partitions </li></ul><ul><li>File System recommendations </li></ul><ul><li>Storage Layout Recommendations </li></ul><ul><li>Backup and Recovery best practices </li></ul><ul><li>Day to day Oracle DBA hints and best practices </li></ul><ul><li>High Availability for Oracle on HPUX </li></ul><ul><li>Monitoring your Oracle environment </li></ul><ul><li>Support for Oracle on HPUX and Oracle information </li></ul>
    3. 3. Before Installing Oracle on HPUX
    4. 4. Brief review HPUX SysAdm tool - SAM
    5. 5. H/W and S/W requirements Overall system requirements <ul><li>Operating System </li></ul><ul><ul><li>HPUX 11.0 and HPUX 11i. </li></ul></ul><ul><ul><li>Oracle 9i onward is only available for 64 bit HPUX </li></ul></ul><ul><li>S/W requirements </li></ul><ul><ul><li>Any X server supported by the UNIX system </li></ul></ul><ul><ul><li>Need to have ar, cc and ld under /usr/ccs/bin </li></ul></ul><ul><ul><li>Need to have Java make installed </li></ul></ul><ul><li>Physical Memory </li></ul><ul><ul><li>Minimum 256 MB for 9i Server (as per release notes). Suggested to have around 4 GB or more </li></ul></ul><ul><li>Swap Space </li></ul><ul><ul><li>Twice the amount of physical Memory or Minimum 400 MB </li></ul></ul><ul><li>Disk Space </li></ul><ul><ul><li>Approx 3 GB for Database S/W and additional space for seed database etc. </li></ul></ul>
    6. 6. <ul><li>Install the necessary HP-UX patches </li></ul><ul><ul><li>Listed in the Oracle Installation Guide </li></ul></ul><ul><ul><li>For patch bundles: </li></ul></ul><ul><ul><ul><li>http ://www.software.hp.com/SUPPORT_PLUS </li></ul></ul></ul><ul><ul><li>For individual patches: </li></ul></ul><ul><ul><ul><li>http://itresourcecenter.hp.com </li></ul></ul></ul><ul><ul><li>To check on patches: </li></ul></ul><ul><ul><ul><ul><li>$ /usr/sbin/swlist -l patch </li></ul></ul></ul></ul><ul><ul><ul><ul><li>$ /usr/sbin/swlist -l patch patch_number </li></ul></ul></ul></ul><ul><ul><ul><ul><li>$ /usr/sbin/swlist -l bundle </li></ul></ul></ul></ul><ul><li>Increase /tmp space to at least 2GB </li></ul><ul><li>Modify Kernel parameters for Oracle </li></ul>H/W and S/W requirements Patches are extremely important
    7. 7. Enhancing performance of Oracle enable Sched_Noage <ul><li>As root, create the file /etc/privgroup </li></ul><ul><ul><ul><ul><li>dba MLOCK (for asynch IO) </li></ul></ul></ul></ul><ul><ul><ul><ul><li>dba RTSCHED RTPRIO (for priority scheduling) </li></ul></ul></ul></ul><ul><li>Issue the commands: </li></ul><ul><ul><ul><ul><li># /usr/sbin/setprivgrp -f /etc/privgroup </li></ul></ul></ul></ul><ul><ul><ul><ul><li># /usr/sbin/getprivgrp dba </li></ul></ul></ul></ul><ul><li>Set the HPUX_SCHED_NOAGE Oracle initialization parameter </li></ul><ul><ul><li>On HP-UX 11.0, the range is 153 to 255 </li></ul></ul><ul><ul><li>On HP-UX 11i, the range is 178 to 255 </li></ul></ul>
    8. 8. Oracle Users and Groups <ul><li>Create UNIX groups ( oinstall, dba ) </li></ul><ul><ul><li># /usr/sbin/groupadd oinstall </li></ul></ul><ul><ul><li># /usr/sbin/groupadd dba </li></ul></ul><ul><li>Create a UNIX account ( oracle ) to own Oracle software </li></ul><ul><ul><li># /usr/sbin/useradd -g oinstall -G dba oracle </li></ul></ul><ul><li>For Oracle10g: Create a Unix account ( extjob ) to own the executable extjob </li></ul><ul><ul><li># /usr/sbin/useradd extjob </li></ul></ul><ul><ul><li>After installing Oracle, make these changes to the extjob file </li></ul></ul><ul><ul><ul><li># cd oracle_home /bin </li></ul></ul></ul><ul><ul><ul><li># mv extjob.nobody extjob </li></ul></ul></ul><ul><ul><ul><li># chown extjob extjob </li></ul></ul></ul><ul><ul><ul><li># chmod 4711 extjob </li></ul></ul></ul>
    9. 9. A Few Important Environment variables <ul><li>ORACLE_HOME </li></ul><ul><li>e.g. export ORACLE_HOME=/oracle/9i_64b </li></ul><ul><li>ORACLE_SID </li></ul><ul><ul><li>e.g. export ORACLE_SID=oratest </li></ul></ul><ul><li>PATH </li></ul><ul><ul><li>e.g. export PATH=$ORACLE_HOME/bin:$PATH </li></ul></ul><ul><li>SHLIB_PATH/LD_LIBRARY_PATH </li></ul><ul><ul><li>e.g. </li></ul></ul><ul><ul><li>export LD_LIBRARY_PATH=$ LD_LIBRARY_PATH:$ORACLE_HOME/lib64 </li></ul></ul>
    10. 10. File and Directory Setup tasks <ul><li>Set the home directory of user oracle to </li></ul><ul><ul><li>$ORACLE_OWNER $HOME directory * </li></ul></ul><ul><li>Set the default shell to /bin/sh for the Bourne shell </li></ul><ul><li>In the .profile, set </li></ul><ul><ul><li>umask 022 </li></ul></ul><ul><ul><li>xhost + </li></ul></ul><ul><li>Check the Oracle Inventory information after installation </li></ul><ul><ul><li># more /var/opt/oracle/oraInst.loc </li></ul></ul><ul><ul><ul><li>inventory_loc=/u01/app/oracle/oraInventory </li></ul></ul></ul><ul><ul><ul><li>inst_group=oinstall </li></ul></ul></ul><ul><li>* See simplified OFA later in presentation </li></ul>
    11. 11. <ul><li>1. Enter the following command as the root user: </li></ul><ul><ul><ul><ul><ul><li># sam </li></ul></ul></ul></ul></ul><ul><li>2. Choose the Kernel Configuration area. </li></ul><ul><li>3. Choose the Drivers area. </li></ul><ul><li>4. Choose the asynchronous disk driver (asyncdsk). </li></ul><ul><li>5. Select Actions>Add Driver to Kernel. </li></ul><ul><li>6. Select List>Configurable Parameters. </li></ul><ul><li>7. Choose the MAX_ASYNC_PORTS parameter. </li></ul><ul><li>8. Select Action>Modify Configurable Parameter. </li></ul><ul><li>9. Specify a new value for the parameter, then choose OK </li></ul><ul><li>Set Oracle initialization parameter DISK_ASYNCH_IO to TRUE. </li></ul>Implementing Asynchronous I/O NOTE: Done ONLY if using RAW (not fs) storage
    12. 12. Implementing Asynchronous I/O
    13. 13. Recommended Kernel parameter settings for Oracle
    14. 14. <ul><li>Specified in the documentation for Oracle </li></ul><ul><li>See “ Oracle Database Installation Guide ” </li></ul><ul><li>To list all kernel parameters </li></ul><ul><ul><li>$ /usr/sbin/kmtune –l |more </li></ul></ul><ul><ul><li>Or use SAM </li></ul></ul><ul><li>To check 32 or 64 bit HP-UX 11.x </li></ul><ul><ul><li>$ /bin/getconf KERNEL_BITS </li></ul></ul><ul><ul><li>$ 64 </li></ul></ul><ul><li>To check 32 or 64 bit Oracle software version </li></ul><ul><ul><li>$ cd $ORACLE_HOME/bin </li></ul></ul><ul><ul><li>$ file oracle </li></ul></ul><ul><ul><li>$ oracle: ELF-64 executable object file - PA-RISC 2.0 (LP64) </li></ul></ul>HP-UX Kernel Settings for Oracle DBs
    15. 15. <ul><li>File System buffer-Cache Parameters: </li></ul>Open or Locked Files Parameters: Recommended Minimum HP-UX Kernel Settings for Oracle DBs between 2 and 5% of memory Min dynamic buffer cache dbc_min_pct between 3 and 10 % of memory Max dynamic buffer cache dbc_max_pct 0 Pages of static buffer cache – older parm bufpages (15 * NPROC + 2048) Open files limit nfile (NPROC) (at least 4096) File lock limit nflocks (8 * NPROC + 2048) Max inodes in memory ninode 1024 (default) Hard limit for open files maxfiles_lim 1024 Soft limit for open files maxfiles
    16. 16. <ul><li>Parameters for Logical Volume Manager: </li></ul>Recommended Minimum HP-UX Kernel Settings for Oracle DBs Parameters for Memory Paging and Variable Page Size: Increase to the number of volume groups you would like to have on the system (maximum 256) Maximum volume groups on the system maxvgs 4096 (up to 65536 for large RAM) Swap chunk size swchunk 64 Max page size in Kbytes vps_ceiling 16384 Max swap space on the system maxswapchunks 1 (on) Enable/disable pseudo-swap swapmem_on
    17. 17. <ul><li>Parameters for Process Management: </li></ul>Recommended Minimum HP-UX Kernel Settings for Oracle DBs 1073741824 bytes or (0x40000000) Max process storage seg size for 64bit procs maxssiz_64bit 256 Max threads/proc max_thread_proc ((NPROC*9)/10) Max # of procs/user maxuprc 128MB Max process text seg size maxtsize (((NPROC * 7) / 4) + 16) Max kernel threads nkthread 4096 Max procs/system nproc 134217728 bytes Max process storage seg size for 32bit procs maxssiz 2147483648 bytes or (0x80000000) Max process data seg size for 64bit procs maxdsiz_64bit 1073741824 bytes or (0x40000000) Max process data seg size for 32bit procs maxdsiz
    18. 18. <ul><li>InterProcessCommunication Message Parameters: </li></ul>Recommended Minimum HP-UX Kernel Settings for Oracle DBs InterProcessCommunication Semaphore Parameters: 32767 # segs in msg queue msgseg nproc Max msg queues msgmni (2+msgmni) Message map size msgmap nproc Total msgs on system msgtql (nproc-4) Max undos/sem semmnu (semmni*2) Max sems for users/sys semmns (semmni+2) Free sem map size semmap 4096 Max sems/system semmni 32767 Max sem value semvmx
    19. 19. <ul><li>InterProcessCommunication Shared Memory Parameters: </li></ul>Recommended Minimum HP-UX Kernel Settings for Oracle DBs Miscellaneous Parameters: 120 per Oracle database Max segments/proc shmseg 512 Max segments/system shmmni Large enough to hold the entire SGA in one shared memory segment Max shared memory segment shmmax max. no of shadow processes + no of parallel query slaves (could go up to nproc) Max ports to asynch driver/system max_async_ports set to number of concurrent Oracle DB users + 64 Max simultaneous users/system maxusers
    20. 20. Updating Kernel Parameters <ul><ul><li>Edit /stand/vmunix </li></ul></ul><ul><ul><li>Backup /stand/system </li></ul></ul><ul><ul><li>Edit/modify /stand/system </li></ul></ul><ul><ul><li>config /stand/system </li></ul></ul><ul><ul><li>? Check that it created vmunix_test in local directory </li></ul></ul><ul><ul><li>Backup /stand/vmunix </li></ul></ul><ul><ul><li>mv /stand/vmunix_test </li></ul></ul><ul><ul><li>/stand/vmunix </li></ul></ul><ul><ul><li>shutdown -r </li></ul></ul>OR Use SAM
    21. 21. Using WLM with Oracle and HPUX partitions
    22. 22. HP Virtual Server Environment: server resource flexing for applications server resource pool security Policy engine provisions resources based on SLOs and business priorities for each application. Reporting Oracle Apps
    23. 23. HP Partitioning Continuum Clustered nodes OS image with HW isolation Virtual partitions vPar hard partition 1 OS image Application 1 with guaranteed compute resources Application 2 with guaranteed compute resources Application n with guaranteed compute resources Based on CPUs or percentages Isolation Highest degree of separation Flexibility Highest degree of dynamic capabilities OS image with HW isolation OS image with HW isolation HP-UX Workload Manager Hard partitions nPar Resources partitions PRM/pSets OS image with SW isolation OS image with SW isolation OS image with SW isolation
    24. 24. vPars logical overview <ul><li>individual “servers” </li></ul><ul><li>different OS revs </li></ul><ul><li>each OS custom-tunable </li></ul><ul><li>dynamic resource allocations </li></ul><ul><li>creates illusion of separate hardware platforms </li></ul><ul><li>manages shared physical resources </li></ul><ul><li>monitors health of operating system instances </li></ul><ul><li>multiple applications, instances or versions </li></ul><ul><li>name space and resource isolation </li></ul>Hardware Platform / Hard Partition vPar Monitor HP-UX Revision A Patch Lvl 1 HP-UX Revision A Patch Lvl 2 HP-UX Revision B Patch Lvl 1 Dept. A App 1 Dept. A App 1’ Dept. B App 2 vPar1 vPar2 vPar3
    25. 25. HP-UX Virtual Partitions <ul><li>Why choose vPars over nPars? </li></ul><ul><ul><li>vPars provides: </li></ul></ul><ul><ul><ul><li>Dynamic processor movement without rebooting the partition </li></ul></ul></ul><ul><ul><ul><li>Single cpu granularity </li></ul></ul></ul><ul><ul><ul><li>Can run within an nPar </li></ul></ul></ul><ul><li>Why choose vPars over resource partitions? </li></ul><ul><ul><li>vPars provides: </li></ul></ul><ul><ul><ul><li>Software fault isolation </li></ul></ul></ul><ul><ul><ul><li>Different versions of the OS </li></ul></ul></ul><ul><ul><ul><li>Application isolation </li></ul></ul></ul>Multiple HP-UX instances running on the same system or in the same nPar rp5470, rp7400, Superdome, rp8400, rp7410, Itanium2 server HP-UX Revision A.1 HP-UX Revision A.2 HP-UX Revision B.3 HP-UX Revision B.3 Dept. A App 1 Dept. A App 1’ Dept. B App 2 Dept. B App 3
    26. 26. <ul><li>System/data center consolidation </li></ul><ul><li>development/test environments </li></ul><ul><li>increased system utilization </li></ul><ul><li>varying workload requirements: </li></ul><ul><ul><li>time of day: order entry during day, batch at night </li></ul></ul><ul><ul><li>time of month (payroll, end-of-month/end-of-year financials </li></ul></ul><ul><ul><li>as particular needs require </li></ul></ul><ul><li>service provider (providing system resources to different users/applications) </li></ul><ul><li>unique application tuning of O/S </li></ul><ul><li>time zoning </li></ul>Where vPars provide the most value
    27. 27. File System recommendations
    28. 28. File system Options <ul><li>Using Online JFS improves performance </li></ul><ul><li>Set these mount options </li></ul><ul><ul><li>delaylog </li></ul></ul><ul><ul><li>nodatainlog </li></ul></ul><ul><ul><li>mincache=direct </li></ul></ul><ul><ul><li>convosync=direct </li></ul></ul><ul><li>Enabling largefiles with Online JFS # fsadm -F vxfs -o largefiles / filesystem </li></ul>
    29. 29. <ul><li>The default Oracle db_block_size on HP-UX is 2048. </li></ul><ul><li>We recommend using: </li></ul><ul><ul><li>db_block_size = 8192 for OLTP Applications </li></ul></ul><ul><ul><li>db_block_size = 8192 to 16384 for DSS/DW Applications </li></ul></ul><ul><li>LVM designed for 8192 blocksize </li></ul><ul><li>Oracle RAC may benefit from a smaller db_block_size to reduce the amount of data to transfer between the nodes for cache fusion </li></ul><ul><li>Must be set when creating database </li></ul>Oracle db_block_size
    30. 30. Data Storage Format Raw or File System? <ul><li>Oracle databases can be implemented either: </li></ul><ul><ul><li>in file systems, or </li></ul></ul><ul><ul><li>in “raw” format </li></ul></ul><ul><li>File Systems: </li></ul><ul><ul><li>overhead (space, CPU, and locking) </li></ul></ul><ul><ul><li>double-buffering & synchronous I/O </li></ul></ul>
    31. 31. Oracle mapping to LVM
    32. 32. Space Layout Raw File System Logical Volume Logical Volume Logical Volume Oracle Data Oracle Data Oracle Data Oracle Data FS overhead empty space FS file FS file
    33. 33. PERCEPTION “ Raw” vs. File System some common misconceptions FACT <ul><li>Raw requires ugly low-level UNIX commands. </li></ul><ul><li>Raw uses LVM just like file systems. </li></ul><ul><li>Need rocket scientists to understand & manage. </li></ul><ul><li>If you manage a FS you already manage raw. </li></ul>
    34. 34. <ul><li>FILE SYSTEM </li></ul><ul><li>create volume group </li></ul><ul><li>create logical volumes </li></ul><ul><ul><li>leave extra room </li></ul></ul><ul><li>‘ newfs’ on logical vols </li></ul><ul><li>mount file systems </li></ul><ul><li>create Oracle files </li></ul><ul><ul><li>tie to fs files </li></ul></ul><ul><ul><li>/orafs/ora/file1.dbf </li></ul></ul>Database Administration Example preparing to create a database <ul><li>RAW </li></ul><ul><li>create volume group </li></ul><ul><li>create logical volumes </li></ul><ul><ul><li>create extra LVs </li></ul></ul><ul><li>create Oracle files </li></ul><ul><ul><li>tie to LVs </li></ul></ul><ul><ul><li>/dev/vgora/rorafile1.dbf </li></ul></ul>
    35. 35. FILE SYSTEM CREATEDB: STARTUP NOMOUNT CREATE CONTROLFILE … DATAFILE '/ora1/data01.dbf' Database Administration Example define oracle data commands RAW CREATEDB: STARTUP NOMOUNT CREATE CONTROLFILE … DATAFILE '/dev/vg9idata/rdata01_2000M.dbf'
    36. 36. FS: OEM + bdf Database Administration Example typical database tasks RAW: OEM MONITOR SPACE ADD MORE SPACE FS: use extra space in FS or create new FS RAW: use extra LVs or create new VGs ACCIDENTALLY DESTROY DATABASE FS: ‘rm -R *’ or ‘newfs’ or ‘lvremove’ RAW: ‘lvremove’
    37. 37. Backup & Recover <ul><li>FILE SYSTEM </li></ul><ul><li>RMAN & (Storage Data Protector) </li></ul><ul><li>fbackup </li></ul><ul><li>tar or cpio or dd </li></ul><ul><ul><li>optionally pipe to compress </li></ul></ul><ul><li>RAW </li></ul><ul><li>RMAN & (Storage Data Protector) </li></ul><ul><li>dd </li></ul><ul><ul><li>optionally pipe to compress </li></ul></ul>RMAN is the preferred method of backing up Oracle RMAN does not distinguish between raw and FS
    38. 38. Summary Raw-based Databases on HP-UX <ul><li>Better performance than HP-UX filesystems </li></ul><ul><ul><li>33-100% better throughput </li></ul></ul><ul><li>No harder to manage than file systems </li></ul><ul><ul><li>same tools & techniques as file-system admin </li></ul></ul>
    39. 39. Storage Layout Recommendations
    40. 40. Storage technology then vs. now
    41. 41. The usual database layout <ul><li>sort database objects by size and expected I/O volume </li></ul><ul><li>implement high-volume objects in striped volumes with various stripe-widths </li></ul><ul><li>squeeze other objects in where they fit </li></ul><ul><li>monitor performance of individual disks and objects </li></ul><ul><li>play ‘chess’ with file placement to find best performance </li></ul>
    42. 42. ‘ SAME’ technique (proposed by Oracle’s Juan Loaiza) <ul><li>S tripe A nd M irror E verything </li></ul><ul><li>stripe all files across maximum # disks </li></ul><ul><li>use 1MB stripe size </li></ul><ul><li>use mirroring for high availability </li></ul><ul><li>place “hot” files on outer edge of disks </li></ul><ul><li>keep it simple </li></ul>
    43. 43. SAME advantages <ul><li>large I/Os minimize impact of disk head movement </li></ul><ul><li>very wide stripe-set allows full I/O throughput capacity to help all transactions </li></ul><ul><li>no need to consider characteristics of individual files/tables/transactions </li></ul>
    44. 44. SAME modified for large storage arrays <ul><li>increase stripe depth to 4-8MB </li></ul><ul><ul><li>implement small critical objects with small stripe </li></ul></ul><ul><li>ignore disk-geometry considerations </li></ul><ul><li>create separate stripesets for different subsets of storage (different RAID levels or disk size/speeds) </li></ul><ul><li>possibly move redo logs to separate device OR use cache LUN </li></ul>
    45. 45. Oracle data organization <ul><li>Objects & Tables </li></ul><ul><ul><li>data, index, etc </li></ul></ul><ul><li>Tablespaces </li></ul><ul><li>Files </li></ul><ul><ul><li>map to space at OS level </li></ul></ul>tablespace tables/objects file file file file ... ‘ logical’ ‘ physical’
    46. 46. Oracle mapping to LVM
    47. 47. PV mapping to the hp xp array Array (RAID) group Array (RAID) group Array (RAID) group EXAMPLE: PV mapping to a disk array tablespace tablespace tablespace oracle LV LV LV PV PV PV PV PV LVM vol. grp. LUN LUN LUN LUN LUN LUN LUN LUN LUN array ...
    48. 48. Implementing SAME <ul><li>decide “stripeset width” (# disks or array groups) </li></ul><ul><ul><li>four to eight groups recommended </li></ul></ul><ul><li>create one volume group per stripeset </li></ul><ul><li>create multiple logical volumes per VG </li></ul><ul><li>(create filesystems on top of LVs) </li></ul><ul><li>allocate Oracle objects among stripesets </li></ul><ul><li>map Oracle files to LVs or to filesystem files </li></ul>
    49. 49. <ul><li>create LVs using LVM ‘extent striping’ </li></ul><ul><ul><li>use 4MB or 8MB extents </li></ul></ul><ul><li>divide objects evenly among VGs </li></ul><ul><ul><li>keep each object wholly contained in a VG </li></ul></ul><ul><li>for raw I/O </li></ul><ul><ul><li>create standard-sized LVs </li></ul></ul><ul><ul><li>use symbolic links in Oracle to point to file locations </li></ul></ul>Implementing SAME
    50. 50. <ul><ul><li>Special Considerations for Oracle Redo Logs </li></ul></ul><ul><li>Online logs deserve very special attention </li></ul><ul><li>multiplex logs in addition to RAID </li></ul><ul><li>locate members so they share no common points of failure at the disk, channel, or board level </li></ul><ul><li>where possible, locate on dedicated (or otherwise quiet) disk spindles </li></ul><ul><li>Possible downside effects </li></ul><ul><li>excess of disk capacity to meet all these constraints </li></ul><ul><li>isolation to single disks or LUNs may not be possible </li></ul><ul><li>peak throughput is reduced </li></ul>
    51. 51. Monitor performance & balance I/O <ul><li>look for high I/O wait states on shadow processes </li></ul><ul><li>monitor volume group I/O rates </li></ul><ul><li>move Oracle files (if necessary) to balance I/O </li></ul>
    52. 52. EXAMPLE: Performance advisor/xp <ul><li>monitors </li></ul><ul><li>LDEV I/O </li></ul><ul><li>port utilization </li></ul><ul><li>internal bus utilization </li></ul><ul><li>cache usage </li></ul>
    53. 53. Optimal Simplified Architecture a variation on OFA * * See Oracle® Database Installation Guide for more information <ul><li>The Optimal Simplified Architecture (OSA) recommends three main directories: </li></ul><ul><li>$ORACLE_OWNER $HOME directory </li></ul><ul><ul><li>environment files, configuration files, administrative scripts, etc. This directory gets taken forward with Oracle version upgrades. </li></ul></ul><ul><li>$ORACLE_HOME $ORACLE_BASE directories </li></ul><ul><ul><li>Oracle product bits. The $ORACLE_HOME directory changes with each release of Oracle. </li></ul></ul><ul><li>$ORACLE_DATA directory </li></ul><ul><ul><li>hot database files, such as DBF files and archive log files which should not be backed up with normal file system backups. </li></ul></ul><ul><li>Also recommend that each $ORACLE_SID be under a unique $ORACLE_OWNER </li></ul>
    54. 54. <ul><li>$ORACLE_HOME and $ORACLE_SID </li></ul><ul><li>uniquely identify an Oracle server </li></ul><ul><ul><li>A user can change these operating system variables to work with a different version of the software or a different instance. </li></ul></ul><ul><ul><li>Table space names should be descriptive and restricted to eight characters plus a two-digit identifier </li></ul></ul><ul><ul><li>Table space names should be used in the name of the data file name. </li></ul></ul>Optimal Simplified Architecture a variation on OFA
    55. 55. Optimal Simplified Architecture Performance Benchmark File I/O Stats BEFORE AFTER GAIN Filename mS/ mS/ mS/ mS/ %age %age Read Write Read Write Read Write /opt/oracle/u05/thadtbs1.dbf 0.09 16.58 0.09 12.45 24% /opt/oracle/u05/thadtbs2.dbf 0.21 15.42 0.19 13.68 9% 11% /opt/oracle/app/ti1tbs.dbf 0.09 12.08 0.09 9.16 24% /opt/oracle/u02/ti2tbs.dbf 0.09 16.34 0.09 11.16 32% /opt/oracle/u03/ti3tbs.dbf 0.12 12.69 0.12 10.54 17% /opt/oracle/u03/ti4tbs.dbf 2.75 20.39 1.61 15.30 41% 25% /opt/oracle/u01/tttdbs.dbf 6.80 9.61 4.86 6.98 29% 27%
    56. 56. Summary <ul><li>stripe everything equally& simply </li></ul><ul><li>use broad stripe size to balance overall I/O </li></ul><ul><li>add capacity in broad stripeset units </li></ul><ul><li>implement using OSA </li></ul><ul><li>monitor performance to ensure balance </li></ul>
    57. 57. Backup and Recovery best practices
    58. 58. HP Recommends <ul><li>Use standard tools utilizing HP’s Data Protector (DP) and Oracle’s Recovery Manager (RMAN) </li></ul><ul><li>Define standard backup/recovery processes utilizing standard tools </li></ul><ul><li>Clearly identify support organization’s roles and responsibilities for database backup and recoveries </li></ul><ul><li>Simplify/standardize backup/recovery procedures </li></ul><ul><li>The overall goal is to reduce support costs </li></ul>
    59. 59. HP OpenView Storage Data Protector 5.1 Features <ul><ul><li>enterprise data protection that automates routine tasks and ensures recovery from any potential disruption </li></ul></ul><ul><ul><li>distributed architecture with centralized control </li></ul></ul><ul><ul><li>the first to integrate disk- and tape-based recovery in a single product </li></ul></ul>for maximum protection at the lowest cost
    60. 60. Newest features in Data Protector 5.1 <ul><li>Enhanced functional capabilities for more control: </li></ul><ul><li>Easy to use Oracle restore GUI allowing administrators to select all or individual RMAN restore options </li></ul><ul><li>Simplified SAN auto-configuration wizard automatically detecting and configuring the backup drives in the SAN </li></ul><ul><li>Increased resilience with recovery from any disruption </li></ul><ul><li>Industry’s first fully-integrated Zero-Downtime backup solution for HP StorageWorks EVA and XP arrays </li></ul><ul><li>Integrated Instant Recovery capability for HP StorageWorks EVA and XP customers enabling the recovery of even terabytes of data in minutes </li></ul>
    61. 61. <ul><ul><li>simple GUI </li></ul></ul><ul><ul><li>complete end-to-end management </li></ul></ul><ul><ul><li>fewer staff managing more systems </li></ul></ul><ul><ul><li>reporting on service levels </li></ul></ul><ul><ul><li>portal for providing status info </li></ul></ul><ul><ul><li>integration with Oracle Recovery Manager (RMAN) for zero-downtime backup </li></ul></ul>Data Protector (Omniback) service-driven management approach enterprise-wide protection management
    62. 62. s recovery image on tape Page data protector automates scheduling & replication of images to optimize storage capacity automatic management & re-cycling of recovery images recovery point objective (how old is the data to which I recover?) within past hour 12 hours 2 hours 3 hours midnight 11:55 a.m. incident time recovery images on disk
    63. 63. EXAMPLE: Oracle9i Backup using Data Protector with Business Copy on HP storage 1. Running Environment “ Zero Downtime Backup” 2. Backup <ul><li>Freeze point in time on the disks and split the mirror </li></ul><ul><li>Unfreeze the disks and perform backup from the mirror </li></ul>S T S T <ul><li>Resynchronize the mirror </li></ul>S T
    64. 64. EXAMPLE: Oracle9i RAC Restore using Data Protector with Business Copy on HP storage S T 1. Database Crash 2. Split the pair S T 3. Restore from tape S T 4. Restore from T to S-Vol S T
    65. 65. Restoring the data takes time minutes several hours how long does it take? Recovery Time Objective tape backup Instant Recovery
    66. 66. Data Protector - Instant Recovery A new approach <ul><li>Fast, automated restore directly from the T-Vol </li></ul><ul><li>Data is instantly available without any data neccessary to be moved </li></ul><ul><li>Neither host involved in data processing </li></ul><ul><li>The backup version will be accessible instantaneously and production can start immediately </li></ul>S T T S
    67. 67. S T Data Protector - Instant Recovery How to use it <ul><li>Using Zero Downtime Backup (ZDB) as a basic concept </li></ul><ul><li>The link stays split after the backup </li></ul><ul><li>This provides a version for fast recovery from disk </li></ul><ul><li>Restore is done automatically by Data Protector </li></ul><ul><li>No manual mirror handling required </li></ul>Running Split Backup Prepare Recoverable Version between Backup and Prepare S T S T S T S T 1 pm 1:15 pm 3 pm 12:30 pm tape
    68. 68. Rotate the Mirrors <ul><li>Using up to 3 different mirrors of XP Business Copy </li></ul><ul><li>Allow instant access to earlier version (t0) at a future time (t2) </li></ul><ul><ul><li>i.e. Restore on Wednesday a version of Monday within minutes </li></ul></ul><ul><li>Data Protector automatically manages the separate mirrors </li></ul><ul><li>Define a policy on how the mirror for the next backup should be prepared </li></ul><ul><ul><li>After the backup finished </li></ul></ul><ul><ul><li>Before the next backup starts </li></ul></ul>S t 0 S t 0 t 1 S t 0 t 1 t 2 S t 0 t 1 t 2
    69. 69. Maintain Version on Disk only <ul><li>The data movement to tape is optional </li></ul><ul><ul><li>Only one version would go to tape (I.e. t1 ) </li></ul></ul><ul><li>In the time t1 is moved to tape other versions could be created ( t2 & t0 ) </li></ul><ul><li>Therefore the tape throughput no longer dictates the backup frequency </li></ul><ul><li>More version could be created in a shorter time </li></ul><ul><li>DB recovery is drastically improved </li></ul>P t 0 P t 0 t 1 P t 0 t 1 t 2 P t 0 t 1 t 2 tape
    70. 70. data protector breaks the link between disk-based protection and tape data protector instant recovery <ul><li>administrators can choose - disk-only protection, - tape-only protection, or - scheduled combinations depending on the business need for a particular application </li></ul><ul><li>once configured, Data Protector fully automates the continuous protection process, including rotation of mirrors </li></ul><ul><li>for recovery, administrator selects the specific recovery image from the GUI </li></ul>production data point-in-time copies (split mirrors or snapshots) t application server backup device server management system t 0 t - 2 t - 1 tape
    71. 71. Recovery Manager (RMAN) features <ul><li>Standard tool supplied and supported by Oracle Corporation </li></ul><ul><li>Backups are done at block level, meaning no need for tablespace backup mode, less overhead </li></ul><ul><li>No extra redo generated during online backup </li></ul><ul><li>Ability to do incremental database backups </li></ul><ul><li>Database and archived log backups are both handled by RMAN </li></ul><ul><li>Corrupt block detection </li></ul><ul><li>Tablespace point in time recovery support </li></ul><ul><li>Ability to do block level recovery (this addresses recovering from corrupt blocks) </li></ul><ul><li>Support for RAW devices </li></ul><ul><li>Support for Real Application Clusters (RAC) </li></ul><ul><li>Complete integration with HP OpenView Data Protector </li></ul>
    72. 72. RMAN vs. OS Backup Feature RMAN Op Sys Offline Backups supported supported Corrupt Block detection supported requires Begin/End Backup stmts. Automatic backup supported not supported Backup catalogs supported not supported Online Backups supported not supported
    73. 73. RMAN/Data Protector Database Backup and Recovery Test Results <ul><li>Test setup </li></ul><ul><li>200GB test database </li></ul><ul><li>backups and recoveries were done using different numbers of tape drives and RMAN channels </li></ul><ul><li>Results </li></ul><ul><li>best database restore performance </li></ul><ul><ul><li>one channel per tape drive during the backup </li></ul></ul><ul><ul><li>double the number of channels during the restore </li></ul></ul><ul><li>Recommendation </li></ul><ul><li>allocate 1 or 2 RMAN channels per tape drive to be used for the backup </li></ul><ul><li>allocate twice as many RMAN channels for the recovery </li></ul>
    74. 74. Backup/Recovery Suggested Responsibilities <ul><li>DBA group responsible for RMAN catalog setup and maintenance </li></ul><ul><li>DBA group responsible for contacting Backup team for backup configuration and scheduling </li></ul><ul><li>RMAN/DP Backups handled by Backup team </li></ul><ul><li>Backup failures and monitoring handled by Backup team </li></ul><ul><li>Recovery responsibility can be handled by DBA group (using RMAN scripts) </li></ul><ul><li>Recovery responsibility will be migrated to Backup team (Data Protector V5.1 Oracle Restore GUI training needed in Backup team) </li></ul>
    75. 75. Other backup considerations <ul><li>RMAN Catalog management and backup strategy </li></ul><ul><li>RMAN/Data Protector training </li></ul><ul><li>Identification of roles/responsibilities of Database and Backup teams in overall backup/recovery process </li></ul><ul><li>Identification of escalation process to Oracle of RMAN problems/bugs </li></ul><ul><li>Identification of escalation process to HP of Data Protector problems/bugs </li></ul>
    76. 76. Another option for Data Replication Use of Quest SharePlex <ul><li>Activate the SharePlex configuration file on the current production database </li></ul><ul><li>Create the interim database using a hot backup and SharePlex Overdrive </li></ul><ul><li>Activate the config file on the backup database </li></ul><ul><li>Begin posting </li></ul>
    77. 77. Another option for Data Replication Use of Quest SharePlex SharePlex <ul><li>Step one is to establish the working database with minimal impact on </li></ul><ul><li>User activity. </li></ul><ul><ul><li>Activate SharePlex configuration file </li></ul></ul><ul><ul><li>Create interim database using a hot-backup and SharePlex Overdrive </li></ul></ul>HP-UX HP-UX
    78. 78. Day to day Oracle DBA hints and best practices
    79. 79. <ul><ul><li>SQLPlus Tips </li></ul></ul><ul><ul><li>simple formatting commands </li></ul></ul><ul><li>SQLPlus - a command line reporting tool </li></ul><ul><ul><li>$ sqlplus ‘/ as sysdba’ </li></ul></ul><ul><ul><li>SQL> select table_name from dba_tables; </li></ul></ul><ul><li>Report formatting commands </li></ul><ul><ul><li>alter line lengths (in characters) </li></ul></ul><ul><ul><li>alter page lengths (in lines) </li></ul></ul><ul><ul><li>alter column widths and styles </li></ul></ul><ul><ul><li>suppress the display of duplicate column rows </li></ul></ul>
    80. 80. <ul><ul><li>SQLPlus Example </li></ul></ul><ul><ul><li>example SQL query </li></ul></ul>Give me a list of all tables with their size and tell me if they’ve been backed up. select owner, table_name, tablespace_name, num_rows, backed_up from dba_tables order by owner, tablespace_name
    81. 81. <ul><ul><li>SQLPlus Example </li></ul></ul><ul><ul><li>without formatting commands </li></ul></ul>
    82. 82. <ul><ul><li>SQLPlus Examples setting formatting commands </li></ul></ul>
    83. 83. <ul><ul><li>SQLPlus Example </li></ul></ul><ul><ul><li>with column formatting </li></ul></ul>
    84. 84. <ul><ul><li>SQLPlus Examples of </li></ul></ul><ul><ul><li>simple formatting commands </li></ul></ul><ul><li>1. Alter line lengths </li></ul><ul><li>SQL> SET LINESIZE 70 </li></ul><ul><li>2. Alter page length </li></ul><ul><li> SQL> SET PAGESIZE 65 </li></ul><ul><li>3. Alter column widths and styles. </li></ul><ul><ul><ul><li>SQL> COLUMN surname FORMAT A15 </li></ul></ul></ul><ul><ul><ul><li>SQL> COLUMN surname FORMAT A15 TRUNC </li></ul></ul></ul><ul><ul><ul><li>Display only the first 15 characters of the column. </li></ul></ul></ul><ul><ul><ul><li>SQL> COLUMN surname FORMAT A15 WRAP </li></ul></ul></ul><ul><ul><ul><li>Display multiple lines, splitting after the 15th character on each line </li></ul></ul></ul><ul><li>For a list of column commands </li></ul><ul><ul><ul><li>SQL> HELP COLUMN </li></ul></ul></ul>
    85. 85. <ul><ul><ul><li>Advice for the DBA </li></ul></ul></ul>DAILY PROCEDURES Verify all instances are up Look for any new alert log entries Verify success of database backup Verify success of database archiving to tape Verify enough resources for acceptable performance Copy Archived Logs to Standby Database and Roll Forward Read DBA manuals for one hour
    86. 86. <ul><ul><ul><li>Advice for the DBA </li></ul></ul></ul>NIGHTLY PROCEDURES Collect volumetric data WEEKLY PROCEDURES Look for objects that break rules Look for security policy violations Look in SQL*Net logs for errors, issues Archive all Alert Logs to history Visit home pages of key vendors
    87. 87. <ul><ul><ul><li>Advice for the DBA </li></ul></ul></ul>MONTHLY PROCEDURES Look for Harmful Growth Rates Review Tuning Opportunities Look for I/O Contention Review Fragmentation Project Performance into the Future Perform Tuning and Maintenance
    88. 88. Example: Hardware for a Customer Oracle Installation
    89. 89. High Availability for Oracle on HPUX
    90. 90. Leading Causes of Data Loss Software Malfunction 14% Viruses 7% Natural Disasters 3% Hardware or System Malfunction 44% Human Error 32% Source: Understanding Data Loss . CBL Data Recovery Technologies Inc. Industry Sources – Data Recovery Report
    91. 91. HP & Oracle handle all causes of downtime Planned Downtime Unplanned Downtime Database Maintenance System Maintenance Human Error Data Failure & Disaster System Failure Online Redefinition, Partitioning, Parallel SQL Dynamic reconfiguration (Patches/Drivers) + MC/SG or RAC Flashback Query, LogMiner DataGuard, Split Mirror 9i RMAN & HP Data Protector integration / Data Guard / ... MC/SG or RAC on HP, Fast Restart
    92. 92. HP ServiceGuard cluster <ul><li>Non-Shared Database </li></ul><ul><li>Provides 16 node failover solution </li></ul>Disk A Disk B Oracle Oracle SystemA SystemB Virtual Server with virtual IP Address <ul><li>Before Failover: </li></ul><ul><li>Virtual IP Address and network name refers to System A </li></ul><ul><li>After Failover: </li></ul><ul><li>Virtual IP address and network name refers to System B </li></ul>
    93. 93. HP ServiceGuard cluster <ul><li>Balance workload after a node failure </li></ul><ul><li>Minimize impact on remaining nodes </li></ul>Node 4 Pkg C Pkg H Pkg I Node 2 Pkg A Pkg D Pkg E Node 3 Pkg B Pkg F Pkg G If Node 1 fails... Node 1 Pkg A Pkg B Pkg C
    94. 94. HP ServiceGuard Software Stack HP-UX 11.x HP MC/ServiceGuard HP MC/ServiceGuard Application 2 Database for Oracle Instance 1 Active Active Oracle Instance 1 Application 2 Storage Exclusive Access Exclusive Access failover HP-UX 11.x
    95. 95. Limits of Cold Failover Clusters <ul><li>Scalability of cluster is limited to scalability of one server </li></ul><ul><li>Load cannot be distributed across all nodes in the cluster </li></ul><ul><li>Cold failover is slow, as many time consuming tasks must be performed as part of failover </li></ul><ul><ul><li>moving and mounting logical volumes </li></ul></ul><ul><ul><li>starting the oracle instance </li></ul></ul><ul><ul><li>opening the data files </li></ul></ul><ul><li>After failover, the instance caches are cold introducing a performance brownout </li></ul>
    96. 96. HP-UX 11.x HP ServiceGuard Extension for RAC HP-UX 11.x HP ServiceGuard Extension for RAC Shared Database for Oracle Instances 1 & 2 Oracle9i EE + RAC Instance 1 Oracle9i EE + RAC Instance 2 Shared Access Shared Access Oracle9i RAC with SG Extension for RAC Cluster Interconnect Redo Redo Redo Thread1 Redo Redo Redo Thread2
    97. 97. Oracle9i RAC Architecture in Detail Global Cache Service USER SGA Node 1, Instance A USER USER DBWR LGWR Database Buffer Cache Redo Log Buffers LCK0 Data Data Data Data USER SGA Node 2, Instance B USER USER DBWR LGWR Database Buffer Cache Redo Log Buffers LCK0 Redo Redo Redo Redo 9i Cache Fusion 9i Cache Fusion Control Redo Thread2 Redo Thread1
    98. 98. Monitoring your Oracle environment
    99. 99. Monitoring with OpenView Operations
    100. 100. Monitoring with Oracle Enterprise Manager
    101. 101. Support for Oracle on HPUX and Oracle information
    102. 102. Joint HP/Oracle Support Simpler, faster problem resolution keeps your business on-line and your staff on-schedule Together, we solve your problem. Reactive Call either HP or Oracle for any interoperability problem Proactive HP and Oracle provide coordinated assessments and optimization
    103. 103. Getting Help – HP’s IT Resource Center (ITRC)
    104. 104. Where to go for more information: http://w ww.hp.com/go/oracle
    105. 105. Where to go for more information: http://otn.oracle.com http://metalink.oracle.com
    106. 106. Where to go for more information: http://www.optimaldba.com/library.html http://www.orapub.com/ http://searchoracle.techtarget.com/ Oracle websites external to HP and Oracle
    107. 107. ACM Transactions on Database Systems (TODS) Canadian Information Processing Society Computer Measurement Group DAMA International IEEE Computer Society International Oracle Users Group NaSPA Home Page Object Database Management Group Home Page Object Management Group (OMG) The OLAP Council Software Publishers Association Software Research SQL Standards Home Page TDWI - The Data Warehouse Institute Transaction Processing Performance Council (TPC) Home XML.com Database-Related Organizations with web sites