State of North Carolina
DB2 for z/OS Utilities Topics
August 27th, 2008

Craig Friske, IBM, friske@us.ibm.com




        ...
Agenda
            Usability with Listdef and Templates
            COPY/RECOVER/QUIESCE/MODIFY
            REORG
        ...
Wildcarding/Allocation Overview
Improved utility processing for groups of objects
 –Avoid job stream changes when applicat...
Wildcarding using LISTDEF
List Example: Process objects x, y, and z in database DBA
With V6
QUIESCE TABLESPACE DBA.X TABLE...
Dynamic Allocation Templates
 Template example:
  – Allocate image copy datasets (2 local, 2 remote)
 V6 - 4 DD statements...
Utilities supported
Utility             LISTDEF   TEMPLATE
CHECK DATA          No        Yes
CHECK INDEX         Yes      ...
Automatic Space Management
 Sliding secondary allocation quantity size
 Applies to DB2 managed pagesets only
 Tries to avo...
COPY Best Practices
            COPY
            – PARALLEL keyword provides parallelism for lists of objects (including
 ...
RECOVER/QUIESCE Best Practices
              RECOVER
              – PARALLEL keyword provides parallelism for lists of ob...
MODIFY RECOVERY Best Practices
           Run MODIFY RECOVERY regularly to clean up old
           records in SYSCOPY and ...
Why do we need to REORG ?
 1)   Reclaim fragmented space, optimize space utilization
 2)   Improve access performance by o...
Online REORG in a Nutshell
                                     Original TS/IXs

                          Reorg
         ...
Use of Log Records in OLR
Step 1     Record current log position, reorg into shadow copy, record new log position
        ...
Reorg Shrlevel Change

            Unload   Reload   Sort   Build     Log               Switch

             UTRW     UTRW...
V9 REORG Partition Parallelism
              REORG TABLESPACE TS
              Three partitions, one PI, one NPSI

       ...
REORG Best Practices
            REORG
            – Use SHRLEVEL REFERENCE or SHRLEVEL CHANGE
            – Inline COPY &...
REORG Best Practices
           REORG
           – Partition parallelism in DB2 9 and NPI processing
             • Parall...
REORG Best Practices continued
             REORG
             – SORTDATA NO only if data is already in or near
          ...
REORG SHRLEVEL CHANGE BPs
              REORG SHRLEVEL CHANGE (sometimes called online REORG)
                •   TIMEOUT ...
Automatically display CLAIMERS when
REORG receives resource unavail
After
DSNUGUTC - REORG TABLESPACE DB1.TS1 COPYDDN(SYSC...
REORG SHRLEVEL CHANGE contd.
            REORG SHRLEVEL CHANGE
             – Consider scheduling SWITCH phase in a mainte...
Real-time Statistics
Introduced in V7
Contain “space” and some “accesspath” statistics in
user-defined tables:
 – SYSIBM.T...
RTS
SYSTABLESPACESTATS                                       SYSINDEXSPACESTATS
–   Global                   Incremental  ...
Utilities On Demand
            Run utilities only when necessary and not on fixed
            schedules
            Infor...
Reorg table space V9 Recommendations
Consider running REORG TABLESPACE in the following situations:
 – Real-time statistic...
Reorg table space V9 recommendations
Consider running REORG TABLESPACE in the following situations:
 – Real-time statistic...
Reorg table space V9 recommendations
Consider running REORG TABLESPACE in the following situations:
 – Real-time statistic...
Reorg Index recommendations
 Consider running REORG INDEX in the following cases:
  – Real-time statistics (SYSINDEXSPACES...
RUNSTATS Best Practices
           RUNSTATS
           – SHRLEVEL CHANGE for availability
           – Collect only column...
RUNSTATS KEYCARD
Collects all of the distinct values in all of the 1 to n key column combinations for the specified
indexe...
When is RUNSTATS needed?
When the data changes sufficiently to warrant
new statistics
–REORG of table space or index (use ...
LOAD Best Practices
               LOAD
               – LOG NO reduces log volume; if REPLACE, then take inline copy
    ...
LOAD Best Practices continued
            LOAD
            – Inline COPY & Inline STATISTICS
            – Use REUSE to lo...
REBUILD INDEX Best Practices
           REBUILD INDEX
           – Indexes are built in parallel
           – Remove SORTW...
More online utilities
           REBUILD INDEX SHRLEVEL CHANGE
               – Great for building new non-unique indexes ...
DB2 Allocated Sort Work Data Sets
            PTFs shipped 02/2008 to enable DB2 to dynamically allocate sort
            ...
Checking all indexes in parallel
                   TS       TS          TS           IX1        IX1      IX1     IX2     ...
CHECK SHRLEVEL CHANGE design
            SHRLEVEL REFERENCE CHECK INDEX causes data and
            indexes to be unavaila...
DB2 9 Utilities Highlights
             More online utilities
              – Rebuild Index SHRLEVEL CHANGE
              ...
DB2 9 Utilities Highlights…
             MODIFY Recovery enhancements
              – “Retain” keyword added to improve ma...
More online utilities (cont…)
                  CHECK DATA SHRLEVEL CHANGE
                  CHECK LOB SHRLEVEL CHANGE
   ...
DB2 9 Utilities
            Support for all new functions in DB2 9 for
            z/OS
             – Universal Table Spa...
Copy Utility Changes
            Always perform CHECKPAGE on the COPY utility
            The COPY utility includes SCOPE ...
Recover Utility Changes
                  Recovery to a point in time with consistency (NFM mode)
                   – Unc...
Recover to a Point In Time (cont…)
             Recovery to a point in time with consistency (contd)
              – Two n...
BACKUP/RESTORE SYSTEM
                What’s new in DB2 9 for BACKUP and RESTORE
                SYSTEM?
                 ...
Future enhancements

                   Simplification, automation, availability, performance, flexibility
               ...
Future enhancements

                   Simplification, automation, availability, performance, flexibility
               ...
References
DB2 for z/OS home page:
http://www-306.ibm.com/software/data/db2/zos/
                DB2 9 for z/OS Technical ...
DB2 for z/OS Information Resources
Information Management Software for z/OS Solutions Information
Center
       http://pub...
Thank you!



 Craig Friske – friske@us.ibm.com
 DB2 for z/OS Utilities Development

 IBM Silicon Valley Lab




         ...
Disclaimer
     © Copyright IBM Corporation 2008. All rights reserved.
     U.S. Government Users Restricted Rights - Use,...
Upcoming SlideShare
Loading in...5
×

Microsoft PowerPoint - STATENC.ppt [Read-Only]

666

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
666
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
18
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Microsoft PowerPoint - STATENC.ppt [Read-Only]

  1. 1. State of North Carolina DB2 for z/OS Utilities Topics August 27th, 2008 Craig Friske, IBM, friske@us.ibm.com 1
  2. 2. Agenda Usability with Listdef and Templates COPY/RECOVER/QUIESCE/MODIFY REORG RTS RUNSTATS LOAD Parallel index build © 2008 IBM Corporation We will cover the best practices, including keyword specifications recommended for those utilities that are most significant within the utilities suite because they are used often or may consume large amounts of resources. For several of these utilities, building indexes is an important part of processing, so we will cover some of the details of that process to see how is may affect space needs and performance. Sorting is a critical part of building indexes, and it’s used at various times for many for other phases of utilities as well. DFSORT is always used as of DB2 V8, so we will look at some of the characteristics and recommendations for DFSORT. This involves using dynamic allocation of sort work data sets and why you should let DB2 manage those data sets when possible. Along the way there will be a few samples to better understand the concepts. 2
  3. 3. Wildcarding/Allocation Overview Improved utility processing for groups of objects –Avoid job stream changes when application changes –Supports table spaces, index spaces, or both Generate object lists –LISTDEF statement to generate a list of objects wildcarding (e.g. dbname*) explicit lists INCLUDE or EXCLUDE objects Dynamically allocate datasets –TEMPLATE statement to define datasets intelligent dataset sizing variables (Job, Utility, Object, Date, Time, ... etc.) Control Statements –OPTIONS PREVIEW, EVENT (halt, skip, RC) © 2008 IBM Corporation 3
  4. 4. Wildcarding using LISTDEF List Example: Process objects x, y, and z in database DBA With V6 QUIESCE TABLESPACE DBA.X TABLESPACE DBA.Y TABLESPACE DBA.Z COPY TABLESPACE DBA.X COPYDDN (C1,C2) RECOVERYDDN(C3,C4) TABLESPACE DBA.Y COPYDDN (..,..) RECOVERYDDN(..,..) TABLESPACE DBA.Z COPYDDN (..,..) RECOVERYDDN(..,..) With V7 LISTDEF X INCLUDE TABLESPACE DBA.* EXCLUDE TABLESPACE DBA.A TEMPLATE T1 DSNAME((&DB..&TS..&LOCREM.&PRIBAC..D&JDATE.) QUIESCE LIST X COPY LIST X COPYDDN(T1,T1) RECOVERYDDN(T1,T1) © 2008 IBM Corporation 4
  5. 5. Dynamic Allocation Templates Template example: – Allocate image copy datasets (2 local, 2 remote) V6 - 4 DD statements for each object //C1 DD DSN=DBA.X.LP.D20020409.,UNIT=..,VOL=SER=..,DISP= //C2 DD DSN=DBA.X.LB.D20020409.,UNIT=..,VOL=SER=..,DISP= //C3 DD DSN=DBA.X.RP.D20020409.,UNIT=..,VOL=SER=..,DISP= //C4 DD DSN=DBA.X.RB.D20020409.,UNIT=..,VOL=SER=..,DISP= //SYSIN DD * V7 - a single template for local and remote //SYSIN DD * TEMPLATE A DSNAME(&DB..&TS..&LOCREM.&PRIBAC..D&DATE.) Generated datasets DSN=DBA.X.LP.D2002099 April 8th Local Primary for DBA.X DSN=DBA.X.LB.D2002099 DSN=DBA.X.RP.D2002099 DSN=DBA.X.RB.D2002099 © 2008 IBM Corporation 5
  6. 6. Utilities supported Utility LISTDEF TEMPLATE CHECK DATA No Yes CHECK INDEX Yes Yes CHECK LOB No Yes COPY Yes Yes COPYTOCOPY Yes Yes LOAD No Yes MERGECOPY Yes Yes MODIFY RECOVERY Yes n/a MODIFY STATISTICS Yes n/a QUIESCE Yes n/a REBUILD INDEX Yes Yes RECOVER Yes n/a REORG INDEX Yes Yes REORG TABLESPACE Yes Yes REPORT Yes n/a RUNSTATS Yes n/a UNLOAD Yes Yes © 2008 IBM Corporation 6
  7. 7. Automatic Space Management Sliding secondary allocation quantity size Applies to DB2 managed pagesets only Tries to avoids VSAM maximum extent limit errors Can reach maximum dataset size before running out of extents Uses cylinder allocation – Default PRIQTY • 1 cylinder for non-LOB tablespaces and indexes • 10 cylinders for LOB tablespaces – Improved SQL SELECT and INSERT performance • 2x improvement relative to track allocation Can be used for – New pagesets: No need for PRIQTY/SECQTY values – Existing pagesets: SQL ALTER PRIQTY/SECQTY values to -1 plus schedule a REORG © 2008 IBM Corporation 7
  8. 8. COPY Best Practices COPY – PARALLEL keyword provides parallelism for lists of objects (including partitions) – CHECKPAGE YES (default in DB2 9, no longer sets copy pending, look for RC=8!) – Maximize other utilities’ access to objects while copying a list with SHRLEVEL CHANGE and OPTIONS EVENT(ITEMERROR,SKIP) • Keeps objects in the list in UTRW state *only* as each object is being copied instead of for the duration of the COPY utility • UTRW – utility allows read/write access by applications, but no access for exclusive utilities – Incremental copy rule-of-thumb: Consider using incremental image copy if • <5% of pages are randomly updated (typically means less than 1% of rows updated) • <80% of pages are sequentially updated • Incremental image copies use list prefetch, so monitor for rid list pool full conditions – Copy indexes on your most critical tables to speed up recovery MERGECOPY – Consider using it © 2008 IBM Corporation PARALLEL must be specified with a list to provide parallel backing up of objects. It can also be used to limit the degree of parallelism if desired, otherwise the COPY utility will determine the optimal number of parallel tasks itself. CHECKPAGE YES is recommended to ensure consistency of the data. After all, one doesn’t want to RECOVER with a bad copy of the data! In addition, since COPY is I/O bound and CHECKPAGE adds CPU, the cost in elapsed time is negligible. CHECKPAGE YES is the default in DB2 9 and it will no longer set the object into copy pending to indicate that problems were found but remember this in a new SYSCOPY record and set RC=8. Look for messages DSNU441I for space map errors or DSNU518I for other errors and check the return codes. Specifying OPTIONS EVENT(ITEMERROR,SKIP) in the same jobstep as a COPY SHRLEVEL CHANGE will prompt the Copy Utility to set the UTRW utility-in- progress state on each object in the list only as it is being copied. This increases the availability of the objects in the list since Utilities that require exclusive use of an object are not allowed when the object is in UTRW status. Two examples of Utilities that may require exclusive use of objects are the Load and Reorg Utilities. 8
  9. 9. RECOVER/QUIESCE Best Practices RECOVER – PARALLEL keyword provides parallelism for lists of objects (including partitions) – Compressed pagesets result in faster restore phase – Enable Fast Log Apply (which can use dual-copy logs) and PAV – <=10 jobs/member with LOGAPSTG=100MB, 20-30 objects per RECOVER – For recovery to a prior point in time • Always recover related sets of objects together (same RECOVER utility statement) – Always run REPORT RECOVERY utility on each object and determine the recovery base before running the Recover Utility – DB2 9 for z/OS: recover to PIT with consistency • Backs out uncommitted changes for the objects specified on the RECOVER utility statement • Significantly reduces the need to run QUIESCE, which can be disruptive to applications QUIESCE – WRITE NO is less disruptive (no quiescing of COPY=NO indexes) – Use TABLESPACESET © 2008 IBM Corporation PARALLEL for RECOVER works the same as for COPY. In addition one should enable the ZPARM LOGAPSTG for fast log apply with a specified amount of space. This fast log apply option also is exploited during DB2 restarting processing. DB2 9 now provides back out capability for point in time recovery. All uncommitted changes for the specified objects will be backed out. So for full application consistency you need to specify all related objects on the RECOVER command (REPORT TABLESPACESET). This should reduce the need to run QUIESCE significantly which can be disruptive to applications. We believe that our BUFNO for COPY and RECOVER is adequate, but there have been certain customers claiming that increasing BUFNO has helped performance. With SVL, we have tested different values for BUFNO without seeing and improvements in performance, but we would certainly appreciate your feedback if you can prove otherwise! If you are using RI, then you should really be specifying TABLESPACESET to get a consistent point for all related tables. This will make it easier for those recoveries to a specific point in time. WRITE NO is also recommended. WRITE YES not only quiesces the table space with a drain, but it also drains all indexes for the table space and writes out the pages for all objects. The writing has the potential for encountering deadlocks with applications, although the application of PQ96628?? With data 1st claiming should help avoid these deadlocks. The main advantage of WRITE YES would only be if you are using an offline process like DSN1COPY that requires the forcing out of all pages. If we crash, we have to go back further and help restart times, but the pages will be externalized on every checkpoint, so the exposure isn’t great. 9
  10. 10. MODIFY RECOVERY Best Practices Run MODIFY RECOVERY regularly to clean up old records in SYSCOPY and SYSLGRNX DB2 9 has RETAIN LAST n or GDGLIMIT Also resets “ALTER_ADD_COLUM” flag in OBD when deleting image copies with previous row versions – MODIFY RECOVERY DELETE AGE/DATE to delete everything before the REORG that follows the ALTER – Will make next REORG more efficient if no more old row versions exist Remember that MODIFY RECOVERY works on day boundaries © 2008 IBM Corporation MODIFY RECOVERY is usually run to clean up old records in SYSLGRNX and SYSCOPY. But it does not only clean up those tables and prevents them from growing infinitely but it also resets a flag we keep in the OBD about alter added columns. This flag is needed for point in time recovery to work correctly. As long as there are image copies out there that can be recovered we need to make sure that we keep the record format information for the row versions that were used when those image copies were taken. Even if the data sets might have gone already DB2 still keeps track of those image copies in SYSCOPY. So make sure that MODIFY RECOVERY is run on a regular basis to get rid of all image copy entries after a new column was added to a table and before the next REORG that follows that ALTER which materialized the new column. If the ALTER_ADD_COLUMN flag is not reset, REORG doesn’t know if the column has been populated or not, so it must fully convert the rows to external format to populate the added column. After the flag is set, REORG doesn’t have to populate any columns, so it can optimize the format of the data the is unload and reloaded without fully converting it. DB2 9 for z/OS has a new option RETAIN that can be used to retain a number of image copies or even retrieve the number of image copies to retain from the GDGLIMIT into which the image copies were written. This makes it even easier to run MODIFY RECOVERY 10
  11. 11. Why do we need to REORG ? 1) Reclaim fragmented space, optimize space utilization 2) Improve access performance by organizing the data rows in clustering order 3) Apply Online Schema changes (Alter) 4) Functions (REBALANCE, DISCARD, UNLOAD) Clustering TS IX TS 2 R3 K1 R1 K2 R2 deleted K3 R3 K4 R4 R2 R4 deleted 1 R1 deleted Before After REORG REORG © 2008 IBM Corporation 11
  12. 12. Online REORG in a Nutshell Original TS/IXs Reorg Drain All Full Reorg Image Copy Log Uses 3 new elements: 1. shadow datasets 2. log records 3. mapping index Reorged TS/IXs (Shadows) © 2008 IBM Corporation 12
  13. 13. Use of Log Records in OLR Step 1 Record current log position, reorg into shadow copy, record new log position User Access READ Unload, Sort, Reload Original Data Shadow Copy WRITE CREATE DB2 Log Image Copy Step Read log between old and new LRSN, apply to shadow, record new LRSN 2->n READ Original Shadow Data Copy Write (except last iteration on Update on last iteration Log) process Image DB2 Log Copy Step Switch Users to access the newly reorged copy n+1 READ New Copy of Data WRITE DB2 Log © 2008 IBM Corporation 13
  14. 14. Reorg Shrlevel Change Unload Reload Sort Build Log Switch UTRW UTRW UTRW UTRW UTRW UTRO UTUT CR CR CR CR CR DW DA Mass Delete TS TS TS IX TS TS TS TS IX IX TS IX IX © 2008 IBM Corporation Serialization for the Shrlevel Change option is quite a bit different. You can see that for the first 4 phase the lock is been completely removed. Reorg sets UTRW as the DBET setting to show utility interest, and but it now claims as a reader instead of draining. Notice that sharelevel change has a switch phase just like that for shrlevel reference. However, it also has the Log phase which updates shadow tablespace and index pages from log records. For most of the Log phase, Reorg acts as a claim writer; however, when gets to a point where it has just only a small number of log records to be applied, it will Drain Writers and set the DBET to UTRO. There's an additional mass delete lock that's
  15. 15. V9 REORG Partition Parallelism REORG TABLESPACE TS Three partitions, one PI, one NPSI UNLOAD RELOAD SORTBLD TS TS Part 1 SORT Part 1 PI IX SORT TS TS Part 2 SORT Part 2 TS TS NPSI Part 3 SORT Part 3 SORT IX - Multiple concurrent jobs no longer needed - No longer allowed with PART specification and NPSIs © 2008 IBM Corporation This diagram illustrates the improvements in elapsed time during the UNLOAD, SORT, and RELOAD phases with REORG. In prior releases, partition level parallelism was not supported, so there would be one task to unload, sort and reload data (only one task handles all three phases). Now when there is enough memory to start a separate set of tasks for each partition, they can now be run in parallel in one job. This parallelism could be achieve in prior release with separate jobs, but then these separate jobs experienced potential availability problems and very inefficient building of NPIs. The performance benefit doesn’t have a downside when reorganizing entire tablespaces or for a part range REORG when no NPIs exist. However, when there are NPIs and SHRLEVEL REFERENCE/CHANGE PART is specified, there may be more work is done to rebuild the entire NPI (it depends on how much work BUILD2 did previously, worst case being reorg of a single partition, best case being reorg of x-1 parts, depends on number/size of keys). Note that with NPIs multiple concurrent REORG PART jobs for the same table space is not allowed. 15
  16. 16. REORG Best Practices REORG – Use SHRLEVEL REFERENCE or SHRLEVEL CHANGE – Inline COPY & Inline STATISTICS – KEEPDICTIONARY (track dictionary effectiveness with history statistics PAGESAVE) – large performance impact – 254 partition limit for compressed table spaces in V8 • PK51853 shipped new ZPARM MAX_UTIL_PARTS (watch virtual storage) • DB2 9 for z/OS no longer has this limit and uses virtual storage more effectively – Index parallelism (SORTKEYS is default and ignored in V8) • Remove SORTWKxx / UTPRINxx, and turn on UTSORTAL=YES • Run REORG against as many partitions as possible in the same job or against the whole table space © 2008 IBM Corporation REORG LOG NO and KEEPDICTIONARY should be used for the same reason as specified in LOAD. KEEPDICTIONARY will prevent uncompressing and re- compressing every single row thus saving on CPU and also on disk space for the sort work datasets. The 254 partition limit for compressed table spaces can be lifted by the DBA using ZPARM MAX_UTIL_PARTS for DB2 for z/OS V8 (provided by PK51853). DB2 9 for z/OS has removed that limit all together as the virtual storage consumption has been improved. The default of REORG will be SORTDATA YES to unload the data and then sort it in clustering order. If the data is already in or near perfect clustering order and disk space for sorting is low you can specify SORTDATA NO to unload data in clustering key order and to skip the sort. Otherwise we recommend to let REORG use the SORT. SORTDATA NO is not available in SHRLEVEL CHANGE. NOSYSREC with SHREVEL NONE also avoids using the SYSREC data set to unload, thus speeding up the process by passing the rows directly into sort. If this option is specified, it should be preceded by COPY as the table space will be left in RECP if the REORG fails. For SHRLEVEL CHANGE this option is always used. 16
  17. 17. REORG Best Practices REORG – Partition parallelism in DB2 9 and NPI processing • Parallel REORG jobs for same table space but different partitions no longer supported if NPIs defined • After REORG PART with no BUILD2 phase, no need for REORG NPI • Watch out for LISTDEFs on partition level with NPIs - full REORG might be more efficient – SHRLEVEL NONE if constrained for disk space • LOG NO reduces log volume; requires an image copy (inline is a good choice) • NOSYSREC to avoid I/O (forced for SHRLEVEL CHANGE) – Take full image copy before REORG SHRLEVEL NONE • Use REUSE to logically reset and reuse DB2-managed data sets without deleting and redefining them (affects elapsed time) © 2008 IBM Corporation REORG LOG NO and KEEPDICTIONARY should be used for the same reason as specified in LOAD. KEEPDICTIONARY will prevent uncompressing and re- compressing every single row thus saving on CPU and also on disk space for the sort work datasets. The 254 partition limit for compressed table spaces can be lifted by the DBA using ZPARM MAX_UTIL_PARTS for DB2 for z/OS V8 (provided by PK51853). DB2 9 for z/OS has removed that limit all together as the virtual storage consumption has been improved. The default of REORG will be SORTDATA YES to unload the data and then sort it in clustering order. If the data is already in or near perfect clustering order and disk space for sorting is low you can specify SORTDATA NO to unload data in clustering key order and to skip the sort. Otherwise we recommend to let REORG use the SORT. SORTDATA NO is not available in SHRLEVEL CHANGE. NOSYSREC with SHREVEL NONE also avoids using the SYSREC data set to unload, thus speeding up the process by passing the rows directly into sort. If this option is specified, it should be preceded by COPY as the table space will be left in RECP if the REORG fails. For SHRLEVEL CHANGE this option is always used. 17
  18. 18. REORG Best Practices continued REORG – SORTDATA NO only if data is already in or near perfect clustering order and disk space is an issue – Set appropriate PRIQTY/SECQTY to minimize extend processing • PK60956 helps to improve SORTBLD elapsed time up to 20x for indexes with small SECQTY!!! • SORTBLD elapsed up to 20x improvement!!! • Affects all utilities that are (re-)building indexes – Run MODIFY RECOVERY some time after ALTER TABLE … ADD COLUMN © 2008 IBM Corporation When ALTER TABLE … ADD COLUMN is used then REORG needs to materialize the new columns and will always decompress the rows even with KEEPDICTIONARY and will unload all rows in external format. This will increase the sort work data set requirement, in some cases significantly. This will be done as long as there are still image copies before the first REORG after the alter added column. Use MODIFY RECOVERY to clean up old image copies with the previous record format according to your backup retention policies. When building indexes the performance for extend processing can be improved by APAR PK60956. Even then it is beneficial to choose the primary and secondary quantity definition for indexes so that the number of extents is kept down to a minimum. Specifying PRIQTY=-1 and SECQTY=-1 will let DB2 pick appropriate values. 18
  19. 19. REORG SHRLEVEL CHANGE BPs REORG SHRLEVEL CHANGE (sometimes called online REORG) • TIMEOUT TERM frees up the objects if timeouts occur in getting drains • DRAIN ALL (better chance of entering SWITCH phase) • MAXRO = IRLMRWT minus 5-10 seconds (to prevent timeouts) • DRAIN_WAIT = IRLMRWT minus 5-10 seconds (to prevent timeouts) • RETRY = utility lock timeout multiplier (6 by default) • RETRY_DELAY = DRAIN_WAIT*RETRY • Enable detection of long running readers (zparm) and activate IFCID 0313 (it’s included in STATS CLASS(3)) – This will report readers that may block command and utilities from draining – It includes “well-behaved” WITH HOLD cursors which a drain cannot break-in on • More Joys of Commitment by Bonnie Baker – http://www.db2mag.com/db_area/archives/2003/q1/programmers.shtml © 2008 IBM Corporation TIMEOUT TERM terminate the utility if timing out trying to acquire drains, and thus avoids leaving the objects in a Utxx state which could result with unavailability. DRAIN ALL gets a drain on all objects once instead of doing it in two steps of draining writers followed by draining readers. If the drain fails, the data is still available in a UTRW state. The two steps may have an advantage if activity is such that it’s hard to get a DRAIN ALL successfully most of the time. The two steps may then quiesce enough activity for the 2nd DRAIN to be more likely to be successful. MAXRO is the estimated time to finish log apply as a UTRO utility before switching data sets. We suggest using something less than the SQL lock timeout value (SPRM???), like the value divided by two. RETRY is the number of times to retry getting drains if unsuccessful. We suggest using the utility multiplier SPRMUTO, which has a default value of 6. RETRY WAIT is the amount of time to wait after a failure before trying to drain again. If your workload has spikes of activity, then you may want to delay considering the average period of those spikes. If activity is fairly consistent, then you may one to retry again immediately. In the absence of any specific workload knowledge, we suggest a value equal to the drain-wait time times the retry value, or a default of 3 minutes. MAXRO ½ log timeout value DRAIN_WAIT – prevent transactions from timing. Should be less (1/2?) RETRY – 6X RETRY 19
  20. 20. Automatically display CLAIMERS when REORG receives resource unavail After DSNUGUTC - REORG TABLESPACE DB1.TS1 COPYDDN(SYSCOPY) SHRLEVEL NONE STATISTICS ) *********************************** ) * DISPLAY DATABASE SUMMARY * GLOBAL CLAIMERS ) *********************************** … NAME TYPE PART STATUS CONNID CORRID CLAIMINFO -------- ---- ----- ----------------- -------- ------------ -------- TS1 TS RW,UTRO BATCH CAPP (WR,C) - AGENT TOKEN 10 - MEMBER NAME V91A Culprits TS1 TS RW,UTRO BATCH CAPP (RR,C) - AGENT TOKEN 10 - MEMBER NAME V91A ******* DISPLAY OF DATABASE DB1 ENDED ********************** ) DSNTDDIS 'DISPLAY DATABASE' NORMAL COMPLETION DSNUGBAC - RESOURCE UNAVAILABLE REASON 00C200EA TYPE 00000200 NAME DB1.TS1 © 2008 IBM Corporation 20
  21. 21. REORG SHRLEVEL CHANGE contd. REORG SHRLEVEL CHANGE – Consider scheduling SWITCH phase in a maintenance window to avoid concurrent workloads that may prevent the utility from breaking in: • MAXRO DEFER and LONGLOG CONTINUE will let REORG do its job except for the last log iteration and the switching • REORG will continue applying log until MAXRO is changed with the ALTER UTILITY command • Many log iterations might reduce the “perfect” organization of the table space, so keep the time until MAXRO is changed to allow final processing down to a minimum © 2008 IBM Corporation When it is still very difficult for the REORG utility to break into heavy concurrent workload it might be possible to use a dedicated maintenance window to execute the switch phase. For that the REORG SHRLEVEL CHANGE job would be submitted with MAXRO DEFER and LONGLOG CONTINUE. This will let the REORG utility perform the reorganization and then loop in the log phase. Submission of the REORG utility should be done so that this status is reached before the scheduled maintenance window. In that window the utility can then be allowed to move forward with the final log phase iteration and the switch phase with the ALTER UTILITY command setting MAXRO to a defined value (after making sure that the workload has been reduced so that the utility has a chance to break in). Since the log phase will update the reorganized shadow table space this might have an effect on the “quality” of the REORG run, when many updates/deletes happen on the existing data. The longer the log phase is executed the more you will see disorganization of data. So the time before switching MAXRO should be kept down to a minimum. 21
  22. 22. Real-time Statistics Introduced in V7 Contain “space” and some “accesspath” statistics in user-defined tables: – SYSIBM.TABLESPACESTATS (one row per partition) – SYSIBM.INDEXSPACESTATS (one row per partition) – In DB2 9, these are moved into the DB2 Catalog (DSNDB06.SYSRTSTS) as • SYSIBM.SYSTABLESPACESTATS • SYSIBM.SYSINDEXSPACESTATS Intended to eliminate running RUNSTATS for reasons of running utilities by exception Access path selection doesn’t use RTS in V7, V8 or V9 © 2008 IBM Corporation 22
  23. 23. RTS SYSTABLESPACESTATS SYSINDEXSPACESTATS – Global Incremental Incremental Statistics – Global • NACTIVE – REORG Statistics – REORG Statistics • NACTIVE • REBUILDLASTTIME • NPAGES • LASTTIME • INSERTS • NLEVELS • LASTTIME • EXTENTS • UPDATES • NPAGES • INSERTS • SPACE • UPDATES • DELETES • NLEAF • TOTALROWS • DISORGLOB • DELETES • EXTENTS • DATASIZE • UNCLUSTINS • APPENDINSERT • SPACE • PSEUDODELETES • UNCOMPRESSEDDATASIZE • MASSDELETE • NEARINDREF • TOTALENTRIES • MASSDELETE • UPDATESTATSTIME • FARINDREF • LASTUSED • LEAFNEAR – COPY Statistics • UPDATESTATSTIME • LEAFFAR • LASTTIME • NUMLEVELS • UPDATEDPAGES – COPY Statistics • CHANGES • LASTTIME • UPDATELRSN • UPDATEDPAGES • UPDATETIME • CHANGES – RUNSTATS Statistics • UPDATELRSN • LASTTIME • UPDATETIME • INSERTS – RUNSTATS Statistics • UPDATES • LASTTIME • DELETES • INSERTS • MASSDELETE • DELETES • MASSDELETE © 2008 IBM Corporation 23
  24. 24. Utilities On Demand Run utilities only when necessary and not on fixed schedules Information on the current status of all objects is contained in Real-Time Statistics (RTS) tables Stored Procedure DSNACCOR applies our suggested thresholds and formulas against a list of objects and recommends utility actions DB2 9 NFM adds Stored Procedure DSNACCOX (PK44133) with additional real-time statistics being used and improved recommendations © 2008 IBM Corporation Real-Time statistics also contains valuable information that can be used to schedule utility operations on demand, as the objects require certain actions, like REORG, COPY or RUNSTATS. DB2 provides the stored procedure DSNACCOR that can be used to apply our suggested thresholds and formulas to the information in the RTS tables to generate recommended utility actions. DB2 9 NFM has the stored procedure DSNACCOX that uses further real-time statistics for improved recommendations. 24
  25. 25. Reorg table space V9 Recommendations Consider running REORG TABLESPACE in the following situations: – Real-time statistics (TABLESPACESTATS) • REORGUNCLUSTINS (number of records inserted since the last Reorg that are not well- clustered)/TOTALROWS > 10% – Irrelevant if predominantly random access – REORGUNCLUSTINS is only an indication of the insert behavior and is correlated to the cluster ratio only if there are no updates or deletes. To prevent DSNACCOR/X from triggering on these, identify such objects and put them in exception list • (REORGNEARINDREF+REORGFARINDREF (number of overflow rows since the last Reorg))/TOTALROWS > 5% in data sharing, >10% in non-data sharing • REORGINSERTS (number of records inserted since the last Reorg)/TOTALROWS > 25% • REORGDELETES (number of records deleted since the last Reorg)/TOTALROWS > 25% • EXTENTS (number of extents) > 254 • REORGDISORGLOB (number of LOBs inserted since the last Reorg that are not perfectly chunked)/TOTALROWS > 50% • SPACE > 2 * (DATASIZE / 1024) (when free space is more than used space) • REORGMASSDELETE > 0 (mass deletes on seg tsp and DROP on multi-table tsps) – RUNSTATS • PERCDROP > 10% • SYSIBM.SYSLOBSTATS.ORGRATIO < 50% (changed to a value 0-100 in PQ96460 on V7/V8) • (NEARINDREF + FARINDREF) / CARDF > 10% non-data-sharing, > 5% if data sharing • FAROFFPOSF / CARDF > 10% – Or, if index is a clustering index, CLUSTERRATIOF < 90% (irrelevant if predominantly random access) – Other • Tsp is in REORP or adv reorg pending status (AREO*) as result of an ALTER TABLE stmnt • Index on the tsp is in adv REBUILD pend state (ARBDP) as result an ALTER stmnt © 2008 IBM Corporation 25
  26. 26. Reorg table space V9 recommendations Consider running REORG TABLESPACE in the following situations: – Real-time statistics (TABLESPACESTATS) • REORGUNCLUSTINS (number of records inserted since the last Reorg that are not well- clustered)/TOTALROWS > 10% – Irrelevant if predominantly random access – REORGUNCLUSTINS is only an indication of the insert behavior and is correlated to the cluster ratio only if there are no updates or deletes. To prevent DSNACCOR/X from triggering on these, identify such objects and put them in exception list • (REORGNEARINDREF+REORGFARINDREF (number of overflow rows since the last Reorg))/TOTALROWS > 5% in data sharing, >10% in non-data sharing • REORGINSERTS (number of records inserted since the last Reorg)/TOTALROWS > 25% • REORGDELETES (number of records deleted since the last Reorg)/TOTALROWS > 25% • EXTENTS (number of extents) > 254 • REORGDISORGLOB (number of LOBs inserted since the last Reorg that are not perfectly chunked)/TOTALROWS > 50% • SPACE > 2 * (DATASIZE / 1024) (when free space is more than used space) • REORGMASSDELETE > 0 (mass deletes on seg tsp and DROP on multi-table tsps) – RUNSTATS use RUNSTATS statistics as a trigger to consider running REORG Don’t • PERCDROP > 10% • SYSIBM.SYSLOBSTATS.ORGRATIO < 50% (changed to a value 0-100 in PQ96460 on V7/V8) • (NEARINDREF + FARINDREF) / CARDF > 10% non-data-sharing, > 5% if data sharing • FAROFFPOSF / CARDF > 10% – Or, if index is a clustering index, CLUSTERRATIOF < 90% (irrelevant if predominantly random access) – Other • Tsp is in REORP or adv reorg pending status (AREO*) as result of an ALTER TABLE stmnt • Index on the tsp is in adv REBUILD pend state (ARBDP) as result an ALTER stmnt © 2008 IBM Corporation 26
  27. 27. Reorg table space V9 recommendations Consider running REORG TABLESPACE in the following situations: – Real-time statistics (TABLESPACESTATS) • REORGUNCLUSTINS (number of records inserted since the last Reorg that are not well-clustered)/TOTALROWS > 10% – Irrelevant if predominantly random access – REORGUNCLUSTINS is only an indication of the insert behavior and is correlated to the cluster ratio only if there are no updates or deletes. To prevent DSNACCOR/X from triggering on these, identify such objects and put them in exception list • (REORGNEARINDREF+REORGFARINDREF (number of overflow rows since the last Reorg))/TOTALROWS > 5% in data sharing, >10% in non-data sharing • REORGINSERTS (# of records inserted since the last Reorg)/TOTALROWS > 25% • REORGDELETES (# of records deleted since the last Reorg)/TOTALROWS > 25% • EXTENTS (number of extents) > 254 • REORGDISORGLOB (number of LOBs inserted since the last Reorg that are not perfectly chunked)/TOTALROWS > 50% • SPACE > 2 * (DATASIZE / 1024) (when free space is more than used space) • REORGMASSDELETE > 0 (mass deletes on seg tsp and DROP on multi-table tsps) – Other • Tsp is in REORP or adv reorg pending status (AREO*) as result of an ALTER TABLE stmnt • Index on the tsp is in adv REBUILD pend state (ARBDP) as result an ALTER stmnt © 2008 IBM Corporation 27
  28. 28. Reorg Index recommendations Consider running REORG INDEX in the following cases: – Real-time statistics (SYSINDEXSPACESTATS) • REORGPSEUDODELETES (number of index entries pseudo-deleted since the last Reorg)/TOTALENTRIES > 10% in non-data sharing, 5% if data sharing as pseudo-deleted entry can cause S-lock/unlock in Insert for unique index • REORGLEAFFAR (number of index leaf page splits since the last Reorg and the new leaf page far from the original leaf page)/NACTIVE > 10% • REORGINSERTS ( number of index entries inserted since the last Reorg)/TOTALENTRIES > 25% • REORGDELETES ( number of index entries inserted since the last Reorg)/TOTALENTRIES > 25% • REORGAPPENDINSERT / TOTALENTRIES > 20% • EXTENTS (number of extents) > 254 – RUNSTATS • LEAFFAR / NLEAF > 10% (NLEAF is a column in SYSIBM.SYSINDEXES and SYSIBM.SYSINDEXPART) • PSEUDO_DEL_ENTRIES / CARDF > 10% for non-data sharing and > 5% for data sharing – Other • The index is in advisory REORG-pending status (AREO*) or advisory- REBUILD-pending status (ARBDP) as the result of an ALTER statement © 2008 IBM Corporation 28
  29. 29. RUNSTATS Best Practices RUNSTATS – SHRLEVEL CHANGE for availability – Collect only column stats on columns used in SQL predicates • Use the Statistics Advisor to detect which stats to collect • SAMPLE reduces CPU time when gathering column stats – KEYCARD provides valuable info for little processing cost (see next slide) © 2008 IBM Corporation If not gathered with inline stats, RUNSTATS should be run with SHRLEVEL CHANGE for availability. For large amounts of data RUNSTATS TABLESPACE with column statistics is very CPU intensive. By specifying the SAMPLE keyword, CPU can be reduced while still gathering enough stats for the optimizer to choose a good access path. 29
  30. 30. RUNSTATS KEYCARD Collects all of the distinct values in all of the 1 to n key column combinations for the specified indexes. n is the number of columns in the index. For example, suppose that you have an index defined on three columns: A, B, and C. If you specify KEYCARD, RUNSTATS collects cardinality statistics for column A, column set A and B, and column set A, B, and C. So these are cardinality statisics across column sets... if we had a 3-column index that had these values: Col1 Col2 Col3 A B C A B D A B E A B E A C A A C A A D A B B B then these stats would be collected: – Col1 cardinality = 2 – Col1 and Col2 cardinality = 4 – Col 1, Col2, and Col3 cardinality = 6 © 2008 IBM Corporation 30
  31. 31. When is RUNSTATS needed? When the data changes sufficiently to warrant new statistics –REORG of table space or index (use inline stats!) –LOAD REPLACE of table space (use inline stats!) –After "significant" application changes for the table space or index •Periodically (weekly, monthly) except for read only data? •Application tracks updates with activity tables? •After percentage of pages changed since last RUNSTATS (RTS)? Understand implications for access paths! © 2008 IBM Corporation 31
  32. 32. LOAD Best Practices LOAD – LOG NO reduces log volume; if REPLACE, then take inline copy – KEEPDICTIONARY (track dictionary effectiveness with history statistics PAGESAVE) - small performance impact if loading lots of data – 254 partition limit for compressed table spaces can be lifted by DBA • PK51853 shipped new ZPARM MAX_UTIL_PARTS (watch virtual storage) – Load Partition Parallelism (V7) • Not individual LOAD part level jobs • Enable Parallel Access Volume (PAV) – Index parallelism (SORTKEYS) • Provide value for SORTKEYS when input is tape/PDS mbr or variable length • SORTKEYS is the sum of ALL indexes (and foreign keys) on the table • Remove SORTWKxx / UTPRINxx, and turn on UTSORTAL=YES © 2008 IBM Corporation LOG NO is the best performance option as well as KEEPDICTIONARY. Dictionaries should not change much. Their effectiveness can be tracked via history statistics PAGESAVE. When loading data into compressed table spaces there is a limit of 254 partitions that can be processed. This limit was introduced because of the amount of virtual storage being used while building compression dictionaries for many partitions in parallel. With PK51853 (UK31488/UK31489) you can now set your own limit in ZPARM MAX_UTIL_PARTS. The ZPARM only changes the limit for the number of partitions, storage consumption was not changed with this PTF, so keep an eye on your virtual storage when loading table spaces with many compressed partitions. Prior to V7, jobs were separated by partition to improved elapsed time in loading large partitioned tables. When NPIs existed, these would be dropped and rebuilt in a separate step. However, now partitioned objects can be loaded in parallel by partition. If the table has NPIs, the keys are merged into a common sort so that KLOAD processing can be used on a single member, thus improving LOAD performance. With multiple indexes, SORTKEYS allows for the indexes to be built in parallel, thus improving elapsed time with some increase in CPU. Usually LOAD should be able to estimate the number of records contained in the input data set. For input on tape, in PDS members or when the input data set has variable length records with great variation of the record length this estimate is not possible. In those cases specify a SORTKEYS estimate which will be required to estimate the required sort work space. Remember that SORTKEYS is the sum for all keys/foreign keys on the table(s) not the number of input records. Foreign keys are only counted if they don’t match an existing index or if there are more than 1 foreign key even if they match an existing index. So if you’re loading into a table space with 1 table and 3 indexes then SORTKEYS needs to be set to three times the number of input records. For the SORT, although you can control the data sets used by specifying SORTWKxx, avoid doing that. By specifying SORTDEVT, DFSORT or DB2 will dynamically allocate what is needed. DFSORT can’t span volumes with sort data sets, so if the amount of data is large, you may want so specify SORTNUM to ensure there are enough data sets available or turn on UTSORTAL=YES. 32
  33. 33. LOAD Best Practices continued LOAD – Inline COPY & Inline STATISTICS – Use REUSE to logically reset and reuse DB2-managed data sets without deleting and redefining them (affects elapsed time) – When using DISCARD, try to avoid having the input on tape • Input is re-read to discard the errant records – Avoid data conversion, use internal representation if possible – Sort data in clustering order (unless data is randomly accessed via SQL) – LOAD RESUME SHRLEVEL CHANGE instead of batch inserts – “LOAD REPLACE SHRLEVEL CHANGE” can be achieved by loading into clone table and then exchanging the tables on DB2 9 – LOAD via Batchpipes to load data that is transferred via FTP from clients © 2008 IBM Corporation A single pass of the data which creates an inline copy and gathers inline statistics while loading is better for elapsed time than running separate COPY and RUNSTATS utilities after the LOAD. Be aware that DISCARD records input block and record numbers and later returns to the input data set to position on each discarded record. This can be really really slow if the input data set is on tape! Sorting data in clustering key order before running LOAD improves performance - and can be done automatically by DB2 Utilities Enhancement Tool for z/OS Version 2.1. DB9 with its clone table support allows a function that comes close to a LOAD REPLACE SHRLEVEL REFERENCE. You can load data into the (named) clone table in the background and as soon as the LOAD has completed you can exchange the clone and the base table and make the new data active. The old data can be read by applications while new data is being loaded, only for the EXCHANGE all readers have to be drained. LOAD via Batchpipes can be used to load data immediately while it is ftp’ed from the client without storing it in data sets first. This helps to reduce the elapsed time as well as the amount of DASD needed for the transfer in total. An article in the IDUG Solutions Journal Volume 14 Number 2 (Summer 2007) describes the process in detail. 33
  34. 34. REBUILD INDEX Best Practices REBUILD INDEX – Indexes are built in parallel – Remove SORTWKxx / UTPRINxx and use SORTDEVT/SORTNUM or UTSORTAL=YES – Inline STATISTICS – Use REORG INDEX SHRLEVEL CHANGE to move index data sets to different volumes – CREATE INDEX DEFER followed by REBUILD INDEX • As of V8, dynamic SQL will not select the index until it is built – DB2 9 allows SHRLEVEL CHANGE • Unique indexes are put in RBDP because uniqueness can not be checked during rebuild process, so no INSERT/ UPDATE/DELETE allowed that affects unique index • No parallel jobs on different indexes of the same table space -> use single job with multiple indexes specified © 2008 IBM Corporation REBUILD INDEX recommendations follow LOAD and REORG. For availability reasons you should not use REBUILD INDEX to move the index data sets to different volumes. REORG INDEX SHRLEVEL CHANGE is much more efficient and provides better availability. Indexes can be created with DEFER YES and then materialized by REBUILD INDEX. Dynamic SQL will not select the index until it is built. In DB2 9 this is even more effective as the index can be rebuilt with SHRLEVEL CHANGE allowing concurrent updates. However this does not allow concurrent updates of data that affects unique indexes as they will still be put in RBDP because uniqueness can not be guaranteed otherwise. REBUILD INDEX SHRLEVEL CHANGE can not be invoked for different indexes of the same table space in multiple jobs concurrently. Instead specify those indexes in a single invocation of the REBUILD INDEX utility. 34
  35. 35. More online utilities REBUILD INDEX SHRLEVEL CHANGE – Great for building new non-unique indexes or when index is in RBDP – Index is built in place with no shadow • To move indexes to different volumes with availability still use REORG INDEX SHRLEVEL CHANGE) – Table space must be in LOGGED state (new log phase) © 2008 IBM Corporation A new enhancement of REBUILD SHRLEVEL CHANGE allows for improved availability when creating a new index, and it’s especially great for creating a new non-unique index on a populated table. Let’s suppose you have a non-unique index with column A, and you have done performance analysis to know that if you also included column B in that same index you would get performance benefits from your applications. The way to accomplish this without a period of unavailability follows: 1) CREATE the new non-unique index with A/B specifying DEFER YES. This places the index in RBDP without building it. 2) RUN REBUILD SHRLEVEL CHANGE to build the new index. The table space is kept in RW mode so that changes can be made. The index being built is in RBDP. Inserts, Updates, and Deletes are avoided for the index when in RBDP. After building the index, there is a log apply to catch all the changes that were made after the table space scan. This log phase is much like that for REORG. Finally, the index is completely built and RBDP is reset. 3) BIND is run to select the superior access path with the new index. 4) The old index is dropped. This type of SHRLEVEL CHANGE utility is different than REORG because the index is build in place rather than using a separate shadow dataset. Therefore, there is no SWITCH phase for REBUILD. Shadows aren’t as useful as in the case of REORG because the index is usually only built when it is newly created or broken, so the original is typically unavailable before the utility starts anyway. Because there are no shadows, don’t try to use REBUILD to move indexes to a new volume like you can do with REORG INDEX. Of course, this utility depends on applying log records, so you can’t specify SHRLEVEL CHANGE for not logged tables. 35
  36. 36. DB2 Allocated Sort Work Data Sets PTFs shipped 02/2008 to enable DB2 to dynamically allocate sort work data sets in utilities: – DB2 for z/OS V8: PK45916 / UK33692 SORTNUM – DB2 9 for z/OS: PK41899 / UK33636 – Enable with UTSORTAL=YES – Used for all sorts in utilities: LOAD, REORG, CHECK INDEX, REBUILD INDEX, CHECK DATA, RUNSTATS – Message “DSNU3340I - UTILITY PERFORMS DYNAMIC ALLOCATION OF SORT DISK SPACE” indicates use – New behavior ignored if hard coded DD cards are found No more need to specify SORTNUM. Existing SORTNUM specification can be honored or ignored (IGNSORTN=YES) Data sets for largest sorts are allocated first Attempts to allocate data sets as large as possible (starting with 2 data sets per sort task, more data sets allocated if necessary) © 2008 IBM Corporation In February 2008 we shipped two PTFs that enable DB2 to allocate the sort work data sets itself instead of having DFSORT allocate them. If DB2 allocates the data sets beforehand it knows exactly how many data sets were allocated and can optimize the degree of parallelism for utilities that use parallelism. Also it takes the burden off the administrator to find a reasonable SORTNUM value that matches many different utilities with different sort work requirements (one size doesn’t fit all). We often refer to that enhancement as “SORTNUM elimination”. To make migration easier this new function needs to be enabled by a new ZPARM UTSORTAL=YES. If only this parameter is set then existing SORTNUM specifications will be honored to make sure existing jobs still run exactly as before. The new behavior will only be used for utilities without SORTNUM specified. If IGNSORTN is also set to YES then all utilities will use the new behavior. SORTNUM values specified will be ignored. When data set allocation starts the largest sort tasks will be handled first as long as the largest chunks are still available on the sort pool disks. DB2 will try to allocate the necessary sort work disk space within 2 data sets first. If the current disk conditions don’t allow data sets of this size it will retry allocating smaller data sets and continue allocating data sets until the required size has been allocated. Don’t worry if you see messages like “IGD17272I VOLUME SELECTION HAS FAILED FOR INSUFFICIENT SPACE FOR DATA SET…”. 36
  37. 37. Checking all indexes in parallel TS TS TS IX1 IX1 IX1 IX2 IX3 Part 1 Part 2 Part 3 Part 1 Part 2 Part 3 NPI NPI Fast replication: Flashcopy/Snapshot UTILINIT IX1 IX1 IX1 Part 1 Part 2 Part 3 UNLOAD TS Part 1 SORT IX2 TS NPI Part 2 TS Part 3 IX3 Shadows NPI SORTCHK © 2008 IBM Corporation In this example of rebuilding all the indexes on a partitioned table space, three unload subtasks will pipe index keys to the appropriate of the three sort subtasks. The three sort subtasks will pipe sorted keys to three companion index build subtasks. If inline statistics are being collected, there will be three statistic subtasks as well. 37
  38. 38. CHECK SHRLEVEL CHANGE design SHRLEVEL REFERENCE CHECK INDEX causes data and indexes to be unavailable for update for the duration of the utility Claim as reader for target data and indexes Create shadow datasets – Same dataset naming convention as REORG SHRLEVEL CHANGE Drain writers for target data and indexes Copy datasets with DFSMSdss ADRDSSU with FCNOCOPY to shadows – Uses dataset-level FlashCopy2 if available – Else, traditional media copy – still smaller r/o outage than SHR REF After logical complete for datasets, – Dedrain target data and indexes – Run CHECK on shadow data and indexes At utilterm delete shadow datasets when DB2 managed © 2008 IBM Corporation Performing a physical copy of the data uses subsystem resources and can impact the performance of other I/O operations that are issued to the ESS. Using the FCNOCOPY keyword on a DFSMSdss™ COPY command prevents the ESS subsystem from performing a physical copy of the data. In general, if you want a temporary copy of the data, specify FCNOCOPY, and then withdraw the FlashCopy relationship when you no longer need the copy. If you want a permanent copy, but want to delay background copy until a convenient time, specify FCNOCOPY to get a point-in-time copy and then perform FCNOCOPYTOCOPY later to start background copy. If you want a permanent copy and do not want to delay background copy, do not specify FCNOCOPY. Allow the ESS subsystem to perform the physical copy and release the subsystem resources that are used to maintain the FlashCopy relationship. 38
  39. 39. DB2 9 Utilities Highlights More online utilities – Rebuild Index SHRLEVEL CHANGE – Reorg LOB now supports SHRLEVEL REFERENCE (space reclamation) – Check data, LOB and repair locate … SHRLEVEL CHANGE – Check index SHRLEVEL REFERENCE supports parallel for > 1 index – Clones for “online LOAD REPLACE” Online REORG BUILD2 phase elimination REORG parallelism for UNLOAD, RELOAD, LOG phases Utility TEMPLATE switching UNLOAD SKIP LOCKED DATA option © 2008 IBM Corporation If you look at all the utility offerings, we have been adding more SHRLEVEL CHANGE and REFERENCE utilities to improve availability in every release. Now with DB2 9, the following utilities have SHRLEVEL CHANGE (RUNSTATS, COPY, REORG TABLESPACE, REORG INDEX, LOAD RESUME, REBUILD, UNLOAD, CHECK INDEX, CHECK DATA, CHECK LOB, and REPAIR). Cloned tables effectually function as LOAD REPLACE SHRLEVEL CHANGE. We have also improved availability for REORG TABLESPACE of a part/part range by removing the BUILD2 phase. From a performance perspective, REORG of partitioned table spaces now has partition parallelism much like already existed in REBUILD INDEX, CHECK INDEX, COPY, RECOVER, and LOAD. Utility template switching allows flexibility in which template applies to datasets based on attributes of the object processed. UNLOAD SKIP LOCKED DATA gives another option besides just UR processing or CS processing. UR will not get locks, and therefore can unload “dirty” data. CS acquires locks and can therefore be slowed with lock contention. SKIP LOCKED is a CS option that will skip those rows/pages that are locked and give consistent data (but not all rows) without being slowed much by contention. 39
  40. 40. DB2 9 Utilities Highlights… MODIFY Recovery enhancements – “Retain” keyword added to improve management of copies • LAST(n), LOGLIMIT, GDGLIMIT Volume-based COPY/RECOVER (BACKUP SYSTEM/RESTORE SYSTEM) – RECOVER modified to enable object-level recovery from volume FlashCopy – Full integration of tape into BACKUP/RESTORE SYSTEM utilities – Incremental FlashCopy, APAR PK41001 Truncate log based on timestamp RECOVER to any point-in-time with consistency RECOVER RESTOREBEFORE to use an earlier image copy Display progress of RECOVER during log apply COPY CHECKPAGE option always active – “Copy Pending” avoided if broken page encountered COPY SCOPE PENDING to copy only objects in “Copy Pending” © 2008 IBM Corporation Prior to DB2 9 the user specified the deletion criteria either as before a specific date or by greater than a given age in days. DB2 9 has an alternative, by which instead of deletion criteria, retention criteria can be specified. There are a number of improvements and extensions to the volume base utilities, or the BACKUP/RESTORE SYSTEM utilities. If you need to do a conditional restart, DSNJU003 now allow you to specify a timestamp instead of an RBA/LRSN. The option is also available for the SYSPITRT for easier prep to truncate the log before running RESTORE SYSTEM. The COPY and RECOVER utilities each had several enhancements in DB2 9. 40
  41. 41. More online utilities (cont…) CHECK DATA SHRLEVEL CHANGE CHECK LOB SHRLEVEL CHANGE REPAIR LOCATE … SHRLEVEL CHANGE REORG LOB now supports SHRLEVEL REFERENCE Clones effectively provide LOAD REPLACE SHRLEVEL CHANGE – UTS only UNLOAD with ISO(CS) supports skipping rows that are locked for transaction updates © 2008 IBM Corporation More improvements to availability are listed here. CHECK DATA and CHECK LOB now have SHRLEVEL CHANGE support which operates the same way described for CHECK INDEX. The best availability is when there are FlashCopy devices availability, but the utilities will still work on other devices. The difference will be that a SLOW copy will be done to make a snapshot before running the CHECK utility. The availability is still improved over running SHRLEVEL REFERENCE, but the superior availability when using FlashCopy devices is the way to go. When running CHECK with SHRLEVEL CHANGE on a snapshot, any error cannot be corrected, but where we can REPAIR statements will be generated to correct the problems. These REPAIR statements also run with SHRLEVEL CHANGE for maximum availability. REORG LOB SHRLEVEL REFERENCE gives better function along with better availability. It turns out that reorganization typically benefits space usage and performance; however, for LOBS the need to reclaim fragmented space is usually much more important, and any performance improvement for better LOB organization is typically negligible. SHRLEVEL REFERENCE reorganizes to a shadow and reclaims spaces where SHRLEVEL NONE did not, therefore it is much better and is recommended as the best practice. In a future release SHRLEVEL NONE will be deprecated. We don’t have support for LOAD REPLACE SHRLEVEL CHANGE, but clone tables provide the same capability in a more flexible manner. Two separate but identical tables are maintained under one logical table space. Both tables can be populated or access using utilities or SQL. At the point a switchover is needed, it can be done with an ALTER statement to switch from the active table to the other. The “switch” is much like the SWITCH phase for online reorg. Note that clones are only supported for Universal Table Spaces (UTS). Another variation of UNLOAD allows a method of access that is somewhat in between UR and CS. For UNLOAD with UR, one can unload “dirty” data that hasn’t been committed, but the advantage is that there is no contention with applications and it will proceed quickly. For UNLOAD with CS, all data is clean and committed, but if locked data is encountered it will skip around it without unloading when SKIP LOCKED DATA is specified. This means rows or pages may be missed during the unload. 41
  42. 42. DB2 9 Utilities Support for all new functions in DB2 9 for z/OS – Universal Table Spaces (UTS) • Partition By Growth (PBG) • Partition By Range (PBR) – XML table spaces (PBG or PBR) – Not logged tables/table spaces – Clone tables – Index on expression – New data types (BIGINT, VARBINARY, DECFLOAT XML) © 2008 IBM Corporation Utility support is extensive for new features such as UTSs, XML, and not logged. For example, one can gather stats, reorg, unload, or rebuild indexes on a PBR or PBG table space. For PBGs, this means that inserts which extend the partitions can be run concurrently with SHRLEVEL CHANGE utilities. For XML on a segmented table space, it resides in a PBG format. Cloned tables are somewhat like Online Load Replace. There is the live table, and then there is the shadow or clone. One can load the clone, and then when ready for the clone to become live, a “switch” can be trigger with DDL. Index on expression is a key that is generated from an SQL or RDS expression. We also must support all the new data types so utilities like LOAD and UNLOAD can exploit them. 42
  43. 43. Copy Utility Changes Always perform CHECKPAGE on the COPY utility The COPY utility includes SCOPE PENDING support to improve usability COPY utility bufferpool usage uses MRU management of those pages read by the COPY utility Template switching (e.g., copy to tape if large; to disk if small) TEMPLATE LRG DSN &DB..&TS..D&DA..T&TI. UNIT=TAPE TEMPLATE SML DSN &DB..&TS..D&DA..T&TI. UNIT=SYSALLDA LIMIT(20 CYL, LRG) COPY TABLESPACE SMALL.TS COPYDDN(SML) COPY TABLESPACE LARGE.TS COPYDDN(SML) © 2008 IBM Corporation CHECKPAGE was an option that is now always done. The value of knowing copies are good outweighs the extra cost of less than 5% CPU. It’s still a bit less than before because of other improvements in our internal logic. CHECK for an index is not the default because the overhead is greater than 2X CPU (700%). These indexes are not as important as the data. COPY Pending is no longer set because this is prohibitive. However, a syscopy record is cut so that an incremental can’t be taken using that base. The RC=8 still applies, but an entire check is done instead of stopping early. COPY now has SCOPE pending like REORG and REBUILD to simply usability. 43
  44. 44. Recover Utility Changes Recovery to a point in time with consistency (NFM mode) – Uncommitted changes are backed out – Significantly reduces (eliminates?) the need to run QUIESCE – Does not apply to RECOVER TOCOPY, TOLASTCOPY and TOLASTFULLCOPY using SHRLEVEL CHANGE copy (consistency is not ensured – use RBA/LRSN after COPY point) – Include all relevant objects in same RECOVER to UW2 ensure data consistency from the application point of view U12 U22 U11 U21 UW1 UWn - Unit of Work number n A Current Unm – Update number n in Unit of Work m © 2008 IBM Corporation The RECOVER utility is changed to improve recovering to a point of time with consistency. If RECOVER is specified for a particular RBA/LRSN, the utility will also ensure that the tablespace and indexes are at a level of consistency. Here’s how it works: 1) As before 9, the RESTORE phase is followed by a fast log apply technique to get to the specified point, in our example that is point A. 2) The utility does a Current Status Rebuild (CSR), just like is done for DB2 restart/recovery to determine if there were any active units of work for the recovery point of A. 3) For anything inflight, inabort, etc that have not been committed, the log will be applied to roll back those units of work. In this case, UW2 and UW1 changes will be rolled back to there respective starting points. This function could greatly improve your procedures today if you use QUIESCE to get recovery points so that you can get a consistent recovery to a point in time. In today’s environment with heavier workloads, some customers were finding it very difficult to get a quiesce point, and there was a requirement to put in timeout/retry logic for that utility. However, the RECOVER solution we provide in 9 is a superior solution! An exception to consistency occurs for RECOVER to an image copy if that image copy was taken with SHRLEVEL CHANGE. In that case there is no log apply. Specify an RBA/LRSN after the COPY point to ensure there is log processing that gets you to a point of consistency. Remember, if there are multiple objects that are logically related (e.g. RI, LOB spaces, XML spaces), they should be included in the same RECOVER utility specification. 44
  45. 45. Recover to a Point In Time (cont…) Recovery to a point in time with consistency (contd) – Two new phases in RECOVER: LOGCSR and LOGUNDO (will each be executed by member) – RECOVER can be restarted in any LOG phase, LOGCSR will restart at beginning, others at last commit point – Fast Log Apply not used for LOGUNDO phase – Progress of LOGUNDO phase can be monitored by regular status message DSNU1555I showing current and target log RBA (e.g. to detect backout pf long running URs) – This function can greatly reduce your need to run QUIESCE • If you want to capture a log point non-disruptively – -DISPLAY LOG, or – QUIESCE against SYSEBCDC (or some other dummy table space), or – SELECT CURRENT_TIMESTAMP from SYSDUMMY1 and then convert it to an LRSN © 2008 IBM Corporation Here are some more details on how RECOVER to a point of consistency works. The status of RECOVER and the LOGUNDO phase can be tracked my regular message which show the status of the backout of the long running URs. This backout will not be as fast as forward logapply. If you run quiesce to capture a log point, there are several ways to do it in a more non-disruptive way. 45
  46. 46. BACKUP/RESTORE SYSTEM What’s new in DB2 9 for BACKUP and RESTORE SYSTEM? – The ability to recover at the object level using system-level backups • RECOVER to point in time or to current • COPY YES indexes can be included in this recovery • Will restore from the previous system level backup or image copy • Requires z/OS V1R8 – Tape Support – Support for Incremental FlashCopy • PK41001 & PK42014 – RTS COPY columns also updated for BACKUP SYSTEM © 2008 IBM Corporation In V8 it was introduced, primarily for ERP vendors. Doesn’t make sense to recover a single object because all related and there is little knowledge of the contents of objects. For example, an SAP systems consists of thousands of objects. The backup info is in the BSDS. For RECOVER CURRENT, they need Jack’s DCR. You can’t specify a PITR of ‘FFFFFFFF’. We can now recover an individual recovery for a single objects. Exploit a DFSMS enhancement. Potentially, no standard DB2 image copies. If the dataset has moved since the flash copy, this one would not work. z/OS is fixing this in 1.11 which is scheduled for Sept 2009. Any time we do a DSM reset. So whenever a REORG, take another system level backup. The ZPARM set means extra overhead to look at the BSDS. How long does restore system take? Stacking 99 volumes. 7 hours for 99 volumes. The Apars are for V8 or DB2 9 DASD RESTORE is like 500 volumes in 2 minutes. 46
  47. 47. Future enhancements Simplification, automation, availability, performance, flexibility Web-based Data Studio Admin Console DB2 Utilities Enhancement Tool (UET) High priority requirements – Autonomic stats – Autonomic compression – Recover to a different table space in a different DB2 with consistency – Reduce RUNSTATS resource consumption – Data set-level FlashCopy support Manual invocation of – Online consistent image copies •RUNSTATS – Policy-based recovery •COPY/BACKUP SYSTEM – UTSERIAL elimination •QUIESCE – REORG enhancements (LOBs, etc.) •MODIFY RECOVERY – … •REORG © 2008 IBM Corporation We will continue to enhancement utilities for the major areas of need, especially in usability. The goal is that periodic utilities of RUNSTATS, COPY/BACKUP, QUIESCE, MODIFY RECOVERY, and REORG will be submitted automatically when needed without manual intervention. The Data Studio Admin Console comes from the Data Servers group. It is designed for alerts and managing systems at a global level. The Utilities Enhancement Tool works closely with utilities to do things like. Sort LOAD input before doing a LOAD Create mapping tables for online reorg jobs. Allow for constant’s in the LOAD input. Auto-Stats Run RUNSTATS when needed, or update with RTS if possible. Determine what options are needed to collect the appropriate stats according to the access paths taken by applications. In the future profiles with the options needed for each object will be kept in the DB2 catalog. There will be better usability between stats advisor and the running of RUNSTATS with the suggested options. Auto-compression Eliminate the need for administration of compressing DB2 table spaces. The DBMS will automatically use compression when it is beneficial, disable compression when it is not beneficial, and not require the use of a utility to implement/disable compression. This becomes the default behavior for the whole data, administrators still can configure and limit the scope for the DBMS automatic compression. 47
  48. 48. Future enhancements Simplification, automation, availability, performance, flexibility Web-based Data Studio Admin Console DB2 Utilities Enhancement Tool (UET) High priority requirements – Autonomic stats – Autonomic compression – Recover to a different table space in a different DB2 with consistency – Reduce RUNSTATS resource consumption – Data set-level FlashCopy support Manual invocation of – Online consistent image copies •RUNSTATS – Policy-based recovery •COPY/BACKUP SYSTEM – UTSERIAL elimination •QUIESCE – REORG enhancements (LOBs, etc.) •MODIFY RECOVERY – … •REORG © 2008 IBM Corporation Recover to alternative TS with consistency. This is targeted with Recover Expert. Dataset level FlashCopy will 1. Exploit dataset-level FlashCopy that will result in VSAM image copies that can be used by the RECOVER utility to instantaneously restore an image copy. UNLOAD and COPYTOCOPY utilities will support these new VSAM image copy formats. 2. With a new keyword on all utilities that create image copies, specify whether log records should be used to create a consistent image copy. FlashCopy Version 2-capable devices are a prerequisite for this function. 3. Provide a utility option that will specify if the recovery should be performed by processing the log backwards versus the traditional restore and forward log-apply process. 4. Provide a DB2 subsystem/data sharing group level option that will disable all tracking of modified pages. UTSERIAL elimination eliminates the majority of timeouts on the global UTSERIAL lock resource. Existing concurrent workloads can be executed as they have previously, with no design changes in the jobs containing IBM DB2 Utilities. Throughout for multiple utilities may be increased, but more concurrent utilities can be run. Policy Based Recover is target for delivery with Recover Expert REORG Enhancements allow multiple part ranges to be specified (n:m, i:j), and allow REORG of a UTS tablespace or subset of partitions of a UTS to move rows with LOB columns between partitions when required by an alter partition key range or if the table space is PBG. Reduce the time spent in the switch phase and provide improved messages and diagnostics to allow customers to better estimate the length of REORG phases and time to completion. 48
  49. 49. References DB2 for z/OS home page: http://www-306.ibm.com/software/data/db2/zos/ DB2 9 for z/OS Technical Overview: http://www.redbooks.ibm.com/abstracts/sg247330.html?Open DB2 9 for z/OS Performance Topics: http://www.redbooks.ibm.com/abstracts/sg247473.html?Open DB2 for z/OS and OS/390 Version 7 Using the Utilities Suite: http://www.redbooks.ibm.com/abstracts/sg246289.html?Open Recommendations for Tuning Large DFSORT Tasks http://www.ibm.com/servers/storage/support/software/sort/mvs/tuning/index.html DFSMSrmm SMS ACS Support http://www.redbooks.ibm.com/abstracts/TIPS0530.html?Open IDUG Solutions Journal http://www.idug.org © 2008 IBM Corporation 49
  50. 50. DB2 for z/OS Information Resources Information Management Software for z/OS Solutions Information Center http://publib.boulder.ibm.com/infocenter/dzichelp/index.jsp DB2 for z/OS Information Roadmap http://ibm.com/software/db2zos/roadmap.html DB2 for z/OS library page http://ibm.com/software/db2zos/library.html DB2 for z/OS Exchange (examples trading post) http://ibm.com/software/db2zos/exHome.html DB2 for z/OS support http://ibm.com/software/db2zos/support.html Official Introduction to DB2 for z/OS http://ibm.com/software/data/education/bookstore © 2008 IBM Corporation 50
  51. 51. Thank you! Craig Friske – friske@us.ibm.com DB2 for z/OS Utilities Development IBM Silicon Valley Lab © 2008 IBM Corporation 51
  52. 52. Disclaimer © Copyright IBM Corporation 2008. All rights reserved. U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, NOR SHALL HAVE THE EFFECT OF, CREATING ANY WARRANTIES OR REPRESENTATIONS FROM IBM (OR ITS SUPPLIERS OR LICENSORS), OR ALTERING THE TERMS AND CONDITIONS OF ANY AGREEMENT OR LICENSE GOVERNING THE USE OF IBM PRODUCTS AND/OR SOFTWARE. IBM, the IBM logo, ibm.com, z/OS, DB2, DFSMS, and DFSORT are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml © 2008 IBM Corporation 52

×