How Does DB2 Get Data? What’s a Page Anyway? DBM1 Address Space – Detailed Review Buffer Manager (Detailed review of Dataspace management, Data Manager, RDS) DSNDB07 – DB2’s Work File Explained Index Considerations Balance Tree Structure & Index Basics Index Considerations (Clustered Vs. Non-Clustered, NPI’s & Piecesize) When to Add Columns to an Index Index Statistics & List Prefetch Tablespace Considerations Table Space Scan Sequential Prefetch & Sequential Detection Techniques to Minimize Logging Using Statistics to help DB2 Eliminate I/O Tune the SQL to get the Biggest Bang for your Buck Tuning Steps Why we should limit what we ask DB2 to do in our SELECTs and UPDATEs Reoptimization of Host Variables Explain Processing (PLAN_TABLE, Optimization Hints) Access Paths – What are the Choices? What are DB2’s choices for getting our data? Understanding Predicates Pinning tables into their Buffer Pool to Eliminate I/O
Most DB2 subsystems will have many databases. DB2 allows up to 65,279 databases. Associating objects that have similar maintenance patterns, recovery needs and access needs to a database has several benefits. Certain operations require locks on the DBD; isolating objects that do not change frequently to their own DBD can help minimize catalog contention. This example shows one table per tablespace, with one index defined on the table. The index name can be 18 characters long. DB2 uses the index name to generate the 8 character indexspace name. It is a good practice to keep your index names to 8 characters to ensure the indexspace and index share the same value. The YELLOW boxes represent the underlying VSAM data sets, the physical objects DB2 manages. A page can have a maximum of 255 rows. Normally you will have fewer rows than the max due to the size of each row. Each row is comprised of the columns. Most programmers understand the tables and columns their SQL touches. Understanding of the indexes involved requires a little more work on our part. SELECT PEAKNAME, ELEVATION FROM PEAKS; PEAKNAME and ELEVATION are columns of a row in the PEAKS table.
DB2 has 4 page sizes. 4K, 8K, 16K and 32K. Row length is the greatest determining factor when you select a page size. Remember that all DB2 I/O against your tablespaces and indexspaces is done at the page level. DB2 reads a 4K page into a 4K buffer, even if you want just one row. When evaluating object size, the number of pages should be used versus the number of rows. I.E. a table with a small row size could have 2 million rows, but only occupy 10,000 pages. A table with a large row size could have only 1 million rows but occupy 100,000 pages. Compare apples to apples. Maximum bytes per page size (how big one row can be): EditProc 4K 8K 16K 32K no 4,056 8,138 16,330 32,714 yes 4,046 8,128 16,320 32,704
Database Services (DBM1): -SQL processing (DDL, DML and DCL) -Maintains system tables -Data space management -Data manager (stage I predicates) -Relational data system (stage II predicates) -Buffer management -Sort Pool – integration with work files (DSNDB07) -EDM Pool -RID Pool -Interaction with IRLM to control locking of data. Concurrency control to maintain data integrity. In IOF or SDSF, you can view DB2’s started tasks. Assuming your subsystem name is DSNA, the database services address space would be DSNADBM1.
Since DB2 does I/O against your tablespaces and indexspaces, it is important to understand how these underlying vsam files are created and managed. Most DBA’s are using stogroups to define the volumes used when space allocations are needed by DB2. A DBA could also specify VCAT defined objects, creating the underlying VSAM clusters prior to running the DDL that creates the object. When using stogroups, you can use two flavors: 1) SMS managed stogroups, where SMS Storage groups handle the volume allocations. CREATE STOGROUP GMYTBLS VOLUMES(“*”) VCAT S0; 2) User Defined & Managed stogroups, where volumes are explicitly denoted and managed by people. ADDVOL to increase volumes in the stogroup CREATE STOGROUP GSPVOLS VOLUMES(B2RP01 ,B2RP02 ,B2RP03) VCAT P0
SMS = System Managed Storage. Manage storage growth, improve utilization, central control of external storage, hardware exploitation and management of data availability. Storage management policies are defined by use of SMS classes. Data Class, Storage Class and Management Class will determine the Storage Group and, therefore, volume allocation. Automatic Class Selection (ACS) routines are the mechanisms for assigning SMS classes and Storage Groups. Data set filtering can occur based on the following criteria: data set name, size of the data set, SMS classes (data, storage, management), as well as other filtering options based on the level of SMS your shop is running. EA Enabled Tablespaces and Indexspaces require DFSMS V1.5 or higher, which allows the STOGROUP to be managed by DFSMS instead of VSAM. You are also required to define the stogroup with (“*”) for the volume clause. You can still set the type of devices through use of ACS routines. The SMS DATACLASS gets setup for extended format, which allows extended addressability. DFSMS can also be used for your active logs. With DFSMS & OS/390 V2R10 and above, DFSMS can be used to implement VSAM DATA STRIPING of your active log data sets. The idea behind striping is to improve performance of data sets which are processed sequentially, splitting the data set into segments/stripes and placing those stripes on multiple volumes. This allows multiple channels, multiple disk arrays and multiple storage controllers. Striping requires extended format of SMS and can have up to 16 stripes. In high update shops, the writing of active logs can be a performance bottle neck. This option should help with those performance concerns.
DB2 performs I/O against tablespaces and indexspace pages via buffers. DB2 allows 50 - 4K, 10 - 8K, 10 - 16K, and 10 - 32K buffer pools. SET DSNZPARM value for default TS BP to BP48, default IX BP to BP49. Keep BP0 for Directory and Catalog objects only. Keep the default buffer pools small, as these should only be used to identify where an object was created without specifying the proper buffer pool (BP4, BP48, BP49) DB2 Tablespaces & Indexspaces = DSNDB01 & DSNDB06 - BP0 (directory & catalog databases), DSNDB07 - BP7 (work file database), DSNDB04 - BP4 (default database). TS - even # buffer pools (purple in diagram above) IX - odd # buffer pools (green in diagram above) Random spaces, Critical spaces, Code spaces Consider isolating your LOB tablespaces to their own buffer pools to better understand how they are being used. DB2 for z/OS and OS/390 V7 Performance Topics – Appendix C – “A method to assess the size of the buffer pools”. Very useful approach to evaluating your buffer pools, regardless of tool set.
The buffer manager does a considerable amount of work. Having the correct number of buffer pools and appropriate buffer pools associated to the correct objects is critical for the subsystem performance. Having DB2 find the page in the buffer pool can save a significant amount of time for the query, since the physical I/O to disk did not have to occur. Buffer Pool Thresholds : Fixed Thresholds: IWTH Immediate write threshold (97.5%) DMTH Data management threshold (95%) SPTH Sequential prefetch threshold (90%) Thresholds We Can Change: VPSEQT Sequential steal threshold (80%), 100% sequential pages monopolize the buffer pool, 0 disables parallel operation HPSEQT Hiperpool sequential steal threshold (80%), dependent on VPSEQT not being 0 VPPSEQT Virtual buffer pool parallel sequential threshold (50% VPSEQT) VPXPSEQT Virtual buffer pool assisting parallel sequential threshold (assist parallel operations initiated from another member of the data sharing) DWQT Deferred write threshold (50%), unavailable pages (updated pages & pages in use), when reached the data sets with the oldest pages are written asynchronously until below proper level. Proper level = DWQT – 10%. VDWQT Vertical deferred write threshold (10%), number of updated pages in a single page set
DB2’s sort pool is established with the parameters denoted in DSNZPARM(SRTPOOL). Increasing the sort pool can help performance of sort, as well as satisfy small sorts without having to use the files in DSNDB07. Only so much memory is available for the DBM1 address space, so test out increase in sort pool allocations thoroughly. Default is 1 mg, max is 64 mg. DB2 may use work files when SQL operations require sorts and materialization. These work files outside of the sort pool are associated to DSNDB07. DSNDB07 can have multiple 4K and 32K tablespaces associated to it. Sort Required (yes) INPUT PHASE – Ordered sets of rows written to work files (could be entirely in the buffer at this point). MERGE – Once input phase is complete, all work files created are merged. Merge does not occur with just one file. Intermediate merges can occur if dealing with many work files. If you have sort assist hardware, the number of work files is only limited by the size of BP7. Without the hardware, you are limited to 140 work files per sort, 255 work files per users. Caveat of Create Index without Defer Yes! Use CREATE DEFER on all NPI’s. The REBUILD utility can use an external sort when populating the index. When DB2 creates and populates the index, it has to use an internal sort which is significantly slower. By using DEFER YES/REBUILD, you won't clobber any concurrent tasks trying to use the 4k work files (sort space - DSNDB07).
**Each new index adds a data set DB2 has to open, close and maintain! Favor naturally occurring keys over generated keys. Natural keys typically ensure consistent clustering between the parent and children tables which assists in join processing. Place highest cardinality columns first in the index. Evaluate the multicolumn stats collection that can assist WHERE clauses. Look at syscoldist for the composite indexes. Don’t forget about the HAVING clause of GROUP BY. Look at ORDER BY, GROUP BY to avoid sorts. Use a composite index to facilitate GROUP BY. Avoid varchar columns in the index. An index has to store the maximum length, as well as access data pages to determine the true length of the column. DSNZPARM RETVLCFK(YES) is the only way to obtain index only access on indexes with varchar columns on V6.
SQL Breakdown: 80% - SELECT PEAKNAME, ELEVATION FROM PEAKS WHERE STATE=‘AK’ ORDER BY ELEVATION SELECT PEAKNAME, ELEVATION, STATE FROM PEAKS ORDER BY ELEVATION, STATE SELECT MAX(EVELATION) GROUP BY STATE; SELECT * FROM PEAKS WHERE STATE=‘CO’ ORDER BY ELEVATION 15% - SELECT PEAKNAME, ELEVATION FROM PEAKS WHERE PEAKNAME = ‘PIKES PEAK’; 5% - SELECT PEAKNAME, STATE FROM PEAKS WHERE LATITUDE = ‘401517N’ AND LONGITUDE = ‘1053655W’ GOAL OF INDEXES IS TO MINIMIZE I/O.
The clustering index is one of the most critical decisions made when defining indexes against your tables. You need to understand how programs and ad hoc users will be coming after the data in the table. Cluster for most frequent access by the majority of your SQL requests. What data elements will be predicate candidates? Always favor naturally occurring keys that help enforce business logic. If possible, avoid columns that are updated frequently. Similar clustering on tables involved in join processing should also be considered. CLUSTERRATIOF indicates how closely the data pages match the ordering maintained in the clustering index. Higher cluster ratios yield more efficient data page access. A cluster ratio below 80% will disable sequential prefetch of data pages. If you have INDEXONLY access, then CLUSTERRATIOF is not considered. Always define a clustering index. If you do not define a clustering index, DB2 is forced to select the index with the lowest OBID to assist in insert row placement. When you have not specified a clustering index, the one with the lowest OBID is normally the first index created on that table. Regaining clustering order when you have not defined a clustering index is a crap shoot at best. Duplicate index entries increase the CLUSTERRATIOF. Normally the clustering index is not a unique index. The clustering index is also the partitioning index if the tablespace is partitioned.
Size of NPI’s (Non-Partitioning Index) TABLESPACE NPI PAGESET SIZE NON LARGE 2G x 32 data sets = 64G LARGE 4G x 254 data sets = 1,016G / 1Tera byte(TB) DSSIZE xxG DSSIZE 8G -> 8G x 254 data sets = 2,032G / 2TB DSSIZE 16G -> 16G x 254 data sets = 4,064G / 4TB DSSIZE 32G -> 32G x 254 data sets = 8,128G / 8TB DSSIZE 64G -> 64G x 254 data sets = 16,256G / 16TB **This assumes your piecesize would match your DSSIZE… A piecesize of 2 gig for a non-large NPI will allow 32 - 2 gig data sets = 64 gig. PIECESIZE / (PRIQTY + (Number of SECQTY * SECQTY)) = Result should be an integer (meaning divides evenly with no wasted space, don’t want a remainder. Remainder = wasted space.)
Column Placement in the Index 1) High Key Column should strive to be the one with the highest cardinality. 2) Look at your GROUP BY and ORDER BY clauses to help pick the order of composite indexes. 3) Use a matrix to find columns used the most in your WHERE and HAVING clauses. Give more weight to the ones executed frequently. 4) Make sure your clustering/partitioning indexes are matching your business need. Random I/O should be minimal. Increasing the RID chain can add to the cost of your Update, Delete, and Insert statements. Is the increased RID chain length negatively impacting your non-read transactions? Adding a column to an index which increasing the filtering of the data can assist those same operations in certain scenarios. Better filtering means fewer rows qualifying and less I/O. Low Cardinality = Not many distinct choices, meaning more data could qualify. For example; gender (male/female). The cardinality is low, meaning only 2 choices. Having gender as the first column of an index would not filter many rows. High Cardinality = More distinct choices, meaning less data would qualify. A person finger print or DNA would have a high cardinality. Having DNA as the first column of an index would provide a great deal of filtering.
When to execute RUNSTATS? After loading a table After creating indexes After a REORG of a tablespace or index space and in-line stats were not gathered After RECOVER tablespace / indexspace After REBUILD of indexspace After massive insert, update and delete programs have run Periodically. Monthly, quarterly at a minimum for non-static tables. Don’t forget DSNDB06 and your various PLAN_TABLE tablespaces and indexes Two basic flavors: RUNSTATS TABLESPACE – gathers stats on tablespace and optionally indexes and columns RUNSTATS INDEX – gathers stats on indexes Caveat: Exclude volatile objects from your runstats to preserve your -1 in the various stats tables. If you have updated the catalog stats manually, RUNSTATS will be overlaying those updates. RUNSTATS will inactivate statements in Dynamic Cache. Don’t forget about the history stats. Can be very helpful with trend analysis and performance tuning!
Number Rows = SELECT COUNT(*) FROM PEAKS SYSIBM.SYSCOLDIST / SYSIBM.SYSCOLDISTSTATS CARDF = Number of distinct columns for column group. CARDF = SELECT DISTINCT COUNT(*) FROM PEAKS GROUP BY STATE CARDF = SELECT DISTINCT COUNT(*) FROM PEAKS GROUP BY STATE, ELEVATION FREQUENCYF = CARDF / Number of rows NUMCOLUMNS = Number of columns associated to statistics (from FREQVAL COUNT xx syntax of RUNSTATS) COLGROUPCOLNO = Array of smallinit numbers with the dimension equal to NUMCOLUMNS. In my examples, I am showing the actual values for understanding. Smaller filter factors mean less rows qualify. Fewer qualifying rows helps DB2 favor index access. If a large percent of rows qualify, the filtering based on that column or index is poor; therefore, DB2 would likely favor a relational scan. **Based on the queries, this index (DB2.STELEV) may be a better clustering choice than the original clustering index.
RID – Record Identifier The RID pool is used for record identifier processing. Enforces unique keys while updating multiple rows and sorting of RIDs for: 1) list prefetch, 2) access via multiple indexes, and 3) hybrid joins. On-line transactions generally do not want lots of rows returned and would rather not have List Prefetch used. OPTIMIZE FOR 1 ROW will disable LIST PREFETCH. FETCH FIRST 1 ROW ONLY will use an implied OPTIMIZE FOR 1 ROW.
Also have LOB tablespaces. Simple tablespace – Tablespace Scan has the potential to touch all pages. Segmented tablespace – Spacemap page helps determine which segments need to be scanned. Partitioned tablespace – Potential to touch all pages for all partitions. Predicates could enable partition scanning, limiting scan to a subset of partitions. SEGSIZE Number of Pages 4 to 28 4 to 28 pages, match to max table size 32 >28 pages and <128 pages 64 >= 128 pages Indexspace Types: Non-segmented (simple) for every index except the partitioning index, which uses a partitioned data set. Advantages to Segmented: Handles TS Scans more efficiently by determining which segments need to be scanned, allows multiple tables, and locking by table. Spacemap page does more for you. Advantages to Partitioned: Allows Extended Addressability. Table sizes greater then 64gig allows use of several stogroups tied to the same table to spread I/O to specific devices, gives utility independence, makes read only parallelism possible, and gives larger LOB data (16TB for non-partitioned, 1,000TB for partitioned, 4,000TB for large). The down side is tablespace scan performance, administrating the clustering/partitioning index and having to define it up front.
RSCAN – Access path 1) No Qualifying Index 2) 0-10 pages (empty or small table) 3) Optimizer determines cost of scan is more favorable than cost of index processing DB2 V6 introduced a way to have the optimizer favor index access even though the normal cost based optimization would have lead to a tablespace scan. This option can be very helpful when you have volatile tables, where the number of rows varies from <10 to >10,000. The idea is that at runstats time, less than 10 pages or an empty table existed. DB2 by default will elect to use a tablespace scan. That is fine when there are less than 10 pages, but when the table grows to 10,000 pages the performance can really take a hit. ZPARM NPGTHRSH (number of pages threshold) can be set to help influence index access. 0 = default value, DB2 uses standard cost optimization n = if npages < n, then prefer index access -1 = always prefer index access (not recommended) Assuming you know the list of volatile tables, you can set the NPAGES = -1, and set NPGTHRSH = 502, so the optimizer will favor index access on tables with less than 502 pages. With NPAGES = -1, DB2 assumes 501 pages. Some shops will drop/create the volatile tables and omit them from their runstats jobs so NPAGES is always -1, rather than manipulating the stats tables themselves. V8 allows CREATE TABLE MYTABLE ….. VOLATILE and in turn the optimizer would naturally favor index access for those objects. ftp://ftp.software.ibm.com/software/db2storedprocedure/db2zos390/techdocs/v8overview.pdf
P = DB2 Prefetch Quantity = 32 pages (4K page size, 4K buffer pool with 1,000 buffers). The prefetch quantity will vary by page size and number of buffers in the buffer pool for the object in question. Page-sequential = P/2 32/2 = 16 Is the current page within 16 pages of the next page? If the answer is yes, the relationship is considered page-sequential. Once DB2 has determined it is page-sequential, it determines if the data access is sequential. Data access is considered sequential if 4 out of the last 8 pages are page-sequential. Assuming this is true, sequential prefetch (getting 32 pages) will continue until the pattern stops.
Logging considerations tied to your DML: INSERTs - Logs the after image of the entire row. DELETEs - Logs the before image of the entire row. (exception for mass deletes) UPDATE – Logs just the changed data. If data capture changes is specified, logs both the before and after image of the entire row. For fixed length rows (tables containing no variable data types), DB2 will log the first byte updated through the last byte updated. For tables containing variable data types, DB2 will log from the first byte updated to the end of the entire row on V6. With V7, it logs from the first byte of the first column changed to the last byte of the last column updated, assuming none of the restrictions have been triggered. If restrictions exist, then it will log to end of the row. 1) The update does not change the row length 2) Not using hardware compression 3) Not using editprocs The physical order of the columns in your table can aid in reducing the amount of data logged. The idea is to keep highly updated columns grouped together rather than spread apart. Keep columns which are not updated frequently at the beginning of the table. Place static fixed length columns first, followed by static variable (varchar) columns, followed by fixed length columns that are updated frequently, followed by variable length columns that are updated. Pay attention to update sets or groupings! Placing columns next to each other when they are always updated in a group is a good idea. Compressed objects are logged as compressed data. Compressed data is also treated like variable data.
Typical tuning steps and order: 1) Tune the existing SQL (apps, stored procs, triggers, udf) 2) Tune the objects (based on input from the SQL & Utility flow) 3) Create additional objects (indexes, de-normalized tables) 4) Tune the subsystem 5) Consider retention, growth factors, archiving logic, space impact Create (insert), Read (select), Update and Delete matrixes allow you to forecast work load and commit frequency. You can use the numbers from your restart tables to get a feel for the numbers in your CRUD matrix, etc. STATIC SQL should always be your first choice. The advantages to STATIC SQL arise out of the ability to perform costly operations (validation, authorization checking, access path determination) once up front at BIND time. DYNAMIC SQL has many good implementations. Using DYNAMIC SQL in COBOL programs can significantly reduce the number of SQL statements that need to be written and maintained. DYNAMIC SQL can turn BLACK boxes into SMART boxes. Obtaining DYNAMIC SQL for code reviews can be a challenge. Typically you have to perform all scenario’s on the user interface to capture the various SQL that can be generated to ensure it is performing well.
Do not under estimate the importance of having a good statistics scheme. Let the Optimizer do the majority of the work, provided you have given the optimizer the best possible picture of your data and application needs. How frequently are you taking runstats? What type of runstats are you taking? Does it vary by object? Are you rebinding after your runstats? Get your way by giving DB2 the stats it needs to select the optimal access path. Gain an understanding of which columns do not change frequently. What will RUNSTATS UPDATE in the catalog tables? ALL would include ACCESSPATH and SPACE. ACCESSPATH indicates that only the catalog table columns that provide statistics used for access path selection are updated. SPACE indicates that only the catalog table columns that provide statistics to help the database administrator assess the status of a particular table space or index are updated. NONE indicates that no catalog tables are updated with the collected statistics. This option is only valid when REPORT YES is specified. KEYCARD, FREQVAL NUMCOLS(1) COUNT(10), TABLE(ALL) COLUMN(xx), FORCEROLLUP(YES) DB2 UDB for z/OS and OS/390 Administration Guide, Table 109, page 764, “Catalog data used for access path selection.”
Each version of DB2 makes improvements and pushes more logic into stage one processing. Regardless of DB2 version, select only the columns you need and the rows you need. Your SQL should do a considerable amount of filtering up front. Start by writing efficient SQL! Don’t wait to see explain output, encourage SQL coding habits that exploit the optimizers strongest features. If you knew the winning lotto numbers, wouldn’t you play? Assuming the statement is not able to use an index, there is still considerable savings from being stage one vs. stage two. Learn the tricks of the trade to convert stage two into stage one statements. Pay attention to predicate usage and classification. Pushing even part of the query to stage one can help in reducing the complexity and amount of stage two filtering that has to occur.
Column Constraints 1) Consider when performance and control are essential. 2) Consider when values are static and few. I.E. Gender, Blood Type, Switch Data (yes/no, on/off, 0/1). 3) If values are not static, or if they will increase if your company expands its business, consider maintenance costs vs. performance gains. 4) If users need access to the business rules, implementation of domain tables may be more practical, especially with security considerations. Consider creating a view on appropriate catalog tables to externalize business rules placed in column constraints. 5) FK violation vs. Column Constraint violation - which is easier for a programmer to understand and take appropriate action? DB2 RI 1) Ease of maintenance 2) Ability to grow with the business 3) Control of ad hoc users 4) Data integrity Application RI 1) Can you limit ad hoc insert, update and delete access to your data? 2) Is there really a need for DB2 RI on code tables? 3) Is your application enforcement efficient?
REOPTIMIZATION The PLAN_TABLE has column WHEN_OPTIMIZE. This can be helpful in identifying SQL that plans to use re-optimization. Enables you to track its performance. Blank - Access path was determined at bind time. B - Access path was determined at bind time; will re-optimize at execution. R - Access path was determined at execution time with real values for the variables. Reopt(vars) allows partition scans, the possibility of more efficient access path, and gives accurate filter factor and cost estimation. Be aware that rebind time can experience more overhead tied to static statements. Use to solve a performance problem versus globally on every package. If the prepare cost is your main concern, do not re-optimize. OPTIMIZATION HINTS +394 = User specified optimization hints used during access path selection. (SQLSTATE 01629) +395 = User specified optimization hints are invalid (reason code = ‘ ‘). The optimization hints are ignored. (SQLSTATE 01628) DSNZPARM change required to support optimization hints. YES in the OPTIMIZATION HINTS field of installation panel DSNTIP4.
Using the QUERYNO clause to assign unique numbers to the SQL statements in your programs is very helpful for: 1) simplifying the use of optimization hints for access path selection, and 2) correlating SQL statement text with EXPLAIN output in the plan table. Dynamic Statements Query number is the statement number in the application where the prepare occurs. For some applications, such as DSNTEP2, the same statement in the application prepares each dynamic SQL statement. This makes the query number for each dynamic statement identical. Using QUERYNO eliminates ambiguity as to which rows in the PLAN_TABLE are associated with each SQL statement in the application. Static Statements If your application change results in the statement number changing then the QUERYNO would change as well (if you have not hard coded it). This would mean updates to your opt hints in your old PLAN_TABLE rows. Assigning a QUERYNO will ensure you are controlling and tagging your SQL properly. PLAN_TABLE - information about the access paths associated to your packages (and plans if using plan based dbrms). DSN_STATEMNT_TABLE - information about a statement’s estimated cost will be collected when explain yes is specified on a plan or package bind. DSN_FUNCTION_TABLE - information on User-Defined Functions referenced in SQL will be collected when explain yes is specified on a plan or package bind. Recommend creating an index on PLAN_TABLE. CREATE INDEX owner.ix_name on owner.PLAN_TABLE (QUERYNO, APPLNAME, PROGNAME, VERSION, COLLID, OPTHINT); DB2 sample library member DSNTESC has a sample create index for the plan table. Download Visual Explain – www-3.ibm.com/software/data/db2/os390/downloads.html
MATCHCOLS - Number of columns of index matching where predicates. Normally increasing matchcols will further limit the result set. PREFETCH - Indicates if a prefetch method was invoked by DB2. Normally acceptable for batch work where lots of data has to be touched. Typically not suited for on-line transactions. L – list prefetch S – sequential prefetch INDEXONLY - Select clause was satisfied from the index, access to the tablespace (data page) did not occur. ACCESSNAME - The index name. Make sure your index access is using the index you intended. OPTHINT - Ability to point a package (via bind or prepare) to an access path stored in your PLAN_TABLE. Useful for returning to an access path that previously executed well. The string identifies the row in the plan table as an optimization hint. You will have to mange opthints yourself, which can become overwhelming. RID - Record identifier, uniquely identifies rows on a page, also useful in other operations (sorting, list prefetch, etc.) ROWID - Stored as 17 bytes. When selected, you get 40 bytes returned. ROWID does allow direct access, assuming no reorgs have occurred that would have changed the location of the row and the ROWID trailing bytes. ROWID(expression) to convert a previously selected ROWID back into ROWID format. JOIN Methods: For more details on performance of joins, please see IDUG 2002 presentation “D5: SQL Tips for the Application Developer” by Mike Vaughan. Nested Loop – Small result set or well clustered tables. Merge Scan – Poor cluster ratio, large result set. Hybrid – Poor cluster ratio, large number of columns on inner table.
The PEAKS table is a subset of the entire table. There are actually 91 peaks in the United States above 13,999 feet, I have only shown 16 in my examples. Colorado has 54 Alaska has 21 California has 15 Washington has 1 *Peaks” table built from data found @ www.americasroof.com. Data is for teaching concepts and not meant for navigation or mountaineering. Rating has been solely established by Rob’s SWAG. --------------------------------------------------------------------- Ibm.com edbooks search on DB2 DB2 UDB Server for OS/390 Version 6 Technical Update DB2 for z/OS and OS/390 Version 7 Performance Topics DB2 for z/OS and OS/390: Squeezing the Most Out of Dynamic SQL DB2 for z/OS and OS/390 Version 7 Selected Performance Topics SAP R/3 on DB2 for OS/390: DB2 Features That Benefit SAP Storage Management with DB2 for OS/390
DB2 Performance Primer
DB2 Performance Primer Baltimore/Washington DB2 Users Group December 10, 2003 Michael Murray Senior Consultant [email_address]
Presentation Overview <ul><li>How Does DB2 Get Data? </li></ul><ul><li>Index Considerations </li></ul><ul><li>Tablespace Considerations </li></ul><ul><li>Tune the SQL to Get the Biggest Bang for your Buck </li></ul><ul><li>Access Paths – What are the Choices? </li></ul>
What is a Page? 4K page, maximum number of data bytes on the page 4,074. Maximum row size 4,056. Row size greater than 2,020 bytes will end up with just one row per page. 4K 8K 16K 32K GETPAGE Physical I/O Your Data Row Limits PSID – page set identifier
DBM1 Address Space <ul><li>SQL, DDL, DML, DCL </li></ul><ul><li>Optimization </li></ul><ul><li>Data Manager </li></ul><ul><li>Relational Data System </li></ul><ul><li>Buffer Manager </li></ul><ul><li>Sort Pool </li></ul><ul><li>EDM Pool </li></ul><ul><li>RID Pool </li></ul><ul><li>Bind/Rebind </li></ul><ul><li>IRLM Interaction </li></ul>Locking IRLM Directory Catalog WorkFile User Data DSNDB01 DSNDB06 DSNDB07 Database Services DBM1 SQL stored procedures
SMS Allocations indexspace volume PBP2 Storage Group I cache indexspace volume PBP8 cache tablespace volume PB34 Storage Group T cache tablespace volume PBL8 cache ACS Routines (4 th node is NOT X or I) (4 th node is X or I) Tablespace or Indexspace Allocations (Create, Extent Allocation, Load, Reorg) DB2 SMS
Buffer Manager indexspace tablespace Hiperpool Group Buffer Pool BUFFER POOL DATA MANAGER Relational Data System STAGE 1 PREDICATES STAGE 2 PREDICATES sargable indexable Get Page Request Physical I/O (page not found) (page not found) (page not found) (page found) (page found) cache cache
DB2’s Work File Explained DATA MANAGER Relational Data System STAGE 1 PREDICATES STAGE 2 PREDICATES sargable indexable Filtered Result Set DSNDB07 SORT required CREATE INDEX DEFER NO <ul><li>No matching index </li></ul><ul><li>DISTINCT </li></ul><ul><li>ORDER BY </li></ul><ul><li>UNION </li></ul><ul><li>GROUP BY </li></ul><ul><li>Some Joins </li></ul>SORT POOL 3. sort columns 1. selected columns 2. calc/derived values key columns, RIDs BP7 sort assist hardware Sort assist hardware?
Index Considerations <ul><li>Clustering Vs. Non Clustering </li></ul><ul><li>Unique Vs. Duplicate </li></ul><ul><li>Piecesize with NPI’s </li></ul><ul><li>Uniqueness, Performance, RI, Sorting </li></ul><ul><li>Table Size </li></ul><ul><li>Minimize I/O </li></ul>
Peaks > 13,999 Feet Located in US <ul><li>Indexes: </li></ul><ul><li>Clustered Index on: Elevation (desc) , State, Peak Name </li></ul><ul><li>Non-Clustered Index on: Peak Name </li></ul><ul><li>Unique PK Index on: Latitude, Longitude </li></ul>Goal of indexes is to minimize I/O
<ul><ul><ul><li>Elevation (desc) </li></ul></ul></ul><ul><ul><ul><li>State </li></ul></ul></ul><ul><ul><ul><li>Peak Name </li></ul></ul></ul>ROOT Page CLUSTERED INDEX Page 552 14831 AK Mount Bear 14420 CO Mount Harvard p. 501, 2 p. 502, 2 LEAF & DATA Page Page 553 14270 CO Grays Peak 14196 CO Mount Yale p. 504, 2 p. 503, 2 Page 554 14110 CO Pikes Peak 14059 CO Sunlight Peak p. 505, 2 Page 555 14037 CO Little Bear Peak 14000 CA Thunderbolt Peak p. 507, 2 NON-LEAF Page Page 717 14420 CO Mount Harvard 14196 CO Mount Yale 14059 CO Sunlight Peak 14000 CA Thunderbolt Peak p. 553, 2 p. 554, 2 p. 552, 2 p. 508, 2 20320 AK Mount McKinley … 14831 AK Mount Bear … Page 502 14494 CA Mount Whitney … 14420 CO Mount Harvard … Page 503 14410 WA Mount Rainier … 14270 CO Grays Peak … Page 504 14255 CO Longs Peak … 14196 CO Mount Yale … Page 505 14172 CO Mount Bross … 14110 CO Pikes Peak … Page 506 14080 CA Starlight Peak … 14059 CO Sunlight Peak … Page 507 14058 CA Split Mountain … 14037 CO Little Bear Peak … Page 508 14000 CA Thunderbolt Peak … Page 501 14005 CO Mount o/t Holy Cross … p. 555, 2 p. 506, 2
<ul><ul><ul><li>Peak Name </li></ul></ul></ul>p. 401, 2 Page 301 Little Bear Peak Mount Bear p. 402, 2 ROOT Page NON-LEAF Page LEAF Page NON-CLUSTERED INDEX 20320 AK Mount McKinley … 14831 AK Mount Bear … Page 502 14494 CA Mount Whitney … 14420 CO Mount Harvard … Page 503 14410 WA Mount Rainier … 14270 CO Grays Peak … Page 504 14255 CO Longs Peak … 14196 CO Mount Yale … Page 505 14172 CO Mount Bross … 14110 CO Pikes Peak … Page 506 14080 CA Starlight Peak … 14059 CO Sunlight Peak … Page 507 14058 CA Split Mountain … 14037 CO Little Bear Peak … Page 508 14005 CO Mount o/t Holy Cross … 14000 CA Thunderbolt Peak … DATA Page Page 501 p. 403, 2 Page 302 Mount Harvard Mount o/t Holy Cross p. 404, 2 p. 405, 2 Page 303 Mount Whitney Pikes Peak p. 406, 2 p. 407, 2 Page 304 Starlight Peak Thunderbolt Peak p. 408, 2 p. 301, 2 Page 201 Mount Bear Mount o/t Holy Cross p. 302, 2 p. 303, 2 Page 202 Pikes Peak Thunderbolt Peak p. 304, 2 p. 201, 2 Page 101 Mount o/t Holy Cross Thunderbolt Peak p. 202, 2 p. 503, 2 Page 401 Grays Peak Little Bear Peak p. 507, 2 p. 504, 1 Page 402 Longs Peak Mount Bear p. 501, 2 p. 505, 1 Page 403 Mount Bross Mount Harvard p. 502, 2 p. 501, 1 Page 404 Mount McKinley Mount o/t Holy Cross p. 508, 1 p. 503, 1 Page 405 Mount Rainier Mount Whitney p. 502, 1 p. 504, 2 Page 406 Mount Yale Pikes Peak p. 505, 2 p. 507, 1 Page 407 Split Mountain Starlight Peak p. 506, 1 p. 506, 2 Page 408 Sunlight Peak Thunderbolt Peak p. 508, 2 NON-LEAF Page
p. 461, 2 Page 331 370116N 1182520W 370539N 1183124W p. 462, 2 ROOT Page NON-LEAF Page LEAF Page PRIMARY NON-CLUSTERED INDEX 20320 AK Mount McKinley … 14831 AK Mount Bear … Page 502 14494 CA Mount Whitney … 14420 CO Mount Harvard … Page 503 14410 WA Mount Rainier … 14270 CO Grays Peak … Page 504 14255 CO Longs Peak … 14196 CO Mount Yale … Page 505 14172 CO Mount Bross … 14110 CO Pikes Peak … Page 506 14080 CA Starlight Peak … 14059 CO Sunlight Peak … Page 507 14058 CA Split Mountain … 14037 CO Little Bear Peak … Page 508 14005 CO Mount o/t Holy Cross … 14000 CA Thunderbolt Peak … DATA Page Page 501 <ul><ul><ul><li>Latitude </li></ul></ul></ul><ul><ul><ul><li>Longitude </li></ul></ul></ul>p. 463, 2 Page 332 373738N 1073543W 385039N 1061848W p. 464, 2 p. 465, 2 Page 333 392007N 1060625W 393802N 1054901W p. 466, 2 p. 467, 2 Page 334 465110N 1214531W 630410N 1510013W p. 468, 2 p. 331, 2 Page 281 370539N 1183124W 385039N 1061848W p. 332, 2 p. 333, 2 Page 282 393802N 1054901W 630410N 1510013W p. 334, 2 p. 281, 2 Page 101 385039N 1061848W 630410N 1510013W p. 282, 2 p. 502, 1 Page 461 363438N 1181733W 370116N 1182520W p. 507, 1 p. 506, 1 Page 462 370539N 1183119W 370539N 1183124W p. 508, 2 p. 507, 2 Page 463 373400N 1052948W 373738N 1073543W p. 506, 2 p. 505, 2 Page 464 385026N 1050238W 385039N 1061848W p. 504, 2 p. 502, 2 Page 465 385528N 1061912W 392007N 1060625W p. 505, 1 p. 508, 1 Page 466 392805N 1062845W 393802N 1054901W p. 503, 2 p. 504, 1 Page 467 401517N 1053655W 465110N 1214531W p. 503, 1 p. 501, 2 Page 468 611702N 1410832W 630410N 1510013W p. 501, 1 NON-LEAF Page
Adding Columns to an Index <ul><li>Improves filtering </li></ul><ul><li>Provides index only access </li></ul><ul><li>Increases matched columns </li></ul><ul><li>Provides a sequence to avoid SORTS </li></ul><ul><li>Increases cluster ratio </li></ul><ul><li>Helps in index screening </li></ul><ul><li>Is the increased RID chain length negatively impacting your non-read transactions? </li></ul>
Index Statistics <ul><li>Indexes & Stats: </li></ul><ul><li>PK Index on: Latitude, Longitude, Climber_ID, Strt_DT </li></ul><ul><li>NPI: Climber_ID, Summit, O2, Guided </li></ul><ul><li>RUNSTATS TABLESPACE dbname.tsname INDEX(ALL) </li></ul><ul><ul><li>1 st Column & Fullkey Cardinality on all indexes </li></ul></ul><ul><ul><li>Top 10 most frequent values for 1 st column of all indexes </li></ul></ul><ul><li>RUNSTATS INDEX owner.ixname FREQVAL xx NUMCOLS xx </li></ul><ul><ul><li>Give me the top 15 climbed peaks (FREQVAL 15 NUMCOLS 2 for PK index) </li></ul></ul>PEAKS CLIMBED TABLE <ul><ul><ul><li>SELECT CLIMBER_ID, LATITUDE, LONGITUDE, O2, GUIDED FROM PEAKS_CLIMBED </li></ul></ul></ul><ul><ul><ul><li>WHERE CLIMBER_ID = ‘CE000008’ AND </li></ul></ul></ul><ul><ul><ul><li>SUMMIT = ‘YES’ ORDER BY O2, GUIDED WITH UR; </li></ul></ul></ul>SORT Required?
SELECT PEAKNAME FROM PEAKS WHERE STATE = ‘CO’; 20320 AK Mount McKinley … 14831 AK Mount Bear … Page 502 14494 CA Mount Whitney … 14420 CO Mount Harvard … Page 503 14410 WA Mount Rainier … 14270 CO Grays Peak … Page 504 14255 CO Longs Peak … 14196 CO Mount Yale … Page 505 14172 CO Mount Bross … 14110 CO Pikes Peak … Page 506 14080 CA Starlight Peak … 14059 CO Sunlight Peak … Page 507 14058 CA Split Mountain … 14037 CO Little Bear Peak … Page 508 14005 CO Mount o/t Holy Cross … 14000 CA Thunderbolt Peak … Page 501 p. 501, 1 Page 307 AK 20320 p. 501, 2 AK 14831 p. 502, 1 Page 308 CA 14494 p. 506, 1 CA 14080 p. 507, 1 Page 309 CA 14058 p. 508, 2 CA 14000 p. 502, 2 Page 310 CO 14420 p. 503, 2 CO 14270 p. 504, 1 Page 311 CO 14255 p. 504, 2 CO 14196 p. 505, 1 Page 312 CO 14172 p. 505, 2 CO 14110 p. 506, 2 Page 313 CO 14059 p. 507, 2 CO 14037 p. 508, 1 Page 314 CO 14005 p. 503, 1 WA 14410 SELECT PEAKNAME FROM PEAKS WHERE STATE = ‘CO’ AND ELEVATION =‘14110’; RUNSTATS INDEX DB2.STELEV FREQVAL 10 NUMCOLS 2 RUNSTATS TABLESPACE dbname.tsname INDEX(ALL) DATA Page LEAF Page CREATE INDEX DB2.STELEV ON DB2.PEAKS STATE, ELEVATION DEFER YES PIECESIZE 2G;
List Prefetch 20320 AK Mount McKinley … 14831 AK Mount Bear … Page 502 14494 CA Mount Whitney … 14420 CO Mount Harvard … Page 503 14410 WA Mount Rainier … 14270 CO Grays Peak … Page 504 14255 CO Longs Peak … 14196 CO Mount Yale … Page 505 14172 CO Mount Bross … 14110 CO Pikes Peak … Page 506 14080 CA Starlight Peak … 14059 CA Sunlight Peak … Page 507 14058 CA Split Mountain … 14037 CO Little Bear Peak … Page 508 14005 CO Mount o/t Holy Cross … 14000 CA Thunderbolt Peak … Page 501 p. 502, 1 Page 461 363438N 1181733W 370116N 1182520W p. 507, 1 p. 506, 1 Page 462 370539N 1183119W 370539N 1183124W p. 508, 2 p. 507, 2 Page 463 373400N 1052948W 373738N 1073543W p. 506, 2 p. 505, 2 Page 464 385026N 1050238W 385039N 1061848W p. 504, 2 p. 502, 2 Page 465 385528N 1061912W 392007N 1060625W p. 505, 1 p. 508, 1 Page 466 392805N 1062845W 393802N 1054901W p. 503, 2 p. 504, 1 Page 467 401517N 1053655W 465110N 1214531W p. 503, 1 p. 501, 2 Page 468 611702N 1410832W 630410N 1510013W p. 501, 1 p. 502, 1 RID LIST 363438N 1181733W 370116N 1182520W p. 507, 1 p. 506, 1 370539N 1183119W 370539N 1183124W p. 508, 2 373738N 1073543W p. 506, 2 p. 502, 1 Sorted RID LIST 363438N 1181733W p. 506, 1 370539N 1183119W 370539N 1183124W p. 508, 2 373738N 1073543W p. 506, 2 physical I/O order 370116N 1182520W p. 507, 1 (poor cluster ratio) = unwanted disk head movement SELECT ELEVATION, PEAKNAME FROM PEAKS WHERE LATITUDE IN (‘363438N’,’370116N’,’370539N’,’373738N’)
Tablespace Types Segmented Tablespace - 3 table example Segmented Tablespace - 1 table example Spacemap keeps track of segments (pages) by table. Spacemap page in a segmented tablespace helps when a program deletes without a where clause by marking those segments logically deleted. Spacemap page also makes segments tied to dropped tables accessible. Segsize between 4-64 increasing in increments of 4. See recommendations in notes section. Simple Tablespace - 1 table example Simple Tablespaces can have multiple tables; however, rows can be intermingled. Use with caution on small static read only tables. Partitioned Tablespace - NUMPARTS 18 Partitioned tablespaces allow only 1 table. Maximum number of partitions 254. Above 64 considered Large.
Table Space Scans Sequential I/O Random I/O SELECT PEAKNAME FROM PEAKS ORDER BY RATING; SELECT PEAKNAME FROM PEAKS WHERE STATE = ‘CA’ AND RATING = ‘MODERATE’; AK CA CO WA LEGEND Buffer Manager initiates a GETPAGE request for P1 through P16. Buffer Manager initiates a GETPAGE request for P3 and P13.
Sequential Detection 20320 AK Mount McKinley … 14831 AK Mount Bear … Page 502 14494 CA Mount Whitney … 14420 CO Mount Harvard … Page 503 14410 WA Mount Rainier … 14270 CO Grays Peak … Page 504 14255 CO Longs Peak … 14196 CO Mount Yale … Page 505 14172 CO Mount Bross … 14110 CO Pikes Peak … Page 506 14080 CA Starlight Peak … 14059 CO Sunlight Peak … Page 507 14058 CA Split Mountain … 14037 CO Little Bear Peak … Page 508 14000 CA Thunderbolt Peak … Page 501 SELECT PEAKNAME FROM PEAKS WHERE STATE = ‘CO’; 14005 CO Mount o/t Holy Cross … p. 507 p. 502 p. 503 p. 504 p. 505 p. 506 p. 508 p. 532 SELECT PEAKNAME FROM PEAKS WHERE STATE = ‘AK’; p. 564 p. 501 p. 502 p. 522 p. 542 p. 562 p. 566 p. 583 ACTIVE 6 out of 8 NOT ACTIVE 3 out of 8 (data access sequential = 4 out of last 8 pages are page-sequential) 1 1 1 1 1 1 24 1 20 20 20 2 2 17 p. 570 p. 630 38 47
Minimize Logging Default column placement from data model. Blue represents static columns that do not change once the row is created Purple represents column grouping updated associated to daily billing transactions Teal represents column grouping updated associated to monthly inventory transactions Dashed box represents variable data Adjusted column placement taking logging considerations into play.
Tuning SQL <ul><li>Tuning Steps </li></ul><ul><li>Static and Dynamic SQL </li></ul><ul><li>Get What You Need, Not What You Want </li></ul><ul><li>Reoptimization </li></ul><ul><li>Explain & Optimization Hints </li></ul>
WHAT is “Estimate & Select Access Path with Cheapest Estimated Cost ” <ul><li>Proper statistics can reduce the amount of SQL tuning required </li></ul><ul><li>Lackluster statistics can be a Pandora's box! </li></ul><ul><ul><li>REBIND DATE < RUNSTATS DATE </li></ul></ul><ul><ul><li>RUNSTATS performed yearly </li></ul></ul><ul><ul><li>EXPLAIN(NO) </li></ul></ul><ul><li>Optimizer does not use Real Time Stats for access path selection </li></ul><ul><li>Better filtering = fewer rows returned </li></ul>
Get Only What You Need data manager relational data system filtered result set GETPAGE REQUEST BP3 sort DBM1 Address Space buffer manger DSNDB07 DASD SELECT * FROM TBL data manager relational data system filtered result set GETPAGE REQUEST BP3 sort buffer manger DSNDB07 DASD SELECT C5,C6,C7 FROM TBL
Performance, Maintenance, Control DELETE RESTRICT IT_ROLE (parent) PROJECT_ROLE (child) PROJECT_ROLE (column constraint) . DDL change to add a new role (data modeler) . DBA involvement needed for the table change . Better Performance, Tighter Control . Insert new row to add a new role (data modeler) . User, Programmer, or DBA can insert the row . Slower performance (DB2 RI)
Getting Your Way <ul><li>REOPT(VARS) – </li></ul><ul><ul><li>Useful when DB2’s estimate of qualifying rows would benefit if the real host variable was available to the optimizer at execution </li></ul></ul><ul><ul><li>Evaluate data values at runtime </li></ul></ul><ul><ul><li>Limit parts for partition scans </li></ul></ul><ul><ul><li>Influence join sequence </li></ul></ul><ul><li>Reoptimization is done at open cursor </li></ul><ul><li>Static SQL – package level. Consider isolating/consolidating your reopt statements to a few static packages. </li></ul><ul><li>Dynamic SQL – statement level. More granular and easier to invoke on a smaller scale. </li></ul><ul><li>IFCID 0022 – Did you get a better access path? </li></ul><ul><li>Would static SQL benefit from being dynamic and taking advantage of dynamic cache? </li></ul><ul><li>Sounds Great! Difficult to implement and maintain. </li></ul><ul><li>DSNZPARM change to activate </li></ul><ul><li>Programmers should add QUERYNO to their code. </li></ul><ul><li>How are you going to name and manage your opthints? CHAR(8) </li></ul><ul><li>After code has been bound with explain yes, need to update plan_table rows to add a OPTHINT. </li></ul><ul><li>To return to the good access path you have previously established, you would rebind your package with OPTHINT(‘ohstring’). </li></ul><ul><li>Verify hint is in use! SQL code +394, HINT_USED column of PLAN_TABLE, or Query special register: CURRENT OPTIMIZATION HINT </li></ul>REOPTIMIZATION OPTIMIZATION HINTS
Access Paths <ul><li>More on QUERYNO = 103; </li></ul><ul><li>DB2’s Choices </li></ul><ul><li>Understanding Predicates </li></ul><ul><li>Small Tables in Memory </li></ul><ul><li>Plan_Table Considerations </li></ul>
ACCESSTYPE in Plan Table RID = record identifier ACCESNAME R Table space scan, segmented tablespace Table space scan, non-segmented tablespace Table space scan, partition scan (scanning limited parts) I mc=0 Non-matching index scan (scan of index leaf pages) One fetch index access (min/max) Perform 1 matching index scan for each value in (LIST) I1 MATCHCOLS PREFETCH N M Multiple index scan (same index twice or two different indexes) MX Qualifying RIDs for each index used in multiple index scan MI AND predicate, intersection of the qualifying RIDs from each index MU OR predicate, union of the qualifying RIDs from each index OPTHINT INDEXONLY PRIMARY_ACCESSTYPE D Direct row access via ROWID (page#, rid). One getpage! I mc>0 Matching index scan (using IX structure - root, non-leaf, leaf)
Predicates <ul><li>Know Your Predicates </li></ul><ul><ul><li>Indexable, matching predicates on index key columns </li></ul></ul><ul><ul><li>Stage 1, not picked as matching but still refer to index columns (index screening) </li></ul></ul><ul><ul><li>Stage 1, operating on a data page </li></ul></ul><ul><ul><li>Stage 2, everything else </li></ul></ul><ul><li>Application Programming and SQL Guide </li></ul><ul><ul><ul><li>Table 68, page 629, DB2 for OS/390 and z/OS V7 </li></ul></ul></ul>