The document discusses optimization of dynamic SQL statements through the use of SQL packages. SQL packages allow the access plans for dynamic SQL statements to be shared across users and connections, improving performance over traditional dynamic SQL. When a prepared dynamic SQL statement is executed, the optimizer can leverage the existing access plan in the SQL package rather than generating a new plan. This approach makes the performance of dynamic SQL more comparable to static SQL.

3. APPLICATION ENVIRONMENT
INVESTIGATION
TUNNING TUNNING
• Client: OS, Environment • Data Providers (5) • Host Servers
• Application: Design, Code • Data Access Optimization • Work Management
• Server: OS, DB • Rendering Optimization • Memory Utilization
• Network • Coding Best Practices • Query Optimization
• CPU Utilization
• Application Stress
• Database Optimization
• Database Monitors
• Communication Optimization

4. MySales Web Application MySales Subsystem Shared Pool Environment
Connection Pool 1 MySales DB/ QZDASOINIT
Temp
JOBD
Results
Connection Pool 2 MySales SignOn
Temp
CLASD
Indexes
MySales CMD
Job QAQQI
Cache NI
MySales DRDA
Event Visual Job Plan DB
Viewer Explain Log Cache Monitor
Monitor Monitor

6. CLIENT SERVER
Application Application
Data Provider/Driver Database
Runtime Environment Runtime Environment
Java/.NET Subsystems
Operating System Operating System
Network Software Network Software
Hardware Hardware
Hardware Software Protocol Bandwidth

7. • System Configuration
• Topology
• Network I/O
System •
•
Disk I/O
Database Optimization
• Operation System
• Application Design/Code
• Application/Web Server Tuning
• Data Providers/Drivers
Application •
•
Data Access Optimization
Clustering
• Caching
• Processors
• Memory
• Disks
Machine • JVM/CLR Tuning
• Communication
• Cache Architecture
• Other Hardware

9. Data
Processing
Host Server/
Network
DBMS
Performance
Factors
Application Query

10. Application
Design
Garbage Data
Collection Providers
Connection
Caching
Pooling
Application
Resource
Locking
Pooling
SQL Managed
Statements Code

11. Journaling
CPU Index
Operations Maintenance
Data
Processing
I/O Constraint
Operations Enforcement
Trigger
Locking
Processing

12. Server
Attributes
Server OS Version
Configuration
Server SMP
Performance
Job, Query System i Database
attributes Design
DBMS
SQL
Table sizes
Request
SQL Views &
Interfaces Indexes
Work
Management

13. Memory
Utilization
Fixed/Calc
Work Prestart
Managemen Jobs
t QZDASOINIT
Data SQL
Providers Statements
Query Index
Options Query Recomme-
QAQQINI ndations
Journaling Compilation
Analysis Options
Unused
System
Indexes &
Values
Views
Query
Engines
(SQE/CQE)

14. Indexes
Advised
Print SQL
SQE Plan
Information
Cache
Messages
Visual
Explain
Debug Tools & SQE Plan
Job log Cache
Messages Methods Snapshots
Detailed DB
Views &
Monitor
Indexes
Data
Summarized
DB Monitor
Data

15. Time spent in data access:
• In the processor
• Waiting for disk I/O
• Waiting for communications I/O
• Waiting for some resource that some other job is using
In most environments the first two are the largest factors.

16. Parse Optimize Open Run

17. SQL Interface
SQL Request processing
Static SQL
Requester Syntax Gather Build internal query Dynamic SQL
Communication Validation Attributes structures Extended Dynamic
SQL
SQL Optimizer
Create & implement the data manipulations
Query
Query Query Query methods Query
implementation
validation dispatching costing plan creation
ODP
Database Engine
executing the query plan that the optimizer provides
Build the structures Build the structures Build & activate Generate any
Debug messages in Gather database Use DB2 Visual
needed for query for any temporary query cursor feedback
the job log monitor records Explain
cursor indexes (if needed) (ODP) requested

19. • Static SQL
SQL Code hardcoded (embedded ) into the application.
SQL that has been pre-compiled into a package or plan may be executed
directly by DB2 without additional preparation.
• Dynamic SQL
SQL text is provided by the user or generated by the application at execution
time SQL that must be prepared for execution every time at run time.
• Extended Dynamic SQL
Prepare once then reference.
Host Server CLI/JDBC
Static Extended
Dynamic
Compiled
Prepare every
Dynamic
Embedded Prepare once then
time
Statements reference

22. • Check whether this is a new request.
• If this request has not been optimized previously, no plan currently exists, thus full optimization is required.
• If this request has previously been optimized, the saved plan must be validated to ensure that it is viable for the current request.
• If the plan is valid, optimization can be eliminated or minimized.
Query • If the plan is not valid, full optimization is required. A plan can be invalidated when the query environment changes or the database
objects change.
validation
• Determine which query engine can complete the processing.
• The goal is to let the SQE optimize the query.
• Depending on the DB2 release level, the SQE might be incapable of running a particular query.
Query
dispatcher
• Determine what methods are available to access the data.
• Choose the best strategy to be employed.
Query Costing • The methods and strategy are assembled into a Access Plan that the database engine executes.
Plan Creation
• DB2 optimizer creates the ODP for the query.
• The ODP consists of the cursor, cursor behavior, and data mapping constructs.
• This phase is also where the optimizer manifests any feedback information about the query and query plan.
Query • Finally, the plan is handed off to the database engine for execution.
Implementation

24. • Two SQL/Query engines.
• The Optimizer now determines which engine will handle the query request through the query dispatcher:
• Classic Query Engine (CQE)
• SQL Query Engine (SQE)
• There is still only one interface into the optimizer.

25. ODBC/JDBC/ADO/DRDA/XDA
Network
Host Server CLI/JDBC
Static Dynamic Extended
Compiled Prepare every Dynamic
Embedded time Prepare once then
Statements reference
SQL
Native
Record I/O
Optimizer
Machine Interface (MI)
DB2 UDB (Data Storage & Management)
SLIC (System Licensed Internal Code)
CQE Database Engine SQE Database Engine

26. ODBC/JDBC/ADO/DRDA/XDA
Network
The Only way for HLL Host Server CLI/JDBC
programs to use the
Static Dynamic Extended
new engine (SQE) is via Compiled Prepare every Dynamic
embedded SQL Embedded time Prepare once then
Statements reference
SQL
Optimizer
Query Dispatcher
Native
Record I/O
CQE Optimizer SQE Optimizer
Machine Interface (MI) The optimizer &
database engine
SLIC (System Licensed Internal Code)
merged, to form the
DB2 UDB (Data Storage & Management) Database Engine SQL Query Engine &
much of the work was
SQE Optimizer moved to
SLIC
SQE Statistics Manager
CQE Database Engine SQE Data Access Primitives

27. SQL Options V5R2 V5R2 V5R3 V5R3 V5R4 V5R4 V6R1 V6R1
CQE SQE CQE SQE CQE SQE CQE SQE
LIKE Predicates Y Y Y Y
Logical File references Y Y Y Y
UDTFs Y Y Y Y
LOB columns Y Y Y Y
LOWER, TRANSLATE or UPPER scalar Y Y Y Y
CHARACTER_LENGTH, POSITION, or Y Y Y Y
SUBSTRING scalar using UTF-8/16
Alternate sort sequences Y Y Y Y
Derived Logical Files over Physical (S/O) Y Y Y Y
Non-SQL : QQQQry API, Query/400, OPNQRYF Y Y Y Y
ALWCPYDTA(*NO) Y Y Y Y
Sensitive Cursor Y Y Y Y
VIEWS, UNIONS, SubQueries Y Y Y Y
INSERT, UPDATE, DELETE Y Y Y Y
Star Schema Join queries Y Y Y Y

29. Query Dispatcher,
• Determines which engine optimizes and processes query requests
• Considers SQL requests only for the query engines
• Serves as the initial step for all query optimization in the i5/OS environment
• Provides support to use the classic SQE when encountering nonstandard indexes during optimization
Remaining SQE
Older Technolo Restrictions as of V5R4:
DDS LF References, DDS LF References,
DDS Select/Omit LF, DDS Select/Omit LF,
Non-SQL Interfaces Non-SQL Interfaces
Optimizer
Query Dispatcher
CQE Optimizer SQE Optimizer
DB2 UDB (Data Storage & Management) Database Engine
CQE SQE

30. Query Request is Dispatched to CQE if:
• Logical file references
• UDTFs
• LOWER, TRANSLATE, or UPPER scalar function
• CHARACTER_LENGTH, POSITION, or SUBSTRING scalar function
using UTF-8/UTF-16
• Sort sequences and CCSID translation between columns
• DB2 Multisystem
• Non-SQL queries (QQQQry API, Query/400, OPNQRYF)
SQE now (V5R4) optimizes:
• LIKE predicates
• LOB columns
• Sensitive cursors
• ALWCPYDTA(*NO)

31. Revert from SQE to CQE if the optimizer encounters:
• Select/omit logical file
• Logical file over multiple members
• Join logical file
• Derived key(s)
• Native logical files that perform some intermediate mapping of the fields
referenced in the key: renaming fields, adding a translate, or only selecting a
subset of the columns
• Specifying an alternate collating sequence (ACS) on a field used for a key will also
make a derived key(an implied map occurs within the index).
• Sort sequence (NLS) specified for index or logical file
• Cost to back up and revert to CQE adds about 15% to the total optimization time.
• QAQQINI parameter to ignore unsupported logical files
IGNORE_DERIVED_INDEX = *YES

33. We can affect
THIS!
Query Query
dispatcher • Choose most efficient execution
• Validate the query
request access method
• Validate existing query • Determine which query • Build access plan
access plan engine can complete
the processing • Build the structures needed for
query cursor
• Build internal query
structures • Build the structures for any
temporary indexes (if needed)
Query Query • Builds-and activate query cursor
Parsing & (ODP)
optimization
validation
• Generate any feedback
requested
• Debug messages in the job log
• Gather database monitor records
• Use DB2 Visual Explain

34. • Validate the query
request
• Validate existing query
access plan
• Build internal query
structures
Query
Parsing &
validation

36. • Non-dynamic SQL statements embedded in application programs.
• Validation:
• SQL pre-compiler validates and parses the SQL statements in the HLL
program.
• Turns them into host language statements.
• Optimization:
• An access plan is then created for each SQL statement and is stored in the
program object or module object.
• Since all optimization parameters are not known prior to actual execution,
the optimizer creates only a generic optimization plan during pre-compile.
• Execution:
• Binding: Host language statements interface with the database manager
upon execution of the program.
• The plan receives detailed analysis on the first execution of the statement,
and then the optimizer updates the original access plan.

37. • Languages Supported:
• RPG
• COBOL
• C, C++
• PL/I
• SQLJ
• Embedded SQL is the most efficient SQL interface to use on System i
since SQL Access Plan is stored in the program objects.
• When the program is run, the validation and access-plan creation steps
can be skipped. thus, improving performance and making the entire
query effort much faster.

38. CRTSQLxxx
Create
Executable
Parse
&
Access Executable
Plan Validate
Optimize &
Create
Access Plan
* Generic plan quickly generated at compile time
* Complete, optimized plan first execution time

40. • With dynamic execution, the SQL statement is completely ignored at program preparation time and
constructs SQL statements on the fly – at runtime.
• So, at runtime the application has to communicate more information with the database, before it
executes the statement.
• The operational form of the statement (ODP) persists for the duration of the connection or until the
last SQL program leaves the call stack.
• Access plans associated with dynamic SQL might not persist after a database connection or job is
ended.
• DB2 for i interfaces that utilize Dynamic SQL:
• CLI • Embedded Dynamic SQL
• JDBC • RUNSQLSTM
• ODBC, OLEDB, .NET • Interactive SQL (STRSQL)
• PHP • System i Navigator SQL requests
• SQLJ • DB2 Web Query
• REXX • Net.Data
• Greater performance overhead.
• Less sharing & reuse of resources – using JOB Working Memory

41. PREPARE and EXECUTE
EXECUTE IMMEDIATE
Create
Executable
(Prepare)
Parse
&
Access Validate
Plan
Optimize &
Create
Access Plan
* Generic plan quickly generated at prepare time
* Complete, optimized plan first execution time

42. One simple way to think of this is that static vs. dynamic SQL is similar to compiled
vs. interpretive applications in that the executable code is generated before
execution time rather than during execution time. Because there is a CPU cost
associated with analyzing an SQL statement and creating an access plan, dynamic
SQL can hinder application performance.
Cost of
Application
PREAPARE &
Requirements
OPTIMIZATION
• CPU cost of Preparation • Needs of the application
• Cost of doing repetitive Optimization • Skill set of developers
• Cost of caching options • Supporting infrastructure

44. • An extended dynamic SQL statement is neither fully static nor fully dynamic.
• Permanent system object (SQL package) is used to store the access plans.
• SQL packages are used to make the implementation of the dynamic SQL
statements similar to that of static SQL.
• SQLPKG allows Dynamic SQL access plans to be shared across
users and to remain intact after a job or connections ends.

45. QSQPRCED API
ODBC/JDBC
Has this Dynamic
request been
Create
previously
Executable
executed?
(Prepare)
Parse
&
Access
Plan Validate
Optimize &
Create
Access Plan
* Generic plan quickly generated at prepare time
* Complete, optimized plan first execution time

46. • SQL packages are permanent objects with the object type *SQLPKG used to store information related to
prepared, extended dynamic SQL statements.
• SQL package contains all the necessary information to execute the prepared statement.
• When using embedded SQL, no separate SQL package is created, but the access plan is integrated into
the program or service program object.
• For environments which make use of the SQL Query Engine (SQE), SQL packages no longer contain
the entire access plan which is instead stored in the SQL Plan Cache. Even in these environments, the
information in the SQL package provides better reuse of SQL statement information.
• An SQL statement goes into the package only if one of the following is true:
• The statement contains parameter markers.
• It is an INSERT with sub-select (INSERT INTO table1 SELECT FROM table2 WHERE...)
• It is a positioned UPDATE or DELETE.
• It is a SELECT FOR UPDATE

47. • Any SQL statement that can be prepared is eligible
• Unlike SQL programs where all access plans are rebuilt when the associated DB2 table definitions are
changed, there are times when the plans stored in an SQL Package are not automatically rebuilt by DB2
UDB for iSeries. So, recreate your SQL packages any time the definitions for your database objects have
changed.
• Size limitations
• Current size limit is 500 MB, about 16K statements
• Package can grow without new statements being added.
• Access plan rebuilds require additional storage
• DB2 does try to perform package compression in the background to increase life & usefulness of
package objects

48. Advantages of extended dynamic SQL packages:
• Shared resource available to all users
• Access information reuse eliminates need for others to "relearn" SQL statement
• Permanent object saves information across job/system termination (IPL)
• Can be saved/restored to other systems
• Improved performance decisions since statistical information accumulates for each SQL statement
The statistics kept in the object include:
• number of times the package is used,
• number of times a packaged statement is executed,
• number of rows fetched by a statement.
With these pieces of information, statements in an SQL package tend to go into reusable ODP mode
after the first execution, which rapidly improves performance, automatically!

49. System API — QSQPRCED
• API user responsible for creating package
• API user responsible for preparing and describing statement into package
• API user responsible for checking existence of statement and executing statements in package
XDA API set
• Abstraction layer built on top of QSQPRCED for local and remote access
IBM Client Access ODBC driver & IBM Toolbox for Java JDBC driver Drivers
• Handle package creation
• Drivers automate the process of adding statements into package
• Drivers automate process of checking for existing statement and executing statements in package

50. • Choose most efficient
access method
• Build access plan
Query
optimization

52. • An access plan is a control structure that describes the actions necessary to satisfy each query request.
• Think of this plan as the program that the database engine runs.
• An access plan or query plan is the output of the query optimization process.
• An access plan includes all of the optimized information that is necessary to accomplish the query.
Statement
Name &
Text
Associated Internal
Tables & parse tree
fields of statement
Query
Access Plan
isolation
level &
Statistics
commitment
control level

53. The access plan is validated when the query is opened. In validation the following is verified:
• Different File Or Member(change in library list or default schema)
• More Than Ten Percent Change In Number Of Rows
• New Access Path Found(Index created)
• Access Path No Longer Found Or Valid(Index deleted)
• Different CCSID
• Different Date Or Time Format
• Different Sort Sequence Table
• Different Storage Pool or Paging Option(Optimizer‘s fair share of memory pool change)
• Symmetric Multi Processing Change
• QAQQINI Change
• Different Isolation Level or Scroll Option
• New Release

54. The DB2 for i Optimizer performs ―Cost Based" optimization.
The goal for the DB2 for i optimizer is to produce an Access Plan that will
allow the query to execute in the shortest time period possible.
The optimizer chooses an optimal access method for the query by calculating an
implementation cost based on the current state of the database.
The optimizer uses 2 costs when making decisions:
• I/O cost
• CPU cost.
The goal of the optimizer is to minimize both I/O and CPU cost.
The optimizer has the ability and freedom to ―Rewrite the Query”.

55. Determining cost of using existing indexes:
The Optimizer orders the indexes:
• For SQE, the indexes are ordered in general such that the indexes that access the smallest number
of entries are examined first.
• For CQE, the indexes are generally ordered from mostly recently created to oldest.
For each index available, the optimizer does the following:
• Determines if the index meets the selection criteria.
• Determines the cost of using the index by estimating the number of I/Os and the CPU cost that will be
needed to perform the Index Probe (or the Index Scan and the possible Table Probes).
• Compares the cost of using this index with the previous cost (current best).
• Picks the cheaper one.
• Continues to search for best index until the optimizer decides to look at no more indexes:
• For SQE, since the indexes are ordered so that the best indexes are examined first, once an index
that is more expensive than the previously chosen best index, the search is ended.
• For CQE, the time limit controls how much time the optimizer spends choosing an implementation.

56. A given query plan can be thought of as an
intersection of all the factors that affect cost based
optimization on a given server with a given
database design.
To really understand a given implementation plan
and it performance, one must know and understand
all the various factors and settings in effect at the
time of query optimization and execution.
Change any one or more of the factors and the
implementation plan and performance may change.

57. Job &
Query
Server QUERY
ACCESS Database
Software
Hardware PLAN
Request

58. Server
Attributes
QPFRADJ,
Server QQRYDEGREE
Configuration Server
Sub-Systems, Performance
Pools,
Activity Levels
CPU
Work
No. of Management
Processors QUERY
ACCESS
PLAN
Memory
OS
Main Storage
Version
Size
Server Model
& SMP
Architecture

59. Query Time
Limit
Allow Copy
Data
QAQQINI
ALWCPYDTA
Naming
CCSID
Convention
QUERY
ACCESS
PLAN
JOB
Optimization Description
Goal
*LIBL, Priority,…
Data Source
Driver
Configuration
JDBC,ODBC

60. Views
Indexes
MQTs
Radix, EVI
QUERY
Table
Statistics ACCESS Sizes
PLAN
Cached
Results Database
Design
MTI
Optimizer
CQE/SQE

61. Optimization
Goal
SQL Result t set
Statement Size
QUERY
Commitment
Static ACCESS Control
PLAN
Sensitivity
Dynamic
(Live Data)
Extended
Dynamic

62. How does the optimizer know which choice to make?

63. Many data access methods can satisfy a query (each with its own strengths
and weaknesses)
What data access method can be used to find the rows that contain the
CITY TELAVIV within a 1-million-row database table?
... WHERE CITY = ‘TELAVIV’
When...
• 1 row contains the city ‗TELAVIV‘.
• 1000 rows contain the city ‗TELAVIV‘.
• 100,000 rows contain the city ‗TELAVIV‘.
• 1,000,000 rows contain the city ‗TELAVIV‘.
How does the optimizer know which choice to make?
TABLE SCAN, INDEX, …

64. • If the correct statistics are collected and available, the cost-based optimizer more accurately
estimates the number of rows to process.
• Better estimates allow for better query optimization and the selection of the best query plan
• All query optimizers rely on statistics to make plan decisions.
• DB2 for i5/OS has always relied on indexes as its source for statistics.
• Other databases rely on manual statistics collection for their source.
• SQE offers a hybrid approach.
• Column statistics are automatically collected when indexes do not already exist.

65. ...WHERE Customer_No > 112358 GROUP BY Customer_No...
• No Index • No Index • Index
• No Statistics • Statistics • Statistics
• Q1 optimizes with • Q1 optimizes with • Q1 optimizes with
defaults statistics statistics & index
• runs without index • runs without index • runs with index
• Queues up statistics
request for Customer_No
Generate Create
Statistics for Index for
Customer_No Customer_No

66. Table Radix EVI
CQE Optimizer
Statistics
Analysis Access
SQL Query
Plan
Costing

67. Table
With Radix EVI
Statistics
Statistics
Manager
Q&A
SQE
SQL Query Optimizer Access
Plan
Costing

68. Selectivity
Cardinality
Metadata
I/O
Estimation

69. • More accurately describes multicolumn key value BEST
• Available immediately as index maintenance occurs
• Selectivity estimates from radix by reading n keys
• Selectivity from EVI by reading symbol table values
Existing Indexed
(Radix or EVI)
• SQE only
• Column cardinality, histograms, and frequent values list
• Constructed over a single column in a table
• Stored internally as a part of the table object after created
• Collected automatically by default for the system
• Statistics are not immediately maintained as the table changes
Column Statistics
• Statistics are refreshed as they become stale over time
• No representation of actual values in columns
Default sources WORST

70. • i5/OS statistics collection job - QDBFSTCCOL
• This job is reactive, based on query requests.
• Automatic collection runs in this background job at a very low priority.
• The statistics manager continuously analyzes entries in the plan cache and queues
up requests for the collection job.
• Options:
• *ALL Allow both user & system requested stats collections
• *NONE No stats collection is allowed
• *USER Only user requested stats collection allowed
• *SYSTEM Only system requested stats collection allowed
• iSeries Navigator GUI helps manage statistics collected by the system.
• There are also APIs to manage the statistics.

71. How does the optimizer know which choice to make?

72. If the goal is: First vehicle to the 10 meter mark, who will win?
The goal
determines
the plan
10 Meter
If the goal is: First vehicle to the 10 kilometer mark, who will win?
10 kilometer

73. The optimization goal:
• Tells the optimizer how many rows you expect to fetch per transaction.
• Optimizer builds a plan that is optimal for returning n or all rows expected
• Affects the query startup time and overall runtime
All rows
Next n rows
Next n rows
Next n rows
First n rows
First I/O All I/O
Read by Build and
key thru an use a hash
index table

74. Optimization goal will affect the optimizer's decisions, such as Use of
indexes, SMP, temporary intermediate results like hash tables
• Set via optional SQL statement clause
• OPTIMIZE FOR n ROWS
• OPTIMIZE FOR ALL ROWS
• Set via QAQQINI options file
• *FIRSTIO
• *ALLIO
• Default for dynamic interfaces is First I/O
• ODBC, JDBC, STRSQL, dynamic SQL in programs
• CQE - 3% of expected rows
• SQE - 30 rows
• Otherwise default is ALL I/O
• Extended dynamic, RUNSQLSTM, INSERT + subSELECT, CLI, static SQL in programs
• All expected rows

75. The Open Data Path is the actual pipe for moving data between the database and applications.
The creation of the Open Data Path is very expensive, in terms of performance, on iSeries servers.
Avoiding the creation of ODPs (Full Open) is the key to delivering high-performing SQL solutions on
iSeries server.
ODPs live in working memory of the job associated with the SQL request.

77. • The SQE Plan Cache is always on, automatic and has no Database Monitor Overhead.
• Initially created with an overall size of 512 MB.
• The purposes of the Plan Cache are to:
• Reuse of a query access plan when the same query is re-executed
• Store runtime information for subsequent use in future query optimizations
• Once an access plan is created, it is available for use by all users and all queries, regardless
of where the query originates.
• Plans are optimized on-demand as new statistics or indexes become available.
• Foundation for a self-learning query optimizer to interrogate the plans to make wiser costing.
• Caches all access plans optimized by the SQE Optimizer.
• Access plans generated by CQE are not stored in the Plan Cache, instead, they are stored in
SQL Packages, the system-wide statement cache, and job cache.
• Works in conjunction with the system wide statement cache and the SQL programs,
packages and service programs.

78. • Multiple access plans (3) can be maintained for a single SQL statement, processed by SQE.
• Although the SQL statement is the primary hash key to the plan cache, a different
environmental setting can cause different access plans to be stored in the plan cache,
each one matching the specific environment.
Examples of these environmental settings include:
• Different SMP degree settings for the same query
• Different library lists specified for the same SQL request
• Different settings for the job‘s share of available memory in the current pool
• Different ALWCPYDTA settings
• Plan cache is automatically maintained to keep most active queries available for reuse.
As new access plans are created for the same SQL statement, older and less frequently
used access plans are discarded to make room for the new access plans.
• Conditions that can cause an existing access plan to be invalidated.
Examples:
• Specifying REOPTIMIZE_ACCESS_PLAN(*YES) or (*FORCE) in the QAQQINI
• Deleting or recreating the table that the access plan refers to
• Deleting an index that is used by the access plan
Plan cache is cleared during an IPL.

81. • DB2 for i also caches access plans for Dynamic SQL requests in the System Wide Statement Cache
(SWSC)
• Only access plans are reused (No ODP reuse)
• SWSC requires no administration
• Cache storage allocation & management handled by DB2
• Cache is created from scratch each IPL
• Cache contents cannot be viewed, max of 165,000+ statements
• SWSC cache does interact with the job cache

83. • With Dynamic interfaces, full opens are avoided by using a "PREPARE once, EXECUTE many―.
statement
• A PREPARE does NOT automatically create a new statement and full open on each execution
• DB2 UDB performs caching on Dynamic SQL PREPAREs within a job
• DB2 UDB caching is not perfect (and subject to change), good application design is the only way to
guarantee ODP reuse

84. Perform Parameter
Marker Conversion
No MATCH (ODP or access plan)
NOT FOUND
found in job cache
Search the
SWSC Add statement
and its plan to
Search JOB Cache
for ODP‘s or
SWSC
Access Plans that
can be reused FOUND Match
in SWSC
Check that job
attributes match
Update the JOB Cache
entry for the statement SWSC attributers
to point at the plan in
the SWSC Attributes did
NOT match
Job & SWSC Create new plan
attributes
MATCHED for statement and
add plan to the
JOB Cache
Use SWSC
access plan

86. Dynamic SQLs

88. • Maintained Temporary Index (MTI) = Autonomic Indexes = Result Set Cache
• Classic Query Engine (CQE) has had the ability to create a temporary index but its usage is restricted
to a single job (not across queries) and single query only.
• MTIs created by SQE can be shared across queries and jobs like permanent indexes.
• The amount of temporary storage used by the database can be noticeably more than in previous
releases.
• SQE automatically controls the creation, maintenance and deletion of MTIs.
• Creation occurs when SQE can justify that a query's performance can be enhanced sufficiently by
using an MTI.
• An MTI is updated as the base table is updated.
The system cost of maintaining an MTI is similar to the cost of maintaining a permanent index.
• An MTI is deleted when:
• The last access plan, in the system plan cache, that refers to that MTI is removed.
• On IPL:
• MTIs are deleted at IPL time, so you may experience a post-IPL warm-up effect for
the queries that leverage MTIs.
• When a permanent index is created that covers the same columns as the MTI

89. How does the optimizer know which choice to make?

90. Indexes
Advised
Print SQL
SQE Plan
Information
Cache
Messages
Visual
Explain
System i
Debug Query SQE Plan
Job log Cache
Messages Optimization Snapshots
Tools
Views & Detailed DB
Indexes Monitor Data
Summarized
DB Monitor
Data

92. System
Resources
CPU Memory Disks

93. • Work management influences SQL performance.
• Size of pool determines which algorithms are used by the query optimizer.
• Use of Expert Cache *CALC helps database optimization and runtime.
• Lower activity levels let SQL use more resources.
Subsystem
Memory Pool Activity
Pool Size Tuning Level

94. CPU:
• SQE uses threads instead of tasks for splitting up a job
• Then, SQE uses fewer tasks.
• SQE can use SMP for executing the threads within an access plan
• SMP can also be used by the statistics engine to collect statistics for SQE
Memory:
• SQE uses a slightly smaller main storage footprint for a query compared to CQE.
• SQE can make its best computation only when the associated memory pool is
defined with a pool paging option of *CALC (expert cache)
• SQE Optimizer responds differently to changes in the memory pool size in
terms of rebuilding an access plan.
Disk Storage:
• SQE performs asynchronous I/O far more aggressively than CQE and fully uses
parallel pre-fetching of data.
• Therefore, any under configuration of the number of disk drives on a system is
accentuated by queries that use SQE, particularly those that access a large
number of table rows.

95. How does the optimizer know which choice to make?

96. FAIR SHARE MEMORY is:
Maximum amount of memory that can be allocated to a
query without having a detrimental effect on other jobs
running in the same memory pool.

97. Plan 1
Index Probe into Index
Index
Memory
Footprint
Query‘s
Fair Share
Plan 2
Hash probe into hash table
Hash Table
Memory
Footprint

98. • CQE fair share = memory pool size / MAX activity level value.
• CQE Optimizer rebuilds an access plan if there is a 2-fold change in size.
• SQE fair share = memory pool size / AVG activity level value
• SQE looks for a 10-fold change.
• AVG = min { max-active, max(avg-active, 5) }
• if Pool paging = *CALC THEN
• 15 minute AVERAGE number of users
avg-active =
• ELSE IF Pool paging = *FIXED THEN
• MAX activity level value
• If query degree is set to *MAX, then fair share = entire pool size

100. • Enabling SMP for a given job allows that job to use multiple tasks or threads to
perform the work.
• Those multiple tasks or threads consume more resources with the goal of
faster response times.
• Multiple processors allow the tasks or threads to run in parallel.
In other words, more work is accomplished in the same unit of time.

101. 64 BIT
Processors
• Multiple processors
Memory
• N-way
Single Level
Query
Request
Storage • SMP
Storage Management
IOP IOP IOP IOP IOP IOP IOP IOP
Table

102. 64 BIT Processors
Job A Thread J
Job B Thread I
Job C Thread H
Job D Thread G
Job E Job F
• n processors can work on several jobs or threads at one time without any special programming.
• No one job is running on more than one processor.

103. Job A 64 BIT Processors
Thread J
Thread Aa
Thread Ab
Thread Ac Thread I
Thread Ad
Thread Ae
Thread H
Thread Af
Thread Ag
Thread Ah
Thread G
Job B Job C
•The system automatically divides the query work into multiple tasks or threads.
• Multiple processors can work on one job’s tasks or threads.

104. Parallel processing allows a user to specify that queries should be able to
use either I/O or CPU parallel processing as determined by the optimizer.
• I/O parallelism allows for the accessing of data in parallel, but the processing of that data does not
occur in parallel.
• Processor/CPU parallelism allows for both the accessing of data and the processing of that data in
parallel.
• CPU parallelism is only available when DB2 Symmetric Multiprocessing is installed
• CPU parallelism does not necessarily require multiple processors
• Parallel processing is set on a per-job basis:
• The parameter DEGREE on the CHGQRYA CL command.
• The parameter PARALLEL_DEGREE in the QAQQINI file.
• The system value QQRYDEGREE.
• Each job will default to the system value (*NONE is the default).
• SMP can be used for parallel index creation on restore or recovery at IPL.
System value QQRYDEGREE controls rebuild of access paths on restore and IPL.
• DB2 Symmetric Multiprocessing is option 26 of System i

106. • *NONE
No parallel processing is allowed for database query processing.
• *IO
Any number of tasks may be used when the database query optimizer
chooses to use I/O parallel processing for queries. CPU parallel processing is
not allowed. SQE always considers IO parallelism.
• *OPTIMIZE
The query optimizer can choose to use any number of tasks or threads for either
I/O or CPU parallel processing to process the query. Use of parallel processing
and the number of tasks or threads used will be determined with respect to the
number of processors available in the system, this job's share of the amount of
active memory available in the pool which the job is run, and whether the
expected elapsed time for the query is limited by CPU processing or I/O
resources.

107. • *MAX
The query optimizer can choose to use either I/O or CPU parallel processing
to process the query. The optimizer will assume that all active memory in the
pool can be used to process the query.
• *SYSVAL
Use current value of the system value QQRYDEGREE.
• *NBRTASKS nn
• Specifies the number of tasks or threads to be used when the query
optimizer chooses to use CPU parallel processing to process a query. I/O
parallelism will also be allowed.
• Used to manually control the degree value

108. • Application environments that can use and benefit from parallelism
• SQL requests that use methods that are parallel enabled
• Longer running or complex SQL queries
• Longer running requests like index creation
• Few or no concurrent users running in the same memory pool
• Willing to dedicate most or all the resources to the specific SQL request(s)
• Native, record level access from within HLL programs is not enabled for SMP.
• Computing resources
• > 1 (physical) CPUs
• 4-8GB memory per CPU
• 10-20 disk units per CPU
• 60% or less average CPU utilization during the time interval of the request
• Setting the memory pool‘s paging option to *CALC allows the database engine to be
more intuitive and more aggressive with I/O requests.
• The optimization goal "ALL I/O" tends to allow SMP, while "FIRST I/O" does not.

109. • Parallel Data load fully utilizes SMP capabilities
• CPYFRMIMPF and CPYTOIMPF CL commands
• Works with fixed format and delimited files
• Import from stream files (IFS), source files, tape files and more
CPYFRMIMPF
FROMSTMF('~mydir/myimport.txt')
TOFILE(MYLIB/MYTABLE)
DTAFMT(*DLM) FLDDLM(',’)

110. • Lazy Close
• Reuse open connections
• Good for applications such as MS Access
• Data Compression
• Enabled by default
• For clients not CPU bound
• Block with a fetch of 1 row
• Advanced option
• Test, incompatible with some applications
• Record blocking
• Default 32kb
• For read only increase dramatically
• Query Optimization Goal (V5R4)
• *ALLIO or *FIRSTIO
• Extended Dynamic
• For subsequent requests of the same query
• Connection Pooling

111. How does the optimizer know which choice to make?

112.  Previous environment
 Data Warehouse Load programs written in RPG using non-SQL I/O
 Add/Update 3 tables, total refresh of an additional 5 summary tables
 Read logical file, chain to another file to get ―group by‖, and add/update
 7 Hours to complete
 Current Environment
 RPG altered to use embedded SQL for 3 table add/updates
 5 Summary tables replaced by MQTs (Materialized Query Tables)
 Refreshed with ―Refresh Table‖ SQL Command
 Binary Radix and Encoded Vector Index techniques used
 Result
 7 Hour job reduced to…..

114. DESCRIPTION OF PROBLEM FIXED FOR APAR MA : -----------------------------------------------Users report drastic
spikes in CPU when the SQE plan cache reaches it's maximum size. When the max size of the SQE plan cache is reached
two things happen First, the plan cache pruner task is no longer allowed to sleep. Second, no more plans are allowed to be
inserted in the plan cache. This results in excessive CPU in full opens as plans are removed and no new plans are allowed
in. The observations on the system are When plan cache Memory usage becomes near the max over size, it starts
decreasing When plan cache Memory usage starts decreasing, the rate of "Number of times Hit" starts decreasing too.
" Total size of all MTIs" starts decreasing too. " . Full Optimizations" starts increasing at the same time. These symptoms
can cause CPU usage to increase significantly. In reviewing the code a couple of changes were decided on. ) The pruner
task not sleeping and no inserts action in the plan cache are always controlled together when the over max size is
exceeded and turn this combination off much sooner than is done currently. CORRECTION FOR APAR MA : ------
-----------------------The operating system code will be changed to limit the high CPU occurances that occur when the
percent over maximum memory usage for the plan cache is met. Specifically, the amount of time that plans are not allowed
to be insert in the plan cache and the amount of time that plans are forcefully removed from the plan cache, will be limited
MF40965 - LIC-DB CPU SPIKES WHEN PLAN CACHE REACHES MAXIMUM SIZE

116. What’s wrong with the DataSet?
I‘m not saying that there‘s anything inherently wrong with the DataSet object. But it‘s like any other tool—
you need to understand how to use it appropriately. Although it‘s a useful tool for Windows Forms
applications, it‘s much less useful for Web application development.
Let‘s look at a simple example. Suppose you use a DataSet to return a set of 1,000 products to display in
a DataGrid on a form. Since you might want to sort or filter the data later, you choose to save the DataSet
in a session variable. Not knowing any better, you also leave the default page ViewState turned on. When
a user navigates to this page, there are three copies of the data somewhere in memory. It‘s on the server
saved in a session-level variable. It‘s in the ViewState stored as the contents of the DataGrid. And it‘s in
the rendered HTML stream in the form of HTML table directives that render the table. Now multiply the
server memory by the number of users to assess impact on server memory, and multiply the two copies of
the data by the number of users to assess the impact on bandwidth utilization. You can quickly overload a
server and its available network bandwidth on a high-traffic site.
The answer: Use the DataReader
Though not as sexy, the DataReader is much more functional for a Web application. Because the DataSet
object‘s cursor is designed to iterate in a forward-only, read-only fashion over the results of a query, it‘s
very fast. Moreover, the DataReader only holds the current record in memory at any one time—never the
entire results set. The DataSet object can be bound to ASP.NET Server Controls (like the DataGrid). More
importantly, server resources and connection resources are released as soon as you‘re finished traversing
them. Build your data-bound pages using DataReaders to retrieve data from an underlying database
whenever it‘s important for the data to be as fresh as possible.
http://articles.techrepublic.com.com/5100-
10878_11-1045330.html

129. <add name="GAL_SM_KS―
connectionString="Provider=IBMDA400; Password=OPTODBC; Persist Security Info=True;
User ID=OPTODBC; Data Source=192.168.240.1; Force Translate=0"/>
<add name="ODBC2_GAL_SM_KS―
connectionString="Driver={Client Access ODBC Driver (32-bit)}; System=192.168.240.1;
Uid=OPTODBC; Pwd=OPTODBC; Cursor Sensitivity=0; QueryTimeOut=0; TRANSLATE=1"/>
<add name="ODBC_GAL_SM_KS―
connectionString="Driver={Client Access ODBC Driver (32-bit)}; System=192.168.240.1;
Uid=optodbc; Pwd=optodbc; QueryTimeOut=0; TRANSLATE=1; XDYNAMIC=1;
DFTPKGLIB=OPTODBC; PKG=LIBRARY/DEFAULT(IBM),2,0,0,0,0; CMT=0; LAZYCLOSE=1;
QAQQINILIB=OPTODBC; COMPRESSION=1; PREFETCH=1; Cursor Sensitivity=0;
DEBUG=4;TRACE=12"/>
<add name="IDB2_GAL_SM_KS―
connectionString="DataSource=192.168.240.1; UserID=OPTIDB2; Password=OPTIDB2;
DataCompression=true; Pooling=true; MaximumPoolSize=-1; MaximumUseCount=1000;
MinimumPoolSize=2; CheckConnectionOnOpen=false; Trace=StartDebug, PrintJobLog;"/>
<add name="DB2_GAL_SM_KS―
connectionString="Database=S653afa2; Pooling=true; Min Pool Size=2;"/>
<add name="DD_GAL_SM_KS"
connectionString="Charset For 65535=37; Host=192.168.240.1; Port=446;
Database Name=S653afa2; User ID=OPTDD; Password=OPTDD; Persist Security Info=True;
Pooling=True; Min Pool Size=2; Connection Reset=False; Statement Cache Mode=Auto;
Max Statement Cache Size=20;"/>

130. Before:
Dim con As OdbcConnection
Dim cmd As OdbcCommand
Dim da As OdbcDataAdapter
Dim dt As DataTable
con = New OdbcConnection(ConnectionString)
cmd = New OdbcCommand(SqlString, con)
da.Fill(dt)
After:
Dim con As iDB2Connection
Dim cmd As iDB2Command
Dim da As iDB2DataAdapter
Dim dt As DataTable
con = New iDB2Connection(ConnectionString)
iDB2ProviderSettings.DecNumericAsString = True
iDB2ProviderSettings.CharBitDataAsString = True
cmd = New iDB2Command(SqlString, con)
da.Fill(dt)

131. Investigation:
Application Code Review - .Net
Backend Review – OS, Database, Code
Web Stress Load
Profiling
Protocol Sniffing
Query Feedback Mechanisms
Application Tuning:
Data Providers: Configuration & Implementation (5)
Data processing Optimization: Prepared Statements, Connection Pooling, Blocking, ….
Rendering Optimization: Data Readers,…
Communication Optimization: IP/Port Mapping, Authentication, …
Code Rewrite
iSeries Tuning:
Host Servers/Subsystems
Query Optimization & Execution: SQL Packages,
Memory Usage Optimization: Caching, …
CPU Utilization
Database Indexes

132. The memory-sharing algorithms discussed above provide balanced performance for all the jobs running in
a memory pool. Running short transactional queries in the same memory pool as long running, data
intensive queries is acceptable. However, if it is desirable to get maximum performance for long-running,
data-intensive queries it may be beneficial to run these types of queries in a memory pool dedicated to
this type of workload. Executing long-running, data-intensive queries in the same memory pool with a
large volume of short transactional queries will limit the amount of memory available for execution of the
long-running query. The plan choice and engine execution of the long-running query will be tuned to run
in the amount of memory comparable to that available to the jobs running the short transactional queries.
In many cases, data-intensive, long-running queries will get improved performance with larger amounts
of memory. With more memory available the optimizer is able to consider access plans which may use
more memory, but will minimize runtime. The query engine will also be able to take advantage of
additional memory by keeping more data in memory potentially eliminating a large number of DASD
I/Os. Also, for a job executing long-running performance critical queries in a separate pool, it may be
beneficial to set QQRYDEGREE=*MAX. This will allow all memory in the pool to be used by the job to
process a query. Thus running the longer-running, data intensive queries in a separate pool may
dramatically reduce query runtime.

133. LAN Performance Considerations
The following parameters for the line description (LIND) and the controller
description (CTLD) have a significant impact on performance. Select one of the
parameters for performance related information.
MAXFRAME (LIND and CTLD)
For a more detailed discussion on iSeries LAN support and for additional LAN
performance considerations, refer to the following manuals:
· iSeries Communications Management, SC41-3406
· iSeries Local Area Network Support, SC41-3404

134. TCP/IP Performance Considerations
When connecting to the server using TCP/IP, you may be able to improve performance with a few simple changes to the server TCP/IP
configuration.
TCP/IP Interface and Route Configuration
Currently the server defaults to a Maximum Transmission Unit (MTU) of 576 when you add a route to the configuration (through CFGTCP
option 2 or ADDTCPRTE). This value ensures packets will not be dropped over this route because all TCP/IP implementations have to
support at least a 576-byte transmission unit.
In many cases, however, this value is unnecessarily small because this route will only be used on this Ethernet or token-ring, and there
are no intermediate hops that only support a 576-byte packet. If this is the case, you should change the Route Maximum Transmission
Unit size to *IFC. This will change the MTU on the Route to the Interface MTU size which defaults to the Line Description Frame Size.
This defaults to approximately 2000 for token-ring and 1500 for Ethernet. There are also cases where the 576-byte size can cause
adapter overruns that put TCP/IP in retransmit mode and slow things down considerably.
This change often results in a considerable increase in your throughput over TCP/IP, especially when installing products.
TCP/IP Send and Receive Buffer Size
Use CFGTCP option 3, Change TCP/IP attributes to change the defaults used for the following:
· TCP/IP receive buffer size
Consider increasing the TCP receive buffer size from the default size of 8192 bytes to a larger value, for example, 64384 bytes. This
value specifies the amount of data the remote system can send before being read by the local application. If a large number of buffer
overruns are occurring on the network adapter, decreasing this value, instead of increasing it, could help performance.
· TCP/IP send buffer size
Consider increasing the TCP send buffer size from the default size of 8192 bytes to a larger value, for example, 64384 bytes. This value
provides a limit on the number of outgoing bytes that are buffered by TCP. If TCP has to resend the data many times due to a buffer
overrun condition of a network adapter, decreasing this value, instead of increasing it, could help performance.

145. p=0c50 t=0bd0 206960671+ 0 [08] Comm-Base 32-bit :
PiCoParms Dump: systemName: 199.1.1.200 ipAddressLookupMode: 0
portLookupMode: 2 sslEnabled: 0 ipAddrOverride: callback: 0xA6792E0
pSecurity: 0xA6761E8 timeout: 30 rcvThread: 0 flowStartServer: 1 workQ: 1
serverID: 0xE004 remotePort: 8471 perfType: 2 service: 4 serviceName:
as-database recvCacheSize: 0 sendCacheSize: 16384 sendThreshHold:
16384 sendMaxCount: 999 flushSendsAll: 1 wsSendBufferSize: 16384
wsRecvBufferSize: 4294967295 nagleEnabled: 0 keepAlivesEnabled: 0
bindRandom: 0 sendTimeout: 0 recvTimeout: 0 sendMaxSize: 2147483647