The document provides an overview of new features and enhancements in DB2 10 and 11 for z/OS related to query optimization and performance. Key highlights include improvements to predicate application such as support for IN-list matching, predicate pushdown, and transitive closure. Other areas covered are safe optimization techniques, parallelism enhancements including dynamic partitioning, plan management capabilities, and additional features in DB2 11 related to predicate indexability and duplicate removal.
In this presentation Skillwise provides you the Cobol Programming Basics. COBOL is primarily used in business, finance, and administrative systems for companies and governments.
Easytrieve plus is an information retrieval and data management system. It is a primitive form of 4GL whose English like language and simple syntax provide the tools for easy data retrieval and report generation, Easytrieve Plus can now produce reports in HTML format. This feature allows you to create reports that can be viewed either from a local disk or LAN or from a World Wide Web server using the Web browser.
A command macro is a text file that contains one or more commands and/or command macro statements and that has a name ending with the suffix .cm. You invoke a macro by entering its name. The operating system then sequentially executes the commands and macro statements in the file.
A command macro is useful to replace one of the following:
a long command string that you issue frequently
a group of internal commands, command macros, and/or program modules that together perform an operation you regularly require
a sequence of internal commands, command macros, and/or program modules that you execute in a certain way depending on external factors
IBM Db2 11.5 External Tables to Db2 customers provides a powerful secure and high performing way to run SQL against NoSQL and SQL data external to the database and drastically improve data ingestion rates as well as unloading of data for single instance and MPP deployments. Increasing both performance , simplicity and efficiency of day to Day to day operations. It also opens up a wide array of scenarios for users. This Presentation takes you through the detail of the Db2 11.5 implementation.
This slide contains all the basic concepts of ISPF. It's giving the simple and easy step to get the knowledge of Interactive system productivity facility. If u like it then give me feedback on email anilbharti85@gmail.com Thanks v much.
A K Bharti
In this presentation Skillwise provides you the Cobol Programming Basics. COBOL is primarily used in business, finance, and administrative systems for companies and governments.
Easytrieve plus is an information retrieval and data management system. It is a primitive form of 4GL whose English like language and simple syntax provide the tools for easy data retrieval and report generation, Easytrieve Plus can now produce reports in HTML format. This feature allows you to create reports that can be viewed either from a local disk or LAN or from a World Wide Web server using the Web browser.
A command macro is a text file that contains one or more commands and/or command macro statements and that has a name ending with the suffix .cm. You invoke a macro by entering its name. The operating system then sequentially executes the commands and macro statements in the file.
A command macro is useful to replace one of the following:
a long command string that you issue frequently
a group of internal commands, command macros, and/or program modules that together perform an operation you regularly require
a sequence of internal commands, command macros, and/or program modules that you execute in a certain way depending on external factors
IBM Db2 11.5 External Tables to Db2 customers provides a powerful secure and high performing way to run SQL against NoSQL and SQL data external to the database and drastically improve data ingestion rates as well as unloading of data for single instance and MPP deployments. Increasing both performance , simplicity and efficiency of day to Day to day operations. It also opens up a wide array of scenarios for users. This Presentation takes you through the detail of the Db2 11.5 implementation.
This slide contains all the basic concepts of ISPF. It's giving the simple and easy step to get the knowledge of Interactive system productivity facility. If u like it then give me feedback on email anilbharti85@gmail.com Thanks v much.
A K Bharti
Online Statistics Gathering for Bulk Loads - the official name of the feature - was introduced in Oracle 12.1. The idea is to gather optimizer statistics "on the fly" for direct path loads. Sounds good for ETL? In certain scenarios it makes sense but even then there are many points to consider so that it becomes a reliable part of your ETL processes. When exactly will it be working and when not? Do you prevent it yourself? Documented, undocumented cases, known bugs. Which statistics are gathered and which are not? What has to be considered with partitioned tables? Interval partitioning - special case?
Mainframe jcl exec and dd statements part - 3janaki ram
EXEC STATEMENT(EXEC)
EXEC Statement is used to identify program name or procedure name.
Maximum we can code 255 EXEC statements in a JOB.
In EXEC statement has two kinds of parameters
Position parameter Keyword parameter
PGM TIME , COND
PROC REGION , PARM
If you not mention any thing default it takes PROC
PGM
This is a positional parameter which must be coded after EXEC with one blank. This parameter indicates the name of program (or) procedure name
Syntax
//STEPNAME EXEC PGM=REPORT (OR)
PROC = PROCEDURE NAME
PARM
It is keyword parameter and it is mainly used for passing the data to another program.
To pass input to Application program
To invoke complier option.
This is a keyword parameter at activity / step level must be coded with exec operand after PGM parameter.
This parameter allows MAX(100) characters.
To receive PARM parameter data, Cobol program must be coded with “Procedure Division Using Parameter ” option.
These parameters must be declared in the linkage section.
DATA DESCRIPTOR STATEMENT ( DD STMT)
It is used to identify files( input and output ) used in JCL
DD name act as a bridge b/w Cobol program and execution JCL
In DD statement has two kinds of parameters.
Position Parameter Keyword Parameter
* DSN,DISP
DATA SPACE,UNIT
DUMMY DCB,VOLUME
*
Position parameter at DD level used with SYSIN to pass data to Cobol programs this is known as In stream data any number of records can be passed to the program.
In stream data is used to pass values from JCL to Cobol dynamically.
To accept the values in Cobol program, we should have equivalent accept verbs.
Syntax
//SYSIN DD * ------- Entry of in stream data
100
200
/* ---------------------- End of in stream data
DATA
We can pass any thing to another program including special character
Syntax
//SYSIN DD DATA
100
/*
200
/*
DUMMY
All files are treated as End of the file
Syntax
//SYSIN DD DUMMY
NOTE
There is no input to the application all input files treated as End of the file.
DSN
Through DSN we can refer temporary / permanent file
Is a physical space or file where the records will be stored.
DISP
DISP parameter is used to identify the mode of the file.
DISP= ( STATUS, NORMAL , ABNORMAL )
TERMINATION TERMINATION
NEW DELETE DELETE
OLD KEEP KEEP
SHR CATLG CATLG
MOD UNCATLG UNCATLG
NEW Creating the data set first time
OLD Already created accessing the data set all resources are allocated exclusively . No other user can until is released by the current user. If dataset is not existing this creates.
SHR The data set can be accessed by multiple users at a time.
MOD Appending the records to the existing record
NOTE If the dataset is not existing , then MOD is assumed as NEW and writes records to the dataset.
CICS is the power of mainframe. It has all the capabilities to handle online transactions. The ppt covers highly useful CICS concepts to refresh your CICS knowledge quickly.
JCL
Job Control Language (JCL) is a name for scripting languages used On IBM mainframe operating systems to instruct the system on how to run a batch job or start a subsystem.
JCL acts as an interface between application programming and MVS Operating system.
Jcl is used for compilation and execution of batch programs.
Apart from the above functionalities JCL can also be used for,
1. Controlling the jobs.
2. Create GDG’S.
3. Allocate PDS,PS file with IBM Utilities.
4. Create Procs.
5. Sort the files.
JCL Coding Sheet
1,2,3----------Column Numbers----------72,73------------80
//JOBNAME JOB PARAMETERS------COMMENTS
// EXEC
// DD
//* ------------ Comment (* in 3rd column indicates line in comment)
//------------ End of JCL
Where // ----- Identification Field
job name------- Naming field
JOB,EXEC,DD - Statement / Operation
NOTE
If we want to continue parameters in the next line end the last parameter with “,” and continue next parameter only in 4-16 columns.
There are three statements in JCL.
JOB
EXEC
DD
JOB Statement:
Job statement is used to identify job name and job related parameters
JOBCARD = job name + job related parameters.
Syntax
//JOBNAME JOB ACCOUNT INFORMATION,’USERNAME’,CLASS=A-Z/0-9,
// NOTIFY =&SYSUID/RACF ID,MSGCLASS,
// MSGLEVEL=(X,Y),PRTY=0-15,
// TIME=(M,S),REGION=MB/KB,TYPRUN=SCAN/
// HOLD/COPY,COND=(RC,OPERATOR,STEPNAME)
// COND=ONLY OR COND=EVEN,RESTART=STEPNAME
JOBNAME
It is required to identify this job from other jobs in the SPOOL
1 to 8 characters minimum 1 character and maximum is 8 character.
1st character must be alphabet.
Other characters can be alphabets or numeric or $,&,#.
Example
Job names for personal or lab sessions
Userid + 1 / 2 chars
KC03P83$ ------- Userid is KC03P83
KC03P84&--------Userid is KCO3P84
ACCOUNTING INFORMATION
It is a keyword parameter and codes it after JOB statement.
It is used for billing purpose, in real time when we submit any job it is going to take some CPU time. Based on the CPU time there will be some amount involved where this amount has to go will be decided by A/C information parameters.
Examples
(8012T)
(80121I)
(8012M)
USERNAME
It is used to identify the user who has written the JCL.
It can be maximum of 20 characters.
Note Both A/C information and user name are positional parameters and the remaining job
card parameters are keyword parameters.
EX1: //KC03P83A JOB (487A),’JANAKI RAM’
EX2: //KC03P84& JOB (488T),’SOMISETTY’
NOTIFY
To which user id the job has to be notification after successful or unsuccessful completion.
Successful completion means MAXCC = 0 (or) 04 unsuccessful completion means MAXCC > 04.
If it is not coded,then user has to check the status of the job from the spo
Online Statistics Gathering for Bulk Loads - the official name of the feature - was introduced in Oracle 12.1. The idea is to gather optimizer statistics "on the fly" for direct path loads. Sounds good for ETL? In certain scenarios it makes sense but even then there are many points to consider so that it becomes a reliable part of your ETL processes. When exactly will it be working and when not? Do you prevent it yourself? Documented, undocumented cases, known bugs. Which statistics are gathered and which are not? What has to be considered with partitioned tables? Interval partitioning - special case?
Mainframe jcl exec and dd statements part - 3janaki ram
EXEC STATEMENT(EXEC)
EXEC Statement is used to identify program name or procedure name.
Maximum we can code 255 EXEC statements in a JOB.
In EXEC statement has two kinds of parameters
Position parameter Keyword parameter
PGM TIME , COND
PROC REGION , PARM
If you not mention any thing default it takes PROC
PGM
This is a positional parameter which must be coded after EXEC with one blank. This parameter indicates the name of program (or) procedure name
Syntax
//STEPNAME EXEC PGM=REPORT (OR)
PROC = PROCEDURE NAME
PARM
It is keyword parameter and it is mainly used for passing the data to another program.
To pass input to Application program
To invoke complier option.
This is a keyword parameter at activity / step level must be coded with exec operand after PGM parameter.
This parameter allows MAX(100) characters.
To receive PARM parameter data, Cobol program must be coded with “Procedure Division Using Parameter ” option.
These parameters must be declared in the linkage section.
DATA DESCRIPTOR STATEMENT ( DD STMT)
It is used to identify files( input and output ) used in JCL
DD name act as a bridge b/w Cobol program and execution JCL
In DD statement has two kinds of parameters.
Position Parameter Keyword Parameter
* DSN,DISP
DATA SPACE,UNIT
DUMMY DCB,VOLUME
*
Position parameter at DD level used with SYSIN to pass data to Cobol programs this is known as In stream data any number of records can be passed to the program.
In stream data is used to pass values from JCL to Cobol dynamically.
To accept the values in Cobol program, we should have equivalent accept verbs.
Syntax
//SYSIN DD * ------- Entry of in stream data
100
200
/* ---------------------- End of in stream data
DATA
We can pass any thing to another program including special character
Syntax
//SYSIN DD DATA
100
/*
200
/*
DUMMY
All files are treated as End of the file
Syntax
//SYSIN DD DUMMY
NOTE
There is no input to the application all input files treated as End of the file.
DSN
Through DSN we can refer temporary / permanent file
Is a physical space or file where the records will be stored.
DISP
DISP parameter is used to identify the mode of the file.
DISP= ( STATUS, NORMAL , ABNORMAL )
TERMINATION TERMINATION
NEW DELETE DELETE
OLD KEEP KEEP
SHR CATLG CATLG
MOD UNCATLG UNCATLG
NEW Creating the data set first time
OLD Already created accessing the data set all resources are allocated exclusively . No other user can until is released by the current user. If dataset is not existing this creates.
SHR The data set can be accessed by multiple users at a time.
MOD Appending the records to the existing record
NOTE If the dataset is not existing , then MOD is assumed as NEW and writes records to the dataset.
CICS is the power of mainframe. It has all the capabilities to handle online transactions. The ppt covers highly useful CICS concepts to refresh your CICS knowledge quickly.
JCL
Job Control Language (JCL) is a name for scripting languages used On IBM mainframe operating systems to instruct the system on how to run a batch job or start a subsystem.
JCL acts as an interface between application programming and MVS Operating system.
Jcl is used for compilation and execution of batch programs.
Apart from the above functionalities JCL can also be used for,
1. Controlling the jobs.
2. Create GDG’S.
3. Allocate PDS,PS file with IBM Utilities.
4. Create Procs.
5. Sort the files.
JCL Coding Sheet
1,2,3----------Column Numbers----------72,73------------80
//JOBNAME JOB PARAMETERS------COMMENTS
// EXEC
// DD
//* ------------ Comment (* in 3rd column indicates line in comment)
//------------ End of JCL
Where // ----- Identification Field
job name------- Naming field
JOB,EXEC,DD - Statement / Operation
NOTE
If we want to continue parameters in the next line end the last parameter with “,” and continue next parameter only in 4-16 columns.
There are three statements in JCL.
JOB
EXEC
DD
JOB Statement:
Job statement is used to identify job name and job related parameters
JOBCARD = job name + job related parameters.
Syntax
//JOBNAME JOB ACCOUNT INFORMATION,’USERNAME’,CLASS=A-Z/0-9,
// NOTIFY =&SYSUID/RACF ID,MSGCLASS,
// MSGLEVEL=(X,Y),PRTY=0-15,
// TIME=(M,S),REGION=MB/KB,TYPRUN=SCAN/
// HOLD/COPY,COND=(RC,OPERATOR,STEPNAME)
// COND=ONLY OR COND=EVEN,RESTART=STEPNAME
JOBNAME
It is required to identify this job from other jobs in the SPOOL
1 to 8 characters minimum 1 character and maximum is 8 character.
1st character must be alphabet.
Other characters can be alphabets or numeric or $,&,#.
Example
Job names for personal or lab sessions
Userid + 1 / 2 chars
KC03P83$ ------- Userid is KC03P83
KC03P84&--------Userid is KCO3P84
ACCOUNTING INFORMATION
It is a keyword parameter and codes it after JOB statement.
It is used for billing purpose, in real time when we submit any job it is going to take some CPU time. Based on the CPU time there will be some amount involved where this amount has to go will be decided by A/C information parameters.
Examples
(8012T)
(80121I)
(8012M)
USERNAME
It is used to identify the user who has written the JCL.
It can be maximum of 20 characters.
Note Both A/C information and user name are positional parameters and the remaining job
card parameters are keyword parameters.
EX1: //KC03P83A JOB (487A),’JANAKI RAM’
EX2: //KC03P84& JOB (488T),’SOMISETTY’
NOTIFY
To which user id the job has to be notification after successful or unsuccessful completion.
Successful completion means MAXCC = 0 (or) 04 unsuccessful completion means MAXCC > 04.
If it is not coded,then user has to check the status of the job from the spo
Performance Tuning Oracle's BI ApplicationsKPI Partners
http://www.kpipartners.com/webinar-Performance-Tuning-Oracle-BI-Applications/ ... From a virtual event that discusses techniques that can be used to optimize performance of the Oracle BI Apps.
The BI Apps from Oracle present customers with a nice head start to getting their BI environment up and running. But for many customers, their user community demands lighting-fast speeds while running dashboards, reports and ad-hoc queries. Learn about some of the key techniques you can use to take the BI Apps to performance levels you didn’t think were possible.
The discussion begins with a conceptual understanding of why performance problems can exist and the counteracting design considerations. Special attention will be paid to the concept of a Performance Layer, describing what it is, what it is comprised of and how to build it. The presentation includes several real world examples of the significant performance gains that can be had from a Performance Layer.
Objective 1: Learn about the concept of a performance layer and what is involved with building one.
Objective 2: Understand the most important steps to improve the performance of your system.
Optimizer is the component of the DB2 SQL compiler responsible for selecting an optimal access plan for an SQL statement. The optimizer works by calculating the execution cost of many alternative access plans, and then choosing the one with the minimal estimated cost. Understanding how the optimizer works and knowing how to influence its behaviour can lead to improved query performance and better resource usage.
This presentation was created for the workshop delivered at the CASCON 2011 conference. Its aim is to introduce basic optimizer and related concepts, and to serve as a starting point for further study of the optimizer techniques.
Stephan Hummel – IT-Tage 2015 – DB2 In-Memory - Eine Technologie nicht nur fü...Informatik Aktuell
Die DB2 In-Memory-Technologie (BLU Acceleration) beschleunigt analytische Abfragen um ein Vielfaches. Dies gilt sowohl für OLAP als auch für Analysen in einer OLTP-Umgebung. Durch die Integration in den DB2-Kernel ab Version 10.5 sind bestehende Datenbanksysteme bereits für die In-Memory Nutzung vorbereitet. Dadurch ist eine flexible und schnelle Umsetzung gewährleistet.
Ims12 workbench data visualization - IMS UG May 2014 Sydney & MelbourneRobert Hain
Analyzing problems with transactions on z/OS can feel like measuring a strand of cotton when your starting point is a shirt: you need to dissect individual aspects of the transaction without losing the overall picture of how they fit together. That means knowing where and how to get logs for various subsystems, relating these logs together, and finally interpreting the combined output.
IBM Transaction Analysis Workbench for z/OS is a tool that provides a coherent picture of a transaction across subsystems - including IMS, DB2, CICS, WebSphere MQ, and z/OS itself - helping you to pinpoint the source of problems. We demonstrate a step-by-step proof-of-concept model for visually interacting with composite log data to help identify and resolve problems involving multiple subsystems.
This session is aimed at the regular ISPF user who wants to learn about recent features of ISPF that can make life easier, and also at those that want to learn about the new features for ISPF in z/OS V2R2.
This session is aimed at the regular ISPF user who wants to learn about recent features of ISPF that can make life easier, and also at those that want to learn about the new features for ISPF in z/OS V2R2.
This session is aimed at the regular ISPF user who wants to learn about recent features of ISPF that can make life easier, and also at those that want to learn about the new features for ISPF in z/OS V2R2.
ERP Magazine is a monthly publication focusing on ERP/SAP software technical insights and innovations, for both new and advanced learners. In every issue, leading authors and consultants from a diverse set of companies present their ideas and/or achievements in a clear fashion. Each article is carefully reviewed both by technical and English-language experts. The graphics are rich in content and visual style, and the articles feature illustrative real-world coding. Our readers are Basis, ABAP, Workflow, Web Dynpro, HANA, Fiori and Mobility consultants. We publish SAP/ERP technical articles in a variety of lengths and sizes. And we include general tips and feature-length articles as well. We will be showcasing a number of published book chapters in upcoming editions, along with including white papers about useful tools and products.
ERP Magazine April 2018 - The magazine for SAP ABAP ProfessionalsRehan Zaidi
ERP Magazine is a monthly publication focusing on ERP/SAP software technical insights and innovations, for both new and advanced learners. In every issue, leading authors and consultants from a diverse set of companies present their ideas and/or achievements in a clear fashion. Each article is carefully reviewed both by technical and English-language experts. The graphics are rich in content and visual style, and the articles feature illustrative real-world coding. Our readers are Basis, ABAP, Workflow, Web Dynpro, HANA, Fiori and Mobility consultants. We publish SAP/ERP technical articles in a variety of lengths and sizes. And we include general tips and feature-length articles as well. We will be showcasing a number of published book chapters in upcoming editions, along with including white papers about useful tools and products. Available on Amazon Worldwide in Kindle format.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
5. Improvements to predicate application
•
Major enhancements to OR and IN predicates
• Improved performance for AND/OR combinations and long IN-lists
• General performance improvement to stage 1 predicate processing
• IN-list matching
•
•
•
•
Matching on multiple IN-lists
Transitive closure support for IN-list predicates
List prefetch support
Trim IN-lists from matching when preceding equals are highly filtering
• Range-list Access for SQL pagination
• Single index matching for complex OR conditions
•
Many stage 2 expressions to be executed at stage 1
• Stage 2 expressions eligible for index screening
5
6. IN-list Table - Table Type 'I' and Access Type 'IN'
• The IN-list predicate will be represented as an in-memory table if:
– List prefetch is chosen, OR
– More than one IN-list is chosen as matching.
– The EXPLAIN output associated with the in-memory table will have:
• New Table Type: TBTYPE – ‘I’
• New Access Type: ACTYPE – ‘IN’
SELECT *
FROM T1
WHERE T1.C1 IN (?, ?, ?);
QBNO PLANNO METHOD TNAME
1
1
6
1
2
0
1
ACTYPE MC ACNAME
DSNIN001(01) IN
T1
I
0
1
QBTYPE
TBTYPE PREFETCH
SELECT I
T1_IX_C1 SELECT T
L
7. IN-list Predicate Transitive Closure (PTC)
SELECT *
FROM T1, T2
WHERE T1.C1 = T2.C1
AND T1.C1 IN (?, ?, ?)
AND T2.C1 IN (?, ?, ?)
Optimizer can generate
this predicate via PTC
• Without IN-list PTC (DB2 9)
– Optimizer will be unlikely to consider T2 is the first table accessed
• With IN-list PTC (DB2 10)
– Optimizer can choose to access T2 or T1 first.
7
8. Range-list Access for SQL Pagination
•
Scroll forward to obtain the next 20 rows
• Assumes index is available on (LASTNAME, FIRSTNAME)
• WHERE clause may appear as:
WHERE (LASTNAME='JONES' AND FIRSTNAME>'WENDY')
OR (LASTNAME>'JONES')
ORDER BY LASTNAME, FIRSTNAME;
• DB2 10 supports
• Single matching index access with sort avoided
• DB2 9 requires
• Multi-index access, list prefetch and sort
• OR, extra predicate (AND LASTNAME >= ‘JONES’) for matching single index
access and sort avoidance
•
8
NOTE: APAR PM56355 to encourage range-list access with OFnR and extra predicate
9. Stage 2 predicates “pushed down” to IM/DM
• Most Stage 2 (residual) predicates can execute as index screening
(indexable) or as stage 1 (sargable)
– CPU time improvement
– Reduced data getpages if stage 2 predicate becomes index screening
– Applies to
• Arithmetic/datetime expressions, scalar built-in functions, CASE, CAST, (essentially
all expressions without subqueries)
• OR’d predicates cannot span different predicate stages
• Externalized in DSN_FILTER_TABLE column PUSHDOWN
• Not enabled for List Prefetch access type
9
10. Stage 2 predicates “pushed down” to IM/DM
• Examples
– Suppose there exists index on (C1,C3)
– … WHERE SUBSTR(C1,1,1) = ?
==> index screening
– … WHERE SUBSTR(C1,1,1) = ? OR C3 = ? ==> index screening
– … WHERE SUBSTR(C1,1,1) = ? OR C4 = ? ==> stage 1
– … WHERE SUBSTR(C1,1,1) = ? AND C4 = ? ==> index screening
and stage 1
– … WHERE SUBSTR(C1,1,1) = ? OR C3 = (SELECT...) ==> stage 2
– … WHERE SUBSTR(C1,1,1) = ? AND C3 = (SELECT...) ==> index scr.
and stage 2
10
12. Minimizing Optimizer Challenges
• Potential causes of sub-optimal plans
– Insufficient statistics
– Unknown literal values used for host variables or parameter markers
• Optimizer will evaluate the risk for each predicate
– For example: WHERE BIRTHDATE < ?
• Could qualify 0-100% of data depending on literal value used
• As part of access path selection
– Compare access paths with close cost and choose lowest risk plan
12
13. Minimizing impact of RID failure
• RID overflow can occur for
– Concurrent queries each consuming shared RID pool
– Single query requesting > 25% of table or hitting RID pool limit
• DB2 9 will fallback to tablespace scan*
• DB2 10 will continue by writing new RIDs to workfile
– Work-file usage may increase
• Mitigate by increasing RID pool size (default increased in DB2 10).
• MAXTEMPS_RID zparm for maximum WF usage for each RID listd
* Hybrid join can incrementally process. Dynamic Index ANDing will use WF for failover.
13
15. Parallelism Enhancements - Effectiveness
• Previous Releases of DB2 use Key Range Partitioning
– Key Ranges Decided at Bind Time
– Based on Statistics (low2key, high2key, column cardinality)
• Complaint is often that data is not evenly distributed across child tasks.
• DB2 10 solutions available to the optimizer
– Dynamic Record Range partitioning
• Introduce a sort to redistribute the data evenly at execution time
– Straw Model Parallelism
• Create more work elements than there are concurrent tasks
– As one child task completes, it takes the next off the queue
15
16. Key range partition - Today
SELECT *
FROM Medium_T M,
Large_T L
WHERE M.C2 = L.C2
AND M.C1 BETWEEN (CURRENTDATE-90) AND CURRENTDATE
Large_T
10,000,000 rows
C2
C3
Medium_T
10,000 rows
C1
C2
Workfile
12-31-2007
3-degree parallelism
25%
SORT
ON C2
09-30-2007
08-31-2007
2,500 rows
05-01-2007
04-30-2007
01-01-2007
Partition the
records according
to the key ranges
M.C1 is date column, assume currentdate is 8-31-2007, after the
between predicate is applied, only rows with date between
06-03-2007 and 8-31-2007 survived, but optimizer chops up the key
ranges within the whole year after the records are sorted :-(
16
5,000,000 rows
17. Dynamic record range partition
SELECT *
FROM Medium_T M,
Large_T L
WHERE M.C2 = L.C2
AND M.C1 BETWEEN (CURRENTDATE-90) AND CURRENTDATE
Medium_T
10,000 rows
C1
C2
3-degrees parallelism
Workfile
SORT
ON C2
2,500 rows
Partition the records each range has same
number of records
17
Large_T
10,000,000 rows
C2
C3
18. STRAW Model
SELECT *
FROM
Medium_T M
WHERE M.C1 BETWEEN 20 AND 50
Medium_T
10,000 rows
C1
C2
Medium_T
T a s k
2
1
50
40
30
degree = 3
20
degree=3
#ranges=10
0
index on C1
Divided in key ranges before DB2 10
18
10,000 rows
C1
C2
3
T a s k
100
T a s k
index on C1
100
50
47
44
41
38
35
32
29
26
23
20
0
Divided in key ranges with Straw Model
20. Extending VOLATILE TABLE usage
•
VOLATILE TABLE support added in DB2 V8
• Targeted to SAP Cluster Tables
• Use Index access whenever possible
• Avoids list prefetch
• Can be a problem for OR predicates or UPDATEs at risk of loop
•
DB2 10 provides VOLATILE to general cases
• Tables matching SAP cluster tables will maintain original limitations
• Table with 1 unique index
• Tables with > 1 index will follow NPGTHRSH rules
• Use Index access whenever possible
• No limitation on list prefetch
• Less chance of getting r-scan when list-prefetch plan is only alternative
20
21. Misc Performance enhancements
• Index INCLUDE columns
– Create an Index as UNIQUE, and add additional columns
– Ability to consolidate redundant indexes
INDEX1 UNIQUE (C1)
INDEX2 (C1,C2)
21
Consolidate to
INDEX1 UNIQUE (C1) INCLUDE (C2)
23. Plan Management (aka Access Path Stability)
• Plan management provides protection from access path
(performance) regression across REBIND/BIND
– Access path fallback to prior (good) access path after REBIND
• DB2 9 PLANMGMT(EXTENDED/BASIC) with SWITCH capability
– DB2 10 (APAR PM25679 – July 2011)
• Freeze access path across BIND/REBIND
– BIND/REBIND PACKAGE … APREUSE(ERROR)
• Access path comparison with BIND/REBIND
– BIND/REBIND PACKAGE… APCOMPARE(WARN | ERROR)
– DB2 11
– BIND/REBIND PACKAGE … APREUSE(WARN)
23
24. Static Plan Management
• DB2 10 delivered APREUSE(ERROR)
– Allowed potential for reuse of prior plan to generate new runtime structure
– Failure of reuse failed the entire package
– APREUSE(ERROR) EXPLAIN(ONLY) failure may not represent a valid plan in DB2 10
•
Failed access path is written to PLAN_TABLE
• DB2 11 delivers APREUSE(WARN)
– Upon failure of reuse, Optimizer will generate a new access path choice
•
Thus failure of 1 SQL will not fail the entire package
– PLAN_TABLE output will represent a valid plan
•
24
For both ERROR or WARN
25. Access Path Stability with statement level hints in V10
• Current limitations in hint matching
– QUERYNO is used to link queries to their hints – a bit fragile
• For dynamic SQL, require a change to apps – can be impractical
• New mechanisms:
– Associate query text with its corresponding hint … more robust
– Hints can be enforced for the entire DB2 subsystem
• irrespective of static vs. dynamic, etc.
– Hints integrated into the access path repository
• PLAN_TABLE isn’t going away
– Only the “hint lookup” mechanism is being improved.
25
26. Statement level hints (cont.)
• Steps to use new hints mechanism
– Enable OPTHINTS zparm
– Populate a user table DSN_USERQUERY_TABLE with query text
• Insert from SYSPACKSTMT (static) or DSN_STATEMENT_CACHE_TABLE
(dynamic)
– Populate PLAN_TABLE with the corresponding hints
• QUERYNO must match between PLAN_TABLE & DSN_USERQUERY_TABLE
– Run new command BIND QUERY
• To integrate the hint into the repository.
• Next package bind or dynamic prepare can pickup hint.
– FREE QUERY can be used to remove the query.
26
27. Statement-level BIND options
• Statement-level granularity may be required rather than:
– Subsystem level ZPARMs (STARJOIN, SJTABLES, MAX_PAR_DEGREE)
– Package level BIND options (REOPT, DEF_CURR_DEGREE)
• For example
– Only one statement in the package needs REOPT(ALWAYS)
• New mechanism for statement-level bind options:
– Similar to mechanism used for hints
• Enable OPTHINTS zparm
• Populate DSN_USERQUERY_TABLE with query text and desired option
– Use a QUERYNO that is NOT in PLAN_TABLE
• Issue BIND QUERY
• Next package bind/rebind or dynamic prepare can pickup statement option
• FREE QUERY can be used to remove the query
27
30. Rewrite Common Stage 2 predicates to indexable
• Targeted Stage 2 predicates
– YEAR(DATE_COL)
– DATE(TIMESTAMP_COL)
– value BETWEEN C1 AND C2
– SUBSTR(C1,1,10)
SUBSTR from position 1 only
• Stage 2 predicates are ONLY rewritten if there is no candidate Index On
Expression to support the predicate
– Regardless of whether the optimizer chooses that IOE
• Applies to literals or host variables/parameter markers
– Requires REBIND for static
30
31. Indexability for OR/IN and OR COL IS NULL combinations
• Improved single matching index access for OR C1 IS NULL
– Examples
WHERE C1 = ? OR C1 IS NULL
WHERE C1 IN (1, 2) OR C1 IS NULL
WHERE C1 > ? OR C1 IS NULL
• IN/OR combination to allow multi-index access……
WHERE C1 = ? OR C2 IN (1,2)
Becomes
WHERE C1 = ? OR C2 = 1 OR C2 = 2
31
32. Prune always true predicates
• Example WHERE 1=1
– So what’s the problem with this harmless predicate?
• DB2 will execute the WHERE 1=1 predicate for every qualified row
• SELECT *
FROM TABLE
WHERE 1=1
AND CUSTNO = ?
– Prune always true predicate to become
• SELECT *
FROM TABLE
WHERE CUSTNO = ?
32
33. Prune always false predicates
• DB2 10 already prunes “always false” equal/IN under OR
– WHERE C1 = ? OR ‘A’ = ‘B’
• DB2 11 extends to “always false” underneath parent “AND”
• SELECT *
FROM TABLE1 T1, TABLE2 T2
WHERE (1=1 AND T1.C1 = T2.C1)
OR (1=2 AND T1.C2 = T2.C2)
– Prune always true/false predicates to become
• SELECT *
FROM TABLE1 T1, TABLE2 T2
WHERE T1.C1 = T2.C1
• NOTE: “OR 0=1” is NOT pruned
33
34. Indexability for CASE predicates
• Case can now be indexable (formerly stage 2)
– For local predicate
– SELECT * FROM T1
WHERE COL =
CASE
WHEN
WHEN
ELSE
(CAST(? AS INT))
1 THEN 'CA'
2 THEN 'NY'
'AL' END;
– For JOIN predicate
• CASE expression must be evaluated before the join.
• In example below, join predicate is indexable if T1 accessed before T2.
– SELECT * FROM T1, T2
WHERE T2.COL = CASE WHEN T1.COL = ‘Y’
THEN T1.COL2
ELSE T1.COL3
END;
34
35. Predicate Pushdown
• DB2 11 supports pushdown into materialized views/TEs of
– Non-boolean term (OR) predicate
SELECT EMPNO, SALARY, DEPTCOUNT
FROM
EMP A ,
(SELECT WORKDEPT, COUNT(*)
FROM
EMP
GROUP BY WORKDEPT) AS B(WORKDEPT, DEPTCOUNT)
WHERE A.WORKDEPT = B.WORKDEPT
AND (B.WORKDEPT LIKE 'C%' OR B.WORKDEPT LIKE 'A%‘);
– Stage 2 predicates (expressions)
SELECT EMPNO, SALARY, DEPTCOUNT
FROM
EMP A ,
(SELECT WORKDEPT, COUNT(*)
FROM
EMP
GROUP BY WORKDEPT)
AS B(WORKDEPT, DEPTCOUNT)
WHERE A.WORKDEPT = B.WORKDEPT
AND UPPER( B.WORKDEPT) = 'C01'
35
36. Predicate Pushdowns (cont.)
• DB2 11 supports pushdown into materialized views/TEs of
– Predicate in the ON clause of an outer join
SELECT EMPNO, SALARY, DEPTCOUNT
FROM
EMP A
LEFT OUTER JOIN
(SELECT WORKDEPT, COUNT(*)
FROM
EMP
GROUP BY WORKDEPT)
AS B(WORKDEPT, DEPTCOUNT)
ON A.WORKDEPT = B.WORKDEPT
AND B.WORKDEPT = 'C01';
– Also – when the view/TE contains a scalar function in the SELECT list
– Some restrictions still remain, such as:
• If all 3 examples had predicates against table A – predicate not pushed in
• Expression pushdown may not qualify for index on expression
36
38. In Memory Data Cache (Workfile) vs. Sparse Index
• IMDC
• Sparse Index
• When insufficient memory for IMDC
T1
NLJ
T2 (WF)
T1
NLJ
T2 (WF)
t1.c = t2.c
t1.c = t2.c
IMDC sorted in t2.c
order
T2
Key RID
... ...
T2
(WF)
(WF)
Binary Search of WF to look up exact
location of qualified key (Hash used if
sufficient memory)
38
Sparse Index sorted
in t2.c order
Binary Search of sparse index to look up
“approximate “ location of qualified key
Workfile sorted
in t2.c order
39. IMDC/Sparse Index DB2 11 Enhancements
• Improved memory management by optimizer and runtime
• Controlled by zparm MXDTCACH (default 20MB)
• Each sparse index/IMDC is given a % of MXDTCACH
• From optimizer cost perspective
• At runtime (based upon cost estimation)
• Runtime will choose appropriate implementation based upon available storage
• Hash, binary search, or spill over to workfile
• Non‐correlated subqueries will also share MXDTCACH
• Improved optimizer cost model
• Allowing this to be opened up in more join scenarios
• Improvements to IFCID 27 for detail, 2 & 3 for summary
39
40. IMDC/Sparse index – Tuning basics
• DB2 11 provides simple accounting/statistics data for sparse index
– Sparse IX disabled
• indicates main memory was insufficient for the MXDTCACH memory request
• Suggest reducing MXDTCACH or allocating more memory to the system
– Sparse IX built WF
• MXDTCACH was insufficient to contain sparse index
– Increase MXDTCACH
• Look at sort BP sync I/O
– If high, also reduce VPSEQT in sort BP (do not use VPSEQT=100)
MISCELLANEOUS
-------------------SPARSE IX DISABLED
SPARSE IX BUILT WF
40
AVERAGE
-------0.00
0.36
TOTAL
-------0
8
42. Index skipping and Early‐out – DB2 11 Enhancements
• Improvements to queries involving GROUP BY, DISTINCT or non‐correlated subq
• Where an index can be used for sort avoidance
• By skipping over duplicates (see next few slides)
• Improvement to join queries using GROUP BY, DISTINCT
• By NOT accessing duplicates from inner table of a join if DISTINCT/GROUP BY will remove those
duplicates
• Improvement to correlated subqueries
• Early‐out for ordered access to MAX/MIN correlated subqueries
• When I1‐fetch is not available
• Optimize usage of the “result cache” for access to subquery with duplicate keys from
the outer query
• 100 element result cache dates back to DB2 V2 as a runtime optimization
• DB2 11 adds optimizer recognition of benefit
42
44. DB2 11 ‐ Duplicate Removal with Index Skipping
SELECT C1
FROM T
GROUP BY C1
Non-leaf
Leaf
100.RID.RID.RID.RID
100.101.101.102
100.RID.RID.101.RID
101.RID.RID.RID.RID
101.RID.102.RID.RID
Index Skipping (over-simplified)
Use index lookaside (current leaf high key and non-leaf) to get
the next key greater than current key
44
45. Early-out join
• DB2 11 supports early-out for join tables where duplicates are not
necessary
– Previously only available for correlated EXISTS subquery transformed to join.
– For below example: Duplicates from T2 are removed by DISTINCT
• In DB2 11, each inner table probe will stop after 1st match is found
– NOTE: For LEFT OUTER JOIN V10 will prune T2
SELECT DISTINCT T1.*
FROM T1, T2
WHERE T1.C1 = T2.C1
• Also applies to Non-Boolean Term join conditions with “early-out” table
SELECT DISTINCT T1.*
FROM T1, T2
WHERE T1.C1 = 1
OR T1.C1 = T2.C1
45
46. Optimize usage of subquery result cache
• DB2 V2 introduced a result cache for saving the 100 most recent
correlated subquery execution results
– Each subquery execution would 1st scan the cache to find the result
• If found, cache value is used
• If not found, subquery is executed, and result saved in cache
• DB2 11 adds optimizer recognition of the cache
– And the benefit to accessing the subquery in order
• Ordered access will reduce the cache size from 100
– For example below, the optimizer recognizes that accessing the
outer in CUSTNO order (via CUSTNO index or tablespace scan if
CUSTNO is clustering) would result in cache hits for repeat
CUSTNO values
SELECT *
FROM POLICY P1
WHERE P1.POLICY_DATE =
(SELECT MAX(P2.POLICY_DATE)
FROM POLICY P2
WHERE P2.CUSTNO = P1.CUSTNO)
46
48. Page Range Screening – DB2 11 Enhancements
• Page range performance Improvements
– Page Range Screening on Join Predicates
• Access only qualified partitions
– Pre-DB2 11, page range screening only applied to local predicates
• With literals, host variables or parameter markers
– Applies to index access or tablespace scan
• Benefits NPIs by reducing data access only to qualified parts
• Biggest benefit to DPSIs by reducing access only to qualified DPSI parts
• Only for equal predicates, same datatype/length only
48
49. Pre-V11 DPSI Join Probing (Join on partitioning Col)
• Current challenge
– 1st composite row probes all parts
– 2nd composite row probes all parts
– Etc
YEAR
C1
2009
1
2010
SELECT *
2011
FROM T1, T2
WHERE T1.C1 = T2.C1
AND T1.YEAR = T2.YEAR
T2
DPSI on C1
Partition by YEAR
2
3
2009 2010 2011 2012 2013
All parts are accessed for each composite row
49
50. V11 DPSI Join Probing (Join on Partitioning Col)
• Join recognizes page range screening
– 1st composite row probes 1 part.
– 2nd composite row probes 1 part.
– And so on.
YEAR
C1
2009
1
2010
T2
DPSI on C1
Partition by YEAR
2
SELECT *
2011
FROM T1, T2
WHERE T1.C1 = T2.C1
AND T1.YEAR = T2.YEAR
3
2009 2010 2011 2012 2013
Only qualified parts are probed on the inner table.
50
51. DPSI – DB2 11 Enhancements
• DPSI can benefit from page range screening from join
– Assuming you partition by columns used in joins (see previous slides)
• For DPSIs on join columns and partition by other columns
– DB2 11 Improves DPSI Join Performance (using parallelism)
• Controlled by ZPARM PARAMDEG_DPSI
• Sort avoidance for DPSIs (also known as DPSI merge)
– Use of Index On Expression (IOE)
• Ability to avoid sorting with DPSI IOE (already available for DPSI non-IOE)
– Index lookaside when DPSI used for sort avoidance
• Straw-model parallelism support for DPSI
– Straw-model (delivered in V10) implies that DB2 creates more work elements than there are
degrees on parallelism.
51
52. Pre-V11 DPSI Probing Challenge for Joins
• NOTE: No page range join predicate
• Current challenge for join to a DPSI
– 1st composite row probes all parts
– 2nd composite row probes all parts
– Etc
SELECT *
FROM T1, T2
WHERE T1.C1 = T2.C1
C1
1
T2
DPSI on C1
Partition by YEAR
2
3
All parts are accessed for each composite row
2009 2010 2011 2012 2013
52
53. DPSI Probing – DB2 11 Join Solution
• DPSI part-level Nested Loop Join
– Repeat composite table for each child task
• Each child task is a 2 table join
• Allows each join to T2 to access index sequentially (and data if high CR)
SELECT *
FROM T1, T2
WHERE T1.C1 = T2.C1
T2
DPSI on C1
C1
C1
C1
C1
C1
1
1
1
1
1
2
2
2
2
2
2009
3
53
53
2010
3
2011
3
2012
3
2013
3
55. Sort / Workfile Performance Improvement
• In memory workfile support in DB2 9 and 10
– DB2 9 RDS simple sort up to 32K
– DB2 10 RDS simple sort up to 1MB (no user control)
– DB2 10 intermediate workfile usage up to 32K for selective path
• More in memory operation in DB2 11
– RDS simple sort up to 128MB
• Controlled by a new zparm MAXSORT_IN_MEMORY (default 1MB)
– Wider range of usage for in memory
• Materialized view, table expression, outer Join, EXISTS, etc.
• In memory up to 32K then continue with physical workfiles
• Avoid workfile usages for final merge on top level sort
– Reduces physical workfile usage for large top level sort
55
56. RID processing enhancments
• Pre‐DB2 11
• DB2 10 added RID failover to WF
• Did not apply to queries involving column function
• A single Hybrid Join query could consume 100% of the RID pool
• Causing other concurrent queries to hit RID limit if > 1 RID block needed
• DB2 11
• RID failover to WF extended to all scenarios when RID limit is hit
• Hybrid join limited to 80% of the RID pool
56
57. Other interesting performance items
• DGTT NOT LOGGED support
• EXCLUDE NULL indexes
• Pseudo-deleted index entry cleanup
• Reduction of indirect references
• Decompression performance improvements
• DECFLOAT performance improvements (used extensively in XML)
57
59. DB2 Optimizer and Statistics - Challenge
• DB2 cost-based optimizer relies on statistics about tables and
indexes
• Customers often gather only standard or default statistics
– E.g. RUNSTATS TABLE(ALL) INDEX(ALL) KEYCARD
• Queries would often perform better if DB2 optimizer could exploit
more complete statistics
• Customers have difficulty knowing which statistics are needed
59
61. DB2 11 Solution: Optimizer Externalization
• During access path calculation, optimizer will identify missing or
conflicting statistics
– On every BIND, REBIND or PREPARE
• Asynchronously writes recommendations to SYSIBM.SYSSTATFEEDBACK
– DB2 also provides statistics recommendations on EXPLAIN
• Populates DSN_STAT_FEEDBACK synchronously
• Contents of SYSSTATFEEDBACK or DSN_STAT_FEEDBACK
can be used to generate input to RUNSTATS
– Contents not directly consumable by RUNSTATS
– Requires DBA or tooling to convert to RUNSTATS input
61
62. Optimizer selectivity - The Filter Factor Problem
• Query optimization challenges
– Cost based query optimization
• Estimate cost of available choices to identify choice with cheapest cost
– The optimizer needs to know how many rows are filtered at every step
• How much does it cost to scan index ? Matching, screen filtering
• Which table should be outer?
• Sometimes, the optimizer is unable to accurately estimate selectivity
–
–
–
–
–
62
Lack of statistics
Join skew, join correlation
Complex predicates
Predicate combinations
Unknowns (host variables, parameter markers, special registers)
63. DB2 11 Selectivity Overrides (FF hints)
• Process of supplying more robust selectivity (Filter Factor) input
– Rather than a whole OPTHINT – just FF hints
• Selectivity profile allows User to
– Provide optimizer with a more accurate view of selectivities used in query
execution
• For one, some or all predicates in a query
• For one or more representative “executions” of a query
- Weighted by frequency of occurrence
• Similar to the SELECTIVITY clause on SQL statement, but ...
• Doesn’t require changing applications
• Handle variations in execution
• Also has OQWT tooling support
63
65. Virtual Index Enhancements – Table Changes
• DSN_VIRTUAL_INDEXES enhanced
– Columns added to complete index modelling capabilities
• UNIQUE_COUNT
To support INCLUDE index columns
• SPARSE
To support NULL Supressed indexes
• DATAREPEATFACTORF
To support enhanced statistics gathering
• KEY_TARGET_COUNT & IX_EXTENSION
To support Index on Expression and XML Index
• DSN_VIRTUAL_KEYTARGETS
– New EXPLAIN table used for Index Advisor support for IOE and XML
indexes
65