This presentation will provide an overview of the Recursive SQL with the CONNECT BY clause feature. We will provide examples of typical practical database problems and describe in detail how they can be solved with recursive SQL. The problems discussed include for bill of materials, obtaining the number of employees for each manager in a particular sub-organization, converting linked dimension hierarchies in a star schema to fixed dimension hierarchies, tracking packages, and generating test data. This presentation compares the new solutions with traditional solutions of these problems and discusses the advantages and disadvantages of the various methods. This presentation will also discuss the query transformation techniques with Informix 12.10 features which will focus on how query blocks are moved between different levels and optimized using examples and diagrams. Users will learn how to analyze complex examples based on various Informix 12.10 features. Examples included in this session are query block movement, table re-ordering, complex ANSI joins, sub-queries, derived tables, views, connect by, OLAP functions, setops cases.

1. Discover the power of Recursive SQL and
query transformation with IBM Informix 12.1
Ajaykumar Gupte
IBM

2. Agenda
• Overview
• ANSI JOIN View Folding
• ANSI to IFMX JOIN cases
• OLAP & SETOPS cases
• HASH JOIN for “IS NULL” Expression
• TEMP Table Optimization
• Recursive SQL (Hierarchical Query Rewrite )

3. Overview

Query transformation (rewrite)

Simple way to execute query

Optimizer -

select better plan

More choice for table join order

Avoid internal temp table

4. Views with ANSI joins
• create view v1(vc1, vc2) as
• select t1.a , t2.a from t1 left join t2 on t1.a = t2.a
where t1.b = 5;
• create view v2(vc1, vc2) as
• select v1.vc1 , t3.a from (v1 left join t3 on v1.vc1 =
t3.a);

5. Simple case
select * from v1 left join t3 on v1.vc1 = t3.a where v1.vc2 = 10
1) informix.t1: INDEX PATH
Filters: informix.t1.b = 5
(1) Index Name: informix.ind1
Index Keys: a (Serial, fragments: ALL)
Lower Index Filter: informix.t1.a = 10
2) informix.t2: INDEX PATH
(1) Index Name: informix.ind2
Index Keys: a (Key-Only) (Serial, fragments: ALL)
Lower Index Filter: informix.t1.a = informix.t2.a
NESTED LOOP JOIN
3) informix.t3: INDEX PATH
(1) Index Name: informix.ind3
Index Keys: a (Serial, fragments: ALL)
Lower Index Filter: informix.t1.a = informix.t3.a
NESTED LOOP JOIN
V1 → t1 left join t2

6. Table map
V1 T2 T3 T4T1
1 2 3 4 5
T2 T3 T4T1 T5 T6 T7
Table order before
View folding
Table order after
View folding
1 2 3 4 5 6 7
T5 LEFT JOIN T6 LEFT JOIN T7

7. Nested ANSI join views
select * from v2
1) informix.t1: SEQUENTIAL SCAN
Filters: informix.t1.b = 5
2) informix.t2: INDEX PATH
(1) Index Name: informix.ind2
Index Keys: a (Key-Only) (Serial, fragments: ALL)
Lower Index Filter: informix.t1.a = informix.t2.a
NESTED LOOP JOIN
3) informix.t3: INDEX PATH
(1) Index Name: informix.ind3
Index Keys: a (Key-Only) (Serial, fragments: ALL)
Lower Index Filter: informix.t1.a = informix.t3.a
NESTED LOOP JOIN
V2 → V1 left join t3
V1 → t1 left join t2

8. ANSI to IFMX JOIN cases

Transform ANSI OUTER JOIN --> ANSI INNER JOIN

Transform ANSI INNER JOIN -->INFORMIX INNER JOIN

Transform ANSI OUTER JOIN --> INFORMIX OUTER
JOIN

Applicable to tables inside view or derived table definition

9. ANSI to IFMX JOIN case
select * from ((t1 left join t2 on t1.a = t2.b) left join t3 on t1.a = t3.b)
Estimated Cost: 8
Estimated # of Rows Returned: 3
1) informix.t1: SEQUENTIAL SCAN
2) informix.t3: SEQUENTIAL SCAN
DYNAMIC HASH JOIN
Dynamic Hash Filters: informix.t1.a = informix.t3.b
3) informix.t2: SEQUENTIAL SCAN
DYNAMIC HASH JOIN
Dynamic Hash Filters: informix.t1.a = informix.t2.b
ANSI OUTER JOIN Transformation

10. Sub-query case
select * from v1 where vc1 in (select t5.a from t5 where t5.b > 10)
Estimated Cost: 4
Estimated # of Rows Returned: 1
1) informix.t1: INDEX PATH
Filters: informix.t1.b = 5
(1) Index Name: informix.ind1
Index Keys: a (Serial, fragments: ALL)
Lower Index Filter: informix.t1.a = ANY <subquery>
2) informix.t2: INDEX PATH
(1) Index Name: informix.ind2
Index Keys: a (Key-Only) (Serial, fragments: ALL)
Lower Index Filter: informix.t1.a = informix.t2.a
NESTED LOOP JOIN

11. OLAP case
select vc1, sum(vc2) over (partition by vc2 order by vc1) from v1
Estimated Cost: 3
Estimated # of Rows Returned: 1
1) informix.t1: SEQUENTIAL SCAN
Filters: informix.t1.b = 5
2) informix.t2: INDEX PATH
(1) Index Name: informix.ind2
Index Keys: a (Key-Only) (Serial, fragments: ALL)
Lower Index Filter: informix.t1.a = informix.t2.a
NESTED LOOP JOIN
V1 → t1 left join t2
type rows_sort est_rows rows_cons
time
-------------------------------------------------
sort 1 0 1 00:00.00
type it_count time
----------------------------
olap 1 00:00.00

12. Setops case
select vc1, vc2 from v1 intersect select t3.a, t3.b from t3
1) informix.t1: SEQUENTIAL SCAN
Filters: informix.t1.b = 5
2) informix.t2: INDEX PATH
(1) Index Name: informix.ind2
Index Keys: a (Key-Only) (Serial, fragments: ALL)
Lower Index Filter: informix.t1.a = informix.t2.a
NESTED LOOP JOIN
1) (Temp Table For Collection Subquery): SEQUENTIAL SCAN
2) informix.t3: INDEX PATH (First Row)
Filters: (Temp Table For Collection Subquery).vc2 == informix.t3.b
(1) Index Name: informix.ind3
Index Keys: a (Serial, fragments: ALL)
Lower Index Filter: (Temp Table For Collection Subquery).vc1 == informix.t3.a
NESTED LOOP JOIN (Semi Join)
V1 → t1 left join t2
Temp table for view

13. When Temp table is used ?
• ANSI OUTER JOIN view Restrictions
– Full join
– Multiple views
– View on subservient side
– Select * from t1 left join v1 on t1.a = v1.vc1;
– Union all views with OUTER JOIN
– Complex cases - Sub-query inside view

14. ANSI JOIN View Folding
• Performance tips
– ANSI OUTER view or derived table on dominant
side
– Avoid complex views (aggregate, correlated sub-
query ) with large data set
– Analyze views or derived tables with large data set

15. HASH JOIN for “IS NULL”
• Use HASH JOIN for “IS NULL” expression
• select count(*) from t1, t2 where a=x or (a is null
and x is null)
• Generalized expression :
• (expr1 = expr2) or ((expr1) IS NULL AND (expr2) IS
NULL)

16. HASH JOIN for “IS NULL”

17. HASH JOIN for “IS NULL”

18. HASH JOIN for “IS NULL”
Complex case with 3 table joins :
select count (*) from T0, T1, T2 where
(T2.C0=T0.C0 or (T2.C0 is null and T0.C0 is null)) and
(T2.C1=T0.C1 or (T2.C1 is null and T0.C1 is null)) and
(T2.C2=T1.C0 or (T2.C2 is null and T1.C0 is null)) and
(T2.C3=T1.C1 or (T2.C3 is null and T1.C1 is null)) and
(T2.C4=T1.C2 or (T2.C4 is null and T1.C2 is null))

19. Temp table optimization
• Large number of columns
• performance degradation due to increased memory
• The BI tools - reuse templates based on tables
• Query will use all columns at lower level, when top level
columns are less
• Large number of derived table : multiple materialization of
intermediate result sets into temporary tables

20. Select v1.vc1, v1.vc2, v1.vc3 from v1
Temp table for view v2
(v2c1, v2c2, v2c3)
Temp table for view v1
(vc1, vc2, vc3, vc4)
Temp table for view v3
(v3c1, v3c2, v3c3)
Temp table for view v4
(v4c1, v4c2, v4c3)
Temp table for view v5
(v5c1, v5c2, v5c3)
Temp table for view v6
(v6c1, v6c2, v6c3)
Base tables t1 , t2
(t1.c1, t2.c1, t2.c2)
Base tables t3 , t4
(t3.c1, t4.c1, t4.c2)
Base tables t5 , t6
(t5.c1, t6.c1, t6.c2)
Original column map

21. Select v1.vc1, v1.vc2, v1.vc3 from v1
Temp table for view v2
(v2c1, v2c2)
Temp table for view v1
(vc1, vc2, vc3)
Temp table for view v3
(v3c2)
Temp table for view v4
(v4c1)
Temp table for view v5
(v5c2)
Temp table for view v6
(v6c1)
Base tables t1 , t2
(t1.c1)
Base tables t3 , t4
( t4.c1)
Base tables t5 , t6
(t5.c1)
Optimized column map

22. Recursive SQL
(Hierarchical Query Rewrite )

23. Problem of querying hierarchical data
• Common technique of storing hierarchical data in
relational tables is self-reference
– Employee-Manager
• Employee table (key – empid)
• Every employee has a manager (indicated by mgrid)
• Manager is also an employee (with a valid empid)
– Shipment
• Inbound shipment table (key – item_id)
• Each item can belong to a package ( key –
package_id)
• Every package is itself an item (with a valid item_id)
CREATE TABLE employee (
empid INTEGER NOT NULL
PRIMARY KEY,
name VARCHAR(10),
salary DECIMAL(9, 2),
mgrid INTEGER);
CREATE TABLE employee (
empid INTEGER NOT NULL
PRIMARY KEY,
name VARCHAR(10),
salary DECIMAL(9, 2),
mgrid INTEGER);
CREATE TABLE inbound_shipment (
shipment_id VARCHAR(50),
item_id VARCHAR(20) ,
package_id VARCHAR(20),
.......
);
CREATE TABLE inbound_shipment (
shipment_id VARCHAR(50),
item_id VARCHAR(20) ,
package_id VARCHAR(20),
.......
);

24. SELECT level as package_level, item_id,
package_id
FROM inbound_shipment
START WITH item_id = 'pallet_BX505'
CONNECT BY PRIOR
item_id = package_id
Table with
hierarchical
data
Seed of
recursion
Condition to
recurse on
Using CONNECT BY to discover
data hierarchy

25. Results of CONNECT BY Query
packing_level item_id package_id
1 pallet_BX505 ship_CX2555
2 box_C1255 pallet_BX505
3 band_aid_H10 box_C1255
3 band_aid_H12 box_C1255
3 A1_pharma_F23 box_C1255
3 A1_pharma_F33 box_C1255

26. sqexplain• QUERY:
• SELECT level as package_level, item_id, package_id FROM inbound_shipment
START WITH item_id = 'pallet_BX505' CONNECT BY PRIOR item_id =
package_id
• Connect by Query Rewrite:
• select x0.level ,x0.item_id ,x0.package_id from
• (select x1.item_id ,x1.package_id ,x1.item_id ,1 ,1 ,0 from
"informix".inbound_shipment x1 where (x1.item_id = 'pallet_BX505' )
• union all
• select x2.item_id ,x2.package_id ,x2.item_id ,(level + 1 ) ::integer
,connect_by_isleaf ,dtab_30093_173_stkcol from
"informix".inbound_shipment x2 ,"informix".dtab_30093_173 x0 where
(dtab_30093_173_p_item_id = x2.package_id ) )
• X0
(item_id,package_id,dtab_30093_173_p_item_id,level,connect_by_isleaf,dtab
_30093_173_stkcol)
START WITH

27. Questions?
Discover the power of Recursive SQL and query transformation with IBM Informix 12.1
Ajaykumar Gupte IBM gupte@us.ibm.com