Introduction Oracle Database 11g Release 2 for developers

Introducing Oracle Database 11g Release 2 for DevelopersOn SQL, PL/SQL and Application Development
Lucas Jellema
AMIS
29th September 2009

Quick Overview
Analytical stuff
LISTAGG, NTH_VALUE, Ignore nulls
Recursive Subquery Factoring
The new hierarchical query
Parallel processing of statements
Small Fry
Big Fry: Edition Based Redefinition

Analytical Functions FIRST_VALUE and LAST_VALUE return the first and last value in a WINDOW
For example a query to return the colleague (same department) who has the highest salarya similar query with last_value(or with order by salasc) would return the colleague with the lowest salary
select deptno
, ename
, first_value(ename) over (partition by deptno order by saldesc)
most_earning_colleague
from emp
NTH_VALUE complements FIRST_VALUE and LAST_VALUE

Analytical Function NTH_VALUE returns the NTH_VALUE in a WINDOW
FIRST_VALUE(column) == NTH_VALUE(column,1)
LAST_VALUE(column) == NTH_VALUE(column,1) FROM LAST
For example a query to return the colleague (same department) who has the second highest salary
select deptno
, ename
, nth_value(ename,2) from first over (partition by deptno order by saldesc)
almost_highest_earning_colleague
from emp
NTH_VALUE complements FIRST_VALUE and LAST_VALUE

NTH_VALUE
Measure_Expr and N are both expressions
composed of columns, SQL functions, PL/SQL function call, constants, scalar subqueries
FROM LAST can be used to start counting at the end of the window
IGNORE NULLS can be used to ignore rows that have NULL as measure_expr

More on NTH_VALUE
FROM FIRST (default) and FROM LAST can be used to start counting at either end of the window
NTH_VALUE( sal ,2) over (order by saldesc) ==NTH_VALUE( sal ,2) FROM FIRST over (order by saldesc)==NTH_VALUE( sal ,2) FROM LAST over (order by salasc) ROWS UNBOUNDED PRECEDING and UNBOUNDED FOLLOWING

More on NTH_VALUE
IGNORE NULLS can be added to not count rows for which the expression evaluates to NULL
The default is RESPECT NULLS
NTH_VALUE(COMM, 2) IGNORE NULLSOVER (ORDER BY SAL DESC)means:
Order the employees by salary, the top earner first
Find the second row where COMM is not null
Return the value of COMM in that row

Example: difference between top 2
For example a query to return all employees and per employee the difference between the highest earning colleague and the second highest earning colleague
select deptno
, ename
, first_value(sal) over (partition by deptno order by saldesc) - nth_value(sal,2) from first over (partition by deptno order by saldesc)
top2_difference
from emp

LAG & LEAD – IGNORE NULLS
In the same way as for NTH_VALUE, IGNORE NULLS is used to skip rows for which the measure is NULL
This allows us to retrieve values from other rows in a more precise manner
For example:
Get the difference in COMM value between the current employee and the first less-experienced (later Hiredate) employee who also has a COMM value
IGNORE NULLS (together with a CASE expression) gives us a row filter for the LAG, LEAD and NTH_VALUE functions

LAG without IGNORE NULLS
Show the Salary of the next Hire
select ename
, sal
, job
, lead (sal)
over (order by hiredate)
sal_of_next_hire
from emp
order
by hiredate

IGNORE NULLS & CASE– row filter
Give me for each employee the salary of the next hire – but ignore CLERKs as they are in a league of their own
select ename
, sal
, job
, lead ( case
when job ='CLERK'
then null
else sal
end
) ignore nulls
over (order by hiredate)
sal_next_non_clerk_hire
from emp
order
by hiredate

COLLECT aggregates records into collection
create or replace type
ename_type as table of varchar2(30)
select deptno
, avg(sal) avg_sal
, cast
( collect(ename)
as ename_type
) enames
from emp
group
by deptno

LISTAGG – aggregating into string
We already had COLLECT for aggregating multiple values into a Collection
Requires a TYPE definition (usually TABLE OF VARCHAR2)
Only Aggregate Function (group by), not Analytical
Can do an order by especially for the Collected Values
LISTAGG creates a delimiter separated string of values
Produces a single VARCHAR2 (does not require a table type)
Available as Aggregate (with GROUP BY) and Analytical Function (with PARTITION clause)
Also includes an order by

select deptno
, avg(sal) avg_sal
, listagg( ename, ',') within group (order by sal)
enames
from emp
group
by deptno
LISTAGG aggregates values into string

select deptno
, ename
, sal
, listagg( ename, ',') within group (order by ename) over (partition by deptno)
colleagues
from emp
order
by deptno
, ename
LISTAGG aggregates values into string

Oracle goes industry standard…

Hierarchical Query
Oracle specific (i.e. proprietary) approach
Since Oracle 2 or 3 (very early on)
Later extensions to hierarchical queries:
order siblings by
sys_connect_path
connect_by_root, connect_by_is_leaf,
nocycle, connect_by_is_cycle
with employees (empno, name, level) as
( select empno, ename, job, mgr, level
from emp
start with mgr is null
connect by prior empno = mgr
)
select *
from employees

Hierarchical Query in ANSI SQL
ANSI SQL defines an alternative approach to Hierarchical Queries
It is called Recursive Subquery Factoring (or Recursive With Clause)
Apparently supported (to some extend) in DB2, SQL Server and PostgreSQL
Extends WITH clause to enable formulation of recursive queries.
“The query in the WITH clause (subquery) can refer to itself”
Recursive WITH clause is ANSI. This makes Oracle ANSI-compatible for recursive queries.

with employees (empno, name, job, mgr, hierlevel) as
( select empno, ename, job, mgr, 1
from emp
where mgr is null
union all
select e.empno, e.ename, e.job
, e.mgr, m.hierlevel + 1
from emp e
join
employees m
on (m.empno = e.mgr)
)
select *
from employees
Root nodes
Recursive Child nodes

What about the Connect by goodies?
SYS_CONNECT_PATH
Used to build the path from a node all the way to its root
with employees (empno, name, mgr, hierlevel, path) as
( select empno, ename, mgr, 1, ename
from emp
where mgr is null
union all
select e.empno, e.ename
, m.path||'/'||e.ename
from emp e
join
employees m
)
select *
from employees

CONNECT_BY_ROOT
Used to retrieve a value from a node’s ultimate root
with employees (empno, name, mgr, hierlevel, root) as
( select empno, ename, mgr, 1, ename
from emp
where mgr is null
union all
, m.root
from emp e
join
employees m
)
select *
from employees

ORDER SIBLINGS BY
Set the sort order of child nodes under a parent node
with employees (empno, name, mgr, hierlevel)
as
( select empno, ename, mgr, 1
from emp
where mgr is null
union all
from emp e join employees m
)
search depth first by name set seq
select *
from employees

Search Clause for organizing rows
Use the SEARCH clause to specify ordering for rows
Specify BREADTH FIRST BY if you want sibling rows returned before any child rows are returned.
Specify DEPTH FIRST BY if you want child rows returned before any siblings rows are returned.
Sibling rows are ordered by the columns listed after the BY keyword.
The ordering_column is automatically added to the column list for the query name. Queries that select from the subquery can use an ORDER BY that refers to it

ORDER SIBLINGS BY
with employees (empno, name, mgr, hierlevel)
as
from emp
where mgr is null
union all
from emp e join employees m
)
search breadth first by name set seq
select *
from employees
order by seq

with employees (empno, name, mgr, hierlevel) as
from emp
where mgr is null
union all
select e.empno, e.ename, e.mgr, m.hierlevel + 1
from emp e
join
employees m
on (m.empno = e.mgr) order
by e.ename
)
select *
from employees

New PL/SQL Packages
DBMS_DBFS_HS
DBMS_COMPRESSION
DBMS_DBFS_CONTENT
DBMS_DBFS_CONTENT_SPI
DBMS_DBFS_HS
DBMS_DBFS_SFS
DBMS_DST
DBMS_PARALLEL_EXECUTE
DBMS_STREAMS_HANDLER_ADM
SEM_RDFCTX
SEM_RDFSA

Some changes in PL/SQL packages
dbms_session.GET_PACKAGE_MEMORY_UTILIZATION - memory usage instantiated packages
dbms_space.ISDATAFILEDROPPABLE
dbms_utility.GET_SQL_HASH
dbms_utility.WAIT_ON_PENDING_DML
Waits until all transactions (other than the caller's own) that have locks on the listed tables and began prior to the specified SCN have either committed or been rolled back

DBMS_PARALLEL_EXECUTE
Run a statement in parallel on chunks of data
Steps
Create a task label
Generate the chunks – rowid or value ranges
Run the task: a SQL statement that has :start_id and :end_id bind parameters
The task is executed with a certain degree of parallellism using job slaves

UTL_MATCH
To find the similarity between strings
Used for example in deduplication of records or finding fuzzy matches

Smaller Fry & Miscellaneous
Create or Replace Type with FORCE option
To replace even in the presence of dependents
Automatic Function Result Cache reset (automatic data source detection)
Function Result Cache accessible across RAC
APPEND_VALUES Hint – to usewith INSERT INTO VALUES …. (especially PL/SQL FORALL)
Fine Grained Invalidation extended to triggers

Semantic Indexing of unstructured documents and the SEM_CONTAINS operator in SQL
And many more new Semantic Technologies
Handle Foreign Key dependencies for DBMS_FLASHBACK TRANSACTION_BACKOUT with CASCADE to roll back a transaction
Internet Protocol version 6 (IPv6) Support
IPv6 address has 128 bits, IPv4 address only 32 bits. IPv6 address must be in brackets in a URL

IGNORE_ROW_ON_DUPKEY_INDEX and CHANGE_DUPKEY_ERROR_INDEX mandates
To instruct Oracle to ignore duplicate keys (just skip the row) or report an ORA-38911 to find out just which UNIQUE KEY caused the exception
Similar to 10gR2 DML Error Logging
More setting up and restrictions for Unique Key on Update & Merge and Direct Path Insert

Release 2
Release 3
Base Release
Introducing Oracle 11gR2 Editioningor:On Parallel Application Universes

Availability
Availability = Performance * (Up or Down)
Up = !Down
Down = Unplanned Down + Planned Down
Planned Down???
System Maintenance
Power-supply, Hardware & Network, O/S upgrade
Database patching & Upgrade
Application Upgrades

Maximum Availability Architecture

Application Upgrade
Creation of new objects
Changing existing objects (alter and create or replace)
Add, Modify or Drop columns or constraints
Change packages and stored procedures
Recompile
Drop redundant objects
Convert or migrate data
Resume normal operations
Application is DOWN

Compare with road maintenance

Restructuring A12
Build new road next to current one (A12-B)
Same “functionality” as the old road
At the cut-over moment
Open new A12-B:
Newly arriving traffic travels on temporary road
Close A12 (original)
Cars already on old road keep going

11gR2 Editioning is similar
Application Upgrade:
Prepare new release
Construct the release in parallel to the existing
Operations in existing application ‘edition’ continue normally
From the cut-over point:
Have new sessions operate on new release
Existing sessions can continue to run on existing release

11gR2 Editions introduces a new dimension in the database
Release 3
Release 2
Base Release

Editions introduce or inherit versions of objects
Release 2
Release 3
Base Release

Editions are parallel universes
Database Objects are identified by
Object Type
Schema
Object Name
…. Edition!
(release, application version, stripe, parallel universe id)
Database Sessions run in the context of a specific edition
Using a specific collection of versions of objects

Database sessions run in a specific edition –one version of each object
Release 2
Release 3
Base Release
The database as seen by a sessionrunning in the context of Release 3
3
2
3
B
2
2
3
3
2

Demo of Application Upgrade
Existing base application, in base edition
Create new editon – release_2
Create and Alter database objects
Base application continues running
Cut-over point
New sessions run in release_2 edition
Current sessions continue in base

Some Rules for EBR (Edition Based Redefinition)
Editioning acts on packages, functions, triggers, procedures, views, types and synonyms
Editioning does not apply to tables
Data is not versionend, cloned, migrated
Different incarnations of a table are suggested through editionable views – there is only one table
Applications should never access tables directly!
Cross Edition Triggers on the table synchronize DML from different editions with the current state of the table
CET are temporary objects – not part of the application

But wait, there is more
After the release of a new edition – there is no reason why you cannot keep the previous one going for some time
And multiple previous ones!
That means – END OF THE BIG BANG upgrade!
Multiple versions of the application can continue running to suport various user groups (e.g. SaaS)
Without data migration and additional downtime upon later move over of user groups

Parallel Application Versions
Application X
VERSION 1
Application X
VERSION 2
Release 2
Release 3
Base Release

Version 1 on Base Edition

Version 1 on Edition Release 2

Upgrade Base to Release 2
Authors
New columns COUNTRY and BIOGRAPHY
Books
Drop column NUM_OF_PAGES
Modified column ISBN (10 to 20 characters)
New columns LANGUAGE and PUBLICATION_YEAR

Version 2 (on Edition Release 2)

Version 2 on Base Edition

Interesting Edition Tid-Bits
Drop Object in an Edition stops the inheritance from previous Edition. Object no longer is carried forward
Edition can have only one child – no branches (yet)
DBMS_SQL.PARSE can be executed in a specific Edition
Use parameter edition to specify other than current edition
If no explicit edition is set for a session, the default edition is used
ALTER DATABASE DEFAULT EDITION = edition_name;
Data Dictionary Views
DBA_/ALL_EDITIONS – editions in the database
DBA_/ALL_OBJECTS – objects (inherited) in current edition
DBA_/ALL_OBJECTS_AE – actual objects across all editions

Editions and Tables
Tables cannot be versioned: there is only one definition of a table across all Editions
Data is not versioned: a record exists once and only once
Note: Workspace Management supports a form of versioning tables and data
The solution for managing changes to Tables: Editioning Views and Cross Edition Triggers
Editioning Views are defined on base table (no joins)
Editioning Views can have DML triggers defined (just like base table)
Using an editioning view in a query or DML statement does not impact the performance of the query or DML

Editions and Tables
Application A
Application B
Table EMP
SAL
MGR
JOB
ENAME
HIREDATE
COMM

Editions and Tables
Application A
Application B
Editioning View EMP
Table EMP_BASE
SAL
MGR
JOB
ENAME
HIREDATE
COMM

Migrate to Editioning Views
Rename Table (for example to <old table name>_BASE)
(suggested but optional): changes the names of the columns in the table – to reflect change history
For example JOB_1_0, JOB_1_1, JOB_1_2 …
Create the Editioning View with the old table name
Using the ‘CREATE OR REPLACE EDITIONING VIEW <view name>’ statement
Recompile the triggers on the table
These now become triggers on the Editioning View
Recompile all invalid PL/SQL program units and Views
They now refer to the Editioning View instead of the table
VPD policies are reassigned to the View
Regular auditing and FGA is on the table

Demo
ALTER TABLE EMP RENAME TO EMP_BASE;
CREATE OR REPLACE EDITIONING VIEW EMP
AS
SELECT ...
FROM EMP_BASE
/
DROP TRIGGER EMP_BRI
/
Rem recreate trigger on Editioning View
@<EMP_BRI>.trg
Rem recompile invalid Views and PL/SQL units

Multiple versions of Editioning View
Application A
Application B
Edition R2
Edition R1
Editioning View EMP
Editioning View EMP
Table EMP_BASE
SAL
MGR
JOB
ENAME
HIREDATE
COMM

Application A
Application B
Edition R2
Edition R1
View EMP (1.1)
…* Language
View EMP (1.0)
Table EMP_BASE
SAL
MGR
JOB
ENAME
LANGUAGE
HIREDATE
COMM
Forward CrosseditionTrigger

Demo
CREATE EDITION R2 AS CHILD OF R1;
ALTER SESSION SET EDITION R2;
SELECT SYS_CONTEXT('Userenv', 'Current_Edition_Name') "Current_Edition" FROM DUAL;
ALTER TABLE EMP_BASE
ADD (LANGUAGE VARCHAR2(2) NULL);
Rem function for deriving value for language
CREATE OR REPLACE FUNCTION
GET_LANGUAGE( p_job in varchar2) return varchar2
is
begin
return case p_job when 'MANAGER' then 'fr'
else 'en' end;
end;

Demo
Rem Create Forward Crossedition Trigger
Rem Applies to DML operations on EMP_BASE from
Rem Earlier Editions
CREATE OR REPLACE TRIGGER EMP_1_1_Fwd_Xed
BEFORE INSERT OR UPDATE ON EMP_BASE
FOR EACH ROW
FORWARD CROSSEDITION
DISABLE
BEGIN
:new.language = get_language(:new.job);
END EMP_1_1_Fwd_Xed;
Rem Enable the Forward Crossedition Trigger
ALTER TRIGGEREMP_1_1_Fwd_Xed ENABLE;

Demo
Rem Use Forward Crossedition trigger to
Rem upgrade existing table records according to new table
Rem version (for large # records use dbms_parallel_execute)
DECLARE
c NUMBER := DBMS_SQL.OPEN_CURSOR();
x NUMBER;
BEGIN
DBMS_SQL.PARSE
( c => c
, Language_Flag => DBMS_SQL.NATIVE
, Statement => 'UPDATE EMP_BASE SET EMPNO = EMPNO'
, Apply_Crossedition_Trigger => 'EMP_1_1_Fwd_Xed'
);
x := DBMS_SQL.EXECUTE(c);
DBMS_SQL.CLOSE_CURSOR(c);
COMMIT;
END;

Upgrade Table Definition
(Create Edition,) Switch to Edition
Make desired changes to the table
Add columns, modify Columns, …
Modify the Editioning View in the Edition
To reflect the table as its latest version should look
Perhaps hiding columns you eventually want to drop
(optional) Modify Editioning Views in previous Edition(s)
If the table change invalidates those Views
(optional) Create Forward Crossedition Trigger on base table to have DML on previous Editions made valid
(optional) Create Reverse Crossedition Trigger on base table to have DML on current Edition made valid

Cross Edition Triggers
If you remove a (mandatory) column from the current Editioning View…
a Reverse Crossedition Trigger ensures that new records get some value in that (invisible) column
If you add a (mandatory) column to the table (and the current Editioning View)…
a Forward Crossedition Trigger ensures that records DMLed through previous Editioning View versions are made valid
(optionally) Apply Forward Crossedition Trigger for all existing records (created in old edition of the table)
Use DBMS_SQL.parse (with parameter Apply_Crossedition_Trigger set to name of trigger)
Use DBMS_PARALLEL_EXECUTE

Application A
Application B
Edition R3
View EMP (1.1)
… (minus ENAME)* Language
o FIRST_NAME
* LAST_NAME
Edition R1
Edition R2
View EMP (1.1)
…* Language
View EMP (1.0)
Table EMP_BASE
Reverse Crossedition Trigger
SAL
MGR
JOB
ENAME
LANGUAGE
FIRSTNAME
LASTNAME
HIREDATE
COMM
Forward CrosseditionTrigger

“Suggested (best) Practices”
Every application (release) sets the Edition it requires when it connects to a session
At the same time it calls dbms_application_info
And sets other Context details
Applications should never access tables – all access should be through views
Only through views can the data structure itself be Editioned
Even triggers should be on the Editioning View

Summary
11gR2 Editions are parallel, co-existing universes with incarnations of database objects
The new release can be constructed, tested and run in a new edition
The old edition can be switched off at cut-over
Editions also allow long time co-existence of multiple releases of an application
Application Upgrade no longer needs to disrupt the operation through planned downtime

Conclusion
See http://www.amis.nl
To do an 11gR2 PoC together with us
To learn about 11gR2 training (DBA & Developer)
For more on database Migration services
See Blog for 11gR2 articles
http://technology.amis.nl/blog
Contact me: lucas.jellema@amis.nl