Overview of Oracle Database 12c R1 for Developers

on

  • 2,755 views

June 26th 2013 saw the GA of the Oracle Database Release 12.1. This presentation shows the most interesting new features and facilities for application development. Among these features are Top-N ...

June 26th 2013 saw the GA of the Oracle Database Release 12.1. This presentation shows the most interesting new features and facilities for application development. Among these features are Top-N queries and SQL pagination, Inline PL/SQL Functions, Temporal Validity, Flashback Query improvements, PL/SQL Callstack API, SQL <=> PL/SQL integration, security improvements, data redaction, SQL Pattern Matching and more.

Statistics

Views

Total Views
2,755
Views on SlideShare
2,742
Embed Views
13

Actions

Likes
2
Downloads
168
Comments
1

1 Embed 13

https://twitter.com 13

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

CC Attribution License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Lucas,

    thanks for this great slideshare !

    Could you please point out to me where in the docs I can read about
    - Temporal Primary and Unique Key constraints that allow multiple non-overlapping entries
    - Temporal referential constraints ...

    Thanks
    Matthias
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • 12c:All table definition, partitioning, and space management DDLs are supported on FDA-enabled tables. Retain session context with transaction Data Pump supports flashback history import ‘user defined history’ as flashback history restore pointsTemporal modeling/ valid time definition specifying the validity period for records unique keys that allow multiple entries with non-overlapping validity periods, foreign keys that include period, flashback based on valid time (not transaction time); joins, aggregation, … valid time aware

Overview of Oracle Database 12c R1 for Developers Overview of Oracle Database 12c R1 for Developers Presentation Transcript

  • Lucas Jellema 15th July 2013 Oracle Database 12c for Developers
  • Overview • The Fourth Dimension: Time • Invisibility • Security • Data masking (data redaction) • The evolution of SQL • Pattern Matching • Miscellaneous
  • The Fourth Dimension
  • The Fourth Dimension • Oracle Database 12c is time aware in two ways – Transaction time – through flashback – Business time – through temporal validity (brand new in 12c)
  • Flashback • Introduced in 9i • Based on UNDO • Initially only for recovery • As of 11g – Total Recall option with Flashback Data Archive – Controlled history keeping • Look back into history – Query trends (version history) – Difference reporting – Audit trails (Replace journaling tables) • Require trick for transaction history: WHO? • Also: when is the start of history? • By the way: Flashback Data Archive requires EE & Advanced Compression database option
  • Total Recall • Flashback Data Archive Improvements The following improvements have been made to Flashback Data Archive (FDA): – Complete schema evolution support All table definition, partitioning, and space management DDLs are supported on FDA-enabled tables. – The metadata information for tracking transactions including the user context is now tracked. The addition of user-context tracking makes it easier to determine which user made which changes to a table. • This could mean that journaling tables are now officially deprecated • Also given that the current contents of journaling tables can even be migrated to Flashback Data Archive
  • Total Recall (2) • Import and export of history – Support for import and export using Data Pump for FDA-enabled tables. Data Pump can now be used to export and import an FDA- enabled base table along with its schema-evolution metadata and historical row versions. • User generated history – Support for importing user-generated history has been added. Customers who have been maintaining history using other mechanisms, such as triggers, can now import that history into Total Recall. • Database Hardening – Register ―Application‖ (a group of tables) and enable/disable flashback data archive for the application (also available: lock an application – make all tables read only)
  • Flashback Data Archive in all database editions
  • Valid time temporal modeling • Validity (or effectivity) of facts recorded in a database is frequently specified through dates or timestamps – For example begin date and [derived] end date of a price, membership, allocation, certificate, agreement • This valid time can differ from the transaction time at which a record is entered into the database • Multiple entries with different, non-overlapping valid-time periods can exist for a single entity • In 12c the notion of Valid Time is introduced into the Oracle Database – The valid-time dimension consists of two date-time columns specified in the table definition (create or alter) – These Valid Time columns specify the period during which a record is valid – A table can have multiple valid_time markers • Valid Time provides the optimizer with additional information and makes/will make many query operations easier to perform
  • Creating a table with valid time dimension • Table with explicit valid time columns: • Table with valid time dimension and implicit columns: columns valid_time_start and valid_time_end (TIMESTAMP) are added implicitly CREATE TABLE EMP ( employee_number NUMBER , salary NUMBER , department_id NUMBER , name VARCHAR2(30) , hiredate TIMESTAMP , firedate TIMESTAMP , PERIOD FOR user_time (hiredate, firedate) ); CREATE TABLE EMP ( employee_number NUMBER , salary NUMBER , department_id NUMBER , name VARCHAR2(30) , PERIOD FOR contract_time );
  • Valid time aware flashback queries • Select all employees who were employed at a certain moment in time • Perform all queries for records that are valid at a certain point in time • Return all records currently (session time) valid • Return all records (default) SELECT * FROM EMP AS OF PERIOD FOR user_time TO_TIMESTAMP('01-JUN-2012 12.00.01 PM') EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time ( 'ASOF' , TO_TIMESTAMP('29-JUL-12 12.00.01 PM') ); EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time('CURRENT'); EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time('ALL');
  • Scope of Valid time temporal modeling in 12c database • Valid time aware DML • Flashback Queries that do not drive from Transaction Time but from Valid Time (or a mix) • Temporal Primary and Unique Key constraints that allow multiple non- overlapping entries • Temporal referential constraints that take into account the valid-time during which the rows exist. – Child needs to have a valid Master at any time during its own validity • Temporal Aggregation - group or order by valid-time • Normalization - coalescing rows which are in adjacent or overlapping time periods • Temporal joins – joins between tables with valid-time semantics based on ‗simultaneous validity‘ • For Information Lifecycle Management (ILM), the Valid Time information is used to assess records to move
  • Invisibility…
  • Nebulous features
  • White List • A white list of allowed invokers can be defined for a PL/SQL unit – supports the robust implementation of a module, consisting of a main unit and helper units, by allowing the helper units to be inaccessible from anywhere except the unit they are intended to help.
  • accessible by clause package Helper authid Definer accessible by (Good_Guy, Bad_Guy) is procedure p; end Helper; package body Good_Guy is procedure p is begin Helper.p(); ... end p; end Good_Guy; package body Bad_Guy is procedure p is begin Helper.p(); ... end p; end Bad_Guy; PLS-00904: insufficient privilege to access object HELPER _______
  • Invisible columns • You can make individual table columns invisible and visible again. – CREATE TABLE EMP (…, SAL NUMBER(5) INVISIBLE,…) – ALTER TABLE EMP MODIFY (b VISIBLE); • Any generic access of a table does not show the invisible columns in the table. – SELECT * FROM statements in SQL – DESCRIBE commands in SQL*Plus – %ROWTYPE attribute declarations in PL/SQL – Describes in Oracle Call Interface (OCI) • Invisible columns are not the same as system-generated hidden columns. You can make invisible columns visible, but you cannot make hidden columns visible. • Note: select * from table order by 1,3 can suffer when columns are made (in)visible • When a column is made visible again, it will show up at the end of the column list (in select * and desc)
  • More visible… • PL/SQL DBMS_UTILITY.EXPAND_SQL_TEXT can be used to uncover the real SQL executed for a given query • It returns the query in full – All views replaced by their core table based queries – All VPD policies explicitly added to the query • This features provides true insight in what a query actually does – Functionality – From a performance analysis point of view
  • SQL Text Expansion ops$tkyte%ORA12CR1> variable x clob ops$tkyte%ORA12CR1> begin 2 dbms_utility.expand_sql_text 3 ( input_sql_text => 'select * from all_users', 4 output_sql_text => :x ); 5 end; 6 / PL/SQL procedure successfully completed.
  • SQL Text Expansion ops$tkyte%ORA12CR1> print x X -------------------------------------------------------------------------------- SELECT "A1"."USERNAME" "USERNAME","A1"."USER_ID" "USER_ID","A1"."CREATED" "CREAT ED","A1"."COMMON" "COMMON" FROM (SELECT "A4"."NAME" "USERNAME","A4"."USER#" "US ER_ID","A4"."CTIME" "CREATED",DECODE(BITAND("A4"."SPARE1",128),128,'YES','NO') " COMMON" FROM "SYS"."USER$" "A4","SYS"."TS$" "A3","SYS"."TS$" "A2" WHERE "A4"."DA TATS#"="A3"."TS#" AND "A4"."TEMPTS#"="A2"."TS#" AND "A4"."TYPE#"=1) "A1"
  • SQL Text Expansion ops$tkyte%ORA12CR1> create or replace 2 function my_security_function( p_schema in varchar2, 3 p_object in varchar2 ) 4 return varchar2 5 as 6 begin 7 return 'owner = USER'; 8 end; 9 / Function created.
  • SQL Text Expansion ops$tkyte%ORA12CR1> create table my_table 2 ( data varchar2(30), 3 OWNER varchar2(30) default USER 4 ) 5 / Table created.
  • SQL Text Expansion ops$tkyte%ORA12CR1> begin 2 dbms_rls.add_policy 3 ( object_schema => user, 4 object_name => 'MY_TABLE', 5 policy_name => 'MY_POLICY', 6 function_schema => user, 7 policy_function => 'My_Security_Function', 8 statement_types => 'select, insert, update, delete' , 9 update_check => TRUE ); 10 end; 11 / PL/SQL procedure successfully completed.
  • SQL Text Expansion ops$tkyte%ORA12CR1> begin 2 dbms_utility.expand_sql_text 3 ( input_sql_text => 'select * from my_table', 4 output_sql_text => :x ); 5 end; 6 / PL/SQL procedure successfully completed. ops$tkyte%ORA12CR1> print x X -------------------------------------------------------------------------------- SELECT "A1"."DATA" "DATA","A1"."OWNER" "OWNER" FROM (SELECT "A2"."DATA" "DATA", "A2"."OWNER" "OWNER" FROM "OPS$TKYTE"."MY_TABLE" "A2" WHERE "A2"."OWNER"=USER@!) "A1"
  • API for inspecting the PL/SQL callstack • New PL/SQL Package UTL_CALL_STACK provides API for inspecting the PL/SQL Callstack – Complements the DBMS_ UTILITY.FORMAT_CALL_STACK that returns a pretty print human readable overview of the callstack procedure tell_on_call_stack is l_prg_uqn UTL_CALL_STACK.UNIT_QUALIFIED_NAME; begin dbms_output.put_line('==== TELL ON CALLSTACK ==== ' ||UTL_CALL_STACK.DYNAMIC_DEPTH ); for i in 1..UTL_CALL_STACK.DYNAMIC_DEPTH loop l_prg_uqn := UTL_CALL_STACK.SUBPROGRAM(i); dbms_output.put_line( l_prg_uqn(1) ||' line '||UTL_CALL_STACK.UNIT_LINE(i) ); end loop; end tell_on_call_stack;
  • API for inspecting the PL/SQL callstack create or replace package body callstack_demo as function b( p1 in number, p2 in number) return number is l number:=1; begin tell_on_call_stack; return l; end b; procedure a ( p1 in number, p2 out number) is begin tell_on_call_stack; for i in 1..p1 loop p2:= b(i, p1); end loop; end a; function c( p_a in number) return number is l number; begin tell_on_call_stack; a(p_a, l); return l; end c; end callstack_demo;
  • UTL_CALL_STACK • Functions for retrieving – BACKTRACE: • DEPTH, LINE and UNIT – ERROR: • DEPTH, MSG and NUMBER – OWNER, SUBPROGRAM, UNIT_LINE – LEXICAL DEPTH (NESTING LEVEL)
  • Security
  • View with invoker‘s rights • As of Oracle Database Release 12c, a view can be either – BEQUEATH DEFINER (the default), which behaves like a Definer‘s Rights unit (functions in the view are executed using the view owner‘s rights) – or BEQUEATH CURRENT_USER, which behaves somewhat like an invoker‘s rights unit (functions in the view are executed using the current user‘s rights) create or replace view managers ( name, sal, deptno, experience) BEQUEATH CURRENT_USER as select ename, sal, deptno, some_function(hiredate) from emp where job = ‘MANAGER’
  • Invoker‘s rights function results can be cached • An Invoker's Rights Function Can Be Result-Cached – Through Oracle Database 11g Release 2 (11.2), only definer's rights PL/SQL functions could be result cached. • Now, invoker's rights PL/SQL functions can also be result cached. – The identity of the invoking user is implicitly added to the key of the result
  • Ability to Attach Roles to Program Units • You now can attach database roles to the program units functions, procedures, packages, and types. • The role then becomes enabled during execution of the program unit (but not during compilation of the program unit). • This feature enables you to temporarily escalate privileges in the PL/SQL code without granting the role directly to the user. The benefit of this feature is that it increases security for applications and helps to enforce the principle of least privilege. • Also: the new INHERIT PRIVILEGES privilege GRANT clerk_admin TO procedure scott.process_salaries;
  • Inherit or not in Invoker rights program units • When a user runs an invoker's rights procedure (or program unit), it runs with the privileges of the invoking user. • As the procedure runs, the procedure‘s owner temporarily has access to the invoking user's privileges. • [If the procedure owner has fewer privileges than an invoking user,] the procedure owner could use the invoking user‘s privileges to perform operations procedure owner Invoker‘s rights invoker procedure Special_ Table Tap_ Table
  • Inherit or not in Invoker rights program units • In previous releases, the invoking user had no control over who could have this access when he or she runs an invoker‘s rights procedure. • Starting with 12c, invoker‘s rights procedure calls only can run with the privileges of the invoker if the procedure‘s owner has the INHERIT PRIVILEGES privilege on the invoker or if the procedure‘s owner has the INHERIT ANY PRIVILEGES privilege. – This gives invoking users control over who has access to their privileges when they run invoker‘s rights procedures or query BEQUEATH CURRENT_USER views. • Any user can grant or revoke the INHERIT PRIVILEGES privilege on themselves to the user whose invoker‘s rights procedures they want to run. GRANT INHERIT PRIVILEGES ON USER invoking_user TO procedure_owner REVOKE INHERIT PRIVILEGES ON invoking_user FROM procedure_owner;
  • SYS_SESSION_ROLES • A new built-in namespace, SYS_SESSION_ROLES, allows you to determine if a specified role is enabled for the querying user. • The following example determines if the DBA role is enabled for user oe: • CONNECT OE/password • SELECT SYS_CONTEXT('SYS_SESSION_ROLES', 'DBA') FROM DUAL; • SYS_CONTEXT('SYS_SESSION_ROLES','DBA') -------- FALSE
  • Audit the real privilege requirements of an application • Objective: restrict privileges for a role to those that are really required for using an application • A privilege capture displays privilege usage for a database according to a specified condition – such as privileges to run an application module – or privileges used in a given user session. • When a user performs an action and you want to monitor the privileges that are used for this action, you can create a privilege capture to capture privilege usage. • Afterwards, you can view a report that describes the behavior the privilege capture defined. – The privilege capture includes both system privileges and object privileges. • Part of Database Vault option
  • Data Masking
  • Data Redaction • At runtime, you can optionally have the query results modified to reset/scramble/randomize sensitive data – Through ‗data redaction‘ policies associated with tables and view and applied at query time • Because the data is masked in real-time, Data Redaction is well suited to environments in which data is constantly changing. • You can create the Data Redaction policies in one central location and easily manage them from there. SQL engine SQL POLICY POLICY RESULTS
  • My first Data redaction policy • Access to DBMS_REDACT package • Create Data Redaction Policy for SAL column in EMP table – hide salaries from view • Find that querying EMP has changed forever… – Note: the expression can be used to dynamically decide whether or not to apply the policy grant execute on dbms_redact to scott; BEGIN DBMS_REDACT.ADD_POLICY( object_schema => 'scott', object_name => 'emp', column_name => 'sal', policy_name => 'hide_salary', function_type => DBMS_REDACT.FULL, expression => '1=1' ); END;
  • Querying EMP with DATA REDACTIOn in place • Note: drop Redaction Policy DBMS_REDACT.DROP_POLICY ( object_schema => 'scott' , object_name => 'emp' , policy_name => 'hide_salary' );
  • Partial Column masking – mask Day and month from hiredate BEGIN DBMS_REDACT.ADD_POLICY( object_schema => 'scott', object_name => 'emp', column_name => 'hiredate', policy_name => 'partially mask hiredate', function_type => DBMS_REDACT.PARTIAL, function_parameters => 'm1d31YHMS', expression => '1=1' ); END;
  • The Evolution of Oracle SQL
  • The Top-3 Earning Employees • What can you say about the result of this query with respect to the question: ―Who are our top three earning employees?‖ A. Correct Answer B. Sometimes correct C. Correct if there are never duplicate salaries D. Not Correct
  • In-Line Views
  • TOP-N Queries in 12c • Last part of a query to be evaluated – to fetch only selected rows from the result set: – To select the next set of rows: select * from emp order by sal desc FETCH FIRST 3 ROWS ONLY; select * from emp order by sal desc OFFSET 3 FETCH NEXT 4 ROWS ONLY;
  • Pagination is just a convenient syntax… • OFFSET and FETCH NEXT are replaced by optimizer with Analytic Functions such as ROW_NUMBER() c##tkyte%CDB1> select /*+ first_rows(5) */ owner, object_name, object_id 2 from t 3 order by owner, object_name 4 OFFSET 5 ROWS FETCH NEXT 5 ROWS ONLY; … --------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | --------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 5 | 1450 | 7 (0)| 00:00:01 | |* 1 | VIEW | | 5 | 1450 | 7 (0)| 00:00:01 | |* 2 | WINDOW NOSORT STOPKEY | | 5 | 180 | 7 (0)| 00:00:01 | | 3 | TABLE ACCESS BY INDEX ROWID| T | 87310 | 3069K| 7 (0)| 00:00:01 | | 4 | INDEX FULL SCAN | T_IDX | 5 | | 3 (0)| 00:00:01 | --------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter("from$_subquery$_003"."rowlimit_$$_rownumber"<=CASE WHEN (5>=0) THEN 5 ELSE 0 END +5 AND "from$_subquery$_003"."rowlimit_$$_rownumber">5) 2 - filter(ROW_NUMBER() OVER ( ORDER BY "OWNER","OBJECT_NAME")<=CASE WHEN (5>=0) THEN 5 ELSE 0 END +5)
  • BOTTOM-N QUERY in 12c • Return only the last three rows in the ordered result set (in the proper order) – or: select * from emp order by sal desc OFFSET ((select count(*) from emp)-3) ROWS FETCH NEXT 3 ROWS ONLY select * from ( select * from emp order by sal asc FETCH FIRST 3 ROWS ONLY ) order by sal desc;
  • TOP-n% querying • To query for a percentage of the result set (rather than an absolute number of rows) • And the next batch select * from emp order by sal desc FETCH FIRST 30 PERCENT ROWS ONLY; select * from emp order by sal desc OFFSET (0.3*(select count(*) from emp)) ROWS FETCH NEXT (0.3*(select count(*) from emp)) ROWS ONLY;
  • Ties • When ordering rows, multiple rows may have the same ‗ranking‘ – For example order by sal will be unable to order records with the same salary in a meaningful ways • Using WITH TIES instead of ONLY will make sure that if one record with a certain position in the order is included, they all are select * from emp order by sal desc FETCH FIRST 30 PERCENT ROWS WITH TIES;
  • In-line PL/SQL Functions and procedures • Procedures are also allowed in-line • In-Line Functions and Procedures can invoke each other WITH procedure increment( operand in out number , incsize in number) is begin operand:= operand + incsize; end; FUNCTION inc(value number) RETURN number IS l_value number(10):= value; BEGIN increment(l_value, 100); RETURN l_value; end; SELECT inc(sal) from emp
  • In-line PL/SQL Functions and procedures • In-Line Functions and Procedures can invoke each other – And themselves (recursively) WITH FUNCTION inc(value number) RETURN number IS BEGIN if value < 6000 then return inc(value+100); else return value + 100; END if; end; SELECT inc(sal) from emp
  • Faculty calculation • Inline PL/SQL function • Recursively invoked • Using APEX as IDE
  • Dynamic (PL/)SQL is allowed inside inline functions • EXECUTE IMMEDIATE can be used inside an inline PL/SQL function to dynamically construct and execute SQL and PL/SQL WITH FUNCTION EMP_ENRICHER(operand varchar2) RETURN varchar2 IS sql_stmt varchar2(500); job varchar2(500); BEGIN sql_stmt := 'SELECT job FROM emp WHERE ename = :param'; EXECUTE IMMEDIATE sql_stmt INTO job USING operand; RETURN ' has job '||job; END; SELECT ename || EMP_ENRICHER(ename) from emp
  • PL/SQL Functions That Run Faster in SQL • As of Oracle Database Release 12c, two kinds of PL/SQL functions might run faster in SQL: – PL/SQL functions that are defined in the WITH clauses of SQL SELECT statements, described in Oracle Database SQL Language Reference – PL/SQL functions that are defined with the "UDF Pragma― • Pragma UDF means: compile in the ‗SQL way‘ as to eliminate SQL  PL/SQL context switch FUNCTION inc(string VARCHAR2) RETURN VARCHAR2 IS PRAGMA UDF; value number(10):= to_number(string); BEGIN if value < 6000 then return inc(value+100); else return to_char(value + 100); end if; end;
  • Lateral inline view • ANSI SQL feature: A lateral view is an inline view that contains correlation referring to other tables that precede it in the FROM clause – You can specify this left correlation anywhere within subquery (such as the SELECT, FROM, and WHERE clauses), and at any nesting level. SELECT * FROM emp e , LATERAL( SELECT * FROM dept d WHERE e.deptno = d.deptno )
  • Lateral inline view enrich joins • Lateral In-line view can be used for ‗enrichment‘ inside multi-table join SELECT * FROM emp e cross join LATERAL( SELECT * FROM dept d WHERE e.deptno = d.deptno ) cross join LATERAL( SELECT e2.ename manager FROM emp e2 WHERE e2.deptno = e.deptno and e2.job ='MANAGER' )
  • Apply for joining • APPLY is used to join with a Collection • The function employees_in_department returns a collection (TABLE OF VARCHAR2 in this case) • The function takes a value from the DEPT records as input • Only when the returned collection is not empty will the DEPT record be produced by this join SELECT * FROM DEPT d CROSS APPLY employees_in_department(deptno) staff
  • Function that produces the collection create or replace type string_tbl as table of varchar2(200) create or replace function employees_in_department ( p_deptno in number ) return string_tbl as l_enames string_tbl; begin select cast(collect(ename) as string_tbl) into l_enames from emp where deptno = p_deptno ; return l_enames; end;select dname , deptno , employees_in_department(deptno) staff from dept
  • OUTER Apply for joining • APPLY is used to join with a Collection • With OUTER APPLY, each DEPT record is produced at least once – depending on the collection returned by the function for the DEPT record, multiple joins may be produced SELECT * FROM DEPT d OUTER APPLY employees_in_department(deptno) staff
  • Rules for APPLY • The APPLY keyword allows you to specify a table_reference or collection_expression on the right side of a join clause. – The table_reference can be a table, inline view, or TABLE collection expression; it cannot be a lateral inline view. – The collection_expression can be a subquery, a column, a function, or a collection constructor. • Regardless of its form, it must return a collection value—that is a nested table or varray. • Left correlations are allowed on the right side of APPLY. • With keyword CROSS – only rows from table that are joined with result in right side are returned • With keyword OUTER – both rows that join and those that do not are returned from the table (on the left)
  • Pattern Matching
  • Who is afraid of Red, Yellow and blue • Table Events – Column Seq number(5) – Column Payload varchar2(200)
  • Solution using Lead • With LEAD it is easy to compare a row with its successor(s) – As long as the pattern is fixed, LEAD will suffice with look_ahead_events as ( SELECT e.* , lead(payload) over (order by seq) next_color , lead(payload,2) over (order by seq) second_next_color FROM events e ) select seq from look_ahead_events where payload ='red' and next_color ='yellow' and second_next_color='blue'
  • Find the pattern red, yellow and blue • Using the new 12c Match Recognize operator for finding patterns in relational data SELECT * FROM events MATCH_RECOGNIZE ( ORDER BY seq MEASURES RED.seq AS redseq , MATCH_NUMBER() AS match_num ALL ROWS PER MATCH PATTERN (RED YELLOW BLUE) DEFINE RED AS RED.payload ='red', YELLOW AS YELLOW.payload ='yellow', BLUE AS BLUE.payload ='blue' ) MR ORDER BY MR.redseq , MR.seq;
  • Match_recognize for finding patterns in relational data • The expression MATCH_RECOGNIZE provides native SQL support to find patterns in sequences of rows • Match_recognize returns Measures for selected (pattern matched) rows – Similar to MODEL clause • Match Conditions are expressed in columns from the Table Source, aggregate functions and pattern functions FIRST, PREV, NEXT, LAST • Patterns are regular expressions using match conditions to express a special sequence of rows satisfying the conditions Table Source & Where Match_ Recognize Process and Filter Select & Order By
  • Did we ever consecutively hire three employees in the same job? • Find a string of three subsequent hires where each hire has the same job • Order by hiredate, pattern is two records that each have the same job as their predecessor SELECT * FROM EMP MATCH_RECOGNIZE ( ORDER BY hiredate MEASURES SAME_JOB.hiredate AS hireday , MATCH_NUMBER() AS match_num ALL ROWS PER MATCH PATTERN (SAME_JOB{3}) DEFINE SAME_JOB AS SAME_JOB.job = FIRST(SAME_JOB.job) ) MR
  • Row Pattern Matching c##tkyte%CDB1> select symbol, tstamp, price, 2 rpad('*',price,'*') hist 3 from stocks 4 order by symbol, tstamp; SYMBOL TSTAMP PRICE HIST ---------- --------- ---------- ---------------------------------------- ORCL 01-SEP-12 35 *********************************** ORCL 02-SEP-12 34 ********************************** ORCL 03-SEP-12 33 ********************************* ORCL 04-SEP-12 34 ********************************** ORCL 05-SEP-12 35 *********************************** ORCL 06-SEP-12 36 ************************************ ORCL 07-SEP-12 37 ************************************* ORCL 08-SEP-12 36 ************************************ ORCL 09-SEP-12 35 *********************************** ORCL 10-SEP-12 34 ********************************** ORCL 11-SEP-12 35 *********************************** ORCL 12-SEP-12 36 ************************************ ORCL 13-SEP-12 37 ************************************* 13 rows selected.
  • Row Pattern Matching c##tkyte%CDB1> SELECT * 2 FROM stocks MATCH_RECOGNIZE 3 ( PARTITION BY symbol 4 ORDER BY tstamp 5 MEASURES 6 STRT.tstamp AS start_tstamp, 7 LAST(DOWN.tstamp) AS bottom_tstamp, 8 LAST(UP.tstamp) AS end_tstamp 9 ONE ROW PER MATCH 10 AFTER MATCH SKIP TO LAST UP 11 PATTERN (STRT DOWN+ UP+) 12 DEFINE 13 DOWN AS DOWN.price < PREV(DOWN.price), 14 UP AS UP.price > PREV(UP.price) 15 ) MR 16 ORDER BY MR.symbol, MR.start_tstamp; SYMBOL START_TST BOTTOM_TS END_TSTAM ---------- --------- --------- --------- ORCL 01-SEP-12 03-SEP-12 07-SEP-12 ORCL 07-SEP-12 10-SEP-12 13-SEP-12 SYMBOL TSTAMP PRICE HIST ---------- --------- ---------- ---------------------------------------- ORCL 01-SEP-12 35 *********************************** ORCL 02-SEP-12 34 ********************************** ORCL 03-SEP-12 33 ********************************* ORCL 04-SEP-12 34 ********************************** ORCL 05-SEP-12 35 *********************************** ORCL 06-SEP-12 36 ************************************ ORCL 07-SEP-12 37 ************************************* ORCL 08-SEP-12 36 ************************************ ORCL 09-SEP-12 35 *********************************** ORCL 10-SEP-12 34 ********************************** ORCL 11-SEP-12 35 *********************************** ORCL 12-SEP-12 36 ************************************ ORCL 13-SEP-12 37 ************************************* 13 rows selected.
  • Pattern clause is a regular expression • Supported operators for the pattern clause include: – * for 0 or more iterations – + for 1 or more iterations – ? for 0 or 1 iterations – { n } for exactly n iterations (n > 0) – { n, } for n or more iterations (n >= 0) – { n, m } for between n and m (inclusive) iterations (0 <= n <= m, 0 < m) – { , m } for between 0 and m (inclusive) iterations (m > 0) – reluctant qualifiers - *?, +?, ??, {n}?, {n,}?, { n, m }?, {,m}? – | for alternation (OR) – grouping using () parentheses – exclusion using {- and -} – empty pattern using () – ^ and $ for start and end of a partition
  • Find the longest sequence of related observations • Records are ordered • Each record is qualified: assigned to a certain category • Examples: – Voting records – Ball possession in football – Days with or without rain – Passing vehicles (make and model or category) – DNA records • The challenge: find the longest string of consecutive observations in the same category
  • Find the longest sequence of related observations SELECT section_category , section_start FROM observations MATCH_RECOGNIZE ( ORDER BY seq MEASURES SAME_CATEGORY.category as section_category , FIRST(SAME_CATEGORY.seq) as section_start ONE ROW PER MATCH PATTERN (SAME_CATEGORY* DIFFERENT_CATEGORY) -- as many times as possible DEFINE SAME_CATEGORY AS SAME_CATEGORY.category = FIRST(SAME_CATEGORY.category) , DIFFERENT_CATEGORY AS DIFFERENT_CATEGORY.category != NEXT(DIFFERENT_CATEGORY.category) ) MR order by rows_in_section desc )
  • Solution using lead with observed_sections as ( select seq , case lag(category) over (order by seq) when category then 'N' -- next row has same category as current row else 'Y' end start_of_next_category – is row beginning of a section? , category from observations o ) , sectioned_observations as ( select category , seq , lead( case start_of_next_category when 'Y' then seq end) ignore nulls over (order by seq) start_next from observed_sections ) select so.category , so.seq start_position , so.start_next -1 last_position , start_next - seq section_size from sectioned_observations so order by start_next - seq desc nulls last FETCH FIRST 1 ROWS ONLY
  • Suppose we allow a single interruption of a sequence • One record with a different category will not end the sequence – it might after all be a fluke or an incident • Rewrite the pattern match to also accept one entry with a different category ONE ROW PER MATCH AFTER MATCH SKIP TO NEXT ROW -- a next row in the current match may be start of a next string PATTERN (SAME_CATEGORY* DIFFERENT_CATEGORY{0,1} SAME_CATEGORY* ) DEFINE SAME_CATEGORY AS SAME_CATEGORY.category = FIRST(SAME_CATEGORY.category) , DIFFERENT_CATEGORY AS DIFFERENT_CATEGORY.category != SAME_CATEGORY.category
  • Find sequence (with one accepted interruption) from all records SELECT substr(section_category,1,1) cat , section_start , seq FROM observations MATCH_RECOGNIZE ( ORDER BY seq MEASURES SAME_CATEGORY.category as section_category , FIRST(SAME_CATEGORY.seq) as section_start , seq as seq ONE ROW PER MATCH AFTER MATCH SKIP TO NEXT ROW -- a next row in the current match may be -- start of a next string PATTERN (SAME_CATEGORY* DIFFERENT_CATEGORY{0,1} SAME_CATEGORY* ) DEFINE SAME_CATEGORY AS SAME_CATEGORY.category = FIRST(SAME_CATEGORY.category) , DIFFERENT_CATEGORY AS DIFFERENT_CATEGORY.category != SAME_CATEGORY.category ) MR order by rows_in_section desc
  • Miscellaneous
  • Miscellaneous • Default • Varchar2 • Java in the Database • SQL Translation • Export View as Table • DICOM support for WebCenter Content • New package dbms_monitor for fine grained trace collection
  • Export View as Table • Through the Data Pump export facility, a view can be exported as a table • Upon import, a table is created – With the columns exposed by the view – And with all the data returned by the view • This provides a lot of control over what exactly is exported – and will subsequently be imported • Available in PL/SQL DBMS_DATAPUMP package and in expdp command line – Also available in network mode import
  • Varchar2 expansion • VARCHAR2 maximum size to 32Kb – In Column definition – In Type definition • On par with VARCHAR2 in PL/SQL programs • Note: anything over 4000 bytes is stored out of line, just like CLOBs are
  • Database Java Engine (JVM) • The Java engine (JVM) in the database is now JDK 6 (default) or JDK 7 (when explicitly set) • Starting from Oracle Database 12c Release 1, Oracle JVM provides support for multiple JDK versions, including the latest JDK version. – Typically, the supported versions are a default version and the next higher version. – Oracle Database 12c Release 1 (12.1) supports JDK 6 and JDK 7, where JDK 6 is the default JDK version. So, if you do not set the JDK version explicitly, the JDK version in effect is JDK 6. • You can set the JDK version prior to database creation – Or update an already created database
  • SQL Statement Preprocessor • A new mechanism is provided to allow the text of a SQL statement, submitted from a client program using an open application programming interface (API) such as ODBC or JDBC, to be translated by user-supplied code before it is submitted to the Oracle Database SQL compiler. Application SQLPre proce ssor SQL engine SQL
  • SQL Statement Preprocessor • The translation code is named and is installed in the database using a PL/SQL API. It can be implemented programmatically, or by look-up, or by a suitable mixture of these. • The mechanism also allows Oracle error codes and American National Standards Institute (ANSI) SQLSTATES to be translated by user-supplied code. • The motivating use case is to allow extant client-side application code, written for a different vendor's database (and therefore for a SQL dialect other than Oracle's), to run unchanged against an Oracle Database by emulating the syntax and semantics of the other SQL dialect thereby greatly reducing the cost of migration. • Additionally, this feature can satisfy any other use case where it is expedient to intervene between the SQL statement that the client submits and what is actually executed. • See: Oracle Database SQL Translation Installation, Configuration, and User's Guide for details
  • SQL Translation • Objective: turn SQL issued by client applications into better Oracle SQL • Package DBMS_SQL_TRANSLATOR • Register SQL for Profile and specify the SQL to execute in that case • Register Error for Profile and specify the Error Code to return in that case • Questions – Deal with bind parameters?
  • DEFAULT
  • DEFAULT • New in 12c – Default based on Sequence – Default applied (also) when NULL was explicitly specified – Identity Column that is automatically assigned generated sequence number value – Meta Data Only Defaults
  • Specify default to be applied even when insert stipulated NULL • For example: – When no salary is provided for a new employee, either because the salary is missing from the INSERT or when it is provided but set to NULL, then set a default value [of 1000] alter table emp modify (sal number(10,2) DEFAULT ON NULL 1000 )
  • DERIVE DEFAULT value from a sequence • For example: – The default value for a column that is specified as DEFAULT ON NULL can be derived from a sequence – so no before row insert trigger is needed for this! alter table emp modify (empno number(5) NOT NULL DEFAULT ON NULL EMPNO_SEQ.NEXTVAL )
  • Define column as ‗identity column‘ • Specify an identity column: .the identity column will be assigned an increasing or decreasing integer value from a sequence generator for each subsequent INSERT statement – Use ALWAYS instead of BY DEFAULT to force the generated value over any value that may have been provided in the insert statement create table emp ( empno NUMBER GENERATED BY DEFAULT AS IDENTITY (START WITH 100 INCREMENT BY 10) , ... )
  • Improved Defaults – metadata only defaults c##tkyte%CDB1> create table t 2 as 3 select * 4 from stage; Table created. c##tkyte%CDB1> exec show_space('T') … Full Blocks ..................... 1,437 Total Blocks............................ 1,536 Total Bytes............................. 12,582,912 Total MBytes............................ 12 … PL/SQL procedure successfully completed.
  • Improved Defaults – metadata only defaults c##tkyte%CDB1> set timing on c##tkyte%CDB1> alter table t add (data char(2000) default 'x'); Table altered. Elapsed: 00:00:00.07 ops$tkyte%ORA11GR2> set timing on ops$tkyte%ORA11GR2> alter table t add (data char(2000) default 'x'); Table altered. Elapsed: 00:00:28.59 11g 12c
  • Improved Defaults – metadata only defaults c##tkyte%CDB1> exec show_space('T') … Full Blocks ..................... 1,437 Total Blocks............................ 1,536 Total Bytes............................. 12,582,912 Total MBytes............................ 12 … PL/SQL procedure successfully completed. ops$tkyte%ORA11GR2> exec show_space('T') … Total MBytes............................ 9 <<<= before Total MBytes............................ 192 <<<= after … PL/SQL procedure successfully completed. 11g 12c
  • Summary
  • Best Take Aways • Flashback and Temporal Validity • In-line PL/SQL Functions • SQL Pattern Match • Default value derived from sequence (identity column) • Top-N Query (―SQL pagination‖) • Varchar2(32768) • Data Masking to create representative test data sets • Multitenant architecture allowing for fast cloning of databases • DBMS_UTILITY.EXPAND_SQL_TEXT for full query discovery