The Oracle Database 12c release introduced a number of improvements to the Flashback mechanism. For the first time, this feature is now ready for use by application developers. This presentation introduces flashback and spells out the limitation up to 11g and the improvements in 12c. It makes clear what now can be done with Flashback [Data Archive] and how that can be done. The presentation also introduces the new Temporal Validity feature in the Oracle Database that allows a form of Flash Forward (and Flashback) based on business validity of data rather than transaction timestamp. This session was presented to the SQL and PL/SQL SIG of the Dutch Oracle user group OGh on May 10th 2016.

1. Lucas Jellema
OGH – SIG SQL & PL/SQL – 10 mei 2016
The Fourth Dimension of the Database
- On Flashback and Temporal Validity in
the Oracle 12c database

2. 2
Time

3. 3
The Pension Fund

4. 4
The Pension Fund
• Administration for 2 milion participants
• Record their complete employment history
– employer, job role, salary and benefits
• Used to calculate invoices for monthly premium as well as the benefits
that have been accrued
• MN has to be able to
– Report on the situation per end of each quarter
– Account for any invoice that has been sent (and the way it was calculated)
– Provide an audit trail of all mutations in the employment history including the nature
and timestamp of the mutation and the identity of the perpetrator
– Retain history for specified periods of time

5. 5
The Pension Fund
Current approach
• Journal tables
• Audit columns to record identity of person responsible for change
• DML Triggers on all relevant tables to record history
• Long reporting batches at the end of the quarter
• Complex queries to reconstruct a certain moment in time that caused a
certain invoice
• Logic to exclude data that is no longer relevant/allowed
• [lack of] scheduled batches to remove data once beyond retention

6. Glimpses of the past
Session 1 Session 2

7. Glimpses of the past
Session 1 Session 2

8. Glimpses of the past – same
session

9. Glimpses from the past
How a session can see its own past
Session 1

10. Formalized Glimpses of the past
Flashback Query

11. Compare current with previous

12. Powered by UNDO
UNDO

13. FLASHBACK facilities
• Flashback Database
• Flashback Standby Database
• Flashback Table
• Flashback Table to Before Undrop
(Recycle Bin)
• Flashback Query
• Flashback Versions Query
• Flashback Transaction
DBA
Developer

14. 11g Flashback Data Archive
1 Month
10 Years
UNDO5 Days

15. Configuring Total Recall for
Table EMP
• Set up Flashback Data Archive
• Enable Total Recall for EMP
• Some requirements:
– Tablespace must be under Automatic Segment Space Management to create the
Flashback Data Archive in it
– Automatic UNDO management must be enabled
– The FLASHBACK ARCHIVE ADMINISTER privilege
CREATE FLASHBACK ARCHIVE LONGTERM_ARCHIVE
TABLESPACE HISTORY
RETENTION 15 YEAR
ALTER TABLE EMP
FLASHBACK ARCHIVE LONGTERM_ARCHIVE;

16. Sessions Travel back in time
• Using Flashback Query and Flashback Versions Query, the past can be
inspected
– Queries have to adapted
– Every table referenced in a query needs to be explicitly 'flashbacked'
• Non intrusive time travelling for an entire session is available too –
through dbms_flashback
– Only travelling to the past
– The past is read-only
– Travel to Timestamp or SCN
– Note: SYSDATE is still the current system time, not the historic timestamp; Views or
Queries that use SYSDATE may need tending to
• For example to set from after-logon trigger or as option in a client
application to set the “time scope”
– For example to create reports for the end of the year – no need to stop new data
coming in!

17. Move entire session to point in time

18. Flashback Row History aka
flashback versions query
• See all flashback history data for each row, including:
– SCN range the row was “valid for”
– Time range (approx) the row was valid for
– The transaction id (XID) that modified the row
– The operation (I/U/D) that was performed on the row
– special constants: minvalue and maxvalue
select ename, sal,
versions_operation,
versions_starttime,
versions_endtime,
versions_startscn,
versions_endscn,
versions_xid
from emp versions between timestamp &A and &B
order by versions_startscn nulls first
from emp versions between timestamp minvalue and maxvalue

19. Trac-king's Salary History

20. Provide insight in trends and recent
changes using flashback versions
10 Years

21. MOST_RECENT_SALARY_CHANGES

22. “Flashback Transaction”
• Select the UNDO_SQL for a certain transaction
select xid
, start_scn
, commit_scn
, operation
, undo_sql
, table_name
from
flashback_transaction_query
where xid = '0004000400000003D';

23. Transaction Back Out
• To undo the effects of one or more transactions, we can invoke
DBMS_FLASHBACK.TRANSACTION_BACKOUT
– with a list of Transaction IDs
– and options for dealing with dependent transactions (later transactions partially on
the same records)
• NonConflict_Only: only backout those parts of selected transaction that do not conflict with later transactions
• Nocascade: raise error if there is dependent transaction
• NoCascade_Force: only backout selected transaction
• Cascade: back out of dependent transaction too
– Backout does not commit: it performs compensating DML, constraints are enforced,
holds locks and reports
– Back out reports are available in views
*_FLASHBACK_TXN_STATE and *_FLASHBACK_TXN_REPORT

24. …_JOURNAL…
Date_modified
User_modified
Date_modified
User_modified
+ columns for
… columns
…_JOURNAL…
Date_modified
User_modified
Date_modified
User_modified
+ columns for
… columns
History is frequently kept in Audit
columns and journaling tables
EMP_JOURNALEMP
Date_modified
User_modified
Date_modified
User_modified
+ columns for
… columns
…_JOURNAL…
Date_modified
User_modified
Date_modified
User_modified
+ columns for
… columns

25. Flashback data archive could mean
The end of journaling
• Get rid of all journaling tables & triggers
• Productivity Gain
• Ease of use of archived data
– AS OF queries
– DBMS_FLASHBACK – transparent
• Performance Benefits
– Maintain flashback data archive is much more efficient than maintaining journaling
tables with triggers
10 YearsEMP
… columns
Date_modified
… columns

26. Overhead of populating Flashback
Data archive compared to triggers
Total Recall vs Triggers
6.0%
8.9%
2.5%
4.5%
54.0%53.2%
56.9%
52.8%
0
10
20
30
40
50
60
70
100k/1k 100k/10k 1M/1k 1M/10k
# of Rows / Commit Interval
%IncreaseinResponseTime
Total Recall Triggers

27. Whodunnit?
• Flashback Data Archive does not record WHO made the change – only
the changes that were made!
• Database Auditing does record the username and the client identifier for
each completed transaction
– AUDIT INSERT,UPDATE,DELETE ON EMP
– Join Flashback Versions Query with user_audit_trail on audit.transactionid =
version.xid
10 YearsEMP
… columns xid Date_modified
… columns
USER_AUDIT
_TRAIL
Client Identifier

28. 28
Whodunnit:
Alternative Approaches
• USER_AUDIT_TRAIL is not meant to be part of business data – nor for
long term storage and historical purposes
– If we need historical record of who performed transactions, we need to do something
in addition to or instead of USER_AUDIT_TRAIL
• One approach: maintain a custom transaction registry that records each
(relevant) transaction with relevant values
– Record transaction identifier, SCN and relevant generic transaction context values
such as client identifier, db_user and os_user
– Options 1: Use FGA policy on each table with FLASHBACK ambition to record
transaction (requires Database Enterprise Edition to use dbms_fga)
– Option 2: Generate custom DML statement level trigger on each table with
FLASHBACK ambition to record transaction
– Join Flashback Versions Query with custom transaction registry on
audit.transactionid = version.xid

29. Flashback Weak Spots as of
Oracle Database 11g R2
• Whodunnit?
• Also:
– when is the start of history?
– What went on before? What to do with existing archives?
– How to deal with Flashback Data during export/import?
– Some DDL is hard to absorb
• Oh, and by the way: Flashback Data Archive
requires Enterprise Edition
& Advanced Compression database option

30. 30
License change in 12c
• Flashback Data Archive in all database editions

31. 31
• Flashback Data Archive also available in 11.2.0.4 – all editions

32. Total Recall 12c
• Flashback Data Archive Improvements:
– Complete schema evolution support: all table definition, partitioning,
and space management DDLs are supported on FDA-enabled
tables.
• For example:
– add column – results from before the column was added will show
NULL for that column
– drop column – historic results will show the dropped column with a
system generated prefix (D_<number>_ColumnName)
– Note: drop table can only be done by user with extended privileges

33. Total Recall 12c
• Flashback Data Archive Improvements:
– The metadata information for tracking transactions including the
user context is now tracked. The addition of user-context tracking
makes it easier to determine which user made which changes to a
table.
• This could mean that journaling tables can now officially be deprecated
• Especially given the fact that the current contents of journaling tables can even be
migrated to Flashback Data Archive

34. 34
Ensure transaction context is
recorded (and set)
exec dbms_flashback_archive.set_context_level(level=> 'ALL');
exec dbms_session.set_identifier('The Creepy User from Finance ');
update oow.emp
set sal = sal * 1.4
where ename = 'ALLEN'
/
commit;
exec dbms_session.set_identifier('Scary Janitor from the Annex');
update oow.emp
set sal = sal * 0.7
where ename = 'MILLER'
/
commit;

35. 35
Audit the generated history
SELECT versions_xid
, versions_starttime
, empno, ename, sal new_sal
, s.client_identifier
FROM oow.emp VERSIONS BETWEEN TIMESTAMP minvalue
AND maxvalue
join
sys.sys_fba_context_aud s
on (versions_xid = s.xid)

36. 36
Alternative: retrieve context with
dbms_flashback_archive.get_sys_context
SELECT versions_xid
, versions_starttime
, empno, ename, sal new_sal
, dbms_flashback_archive.get_sys_context
(versions_xid,'USERENV','CLIENT_IDENTIFIER') who
FROM emp VERSIONS BETWEEN TIMESTAMP minvalue
AND maxvalue

37. Total Recall (2)
• Import and export of Flashback Data Archive history using
Data Pump
– Support for import and export using Data Pump for FDA-enabled
tables. Data Pump can now be used to export and import an FDA-
enabled base table along with its schema-evolution metadata and
historical row versions.

38. Total Recall (3)
• User generated history
– Support for importing user-generated history has been added.
Customers who have been maintaining history using other
mechanisms, such as triggers, can now import that history into Total
Recall.
• The solution for that question of 'what went on before
history started' with Flashback Database Archive prior to
12c

39. 39
Generate History –
All actions by SYS
create table oow.emp as select * from scott.emp
grant execute on dbms_flashback_archive to oow;
grant execute on dbms_flashback to oow;
CREATE FLASHBACK ARCHIVE DEFAULT one_year TABLESPACE users
QUOTA 100M RETENTION 1 YEAR;
grant flashback archive on one_year to oow;
exec dbms_flashback_archive.create_temp_history_table('OOW', 'EMP');
-- This statement needs execution once in a database instance. This
will extend mappings to the past so that import of old history can
be done. Goes back to 01-JAN-88.
EXEC DBMS_FLASHBACK_ARCHIVE.extend_mappings();
---
-- after some history has been created:
EXEC DBMS_FLASHBACK_ARCHIVE.IMPORT_HISTORY('oow','EMP');

40. 40
Generate History –
Actions by Application
• Insert records describing each stage of history that has existed
– Including start and end time of historic state
– then import history [from temporary table into Flashback Data Archive]
insert into temp_history
(RID , STARTSCN , ENDSCN , XID, OPERATION
,EMPNO, ename, job, hiredate, sal, deptno )
values (NULL, timestamp_to_scn(to_date('01-04-2001', 'DD-MM-YYYY')),
timestamp_to_scn(to_date('01-07-2003', 'DD-MM-YYYY')), NULL, 'I'
,1567,'SELLERS','CLERK',to_date('01-04-2001','DD-MM-YYYY'),2200, 10);
insert into temp_history
(RID , STARTSCN , ENDSCN , XID, OPERATION
,EMPNO, ename, job, hiredate, sal, deptno)
values (NULL, timestamp_to_scn(to_date('01-07-2003', 'DD-MM-YYYY')),
timestamp_to_scn(to_date('01-10-2006', 'DD-MM-YYYY')), NULL, 'U'
,1567,'SELLERS','CLERK',to_date('01-04-2001','DD-MM-YYYY'),2200, 20);
…

41. 41
Query the generated history
select ename
, job
from emp as of timestamp (sysdate - INTERVAL '10' YEAR)
minus
select ename
, job
from emp
select ename
, job
from emp as of timestamp (systimestamp - INTERVAL '3' YEAR)
minus
select ename
, job
from emp

42. DBMS_COMPARISON
• Supplied package DBMS_COMPARISON
• To compare tables or views
– Within the same schema or database or across databases (over a database link)
– Reports the number of differences – if any
– Reports the individual ROWIDs for all
rows that have differences
• To merge tables or
views in order to
synchronize one
table with the
other (master)

43. DBMS_COMPARISON
• First define a comparison:
• Execute the comparison:
DBMS_COMPARISON.CREATE_COMPARISON
( comparison_name => 'comp_emp_n_clone'
, schema_name => 'scott'
, object_name => 'emp'
, dblink_name => null –- same database
, column_list => null –- all columns participate
, remote_schema_name=>'scott'
, remote_object_name=>'EMP_COPY'
);
consistent := DBMS_COMPARISON.COMPARE
( comparison_name => 'comp_emp_n_clone'
, scan_info => scan_info
, perform_row_dif => TRUE );
dbms_output.put_line(' scan id = '||scan_info.id);

44. DBMS_COMPARISON
• Report differences:
SELECT r.INDEX_VALUE -- PK value
, case when r.LOCAL_ROWID is null
then 'REMOTE'
else case when r.REMOTE_ROWID is null
then 'LOCAL'
else 'BOTH'
end
end LOCAL_OR_REMOTE
FROM USER_COMPARISON_COLUMNS c
, USER_COMPARISON_ROW_DIF r
, USER_COMPARISON_SCAN s
WHERE c.COMPARISON_NAME = 'COMP_EMP_N_CLONE'
AND r.SCAN_ID = s.SCAN_ID
AND s.scan_id = &scan_id
AND r.STATUS = 'DIF'
AND c.INDEX_COLUMN = 'Y'
AND c.COMPARISON_NAME = r.COMPARISON_NAME

45. DBMS_COMPARISON
• Synchronize Local with Remote
– (Replicate Remote to Local):
DECLARE
scan_info DBMS_COMPARISON.COMPARISON_TYPE;
BEGIN
DBMS_COMPARISON.CONVERGE
( comparison_name => 'compare_emp_and_clone'
, scan_id => &scan_id
, scan_info => scan_info
, converge_options =>
DBMS_COMPARISON.CMP_CONVERGE_REMOTE_WINS
);
DBMS_OUTPUT.PUT_LINE('Remote Rows Merged: '
||scan_info.rmt_rows_merged);
DBMS_OUTPUT.PUT_LINE('Remote Rows Deleted: '
||scan_info.rmt_rows_deleted);
END;

46. Fine Grained Flashback Table
• Package DBaMiS_PARTIAL_FLASHBACK allows partial flashback
revert [parts of] selected records to
a previous state:
– only certain columns
– only specific rows
– Based on DBMS_COMPARISON
and Flashback Query
– Note: works flawlessly in 11g.
In 12c – dbms_comparison can
not run against a view that uses
AS OF in the from clause
Partial Table Flashback
T
a
b
l
e
A
r
c
h
i
v
e

47. Fine Grained Flashback Table
• Flashback DEPARTMENT 10
– and do not touch records from other departments
• Flashback Salary Data – but not any other columns
dbamis_partial_flashback.partial_flashback_table
( p_table_name => 'EMP'
, p_schema_name => 'SCOTT'
, p_to_timestamp => SYSTIMESTAMP – 30 –- go back one month
, p_column_list => 'EMPNO,SAL,COMM' -- only undo SAL & COMM
, p_undo_insert_flag => false — do not undo new inserts
);
dbamis_partial_flashback.partial_flashback_table
( p_table_name => 'EMP'
, p_schema_name => 'SCOTT'
, p_to_timestamp => SYSTIMESTAMP - 1/24*4 -- 4 hours back
, p_filter_condition => 'DEPTNO=10'
);

48. 48
The Pension Fund
Future approach
• Flashback Data Archives
– Per retention period
• Associate relevant tables to FDA of
applicable retention time
• Configure tracking of meta-data
– To ensure transaction context is retained
• Report batches can be run at any
moment against any moment
– Using dbms_flashback.enable_at_time
• Queries can easily be performed to
reproduce a moment in the past
– No impact of historic queries on SQL
• FDA management takes care of
deprecated data
• Better transaction performance through
lower overhead
• Journal tables
• Audit columns
• DML Journaling Triggers
• Batches at the end of the
quarter
• Complex queries to reconstruct
• Logic to exclude deprecated
data
• batches to purge data

49. 49
Commercial Interval
• Donderdag en vrijdag 2 en 3 juni – Vliegveld Valkenburg
• 120 sessies, 8 zalen, 1 keynote & 1 feestavond
• 80 sprekers uit 5 continenten
• Sprekers o.a.: Jonathan Lewis, Tim Hall, Graham Wood, Alex Nuijten, Patrick Barel, Heli
Helskyaho, Bryn Llewellyn, Pete Finnigan, Björn Rost, Alan Arentsen, Francisco Munoz
Alvarez, Toon Koppelaars, Frits Hoogland, Marco Gralike, Iloon Ellen-Wolff, Kellyn Potvin,…

50. 50
Commercial Interval
• Registratie op http://www.amis-conference.com
• Kosten: 795 euro
– Maar bij mij kan je een kortingscode krijgen (voor de Early Bird fee)

51. 51
Looking into the future…
OUR_PRODUCTS
NAME PRICE
select name, price
from our_products

52. 52
Looking further into the
future…
OUR_PRODUCTS
NAME PRICE
select name, price
from our_products
begin
DBMS_FLASHBACK_ARCHIVE.ENABLE_AT_VALID_TIME (
level => 'ASOF'
, query_time => TO_TIMESTAMP('01-10-2018', 'DD-MM-YYYY')
);
end;

53. 53
Current situation …
OUR_PRODUCTS
NAME PRICE
select name, price
from our_products
begin
DBMS_FLASHBACK_ARCHIVE.ENABLE_AT_VALID_TIME (
level => 'CURRENT'
);
end;

54. 54
All data in the table
(the default setting)
OUR_PRODUCTS
NAME PRICE
select name, price
from our_products
begin
DBMS_FLASHBACK_ARCHIVE.ENABLE_AT_VALID_TIME (
level => 'ALL'
);
end;

55. 55
All data in the table
(the default setting)
OUR_PRODUCTS
NAME PRICE
select name, price, start_date, end_date
from our_products
order
by start_date
START_DATE END_DATE
begin
DBMS_FLASHBACK_ARCHIVE.ENABLE_AT_VALID_TIME (
level => 'ALL'
);
end;

56. 56
Part of SQL 2011 standard:
Temporal Database

57. Make the database aware of the time
based business validity of records
• Add timestamp columns indicating start and end of valid time for a record
• Specify a PERIOD for the table
• Note:
– A table can have multiple sets of columns, describing multiple types of temporal
business validity
create table our_products
( name varchar2(100)
, price number(7,2)
, start_date timestamp
, end_date timestamp
, PERIOD FOR offer_time (start_date, end_date)
);

58. Valid time aware
flashback and flash foward queries
• Select all product prices on offer at a certain moment in time
• Perform all queries for records that are valid at a certain point in time –
past or future
• Return all records currently (session time) valid
• Return all records (default)
SELECT *
FROM OUR_PRODUCTS AS OF PERIOD FOR offer_time
TO_TIMESTAMP('01-10-2014','DD-MM-YYYY')
EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time
( 'ASOF'
, TO_TIMESTAMP('01-05-2017','DD-MM-YYYY')
);
EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time('CURRENT');
EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time('ALL');

59. 59
Temporal Validity –
Chairmanships with gaps and overlaps
• Chairperson of the board
– At any time there must be a chairperson – and at no time that can be more than one
• Table to hold chairperson-ships

60. 60
Temporal Validity –
Chairmanships with gaps and overlaps
• Select chairperson at a certain moment in time
• Perform all queries for records that are valid at a certain point in time –
past or future
SELECT *
FROM chairmanships AS OF PERIOD FOR chairmanship
TO_TIMESTAMP('01-02-1980', 'DD-MM-YYYY')
EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time
( 'ASOF'
, TO_TIMESTAMP('01-10-2018','DD-MM-YYYY')
);

61. 61
The future of temporal validity
Beyond 12.1.0.2 many Temporal Validity enhancements are expected:
• Gap and overlap checks ; support for PACKing consecutive records
• Valid time aware DML
• Temporal Primary and Unique Key constraints that allow multiple non-
overlapping entries
• Temporal referential constraints that take into account the valid-time
during which the rows exist.
– Child needs to have a valid Master at any time during its own validity
• Temporal Aggregation - group or order by valid-time
• Normalization - coalescing rows which are in adjacent or overlapping time
periods
• Temporal joins – joins between tables with valid-time semantics based on
'simultaneous validity'
• For Information Lifecycle Management (ILM), the Valid Time information is
used to assess records to move

62. 62
Summary
• Time and History are supported in Oracle Database 12c along two
dimensions:
– Transaction time – what been the committed state of the database at any point
– Business (valid) time – what was, is or will be the relevant data at any point
• Flashback is concerned with transaction time. As of 12c:
– Available in all editions (no license restrictions)
– Includes transaction (session) context at time of commit (whodunnit)
– Supports user defined history (as well as import and export)
– Flashback Query, Flashback Versions Query, Enable database session at timestamp
 Flashback is ready for inclusion in database design and application development
• Valid Time Modeling (Temporal Database) is concerned with business
validity of data – begin date and end date
– Based on SQL 2011
– Low impact use: alter table add (period for …)
– Easy querying across time – past, present and future
– Next steps are expected in enriching temporal database support (12.2?)