SlideShare a Scribd company logo
1 of 65
Download to read offline
Confoo 2015
Tracking your data across
the fourth dimension
Jeremy Cook
This talk is about temporal
databases
–Wikipedia
“A temporal database is a database with built-in
support for handling data involving time…”
– Jeff Carouth, https://twitter.com/jcarouth/status/496842218674470912
“Tonight @JCook21 explained temporal databases
and I’m sure my brain is now leaking out of my
nose.”
What problem do temporal
databases solve?
Databases are good at ‘now’
❖ Create
❖ Read
❖ Update
❖ Delete
❖ At any point we only see the current state of the data
Databases are good at ‘now’
❖ How many people work in each department of the
company?
❖ For each product category how many products are in
stock? Where is the stock located at?
❖ How many orders are currently in each fulfilment state?
The fourth dimension
❖ Show me how salaries paid have changed by
department for each quarter over the last 4 years and
how they’re forecast to change next year
❖ Show me how stock levels have changed over time. How
much stock are we forecast to have at any point in the
future?
❖ For audit purposes show me a complete history of every
change to this data, what period of time each change
was valid for and when we knew about any changes
The fourth dimension
Some Temporal Database Theory
Temporal aspects
Decision Time
❖ Records the time at which a decision was made
❖ Modelled as a single value
❖ Allows for granularity through the data type used
Decision Time
EmpId Name Hire Date Decision to Hire
1 Jeremy 2014-03-03 2014-01-20
2 Anna 2015-01-02 2013-12-15
3 Yann 2013-08-20 2013-08-20
Valid Time
“In temporal databases, valid time (VT) is the time
period during which a database fact is valid in the
modelled reality.”
–Wikipedia
Valid Time
❖ Modelled as a period of time between two dates
❖ Lower bound is always closed but upper bound can be
open
Valid Time
EmpId Name Hire date Termination date
1 Jeremy 2014-03-03 2015-01-20
2 Anna 2015-01-02 ∞
3 Yann 2013-08-20 2015-12-22
4 Colin 2015-05-01 ∞
Valid Time
EmpId Name Dept Hire date Term date StartVT EndVT
1 Jeremy Dev 2014-03-03 ∞ 2014-03-03 2014-07-30
1 Jeremy QA 2014-03-03 2015-01-20 2015-01-21 2015-01-20
2 Anna Dev 2015-01-02 ∞ 2015-01-02 2015-01-30
2 Anna Mgmt 2015-01-02 ∞ 2015-01-31 ∞
3 Yann Mgmt 2013-08-20 2015-12-22 2013-08-20 ∞
4 Colin Dev 2015-05-01 ∞ 2015-05-01 ∞
Job done?
Valid-time on its own may not be enough!
Name Type StartVT EndVT
Saturn Planet
Billions of years
ago
∞
Pluto Planet
Billions of years
ago
∞
Valid-time on its own may not be enough!
Name Type StartVT EndVT
Saturn Planet
Billions of years
ago
∞
Pluto Dwarf planet
Billions of years
ago
∞
Valid-time on its own may not be enough!
Name Type StartVT EndVT
Saturn Planet
Billions of years
ago
∞
Pluto Plutoid
Billions of years
ago
∞
Valid-time on its own may not be enough!
Name Type StartVT EndVT
Saturn Planet
Billions of years
ago
∞
Pluto Planet
Billions of years
ago
2006
Pluto Dwarf planet 2006 2008
Pluto Plutoid 2008 ∞
Transaction Time
“In temporal databases, transaction time (TT) is the
time period during which a fact stored in the
database is considered to be true.”
–Wikipedia
Transaction Time
❖ Modelled as a period of time between two dates
❖ Lower bound is always closed but upper bound can be
open
Transaction Time
Name Type StartVT EndVT StartTT EndTT
Pluto Planet
Billions of
years ago
∞ 1930 2006
Pluto
Dwarf
planet
Billions of
years ago
∞ 2006 2008
Pluto Plutoid
Billions of
years ago
∞ 2008 ∞
Valid Time != Transaction Time
Name Clothing StartVT EndVT StartTT EndTT
Father
Christmas
null
A long time
ago
∞ 1973 1975
Santa Claus red
A long time
ago
∞ 1975 1980
Saint
Nicholas
red 270 AD ∞ 1980 1982
How many temporal aspects should you use?
❖ As many or few as your application needs!
❖ Tables that implement two aspects are bi-temporal
❖ You can implement more aspects, in which case you
have multi temporal tables
Is your head spinning?
❖ Decision time records when a decision was taken
❖ Valid Time records the period of time for which the fact
is valid
❖ Transaction Time records the period of time for which
the fact is considered to be true
SQL:2011 Temporal
A note on the example tables
CREATE TABLE dept (
DNo INTEGER,
DName VARCHAR(255)
);
CREATE TABLE emp (
ENo INTEGER,
EName VARCHAR(255),
EDept INTEGER
);
Periods
❖ Table component, capturing a pair of columns defining
a start and end date
❖ Not a new data type, but metadata about columns in the
table
❖ Closed-open constraint
❖ Enforces that end time > start time
Valid time
❖ Also called application time in SQL:2011
❖ Modelled as a pair of date time columns with a period
❖ Name of the columns and period is up to you
Valid time
ALTER TABLE emp ADD (
EStart DATE,
EEnd DATE,
PERIOD FOR EPeriod (EStart, EEnd)
);
Temporal primary keys
❖ SQL:2011 allows a valid time period to be named as part
of a primary key
❖ Can also enforce that the valid time periods do not
overlap
Temporal primary keys
ALTER TABLE emp
ADD PRIMARY KEY (ENo, EPeriod);
Temporal primary keys
ALTER TABLE emp
ADD PRIMARY KEY
(ENo, EPeriod WITHOUT OVERLAPS);
Temporal foreign keys
❖ What happens if a parent and child table both define
valid time periods?
❖ It doesn’t make sense to allow a row in a child table to
reference a row in a parent table where the valid time
does not overlap
❖ SQL:2011 allows valid time periods to be part of foreign
key constraints
Temporal foreign keys
ALTER TABLE dept ADD (
DStart DATE,
DEnd DATE,
PERIOD FOR DPeriod (DStart, DEnd)
);
ALTER TABLE emp
ADD FOREIGN KEY (Edept, EPeriod)
REFERENCES dept (DNo, PERIOD DPeriod);
Querying valid time tables
❖ Can query against valid time columns as normal -
they’re just normal table columns
❖ Updates and deletes can be performed for a period of a
valid time time period
Querying valid time tables
❖ SQL:2011 allows you to create periods to use in your queries
and use new predicates:
❖ CONTAINS
❖ OVERLAPS
❖ EQUALS
❖ PRECEDES
❖ SUCCEEDS
❖ IMMEDIATELY SUCCEEDS and IMMEDIATELY PRECEDES
Querying valid time tables
UPDATE Emp
FOR PORTION OF EPeriod
FROM DATE '2011-02-03'
TO DATE '2011-09-10'
SET EDept = 4
WHERE ENo = 22217;
Querying valid time tables
DELETE Emp
FOR PORTION OF EPeriod
FROM DATE '2011-02-03'
TO DATE '2011-09-10'
WHERE ENo = 22217;
Querying valid time tables
SELECT EName, Edept
FROM Emp
WHERE ENo = 22217
AND EPeriod CONTAINS DATE '2015-01-23';
Querying valid time tables
SELECT EName, Edept
FROM Emp
WHERE ENo = 31
AND EPeriod OVERLAPS
PERIOD (DATE '2015-01-01',
DATE '2015-01-31');
Transaction time
❖ Also known as system time in SQL:2011
❖ Modelled as two DATE or TIMESTAMP columns
❖ Management of the columns for the period is handled
by the database for you
Transaction time
❖ When data is inserted:
❖ Start of transaction time is set to current time
Transaction time
❖ When data is updated:
❖ Transaction time end is set to current time on the
existing row
❖ A new row is added with the updated date and a
transaction time start of the current time
Transaction time
❖ When data is deleted:
❖ Transaction time end is set to current time in the
existing row
Transaction time
❖ Because the system manages transaction time:
❖ Not possible to alter transaction time values in the
past
❖ Not possible to add future dated transaction time
values
❖ Referential constraints on historical data are never
checked
Transaction time
CREATE TABLE emp (
…,
Sys_start TIMESTAMP(12) GENERATED ALWAYS
AS ROW START,
Sys_end TIMESTAMP(12) GENERATED ALWAYS
AS ROW END,
PERIOD FOR SYSTEM_TIME (Sys_start,
Sys_end)
) WITH SYSTEM VERSIONING;
Querying transaction time tables
❖ New predicates to be used with transaction time:
❖ FOR SYSTEM_TIME AS OF
❖ FOR SYSTEM_TIME FROM
❖ FOR SYSTEM_TIME BETWEEN
❖ If none of the above supplied the database should only
return rows for the current system time
Querying transaction time tables
SELECT ENo, EName
FROM emp
WHERE Eno = 22;
Querying transaction time tables
SELECT ENo, EName
FROM emp
WHERE ENo = 22
FOR SYSTEM_TIME AS OF
TIMESTAMP '2015-01-28 12:45:00';
Querying transaction time tables
SELECT ENo, EName
FROM emp
WHERE ENo = 22
AND EPeriod CONTAINS DATE '2014-08-27'
FOR SYSTEM_TIME AS OF
TIMESTAMP '2015-01-28 12:45:00';
Grey areas/not implemented yet
❖ Evolving schema over time
❖ Support for period joins
❖ Support for period aggregates or period grouped
queries
❖ Support for period normalization
❖ Support for multiple valid time periods per table
Which vendors support SQL:
2011?
Current support
❖ Oracle 12c
❖ SQL:2011 compliant but not even nearly complete
❖ PostgreSQL
❖ 9.1 and earlier: temporal contributed package
❖ 9.2 native ranged data types
❖ IBM DB2 through ‘time travel query’ feature
❖ Teradata 13.10 and 14
❖ Handful of others implemented as extensions
How do I add this stuff to my
current schema?
Implementing valid time
❖ Add a pair of date time columns to your table for the
valid time period.
❖ Can make these part of your primary key
Implementing valid time
❖ Things to consider:
❖ Have to check for end time > start time
❖ Have to check for overlaps in valid time periods
❖ Temporal foreign keys have to be implemented
yourself
❖ Queries become potentially more complex
Implementing transaction time
❖ Add a column recording transaction time start to your table
❖ For each table create a backup table mirroring the columns
in the main table, adding a transaction time end column too
❖ Create a trigger that fires on each update or delete to copy
old values from the main table to the backup table
❖ Should add transaction time end to the backup table
❖ Should also update the transaction time start to now in
the main table if the operation is an update
Implementing transaction time
❖ Things to consider:
❖ Extra complexity
❖ How long should backup data be kept for?
❖ Do you optimize for fast reads or writes?
❖ Should truncating the main table delete the data from
the backup?
More information
❖ Wikipedia article on Temporal Databases
❖ Temporal features in SQL:2011 (PDF)
❖ Time and Relational Theory
Thanks for listening!
❖ Any questions?
❖ I’d love some feedback
❖ https://joind.in/talk/view/13294
❖ Contact me:
❖ @JCook21
❖ jeremycook0@icloud.com

More Related Content

Similar to Tracking your data across the fourth dimension

Back to the future - Temporal Table in SQL Server 2016
Back to the future - Temporal Table in SQL Server 2016Back to the future - Temporal Table in SQL Server 2016
Back to the future - Temporal Table in SQL Server 2016Stéphane Fréchette
 
Timo Walther - Table & SQL API - unified APIs for batch and stream processing
Timo Walther - Table & SQL API - unified APIs for batch and stream processingTimo Walther - Table & SQL API - unified APIs for batch and stream processing
Timo Walther - Table & SQL API - unified APIs for batch and stream processingVerverica
 
SQL Extensions to Support Streaming Data With Fabian Hueske | Current 2022
SQL Extensions to Support Streaming Data With Fabian Hueske | Current 2022SQL Extensions to Support Streaming Data With Fabian Hueske | Current 2022
SQL Extensions to Support Streaming Data With Fabian Hueske | Current 2022HostedbyConfluent
 
Automation in Excel Reporting - Marco Kiesewetter
Automation in Excel Reporting - Marco KiesewetterAutomation in Excel Reporting - Marco Kiesewetter
Automation in Excel Reporting - Marco KiesewetterMarco M. Kiesewetter, MBA
 
Delta: Building Merge on Read
Delta: Building Merge on ReadDelta: Building Merge on Read
Delta: Building Merge on ReadDatabricks
 
Adminlicious - A Guide To TCO Features In Domino v10
Adminlicious - A Guide To TCO Features In Domino v10Adminlicious - A Guide To TCO Features In Domino v10
Adminlicious - A Guide To TCO Features In Domino v10Gabriella Davis
 
Perfect trio : temporal tables, transparent archiving in db2 for z_os and idaa
Perfect trio : temporal tables, transparent archiving in db2 for z_os and idaaPerfect trio : temporal tables, transparent archiving in db2 for z_os and idaa
Perfect trio : temporal tables, transparent archiving in db2 for z_os and idaaCuneyt Goksu
 
Value streammapping cascadiait2014-mceniry
Value streammapping cascadiait2014-mceniryValue streammapping cascadiait2014-mceniry
Value streammapping cascadiait2014-mceniryChris McEniry
 
Ultimate Real-Time — Monitor Anything, Update Anything
Ultimate Real-Time — Monitor Anything, Update AnythingUltimate Real-Time — Monitor Anything, Update Anything
Ultimate Real-Time — Monitor Anything, Update AnythingSafe Software
 
ApacheCon 2020 - Flink SQL in 2020: Time to show off!
ApacheCon 2020 - Flink SQL in 2020: Time to show off!ApacheCon 2020 - Flink SQL in 2020: Time to show off!
ApacheCon 2020 - Flink SQL in 2020: Time to show off!Timo Walther
 
Julian Hyde - Streaming SQL
Julian Hyde - Streaming SQLJulian Hyde - Streaming SQL
Julian Hyde - Streaming SQLFlink Forward
 
Streaming SQL (at FlinkForward, Berlin, 2016/09/12)
Streaming SQL (at FlinkForward, Berlin, 2016/09/12)Streaming SQL (at FlinkForward, Berlin, 2016/09/12)
Streaming SQL (at FlinkForward, Berlin, 2016/09/12)Julian Hyde
 
Sql 2016 - What's New
Sql 2016 - What's NewSql 2016 - What's New
Sql 2016 - What's Newdpcobb
 
Dev112 let's calendar that
Dev112   let's calendar thatDev112   let's calendar that
Dev112 let's calendar thatHoward Greenberg
 
Back to FME School - Day 2: Your Data and FME
Back to FME School - Day 2: Your Data and FMEBack to FME School - Day 2: Your Data and FME
Back to FME School - Day 2: Your Data and FMESafe Software
 
Azure stream analytics by Nico Jacobs
Azure stream analytics by Nico JacobsAzure stream analytics by Nico Jacobs
Azure stream analytics by Nico JacobsITProceed
 
Data modeling trends for Analytics
Data modeling trends for AnalyticsData modeling trends for Analytics
Data modeling trends for AnalyticsIke Ellis
 
Temporal Tables, Transparent Archiving in DB2 for z/OS and IDAA
Temporal Tables, Transparent Archiving in DB2 for z/OS and IDAATemporal Tables, Transparent Archiving in DB2 for z/OS and IDAA
Temporal Tables, Transparent Archiving in DB2 for z/OS and IDAACuneyt Goksu
 

Similar to Tracking your data across the fourth dimension (20)

Back to the future - Temporal Table in SQL Server 2016
Back to the future - Temporal Table in SQL Server 2016Back to the future - Temporal Table in SQL Server 2016
Back to the future - Temporal Table in SQL Server 2016
 
Timo Walther - Table & SQL API - unified APIs for batch and stream processing
Timo Walther - Table & SQL API - unified APIs for batch and stream processingTimo Walther - Table & SQL API - unified APIs for batch and stream processing
Timo Walther - Table & SQL API - unified APIs for batch and stream processing
 
SQL Extensions to Support Streaming Data With Fabian Hueske | Current 2022
SQL Extensions to Support Streaming Data With Fabian Hueske | Current 2022SQL Extensions to Support Streaming Data With Fabian Hueske | Current 2022
SQL Extensions to Support Streaming Data With Fabian Hueske | Current 2022
 
Automation in Excel Reporting - Marco Kiesewetter
Automation in Excel Reporting - Marco KiesewetterAutomation in Excel Reporting - Marco Kiesewetter
Automation in Excel Reporting - Marco Kiesewetter
 
Oracle Primavera P6 PRO R8 Tips & tricks
Oracle Primavera P6 PRO R8 Tips & tricksOracle Primavera P6 PRO R8 Tips & tricks
Oracle Primavera P6 PRO R8 Tips & tricks
 
Delta: Building Merge on Read
Delta: Building Merge on ReadDelta: Building Merge on Read
Delta: Building Merge on Read
 
Adminlicious - A Guide To TCO Features In Domino v10
Adminlicious - A Guide To TCO Features In Domino v10Adminlicious - A Guide To TCO Features In Domino v10
Adminlicious - A Guide To TCO Features In Domino v10
 
Perfect trio : temporal tables, transparent archiving in db2 for z_os and idaa
Perfect trio : temporal tables, transparent archiving in db2 for z_os and idaaPerfect trio : temporal tables, transparent archiving in db2 for z_os and idaa
Perfect trio : temporal tables, transparent archiving in db2 for z_os and idaa
 
Value streammapping cascadiait2014-mceniry
Value streammapping cascadiait2014-mceniryValue streammapping cascadiait2014-mceniry
Value streammapping cascadiait2014-mceniry
 
Ultimate Real-Time — Monitor Anything, Update Anything
Ultimate Real-Time — Monitor Anything, Update AnythingUltimate Real-Time — Monitor Anything, Update Anything
Ultimate Real-Time — Monitor Anything, Update Anything
 
Streaming SQL
Streaming SQLStreaming SQL
Streaming SQL
 
ApacheCon 2020 - Flink SQL in 2020: Time to show off!
ApacheCon 2020 - Flink SQL in 2020: Time to show off!ApacheCon 2020 - Flink SQL in 2020: Time to show off!
ApacheCon 2020 - Flink SQL in 2020: Time to show off!
 
Julian Hyde - Streaming SQL
Julian Hyde - Streaming SQLJulian Hyde - Streaming SQL
Julian Hyde - Streaming SQL
 
Streaming SQL (at FlinkForward, Berlin, 2016/09/12)
Streaming SQL (at FlinkForward, Berlin, 2016/09/12)Streaming SQL (at FlinkForward, Berlin, 2016/09/12)
Streaming SQL (at FlinkForward, Berlin, 2016/09/12)
 
Sql 2016 - What's New
Sql 2016 - What's NewSql 2016 - What's New
Sql 2016 - What's New
 
Dev112 let's calendar that
Dev112   let's calendar thatDev112   let's calendar that
Dev112 let's calendar that
 
Back to FME School - Day 2: Your Data and FME
Back to FME School - Day 2: Your Data and FMEBack to FME School - Day 2: Your Data and FME
Back to FME School - Day 2: Your Data and FME
 
Azure stream analytics by Nico Jacobs
Azure stream analytics by Nico JacobsAzure stream analytics by Nico Jacobs
Azure stream analytics by Nico Jacobs
 
Data modeling trends for Analytics
Data modeling trends for AnalyticsData modeling trends for Analytics
Data modeling trends for Analytics
 
Temporal Tables, Transparent Archiving in DB2 for z/OS and IDAA
Temporal Tables, Transparent Archiving in DB2 for z/OS and IDAATemporal Tables, Transparent Archiving in DB2 for z/OS and IDAA
Temporal Tables, Transparent Archiving in DB2 for z/OS and IDAA
 

Recently uploaded

2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch TuesdayIvanti
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkPixlogix Infotech
 
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...panagenda
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...itnewsafrica
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI AgeCprime
 
A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxAna-Maria Mihalceanu
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...Wes McKinney
 
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...BookNet Canada
 
All These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFAll These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFMichael Gough
 
Laying the Data Foundations for Artificial Intelligence!
Laying the Data Foundations for Artificial Intelligence!Laying the Data Foundations for Artificial Intelligence!
Laying the Data Foundations for Artificial Intelligence!Memoori
 
Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Karmanjay Verma
 
Deliver Latency Free Customer Experience
Deliver Latency Free Customer ExperienceDeliver Latency Free Customer Experience
Deliver Latency Free Customer ExperienceOpsTree solutions
 
Why Agile? - A handbook behind Agile Evolution
Why Agile? - A handbook behind Agile EvolutionWhy Agile? - A handbook behind Agile Evolution
Why Agile? - A handbook behind Agile EvolutionDEEPRAJ PATHAK
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfIngrid Airi González
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra
 
Digital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentDigital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentMahmoud Rabie
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...Nikki Chapple
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integrationmarketing932765
 
QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesQCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesBernd Ruecker
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Hiroshi SHIBATA
 

Recently uploaded (20)

2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch Tuesday
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App Framework
 
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI Age
 
A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance Toolbox
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
 
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
 
All These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFAll These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDF
 
Laying the Data Foundations for Artificial Intelligence!
Laying the Data Foundations for Artificial Intelligence!Laying the Data Foundations for Artificial Intelligence!
Laying the Data Foundations for Artificial Intelligence!
 
Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#
 
Deliver Latency Free Customer Experience
Deliver Latency Free Customer ExperienceDeliver Latency Free Customer Experience
Deliver Latency Free Customer Experience
 
Why Agile? - A handbook behind Agile Evolution
Why Agile? - A handbook behind Agile EvolutionWhy Agile? - A handbook behind Agile Evolution
Why Agile? - A handbook behind Agile Evolution
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdf
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
 
Digital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentDigital Tools & AI in Career Development
Digital Tools & AI in Career Development
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
 
QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesQCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architectures
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024
 

Tracking your data across the fourth dimension

  • 1. Confoo 2015 Tracking your data across the fourth dimension Jeremy Cook
  • 2.
  • 3. This talk is about temporal databases
  • 4. –Wikipedia “A temporal database is a database with built-in support for handling data involving time…”
  • 5. – Jeff Carouth, https://twitter.com/jcarouth/status/496842218674470912 “Tonight @JCook21 explained temporal databases and I’m sure my brain is now leaking out of my nose.”
  • 6. What problem do temporal databases solve?
  • 7. Databases are good at ‘now’ ❖ Create ❖ Read ❖ Update ❖ Delete ❖ At any point we only see the current state of the data
  • 8. Databases are good at ‘now’ ❖ How many people work in each department of the company? ❖ For each product category how many products are in stock? Where is the stock located at? ❖ How many orders are currently in each fulfilment state?
  • 9. The fourth dimension ❖ Show me how salaries paid have changed by department for each quarter over the last 4 years and how they’re forecast to change next year ❖ Show me how stock levels have changed over time. How much stock are we forecast to have at any point in the future? ❖ For audit purposes show me a complete history of every change to this data, what period of time each change was valid for and when we knew about any changes
  • 13. Decision Time ❖ Records the time at which a decision was made ❖ Modelled as a single value ❖ Allows for granularity through the data type used
  • 14. Decision Time EmpId Name Hire Date Decision to Hire 1 Jeremy 2014-03-03 2014-01-20 2 Anna 2015-01-02 2013-12-15 3 Yann 2013-08-20 2013-08-20
  • 15. Valid Time “In temporal databases, valid time (VT) is the time period during which a database fact is valid in the modelled reality.” –Wikipedia
  • 16. Valid Time ❖ Modelled as a period of time between two dates ❖ Lower bound is always closed but upper bound can be open
  • 17. Valid Time EmpId Name Hire date Termination date 1 Jeremy 2014-03-03 2015-01-20 2 Anna 2015-01-02 ∞ 3 Yann 2013-08-20 2015-12-22 4 Colin 2015-05-01 ∞
  • 18. Valid Time EmpId Name Dept Hire date Term date StartVT EndVT 1 Jeremy Dev 2014-03-03 ∞ 2014-03-03 2014-07-30 1 Jeremy QA 2014-03-03 2015-01-20 2015-01-21 2015-01-20 2 Anna Dev 2015-01-02 ∞ 2015-01-02 2015-01-30 2 Anna Mgmt 2015-01-02 ∞ 2015-01-31 ∞ 3 Yann Mgmt 2013-08-20 2015-12-22 2013-08-20 ∞ 4 Colin Dev 2015-05-01 ∞ 2015-05-01 ∞
  • 20. Valid-time on its own may not be enough! Name Type StartVT EndVT Saturn Planet Billions of years ago ∞ Pluto Planet Billions of years ago ∞
  • 21. Valid-time on its own may not be enough! Name Type StartVT EndVT Saturn Planet Billions of years ago ∞ Pluto Dwarf planet Billions of years ago ∞
  • 22. Valid-time on its own may not be enough! Name Type StartVT EndVT Saturn Planet Billions of years ago ∞ Pluto Plutoid Billions of years ago ∞
  • 23. Valid-time on its own may not be enough! Name Type StartVT EndVT Saturn Planet Billions of years ago ∞ Pluto Planet Billions of years ago 2006 Pluto Dwarf planet 2006 2008 Pluto Plutoid 2008 ∞
  • 24. Transaction Time “In temporal databases, transaction time (TT) is the time period during which a fact stored in the database is considered to be true.” –Wikipedia
  • 25. Transaction Time ❖ Modelled as a period of time between two dates ❖ Lower bound is always closed but upper bound can be open
  • 26. Transaction Time Name Type StartVT EndVT StartTT EndTT Pluto Planet Billions of years ago ∞ 1930 2006 Pluto Dwarf planet Billions of years ago ∞ 2006 2008 Pluto Plutoid Billions of years ago ∞ 2008 ∞
  • 27. Valid Time != Transaction Time Name Clothing StartVT EndVT StartTT EndTT Father Christmas null A long time ago ∞ 1973 1975 Santa Claus red A long time ago ∞ 1975 1980 Saint Nicholas red 270 AD ∞ 1980 1982
  • 28. How many temporal aspects should you use? ❖ As many or few as your application needs! ❖ Tables that implement two aspects are bi-temporal ❖ You can implement more aspects, in which case you have multi temporal tables
  • 29. Is your head spinning? ❖ Decision time records when a decision was taken ❖ Valid Time records the period of time for which the fact is valid ❖ Transaction Time records the period of time for which the fact is considered to be true
  • 31. A note on the example tables CREATE TABLE dept ( DNo INTEGER, DName VARCHAR(255) ); CREATE TABLE emp ( ENo INTEGER, EName VARCHAR(255), EDept INTEGER );
  • 32. Periods ❖ Table component, capturing a pair of columns defining a start and end date ❖ Not a new data type, but metadata about columns in the table ❖ Closed-open constraint ❖ Enforces that end time > start time
  • 33. Valid time ❖ Also called application time in SQL:2011 ❖ Modelled as a pair of date time columns with a period ❖ Name of the columns and period is up to you
  • 34. Valid time ALTER TABLE emp ADD ( EStart DATE, EEnd DATE, PERIOD FOR EPeriod (EStart, EEnd) );
  • 35. Temporal primary keys ❖ SQL:2011 allows a valid time period to be named as part of a primary key ❖ Can also enforce that the valid time periods do not overlap
  • 36. Temporal primary keys ALTER TABLE emp ADD PRIMARY KEY (ENo, EPeriod);
  • 37. Temporal primary keys ALTER TABLE emp ADD PRIMARY KEY (ENo, EPeriod WITHOUT OVERLAPS);
  • 38. Temporal foreign keys ❖ What happens if a parent and child table both define valid time periods? ❖ It doesn’t make sense to allow a row in a child table to reference a row in a parent table where the valid time does not overlap ❖ SQL:2011 allows valid time periods to be part of foreign key constraints
  • 39. Temporal foreign keys ALTER TABLE dept ADD ( DStart DATE, DEnd DATE, PERIOD FOR DPeriod (DStart, DEnd) ); ALTER TABLE emp ADD FOREIGN KEY (Edept, EPeriod) REFERENCES dept (DNo, PERIOD DPeriod);
  • 40. Querying valid time tables ❖ Can query against valid time columns as normal - they’re just normal table columns ❖ Updates and deletes can be performed for a period of a valid time time period
  • 41. Querying valid time tables ❖ SQL:2011 allows you to create periods to use in your queries and use new predicates: ❖ CONTAINS ❖ OVERLAPS ❖ EQUALS ❖ PRECEDES ❖ SUCCEEDS ❖ IMMEDIATELY SUCCEEDS and IMMEDIATELY PRECEDES
  • 42. Querying valid time tables UPDATE Emp FOR PORTION OF EPeriod FROM DATE '2011-02-03' TO DATE '2011-09-10' SET EDept = 4 WHERE ENo = 22217;
  • 43. Querying valid time tables DELETE Emp FOR PORTION OF EPeriod FROM DATE '2011-02-03' TO DATE '2011-09-10' WHERE ENo = 22217;
  • 44. Querying valid time tables SELECT EName, Edept FROM Emp WHERE ENo = 22217 AND EPeriod CONTAINS DATE '2015-01-23';
  • 45. Querying valid time tables SELECT EName, Edept FROM Emp WHERE ENo = 31 AND EPeriod OVERLAPS PERIOD (DATE '2015-01-01', DATE '2015-01-31');
  • 46. Transaction time ❖ Also known as system time in SQL:2011 ❖ Modelled as two DATE or TIMESTAMP columns ❖ Management of the columns for the period is handled by the database for you
  • 47. Transaction time ❖ When data is inserted: ❖ Start of transaction time is set to current time
  • 48. Transaction time ❖ When data is updated: ❖ Transaction time end is set to current time on the existing row ❖ A new row is added with the updated date and a transaction time start of the current time
  • 49. Transaction time ❖ When data is deleted: ❖ Transaction time end is set to current time in the existing row
  • 50. Transaction time ❖ Because the system manages transaction time: ❖ Not possible to alter transaction time values in the past ❖ Not possible to add future dated transaction time values ❖ Referential constraints on historical data are never checked
  • 51. Transaction time CREATE TABLE emp ( …, Sys_start TIMESTAMP(12) GENERATED ALWAYS AS ROW START, Sys_end TIMESTAMP(12) GENERATED ALWAYS AS ROW END, PERIOD FOR SYSTEM_TIME (Sys_start, Sys_end) ) WITH SYSTEM VERSIONING;
  • 52. Querying transaction time tables ❖ New predicates to be used with transaction time: ❖ FOR SYSTEM_TIME AS OF ❖ FOR SYSTEM_TIME FROM ❖ FOR SYSTEM_TIME BETWEEN ❖ If none of the above supplied the database should only return rows for the current system time
  • 53. Querying transaction time tables SELECT ENo, EName FROM emp WHERE Eno = 22;
  • 54. Querying transaction time tables SELECT ENo, EName FROM emp WHERE ENo = 22 FOR SYSTEM_TIME AS OF TIMESTAMP '2015-01-28 12:45:00';
  • 55. Querying transaction time tables SELECT ENo, EName FROM emp WHERE ENo = 22 AND EPeriod CONTAINS DATE '2014-08-27' FOR SYSTEM_TIME AS OF TIMESTAMP '2015-01-28 12:45:00';
  • 56. Grey areas/not implemented yet ❖ Evolving schema over time ❖ Support for period joins ❖ Support for period aggregates or period grouped queries ❖ Support for period normalization ❖ Support for multiple valid time periods per table
  • 57. Which vendors support SQL: 2011?
  • 58. Current support ❖ Oracle 12c ❖ SQL:2011 compliant but not even nearly complete ❖ PostgreSQL ❖ 9.1 and earlier: temporal contributed package ❖ 9.2 native ranged data types ❖ IBM DB2 through ‘time travel query’ feature ❖ Teradata 13.10 and 14 ❖ Handful of others implemented as extensions
  • 59. How do I add this stuff to my current schema?
  • 60. Implementing valid time ❖ Add a pair of date time columns to your table for the valid time period. ❖ Can make these part of your primary key
  • 61. Implementing valid time ❖ Things to consider: ❖ Have to check for end time > start time ❖ Have to check for overlaps in valid time periods ❖ Temporal foreign keys have to be implemented yourself ❖ Queries become potentially more complex
  • 62. Implementing transaction time ❖ Add a column recording transaction time start to your table ❖ For each table create a backup table mirroring the columns in the main table, adding a transaction time end column too ❖ Create a trigger that fires on each update or delete to copy old values from the main table to the backup table ❖ Should add transaction time end to the backup table ❖ Should also update the transaction time start to now in the main table if the operation is an update
  • 63. Implementing transaction time ❖ Things to consider: ❖ Extra complexity ❖ How long should backup data be kept for? ❖ Do you optimize for fast reads or writes? ❖ Should truncating the main table delete the data from the backup?
  • 64. More information ❖ Wikipedia article on Temporal Databases ❖ Temporal features in SQL:2011 (PDF) ❖ Time and Relational Theory
  • 65. Thanks for listening! ❖ Any questions? ❖ I’d love some feedback ❖ https://joind.in/talk/view/13294 ❖ Contact me: ❖ @JCook21 ❖ jeremycook0@icloud.com