This document discusses handling late arriving fact data in big data warehouses. It provides an example of receiving old telephone call records months after the actual calls occurred. It then outlines steps to correctly insert the late data into the appropriate historical partitions, including: 1) Creating a partitioned output table 2) Inserting data into a temporary table with additional columns 3) Dropping existing partitions prior to the late data date 4) Inserting from the temporary table into the partitioned output table based on the new date column.

1. Late Arrival Facts in BIG DATA
Hive/Impala
1

2. Late Arriving Fact scenario occurs when the transaction or fact data comes to data warehouse way later than the
actual transaction occurred in the source application.
In facts data scenario, actual fact data created prior & sent later from source application to warehouse cause late arrival
facts, the situation become messy because we have to search back in history within the dimensions to decide how to assign
the right dimension keys that were in effect when the activity occurred in the past. It is important to be conceptually clear
upon the nature of business process & the source application behavior.
Let's have a practical example for further illustration for Telecom Domain:
Using telecommunication domain, we receive the conversation record (voice data) today that is several months old. We
suppose to insert this old data into its correct historical position, even though our total records summary for the prior month is
now changed. Elaboration, here you go:
Insert the late-arriving fact record into the correct physical partition of the database containing the other fact records from the
time of the late-arriving data. An interesting compromise we have used in this situation is to carry two sets of data
dimensions that is partitioning datasets. on voice data. One refers to the actual data record, and the other refers to the data
receiving date. Now you can roll up the data by applying a particular logic, as all fact records in a time span occupy the same
physical partition if you declare the physical partitioning of the fact table to be based on the date dimension. Since you
should be using surrogate keys for the date dimension, surrogate keys of a dimension should be assigned in a particular
logical order.
2By Punya Sloka Muduli

3. How to handle ?
Partition Table: (Output Table)
CREATE TABLE poc.master_part (cellno
BIGINT,name String,duration INT,exedate String)
COMMENT 'PARTITIONED OUTPUT Table'
PARTITIONED BY (calldate String) LOCATION
'/user/sps/taspsdal-hadoop/punya/poc/hive/lafact/part';
Following are the steps recommended by Punyasloka Muduli.
3By Punya Sloka Muduli

4. Temporary Table:
DROP TABLE poc.master_temp;
CREATE TABLE poc.master_temp (cellno BIGINT,name String,duration INT,calldate String,exedate BIGINT,rank INT)
COMMENT 'Master Table' LOCATION '/user/sps/taspsdal-hadoop/punya/poc/hive/lafact/master_temp/';
Insert OVERWRITE TABLE poc.master_temp select cellno,name,duration,calldate,exedate,row_number() over (partition
by cellno order by name,duration,calldate,exedate) rn from poc.master_part;
4By Punya Sloka Muduli

5. Drop All Partition:
ALTER TABLE poc.master_part drop if exists partition(calldate<='19-11-2017');
Insert to Partitioned Output Table:
Insert OVERWRITE TABLE poc.master_part PARTITION(calldate) select cellno,name,max(case when rank >= 1 then
duration end) duration1, max(case when rank >= 1 then exedate end) exedate1, max(case when rank >= 1 then calldate
end)calldate1 from poc.master_temp group by cellno,name;
5By Punya Sloka Muduli

6. Thank You
6
Punya Sloka Muduli
+91-9560605432
punyaslokamuduli@gmail.com