Recommended
More Related Content
Similar to Enterprise Data Warehouse
Similar to Enterprise Data Warehouse (20)
Recently uploaded
Recently uploaded (8)
Enterprise Data Warehouse
- 1. Enterprise Data Warehouse Table of Contents Enterprise Data Warehouse........................................................................................................................1 Architecture:.............................................................................................................................................2 Data Models: ............................................................................................................................................3 Conceptual Data Model .......................................................................................................................3 Logical Data Model...............................................................................................................................3 Physical Data Model.............................................................................................................................4 Production and Staging Databases:.........................................................................................................9 Fact & Dimension tables in DWProd and Staging tables in DWStage ................................................9 Referential Integrity Constraints: ............................................................................................................9 Primary and Foreign Keys ....................................................................................................................9 Indexes:...................................................................................................................................................10 Data Mapping.........................................................................................................................................10 ETL Process: ............................................................................................................................................13 Data Extraction from Text Files and Loading Staging Tables............................................................13 Data Extraction from one Text File and Loading Item Master Staging Table...................................13 Data Extraction from all Text File and Loading Order Item Staging Table .......................................13 SSIS Project Parameters.....................................................................................................................13 SSIS Package Variables.......................................................................................................................14 Example Source of Text Files for One Store ......................................................................................14 Extrating Data from Staging Tables → Transformoing data to Join tables → Loading Dimension.15 Natural Keys Algorithm......................................................................................................................16 Derived Column Transformation to generate Key Values ................................................................19 Data Flow: Efficient Data Loading to Fact tables without LookUp transformation. Error! Bookmark not defined. Loading Fact tables.............................................................................................................................19 Control Flow: Loading FactOrderSummary table..............................................................................20 Data Flow: Loading FactOrderSummary table ..................................................................................20 Data Warehouse Size after Initial/Full Load......................................................................................21
- 4. Data Models: Conceptual Data Model Logical Data Model
- 5. Physical Data Model OrderDetailFact OrderDetailKey bigint OrderSummaryKey bigint StoreKey int RegisterKey int ItemKey int OrderTypeKey int DiscountKey int DateKey int OrderDateKey int OrderTimeKey int SendDateKey int SendTimeKey int CustomerKey int EmployeeKey int OrderNumber int SerialNumber int Quantity smallint Weight float Price money ExtPrice money PrepCost money CarryOutFee money Tax money OrigPrice money Surcharge money TipsAmount money TippableSales money ComboSavings money OrderDiscount money DiscountAmount money RecordCreateDate smalldatetime Column Name Data Type OrderDiscountFact OrderDiscountKey bigint OrderSummaryKey bigint StoreKey int DateKey int OrderTypeKey int DiscountKey int OwnerKey int RegisterKey int SerialNumber int TransactionDateKey int TransactionTimeKey int OrderNumber int DiscountCount int DiscountAmount money RecordCreateDate smalldatetime Column Name Data Type OrderPaymentFact PaymentKey int StoreKey int DateKey int PaymentTypeKey int OrderSummaryKey bigint RegisterKey int EmployeeKey int CustomerKey int OriginalDateKey int OriginalTimeKey int OrderNumber int SerialNumber smallint CustomerName varchar(40) CardAccount varchar(255) AuthCode varchar(50) CardExpDate datetime Amount money TipsAmount money AmountUsed money SurchargeAmount money CashBack money RecordCreateDate smalldateti... Column Name Data Type OrderSummaryFact OrderSummaryKey bigint StoreKey int DateKey int OrderTypeKey int EmployeeKey int OwnerKey int CashierKey int CustomerKey int RegisterKey int CashRegisterKey int KitchenSentDateKey int KitchenSentTimeKey int PickupDateKey int PickupTimeKey int FirstDateKey int FirstTimeKey int SaveDateKey int SaveTimeKey int OrderNumber int ItemCount int CustomerCount int SubTotal money Tax money NetTotal money NetExemptTotal money DeliveryFee money OrderDiscountTotal money LineDiscountTotal money CreditTips money CashTips money TipsPaid money ItemRefunds money ItemRefundTax money SplitFrom int CombinedTo int RecordCreateDate smalldatetime Column Name Data Type
- 10. Production and Staging Databases: Fact & Dimension tables in DWProd and Staging tables in DWStage Referential Integrity Constraints: Primary and Foreign Keys
- 12. Data Cleansing
- 13. Data Quality Issues: (Reused Codes)
- 14. ETL Process: Data Extraction from Text Files and Loading Staging Tables Control Flow for one Text File to Load Item Master Staging Table Control Flow for all Text Files to Load Order Item Staging Table SSIS Project Parameters
- 15. SSIS Package Variables Example Source of Text Files for One Store
- 16. Extracting Data from Staging Tables → Transforming data to Join tables → Loading Dimension Extracting Data from Staging Tables → Transforming data to generate Key → Loading Dimension
- 17. Loading Dimension → LookUp Transformation → No Match Output Loading Dimension → LookUp Transformation → EmployeeKey Lookup
- 18. Loading Dimension → OLEDB Command → SQL Command Loading Dimension → OLEDB Command → Column Mapping
- 20. Natural Keys Algorithm Derived Column Transformation to generate Key Values Loading Fact tables
- 21. Control Flow: Loading FactOrderSummary table SQL Merge Statement for (EST) Delete Existing Matching Target Records MERGE [DWProd].[dbo].[FactOrderSummary] TT USING [DWStage].[dbo].[StgTblOrderFull] ST ON TT.[StoreKey] = ST.[StoreKey] AND TT.[DateKey] = ST.[DateKey] WHEN MATCHED THEN DELETE; Data Flow: Efficient Data Loading to FactOrderSummary table
- 22. Data Warehouse Size after Initial/Full Load