Introduction to Data Warehousing
Data Warehouse





Maintain historic data
Analysis to get better understanding of business
Better Decision making
Def...
Subject Oriented
•
•
•

Data warehouse is organized around subjects such as
sales, product, customer.
It focuses on modeli...
Integrated
•
•

Data Warehouse is constructed by integrating multiple
heterogeneous sources.
Data Preprocessing are applie...
Non-volatile
•

Mostly, data once recorded will not be updated.
• Data warehouse requires two operations in data accessing...
Time Variant
•
•

Provides information from historical perspective e.g. past 510 years
Every key structure contains either...
Why Data Warehouse?
Problem Statement:
• ABC Pvt Ltd is a company with branches at Mumbai, Delhi,
Chennai and Bangalore.
•...
Why Data Warehouse?
Mumbai

Delhi

Get quarterly sales figure
for each branch
and manually calculate
sales figure across b...
Why Data Warehouse?
•
•

Solution:
Extract sales information from each database.
Store the information in a common reposit...
Why Data Warehouse?
Mumbai

Delhi
Data
Warehouse
Chennai

Banglore

Query &
Analysis tools

Sales
Manager
Characteristics of Data Warehouse









Relational / Multidimensional database
Query and Analysis rather than tr...
When we say large - we mean it!
• Terabytes -- 10^12 bytes:

Yahoo! – 300 Terabytes and
growing

• Petabytes -- 10^15 byte...
OLTP Vs Data Warehouse (OLAP)
OLTP

Data Warehouse (OLAP)

Indexes

Few

Many

Data

Normalized

Generally De-normalized

...
Data Warehouse Architecture
Operational
System

Sales
Data Mart
Analysis

Operational
System
ETL
(Extract
Transform
and Lo...
ETL
ETL stands for Extract, Transform and
Load






Data is distributed across different sources
– Flat files, Streami...
ETL Functions






Extract
– Collect data from different sources
– Parse data
– Remove unwanted data
Transform
– Proje...
ETL Steps
•

The first step in ETL process is mapping the data between
source systems and target database.
• The second st...
ETL Glossary
Mapping:
Defining relationship between source and target objects.
Cleansing:
The process of resolving inconsi...
Dimension






Categorizes the data. For example - time, location, etc.
A dimension can have one or more attributes. F...
Types of dimensions
•

•

•

•

•

Conformed Dimension - A dimension that is shared across
fact tables.
Junk Dimension - A...
Types of Slowly Changing Dimension
•

•

•

•

Type1 - The Type 1 methodology overwrites old data with
new data, and there...
Facts







Facts are values that can be examined and analyzed.
For Example - Page Views, Unique Users, Pieces
Sold, ...
How to store data?
Facts and Dimensions:
1. Select the business process to model
2. Declare the grain of the business proc...
Dimension Table





Contains attributes of dimensions e.g. Month is an attribute
of Time dimension.
Can also have fore...
Fact Table




Contains Facts
Foreign keys to dimension tables
Primary Key: usually composite key of all FKs
Types of schema used in data
warehouse
Star Schema





Snowflake Schema
Fact Constellation Schema
Star Schema





Multi-dimensional Data
Dimension and Fact Tables
A fact table with pointers to Dimension tables
Star Schema
Snowflake Schema




An extension of star schema in which the dimension tables
are partly or fully normalized.
Dimension...
Snowflake Schema
Fact Constellation Schema
•
•



A fact constellation schema allows dimension tables to be
shared between fact tables.
Th...
Fact Constellation Schema

In this example, the dimensions tables for time, item, and location are
shared between both the...
Operations on Data Warehouse






Drill Down
Roll up
Slice & Dice
Pivoting
Drill Down
Product
Category e.g Home Appliances
Sub Category e.g Kitchen Appliances
Product e.g Toaster

Time
Roll Up
Year

Fiscal Year

Quarter

Fiscal Quarter

Month

Fiscal Month

Fiscal Week

Day
Slice & Dice
Product = Toaster
Product

Time
Time
Pivoting
Product

Product

Time

•
•
•

Also called rotation
Rotate on an axis
Interchange Rows and Columns

Region
Advantages of Data Warehouse
•
•
•
•
•

One consistent data store for reporting, forecasting, and
analysis
Easier and time...
Disadvantages of Data Warehouse
•

Preparation may be time consuming.
• High associated cost
Case Study: Why Data Warehouse
•

•

G2G Courier Pvt. Ltd. is an established brand in courier
industry which has its own n...
Challenges
Apart from G2G’s own transaction system, each partner has
their own system which make the data very heterogeneo...
“Looks like we are doing good in
South, is there any scope of further
improvement???”

“We are getting lot of complaints
f...
Sales Information
Report: Revenue by region
Region

Revenue (lacs)

% Change

South

41

+ 8.1

North

34

+ 5.2

East

25...
Case Study: Data Warehouse Design
•
•
•

ABC Pvt Ltd is a new company which produces stationary
products with production u...
Sales Information
Report: The number of units sold.
113

Report: The number of units sold over time
January

February

Mar...
Sales Information
Report: The number of items sold in each City for each product with time
City

Item

Delhi

Jan

Feb Mar...
Identify sales Facts & Dimensions


Facts – Units sold



Dimensions – Product, Time, Region.



Fact Table
City_ID Pro...
Identify sales Facts & Dimensions
Region Dimension Table

City_ID

City

Region

Country

1

Delhi

North

India

2

Banga...
Data warehouse model
Time

Product

Sales Fact

Region

Product
Category
Thank You
Upcoming SlideShare
Loading in …5
×

Introduction to Data Warehousing

1,140 views

Published on

Published in: Education, Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,140
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
70
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • CRMERP change to something simple
  • Introduction to Data Warehousing

    1. 1. Introduction to Data Warehousing
    2. 2. Data Warehouse     Maintain historic data Analysis to get better understanding of business Better Decision making Definition: A data warehouse is a  subject-oriented  integrated  time-varying  non-volatile collection of data that is used primarily in organizational decision making. -- Bill Inmon, Building the Data Warehouse 1996
    3. 3. Subject Oriented • • • Data warehouse is organized around subjects such as sales, product, customer. It focuses on modeling and analysis of data for decision makers. Excludes data not useful in decision support process.
    4. 4. Integrated • • Data Warehouse is constructed by integrating multiple heterogeneous sources. Data Preprocessing are applied to ensure consistency. RDBMS Data Processing Data Transformation Legacy System Flat File Data Warehouse Data Processing Data Transformation
    5. 5. Non-volatile • Mostly, data once recorded will not be updated. • Data warehouse requires two operations in data accessing - Incremental loading of data - Access of data load access
    6. 6. Time Variant • • Provides information from historical perspective e.g. past 510 years Every key structure contains either implicitly or explicitly an element of time
    7. 7. Why Data Warehouse? Problem Statement: • ABC Pvt Ltd is a company with branches at Mumbai, Delhi, Chennai and Bangalore. • The Sales Manager wants quarterly sales report across the branches. • Each branch has a separate operational system where sales transactions are recorded.
    8. 8. Why Data Warehouse? Mumbai Delhi Get quarterly sales figure for each branch and manually calculate sales figure across branches. Sales Manager Chennai Banglore What if he need daily sales report across the branches?
    9. 9. Why Data Warehouse? • • Solution: Extract sales information from each database. Store the information in a common repository at a single site.
    10. 10. Why Data Warehouse? Mumbai Delhi Data Warehouse Chennai Banglore Query & Analysis tools Sales Manager
    11. 11. Characteristics of Data Warehouse        Relational / Multidimensional database Query and Analysis rather than transaction Historical data from transactions Consolidates Multiple data sources Separates query load from transactions Mostly non volatile Large amount of data in order of TBs
    12. 12. When we say large - we mean it! • Terabytes -- 10^12 bytes: Yahoo! – 300 Terabytes and growing • Petabytes -- 10^15 bytes: Geographic Information Systems • Exabytes -- 10^18 bytes: National Medical Records • Zettabytes -- 10^21 bytes: Weather images • Zottabytes -- 10^24 bytes: Intelligence Agency Videos
    13. 13. OLTP Vs Data Warehouse (OLAP) OLTP Data Warehouse (OLAP) Indexes Few Many Data Normalized Generally De-normalized Joins Many Some Derived data and aggregates Rare Common
    14. 14. Data Warehouse Architecture Operational System Sales Data Mart Analysis Operational System ETL (Extract Transform and Load) Data Warehouse Generic Data Mart Flat Files Flat Files Data Mining Inventory Data Mart Reporting
    15. 15. ETL ETL stands for Extract, Transform and Load    Data is distributed across different sources – Flat files, Streaming Data, DB Systems, XML, JSON Data can be in different format – CSV, Key Value Pairs Different units and representation – Country: IN or India – Date: 20 Nov 2010 or 20101020
    16. 16. ETL Functions    Extract – Collect data from different sources – Parse data – Remove unwanted data Transform – Project – Generate Surrogate keys – Encode data – Join data from different sources – Aggregate Load
    17. 17. ETL Steps • The first step in ETL process is mapping the data between source systems and target database. • The second step is cleansing of source data in staging area. • The third step is transforming cleansed source data. • Fourth step is loading into the target system.  Data before ETL Processing:  Data after ETL Processing:
    18. 18. ETL Glossary Mapping: Defining relationship between source and target objects. Cleansing: The process of resolving inconsistencies in source data. Transformation: The process of manipulating data. Any manipulation beyond copying is a transformation. Examples include aggregating, and integrating data from multiple sources. Staging Area: A place where data is processed before entering the warehouse.
    19. 19. Dimension    Categorizes the data. For example - time, location, etc. A dimension can have one or more attributes. For example - day, week and month are attributes of time dimension. Role of dimensions in data warehousing. - Slice and dice - Filter by dimensions
    20. 20. Types of dimensions • • • • • Conformed Dimension - A dimension that is shared across fact tables. Junk Dimension - A junk dimension is a convenient grouping of flags and indicators. For example, payment method, shipping method. De-generated Dimension - A dimension key, that has no attributes and hence does not have its own dimension table. For example, transaction number, invoice number. Value of these dimension is mostly unique within a fact table. Role Playing Dimensions - Role Playing dimension refers to a dimension that play different roles in fact tables depending on the context. For example, the Date dimension can be used for the ordered date, shipment date, and invoice date. Slowly Changing Dimensions - Dimensions that have data
    21. 21. Types of Slowly Changing Dimension • • • • Type1 - The Type 1 methodology overwrites old data with new data, and therefore does not track historical data at all. Type 2 - The Type 2 method tracks historical data by creating multiple records for a given value in dimension table with separate surrogate keys. Type 3 - The Type 3 method tracks changes using separate columns. Whereas Type 2 had unlimited history preservation, Type 3 has limited history preservation, as it's limited to the number of columns we designate for storing historical data. Type 4 - The Type 4 method is usually referred to as using "history tables", where one table keeps the current data, and an additional table is used to keep a record of all changes. Type 1, 2 and 3 are commonly used.
    22. 22. Facts     Facts are values that can be examined and analyzed. For Example - Page Views, Unique Users, Pieces Sold, Profit. Fact and measure are synonymous. Types of facts: – Additive - Measures that can be added across all dimensions. – Non Additive - Measures that cannot be added across all dimensions. – Semi Additive - Measures that can be added across few dimensions and not with others.
    23. 23. How to store data? Facts and Dimensions: 1. Select the business process to model 2. Declare the grain of the business process 3. Choose the dimensions that apply to each fact table row 4. Identify the numeric facts that will populate each fact table row
    24. 24. Dimension Table    Contains attributes of dimensions e.g. Month is an attribute of Time dimension. Can also have foreign keys to another dimension table Usually identified by a unique integer primary key called surrogate key
    25. 25. Fact Table    Contains Facts Foreign keys to dimension tables Primary Key: usually composite key of all FKs
    26. 26. Types of schema used in data warehouse Star Schema    Snowflake Schema Fact Constellation Schema
    27. 27. Star Schema    Multi-dimensional Data Dimension and Fact Tables A fact table with pointers to Dimension tables
    28. 28. Star Schema
    29. 29. Snowflake Schema   An extension of star schema in which the dimension tables are partly or fully normalized. Dimension table hierarchies broken down into simpler tables.
    30. 30. Snowflake Schema
    31. 31. Fact Constellation Schema • •  A fact constellation schema allows dimension tables to be shared between fact tables. This Schema is used mainly for the aggregate fact tables, OR where we want to split a fact table for better comprehension. For example, a separate fact table for daily, weekly and monthly reporting requirement.
    32. 32. Fact Constellation Schema In this example, the dimensions tables for time, item, and location are shared between both the sales and shipping fact tables.
    33. 33. Operations on Data Warehouse     Drill Down Roll up Slice & Dice Pivoting
    34. 34. Drill Down Product Category e.g Home Appliances Sub Category e.g Kitchen Appliances Product e.g Toaster Time
    35. 35. Roll Up Year Fiscal Year Quarter Fiscal Quarter Month Fiscal Month Fiscal Week Day
    36. 36. Slice & Dice Product = Toaster Product Time Time
    37. 37. Pivoting Product Product Time • • • Also called rotation Rotate on an axis Interchange Rows and Columns Region
    38. 38. Advantages of Data Warehouse • • • • • One consistent data store for reporting, forecasting, and analysis Easier and timely access to data Scalability Trend analysis and detection Drill down analysis
    39. 39. Disadvantages of Data Warehouse • Preparation may be time consuming. • High associated cost
    40. 40. Case Study: Why Data Warehouse • • G2G Courier Pvt. Ltd. is an established brand in courier industry which has its own network in main cities and also have sub contracted in rural areas across the country to various partners. The President of the company wants to look deep into the financial health of the company and different performance aspects.
    41. 41. Challenges Apart from G2G’s own transaction system, each partner has their own system which make the data very heterogeneous. • Granularity of data in various systems is also different. For eg: minute accuracy and day accuracy. • To do analysis on metrics like Revenue and Timely delivery across various geographical locations and partner, we need to have a unified system. •
    42. 42. “Looks like we are doing good in South, is there any scope of further improvement???” “We are getting lot of complaints from the East, who exactly is the black sheep???”
    43. 43. Sales Information Report: Revenue by region Region Revenue (lacs) % Change South 41 + 8.1 North 34 + 5.2 East 25 - 6.8 West 12 + 2.7 Report: Performance by partner Partner On Time Delivery Rate No. of complaints A 100 % 0 B 98 % 90 C 60 % 521
    44. 44. Case Study: Data Warehouse Design • • • ABC Pvt Ltd is a new company which produces stationary products with production unit located at Ludhiana. They have sales units at Delhi, Bangalore. The President of the company wants sales information.
    45. 45. Sales Information Report: The number of units sold. 113 Report: The number of units sold over time January February March April 14 41 33 25 Report : The number of items sold for each product with time Jan Feb Apr 6 Black Cartridge Mar 17 8 Long notebook 6 16 6 Short notebook 8 25 21 Product
    46. 46. Sales Information Report: The number of items sold in each City for each product with time City Item Delhi Jan Feb Mar Apr Black Cartridge 3 16 6 Short Notebook 4 16 6 Bangalore Black Cartridge 3 Time Long Notebook 3 10 7 Long Notebook 3 8 Short Notebook 4 9 Product 15 City Item Jan Feb Mar Apr Delhi General Stationary 7 12 Ink & Toners Bangalore General Stationary Ink & Toners 3 7 9 10 15 8 3 7 Time 32 Product Category
    47. 47. Identify sales Facts & Dimensions  Facts – Units sold  Dimensions – Product, Time, Region.  Fact Table City_ID Prod_ID 1 589 1 3 1 1218 1 4 2 589 1 3 2 1218 1 4 1  Time_Id Units 589 2 16 Time dimension table Time_Id Month 1 January 2012 2 February 2012
    48. 48. Identify sales Facts & Dimensions Region Dimension Table City_ID City Region Country 1 Delhi North India 2 Bangalore South India Product Dimension Tables Prod_ID Product_Name Product_Category_ID 589 Black Cartridge 2 590 Long Notebook 1 288 Short Notebook 1 Product_Category_ID Product_Category 1 General Stationary 2 Ink & Toners
    49. 49. Data warehouse model Time Product Sales Fact Region Product Category
    50. 50. Thank You

    ×