DATASTAGE AND QUALITY STAGE 9.1 ONLINE TRAINING

®
IBM Software Group
© 2007 IBM Corporation

IBM Software Group | WebSphere software
3
06/15/14 TCS Confidential 3

4
Course Roadmap
• Why we use Data warehousing
• Difference between Operational System and Data Warehouse
• Introduction to Data warehousing
• Data Warehousing Approaches
• Data Warehouse Technical Architecture
• Data Modelling concepts
• Operational Data Store
• Schema Design of Data warehouse
• Data Acquisation
• ETL Products
• Project Life Cycle

5
Why We Need Data Warehousing ?
 Better business intelligence for end-users
 Reduction in time to locate, access, and analyze information
 Consolidation of disparate information sources
 To Store Large Volumes of Historical Detail Data from Mission Critical Applications
 Strategic advantage over competitors
 Faster time-to-market for products and services
 Replacement of older, less-responsive decision support systems
 Reduction in demand on IS to generate reports

6
What is an Operational System?
 Operational systems are just what their name implies; they are the systems that
help us run the day-to-day enterprise operations.
 These are the backbone systems of any enterprise, such as order entry inventory
etc.
 The classic examples are airline reservations, credit-card authorizations, and ATM
withdrawals etc.,

7
Characteristics of Operational Systems
• Continuous availability
• Predefined access paths
• Transaction integrity
• Volume of transaction - High
• Data volume per query - Low
• Used by operational staff
• Supports day to day control operations
• Large number of users

8
OLTP Vs Data Warehouse
Operational System Data Warehouse
Transaction Processing Query Processing
Predictable CPU Usage Random CPU Usage
Time Sensitive History Oriented
Operator View Managerial View
Normalized Efficient
Design for TP
Denormalized Design for
Query Processing

9
OLTP Vs Warehouse
Designed for Atmocity,
Consistency, Isolation and
Durability
Designed for quite or static
database
Organized by transactions
(Order, Input, Inventory)
Organized by subject
(Customer, Product)
Relatively smaller database Large database size
Many concurrent users Relatively few concurrent
users
Volatile Data Non Volatile Data

10
Stores all data Stores relevant data
Performance Sensitive Less Sensitive to performance
Not Flexible Flexible
Efficiency Effectiveness

11
What is a Data Warehouse ?
 Data WarehouseData Warehouse is a
 Subject-Oriented
 Integrated
 Time-Variant
 Non-volatile
WH Inmon - Regarded As Father Of Data WarehousingWH Inmon - Regarded As Father Of Data Warehousing

13
13
Subject Oriented Analysis
Data Warehouse StorageTransactional Storage
SalesSales
CustomersCustomers
ProductsProducts
Entry
Sales Rep
Quantity Sold
Part Number
Date
Customer Name
Product Description
Unit Price
Mail Address
Process Oriented Subject Oriented

14
14
Integration of Data
Appl. A - M, F
Appl. B - 1, 0
Appl. C - X, Y
Appl. A - pipeline cm.
Appl. B - pipeline inches
Appl. C - pipeline mcf
Appl. A - balance dec(13,2)
Appl. B - balance PIC 9(9)V99
Appl. C - balance float
Appl. A - bal-on-hand
Appl. B - current_balance
Appl. C - balance
Appl. A - date (Julian)
Appl. B - date (yymmdd)
Appl. C - date (absolute)
M, F
pipeline cm
balance dec(13, 2)
balance
date (Julian)
Integration
Encoding
Unit of
Attributes
Physical
Attributes
Naming
Conventions
Data
Consistency

15
15
Load
Access
Mass Load / Access of DataRecord-by-Record Data Manipulation
Insert
Access
Insert
Change
Delete
Change
Volatile Non-Volatile
Volatility of Data

16
16
Time Variant Data Analysis
Current Data Historical Data
0
5
10
15
20
Sales ( in lakhs
)
January February March
Year97
Sales ( Region , Year - Year 97 - 1st Qtr)
East
West
North

Load/
Update
Consistent Points in Time
Updated constantly
Data changes according to
need, not a fixed schedule
Added to regularly, but loaded data
is rarely directly changed
Does NOT mean the Data
warehouse is never updated or
never changes!!
Constant Change
Operational systems
Database
Data warehouse
Datawarehouse- Differences from Operational
Systems
Insert
Insert
Update
Initial Load
Incremental Load
Incremental Load
Update
Delete

18
Difference B/W OLTP AND OLAP

19
DW Implementation Approaches
 Top Down
 Bottom-up
 Combination of both
 Choices depend on:
current infrastructure
resources
architecture
ROI
Implementation speed

20
Heterogeneous Source Systems
Staging
Common Staging interface Layer
EDW- “Top Down”Approach
Data mart bus architecture Layer
Enterprise Datawarehouse
Source
1
Source
2
Source
3
Incremental Architected data marts
DM 1 DM 3DM 2

21
Heterogeneous Source Systems
Staging
Common Staging interface Layer
EDW- “Bottom up”Approach
Data mart bus architecture Layer
Source
1
Source
2
Source
3
Incremental Architected data marts
DM 1 DM 3DM 2
Enterprise Datawarehouse

22
Source System Data Staging Area Presentation Area
Services:
Transform from
source-to-Target
Maintain Conform
Dimensions
No user query
support
Data Store:
Flat files or
relational tables
Design Goals:
Staging
Throughput
integrity/
consistency
Load
Access
Ad Hoc Query Tools
Report Writers
Analytic Applications
Modeling:
Forecasting
Scoring
Data
Mining
Data Mart #1
Dimensional
Atomic AND
summery data
Business
Process Centric
Design Goals:
Easy-of -use
Query
Performance
Data Mart #2
Data Mart #.....
Data Mart Bus:
Conformed facts and dims
Extract
Extract
Extract
Data Access Tools
Independent Data Marts: Ralph Kimball’s Ideology
Ralph Kimball’ Approach

23
•E/R Design or Flat File
•Retain History Needed
for
regular processing
•No end user access
• Dimensional
•Transaction &
Summary data
•Data Mart Single
subject area
(i.e. Fact table)
•Multiple Marts May
exist in a
Single Database
Instance
Bottom Up Approach
Staging Data Store
Data Warehouse
Data Mart Data Mart Data Mart
Data Mart Data MartData Mart
•Integrated Data
•Timely User Access
•Conformed Dimensions
•Single Process to
Build Dimension

25
Bill Inmon’ Approach
Source
System
Data Staging
Area
Presentation
Area
“Enterprise Data
Warehouse”
Normalized
tables
Atomic Data
User query
support to
atomic data
Extract
Extract
Extract
Load
Data Mart #1
Dimensional
summery data
Departmental
Centric
Access
Access
Data Access
Tools
Data Mart #2
Data Mart #...
ETL
Dependent Data Marts: Bill Inmon’s Ideology
DWH

26
Top Down Approach
• Raw Input Data
• E/R Model
• Subject Areas
• Transaction Level Detail
• Historical Persistency As justified- Archive
for Retrieval if Needed
• Most are dimensional
• Data Mart Design by Business
Function
• Summary Level Data
•
Data Mart Data Mart
Staging Data Store
Data
Warehouse
Data Mart
Data
Mart
Flat
File
•Integrated Data
•Timely user Access
•Single Process to build dimension

27
Top Down
 More planning and design initially
 Involve people from different work-
groups, departments
 Data marts may be built later from
Global DW
 Overall data model to be decided up-
front
Bottom Up
 Can plan initially without waiting for
global infrastructure
 built incrementally
 can be built before or in parallel with
Global DW
 Less complexity in design

28
Top Down
 Consistent data definition and
enforcement of business rules across
enterprise
 High cost, lengthy process, time
consuming
 Works well when there is centralized IS
department responsible for all H/W and
resources
Bottom Up
 Data redundancy and
inconsistency between data marts
may occur
 Integration requires great planning
 Less cost of H/W and other
resources
 Faster pay-back

29
29
DW Architectures

30
Prod
Mkt
HR
Fin
Acctg
Data Sources
Transaction Data
IBM
IMS
VSAM
Oracle
Sybase
ETL Software Data Stores Data Analysis
Tools and
Applications
Users
Other Internal Data
ERP SAP
Clickstream Informix
Web Data
External Data
Demographic Harte-
Hanks
S
T
A
G
I
N
G
A
R
E
A
O
P
E
R
A
T
I
O
N
A
L
D
A
T
A
S
T
O
R
E
Ascential
Extract
Sagent
SAS
Clean/Scrub
Transform
Firstlogic
Load
DATASTAGE
Data Marts
Teradata
IBM
Data
Warehouse
Meta
Data
Finance
Marketing
Sales
Essbase
Microsoft
ANALYSTS
MANAGERS
EXECUTIVES
OPERATIONAL
PERSONNEL
CUSTOMERS/
SUPPLIERS
SQL
Cognos
SAS
Queries,Reporting,
DSS/EIS,
Data Mining
Micro Strategy
Siebel
Business
Objects
Web
Browser

31
Benefits of DWH
To formulate effective business, marketing
and sales strategies.
To precisely target promotional activity.
To discover and penetrate new markets.
To successfully compete in the marketplace
from a position of informed strength.
To build predictive rather than retrospective models.

32
Data Modeling

33
Data Modeling
 WHAT IS A DATA MODEL?
A data model is an abstraction of some aspect of the real
world (system).
 WHY A DATA MODEL?
• Helps to visualize the business
• A model is a means of communication.
• Models help elicit and document requirements.
• Models reduce the cost of change.
• Model is the essence of DW architecture based on which
DW will be implemented

34
STEPS in DATA MODELING
Problem & scope definition
Requirement Gathering
Analysis
Logical Database Design
Deciding Database
Physical Database design
Schema Generation

35
Levels of modeling
 Conceptual modeling
Describe data requirements from a
business point of view without technical
details
 Logical modeling
Refine conceptual models
Data structure oriented, platform
independent
 Physical modeling
Detailed specification of what is physically
implemented using specific technology

36
Modeling Techniques
 Entity-Relationship Modeling
Traditional modeling technique
Technique of choice for OLTP
Suited for corporate data warehouse
 Dimensional Modeling
Analyzing business measures in the specific business context
Helps visualize very abstract business questions
End users can easily understand and navigate the data
structure

37
 Relationship
Relationship between entities - structural interaction
and association
described by a verb
Cardinality
 1-1
 1-M
 M-M
Example : Books belong to Printed Media
Entity-Relationship Modeling - Basic Concepts

38
Entity-Relationship Modeling - Basic Concepts
 Attributes
Characteristics and properties of entities
Example :
 Book Id, Description, book category are
attributes of entity “Book”
Attribute name should be unique and self-
explanatory
Primary Key, Foreign Key, Constraints are defined
on Attributes

Review of Logical Modeling Terms & Symbols
 Entities define specific groups of information
Sales Organization
Sales Org ID
Distribution Channel
Entity

 One or more attribute uniquely identifies an instance of an
entity
Sales Organization
Sales Org ID
Identifier

 The logical model identifies relationships between
entities
Sales Detail
Sales Record ID
Sales Rep
Sales Rep ID
Relationship
{

Logical Data Model
Sales Detail
Sales Record ID
Customer
Customer ID
Product
Product SKU
Suppliers
Supplier ID
Manufacturing Group
Manufacturing Org ID
Factory
Factory ID
Sales Organization
Sales Org ID
Sales Rep
Sales Rep ID
Retail
Market
Product Sales Plan
Plan ID
Wholesale
Industry

44
44
Examples: ER Model

45
Limitations of E-R Modeling
 Poor Performance
 Tend to be very complex and difficult to navigate.

47
47
Dimensional Modeling

48
Dimensional Modeling
 Dimensional modeling uses three basic concepts : measures,
facts, dimensions.
 Is powerful in representing the requirements of the business
user in the context of database tables.
 Focuses on numeric data, such as values counts, weights,
balances and occurences.

49
What is a Facts
 A fact is a collection of related data items, consisting of measures
and context data.
 Each fact typically represents a business item, a business
transaction, or an event that can be used in analyzing the business
or business process.
 Facts are measured, “continuously valued”, rapidly changing
information. Can be calculated and/or derived.
 Granularity
The level of detail of data contained in the data warehouse
e.g. Daily item totals by product, by store

50
Types of Facts
 Additive
Able to add the facts along all the dimensions
Discrete numerical measures eg. Retail sales in $
 Semi Additive
Snapshot, taken at a point in time
Measures of Intensity
Not additive along time dimension eg. Account balance, Inventory
balance
Added and divided by number of time period to get a time-average
 Non Additive
Numeric measures that cannot be added across any dimensions
Intensity measure averaged across all dimensions eg. Room
temperature
Textual facts - AVOID THEM

51
Dimensions
 A dimension is a collection of members or units of the same type
of views.
 Dimensions determine the contextual background for the facts.
 Dimensions represent the way business people talk about the
data resulting from a business process, e.g., who, what, when,
where, why, how

52
52
Dimensional Hierarchy
World
America AsiaEurope
USA
FL
Canada Argentina
GA VA CA WA
TampaMiami Orlando Naples
Continent Level
State Level
City Level
World Level
Country Level
ParentRelation
Dimension Member /
Business Entity
Geography Dimension
Attributes: Population, Tourist’s Place

53
Dimensions Types
 Conformed Dimension
 Junk Dimension
 Fast Changing Dimension
 Role Playing Dimension
 ‘Garbage’ Dimension
 Slowly Changing Dimension
 Degenerated Dimension
53

54
What is a Slowly Changing Dimension?
 Although dimension tables are typically static lists, most dimension tables do change over
time.
 Since these changes are smaller in magnitude compared to changes in fact tables, these
dimensions are known as slowly growing or slowly changing dimensions.

55
Slowly Changing Dimension -Classification
Slowly changing dimensions are classified into three different
types
 TYPE I
 TYPE II
 TYPE III

56
Slowly Changing Dimensions Type I
Shane
Name
Shane@xyz.com1001
EmailEmp id
Shane
Name
Shane@xyz.com1001
EmailEmp id
Shane
Name
Shane@
abc.co.in
1001
EmailEmp id
Shane
Name
Shane@
abc.co.in
1001
EmailEmp id
Source
Source Target
Target
Shane@
xyz.com

57
Slowly Changing Dimensions Type II
Shane
Name
Shane@xyz.com10
EmailEmp id
Shane@x
yz.
com
Email
Shane
Name
10
Emp id
1000
PM_PRI
MARYK
EY
0
PM_VER
SION_N
UMBER
Source Target

59
Slowly Changing Dimensions -Versioning
Shane
Name
Shane@
abc.co.in
10
EmailEmp id
Source
Target
0Shane@
xyz.com
Shane101000
1Shane@
abc.co.in
Shane101001
EmailNameEmp idPM_PRIMA
RYKEY
PM_VERSION_NUMBER

60
Slowly Changing Dimensions -Versioning
Shane
Name
Shane@
abc.com
10
EmailEmp id
Source
Target
1Shane@
abc.co.in
Shane101001
2Shane@
abc.com
Shane101003
0Shane@
xyz.com
Shane101000
EmailNameEmp idPM_PRIM
ARYKEY
PM_VERSION_NUM
BER

61
Slowly Changing Dimensions Type II -
Flag
Shane
Name
Shane@xyz.com10
EmailEmp id
Shane@
xyz.
com
Email
Shane
Name
10
Emp id
1000
PM_PR
IMAR
YKEY
Y
PM_CUR
RENT_FL
AG
Source
Target

62
Slowly Changing Dimensions - Flag Current
Shane
Name
Shane@
abc.co.in
10
EmailEmp id
Source
Target
NShane@
xyz.com
Shane101000
YShane@
abc.co.in
Shane101001
RYKEY
PM_CURRENT_FLAG

63
Slowly Changing Dimensions - Flag Current
Shane
Name
Shane@
abc.com
10
EmailEmp id
Source
Target
NShane@
abc.co.in
Shane101001
YShane@
abc.com
Shane101003
NShane@
xyz.com
Shane101000
RYKEY
PM_CURRENT_FLAG

65
Slowly Changing Dimensions Type II
Shane
Name
Shane@xyz.c
om
10
EmailEmp id
01/01/00
PM_BEG
IN_DAT
E
Shane@x
yz.com
Email
Shane
Name
10
Emp id
1000
PM_PRI
MARYK
EY
PM_EN
D_DATE
Source
Target

66
Slowly Changing Dimensions -Effective Date
Shane
Name
Shane@
abc.co.in10
Email
Emp id
Source
Target
03/01/00
01/01/00
PM_BEGIN_D
ATE
03/01/00Shane@x
yz.com
Shane101000
Shane@
abc.co.in
Shane101001
EmailNameEmp idPM_PRIMAR
YKEY
PM_END_D
ATE

67
Slowly Changing Dimensions - Effective Date
Shane
Name
Shane@
abc.com10
EmailEmp id
Source
Target
05/02/00
03/01/00
01/01/00
PM_BEGIN_D
ATE
05/02/00Shane@
abc.co.in
Shane101001
Shane@
abc.com
Shane101003
03/01/00Shane@
xyz.com
Shane101000
ARYKEY
PM_END_DA
TE

69
Slowly Changing Dimensions Type III
Shane
Name
Shane@xyz.c
om
10
EmailEmp id
PM_Prev_
Column
Name
Shane@xyz.
com
Email
Shane
Name
10
Emp id
1
PM_PRI
MARYKE
Y
01/01/00
PM_EFFEC
T_DATE
Source
Target

70
Shane
Name
Shane@
abc.co.in10
EmailEmp id
Source
Target
Shane@xyz.co
m
PM_Prev_Colu
mnName
01/02/00Shane@
abc.co.in
Shane101
EmailNameEmp idPM_PRIMAR
YKEY
PM_EFFEC
T_DATE

71
Shane
Name
Shane@
abc.com10
EmailEmp id
Source
Target
Shane@
abc.co.in
PM_Prev_Colu
mnName
01/03/00Shane@
abc.com
Shane101
ARYKEY
PM_EFFECT_
DATE

72
Degenerate Dimension
 Dimension keys in fact table without corresponding dimension tables are
called Degenerate Dimensions
 Purpose of Degenerate Dimensions
1. Generally used when each record in fact represents transaction line item
2. Useful for grouping transaction line items belonging to a single
transaction

73
Fast Changing Dimension
A fast changing dimension is a dimension whose attribute or
attributes for a record (row) change rapidly over time.
1. Example: Age of associates, Income, Daily balance etc.
2. Technique to handle fast changing dimension: Create band
tables

74
Role Playing Dimension
A single dimension which is expressed differently in a fact table using
views is called a role-playing dimension. This can be achieved by
creating views on dimension table.

75
Conformed Dimension
A conformed dimension means the same thing to
each fact table to which it can be joined.
Typically, dimension tables that are referenced or are
likely to be referenced by multiple fact tables
(multiple dimensional models) are called conformed
dimensions
.

76
Conformed Dimension Option #1
 Identical dimensions with same keys, labels, definitions and Values
Sales Schema
Inventory Schema
SALES Facts
DATE KEY
PRODUCT KEY
STORE KEY
PROMO KEY
Product Desc
Brand Desc
Category Desc
PRODUCT KEY
INVENTORY
Facts
DATE KEY
PRODUCT KEY
STORE KEY
Product Desc
Brand Desc
Category Desc
PRODUCT KEY

77
Conformed Dimension Option #2
Subset of base dimension with common labels, definitions
and values
Sales
Schema
Forecast
Schema
SALES $
DATE KEY
PRODUCT KEY
STORE KEY
PROMO KEY
Product Desc
Brand Desc
Category Desc
PRODUCT KEY
DATE KEY
Day-of-week
Week Desc
Month Desc
SALES $
MONTH KEY
BRAND KEYBrand Desc
Category Desc
BRAND KEY MONTH KEY
Month Desc
BRAND KEY Brand Desc Category Desc
12345 Cherriors Cereal
PROD KEY Prod Desc Brand Desc Category Desc
12345 Cherriors 10 Cherriors Cereal

78
‘Garbage’ Dimension
A garbage dimension is a dimension that consists of low-cardinality columns
such as codes, indicators, and status flags.
Approach to handle Garbage dimension:
• Put the new attributes into existing dimension tables.
• Put the new attributes into the fact table.
• Create new separate dimension tables garbage dimension
• Create a separate ‘Garbage Dimension’ table

79
Junk Dimensions
 Whether to use junk dimension
5 indicators, each has 3 values -> 243 (35
) rows
5 indicators, each has 100 values -> 100 million (1005
) rows
 When to insert rows in the dimension

80
Factless Fact Tables
The two types of factless fact tables are:
 Coverage tables
 Event tracking tables

81
Factless Fact Tables - Coverage Tables
Coverage tables are required when a primary fact
table is sparse
Example: Tracking products in a store that did not sell

83
Factless Fact Tables - Event Tracking
These tables are used for tracking a event:
Example: Tracking student attendance

84
Fact Constellation
 Fact constellations: Multiple fact tables share dimension tables,viewed as
a collection of stars, therefore called galaxy schema or fact constellation

85
What is a Data mart?
 Data mart is a decentralized subset of data found either in a data warehouse
or as a standalone subset designed to support the unique business unit
requirements of a specific decision-support system.
 Data marts have specific business-related purposes such as measuring the
impact of marketing promotions, or measuring and forecasting sales
performance etc,.
Data Mart
Data Mart
Enterprise
Data Warehouse

86
Data marts - Main Features
Main Features:
 Low cost
 Controlled locally rather than centrally, conferring power on the user group.
 Contain less information than the warehouse
 Rapid response
 Easily understood and navigated than an enterprise data warehouse.
 Within the range of divisional or departmental budgets

88
Datamart Advantages :
 Typically single subject area and fewer dimensions
 Limited feeds
 Very quick time to market (30-120 days to pilot)
 Quick impact on bottom line problems
 Focused user needs
 Limited scope
 Optimum model for DW construction
 Demonstrates ROI
 Allows prototyping
Advantages of Datamart over Datawarehouse

89
Data Mart disadvantages :
• Does not provide integrated view of business information.
• Uncontrolled proliferation of data marts results in redundancy
• More number of data marts complex to maintain
• Scalability issues for large number of users and increased data volume
Disadvantages of Data Mart

90
90
Data marts
• Embedded data marts are marts that are stored within
the central DW. They can be stored relationally as files or
cubes.
• Dependent data marts are marts that are fed directly by
the DW, sometimes supplemented with other feeds, such
as external data.
• Independent data marts are marts that are fed directly
by external sources and do not use the DW.
DM - Types

®
IBM Software Group
The Operational Data Store

92

93
Why We Need Operational Data Store?
Need
 To obtain a “system of record” that contains the best data that
exists in a legacy environment as a source of information
 Best here implies data to be
Complete
Up to date
Accurate
 In conformance with the organization’s information model

 ODS data resolves data integration issues
 Data physically separated from production
environment to insulate it from the processing
demands of reporting and analysis
 Access to current data facilitated.
Operational Data Store - Insulated from OLTP
Tactical
Analysis
OLTP Server
ODS

95
 Detailed data
 Records of Business Events
(e.g. Orders capture)
 Data from heterogeneous sources
 Does not store summary data
 Contains current data
Operational Data Store - Data

97
ODS- Benefits
 Integrates the data
 Synchronizes the structural differences in data
 High transaction performance
 Serves the operational and DSS environment
 Transaction level reporting on current data
Flat
files
Relational
Database
Operational
Data Store
60,5.2,”JOHN”
72,6.2,”DAVID”
Excel files

 Update schedule - Daily or less
time frequency
 Detail of Data is mostly between
30 and 90 days
 Addresses operational needs
 Weekly or greater time frequency
 Potentially infinite history
 Address strategic needs
Operational Data Store- Update schedule
ODS
Data
Data warehouse
Data

100
OLTP Vs ODS Vs DWH
Characteristic OLTP ODS Data Warehouse
Data redundancy Non-redundant
within system;
Unmanaged
redundancy among
systems
Somewhat
redundant with
operational
databases
Managed
redundancy
Data stability Dynamic Somewhat dynamic Static
Data update Field by field Field by field Controlled batch
Data usage Highly structured,
repetitive
Somewhat
structured, some
analytical
Highly
unstructured,
heuristic or
analytical
Database size Moderate Moderate Large to very large
Database
structure stability
Stable Somewhat stable Dynamic

101
Star Schema Design
Single fact table surrounded by denormalized dimension
tables
The fact table primary key is the composite of the
foreign keys (primary keys of dimension tables)
Fact table contains transaction type information.
Many star schemas in a data mart
Easily understood by end users, more disk storage
required

102
EXAMPLE OF STAR SCHEMA

103
Snowflake Schema
Single fact table surrounded by normalized dimension
tables
Normalizes dimension table to save data storage space.
When dimensions become very very large
Less intuitive, slower performance due to joins
 May want to use both approaches, especially if supporting multiple
end-user tools.

104
Example of Snow flake schema

105
Snowflake - Disadvantages
 Normalization of dimension makes it difficult for user to
understand
 Decreases the query performance because it involves more
joins
 Dimension tables are normally smaller than fact tables - space
may not be a major issue to warrant snowflaking

106
Data Acquisation
 Data Extraction
 Data Transformation
 Data Loading
106

107
Tool Category Products
ETL Tools ETI Extract, Informatica, IBM Visual Warehouse
Oracle Warehouse Builder
OLAP Server Oracle Express Server, Hyperion Essbase, IBM DB2
OLAP Server, Microsoft SQL Server OLAP Services,
Seagate HOLOS, SAS/MDDB
OLAP Tools Oracle Express Suite, Business Objects, Web
Intelligence, SAS, Cognos Powerplay/Impromtu,
KALIDO, MicroStrategy, Brio Query, MetaCube
Data Warehouse Oracle, Informix, Teradata, DB2/UDB, Sybase,
Microsoft SQL Server, RedBricks
Data Mining &
Analysis
SAS Enterprise Miner, IBM Intelligent Miner,
SPSS/Clementine, TCS Tools
Representative DW Tools

108
ETL PRODUCTS
 CODE BASED ETL TOOLS
 GUI BASED ETL TOOLS
108

109
CODE BASED ETL TOOLS
 SAS ACCESS
 SAS BASE
 TERADATA ETL TOOLS
1. BTEQ
2. TPUMP
3. FAST LOAD
4. MULTI LOAD

110
GUI BASED ETL TOOLS
 Informatica
 DT/Studio
 Data Stage
 Business Objects Data Integrator (BODI)
 AbInitio
 Data Junction
 Oracle Warehouse Builder
 Microsoft SQL Server Integration Services
 IBM DB2 Ware house Center

®
IBM Software Group
Extraction Types

112
Extraction Types
Extraction
Full Extract
Periodic/
Incremental
Extract

113
Full Extract
Source System
Full Extract
Data Mart
New data

115
Incremental Extract
Data Mart
Source System
Incremental Extract
Existing data
Incremental
Data

116
Incremental Extract
Data Mart
Source System
Incremental Extract
New data
Changed data
Existing data
Incremental
Data

117
Incremental Extract
Data Mart
Source System
Incremental Extract
New data
Changed data
Existing data updated
using changed data
Incremental
Data
Incremental addition
to data mart

118
DATAWARE LOADING

120
Types of Data warehouse Loading
 Target update types
Insert
Update

Types of Data Warehouse Updates
Insert
Full Replace
Selective Replace
Update plus Retain History
Update
Point in Time
Snapshots
New Data
Changed Data
Data Warehouse
Source data Data Staging

New Data and Point-In-Time Data Insert
Source data
New data
OR
Point-in-Time
Snapshot
(e.g.. Monthly)
New Data Added to
Existing Data

Changed Data Insert
Source data
Changed Data Added to
Existing Data
Changed
data

124
DataData
WareWare
househouse
DataData
WareWare
househouse
Enterprise
Data
Warehouse
InfoInfo
AccessAccess
InfoInfo
AccessAccess
Reporting tools
Web
Browsers
OLAP
Mining
ETLETLETLETL
External DataExternal Data
StorageStorage
BusinessBusiness
RequirementRequirement
Map DataMap Data
sourcessources
ReverseReverse
Engg.Engg.
MapMap
Req. toReq. to
OLTPOLTP
OLTPOLTP
SystemSystem
LogicalLogical
ModelingModeling
RefineRefine
ModelModel
Data Warehouse Life cycle

125
Project Life Cycle
 Software Requirement Specification
 High level Design(HLD)
 Low level Design(LLD)
 Development
 Unit Testing
 System Integration Testing
 Peer Review
 User Acceptance Testing
 Production
 Maintenance
125

®
IBM Software Group
Meta Data in a
Data Warehouse

127
• Data about data and the processes
• Metadata is stored in a data dictionary and repository.
• Insulates the data warehouse from changes in the schema of
operational systems.
• It serves to identify the contents and location of data in the
data warehouse
What is Metadata?

128
 Share resources
 Users
 Tools
 Document system
 Without meta data
 Not Sustainable
 Not able to fully utilize resource
Why Do You Need Meta Data?

The Role of Meta Data in the Data Warehouse
 Know what data you have and
 You can trust it!
Meta Data enables data to become information, because with it you

Meta Data Answers….
How have business definitions and terms changed over time?
How do product lines vary across organizations?
What business assumptions have been made?
How do I find the data I need?
What is the original source of the data?
How was this summarization created?
What queries are available to access the data

131
Meta Data Process
 Integrated with entire process and data flow
 Populated from beginning to end
 Begin population at design phase of project
 Dedicated resources throughout
 Build
 Maintain
•Design
•Mapping
•Design
•Mapping
•Extract
•Scrub
•Transform
•Extract
•Scrub
•Transform
•Load
•Index
•Aggregation
•Load
•Index
•Aggregation
•Replication
•Data Set Distribution
•Replication
•Data Set Distribution
•Access & Analysis
•Resource Scheduling & Distribution
•Access & Analysis
•Resource Scheduling & Distribution
Meta DataMeta Data
System MonitoringSystem Monitoring

132
Types of ETL Meta Data
.
ETL Meta data
Technical
Meta data
Operational
Meta data

 Data Warehouse Meta data
This Meta data stores descriptive information about the physical
implementation details of data warehouse.
 Source Meta data
This Meta data stores information about the source data and the mapping of source
data to data warehouse data
Classification of ETL Meta Data

 Transformations & Integrations.
This Meta data describes comprehensive information about the Transformation and
loading.
 Processing Information
This Meta data stores information about the activities involved in the processing of data
such as scheduling and archives etc
 End User Information
This Meta data records information about the user profile and security.
ETL Meta Data

135
ETL -Planning for the Movement
The following may be helpful for planning the movement
 Develop a ETL plan
 Specifications
 Implementation

DATASTAGE AND QUALITY STAGE 9.1 ONLINE TRAINING

More Related Content

What's hot

Viewers also liked

Similar to DATASTAGE AND QUALITY STAGE 9.1 ONLINE TRAINING

Recently uploaded

DATASTAGE AND QUALITY STAGE 9.1 ONLINE TRAINING