Data Warehousing and Mining - CS437 GIK Institute

1. Lecture 1
Dr. Fawad Hussain
GIK Institute
Fall 2015
Data Warehousing and MiningData Warehousing and MiningData Warehousing and MiningData Warehousing and Mining
(CS437)(CS437)(CS437)(CS437)
Some lectures in this course have been partially adapted from lecture series by StephenA. Brobst, Chief Technology Officer atTeradata and
professor at MIT.

2. General Course Description
Datawarehousing
What is the motivation behind DataWarehousing and Mining?
Advanced Indexing, Query Processing and Optimization.
Building DataWarehouses.
Data Cubes, OLAP, De-Normalization,etc.
Data MiningTechniques
Regression
Clustering
DecisionTrees
Other Information
Office Hours (Pasted on the office door)
Office: G03 (FCSE)
CourseTA (Mr. Bilal)

3. Text Books (Optional)
Introduction to Data Mining;Tan, Steinbach & Kumar.
Data Mining: Concepts andTechniques by Jiawei Han and
Micheline Kamber Morgan Kaufmann Publishers, 2nd Edition,
March 2006, ISBN 1-55860-901-6.
Building a DataWarehouse for Decision Support byVidette
Poe.
Fundamentals of Database Systems by Elmasri and Navathe
Addison-Wesley, 5th Edition, 2007.

4. Grading Plan
Grading Plan for Course %
Tentative
Number(s)
Midterm Exam 25 01
Quizzes 10 06
Project 20 02
Final Exam 45 01

5. Tentative Schedule

6. Tentative Schedule

7. Tentative Schedule

8. Lecture 1
Introduction and Overview

9. Why this Course?
The world is changing (actually changed), either change or be
left behind.
Missing the opportunities or going in the wrong direction has
prevented us from growing.
What is the right direction?
Harnessing the data, in a knowledge driven economy.

10. The Need
Knowledge is power, Intelligence is
absolute power!
“Drowning in data and starving for
information”

11. Data Processing Steps
DATA
INFORMATION
POWER
INTELLIGENCE
$
End goal?

12. Historical Overview
1960
Master Files & Reports
1965
Lots of Master files!
1970
Direct Access Memory & DBMS
1975
Online high performance transaction processing
1980
PCs and 4GL Technology (MIS/DSS)
Post 1990
Data Warehousing and Data Mining

13. Crises of Credibility
What is the financial health of our company?
-10%
+10%
??

14. Why a Data Warehouse?
Data recording and storage is growing.
History is excellent predictor of the future.
Gives total view of the organization.
Intelligent decision-support is required for decision-
making.

15. Why Data Warehouse?
Size of Data Sets are going up ↑.
Cost of data storage is coming down ↓.
The amount of data average business collects and stores
is doubling every year
Total hardware and software cost to store and manage 1
Mbyte of data
1990: ~ $15
2002: ~ ¢15 (Down 100 times)
By 2007: < ¢1 (Down 150 times)

16. Why Data Warehouse?
A Few Examples
WalMart: 24TB
FranceTelecom: ~ 100TB
CERN: Up to 20 PB by 2006
Stanford LinearAccelerator Center (SLAC): 500TB
Businesses demand Intelligence (BI).
Complex questions from integrated data.
“Intelligent Enterprise”

17. List of all items that were sold last month?
List of all items purchased by X?
The total sales of the last month grouped by branch?
How many sales transactions occurred during the month of
January?
DBMS Approach

18. Which items sell together? Which items to stock?
Where and how to place the items? What
discounts to offer?
How best to target customers to increase sales at
a branch?
Which customers are most likely to respond to
my next promotional campaign, and why?
Intelligent Enterprise

19. What is a Data Warehouse?
A complete repository of historical corporate data extracted
from transaction systems that is available for ad-hoc access by
knowledge workers.

20. What is Data Mining?
“There are things that we know that we know…
there are things that we know that we don’t know…
there are things that we don’t know we don’t know.”
Donald Rumsfield
Former US Secretary of Defence

21. What is Data Mining?
Tell me something that I should know.
When you don’t know what you should be knowing,
how do you write SQL?
You cant!!

22. What is Data Mining?
Knowledge Discovery in Databases (KDD).
Data mining digs out valuable non-trivial information from large
multidimensional apparently unrelated data bases(sets).
It’s the integration of business knowledge, people, information,
algorithms, statistics and computing technology.
Discovering useful hidden patterns and relationships in data.

23. HUGE VOLUME THERE IS WAY TOO MUCH
DATA & GROWING!
Data collected much faster than it can be processed or
managed. NASA Earth Observation System (EOS), will
alone, collect 15 Peta bytes by 2007
(15,000,000,000,000,000 bytes).
• Much of which won't be used - ever!
• Much of which won't be seen - ever!
• Why not?
There's so much volume, usefulness of some of it will never
be discovered
SOLUTION: Reduce the volume and/or raise the
information content by structuring, querying, filtering,
summarizing, aggregating, mining...

24. Requires solution of fundamentally new
problems
1. developing algorithms and systems to mine large, massive
and high dimensional data sets;
2. developing algorithms and systems to mine new types of
data (images, music, videos);
3. developing algorithms, protocols, and other infrastructure
to mine distributed data; and
4. improving the ease of use of data mining systems;
5. developing appropriate privacy and security techniques
for data mining.

25. Future of Data Mining
10 Hottest Jobs of year 2025
TIME Magazine,22 May,2000
10 emerging areas of technology
MIT’s Magazine ofTechnology Review,
Jan/Feb,2001

26. Data Mining
Data Mining
Machine
Learning
Database
Technology
Statistics
Visualization
Other
Disciplines
Information
Science

27. Logical and Physical DatabaseLogical and Physical DatabaseLogical and Physical DatabaseLogical and Physical Database
DesignDesignDesignDesign

28. Data Mining is one step of Knowledge
Discovery in Databases (KDD)
Raw
Data
Preprocessing
• Extraction
• Transformation
• Cleansing
• Validation
Data Mining
• Identify Patterns
• Create Models
Interpretation/
Evaluation
• Visualization
• Feature Extraction
• Analysis
Clean
Data
$ $ $
Knowledge

30. Information Evolution in a Data
Warehouse Environment
Primarily Batch Event Based
Triggering
Takes Hold
Increase in
Ad Hoc
Queries
Analytical
Modeling
Grows
Continuous Update &
Time Sensitive Queries
Become Important
Batch Ad Hoc Analytics Continuous Update/Short Queries Event-Based Triggering
STAGE 2:
ANALYZE
WHY did
it happen?
STAGE 3:
PREDICT
What WILL
happen?
STAGE 1:
REPORT
WHAT happened?
STAGE 4:
OPERATIONALIZE
What IS happening?
STAGE 5:
ACTIVATE
What do you WANT to
happen?

31. Normalization and Denormalization
Normalization
A relational database relates subsets of a dataset to each other.
A dataset is a set of tables (or schema in Oracle)
A table defines the structure and contains the row and column data for each
subset.
Tables are related to each other by linking them based on common items and
values between two tables.
Normalization is the optimization of record keeping for insertion, deletion
and updation (in addition to selection, ofcourse)
De-normalization
Why denormalize?
When to denormalize
How to denormalize

34. Why De-normalization?
Do you have performance problems?
If not, then you shouldn’t be studying this course!
The root cause of 99% of database performance problems is
poorly written SQL code.
Usually as a result of poorly optimized underlying structure
Do you have disk storage problems?
Consider separating large, less used datasets and frequently used
datasets.

35. When to Denormalize?
Denormalization sometimes implies the undoing of some of the
steps of Normalization
Denormalization is not necessarily the reverse of the steps of
Normalization.
Denormalization does not imply complete removal of specific
Normal Form levels.
Denormalization results in duplication.
It is quite possible that table structure is much too granular or possibly even
incompatible with structure imposed by applications.
Denormalization usually involves merging of multiple
transactional tables or multiple static tables into single

36. When to Denormalize?
Look for one-to-one relationships.
These may be unnecessary if the required removal of null values
causes costly joins. Disk space is cheap. Complex SQL join statements
can destroy performance.
Do you have many-to-many join resolution entities? Are they all
necessary? Are they all used by applications?
When constructing SQL statement joins are you finding many
tables in joins where those tables are scattered throughout the
entity relationship diagram?
When searching for static data items such as customer details are
you querying a single or multiple tables?
A single table is much more efficient than multiple tables.

37. How to Denormalize?
Common Forms of Denormalization
Pre-join de-normalization.
Column replication or movement.
Pre-aggregation.

38. Considerations in Assessing
De-normalization
Performance implications
Storage implications
Ease-of-use implications
Maintenance implications
Most commonly missed/disregarded.

39. Pre-join Denormalization
Take tables which are frequently joined and “glue” them together
into a single table.
Avoids performance impact of the frequent joins.
Typically increases storage requirements.

40. Pre-join Denormalization
A simplified retail example...
Before denormalization:
sale_id store_id sale_dt …
tx_id sale_id item_id … item_qty sale$
1
m

41. Pre-join Denormalization
tx_id sale_id store_id sale_dt item_id … item_qty $
A simplified retail example...
After denormalization:
Points to Ponder
Which Normal Form is being violated?
Will there be maintenance issues?

42. Pre-join Denormalization
Storage implications...
Assume 1:3 record count ratio between sales header and detail.
Assume 1 billion sales (3 billion sales detail).
Assume 8 byte sales_id.
Assume 30 byte header and 40 byte detail records.
Which businesses will be most hurt, in terms of storage capacity, by
this form of denormalization?

43. Pre-join Denormalization
Storage implications...
Before denormalization: 150 GB raw data.
After denormalization: 186 GB raw data.
Net result is 24% increase in raw data size for the database.
Pre-join may actually result in space saving, if many concurrent queries are
demanding frequent joins on the joined tables! HOW?

44. Pre-join Denormalization
Sample Query:
What was my total $ volume betweenThanksgiving and Christmas in
1999?

45. Pre-join Denormalization
Before de-normalization:
select sum(sales_detail.sale_amt)
from sales
,sales_detail
where sales.sales_id =
sales_detail.sales_id
and sales.sales_dt between '1999-11-26'
and '1999-12-25'
;

46. Pre-join Denormalization
After de-normalization:
select sum(d_sales_detail.sale_amt)
from d_sales_detail
where d_sales_detail.sales_dt between '1999-
11-26' and '1999-12-25'
;
No join operation performed.
How to compare performance?

47. Pre-join Denormalization
But consider the question...
How many sales (transactions) did I make betweenThanksgiving and
Christmas in 1999?

48. Pre-join Denormalization
Before denormalization:
select count(*)
from sales
where sales.sales_dt between '1999-11-26' and
'1999-12-25';
After denormalization:
select count(distinct d_sales_detail.sales_id)
from d_sales_detail
where d_sales_detail.sales_dt between '1999-11-
26' and '1999-12-25';
Which query will perform better?

49. Pre-join Denormalization
Performance implications...
Performance penalty for count distinct (forces sort) can be quite large.
May be worth 30 GB overhead to keep sales header records if this is a common
query structure because both ease-of-use and performance will be enhanced (at
some cost in storage)?

50. Considerations in Assessing
De-normalization
Performance implications
Storage implications
Ease-of-use implications
Maintenance implications
Most commonly missed/disregarded.

51. Column Replication or Movement
Take columns that are frequently accessed via large scale joins and
replicate (or move) them into detail table(s) to avoid join
operation.
Avoids performance impact of the frequent joins.
Increases storage requirements for database.
Possible to “move” frequently accessed column to detail instead of
replicating it.
Note: This technique is no different than a limited form of the pre-
join denormalization described previously.

52. ColA ColB
Table_1
ColA ColC ColD … ColZ
Table_2
ColA ColB
Table_1’
ColA ColC ColD … ColZ
Table_2
ColC

53. Column Replication or Movement
Health Care DW Example: Take member_id from claim header
and move it to claim detail.
Result: An extra ten bytes per row on claim line table allows
avoiding join to claim header table on some (many?) queries.
Which normal form does this technique violates?

54. Column Replication or Movement
Beware of the results of de-normalization:
Assuming a 100 byte record before the denormalization, all scans
through the claim line detail will now take 10% longer than
previously.
A significant percentage of queries must get benefit from access to
the denormalized column in order to justify movement into the
claim line table.
Need to quantify both cost and benefit of each denormalization
decision.

55. Column Replication or Movement
May want to replicate columns in order to facilitate co-location of commonly joined
tables.
Before denormalization:
A three table join requires re-distribution of significant amounts of data to answer many
important questions related to customer transaction behavior.
Customer_Id Customer_Nm Address Ph …
Account_Id Customer_Id Balance$ Open_Dt …
Tx_Id Account_Id Tx$ Tx_Dt Location_Id …
1
m
1
m
CustTable
AcctTable
TrxTable