Defined in many different ways, but not rigorously.
A decision support database that is maintained separately from the organization’s operational database
Support information processing by providing a solid platform of consolidated, historical data for analysis.
“ A data warehouse is a subject-oriented , integrated , time-variant , and nonvolatile collection of data in support of management’s decision-making process.”—W. H. Inmon
Data warehousing:
The process of constructing and using data warehouses
Build wrappers/mediators on top of heterogeneous databases
Query driven approach
When a query is posed to a client site, a meta-dictionary is used to translate the query into queries appropriate for individual heterogeneous sites involved, and the results are integrated into a global answer set
Complex information filtering, compete for resources
Data warehouse: update-driven , high performance
Information from heterogeneous sources is integrated in advance and stored in warehouses for direct query and analysis
A data warehouse is based on a multidimensional data model which views data in the form of a data cube
A data cube, such as sales , allows data to be modeled and viewed in multiple dimensions
Dimension tables, such as item (item_name, brand, type), or time(day, week, month, quarter, year)
Fact table contains measures (such as dollars_sold ) and keys to each of the related dimension tables
In data warehousing literature, an n-D base cube is called a base cuboid . The top most 0-D cuboid, which holds the highest-level of summarization, is called the apex cuboid . The lattice of cuboids forms a data cube.
14.
Cube: A Lattice of Cuboids all time item location supplier time,item time,location time,supplier item,location item,supplier location,supplier time,item,location time,item,supplier time,location,supplier item,location,supplier time, item, location, supplier 0-D(apex) cuboid 1-D cuboids 2-D cuboids 3-D cuboids 4-D(base) cuboid
Star schema : A fact table in the middle connected to a set of dimension tables
Snowflake schema : A refinement of star schema where some dimensional hierarchy is normalized into a set of smaller dimension tables , forming a shape similar to snowflake
Fact constellations : Multiple fact tables share dimension tables , viewed as a collection of stars, therefore called galaxy schema or fact constellation
16.
Example of Star Schema Sales Fact Table time_key item_key branch_key location_key units_sold dollars_sold avg_sales Measures time_key day day_of_the_week month quarter year time location_key street city state_or_province country location item_key item_name brand type supplier_type item branch_key branch_name branch_type branch
17.
Example of Snowflake Schema Sales Fact Table time_key item_key branch_key location_key units_sold dollars_sold avg_sales Measures time_key day day_of_the_week month quarter year time location_key street city_key location item_key item_name brand type supplier_key item branch_key branch_name branch_type branch supplier_key supplier_type supplier city_key city state_or_province country city
18.
Example of Fact Constellation Sales Fact Table time_key item_key branch_key location_key units_sold dollars_sold avg_sales Measures Shipping Fact Table time_key item_key shipper_key from_location to_location dollars_cost units_shipped time_key day day_of_the_week month quarter year time location_key street city province_or_state country location item_key item_name brand type supplier_type item branch_key branch_name branch_type branch shipper_key shipper_name location_key shipper_type shipper
distributive : if the result derived by applying the function to n aggregate values is the same as that derived by applying the function on all the data without partitioning.
E.g., count(), sum(), min(), max().
algebraic : if it can be computed by an algebraic function with M arguments (where M is a bounded integer), each of which is obtained by applying a distributive aggregate function.
E.g., avg(), min_N(), standard_deviation().
holistic : if there is no constant bound on the storage size needed to describe a subaggregate.
E.g., median(), mode(), rank().
24.
A Concept Hierarchy: Dimension (location) all Europe North_America Mexico Canada Spain Germany Vancouver M. Wind L. Chan ... ... ... ... ... ... all region office country Toronto Frankfurt city
Sales volume as a function of product, month, and region
Product Region Month Dimensions: Product, Location, Time Hierarchical summarization paths Industry Region Year Category Country Quarter Product City Month Week Office Day Pick one node from each dimension hierarchy, you get a data cube!
27.
A data cube all product quarter country product, quarter product,country quarter, country product, quarter, country 0-D(apex) cuboid 1-D cuboids 2-D cuboids 3-D(base) cuboid
How many values does the base cuboid (Product, Quarter, Country) has?
How many values does the cuboid (Quarter, Country) have?
How many values does the complete cube have?
29.
A Sample Data Cube Total annual sales of TV in U.S.A. Quarter Product Country All, All, All sum sum TV VCR PC 1Qtr 2Qtr 3Qtr 4Qtr U.S.A Canada Mexico sum
Top-down, bottom-up approaches or a combination of both
Top-down : Starts with overall design and planning (mature)
Bottom-up : Starts with experiments and prototypes (rapid)
From software engineering point of view
Waterfal l: structured and systematic analysis at each step before proceeding to the next
Spiral : rapid generation of increasingly functional systems, short turn around time, quick turn around
Typical data warehouse design process
Choose a business process to model, e.g., orders, invoices, etc.
Choose the grain ( atomic level of data ) of the business process
Choose the dimensions that will apply to each fact table record
Choose the measure that will populate each fact table record
35.
Multi-Tiered Architecture Data Warehouse OLAP Engine Analysis Query Reports Data mining Monitor & Integrator Metadata Data Sources Front-End Tools Serve Data Marts Data Storage OLAP Server Extract Transform Load Refresh Operational DBs other sources
collects all of the information about subjects spanning the entire organization
Data Mart
a subset of corporate-wide data that is of value to a specific groups of users. Its scope is confined to specific, selected groups, such as marketing data mart
Independent vs. dependent (directly from warehouse) data mart
Virtual warehouse
A set of views over operational databases
Only some of the possible summary views may be materialized
37.
Data Warehouse Development: A Recommended Approach Define a high-level corporate data model Data Mart Data Mart Distributed Data Marts Multi-Tier Data Warehouse Enterprise Data Warehouse Model refinement Model refinement
Compute aggregates in “multiway” by visiting cube cells in the order which minimizes the # of times to visit each cell, and reduces memory access and storage cost.
What is the best traversing order to do multi-way aggregation? A B 29 30 31 32 1 2 3 4 5 9 13 14 15 16 64 63 62 61 48 47 46 45 a1 a0 c3 c2 c1 c 0 b3 b2 b1 b0 a2 a3 C B 44 28 56 40 24 52 36 20 60
45.
Multi-way Array Aggregation for Cube Computation B A B 29 30 31 32 1 2 3 4 5 9 13 14 15 16 64 63 62 61 48 47 46 45 a1 a0 c3 c2 c1 c 0 b3 b2 b1 b0 a2 a3 C 44 28 56 40 24 52 36 20 60
46.
Multi-way Array Aggregation for Cube Computation A B 29 30 31 32 1 2 3 4 5 9 13 14 15 16 64 63 62 61 48 47 46 45 a1 a0 c3 c2 c1 c 0 b3 b2 b1 b0 a2 a3 C 44 28 56 40 24 52 36 20 60 B Order: A B C AB: plane AC: line BC: point
47.
Multi-Way Array Aggregation for Cube Computation (Cont.)
Let A: 40 values, B: 400 values, C: 4000 values.
One chunk contains 10*100*1000 = 1,000,000 values.
A B C needs how much memory?
AB plane: 40*400=16,000
AC line: 40*(4000/4) = 40,000
BC point: (400/4)*(4000/4) = 100,000
total: 156,000
C B A needs how much memory?
CB plane: 4000*400=1,600,000
CA line: 4000*(40/4) = 40,000
BA point: (400/4)*(40/4) = 1000
total: 1,641,000 --- 10 times more!
48.
Chapter 3: Data Warehousing and OLAP Technology for Data Mining
Especially when #dimensions is large, should store in a compressed way.
SELECT P, L, M, COUNT(*) FROM Sales CUBE BY P, L, M HAVING COUNT(*)>=2 Iceburg cube query asks for non-empty cuboids! E.g. is cuboid (P,L) empty? * m1 l3 p6 * m2 l2 p5 * m1 l2 p4 * m2 l1 p3 * m1 l1 p2 * m1 l1 p1 sale M L P
59.
What we have Can we do better? O(n 2 ) O(1) store prefix-sum O(1) O(n 2 ) store original cube update cost query cost O(n) O(log(n)) dynamic data cube (naïve version)
60.
The Dynamic Data Cube [EDBT’00] 1 1 1 1 1 2 3 4
Organize the original cube into a tree structure with fanout = 4.
69.
What we have Can we do better? Can we do better? O(n 2 ) O(1) store prefix-sum O(1) O(n 2 ) store original cube update cost query cost O(n) O(log(n)) dynamic data cube (naïve version) O(log 2 n) O(log 2 n) dynamic data cube
Be the first to comment