IOUG93 - Technical Architecture for the Data Warehouse - Presentation
David M Walker Consultant Data Management & Warehousing ATechnical Architecture For The Data Warehouse
Data Warehouse Implementation Strategy Project Management Business Analysis Database Schema Design Technical Architecture
Business Analysis•! End user driven•! Cross Functional Workshops•! Iterative design principle (80/20 rules)•! Determine the Key Performance Indicators (KPI)•! Determine constraints on KPI
Database Schema Design•! Identify sources of information•! Qualify external sources of information•! Translate KPI into facts•! Translate constraints into dimensions•! Choose required aggregations•! Build Meta Data and Security Model
Project Management•! Iterative Process•! Rapid Application Development (RAD) techniques•! Arbitration when 80/20 rule used•! Conflict of short and long term goals
The Data Warehouse Systems Logical Architecture Presentation Third Party Tools Third Party Tools Layer The Data Warehouse Middleware Middleware Security EIS EIS Meta Data Decision Decision Support Systems Support Systems Transaction Repository Data Acquisition Operational Systems OLTP Legacy External System System Data Sources
Data Acquisition Data Extraction Data Load •!Extraction •!Loading •!Transformation •!Exception Processing •!Collation •!Quality Assurance •!Migration •!Publication
Data Aggregation Year Executive Information Systems Quarter Month Decision Support System Week Transaction Repository Day
The Cost Of AggregationA very simple schema:100 Stores 1095 Days 100000 Products 10 Regions 157 Weeks 1000 Categories 1 Company 36 Month 10 Groups 12 Quarters 1 Type 3 YearsRows: No aggregation, No sparsity: 10950000000 Aggregation, No sparsity: 14609523963 Growth 33% No aggregation,30% sparsity: 7665000000 Aggregation, Variable sparsity: 10574481741 Growth 38%If each row is 64 bytes long, a 10Billion row schema without indexesand other overheads would be 630Gb!
Data Mart Time Dimension Associated Another Dimension Day Facts Week Month Quarter Year Another Dimension Another Dimension
Meta Data Dictionary And Security Meta Data •!Master schema Security •!Star schema Control of •!Star schema description user access •!Table to the data •!Table description •!Table row count •!Column •!Column description •!Column derivation •!Column format
Middleware and Presentation•! Use a common middleware•! Group users based on their requirements•! Try a number of tools for each group•! Final solution will have more than one front end, but not an infinite number•! Add value with alert systems
ConclusionStrategy Technical Architeture •! Project Managment •! Source Systems •! Business Analysis •! Data Acquisition •! Schema Design •! Transaction Repository •! Technical Architecture •! Data Aggregation •! Data Mart •! Meta Data & Security •! Middleware & Presentation Help your users find it !
Contacts•! Data Management & Warehousing –! WWW http://www.datamgmt.com –! Mail firstname.lastname@example.org –! Telephone +44 1734 771291 –! Fax +44 1734 773058•! The Data Warehouse Institute –! WWW http://www.tekptnr.com/tpi/tdwi –! Mail email@example.com•! The Data Warehouse Information Center –! WWW http://pwp.starnetinc.com/larryg/index.html