Data Warehouse Design & Dimensional Modeling


Published on

At Code Mastery Boston Aaron Lowe, Principal Consultant at Magenic, talks Data Warehouse Design & Dimensional Modeling

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Data Warehouse Design & Dimensional Modeling

  1. 1. Data Warehouse Design& Dimensional ModelingAaron LowePrincipal Consultant@Vendoran@SQLFriends
  2. 2. Who am I» Aaron Lowe » Husband » Father of 5 » Principal Consultant at Magenic » Working with SQL Server since 1998, version 6.5 » MCITP 2005 and 2008 » Co-organizer of SQLSaturday Chicago » Masters in Information Systems Management » / @Vendoran » / @SQLFriends
  3. 3. Data, Data everywhere, but not a drop ofInformation
  4. 4. The Data Person
  5. 5. How can we get more out of our data?
  6. 6. Leverage data to provide business insight
  7. 7. Create a new Data Model in a Data Warehouse
  8. 8. Why a new Data Model?
  9. 9. What do we need?
  10. 10. Information – not just data» Collecting data » Log Files » Clicks » How long? » How much?» Prediction? » How Target Figured out Teen was pregnant - out-a-teen-girl-was-pregnant-before-her-father-did/ » The Numerati - Baker/dp/B003TO6G20/ - published 2009!
  11. 11. Relate data from multiple systems» The purpose of a data warehouse is to house standardized, structured, consistent, integrated, correct, cleansed and timely data, extracted from various operational systems in an organization» True picture of the business process» Source Systems » Financial – AR/AP » Sales » CRM » HR » Application
  12. 12. Fast» It’s my information and I want it Now!» Empower Users» Exploratory» Reads» Large datasets
  13. 13. Why won’t existing models work?
  14. 14. What are they designed for?» Operational » Preservation of data integrity » Speed of recording of business transactions » Often Many tables» To free the collection of relations from undesirable insertion, update and deletion dependencies;» To reduce the need for restructuring the collection of relations, as new types of data are introduced, and thus increase the life span of application programs;» To make the relational model more informative to users;» To make the collection of relations neutral to the query statistics, where these statistics are liable to change as time goes by.» —E.F. Codd, "Further Normalization of the Data Base Relational Model"
  15. 15. Consistent» Partial data across » Have the sale in the sales system » Represented in the inventory system » Don’t have the $ in the financial system yet» Deleted on sources » Removed transactions » Archive » Legally destroy records can remove work product» Incomplete data on source » Changes over time
  16. 16. Silo’d» How do we get the entire picture?» Example: » Cost of Sales? » Sales system – Sale Price » Marketing System – $$ spent on Marketing » Inventory System – $$ spent on inventory » HR System – $$ spent on Employee » IT Systems – $$ spent on Infrastructure
  17. 17. What will work?
  18. 18. Designed for Users» De-normalized » Fast Reads » Fast Reports » Limited JOINs» Information » Scheduled » On Demand » Exploratory» Information » Cross Functional » The more the better!
  19. 19. Inter-related data» Specifications for my Current Data Warehouse
  20. 20. Independent from Operational» Operational systems change » Data will outlive Application » Crashes » Upgrades » Breaking changes» Single Source of truth
  21. 21. Logical Data Model
  22. 22. Terminology
  23. 23. Metadata Management» Business metadata » What’s out there? » Identify/Define » Overloaded terms » What is a customer?» Process metadata » DW process operations » Asses system status » Investigate problems» Technical metadata » Tables » Fields » Datatypes
  24. 24. Dimensions and FactsDimensions FactsThing/Objects Measurements/EventsNouns VerbsWide but short Skinny but longRows can exist independently Rows cannot exist independentlyDescriptive Mostly Numeric and Additive “By” words – FACT by Dimension Quantity Ordered by Product by Customer by Date
  25. 25. Grain• Level of detail• What is needed to meet business requirements?• What is possible to collect?• How do you describe it?• One row per X where X is the business event • One row per customer call • One row per time sheet entry • One row per employee status change • One row per order line item
  26. 26. Methodology
  27. 27. Requirements – business focused» “Must embrace the goal of enhancing business value as the primary purpose.” – Kimball» “If your job is BI and you speak mostly to technical people all day, you are doing it wrong. Focus on first word - BUSINESS.” – Whitney Weaver (former Magenicon)» Never ask “What do you want in the data warehouse?” Only one right answer - “Everything.”» Ask questions that help you learn what the end user does
  28. 28. Kimball v. InmonRalph Kimball Bill InmonKimballites InmonitesBottom Up Top DownDimensional NormalizedStar Schema 3rd Normal FormEasier for the User More Difficult for the UsersFew JOINs Many JOINsDimension/Facts EntitiesComplicated ETL Not as complicated ETLDifficult to modify structure Easier to adapt Not mutually Exclusive
  29. 29. Star vs. SnowflakeStar SnowflakeER resembles Star ER resembles SnowflakeEasier for the User More Difficult for the UsersFew JOINs Many JOINsFaster Aggregations Slower Aggregations Children with multiple parent tables Normalized Dimensions Snowflake is a variation on a Star, not an alternative
  30. 30. History (ology?)
  31. 31. Dimension Types» 0 – Inserts only, no updates or delete» 1 – Insert and updated to reflect current state» 2 – Slowly Changing Dimension (SCD)- multiple records to indicate different points in time Source Key Value StartDate EndDate 14 Blue 2012-01-01 2012-03-01 14 Green 2012-03-02» 3 – multiple columns to indicate different point in time Source Key Value OldValue EffectiveDate 14 Green Blue 2012-03-02» 4 – current value table and a history table» UNKNOWN values
  32. 32. Date and Time» Date » Fundamental dimensions across all organizations and industries » Allows for trending across dates or periods » 1 row for every date in the years = 365 or 366 row/year » Use your words » WeekDay » EndofMonth » Quarter » FiscalYear?» Time » Not often needed, but becoming more popular » Allows for time based analysis for things like Status » 1 row for every time slice in a day – minutes? Seconds?
  33. 33. Surrogate Keys» New set of keys in the DW» Protects against » Source systems changes » Single key for multiple source systems » New rows that only exist in DW (UNKNOWN) » Tracking over time (SCD)
  34. 34. Physical Data Model
  35. 35. Approach
  36. 36. Null – yay or nay» Same discussion as OLTP with a twist » Purpose of DW is for reporting » Building on top of with : » SSIS » SSAS » Purpose of the Dimension UNKNOWN values» Best practice is to avoid if you can, otherwise document » Some have separate values for UNKNOWN and NOT POPULATED » Default value instead
  37. 37. Aggregates» Minimize number of aggregates while maximize effectiveness» Store or» Can aggregate Facts» Roll-up Dimension hierarchies?» Can still be relational to other tables when necessary
  38. 38. Hierarchies» Example:» Date - Roll up by Month, Quarter or Year Key Day Month Quarter Year 364 30 12 4 2011 365 31 12 4 2011 366 1 1 1 2012 367 2 1 1 2012» Variable depth – Self-referencing» Variable depth with historical – changing surrogate keys – ouch » Track business process separately
  39. 39. Size Matters
  40. 40. Data amount and size» Data Types? » BLOB data? » Identity columns (do you need bigint?)» Data Profiling » Collect source system sizes for data bringing over » Add sizes of new row» Don’t forget index size!!!
  41. 41. Partitioning» Usually lends naturally to partitioning large Fact tables by Date» Larger Dimension tables can be partitioned as well» Sometimes Old (SQL 2000) Partitioning is still better than SQL 2005+ partitioning» Take ETL process into consideration
  42. 42. Archiving» Question: When is big too big?» Answer: When performance impact outweighs need for data availability» Many options: » Backup to tape offline » keep “Archived” DW available» Records Retention – this could be your work product
  43. 43. Performance
  44. 44. Hardware» Remember when the user said “It’s my data and I want it now”?» Buy » Reference Architecture (Fast Track) » Appliances » HP » Enterprise Data Warehouse » Business Decision » Business Data Warehouse » Enterprise Database Consolidation » Dell » PDW» Build » Reference Architecture (Fast Track) » SQLIO » Benchmark
  45. 45. Throughput» Amounts of data » Not all of it will be in memory » Between ETL and reports, SP Cache might not be efficient » Need to tune those disks» Reference Architecture(Fast Track) » Accepts that Procedure cache will stink due to data sizes » Instead small amount of RAM » Requires bandwidth of 400 GB/s per LUN» Materialize data that makes reporting faster!! » More Denormalization » More Aggregations» ReadOnly while not processing ETLs? (switch)
  46. 46. Parallelism» Multiple Data Files » SQL writes proportional fill» Multiple Filegroups » Partitioning scheme » Facts/Dimensions » Tables that are often joined » Big tables » NCIX vs. data» Multiple LUNs » I am not a SAN admin nor play one on TV» Normal SQL performance
  47. 47. Questions and Discussion time!