Your SlideShare is downloading. ×
0
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Data Warehouse Design & Dimensional Modeling
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Data Warehouse Design & Dimensional Modeling

3,725

Published on

At Code Mastery Boston Aaron Lowe, Principal Consultant at Magenic, talks Data Warehouse Design & Dimensional Modeling

At Code Mastery Boston Aaron Lowe, Principal Consultant at Magenic, talks Data Warehouse Design & Dimensional Modeling

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,725
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
91
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Data Warehouse Design& Dimensional ModelingAaron LowePrincipal Consultant@Vendoran@SQLFriends
  • 2. Who am I» Aaron Lowe » Husband » Father of 5 » Principal Consultant at Magenic » Working with SQL Server since 1998, version 6.5 » MCITP 2005 and 2008 » Co-organizer of SQLSaturday Chicago » Masters in Information Systems Management » www.aaronlowe.net / @Vendoran » sqlfriends.org / @SQLFriends
  • 3. Data, Data everywhere, but not a drop ofInformation http://www.flickr.com/photos/walkingsf/5993167874/
  • 4. The Data Person http://www.flickr.com/photos/tantek/1360323838/
  • 5. How can we get more out of our data? http://www.flickr.com/photos/danahlongley/4472897115/
  • 6. Leverage data to provide business insight http://www.flickr.com/photos/juhansonin/4646203016/
  • 7. Create a new Data Model in a Data Warehouse
  • 8. Why a new Data Model?
  • 9. What do we need?
  • 10. Information – not just data» Collecting data » Log Files » Clicks » How long? » How much?» Prediction? » How Target Figured out Teen was pregnant - http://www.forbes.com/sites/kashmirhill/2012/02/16/how-target-figured- out-a-teen-girl-was-pregnant-before-her-father-did/ » The Numerati - http://www.amazon.com/The-Numerati-Stephen- Baker/dp/B003TO6G20/ - published 2009!
  • 11. Relate data from multiple systems» The purpose of a data warehouse is to house standardized, structured, consistent, integrated, correct, cleansed and timely data, extracted from various operational systems in an organization» True picture of the business process» Source Systems » Financial – AR/AP » Sales » CRM » HR » Application
  • 12. Fast» It’s my information and I want it Now!» Empower Users» Exploratory» Reads» Large datasets
  • 13. Why won’t existing models work?
  • 14. What are they designed for?» Operational » Preservation of data integrity » Speed of recording of business transactions » Often Many tables» To free the collection of relations from undesirable insertion, update and deletion dependencies;» To reduce the need for restructuring the collection of relations, as new types of data are introduced, and thus increase the life span of application programs;» To make the relational model more informative to users;» To make the collection of relations neutral to the query statistics, where these statistics are liable to change as time goes by.» —E.F. Codd, "Further Normalization of the Data Base Relational Model"
  • 15. Consistent» Partial data across » Have the sale in the sales system » Represented in the inventory system » Don’t have the $ in the financial system yet» Deleted on sources » Removed transactions » Archive » Legally destroy records can remove work product» Incomplete data on source » Changes over time
  • 16. Silo’d» How do we get the entire picture?» Example: » Cost of Sales? » Sales system – Sale Price » Marketing System – $$ spent on Marketing » Inventory System – $$ spent on inventory » HR System – $$ spent on Employee » IT Systems – $$ spent on Infrastructure
  • 17. What will work? http://www.flickr.com/photos/d-y-f/2870942257/
  • 18. Designed for Users» De-normalized » Fast Reads » Fast Reports » Limited JOINs» Information » Scheduled » On Demand » Exploratory» Information » Cross Functional » The more the better!
  • 19. Inter-related data» Specifications for my Current Data Warehouse http://www.flickr.com/photos/ross_goodman/3276964270/
  • 20. Independent from Operational» Operational systems change » Data will outlive Application » Crashes » Upgrades » Breaking changes» Single Source of truth
  • 21. Logical Data Model http://www.flickr.com/photos/doctorlizardo/6812846803/
  • 22. Terminology http://www.flickr.com/photos/doctorlizardo/6809564765/
  • 23. Metadata Management» Business metadata » What’s out there? » Identify/Define » Overloaded terms » What is a customer?» Process metadata » DW process operations » Asses system status » Investigate problems» Technical metadata » Tables » Fields » Datatypes
  • 24. Dimensions and FactsDimensions FactsThing/Objects Measurements/EventsNouns VerbsWide but short Skinny but longRows can exist independently Rows cannot exist independentlyDescriptive Mostly Numeric and Additive “By” words – FACT by Dimension Quantity Ordered by Product by Customer by Date
  • 25. Grain• Level of detail• What is needed to meet business requirements?• What is possible to collect?• How do you describe it?• One row per X where X is the business event • One row per customer call • One row per time sheet entry • One row per employee status change • One row per order line item http://www.flickr.com/photos/frederikvanroest/3842334310/
  • 26. Methodology http://www.flickr.com/photos/doctorlizardo/6812847973/
  • 27. Requirements – business focused» “Must embrace the goal of enhancing business value as the primary purpose.” – Kimball» “If your job is BI and you speak mostly to technical people all day, you are doing it wrong. Focus on first word - BUSINESS.” – Whitney Weaver (former Magenicon)» Never ask “What do you want in the data warehouse?” Only one right answer - “Everything.”» Ask questions that help you learn what the end user does
  • 28. Kimball v. InmonRalph Kimball Bill InmonKimballites InmonitesBottom Up Top DownDimensional NormalizedStar Schema 3rd Normal FormEasier for the User More Difficult for the UsersFew JOINs Many JOINsDimension/Facts EntitiesComplicated ETL Not as complicated ETLDifficult to modify structure Easier to adapt Not mutually Exclusive
  • 29. Star vs. SnowflakeStar SnowflakeER resembles Star ER resembles SnowflakeEasier for the User More Difficult for the UsersFew JOINs Many JOINsFaster Aggregations Slower Aggregations Children with multiple parent tables Normalized Dimensions Snowflake is a variation on a Star, not an alternative http://www.flickr.com/photos/wandrus/6283157711/
  • 30. History (ology?) http://www.flickr.com/photos/doctorlizardo/6809564335/
  • 31. Dimension Types» 0 – Inserts only, no updates or delete» 1 – Insert and updated to reflect current state» 2 – Slowly Changing Dimension (SCD)- multiple records to indicate different points in time Source Key Value StartDate EndDate 14 Blue 2012-01-01 2012-03-01 14 Green 2012-03-02» 3 – multiple columns to indicate different point in time Source Key Value OldValue EffectiveDate 14 Green Blue 2012-03-02» 4 – current value table and a history table» UNKNOWN values
  • 32. Date and Time» Date » Fundamental dimensions across all organizations and industries » Allows for trending across dates or periods » 1 row for every date in the years = 365 or 366 row/year » Use your words » WeekDay » EndofMonth » Quarter » FiscalYear?» Time » Not often needed, but becoming more popular » Allows for time based analysis for things like Status » 1 row for every time slice in a day – minutes? Seconds?
  • 33. Surrogate Keys» New set of keys in the DW» Protects against » Source systems changes » Single key for multiple source systems » New rows that only exist in DW (UNKNOWN) » Tracking over time (SCD)
  • 34. Physical Data Model http://www.flickr.com/photos/flying_cloud/2667218708/
  • 35. Approach http://www.flickr.com/photos/7506006@N07/7021456259/
  • 36. Null – yay or nay» Same discussion as OLTP with a twist » Purpose of DW is for reporting » Building on top of with : » SSIS » SSAS » Purpose of the Dimension UNKNOWN values» Best practice is to avoid if you can, otherwise document » Some have separate values for UNKNOWN and NOT POPULATED » Default value instead
  • 37. Aggregates» Minimize number of aggregates while maximize effectiveness» Store or» Can aggregate Facts» Roll-up Dimension hierarchies?» Can still be relational to other tables when necessary
  • 38. Hierarchies» Example:» Date - Roll up by Month, Quarter or Year Key Day Month Quarter Year 364 30 12 4 2011 365 31 12 4 2011 366 1 1 1 2012 367 2 1 1 2012» Variable depth – Self-referencing» Variable depth with historical – changing surrogate keys – ouch » Track business process separately
  • 39. Size Matters http://starwars.wikia.com/wiki/Rancor?image=Rancor-jpg
  • 40. Data amount and size» Data Types? » BLOB data? » Identity columns (do you need bigint?)» Data Profiling » Collect source system sizes for data bringing over » Add sizes of new row» Don’t forget index size!!!
  • 41. Partitioning» Usually lends naturally to partitioning large Fact tables by Date» Larger Dimension tables can be partitioned as well» Sometimes Old (SQL 2000) Partitioning is still better than SQL 2005+ partitioning» Take ETL process into consideration
  • 42. Archiving» Question: When is big too big?» Answer: When performance impact outweighs need for data availability» Many options: » Backup to tape offline » keep “Archived” DW available» Records Retention – this could be your work product
  • 43. Performance http://www.flickr.com/photos/elfidomx/6026943114/
  • 44. Hardware» Remember when the user said “It’s my data and I want it now”?» Buy » Reference Architecture (Fast Track) » Appliances » HP » Enterprise Data Warehouse » Business Decision » Business Data Warehouse » Enterprise Database Consolidation » Dell » PDW» Build » Reference Architecture (Fast Track) » SQLIO » Benchmark
  • 45. Throughput» Amounts of data » Not all of it will be in memory » Between ETL and reports, SP Cache might not be efficient » Need to tune those disks» Reference Architecture(Fast Track) » Accepts that Procedure cache will stink due to data sizes » Instead small amount of RAM » Requires bandwidth of 400 GB/s per LUN» Materialize data that makes reporting faster!! » More Denormalization » More Aggregations» ReadOnly while not processing ETLs? (switch)
  • 46. Parallelism» Multiple Data Files » SQL writes proportional fill» Multiple Filegroups » Partitioning scheme » Facts/Dimensions » Tables that are often joined » Big tables » NCIX vs. data» Multiple LUNs » I am not a SAN admin nor play one on TV» Normal SQL performance
  • 47. Questions and Discussion time!

×