Data vault: What's Next


Published on

This was a presentation about Data Warehousing, where it's going - covers operational Data Vault. I gave this presentation in 2009 at an Array Conference in the Netherlands.

IF you want to use these slides, then please let me know, and add: "(C) Dan Linstedt, all rights reserved,"

Published in: Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Data vault: What's Next

  1. 1. Data Vault ModelingWHAT’S NEXT???<br />© Dan Linstedt 2009-2012<br />This was a presentation I gave at an Array Conference In the Netherlands, in 2009.<br />
  2. 2. A bit about me…<br />2<br />Author, Inventor, Speaker – and part time photographer…<br />25+ years in the IT industry<br />Worked in DoD, US Gov’t, Fortune 50, and so on…<br />Find out more about the Data Vault:<br /><br /><br />Full profile on<br />10/5/2011<br /><br />
  3. 3. Who’s Using It?<br />10/5/2011<br /><br />3<br />
  4. 4. Featured Word from Our Customer<br />10/5/2011<br /><br />4<br />Jonathan Rice, BSc<br />ING Real Estate/Development International/Operational Support <br />Location HP A.05.071<br />PO Box 90463, 2509 LL The Hague, The Netherlands<br />They are connecting MS SharePoint back-end to Data Vault Modeling EDW.<br />
  5. 5. What’s Changing the Game?<br />Solid State Disk (SSD)<br />Hosted Cloud Computing<br />Column Based Databases<br />Unstructured Information<br />Ontologies / Taxonomies<br />Mining Engines<br />Broad Based Web Services<br />Business Demand for Immediate Answers (pushing real-time)<br />Compliance and Auditability (pushing volume)<br />10/5/2011<br /><br />5<br /><ul><li>Data Visualization
  6. 6. Flash & Silverlight Front-Ends
  7. 7. Analytic functions in Database Engines
  8. 8. Business Rules Engines Melding with ETL and Web Services
  9. 9. DW Appliances</li></li></ul><li>Current EDW + DV Architecture<br />10/5/2011<br /><br />6<br />SOA<br />Enterprise BI Solution<br />Star<br />Schemas<br />(real-time)<br />Sales<br />(batch)<br />DV<br />EDW<br />(batch)<br />Staging<br />Error<br />Marts<br />Finance<br />Contracts<br />Report<br />Collections<br />Business Rules Downstream!<br />(the Lens Filter)<br />
  10. 10. First Change: Business Data Vault<br />10/5/2011<br /><br />7<br />SOA<br />Enterprise BI Solution<br />Star<br />Schemas<br />(real-time)<br />Sales<br />Batch<br />DV<br />EDW<br />BDV<br />EDW<br />Staging<br />Error<br />Marts<br />Finance<br />Contracts<br />Report<br />Collections<br />Business Rules Downstream!<br />(the Lens Filter)<br />
  11. 11. Next Change: Staging Removal<br />10/5/2011<br /><br />8<br />Enterprise BI Solution<br />SOA<br />Write Back<br />(real-time)<br />Star<br />Schemas<br />Sales<br />DV<br />EDW<br />BDV<br />EDW<br />Error<br />Marts<br />Finance<br />Contracts<br />Report<br />Collections<br />Business Rules Downstream!<br />(the Lens Filter)<br />
  12. 12. Next Change: Virtual Marts<br />10/5/2011<br /><br />9<br />Enterprise BI Solution<br />SOA<br />Write Back<br />Virtual <br />Marts<br />& Dynamic Cubes<br />(real-time)<br />Sales<br />Finance<br />DV<br />EDW<br />BDV<br />EDW<br />Contracts<br />Business Rules Downstream!<br />(the Lens Filter)<br />
  13. 13. Next Change: Unstructured Data<br />10/5/2011<br /><br />10<br />Unstructured Data Sets<br />Ontologies/Taxonomies<br />Unstructured <br />Processing Engine<br /><ul><li>Email
  14. 14. Docs
  15. 15. Images
  16. 16. Movies
  17. 17. Sound</li></ul>On-Demand<br />Cubes<br />Joins through LINK Structures<br />Raw Data Vault EDW<br />
  18. 18. A Look at Ontologies<br />10/5/2011<br /><br />11<br />Hierarchies of Data:<br />Synonyms, Homonyms, Antonyms, Related Terms, Definitions, Categorizations, Organizations, Views of the data world<br />Ontologies HOLD THE KEY to understanding/conceptual relevance<br />Ontologies can PIVOT raw data in to many different results<br />
  19. 19. Plateau: Operational Data Warehouse <br />10/5/2011<br /><br />12<br />Operational<br />Metadata<br />Management<br />Operational<br />Applications<br />Master Data<br />Strategic<br />Reports<br />& OLAP<br />& W.SVCS<br />Direct Edits<br />Dynamic <br />Cubes<br />Web Interface (usually)<br />Direct Edits<br />Real-Time<br />Collector<br />SOR<br />Real-Time Data<br />Data Vault EDW<br /><ul><li>Stored
  20. 20. Analyzed / Scored</li></ul>Operational<br />Systems<br />Unstructured<br />Semi-Structured<br />Staging <br />Area<br />Non-S.O.R.<br />Historical Batch Data<br />Non-SOR<br />Batch Data<br />Operational<br />Alerts<br />Operational <br />Systems<br />Strategic<br />Reports<br />& OLAP<br />& W.SVCS<br />Virtual <br />Marts<br />Real-Time<br />Mining<br />Engine<br /><ul><li>Flexible
  21. 21. Accountable
  22. 22. Compliant
  23. 23. Scalable
  24. 24. Normalized
  25. 25. Dynamic
  26. 26. Granular
  27. 27. Historic</li></li></ul><li>Data Vault<br />(EDW)<br />Operational Data Vault<br />10/5/2011<br /><br />13<br />Operational<br />Application<br />Update/Unlock<br />Read/Lock<br />Common Data Access Layer<br />Insert Changes<br />Read Current<br />
  28. 28. Why go Operational Data Vault?<br />Benefits Include<br />Direct access to data<br />Removal of 90% of batch streams, replaced by Transactions<br />Better/faster alerting capabilities<br />Direct control over changes to BI answer sets<br />What are the risks?<br />The Data Vault EDW becomes an ODV, brings with it ALL responsibilities of the Operational System<br />What are the problems?<br />Row Locking<br />Consistent Data Access<br />Data Version Control<br />Security of Data through access points<br />Has it been done before?<br />Yes, Cendant Timeshare Resource Group did it in 2002!<br />10/5/2011<br /><br />14<br />
  29. 29. Plateau: Dynamic Data Warehouse <br />10/5/2011<br /><br />15<br />Dynamically Constructed Links<br />Dynamically Created Hubs<br />Dynamically Altered Satellites<br />Dynamically changed ETL/ELT<br />Dynamically Altered Queries<br />Automatically Updated Cubes<br />= Self Morphing (guided) Data Vault<br />H<br />L<br />DL<br />H<br />L<br />Meta<br />Mining<br />Engine<br />H<br />L<br />H<br />H<br />DL<br />L<br />H<br />
  30. 30. Impacts of Dynamic Data Vaults<br />Business<br />New found knowledge<br />Evolving Data Warehouse models over time<br />Faster Reporting<br />Self-Maintaining Back-Ends<br />Technical<br />Guided Changes to Structures<br />Auto-Adapting Load Routines<br />Auto-Adapting Query Sets<br />Auto-Suggested Star Schemas<br />10/5/2011<br /><br />16<br />
  31. 31. Why go Dynamic?<br />Benefits include:<br />Faster Turn Around Time<br />Auto Configuration of “mundane tasks”<br />Discovery of new relationships (could result in increased revenue)<br />Self Healing Structures<br />How Long before we See It?<br />5 to 7 years out<br />How do we get there?<br />Meta-Mining<br />Use of Ontologies inside the EDW<br />What’s driving us there?<br />Business Users want faster turn around time<br />Business Users want ingestion of Unstructured Data Sets<br />Business Users need more control over their systems<br />10/5/2011<br /><br />17<br />
  32. 32. Where To Learn More<br />The Technical Modeling Book:<br />The Discussion Forums: & events – Data Vault Discussions<br />Contact me: - web - email<br />World wide User Group (Free)<br />18<br />10/5/2011<br /><br />