Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Data vault: What's Next


Published on

This was a presentation about Data Warehousing, where it's going - covers operational Data Vault. I gave this presentation in 2009 at an Array Conference in the Netherlands.

IF you want to use these slides, then please let me know, and add: "(C) Dan Linstedt, all rights reserved,"

Published in: Business
  • Be the first to comment

Data vault: What's Next

  1. 1. Data Vault ModelingWHAT’S NEXT???<br />© Dan Linstedt 2009-2012<br />This was a presentation I gave at an Array Conference In the Netherlands, in 2009.<br />
  2. 2. A bit about me…<br />2<br />Author, Inventor, Speaker – and part time photographer…<br />25+ years in the IT industry<br />Worked in DoD, US Gov’t, Fortune 50, and so on…<br />Find out more about the Data Vault:<br /><br /><br />Full profile on<br />10/5/2011<br /><br />
  3. 3. Who’s Using It?<br />10/5/2011<br /><br />3<br />
  4. 4. Featured Word from Our Customer<br />10/5/2011<br /><br />4<br />Jonathan Rice, BSc<br />ING Real Estate/Development International/Operational Support <br />Location HP A.05.071<br />PO Box 90463, 2509 LL The Hague, The Netherlands<br />They are connecting MS SharePoint back-end to Data Vault Modeling EDW.<br />
  5. 5. What’s Changing the Game?<br />Solid State Disk (SSD)<br />Hosted Cloud Computing<br />Column Based Databases<br />Unstructured Information<br />Ontologies / Taxonomies<br />Mining Engines<br />Broad Based Web Services<br />Business Demand for Immediate Answers (pushing real-time)<br />Compliance and Auditability (pushing volume)<br />10/5/2011<br /><br />5<br /><ul><li>Data Visualization
  6. 6. Flash & Silverlight Front-Ends
  7. 7. Analytic functions in Database Engines
  8. 8. Business Rules Engines Melding with ETL and Web Services
  9. 9. DW Appliances</li></li></ul><li>Current EDW + DV Architecture<br />10/5/2011<br /><br />6<br />SOA<br />Enterprise BI Solution<br />Star<br />Schemas<br />(real-time)<br />Sales<br />(batch)<br />DV<br />EDW<br />(batch)<br />Staging<br />Error<br />Marts<br />Finance<br />Contracts<br />Report<br />Collections<br />Business Rules Downstream!<br />(the Lens Filter)<br />
  10. 10. First Change: Business Data Vault<br />10/5/2011<br /><br />7<br />SOA<br />Enterprise BI Solution<br />Star<br />Schemas<br />(real-time)<br />Sales<br />Batch<br />DV<br />EDW<br />BDV<br />EDW<br />Staging<br />Error<br />Marts<br />Finance<br />Contracts<br />Report<br />Collections<br />Business Rules Downstream!<br />(the Lens Filter)<br />
  11. 11. Next Change: Staging Removal<br />10/5/2011<br /><br />8<br />Enterprise BI Solution<br />SOA<br />Write Back<br />(real-time)<br />Star<br />Schemas<br />Sales<br />DV<br />EDW<br />BDV<br />EDW<br />Error<br />Marts<br />Finance<br />Contracts<br />Report<br />Collections<br />Business Rules Downstream!<br />(the Lens Filter)<br />
  12. 12. Next Change: Virtual Marts<br />10/5/2011<br /><br />9<br />Enterprise BI Solution<br />SOA<br />Write Back<br />Virtual <br />Marts<br />& Dynamic Cubes<br />(real-time)<br />Sales<br />Finance<br />DV<br />EDW<br />BDV<br />EDW<br />Contracts<br />Business Rules Downstream!<br />(the Lens Filter)<br />
  13. 13. Next Change: Unstructured Data<br />10/5/2011<br /><br />10<br />Unstructured Data Sets<br />Ontologies/Taxonomies<br />Unstructured <br />Processing Engine<br /><ul><li>Email
  14. 14. Docs
  15. 15. Images
  16. 16. Movies
  17. 17. Sound</li></ul>On-Demand<br />Cubes<br />Joins through LINK Structures<br />Raw Data Vault EDW<br />
  18. 18. A Look at Ontologies<br />10/5/2011<br /><br />11<br />Hierarchies of Data:<br />Synonyms, Homonyms, Antonyms, Related Terms, Definitions, Categorizations, Organizations, Views of the data world<br />Ontologies HOLD THE KEY to understanding/conceptual relevance<br />Ontologies can PIVOT raw data in to many different results<br />
  19. 19. Plateau: Operational Data Warehouse <br />10/5/2011<br /><br />12<br />Operational<br />Metadata<br />Management<br />Operational<br />Applications<br />Master Data<br />Strategic<br />Reports<br />& OLAP<br />& W.SVCS<br />Direct Edits<br />Dynamic <br />Cubes<br />Web Interface (usually)<br />Direct Edits<br />Real-Time<br />Collector<br />SOR<br />Real-Time Data<br />Data Vault EDW<br /><ul><li>Stored
  20. 20. Analyzed / Scored</li></ul>Operational<br />Systems<br />Unstructured<br />Semi-Structured<br />Staging <br />Area<br />Non-S.O.R.<br />Historical Batch Data<br />Non-SOR<br />Batch Data<br />Operational<br />Alerts<br />Operational <br />Systems<br />Strategic<br />Reports<br />& OLAP<br />& W.SVCS<br />Virtual <br />Marts<br />Real-Time<br />Mining<br />Engine<br /><ul><li>Flexible
  21. 21. Accountable
  22. 22. Compliant
  23. 23. Scalable
  24. 24. Normalized
  25. 25. Dynamic
  26. 26. Granular
  27. 27. Historic</li></li></ul><li>Data Vault<br />(EDW)<br />Operational Data Vault<br />10/5/2011<br /><br />13<br />Operational<br />Application<br />Update/Unlock<br />Read/Lock<br />Common Data Access Layer<br />Insert Changes<br />Read Current<br />
  28. 28. Why go Operational Data Vault?<br />Benefits Include<br />Direct access to data<br />Removal of 90% of batch streams, replaced by Transactions<br />Better/faster alerting capabilities<br />Direct control over changes to BI answer sets<br />What are the risks?<br />The Data Vault EDW becomes an ODV, brings with it ALL responsibilities of the Operational System<br />What are the problems?<br />Row Locking<br />Consistent Data Access<br />Data Version Control<br />Security of Data through access points<br />Has it been done before?<br />Yes, Cendant Timeshare Resource Group did it in 2002!<br />10/5/2011<br /><br />14<br />
  29. 29. Plateau: Dynamic Data Warehouse <br />10/5/2011<br /><br />15<br />Dynamically Constructed Links<br />Dynamically Created Hubs<br />Dynamically Altered Satellites<br />Dynamically changed ETL/ELT<br />Dynamically Altered Queries<br />Automatically Updated Cubes<br />= Self Morphing (guided) Data Vault<br />H<br />L<br />DL<br />H<br />L<br />Meta<br />Mining<br />Engine<br />H<br />L<br />H<br />H<br />DL<br />L<br />H<br />
  30. 30. Impacts of Dynamic Data Vaults<br />Business<br />New found knowledge<br />Evolving Data Warehouse models over time<br />Faster Reporting<br />Self-Maintaining Back-Ends<br />Technical<br />Guided Changes to Structures<br />Auto-Adapting Load Routines<br />Auto-Adapting Query Sets<br />Auto-Suggested Star Schemas<br />10/5/2011<br /><br />16<br />
  31. 31. Why go Dynamic?<br />Benefits include:<br />Faster Turn Around Time<br />Auto Configuration of “mundane tasks”<br />Discovery of new relationships (could result in increased revenue)<br />Self Healing Structures<br />How Long before we See It?<br />5 to 7 years out<br />How do we get there?<br />Meta-Mining<br />Use of Ontologies inside the EDW<br />What’s driving us there?<br />Business Users want faster turn around time<br />Business Users want ingestion of Unstructured Data Sets<br />Business Users need more control over their systems<br />10/5/2011<br /><br />17<br />
  32. 32. Where To Learn More<br />The Technical Modeling Book:<br />The Discussion Forums: & events – Data Vault Discussions<br />Contact me: - web - email<br />World wide User Group (Free)<br />18<br />10/5/2011<br /><br />