PDI data vault framework #pcmams 2012
Upcoming SlideShare
Loading in...5

PDI data vault framework #pcmams 2012



Presentation given by Edwin Weber at #pcmams 2012

Presentation given by Edwin Weber at #pcmams 2012



Total Views
Views on SlideShare
Embed Views



3 Embeds 6

http://www.docshut.com 3
http://www.linkedin.com 2
https://twitter.com 1



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

PDI data vault framework #pcmams 2012 PDI data vault framework #pcmams 2012 Presentation Transcript

  • Introductionneacweber@gmail.com
  • Data Vault DefinitionThe Data Vault is a detail oriented, historical tracking and uniquelylinked set of normalized tables that support one or more functionalareas of business. It is a hybrid approach encompassing the bestof breed between 3rd normal form (3NF) and star schema. Thedesign is flexible, scalable, consistent and adaptable to the needsof the enterprise. It is a data model that is architected specificallyto meet the needs of enterprise data warehouses.Source: Dan Linstedthttp://www.tdan.com/view-articles/5054/
  • Data Vault Building Blocks different sources/rate of changeSource: Dan Linstedthttp://www.slideshare.net/dlinstedt/introduction-to-data-vault-dama-oregon-2012
  • Data Vault Fundamentals: HubSource: data-vault-modeling-guideGENESEE ACADEMY, LLC, Hans Hultgren
  • Data Vault Fundamentals: LinkSource: data-vault-modeling-guideGENESEE ACADEMY, LLC, Hans Hultgren
  • Data Vault Fundamentals: Satellite Source: data-vault-modeling-guide GENESEE ACADEMY, LLC, Hans Hultgren
  • Data Vault Fundamentals: ModelSource: data-vault-modeling-guideGENESEE ACADEMY, LLC, Hans Hultgren
  • Data Vault ETLMany objects to load, standardized proceduresThis screams for a generic solution!I dont want to: throw ETL tool away and code it all myself manage too many ETL objects connect similar columns in mappings by handI do want to: generate ETL (Kettle) objects? No Take it one step further: theres only 1 parameterised hub load object. Dont need to know xml structure of PDI objects
  • ToolsOperating System Database Virtualization Data Integration ProductivitySql Development Version Control
  • Place of framework in architecture Files MySQL ETL: Kettle MySQL DBMS ETL Data Data ETL Vault Vault CSV Frame Files work ERP Staging Central DWH & Area Data MartsSources ETL Process Data Warehouse EUL
  • What has to be taken care of?Data Vault designed and implemented in databaseStaging tables and loading procedures in place(can also be generic, we use PDI Metadata Injection step for loadingfiles)Mapping from source to Data Vault specified(now in an Excel sheet)
  • Framework componentsPDI repository (file based), jobs and transformationsConfiguration files:kettle.propertiesshared.xmlrepositories.xmlExcel sheet that contains the specificationsMySQL database for metadataVirtual machine with Ubuntu 12.04 Server
  • Design decisionsUpdateable views with generic column names(MySQL more lenient than PostgreSQL)Compare satellite attributes via string comparison(concatenate all columns, with | (pipe) as delimiter)inject the metadata using Kettle parametersGenerate and use an error table for each Data Vaulttable
  • Metadata tablesAll have history tables
  • Metadata in Excel Data Vault connections source systems source tables
  • Metadata in Excel (hub + sat) x 200 (max)
  • Metadata in Excel (link) x 10link attributes
  • Metadata in Excel (link satellite) x 10 x5 x 200 (max)
  • Last seen dateapplicable for hubs and linksexisting hubs and links: update last_seen_dts!
  • Link validity satelliteLink has business key: not all hub ids
  • Loading the metadata
  • design errorsChecks to avoid debugging:(compares design metadata with Data Vault DB information_schema) hubs, links, satellites that dont exist in the DV key columns that do not exist in the DV missing connection data (source db) missing attribute columns
  • A complete run
  • Metadata needed for a hubnamekey columnbusiness key columnsource tablesource table business key column(can be expression, e.g. concatenate for composite key)
  • Job for hub
  • Transformation for hub
  • Metadata needed for a linknamekey columnfor each hub (maximum 10, can be a ref-table) hub name column name for the hub key in the link (roles!) column in the source table → business key of hublink attributes (part of key, no hub, maximum 5)link validity satellite needed?last seen date needed?source table
  • Job for link
  • Transformation for link Last seen?Lookup hubs Remove columns not in link Run table needed for validity sat ?
  • Metadata needed for a hub satellite name key column hub name column in the source table → business key of hub for each attribute (maximum 200) source column target column source table
  • Job for hub satellite
  • Transformation for hub satellite
  • Metadata needed for a link satellitenamekey columnlink namefor each hub of the link:column in the source table → business key of hubfor each key attribute: source columnfor each attribute: source column → target columnsource table
  • Job for link satellite
  • Transformation for link satellite
  • Executing in a loop ..
  • .. and parallel
  • LoggingCustom logging PDI loggingConfiguring log tablesfor concurrent access
  • Version Control: PDI objects
  • Version Control: database objects
  • Some points of interestEasy to make mistake in design sheetGeneric → a bit harder to maintain and debugApplication/tool to maintain metadata?Data Vault generators (e.g. Quipu)?Spinoff using Informatica and Oracle: Sander RobijnsThanks to: Jos van Dongen Kasper de Graaf
  • Sourceforge!