Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1 of 21

Biml Academy 2 - Lesson 5: Importing source metadata into Biml

3

Share

Download to read offline

Biml Academy 2 - Lesson 5: Importing source metadata into Biml (Presented by Cathrine Wilhelmsen on August 15th 2016)

Biml Academy 2 - Lesson 5: Importing source metadata into Biml

  1. 1. Biml Academy 2 Importing, Staging, and Loading Data with Biml Andy Leonard · Scott Currie · Cathrine Wilhelmsen
  2. 2. Biml Academy 2 Lesson 5: Importing source metadata into Biml (Cathrine) Lesson 6: Staging data with Biml (Andy) Lesson 7: Metadata-driven staging pattern with Biml (Andy) Lesson 8: BimlFlex for Data Vault data warehouses (Scott) Lesson 9: Biml Q&A with several BimlHeroes!
  3. 3. Cathrine Wilhelmsen @cathrinew cathrinew.net Data Warehouse Architect Business Intelligence Developer
  4. 4. Lesson 5: Importing source metadata into Biml
  5. 5. Methods for importing metadata ImportTableNodes ImportDB GetDatabaseSchema
  6. 6. Methods for importing metadata ImportTableNodes Import tables and views in one schema Filter tables using LIKE % wildcard syntax
  7. 7. Methods for importing metadata ImportDB Import all schemas, tables and views Filter using LIKE % wildcard syntax
  8. 8. Methods for importing metadata GetDatabaseSchema Import all schemas, tables and views Filter using collections
  9. 9. Why ImportDB and GetDatabaseSchema? ImportDB / ImportTableNodes • SELECT * on each imported table • SELECT to get defaults, indexes etc. • Using SchemaOnly mode • Doesn't actually run the query, but returns schema for the result set GetDatabaseSchema • Figures out target database type (SQL Server, Oracle, Teradata, DB2, PostgreSQL, Ingres, OpenEdge etc.) • Queries the relevant information schema for that database type A few queries for the entire importMultiple queries per table
  10. 10. But why keep both if one is faster? Support for legacy scripts :) They can return different results – where both are valid/correct Different data type mapping defaults in different systems (Bulk Copy, SSIS and Excel are all different) Biml supports the two most commonly used approaches
  11. 11. Methods and parameters ImportTableNodes(schema, tableFilter) ImportTableNodes(schema, tableFilter, importOptions) ImportDB() ImportDB(schemaFilter, tableFilter) ImportDB(schemaFilter, tableFilter, importOptions) GetDatabaseSchema() GetDatabaseSchema(importOptions) GetDatabaseSchema(includedSchemas, includedTables, importOptions)
  12. 12. Methods and parameters: string value ImportTableNodes(schema, tableFilter) ImportTableNodes(schema, tableFilter, importOptions) ImportDB() ImportDB(schemaFilter, tableFilter) ImportDB(schemaFilter, tableFilter, importOptions) GetDatabaseSchema() GetDatabaseSchema(importOptions) GetDatabaseSchema(includedSchemas, includedTables, importOptions)
  13. 13. Methods and parameters: string filter ImportTableNodes(schema, tableFilter) ImportTableNodes(schema, tableFilter, importOptions) ImportDB() ImportDB(schemaFilter, tableFilter) ImportDB(schemaFilter, tableFilter, importOptions) GetDatabaseSchema() GetDatabaseSchema(importOptions) GetDatabaseSchema(includedSchemas, includedTables, importOptions)
  14. 14. Methods and parameters: collections ImportTableNodes(schema, tableFilter) ImportTableNodes(schema, tableFilter, importOptions) ImportDB() ImportDB(schemaFilter, tableFilter) ImportDB(schemaFilter, tableFilter, importOptions) GetDatabaseSchema() GetDatabaseSchema(importOptions) GetDatabaseSchema(includedSchemas, includedTables, importOptions)
  15. 15. Methods and parameters: ImportOptions ImportTableNodes(schema, tableFilter) ImportTableNodes(schema, tableFilter, importOptions) ImportDB() ImportDB(schemaFilter, tableFilter) ImportDB(schemaFilter, tableFilter, importOptions) GetDatabaseSchema() GetDatabaseSchema(importOptions) GetDatabaseSchema(includedSchemas, includedTables, importOptions)
  16. 16. ImportOptions specify objects to exclude Don't exclude anything Views Identity Specifications Primary Keys Foreign Keys Unique Keys Indexes Column Defaults Check Constraints ImportOptions.None ImportOptions.ExcludeViews ImportOptions.ExcludeIdentity ImportOptions.ExcludePrimaryKey ImportOptions.ExcludeForeignKey ImportOptions.ExcludeUniqueKey ImportOptions.ExcludeIndex ImportOptions.ExcludeColumnDefault ImportOptions.ExcludeCheckConstraint
  17. 17. Demo time!
  18. 18. Additional methods for importing metadata GenerateSchemaNode Returns AstSchemaNode GenerateSchemaNodes Returns IEnumerable<AstSchemaNode>
  19. 19. Additional methods for importing metadata GenerateTableNode Returns AstTableNode GenerateTableNodes Returns IEnumerable<AstTableNode>
  20. 20. Demo time!
  21. 21. Biml Academy 2 Lesson 5: Importing source metadata into Biml (Cathrine) Lesson 6: Staging data with Biml (Andy) Lesson 7: Metadata-driven staging pattern with Biml (Andy) Lesson 8: BimlFlex for delivering Data Vault data warehouses (Scott) Lesson 9: Biml Q&A with several BimlHeroes!

×