Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018)

3,035 views

Published on

Level Up Your Biml: Best Practices and Coding Techniques (Presented at PASS Summit on November 8th, 2018)

Published in: Data & Analytics
  • Be the first to comment

Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018)

  1. 1. Level Up Your Biml: Best Practices and Coding Techniques Cathrine Wilhelmsen, Inmeta @cathrinew | cathrinew.net
  2. 2. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Abstract Is your Biml solution starting to remind you of a bowl of tangled spaghetti code? Good! That means you are solving real problems while saving a lot of time. The next step is to make sure that your solution does not grow too complex and confusing - you do not want to waste all that saved time on future maintenance! Attend this session for an overview of Biml best practices and coding techniques. Learn how to improve and simplify your solution by using some common and some lesser-known Biml features. If standard Biml is not enough, you can implement custom logic by creating your own C# classes and methods. Finally, see how to bring everything together in an example project for creating and loading a data mart with facts and dimensions. Start improving your code today and level up your Biml in no time!
  3. 3. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) @cathrinew cathrinew.net
  4. 4. Free online webinar events Free 1-day local training events Local user groups around the world Online special interest user groups Business analytics training Get involved Explore everything PASS has to offer Free Online Resources Newsletters PASS.org
  5. 5. Download the GuideBook App and search: PASS Summit 2018 Follow the QR code link displayed on session signage throughout the conference venue and in the program guide Session evaluations Your feedback is important and valuable. Go to passSummit.com 3 Ways to Access: Submit by 5pm Friday, November 16th to win prizes.
  6. 6. Please silence cell phones
  7. 7. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
  8. 8. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Biml Tools Staging Biml for Beginners
  9. 9. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Code ModelManage Level Up Your Biml
  10. 10. Biml Code Management
  11. 11. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Don't Repeat Yourself Refactor and centralize code Update once for all projects 1. Include Files 2. CallBimlScript 3. C# Code
  12. 12. Include Files Automated Copy & Paste
  13. 13. Include Files Works like an automated Copy & Paste <#@ include file="CommonCode.biml" #> Code from included file replaces this include directive
  14. 14. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Include Files
  15. 15. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Include Files
  16. 16. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Include Files
  17. 17. CallBimlScript Parameterized Include with Logic
  18. 18. CallBimlScript Parameterized include (like a stored procedure) CallBimlScript file specifies accepted parameters: <#@ property name="Parameter" type="String" #> Main file calls and passes parameters: <#=CallBimlScript("CommonCode.biml", Parameter)#>
  19. 19. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) CallBimlScript
  20. 20. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) CallBimlScript
  21. 21. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) CallBimlScript
  22. 22. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) CallBimlScript
  23. 23. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) CallBimlScript
  24. 24. CallBimlScriptWithOutput Return Code and Object
  25. 25. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) CallBimlScriptWithOutput CallBimlScript file specifies accepted parameters: <#@ property name="ProjectName" type="String" #> CallBimlScriptWithOutput file also specifies object: <# CustomOutput.Project = metadata.Project; #>
  26. 26. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) CallBimlScriptWithOutput Main file defines, passes, and uses output object in addition to regular parameters: <# dynamic project; #> <#=CallBimlScriptWithOutput("Metadata.biml", out project, "Staging Project")#> <#=project.SourceConnection#>
  27. 27. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) CallBimlScriptWithOutput
  28. 28. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) CallBimlScriptWithOutput
  29. 29. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) CallBimlScriptWithOutput
  30. 30. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) CallBimlScriptWithOutput
  31. 31. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) CallBimlScriptWithOutput use CustomOutput
  32. 32. C# Code Custom Classes and Methods
  33. 33. C# Code: Inline Blocks <#+ ... #> <#+ #>
  34. 34. C# Code: Included Files <#@ include file="CodeBlock.biml" #> <#+ #>
  35. 35. C# Code: Separate Code Files <#@ code file="Code.cs" #>
  36. 36. EXAMPLE Refactoring Biml Code
  37. 37. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Two Approaches to Refactoring 1. Identify Identical Code → Include Files 2. Identify Patterns → CallBimlScript
  38. 38. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Log Package Execution (Start) Log Package Execution (Stop)
  39. 39. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Log Package Execution (Start) Log Package Execution (Stop)
  40. 40. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Two Approaches to Refactoring 1. Identify Identical Code → Include Files 2. Identify Patterns → CallBimlScript
  41. 41. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Log Package Execution Log Package Execution
  42. 42. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Log Package Execution Log Package Execution
  43. 43. DEMO Auditing Framework
  44. 44. Don't Repeat Yourself If you reuse code more than 3 times, refactor :)
  45. 45. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) +10000 XP Congratulations! You reached level 60!
  46. 46. Practical Biml Coding
  47. 47. Annotations and ObjectTags Store and Pass Metadata Between Biml Files
  48. 48. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Annotations and ObjectTags Store metadata by attaching tags to Biml objects Annotations: String values ObjectTags: Objects Higher tier files can get tags from lower tier files
  49. 49. Annotations Create annotations directly in Biml: <OleDbConnection Name="Dest"> <Annotations> <Annotation Tag="Schema">stg</Annotation> </Annotations> </OleDbConnection> Use annotations in BimlScript: RootNode.OleDbConnections["Dest"].GetTag("Schema");
  50. 50. ObjectTags Create ObjectTags in BimlScript: RootNode.ObjectTag["Filter"] = new List<string>{"Product","ProductCategory"}; Use ObjectTags in BimlScript: var FilteredTables = RootNode.ObjectTag["Filter"];
  51. 51. LINQ Query and Filter Collections
  52. 52. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) LINQ One language to query: SQL Server Databases XML Documents Datasets Collections Two ways to write queries: SQL-ish Syntax Methods
  53. 53. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) LINQ Methods Sort OrderBy, Reverse Filter Where, OfType Get First, Last Check Collections All, Any, Contains Set Operations Except, Intersect Project Collections Select, SelectMany
  54. 54. LINQ Methods Example var firstConnection = RootNode.Connections.First() foreach (var table in RootNode.Tables.Where(…)) if (RootNode.Packages.Any(…))
  55. 55. LINQ Methods Example foreach (var table in RootNode.Tables.Where(…)) if (RootNode.Packages.Any(…)) But what do you put in here?
  56. 56. Lambda Expressions "A lambda expression is an anonymous function that you can use to create delegates or expression tree types"
  57. 57. Lambda Expressions "A lambda expression is an anonymous function that you can use to create delegates or expression tree types"
  58. 58. Lambda Expressions column => column.Name == "ColumnID"
  59. 59. Lambda Expressions column => column.Name == "ColumnID" The arrow ("goes to") is the lambda operator
  60. 60. Lambda Expressions column => column.Name == "ColumnID" Input parameter is on the left side
  61. 61. Lambda Expressions column => column.Name == "ColumnID" Expression is on the right side
  62. 62. Lambda Expressions column => column.Name == "ColumnID"
  63. 63. LINQ and Lambda Chain LINQ Methods and use Lambda Expressions for simple and powerful querying of collections: .Where(table => table.Schema.Name == "Production") .OrderBy(table => table.Name)
  64. 64. Column Methods Generate SQL from Biml
  65. 65. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) GetColumnList() GetColumnAssignmentList() GetColumnComparisonList()
  66. 66. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) GetColumnList Get columns for SELECT: [Column1], [Column2], [Column3]
  67. 67. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) GetColumnAssignmentList Get columns for UPDATE … SET: [left].[Column1] = [right].[Column1] ,[left].[Column2] = [right].[Column2] ,[left].[Column3] = [right].[Column3]
  68. 68. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) GetColumnComparisonList Get columns for JOIN … ON: [left].[Column1] = [right].[Column1] AND [left].[Column2] = [right].[Column2] AND [left].[Column3] = [right].[Column3]
  69. 69. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Column Methods and Lambda Use Lambda Expressions to filter columns: GetColumnList(column => column.IsUsedInPrimaryKey)
  70. 70. C# Extension Methods Custom LINQ-style!
  71. 71. Extension Methods "Make it look like the method belongs to an object instead of a helper class"
  72. 72. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) From this… public static string GetName(AstTableNode table) {...}
  73. 73. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) …to this public static string GetName(this AstTableNode table) {...}
  74. 74. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) From this… HelperClass.GetName(table)
  75. 75. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) …to this table.GetName()
  76. 76. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) +100000 XP Congratulations! You reached level 80!
  77. 77. Facts and Dimensions
  78. 78. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Solution Overview
  79. 79. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Solution Overview Dimensions SCD2 Hybrid SCD1 SCD2 Facts Trans- actional
  80. 80. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Facts Source View Lookup Dimension Keys Insert New Rows Load Fact Get Latest Key
  81. 81. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Dimensions Source View Lookup Existing Rows Insert New Rows Insert Updated Rows Truncate Update Table Load Dimension Update Rows
  82. 82. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Solution Overview Data Warehouse Facts Dims Update Staging Source Views
  83. 83. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Solution Process 1. Model Facts and Dimensions 2. Generate SQL Scripts 3. Modify Source Views 4. Generate SSIS Packages
  84. 84. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Solution Process 1. Model Facts and Dimensions 2. Generate SQL Scripts 3. Modify Source Views 4. Generate SSIS Packages
  85. 85. 80 / 20 Automated Manual
  86. 86. Facts and Dimensions Metadata Model in Excel
  87. 87. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Metadata Model Connections Projects Tables Columns
  88. 88. EXAMPLE Metadata Model in Excel
  89. 89. Facts and Dimensions Metadata Model in Biml
  90. 90. Facts and Dimensions Use Biml features to simplify code: • ScdType on columns • StaticSource for static and unknown dimension rows • CloneTable for update tables
  91. 91. ScdType Define ScdType per column: <Column Name="DimSK" DataType="Int32" IsNullable="false" ScdType="SurrogateKey" /> Use instead of custom annotations :)
  92. 92. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) ScdType Values SurrogateKey Key Update Historical Audit Other Often IDENTITY Source IDs SCD1 Columns SCD2 Columns ETL Columns Not used in SCD logic
  93. 93. StaticSource Define rows to be inserted when table is created: <Table Name="Table"> <Sources> <StaticSource Name="TableRows">...</StaticSource> </Sources> </Table> Use instead of separate T-SQL scripts :)
  94. 94. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) StaticSource Values <StaticSource Name="TableRows"> <Rows> <Row> <ColumnValues> <ColumnValue ColumnName="Col1" Value="-1" /> </ColumnValues> </Row> </Rows> </StaticSource>
  95. 95. CloneTable Create a copy of a table: <CloneTable Name="NewTable" TableName="OriginalSchema.OriginalTable" SchemaName="Database.NewSchema" /> Use instead of defining tables twice :)
  96. 96. DEMO Metadata Model in Biml
  97. 97. SQL Scripts Drop and Create
  98. 98. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Could not drop object 'Schema.Table' because it is referenced by a FOREIGN KEY constraint.
  99. 99. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Drop and Create Create Order: Dim1 Dim2 Fact1 Fact2 Fact2 Fact1 Dim2 Dim1Drop Order:
  100. 100. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Drop and Create Create Order: Dim1 Dim2 Fact1 Fact2 Fact2 Fact1 Dim2 Dim1Drop Order:
  101. 101. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Topological Sort "A topological sort of a directed graph is a linear ordering of its vertices such that for every directed edge uv from vertex u to vertex v, u comes before v in the ordering"
  102. 102. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Topological Sort "A topological sort of a directed graph is a linear ordering of its vertices such that for every directed edge uv from vertex u to vertex v, u comes before v in the ordering"
  103. 103. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Topological Sort "Sorts objects by dependencies. For example dimensions before fact tables."
  104. 104. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Topological Sort "Sorts objects by dependencies. For example dimensions before fact tables."
  105. 105. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Topological Sort var dependentTables = DependencyAnalysis.TopologicalSort<AstTableNode> ( RootNode.Tables, t => t.Columns .OfType<AstTableColumnTableReferenceNode>() .Select(c => c.ForeignTable) );
  106. 106. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Topological Sort: Method var dependentTables = DependencyAnalysis.TopologicalSort<AstTableNode> ( RootNode.Tables, t => t.Columns .OfType<AstTableColumnTableReferenceNode>() .Select(c => c.ForeignTable) );
  107. 107. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Topological Sort: Collection to Sort var dependentTables = DependencyAnalysis.TopologicalSort<AstTableNode> ( RootNode.Tables, t => t.Columns .OfType<AstTableColumnTableReferenceNode>() .Select(c => c.ForeignTable) );
  108. 108. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Topological Sort: How to Sort var dependentTables = DependencyAnalysis.TopologicalSort<AstTableNode> ( RootNode.Tables, t => t.Columns .OfType<AstTableColumnTableReferenceNode>() .Select(c => c.ForeignTable) );
  109. 109. DEMO Facts and Dimensions
  110. 110. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) +1000000 XP Congratulations! You reached level 100!
  111. 111. Get things done Start small Start simple Solve a problem Keep going Expand Improve Deliver often
  112. 112. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
  113. 113. Download the GuideBook App and search: PASS Summit 2018 Follow the QR code link displayed on session signage throughout the conference venue and in the program guide Session evaluations Your feedback is important and valuable. Go to passSummit.com 3 Ways to Access: Submit by 5pm Friday, November 16th to win prizes.
  114. 114. @cathrinew cathrinew.net hi@cathrinew.net cathrinew.net/biml

×