Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Level Up Your Biml: Best Practices and Coding Techniques (SQLDay 2018)

112 views

Published on

Level Up Your Biml: Best Practices and Coding Techniques (Presented at SQLDay in Poland on May 16th 2018)

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Level Up Your Biml: Best Practices and Coding Techniques (SQLDay 2018)

  1. 1. SQLDay 2018 Level Up Your Biml Best Practices and Coding Techniques Cathrine Wilhelmsen @cathrinew
  2. 2. SQLDay 2018 GOLD SPONSORS SILVER SPONSORS BRONZE SPONSOR STRATEGIC PARTNER
  3. 3. SQLDay 2018 Session Description Is your Biml solution starting to remind you of a bowl of tangled spaghetti code? Good! That means you are solving real problems while saving a lot of time. The next step is to make sure that your solution does not grow too complex and confusing – you do not want to waste all that saved time on future maintenance! Attend this session for an overview of Biml best practices and coding techniques. Learn how to centralize and reuse code with Include files and the CallBimlScript method. Make your code easier to read and write by utilizing LINQ (Language-Integrated Queries). Share code between files by using Annotations and ObjectTags. And finally, if standard Biml is not enough to solve your problems, you can create your own C# helper classes and extension methods to implement custom logic. Start improving your code today and level up your Biml in no time!
  4. 4. SQLDay 2018 Cathrine Wilhelmsen @cathrinew cathrinew.net Business Intelligence Consultant
  5. 5. SQLDay 2018 You… …? Know basic Biml and BimlScript Created a staging environment using Biml Created a metadata-driven Biml framework
  6. 6. SQLDay 2018 …the next 60 minutes… Practical Biml Coding C# Classes and Methods Code Management
  7. 7. SQLDay 2018 Biml Tools
  8. 8. SQLDay 2018 Code Management
  9. 9. SQLDay 2018 Don't Repeat Yourself Centralize and reuse code Update once in one file 1. Tiered Biml Files 2. Include Files 3. CallBimlScript
  10. 10. SQLDay 2018 Tiered Biml Files Multiple Biml Files Working Together
  11. 11. SQLDay 2018 What are Tiered Biml Files? Think of tiers as stacked layers or sequential steps Tier (Layer) 1 Tier 0 Tier (Layer) 2 Tier (Step) 1 Tier (Step) 2
  12. 12. SQLDay 2018 Tiered Biml Files Split Biml code in multiple files to: • Solve logical dependencies • Build solutions in multiple steps behind the scenes Specify the tier per file by using the template directive: <#@ template tier="2" #>
  13. 13. SQLDay 2018 Tiered Biml Files: Behind the Scenes Connections Load Packages Master Package Connections Databases Schemas Tables Packages
  14. 14. SQLDay 2018 Tiered Biml Files: Behind the Scenes Connections Databases Schemas Tables Packages Connections Load Packages Master Package
  15. 15. SQLDay 2018 Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Connections Load Packages Master Package
  16. 16. SQLDay 2018 Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Connections Load Packages Master Package
  17. 17. SQLDay 2018 Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Connections Load Packages Master Package
  18. 18. SQLDay 2018 Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Connections Load Packages Master Package
  19. 19. SQLDay 2018 Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Connections Load Packages Master Package
  20. 20. SQLDay 2018 Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Connections Load Packages Master Package
  21. 21. SQLDay 2018 Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Master Connections Load Packages Master Package
  22. 22. SQLDay 2018 Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Master Connections Load Packages Master Package
  23. 23. SQLDay 2018 Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Master
  24. 24. SQLDay 2018 How do you use Tiered Biml Files? 1. Create Biml files with specified tiers 2. Select all the tiered Biml files 3. Right-click and click Generate SSIS Packages 1 2 3
  25. 25. SQLDay 2018 Annotations and ObjectTags Store and Pass Metadata Between Biml Files
  26. 26. SQLDay 2018 Annotations and ObjectTags Annotations and ObjectTags are Key / Value pairs Annotations: String / String ObjectTags: String / Object Store metadata by attaching tags to Biml objects Higher tier files can get tags from lower tier files
  27. 27. SQLDay 2018 Annotations Create annotations: <OleDbConnection Name="Dest"> <Annotations> <Annotation Tag="Schema">stg</Annotation> </Annotations> </OleDbConnection> Use annotations: RootNode.OleDbConnections["Dest"].GetTag("Schema");
  28. 28. SQLDay 2018 ObjectTags Create ObjectTags: RootNode.OleDbConnections["Dest"].ObjectTag["Filter"] = new List<string>{"Product","ProductCategory"}; Use ObjectTags: RootNode.OleDbConnections["Dest"].ObjectTag["Filter"];
  29. 29. SQLDay 2018 Include Files Automated Copy & Paste
  30. 30. SQLDay 2018 Include Files Include common code in multiple files and projects Can include many file types: .biml .txt .sql .cs Use the include directive <#@ include file="CommonCode.biml" #> The directive will be replaced by the included file Works like an automated Copy & Paste
  31. 31. SQLDay 2018 Include Files
  32. 32. SQLDay 2018 Include Files
  33. 33. SQLDay 2018 Include Files
  34. 34. SQLDay 2018 Global Include Files Automated Copy & Paste Everywhere
  35. 35. SQLDay 2018 Global Include Files Use the global directive <#@ global #> Use instead of adding include directive to all files to • Create global variables • Include code files • Make VB default language
  36. 36. SQLDay 2018 Global Include Files Specify attributes to control global include files: <#@ global order="1" location="top" active="true" #>
  37. 37. SQLDay 2018 Global Include Files
  38. 38. SQLDay 2018 Global Include Files
  39. 39. SQLDay 2018 How do you use Global Include Files? 1. Create Global Include File 2. Select Biml files and Global Include File 3. Right-click and click Generate SSIS Packages 1 2 3
  40. 40. SQLDay 2018 CallBimlScript Parameterized Control over Returned Code
  41. 41. SQLDay 2018 CallBimlScript Control and limit the code returned Works like a parameterized include (or stored procedure) CallBimlScript file specifies accepted parameters: <#@ property name="Parameter" type="String" #> Main file calls and passes parameters: <#=CallBimlScript("CommonCode.biml", Parameter)#>
  42. 42. SQLDay 2018 CallBimlScript
  43. 43. SQLDay 2018 CallBimlScript
  44. 44. SQLDay 2018 CallBimlScript
  45. 45. SQLDay 2018 CallBimlScript
  46. 46. SQLDay 2018 CallBimlScript
  47. 47. SQLDay 2018 CallBimlScriptWithOutput Parameterized Control over Returned Code and Objects
  48. 48. SQLDay 2018 CallBimlScriptWithOutput CallBimlScript file specifies custom output object: <# CustomOutput.BimlComment = "Comment"; #> Main file defines, passes and uses output object: <# dynamic outObj; #> <#=CallBimlScriptWithOutput("CommonCode.biml", out outObj)#> <#=outObj.BimlComment#>
  49. 49. SQLDay 2018 CallBimlScriptWithOutput
  50. 50. SQLDay 2018 CallBimlScriptWithOutput
  51. 51. SQLDay 2018 CallBimlScriptWithOutput
  52. 52. SQLDay 2018 CallBimlScriptWithOutput
  53. 53. SQLDay 2018 CallBimlScriptWithOutput use CustomOutput
  54. 54. SQLDay 2018 Wait! When do you use what? In larger projects, you often see a combination of Tiered Biml Files, Include Files, and CallBimlScript Rule of Thumb: If you reuse code more than 3 times, refactor so you Don't Repeat Yourself
  55. 55. SQLDay 2018 Include Files Automated Copy & Paste Code Included As-Is CallBimlScript Parameterized Control Returned Code Tiered Biml Files Solve Dependencies Multi-Step Builds
  56. 56. SQLDay 2018 DEMO Code Management
  57. 57. SQLDay 2018 Practical Biml Coding
  58. 58. SQLDay 2018 LINQ
  59. 59. SQLDay 2018 LINQ (Language-Integrated Query) One language to query: SQL Server Databases Datasets Collections Two ways to write queries: SQL-ish Syntax Extension Methods
  60. 60. SQLDay 2018 LINQ Extension Methods Sort OrderBy, ThenBy Filter Where, OfType Group GroupBy Aggregate Count, Sum Check Collections All, Any, Contains Get Elements First, Last, ElementAt Project Collections Select, SelectMany
  61. 61. SQLDay 2018 LINQ Extension Methods Example var numConnections = RootNode.Connections.Count() foreach (var table in RootNode.Tables.Where(…)) if (RootNode.Packages.Any(…))
  62. 62. SQLDay 2018 LINQ Extension Methods Example foreach (var table in RootNode.Tables.Where(…)) if (RootNode.Packages.Any(…)) But what do you put in here?
  63. 63. SQLDay 2018 Lambda Expressions "A lambda expression is an anonymous function that you can use to create delegates or expression tree types"
  64. 64. SQLDay 2018 Lambda Expressions …huh? o_O
  65. 65. SQLDay 2018 Lambda Expressions table => table.Name == "Product"
  66. 66. SQLDay 2018 Lambda Expressions table => table.Name == "Product" The arrow is the lambda operator
  67. 67. SQLDay 2018 Lambda Expressions table => table.Name == "Product" Input parameter is on the left side
  68. 68. SQLDay 2018 Lambda Expressions table => table.Name == "Product" Expression is on the right side
  69. 69. SQLDay 2018 LINQ and Lambda Chain LINQ Methods and use Lambda Expressions for simple and powerful querying of collections: .Where(table => table.Schema.Name == "Production") .OrderBy(table => table.Name)
  70. 70. SQLDay 2018 LINQ: Filter collections Where() Returns the filtered collection with all elements that meet the criteria RootNode.Tables.Where(t => t.Schema.Name == "Production") OfType() Returns the filtered collection with all elements of the specified type RootNode.Connections.OfType<AstExcelOleDbConnectionNode>()
  71. 71. SQLDay 2018 LINQ: Sort collections OrderBy() Returns the collection sorted by key… RootNode.Tables.OrderBy(t => t.Name) ThenBy() …then sorted by secondary key RootNode.Tables.OrderBy(t => t.Schema.Name) .ThenBy(t => t.Name)
  72. 72. SQLDay 2018 LINQ: Sort collections OrderByDescending() Returns the collection sorted by key… RootNode.Tables.OrderByDescending(t => t.Name) ThenByDescending() …then sorted by secondary key RootNode.Tables.OrderBy(t => t.Schema.Name) .ThenByDescending(t => t.Name)
  73. 73. SQLDay 2018 LINQ: Sort collections Reverse() Returns the collection sorted in reverse order RootNode.Tables.Reverse()
  74. 74. SQLDay 2018 LINQ: Group collections GroupBy() Returns a collection of key-value pairs where each value is a new collection RootNode.Tables.GroupBy(t => t.Schema.Name)
  75. 75. SQLDay 2018 LINQ: Aggregate collections Count() Returns the number of elements in the collection RootNode.Tables.Count() RootNode.Tables.Count(t => t.Schema.Name == "Production")
  76. 76. SQLDay 2018 LINQ: Aggregate collections Sum() Returns the sum of the (numeric) values in the collection RootNode.Tables.Sum(t => t.Columns.Count) Average() Returns the average value of the (numeric) values in the collection RootNode.Tables.Average(t => t.Columns.Count)
  77. 77. SQLDay 2018 LINQ: Aggregate collections Min() Returns the minimum value of the (numeric) values in the collection RootNode.Tables.Min(t => t.Columns.Count) Max() Returns the maximum value of the (numeric) values in the collection RootNode.Tables.Max(t => t.Columns.Count)
  78. 78. SQLDay 2018 LINQ: Check collections All() Returns true if all elements in the collection meet the criteria RootNode.Databases.All(d => d.Name.StartsWith("A")) Any() Returns true if any element in the collection meets the criteria RootNode.Databases.Any(d => d.Name.Contains("DW"))
  79. 79. SQLDay 2018 LINQ: Check collections Contains() Returns true if collection contains element RootNode.Databases.Contains(AdventureWorks2014)
  80. 80. SQLDay 2018 LINQ: Get elements First() Returns the first element in the collection (that meets the criteria) RootNode.Tables.First() RootNode.Tables.First(t => t.Schema.Name == "Production") FirstOrDefault() Returns the first element in the collection or default value (that meets the criteria) RootNode.Tables.FirstOrDefault() RootNode.Tables.FirstOrDefault(t => t.Schema.Name == "Production")
  81. 81. SQLDay 2018 LINQ: Get elements Last() Returns the last element in the collection (that meets the criteria) RootNode.Tables.Last() RootNode.Tables.Last(t => t.Schema.Name == "Production") LastOrDefault() Returns the last element in the collection or default value (that meets the criteria) RootNode.Tables.LastOrDefault() RootNode.Tables.LastOrDefault(t => t.Schema.Name == "Production")
  82. 82. SQLDay 2018 LINQ: Get elements ElementAt() Returns the element in the collection at the specified index RootNode.Tables.ElementAt(42) ElementAtOrDefault() Returns the element in the collection or default value at the specified index RootNode.Tables.ElementAtOrDefault(42)
  83. 83. SQLDay 2018 LINQ: Project collections Select() Creates a new collection from one collection A list of table names: RootNode.Tables.Select(t => t.Name) A list of table and schema names: RootNode.Tables.Select(t => new {t.Name, t.Schema.Name})
  84. 84. SQLDay 2018 LINQ: Project collections SelectMany() Creates a new collection from many collections and merges the collections A list of all columns from all tables: RootNode.Tables.SelectMany(t => t.Columns)
  85. 85. SQLDay 2018 Preview Pane The power is in the…
  86. 86. SQLDay 2018 BimlExpress Preview Pane
  87. 87. SQLDay 2018 BimlScript to Biml to SSIS
  88. 88. SQLDay 2018 BimlScript to Biml Preview Pane
  89. 89. SQLDay 2018 BimlScript to …? Preview Pane
  90. 90. SQLDay 2018
  91. 91. SQLDay 2018 DEMO Practical Biml Coding
  92. 92. SQLDay 2018 C# Classes and Methods
  93. 93. SQLDay 2018 C# Classes and Methods BimlScript and LINQ not enough? Need to reuse C# code? Create your own classes and methods!
  94. 94. SQLDay 2018 C# Classes and Methods public static class Utilities { public static string GetName(AstTableNode table) { return table.Schema.Name.ToUpper() + "_" + table.Name.ToUpper(); } }
  95. 95. SQLDay 2018 C# Classes and Methods: Class public static class Utilities { public static string GetName(AstTableNode table) { return table.Schema.Name.ToUpper() + "_" + table.Name.ToUpper(); } }
  96. 96. SQLDay 2018 C# Classes and Methods: Method public static class Utilities { public static string GetName(AstTableNode table) { return table.Schema.Name.ToUpper() + "_" + table.Name.ToUpper(); } }
  97. 97. SQLDay 2018 C# Classes and Methods: Logic public static class Utilities { public static string GetName(AstTableNode table) { return table.Schema.Name.ToUpper() + "_" + table.Name.ToUpper(); } }
  98. 98. SQLDay 2018 C# Code Management
  99. 99. SQLDay 2018 Inline Code Blocks <#+ ... #> <#+ ... #>
  100. 100. SQLDay 2018 Included Biml Files with Code Blocks <#@ include file="CodeBlock.biml" #> <#+ ... #>
  101. 101. SQLDay 2018 Code Files <#@ code file="Code.cs" #>
  102. 102. SQLDay 2018 Code Blocks vs. Code Files Code Blocks • Used when logic is specific to one file • Reuse by including files with code blocks Code Files • Use code editor of choice • Create extension methods
  103. 103. SQLDay 2018 C# Extension Methods
  104. 104. SQLDay 2018 "Make it look like the method belongs to an object instead of a helper class"
  105. 105. SQLDay 2018 Extension Methods: From this… <#@ code file="HelperClass.cs" #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #> ... <# } #> <# } #> </Biml> public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } }
  106. 106. SQLDay 2018 Extension Methods: …to this <#@ code file="HelperClass.cs" #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #> ... <# } #> <# } #> </Biml> public static class HelperClass { public static bool AnnotationTagExists(this AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } }
  107. 107. SQLDay 2018 Extension Methods: …to this <#@ code file="HelperClass.cs" #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables) { #> <# if (table.AnnotationTagExists("SourceSchema")) { #> ... <# } #> <# } #> </Biml> public static class HelperClass { public static bool AnnotationTagExists(this AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } }
  108. 108. SQLDay 2018 Extension Methods: …to this :) <#@ code file="HelperClass.cs" #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables.Where(t => t.AnnotationTagExists("SourceSchema")) { #> ... <# } #> <# } #> </Biml> public static class HelperClass { public static bool AnnotationTagExists(this AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } }
  109. 109. SQLDay 2018 DEMO C# Classes and Methods
  110. 110. SQLDay 2018 …the past 60 minutes… Practical Biml Coding C# Classes and Methods Code Management
  111. 111. SQLDay 2018 Where can you learn more? cathrinew.net/BimlBook
  112. 112. SQLDay 2018 Evaluations What can we do better? How can we improve? :):( :|
  113. 113. SQLDay 2018 Get things done Start small Start simple Start with ugly code Keep going Expand Improve Deliver often
  114. 114. SQLDay 2018 @cathrinew cathrinew.net hi@cathrinew.net Biml resources and demo files: cathrinew.net/biml

×