Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Oslo)

423 views

Published on

Level Up Your Biml: Best Practices and Coding Techniques (Presented at SQLSaturday Oslo on September 3rd 2016)

Published in: Data & Analytics
  • Be the first to comment

Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Oslo)

  1. 1. Level Up Your Biml: Best Practices and Coding Techniques Cathrine Wilhelmsen
  2. 2. Thank you to our sponsors!
  3. 3. Session Description Is your Biml solution starting to remind you of a bowl of tangled spaghetti code? Good! That means you are solving real problems while saving a lot of time. The next step is to make sure that your solution does not grow too complex and confusing – you do not want to waste all that saved time on future maintenance! Attend this session for an overview of Biml best practices and coding techniques. Learn how to centralize and reuse code with Include files and the CallBimlScript method. Make your code easier to read and write by utilizing LINQ (Language-Integrated Queries). Share code between files by using Annotations and ObjectTags. And finally, if standard Biml is not enough to solve your problems, you can create your own C# helper classes and extension methods to implement custom logic. Start improving your code today and level up your Biml in no time!
  4. 4. Code Management …the next 60 minutes… C# Classes and MethodsPractical Biml Coding
  5. 5. Cathrine Wilhelmsen @cathrinew cathrinew.net Data Warehouse Architect Business Intelligence Developer
  6. 6. Biml vs. BimlScript
  7. 7. Biml vs. BimlScript Generate, control and manipulate Biml with C# XML Language "Just plain text"
  8. 8. Biml Biml is an XML language – it's just plain text Easy to read and write – but can be verbose You probably don't want to write 50000 lines of Biml code manually...?
  9. 9. Generating Biml You can use any tool to generate Biml for you: • Text editor macros • Excel • PowerShell • T-SQL …but the recommended tool is BimlScript :)
  10. 10. What is BimlScript? Extend Biml with C# or VB code blocks Import database structure and metadata Loop over tables and columns Expressions replace static values Allows you to generate, control and manipulate Biml code
  11. 11. BimlScript Code Nuggets <# … #> Control Nuggets (Control logic) <#= … #> Text Nuggets (Returns string) <#@ … #> Directives (Compiler instructions) <#+ … #> Class Nuggets (Create C# classes)
  12. 12. BimlScript Syntax <# var con = SchemaManager.CreateConnectionNode(...); #> <# var metadata = con.GetDatabaseSchema(); #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in metadata.TableNodes) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml>
  13. 13. BimlScript Syntax: Import metadata <# var con = SchemaManager.CreateConnectionNode(...); #> <# var metadata = con.GetDatabaseSchema(); #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in metadata.TableNodes) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml>
  14. 14. BimlScript Syntax: Loop over tables <# var con = SchemaManager.CreateConnectionNode(...); #> <# var metadata = con.GetDatabaseSchema(); #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in metadata.TableNodes) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml>
  15. 15. BimlScript Syntax: Replace static values <# var con = SchemaManager.CreateConnectionNode(...); #> <# var metadata = con.GetDatabaseSchema(); #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in metadata.TableNodes) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml>
  16. 16. How does it work?
  17. 17. Yes, but how does it work?
  18. 18. Yes, but how does it actually work? <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in RootNode.Tables) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="Load_Customer"></Package> <Package Name="Load_Product"></Package> <Package Name="Load_Sales"></Package> </Packages> </Biml>
  19. 19. Biml Tools Comparison
  20. 20. What do you need?
  21. 21. BIDS Helper Free open-source add-in for Visual Studio 60+ features for SSIS, SSAS and SSRS Includes basic Biml package generator bidshelper.codeplex.com
  22. 22. …or you can use the new Biml tools…
  23. 23. BimlExpress Free add-in for Visual Studio Code editor with syntax highlighting and Biml Intellisense More frequent updates than BIDS Helper varigence.com/bimlexpress
  24. 24. BimlOnline Free browser-based Biml editor Code editor with Biml and C# Intellisense Reverse-engineer from SSIS to Biml bimlonline.com
  25. 25. BimlStudio Licensed full-featured development environment for Biml Visual designer and metadata modeling Full-stack automation and transformers varigence.com/bimlstudio
  26. 26. Biml Tools Comparison Free Biml Intellisense "Black Box" Free Full Intellisense Logical View Licensed Full IDE Preview BimlScript
  27. 27. Code Management
  28. 28. Don't Repeat Yourself Move common code to separate files Centralize and reuse in many projects Update code once for all projects 1. Include files 2. CallBimlScript with parameters 3. Tiered Biml files
  29. 29. Include Files Include common code in multiple files and projects Can include many file types: .biml .txt .sql .cs Use the include directive <#@ include file="CommonCode.biml" #> The directive will be replaced by the included file Works like an automated Copy & Paste
  30. 30. Include Files
  31. 31. Include Files
  32. 32. Include Files
  33. 33. CallBimlScript with Parameters Works like a parameterized include (or stored procedure) File to be called (callee) specifies accepted parameters: <#@ property name="Parameter" type="String" #> File that calls (caller) passes parameters: <#=CallBimlScript("Common.biml", Parameter)#>
  34. 34. CallBimlScript with Parameters
  35. 35. CallBimlScript with Parameters
  36. 36. CallBimlScript with Parameters
  37. 37. CallBimlScript with Parameters
  38. 38. CallBimlScript with Parameters
  39. 39. Tiered Biml Files Split Biml code in multiple files Specify the tier per file by using the template directive: <#@ template tier="2" #> Biml is compiled from lowest to highest tier to: • Solve logical dependencies • Build solutions in multiple steps behind the scenes
  40. 40. Tiered Biml Files Biml is compiled step-by-step from lowest to highest tier • Biml files with no BimlScript are implicitly tier 0 • Biml files with BimlScript are implicitly tier 1 For each tier, objects are added to the in-memory RootNode Higher tiers can use objects defined in lower tiers
  41. 41. What is this RootNode? When working with Biml, the <Biml> root element contains collections of elements: When working with BimlScript, the RootNode object contains collections of objects: <Biml> <Connections>...</Connections> <Databases>...</Databases> <Schemas>...</Schemas> <Tables>...</Tables> <Projects>...</Projects> <Packages>...</Packages> </Biml>
  42. 42. Tiered Biml Files: Behind the Scenes <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  43. 43. Tiered Biml Files: Behind the Scenes <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  44. 44. Tiered Biml Files: Behind the Scenes <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  45. 45. Tiered Biml Files: Behind the Scenes <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  46. 46. Tiered Biml Files: Behind the Scenes <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  47. 47. Tiered Biml Files: Behind the Scenes <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  48. 48. Tiered Biml Files: Behind the Scenes <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  49. 49. Tiered Biml Files: Behind the Scenes <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  50. 50. Tiered Biml Files: Behind the Scenes <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  51. 51. Tiered Biml Files: Behind the Scenes <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  52. 52. Tiered Biml Files: Behind the Scenes
  53. 53. How do you use Tiered Biml files? 1. Create Biml files with specified tiers 2. Select all the tiered Biml files 3. Right-click and click Generate SSIS Packages 1 2 3
  54. 54. How does this actually work?
  55. 55. Annotations and ObjectTags
  56. 56. Annotations and ObjectTags Annotations and ObjectTags are Key / Value pairs Use them to pass code between Biml files Annotations: String / String ObjectTags: String / Object
  57. 57. Annotations Create annotations: <OleDbConnection Name="Destination" ConnectionString="…"> <Annotations> <Annotation Tag="Schema">AW2014</Annotation> </Annotations> </OleDbConnection> Use annotations: RootNode.OleDbConnections["Destination"].GetTag("Schema");
  58. 58. ObjectTags Create ObjectTags: RootNode.OleDbConnections["Destination"].ObjectTag["Filter"] = new List<string>{"Product","ProductCategory"}; Use ObjectTags: RootNode.OleDbConnections["Destination"].ObjectTag["Filter"];
  59. 59. LINQ
  60. 60. LINQ (Language-Integrated Query) One language to query: SQL Server Databases Datasets Collections Two ways to write queries: SQL-ish Syntax Extension Methods
  61. 61. LINQ Extension Methods Sort OrderBy, ThenBy Filter Where, OfType Group GroupBy Aggregate Count, Sum Check Collections All, Any, Contains Get Elements First, Last, ElementAt Project Collections Select, SelectMany
  62. 62. LINQ Extension Methods Example var numConnections = RootNode.Connections.Count() foreach (var table in RootNode.Tables.Where(…)) if (RootNode.Packages.Any(…))
  63. 63. LINQ Extension Methods Example foreach (var table in RootNode.Tables.Where(…)) if (RootNode.Packages.Any(…)) But what do you put in here?
  64. 64. Lambda Expressions "A lambda expression is an anonymous function that you can use to create delegates or expression tree types"
  65. 65. Lambda Expressions …huh? o_O
  66. 66. Lambda Expressions table => table.Name == "Product"
  67. 67. Lambda Expressions table => table.Name == "Product" The arrow is the lambda operator
  68. 68. Lambda Expressions table => table.Name == "Product" Input parameter is on the left side
  69. 69. Lambda Expressions table => table.Name == "Product" Expression is on the right side
  70. 70. LINQ and Lambda Chain LINQ Methods and use Lambda Expressions for simple and powerful querying of collections: .Where(table => table.Schema.Name == "Production") .OrderBy(table => table.Name)
  71. 71. LINQ: Filter collections Where() Returns the filtered collection with all elements that meet the criteria RootNode.Tables.Where(t => t.Schema.Name == "Production") OfType() Returns the filtered collection with all elements of the specified type RootNode.Connections.OfType<AstExcelOleDbConnectionNode>()
  72. 72. LINQ: Sort collections OrderBy() Returns the collection sorted by key… RootNode.Tables.OrderBy(t => t.Name) ThenBy() …then sorted by secondary key RootNode.Tables.OrderBy(t => t.Schema.Name) .ThenBy(t => t.Name)
  73. 73. LINQ: Sort collections OrderByDescending() Returns the collection sorted by key… RootNode.Tables.OrderByDescending(t => t.Name) ThenByDescending() …then sorted by secondary key RootNode.Tables.OrderBy(t => t.Schema.Name) .ThenByDescending(t => t.Name)
  74. 74. LINQ: Sort collections Reverse() Returns the collection sorted in reverse order RootNode.Tables.Reverse()
  75. 75. LINQ: Group collections GroupBy() Returns a collection of key-value pairs where each value is a new collection RootNode.Tables.GroupBy(t => t.Schema.Name)
  76. 76. LINQ: Aggregate collections Count() Returns the number of elements in the collection RootNode.Tables.Count() RootNode.Tables.Count(t => t.Schema.Name == "Production")
  77. 77. LINQ: Aggregate collections Sum() Returns the sum of the (numeric) values in the collection RootNode.Tables.Sum(t => t.Columns.Count) Average() Returns the average value of the (numeric) values in the collection RootNode.Tables.Average(t => t.Columns.Count)
  78. 78. LINQ: Aggregate collections Min() Returns the minimum value of the (numeric) values in the collection RootNode.Tables.Min(t => t.Columns.Count) Max() Returns the maximum value of the (numeric) values in the collection RootNode.Tables.Max(t => t.Columns.Count)
  79. 79. LINQ: Check collections All() Returns true if all elements in the collection meet the criteria RootNode.Databases.All(d => d.Name.StartsWith("A")) Any() Returns true if any element in the collection meets the criteria RootNode.Databases.Any(d => d.Name.Contains("DW"))
  80. 80. LINQ: Check collections Contains() Returns true if collection contains element RootNode.Databases.Contains(AdventureWorks2014)
  81. 81. LINQ: Get elements First() Returns the first element in the collection (that meets the criteria) RootNode.Tables.First() RootNode.Tables.First(t => t.Schema.Name == "Production") FirstOrDefault() Returns the first element in the collection or default value (that meets the criteria) RootNode.Tables.FirstOrDefault() RootNode.Tables.FirstOrDefault(t => t.Schema.Name == "Production")
  82. 82. LINQ: Get elements Last() Returns the last element in the collection (that meets the criteria) RootNode.Tables.Last() RootNode.Tables.Last(t => t.Schema.Name == "Production") LastOrDefault() Returns the last element in the collection or default value (that meets the criteria) RootNode.Tables.LastOrDefault() RootNode.Tables.LastOrDefault(t => t.Schema.Name == "Production")
  83. 83. LINQ: Get elements ElementAt() Returns the element in the collection at the specified index RootNode.Tables.ElementAt(42) ElementAtOrDefault() Returns the element in the collection or default value at the specified index RootNode.Tables.ElementAtOrDefault(42)
  84. 84. LINQ: Project collections Select() Creates a new collection from one collection A list of table names: RootNode.Tables.Select(t => t.Name) A list of table and schema names: RootNode.Tables.Select(t => new {t.Name, t.Schema.Name})
  85. 85. LINQ: Project collections SelectMany() Creates a new collection from many collections and merges the collections A list of all columns from all tables: RootNode.Tables.SelectMany(t => t.Columns)
  86. 86. Debugging
  87. 87. Debugging Biml BimlExpress is a "black box": • You only see the generated SSIS packages • Not possible to view expanded BimlScript or compiled Biml Add high-tier helper files to debug and save compiled Biml to files • Use Check Biml For Errors to debug without generating packages
  88. 88. SaveCompiledBimlToFile.biml Add the helper file to your project… <#@ template tier="999" #> <# System.IO.File.WriteAllText( @"C:BimlCompiledBiml.xml", RootNode.GetBiml() ); #>
  89. 89. SaveCompiledBimlToFile.biml …with a high tier so it is executed as the last step <#@ template tier="999" #> <# System.IO.File.WriteAllText( @"C:BimlCompiledBiml.xml", RootNode.GetBiml() ); #>
  90. 90. SaveCompiledBimlToFile.biml It creates a file… <#@ template tier="999" #> <# System.IO.File.WriteAllText( @"C:BimlCompiledBiml.xml", RootNode.GetBiml() ); #>
  91. 91. SaveCompiledBimlToFile.biml …at the specified path… <#@ template tier="999" #> <# System.IO.File.WriteAllText( @"C:BimlCompiledBiml.xml", RootNode.GetBiml() ); #>
  92. 92. SaveCompiledBimlToFile.biml …with all the Biml for all the objects in RootNode :) <#@ template tier="999" #> <# System.IO.File.WriteAllText( @"C:BimlCompiledBiml.xml", RootNode.GetBiml() ); #>
  93. 93. How do you use this helper file? 1. Create the helper file 2. Select all the Biml files and the helper file 3. Right-click and click Check Biml For Errors 1 2 3
  94. 94. How do you debug Biml?
  95. 95. C# Classes and Methods
  96. 96. C# Classes and Methods BimlScript and LINQ not enough? Need to reuse C# code? Create your own classes and methods!
  97. 97. C# Classes and Methods: From this… public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { if (node.GetTag(tag) != "") { return true; } else { return false; } } }
  98. 98. C# Classes and Methods: …to this public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { if (node.GetTag(tag) != "") { return true; } return false; } }
  99. 99. C# Classes and Methods: …to this public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } }
  100. 100. C# Classes and Methods: …to this public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { return (node.GetTag(tag) != ""); } }
  101. 101. Where do you put your C# code? Inline code nuggets Included Biml files with code nuggets Reference code files
  102. 102. C# Classes and Methods: Inline <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #> ... <# } #> <# } #> </Biml> <#+ public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } } #>
  103. 103. C# Classes and Methods: Included Files <#@ include file="HelperClass.biml" #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #> ... <# } #> <# } #> </Biml> <#+ public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } } #>
  104. 104. C# Classes and Methods: Code Files <#@ code file="HelperClass.cs" #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #> ... <# } #> <# } #> </Biml> public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } }
  105. 105. Extension Methods
  106. 106. Extension Methods "Make it look like the method belongs to an object instead of a helper class"
  107. 107. Extension Methods: From this… <#@ code file="HelperClass.cs" #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #> ... <# } #> <# } #> </Biml> public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } }
  108. 108. Extension Methods: …to this <#@ code file="HelperClass.cs" #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #> ... <# } #> <# } #> </Biml> public static class HelperClass { public static bool AnnotationTagExists(this AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } }
  109. 109. Extension Methods: …to this <#@ code file="HelperClass.cs" #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables) { #> <# if (table.AnnotationTagExists("SourceSchema")) { #> ... <# } #> <# } #> </Biml> public static class HelperClass { public static bool AnnotationTagExists(this AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } }
  110. 110. Extension Methods: …to this :) <#@ code file="HelperClass.cs" #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables.Where(t => t.AnnotationTagExists("SourceSchema")) { #> ... <# } #> <# } #> </Biml> public static class HelperClass { public static bool AnnotationTagExists(this AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } }
  111. 111. Code Management …the past 60 minutes… C# Classes and MethodsPractical Biml Coding
  112. 112. Get things done Start small Start simple Start with ugly code Keep going Expand Improve Deliver often
  113. 113. Biml on Monday… …BimlBreak the rest of the week 
  114. 114. Don't forget…  Session Evaluations Sponsor Raffles After Party
  115. 115. @cathrinew cathrinew.net linkedin.com/in/cathrinewilhelmsen hi@cathrinew.net slideshare.net/cathrinewilhelmsen Biml resources and references: cathrinew.net/biml

×