Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
S.M.A.R.T. Biml
Cathrine Wilhelmsen
October 24th 2015
Have you ever wanted to build a Data Warehouse simply by pushing a button? It might not be quite that easy
yet, but gone a...
Please Support Our Sponsors
SQL Saturday is made possible with the generous support of these sponsors.
You can support the...
Cathrine Wilhelmsen
@cathrinew
cathrinewilhelmsen.net
Data Warehouse Architect
Business Intelligence Developer
Know basic Biml and BimlScript
Completed BimlScript.com lessons
Have created a staging environment
You…
…?
S.M.A.R.T. Biml
Standardize • Model • Automate • Reuse • Transform
C# Classes and Methods
Metadata Modeling
Transformers and Frameworks
Quick Recap
Biml Tools
Automate, control and
manipulate Biml with C#
Flat XML
"Just text"
Biml vs. BimlScript
How does it work?
Yes, but how does it work?
Yes, but how does it actually work?
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<# foreach (var table ...
Don't Repeat Yourself
Move common code to separate files
Centralize and reuse in many projects
Update code once for all projects
Don't Repeat Yo...
Solve logical dependencies and simulate manual workflows by using tiers
Tiers instruct the BimlCompiler to compile files f...
Include common code in multiple files and projects
Can include many file types: .biml .txt .sql .cs
Use the include direct...
Include Files
Include Files
Include Files
Works like a parameterized include
File to be called (callee) specifies the input parameters it accepts
<#@ property name=...
CallBimlScript with Parameters
CallBimlScript with Parameters
CallBimlScript with Parameters
CallBimlScript with Parameters
CallBimlScript with Parameters
BIDS Helper vs. Mist
"Black Box"
Only SSIS packages visible
Save Biml to file for debugging
Visual IDE
All in-memory objec...
C# Classes and Methods
One language to query:
SQL Server Databases
XML Documents
Datasets
Collections
LINQ (Language-Integrated Query)
Two ways t...
LINQ Extension Methods
..and many, many more!
Sort
OrderBy, ThenBy
Filter
Where, OfType
Group
GroupBy
Aggregate
Count, Sum...
var numConnections = RootNode.Connections.Count()
foreach (var table in RootNode.Tables.Where(…))
if (RootNode.Packages.An...
Use lambda expressions to filter or specify values:
.Where(table => table.Schema.Name == "Production")
.OrderBy(table => t...
For each element in the collection…
.Where(table => table.Schema.Name == "Production")
.OrderBy(table => table.Name)
LINQ ...
…evaluate a criteria or get a value:
.Where(table => table.Schema.Name == "Production")
.OrderBy(table => table.Name)
LINQ...
You can name the element anything…
.Where(x => x.Schema.Name == "Production")
.OrderBy(x => x.Name)
LINQ and Lambda expres...
…but try to avoid confusing code 
.Where(cat => cat.Schema.Name == "Production")
.OrderBy(cat => cat.Name)
LINQ and Lambd...
LINQ: Filter collections
Where()
Returns the filtered collection with all elements that meet the criteria
RootNode.Tables....
LINQ: Sort collections
OrderBy()
Returns the collection sorted by key…
RootNode.Tables.OrderBy(t => t.Name)
ThenBy()
…then...
LINQ: Sort collections
OrderByDescending()
Returns the collection sorted by key…
RootNode.Tables.OrderByDescending(t => t....
LINQ: Sort collections
Reverse()
Returns the collection sorted in reverse order
RootNode.Tables.Reverse()
LINQ: Group collections
GroupBy()
Returns a collection of key-value pairs where each value is a new collection
RootNode.Ta...
LINQ: Aggregate collections
Count()
Returns the number of elements in the collection
RootNode.Tables.Count()
RootNode.Tabl...
LINQ: Aggregate collections
Sum()
Returns the sum of the (numeric) values in the collection
RootNode.Tables.Sum(t => t.Col...
LINQ: Aggregate collections
Min()
Returns the minimum value of the (numeric) values in the collection
RootNode.Tables.Min(...
LINQ: Check collections
All()
Returns true if all elements in the collection meet the criteria
RootNode.Databases.All(d =>...
LINQ: Check collections
Contains()
Returns true if collection contains element
RootNode.Databases.Contains(AdventureWorks2...
LINQ: Get elements
First()
Returns the first element in the collection (that meets the criteria)
RootNode.Tables.First()
R...
LINQ: Get elements
Last()
Returns the last element in the collection (that meets the criteria)
RootNode.Tables.Last()
Root...
LINQ: Get elements
ElementAt()
Returns the element in the collection at the specified index
RootNode.Tables.ElementAt(42)
...
LINQ: Project collections
Select()
Creates a new collection from one collection
A list of table names:
RootNode.Tables.Sel...
LINQ: Project collections
SelectMany()
Creates a new collection from many collections and merges the collections
A list of...
C# Classes and Methods
BimlScript and LINQ not enough?
Need to reuse C# code?
Create your own classes and methods!
public static class HelperClass {
public static bool AnnotationTagExists(AstNode node, string tag) {
if (node.GetTag(tag) ...
public static class HelperClass {
public static bool AnnotationTagExists(AstNode node, string tag) {
return (node.GetTag(t...
public static class HelperClass {
public static string AnnotationTagExists(AstNode node, string tag) {
return (node.GetTag...
C# Classes and Methods
Inline code blocks
Included Biml files with code blocks
Reference code files
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<# foreach (var table in RootNode.Tables) { #>
<# if (HelperClass.Ann...
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<# foreach (var table in RootNode.Tables) { #>
<# if (HelperClass.Ann...
<#@ include file="HelperClass.biml" #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<# foreach (var table in RootN...
<#@ code file="..CodeHelperClass.cs" #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<# foreach (var table in Root...
"Make it look like the method belongs to
an object instead of a helper class"
Extension Methods
<#@ code file="..CodeHelperClass.cs" #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<# foreach (var table in Root...
<#@ code file="..CodeHelperClass.cs" #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<# foreach (var table in Root...
<#@ code file="..CodeHelperClass.cs" #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<# foreach (var table in Root...
<#@ code file="..CodeHelperClass.cs" #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<# foreach (var table in Root...
C# Classes and Methods
Metadata Modeling
Lesson learned:
Start small, start simple
If you try to create a complete and perfect
metadata model from day 1…
…there is a risk you might not get anything done 
Metadata Models
Entities
Relationships
Properties
Metadata Modeling in Mist
Metadata Instances
Data Items
Properties
Metadata Model
Entities Relationships
Properties
Metadata Instance
Properties
Data
Items
Metadata Modeling
Transformers and Frameworks
Transformers: Modify existing Biml
<#@ target type="Package" mergemode="LocalMerge" #>
<Node>
<Variables>
<Variable Name="NewRows" DataType="Int32">0</Variab...
<#@ target type="Package" mergemode="LocalMerge" #>
<Node>
<Variables>
<Variable Name="NewRows" DataType="Int32">0</Variab...
<#@ target type="Package" mergemode="LocalMerge" #>
<Node>
<Variables>
<Variable Name="NewRows" DataType="Int32">0</Variab...
LocalMerge (update object)
Keep original object and merge changes
LocalReplace (replace object)
Replace original object wi...
Transformers and Frameworks
Get things done
Start small
Start simple
Start with ugly code
Keep going
Expand
Improve
Deliver often
@cathrinew
cathrinewilhelmsen.net
no.linkedin.com/in/cathrinewilhelmsen
contact@cathrinewilhelmsen.net
slideshare.net/cath...
S.M.A.R.T. Biml - Standardize, Model, Automate, Reuse and Transform (SQLSaturday Oregon)
S.M.A.R.T. Biml - Standardize, Model, Automate, Reuse and Transform (SQLSaturday Oregon)
Upcoming SlideShare
Loading in …5
×

S.M.A.R.T. Biml - Standardize, Model, Automate, Reuse and Transform (SQLSaturday Oregon)

5,477 views

Published on

S.M.A.R.T. Biml - Standardize, Model, Automate, Reuse and Transform (Presented at SQLSaturday Oregon on October 24th, 2015)

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

S.M.A.R.T. Biml - Standardize, Model, Automate, Reuse and Transform (SQLSaturday Oregon)

  1. 1. S.M.A.R.T. Biml Cathrine Wilhelmsen October 24th 2015
  2. 2. Have you ever wanted to build a Data Warehouse simply by pushing a button? It might not be quite that easy yet, but gone are the days of repetitive development. Stop wasting your time on dragging, dropping, connecting, aligning and creating the same SSIS package over and over and over again and start working S.M.A.R.T. with Biml. You already know how to build a staging environment in an hour, so let us dive straight into some advanced features of Biml. We will start by looking at how to create our own C# classes and methods, and how to centralize and reuse code. Then we will explore the metadata modeling feature in Mist. Finally, we will create a framework of transformers that allow you to modify existing objects both interactively and automatically. If you already think Biml is powerful, just wait until you have a toolbox full of transformers ready to do the heavy lifting for you! (No Autobots were harmed in the making of this session.) Session Description
  3. 3. Please Support Our Sponsors SQL Saturday is made possible with the generous support of these sponsors. You can support them by opting-in and visiting them in the sponsor area.
  4. 4. Cathrine Wilhelmsen @cathrinew cathrinewilhelmsen.net Data Warehouse Architect Business Intelligence Developer
  5. 5. Know basic Biml and BimlScript Completed BimlScript.com lessons Have created a staging environment You… …?
  6. 6. S.M.A.R.T. Biml Standardize • Model • Automate • Reuse • Transform
  7. 7. C# Classes and Methods Metadata Modeling Transformers and Frameworks
  8. 8. Quick Recap
  9. 9. Biml Tools
  10. 10. Automate, control and manipulate Biml with C# Flat XML "Just text" Biml vs. BimlScript
  11. 11. How does it work?
  12. 12. Yes, but how does it work?
  13. 13. Yes, but how does it actually work? <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in RootNode.Tables) { #> <Package Name="Load<#=table.Name#>"></Package> <# } #> </Packages> </Biml> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="LoadCustomer"></Package> <Package Name="LoadProduct"></Package> <Package Name="LoadSales"></Package> </Packages> </Biml>
  14. 14. Don't Repeat Yourself
  15. 15. Move common code to separate files Centralize and reuse in many projects Update code once for all projects Don't Repeat Yourself
  16. 16. Solve logical dependencies and simulate manual workflows by using tiers Tiers instruct the BimlCompiler to compile files from lowest to highest tier <#@ template tier="1" #> Higher tiers can use and might depend on objects from lower tiers Tier 1 - Create database connections Example: Tier 2 - Create loading packages Tier 3 - Create master package to execute loading packages Split and Combine Biml Files
  17. 17. Include common code in multiple files and projects Can include many file types: .biml .txt .sql .cs Use the include directive <#@ include file="CommonCode.biml" #> The include directive will be replaced by the content of the included file Include pulls code from the included file into the main file Include Files
  18. 18. Include Files
  19. 19. Include Files
  20. 20. Include Files
  21. 21. Works like a parameterized include File to be called (callee) specifies the input parameters it accepts <#@ property name="Table" type="AstTableNode" #> File that calls (caller) passes input parameters <#=CallBimlScript("CommonCode.biml", Table)#> CallBimlScript pushes parameters from the caller to the callee, and the callee returns code CallBimlScript with Parameters
  22. 22. CallBimlScript with Parameters
  23. 23. CallBimlScript with Parameters
  24. 24. CallBimlScript with Parameters
  25. 25. CallBimlScript with Parameters
  26. 26. CallBimlScript with Parameters
  27. 27. BIDS Helper vs. Mist "Black Box" Only SSIS packages visible Save Biml to file for debugging Visual IDE All in-memory objects visible Preview Expanded BimlScript
  28. 28. C# Classes and Methods
  29. 29. One language to query: SQL Server Databases XML Documents Datasets Collections LINQ (Language-Integrated Query) Two ways to write queries: SQL-like Syntax Extension Methods
  30. 30. LINQ Extension Methods ..and many, many more! Sort OrderBy, ThenBy Filter Where, OfType Group GroupBy Aggregate Count, Sum Check Collections All, Any, Contains Get Elements First, Last, ElementAt Project Collections Select, SelectMany
  31. 31. var numConnections = RootNode.Connections.Count() foreach (var table in RootNode.Tables.Where(…)) if (RootNode.Packages.Any(…)) LINQ Extension Methods
  32. 32. Use lambda expressions to filter or specify values: .Where(table => table.Schema.Name == "Production") .OrderBy(table => table.Name) LINQ and Lambda expressions
  33. 33. For each element in the collection… .Where(table => table.Schema.Name == "Production") .OrderBy(table => table.Name) LINQ and Lambda expressions
  34. 34. …evaluate a criteria or get a value: .Where(table => table.Schema.Name == "Production") .OrderBy(table => table.Name) LINQ and Lambda expressions
  35. 35. You can name the element anything… .Where(x => x.Schema.Name == "Production") .OrderBy(x => x.Name) LINQ and Lambda expressions
  36. 36. …but try to avoid confusing code  .Where(cat => cat.Schema.Name == "Production") .OrderBy(cat => cat.Name) LINQ and Lambda expressions
  37. 37. LINQ: Filter collections Where() Returns the filtered collection with all elements that meet the criteria RootNode.Tables.Where(t => t.Schema.Name == "Production") OfType() Returns the filtered collection with all elements of the specified type RootNode.Connections.OfType<AstExcelOleDbConnectionNode>()
  38. 38. LINQ: Sort collections OrderBy() Returns the collection sorted by key… RootNode.Tables.OrderBy(t => t.Name) ThenBy() …then sorted by secondary key RootNode.Tables.OrderBy(t => t.Schema.Name) .ThenBy(t => t.Name)
  39. 39. LINQ: Sort collections OrderByDescending() Returns the collection sorted by key… RootNode.Tables.OrderByDescending(t => t.Name) ThenByDescending() …then sorted by secondary key RootNode.Tables.OrderBy(t => t.Schema.Name) .ThenByDescending(t => t.Name)
  40. 40. LINQ: Sort collections Reverse() Returns the collection sorted in reverse order RootNode.Tables.Reverse()
  41. 41. LINQ: Group collections GroupBy() Returns a collection of key-value pairs where each value is a new collection RootNode.Tables.GroupBy(t => t.Schema.Name)
  42. 42. LINQ: Aggregate collections Count() Returns the number of elements in the collection RootNode.Tables.Count() RootNode.Tables.Count(t => t.Schema.Name == "Production")
  43. 43. LINQ: Aggregate collections Sum() Returns the sum of the (numeric) values in the collection RootNode.Tables.Sum(t => t.Columns.Count) Average() Returns the average value of the (numeric) values in the collection RootNode.Tables.Average(t => t.Columns.Count)
  44. 44. LINQ: Aggregate collections Min() Returns the minimum value of the (numeric) values in the collection RootNode.Tables.Min(t => t.Columns.Count) Max() Returns the maximum value of the (numeric) values in the collection RootNode.Tables.Max(t => t.Columns.Count)
  45. 45. LINQ: Check collections All() Returns true if all elements in the collection meet the criteria RootNode.Databases.All(d => d.Name.StartsWith("A")) Any() Returns true if any element in the collection meets the criteria RootNode.Databases.Any(d => d.Name.Contains("DW"))
  46. 46. LINQ: Check collections Contains() Returns true if collection contains element RootNode.Databases.Contains(AdventureWorks2014)
  47. 47. LINQ: Get elements First() Returns the first element in the collection (that meets the criteria) RootNode.Tables.First() RootNode.Tables.First(t => t.Schema.Name == "Production") FirstOrDefault() Returns the first element in the collection or default value (that meets the criteria) RootNode.Tables.FirstOrDefault() RootNode.Tables.FirstOrDefault(t => t.Schema.Name == "Production")
  48. 48. LINQ: Get elements Last() Returns the last element in the collection (that meets the criteria) RootNode.Tables.Last() RootNode.Tables.Last(t => t.Schema.Name == "Production") LastOrDefault() Returns the last element in the collection or default value (that meets the criteria) RootNode.Tables.LastOrDefault() RootNode.Tables.LastOrDefault(t => t.Schema.Name == "Production")
  49. 49. LINQ: Get elements ElementAt() Returns the element in the collection at the specified index RootNode.Tables.ElementAt(42) ElementAtOrDefault() Returns the element in the collection or default value at the specified index RootNode.Tables.ElementAtOrDefault(42)
  50. 50. LINQ: Project collections Select() Creates a new collection from one collection A list of table names: RootNode.Tables.Select(t => t.Name) A list of table and schema names: RootNode.Tables.Select(t => new {t.Name, t.Schema.Name})
  51. 51. LINQ: Project collections SelectMany() Creates a new collection from many collections and merges the collections A list of all columns from all tables: RootNode.Tables.SelectMany(t => t.Columns)
  52. 52. C# Classes and Methods BimlScript and LINQ not enough? Need to reuse C# code? Create your own classes and methods!
  53. 53. public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { if (node.GetTag(tag) != "") { return true; } else { return false; } } } C# Classes and Methods: From this…
  54. 54. public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } } C# Classes and Methods: …to this * For bools you can just use: return (node.GetTag(tag) != ""); But in this example we'll use the verbose, SSIS-like syntax because it can be reused with other data types, like…
  55. 55. public static class HelperClass { public static string AnnotationTagExists(AstNode node, string tag) { return (node.GetTag(tag) != "") ? "Yes" : "No"; } } C# Classes and Methods: …or this
  56. 56. C# Classes and Methods Inline code blocks Included Biml files with code blocks Reference code files
  57. 57. <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #> ... <# } #> <# } #> </Biml> C# Classes and Methods
  58. 58. <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #> ... <# } #> <# } #> </Biml> <#+ public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } } #> C# Classes and Methods: Inline
  59. 59. <#@ include file="HelperClass.biml" #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #> ... <# } #> <# } #> </Biml> C# Classes and Methods: Included Files <#+ public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } } #>
  60. 60. <#@ code file="..CodeHelperClass.cs" #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #> ... <# } #> <# } #> </Biml> C# Classes and Methods: Code Files public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } }
  61. 61. "Make it look like the method belongs to an object instead of a helper class" Extension Methods
  62. 62. <#@ code file="..CodeHelperClass.cs" #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #> ... <# } #> <# } #> </Biml> Extension Methods: From this… public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } }
  63. 63. <#@ code file="..CodeHelperClass.cs" #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #> ... <# } #> <# } #> </Biml> Extension Methods: …to this public static class HelperClass { public static bool AnnotationTagExists(this AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } }
  64. 64. <#@ code file="..CodeHelperClass.cs" #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables) { #> <# if (table.AnnotationTagExists("SourceSchema")) { #> ... <# } #> <# } #> </Biml> Extension Methods: …to this public static class HelperClass { public static bool AnnotationTagExists(this AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } }
  65. 65. <#@ code file="..CodeHelperClass.cs" #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables.Where(t => t.AnnotationTagExists("SourceSchema")) { #> ... <# } #> </Biml> Extension Methods: …to this public static class HelperClass { public static bool AnnotationTagExists(this AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } }
  66. 66. C# Classes and Methods
  67. 67. Metadata Modeling
  68. 68. Lesson learned: Start small, start simple If you try to create a complete and perfect metadata model from day 1…
  69. 69. …there is a risk you might not get anything done 
  70. 70. Metadata Models Entities Relationships Properties Metadata Modeling in Mist Metadata Instances Data Items Properties
  71. 71. Metadata Model Entities Relationships Properties
  72. 72. Metadata Instance Properties Data Items
  73. 73. Metadata Modeling
  74. 74. Transformers and Frameworks
  75. 75. Transformers: Modify existing Biml
  76. 76. <#@ target type="Package" mergemode="LocalMerge" #> <Node> <Variables> <Variable Name="NewRows" DataType="Int32">0</Variable> </Variables> </Node> Transformers syntax
  77. 77. <#@ target type="Package" mergemode="LocalMerge" #> <Node> <Variables> <Variable Name="NewRows" DataType="Int32">0</Variable> </Variables> </Node> Transformers syntax: Target Type
  78. 78. <#@ target type="Package" mergemode="LocalMerge" #> <Node> <Variables> <Variable Name="NewRows" DataType="Int32">0</Variable> </Variables> </Node> Transformers syntax: Merge Mode
  79. 79. LocalMerge (update object) Keep original object and merge changes LocalReplace (replace object) Replace original object with new object(s) LocalMergeAndTypeReplace (convert object) Replace original object, but copy shared properties Merge Modes
  80. 80. Transformers and Frameworks
  81. 81. Get things done Start small Start simple Start with ugly code Keep going Expand Improve Deliver often
  82. 82. @cathrinew cathrinewilhelmsen.net no.linkedin.com/in/cathrinewilhelmsen contact@cathrinewilhelmsen.net slideshare.net/cathrinewilhelmsen Biml resources and references: cathrinewilhelmsen.net/biml

×