Level Up Your Biml: Best Practices and Coding Techniques (NTK 2016)

Cathrine Wilhelmsen
Cathrine WilhelmsenData & Analytics Solutions Architect in Evidi | Microsoft Data Platform MVP
Level Up Your Biml: Best Practices and Coding Techniques (NTK 2016)
Level Up Your Biml:
Best Practices and Coding Techniques
Cathrine Wilhelmsen
Session Description
You already know how to use Biml to build a staging environment in an hour, so
let's dive straight into some of the more advanced features of Biml.
Attend this session for an overview of Biml best practices and coding techniques.
Learn how to centralize and reuse code with include files and the CallBimlScript
method. Make your code easier to read and write by utilizing LINQ (Language-
Integrated Queries). Share code between files by using Annotations and
ObjectTags. And finally, if standard Biml is not enough to solve your problems,
you can create your own C# helper classes and extension methods to implement
custom logic.
Start improving your code today and level up your Biml in no time!
Cathrine Wilhelmsen
@cathrinew
cathrinewilhelmsen.net
Data Warehouse Architect
Business Intelligence Developer
You…
Know basic Biml and BimlScript
Completed BimlScript.com lessons
Have created a staging environment
…?
Today…
Code Management
Practical Biml Programming
C# Classes and Methods
… :)
Quick Recap of
Basic Biml
What is Biml?
Business Intelligence Markup Language
Easy to read and write XML language
Describes business intelligence objects:
• Databases, Schemas, Tables, Views, Columns
• SSIS Packages
• SSAS Cubes
• Metadata
What do you need?
…or you can use the new Biml tools
How does it work?
demotime!
Let's generate
some packages!
Ok, so we can go from Biml to SSIS…
…can we go from SSIS to Biml?
Yes! :)
demotime!
Let's reverse-engineer
some packages!
The magic is in the BimlScript!
Extend Biml with C# or VB code blocks
Import database structure and metadata
Loop over tables and columns
Expressions replace static values
BimlScript allows you to control and manipulate Biml code
BimlScript Code Nuggets
<# … #> Control Nuggets (Control logic)
<#= … #> Text Nuggets (Returns string)
<#@ … #> Directives (Compiler instructions)
<#+ … #> Class Nuggets (Create C# classes)
How does it work?
Yes, but how does it work?
Yes, but how does it actually work?
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<# foreach (var table in RootNode.Tables) { #>
<Package Name="Load_<#=table.Name#>"></Package>
<# } #>
</Packages>
</Biml> <Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<Package Name="Load_Customer"/>
<Package Name="Load_Product"/>
<Package Name="Load_Sales"/>
</Packages>
</Biml>
Biml vs. BimlScript
Automate, control and
manipulate Biml with C#
Flat XML
"Just text"
demotime!
Let's generate
a lot of packages!
Code
Management
Don't Repeat Yourself
Move common code to separate files
Centralize and reuse in many projects
Update code once for all projects
1. Include files
2. CallBimlScript with Parameters
3. Tiered Biml files
BimlExpress vs. BimlOnline / BimlStudio
"Black Box"
Only SSIS packages visible
Visual Editors
All in-memory objects visible
Include Files
Include common code in multiple files and projects
Can include many file types: .biml .txt .sql .cs
Use the include directive
<#@ include file="CommonCode.biml" #>
The directive will be replaced by the included file
Include pulls code from the included file into the main file
Works like an automated Copy & Paste
Include Files
Include Files
Include Files
CallBimlScript with Parameters
Works like a parameterized include
File to be called (callee) specifies input parameters it accepts
<#@ property name="Parameter" type="String" #>
File that calls (caller) passes input parameters
<#=CallBimlScript("CommonCode.biml", Parameter)#>
CallBimlScript pushes parameters from the caller to the callee, and the
callee returns code
CallBimlScript with Parameters
CallBimlScript with Parameters
CallBimlScript with Parameters
CallBimlScript with Parameters
CallBimlScript with Parameters
Tiered Biml Files
Split Biml code in multiple files and use the template directive:
<#@ template tier="1" #>
Create objects in-memory from lowest to highest tier to:
• Solve logical dependencies
• Simulate manual workflows
In-memory objects are added to the RootNode
Higher tiers can get objects added to RootNode in lower tiers
What is this RootNode?
The RootNode contains all in-memory objects:
• Connections, Databases, Schemas, Tables
• Projects, Packages
• Annotations, Metadata
Query the RootNode to loop over collections:
<# foreach (var table in RootNode.Tables) { #>
Query the RootNode to get specific objects:
<#=RootNode.Tables["Product"].Schema#>
Inside the Black Box: Tiered Biml Files
<#@ template tier="1" #>
<Connections>...</Connections>
<#@ template tier="2" #>
<Packages>...</Packages>
<#@ template tier="3" #>
<Package>...</Package>
Inside the Black Box: Tiered Biml Files
<#@ template tier="1" #>
<Connections>...</Connections>
<#@ template tier="2" #>
<Packages>...</Packages>
<#@ template tier="3" #>
<Package>...</Package>
Inside the Black Box: Tiered Biml Files
<#@ template tier="1" #>
<Connections>...</Connections>
<#@ template tier="2" #>
<Packages>...</Packages>
<#@ template tier="3" #>
<Package>...</Package>
Inside the Black Box: Tiered Biml Files
<#@ template tier="1" #>
<Connections>...</Connections>
<#@ template tier="2" #>
<Packages>...</Packages>
<#@ template tier="3" #>
<Package>...</Package>
Inside the Black Box: Tiered Biml Files
<#@ template tier="1" #>
<Connections>...</Connections>
<#@ template tier="2" #>
<Packages>...</Packages>
<#@ template tier="3" #>
<Package>...</Package>
Inside the Black Box: Tiered Biml Files
<#@ template tier="1" #>
<Connections>...</Connections>
<#@ template tier="2" #>
<Packages>...</Packages>
<#@ template tier="3" #>
<Package>...</Package>
Inside the Black Box: Tiered Biml Files
<#@ template tier="1" #>
<Connections>...</Connections>
<#@ template tier="2" #>
<Packages>...</Packages>
<#@ template tier="3" #>
<Package>...</Package>
Inside the Black Box: Tiered Biml Files
<#@ template tier="1" #>
<Connections>...</Connections>
<#@ template tier="2" #>
<Packages>...</Packages>
<#@ template tier="3" #>
<Package>...</Package>
Inside the Black Box: Tiered Biml Files
<#@ template tier="1" #>
<Connections>...</Connections>
<#@ template tier="2" #>
<Packages>...</Packages>
<#@ template tier="3" #>
<Package>...</Package>
Inside the Black Box: Tiered Biml Files
<#@ template tier="1" #>
<Connections>...</Connections>
<#@ template tier="2" #>
<Packages>...</Packages>
<#@ template tier="3" #>
<Package>...</Package>
Inside the Black Box: Tiered Biml Files
How do you use Tiered Biml files?
1. Create Biml files with specified tiers
2. Select all the tiered Biml files
3. Right-click and click Generate SSIS Packages
1
2
3
demotime!
How does this
actually work?
Debugging Biml
Debugging Biml
BimlExpress is a "black box":
• You can only see the generated SSIS packages
• It is not possible to see the compiled Biml first
Add a high-tier helper file to save compiled, flat Biml to file
• Check Biml For Errors to save flat Biml without generating packages
SaveFlatBimlToFile.biml
Add the helper file to your project…
<#@ template tier="999" #>
<# System.IO.File.WriteAllText(
@"C:BimlFlatBiml.xml",
RootNode.GetBiml()
); #>
SaveFlatBimlToFile.biml
…with a high tier so it is executed as the last step
<#@ template tier="999" #>
<# System.IO.File.WriteAllText(
@"C:BimlFlatBiml.xml",
RootNode.GetBiml()
); #>
SaveFlatBimlToFile.biml
It creates a file…
<#@ template tier="999" #>
<# System.IO.File.WriteAllText(
@"C:BimlFlatBiml.xml",
RootNode.GetBiml()
); #>
SaveFlatBimlToFile.biml
…at the specified path…
<#@ template tier="999" #>
<# System.IO.File.WriteAllText(
@"C:BimlFlatBiml.xml",
RootNode.GetBiml()
); #>
SaveFlatBimlToFile.biml
…with all the Biml for all the object in RootNode
<#@ template tier="999" #>
<# System.IO.File.WriteAllText(
@"C:BimlFlatBiml.xml",
RootNode.GetBiml()
); #>
How do you use this helper file?
1. Create the helper file
2. Select all the Biml files and the helper file
3. Right-click and click Check Biml For Errors
1
2
3
demotime!
How is this
helper file used?
Annotations and
ObjectTags
Annotations and ObjectTags
Biml Annotations != SSIS Annotations
Annotations are string/string Key/Value pairs
ObjectTags are string/object Key/Value pairs
Use Annotations and ObjectTags to pass code
between Biml files
Annotations
Create annotations:
<OleDbConnection Name="Destination" ConnectionString="…">
<Annotations>
<Annotation Tag="Schema">AW2014</Annotation>
</Annotations>
</OleDbConnection>
Use annotations:
<# var destinationSchema =
RootNode.OleDbConnections["Destination"].GetTag("Schema"); #>
ObjectTags
Create ObjectTags:
<#
RootNode.OleDbConnections["Destination"].ObjectTag["TableFilter"] =
new List<string> {"Product","ProductSubcategory","ProductCategory"};
#>
Use ObjectTags:
<#
var TableFilter = (List<string>)
RootNode.OleDbConnections["Destination"].ObjectTag["TableFilter"];
#>
LINQ
LINQ (Language-Integrated Query)
One language to query:
SQL Server Databases
XML Documents
Datasets
Collections
Two ways to write queries:
SQL-like Syntax
Extension Methods
LINQ Extension Methods
..and many, many more!
Sort
OrderBy, ThenBy
Filter
Where, OfType
Group
GroupBy
Aggregate
Count, Sum
Check Collections
All, Any, Contains
Get Elements
First, Last, ElementAt
Project Collections
Select, SelectMany
LINQ Extension Methods
var numConnections = RootNode.Connections.Count()
foreach (var table in RootNode.Tables.Where(…))
if (RootNode.Packages.Any(…))
LINQ and Lambda expressions
Use lambda expressions to filter or specify values:
.Where(table => table.Schema.Name == "Production")
.OrderBy(table => table.Name)
LINQ and Lambda expressions
For each element in the collection…
.Where(table => table.Schema.Name == "Production")
.OrderBy(table => table.Name)
LINQ and Lambda expressions
…evaluate a criteria or get a value:
.Where(table => table.Schema.Name == "Production")
.OrderBy(table => table.Name)
LINQ: Filter collections
Where()
Returns the filtered collection with all elements that meet the criteria
RootNode.Tables.Where(t => t.Schema.Name == "Production")
OfType()
Returns the filtered collection with all elements of the specified type
RootNode.Connections.OfType<AstExcelOleDbConnectionNode>()
LINQ: Sort collections
OrderBy()
Returns the collection sorted by key…
RootNode.Tables.OrderBy(t => t.Name)
ThenBy()
…then sorted by secondary key
RootNode.Tables.OrderBy(t => t.Schema.Name)
.ThenBy(t => t.Name)
LINQ: Sort collections
OrderByDescending()
Returns the collection sorted by key…
RootNode.Tables.OrderByDescending(t => t.Name)
ThenByDescending()
…then sorted by secondary key
RootNode.Tables.OrderBy(t => t.Schema.Name)
.ThenByDescending(t => t.Name)
LINQ: Sort collections
Reverse()
Returns the collection sorted in reverse order
RootNode.Tables.Reverse()
LINQ: Group collections
GroupBy()
Returns a collection of key-value pairs where each value is a new collection
RootNode.Tables.GroupBy(t => t.Schema.Name)
LINQ: Aggregate collections
Count()
Returns the number of elements in the collection
RootNode.Tables.Count()
RootNode.Tables.Count(t => t.Schema.Name == "Production")
LINQ: Aggregate collections
Sum()
Returns the sum of the (numeric) values in the collection
RootNode.Tables.Sum(t => t.Columns.Count)
Average()
Returns the average value of the (numeric) values in the collection
RootNode.Tables.Average(t => t.Columns.Count)
LINQ: Aggregate collections
Min()
Returns the minimum value of the (numeric) values in the collection
RootNode.Tables.Min(t => t.Columns.Count)
Max()
Returns the maximum value of the (numeric) values in the collection
RootNode.Tables.Max(t => t.Columns.Count)
LINQ: Check collections
All()
Returns true if all elements in the collection meet the criteria
RootNode.Databases.All(d => d.Name.StartsWith("A"))
Any()
Returns true if any element in the collection meets the criteria
RootNode.Databases.Any(d => d.Name.Contains("DW"))
LINQ: Check collections
Contains()
Returns true if collection contains element
RootNode.Databases.Contains(AdventureWorks2014)
LINQ: Get elements
First()
Returns the first element in the collection (that meets the criteria)
RootNode.Tables.First()
RootNode.Tables.First(t => t.Schema.Name == "Production")
FirstOrDefault()
Returns the first element in the collection or default value (that meets the criteria)
RootNode.Tables.FirstOrDefault()
RootNode.Tables.FirstOrDefault(t => t.Schema.Name == "Production")
LINQ: Get elements
Last()
Returns the last element in the collection (that meets the criteria)
RootNode.Tables.Last()
RootNode.Tables.Last(t => t.Schema.Name == "Production")
LastOrDefault()
Returns the last element in the collection or default value (that meets the criteria)
RootNode.Tables.LastOrDefault()
RootNode.Tables.LastOrDefault(t => t.Schema.Name == "Production")
LINQ: Get elements
ElementAt()
Returns the element in the collection at the specified index
RootNode.Tables.ElementAt(42)
ElementAtOrDefault()
Returns the element in the collection or default value at the specified index
RootNode.Tables.ElementAtOrDefault(42)
LINQ: Project collections
Select()
Creates a new collection from one collection
A list of table names:
RootNode.Tables.Select(t => t.Name)
A list of table and schema names:
RootNode.Tables.Select(t => new {t.Name, t.Schema.Name})
LINQ: Project collections
SelectMany()
Creates a new collection from many collections and merges the collections
A list of all columns from all tables:
RootNode.Tables.SelectMany(t => t.Columns)
demotime!
How is LINQ used
in Biml projects?
C#
Classes and
Methods
C# Classes and Methods
BimlScript and LINQ not enough?
Need to reuse C# code?
Create your own classes and methods!
C# Classes and Methods: From this…
public static class HelperClass {
public static bool AnnotationTagExists(AstNode node, string tag) {
if (node.GetTag(tag) != "") {
return true;
} else {
return false;
}
}
}
C# Classes and Methods: …to this
public static class HelperClass {
public static bool AnnotationTagExists(AstNode node, string tag) {
return (node.GetTag(tag) != "") ? true : false;
}
} * For bools you can just use:
return (node.GetTag(tag) != "");
But in this example we'll use the verbose, SSIS-like syntax
because it can be reused with other data types, like…
C# Classes and Methods: …or this
public static class HelperClass {
public static string AnnotationTagExists(AstNode node, string tag) {
return (node.GetTag(tag) != "") ? "Yes" : "No";
}
}
Where do you put your code?
Inline code nuggets
Included Biml files with code nuggets
Reference code files
C# Classes and Methods: Inline
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<# foreach (var table in RootNode.Tables) { #>
<# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #>
...
<# } #>
<# } #>
</Biml>
<#+
public static class HelperClass {
public static bool AnnotationTagExists(AstNode node, string tag) {
return (node.GetTag(tag) != "") ? true : false;
}
}
#>
C# Classes and Methods: Included Files
<#@ include file="HelperClass.biml" #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<# foreach (var table in RootNode.Tables) { #>
<# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #>
...
<# } #>
<# } #>
</Biml>
<#+
public static class HelperClass {
public static bool AnnotationTagExists(AstNode node, string tag) {
return (node.GetTag(tag) != "") ? true : false;
}
}
#>
C# Classes and Methods: Code Files
<#@ code file="HelperClass.cs" #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<# foreach (var table in RootNode.Tables) { #>
<# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #>
...
<# } #>
<# } #>
</Biml>
public static class HelperClass {
public static bool AnnotationTagExists(AstNode node, string tag) {
return (node.GetTag(tag) != "") ? true : false;
}
}
C#
Extension
Methods
Extension Methods
"Make it look like the method belongs to
an object instead of a helper class"
Extension Methods: From this…
<#@ code file="HelperClass.cs" #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<# foreach (var table in RootNode.Tables) { #>
<# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #>
...
<# } #>
<# } #>
</Biml>
public static class HelperClass {
public static bool AnnotationTagExists(AstNode node, string tag) {
return (node.GetTag(tag) != "") ? true : false;
}
}
Extension Methods: …to this
<#@ code file="HelperClass.cs" #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<# foreach (var table in RootNode.Tables) { #>
<# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #>
...
<# } #>
<# } #>
</Biml>
public static class HelperClass {
public static bool AnnotationTagExists(this AstNode node, string tag) {
return (node.GetTag(tag) != "") ? true : false;
}
}
Extension Methods: …to this
<#@ code file="HelperClass.cs" #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<# foreach (var table in RootNode.Tables) { #>
<# if (table.AnnotationTagExists("SourceSchema")) { #>
...
<# } #>
<# } #>
</Biml>
public static class HelperClass {
public static bool AnnotationTagExists(this AstNode node, string tag) {
return (node.GetTag(tag) != "") ? true : false;
}
}
Extension Methods: …to this :)
<#@ code file="HelperClass.cs" #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<# foreach (var table in RootNode.Tables.Where(t =>
t.AnnotationTagExists("SourceSchema")) { #>
...
<# } #>
<# } #>
</Biml>
public static class HelperClass {
public static bool AnnotationTagExists(this AstNode node, string tag) {
return (node.GetTag(tag) != "") ? true : false;
}
}
Questions?
Get things done
Start small
Start simple
Start with ugly code
Keep going
Expand
Improve
Deliver often
Izpolnite anketo!
Vam je bilo predavanje všeč?
Ste se naučili kaj novega?
Vaše mnenje nam veliko pomeni!
Da bo NT konferenca prihodnje leto še boljša, vas
prosimo, da izpolnite anketo o zadovoljstvu, ki jo
najdete v svojem NTK spletnem profilu.
@cathrinew
cathrinewilhelmsen.net
linkedin.com/in/cathrinewilhelmsen
contact@cathrinewilhelmsen.net
slideshare.net/cathrinewilhelmsen
Biml resources and references:
cathrinewilhelmsen.net/biml
Level Up Your Biml: Best Practices and Coding Techniques (NTK 2016)
1 of 107

Recommended

Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Oslo) by
Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Oslo)Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Oslo)
Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Oslo)Cathrine Wilhelmsen
1.6K views115 slides
Biml for Beginners: Speed up your SSIS development (Malta Microsoft Data Plat... by
Biml for Beginners: Speed up your SSIS development (Malta Microsoft Data Plat...Biml for Beginners: Speed up your SSIS development (Malta Microsoft Data Plat...
Biml for Beginners: Speed up your SSIS development (Malta Microsoft Data Plat...Cathrine Wilhelmsen
3.4K views96 slides
Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Sacrame... by
Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Sacrame...Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Sacrame...
Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Sacrame...Cathrine Wilhelmsen
7.1K views113 slides
Biml for Beginners - Generating SSIS Packages with BimlScript (SQLSaturday Ex... by
Biml for Beginners - Generating SSIS Packages with BimlScript (SQLSaturday Ex...Biml for Beginners - Generating SSIS Packages with BimlScript (SQLSaturday Ex...
Biml for Beginners - Generating SSIS Packages with BimlScript (SQLSaturday Ex...Cathrine Wilhelmsen
5.9K views63 slides
Generate SSIS packages automatically with Biml and BimlScript (SQLKonferenz 2... by
Generate SSIS packages automatically with Biml and BimlScript (SQLKonferenz 2...Generate SSIS packages automatically with Biml and BimlScript (SQLKonferenz 2...
Generate SSIS packages automatically with Biml and BimlScript (SQLKonferenz 2...Cathrine Wilhelmsen
3.7K views59 slides
Level Up Your Biml: Best Practices and Coding Techniques (TUGA IT 2016) by
Level Up Your Biml: Best Practices and Coding Techniques (TUGA IT 2016)Level Up Your Biml: Best Practices and Coding Techniques (TUGA IT 2016)
Level Up Your Biml: Best Practices and Coding Techniques (TUGA IT 2016)Cathrine Wilhelmsen
1.9K views108 slides

More Related Content

What's hot

Don't Repeat Yourself - An Introduction to Agile SSIS Development (24 Hours o... by
Don't Repeat Yourself - An Introduction to Agile SSIS Development (24 Hours o...Don't Repeat Yourself - An Introduction to Agile SSIS Development (24 Hours o...
Don't Repeat Yourself - An Introduction to Agile SSIS Development (24 Hours o...Cathrine Wilhelmsen
3.6K views55 slides
Biml Academy 2 - Lesson 5: Importing source metadata into Biml by
Biml Academy 2 - Lesson 5: Importing source metadata into BimlBiml Academy 2 - Lesson 5: Importing source metadata into Biml
Biml Academy 2 - Lesson 5: Importing source metadata into BimlCathrine Wilhelmsen
8.6K views21 slides
Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Denver) by
Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Denver)Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Denver)
Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Denver)Cathrine Wilhelmsen
3.7K views133 slides
Biml for Beginners: Speed up your SSIS development (SQLBits XV) by
Biml for Beginners: Speed up your SSIS development (SQLBits XV)Biml for Beginners: Speed up your SSIS development (SQLBits XV)
Biml for Beginners: Speed up your SSIS development (SQLBits XV)Cathrine Wilhelmsen
1.5K views57 slides
Biml for Beginners: Speed up your SSIS development (SQLSaturday Vancouver) by
Biml for Beginners: Speed up your SSIS development (SQLSaturday Vancouver)Biml for Beginners: Speed up your SSIS development (SQLSaturday Vancouver)
Biml for Beginners: Speed up your SSIS development (SQLSaturday Vancouver)Cathrine Wilhelmsen
3.4K views95 slides
Biml for Beginners: Speed up your SSIS development (SQLSaturday Chicago) by
Biml for Beginners: Speed up your SSIS development (SQLSaturday Chicago)Biml for Beginners: Speed up your SSIS development (SQLSaturday Chicago)
Biml for Beginners: Speed up your SSIS development (SQLSaturday Chicago)Cathrine Wilhelmsen
2.7K views82 slides

What's hot(20)

Don't Repeat Yourself - An Introduction to Agile SSIS Development (24 Hours o... by Cathrine Wilhelmsen
Don't Repeat Yourself - An Introduction to Agile SSIS Development (24 Hours o...Don't Repeat Yourself - An Introduction to Agile SSIS Development (24 Hours o...
Don't Repeat Yourself - An Introduction to Agile SSIS Development (24 Hours o...
Cathrine Wilhelmsen3.6K views
Biml Academy 2 - Lesson 5: Importing source metadata into Biml by Cathrine Wilhelmsen
Biml Academy 2 - Lesson 5: Importing source metadata into BimlBiml Academy 2 - Lesson 5: Importing source metadata into Biml
Biml Academy 2 - Lesson 5: Importing source metadata into Biml
Cathrine Wilhelmsen8.6K views
Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Denver) by Cathrine Wilhelmsen
Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Denver)Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Denver)
Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Denver)
Cathrine Wilhelmsen3.7K views
Biml for Beginners: Speed up your SSIS development (SQLBits XV) by Cathrine Wilhelmsen
Biml for Beginners: Speed up your SSIS development (SQLBits XV)Biml for Beginners: Speed up your SSIS development (SQLBits XV)
Biml for Beginners: Speed up your SSIS development (SQLBits XV)
Cathrine Wilhelmsen1.5K views
Biml for Beginners: Speed up your SSIS development (SQLSaturday Vancouver) by Cathrine Wilhelmsen
Biml for Beginners: Speed up your SSIS development (SQLSaturday Vancouver)Biml for Beginners: Speed up your SSIS development (SQLSaturday Vancouver)
Biml for Beginners: Speed up your SSIS development (SQLSaturday Vancouver)
Cathrine Wilhelmsen3.4K views
Biml for Beginners: Speed up your SSIS development (SQLSaturday Chicago) by Cathrine Wilhelmsen
Biml for Beginners: Speed up your SSIS development (SQLSaturday Chicago)Biml for Beginners: Speed up your SSIS development (SQLSaturday Chicago)
Biml for Beginners: Speed up your SSIS development (SQLSaturday Chicago)
Cathrine Wilhelmsen2.7K views
Don't Repeat Yourself - Agile SSIS Development with Biml and BimlScript (SQL ... by Cathrine Wilhelmsen
Don't Repeat Yourself - Agile SSIS Development with Biml and BimlScript (SQL ...Don't Repeat Yourself - Agile SSIS Development with Biml and BimlScript (SQL ...
Don't Repeat Yourself - Agile SSIS Development with Biml and BimlScript (SQL ...
S.M.A.R.T. Biml - Standardize, Model, Automate, Reuse and Transform (SQLSatur... by Cathrine Wilhelmsen
S.M.A.R.T. Biml - Standardize, Model, Automate, Reuse and Transform (SQLSatur...S.M.A.R.T. Biml - Standardize, Model, Automate, Reuse and Transform (SQLSatur...
S.M.A.R.T. Biml - Standardize, Model, Automate, Reuse and Transform (SQLSatur...
Cathrine Wilhelmsen6.4K views
Biml for Beginners: Speed up your SSIS development (SQLSaturday Vienna) by Cathrine Wilhelmsen
Biml for Beginners: Speed up your SSIS development (SQLSaturday Vienna)Biml for Beginners: Speed up your SSIS development (SQLSaturday Vienna)
Biml for Beginners: Speed up your SSIS development (SQLSaturday Vienna)
Cathrine Wilhelmsen2.8K views
Biml for Beginners - Generating SSIS Packages with BimlScript (SQLSaturday Go... by Cathrine Wilhelmsen
Biml for Beginners - Generating SSIS Packages with BimlScript (SQLSaturday Go...Biml for Beginners - Generating SSIS Packages with BimlScript (SQLSaturday Go...
Biml for Beginners - Generating SSIS Packages with BimlScript (SQLSaturday Go...
Cathrine Wilhelmsen2.2K views
Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLSat... by Cathrine Wilhelmsen
Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLSat...Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLSat...
Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLSat...
Cathrine Wilhelmsen1.4K views
Upgrading from SSIS Package Deployment to Project Deployment (SQLSaturday Den... by Cathrine Wilhelmsen
Upgrading from SSIS Package Deployment to Project Deployment (SQLSaturday Den...Upgrading from SSIS Package Deployment to Project Deployment (SQLSaturday Den...
Upgrading from SSIS Package Deployment to Project Deployment (SQLSaturday Den...
Cathrine Wilhelmsen3.4K views
Biml for Beginners: Speed up your SSIS development (SQL PASS Edmonton ) by Cathrine Wilhelmsen
Biml for Beginners: Speed up your SSIS development (SQL PASS Edmonton )Biml for Beginners: Speed up your SSIS development (SQL PASS Edmonton )
Biml for Beginners: Speed up your SSIS development (SQL PASS Edmonton )
Biml for Beginners: Speed up your SSIS development (SQLSaturday Tallinn) by Cathrine Wilhelmsen
Biml for Beginners: Speed up your SSIS development (SQLSaturday Tallinn)Biml for Beginners: Speed up your SSIS development (SQLSaturday Tallinn)
Biml for Beginners: Speed up your SSIS development (SQLSaturday Tallinn)
Cathrine Wilhelmsen1.2K views
Level Up Your Biml: Best Practices and Coding Techniques (SQLBits 2018) by Cathrine Wilhelmsen
Level Up Your Biml: Best Practices and Coding Techniques (SQLBits 2018)Level Up Your Biml: Best Practices and Coding Techniques (SQLBits 2018)
Level Up Your Biml: Best Practices and Coding Techniques (SQLBits 2018)
Cathrine Wilhelmsen4.3K views
Biml for Beginners: Script and Automate SSIS development (SQLSaturday Chicago) by Cathrine Wilhelmsen
Biml for Beginners: Script and Automate SSIS development (SQLSaturday Chicago)Biml for Beginners: Script and Automate SSIS development (SQLSaturday Chicago)
Biml for Beginners: Script and Automate SSIS development (SQLSaturday Chicago)
Biml for Beginners: Script and Automate SSIS development (Capital Area SQL Se... by Cathrine Wilhelmsen
Biml for Beginners: Script and Automate SSIS development (Capital Area SQL Se...Biml for Beginners: Script and Automate SSIS development (Capital Area SQL Se...
Biml for Beginners: Script and Automate SSIS development (Capital Area SQL Se...
Cathrine Wilhelmsen2.5K views
Biml for Beginners: Speed up your SSIS development (SQLSaturday Nashville) by Cathrine Wilhelmsen
Biml for Beginners: Speed up your SSIS development (SQLSaturday Nashville)Biml for Beginners: Speed up your SSIS development (SQLSaturday Nashville)
Biml for Beginners: Speed up your SSIS development (SQLSaturday Nashville)
Biml for Beginners: Script and Automate SSIS development (Malibu SQL Server U... by Cathrine Wilhelmsen
Biml for Beginners: Script and Automate SSIS development (Malibu SQL Server U...Biml for Beginners: Script and Automate SSIS development (Malibu SQL Server U...
Biml for Beginners: Script and Automate SSIS development (Malibu SQL Server U...
Cathrine Wilhelmsen1.1K views
Tools and Tips For Data Warehouse Developers (SQLSaturday Slovenia) by Cathrine Wilhelmsen
Tools and Tips For Data Warehouse Developers (SQLSaturday Slovenia)Tools and Tips For Data Warehouse Developers (SQLSaturday Slovenia)
Tools and Tips For Data Warehouse Developers (SQLSaturday Slovenia)
Cathrine Wilhelmsen1.7K views

Similar to Level Up Your Biml: Best Practices and Coding Techniques (NTK 2016)

Level Up Your Biml: Best Practices and Coding Techniques (SQLDay 2018) by
Level Up Your Biml: Best Practices and Coding Techniques (SQLDay 2018)Level Up Your Biml: Best Practices and Coding Techniques (SQLDay 2018)
Level Up Your Biml: Best Practices and Coding Techniques (SQLDay 2018)Cathrine Wilhelmsen
880 views114 slides
Biml for Beginners: Script and Automate SSIS development (24 Hours of PASS: S... by
Biml for Beginners: Script and Automate SSIS development (24 Hours of PASS: S...Biml for Beginners: Script and Automate SSIS development (24 Hours of PASS: S...
Biml for Beginners: Script and Automate SSIS development (24 Hours of PASS: S...Cathrine Wilhelmsen
1.1K views63 slides
Magento - Design Integration Guideline - Bysoft China by
Magento - Design Integration Guideline - Bysoft ChinaMagento - Design Integration Guideline - Bysoft China
Magento - Design Integration Guideline - Bysoft ChinaBysoft Technologies
1.5K views26 slides
Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) by
Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018)Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018)
Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018)Cathrine Wilhelmsen
19.5K views114 slides
Open Writing! Collaborative Authoring for CloudStack Documentation by Jessica... by
Open Writing! Collaborative Authoring for CloudStack Documentation by Jessica...Open Writing! Collaborative Authoring for CloudStack Documentation by Jessica...
Open Writing! Collaborative Authoring for CloudStack Documentation by Jessica...buildacloud
2.2K views34 slides
Open writing-cloud-collab by
Open writing-cloud-collabOpen writing-cloud-collab
Open writing-cloud-collabKaren Vuong
731 views34 slides

Similar to Level Up Your Biml: Best Practices and Coding Techniques (NTK 2016)(15)

Level Up Your Biml: Best Practices and Coding Techniques (SQLDay 2018) by Cathrine Wilhelmsen
Level Up Your Biml: Best Practices and Coding Techniques (SQLDay 2018)Level Up Your Biml: Best Practices and Coding Techniques (SQLDay 2018)
Level Up Your Biml: Best Practices and Coding Techniques (SQLDay 2018)
Biml for Beginners: Script and Automate SSIS development (24 Hours of PASS: S... by Cathrine Wilhelmsen
Biml for Beginners: Script and Automate SSIS development (24 Hours of PASS: S...Biml for Beginners: Script and Automate SSIS development (24 Hours of PASS: S...
Biml for Beginners: Script and Automate SSIS development (24 Hours of PASS: S...
Cathrine Wilhelmsen1.1K views
Magento - Design Integration Guideline - Bysoft China by Bysoft Technologies
Magento - Design Integration Guideline - Bysoft ChinaMagento - Design Integration Guideline - Bysoft China
Magento - Design Integration Guideline - Bysoft China
Bysoft Technologies1.5K views
Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) by Cathrine Wilhelmsen
Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018)Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018)
Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018)
Cathrine Wilhelmsen19.5K views
Open Writing! Collaborative Authoring for CloudStack Documentation by Jessica... by buildacloud
Open Writing! Collaborative Authoring for CloudStack Documentation by Jessica...Open Writing! Collaborative Authoring for CloudStack Documentation by Jessica...
Open Writing! Collaborative Authoring for CloudStack Documentation by Jessica...
buildacloud2.2K views
Open writing-cloud-collab by Karen Vuong
Open writing-cloud-collabOpen writing-cloud-collab
Open writing-cloud-collab
Karen Vuong731 views
Extending the build workflow of TFS 2010 by Gian Maria Ricci
Extending the build workflow of TFS 2010Extending the build workflow of TFS 2010
Extending the build workflow of TFS 2010
Gian Maria Ricci328 views
Simple build tool by Knoldus Inc.
Simple build toolSimple build tool
Simple build tool
Knoldus Inc.1.2K views
Le Wagon Tokyo | Build your Landing Page in 2 hours by YannKlein2
Le Wagon Tokyo | Build your Landing Page in 2 hoursLe Wagon Tokyo | Build your Landing Page in 2 hours
Le Wagon Tokyo | Build your Landing Page in 2 hours
YannKlein266 views
Stencil the time for vanilla web components has arrived by Gil Fink
Stencil the time for vanilla web components has arrivedStencil the time for vanilla web components has arrived
Stencil the time for vanilla web components has arrived
Gil Fink290 views
Joomla! Day UK 2009 Basic Templates by Andy Wallace
Joomla! Day UK 2009 Basic TemplatesJoomla! Day UK 2009 Basic Templates
Joomla! Day UK 2009 Basic Templates
Andy Wallace1.1K views
Joomla Day UK 2009 Basic Templates by Chris Davenport
Joomla Day UK 2009 Basic TemplatesJoomla Day UK 2009 Basic Templates
Joomla Day UK 2009 Basic Templates
Chris Davenport934 views
Lotusphere 2007 AD505 DevBlast 30 LotusScript Tips by Bill Buchan
Lotusphere 2007 AD505 DevBlast 30 LotusScript TipsLotusphere 2007 AD505 DevBlast 30 LotusScript Tips
Lotusphere 2007 AD505 DevBlast 30 LotusScript Tips
Bill Buchan804 views

More from Cathrine Wilhelmsen

Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac... by
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...Cathrine Wilhelmsen
131 views43 slides
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D... by
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D...Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D...
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D...Cathrine Wilhelmsen
47 views53 slides
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S... by
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...Cathrine Wilhelmsen
89 views51 slides
"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ... by
"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ..."I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ...
"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ...Cathrine Wilhelmsen
97 views62 slides
Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S... by
Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...
Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...Cathrine Wilhelmsen
687 views49 slides
6 Tips for Building Confidence as a Public Speaker (SQLBits 2022) by
6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)
6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)Cathrine Wilhelmsen
118 views30 slides

More from Cathrine Wilhelmsen(20)

Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac... by Cathrine Wilhelmsen
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D... by Cathrine Wilhelmsen
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D...Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D...
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D...
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S... by Cathrine Wilhelmsen
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...
"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ... by Cathrine Wilhelmsen
"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ..."I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ...
"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ...
Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S... by Cathrine Wilhelmsen
Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...
Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...
6 Tips for Building Confidence as a Public Speaker (SQLBits 2022) by Cathrine Wilhelmsen
6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)
6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)
Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu... by Cathrine Wilhelmsen
Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...
Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...
Cathrine Wilhelmsen3.8K views
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A... by Cathrine Wilhelmsen
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A... by Cathrine Wilhelmsen
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Understanding Azure Data Factory: The What, When, and Why (NIC 2020) by Cathrine Wilhelmsen
Understanding Azure Data Factory: The What, When, and Why (NIC 2020)Understanding Azure Data Factory: The What, When, and Why (NIC 2020)
Understanding Azure Data Factory: The What, When, and Why (NIC 2020)
Cathrine Wilhelmsen1.8K views
Azure Data Factory for the SSIS Developer (SentryOne Webinar) by Cathrine Wilhelmsen
Azure Data Factory for the SSIS Developer (SentryOne Webinar)Azure Data Factory for the SSIS Developer (SentryOne Webinar)
Azure Data Factory for the SSIS Developer (SentryOne Webinar)
Cathrine Wilhelmsen1.1K views
Azure Synapse Analytics Teaser (Microsoft TechX Oslo 2019) by Cathrine Wilhelmsen
Azure Synapse Analytics Teaser (Microsoft TechX Oslo 2019)Azure Synapse Analytics Teaser (Microsoft TechX Oslo 2019)
Azure Synapse Analytics Teaser (Microsoft TechX Oslo 2019)
Cathrine Wilhelmsen1.1K views
Lessons Learned: Understanding Azure Data Factory Pricing (Microsoft Ignite 2... by Cathrine Wilhelmsen
Lessons Learned: Understanding Azure Data Factory Pricing (Microsoft Ignite 2...Lessons Learned: Understanding Azure Data Factory Pricing (Microsoft Ignite 2...
Lessons Learned: Understanding Azure Data Factory Pricing (Microsoft Ignite 2...
Cathrine Wilhelmsen15.8K views
Building Dynamic Data Pipelines in Azure Data Factory (Microsoft Ignite 2019) by Cathrine Wilhelmsen
Building Dynamic Data Pipelines in Azure Data Factory (Microsoft Ignite 2019)Building Dynamic Data Pipelines in Azure Data Factory (Microsoft Ignite 2019)
Building Dynamic Data Pipelines in Azure Data Factory (Microsoft Ignite 2019)
Cathrine Wilhelmsen4.1K views
Creating Visual Transformations in Azure Data Factory (dataMinds Connect) by Cathrine Wilhelmsen
Creating Visual Transformations in Azure Data Factory (dataMinds Connect)Creating Visual Transformations in Azure Data Factory (dataMinds Connect)
Creating Visual Transformations in Azure Data Factory (dataMinds Connect)
Cathrine Wilhelmsen1.2K views
Building Dynamic Pipelines in Azure Data Factory (Data Saturday Holland) by Cathrine Wilhelmsen
Building Dynamic Pipelines in Azure Data Factory (Data Saturday Holland)Building Dynamic Pipelines in Azure Data Factory (Data Saturday Holland)
Building Dynamic Pipelines in Azure Data Factory (Data Saturday Holland)
Pipelines and Packages: Introduction to Azure Data Factory (Techorama NL 2019) by Cathrine Wilhelmsen
Pipelines and Packages: Introduction to Azure Data Factory (Techorama NL 2019)Pipelines and Packages: Introduction to Azure Data Factory (Techorama NL 2019)
Pipelines and Packages: Introduction to Azure Data Factory (Techorama NL 2019)
Cathrine Wilhelmsen1.4K views
Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) by Cathrine Wilhelmsen
Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019)Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019)
Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019)
Cathrine Wilhelmsen4.1K views
Building Dynamic Pipelines in Azure Data Factory (SQLSaturday Oslo) by Cathrine Wilhelmsen
Building Dynamic Pipelines in Azure Data Factory (SQLSaturday Oslo)Building Dynamic Pipelines in Azure Data Factory (SQLSaturday Oslo)
Building Dynamic Pipelines in Azure Data Factory (SQLSaturday Oslo)
Cathrine Wilhelmsen2.7K views
Uhms and Bunny Hands: Tips for Improving Your Presentation Skills (DataGrille... by Cathrine Wilhelmsen
Uhms and Bunny Hands: Tips for Improving Your Presentation Skills (DataGrille...Uhms and Bunny Hands: Tips for Improving Your Presentation Skills (DataGrille...
Uhms and Bunny Hands: Tips for Improving Your Presentation Skills (DataGrille...
Cathrine Wilhelmsen1.7K views

Recently uploaded

3196 The Case of The East River by
3196 The Case of The East River3196 The Case of The East River
3196 The Case of The East RiverErickANDRADE90
11 views4 slides
Building Real-Time Travel Alerts by
Building Real-Time Travel AlertsBuilding Real-Time Travel Alerts
Building Real-Time Travel AlertsTimothy Spann
109 views48 slides
Introduction to Microsoft Fabric.pdf by
Introduction to Microsoft Fabric.pdfIntroduction to Microsoft Fabric.pdf
Introduction to Microsoft Fabric.pdfishaniuudeshika
24 views16 slides
How Leaders See Data? (Level 1) by
How Leaders See Data? (Level 1)How Leaders See Data? (Level 1)
How Leaders See Data? (Level 1)Narendra Narendra
13 views76 slides
PTicketInput.pdf by
PTicketInput.pdfPTicketInput.pdf
PTicketInput.pdfstuartmcphersonflipm
376 views1 slide
Vikas 500 BIG DATA TECHNOLOGIES LAB.pdf by
Vikas 500 BIG DATA TECHNOLOGIES LAB.pdfVikas 500 BIG DATA TECHNOLOGIES LAB.pdf
Vikas 500 BIG DATA TECHNOLOGIES LAB.pdfvikas12611618
8 views30 slides

Recently uploaded(20)

3196 The Case of The East River by ErickANDRADE90
3196 The Case of The East River3196 The Case of The East River
3196 The Case of The East River
ErickANDRADE9011 views
Building Real-Time Travel Alerts by Timothy Spann
Building Real-Time Travel AlertsBuilding Real-Time Travel Alerts
Building Real-Time Travel Alerts
Timothy Spann109 views
Introduction to Microsoft Fabric.pdf by ishaniuudeshika
Introduction to Microsoft Fabric.pdfIntroduction to Microsoft Fabric.pdf
Introduction to Microsoft Fabric.pdf
ishaniuudeshika24 views
Vikas 500 BIG DATA TECHNOLOGIES LAB.pdf by vikas12611618
Vikas 500 BIG DATA TECHNOLOGIES LAB.pdfVikas 500 BIG DATA TECHNOLOGIES LAB.pdf
Vikas 500 BIG DATA TECHNOLOGIES LAB.pdf
vikas126116188 views
JConWorld_ Continuous SQL with Kafka and Flink by Timothy Spann
JConWorld_ Continuous SQL with Kafka and FlinkJConWorld_ Continuous SQL with Kafka and Flink
JConWorld_ Continuous SQL with Kafka and Flink
Timothy Spann100 views
UNEP FI CRS Climate Risk Results.pptx by pekka28
UNEP FI CRS Climate Risk Results.pptxUNEP FI CRS Climate Risk Results.pptx
UNEP FI CRS Climate Risk Results.pptx
pekka2811 views
Data structure and algorithm. by Abdul salam
Data structure and algorithm. Data structure and algorithm.
Data structure and algorithm.
Abdul salam 18 views
Understanding Hallucinations in LLMs - 2023 09 29.pptx by Greg Makowski
Understanding Hallucinations in LLMs - 2023 09 29.pptxUnderstanding Hallucinations in LLMs - 2023 09 29.pptx
Understanding Hallucinations in LLMs - 2023 09 29.pptx
Greg Makowski13 views
Cross-network in Google Analytics 4.pdf by GA4 Tutorials
Cross-network in Google Analytics 4.pdfCross-network in Google Analytics 4.pdf
Cross-network in Google Analytics 4.pdf
GA4 Tutorials6 views
Organic Shopping in Google Analytics 4.pdf by GA4 Tutorials
Organic Shopping in Google Analytics 4.pdfOrganic Shopping in Google Analytics 4.pdf
Organic Shopping in Google Analytics 4.pdf
GA4 Tutorials10 views
RuleBookForTheFairDataEconomy.pptx by noraelstela1
RuleBookForTheFairDataEconomy.pptxRuleBookForTheFairDataEconomy.pptx
RuleBookForTheFairDataEconomy.pptx
noraelstela167 views
Advanced_Recommendation_Systems_Presentation.pptx by neeharikasingh29
Advanced_Recommendation_Systems_Presentation.pptxAdvanced_Recommendation_Systems_Presentation.pptx
Advanced_Recommendation_Systems_Presentation.pptx
Chapter 3b- Process Communication (1) (1)(1) (1).pptx by ayeshabaig2004
Chapter 3b- Process Communication (1) (1)(1) (1).pptxChapter 3b- Process Communication (1) (1)(1) (1).pptx
Chapter 3b- Process Communication (1) (1)(1) (1).pptx
ayeshabaig20045 views
Short Story Assignment by Kelly Nguyen by kellynguyen01
Short Story Assignment by Kelly NguyenShort Story Assignment by Kelly Nguyen
Short Story Assignment by Kelly Nguyen
kellynguyen0118 views

Level Up Your Biml: Best Practices and Coding Techniques (NTK 2016)

  • 2. Level Up Your Biml: Best Practices and Coding Techniques Cathrine Wilhelmsen
  • 3. Session Description You already know how to use Biml to build a staging environment in an hour, so let's dive straight into some of the more advanced features of Biml. Attend this session for an overview of Biml best practices and coding techniques. Learn how to centralize and reuse code with include files and the CallBimlScript method. Make your code easier to read and write by utilizing LINQ (Language- Integrated Queries). Share code between files by using Annotations and ObjectTags. And finally, if standard Biml is not enough to solve your problems, you can create your own C# helper classes and extension methods to implement custom logic. Start improving your code today and level up your Biml in no time!
  • 5. You… Know basic Biml and BimlScript Completed BimlScript.com lessons Have created a staging environment …?
  • 6. Today… Code Management Practical Biml Programming C# Classes and Methods … :)
  • 8. What is Biml? Business Intelligence Markup Language Easy to read and write XML language Describes business intelligence objects: • Databases, Schemas, Tables, Views, Columns • SSIS Packages • SSAS Cubes • Metadata
  • 9. What do you need?
  • 10. …or you can use the new Biml tools
  • 11. How does it work?
  • 13. Ok, so we can go from Biml to SSIS…
  • 14. …can we go from SSIS to Biml?
  • 17. The magic is in the BimlScript! Extend Biml with C# or VB code blocks Import database structure and metadata Loop over tables and columns Expressions replace static values BimlScript allows you to control and manipulate Biml code
  • 18. BimlScript Code Nuggets <# … #> Control Nuggets (Control logic) <#= … #> Text Nuggets (Returns string) <#@ … #> Directives (Compiler instructions) <#+ … #> Class Nuggets (Create C# classes)
  • 19. How does it work?
  • 20. Yes, but how does it work?
  • 21. Yes, but how does it actually work? <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in RootNode.Tables) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="Load_Customer"/> <Package Name="Load_Product"/> <Package Name="Load_Sales"/> </Packages> </Biml>
  • 22. Biml vs. BimlScript Automate, control and manipulate Biml with C# Flat XML "Just text"
  • 25. Don't Repeat Yourself Move common code to separate files Centralize and reuse in many projects Update code once for all projects 1. Include files 2. CallBimlScript with Parameters 3. Tiered Biml files
  • 26. BimlExpress vs. BimlOnline / BimlStudio "Black Box" Only SSIS packages visible Visual Editors All in-memory objects visible
  • 27. Include Files Include common code in multiple files and projects Can include many file types: .biml .txt .sql .cs Use the include directive <#@ include file="CommonCode.biml" #> The directive will be replaced by the included file Include pulls code from the included file into the main file Works like an automated Copy & Paste
  • 31. CallBimlScript with Parameters Works like a parameterized include File to be called (callee) specifies input parameters it accepts <#@ property name="Parameter" type="String" #> File that calls (caller) passes input parameters <#=CallBimlScript("CommonCode.biml", Parameter)#> CallBimlScript pushes parameters from the caller to the callee, and the callee returns code
  • 37. Tiered Biml Files Split Biml code in multiple files and use the template directive: <#@ template tier="1" #> Create objects in-memory from lowest to highest tier to: • Solve logical dependencies • Simulate manual workflows In-memory objects are added to the RootNode Higher tiers can get objects added to RootNode in lower tiers
  • 38. What is this RootNode? The RootNode contains all in-memory objects: • Connections, Databases, Schemas, Tables • Projects, Packages • Annotations, Metadata Query the RootNode to loop over collections: <# foreach (var table in RootNode.Tables) { #> Query the RootNode to get specific objects: <#=RootNode.Tables["Product"].Schema#>
  • 39. Inside the Black Box: Tiered Biml Files <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  • 40. Inside the Black Box: Tiered Biml Files <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  • 41. Inside the Black Box: Tiered Biml Files <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  • 42. Inside the Black Box: Tiered Biml Files <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  • 43. Inside the Black Box: Tiered Biml Files <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  • 44. Inside the Black Box: Tiered Biml Files <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  • 45. Inside the Black Box: Tiered Biml Files <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  • 46. Inside the Black Box: Tiered Biml Files <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  • 47. Inside the Black Box: Tiered Biml Files <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  • 48. Inside the Black Box: Tiered Biml Files <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  • 49. Inside the Black Box: Tiered Biml Files
  • 50. How do you use Tiered Biml files? 1. Create Biml files with specified tiers 2. Select all the tiered Biml files 3. Right-click and click Generate SSIS Packages 1 2 3
  • 53. Debugging Biml BimlExpress is a "black box": • You can only see the generated SSIS packages • It is not possible to see the compiled Biml first Add a high-tier helper file to save compiled, flat Biml to file • Check Biml For Errors to save flat Biml without generating packages
  • 54. SaveFlatBimlToFile.biml Add the helper file to your project… <#@ template tier="999" #> <# System.IO.File.WriteAllText( @"C:BimlFlatBiml.xml", RootNode.GetBiml() ); #>
  • 55. SaveFlatBimlToFile.biml …with a high tier so it is executed as the last step <#@ template tier="999" #> <# System.IO.File.WriteAllText( @"C:BimlFlatBiml.xml", RootNode.GetBiml() ); #>
  • 56. SaveFlatBimlToFile.biml It creates a file… <#@ template tier="999" #> <# System.IO.File.WriteAllText( @"C:BimlFlatBiml.xml", RootNode.GetBiml() ); #>
  • 57. SaveFlatBimlToFile.biml …at the specified path… <#@ template tier="999" #> <# System.IO.File.WriteAllText( @"C:BimlFlatBiml.xml", RootNode.GetBiml() ); #>
  • 58. SaveFlatBimlToFile.biml …with all the Biml for all the object in RootNode <#@ template tier="999" #> <# System.IO.File.WriteAllText( @"C:BimlFlatBiml.xml", RootNode.GetBiml() ); #>
  • 59. How do you use this helper file? 1. Create the helper file 2. Select all the Biml files and the helper file 3. Right-click and click Check Biml For Errors 1 2 3
  • 62. Annotations and ObjectTags Biml Annotations != SSIS Annotations Annotations are string/string Key/Value pairs ObjectTags are string/object Key/Value pairs Use Annotations and ObjectTags to pass code between Biml files
  • 63. Annotations Create annotations: <OleDbConnection Name="Destination" ConnectionString="…"> <Annotations> <Annotation Tag="Schema">AW2014</Annotation> </Annotations> </OleDbConnection> Use annotations: <# var destinationSchema = RootNode.OleDbConnections["Destination"].GetTag("Schema"); #>
  • 64. ObjectTags Create ObjectTags: <# RootNode.OleDbConnections["Destination"].ObjectTag["TableFilter"] = new List<string> {"Product","ProductSubcategory","ProductCategory"}; #> Use ObjectTags: <# var TableFilter = (List<string>) RootNode.OleDbConnections["Destination"].ObjectTag["TableFilter"]; #>
  • 65. LINQ
  • 66. LINQ (Language-Integrated Query) One language to query: SQL Server Databases XML Documents Datasets Collections Two ways to write queries: SQL-like Syntax Extension Methods
  • 67. LINQ Extension Methods ..and many, many more! Sort OrderBy, ThenBy Filter Where, OfType Group GroupBy Aggregate Count, Sum Check Collections All, Any, Contains Get Elements First, Last, ElementAt Project Collections Select, SelectMany
  • 68. LINQ Extension Methods var numConnections = RootNode.Connections.Count() foreach (var table in RootNode.Tables.Where(…)) if (RootNode.Packages.Any(…))
  • 69. LINQ and Lambda expressions Use lambda expressions to filter or specify values: .Where(table => table.Schema.Name == "Production") .OrderBy(table => table.Name)
  • 70. LINQ and Lambda expressions For each element in the collection… .Where(table => table.Schema.Name == "Production") .OrderBy(table => table.Name)
  • 71. LINQ and Lambda expressions …evaluate a criteria or get a value: .Where(table => table.Schema.Name == "Production") .OrderBy(table => table.Name)
  • 72. LINQ: Filter collections Where() Returns the filtered collection with all elements that meet the criteria RootNode.Tables.Where(t => t.Schema.Name == "Production") OfType() Returns the filtered collection with all elements of the specified type RootNode.Connections.OfType<AstExcelOleDbConnectionNode>()
  • 73. LINQ: Sort collections OrderBy() Returns the collection sorted by key… RootNode.Tables.OrderBy(t => t.Name) ThenBy() …then sorted by secondary key RootNode.Tables.OrderBy(t => t.Schema.Name) .ThenBy(t => t.Name)
  • 74. LINQ: Sort collections OrderByDescending() Returns the collection sorted by key… RootNode.Tables.OrderByDescending(t => t.Name) ThenByDescending() …then sorted by secondary key RootNode.Tables.OrderBy(t => t.Schema.Name) .ThenByDescending(t => t.Name)
  • 75. LINQ: Sort collections Reverse() Returns the collection sorted in reverse order RootNode.Tables.Reverse()
  • 76. LINQ: Group collections GroupBy() Returns a collection of key-value pairs where each value is a new collection RootNode.Tables.GroupBy(t => t.Schema.Name)
  • 77. LINQ: Aggregate collections Count() Returns the number of elements in the collection RootNode.Tables.Count() RootNode.Tables.Count(t => t.Schema.Name == "Production")
  • 78. LINQ: Aggregate collections Sum() Returns the sum of the (numeric) values in the collection RootNode.Tables.Sum(t => t.Columns.Count) Average() Returns the average value of the (numeric) values in the collection RootNode.Tables.Average(t => t.Columns.Count)
  • 79. LINQ: Aggregate collections Min() Returns the minimum value of the (numeric) values in the collection RootNode.Tables.Min(t => t.Columns.Count) Max() Returns the maximum value of the (numeric) values in the collection RootNode.Tables.Max(t => t.Columns.Count)
  • 80. LINQ: Check collections All() Returns true if all elements in the collection meet the criteria RootNode.Databases.All(d => d.Name.StartsWith("A")) Any() Returns true if any element in the collection meets the criteria RootNode.Databases.Any(d => d.Name.Contains("DW"))
  • 81. LINQ: Check collections Contains() Returns true if collection contains element RootNode.Databases.Contains(AdventureWorks2014)
  • 82. LINQ: Get elements First() Returns the first element in the collection (that meets the criteria) RootNode.Tables.First() RootNode.Tables.First(t => t.Schema.Name == "Production") FirstOrDefault() Returns the first element in the collection or default value (that meets the criteria) RootNode.Tables.FirstOrDefault() RootNode.Tables.FirstOrDefault(t => t.Schema.Name == "Production")
  • 83. LINQ: Get elements Last() Returns the last element in the collection (that meets the criteria) RootNode.Tables.Last() RootNode.Tables.Last(t => t.Schema.Name == "Production") LastOrDefault() Returns the last element in the collection or default value (that meets the criteria) RootNode.Tables.LastOrDefault() RootNode.Tables.LastOrDefault(t => t.Schema.Name == "Production")
  • 84. LINQ: Get elements ElementAt() Returns the element in the collection at the specified index RootNode.Tables.ElementAt(42) ElementAtOrDefault() Returns the element in the collection or default value at the specified index RootNode.Tables.ElementAtOrDefault(42)
  • 85. LINQ: Project collections Select() Creates a new collection from one collection A list of table names: RootNode.Tables.Select(t => t.Name) A list of table and schema names: RootNode.Tables.Select(t => new {t.Name, t.Schema.Name})
  • 86. LINQ: Project collections SelectMany() Creates a new collection from many collections and merges the collections A list of all columns from all tables: RootNode.Tables.SelectMany(t => t.Columns)
  • 87. demotime! How is LINQ used in Biml projects?
  • 89. C# Classes and Methods BimlScript and LINQ not enough? Need to reuse C# code? Create your own classes and methods!
  • 90. C# Classes and Methods: From this… public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { if (node.GetTag(tag) != "") { return true; } else { return false; } } }
  • 91. C# Classes and Methods: …to this public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } } * For bools you can just use: return (node.GetTag(tag) != ""); But in this example we'll use the verbose, SSIS-like syntax because it can be reused with other data types, like…
  • 92. C# Classes and Methods: …or this public static class HelperClass { public static string AnnotationTagExists(AstNode node, string tag) { return (node.GetTag(tag) != "") ? "Yes" : "No"; } }
  • 93. Where do you put your code? Inline code nuggets Included Biml files with code nuggets Reference code files
  • 94. C# Classes and Methods: Inline <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #> ... <# } #> <# } #> </Biml> <#+ public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } } #>
  • 95. C# Classes and Methods: Included Files <#@ include file="HelperClass.biml" #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #> ... <# } #> <# } #> </Biml> <#+ public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } } #>
  • 96. C# Classes and Methods: Code Files <#@ code file="HelperClass.cs" #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #> ... <# } #> <# } #> </Biml> public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } }
  • 98. Extension Methods "Make it look like the method belongs to an object instead of a helper class"
  • 99. Extension Methods: From this… <#@ code file="HelperClass.cs" #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #> ... <# } #> <# } #> </Biml> public static class HelperClass { public static bool AnnotationTagExists(AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } }
  • 100. Extension Methods: …to this <#@ code file="HelperClass.cs" #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #> ... <# } #> <# } #> </Biml> public static class HelperClass { public static bool AnnotationTagExists(this AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } }
  • 101. Extension Methods: …to this <#@ code file="HelperClass.cs" #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables) { #> <# if (table.AnnotationTagExists("SourceSchema")) { #> ... <# } #> <# } #> </Biml> public static class HelperClass { public static bool AnnotationTagExists(this AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } }
  • 102. Extension Methods: …to this :) <#@ code file="HelperClass.cs" #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <# foreach (var table in RootNode.Tables.Where(t => t.AnnotationTagExists("SourceSchema")) { #> ... <# } #> <# } #> </Biml> public static class HelperClass { public static bool AnnotationTagExists(this AstNode node, string tag) { return (node.GetTag(tag) != "") ? true : false; } }
  • 104. Get things done Start small Start simple Start with ugly code Keep going Expand Improve Deliver often
  • 105. Izpolnite anketo! Vam je bilo predavanje všeč? Ste se naučili kaj novega? Vaše mnenje nam veliko pomeni! Da bo NT konferenca prihodnje leto še boljša, vas prosimo, da izpolnite anketo o zadovoljstvu, ki jo najdete v svojem NTK spletnem profilu.