SlideShare a Scribd company logo
1 of 98
Download to read offline
Biml Tips and Tricks
Not just for SSIS Packages!
Cathrine Wilhelmsen
SQLGrillen · June 22nd 2018
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Session Description
"Wait, what? Biml is not just for generating SSIS packages?"
Absolutely not! Come and see how you can use Biml (Business Intelligence
Markup Language) to save time and speed up other Data Warehouse
development tasks. You can generate complex T-SQL statements with Biml
instead of using dynamic SQL, create test data and populate static dimensions,
and even compare tables and views across multiple servers and databases.
Don't Repeat Yourself, start automating those boring, manual tasks today!
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Cathrine Wilhelmsen
@cathrinew cathrinew.net
Data Warehouse & Business Intelligence Consultant
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
…the next 60 minutes…
T-SQL
Test Data Comparing
Dimensions
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Wait…
Can't I use
<something else>?
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Of course!
But Biml works well
with source metadata
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
And…
Biml is fun!
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
But… Biml? How?
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Traditional Biml
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Crazy Fun Biml
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
What do you need?
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Preview Pane
The power is in the…
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Traditional BimlScript
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Traditional BimlScript
<# foreach (var table in RootNode.Tables) { #>
<Package Name="Load_<#=table.Name#>" />
<# } #>
<Package Name="Load_Customer" />
<Package Name="Load_Product" />
<Package Name="Load_Sales" />
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
BimlScript to Biml
Preview Pane
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
BimlScript to... ???
Preview Pane
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Crazy Fun BimlScript
<# foreach (var table in RootNode.Tables) { #>
Yay, a package! <#=table.Name#> :)
<# } #>
Yay, a package! Load_Customer :)
Yay, a package! Load_Product :)
Yay, a package! Load_Sales :)
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
T-SQL from Biml
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Built-in
T-SQL from Biml
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Built-in T-SQL from Biml
Functional T-SQL
GetSelectSql
GetDropAndCreateDdl
GetTableSql
SSMS Snippets
GetInsertSql
GetUpdateSql
GetDeleteSql
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Functional T-SQL: GetSelectSql
SELECT
[Column1]
,[Column2]
FROM [Schema].[Table]
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Functional T-SQL: GetDropAndCreateDdl
IF EXISTS (...)
DROP TABLE [Schema].[Table]
GO
CREATE TABLE [Schema].[Table] (
[Column1] int IDENTITY(1,1) NOT NULL
,[Column2] nvarchar(50) NOT NULL
-- Constraints
)
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Functional T-SQL: GetTableSql
IF EXISTS (...)
DROP TABLE [Schema].[Table]
GO
CREATE TABLE [Schema].[Table] (
[Column1] int IDENTITY(1,1) NOT NULL
,[Column2] nvarchar(50) NOT NULL
-- Constraints
)
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Functional T-SQL
Use directly in SSIS
Copy and run in SSMS
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
SSMS Snippets: GetDeleteSql
DELETE FROM [Schema].[Table]
WHERE <Search Conditions,,>
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
SSMS Snippets: GetInsertSql
INSERT INTO [Schema].[Table] (
[Column1]
,[Column1]
) VALUES (
<Column1,int,>,
<Column2,nvarchar(50),>
)
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
SSMS Snippets: GetUpdateSql
UPDATE [Schema].[Table] SET
[Column1] = <Column1, int,>,
[Column2] = <Column2, nvarchar(50),>
WHERE <Search Conditions,,>
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Copy and run in SSMS
Replace Template Parameters with actual values
CTRL SHIFT M
SSMS Snippets
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Custom
T-SQL from Biml
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Custom T-SQL from Biml
Table Metadata
<#=table.Name#>
<#=table.SchemaQualifiedName#>
SELECT *
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Custom T-SQL from Biml
Column Methods
GetColumnList
GetColumnAssignmentList
GetColumnComparisonList
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Column Methods
Return code fragments
Use as building blocks in custom T-SQL
Filter columns by using lambda expressions
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Column Methods
Return code fragments
Use as building blocks in custom T-SQL
Filter columns by using lambda expressions
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
"A lambda expression is an anonymous
function that you can use to create
delegates or expression tree types"
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
"A lambda expression is an anonymous
function that you can use to create
delegates or expression tree types"
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
column => column.Name == "ColumnID"
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
column => column.Name == "ColumnID"
The arrow is the lambda operator
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
column => column.Name == "ColumnID"
Input parameter is on the left side
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
column => column.Name == "ColumnID"
Expression is on the right side
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
column => column.Name == "ColumnID"
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnList
Get columns for SELECT
[Column1], [Column2], [Column3]
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnList
GetColumnList()
GetColumnList("src")
GetColumnList(c => c.Name == "ColumnID")
GetColumnList(c => c.DataType != DbType.Guid, "src")
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnAssignmentList
Get columns for UPDATE … SET
[l].[Column1] = [r].[Column1]
,[l].[Column2] = [r].[Column2]
,[l].[Column3] = [r].[Column3]
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnAssignmentList
GetColumnAssignmentList()
GetColumnAssignmentList("dst", "src")
GetColumnAssignmentList(c => !c.IsUsedInPrimaryKey)
GetColumnAssignmentList(c => !c.IsUsedInPrimaryKey, "dst", "src")
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnComparisonList
Get columns for JOIN … ON
[l].[Column1] = [r].[Column1]
AND [l].[Column2] = [r].[Column2]
AND [l].[Column3] = [r].[Column3]
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnComparisonList
GetColumnComparisonList()
GetColumnComparisonList("!=")
GetColumnComparisonList("dst", "src")
GetColumnComparisonList(c => c.IsUsedInPrimaryKey)
GetColumnComparisonList(c => c.IsUsedInPrimaryKey, "dst", "src")
GetColumnComparisonList(c => c.IsUsedInPrimaryKey, "!=", "dst", "src")
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
DEMO
T-SQL from Biml
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Test Data
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml + Random()
Create Random Test Data
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Random Class
Random()
Pseudo-random number generator
Random enough for practical purposes
Random(12345)
Always generates the same series of random numbers
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Random Methods
Next() Random number
Next(100) Random number smaller than specified value
Next(1000, 2000) Random number within specified range
NextDouble() Random decimal number between 0.0 and 1.0
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml Helper Method
<#+
Random r = new Random();
public string GetTestData(AstTableColumnBaseNode column) {
switch (column.DataType) {
case DbType.Int32 :
return r.Next(Int32.MaxValue).ToString();
case DbType.String :
return "'" + GetRandomString(column.Length) + "'"
default :
return "NULL";
}
}
#>
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml Helper Helper Method
<#+
private string GetRandomString(int length) {
var allowedChars = "ABC123";
var randomChars = new char[length];
for (int i = 0; i < length; i++) {
randomChars[i] = allowedChars[ r.Next(allowedChars.Length) ];
}
return new String(randomCharacters);
}
#>
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml + Random()
Very limited functionality
No dependencies
Works out-of-the box
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml + Bogus
Create Random and Specific Test Data
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Bogus Project
Simple and sane fake data generator for .NET
https://github.com/bchavez/Bogus
Advanced functionality for custom objects
…or simple functionality for Biml hacks :)
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Address
Commerce
Company
Database
Date
Finance
Internet
Lorem
Name
Person
Phone
System
Bogus API
Supports all data types plus built-in test data, including:
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
The Bogus Methods
<#@ assembly name="Bogus.dll" #>
<#@ import namespace="Bogus" #>
<# var f = new Faker(); #>
<# Person p = new Person(); #>
INSERT INTO dbo.Employee(FirstName, LastName, Birthday) VALUES
(
'<#=p.FirstName#>',
'<#=p.LastName#>',
'<#=p.DateOfBirth.ToString("yyyyMMdd")#>'
)
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml + Bogus()
Better functionality than Random()
Easier to understand, frequently updated
Requires additional files
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Installing Bogus
Install via Nuget:
Install-Package Bogus
Or download latest release .zip:
https://github.com/bchavez/Bogus/releases
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
DEMO
Test Data with
Random() and Bogus
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Alternative (Better) Solutions
Mockaroo (free, but only 1000 rows)
https://mockaroo.com/
Redgate SQL Data Generator (licensed)
https://www.red-gate.com/products/sql-
development/sql-data-generator/index
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Static Dimensions
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Static Dimensions (or any kind of static table)
SCD Type 0: Not changing
Lookup Tables
1. Create table definition
2. Add static source rows
3. Generate INSERT statements
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml Static Sources
Part of <Table> objects
Defines rows to be inserted when table is created
• Unknown dimension members
• Date dimension
• Code tables
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml Static Source
<Table Name="Table" SchemaName="Database.Schema">
<Columns>
<Column Name="Column1" DataType="Int32" />
<Column Name="Column2" DataType="String" Length="10" />
</Columns>
<Sources>
<StaticSource Name="TableRows">
<Rows>
<Row>
<ColumnValues>
<ColumnValue ColumnName="Column1" Value="-1" />
<ColumnValue ColumnName="Column2" Value="'Unknown'" />
</ColumnValues>
</Row>
</Rows>
</StaticSource>
</Sources>
</Table>
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml Static Source
<# if (table.Sources.Any()) { #>
<#=TableToPackageLowerer.GetStaticSourceInsertStatements(
table.SchemaQualifiedName,
table.HasIdentity,
(AstTableStaticSourceNode)table.Sources.FirstOrDefault()
)#>
<# } #>
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Drop and Create
Facts and Dimensions
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Could not drop object 'Schema.Table'
because it is referenced by a
FOREIGN KEY constraint.
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Drop and Create
Create Order: Dim1 Dim2 Fact1 Fact2
Fact2 Fact1 Dim2 Dim1Drop Order:
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Drop and Create
Create Order: Dim1 Dim2 Fact1 Fact2
Fact2 Fact1 Dim2 Dim1Drop Order:
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Topological Sort
"A topological sort of a directed graph is a
linear ordering of its vertices such that for
every directed edge uv from vertex u to vertex
v, u comes before v in the ordering"
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Topological Sort
"A topological sort of a directed graph is a
linear ordering of its vertices such that for
every directed edge uv from vertex u to vertex
v, u comes before v in the ordering"
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Topological Sort
"Sorts objects by dependencies.
For example dimensions before fact tables."
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Topological Sort
"Sorts objects by dependencies.
For example dimensions before fact tables."
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Topological Sort
IEnumerable<T> DependencyAnalysis.TopologicalSort<T> (
IEnumerable<T> source,
Expression<Func<T, T>> relationProperty
)
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Topological Sort
var dependentTables =
DependencyAnalysis.TopologicalSort<AstTableNode> (
RootNode.Tables,
t => t.Columns
.OfType<AstTableColumnTableReferenceNode>()
.Select(c => c.ForeignTable)
);
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
DEMO
Static Dimensions
and Topological Sort
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Comparing
Databases
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Once upon a time…
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Panic!
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
How to Compare
Tables and Columns
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
LINQ
Sort
OrderBy, ThenBy
Filter
Where, OfType
Group
GroupBy
Aggregate
Count, Sum
Check Collections
All, Any, Contains
Set Operations
Except, Intersect
Project Collections
Select, SelectMany
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
LINQ
Sort
OrderBy, ThenBy
Filter
Where, OfType
Group
GroupBy
Aggregate
Count, Sum
Check Collections
All, Any, Contains
Set Operations
Except, Intersect
Project Collections
Select, SelectMany
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
LINQ: Filter Collections
Where()
Returns the filtered collection with all elements that meet the criteria
table.Where(t => t.Schema.Name == "dim")
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
LINQ: Set Operations
Except()
Returns elements in first collection that are not in second collection
table1.Except(table2)
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
LINQ: Set Operations
Intersect()
Returns elements in first collection that are also in second collection
table1.Intersect(table2)
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
LINQ: Project Collections
Select()
Creates a new collection from specified attributes
table.Select(t => t.Name)
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
DEMO
Comparing
Databases
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Alternative (Better) Solutions
Redgate SQL Compare (licensed)
https://www.red-gate.com/products/sql-
development/sql-compare/
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
…the past 60 minutes…
T-SQL
Test Data Comparing
Dimensions
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Is this useful?
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
What other ideas
do you have?
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Have fun!
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
@cathrinew
cathrinew.net
hi@cathrinew.net
Biml resources and demo files:
cathrinew.net/biml

More Related Content

What's hot

Alternatives for Systems Integration in the NoSQL Era - NoSQL Roadshow 2013
Alternatives for Systems Integration in the NoSQL Era - NoSQL Roadshow 2013Alternatives for Systems Integration in the NoSQL Era - NoSQL Roadshow 2013
Alternatives for Systems Integration in the NoSQL Era - NoSQL Roadshow 2013
Kai Wähner
 
O365Engage17 - Smart Email Migration Knowing What’s Lurking in the ‘Dark Corn...
O365Engage17 - Smart Email Migration Knowing What’s Lurking in the ‘Dark Corn...O365Engage17 - Smart Email Migration Knowing What’s Lurking in the ‘Dark Corn...
O365Engage17 - Smart Email Migration Knowing What’s Lurking in the ‘Dark Corn...
NCCOMMS
 

What's hot (20)

Biml for Beginners: Script and Automate SSIS development (Malibu SQL Server U...
Biml for Beginners: Script and Automate SSIS development (Malibu SQL Server U...Biml for Beginners: Script and Automate SSIS development (Malibu SQL Server U...
Biml for Beginners: Script and Automate SSIS development (Malibu SQL Server U...
 
Building Dynamic Pipelines in Azure Data Factory (SQLSaturday Oslo)
Building Dynamic Pipelines in Azure Data Factory (SQLSaturday Oslo)Building Dynamic Pipelines in Azure Data Factory (SQLSaturday Oslo)
Building Dynamic Pipelines in Azure Data Factory (SQLSaturday Oslo)
 
Creating Visual Transformations in Azure Data Factory (dataMinds Connect)
Creating Visual Transformations in Azure Data Factory (dataMinds Connect)Creating Visual Transformations in Azure Data Factory (dataMinds Connect)
Creating Visual Transformations in Azure Data Factory (dataMinds Connect)
 
Understanding Azure Data Factory: The What, When, and Why (NIC 2020)
Understanding Azure Data Factory: The What, When, and Why (NIC 2020)Understanding Azure Data Factory: The What, When, and Why (NIC 2020)
Understanding Azure Data Factory: The What, When, and Why (NIC 2020)
 
Azure Data Factory for the SSIS Developer (SentryOne Webinar)
Azure Data Factory for the SSIS Developer (SentryOne Webinar)Azure Data Factory for the SSIS Developer (SentryOne Webinar)
Azure Data Factory for the SSIS Developer (SentryOne Webinar)
 
Building Dynamic Pipelines in Azure Data Factory (Data Saturday Holland)
Building Dynamic Pipelines in Azure Data Factory (Data Saturday Holland)Building Dynamic Pipelines in Azure Data Factory (Data Saturday Holland)
Building Dynamic Pipelines in Azure Data Factory (Data Saturday Holland)
 
Biml for Beginners: Script and Automate SSIS development (Capital Area SQL Se...
Biml for Beginners: Script and Automate SSIS development (Capital Area SQL Se...Biml for Beginners: Script and Automate SSIS development (Capital Area SQL Se...
Biml for Beginners: Script and Automate SSIS development (Capital Area SQL Se...
 
Biml for Beginners: Script and Automate SSIS development (SQLSaturday Chicago)
Biml for Beginners: Script and Automate SSIS development (SQLSaturday Chicago)Biml for Beginners: Script and Automate SSIS development (SQLSaturday Chicago)
Biml for Beginners: Script and Automate SSIS development (SQLSaturday Chicago)
 
Level Up Your Biml: Best Practices and Coding Techniques (SQLBits 2018)
Level Up Your Biml: Best Practices and Coding Techniques (SQLBits 2018)Level Up Your Biml: Best Practices and Coding Techniques (SQLBits 2018)
Level Up Your Biml: Best Practices and Coding Techniques (SQLBits 2018)
 
About Pragmatic Works
About Pragmatic WorksAbout Pragmatic Works
About Pragmatic Works
 
O365Engage17 - Using Exchange Online to Classify and Secure Mail
O365Engage17 - Using Exchange Online to Classify and Secure MailO365Engage17 - Using Exchange Online to Classify and Secure Mail
O365Engage17 - Using Exchange Online to Classify and Secure Mail
 
Tools and Tips For Data Warehouse Developers (SQLGLA)
Tools and Tips For Data Warehouse Developers (SQLGLA)Tools and Tips For Data Warehouse Developers (SQLGLA)
Tools and Tips For Data Warehouse Developers (SQLGLA)
 
O365Engage17 - Options for staying compliant in exchange online
O365Engage17 - Options for staying compliant in exchange onlineO365Engage17 - Options for staying compliant in exchange online
O365Engage17 - Options for staying compliant in exchange online
 
Reach New Heights with Amazon Redshift
Reach New Heights with Amazon RedshiftReach New Heights with Amazon Redshift
Reach New Heights with Amazon Redshift
 
Alternatives for Systems Integration in the NoSQL Era - NoSQL Roadshow 2013
Alternatives for Systems Integration in the NoSQL Era - NoSQL Roadshow 2013Alternatives for Systems Integration in the NoSQL Era - NoSQL Roadshow 2013
Alternatives for Systems Integration in the NoSQL Era - NoSQL Roadshow 2013
 
Pick a Winner: How to Choose a Data Warehouse
Pick a Winner: How to Choose a Data WarehousePick a Winner: How to Choose a Data Warehouse
Pick a Winner: How to Choose a Data Warehouse
 
O365Engage17 - Smart Email Migration Knowing What’s Lurking in the ‘Dark Corn...
O365Engage17 - Smart Email Migration Knowing What’s Lurking in the ‘Dark Corn...O365Engage17 - Smart Email Migration Knowing What’s Lurking in the ‘Dark Corn...
O365Engage17 - Smart Email Migration Knowing What’s Lurking in the ‘Dark Corn...
 
O365Engage17 - Protecting O365 Data in a Modern World
O365Engage17 - Protecting O365 Data in a Modern WorldO365Engage17 - Protecting O365 Data in a Modern World
O365Engage17 - Protecting O365 Data in a Modern World
 
How to Choose a Data Warehouse
How to Choose a Data WarehouseHow to Choose a Data Warehouse
How to Choose a Data Warehouse
 
Hassle-Free Data Lake Governance: Automating Your Analytics with a Semantic L...
Hassle-Free Data Lake Governance: Automating Your Analytics with a Semantic L...Hassle-Free Data Lake Governance: Automating Your Analytics with a Semantic L...
Hassle-Free Data Lake Governance: Automating Your Analytics with a Semantic L...
 

Similar to Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018)

Similar to Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) (20)

Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Denver)
Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Denver)Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Denver)
Level Up Your Biml: Best Practices and Coding Techniques (SQLSaturday Denver)
 
Pipelines and Packages: Introduction to Azure Data Factory (24HOP)
Pipelines and Packages: Introduction to Azure Data Factory (24HOP)Pipelines and Packages: Introduction to Azure Data Factory (24HOP)
Pipelines and Packages: Introduction to Azure Data Factory (24HOP)
 
Deep Dive into Concepts and Tools for Analyzing Streaming Data on AWS
Deep Dive into Concepts and Tools for Analyzing Streaming Data on AWS Deep Dive into Concepts and Tools for Analyzing Streaming Data on AWS
Deep Dive into Concepts and Tools for Analyzing Streaming Data on AWS
 
OOP Best Practices in JavaScript
OOP Best Practices in JavaScriptOOP Best Practices in JavaScript
OOP Best Practices in JavaScript
 
AppSync in real world - pitfalls, unexpected benefits & lessons learnt
AppSync in real world - pitfalls, unexpected benefits & lessons learntAppSync in real world - pitfalls, unexpected benefits & lessons learnt
AppSync in real world - pitfalls, unexpected benefits & lessons learnt
 
Serverless:It All Started in Vegas (DVC306) - AWS re:Invent 2018
Serverless:It All Started in Vegas (DVC306) - AWS re:Invent 2018Serverless:It All Started in Vegas (DVC306) - AWS re:Invent 2018
Serverless:It All Started in Vegas (DVC306) - AWS re:Invent 2018
 
Creating a Machine Learning Factory
Creating a Machine Learning FactoryCreating a Machine Learning Factory
Creating a Machine Learning Factory
 
Analyzing Streams
Analyzing StreamsAnalyzing Streams
Analyzing Streams
 
AWS IoT for Frictionless Consumer Experiences in Retail (RET201) - AWS re:Inv...
AWS IoT for Frictionless Consumer Experiences in Retail (RET201) - AWS re:Inv...AWS IoT for Frictionless Consumer Experiences in Retail (RET201) - AWS re:Inv...
AWS IoT for Frictionless Consumer Experiences in Retail (RET201) - AWS re:Inv...
 
ServerlessConf 2018 Keynote - Debunking Serverless Myths
ServerlessConf 2018 Keynote - Debunking Serverless MythsServerlessConf 2018 Keynote - Debunking Serverless Myths
ServerlessConf 2018 Keynote - Debunking Serverless Myths
 
Architecting for Real-Time Insights with Amazon Kinesis (ANT310) - AWS re:Inv...
Architecting for Real-Time Insights with Amazon Kinesis (ANT310) - AWS re:Inv...Architecting for Real-Time Insights with Amazon Kinesis (ANT310) - AWS re:Inv...
Architecting for Real-Time Insights with Amazon Kinesis (ANT310) - AWS re:Inv...
 
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...
 
The Theory and Math Behind Data Privacy and Security Assurance (SEC301) - AWS...
The Theory and Math Behind Data Privacy and Security Assurance (SEC301) - AWS...The Theory and Math Behind Data Privacy and Security Assurance (SEC301) - AWS...
The Theory and Math Behind Data Privacy and Security Assurance (SEC301) - AWS...
 
IBM Insight 2015 - 1824 - Using Bluemix and dashDB for Twitter Analysis
IBM Insight 2015 - 1824 - Using Bluemix and dashDB for Twitter AnalysisIBM Insight 2015 - 1824 - Using Bluemix and dashDB for Twitter Analysis
IBM Insight 2015 - 1824 - Using Bluemix and dashDB for Twitter Analysis
 
Understanding Graph Databases: AWS Developer Workshop at Web Summit
Understanding Graph Databases: AWS Developer Workshop at Web SummitUnderstanding Graph Databases: AWS Developer Workshop at Web Summit
Understanding Graph Databases: AWS Developer Workshop at Web Summit
 
Building Serverless ETL Pipelines
Building Serverless ETL PipelinesBuilding Serverless ETL Pipelines
Building Serverless ETL Pipelines
 
Ten Tips And Tricks for Improving Your GraphQL API with AWS AppSync (MOB401) ...
Ten Tips And Tricks for Improving Your GraphQL API with AWS AppSync (MOB401) ...Ten Tips And Tricks for Improving Your GraphQL API with AWS AppSync (MOB401) ...
Ten Tips And Tricks for Improving Your GraphQL API with AWS AppSync (MOB401) ...
 
Tsar tech talk
Tsar tech talkTsar tech talk
Tsar tech talk
 
TSAR (TimeSeries AggregatoR) Tech Talk
TSAR (TimeSeries AggregatoR) Tech TalkTSAR (TimeSeries AggregatoR) Tech Talk
TSAR (TimeSeries AggregatoR) Tech Talk
 
Supercell – Scaling Mobile Games (GAM301) - AWS re:Invent 2018
Supercell – Scaling Mobile Games (GAM301) - AWS re:Invent 2018Supercell – Scaling Mobile Games (GAM301) - AWS re:Invent 2018
Supercell – Scaling Mobile Games (GAM301) - AWS re:Invent 2018
 

More from Cathrine Wilhelmsen

Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...
Cathrine Wilhelmsen
 

More from Cathrine Wilhelmsen (20)

Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)
 
Getting Started: Data Factory in Microsoft Fabric (Microsoft Fabric Community...
Getting Started: Data Factory in Microsoft Fabric (Microsoft Fabric Community...Getting Started: Data Factory in Microsoft Fabric (Microsoft Fabric Community...
Getting Started: Data Factory in Microsoft Fabric (Microsoft Fabric Community...
 
Website Analytics in My Pocket using Microsoft Fabric (SQLBits 2024)
Website Analytics in My Pocket using Microsoft Fabric (SQLBits 2024)Website Analytics in My Pocket using Microsoft Fabric (SQLBits 2024)
Website Analytics in My Pocket using Microsoft Fabric (SQLBits 2024)
 
Data Integration using Data Factory in Microsoft Fabric (ESPC Microsoft Fabri...
Data Integration using Data Factory in Microsoft Fabric (ESPC Microsoft Fabri...Data Integration using Data Factory in Microsoft Fabric (ESPC Microsoft Fabri...
Data Integration using Data Factory in Microsoft Fabric (ESPC Microsoft Fabri...
 
Choosing between Fabric, Synapse and Databricks (Data Left Unattended 2023)
Choosing between Fabric, Synapse and Databricks (Data Left Unattended 2023)Choosing between Fabric, Synapse and Databricks (Data Left Unattended 2023)
Choosing between Fabric, Synapse and Databricks (Data Left Unattended 2023)
 
Data Integration with Data Factory (Microsoft Fabric Day Oslo 2023)
Data Integration with Data Factory (Microsoft Fabric Day Oslo 2023)Data Integration with Data Factory (Microsoft Fabric Day Oslo 2023)
Data Integration with Data Factory (Microsoft Fabric Day Oslo 2023)
 
The Battle of the Data Transformation Tools (PASS Data Community Summit 2023)
The Battle of the Data Transformation Tools (PASS Data Community Summit 2023)The Battle of the Data Transformation Tools (PASS Data Community Summit 2023)
The Battle of the Data Transformation Tools (PASS Data Community Summit 2023)
 
Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PAS...
Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PAS...Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PAS...
Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PAS...
 
Building an End-to-End Solution in Microsoft Fabric: From Dataverse to Power ...
Building an End-to-End Solution in Microsoft Fabric: From Dataverse to Power ...Building an End-to-End Solution in Microsoft Fabric: From Dataverse to Power ...
Building an End-to-End Solution in Microsoft Fabric: From Dataverse to Power ...
 
Website Analytics in my Pocket using Microsoft Fabric (AdaCon 2023)
Website Analytics in my Pocket using Microsoft Fabric (AdaCon 2023)Website Analytics in my Pocket using Microsoft Fabric (AdaCon 2023)
Website Analytics in my Pocket using Microsoft Fabric (AdaCon 2023)
 
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...
 
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D...
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D...Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D...
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D...
 
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...
 
"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ...
"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ..."I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ...
"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ...
 
Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...
Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...
Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...
 
6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)
6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)
6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)
 
Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...
Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...
Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...
 
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
 
Azure Synapse Analytics Teaser (Microsoft TechX Oslo 2019)
Azure Synapse Analytics Teaser (Microsoft TechX Oslo 2019)Azure Synapse Analytics Teaser (Microsoft TechX Oslo 2019)
Azure Synapse Analytics Teaser (Microsoft TechX Oslo 2019)
 
Lessons Learned: Understanding Azure Data Factory Pricing (Microsoft Ignite 2...
Lessons Learned: Understanding Azure Data Factory Pricing (Microsoft Ignite 2...Lessons Learned: Understanding Azure Data Factory Pricing (Microsoft Ignite 2...
Lessons Learned: Understanding Azure Data Factory Pricing (Microsoft Ignite 2...
 

Recently uploaded

一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理
pyhepag
 
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotecAbortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
Abortion pills in Riyadh +966572737505 get cytotec
 
一比一原版麦考瑞大学毕业证成绩单如何办理
一比一原版麦考瑞大学毕业证成绩单如何办理一比一原版麦考瑞大学毕业证成绩单如何办理
一比一原版麦考瑞大学毕业证成绩单如何办理
cyebo
 
Fuzzy Sets decision making under information of uncertainty
Fuzzy Sets decision making under information of uncertaintyFuzzy Sets decision making under information of uncertainty
Fuzzy Sets decision making under information of uncertainty
RafigAliyev2
 
一比一原版西悉尼大学毕业证成绩单如何办理
一比一原版西悉尼大学毕业证成绩单如何办理一比一原版西悉尼大学毕业证成绩单如何办理
一比一原版西悉尼大学毕业证成绩单如何办理
pyhepag
 
Exploratory Data Analysis - Dilip S.pptx
Exploratory Data Analysis - Dilip S.pptxExploratory Data Analysis - Dilip S.pptx
Exploratory Data Analysis - Dilip S.pptx
DilipVasan
 
一比一原版纽卡斯尔大学毕业证成绩单如何办理
一比一原版纽卡斯尔大学毕业证成绩单如何办理一比一原版纽卡斯尔大学毕业证成绩单如何办理
一比一原版纽卡斯尔大学毕业证成绩单如何办理
cyebo
 

Recently uploaded (20)

2024 Q2 Orange County (CA) Tableau User Group Meeting
2024 Q2 Orange County (CA) Tableau User Group Meeting2024 Q2 Orange County (CA) Tableau User Group Meeting
2024 Q2 Orange County (CA) Tableau User Group Meeting
 
Machine Learning for Accident Severity Prediction
Machine Learning for Accident Severity PredictionMachine Learning for Accident Severity Prediction
Machine Learning for Accident Severity Prediction
 
Webinar One View, Multiple Systems No-Code Integration of Salesforce and ERPs
Webinar One View, Multiple Systems No-Code Integration of Salesforce and ERPsWebinar One View, Multiple Systems No-Code Integration of Salesforce and ERPs
Webinar One View, Multiple Systems No-Code Integration of Salesforce and ERPs
 
一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理
 
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotecAbortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
 
Artificial_General_Intelligence__storm_gen_article.pdf
Artificial_General_Intelligence__storm_gen_article.pdfArtificial_General_Intelligence__storm_gen_article.pdf
Artificial_General_Intelligence__storm_gen_article.pdf
 
Pre-ProductionImproveddsfjgndflghtgg.pptx
Pre-ProductionImproveddsfjgndflghtgg.pptxPre-ProductionImproveddsfjgndflghtgg.pptx
Pre-ProductionImproveddsfjgndflghtgg.pptx
 
2024 Q1 Tableau User Group Leader Quarterly Call
2024 Q1 Tableau User Group Leader Quarterly Call2024 Q1 Tableau User Group Leader Quarterly Call
2024 Q1 Tableau User Group Leader Quarterly Call
 
一比一原版麦考瑞大学毕业证成绩单如何办理
一比一原版麦考瑞大学毕业证成绩单如何办理一比一原版麦考瑞大学毕业证成绩单如何办理
一比一原版麦考瑞大学毕业证成绩单如何办理
 
basics of data science with application areas.pdf
basics of data science with application areas.pdfbasics of data science with application areas.pdf
basics of data science with application areas.pdf
 
Data analytics courses in Nepal Presentation
Data analytics courses in Nepal PresentationData analytics courses in Nepal Presentation
Data analytics courses in Nepal Presentation
 
Fuzzy Sets decision making under information of uncertainty
Fuzzy Sets decision making under information of uncertaintyFuzzy Sets decision making under information of uncertainty
Fuzzy Sets decision making under information of uncertainty
 
Easy and simple project file on mp online
Easy and simple project file on mp onlineEasy and simple project file on mp online
Easy and simple project file on mp online
 
Atlantic Grupa Case Study (Mintec Data AI)
Atlantic Grupa Case Study (Mintec Data AI)Atlantic Grupa Case Study (Mintec Data AI)
Atlantic Grupa Case Study (Mintec Data AI)
 
一比一原版西悉尼大学毕业证成绩单如何办理
一比一原版西悉尼大学毕业证成绩单如何办理一比一原版西悉尼大学毕业证成绩单如何办理
一比一原版西悉尼大学毕业证成绩单如何办理
 
How I opened a fake bank account and didn't go to prison
How I opened a fake bank account and didn't go to prisonHow I opened a fake bank account and didn't go to prison
How I opened a fake bank account and didn't go to prison
 
Exploratory Data Analysis - Dilip S.pptx
Exploratory Data Analysis - Dilip S.pptxExploratory Data Analysis - Dilip S.pptx
Exploratory Data Analysis - Dilip S.pptx
 
一比一原版纽卡斯尔大学毕业证成绩单如何办理
一比一原版纽卡斯尔大学毕业证成绩单如何办理一比一原版纽卡斯尔大学毕业证成绩单如何办理
一比一原版纽卡斯尔大学毕业证成绩单如何办理
 
Slip-and-fall Injuries: Top Workers' Comp Claims
Slip-and-fall Injuries: Top Workers' Comp ClaimsSlip-and-fall Injuries: Top Workers' Comp Claims
Slip-and-fall Injuries: Top Workers' Comp Claims
 
Generative AI for Trailblazers_ Unlock the Future of AI.pdf
Generative AI for Trailblazers_ Unlock the Future of AI.pdfGenerative AI for Trailblazers_ Unlock the Future of AI.pdf
Generative AI for Trailblazers_ Unlock the Future of AI.pdf
 

Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018)

  • 1. Biml Tips and Tricks Not just for SSIS Packages! Cathrine Wilhelmsen SQLGrillen · June 22nd 2018
  • 2. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
  • 3. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Session Description "Wait, what? Biml is not just for generating SSIS packages?" Absolutely not! Come and see how you can use Biml (Business Intelligence Markup Language) to save time and speed up other Data Warehouse development tasks. You can generate complex T-SQL statements with Biml instead of using dynamic SQL, create test data and populate static dimensions, and even compare tables and views across multiple servers and databases. Don't Repeat Yourself, start automating those boring, manual tasks today!
  • 4. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Cathrine Wilhelmsen @cathrinew cathrinew.net Data Warehouse & Business Intelligence Consultant
  • 5. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) …the next 60 minutes… T-SQL Test Data Comparing Dimensions
  • 6. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
  • 7. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Wait… Can't I use <something else>?
  • 8. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Of course! But Biml works well with source metadata
  • 9. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) And… Biml is fun!
  • 10. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) But… Biml? How?
  • 11. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Traditional Biml
  • 12. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Crazy Fun Biml
  • 13. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What do you need?
  • 14. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Preview Pane The power is in the…
  • 15. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Traditional BimlScript
  • 16. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Traditional BimlScript <# foreach (var table in RootNode.Tables) { #> <Package Name="Load_<#=table.Name#>" /> <# } #> <Package Name="Load_Customer" /> <Package Name="Load_Product" /> <Package Name="Load_Sales" />
  • 17. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlScript to Biml Preview Pane
  • 18. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlScript to... ??? Preview Pane
  • 19. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Crazy Fun BimlScript <# foreach (var table in RootNode.Tables) { #> Yay, a package! <#=table.Name#> :) <# } #> Yay, a package! Load_Customer :) Yay, a package! Load_Product :) Yay, a package! Load_Sales :)
  • 20. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
  • 21. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) T-SQL from Biml
  • 22. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Built-in T-SQL from Biml
  • 23. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Built-in T-SQL from Biml Functional T-SQL GetSelectSql GetDropAndCreateDdl GetTableSql SSMS Snippets GetInsertSql GetUpdateSql GetDeleteSql
  • 24. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Functional T-SQL: GetSelectSql SELECT [Column1] ,[Column2] FROM [Schema].[Table]
  • 25. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Functional T-SQL: GetDropAndCreateDdl IF EXISTS (...) DROP TABLE [Schema].[Table] GO CREATE TABLE [Schema].[Table] ( [Column1] int IDENTITY(1,1) NOT NULL ,[Column2] nvarchar(50) NOT NULL -- Constraints )
  • 26. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Functional T-SQL: GetTableSql IF EXISTS (...) DROP TABLE [Schema].[Table] GO CREATE TABLE [Schema].[Table] ( [Column1] int IDENTITY(1,1) NOT NULL ,[Column2] nvarchar(50) NOT NULL -- Constraints )
  • 27. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Functional T-SQL Use directly in SSIS Copy and run in SSMS
  • 28. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) SSMS Snippets: GetDeleteSql DELETE FROM [Schema].[Table] WHERE <Search Conditions,,>
  • 29. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) SSMS Snippets: GetInsertSql INSERT INTO [Schema].[Table] ( [Column1] ,[Column1] ) VALUES ( <Column1,int,>, <Column2,nvarchar(50),> )
  • 30. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) SSMS Snippets: GetUpdateSql UPDATE [Schema].[Table] SET [Column1] = <Column1, int,>, [Column2] = <Column2, nvarchar(50),> WHERE <Search Conditions,,>
  • 31. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Copy and run in SSMS Replace Template Parameters with actual values CTRL SHIFT M SSMS Snippets
  • 32. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Custom T-SQL from Biml
  • 33. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Custom T-SQL from Biml Table Metadata <#=table.Name#> <#=table.SchemaQualifiedName#> SELECT *
  • 34. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Custom T-SQL from Biml Column Methods GetColumnList GetColumnAssignmentList GetColumnComparisonList
  • 35. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Column Methods Return code fragments Use as building blocks in custom T-SQL Filter columns by using lambda expressions
  • 36. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Column Methods Return code fragments Use as building blocks in custom T-SQL Filter columns by using lambda expressions
  • 37. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions "A lambda expression is an anonymous function that you can use to create delegates or expression tree types"
  • 38. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions "A lambda expression is an anonymous function that you can use to create delegates or expression tree types"
  • 39. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions column => column.Name == "ColumnID"
  • 40. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions column => column.Name == "ColumnID" The arrow is the lambda operator
  • 41. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions column => column.Name == "ColumnID" Input parameter is on the left side
  • 42. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions column => column.Name == "ColumnID" Expression is on the right side
  • 43. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions column => column.Name == "ColumnID"
  • 44. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnList Get columns for SELECT [Column1], [Column2], [Column3]
  • 45. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnList GetColumnList() GetColumnList("src") GetColumnList(c => c.Name == "ColumnID") GetColumnList(c => c.DataType != DbType.Guid, "src")
  • 46. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnAssignmentList Get columns for UPDATE … SET [l].[Column1] = [r].[Column1] ,[l].[Column2] = [r].[Column2] ,[l].[Column3] = [r].[Column3]
  • 47. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnAssignmentList GetColumnAssignmentList() GetColumnAssignmentList("dst", "src") GetColumnAssignmentList(c => !c.IsUsedInPrimaryKey) GetColumnAssignmentList(c => !c.IsUsedInPrimaryKey, "dst", "src")
  • 48. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnComparisonList Get columns for JOIN … ON [l].[Column1] = [r].[Column1] AND [l].[Column2] = [r].[Column2] AND [l].[Column3] = [r].[Column3]
  • 49. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnComparisonList GetColumnComparisonList() GetColumnComparisonList("!=") GetColumnComparisonList("dst", "src") GetColumnComparisonList(c => c.IsUsedInPrimaryKey) GetColumnComparisonList(c => c.IsUsedInPrimaryKey, "dst", "src") GetColumnComparisonList(c => c.IsUsedInPrimaryKey, "!=", "dst", "src")
  • 50. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO T-SQL from Biml
  • 51. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Test Data
  • 52. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml + Random() Create Random Test Data
  • 53. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Random Class Random() Pseudo-random number generator Random enough for practical purposes Random(12345) Always generates the same series of random numbers
  • 54. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Random Methods Next() Random number Next(100) Random number smaller than specified value Next(1000, 2000) Random number within specified range NextDouble() Random decimal number between 0.0 and 1.0
  • 55. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Helper Method <#+ Random r = new Random(); public string GetTestData(AstTableColumnBaseNode column) { switch (column.DataType) { case DbType.Int32 : return r.Next(Int32.MaxValue).ToString(); case DbType.String : return "'" + GetRandomString(column.Length) + "'" default : return "NULL"; } } #>
  • 56. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Helper Helper Method <#+ private string GetRandomString(int length) { var allowedChars = "ABC123"; var randomChars = new char[length]; for (int i = 0; i < length; i++) { randomChars[i] = allowedChars[ r.Next(allowedChars.Length) ]; } return new String(randomCharacters); } #>
  • 57. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml + Random() Very limited functionality No dependencies Works out-of-the box
  • 58. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml + Bogus Create Random and Specific Test Data
  • 59. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Bogus Project Simple and sane fake data generator for .NET https://github.com/bchavez/Bogus Advanced functionality for custom objects …or simple functionality for Biml hacks :)
  • 60. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Address Commerce Company Database Date Finance Internet Lorem Name Person Phone System Bogus API Supports all data types plus built-in test data, including:
  • 61. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) The Bogus Methods <#@ assembly name="Bogus.dll" #> <#@ import namespace="Bogus" #> <# var f = new Faker(); #> <# Person p = new Person(); #> INSERT INTO dbo.Employee(FirstName, LastName, Birthday) VALUES ( '<#=p.FirstName#>', '<#=p.LastName#>', '<#=p.DateOfBirth.ToString("yyyyMMdd")#>' )
  • 62. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml + Bogus() Better functionality than Random() Easier to understand, frequently updated Requires additional files
  • 63. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Installing Bogus Install via Nuget: Install-Package Bogus Or download latest release .zip: https://github.com/bchavez/Bogus/releases
  • 64. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO Test Data with Random() and Bogus
  • 65. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Alternative (Better) Solutions Mockaroo (free, but only 1000 rows) https://mockaroo.com/ Redgate SQL Data Generator (licensed) https://www.red-gate.com/products/sql- development/sql-data-generator/index
  • 66. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Static Dimensions
  • 67. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Static Dimensions (or any kind of static table) SCD Type 0: Not changing Lookup Tables 1. Create table definition 2. Add static source rows 3. Generate INSERT statements
  • 68. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Static Sources Part of <Table> objects Defines rows to be inserted when table is created • Unknown dimension members • Date dimension • Code tables
  • 69. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Static Source <Table Name="Table" SchemaName="Database.Schema"> <Columns> <Column Name="Column1" DataType="Int32" /> <Column Name="Column2" DataType="String" Length="10" /> </Columns> <Sources> <StaticSource Name="TableRows"> <Rows> <Row> <ColumnValues> <ColumnValue ColumnName="Column1" Value="-1" /> <ColumnValue ColumnName="Column2" Value="'Unknown'" /> </ColumnValues> </Row> </Rows> </StaticSource> </Sources> </Table>
  • 70. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Static Source <# if (table.Sources.Any()) { #> <#=TableToPackageLowerer.GetStaticSourceInsertStatements( table.SchemaQualifiedName, table.HasIdentity, (AstTableStaticSourceNode)table.Sources.FirstOrDefault() )#> <# } #>
  • 71. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Drop and Create Facts and Dimensions
  • 72. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Could not drop object 'Schema.Table' because it is referenced by a FOREIGN KEY constraint.
  • 73. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Drop and Create Create Order: Dim1 Dim2 Fact1 Fact2 Fact2 Fact1 Dim2 Dim1Drop Order:
  • 74. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Drop and Create Create Order: Dim1 Dim2 Fact1 Fact2 Fact2 Fact1 Dim2 Dim1Drop Order:
  • 75. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Topological Sort "A topological sort of a directed graph is a linear ordering of its vertices such that for every directed edge uv from vertex u to vertex v, u comes before v in the ordering"
  • 76. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Topological Sort "A topological sort of a directed graph is a linear ordering of its vertices such that for every directed edge uv from vertex u to vertex v, u comes before v in the ordering"
  • 77. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Topological Sort "Sorts objects by dependencies. For example dimensions before fact tables."
  • 78. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Topological Sort "Sorts objects by dependencies. For example dimensions before fact tables."
  • 79. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Topological Sort IEnumerable<T> DependencyAnalysis.TopologicalSort<T> ( IEnumerable<T> source, Expression<Func<T, T>> relationProperty )
  • 80. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Topological Sort var dependentTables = DependencyAnalysis.TopologicalSort<AstTableNode> ( RootNode.Tables, t => t.Columns .OfType<AstTableColumnTableReferenceNode>() .Select(c => c.ForeignTable) );
  • 81. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO Static Dimensions and Topological Sort
  • 82. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Comparing Databases
  • 83. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Once upon a time…
  • 84. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Panic!
  • 85. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) How to Compare Tables and Columns
  • 86. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) LINQ Sort OrderBy, ThenBy Filter Where, OfType Group GroupBy Aggregate Count, Sum Check Collections All, Any, Contains Set Operations Except, Intersect Project Collections Select, SelectMany
  • 87. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) LINQ Sort OrderBy, ThenBy Filter Where, OfType Group GroupBy Aggregate Count, Sum Check Collections All, Any, Contains Set Operations Except, Intersect Project Collections Select, SelectMany
  • 88. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) LINQ: Filter Collections Where() Returns the filtered collection with all elements that meet the criteria table.Where(t => t.Schema.Name == "dim")
  • 89. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) LINQ: Set Operations Except() Returns elements in first collection that are not in second collection table1.Except(table2)
  • 90. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) LINQ: Set Operations Intersect() Returns elements in first collection that are also in second collection table1.Intersect(table2)
  • 91. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) LINQ: Project Collections Select() Creates a new collection from specified attributes table.Select(t => t.Name)
  • 92. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO Comparing Databases
  • 93. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Alternative (Better) Solutions Redgate SQL Compare (licensed) https://www.red-gate.com/products/sql- development/sql-compare/
  • 94. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) …the past 60 minutes… T-SQL Test Data Comparing Dimensions
  • 95. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Is this useful?
  • 96. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What other ideas do you have?
  • 97. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Have fun!
  • 98. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) @cathrinew cathrinew.net hi@cathrinew.net Biml resources and demo files: cathrinew.net/biml