More Related Content
Similar to Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) (20)
More from Cathrine Wilhelmsen (20)
Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018)
- 1. Biml Tips and Tricks
Not just for SSIS Packages!
Cathrine Wilhelmsen
SQLGrillen · June 22nd 2018
- 3. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Session Description
"Wait, what? Biml is not just for generating SSIS packages?"
Absolutely not! Come and see how you can use Biml (Business Intelligence
Markup Language) to save time and speed up other Data Warehouse
development tasks. You can generate complex T-SQL statements with Biml
instead of using dynamic SQL, create test data and populate static dimensions,
and even compare tables and views across multiple servers and databases.
Don't Repeat Yourself, start automating those boring, manual tasks today!
- 4. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Cathrine Wilhelmsen
@cathrinew cathrinew.net
Data Warehouse & Business Intelligence Consultant
- 5. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
…the next 60 minutes…
T-SQL
Test Data Comparing
Dimensions
- 7. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Wait…
Can't I use
<something else>?
- 8. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Of course!
But Biml works well
with source metadata
- 9. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
And…
Biml is fun!
- 10. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
But… Biml? How?
- 11. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Traditional Biml
- 12. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Crazy Fun Biml
- 13. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
What do you need?
- 14. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Preview Pane
The power is in the…
- 15. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Traditional BimlScript
- 16. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Traditional BimlScript
<# foreach (var table in RootNode.Tables) { #>
<Package Name="Load_<#=table.Name#>" />
<# } #>
<Package Name="Load_Customer" />
<Package Name="Load_Product" />
<Package Name="Load_Sales" />
- 17. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
BimlScript to Biml
Preview Pane
- 18. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
BimlScript to... ???
Preview Pane
- 19. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Crazy Fun BimlScript
<# foreach (var table in RootNode.Tables) { #>
Yay, a package! <#=table.Name#> :)
<# } #>
Yay, a package! Load_Customer :)
Yay, a package! Load_Product :)
Yay, a package! Load_Sales :)
- 21. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
T-SQL from Biml
- 22. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Built-in
T-SQL from Biml
- 23. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Built-in T-SQL from Biml
Functional T-SQL
GetSelectSql
GetDropAndCreateDdl
GetTableSql
SSMS Snippets
GetInsertSql
GetUpdateSql
GetDeleteSql
- 24. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Functional T-SQL: GetSelectSql
SELECT
[Column1]
,[Column2]
FROM [Schema].[Table]
- 25. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Functional T-SQL: GetDropAndCreateDdl
IF EXISTS (...)
DROP TABLE [Schema].[Table]
GO
CREATE TABLE [Schema].[Table] (
[Column1] int IDENTITY(1,1) NOT NULL
,[Column2] nvarchar(50) NOT NULL
-- Constraints
)
- 26. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Functional T-SQL: GetTableSql
IF EXISTS (...)
DROP TABLE [Schema].[Table]
GO
CREATE TABLE [Schema].[Table] (
[Column1] int IDENTITY(1,1) NOT NULL
,[Column2] nvarchar(50) NOT NULL
-- Constraints
)
- 27. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Functional T-SQL
Use directly in SSIS
Copy and run in SSMS
- 28. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
SSMS Snippets: GetDeleteSql
DELETE FROM [Schema].[Table]
WHERE <Search Conditions,,>
- 29. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
SSMS Snippets: GetInsertSql
INSERT INTO [Schema].[Table] (
[Column1]
,[Column1]
) VALUES (
<Column1,int,>,
<Column2,nvarchar(50),>
)
- 30. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
SSMS Snippets: GetUpdateSql
UPDATE [Schema].[Table] SET
[Column1] = <Column1, int,>,
[Column2] = <Column2, nvarchar(50),>
WHERE <Search Conditions,,>
- 31. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Copy and run in SSMS
Replace Template Parameters with actual values
CTRL SHIFT M
SSMS Snippets
- 32. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Custom
T-SQL from Biml
- 33. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Custom T-SQL from Biml
Table Metadata
<#=table.Name#>
<#=table.SchemaQualifiedName#>
SELECT *
- 34. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Custom T-SQL from Biml
Column Methods
GetColumnList
GetColumnAssignmentList
GetColumnComparisonList
- 35. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Column Methods
Return code fragments
Use as building blocks in custom T-SQL
Filter columns by using lambda expressions
- 36. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Column Methods
Return code fragments
Use as building blocks in custom T-SQL
Filter columns by using lambda expressions
- 37. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
"A lambda expression is an anonymous
function that you can use to create
delegates or expression tree types"
- 38. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
"A lambda expression is an anonymous
function that you can use to create
delegates or expression tree types"
- 39. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
column => column.Name == "ColumnID"
- 40. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
column => column.Name == "ColumnID"
The arrow is the lambda operator
- 41. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
column => column.Name == "ColumnID"
Input parameter is on the left side
- 42. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
column => column.Name == "ColumnID"
Expression is on the right side
- 43. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
column => column.Name == "ColumnID"
- 44. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnList
Get columns for SELECT
[Column1], [Column2], [Column3]
- 45. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnList
GetColumnList()
GetColumnList("src")
GetColumnList(c => c.Name == "ColumnID")
GetColumnList(c => c.DataType != DbType.Guid, "src")
- 46. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnAssignmentList
Get columns for UPDATE … SET
[l].[Column1] = [r].[Column1]
,[l].[Column2] = [r].[Column2]
,[l].[Column3] = [r].[Column3]
- 47. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnAssignmentList
GetColumnAssignmentList()
GetColumnAssignmentList("dst", "src")
GetColumnAssignmentList(c => !c.IsUsedInPrimaryKey)
GetColumnAssignmentList(c => !c.IsUsedInPrimaryKey, "dst", "src")
- 48. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnComparisonList
Get columns for JOIN … ON
[l].[Column1] = [r].[Column1]
AND [l].[Column2] = [r].[Column2]
AND [l].[Column3] = [r].[Column3]
- 49. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnComparisonList
GetColumnComparisonList()
GetColumnComparisonList("!=")
GetColumnComparisonList("dst", "src")
GetColumnComparisonList(c => c.IsUsedInPrimaryKey)
GetColumnComparisonList(c => c.IsUsedInPrimaryKey, "dst", "src")
GetColumnComparisonList(c => c.IsUsedInPrimaryKey, "!=", "dst", "src")
- 50. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
DEMO
T-SQL from Biml
- 52. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml + Random()
Create Random Test Data
- 53. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Random Class
Random()
Pseudo-random number generator
Random enough for practical purposes
Random(12345)
Always generates the same series of random numbers
- 54. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Random Methods
Next() Random number
Next(100) Random number smaller than specified value
Next(1000, 2000) Random number within specified range
NextDouble() Random decimal number between 0.0 and 1.0
- 55. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml Helper Method
<#+
Random r = new Random();
public string GetTestData(AstTableColumnBaseNode column) {
switch (column.DataType) {
case DbType.Int32 :
return r.Next(Int32.MaxValue).ToString();
case DbType.String :
return "'" + GetRandomString(column.Length) + "'"
default :
return "NULL";
}
}
#>
- 56. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml Helper Helper Method
<#+
private string GetRandomString(int length) {
var allowedChars = "ABC123";
var randomChars = new char[length];
for (int i = 0; i < length; i++) {
randomChars[i] = allowedChars[ r.Next(allowedChars.Length) ];
}
return new String(randomCharacters);
}
#>
- 57. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml + Random()
Very limited functionality
No dependencies
Works out-of-the box
- 58. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml + Bogus
Create Random and Specific Test Data
- 59. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Bogus Project
Simple and sane fake data generator for .NET
https://github.com/bchavez/Bogus
Advanced functionality for custom objects
…or simple functionality for Biml hacks :)
- 60. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Address
Commerce
Company
Database
Date
Finance
Internet
Lorem
Name
Person
Phone
System
Bogus API
Supports all data types plus built-in test data, including:
- 61. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
The Bogus Methods
<#@ assembly name="Bogus.dll" #>
<#@ import namespace="Bogus" #>
<# var f = new Faker(); #>
<# Person p = new Person(); #>
INSERT INTO dbo.Employee(FirstName, LastName, Birthday) VALUES
(
'<#=p.FirstName#>',
'<#=p.LastName#>',
'<#=p.DateOfBirth.ToString("yyyyMMdd")#>'
)
- 62. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml + Bogus()
Better functionality than Random()
Easier to understand, frequently updated
Requires additional files
- 63. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Installing Bogus
Install via Nuget:
Install-Package Bogus
Or download latest release .zip:
https://github.com/bchavez/Bogus/releases
- 64. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
DEMO
Test Data with
Random() and Bogus
- 65. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Alternative (Better) Solutions
Mockaroo (free, but only 1000 rows)
https://mockaroo.com/
Redgate SQL Data Generator (licensed)
https://www.red-gate.com/products/sql-
development/sql-data-generator/index
- 66. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Static Dimensions
- 67. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Static Dimensions (or any kind of static table)
SCD Type 0: Not changing
Lookup Tables
1. Create table definition
2. Add static source rows
3. Generate INSERT statements
- 68. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml Static Sources
Part of <Table> objects
Defines rows to be inserted when table is created
• Unknown dimension members
• Date dimension
• Code tables
- 69. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml Static Source
<Table Name="Table" SchemaName="Database.Schema">
<Columns>
<Column Name="Column1" DataType="Int32" />
<Column Name="Column2" DataType="String" Length="10" />
</Columns>
<Sources>
<StaticSource Name="TableRows">
<Rows>
<Row>
<ColumnValues>
<ColumnValue ColumnName="Column1" Value="-1" />
<ColumnValue ColumnName="Column2" Value="'Unknown'" />
</ColumnValues>
</Row>
</Rows>
</StaticSource>
</Sources>
</Table>
- 70. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml Static Source
<# if (table.Sources.Any()) { #>
<#=TableToPackageLowerer.GetStaticSourceInsertStatements(
table.SchemaQualifiedName,
table.HasIdentity,
(AstTableStaticSourceNode)table.Sources.FirstOrDefault()
)#>
<# } #>
- 71. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Drop and Create
Facts and Dimensions
- 72. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Could not drop object 'Schema.Table'
because it is referenced by a
FOREIGN KEY constraint.
- 73. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Drop and Create
Create Order: Dim1 Dim2 Fact1 Fact2
Fact2 Fact1 Dim2 Dim1Drop Order:
- 74. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Drop and Create
Create Order: Dim1 Dim2 Fact1 Fact2
Fact2 Fact1 Dim2 Dim1Drop Order:
- 75. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Topological Sort
"A topological sort of a directed graph is a
linear ordering of its vertices such that for
every directed edge uv from vertex u to vertex
v, u comes before v in the ordering"
- 76. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Topological Sort
"A topological sort of a directed graph is a
linear ordering of its vertices such that for
every directed edge uv from vertex u to vertex
v, u comes before v in the ordering"
- 77. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Topological Sort
"Sorts objects by dependencies.
For example dimensions before fact tables."
- 78. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Topological Sort
"Sorts objects by dependencies.
For example dimensions before fact tables."
- 79. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Topological Sort
IEnumerable<T> DependencyAnalysis.TopologicalSort<T> (
IEnumerable<T> source,
Expression<Func<T, T>> relationProperty
)
- 80. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Topological Sort
var dependentTables =
DependencyAnalysis.TopologicalSort<AstTableNode> (
RootNode.Tables,
t => t.Columns
.OfType<AstTableColumnTableReferenceNode>()
.Select(c => c.ForeignTable)
);
- 81. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
DEMO
Static Dimensions
and Topological Sort
- 82. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Comparing
Databases
- 83. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Once upon a time…
- 85. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
How to Compare
Tables and Columns
- 86. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
LINQ
Sort
OrderBy, ThenBy
Filter
Where, OfType
Group
GroupBy
Aggregate
Count, Sum
Check Collections
All, Any, Contains
Set Operations
Except, Intersect
Project Collections
Select, SelectMany
- 87. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
LINQ
Sort
OrderBy, ThenBy
Filter
Where, OfType
Group
GroupBy
Aggregate
Count, Sum
Check Collections
All, Any, Contains
Set Operations
Except, Intersect
Project Collections
Select, SelectMany
- 88. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
LINQ: Filter Collections
Where()
Returns the filtered collection with all elements that meet the criteria
table.Where(t => t.Schema.Name == "dim")
- 89. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
LINQ: Set Operations
Except()
Returns elements in first collection that are not in second collection
table1.Except(table2)
- 90. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
LINQ: Set Operations
Intersect()
Returns elements in first collection that are also in second collection
table1.Intersect(table2)
- 91. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
LINQ: Project Collections
Select()
Creates a new collection from specified attributes
table.Select(t => t.Name)
- 92. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
DEMO
Comparing
Databases
- 93. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Alternative (Better) Solutions
Redgate SQL Compare (licensed)
https://www.red-gate.com/products/sql-
development/sql-compare/
- 94. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
…the past 60 minutes…
T-SQL
Test Data Comparing
Dimensions
- 95. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Is this useful?
- 96. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
What other ideas
do you have?
- 98. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
@cathrinew
cathrinew.net
hi@cathrinew.net
Biml resources and demo files:
cathrinew.net/biml