Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Biml Tips and Tricks
Not just for SSIS Packages!
Cathrine Wilhelmsen
SQLGrillen · June 22nd 2018
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Session Description
"Wait, what? Biml is not just for generati...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Cathrine Wilhelmsen
@cathrinew cathrinew.net
Data Warehouse & ...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
…the next 60 minutes…
T-SQL
Test Data Comparing
Dimensions
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Wait…
Can't I use
<something else>?
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Of course!
But Biml works well
with source metadata
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
And…
Biml is fun!
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
But… Biml? How?
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Traditional Biml
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Crazy Fun Biml
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
What do you need?
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Preview Pane
The power is in the…
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Traditional BimlScript
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Traditional BimlScript
<# foreach (var table in RootNode.Table...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
BimlScript to Biml
Preview Pane
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
BimlScript to... ???
Preview Pane
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Crazy Fun BimlScript
<# foreach (var table in RootNode.Tables)...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
T-SQL from Biml
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Built-in
T-SQL from Biml
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Built-in T-SQL from Biml
Functional T-SQL
GetSelectSql
GetDrop...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Functional T-SQL: GetSelectSql
SELECT
[Column1]
,[Column2]
FRO...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Functional T-SQL: GetDropAndCreateDdl
IF EXISTS (...)
DROP TAB...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Functional T-SQL: GetTableSql
IF EXISTS (...)
DROP TABLE [Sche...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Functional T-SQL
Use directly in SSIS
Copy and run in SSMS
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
SSMS Snippets: GetDeleteSql
DELETE FROM [Schema].[Table]
WHERE...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
SSMS Snippets: GetInsertSql
INSERT INTO [Schema].[Table] (
[Co...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
SSMS Snippets: GetUpdateSql
UPDATE [Schema].[Table] SET
[Colum...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Copy and run in SSMS
Replace Template Parameters with actual v...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Custom
T-SQL from Biml
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Custom T-SQL from Biml
Table Metadata
<#=table.Name#>
<#=table...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Custom T-SQL from Biml
Column Methods
GetColumnList
GetColumnA...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Column Methods
Return code fragments
Use as building blocks in...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Column Methods
Return code fragments
Use as building blocks in...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
"A lambda expression is an anonymous
functi...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
"A lambda expression is an anonymous
functi...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
column => column.Name == "ColumnID"
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
column => column.Name == "ColumnID"
The arr...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
column => column.Name == "ColumnID"
Input p...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
column => column.Name == "ColumnID"
Express...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Lambda Expressions
column => column.Name == "ColumnID"
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnList
Get columns for SELECT
[Column1], [Column2], [Co...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnList
GetColumnList()
GetColumnList("src")
GetColumnLi...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnAssignmentList
Get columns for UPDATE … SET
[l].[Colu...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnAssignmentList
GetColumnAssignmentList()
GetColumnAss...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnComparisonList
Get columns for JOIN … ON
[l].[Column1...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
GetColumnComparisonList
GetColumnComparisonList()
GetColumnCom...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
DEMO
T-SQL from Biml
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Test Data
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml + Random()
Create Random Test Data
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Random Class
Random()
Pseudo-random number generator
Random en...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Random Methods
Next() Random number
Next(100) Random number sm...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml Helper Method
<#+
Random r = new Random();
public string ...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml Helper Helper Method
<#+
private string GetRandomString(i...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml + Random()
Very limited functionality
No dependencies
Wor...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml + Bogus
Create Random and Specific Test Data
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Bogus Project
Simple and sane fake data generator for .NET
htt...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Address
Commerce
Company
Database
Date
Finance
Internet
Lorem
...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
The Bogus Methods
<#@ assembly name="Bogus.dll" #>
<#@ import ...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml + Bogus()
Better functionality than Random()
Easier to un...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Installing Bogus
Install via Nuget:
Install-Package Bogus
Or d...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
DEMO
Test Data with
Random() and Bogus
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Alternative (Better) Solutions
Mockaroo (free, but only 1000 r...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Static Dimensions
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Static Dimensions (or any kind of static table)
SCD Type 0: No...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml Static Sources
Part of <Table> objects
Defines rows to be...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml Static Source
<Table Name="Table" SchemaName="Database.Sc...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Biml Static Source
<# if (table.Sources.Any()) { #>
<#=TableTo...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Drop and Create
Facts and Dimensions
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Could not drop object 'Schema.Table'
because it is referenced ...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Drop and Create
Create Order: Dim1 Dim2 Fact1 Fact2
Fact2 Fact...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Drop and Create
Create Order: Dim1 Dim2 Fact1 Fact2
Fact2 Fact...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Topological Sort
"A topological sort of a directed graph is a
...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Topological Sort
"A topological sort of a directed graph is a
...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Topological Sort
"Sorts objects by dependencies.
For example d...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Topological Sort
"Sorts objects by dependencies.
For example d...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Topological Sort
IEnumerable<T> DependencyAnalysis.Topological...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Topological Sort
var dependentTables =
DependencyAnalysis.Topo...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
DEMO
Static Dimensions
and Topological Sort
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Comparing
Databases
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Once upon a time…
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Panic!
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
How to Compare
Tables and Columns
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
LINQ
Sort
OrderBy, ThenBy
Filter
Where, OfType
Group
GroupBy
A...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
LINQ
Sort
OrderBy, ThenBy
Filter
Where, OfType
Group
GroupBy
A...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
LINQ: Filter Collections
Where()
Returns the filtered collecti...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
LINQ: Set Operations
Except()
Returns elements in first collec...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
LINQ: Set Operations
Intersect()
Returns elements in first col...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
LINQ: Project Collections
Select()
Creates a new collection fr...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
DEMO
Comparing
Databases
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Alternative (Better) Solutions
Redgate SQL Compare (licensed)
...
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
…the past 60 minutes…
T-SQL
Test Data Comparing
Dimensions
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Is this useful?
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
What other ideas
do you have?
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
Have fun!
© 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
@cathrinew
cathrinew.net
hi@cathrinew.net
Biml resources and d...
Upcoming SlideShare
Loading in …5
×

of

Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 1 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 2 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 3 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 4 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 5 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 6 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 7 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 8 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 9 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 10 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 11 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 12 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 13 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 14 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 15 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 16 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 17 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 18 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 19 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 20 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 21 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 22 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 23 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 24 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 25 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 26 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 27 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 28 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 29 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 30 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 31 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 32 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 33 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 34 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 35 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 36 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 37 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 38 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 39 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 40 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 41 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 42 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 43 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 44 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 45 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 46 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 47 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 48 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 49 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 50 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 51 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 52 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 53 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 54 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 55 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 56 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 57 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 58 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 59 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 60 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 61 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 62 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 63 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 64 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 65 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 66 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 67 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 68 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 69 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 70 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 71 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 72 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 73 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 74 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 75 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 76 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 77 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 78 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 79 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 80 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 81 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 82 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 83 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 84 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 85 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 86 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 87 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 88 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 89 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 90 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 91 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 92 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 93 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 94 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 95 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 96 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 97 Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018) Slide 98
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

1 Like

Share

Download to read offline

Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018)

Download to read offline

Biml Tips and Tricks: Not Just for SSIS Packages! (Presented at SQLGrillen on June 22nd, 2018)

Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018)

  1. 1. Biml Tips and Tricks Not just for SSIS Packages! Cathrine Wilhelmsen SQLGrillen · June 22nd 2018
  2. 2. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
  3. 3. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Session Description "Wait, what? Biml is not just for generating SSIS packages?" Absolutely not! Come and see how you can use Biml (Business Intelligence Markup Language) to save time and speed up other Data Warehouse development tasks. You can generate complex T-SQL statements with Biml instead of using dynamic SQL, create test data and populate static dimensions, and even compare tables and views across multiple servers and databases. Don't Repeat Yourself, start automating those boring, manual tasks today!
  4. 4. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Cathrine Wilhelmsen @cathrinew cathrinew.net Data Warehouse & Business Intelligence Consultant
  5. 5. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) …the next 60 minutes… T-SQL Test Data Comparing Dimensions
  6. 6. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
  7. 7. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Wait… Can't I use <something else>?
  8. 8. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Of course! But Biml works well with source metadata
  9. 9. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) And… Biml is fun!
  10. 10. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) But… Biml? How?
  11. 11. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Traditional Biml
  12. 12. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Crazy Fun Biml
  13. 13. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What do you need?
  14. 14. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Preview Pane The power is in the…
  15. 15. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Traditional BimlScript
  16. 16. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Traditional BimlScript <# foreach (var table in RootNode.Tables) { #> <Package Name="Load_<#=table.Name#>" /> <# } #> <Package Name="Load_Customer" /> <Package Name="Load_Product" /> <Package Name="Load_Sales" />
  17. 17. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlScript to Biml Preview Pane
  18. 18. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlScript to... ??? Preview Pane
  19. 19. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Crazy Fun BimlScript <# foreach (var table in RootNode.Tables) { #> Yay, a package! <#=table.Name#> :) <# } #> Yay, a package! Load_Customer :) Yay, a package! Load_Product :) Yay, a package! Load_Sales :)
  20. 20. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
  21. 21. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) T-SQL from Biml
  22. 22. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Built-in T-SQL from Biml
  23. 23. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Built-in T-SQL from Biml Functional T-SQL GetSelectSql GetDropAndCreateDdl GetTableSql SSMS Snippets GetInsertSql GetUpdateSql GetDeleteSql
  24. 24. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Functional T-SQL: GetSelectSql SELECT [Column1] ,[Column2] FROM [Schema].[Table]
  25. 25. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Functional T-SQL: GetDropAndCreateDdl IF EXISTS (...) DROP TABLE [Schema].[Table] GO CREATE TABLE [Schema].[Table] ( [Column1] int IDENTITY(1,1) NOT NULL ,[Column2] nvarchar(50) NOT NULL -- Constraints )
  26. 26. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Functional T-SQL: GetTableSql IF EXISTS (...) DROP TABLE [Schema].[Table] GO CREATE TABLE [Schema].[Table] ( [Column1] int IDENTITY(1,1) NOT NULL ,[Column2] nvarchar(50) NOT NULL -- Constraints )
  27. 27. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Functional T-SQL Use directly in SSIS Copy and run in SSMS
  28. 28. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) SSMS Snippets: GetDeleteSql DELETE FROM [Schema].[Table] WHERE <Search Conditions,,>
  29. 29. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) SSMS Snippets: GetInsertSql INSERT INTO [Schema].[Table] ( [Column1] ,[Column1] ) VALUES ( <Column1,int,>, <Column2,nvarchar(50),> )
  30. 30. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) SSMS Snippets: GetUpdateSql UPDATE [Schema].[Table] SET [Column1] = <Column1, int,>, [Column2] = <Column2, nvarchar(50),> WHERE <Search Conditions,,>
  31. 31. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Copy and run in SSMS Replace Template Parameters with actual values CTRL SHIFT M SSMS Snippets
  32. 32. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Custom T-SQL from Biml
  33. 33. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Custom T-SQL from Biml Table Metadata <#=table.Name#> <#=table.SchemaQualifiedName#> SELECT *
  34. 34. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Custom T-SQL from Biml Column Methods GetColumnList GetColumnAssignmentList GetColumnComparisonList
  35. 35. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Column Methods Return code fragments Use as building blocks in custom T-SQL Filter columns by using lambda expressions
  36. 36. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Column Methods Return code fragments Use as building blocks in custom T-SQL Filter columns by using lambda expressions
  37. 37. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions "A lambda expression is an anonymous function that you can use to create delegates or expression tree types"
  38. 38. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions "A lambda expression is an anonymous function that you can use to create delegates or expression tree types"
  39. 39. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions column => column.Name == "ColumnID"
  40. 40. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions column => column.Name == "ColumnID" The arrow is the lambda operator
  41. 41. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions column => column.Name == "ColumnID" Input parameter is on the left side
  42. 42. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions column => column.Name == "ColumnID" Expression is on the right side
  43. 43. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions column => column.Name == "ColumnID"
  44. 44. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnList Get columns for SELECT [Column1], [Column2], [Column3]
  45. 45. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnList GetColumnList() GetColumnList("src") GetColumnList(c => c.Name == "ColumnID") GetColumnList(c => c.DataType != DbType.Guid, "src")
  46. 46. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnAssignmentList Get columns for UPDATE … SET [l].[Column1] = [r].[Column1] ,[l].[Column2] = [r].[Column2] ,[l].[Column3] = [r].[Column3]
  47. 47. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnAssignmentList GetColumnAssignmentList() GetColumnAssignmentList("dst", "src") GetColumnAssignmentList(c => !c.IsUsedInPrimaryKey) GetColumnAssignmentList(c => !c.IsUsedInPrimaryKey, "dst", "src")
  48. 48. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnComparisonList Get columns for JOIN … ON [l].[Column1] = [r].[Column1] AND [l].[Column2] = [r].[Column2] AND [l].[Column3] = [r].[Column3]
  49. 49. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnComparisonList GetColumnComparisonList() GetColumnComparisonList("!=") GetColumnComparisonList("dst", "src") GetColumnComparisonList(c => c.IsUsedInPrimaryKey) GetColumnComparisonList(c => c.IsUsedInPrimaryKey, "dst", "src") GetColumnComparisonList(c => c.IsUsedInPrimaryKey, "!=", "dst", "src")
  50. 50. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO T-SQL from Biml
  51. 51. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Test Data
  52. 52. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml + Random() Create Random Test Data
  53. 53. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Random Class Random() Pseudo-random number generator Random enough for practical purposes Random(12345) Always generates the same series of random numbers
  54. 54. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Random Methods Next() Random number Next(100) Random number smaller than specified value Next(1000, 2000) Random number within specified range NextDouble() Random decimal number between 0.0 and 1.0
  55. 55. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Helper Method <#+ Random r = new Random(); public string GetTestData(AstTableColumnBaseNode column) { switch (column.DataType) { case DbType.Int32 : return r.Next(Int32.MaxValue).ToString(); case DbType.String : return "'" + GetRandomString(column.Length) + "'" default : return "NULL"; } } #>
  56. 56. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Helper Helper Method <#+ private string GetRandomString(int length) { var allowedChars = "ABC123"; var randomChars = new char[length]; for (int i = 0; i < length; i++) { randomChars[i] = allowedChars[ r.Next(allowedChars.Length) ]; } return new String(randomCharacters); } #>
  57. 57. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml + Random() Very limited functionality No dependencies Works out-of-the box
  58. 58. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml + Bogus Create Random and Specific Test Data
  59. 59. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Bogus Project Simple and sane fake data generator for .NET https://github.com/bchavez/Bogus Advanced functionality for custom objects …or simple functionality for Biml hacks :)
  60. 60. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Address Commerce Company Database Date Finance Internet Lorem Name Person Phone System Bogus API Supports all data types plus built-in test data, including:
  61. 61. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) The Bogus Methods <#@ assembly name="Bogus.dll" #> <#@ import namespace="Bogus" #> <# var f = new Faker(); #> <# Person p = new Person(); #> INSERT INTO dbo.Employee(FirstName, LastName, Birthday) VALUES ( '<#=p.FirstName#>', '<#=p.LastName#>', '<#=p.DateOfBirth.ToString("yyyyMMdd")#>' )
  62. 62. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml + Bogus() Better functionality than Random() Easier to understand, frequently updated Requires additional files
  63. 63. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Installing Bogus Install via Nuget: Install-Package Bogus Or download latest release .zip: https://github.com/bchavez/Bogus/releases
  64. 64. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO Test Data with Random() and Bogus
  65. 65. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Alternative (Better) Solutions Mockaroo (free, but only 1000 rows) https://mockaroo.com/ Redgate SQL Data Generator (licensed) https://www.red-gate.com/products/sql- development/sql-data-generator/index
  66. 66. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Static Dimensions
  67. 67. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Static Dimensions (or any kind of static table) SCD Type 0: Not changing Lookup Tables 1. Create table definition 2. Add static source rows 3. Generate INSERT statements
  68. 68. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Static Sources Part of <Table> objects Defines rows to be inserted when table is created • Unknown dimension members • Date dimension • Code tables
  69. 69. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Static Source <Table Name="Table" SchemaName="Database.Schema"> <Columns> <Column Name="Column1" DataType="Int32" /> <Column Name="Column2" DataType="String" Length="10" /> </Columns> <Sources> <StaticSource Name="TableRows"> <Rows> <Row> <ColumnValues> <ColumnValue ColumnName="Column1" Value="-1" /> <ColumnValue ColumnName="Column2" Value="'Unknown'" /> </ColumnValues> </Row> </Rows> </StaticSource> </Sources> </Table>
  70. 70. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Static Source <# if (table.Sources.Any()) { #> <#=TableToPackageLowerer.GetStaticSourceInsertStatements( table.SchemaQualifiedName, table.HasIdentity, (AstTableStaticSourceNode)table.Sources.FirstOrDefault() )#> <# } #>
  71. 71. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Drop and Create Facts and Dimensions
  72. 72. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Could not drop object 'Schema.Table' because it is referenced by a FOREIGN KEY constraint.
  73. 73. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Drop and Create Create Order: Dim1 Dim2 Fact1 Fact2 Fact2 Fact1 Dim2 Dim1Drop Order:
  74. 74. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Drop and Create Create Order: Dim1 Dim2 Fact1 Fact2 Fact2 Fact1 Dim2 Dim1Drop Order:
  75. 75. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Topological Sort "A topological sort of a directed graph is a linear ordering of its vertices such that for every directed edge uv from vertex u to vertex v, u comes before v in the ordering"
  76. 76. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Topological Sort "A topological sort of a directed graph is a linear ordering of its vertices such that for every directed edge uv from vertex u to vertex v, u comes before v in the ordering"
  77. 77. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Topological Sort "Sorts objects by dependencies. For example dimensions before fact tables."
  78. 78. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Topological Sort "Sorts objects by dependencies. For example dimensions before fact tables."
  79. 79. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Topological Sort IEnumerable<T> DependencyAnalysis.TopologicalSort<T> ( IEnumerable<T> source, Expression<Func<T, T>> relationProperty )
  80. 80. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Topological Sort var dependentTables = DependencyAnalysis.TopologicalSort<AstTableNode> ( RootNode.Tables, t => t.Columns .OfType<AstTableColumnTableReferenceNode>() .Select(c => c.ForeignTable) );
  81. 81. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO Static Dimensions and Topological Sort
  82. 82. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Comparing Databases
  83. 83. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Once upon a time…
  84. 84. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Panic!
  85. 85. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) How to Compare Tables and Columns
  86. 86. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) LINQ Sort OrderBy, ThenBy Filter Where, OfType Group GroupBy Aggregate Count, Sum Check Collections All, Any, Contains Set Operations Except, Intersect Project Collections Select, SelectMany
  87. 87. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) LINQ Sort OrderBy, ThenBy Filter Where, OfType Group GroupBy Aggregate Count, Sum Check Collections All, Any, Contains Set Operations Except, Intersect Project Collections Select, SelectMany
  88. 88. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) LINQ: Filter Collections Where() Returns the filtered collection with all elements that meet the criteria table.Where(t => t.Schema.Name == "dim")
  89. 89. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) LINQ: Set Operations Except() Returns elements in first collection that are not in second collection table1.Except(table2)
  90. 90. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) LINQ: Set Operations Intersect() Returns elements in first collection that are also in second collection table1.Intersect(table2)
  91. 91. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) LINQ: Project Collections Select() Creates a new collection from specified attributes table.Select(t => t.Name)
  92. 92. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO Comparing Databases
  93. 93. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Alternative (Better) Solutions Redgate SQL Compare (licensed) https://www.red-gate.com/products/sql- development/sql-compare/
  94. 94. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) …the past 60 minutes… T-SQL Test Data Comparing Dimensions
  95. 95. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Is this useful?
  96. 96. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What other ideas do you have?
  97. 97. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Have fun!
  98. 98. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) @cathrinew cathrinew.net hi@cathrinew.net Biml resources and demo files: cathrinew.net/biml
  • BrentSmith104

    Oct. 12, 2018

Biml Tips and Tricks: Not Just for SSIS Packages! (Presented at SQLGrillen on June 22nd, 2018)

Views

Total views

1,676

On Slideshare

0

From embeds

0

Number of embeds

14

Actions

Downloads

16

Shares

0

Comments

0

Likes

1

×