Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Level Up Your Biml:
Best Practices and Coding Techniques
Cathrine Wilhelmsen, Inmeta
@cathrinew | cathrinew.net
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Abstract
Is your Biml solution starting to remind you of a bowl of tangled s...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
@cathrinew
cathrinew.net
Free online webinar
events
Free 1-day local
training events
Local user groups
around the world
Online special interest
use...
Download the GuideBook App
and search: PASS Summit 2018
Follow the QR code link displayed on session
signage throughout th...
Please silence
cell phones
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Biml Tools Staging
Biml for Beginners
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Code ModelManage
Level Up Your Biml
Biml Code
Management
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Don't Repeat Yourself
Refactor and centralize code
Update once for all proje...
Include Files
Automated Copy & Paste
Include Files
Works like an automated Copy & Paste
<#@ include file="CommonCode.biml" #>
Code from included file replaces ...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Include Files
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Include Files
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Include Files
CallBimlScript
Parameterized Include with Logic
CallBimlScript
Parameterized include (like a stored procedure)
CallBimlScript file specifies accepted parameters:
<#@ prop...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
CallBimlScript
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
CallBimlScript
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
CallBimlScript
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
CallBimlScript
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
CallBimlScript
CallBimlScriptWithOutput
Return Code and Object
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
CallBimlScriptWithOutput
CallBimlScript file specifies accepted parameters:
...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
CallBimlScriptWithOutput
Main file defines, passes, and uses output object
i...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
CallBimlScriptWithOutput
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
CallBimlScriptWithOutput
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
CallBimlScriptWithOutput
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
CallBimlScriptWithOutput
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
CallBimlScriptWithOutput
use CustomOutput
C# Code
Custom Classes and Methods
C# Code: Inline Blocks
<#+ ... #>
<#+ #>
C# Code: Included Files
<#@ include file="CodeBlock.biml" #>
<#+ #>
C# Code: Separate Code Files
<#@ code file="Code.cs" #>
EXAMPLE
Refactoring
Biml Code
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Two Approaches to Refactoring
1. Identify Identical Code → Include Files
2. ...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Log Package Execution (Start)
Log Package Execution (Stop)
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Log Package
Execution (Start)
Log Package
Execution (Stop)
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Two Approaches to Refactoring
1. Identify Identical Code → Include Files
2. ...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Log Package Execution
Log Package Execution
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Log Package
Execution
Log Package
Execution
DEMO
Auditing
Framework
Don't Repeat Yourself
If you reuse code more than 3 times,
refactor :)
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
+10000 XP
Congratulations! You reached level 60!
Practical
Biml Coding
Annotations and ObjectTags
Store and Pass Metadata Between Biml Files
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Annotations and ObjectTags
Store metadata by attaching tags to Biml objects
...
Annotations
Create annotations directly in Biml:
<OleDbConnection Name="Dest">
<Annotations>
<Annotation Tag="Schema">stg<...
ObjectTags
Create ObjectTags in BimlScript:
RootNode.ObjectTag["Filter"]
= new List<string>{"Product","ProductCategory"};
...
LINQ
Query and Filter Collections
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
LINQ
One language to query:
SQL Server Databases
XML Documents
Datasets
Coll...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
LINQ Methods
Sort
OrderBy, Reverse
Filter
Where, OfType
Get
First, Last
Chec...
LINQ Methods Example
var firstConnection = RootNode.Connections.First()
foreach (var table in RootNode.Tables.Where(…))
if...
LINQ Methods Example
foreach (var table in RootNode.Tables.Where(…))
if (RootNode.Packages.Any(…))
But what do you put in ...
Lambda Expressions
"A lambda expression is an anonymous
function that you can use to create delegates
or expression tree t...
Lambda Expressions
"A lambda expression is an anonymous
function that you can use to create delegates
or expression tree t...
Lambda Expressions
column => column.Name == "ColumnID"
Lambda Expressions
column => column.Name == "ColumnID"
The arrow ("goes to") is the lambda operator
Lambda Expressions
column => column.Name == "ColumnID"
Input parameter is on the left side
Lambda Expressions
column => column.Name == "ColumnID"
Expression is on the right side
Lambda Expressions
column => column.Name == "ColumnID"
LINQ and Lambda
Chain LINQ Methods and use Lambda Expressions for
simple and powerful querying of collections:
.Where(tabl...
Column Methods
Generate SQL from Biml
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
GetColumnList()
GetColumnAssignmentList()
GetColumnComparisonList()
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
GetColumnList
Get columns for SELECT:
[Column1], [Column2], [Column3]
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
GetColumnAssignmentList
Get columns for UPDATE … SET:
[left].[Column1] = [ri...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
GetColumnComparisonList
Get columns for JOIN … ON:
[left].[Column1] = [right...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Column Methods and Lambda
Use Lambda Expressions to filter columns:
GetColum...
C# Extension Methods
Custom LINQ-style!
Extension Methods
"Make it look like the method belongs to
an object instead of a helper class"
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
From this…
public static string
GetName(AstTableNode table) {...}
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
…to this
public static string
GetName(this AstTableNode table) {...}
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
From this…
HelperClass.GetName(table)
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
…to this
table.GetName()
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
+100000 XP
Congratulations! You reached level 80!
Facts and
Dimensions
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Solution Overview
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Solution Overview
Dimensions
SCD2
Hybrid
SCD1 SCD2
Facts
Trans-
actional
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Facts
Source View
Lookup
Dimension Keys
Insert
New Rows
Load Fact
Get Latest...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Dimensions
Source View
Lookup
Existing Rows
Insert
New Rows
Insert
Updated R...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Solution Overview
Data Warehouse
Facts Dims Update
Staging
Source
Views
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Solution Process
1. Model Facts and Dimensions
2. Generate SQL Scripts
3. Mo...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Solution Process
1. Model Facts and Dimensions
2. Generate SQL Scripts
3. Mo...
80 / 20
Automated
Manual
Facts and Dimensions
Metadata Model in Excel
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Metadata Model
Connections Projects
Tables Columns
EXAMPLE
Metadata
Model in Excel
Facts and Dimensions
Metadata Model in Biml
Facts and Dimensions
Use Biml features to simplify code:
• ScdType on columns
• StaticSource for static and unknown dimens...
ScdType
Define ScdType per column:
<Column Name="DimSK" DataType="Int32"
IsNullable="false" ScdType="SurrogateKey" />
Use ...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
ScdType Values
SurrogateKey
Key
Update
Historical
Audit
Other
Often IDENTITY...
StaticSource
Define rows to be inserted when table is created:
<Table Name="Table">
<Sources>
<StaticSource Name="TableRow...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
StaticSource Values
<StaticSource Name="TableRows">
<Rows>
<Row>
<ColumnValu...
CloneTable
Create a copy of a table:
<CloneTable Name="NewTable"
TableName="OriginalSchema.OriginalTable"
SchemaName="Data...
DEMO
Metadata
Model in Biml
SQL Scripts
Drop and Create
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Could not drop object 'Schema.Table'
because it is referenced by a
FOREIGN K...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Drop and Create
Create Order: Dim1 Dim2 Fact1 Fact2
Fact2 Fact1 Dim2 Dim1Dro...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Drop and Create
Create Order: Dim1 Dim2 Fact1 Fact2
Fact2 Fact1 Dim2 Dim1Dro...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Topological Sort
"A topological sort of a directed graph is a linear
orderin...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Topological Sort
"A topological sort of a directed graph is a linear
orderin...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Topological Sort
"Sorts objects by dependencies.
For example dimensions befo...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Topological Sort
"Sorts objects by dependencies.
For example dimensions befo...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Topological Sort
var dependentTables =
DependencyAnalysis.TopologicalSort<As...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Topological Sort: Method
var dependentTables =
DependencyAnalysis.Topologica...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Topological Sort: Collection to Sort
var dependentTables =
DependencyAnalysi...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Topological Sort: How to Sort
var dependentTables =
DependencyAnalysis.Topol...
DEMO
Facts and
Dimensions
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
+1000000 XP
Congratulations! You reached level 100!
Get things done
Start small
Start simple
Solve a problem
Keep going
Expand
Improve
Deliver often
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Download the GuideBook App
and search: PASS Summit 2018
Follow the QR code link displayed on session
signage throughout th...
@cathrinew
cathrinew.net
hi@cathrinew.net
cathrinew.net/biml
Upcoming SlideShare
Loading in …5
×

of

Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 1 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 2 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 3 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 4 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 5 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 6 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 7 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 8 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 9 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 10 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 11 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 12 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 13 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 14 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 15 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 16 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 17 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 18 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 19 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 20 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 21 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 22 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 23 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 24 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 25 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 26 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 27 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 28 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 29 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 30 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 31 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 32 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 33 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 34 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 35 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 36 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 37 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 38 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 39 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 40 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 41 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 42 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 43 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 44 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 45 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 46 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 47 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 48 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 49 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 50 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 51 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 52 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 53 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 54 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 55 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 56 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 57 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 58 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 59 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 60 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 61 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 62 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 63 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 64 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 65 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 66 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 67 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 68 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 69 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 70 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 71 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 72 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 73 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 74 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 75 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 76 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 77 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 78 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 79 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 80 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 81 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 82 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 83 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 84 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 85 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 86 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 87 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 88 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 89 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 90 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 91 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 92 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 93 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 94 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 95 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 96 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 97 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 98 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 99 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 100 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 101 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 102 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 103 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 104 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 105 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 106 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 107 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 108 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 109 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 110 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 111 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 112 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 113 Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018) Slide 114
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

3 Likes

Share

Download to read offline

Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018)

Download to read offline

Level Up Your Biml: Best Practices and Coding Techniques (Presented at PASS Summit on November 8th, 2018)

Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018)

  1. 1. Level Up Your Biml: Best Practices and Coding Techniques Cathrine Wilhelmsen, Inmeta @cathrinew | cathrinew.net
  2. 2. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Abstract Is your Biml solution starting to remind you of a bowl of tangled spaghetti code? Good! That means you are solving real problems while saving a lot of time. The next step is to make sure that your solution does not grow too complex and confusing - you do not want to waste all that saved time on future maintenance! Attend this session for an overview of Biml best practices and coding techniques. Learn how to improve and simplify your solution by using some common and some lesser-known Biml features. If standard Biml is not enough, you can implement custom logic by creating your own C# classes and methods. Finally, see how to bring everything together in an example project for creating and loading a data mart with facts and dimensions. Start improving your code today and level up your Biml in no time!
  3. 3. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) @cathrinew cathrinew.net
  4. 4. Free online webinar events Free 1-day local training events Local user groups around the world Online special interest user groups Business analytics training Get involved Explore everything PASS has to offer Free Online Resources Newsletters PASS.org
  5. 5. Download the GuideBook App and search: PASS Summit 2018 Follow the QR code link displayed on session signage throughout the conference venue and in the program guide Session evaluations Your feedback is important and valuable. Go to passSummit.com 3 Ways to Access: Submit by 5pm Friday, November 16th to win prizes.
  6. 6. Please silence cell phones
  7. 7. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
  8. 8. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Biml Tools Staging Biml for Beginners
  9. 9. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Code ModelManage Level Up Your Biml
  10. 10. Biml Code Management
  11. 11. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Don't Repeat Yourself Refactor and centralize code Update once for all projects 1. Include Files 2. CallBimlScript 3. C# Code
  12. 12. Include Files Automated Copy & Paste
  13. 13. Include Files Works like an automated Copy & Paste <#@ include file="CommonCode.biml" #> Code from included file replaces this include directive
  14. 14. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Include Files
  15. 15. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Include Files
  16. 16. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Include Files
  17. 17. CallBimlScript Parameterized Include with Logic
  18. 18. CallBimlScript Parameterized include (like a stored procedure) CallBimlScript file specifies accepted parameters: <#@ property name="Parameter" type="String" #> Main file calls and passes parameters: <#=CallBimlScript("CommonCode.biml", Parameter)#>
  19. 19. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) CallBimlScript
  20. 20. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) CallBimlScript
  21. 21. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) CallBimlScript
  22. 22. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) CallBimlScript
  23. 23. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) CallBimlScript
  24. 24. CallBimlScriptWithOutput Return Code and Object
  25. 25. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) CallBimlScriptWithOutput CallBimlScript file specifies accepted parameters: <#@ property name="ProjectName" type="String" #> CallBimlScriptWithOutput file also specifies object: <# CustomOutput.Project = metadata.Project; #>
  26. 26. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) CallBimlScriptWithOutput Main file defines, passes, and uses output object in addition to regular parameters: <# dynamic project; #> <#=CallBimlScriptWithOutput("Metadata.biml", out project, "Staging Project")#> <#=project.SourceConnection#>
  27. 27. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) CallBimlScriptWithOutput
  28. 28. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) CallBimlScriptWithOutput
  29. 29. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) CallBimlScriptWithOutput
  30. 30. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) CallBimlScriptWithOutput
  31. 31. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) CallBimlScriptWithOutput use CustomOutput
  32. 32. C# Code Custom Classes and Methods
  33. 33. C# Code: Inline Blocks <#+ ... #> <#+ #>
  34. 34. C# Code: Included Files <#@ include file="CodeBlock.biml" #> <#+ #>
  35. 35. C# Code: Separate Code Files <#@ code file="Code.cs" #>
  36. 36. EXAMPLE Refactoring Biml Code
  37. 37. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Two Approaches to Refactoring 1. Identify Identical Code → Include Files 2. Identify Patterns → CallBimlScript
  38. 38. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Log Package Execution (Start) Log Package Execution (Stop)
  39. 39. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Log Package Execution (Start) Log Package Execution (Stop)
  40. 40. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Two Approaches to Refactoring 1. Identify Identical Code → Include Files 2. Identify Patterns → CallBimlScript
  41. 41. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Log Package Execution Log Package Execution
  42. 42. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Log Package Execution Log Package Execution
  43. 43. DEMO Auditing Framework
  44. 44. Don't Repeat Yourself If you reuse code more than 3 times, refactor :)
  45. 45. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) +10000 XP Congratulations! You reached level 60!
  46. 46. Practical Biml Coding
  47. 47. Annotations and ObjectTags Store and Pass Metadata Between Biml Files
  48. 48. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Annotations and ObjectTags Store metadata by attaching tags to Biml objects Annotations: String values ObjectTags: Objects Higher tier files can get tags from lower tier files
  49. 49. Annotations Create annotations directly in Biml: <OleDbConnection Name="Dest"> <Annotations> <Annotation Tag="Schema">stg</Annotation> </Annotations> </OleDbConnection> Use annotations in BimlScript: RootNode.OleDbConnections["Dest"].GetTag("Schema");
  50. 50. ObjectTags Create ObjectTags in BimlScript: RootNode.ObjectTag["Filter"] = new List<string>{"Product","ProductCategory"}; Use ObjectTags in BimlScript: var FilteredTables = RootNode.ObjectTag["Filter"];
  51. 51. LINQ Query and Filter Collections
  52. 52. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) LINQ One language to query: SQL Server Databases XML Documents Datasets Collections Two ways to write queries: SQL-ish Syntax Methods
  53. 53. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) LINQ Methods Sort OrderBy, Reverse Filter Where, OfType Get First, Last Check Collections All, Any, Contains Set Operations Except, Intersect Project Collections Select, SelectMany
  54. 54. LINQ Methods Example var firstConnection = RootNode.Connections.First() foreach (var table in RootNode.Tables.Where(…)) if (RootNode.Packages.Any(…))
  55. 55. LINQ Methods Example foreach (var table in RootNode.Tables.Where(…)) if (RootNode.Packages.Any(…)) But what do you put in here?
  56. 56. Lambda Expressions "A lambda expression is an anonymous function that you can use to create delegates or expression tree types"
  57. 57. Lambda Expressions "A lambda expression is an anonymous function that you can use to create delegates or expression tree types"
  58. 58. Lambda Expressions column => column.Name == "ColumnID"
  59. 59. Lambda Expressions column => column.Name == "ColumnID" The arrow ("goes to") is the lambda operator
  60. 60. Lambda Expressions column => column.Name == "ColumnID" Input parameter is on the left side
  61. 61. Lambda Expressions column => column.Name == "ColumnID" Expression is on the right side
  62. 62. Lambda Expressions column => column.Name == "ColumnID"
  63. 63. LINQ and Lambda Chain LINQ Methods and use Lambda Expressions for simple and powerful querying of collections: .Where(table => table.Schema.Name == "Production") .OrderBy(table => table.Name)
  64. 64. Column Methods Generate SQL from Biml
  65. 65. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) GetColumnList() GetColumnAssignmentList() GetColumnComparisonList()
  66. 66. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) GetColumnList Get columns for SELECT: [Column1], [Column2], [Column3]
  67. 67. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) GetColumnAssignmentList Get columns for UPDATE … SET: [left].[Column1] = [right].[Column1] ,[left].[Column2] = [right].[Column2] ,[left].[Column3] = [right].[Column3]
  68. 68. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) GetColumnComparisonList Get columns for JOIN … ON: [left].[Column1] = [right].[Column1] AND [left].[Column2] = [right].[Column2] AND [left].[Column3] = [right].[Column3]
  69. 69. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Column Methods and Lambda Use Lambda Expressions to filter columns: GetColumnList(column => column.IsUsedInPrimaryKey)
  70. 70. C# Extension Methods Custom LINQ-style!
  71. 71. Extension Methods "Make it look like the method belongs to an object instead of a helper class"
  72. 72. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) From this… public static string GetName(AstTableNode table) {...}
  73. 73. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) …to this public static string GetName(this AstTableNode table) {...}
  74. 74. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) From this… HelperClass.GetName(table)
  75. 75. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) …to this table.GetName()
  76. 76. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) +100000 XP Congratulations! You reached level 80!
  77. 77. Facts and Dimensions
  78. 78. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Solution Overview
  79. 79. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Solution Overview Dimensions SCD2 Hybrid SCD1 SCD2 Facts Trans- actional
  80. 80. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Facts Source View Lookup Dimension Keys Insert New Rows Load Fact Get Latest Key
  81. 81. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Dimensions Source View Lookup Existing Rows Insert New Rows Insert Updated Rows Truncate Update Table Load Dimension Update Rows
  82. 82. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Solution Overview Data Warehouse Facts Dims Update Staging Source Views
  83. 83. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Solution Process 1. Model Facts and Dimensions 2. Generate SQL Scripts 3. Modify Source Views 4. Generate SSIS Packages
  84. 84. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Solution Process 1. Model Facts and Dimensions 2. Generate SQL Scripts 3. Modify Source Views 4. Generate SSIS Packages
  85. 85. 80 / 20 Automated Manual
  86. 86. Facts and Dimensions Metadata Model in Excel
  87. 87. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Metadata Model Connections Projects Tables Columns
  88. 88. EXAMPLE Metadata Model in Excel
  89. 89. Facts and Dimensions Metadata Model in Biml
  90. 90. Facts and Dimensions Use Biml features to simplify code: • ScdType on columns • StaticSource for static and unknown dimension rows • CloneTable for update tables
  91. 91. ScdType Define ScdType per column: <Column Name="DimSK" DataType="Int32" IsNullable="false" ScdType="SurrogateKey" /> Use instead of custom annotations :)
  92. 92. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) ScdType Values SurrogateKey Key Update Historical Audit Other Often IDENTITY Source IDs SCD1 Columns SCD2 Columns ETL Columns Not used in SCD logic
  93. 93. StaticSource Define rows to be inserted when table is created: <Table Name="Table"> <Sources> <StaticSource Name="TableRows">...</StaticSource> </Sources> </Table> Use instead of separate T-SQL scripts :)
  94. 94. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) StaticSource Values <StaticSource Name="TableRows"> <Rows> <Row> <ColumnValues> <ColumnValue ColumnName="Col1" Value="-1" /> </ColumnValues> </Row> </Rows> </StaticSource>
  95. 95. CloneTable Create a copy of a table: <CloneTable Name="NewTable" TableName="OriginalSchema.OriginalTable" SchemaName="Database.NewSchema" /> Use instead of defining tables twice :)
  96. 96. DEMO Metadata Model in Biml
  97. 97. SQL Scripts Drop and Create
  98. 98. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Could not drop object 'Schema.Table' because it is referenced by a FOREIGN KEY constraint.
  99. 99. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Drop and Create Create Order: Dim1 Dim2 Fact1 Fact2 Fact2 Fact1 Dim2 Dim1Drop Order:
  100. 100. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Drop and Create Create Order: Dim1 Dim2 Fact1 Fact2 Fact2 Fact1 Dim2 Dim1Drop Order:
  101. 101. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Topological Sort "A topological sort of a directed graph is a linear ordering of its vertices such that for every directed edge uv from vertex u to vertex v, u comes before v in the ordering"
  102. 102. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Topological Sort "A topological sort of a directed graph is a linear ordering of its vertices such that for every directed edge uv from vertex u to vertex v, u comes before v in the ordering"
  103. 103. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Topological Sort "Sorts objects by dependencies. For example dimensions before fact tables."
  104. 104. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Topological Sort "Sorts objects by dependencies. For example dimensions before fact tables."
  105. 105. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Topological Sort var dependentTables = DependencyAnalysis.TopologicalSort<AstTableNode> ( RootNode.Tables, t => t.Columns .OfType<AstTableColumnTableReferenceNode>() .Select(c => c.ForeignTable) );
  106. 106. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Topological Sort: Method var dependentTables = DependencyAnalysis.TopologicalSort<AstTableNode> ( RootNode.Tables, t => t.Columns .OfType<AstTableColumnTableReferenceNode>() .Select(c => c.ForeignTable) );
  107. 107. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Topological Sort: Collection to Sort var dependentTables = DependencyAnalysis.TopologicalSort<AstTableNode> ( RootNode.Tables, t => t.Columns .OfType<AstTableColumnTableReferenceNode>() .Select(c => c.ForeignTable) );
  108. 108. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Topological Sort: How to Sort var dependentTables = DependencyAnalysis.TopologicalSort<AstTableNode> ( RootNode.Tables, t => t.Columns .OfType<AstTableColumnTableReferenceNode>() .Select(c => c.ForeignTable) );
  109. 109. DEMO Facts and Dimensions
  110. 110. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) +1000000 XP Congratulations! You reached level 100!
  111. 111. Get things done Start small Start simple Solve a problem Keep going Expand Improve Deliver often
  112. 112. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
  113. 113. Download the GuideBook App and search: PASS Summit 2018 Follow the QR code link displayed on session signage throughout the conference venue and in the program guide Session evaluations Your feedback is important and valuable. Go to passSummit.com 3 Ways to Access: Submit by 5pm Friday, November 16th to win prizes.
  114. 114. @cathrinew cathrinew.net hi@cathrinew.net cathrinew.net/biml
  • RaminZarrasvand

    Feb. 15, 2020
  • BarkhaJaved

    Feb. 9, 2019
  • JayMarquez4

    Nov. 12, 2018

Level Up Your Biml: Best Practices and Coding Techniques (Presented at PASS Summit on November 8th, 2018)

Views

Total views

16,659

On Slideshare

0

From embeds

0

Number of embeds

15,270

Actions

Downloads

38

Shares

0

Comments

0

Likes

3

×