Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Biml for Beginners:
Script and Automate
SSIS Development
Cathrine Wilhelmsen
PASS Hybrid VC · July 30th 2018
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Session Description
Are you tired of creating and updating the same SSIS pac...
Cathrine Wilhelmsen
@cathrinew cathrinew.net
Senior BI Consultant, Inmeta
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Biml Basics Tools & Code
SQL and SSIS from Metadata
…the next 60 minutes…
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Once upon a time…
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Meet Sandra,
the SSIS developer
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Sandra has to create a
new staging project with
100+ SSIS packages
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Sandra gets quite bored of
creating the same package
over and over and over
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Yay, Sandra has finally
completed her project!
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
But Sandra's manager
has just a few new
requirements…
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Sandra gets to update
every single SSIS package
one… more… time…
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
…yay
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
It's time for a change!
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Biml?
Business Intelligence Markup Language
Easy to read and write X...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Why use Biml?
SSIS: Plumbing Biml: Business Logic
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
SSIS: Plumbing
Spend time on dragging, dropping, connecting, aligning
Create...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Biml: Business Logic
Spend time on what is unique in a package
Create a patt...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Will Biml solve all your challenges?
Biml is a tool for generating SSIS pack...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
How can Biml help you?
Timesaving: Many SSIS packages from one Biml file
Reu...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
What if you don't have 100+ packages?
Source Control: Keep track of actual l...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
How does SSIS development work?
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
How does Biml development work?
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
What do you need?
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Biml syntax
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Biml syntax: Root Element
<Biml xmlns="http://schemas.varigence.com/biml.xsd...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Biml syntax: Collections of Elements
<Biml xmlns="http://schemas.varigence.c...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Biml syntax: Elements
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Biml syntax: Attributes
<Biml xmlns="http://schemas.varigence.com/biml.xsd">...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Biml syntax: Full vs. Shorthand Syntax
<Biml xmlns="http://schemas.varigence...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
DEMO
Let's generate
some packages!
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Ok, so we can go from Biml to SSIS…
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
…can we go from SSIS to Biml?
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Yes!
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
DEMO
Let's convert some
SSIS to Biml!
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
The magic is in the…
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
What is BimlScript?
Extend Biml with C# or VB code blocks
Import database st...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
How does it work?
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Yes, but how does it work?
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
BimlScript Code Blocks
<# … #> Control Block (Variables and logic)
<#= … #> ...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
BimlScript Syntax
<# var con = SchemaManager.CreateConnectionNode(...); #>
<...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
BimlScript Syntax: Import metadata
<# var con = SchemaManager.CreateConnecti...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
BimlScript Syntax: Loop over tables
<# var con = SchemaManager.CreateConnect...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
BimlScript Syntax: Replace static values
<# var con = SchemaManager.CreateCo...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Yes, but how does it actually work?
<# foreach (var table in RootNode.Tables...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
PreviewPane
The power is in the…
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
DEMO
Let's generate
many packages!
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
What about SQL?
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
We can generate SSIS…
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
…can we also generate SQL?
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Yes!
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Tiered Biml Files
Multiple Biml Files Working Together
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Tiered Biml Files?
Think of tiers as stacked layers or sequential s...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Tiered Biml Files
Split Biml code in multiple files to:
• Solve logical depe...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Tiered Biml Files: Behind the Scenes
Connections
Load Packages
Master Packag...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Tiered Biml Files: Behind the Scenes
Connections
Databases
Schemas
Tables
Pa...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Da...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Da...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Da...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Da...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Da...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Da...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Da...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Da...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Tiered Biml Files: Behind the Scenes
Connections
Admin
Source
Destination
Da...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
How do you use Tiered Biml Files?
1. Create Biml files with specified tiers
...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Annotations
Store and Pass Metadata Between Biml Files
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Annotations
Store custom metadata by attaching tags to Biml objects:
<Annota...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Annotations
Create annotations:
<OleDbConnection Name="Dest">
<Annotations>
...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
DEMO
Let's put it
all together!
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
New project
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Of course I can create
200 SSIS Packages!
…what do you need me
to do after l...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Biml Basics Tools & Code
SQL and SSIS from Metadata
…the past 60 minutes…
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Get things done
Start small
Start simple
Start with ugly code
Keep going
Exp...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Biml on Monday…
…BimlBreak the rest
of the week!
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
@cathrinew
cathrinew.net
hi@cathrinew.net
Biml resources and demo files:
cat...
Upcoming SlideShare
Loading in …5
×

of

Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 1 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 2 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 3 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 4 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 5 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 6 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 7 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 8 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 9 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 10 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 11 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 12 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 13 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 14 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 15 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 16 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 17 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 18 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 19 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 20 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 21 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 22 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 23 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 24 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 25 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 26 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 27 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 28 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 29 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 30 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 31 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 32 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 33 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 34 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 35 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 36 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 37 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 38 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 39 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 40 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 41 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 42 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 43 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 44 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 45 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 46 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 47 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 48 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 49 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 50 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 51 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 52 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 53 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 54 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 55 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 56 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 57 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 58 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 59 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 60 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 61 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 62 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 63 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 64 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 65 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 66 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 67 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 68 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 69 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 70 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 71 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 72 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 73 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 74 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 75 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 76 Biml for Beginners: Script and Automate SSIS development (Hybrid VC) Slide 77
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

2 Likes

Share

Download to read offline

Biml for Beginners: Script and Automate SSIS development (Hybrid VC)

Download to read offline

Biml for Beginners: Script and Automate SSIS development (Presented for the PASS Hybrid VC on July 30th 2018)

Biml for Beginners: Script and Automate SSIS development (Hybrid VC)

  1. 1. Biml for Beginners: Script and Automate SSIS Development Cathrine Wilhelmsen PASS Hybrid VC · July 30th 2018
  2. 2. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Session Description Are you tired of creating and updating the same SSIS packages again and again? Is your wrist hurting from all that clicking, dragging, dropping, connecting and aligning? Do you want to take the next step and really speed up your SSIS development? Say goodbye to repetitive work and hello to Biml, the markup language for Business Intelligence projects. In this session we will look at the basics of Biml. First learn how to use Biml to generate SSIS packages from database metadata. Then see how you can reuse code to implement changes in multiple SSIS packages and projects with just a few clicks. Finally, we will create an example project that you can download and start with to speed up your SSIS development from day one. Stop wasting your valuable time on doing the same things over and over and over again, and see how you can complete in a day what once took more than a week!
  3. 3. Cathrine Wilhelmsen @cathrinew cathrinew.net Senior BI Consultant, Inmeta
  4. 4. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Biml Basics Tools & Code SQL and SSIS from Metadata …the next 60 minutes…
  5. 5. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
  6. 6. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Once upon a time…
  7. 7. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Meet Sandra, the SSIS developer
  8. 8. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Sandra has to create a new staging project with 100+ SSIS packages
  9. 9. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Sandra gets quite bored of creating the same package over and over and over
  10. 10. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Yay, Sandra has finally completed her project!
  11. 11. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) But Sandra's manager has just a few new requirements…
  12. 12. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Sandra gets to update every single SSIS package one… more… time…
  13. 13. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) …yay
  14. 14. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) It's time for a change!
  15. 15. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
  16. 16. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) What is Biml? Business Intelligence Markup Language Easy to read and write XML language Describes business intelligence objects: • Databases, Schemas, Tables, Views, Columns • SSIS Packages • SSAS and Azure Data Factory *
  17. 17. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Why use Biml? SSIS: Plumbing Biml: Business Logic
  18. 18. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) SSIS: Plumbing Spend time on dragging, dropping, connecting, aligning Create the same package over and over again with minor changes Standards, patterns and templates must be defined up-front Changes must be done in every single package Higher risk of manual errors More packages, more time
  19. 19. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Biml: Business Logic Spend time on what is unique in a package Create a pattern once and reuse for all similar packages Handle scope and requirement changes quickly and easily Changes can be applied to all packages at once Lower risk of manual errors Longer time to start, but then automate and reuse
  20. 20. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Will Biml solve all your challenges? Biml is a tool for generating SSIS packages Biml is not a pre-defined Data Warehouse framework Biml is not a tool for automating deployment
  21. 21. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) How can Biml help you? Timesaving: Many SSIS packages from one Biml file Reusable: Write once and run on any platform Flexible: Start simple, expand as you learn
  22. 22. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) What if you don't have 100+ packages? Source Control: Keep track of actual logic changes Simplify: Untangle complex packages for quick edits SQL Scripts: Use Biml instead of dynamic T-SQL
  23. 23. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) How does SSIS development work?
  24. 24. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) How does Biml development work?
  25. 25. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) What do you need?
  26. 26. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Biml syntax <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  27. 27. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Biml syntax: Root Element <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  28. 28. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Biml syntax: Collections of Elements <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  29. 29. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Biml syntax: Elements <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  30. 30. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Biml syntax: Attributes <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  31. 31. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Biml syntax: Full vs. Shorthand Syntax <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  32. 32. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) DEMO Let's generate some packages!
  33. 33. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Ok, so we can go from Biml to SSIS…
  34. 34. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) …can we go from SSIS to Biml?
  35. 35. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Yes!
  36. 36. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) DEMO Let's convert some SSIS to Biml!
  37. 37. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) The magic is in the…
  38. 38. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) What is BimlScript? Extend Biml with C# or VB code blocks Import database structure and metadata Loop over tables and columns Expressions replace static values Generate, control and manipulate Biml code
  39. 39. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) How does it work?
  40. 40. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Yes, but how does it work?
  41. 41. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) BimlScript Code Blocks <# … #> Control Block (Variables and logic) <#= … #> Text Block (Returns string) <#@ … #> Directive (Compiler instructions)
  42. 42. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) BimlScript Syntax <# var con = SchemaManager.CreateConnectionNode(...); #> <# var metadata = con.GetDatabaseSchema(); #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in metadata.TableNodes) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml>
  43. 43. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) BimlScript Syntax: Import metadata <# var con = SchemaManager.CreateConnectionNode(...); #> <# var metadata = con.GetDatabaseSchema(); #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in metadata.TableNodes) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml>
  44. 44. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) BimlScript Syntax: Loop over tables <# var con = SchemaManager.CreateConnectionNode(...); #> <# var metadata = con.GetDatabaseSchema(); #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in metadata.TableNodes) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml>
  45. 45. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) BimlScript Syntax: Replace static values <# var con = SchemaManager.CreateConnectionNode(...); #> <# var metadata = con.GetDatabaseSchema(); #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in metadata.TableNodes) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml>
  46. 46. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Yes, but how does it actually work? <# foreach (var table in RootNode.Tables) { #> <Package Name="Load_<#=table.Name#>" /> <# } #> <Package Name="Load_Customer" /> <Package Name="Load_Product" /> <Package Name="Load_Sales" />
  47. 47. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) PreviewPane The power is in the…
  48. 48. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) DEMO Let's generate many packages!
  49. 49. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) What about SQL?
  50. 50. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) We can generate SSIS…
  51. 51. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) …can we also generate SQL?
  52. 52. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Yes!
  53. 53. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Tiered Biml Files Multiple Biml Files Working Together
  54. 54. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) What are Tiered Biml Files? Think of tiers as stacked layers or sequential steps Tier (Layer) 1 Tier 0 Tier (Layer) 2 Tier (Step) 1 Tier (Step) 2
  55. 55. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Tiered Biml Files Split Biml code in multiple files to: • Solve logical dependencies • Build solutions in multiple steps Specify the tier per file by using template directive: <#@ template tier="2" #>
  56. 56. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Tiered Biml Files: Behind the Scenes Connections Load Packages Master Package Connections Databases Schemas Tables Packages
  57. 57. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Tiered Biml Files: Behind the Scenes Connections Databases Schemas Tables Packages Connections Load Packages Master Package
  58. 58. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Connections Load Packages Master Package
  59. 59. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Connections Load Packages Master Package
  60. 60. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Connections Load Packages Master Package
  61. 61. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Connections Load Packages Master Package
  62. 62. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Connections Load Packages Master Package
  63. 63. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Connections Load Packages Master Package
  64. 64. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Master Connections Load Packages Master Package
  65. 65. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Master Connections Load Packages Master Package
  66. 66. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Master
  67. 67. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) How do you use Tiered Biml Files? 1. Create Biml files with specified tiers 2. Select all the tiered Biml files 3. Right-click and click Generate SSIS Packages 1 2 3
  68. 68. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Annotations Store and Pass Metadata Between Biml Files
  69. 69. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Annotations Store custom metadata by attaching tags to Biml objects: <Annotation Tag="TagName">Custom Value</Annotation> Higher tier files can get tags from lower tier files
  70. 70. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Annotations Create annotations: <OleDbConnection Name="Dest"> <Annotations> <Annotation Tag="Schema">stg</Annotation> </Annotations> </OleDbConnection> Use annotations: RootNode.OleDbConnections["Dest"].GetTag("Schema");
  71. 71. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) DEMO Let's put it all together!
  72. 72. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) New project
  73. 73. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Of course I can create 200 SSIS Packages! …what do you need me to do after lunch?
  74. 74. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Biml Basics Tools & Code SQL and SSIS from Metadata …the past 60 minutes…
  75. 75. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Get things done Start small Start simple Start with ugly code Keep going Expand Improve Deliver often
  76. 76. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Biml on Monday… …BimlBreak the rest of the week!
  77. 77. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) @cathrinew cathrinew.net hi@cathrinew.net Biml resources and demo files: cathrinew.net/biml
  • RaminZarrasvand

    Feb. 15, 2020
  • MonikaTavas

    Nov. 1, 2019

Biml for Beginners: Script and Automate SSIS development (Presented for the PASS Hybrid VC on July 30th 2018)

Views

Total views

18,030

On Slideshare

0

From embeds

0

Number of embeds

16,721

Actions

Downloads

33

Shares

0

Comments

0

Likes

2

×