Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Biml for Beginners: Script and Automate SSIS development (SQLSaturday Finland)

1,107 views

Published on

Biml for Beginners: Script and Automate SSIS development (Presented at SQLSaturday Finland on May 12th 2018)

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Biml for Beginners: Script and Automate SSIS development (SQLSaturday Finland)

  1. 1. Biml for Beginners: Script and Automate SSIS Development Cathrine Wilhelmsen SQLSaturday Finland · May 12th 2018
  2. 2. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Session Description Are you tired of creating and updating the same SSIS packages again and again? Is your wrist hurting from all that clicking, dragging, dropping, connecting and aligning? Do you want to take the next step and really speed up your SSIS development? Say goodbye to repetitive work and hello to Biml, the markup language for Business Intelligence projects. In this session we will look at the basics of Biml. First learn how to use Biml to generate SSIS packages from database metadata. Then see how you can reuse code to implement changes in multiple SSIS packages and projects with just a few clicks. Finally, we will create an example project that you can download and start with to speed up your SSIS development from day one. Stop wasting your valuable time on doing the same things over and over and over again, and see how you can complete in a day what once took more than a week!
  3. 3. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Basics Tools & Code SQL and SSIS from Metadata …the next 60 minutes…
  4. 4. Cathrine Wilhelmsen @cathrinew cathrinew.net Data Warehouse & Business Intelligence Consultant
  5. 5. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) You… …? SSIS developer Easily bored Tired of repetitive work
  6. 6. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Work… …? Long development time Many SSIS packages Frequent requirement changes
  7. 7. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) job done! new standards …yay Ever experienced this?
  8. 8. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) It's time for a change!
  9. 9. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
  10. 10. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What is Biml? Business Intelligence Markup Language Easy to read and write XML language Describes business intelligence objects: • Databases, Schemas, Tables, Views, Columns • SSIS Packages • SSAS and Azure Data Factory *
  11. 11. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Why use Biml? SSIS: Plumbing Biml: Business Logic
  12. 12. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) SSIS: Plumbing Spend time on dragging, dropping, connecting, aligning Create the same package over and over again with minor changes Standards, patterns and templates must be defined up-front Changes must be done in every single package Higher risk of manual errors More packages, more time
  13. 13. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml: Business Logic Spend time on what is unique in a package Create a pattern once and reuse for all similar packages Handle scope and requirement changes quickly and easily Changes can be applied to all packages at once Lower risk of manual errors Longer time to start, but then automate and reuse
  14. 14. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Will Biml solve all your challenges? Biml is a tool for generating SSIS packages Biml is not a pre-defined Data Warehouse framework Biml is not a tool for automating deployment
  15. 15. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) How can Biml help you? Timesaving: Many SSIS packages from one Biml file Reusable: Write once and run on any platform Flexible: Start simple, expand as you learn
  16. 16. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What if you don't have 100+ packages? Source Control: Keep track of actual logic changes Simplify: Untangle complex packages for quick edits SQL Scripts: Use Biml instead of dynamic T-SQL
  17. 17. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) How does SSIS development work?
  18. 18. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) How does Biml development work?
  19. 19. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What do you need?
  20. 20. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlExpress Free add-in for Visual Studio Code editor with syntax highlighting and Biml Intellisense More frequent updates than BIDS Helper varigence.com/bimlexpress
  21. 21. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlStudio Licensed full-featured development environment for Biml Visual designer and metadata modeling Full-stack automation and transformers varigence.com/bimlstudio
  22. 22. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml syntax <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  23. 23. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml syntax: Root Element <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  24. 24. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml syntax: Collections of Elements <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  25. 25. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml syntax: Elements <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  26. 26. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml syntax: Attributes <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  27. 27. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml syntax: Full vs. Shorthand Syntax <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  28. 28. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO Let's generate some packages!
  29. 29. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Add New Biml File from BimlExpress menu…
  30. 30. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) …or right-click on SSIS project to Add New Biml File
  31. 31. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml files are placed under Miscellaneous
  32. 32. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Check Biml For Errors from BimlExpress menu…
  33. 33. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) …or right-click on file to Check Biml For Errors
  34. 34. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Generate SSIS Packages from BimlExpress menu…
  35. 35. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) …or right-click on file to Generate SSIS Packages
  36. 36. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) From Biml to SSIS: Control Flow <Package Name="TruncateLoad" ConstraintMode="Linear"> <Tasks> <ExecuteSQL Name="Truncate Table" ConnectionName="Staging"> <DirectInput>TRUNCATE TABLE DestinationTable</DirectInput> </ExecuteSQL> <Dataflow Name="Load Table"> <Transformations>...</Transformations> </Dataflow> </Tasks> </Package>
  37. 37. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) From Biml to SSIS: Data Flow <Transformations> <OleDbSource Name="Source" ConnectionName="Source"> <ExternalTableInput Table="SourceTable" /> </OleDbSource> <DerivedColumns Name="Add LoadDate"> <Columns> <Column Name="LoadDate" DataType="DateTime"> @[System::StartTime] </Column> </Columns> </DerivedColumns> <OleDbDestination Name="Destination" ConnectionName="Staging"> <ExternalTableOutput Table="DestinationTable" /> </OleDbDestination> </Transformations>
  38. 38. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Ok, so we can go from Biml to SSIS…
  39. 39. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) …can we go from SSIS to Biml?
  40. 40. Yes!
  41. 41. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO Let's convert some SSIS to Biml!
  42. 42. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Convert SSIS to Biml in BimlExpress 2018
  43. 43. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Select Package or project…
  44. 44. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) …import, filter and add to project
  45. 45. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) The magic is in the…
  46. 46. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What is BimlScript? Extend Biml with C# or VB code blocks Import database structure and metadata Loop over tables and columns Expressions replace static values Generate, control and manipulate Biml code
  47. 47. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlScript Code Blocks <# … #> Control Block (Variables and logic) <#= … #> Text Block (Returns string) <#@ … #> Directive (Compiler instructions) <#+ … #> Class Block (C# classes and methods) <#* … *#> Comment Block (Disable BimlScript)
  48. 48. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlScript Syntax <# var con = SchemaManager.CreateConnectionNode(...); #> <# var metadata = con.GetDatabaseSchema(); #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in metadata.TableNodes) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml>
  49. 49. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlScript Syntax: Import metadata <# var con = SchemaManager.CreateConnectionNode(...); #> <# var metadata = con.GetDatabaseSchema(); #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in metadata.TableNodes) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml>
  50. 50. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlScript Syntax: Loop over tables <# var con = SchemaManager.CreateConnectionNode(...); #> <# var metadata = con.GetDatabaseSchema(); #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in metadata.TableNodes) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml>
  51. 51. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlScript Syntax: Replace static values <# var con = SchemaManager.CreateConnectionNode(...); #> <# var metadata = con.GetDatabaseSchema(); #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in metadata.TableNodes) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml>
  52. 52. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml vs. BimlScript Generate, control and manipulate Biml with C# XML Language "Just plain text"
  53. 53. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) How does it work?
  54. 54. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Yes, but how does it work?
  55. 55. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Yes, but how does it actually work? <# foreach (var table in RootNode.Tables) { #> <Package Name="Load_<#=table.Name#>" /> <# } #> <Package Name="Load_Customer" /> <Package Name="Load_Product" /> <Package Name="Load_Sales" />
  56. 56. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) PreviewPane The power is in the…
  57. 57. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlExpress Preview Pane
  58. 58. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO Let's generate many packages!
  59. 59. Of course I can create 200 SSIS Packages! …what do you need me to do after lunch?
  60. 60. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Don't Repeat Yourself Centralize and reuse code Update once in one file 1. Tiered Biml Files 2. Include Files 3. CallBimlScript
  61. 61. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files Multiple Biml Files Working Together
  62. 62. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What are Tiered Biml Files? Think of tiers as stacked layers or sequential steps Tier (Layer) 1 Tier 0 Tier (Layer) 2 Tier (Step) 1 Tier (Step) 2
  63. 63. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files Split Biml code in multiple files to: • Solve logical dependencies • Build solutions in multiple steps behind the scenes Specify the tier per file by using the template directive: <#@ template tier="2" #>
  64. 64. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files Biml is compiled step-by-step from lowest to highest tier • Biml files are implicitly tier 0 • BimlScript files are implicitly tier 1 In each tier, objects are added in-memory to RootNode Higher tiers can call RootNode to use objects from lower tiers
  65. 65. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Load Packages Master Package Connections Databases Schemas Tables Packages
  66. 66. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Databases Schemas Tables Packages Connections Load Packages Master Package
  67. 67. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Connections Load Packages Master Package
  68. 68. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Connections Load Packages Master Package
  69. 69. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Connections Load Packages Master Package
  70. 70. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Connections Load Packages Master Package
  71. 71. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Connections Load Packages Master Package
  72. 72. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Connections Load Packages Master Package
  73. 73. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Master Connections Load Packages Master Package
  74. 74. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Master Connections Load Packages Master Package
  75. 75. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Master
  76. 76. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) How do you use Tiered Biml Files? 1. Create Biml files with specified tiers 2. Select all the tiered Biml files 3. Right-click and click Generate SSIS Packages 1 2 3
  77. 77. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Annotations Store and Pass Metadata Between Biml Files
  78. 78. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Annotations Store custom metadata by attaching tags to Biml objects: <Annotation Tag="TagName">Custom Value</Annotation> Higher tier files can get tags from lower tier files
  79. 79. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Annotations Create annotations: <OleDbConnection Name="Dest"> <Annotations> <Annotation Tag="Schema">stg</Annotation> </Annotations> </OleDbConnection> Use annotations: RootNode.OleDbConnections["Dest"].GetTag("Schema");
  80. 80. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Include Files Automated Copy & Paste
  81. 81. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Include Files Include common code in multiple files and projects Can include many file types: .biml .txt .sql .cs Use the include directive <#@ include file="CommonCode.biml" #> The directive will be replaced by the included file Works like an automated Copy & Paste
  82. 82. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Include Files
  83. 83. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Include Files
  84. 84. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Include Files
  85. 85. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) CallBimlScript Parameterized Control over Returned Code
  86. 86. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) CallBimlScript Control and limit the code returned Works like a parameterized include (or stored procedure) CallBimlScript file specifies accepted parameters: <#@ property name="Parameter" type="String" #> Main file calls and passes parameters: <#=CallBimlScript("CommonCode.biml", Parameter)#>
  87. 87. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) CallBimlScript
  88. 88. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) CallBimlScript
  89. 89. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) CallBimlScript
  90. 90. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) CallBimlScript
  91. 91. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) CallBimlScript
  92. 92. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Wait! When do you use what? In larger projects, you often see a combination of Tiered Biml Files, Include Files, and CallBimlScript Rule of Thumb: If you reuse code more than 3 times, refactor so you Don't Repeat Yourself
  93. 93. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Include Files Automated Copy & Paste Code Included As-Is CallBimlScript Parameterized Control Returned Code Tiered Biml Files Solve Dependencies Multi-Step Builds
  94. 94. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO Let's put it all together!
  95. 95. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Basics Tools & Code SQL and SSIS from Metadata …the past 60 minutes…
  96. 96. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Where can you learn more? Free online training bimlscript.com
  97. 97. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Where can you learn even more? cathrinew.net/BimlBook
  98. 98. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Get things done Start small Start simple Start with ugly code Keep going Expand Improve Deliver often
  99. 99. Biml on Monday… …BimlBreak the rest of the week!
  100. 100. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) @cathrinew cathrinew.net hi@cathrinew.net Biml resources and demo files: cathrinew.net/biml

×