Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Biml for Beginners: Speed up your SSIS development (SQLSaturday Chicago)

1,597 views

Published on

Biml for Beginners: Speed up your SSIS development (Presented at SQLSaturday Chicago on March 5th 2016)

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Biml for Beginners: Speed up your SSIS development (SQLSaturday Chicago)

  1. 1. Biml for Beginners: Speed up your SSIS development Cathrine Wilhelmsen · SQLSaturday Chicago · March 5th 2016
  2. 2. Session Description SSIS is a powerful tool for extracting, transforming and loading data, but creating and maintaining a large number of SSIS packages can be both tedious and time-consuming. Even if you use templates and follow best practices you often have to repeat the same steps over and over and over again. Handling metadata and schema changes is a manual process, and there are no easy ways to implement new requirements in multiple packages at the same time. It is time to bring the Don't Repeat Yourself (DRY) software engineering principle to SSIS projects. First learn how to use Biml and BimlScript to generate SSIS packages from database metadata and implement changes in all packages with just a few clicks. Then take the DRY principle one step further and learn how to update all packages in multiple projects by separating and reusing common code. Speed up your SSIS development by using Biml and BimlScript, and see how you can complete in a day what once took more than a week!
  3. 3. The Basics Tools & Projects Code Management
  4. 4. Cathrine Wilhelmsen @cathrinew cathrinewilhelmsen.net Data Warehouse Architect Business Intelligence Developer
  5. 5. SSIS developer? Easily bored? Tired of repetitive work? You… …?
  6. 6. Long development time? Many SSIS packages? Frequent requirement changes? Work… …?
  7. 7. job done! new standards ...yay Ever experienced this?
  8. 8. Ready for a change?
  9. 9. Business Intelligence Markup Language Easy to read and write XML language Describes business intelligence objects: • Databases, Schemas, Tables, Views, Columns • SSIS Packages • SSAS Cubes What is Biml?
  10. 10. From Traditional SSIS to Agile SSIS Traditional SSIS: Plumbing Agile SSIS: Business Logic
  11. 11. Will Biml solve all your problems? Probably not... Biml is a tool for generating SSIS packages Biml is not a pre-defined ETL framework Biml is not a tool for automated deployment ...but it will solve many problems!
  12. 12. How can Biml help you? Timesaving: Many SSIS Packages from one Biml file Reusable: Write once and run on any platform Flexible: Start simple, expand as you learn
  13. 13. What do you need?
  14. 14. …or you can use
  15. 15. …or you can invest in (Mist will be rebranded as BimlStudio)
  16. 16. How does it work? …generated packages look exactly like manually created packages
  17. 17. <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml> Biml syntax
  18. 18. <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml> Biml syntax: Root Element
  19. 19. <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml> Biml syntax: Collections of Root Objects
  20. 20. <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml> Biml syntax: Elements
  21. 21. <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml> Biml syntax: Attributes
  22. 22. <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml> Biml syntax: Full vs. Shorthand Syntax
  23. 23. Let's generate some packages!
  24. 24. Right-click on SSIS project and click Add New Biml File
  25. 25. Biml files are placed in the Miscellaneous folder <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  26. 26. Right-click on the Biml file to Generate SSIS Packages <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  27. 27. From Biml to SSIS <Package Name="TruncateLoad" ConstraintMode="Linear"> <Tasks> <ExecuteSQL Name="Truncate Table" ConnectionName="Staging"> <DirectInput>TRUNCATE TABLE DestinationTable</DirectInput> </ExecuteSQL> <Dataflow Name="Load Table"> <Transformations>...</Transformations> </Dataflow> </Tasks> </Package>
  28. 28. From Biml to SSIS <Transformations> <OleDbSource Name="Source" ConnectionName="AW2014"> <ExternalTableInput Table="SourceTable" /> </OleDbSource> <DerivedColumns Name="Add LoadDate"> <Columns> <Column Name="LoadDate" DataType="DateTime"> @[System::StartTime] </Column> </Columns> </DerivedColumns> <OleDbDestination Name="Destination" ConnectionName="Staging"> <ExternalTableOutput Table="DestinationTable" /> </OleDbDestination> </Transformations>
  29. 29. .biml vs .dtsx: human-readable vs ALL THE CODE! (20% zoom)(150% zoom)
  30. 30. Ok, so we can go from Biml to SSIS…
  31. 31. …can we go from SSIS to Biml?
  32. 32. Yes! 
  33. 33. Let's reverse-engineer some packages!
  34. 34. Package Importer in BimlOnline
  35. 35. Choose a File
  36. 36. Convert from SSIS to Biml
  37. 37. Choose and filter assets
  38. 38. Copy the Biml…
  39. 39. …or Create /Add to BimlOnline Project
  40. 40. The magic is in the
  41. 41. What is BimlScript? Extend Biml with C# or VB code blocks Import database structure and metadata Loop over tables and columns Expressions replace static values Allows you to control and manipulate Biml code
  42. 42. BimlScript Code Nuggets <#@ … #> Directives (Instructions to BimlCompiler) <# … #> Control Nuggets (Control logic) <#= … #> Text Nuggets (Replace nugget with text value) <#+ … #> Class Nuggets (Create helper classes and methods)
  43. 43. BimlScript Syntax <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in RootNode.Tables) { #> <Package Name="Load<#=table.Name#>"></Package> <# } #> </Packages> </Biml>
  44. 44. BimlScript Syntax: Control Nuggets <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in RootNode.Tables) { #> <Package Name="Load<#=table.Name#>"></Package> <# } #> </Packages> </Biml>
  45. 45. BimlScript Syntax: Text Nuggets <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in RootNode.Tables) { #> <Package Name="Load<#=table.Name#>"></Package> <# } #> </Packages> </Biml>
  46. 46. How does it work?
  47. 47. Yes, but how does it work?
  48. 48. Yes, but how does it actually work? <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in RootNode.Tables) { #> <Package Name="Load<#=table.Name#>"></Package> <# } #> </Packages> </Biml> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="LoadCustomer"></Package> <Package Name="LoadProduct"></Package> <Package Name="LoadSales"></Package> </Packages> </Biml>
  49. 49. Biml vs. BimlScript Automate, control and manipulate Biml with C# Flat XML "Just text"
  50. 50. Let's generate a lot of packages!
  51. 51. Don't Repeat Yourself
  52. 52. Move common code to separate files Centralize and reuse in many projects Update code once for all projects 1. Tiered Biml files 2. Include files 3. CallBimlScript with parameters Don't Repeat Yourself
  53. 53. BIDS Helper vs. BimlOnline / Mist "Black Box" Only SSIS packages visible Visual Editors All in-memory objects visible
  54. 54. Tiered Biml Files Use the template directive: <#@ template tier="1" #> Create objects in-memory from lowest to highest tier to: • Solve logical dependencies • Simulate manual workflows In-memory objects are added to the RootNode Get objects from RootNode in higher tiers
  55. 55. Inside the Black Box: Tiered Biml Files Tier 1: Create database connections Add connections to RootNode Tier 2: Create loading packages Use connections from RootNode Add packages to RootNode Tier 3: Create master package to execute packages Use packages from RootNode Add package to RootNode
  56. 56. Inside the Black Box: Tiered Biml Files <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  57. 57. Inside the Black Box: Tiered Biml Files <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  58. 58. Inside the Black Box: Tiered Biml Files <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  59. 59. Inside the Black Box: Tiered Biml Files <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  60. 60. Inside the Black Box: Tiered Biml Files <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  61. 61. Inside the Black Box: Tiered Biml Files <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  62. 62. Inside the Black Box: Tiered Biml Files <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  63. 63. Inside the Black Box: Tiered Biml Files <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  64. 64. Inside the Black Box: Tiered Biml Files <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  65. 65. Inside the Black Box: Tiered Biml Files <#@ template tier="1" #> <Connections>...</Connections> <#@ template tier="2" #> <Packages>...</Packages> <#@ template tier="3" #> <Package>...</Package>
  66. 66. Inside the Black Box: Tiered Biml Files
  67. 67. 1. Create Biml files with specified tiers 2. Select all the tiered Biml files 3. Right-click and click Generate SSIS Packages How do you use Tiered Biml files? 1 2 3
  68. 68. Include Files Include common code in multiple files and projects Can include many file types: .biml .txt .sql .cs Use the include directive <#@ include file="CommonCode.biml" #> The directive will be replaced by the included file Include pulls code from the included file into the main file
  69. 69. Include Files
  70. 70. Include Files
  71. 71. Include Files
  72. 72. CallBimlScript with Parameters Works like a parameterized include File to be called (callee) specifies input parameters it accepts <#@ property name="Table" type="AstTableNode" #> File that calls (caller) passes input parameters <#=CallBimlScript("CommonCode.biml", Table)#> CallBimlScript pushes parameters from the caller to the callee, and the callee returns code
  73. 73. CallBimlScript with Parameters
  74. 74. CallBimlScript with Parameters
  75. 75. CallBimlScript with Parameters logic based on parameters
  76. 76. CallBimlScript with Parameters
  77. 77. CallBimlScript with Parameters
  78. 78. How does this actually work?
  79. 79. Get things done Start small Start simple Start with ugly code Keep going Expand Improve Deliver often
  80. 80. …BimlBreak the rest of the week  Biml on Monday...
  81. 81. @cathrinew cathrinewilhelmsen.net no.linkedin.com/in/cathrinewilhelmsen contact@cathrinewilhelmsen.net slideshare.net/cathrinewilhelmsen Biml resources and references: cathrinewilhelmsen.net/biml

×