Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Biml for Beginners: Script and Automate SSIS development (SQLSaturday Oslo)

410 views

Published on

Biml for Beginners: Script and Automate SSIS development (Presented at SQLSaturday Oslo on September 1st, 2018)

Published in: Data & Analytics
  • HOW TO UNLOCK HER LEGS! (SNEAK PEAK), learn more... ♥♥♥ http://t.cn/AijLRbnO
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • My special guest's 3-Step "No Product Funnel" can be duplicated to start earning a significant income online. ♥♥♥ https://bit.ly/2kS5a5J
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Legitimate jobs paying $40/h Tap into the booming online job, industry and start working now! ★★★ http://ishbv.com/ezpayjobs/pdf
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Biml for Beginners: Script and Automate SSIS development (SQLSaturday Oslo)

  1. 1. Biml for Beginners: Script and Automate SSIS Development Cathrine Wilhelmsen SQLSaturday Oslo · September 1st 2018
  2. 2. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) A big thanks to all of our sponsors!
  3. 3. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Basics Tools & Code SQL and SSIS from Metadata …the next 60 minutes…
  4. 4. Cathrine Wilhelmsen @cathrinew cathrinew.net Senior Business Intelligence Consultant, Inmeta
  5. 5. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Once upon a time…
  6. 6. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Meet Sandra, the SSIS developer
  7. 7. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Sandra has to create a new staging project with 100+ SSIS packages
  8. 8. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Sandra gets quite bored of creating the same package over and over and over
  9. 9. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Yay, Sandra has finally completed her project!
  10. 10. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) But Sandra's manager has just a few new requirements…
  11. 11. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Sandra gets to update every single SSIS package one… more… time…
  12. 12. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) …yay
  13. 13. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) It's time for a change!
  14. 14. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
  15. 15. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What is Biml? Business Intelligence Markup Language Easy to read and write XML language Describes business intelligence objects: • Databases, Schemas, Tables, Views, Columns • SSIS Packages • SSAS and Azure Data Factory *
  16. 16. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Why use Biml? SSIS: Plumbing Biml: Business Logic
  17. 17. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Will Biml solve all your challenges? Biml is a tool for generating SSIS packages Biml is not a pre-defined Data Warehouse framework Biml is not a tool for automating deployment
  18. 18. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) How can Biml help you? Timesaving: Many SSIS packages from one Biml file Reusable: Write once and run on any platform Flexible: Start simple, expand as you learn
  19. 19. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What if you don't have 100+ packages? Source Control: Keep track of actual logic changes Simplify: Untangle complex packages for quick edits SQL Scripts: Use Biml instead of dynamic T-SQL
  20. 20. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) How does SSIS development work?
  21. 21. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) How does Biml development work?
  22. 22. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What do you need?
  23. 23. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml syntax <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  24. 24. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml syntax: Root Element <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  25. 25. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml syntax: Collections of Elements <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  26. 26. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml syntax: Elements <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  27. 27. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml syntax: Attributes <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  28. 28. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml syntax: Full vs. Shorthand Syntax <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  29. 29. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO Let's generate some packages!
  30. 30. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Ok, so we can go from Biml to SSIS…
  31. 31. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) …can we go from SSIS to Biml?
  32. 32. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Yes!
  33. 33. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO Let's convert some SSIS to Biml!
  34. 34. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) The magic is in the…
  35. 35. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What is BimlScript? Extend Biml with C# or VB code blocks Import database structure and metadata Loop over tables and columns Expressions replace static values Generate, control and manipulate Biml code
  36. 36. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) How does it work?
  37. 37. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Yes, but how does it work?
  38. 38. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlScript Code Blocks <# … #> Control Block (Variables and logic) <#= … #> Text Block (Returns string) <#@ … #> Directive (Compiler instructions)
  39. 39. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlScript Syntax <# var con = SchemaManager.CreateConnectionNode(...); #> <# var metadata = con.GetDatabaseSchema(); #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in metadata.TableNodes) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml>
  40. 40. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlScript Syntax: Import metadata <# var con = SchemaManager.CreateConnectionNode(...); #> <# var metadata = con.GetDatabaseSchema(); #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in metadata.TableNodes) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml>
  41. 41. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlScript Syntax: Loop over tables <# var con = SchemaManager.CreateConnectionNode(...); #> <# var metadata = con.GetDatabaseSchema(); #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in metadata.TableNodes) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml>
  42. 42. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlScript Syntax: Replace static values <# var con = SchemaManager.CreateConnectionNode(...); #> <# var metadata = con.GetDatabaseSchema(); #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in metadata.TableNodes) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml>
  43. 43. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Yes, but how does it actually work? <# foreach (var table in RootNode.Tables) { #> <Package Name="Load_<#=table.Name#>" /> <# } #> <Package Name="Load_Customer" /> <Package Name="Load_Product" /> <Package Name="Load_Sales" />
  44. 44. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) PreviewPane The power is in the…
  45. 45. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO Let's generate many packages!
  46. 46. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What about SQL?
  47. 47. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) We can generate SSIS…
  48. 48. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) …can we also generate SQL?
  49. 49. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Yes!
  50. 50. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files Multiple Biml Files Working Together
  51. 51. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What are Tiered Biml Files? Think of tiers as stacked layers or sequential steps Tier (Layer) 1 Tier 0 Tier (Layer) 2 Tier (Step) 1 Tier (Step) 2
  52. 52. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files Split Biml code in multiple files to: • Solve logical dependencies • Build solutions in multiple steps Specify the tier per file by using template directive: <#@ template tier="2" #>
  53. 53. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Load Packages Master Package Connections Databases Schemas Tables Packages
  54. 54. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Databases Schemas Tables Packages Connections Load Packages Master Package
  55. 55. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Connections Load Packages Master Package
  56. 56. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Connections Load Packages Master Package
  57. 57. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Connections Load Packages Master Package
  58. 58. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Connections Load Packages Master Package
  59. 59. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Connections Load Packages Master Package
  60. 60. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Connections Load Packages Master Package
  61. 61. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Master Connections Load Packages Master Package
  62. 62. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Master Connections Load Packages Master Package
  63. 63. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Master
  64. 64. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Annotations Store and Pass Metadata Between Biml Files
  65. 65. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Annotations Store custom metadata by attaching tags to Biml objects: <Annotation Tag="TagName">Custom Value</Annotation> Higher tier files can get tags from lower tier files
  66. 66. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Annotations Create annotations: <OleDbConnection Name="Dest"> <Annotations> <Annotation Tag="Schema">stg</Annotation> </Annotations> </OleDbConnection> Use annotations: RootNode.OleDbConnections["Dest"].GetTag("Schema");
  67. 67. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO Let's put it all together!
  68. 68. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) New project
  69. 69. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Of course I can create 200 SSIS Packages! …what do you need me to do after lunch?
  70. 70. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Basics Tools & Code SQL and SSIS from Metadata …the past 60 minutes…
  71. 71. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Where can you learn more? Free online training bimlscript.com
  72. 72. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Where can you learn even more? cathrinew.net/BimlBook
  73. 73. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Get things done Start small Start simple Start with ugly code Keep going Expand Improve Deliver often
  74. 74. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml on Monday… …BimlBreak the rest of the week!
  75. 75. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) @cathrinew cathrinew.net hi@cathrinew.net Biml resources and demo files: cathrinew.net/biml

×