Successfully reported this slideshow.
Your SlideShare is downloading. ×

Biml for Beginners: Script and Automate SSIS development (SQLSaturday Oslo)

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 75 Ad

Biml for Beginners: Script and Automate SSIS development (SQLSaturday Oslo)

Download to read offline

Biml for Beginners: Script and Automate SSIS development (Presented at SQLSaturday Oslo on September 1st, 2018)

Biml for Beginners: Script and Automate SSIS development (Presented at SQLSaturday Oslo on September 1st, 2018)

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Similar to Biml for Beginners: Script and Automate SSIS development (SQLSaturday Oslo) (20)

Advertisement

More from Cathrine Wilhelmsen (13)

Recently uploaded (20)

Advertisement

Biml for Beginners: Script and Automate SSIS development (SQLSaturday Oslo)

  1. 1. Biml for Beginners: Script and Automate SSIS Development Cathrine Wilhelmsen SQLSaturday Oslo · September 1st 2018
  2. 2. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) A big thanks to all of our sponsors!
  3. 3. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Basics Tools & Code SQL and SSIS from Metadata …the next 60 minutes…
  4. 4. Cathrine Wilhelmsen @cathrinew cathrinew.net Senior Business Intelligence Consultant, Inmeta
  5. 5. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Once upon a time…
  6. 6. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Meet Sandra, the SSIS developer
  7. 7. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Sandra has to create a new staging project with 100+ SSIS packages
  8. 8. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Sandra gets quite bored of creating the same package over and over and over
  9. 9. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Yay, Sandra has finally completed her project!
  10. 10. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) But Sandra's manager has just a few new requirements…
  11. 11. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Sandra gets to update every single SSIS package one… more… time…
  12. 12. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) …yay
  13. 13. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) It's time for a change!
  14. 14. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
  15. 15. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What is Biml? Business Intelligence Markup Language Easy to read and write XML language Describes business intelligence objects: • Databases, Schemas, Tables, Views, Columns • SSIS Packages • SSAS and Azure Data Factory *
  16. 16. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Why use Biml? SSIS: Plumbing Biml: Business Logic
  17. 17. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Will Biml solve all your challenges? Biml is a tool for generating SSIS packages Biml is not a pre-defined Data Warehouse framework Biml is not a tool for automating deployment
  18. 18. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) How can Biml help you? Timesaving: Many SSIS packages from one Biml file Reusable: Write once and run on any platform Flexible: Start simple, expand as you learn
  19. 19. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What if you don't have 100+ packages? Source Control: Keep track of actual logic changes Simplify: Untangle complex packages for quick edits SQL Scripts: Use Biml instead of dynamic T-SQL
  20. 20. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) How does SSIS development work?
  21. 21. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) How does Biml development work?
  22. 22. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What do you need?
  23. 23. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml syntax <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  24. 24. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml syntax: Root Element <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  25. 25. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml syntax: Collections of Elements <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  26. 26. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml syntax: Elements <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  27. 27. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml syntax: Attributes <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  28. 28. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml syntax: Full vs. Shorthand Syntax <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="EmptyPackage1"></Package> <Package Name="EmptyPackage2"/> </Packages> </Biml>
  29. 29. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO Let's generate some packages!
  30. 30. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Ok, so we can go from Biml to SSIS…
  31. 31. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) …can we go from SSIS to Biml?
  32. 32. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Yes!
  33. 33. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO Let's convert some SSIS to Biml!
  34. 34. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) The magic is in the…
  35. 35. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What is BimlScript? Extend Biml with C# or VB code blocks Import database structure and metadata Loop over tables and columns Expressions replace static values Generate, control and manipulate Biml code
  36. 36. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) How does it work?
  37. 37. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Yes, but how does it work?
  38. 38. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlScript Code Blocks <# … #> Control Block (Variables and logic) <#= … #> Text Block (Returns string) <#@ … #> Directive (Compiler instructions)
  39. 39. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlScript Syntax <# var con = SchemaManager.CreateConnectionNode(...); #> <# var metadata = con.GetDatabaseSchema(); #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in metadata.TableNodes) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml>
  40. 40. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlScript Syntax: Import metadata <# var con = SchemaManager.CreateConnectionNode(...); #> <# var metadata = con.GetDatabaseSchema(); #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in metadata.TableNodes) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml>
  41. 41. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlScript Syntax: Loop over tables <# var con = SchemaManager.CreateConnectionNode(...); #> <# var metadata = con.GetDatabaseSchema(); #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in metadata.TableNodes) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml>
  42. 42. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlScript Syntax: Replace static values <# var con = SchemaManager.CreateConnectionNode(...); #> <# var metadata = con.GetDatabaseSchema(); #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# foreach (var table in metadata.TableNodes) { #> <Package Name="Load_<#=table.Name#>"></Package> <# } #> </Packages> </Biml>
  43. 43. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Yes, but how does it actually work? <# foreach (var table in RootNode.Tables) { #> <Package Name="Load_<#=table.Name#>" /> <# } #> <Package Name="Load_Customer" /> <Package Name="Load_Product" /> <Package Name="Load_Sales" />
  44. 44. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) PreviewPane The power is in the…
  45. 45. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO Let's generate many packages!
  46. 46. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What about SQL?
  47. 47. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) We can generate SSIS…
  48. 48. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) …can we also generate SQL?
  49. 49. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Yes!
  50. 50. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files Multiple Biml Files Working Together
  51. 51. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What are Tiered Biml Files? Think of tiers as stacked layers or sequential steps Tier (Layer) 1 Tier 0 Tier (Layer) 2 Tier (Step) 1 Tier (Step) 2
  52. 52. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files Split Biml code in multiple files to: • Solve logical dependencies • Build solutions in multiple steps Specify the tier per file by using template directive: <#@ template tier="2" #>
  53. 53. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Load Packages Master Package Connections Databases Schemas Tables Packages
  54. 54. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Databases Schemas Tables Packages Connections Load Packages Master Package
  55. 55. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Connections Load Packages Master Package
  56. 56. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Connections Load Packages Master Package
  57. 57. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Connections Load Packages Master Package
  58. 58. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Connections Load Packages Master Package
  59. 59. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Connections Load Packages Master Package
  60. 60. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Connections Load Packages Master Package
  61. 61. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Master Connections Load Packages Master Package
  62. 62. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Master Connections Load Packages Master Package
  63. 63. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Tiered Biml Files: Behind the Scenes Connections Admin Source Destination Databases Schemas Tables Packages Load_Customer Load_Product Load_Sales Master
  64. 64. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Annotations Store and Pass Metadata Between Biml Files
  65. 65. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Annotations Store custom metadata by attaching tags to Biml objects: <Annotation Tag="TagName">Custom Value</Annotation> Higher tier files can get tags from lower tier files
  66. 66. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Annotations Create annotations: <OleDbConnection Name="Dest"> <Annotations> <Annotation Tag="Schema">stg</Annotation> </Annotations> </OleDbConnection> Use annotations: RootNode.OleDbConnections["Dest"].GetTag("Schema");
  67. 67. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO Let's put it all together!
  68. 68. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) New project
  69. 69. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Of course I can create 200 SSIS Packages! …what do you need me to do after lunch?
  70. 70. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Basics Tools & Code SQL and SSIS from Metadata …the past 60 minutes…
  71. 71. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Where can you learn more? Free online training bimlscript.com
  72. 72. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Where can you learn even more? cathrinew.net/BimlBook
  73. 73. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Get things done Start small Start simple Start with ugly code Keep going Expand Improve Deliver often
  74. 74. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml on Monday… …BimlBreak the rest of the week!
  75. 75. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) @cathrinew cathrinew.net hi@cathrinew.net Biml resources and demo files: cathrinew.net/biml

×