Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018)

905 views

Published on

Biml Tips and Tricks: Not Just for SSIS Packages! (Presented at SQLGrillen on June 22nd, 2018)

Published in: Data & Analytics
  • Be the first to comment

Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018)

  1. 1. Biml Tips and Tricks Not just for SSIS Packages! Cathrine Wilhelmsen SQLGrillen · June 22nd 2018
  2. 2. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
  3. 3. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Session Description "Wait, what? Biml is not just for generating SSIS packages?" Absolutely not! Come and see how you can use Biml (Business Intelligence Markup Language) to save time and speed up other Data Warehouse development tasks. You can generate complex T-SQL statements with Biml instead of using dynamic SQL, create test data and populate static dimensions, and even compare tables and views across multiple servers and databases. Don't Repeat Yourself, start automating those boring, manual tasks today!
  4. 4. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Cathrine Wilhelmsen @cathrinew cathrinew.net Data Warehouse & Business Intelligence Consultant
  5. 5. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) …the next 60 minutes… T-SQL Test Data Comparing Dimensions
  6. 6. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
  7. 7. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Wait… Can't I use <something else>?
  8. 8. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Of course! But Biml works well with source metadata
  9. 9. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) And… Biml is fun!
  10. 10. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) But… Biml? How?
  11. 11. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Traditional Biml
  12. 12. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Crazy Fun Biml
  13. 13. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What do you need?
  14. 14. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Preview Pane The power is in the…
  15. 15. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Traditional BimlScript
  16. 16. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Traditional BimlScript <# foreach (var table in RootNode.Tables) { #> <Package Name="Load_<#=table.Name#>" /> <# } #> <Package Name="Load_Customer" /> <Package Name="Load_Product" /> <Package Name="Load_Sales" />
  17. 17. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlScript to Biml Preview Pane
  18. 18. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) BimlScript to... ??? Preview Pane
  19. 19. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Crazy Fun BimlScript <# foreach (var table in RootNode.Tables) { #> Yay, a package! <#=table.Name#> :) <# } #> Yay, a package! Load_Customer :) Yay, a package! Load_Product :) Yay, a package! Load_Sales :)
  20. 20. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net)
  21. 21. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) T-SQL from Biml
  22. 22. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Built-in T-SQL from Biml
  23. 23. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Built-in T-SQL from Biml Functional T-SQL GetSelectSql GetDropAndCreateDdl GetTableSql SSMS Snippets GetInsertSql GetUpdateSql GetDeleteSql
  24. 24. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Functional T-SQL: GetSelectSql SELECT [Column1] ,[Column2] FROM [Schema].[Table]
  25. 25. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Functional T-SQL: GetDropAndCreateDdl IF EXISTS (...) DROP TABLE [Schema].[Table] GO CREATE TABLE [Schema].[Table] ( [Column1] int IDENTITY(1,1) NOT NULL ,[Column2] nvarchar(50) NOT NULL -- Constraints )
  26. 26. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Functional T-SQL: GetTableSql IF EXISTS (...) DROP TABLE [Schema].[Table] GO CREATE TABLE [Schema].[Table] ( [Column1] int IDENTITY(1,1) NOT NULL ,[Column2] nvarchar(50) NOT NULL -- Constraints )
  27. 27. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Functional T-SQL Use directly in SSIS Copy and run in SSMS
  28. 28. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) SSMS Snippets: GetDeleteSql DELETE FROM [Schema].[Table] WHERE <Search Conditions,,>
  29. 29. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) SSMS Snippets: GetInsertSql INSERT INTO [Schema].[Table] ( [Column1] ,[Column1] ) VALUES ( <Column1,int,>, <Column2,nvarchar(50),> )
  30. 30. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) SSMS Snippets: GetUpdateSql UPDATE [Schema].[Table] SET [Column1] = <Column1, int,>, [Column2] = <Column2, nvarchar(50),> WHERE <Search Conditions,,>
  31. 31. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Copy and run in SSMS Replace Template Parameters with actual values CTRL SHIFT M SSMS Snippets
  32. 32. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Custom T-SQL from Biml
  33. 33. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Custom T-SQL from Biml Table Metadata <#=table.Name#> <#=table.SchemaQualifiedName#> SELECT *
  34. 34. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Custom T-SQL from Biml Column Methods GetColumnList GetColumnAssignmentList GetColumnComparisonList
  35. 35. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Column Methods Return code fragments Use as building blocks in custom T-SQL Filter columns by using lambda expressions
  36. 36. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Column Methods Return code fragments Use as building blocks in custom T-SQL Filter columns by using lambda expressions
  37. 37. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions "A lambda expression is an anonymous function that you can use to create delegates or expression tree types"
  38. 38. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions "A lambda expression is an anonymous function that you can use to create delegates or expression tree types"
  39. 39. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions column => column.Name == "ColumnID"
  40. 40. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions column => column.Name == "ColumnID" The arrow is the lambda operator
  41. 41. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions column => column.Name == "ColumnID" Input parameter is on the left side
  42. 42. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions column => column.Name == "ColumnID" Expression is on the right side
  43. 43. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Lambda Expressions column => column.Name == "ColumnID"
  44. 44. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnList Get columns for SELECT [Column1], [Column2], [Column3]
  45. 45. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnList GetColumnList() GetColumnList("src") GetColumnList(c => c.Name == "ColumnID") GetColumnList(c => c.DataType != DbType.Guid, "src")
  46. 46. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnAssignmentList Get columns for UPDATE … SET [l].[Column1] = [r].[Column1] ,[l].[Column2] = [r].[Column2] ,[l].[Column3] = [r].[Column3]
  47. 47. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnAssignmentList GetColumnAssignmentList() GetColumnAssignmentList("dst", "src") GetColumnAssignmentList(c => !c.IsUsedInPrimaryKey) GetColumnAssignmentList(c => !c.IsUsedInPrimaryKey, "dst", "src")
  48. 48. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnComparisonList Get columns for JOIN … ON [l].[Column1] = [r].[Column1] AND [l].[Column2] = [r].[Column2] AND [l].[Column3] = [r].[Column3]
  49. 49. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) GetColumnComparisonList GetColumnComparisonList() GetColumnComparisonList("!=") GetColumnComparisonList("dst", "src") GetColumnComparisonList(c => c.IsUsedInPrimaryKey) GetColumnComparisonList(c => c.IsUsedInPrimaryKey, "dst", "src") GetColumnComparisonList(c => c.IsUsedInPrimaryKey, "!=", "dst", "src")
  50. 50. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO T-SQL from Biml
  51. 51. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Test Data
  52. 52. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml + Random() Create Random Test Data
  53. 53. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Random Class Random() Pseudo-random number generator Random enough for practical purposes Random(12345) Always generates the same series of random numbers
  54. 54. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Random Methods Next() Random number Next(100) Random number smaller than specified value Next(1000, 2000) Random number within specified range NextDouble() Random decimal number between 0.0 and 1.0
  55. 55. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Helper Method <#+ Random r = new Random(); public string GetTestData(AstTableColumnBaseNode column) { switch (column.DataType) { case DbType.Int32 : return r.Next(Int32.MaxValue).ToString(); case DbType.String : return "'" + GetRandomString(column.Length) + "'" default : return "NULL"; } } #>
  56. 56. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Helper Helper Method <#+ private string GetRandomString(int length) { var allowedChars = "ABC123"; var randomChars = new char[length]; for (int i = 0; i < length; i++) { randomChars[i] = allowedChars[ r.Next(allowedChars.Length) ]; } return new String(randomCharacters); } #>
  57. 57. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml + Random() Very limited functionality No dependencies Works out-of-the box
  58. 58. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml + Bogus Create Random and Specific Test Data
  59. 59. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Bogus Project Simple and sane fake data generator for .NET https://github.com/bchavez/Bogus Advanced functionality for custom objects …or simple functionality for Biml hacks :)
  60. 60. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Address Commerce Company Database Date Finance Internet Lorem Name Person Phone System Bogus API Supports all data types plus built-in test data, including:
  61. 61. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) The Bogus Methods <#@ assembly name="Bogus.dll" #> <#@ import namespace="Bogus" #> <# var f = new Faker(); #> <# Person p = new Person(); #> INSERT INTO dbo.Employee(FirstName, LastName, Birthday) VALUES ( '<#=p.FirstName#>', '<#=p.LastName#>', '<#=p.DateOfBirth.ToString("yyyyMMdd")#>' )
  62. 62. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml + Bogus() Better functionality than Random() Easier to understand, frequently updated Requires additional files
  63. 63. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Installing Bogus Install via Nuget: Install-Package Bogus Or download latest release .zip: https://github.com/bchavez/Bogus/releases
  64. 64. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO Test Data with Random() and Bogus
  65. 65. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Alternative (Better) Solutions Mockaroo (free, but only 1000 rows) https://mockaroo.com/ Redgate SQL Data Generator (licensed) https://www.red-gate.com/products/sql- development/sql-data-generator/index
  66. 66. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Static Dimensions
  67. 67. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Static Dimensions (or any kind of static table) SCD Type 0: Not changing Lookup Tables 1. Create table definition 2. Add static source rows 3. Generate INSERT statements
  68. 68. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Static Sources Part of <Table> objects Defines rows to be inserted when table is created • Unknown dimension members • Date dimension • Code tables
  69. 69. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Static Source <Table Name="Table" SchemaName="Database.Schema"> <Columns> <Column Name="Column1" DataType="Int32" /> <Column Name="Column2" DataType="String" Length="10" /> </Columns> <Sources> <StaticSource Name="TableRows"> <Rows> <Row> <ColumnValues> <ColumnValue ColumnName="Column1" Value="-1" /> <ColumnValue ColumnName="Column2" Value="'Unknown'" /> </ColumnValues> </Row> </Rows> </StaticSource> </Sources> </Table>
  70. 70. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Biml Static Source <# if (table.Sources.Any()) { #> <#=TableToPackageLowerer.GetStaticSourceInsertStatements( table.SchemaQualifiedName, table.HasIdentity, (AstTableStaticSourceNode)table.Sources.FirstOrDefault() )#> <# } #>
  71. 71. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Drop and Create Facts and Dimensions
  72. 72. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Could not drop object 'Schema.Table' because it is referenced by a FOREIGN KEY constraint.
  73. 73. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Drop and Create Create Order: Dim1 Dim2 Fact1 Fact2 Fact2 Fact1 Dim2 Dim1Drop Order:
  74. 74. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Drop and Create Create Order: Dim1 Dim2 Fact1 Fact2 Fact2 Fact1 Dim2 Dim1Drop Order:
  75. 75. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Topological Sort "A topological sort of a directed graph is a linear ordering of its vertices such that for every directed edge uv from vertex u to vertex v, u comes before v in the ordering"
  76. 76. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Topological Sort "A topological sort of a directed graph is a linear ordering of its vertices such that for every directed edge uv from vertex u to vertex v, u comes before v in the ordering"
  77. 77. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Topological Sort "Sorts objects by dependencies. For example dimensions before fact tables."
  78. 78. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Topological Sort "Sorts objects by dependencies. For example dimensions before fact tables."
  79. 79. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Topological Sort IEnumerable<T> DependencyAnalysis.TopologicalSort<T> ( IEnumerable<T> source, Expression<Func<T, T>> relationProperty )
  80. 80. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Topological Sort var dependentTables = DependencyAnalysis.TopologicalSort<AstTableNode> ( RootNode.Tables, t => t.Columns .OfType<AstTableColumnTableReferenceNode>() .Select(c => c.ForeignTable) );
  81. 81. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO Static Dimensions and Topological Sort
  82. 82. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Comparing Databases
  83. 83. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Once upon a time…
  84. 84. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Panic!
  85. 85. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) How to Compare Tables and Columns
  86. 86. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) LINQ Sort OrderBy, ThenBy Filter Where, OfType Group GroupBy Aggregate Count, Sum Check Collections All, Any, Contains Set Operations Except, Intersect Project Collections Select, SelectMany
  87. 87. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) LINQ Sort OrderBy, ThenBy Filter Where, OfType Group GroupBy Aggregate Count, Sum Check Collections All, Any, Contains Set Operations Except, Intersect Project Collections Select, SelectMany
  88. 88. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) LINQ: Filter Collections Where() Returns the filtered collection with all elements that meet the criteria table.Where(t => t.Schema.Name == "dim")
  89. 89. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) LINQ: Set Operations Except() Returns elements in first collection that are not in second collection table1.Except(table2)
  90. 90. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) LINQ: Set Operations Intersect() Returns elements in first collection that are also in second collection table1.Intersect(table2)
  91. 91. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) LINQ: Project Collections Select() Creates a new collection from specified attributes table.Select(t => t.Name)
  92. 92. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) DEMO Comparing Databases
  93. 93. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Alternative (Better) Solutions Redgate SQL Compare (licensed) https://www.red-gate.com/products/sql- development/sql-compare/
  94. 94. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) …the past 60 minutes… T-SQL Test Data Comparing Dimensions
  95. 95. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Is this useful?
  96. 96. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) What other ideas do you have?
  97. 97. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) Have fun!
  98. 98. © 2018 Cathrine Wilhelmsen (contact@cathrinewilhelmsen.net) @cathrinew cathrinew.net hi@cathrinew.net Biml resources and demo files: cathrinew.net/biml

×