Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLSaturday Portugal)

496 views

Published on

Tools and Tips: From Accidental to Efficient Data Warehouse Developer (Presented at SQLSaturday Portugal on May 16th 2015)

Published in: Data & Analytics
  • Be the first to comment

Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLSaturday Portugal)

  1. 1. Tools and Tips: From Accidental to Efficient Data Warehouse Developer Cathrine Wilhelmsen - SQLSaturday Lisbon 2015
  2. 2. Session description You have probably heard about the Accidental DBA, but what about the Accidental Data Warehouse developer? We stumbled into the world of data warehousing, learned dimensional modeling and work with T-SQL and SSIS daily. We're masters of googling solutions to our problems and make sure our complex ETL processes run without errors. We deliver data to business users... but we don't deliver data as fast as we want. You might not be able to rewrite your entire data warehouse or change your team's processes over night, but there are many things you can do to increase your own productivity and become a more efficient data warehouse developer. In this session I will show you some of what I've learned and discovered that has made me burst out "Oh wow! Why did I not know this yesterday!?" including query improvements, free tools and scripts, SSMS features and even a couple of things I used to think were only useful for those scary DBAs.
  3. 3. Thank you to the main sponsors!
  4. 4. Say thank you to organizers and volunteers! They spend their FREE time to give you this event Because they are crazy  Because they want YOU to learn from the BEST IN THE WORLD
  5. 5. Paulo Matos:
  6. 6. Pedro Simões:
  7. 7. André Batista:
  8. 8. Paulo Borges:
  9. 9. André Melancia:
  10. 10. Murilo Miranda:
  11. 11. Quilson Antunes:
  12. 12. Important Activities Sponsor Sessions with Raffles • 15:10 - Rumos, BI4ALL, Bold Int, CozyRoc, Pythian WIT (Women in Technology) • 15:10 at BizSpark Room (Ground Floor) SQLClinic Challenges • 17:00 BI (Steph Locke)
  13. 13. Cathrine Wilhelmsen @cathrinew cathrinewilhelmsen.net Data Warehouse Architect Business Intelligence Developer
  14. 14. you 1-3 years? T-SQL? SSIS?
  15. 15. once upon a time...
  16. 16. how I felt...
  17. 17. how I want to be...
  18. 18. what? SSMS Queries Biml for SSIS
  19. 19. Tip #1: Visual Information
  20. 20. Connection Colors
  21. 21. Status Bar and Tab Text
  22. 22. Results in Separate Tab
  23. 23. Tab Groups - Vertical
  24. 24. Tab Groups - Horizontal
  25. 25. Split –one query in two windows
  26. 26. Tip #2: Shortcuts
  27. 27. Query Shortcuts
  28. 28. Oh, by the way... SELECT SUM(rows) FROM sys.partitions WHERE index_id IN (0, 1) AND object_id = OBJECT_ID('Sales'); ...querying sys.partitions can be better and faster than COUNT(*)
  29. 29. Keyboard Shortcuts Assign shortcuts you frequently use Remove shortcuts you accidentally click (no more "ooops") msdn.microsoft.com/en-us/library/ms174205.aspx
  30. 30. Magic keys! HOME END PG UP PG DNCTRL ALT SHIFT TAB
  31. 31. Show / Hide Query Results CTRL R
  32. 32. Toggle Full Screen ALTSHIFT ENTER
  33. 33. Cycle through windows TABCTRL
  34. 34. Change database while writing query CTRL U
  35. 35. Column / Multi-Line Editing SHIFTALT
  36. 36. Comment / Uncomment CTRL K CTRL C Comment Line CTRL K CTRL U Uncomment Line
  37. 37. Tip #3: Search in SSMS
  38. 38. Free Tool: Redgate SQL Search red-gate.com/products/sql-development/sql-search/
  39. 39. Free Tool: Redgate SQL Search
  40. 40. Tip #4: Templates and Snippets
  41. 41. Templates Template Browser Drag & Drop Templates Create Templates CTRL ALT T
  42. 42. Template Parameters Replace Template Parameters with actual values CTRL SHIFT M
  43. 43. Snippets CTRL K CTRL X Insert Snippet CTRL K CTRL S Surround With Snippet
  44. 44. Advanced Snippets and Formatting Redgate SQL Prompt (Licensed) ApexSQL Complete / Refactor SSMS Tools Pack (Licensed) SSMS Boost Poor Man's T-SQL Formatter dbForge SQL Complete (Licensed) red-gate.com apexsql.com ssmstoolspack.com ssmsboost.com poorsql.com devart.com/dbforge
  45. 45. Redgate SQL Prompt Demo
  46. 46. Tip #5: Registered Servers and Multiserver Queries
  47. 47. Registered Servers Save and group servers Is the server running? Multiserver Queries View Registered Servers CTRL ALT G
  48. 48. Manage services from SSMS
  49. 49. Multiserver Queries
  50. 50. Multiserver Queries
  51. 51. Tip #6: SARGable Queries
  52. 52. SARGable queries "The query can efficiently seek using an index to find the correct rows searched for in WHERE or JOIN clauses" Compare it to finding a person in a phone book (We'll pretend we still use phone books)
  53. 53. Adama, Lee Adama, William Agathon, Karl Baltar, Gaius Dualla, Anastasia Gaeta, Felix Henderson, Cally Roslin, Laura Thrace, Kara Tigh, Saul Tyrol, Galen Valerii, Sharon SARGable queries Find all rows where Name starts with "T"
  54. 54. Adama, Lee Adama, William Agathon, Karl Baltar, Gaius Dualla, Anastasia Gaeta, Felix Henderson, Cally Roslin, Laura Thrace, Kara Tigh, Saul Tyrol, Galen Valerii, Sharon SARGable queries Find all rows where Name starts with "T"
  55. 55. Non-SARGable queries "The query has to scan each row in the table to find the correct rows searched for in WHERE or JOIN clauses" Compare it to finding a person in a phone book (We'll keep pretending we still use phone books)
  56. 56. Adama, Lee Adama, William Agathon, Karl Baltar, Gaius Dualla, Anastasia Gaeta, Felix Henderson, Cally Roslin, Laura Thrace, Kara Tigh, Saul Tyrol, Galen Valerii, Sharon Non-SARGable queries Find all rows where Name contains "al"
  57. 57. Adama, Lee Adama, William Agathon, Karl Baltar, Gaius Dualla, Anastasia Gaeta, Felix Henderson, Cally Roslin, Laura Thrace, Kara Tigh, Saul Tyrol, Galen Valerii, Sharon Non-SARGable queries Find all rows where Name contains "al"
  58. 58. WHERE Name LIKE '%al%' WHERE Name LIKE 'T%' WHERE LEFT(Name,1,1) = 'T' SARGable or Non-SARGable?
  59. 59. WHERE Name LIKE '%al%' WHERE Name LIKE 'T%' WHERE LEFT(Name,1,1) = 'T' SARGable or Non-SARGable?
  60. 60. WHERE CAST(EpisodeDate AS DATE) = '20050114' WHERE CONVERT(CHAR(6), EpisodeDate, 112) = '200501' WHERE YEAR(EpisodeDate) = 2005 WHERE EpisodeDate >= '20050101' AND EpisodeDate < '20060101' SARGable or Non-SARGable?
  61. 61. WHERE CAST(EpisodeDate AS DATE) = '20050114' WHERE CONVERT(CHAR(6), EpisodeDate, 112) = '200501' WHERE YEAR(EpisodeDate) = 2005 WHERE EpisodeDate >= '20050101' AND EpisodeDate < '20060101' SARGable or Non-SARGable?
  62. 62. WHERE Survivors < 40000 WHERE @Survivors BETWEEN Survivors-1000 AND Survivors+1000 WHERE Survivors BETWEEN @Survivors-1000 AND @Survivors+1000 SARGable or Non-SARGable?
  63. 63. WHERE Survivors < 40000 WHERE @Survivors BETWEEN Survivors-1000 AND Survivors+1000 WHERE Survivors BETWEEN @Survivors-1000 AND @Survivors+1000 SARGable or Non-SARGable?
  64. 64. sqlbits.com/Sessions/Event7/Understanding_SARGability_to_make_your_queries_run_faster
  65. 65. Tip #7: Query Analysis
  66. 66. Execution Plans Display Estimated Execution Plan CTRL L Include Actual Execution Plan CTRL M
  67. 67. Execution Plans See how a query was or will be executed:
  68. 68. Details in Tooltips (ok)
  69. 69. Details in Properties (better)
  70. 70. Free Tool: SQL Sentry Plan Explorer sqlsentry.com/products/plan-explorer
  71. 71. Free Tool: SQL Sentry Plan Explorer answers.sqlperformance.com
  72. 72. Free Book: SQL Server Execution Plans by Grant Fritchey red-gate.com/community/books
  73. 73. Tip #8: Query Statistics
  74. 74. Statistics IO SET STATISTICS IO OFF; SET STATISTICS IO ON;
  75. 75. Statistics Time SET STATISTICS TIME OFF; SET STATISTICS TIME ON;
  76. 76. Free Tool: Statistics Parser by Richie Rump statisticsparser.com
  77. 77. Client Statistics Include Client Statistics SHIFT SALT
  78. 78. Client Statistics Compare multiple query executions:
  79. 79. Tip #9: Activity Monitoring
  80. 80. Free Script: sp_WhoIsActive by Adam Machanic sqlblog.com/blogs/adam_machanic
  81. 81. Free Script: sp_WhoIsActive by Adam Machanic
  82. 82. Tip #10: Generate SSIS Packages with Biml
  83. 83. Business Intelligence Markup Language Easy to read and write XML dialect Generate SSIS packages from metadata
  84. 84. What do I need? Free add-in for BIDS / SSDT-BI bidshelper.codeplex.com
  85. 85. How does it work? Create many SSIS packages from one Biml file
  86. 86. …what do you need me to do after lunch? Of course I can create 200 SSIS Packages!
  87. 87. Biml syntax <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Connections> <OleDbConnection Name="Source" ConnectionString="…" /> </Connections> <Packages> <Package Name="EmptyPackage"> … </Package> </Packages> </Biml>
  88. 88. From Biml to SSIS
  89. 89. From Biml to SSIS
  90. 90. The magic is in the Extend Biml with C# or VB.NET code blocks Import database structure and metadata Loop over tables and columns Add expressions to replace static values
  91. 91. BimlScript syntax <#@ import namespace="Varigence.Hadron.CoreLowerer.SchemaManagement" #> <# var conAW2014 = SchemaManager.CreateConnectionNode("AW2014", "..."); #> <# var AW2014DB = conAW2014.ImportDB("","", ImportOptions.ExcludeViews); #> <Packages> <# foreach (var table in AW2014DB.TableNodes) { #> <Package Name="Load_<#=table.Name#>"> … </Package> <# } #> </Packages>
  92. 92. Biml for SSIS demo
  93. 93. …BimlBreak the rest of the week  Biml on Monday…
  94. 94. Bonus Tip: Be social in your community! #SQLHelp
  95. 95. cathrinewilhelmsen.net/efficient Not enough details? Too fast? Don't worry! Slide deck, links and resources:
  96. 96. Go talk to the sponsors!
  97. 97. Thank you! Enjoy your lunch  @cathrinew cathrinewilhelmsen.net no.linkedin.com/in/cathrinewilhelmsen contact@cathrinewilhelmsen.net cathrinewilhelmsen.net/efficient

×