Tools and Tips: From Accidental to Efficient Data Warehouse Developer (Presented at SQLBits XIV on March 6th 2015)

1. Tools and Tips
for Data Warehouse Developers
Cathrine Wilhelmsen - SQLBits XIV

2. Session description
You have probably heard about the Accidental DBA, but what about the Accidental Data
Warehouse developer? We stumbled into the world of data warehousing, learned dimensional
modeling and work with T-SQL and SSIS daily. We're masters of googling solutions to our
problems and make sure our complex ETL processes run without errors. We deliver data to
business users... but we don't deliver data as fast as we want.
You might not be able to rewrite your entire data warehouse or change your team's processes
over night, but there are many things you can do to increase your own productivity and become
a more efficient data warehouse developer.
In this session I will show you some of what I've learned and discovered that has made me
burst out "Oh wow! Why did I not know this yesterday!?" including query improvements, free
tools and scripts, SSMS features and even a couple of things I used to think were only useful for
those scary DBAs.

3. Cathrine Wilhelmsen
@cathrinew
cathrinewilhelmsen.net
Data Warehouse Architect
Business Intelligence Developer

4. you
1-3 years?
T-SQL?
SSIS?

5. once upon a time...

7. how I felt...

9. how I want to be...

11. what?
SSMS
Queries
Biml for SSIS

12. Tip #1: Visual Information

13. Connection Colors

14. Status Bar and Tab Text

15. Results in Separate Tab

16. Tab Groups - Vertical

17. Tab Groups - Horizontal

18. Split –one query in two windows

19. Tip #2: Shortcuts

20. Query Shortcuts

21. Keyboard Shortcuts
Assign shortcuts
you frequently use
Remove shortcuts
you accidentally click
(no more "ooops")
msdn.microsoft.com/en-us/library/ms174205.aspx

22. Magic keys!
HOME END
PG UP PG DNCTRL ALT
SHIFT TAB

23. Show / Hide Query Results
CTRL R

24. Toggle Full Screen
ALTSHIFT ENTER

25. Cycle through windows
TABCTRL

26. Change database while writing query
CTRL U

27. Column / Multi-Line Editing
SHIFTALT

28. Comment / Uncomment
CTRL K CTRL C
Comment Line
CTRL K CTRL U
Uncomment Line

29. Tip #3: Search in SSMS

30. Free Tool: Redgate SQL Search
red-gate.com/products/sql-development/sql-search/

31. Free Tool: Redgate SQL Search

32. Tip #4: Templates and Snippets

33. Templates
Template Browser
Drag & Drop Templates
Create Templates
CTRL ALT T

34. Template Parameters
Replace Template Parameters with actual values
CTRL SHIFT M

35. Snippets
CTRL K CTRL X
Insert Snippet
CTRL K CTRL S
Surround With Snippet

36. Advanced Snippets and Formatting
Redgate SQL Prompt (Licensed)
ApexSQL Complete / Refactor
SSMS Tools Pack (Licensed)
SSMS Boost
Poor Man's T-SQL Formatter
dbForge SQL Complete (Licensed)
red-gate.com
apexsql.com
ssmstoolspack.com
ssmsboost.com
poorsql.com
devart.com/dbforge

37. Redgate SQL Prompt Demo

38. Tip #5: Registered Servers and Multiserver Queries

39. Registered Servers
Save and group servers
Is the server running?
Multiserver Queries
View Registered Servers
CTRL ALT G

40. Manage services from SSMS

41. Multiserver Queries

42. Multiserver Queries

43. Tip #6: SARGable Queries

44. SARGable queries
"The query can efficiently seek using an index to find the
correct rows searched for in WHERE or JOIN clauses"
Compare it to finding a person in a phone book
(We'll pretend we still use phone books)

45. Adama, Lee
Adama, William
Agathon, Karl
Baltar, Gaius
Dualla, Anastasia
Gaeta, Felix
Henderson, Cally
Roslin, Laura
Thrace, Kara
Tigh, Saul
Tyrol, Galen
Valerii, Sharon
SARGable queries
Find all rows where Name starts with "T"

46. Adama, Lee
Adama, William
Agathon, Karl
Baltar, Gaius
Dualla, Anastasia
Gaeta, Felix
Henderson, Cally
Roslin, Laura
Thrace, Kara
Tigh, Saul
Tyrol, Galen
Valerii, Sharon
SARGable queries
Find all rows where Name starts with "T"

47. Non-SARGable queries
"The query has to scan each row in the table to find the
correct rows searched for in WHERE or JOIN clauses"
Compare it to finding a person in a phone book
(We'll keep pretending we still use phone books)

48. Adama, Lee
Adama, William
Agathon, Karl
Baltar, Gaius
Dualla, Anastasia
Gaeta, Felix
Henderson, Cally
Roslin, Laura
Thrace, Kara
Tigh, Saul
Tyrol, Galen
Valerii, Sharon
Non-SARGable queries
Find all rows where Name contains "al"

49. Adama, Lee
Adama, William
Agathon, Karl
Baltar, Gaius
Dualla, Anastasia
Gaeta, Felix
Henderson, Cally
Roslin, Laura
Thrace, Kara
Tigh, Saul
Tyrol, Galen
Valerii, Sharon
Non-SARGable queries
Find all rows where Name contains "al"

50. WHERE Name LIKE '%al%'
WHERE Name LIKE 'T%'
WHERE LEFT(Name,1,1) = 'T'
SARGable or Non-SARGable?

51. WHERE Name LIKE '%al%'
WHERE Name LIKE 'T%'
WHERE LEFT(Name,1,1) = 'T'
SARGable or Non-SARGable?

52. WHERE CAST(EpisodeDate AS DATE) = '20050114'
WHERE CONVERT(CHAR(6), EpisodeDate, 112) = '200501'
WHERE YEAR(EpisodeDate) = 2005
WHERE EpisodeDate >= '20050101'
AND EpisodeDate < '20060101'
SARGable or Non-SARGable?

53. WHERE CAST(EpisodeDate AS DATE) = '20050114'
WHERE CONVERT(CHAR(6), EpisodeDate, 112) = '200501'
WHERE YEAR(EpisodeDate) = 2005
WHERE EpisodeDate >= '20050101'
AND EpisodeDate < '20060101'
SARGable or Non-SARGable?

54. WHERE Survivors < 40000
WHERE @Survivors BETWEEN Survivors-1000
AND Survivors+1000
WHERE Survivors BETWEEN @Survivors-1000
AND @Survivors+1000
SARGable or Non-SARGable?

55. WHERE Survivors < 40000
WHERE @Survivors BETWEEN Survivors-1000
AND Survivors+1000
WHERE Survivors BETWEEN @Survivors-1000
AND @Survivors+1000
SARGable or Non-SARGable?

56. sqlbits.com/Sessions/Event7/Understanding_SARGability_to_make_your_queries_run_faster

57. Tip #7: Query Analysis

58. Execution Plans
Display Estimated Execution Plan
CTRL L
Include Actual Execution Plan
CTRL M

59. Execution Plans
See how a query was or will be executed:

60. Details in Tooltips

61. Details in Properties

62. Free Tool: SQL Sentry Plan Explorer
sqlsentry.com/products/plan-explorer

63. Free Tool: SQL Sentry Plan Explorer
answers.sqlperformance.com

64. Free Book: SQL Server Execution Plans
by Grant Fritchey
red-gate.com/community/books

65. Tip #8: Query Statistics

66. Statistics IO
SET STATISTICS IO OFF;
SET STATISTICS IO ON;

67. Statistics Time
SET STATISTICS TIME OFF;
SET STATISTICS TIME ON;

68. Free Tool: Statistics Parser
statisticsparser.com

69. Client Statistics
Include Client Statistics
SHIFT SALT

70. Client Statistics
Compare multiple query executions:

71. Tip #9: Activity Monitoring

72. Free Script: sp_WhoIsActive by Adam Machanic
sqlblog.com/blogs/adam_machanic

73. Free Script: sp_WhoIsActive by Adam Machanic

74. Tip #10: Automatically Generate SSIS Packages with Biml

75. Business Intelligence Markup Language
Easy to read and write XML dialect
Generate SSIS packages from metadata

76. What do I need?
Free add-in for BIDS / SSDT-BI
bidshelper.codeplex.com

77. How does it work?
Create many SSIS packages from one Biml file

78. …what do you need me to do after lunch?
Of course I can create 200 SSIS Packages!

79. Biml syntax
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Connections>
<OleDbConnection Name="Source" ConnectionString="…" />
</Connections>
<Packages>
<Package Name="EmptyPackage">
…
</Package>
</Packages>
</Biml>

80. From Biml to SSIS

81. From Biml to SSIS

82. The magic is in the
Extend Biml with C# or VB.NET code blocks
Import database structure and metadata
Loop over tables and columns
Add expressions to replace static values

83. BimlScript syntax
<#@ import namespace="Varigence.Hadron.CoreLowerer.SchemaManagement" #>
<# var conAW2014 = SchemaManager.CreateConnectionNode("AW2014", "..."); #>
<# var AW2014DB = conAW2014.ImportDB("","", ImportOptions.ExcludeViews); #>
<Packages>
<# foreach (var table in AW2014DB.TableNodes) { #>
<Package Name="Load_<#=table.Name#>">
…
</Package>
<# } #>
</Packages>

84. Biml for SSIS demo

85. …BimlBreak the rest of the week 
Biml on Monday…

86. cathrinewilhelmsen.net/efficient
Not enough details? Too fast? Don't worry!
Slide deck, links and resources:

87. Thank you!
@cathrinew
cathrinewilhelmsen.net
no.linkedin.com/in/cathrinewilhelmsen
contact@cathrinewilhelmsen.net
cathrinewilhelmsen.net/efficient