Tools and Tips For Data Warehouse Developers (SQLGLA)

Tools and Tips For Data Warehouse Developers (Presented at SQLGLA in Glasgow, Scotland on September 14th, 2018)

Data & Analytics

Tools and Tips
for Data Warehouse Developers
Cathrine Wilhelmsen
SQLGLA · September 14th 2018

Gold Sponsors
Silver Sponsors
Bronze Sponsors Local Partners

Cathrine Wilhelmsen
@cathrinew cathrinew.net
Senior Business Intelligence Consultant, Inmeta

Today?
SQL Tools and Tips
Free Scripts and Resources
Biml for SSIS

Keyboard Shortcuts
docs.microsoft.com/en-us/sql/ssms/sql-server-management-studio-keyboard-shortcuts
Assign shortcuts
you frequently use
Remove shortcuts you
accidentally click

HOME END
PG UP PG DNCTRL ALT
SHIFT TAB
Magic keys!

CTRL K CTRL C
Comment Line
CTRL K CTRL U
Uncomment Line
Comment / Uncomment

CTRL SHIFT V
Last 20 text items copied are
saved in memory
Cycle through them to paste
the text you need
Clipboard Ring

Too many shortcuts? Use Quick Launch!
…or just ALT + Keyboard!
CTRL Q

am2.co/shortcuts
Keyboard Shortcuts Cheat Sheet
by Andy Mallon

SQLWorkbooks.com/SSMS
SSMS Shortcuts and Secrets
by Kendra Little

Template Browser
Create Templates
CTRL ALT T
Templates

Replace Template Parameters with actual values
CTRL SHIFT M
Template Parameters

CTRL K CTRL X
Insert Snippet
CTRL K CTRL S
Surround With Snippet
Snippets

Redgate SQL Prompt (Licensed)
SSMS Tools Pack (Licensed)
red-gate.com
ssmstoolspack.com
Advanced Snippets and Formatting
Favorites

ApexSQL Complete / Refactor
SSMS Boost
Poor Man's T-SQL Formatter
dbForge SQL Complete (Licensed)
SQL Formatter
SqlSmash
apexsql.com
ssmsboost.com
poorsql.com
devart.com/dbforge
sql-format.com
sqlsmash.com
Advanced Snippets and Formatting
Alternatives

Free Tool: Redgate SQL Search
red-gate.com/products/sql-development/sql-search/

Redgate SQL Search
ApexSQL Search
SSMS Tools Pack (Licensed)
SSMS Boost
red-gate.com
apexsql.com
ssmstoolspack.com
ssmsboost.com
Tools: Search in SSMS

Registered Servers and
Multiserver Queries in SSMS

Registered Servers
Save and group servers
Is the server running?
CTRL ALT G

Execution Plans
Display Estimated Execution Plan
CTRL L
Include Actual Execution Plan
CTRL M

Execution Plans
See how a query will be executed:

Live Query Statistics
Include Live Query Statistics

Free Tool: SQL Sentry Plan Explorer
sqlsentry.com/products/plan-explorer

Free Tool: SQL Sentry Plan Explorer
answers.sqlperformance.com

red-gate.com/books
Free Book: SQL Server Execution Plans
by Grant Fritchey

Statistics IO
SET STATISTICS IO OFF;
SET STATISTICS IO ON;

Statistics Time
SET STATISTICS TIME OFF;
SET STATISTICS TIME ON;

statisticsparser.com
Free Tool: Statistics Parser
by Richie Rump

Client Statistics
Include Client Statistics
SHIFT SALT

Client Statistics
Compare multiple query executions:

whoisactive.com
Free Script: sp_WhoIsActive
by Adam Machanic

Of course I can create
200 SSIS Packages!
…what do you need me
to do after lunch?

Business Intelligence Markup Language
Easy to read and write XML language
Generate SSIS packages from metadata

Where can I learn more?
Free online training
bimlscript.com

Biml on Monday…
…BimlBreak the rest
of the week!

@cathrinew
cathrinew.net
hi@cathrinew.net
Too fast? Not enough details? Don't worry!
cathrinew.net/efficient

Talk Description: The Modern Data Warehouse architecture is a response to the emergence of Big Data, Machine Learning and Advanced Analytics. DevOps is a key aspect of successfully operationalising a multi-source Modern Data Warehouse. While there are many examples of how to build CI/CD pipelines for traditional applications, applying these concepts to Big Data Analytical Pipelines is a relatively new and emerging area. In this demo heavy session, we will see how to apply DevOps principles to an end-to-end Data Pipeline built on the Microsoft Azure Data Platform with technologies such as Data Factory, Databricks, Data Lake Gen2, Azure Synapse, and AzureDevOps. Resources: https://aka.ms/mdw-dataops

ADF Mapping Data Flow Private Preview Migration

Eugene Polonichko "Architecture of modern data warehouse"

Lviv Startup Club

Big data is the new big thing where storing the data is the easy part. Gaining insights in your pile of data is something different. Based on a data dump of the well-known StackExchange websites, we will store & analyse 150+ GB of data with Azure Data Lake Store & Analytics to gain some insights about their users. After that we will use Power BI to give an at a glance overview of our learnings. If you are a developer that is interested in big data, this is your time to shine! We will use our existing SQL & C# skills to analyse everything without having to worry about running clusters.

Data quality patterns in the cloud with ADF

Modern ETL: Azure Data Factory, Data Lake, and SQL Database

Azure Data Factory for Redmond SQL PASS UG Sept 2018

R in Power BI

BTUG - Dec 2014 - Hybrid Connectivity Options

Using Redash for SQL Analytics on Databricks

Develop scalable analytical solutions with Azure Data Factory & Azure SQL Dat...

Microsoft Tech Community

In this session you will learn how to develop data pipelines in Azure Data Factory and build a Cloud-based analytical solution adopting modern data warehouse approaches with Azure SQL Data Warehouse and implementing incremental ETL orchestration at scale. With the multiple sources and types of data available in an enterprise today Azure Data factory enables full integration of data and enables direct storage in Azure SQL Data Warehouse for powerful and high-performance query workloads which drive a majority of enterprise applications and business intelligence applications.

Modern data warehouse

Rakesh Jayaram

Microsoft Azure Data Factory Hands-On Lab Overview Slides

Databricks: A Tool That Empowers You To Do More With Data

In this talk we will present how Databricks has enabled the author to achieve more with data, enabling one person to build a coherent data project with data engineering, analysis and science components, with better collaboration, better productionalization methods, with larger datasets and faster. The talk will include a demo that will illustrate how the multiple functionalities of Databricks help to build a coherent data project with Databricks jobs, Delta Lake and auto-loader for data engineering, SQL Analytics for Data Analysis, Spark ML and MLFlow for data science, and Projects for collaboration.

Integration Monday - Analysing StackExchange data with Azure Data Lake

Tom Kerkhove

SharePoint User Group - Leeds - 2015-09-02

Spark as a Service with Azure Databricks

Lace Lofranco

Presented at: Global Azure Bootcamp (Melbourne) Participants will get a deep dive into one of Azure’s newest offering: Azure Databricks, a fast, easy and collaborative Apache® Spark™ based analytics platform optimized for Azure. In this session, we will go through Azure Databricks key collaboration features, cluster management, and tight data integration with Azure data sources. We’ll also walk through an end-to-end Recommendation System Data Pipeline built using Spark on Azure Databricks.

Azure Data Factory for Azure Data Week

PowerBI v2, Power to the People, 1 year later

serge luca

ADF Mapping Data Flows Training V2

Azure Stream Analytics

Marco Parenzan

Tools and Tips For Data Warehouse Developers (SQLSaturday Slovenia)

Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLSat...

What's hot

Microsoft Azure BI Solutions in the Cloud

Introduction to Azure Data Factory

Slava Kokaev

Analyzing StackExchange data with Azure Data Lake

BizTalk360

Data quality patterns in the cloud with ADF

Modern ETL: Azure Data Factory, Data Lake, and SQL Database

Azure Data Factory for Redmond SQL PASS UG Sept 2018

R in Power BI

BTUG - Dec 2014 - Hybrid Connectivity Options

Using Redash for SQL Analytics on Databricks

Develop scalable analytical solutions with Azure Data Factory & Azure SQL Dat...

Microsoft Tech Community

Modern data warehouse

Rakesh Jayaram

Microsoft Azure Data Factory Hands-On Lab Overview Slides

Databricks: A Tool That Empowers You To Do More With Data

Integration Monday - Analysing StackExchange data with Azure Data Lake

Tom Kerkhove

SharePoint User Group - Leeds - 2015-09-02

Spark as a Service with Azure Databricks

Lace Lofranco

Azure Data Factory for Azure Data Week

PowerBI v2, Power to the People, 1 year later

serge luca

ADF Mapping Data Flows Training V2

Azure Stream Analytics

Marco Parenzan

What's hot (20)

Microsoft Azure BI Solutions in the Cloud

Introduction to Azure Data Factory

Analyzing StackExchange data with Azure Data Lake

Data quality patterns in the cloud with ADF

Modern ETL: Azure Data Factory, Data Lake, and SQL Database

Azure Data Factory for Redmond SQL PASS UG Sept 2018

R in Power BI

BTUG - Dec 2014 - Hybrid Connectivity Options

Using Redash for SQL Analytics on Databricks

Develop scalable analytical solutions with Azure Data Factory & Azure SQL Dat...

Modern data warehouse

Microsoft Azure Data Factory Hands-On Lab Overview Slides

Databricks: A Tool That Empowers You To Do More With Data

Integration Monday - Analysing StackExchange data with Azure Data Lake

SharePoint User Group - Leeds - 2015-09-02

Spark as a Service with Azure Databricks

Azure Data Factory for Azure Data Week

PowerBI v2, Power to the People, 1 year later

ADF Mapping Data Flows Training V2

Azure Stream Analytics

Similar to Tools and Tips For Data Warehouse Developers (SQLGLA)

Tools and Tips For Data Warehouse Developers (SQLSaturday Slovenia)

Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLSat...

Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBit...

Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLSat...

Tools and Tips: From Accidental to Efficient Data Warehouse Developer (24 Hou...

There's a shortcut for that! SSMS Tips and Tricks (Denver SQL)

SQL PASS BAC - 60 reporting tips in 60 minutes

Ike Ellis

Ultimate Free SQL Server Toolkit

Kevin Kline

Free and useful tools have proliferated since the launch of the CodePlex and SourceForge websites. Join Kevin Kline, long-time author of the SQL Server Magazine column "Tool Time", as he profiles the very best of the free tools covered in his monthly column - dozens of free tools and utilities! Some of the cover tools help to: - Track database growth - Implement logging in SSIS job steps - Stress test your database applications - Automate important preventative maintenance tasks - Automate maintenance tasks for Analysis Services - Help protect against SQL Injection attacks - Graphically manage Extended Events - Utilize PowerShell scripts to ease administration And much more. These tools are all free and independently supported by SQL Server enthusiasts around the world.

Bi Architecture And Conceptual FrameworkSlava Kokaev

Top Ten Siemens S7 Tips and Tricks

DMC, Inc.

Azure Stream Analytics : Analyse Data in Motion

Ruhani Arora

Building Custom Big Data Integrations

Pat Patterson

Big data ingest frameworks ship with an array of connectors for common data origins and destinations, such as flat files, S3, HDFS, Kafka etc, but sometimes, you need to send data to, or receive data from a system that's not on the list. StreamSets includes template code for building your own connectors and processors; we'll walk through the process of building a simple destination that sends data to a REST web service, and show how it can be extended to target more sophisticated systems such as Salesforce Wave Analytics.

Redgate Community Circle: Tools For SQL Server Performance Tuning

Grant Fritchey

This slide deck is in support of the live and video classes held by Redgate Software as part of the Community Circle initiative. This series of classes is meant to teach you the tools that can help to make query tuning easier in SQL Server. There are a number of different tools, all part of the fundamental Microsoft Data Platform offering, that can be used to help tune queries. This class outlines those tools and their uses.

SQL Pass Architecture SQL Tips & Tricks

Ike Ellis

Social media analytics using Azure Technologies

Koray Kocabas

Social media are computer-mediated tools that allow people to create, share or exchange information, ideas, and pictures/videos in virtual communities and networks. To sum up Social Media is everything for your customers and Your company need to listen them to understand, make a custom offer or improve loyalty etc. Azure Stream Analytics and HDInsight platforms can solve this problem for you. We'll focus on how to get Twitter data using Stream Analytics and how to make data enrichment and storing using HDInsight and What is the problem about sentiment analytics using Azure Machine Learning.

Assyst 9 Overview RoadmapDCL1963

AnalysisServiceswebuploader

60 reporting tips in 60 minutes - SQLBits 2018

Ike Ellis

Mstr meetup

Bhavani Akunuri

TestGuild and QuerySurge Presentation -DevOps for Data Testing

RTTS

This slide deck is from one of our 4 webinars in our half-day series in conjunction with Test Guild. Chris Thompson and Mike Calabrese, Senior Solution Architects and QuerySurge experts, provide great information, a demo and lots of humor in this webinar on how to implement DevOps for Data in your DataOps pipeline. This webinar was performed in conjunction with Test Guild. To watch the video, go to: https://youtu.be/1ihuRPgY_rs

Similar to Tools and Tips For Data Warehouse Developers (SQLGLA) (20)

Tools and Tips For Data Warehouse Developers (SQLSaturday Slovenia)

Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLSat...

Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBit...

Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLSat...

Tools and Tips: From Accidental to Efficient Data Warehouse Developer (24 Hou...

There's a shortcut for that! SSMS Tips and Tricks (Denver SQL)

SQL PASS BAC - 60 reporting tips in 60 minutes

Ultimate Free SQL Server Toolkit

Bi Architecture And Conceptual Framework

Top Ten Siemens S7 Tips and Tricks

Azure Stream Analytics : Analyse Data in Motion

Building Custom Big Data Integrations

Redgate Community Circle: Tools For SQL Server Performance Tuning

SQL Pass Architecture SQL Tips & Tricks

Social media analytics using Azure Technologies

Assyst 9 Overview Roadmap

AnalysisServices

60 reporting tips in 60 minutes - SQLBits 2018

Mstr meetup

TestGuild and QuerySurge Presentation -DevOps for Data Testing

More from Cathrine Wilhelmsen

Data Factory in Microsoft Fabric (MsBIP #82)

Getting Started: Data Factory in Microsoft Fabric (Microsoft Fabric Community...

Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...

Website Analytics in My Pocket using Microsoft Fabric (SQLBits 2024)

Data Integration using Data Factory in Microsoft Fabric (ESPC Microsoft Fabri...

Choosing between Fabric, Synapse and Databricks (Data Left Unattended 2023)

Data Integration with Data Factory (Microsoft Fabric Day Oslo 2023)

The Battle of the Data Transformation Tools (PASS Data Community Summit 2023)

Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PAS...

Building an End-to-End Solution in Microsoft Fabric: From Dataverse to Power ...

Website Analytics in my Pocket using Microsoft Fabric (AdaCon 2023)

Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...

Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D...

Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...

Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (Presented at SQLBits on March 18th, 2023) We all experience stress in our lives. When the stress is time-limited and manageable, it can be positive and productive. This kind of stress can help you get things done and lead to personal growth. However, when the stress stretches out over longer periods of time and we are unable to manage it, it can be negative and debilitating. This kind of stress can affect your mental health as well as your physical health, and increase the risk of depression and burnout. The tricky part is that both depression and burnout can hit you hard without the warning signs you might recognize from stress. Where stress barges through your door and yells "hey, it's me!", depression and burnout can silently sneak in and gradually make adjustments until one day you turn around and see them smiling while realizing that you no longer recognize your house. I know, because I've dealt with both. And when I thought I had kicked them out, they both came back for new visits. I don't have the Answers™️ or Solutions™️ to how to keep them away forever. But in hindsight, there were plenty of warning signs I missed, ignored, or was oblivious to at the time. In this deeply personal session, I will share my story of dealing with both depression and burnout. What were the warning signs? Why did I miss them? Could I have done something differently? And most importantly, what can I - and you - do to help ourselves or our loved ones if we notice that something is not quite right?

"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ...

"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced World (Presented at SQLBits on March 17th, 2023) Do you sometimes think the world is moving so fast that you're struggling to keep up? Does it make you feel a little uncomfortable? Awesome! That means that you have ambitions. You want to learn new things, take that next step in your career, achieve your goals. You can do anything if you set your mind to it. It just might not be easy. All growth requires some discomfort. You need to manage and balance that discomfort, find a way to push yourself a little bit every day without feeling overwhelmed. In a fast-paced world, you need to know how to break down your goals into smaller chunks, how to prioritize, and how to optimize your learning. Are you ready to turn your "I can't keep up" into "I can't believe I did all of that in just one year"?

Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...

Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing Startup (Presented at SQLBits on March 11th, 2022) What happens when you mix one rapidly-changing startup, one data analyst, one data engineer, and one hypothesis that Azure Synapse Analytics could be the right tool of choice for gaining business insights? We had no idea, but we gave it a go! Our ambition was to think big, start small, and act fast – to deliver business value early and often. Did we succeed? Join us for an honest conversation about why we decided to implement Azure Synapse Analytics alongside Power BI, how we got started, which areas we completely messed up at first, what our current solution looks like, the lessons learned along the way, and the things we would have done differently if we could start all over again.

6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)

6 Tips for Building Confidence as a Public Speaker (Presented at SQLBits on March 10th, 2022) Do you feel nervous about getting on stage to deliver a presentation? That was me a few years ago. Palms sweating. Hands shaking. Voice trembling. I could barely breathe and talked at what felt like a thousand words per second. Now, public speaking is one of my favorite hobbies. Sometimes, I even plan my vacations around events! What changed? There are no shortcuts to building confidence as a public speaker. However, there are many things you can do to make the journey a little easier for yourself. In this session, I share the top tips I have learned over the years. All it takes is a little preparation and practice. You can do this!

Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...

Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...

Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...