Don't Repeat Yourself - An Introduction to Agile SSIS Development (24 Hours o...Cathrine Wilhelmsen
Don't Repeat Yourself - An Introduction to Agile SSIS Development (24 Hours of PASS) (Presented at 24 HOurs of PASS Growing Our Community Edititon on June 25th 2015)
Don't Repeat Yourself - An Introduction to Agile SSIS Development (24 Hours o...Cathrine Wilhelmsen
Don't Repeat Yourself - An Introduction to Agile SSIS Development (24 Hours of PASS) (Presented at 24 HOurs of PASS Growing Our Community Edititon on June 25th 2015)
In this presentation, you may find necessary information to engage in custom design integration on Magento:
- standard development workflow
- working environment preparation
- layout comprehension
- introducing blocks
- understand header and footer elements
In this presentation I cover some techniques to extend the base build of TFS Build 2010, showing the interation with MSBuild and creation of custom activities.
Simple Build Tool (sbt) is an open source build tool , It is a best choice for Scala projects that aims to do the basics well. It requires Java 1.6 or later.
This fast-paced presentation is for the medium-level and higher LotusScript programmer who wants bite-sized lessons from the field. It's presented in three categories, covering tips you should be using, tips you should know, and finally tips that you should be aware of.
This presentation was also used for the speedgeeking session. I had to give a 60 minute presentation in 4.5 minutes, 12 times. I managed to complete it once, and averaged getting throught 22 tips. The fun thing about speedgeeking is that it does force you to think how you can compress down a presentation, and make each word fight for survival..
The Battle of the Data Transformation Tools (PASS Data Community Summit 2023)Cathrine Wilhelmsen
The Battle of the Data Transformation Tools (Presented as part of the "Batte of the Data Transformation Tools" Learning Path at PASS Data Community Summit on November 16th, 2023)
Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PAS...Cathrine Wilhelmsen
Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (Presented as part of the "Batte of the Data Transformation Tools" Learning Path at PASS Data Community Summit on November 15th, 2023)
Building an End-to-End Solution in Microsoft Fabric: From Dataverse to Power ...Cathrine Wilhelmsen
Building an End-to-End Solution in Microsoft Fabric: From Dataverse to Power BI (Presented at SQLSaturday Oregon & SW Washington on November 11th, 2023)
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...Cathrine Wilhelmsen
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (Presented at SQLBits on March 18th, 2023)
We all experience stress in our lives. When the stress is time-limited and manageable, it can be positive and productive. This kind of stress can help you get things done and lead to personal growth. However, when the stress stretches out over longer periods of time and we are unable to manage it, it can be negative and debilitating. This kind of stress can affect your mental health as well as your physical health, and increase the risk of depression and burnout.
The tricky part is that both depression and burnout can hit you hard without the warning signs you might recognize from stress. Where stress barges through your door and yells "hey, it's me!", depression and burnout can silently sneak in and gradually make adjustments until one day you turn around and see them smiling while realizing that you no longer recognize your house. I know, because I've dealt with both. And when I thought I had kicked them out, they both came back for new visits.
I don't have the Answers™️ or Solutions™️ to how to keep them away forever. But in hindsight, there were plenty of warning signs I missed, ignored, or was oblivious to at the time. In this deeply personal session, I will share my story of dealing with both depression and burnout. What were the warning signs? Why did I miss them? Could I have done something differently? And most importantly, what can I - and you - do to help ourselves or our loved ones if we notice that something is not quite right?
"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ...Cathrine Wilhelmsen
"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced World (Presented at SQLBits on March 17th, 2023)
Do you sometimes think the world is moving so fast that you're struggling to keep up?
Does it make you feel a little uncomfortable?
Awesome!
That means that you have ambitions. You want to learn new things, take that next step in your career, achieve your goals. You can do anything if you set your mind to it.
It just might not be easy.
All growth requires some discomfort. You need to manage and balance that discomfort, find a way to push yourself a little bit every day without feeling overwhelmed. In a fast-paced world, you need to know how to break down your goals into smaller chunks, how to prioritize, and how to optimize your learning.
Are you ready to turn your "I can't keep up" into "I can't believe I did all of that in just one year"?
Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...Cathrine Wilhelmsen
Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing Startup (Presented at SQLBits on March 11th, 2022)
What happens when you mix one rapidly-changing startup, one data analyst, one data engineer, and one hypothesis that Azure Synapse Analytics could be the right tool of choice for gaining business insights?
We had no idea, but we gave it a go!
Our ambition was to think big, start small, and act fast – to deliver business value early and often.
Did we succeed?
Join us for an honest conversation about why we decided to implement Azure Synapse Analytics alongside Power BI, how we got started, which areas we completely messed up at first, what our current solution looks like, the lessons learned along the way, and the things we would have done differently if we could start all over again.
6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)Cathrine Wilhelmsen
6 Tips for Building Confidence as a Public Speaker (Presented at SQLBits on March 10th, 2022)
Do you feel nervous about getting on stage to deliver a presentation?
That was me a few years ago. Palms sweating. Hands shaking. Voice trembling. I could barely breathe and talked at what felt like a thousand words per second. Now, public speaking is one of my favorite hobbies. Sometimes, I even plan my vacations around events! What changed?
There are no shortcuts to building confidence as a public speaker. However, there are many things you can do to make the journey a little easier for yourself. In this session, I share the top tips I have learned over the years. All it takes is a little preparation and practice.
You can do this!
Learn SQL from basic queries to Advance queriesmanishkhaire30
Dive into the world of data analysis with our comprehensive guide on mastering SQL! This presentation offers a practical approach to learning SQL, focusing on real-world applications and hands-on practice. Whether you're a beginner or looking to sharpen your skills, this guide provides the tools you need to extract, analyze, and interpret data effectively.
Key Highlights:
Foundations of SQL: Understand the basics of SQL, including data retrieval, filtering, and aggregation.
Advanced Queries: Learn to craft complex queries to uncover deep insights from your data.
Data Trends and Patterns: Discover how to identify and interpret trends and patterns in your datasets.
Practical Examples: Follow step-by-step examples to apply SQL techniques in real-world scenarios.
Actionable Insights: Gain the skills to derive actionable insights that drive informed decision-making.
Join us on this journey to enhance your data analysis capabilities and unlock the full potential of SQL. Perfect for data enthusiasts, analysts, and anyone eager to harness the power of data!
#DataAnalysis #SQL #LearningSQL #DataInsights #DataScience #Analytics
Analysis insight about a Flyball dog competition team's performanceroli9797
Insight of my analysis about a Flyball dog competition team's last year performance. Find more: https://github.com/rolandnagy-ds/flyball_race_analysis/tree/main
The Building Blocks of QuestDB, a Time Series Databasejavier ramirez
Talk Delivered at Valencia Codes Meetup 2024-06.
Traditionally, databases have treated timestamps just as another data type. However, when performing real-time analytics, timestamps should be first class citizens and we need rich time semantics to get the most out of our data. We also need to deal with ever growing datasets while keeping performant, which is as fun as it sounds.
It is no wonder time-series databases are now more popular than ever before. Join me in this session to learn about the internal architecture and building blocks of QuestDB, an open source time-series database designed for speed. We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read-replicas, or faster batch ingestion.
Enhanced Enterprise Intelligence with your personal AI Data Copilot.pdfGetInData
Recently we have observed the rise of open-source Large Language Models (LLMs) that are community-driven or developed by the AI market leaders, such as Meta (Llama3), Databricks (DBRX) and Snowflake (Arctic). On the other hand, there is a growth in interest in specialized, carefully fine-tuned yet relatively small models that can efficiently assist programmers in day-to-day tasks. Finally, Retrieval-Augmented Generation (RAG) architectures have gained a lot of traction as the preferred approach for LLMs context and prompt augmentation for building conversational SQL data copilots, code copilots and chatbots.
In this presentation, we will show how we built upon these three concepts a robust Data Copilot that can help to democratize access to company data assets and boost performance of everyone working with data platforms.
Why do we need yet another (open-source ) Copilot?
How can we build one?
Architecture and evaluation
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Round table discussion of vector databases, unstructured data, ai, big data, real-time, robots and Milvus.
A lively discussion with NJ Gen AI Meetup Lead, Prasad and Procure.FYI's Co-Found
Level Up Your Biml: Best Practices and Coding Techniques (NTK 2016)
1.
2. Level Up Your Biml:
Best Practices and Coding Techniques
Cathrine Wilhelmsen
3. Session Description
You already know how to use Biml to build a staging environment in an hour, so
let's dive straight into some of the more advanced features of Biml.
Attend this session for an overview of Biml best practices and coding techniques.
Learn how to centralize and reuse code with include files and the CallBimlScript
method. Make your code easier to read and write by utilizing LINQ (Language-
Integrated Queries). Share code between files by using Annotations and
ObjectTags. And finally, if standard Biml is not enough to solve your problems,
you can create your own C# helper classes and extension methods to implement
custom logic.
Start improving your code today and level up your Biml in no time!
8. What is Biml?
Business Intelligence Markup Language
Easy to read and write XML language
Describes business intelligence objects:
• Databases, Schemas, Tables, Views, Columns
• SSIS Packages
• SSAS Cubes
• Metadata
17. The magic is in the BimlScript!
Extend Biml with C# or VB code blocks
Import database structure and metadata
Loop over tables and columns
Expressions replace static values
BimlScript allows you to control and manipulate Biml code
25. Don't Repeat Yourself
Move common code to separate files
Centralize and reuse in many projects
Update code once for all projects
1. Include files
2. CallBimlScript with Parameters
3. Tiered Biml files
26. BimlExpress vs. BimlOnline / BimlStudio
"Black Box"
Only SSIS packages visible
Visual Editors
All in-memory objects visible
27. Include Files
Include common code in multiple files and projects
Can include many file types: .biml .txt .sql .cs
Use the include directive
<#@ include file="CommonCode.biml" #>
The directive will be replaced by the included file
Include pulls code from the included file into the main file
Works like an automated Copy & Paste
31. CallBimlScript with Parameters
Works like a parameterized include
File to be called (callee) specifies input parameters it accepts
<#@ property name="Parameter" type="String" #>
File that calls (caller) passes input parameters
<#=CallBimlScript("CommonCode.biml", Parameter)#>
CallBimlScript pushes parameters from the caller to the callee, and the
callee returns code
37. Tiered Biml Files
Split Biml code in multiple files and use the template directive:
<#@ template tier="1" #>
Create objects in-memory from lowest to highest tier to:
• Solve logical dependencies
• Simulate manual workflows
In-memory objects are added to the RootNode
Higher tiers can get objects added to RootNode in lower tiers
38. What is this RootNode?
The RootNode contains all in-memory objects:
• Connections, Databases, Schemas, Tables
• Projects, Packages
• Annotations, Metadata
Query the RootNode to loop over collections:
<# foreach (var table in RootNode.Tables) { #>
Query the RootNode to get specific objects:
<#=RootNode.Tables["Product"].Schema#>
50. How do you use Tiered Biml files?
1. Create Biml files with specified tiers
2. Select all the tiered Biml files
3. Right-click and click Generate SSIS Packages
1
2
3
53. Debugging Biml
BimlExpress is a "black box":
• You can only see the generated SSIS packages
• It is not possible to see the compiled Biml first
Add a high-tier helper file to save compiled, flat Biml to file
• Check Biml For Errors to save flat Biml without generating packages
54. SaveFlatBimlToFile.biml
Add the helper file to your project…
<#@ template tier="999" #>
<# System.IO.File.WriteAllText(
@"C:BimlFlatBiml.xml",
RootNode.GetBiml()
); #>
55. SaveFlatBimlToFile.biml
…with a high tier so it is executed as the last step
<#@ template tier="999" #>
<# System.IO.File.WriteAllText(
@"C:BimlFlatBiml.xml",
RootNode.GetBiml()
); #>
56. SaveFlatBimlToFile.biml
It creates a file…
<#@ template tier="999" #>
<# System.IO.File.WriteAllText(
@"C:BimlFlatBiml.xml",
RootNode.GetBiml()
); #>
58. SaveFlatBimlToFile.biml
…with all the Biml for all the object in RootNode
<#@ template tier="999" #>
<# System.IO.File.WriteAllText(
@"C:BimlFlatBiml.xml",
RootNode.GetBiml()
); #>
59. How do you use this helper file?
1. Create the helper file
2. Select all the Biml files and the helper file
3. Right-click and click Check Biml For Errors
1
2
3
62. Annotations and ObjectTags
Biml Annotations != SSIS Annotations
Annotations are string/string Key/Value pairs
ObjectTags are string/object Key/Value pairs
Use Annotations and ObjectTags to pass code
between Biml files
63. Annotations
Create annotations:
<OleDbConnection Name="Destination" ConnectionString="…">
<Annotations>
<Annotation Tag="Schema">AW2014</Annotation>
</Annotations>
</OleDbConnection>
Use annotations:
<# var destinationSchema =
RootNode.OleDbConnections["Destination"].GetTag("Schema"); #>
66. LINQ (Language-Integrated Query)
One language to query:
SQL Server Databases
XML Documents
Datasets
Collections
Two ways to write queries:
SQL-like Syntax
Extension Methods
67. LINQ Extension Methods
..and many, many more!
Sort
OrderBy, ThenBy
Filter
Where, OfType
Group
GroupBy
Aggregate
Count, Sum
Check Collections
All, Any, Contains
Get Elements
First, Last, ElementAt
Project Collections
Select, SelectMany
68. LINQ Extension Methods
var numConnections = RootNode.Connections.Count()
foreach (var table in RootNode.Tables.Where(…))
if (RootNode.Packages.Any(…))
69. LINQ and Lambda expressions
Use lambda expressions to filter or specify values:
.Where(table => table.Schema.Name == "Production")
.OrderBy(table => table.Name)
70. LINQ and Lambda expressions
For each element in the collection…
.Where(table => table.Schema.Name == "Production")
.OrderBy(table => table.Name)
71. LINQ and Lambda expressions
…evaluate a criteria or get a value:
.Where(table => table.Schema.Name == "Production")
.OrderBy(table => table.Name)
72. LINQ: Filter collections
Where()
Returns the filtered collection with all elements that meet the criteria
RootNode.Tables.Where(t => t.Schema.Name == "Production")
OfType()
Returns the filtered collection with all elements of the specified type
RootNode.Connections.OfType<AstExcelOleDbConnectionNode>()
73. LINQ: Sort collections
OrderBy()
Returns the collection sorted by key…
RootNode.Tables.OrderBy(t => t.Name)
ThenBy()
…then sorted by secondary key
RootNode.Tables.OrderBy(t => t.Schema.Name)
.ThenBy(t => t.Name)
74. LINQ: Sort collections
OrderByDescending()
Returns the collection sorted by key…
RootNode.Tables.OrderByDescending(t => t.Name)
ThenByDescending()
…then sorted by secondary key
RootNode.Tables.OrderBy(t => t.Schema.Name)
.ThenByDescending(t => t.Name)
78. LINQ: Aggregate collections
Sum()
Returns the sum of the (numeric) values in the collection
RootNode.Tables.Sum(t => t.Columns.Count)
Average()
Returns the average value of the (numeric) values in the collection
RootNode.Tables.Average(t => t.Columns.Count)
79. LINQ: Aggregate collections
Min()
Returns the minimum value of the (numeric) values in the collection
RootNode.Tables.Min(t => t.Columns.Count)
Max()
Returns the maximum value of the (numeric) values in the collection
RootNode.Tables.Max(t => t.Columns.Count)
80. LINQ: Check collections
All()
Returns true if all elements in the collection meet the criteria
RootNode.Databases.All(d => d.Name.StartsWith("A"))
Any()
Returns true if any element in the collection meets the criteria
RootNode.Databases.Any(d => d.Name.Contains("DW"))
82. LINQ: Get elements
First()
Returns the first element in the collection (that meets the criteria)
RootNode.Tables.First()
RootNode.Tables.First(t => t.Schema.Name == "Production")
FirstOrDefault()
Returns the first element in the collection or default value (that meets the criteria)
RootNode.Tables.FirstOrDefault()
RootNode.Tables.FirstOrDefault(t => t.Schema.Name == "Production")
83. LINQ: Get elements
Last()
Returns the last element in the collection (that meets the criteria)
RootNode.Tables.Last()
RootNode.Tables.Last(t => t.Schema.Name == "Production")
LastOrDefault()
Returns the last element in the collection or default value (that meets the criteria)
RootNode.Tables.LastOrDefault()
RootNode.Tables.LastOrDefault(t => t.Schema.Name == "Production")
84. LINQ: Get elements
ElementAt()
Returns the element in the collection at the specified index
RootNode.Tables.ElementAt(42)
ElementAtOrDefault()
Returns the element in the collection or default value at the specified index
RootNode.Tables.ElementAtOrDefault(42)
85. LINQ: Project collections
Select()
Creates a new collection from one collection
A list of table names:
RootNode.Tables.Select(t => t.Name)
A list of table and schema names:
RootNode.Tables.Select(t => new {t.Name, t.Schema.Name})
86. LINQ: Project collections
SelectMany()
Creates a new collection from many collections and merges the collections
A list of all columns from all tables:
RootNode.Tables.SelectMany(t => t.Columns)
89. C# Classes and Methods
BimlScript and LINQ not enough?
Need to reuse C# code?
Create your own classes and methods!
90. C# Classes and Methods: From this…
public static class HelperClass {
public static bool AnnotationTagExists(AstNode node, string tag) {
if (node.GetTag(tag) != "") {
return true;
} else {
return false;
}
}
}
91. C# Classes and Methods: …to this
public static class HelperClass {
public static bool AnnotationTagExists(AstNode node, string tag) {
return (node.GetTag(tag) != "") ? true : false;
}
} * For bools you can just use:
return (node.GetTag(tag) != "");
But in this example we'll use the verbose, SSIS-like syntax
because it can be reused with other data types, like…
92. C# Classes and Methods: …or this
public static class HelperClass {
public static string AnnotationTagExists(AstNode node, string tag) {
return (node.GetTag(tag) != "") ? "Yes" : "No";
}
}
93. Where do you put your code?
Inline code nuggets
Included Biml files with code nuggets
Reference code files
94. C# Classes and Methods: Inline
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<# foreach (var table in RootNode.Tables) { #>
<# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #>
...
<# } #>
<# } #>
</Biml>
<#+
public static class HelperClass {
public static bool AnnotationTagExists(AstNode node, string tag) {
return (node.GetTag(tag) != "") ? true : false;
}
}
#>
95. C# Classes and Methods: Included Files
<#@ include file="HelperClass.biml" #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<# foreach (var table in RootNode.Tables) { #>
<# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #>
...
<# } #>
<# } #>
</Biml>
<#+
public static class HelperClass {
public static bool AnnotationTagExists(AstNode node, string tag) {
return (node.GetTag(tag) != "") ? true : false;
}
}
#>
96. C# Classes and Methods: Code Files
<#@ code file="HelperClass.cs" #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<# foreach (var table in RootNode.Tables) { #>
<# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #>
...
<# } #>
<# } #>
</Biml>
public static class HelperClass {
public static bool AnnotationTagExists(AstNode node, string tag) {
return (node.GetTag(tag) != "") ? true : false;
}
}
104. Get things done
Start small
Start simple
Start with ugly code
Keep going
Expand
Improve
Deliver often
105. Izpolnite anketo!
Vam je bilo predavanje všeč?
Ste se naučili kaj novega?
Vaše mnenje nam veliko pomeni!
Da bo NT konferenca prihodnje leto še boljša, vas
prosimo, da izpolnite anketo o zadovoljstvu, ki jo
najdete v svojem NTK spletnem profilu.