This document discusses using Azure DevOps and Snowflake to enable continuous integration and continuous deployment (CI/CD) of database changes. It covers setting up source control in a repository, implementing pull requests for code reviews, building deployment artifacts in a build pipeline, and deploying artifacts to development, test, and production environments through a release pipeline. The document also highlights key Snowflake features like zero-copy cloning that enable testing deployments before production.

1. Automated
Deployments in
Snowflake
Using Azure DevOps

2. Hello!
Drew Hansen
Data Engineer at ADESA (KAR Global)
Email: drew.hansen@adesa.com
Twitter: @DrewHansen_9
2
Adorable picture so you
don’t have to look at me

3. Agenda
● DevOps and Business Intelligence?
● CI/CD Pipelines: What are they?
● Database Deployments: State based vs Migration based
● Snowflake features for CI/CD
● Azure DevOps: Build and Release Pipelines
● Putting it all together: End to End solution
● Demo
● Other Features we added
3

4. How do
you
deploy?
If you need to deploy a hot fix to a table in
production, what is your process?
4

5. Agile? DevOps? What’s the
Hype?
It is about the Business!

6. Agile vs. DevOps
6
Business Developer Technology
Agile DevOps

7. 7
**Image from: https://skookum.com/blog/agile-transformation-and-devops
Agile vs. DevOps

8. “ An agile culture that better
supports collaboration between
people that uses automation and
tooling in support of process.
8
DevOps is…
- Grant Fritchey
PASS President (the man)

9. Continuous Integration + Continuous Deployment
Cool people call it CI/CD

10. Continuous Integration
● Small incremental changes
● Shift Left
● Test often and early
○ Unit Tests
○ Integration Tests
● Outcome: BUILD
Definitions
Continuous Deployment
● Hands Off deployments
● Consistent throughout
environments
● Outcome: RELEASE
10
By source controlling your objects you can revert back
to a specific time/place.
Helps developers feel confident in their changes.

11. ● Automation means less human mistakes.
● Helps developers feel confident in their changes.
● No one has access to production!
Guardrails
11

12. 12
All together now

13. Look a Pipeline!
13

14. Build / Release
14
Build
• Packages SQL to be
run on DB
• Runs Unit Tests
• Produces an Artifact
Artifact Release Stage
CI CD
• File(s) that are
irrespective of
environment.
• Artifact will be
deployed into DB
• A release pipeline is
process that deploys to
multiple stages.
• Holds environment
variables.
• A single deployment
into an environment
(DB)
• Ex. 3 stages (DEV, TST,
PRD)

15. Database Deployments
State based vs Migration based

16. State Based
● Compare Database to
Repo/Source Control.
● Deploying tool makes
decision on what to deploy
based on differences.
● Source control objects in a
consistent State.
● Ex. CREATE TABLE AS…
● Ex. DACPAC
Definitions
Migration Based
● Simply deploys scripts given by
the developer.
● Source control objects in a
evolving state.
● Ex. ALTER TABLE AS…
16

17. Confused? Pictures help.
17
DB Project/
DACPAC
DB
compare
generate
execute
Difference
Script
State Based Migration Based
State 1 State n
Migration 1
Script
Migration
n
Script
Laptop
Developer updates
DB Project
Laptop
Developer creates
new script
Laptop

18. State Based
● None.
WOMP WOMP WOMP…
Database Management Tools
Available To Snowflake
Migration Based
● Liquibase (XML)
● FlyWay (XML/JSON)
● Sqitch (SQL, but complicated)
● Datical (2020?)
● Make your own!
18
😭
👍

19. Snowflake Features
and Considerations

20. ● Application?
○ Build the app/solution. Easy Peasy.
● Database?
○ Not so fast. Building a script is not testing.
○ You have to Deploy!
○ Annnnnnd break all the DevOps rules we
just discussed.
○ BUT…
How in the world do you
test a deployment?
20

21. Spoiler: Snowflake Wins…
21

22. Testing in Snowflake
Zero-Copy Cloning
● Create clones of tables,
schemas, or entire databases
instantly
● Requires NO additional
storage
● Adds no extra load on the
Production users
22
PRD TESTING_BUILD

23. What does that mean?
23
● I can Create an Artifact (in the Build Process) and Test it (still in
the Build) by Cloning production and trying to deploy to the
clone.
● I can know 100% that the artifact file that is being deployed in the
TESTING_BUILD clone will be deployed successfully to PRD.
● BOOM.

24. How to Connect to Snowflake
What connector should we use?

25. Snowflake Connectors
● Tons to choose from!

27. SnowSQL (CLI) vs. Python
27
● SnowSQL is a command line interface (CLI) built off the Python
Connector.
● Python cannot run multiple statements.
● SnowSQL can run files with over 100 separate queries and over
10K lines of code.
● Once installed, SnowSQL is super simple to use and easy to
configure.
● We choose SnowSQL!

28. SnowSQL Config File
28
● After installation of SnowSQL, a file called config is created.
○ C:Usersuser.name.snowsqlconfig
● In this file, you can set up connections, variables, and other
configuration settings.
● Connections:

29. SnowSQL Configuration Settings
29
● Helpful config settings:
○ exit_on_error
■ exists the program immediately when there is an error.
■ If we deploy something and it fails. STOP! DROP! And ROLLBACK!
○ output_file
■ Instead of outputting to the terminal, it will write to a log file.
○ sfqid
■ Output the Snowflake Query ID. Awesome for troubleshooting.
○ output_format
■ Change the look of how the output is formatted, grid vs. txt.
○ friendly
■ Snowflake greets you with hello and goodbye messages… But
Snowflake isn’t my only friend, so I don’t need this.

30. Azure DevOps
Planning / Repo / Pipelines all in one place!

31. Azure DevOps.
One Stop Shop(s).  sick rhyme
Boards
With the Azure Boards
web service, teams can
manage their software
projects. It provides a
rich set of capabilities
including native support
for Scrum and Kanban,
customizable
dashboards, and
integrated reporting.
Repo
Azure Repos is a set
of version control
tools that you can
use to manage your
code.
Testing
The testing suite
contains tools for
both exploratory
and manual testing
as well as
automated
continuous testing.
31
Pipelines
Azure Pipelines is a
cloud service that you
can use to
automatically build
and test your code
project and make it
available to other
users. It works with
just about any
language or project
type.

32. End to End
From Developer Typing Code to Deploy into Snowflake Production.

33. 33
How do we
get there?

34. 34
Pieces to the Puzzle
1. Source Control.
2. Put Process into place (PRs).
3. Decide how to build artifacts to deploy.
4. Decide stages and gates.
5. Deploy using SnowSql.

35. Set up your Repo
35
● It is crucial to use source control!
● Benefits:
○ Log of every change to your database for all time.
○ Devs can work on the same objects at the same time.
○ Able to trace all changes in a project management tool
(Azure DevOps boards).

36. Pull Requests
● Created when the developer is ready to promote their code.
● Reviewed by peers, architect, management, etc.
● Benefits:
○ Asynchronous Code Reviews
○ Can add comments and follow up later.
○ Team knows what is going into Prod.
○ No one can merge into master directly!
36

37. Build Pipeline
● Purpose: Build Artifacts!
● Steps:
1. Download latest repo (master).
2. Create Artifact.
3. Install SnowSQL.
4. Clone PRD database.
5. Test Deploy Artifact.
6. Publish Artifact.
37

38. Release Pipeline
38
● Purpose: Deploy Artifact to Stages!
● Steps:
1. Download Artifact.
2. Replace Environment Variables.
3. Install SnowSQL.
4. Deploy Artifact to DEV.
5. Deploy Artifact to TST.
6. Final Approval.
7. Deploy Artifact to PRD.

39. Demo
Bout time, am I right?

40. Other Features we added
● Both Change and Rollback Scripts
● Run in Rollback-only Mode.
● Write all Deployment Logs to Storage Container via Azure CLI.
40

41. Other Features…
● Split CICD Pipeline into two: Lower and Upper.
○ Lower:
■ Deploys to DEV.
■ Used for Devs to test deployments.
■ Can be run on-demand.
○ Upper:
■ Deploys to TST and PRD.
■ Locked Down.
■ Only run by merge into master (PR).
● Next Big Enhancement:
○ Automated Testing!
41

42. 42
Thanks!
Any questions?
Email me: drew.hansen@adesa.com
Connect with me: https://www.linkedin.com/in/drew-hansen-09806289/