XpertSolvers: Your Partner in Building Innovative Software Solutions
The Future of Database Development
1. The Future of Database
Development
(with containers)
Steve Jones
Editor, SQLServerCentral
Redgate Software
2. Agenda
Who am I?
Getting Started on a Database Project
Container Primer
Branching for Experiments
Fixing Mistakes
Easy Continuous Integration
3. Steve Jones
DevOps Advocate, Redgate Software
Editor, SQLServerCentral
30 years SQL Server data experience
DBA, developer, manager, writer, speaker in a variety of
companies and industries
Founder, SQLServerCentral
Currently the editor in chief, with the goal of helping you learn
to be a better data professional every day
13 year Microsoft Data Platform MVP
I have been honored to be recognized by Microsoft for the
as a Data Platform MVP working with SQL Server
@way0utwest
/in/way0utwest www.voiceofthedba.com
sjones@sqlservercentral.com
6. The Database Vision - Getting Started
• I got a new gig…
• They're letting me do web development!!!!
• I don’t want to worry about databases
• I don’t have SQL installed (No services running)
• I have cloned the source code (git clone)
13. What is a container?
• Started in Linux as a lightweight way to provide a full fledged Linux
environment without starting a full VM
• LXC were early Linux containers
• Docker built their own container engine
• Uses the host kernel
• Provides a separated environment for applications
• Virtualizes OS resources
• Namespaces
• Control groups
14. What are Images?
• In the container world, an image is a template of the filesystem
• We build these with layers
• A unified file system combines these layers to give a consistent view
• The layers are read only
• Images are downloaded and used to create an instance of a container
15. Images and Containers
• The container is an instance of the Image
• All containers share the image
• Each container gets a separate writeable layer
• Changes exist in this writeable layer
• The unified file system keeps a consistent view
• Changes made to the writeable layer persist through pause/stop
• A volume may be mounted from the host inside the container
17. Containers Help with Getting Started
• Containers are self-contained
• Everything inside
• Monolithic, or multiple containers linked
• Remove the need for dependencies
• Remove (some) of the need for infrastructure
• Known versions ensure stability/repeatability
• For databases
• Smaller data sets work well inside containers
• Larger data sets are a problem
18. Containers help with Branching
• A typical concern is I want to try something
• However, I don’t want to disturb other’s work
• Branches allow me to experiment and commit separately
• I can discard these, recreate these as needed, which means I want the
database to do the same.
• I can merge in changes that work
19. Containers and Branching
• Containers are immutable
• Restarting a container means that I return to a known state
• I can branch, with a new container and make changes
• Once work is done, I can commit the code to VCS
21. Oops, I did it again
• Often, I make some mistakes in development
• Usually, I can’t remember or don’t know the code to reset the
database
• This is frustrating because app code just resets
23. Recap
• DBaaS removes the need to manage the server component
• Offloads resources elsewhere
• Offers more features
• Containers provide a known, consistent starting point
• Integration with easy deployment as a part of app config
• Reset can speed up testing of data issues
25. Independent Validation with CI
• We want to check changes (committed) by developers
• Avoid the "it works on my machine"
• The CI process should be simple and seamless
26. CI with Containers
• Known state for a database app
• Schema
• (test) data
• Start quickly
• Multiple containers for different data stores can be run in parallel
• Lighter weight than VMs
28. Challenges
• DBaaS
• Subsetting
• We need easy ways to get a partial set of data
• Automation and adaptation to schema changes
• Test Data
• Size (subsetting)
• Scale
• Spread
• Container Orchestration (AKS et al)
29. Summary
• DBaaS is the future
• Containers make development easy(-ier)
• Handle branching easily for experimentation
• Reset immediately
• Parallelize testing with a database
• Lightweight and reduce setup
• DBAs still have control
DevOps is a hot topic in today’s software development world. However most of the knowledge and experience with DevOps is based around application software and ignores the database. We will examine how the concepts and principles of DevOps can be applied to database development by looking at both automated comparison analysis as well as migration script management. Automated building, testing, and deployment of database changes will be shown.
https://unsplash.com/photos/IMUwe-p1yqs
steve
This is a recording of a demo system on my laptop. Notice at the beginning, I don't have any SQL Servers running. In fact, I have no data stores, including PostgreSQL, which this app uses.
I have downloaded a repo prior to this, but I can use the yarn package manager to start my application. Once I do that, the yarn configuration starts to look for containers, doesn't find them and creates them. Notice there are MSSQL, SQL Server, images and PostgreSQL images, These are used to create new containers.
This does take a little time, as this is a proof of concept, running in the UK, and not a lot of resources are allocated. Once up, notice that credentials and connection strings are returned. These are plain text, but they are also generated for this container, for this user.
Once we have containers created, there is an upgrade, as migrations, or new SQL Scripts in the repository, are applied to the databases. Flyway is used to check the scripts and apply them to the schemas in the databases. Both SQL Server and PostgreSQL databases are migrated.
As migrations are finished, the app then is started, and a URL returned. I can then switch to a browser and see the app running. I can connect, and begin working. Each time I start the app, the container status is checked, and if they are available, they are reused.
If I need to reset my data store for some reason, say I removed or changed data, I can just remove a container with a command line call and a new one will be recreated the next time the app starts. If I add new SQL code for objects in a file, the new migration script is executed the next time the app starts up.
If I branch, I get a new set of containers. If I switch back to my old branch, the old containers are there. Each one starting from the same, immutable image.
At the end, we see this is from the Spawn project, from Redgate, which is a beta proof of concept for doing database development with the Database as a Service.