Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DevOps for the DBA

439 views

Published on

Moving from manual processes for development and deployment to a more automated approach requires a great deal of work and knowledge. In this all day seminar we will go through the steps to help you along this journey.

We will start with understanding how source control works and end with automated deployments across environments
You’ll learn about processes and tools that not only make it easier and faster to move database changes, but add protection to your production information.
We will discuss tools, process and the fundamental changes in culture necessary to take your database development and deployment into a high functioning DevOps team.

Published in: Technology
  • Be the first to comment

DevOps for the DBA

  1. 1. DEVOPS FOR THE DBA GRANT FRITCHEY PRODUCT ADVOCATE REDGATE SOFTWARE
  2. 2. GOALS • Discuss tools, process and the fundamental changes in culture necessary to make your database development and deployment a part of a high functioning DevOps team
  3. 3. scarydba.com grant@scarydba.com @gfritchey Grant Fritchey youtube.com/c/GrantFritchey
  4. 4. WHAT IS DEVOPS?
  5. 5. DEVOPS • Gartner: Devops represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, lean practices in the context of a system- oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology – especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective.
  6. 6. DEVOPS • Gartner: Devops represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, lean practices in the context of a system-oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology – especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective.
  7. 7. DEVOPS • Gartner: Devops represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, leanpractices in the context of a system-oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology – especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective.
  8. 8. DEVOPS • Gartner: Devops represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, leanpractices in the context of a system-oriented approach. DevOps emphasizes people(and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology – especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective.
  9. 9. DEVOPS • Gartner: Devops represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, leanpractices in the context of a system-oriented approach. DevOps emphasizes people(and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology – especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective.
  10. 10. DEVOPS • Gartner: Devops represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, leanpractices in the context of a system-oriented approach. DevOps emphasizes people(and culture), and seeks to improve collaborationbetween operations and development teams. DevOps implementations utilize technology – especially automation toolsthat can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective.
  11. 11. DEVOPS •An agile culture that better supports collaboration between people that uses automation and tooling in support of process.
  12. 12. WHO IS IN DEVOPS
  13. 13. WHO IS IN DEVOPS
  14. 14. WHAT DEVOPS IS NOT
  15. 15. WHY DEVOPS? Improved communications Increasing efficiency Faster development Fewer failures Lower costs
  16. 16. DEVOPS ADOPTION
  17. 17. WHY DEVOPS FOR THE DBA?
  18. 18. HOLDING BACK THE OCEAN Applying Versions and Patches Missing Backups Bad Code Incorrect Structures Misconfigured Servers Change You!
  19. 19. SOME JUSTIFICATION
  20. 20. FUD IS THE PROBLEM DevOps only works for new systems ‘SA’ for developers NoOps DevOps is incompatible with security and compliance Agile means no design Elimination of process
  21. 21. PROTECTION
  22. 22. DBA INTEGRATION
  23. 23. DATA PAIN POINTS
  24. 24. WHY DEVOPS FOR THE DBA Improved communications Better integration with dev, ops and the business Added protections through superior processes Better support for the organization Faster and safer deployments
  25. 25. A CULTURE OF COMMUNICATION
  26. 26. DEVOPS FUNDAMENTALS Communication is everything All code in source control Shift Left Automated Tests Automated Deployments Monitoring
  27. 27. COMMUNICATION IS CORE TO DEVOPS Eliminating barriers Mitigate, reduce, and eliminate choke points Successful implementation of DevOps involves all the teams
  28. 28. ORGANIZATIONAL CULTURE “We can’t change” “We’ve always done it this way” “Fine, until something goes wrong” “As long as it doesn’t impact the schedule”
  29. 29. CULTURAL SHIFT Transparency What do you do? How do you do it? Write it all down Collaboration Tear down the wall We’re all on the same team Clarity Definitions matter Language matters
  30. 30. MANAGEMENT WITHOUT DIRECT MANAGEMENT BUY-IN AND SUPPORT, THIS ALL COLLAPSES
  31. 31. DEVOPS FOR THE DBA
  32. 32. DEVOPS FUNDAMENTALS Communication is everything All code in source control Shift Left Automated Tests Automated Deployments Monitoring
  33. 33. SOURCE CONTROL IS A BACKUP Every database is code More backups is better Recovery is hard An “undo” button in a database
  34. 34. “SHIFT LEFT” ADDS PROTECTION EARLIER TESTING FASTER FAILURE FREQUENCY OF DEPLOYMENT SELF-SERVICE MEANS LEARNING
  35. 35. AUTOMATED TESTING PROTECTS PRODUCTION • When something breaks deploying in dev, build a test • When something breaks deploying in CI, build a test • When something breaks deploying to QA, build a test • Every test improves protection for production • Testing combined with “shift-left” improves production safety
  36. 36. FREQUENT DEPLOYMENTS MEANS PRACTICE No script should run first in production Every deployment is a test Practice is fundamental to mastery “Shift-left” makes practice deployments mean more All this adds to protection for production
  37. 37. MONITORING ENSURES BEHAVIOR Understand behavior prior to deployment Know when deployments occur Understand behavior after a deployment Monitoring drives innovation Protection starts with monitoring
  38. 38. DEVOPS FUNDAMENTALS Communication is everything All code in source control Shift Left Automated Tests Automated Deployments Monitoring
  39. 39. DEFINING YOUR PROCESS
  40. 40. SOME CAVEATS No two systems are the same What works for me, may not work for you Compliance drives some decisions Business needs drive others
  41. 41. DATABASE CHALLENGES DATA PERSISTENCE SIZE OF DATA VOLUME OF CHANGES LACK OF MONITORING
  42. 42. BUILDING BLOCKS Environments Databases Applications Data movement Compliance
  43. 43. ENVIRONMENTS Development Laptop Shared Development Development Integration Continuous Integration Quality Assurance Testing Financial Testing Performance Testing Pre- Production Staging Production
  44. 44. DATABASES One database or many Linked servers Replication Availability Groups Data size
  45. 45. APPLICATONS One application or many Linked or unlinked deployments Linked or unlinked testing Deployment methodologies
  46. 46. DATA MOVEMENT Shift Left SSIS Replication Compliance Analysis Reporting
  47. 47. COMPLIANCE Test data Development data Clean data Generated data Hybrid
  48. 48. BUILDING A DEPLOYMENT PROCESS WORK BACKWARDS FROM PRODUCTION IF IT’S NOT IN SOURCE CONTROL, IT DOESN’T EXIST ENSURE COMPLIANCE IS ALWAYS COVERED TEST WHAT BREAKS
  49. 49. BUILDING A DEPLOYMENT PROCESS Production deployment script Test script on copy of production Generate script for production from trunk Merge code in source control Add changes to source control Make changes to code Get code from source control Branch code in source control
  50. 50. TOOLS IN SUPPORT OF AUTOMATION
  51. 51. OPERATING SYSTEMS WINDOWS LINUX
  52. 52. LANGUAGES SQL POWERSHELL JAVA/JAVASCRIPT BASH
  53. 53. CONTAINERS OPEN CONTAINER INITIATIVE (OCI) DOCKER
  54. 54. NATIVE TOOLS VISUAL STUDIO SQL SERVER MANAGEMENT STUDIO AZURE DATA STUDIO
  55. 55. FLOW CONTROL CHEF JENKINS AZURE DEVOPS OCTOPUS TEAMCITY GITLAB READYROLL
  56. 56. SOURCE CONTROL Subversion/SVN Git Azure DevOps Team Foundation Services AWS Code Commit Helix Core
  57. 57. DATABASE TOOLING, THE MAGIC Microsoft SQL Server Data Tools Redgate SQL Change Automation DBMaestro DbUp Datical Apex/Idera DevOps Flyway
  58. 58. HOW THE MAGIC WORKS State • Compare between database & database • Compare between database & source control • Compare between ? & ? Migrations • Scripts build on scripts • Which build on scripts Hybrid • Mix & match
  59. 59. EXERCISE
  60. 60. SCENARIO ONE • We have a single database under development, but we use linked servers for reporting. How do we: • A) Automate our processes across multiple environments • B) Prevent use of production data in linked servers
  61. 61. SCENARIO TWO • We have multiple teams developing within a single database. How do we: • A) Support completely independent development and release cycles • B) Ensure that functionality from one team doesn’t affect another • C) Get each teams functionality to the other teams
  62. 62. SCENARIO THREE • Our system consists of four different, interrelated databases. Each one is on it’s own for development and release, but, we need to have all four while we develop. How do we: • A) Keep four databases on every developers machine • B) Ensure that different development teams don’t break others code
  63. 63. SCENARIO FOUR • We’re pretty happily chugging along with our DevOps processes, however, since moving to Azure SQL Database and implementing automated tuning, we’re seeing indexes in production that don’t exist elsewhere. How do we: • A) See what has changed prior to deployment • B) Ensure that we can incorporate these changes into our own processing
  64. 64. SOURCE CONTROL
  65. 65. SOURCE CONTROL SOFTWARE THAT ENABLES DEVELOPMENT THROUGH COLLABORATION AND CHANGE TRACKING
  66. 66. SOURCE CONTROL Subversion/SVN Git Azure DevOps Team Foundation Services AWS Code Commit Helix Core
  67. 67. SOURCE CONTROL FUNCTIONALITY Keep track of code changes Maintain a history of changes Multiple people working simultaneously Isolate code through branching
  68. 68. SOURCE CONTROL FUNCTIONALITY Merge code from different branches on command Show conflicts on merges in order to resolve them Allow reversion to a previous state Works within your toolset
  69. 69. GIT Local version control Central version control Branch/Merge core design concepts Very broad adoption and support It just works
  70. 70. GIT TERMINOLOGY Master Branch Pull/Sync Commit Merge Push Pull Request
  71. 71. GITFLOW • Sourced: A successful Git branching model
  72. 72. GITHUB FLOW • Sourced: Git Workflows That Work
  73. 73. MY SUGGESTION master Online Feature Branch Local Feature Branch Pull/Sync Commit after testing Push Push Pull Request and Merge
  74. 74. A SHORT DISCUSSION ON DATABASE BRANCHING • You can’t!
  75. 75. A SHORT DISCUSSION ON DATABASE BRANCHING ONE DATABASE PER BRANCH ON AN INSTANCE ONE INSTANCE PER BRANCH ONE CONTAINER PER BRANCH JUST BE READY TO REBUILD DATABASES AS NEEDED
  76. 76. DEMONSTRATION
  77. 77. AUTOMATED TESTING
  78. 78. WHY TESTING Things go wrong Mistakes are made Complexity leads to error
  79. 79. GENERAL TESTING PRACTICES TEST EARLY AND OFTEN TESTING STARTS IN DEVELOPMENT SLOWLY ADD TESTS OVER TIME VALIDATE STANDARDS DOCUMENT YOUR INTENTIONS
  80. 80. HOW DO WE TEST? Automatically Define a known starting state • Generated test data • Modified production data • Always work within Compliance requirements Test data must be provided
  81. 81. WHAT DO WE TEST WITH? tSQLt Microsoft Unit Testing Projects DbUnit Custom Scripts
  82. 82. WHAT DO WE TEST? CODE COVERAGE DATA MOVEMENT THINGS THAT BREAK
  83. 83. DEMONSTRATION
  84. 84. AUTOMATING DEPLOYMENTS
  85. 85. WHAT ARE WE AUTOMATING Getting code into source controlGetting Retrieving code from source controlRetrieving Generating an artifact (aka SQL)Generating
  86. 86. DATABASE TOOLING, THE MAGIC Microsoft SQL Server Data Tools Redgate SQL Change Automation DBMaestro DbUp Datical Apex/Idera DevOps Flyway
  87. 87. STATE VS. MIGRATIONS State • Compare between database & database • Compare between database & source control • Compare between ? & ? Migrations • Scripts build on scripts • Which build on scripts Hybrid • Mix & match as needed
  88. 88. STATE: STRENGTHS & WEAKNESSES Strengths • Simple to understand • Merge is easy • Tools generate scripts • Visible history Weaknesses • Domain of problems not solved • Tool ordering of changes can be problematic • Tool choices for changes can be an issue
  89. 89. MIGRATIONS: STRENGTHS & WEAKNESSES Strengths • All changes work • Control over method and changes • All domain problems handled Weaknesses • Harder to understand and work with • History is hard to assemble • Merges may be difficult • All changes in Dev occur in Prod
  90. 90. THE NEED FOR AN ARTIFACT Separate the concept of a build from a deploy Build generates an artifact Deploy uses the artifact to create/update a database Automate both
  91. 91. CONTINUOUS INTEGRATION FREQUENT BUILDS FREQUENT DEPLOYMENTS FREQUENT TESTS FAIL FAST FOR AN IMMEDIATE FEEDBACK LOOP
  92. 92. CONTINUOUS INTEGRATION • Get your code into source control • Pick a tool for the build • Pick a flow control tool • Build your CI process • You’ve now started down the DevOps path
  93. 93. FLOW CONTROL CHEF JENKINS AZURE DEVOPS OCTOPUS TEAMCITY
  94. 94. DEMONSTRATION
  95. 95. CONTINUOUS DELIVERY
  96. 96. CONTINUOUS DELIVERY DELIVER FUNCTIONALITY INTO PRODUCTION QUICKLY, SAFELY, AND CONSTANTLY
  97. 97. ENVIRONMENTS Development Laptop Shared Development Development Integration Continuous Integration Quality Assurance Testing Financial Testing Performance Testing Pre- Production Staging Production
  98. 98. FLOW CONTROL CHEF JENKINS AZURE DEVOPS OCTOPUS TEAMCITY
  99. 99. ENVIRONMENTAL DIFFERENCES SECURITY COMPLIANCE SECONDARY PROCESSES
  100. 100. PRODUCTION DRIFT There will ALWAYS be production fixes Prod changes must go to VCS Drift may block deployments Tooling detect drift Track versions of the db yourself Drift detection is needed
  101. 101. ROLLBACKS Once changes committed, can't be rolled back Old version of code Be prepared Pre-write rollback scripts Have old code versions ready (snapshot production) Be ready to restore
  102. 102. ROLLING BACK CODE The previous version of procs/functions/views can be used Double check logical function with the rest of the application Can we roll back independently? Is application code rollback needed? Previous version from production Previous released version from VCS Auto-generate rollback code
  103. 103. ROLLBACK TABLE AND DATA CHANGES Undoing data changes is complex and risky Pre-write scripts Include DML changes Test in intermediate environments May need to execute manually Copy old data prior to changes Use scripts to restore old data
  104. 104. ROLL FORWARD Faster • Faster than a rollback Safer • Safer than a rollback Easier • Easier than a rollback Harder • Harder to convince management to support
  105. 105. POST RELEASE Any deployment impacts the system Lack of impact is information Monitoring provides a baseline of resource usage Compare the baseline to new resource and business use
  106. 106. DEMONSTRATION
  107. 107. EXERCISE
  108. 108. SCENARIO ONE • We have a single database under development, but we use linked servers for reporting. How do we: • A) Automate our processes across multiple environments • B) Prevent use of production data in linked servers
  109. 109. SCENARIO TWO • We have multiple teams developing within a single database. How do we: • A) Support completely independent development and release cycles • B) Ensure that functionality from one team doesn’t affect another • C) Get each teams functionality to the other teams
  110. 110. SCENARIO THREE • Our system consists of four different, interrelated databases. Each one is on it’s own for development and release, but, we need to have all four while we develop. How do we: • A) Keep four databases on every developers machine • B) Ensure that different development teams don’t break others code
  111. 111. SCENARIO FOUR • We’re pretty happily chugging along with our DevOps processes, however, since moving to Azure SQL Database and implementing automated tuning, we’re seeing indexes in production that don’t exist elsewhere. How do we: • A) See what has changed prior to deployment • B) Ensure that we can incorporate these changes into our own processing
  112. 112. BONUS SCENARIO • We’re using state-based delivery of our databases. Because we have multiple development streams, we have multiple QA systems. We have an emergency release. How do we: • A) Manage getting our code from dev to QA to preprod to production
  113. 113. WRAP UP
  114. 114. DEVOPS FUNDAMENTALS Communication is everything All code in source control Shift Left Automated Tests Automated Deployments Monitoring
  115. 115. DEVOPS FUNDAMENTALS One Team More Backups Added Protection Automated Protection Practice Deployments Monitoring
  116. 116. GOALS • Discuss tools, process and the fundamental changes in culture necessary to make your database development and deployment a part of a high functioning DevOps team
  117. 117. RESOURCES • State of Database Devops 2019 • The Phoenix Project • Redgate University • Microsoft Learning Resources for DevOps • Microsoft DevOps Resource Center • Steve Jones • Kendra Little
  118. 118. scarydba.com grant@scarydba.com @gfritchey Grant Fritchey youtube.com/c/GrantFritchey
  119. 119. PHOTO CREDITS • Programmer • Cat and Dog • Operations • Dba running with scissors • Crowd/Everyone • Peggy in the office • Culture of Communication • Tools • Management • Rube Goldberg • Exercise • Hope • Source control (safe) • Demonstration • Automatic • Delivery • Wrapping

×