Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DevOps for the Database

162 views

Published on

Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2SH1krR.

Baron Schwartz explores real-life stories that answer two questions: “Why is it hard to apply DevOps principles and practices to databases, and how can we get better at it?” He covers topics including: what the research shows about DevOps, databases, and company performance; current & emerging trends in building and managing data tiers; the traditional dedicated DBA role, and more. Filmed at qconsf.com.

Baron Schwartz is the CTO and founder of VividCortex. He has written a lot of open source software, and several books including High Performance MySQL. He’s focused his career on learning and teaching about scalability, performance, and observability of systems generally (including the view that teams are systems and culture influences their performance), and databases specifically.

Published in: Technology
  • Be the first to comment

DevOps for the Database

  1. 1. DevOps for the Database Baron Schwartz • QConSF 2018
  2. 2. InfoQ.com: News & Community Site • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ devops-database
  3. 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
  4. 4. Introduction I’ve been focused on databases for about two decades, rst as a developer, then a consultant, and now a startup founder. I’ve written several books (High Performance MySQL)… and created a lot of open source software, mostly focused around database monitoring, database operations, and database performance: innotop, Percona Toolkit, Percona Monitoring Plugins among others. @xaprb 2
  5. 5. Agenda Overview of database DevOps How companies succeed Tooling, culture, process, people How companies fail Challenges to database DevOps Resources, etc Slides are online at xaprb.com. My Twitter is @xaprb, my email is baron@vividcortex.com @xaprb 3
  6. 6. Three Database DevOps Stories 1. One DBA, hundreds of developers, growing the DBA team @xaprb 4
  7. 7. Three Database DevOps Stories 1. One DBA, hundreds of developers, growing the DBA team 2. One DBA, from 20 to 100 developers @xaprb 4
  8. 8. Three Database DevOps Stories 1. One DBA, hundreds of developers, growing the DBA team 2. One DBA, from 20 to 100 developers 3. Two database ops folks, seventeen developers @xaprb 4
  9. 9. Benefits of Database DevOps DevOps brings the same bene ts to the database as everywhere else. @xaprb 5
  10. 10. Benefits of Database DevOps DevOps brings the same bene ts to the database as everywhere else. For software delivery performance in particular: Faster, better, cheaper—pick all three stability -> speed -> stability -> speed (See the 2018 State of DevOps Report!) @xaprb 5
  11. 11. Detriments of Lacking DevOps Without DevOps, speed and quality suffer: Mismatched responsibility and authority Overburdened database operations personnel Broken feedback loops from production Reduced developer productivity @xaprb 6
  12. 12. What Is Database DevOps? Attributes I’ve seen in companies that apply DevOps to their database: Developers own database schema, workload, and performance Developers debug, troubleshoot, and repair their own outages Schema and data model as code A single fully-automated deployment pipeline App deployment includes automated schema migrations Automated pre-production refresh from production Automation of database operations, to an RDS-like level @xaprb 7
  13. 13. From the 2018 State of DevOps Report Database changes are often a major source of risk and delay when performing deployments… integrating database work into the software delivery process positively contributed to continuous delivery… good communication and comprehensive con guration management that includes the database matter. Teams that do well at continuous delivery store database changes as scripts in version control and manage these changes in the same way as production application changes… when changes to the application require database changes, these teams discuss them with the people responsible for the production database Emphasis mine. See p. 57 @xaprb 8
  14. 14. Bringing DevOps to the Database
  15. 15. Core Elements 1. People 2. Culture 3. Structure and Process 4. Tooling @xaprb 10
  16. 16. Tooling: Deploy/Release You need frequent, automated deploys Eliminate manual work (toil) Continuous integration and deployment @xaprb 11
  17. 17. Tooling: Monitoring and Observability Instrumentation, telemetry, analytics, monitoring, observability Monitoring: the Seven Golden Signals (CELT + USE) Observability is built on these foundations @xaprb 12
  18. 18. Tooling: Shared Knowledge and Process DBRE processes, mentality, rubrics Deploy con dence procedures Documentation to share SME experience & skill Noti cations should link to runbooks @xaprb 13
  19. 19. Structure: Team Orientation Teams work best when they: Are service- or product-oriented Are loosely coupled, autonomous Are highly aligned and trusted Own what they build My personal experience: Conway’s Law is true. @xaprb 14
  20. 20. Process: First, Do No Harm Stabilize the patient, then transport Protect production (no holes below the waterline) @xaprb 15
  21. 21. Process: Plan And Roadmap Work is work—maintain a single backlog Embrace DevOps in small chunks and build on success Lay out the progression in stages Emphasize what’s not changing, too @xaprb 16
  22. 22. Process: Getting Started Pick a place to start. One team One app, service, or product First new, then established/legacy @xaprb 17
  23. 23. Culture: Change Culture is emergent; you can’t operate on it directly Create a new path of least resistance Include everyone in consequences and bene ts @xaprb 18
  24. 24. Culture: Leadership Support Exec mandate sometimes works; so does attraction Starve the old way Leadership isn’t just top of org chart @xaprb 19
  25. 25. Culture: Communication and Trust Align around a North Star: a simple, compelling why Customer-centric culture, customer empathy Psychological safety enables risk-taking @xaprb 20
  26. 26. People: You Need Experts You do need database expertise You do not need a database caretaker Engineers can learn database competency @xaprb 21
  27. 27. Pathways To Failure
  28. 28. Tooling FAIL Fragile or too-ambitious automation Lack of automation / accepting manual toil Two routes to production—code vs DB @xaprb 23
  29. 29. Culture FAIL Any friction in the way of change Failure to create incentives to change Relying on a vendor to bring culture Insisting on adherence to One True Way Clinging to legacy DBA roles and duties @xaprb 24
  30. 30. Aside: Legacy DBA Roles In a traditional sense, the job of the DBA means she is the only person with access to the servers that host the data, the go-to person to create new database cluster for new features, the person to design new schemas, and the only person to contact when anything database related breaks in a production environment. — Silvia Botros, SendGrid @xaprb 25
  31. 31. Leadership FAIL Underinvesting in experience and skill Lack of management support Micromanagement, failure to manage up See also my Kafka Summit talk on advocating technical decisions. @xaprb 26
  32. 32. Planning FAIL All-or-nothing Too much too fast Velocity over resilience @xaprb 27
  33. 33. What’s The Hardest Part?
  34. 34. Challenge: Politics Selling DevOps to the DBA Selling DevOps to leadership Creating culture change @xaprb 29
  35. 35. Challenge: Tooling Legacy databases aren’t cloud-native Data tier ops tooling isn’t as mature Schema changes Integration with e.g. canary deploys, feature ags @xaprb 30
  36. 36. Challenge: HA, Scale, Performance Failover and recovery Locking/blocking Nonblocking schema changes at scale @xaprb 31
  37. 37. Whither The DBA? Become a DBRE instead of a DBA Focus on data platform and architecture Be the subject matter expert supporting product teams @xaprb 32
  38. 38. The Rewards The outcomes The process itself Individual bene ts @xaprb 33
  39. 39. Acknowledgments Many people contributed to this talk. Thank you. (When I get their OK I will add their names.) @xaprb 34
  40. 40. Slides and Contact Information Slides are at https://www.xaprb.com/talks/ or you can scan the QR code. Contact: baron@vividcortex.com, @xaprb @xaprb 35
  41. 41. Appendix: Survey I asked my Twitter followers to respond to an informal, nonscienti c survey. The following two slides summarize some of the quantitative feedback. @xaprb 36
  42. 42. Survey Results How important are each of the following in your view of “Database DevOps”? Automation Is it important to be DevOps-y with databases? Visibility and measurement CI, CD, etc Tight feedback loops Developers own DB performance of their code Avoiding specialized roles/privileges/human bottlenecks Autonomy Infrastructure as code Shared ownership, alignment 0 15 30 45 60 Very important Somewhat important Not important @xaprb 37
  43. 43. Survey Results Cont’d How do you rate the importance of these factors in making progress? Cultural change was required Buy-in from engineers Tooling Individuals championing the cause Leadership from above Databases are hard to DevOps 0 15 30 45 60 Very important Somewhat important Not important @xaprb 38
  44. 44. Resources Slides/Video: How to Monitor a Database including the Seven Golden Signals (CELT+USE) The Phoenix Project Database Reliability Engineering The 2018 DORA State of DevOps Report Three Steps To Psychological Safety My Kafka Summit talk on advocating technical decisions @xaprb 39
  45. 45. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ devops-database

×