Downloaded 55 times
Uploaded byKellyn Pot'Vin-Gorman
DevOps and the DBA
This document discusses DevOps and how it relates to database administrators (DBAs). It begins with a story about data corruption resulting from a lack of formal development processes. It then defines DevOps and discusses how including DBAs is important for efficiency. The document outlines common DevOps terms and tools and how database virtualization fits into the DevOps model. It addresses cultural challenges for DBAs in adopting DevOps and how DBAs can provide value through collaboration, skills updates, and familiarity with the DevOps toolchain.
More Related Content
More from Kellyn Pot'Vin-Gorman
![Getting Started with Apache Spark: Big Data Made Simple [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/apachesparkgettingstarted-260203175547-8361bcc3-thumbnail.jpg?width=640&height=640&fit=bounds)
DevOps and the DBA
- 1. DevOps and theAgile DBA Efficiency Out of Chaos Kellyn Pot’Vin-Gorman | Technical Intelligence Manager for the Office of CTO
- 2. 2© 2017 DelphixCorporation Kellyn Pot’Vin-Gorman Technical Intelligence Manager, Delphix • Multi-platform DBA, (Oracle, MSSQL, MySQL, Sybase, Postgres…..) • Oracle ACE Director, (Alumni) • Oak Table Network • APEX Women in Technology Award, CTA 2014 • STEM education with Raspberry Pi and Python • Board of Director, RMOUG • Training Days Conference Director • Author, blogger, (http://dbakevlar.com)
- 3. This session isan introduction to DevOps for Database Administrators. The focus is not only to discuss the concepts, tools and processes, but cultural changes for DBAs to become more agile in a DevOps world.
- 4. © 2017 DelphixCorporation 4 Agenda Story Time1 What is DevOps2 Agility- Speed is Everything3 Tools of the Trade4 Summary5
- 5. 5© 2017 DelphixCorporation Story Time • Large company • 4TB transactional database, (small by today’s standards) • Financial data, aggregated to other financial systems. • Agile development released, (most often) directly to production, without formal processes, (DevOps) in place. • Archaic development, (1/3 of size of production), rarely, if ever testing outside of production queries.
- 6. 6© 2017 DelphixCorporation After 4-6 Weeks of Research… • Over 40% data corruption to main transactional system that feeds into datamarts. • Even higher percentage of corruption impact in marts due to poor agile development practices and percentage of highly volatile source data. I now have to reveal my findings to senior management as the new Lead DBA…
- 7. 7© 2017 DelphixCorporation “Accuracy is over-rated…”
- 8. 8© 2017 DelphixCorporation Last Invited to the Party… • DevOps is for Development and Operations- where do we, DBAs fit? • When we aren’t included, groups attempt to work around us and the business pays the cost. • Feeling alienated, some are content to promote this scenario. • Results in our education on DevOps lagging behind other groups and high cost to efficiency and productivity.
- 9. 9© 2017 DelphixCorporation https://en.wikipedia.org/wiki/DevOps DevOps DevOps is a short for DEVelopment and IT OPerationS. The term is used to refer to a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment where building, testing, and releasing software can happen rapidly, frequently, and more reliably
- 10. 10© 2017 DelphixCorporation History of DevOps • First called “Agile Infrastructure” in 2008 • 2009, the first time DevOps was coined in Belgium • Has taken over most of the fellow terms of WinOps and in some cases, Agile. • Introduced at relational database events in 2010, but rarely was given much credit until the last five years in the DBA community. https://www.versionone.com/devops-101/what-is-devops/
- 11. 11© 2017 DelphixCorporation Common Terms, (Alphabetically) • Agile • Build Automation • Configuration Management • Continuous Delivery • Functional Testing/Unit Testing • Packages/Containers • Rapid Deploy • Release Coordination/Orchestration • Test Driven Development • Virtualization https://xebialabs.com/glossary/
- 12. 12© 2017 DelphixCorporation DevOps Relationship to Agile • Agile’s success pushed the need for DevOps • Agile and DevOps have similarities, Agile focuses more on culture and DevOps focuses more on organizational changes to create success. • DevOps focuses on the success of releases as the scoring method of success of implementation. We’re VERY close…
- 13. 13© 2017 DelphixCorporation Build Automation Tools or framework that allow source code to be automatically compiled into releasable binaries. Usually includes code-level unit testing to ensure individual pieces of code behave as expected.
- 14. 14© 2017 DelphixCorporation Configuration Management Concept & Development Testing & Verification Operations & Support Deployment to Production
- 15. 15© 2017 DelphixCorporation Continuous Delivery Continuous Delivery is the ability to get changes of all types— including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way. https://continuousdelivery.com/
- 16. 16© 2017 DelphixCorporation Functional Testing/Unit Testing Testing Requirement Analysis Test Planning Traceability Matrix Test Execution Defect Management Coverage Analysis Test Case Design
- 17. 17© 2017 DelphixCorporation Rapid Deploy Software Content Enablement Service Need or Focus Area All Tiers of Specific Environment
- 18. 18© 2017 DelphixCorporation Release Coordination/Orchestration Release Orchestration focuses on the use of tools which manage software releases from the development stage through the deployment to production.
- 19. 19© 2017 DelphixCorporation Test Driven Development Red GreenRefactor TDD Create Test that Fails Update the Test to Succeed Eliminate Redundancy
- 20. 20© 2017 DelphixCorporation Timeline in a Project Waterfall- 5 Releases January February March April Agile with Virtualization- 12 Releases January February March April Database Refresh Development Production Release Testing Cycle Refresh and Retest Skip Test and Release
- 21. 21© 2017 DelphixCorporation Virtualization A technical approach in which users and applications do not use physical machines, but simulated systems running on actual, “real” hardware. Virtualization can be used to eliminate resource usage and enable savings for databases, network, file and application management, along with server infrastructure.
- 22. 22© 2017 DelphixCorporation ▶▶▶ Virtualize and Deployed▶ ▶ ▶ 600GB Many Types of Virtualization Storage Pool for Delphix QA DEV PATCH TEST PRODUCTION Database/App Tier 1 TB 1 TB Read From Production Each Virtual Database takes up around 5-10Gb upon creation, (dependent upon parameters) TEST Read AND Write
- 23. 23© 2017 DelphixCorporation Why Database Virtualization is Different • DBA is “only as good as their last backup”. • Many database tools take considerable time to recover. • DevOps is often about automation- automate an “undo” for development and testing that includes data rewind. • Include a self-service tool that will allow for rewind without DBA intervention.
- 24. 24© 2017 DelphixCorporation Packages/Containers Containers offer the ability to isolate application code and/or the whole infrastructure stack into a package able entity to ease deployment, even from the same kernel. This is a powerful tool for DevOps to ease deployment for complex tiers, applications and multiple data stores.
- 25. 25© 2017 DelphixCorporation “Containerize” for Ease of Delivery Development Delphix Virtualization Engine Same Storage Size as Production NFS iSCSI Testing NFS iSCSI Reporting NFS iSCSI Create “Container” Rsync (UNIX/Linux) Robocopy (Windows) Deployed using Jenkins
- 26. 26© 2017 DelphixCorporation The DevOps Toolchain Code Build Test Package Release Configure Monitor Repeat Areas that will benefit from DBA involvement Code Build Test Package Release Configure Monitor Repeat Areas where involvement will benefit the DBAs career
- 27. 27© 2017 DelphixCorporation A Team Effort Between the following groups: • Development • Management • Business • Database Administration • Testing • Operations • Infrastructure
- 28. 28© 2017 DelphixCorporation Continuous Delivery Involves all these groups and… • Orchestrates processes from different tiers. • Allows for deployments more often and with more success. • Removes latency and adds automation. • Improves communication between all groups involved.
- 29. 29© 2017 DelphixCorporation Goals Are All the Same… • Shorter Delivery Cycles • Higher Success Rate on Releases • Shorten Time to Address Issues • Continued Improvement in Communication • Increased Revenue
- 30. 30© 2017 DelphixCorporation Collaboration Between All Stakeholders • In Agile shops, stakeholders meet on a regular basis. • Those involved in the product development, testing, deployment and support may meet as often as once per day or more. Can you see the value of having the DBA in the room when the requirements, infrastructure, code and other decisions are being made? Yet, we still ask, “What’s in it for me?”
- 31. 31© 2017 DelphixCorporation Cultural Challenges for DBAs • Natural Gatekeepers • Seek Stability • Guardians of company data • Trust archaic processes that ensure success, may not ensure speed of completion. • Viewed as roadblocks.
- 32. 32© 2017 DelphixCorporation Cultural Challenges for Operations • Viewed as security guards • Removers of “waste” • Answer to management and business • Also want stability, (often have similar concerns as the DBA Team)
- 33. 33© 2017 DelphixCorporation Development • Seeking change • Need to meet requirements of the business • Need more access to meet demands and deadlines • Desire the path of least resistance
- 34. 34© 2017 DelphixCorporation Business • Have Needs • Want them fulfilled • Often don’t understand requirements or resource constraints • Pivotal to End User experience and requirement gathering • Often can be the least concerned about security
- 35. 35© 2017 DelphixCorporation Where Can the DBA offer the Most Value? • Update skills to remove bottlenecks • Enhance monitoring utilities that take infrastructure and development into overall view • Recognize a need to become more skilled in scripting and deployment processes • Become familiar with DevOps toolchain DBA DevOpsSkills
- 36. 36© 2017 DelphixCorporation Collaboration Practices/Tools • Methods- Scrum, KanBan, Extreme Programming, Continuous Delivery • Hybrid approaches combine, such as Scrumban and KanPlan • Collaboration- Jira, SlackStorm, Pivotal Tracker, HipChat, Rally, (now Agile Central) • Deployment- RapidDeploy, ElasticBox, JuJu, Otto • Automation- Jenkins, Chef, Ansible • Build Tools- Maven, Gradle, Make, Packer, MSBuild
- 37. 37© 2017 DelphixCorporation Agile Methodologies Scrum- A lightweight PM framework. Benefits include a flexible set of management and project protocols and strong collaboration focus. Lean/Kanban- Strong focus on team and deliverables. Efficiency and ability to see the “whole”. Extreme Programming, (XP)- Now the most popular, with continuous delivery key. Crystal- A lightweight agile framework. A removal of hurdles, including bureaucracy, obstacles and other impediments to the development cycle. Feature Driven Development, (FDD)- client and architecture centric software development, focused on the feature deliverable.
- 38. 38© 2017 DelphixCorporation Example of a Scrum Sprint https://www.scrumalliance.org/community/articles/2014/april/devops-and-agile
- 39. 39© 2017 DelphixCorporation Know Your Tools Incredible variation of tools and knowing tool classifications doesn’t mean that the tools are alike. Many groups are often comparing one feature of a given tool.
- 40. 40© 2017 DelphixCorporation The Periodic Table of DevOps Tools https://xebialabs.com/periodic-table-of-devops-tools/
- 41. 41© 2017 DelphixCorporation The Last Piece- Security!
- 42. 42© 2017 DelphixCorporation Confidential data Production Non- Production Exposure
- 43. 43© 2017 DelphixCorporation As 80% of data in a company are copies, then 80% of data won’t be subject to security like a production environment. Securing this data is not just a priority, but in many cases, subject to legal ramifications, (i.e. PCI/PII) • Masking Requirements • Masking shouldn’t be reversible • The masked data should be representative of the original data type to ensure performance is consistent. • Referential Integrity should be maintained as part of the masking process. Masking should be a simple, repeatable process with a user interface that ensures it is simple. Masking in the Picture
- 44. 44© 2017 DelphixCorporation Do I Have to Mask Data? Nah…. Type of Data Year Passed Ruling Data Masking in the EU 2014 ARTICLE 29 DATA PROTECTION, (GDPR) HIPAA 1996 Health Insurance Portability and Accountability Act PCI 2016, (Updated) Payment Card Industry Standards PII Personably Identifiable Information SOX 2002 Sarbanes-Oxley Act
- 45. 45© 2017 DelphixCorporation Confidential data Production Non- Production Exposure Encryption Masking Solution
- 46. 46© 2017 DelphixCorporation Source/Version Control “A component of software configuration management, version control, also known as revision control or source control, is the management of changes to documents, computer programs, large web sites, and other collections of information.”
- 47. 47© 2017 DelphixCorporation Branching and Bookmarking • The ability to mark each iteration of development with a bookmark • Simplify to lock and deliver while testing a consistent image via a virtual database, (VDB) • If a something goes wrong, the ability to “bookmark”, (and subsequent snapshot) to deliver to development to address.
- 48. 48© 2017 DelphixCorporation DevOps and Source Control- The Delphix Way
- 49. 49© 2017 DelphixCorporation From the DBA Perspective DevOps include the following focus areas for Database Administrators: • Automating repeatable and complex processes • Ensuring the databases and data stores aren’t the bottleneck to the development cycle and critical data is protected. • Locating heterogeneous tools/scripting languages that can support multiple tiers of technology. • Understanding that automation does not lessen your value. • Embracing new skills to add more value to the business and to your career.
- 50. 50© 2017 DelphixCorporation How This All Comes Together… • The DBA, with their knowledge, natural “gate keeper” mindset and area of expertise, should be a pivotal member of the DevOps solution. • There are a number of stakeholders and groups that make up DevOps and DBAs need to find a strong balance between securing the environment and not being viewed as a roadblock. • Understand the terms, the process and methodologies of DevOps to be more integrated into the new age of DevOps! • Collaboration and culture change are always challenging, but the benefit in the end is more productivity, product satisfaction and shorter development cycles.
- 51. Kellyn Pot’Vin-Gorman Technical IntelligenceManager kellyn@delphix.com http://dbakevlar.com
Editor's Notes
- #7 I now have to go tell upper management of my findings
- #8 What I learned from this- When only development is agile or embraces an increased development cycle, it’s not the same as DevOps and this was the type of use case that showed proof of why its needed.
- #10 DevOps derives from both development and operations, groups that DBAs often have a foot in each of. There is a high focus on collaboration, geared on methodologies, process and practice. The goal is to release more frequently, more successfully and with less bugs.
- #11 Agile 2008 conference, Andrew Clay Shafer and Patrick Debois discussed "Agile Infrastructure” The term DevOps was popularized through a series of "devopsdays" starting in 2009 in Belgium
- #13 Agile and DevOps aren’t one in the same, but as it’s well known, DevOps came out of Agile’s success. Agile= culture, where DevOps focuses more on the organization changes.
- #14 Build automation is the process of automating the creation of a software build and the associated processes including: compiling computer source code into binary code, packaging binary code, and running automated tests.
- #15 Configuration management (CM) is a systems engineering process for establishing and maintaining consistency of a product's performance, functional, and physical attributes with its requirements, design, and operational information throughout its life.
- #16 Continuous delivery (CD) is a software engineering approach in which teams produce software ... incremental updates to applications in production. A straightforward and repeatable deployment process is important for continuous delivery.
- #17 Functional testing is a software testing process used within software development in which software is tested to ensure that it conforms with all requirements.Functional testing is a way of checking software to ensure that it has all the required functionality that's specified within its functional requirements. Unit testing is a software development process in which the smallest testable parts of an application, called units, are individually and independently scrutinized for proper operation. Unit testing can be done manually but is often automated.
- #18 A rapid deployment force is a military formation capable of quick deployment of its forces. Such forces typically consist of elite military units (special ops, paratroopers, marines, etc.) and are usually trained at a higher intensity than the rest of their country's military.
- #19 Release Orchestration is the use of tools like XLRelease which manage software releases from the development stage to the actual software release itself.
- #20 Test-driven development (TDD) is a development technique where you must first write a test that fails before you write new functional code. TDD is being quickly adopted by agile software developers fordevelopment of application source code and is even being adopted by Agile DBAs for database development.
- #21 Over 80% of time is waiting for RDBMS, (relational databases) to be refreshed. Developers and Testers are waiting for data to do their primary functions. This allows for faster and less costly migrations to the cloud, too.
- #22 In computing, virtualization means to create a virtual version of a device or resource, such as a server, storage device, network or even a database. The framework divides the resource into one or more execution environments. For data, this can result in a golden copy or source that is used for a centralized location and removal of duplicated data. For read and writes, having unique data for that given copy, while duplicates are kept to singular.
- #23 Point out the engine and size after we’ve compressed and de-duplicated. Note that each of the VDBs will take approximately 5-10G vs. 1TB to offer a FULL read/write copy of the production system It will do so in just a matter of minutes. That this can also be done for the application tier!
- #25 Package software into standardized units for development, shipment and deployment. A container image is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings.
- #26 We refer to a container as a template in our product.
- #27 Why the DBA needs to be part of this? This is a hand-in-hand process. The DBA can provide as much value as DevOps can offer the DBA.
- #28 As the norm, we’re in the center of everything. As data touches so much in the environment, this is expected and why DBAs are pivotal to the success of DevOps
- #29 Releases are continuous, for some companies like Toyota, 100-300 per day. The goal is to remove obstacles and to experience less bugs/issues.
- #34 And this is how we see them, right?
- #37 Methods provide a format or guide to work from. Hybrid approaches often implement best. Collaboration methods ensure that communication continues when team members return to their desks Deployment tools help with documenting and lessons learned Build tools help with automation and orchestration
- #38 Scrum focuses on features, bug fixes and backlog debt. Serves very large teams, including those 800+ Lean’s goal is to eliminate all waste, over demand on resources and ability to deliver faster and more effectively each time. XP is one of the most controversial due to the ability to deliver even to large companies every 1-3 weeks. Very disciplined approach. Crystal is often known under Crystal Clear, Yellow Orange and others.
- #42 First Quarter of 2017
- #44 If the ssn is the reference key, then the numbers should be masked identically across the objects to ensure integrity is maintained.
- #45 Article 29 makes it unlawful in EMEA to not just encrypt, but to mask data in non production systems and when handling data outside of secure environments. HIPAA protects medical information PCI protects payment information, via the internet, inside companies and in the public eye. PII protects personably identify information between systems, (big brother) for demographics and information collections SOX protects investor information
- #47 I’m going to add to this definition with Data version control.
- #48 This is a cornerstone to developers and testers, so as DBAs, we know the pain when a developer comes to us to flashback a database and before that, recover or logically recover, (import or datapump) independent objects. What is The developer/tester could do this for themselves?
- #49 This may appear to be a traffic disaster of changes, but for developers with Agile experience, a “sprint” looks just like this. You have different sprints that are quick runs and merges where developers are working separately on code that must merge successfully at the correct intersection and be deployed. Versioning with source control is displayed at the top, using Virtual images. You can see each iteration of the sprints. In the middle section is the branches of that occur during the development process. A virtual can be spun from a virtual, which means that it’s easier for developers to work from the work another developer has produced. Stopping points and release via a clone is simply minutes vs. hours or days.