Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

New DevOps for the DBA

1,074 views

Published on

ECO 2017 Updated DevOps for the DBA Session

Published in: Technology
  • Be the first to comment

New DevOps for the DBA

  1. 1. © 2017 Delphix. All Rights Reserved. Private and Confidential.© 2017 Delphix. All Rights Reserved. Private and Confidential. Kellyn Pot’Vin-Gorman | Technical Intelligence Manager| October, 2017 The Revolution DevOps to DataOps
  2. 2. © 2017 Delphix. All Rights Reserved. Private and Confidential. 2 Kellyn Pot’Vin-Gorman Technical Intelligence Manager, Delphix • Multi-platform DBA, (Oracle, MSSQL, MySQL, Sybase, Postgres…..) • Oracle ACE Director, (Alumni) • Oak Table Network • APEX Women in Technology Award, CTA 2014 • STEM education with Raspberry Pi and Python • President, Rocky Mtn Oracle User Group • Liaison, Denver SQL Server User Group • DevOps author, instructor and presenter. • Author, blogger, (http://dbakevlar.com)
  3. 3. © 2017 Delphix. All Rights Reserved. Private and Confidential. 3 • https://en.wikipedia.org/wiki/DevOps The Source DevOps is a short for DEVelopment and IT OPerationS. The term is used to refer to a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment where building, testing, and releasing software can happen rapidly, frequently, and more reliably
  4. 4. © 2017 Delphix. All Rights Reserved. Private and Confidential. The Life of a DBA  Database management, automating where able  Provide environments and assist with projects  Secure database environments  Optimize data access  Collaborate to solve business challenges
  5. 5. © 2017 Delphix. All Rights Reserved. Private and Confidential. The Life of a DevOps Engineer  Automation through scripting  Orchestration between environments, including data sources  Security  Optimize projects, improving on each pass  Collaborate and Communicate
  6. 6. © 2017 Delphix. All Rights Reserved. Private and Confidential. 6 Tools Empower? • DevOps requires agility • Agility requires more automation and tools to ease demands • Even the history of DevOps includes tools • What tools are part of DevOps for the DBA?
  7. 7. © 2017 Delphix. All Rights Reserved. Private and Confidential. 7 How DBAs Risk the Change to DevOps • Database Administrators heavily influence decisions. • We’re sometimes identified with manual processing as a way to measure our value. • Introduction of the cloud, especially and SaaS, (Software as a Service) has been seen as competitive to our role. Still, we’re essential to a successful change, or many are working around us…
  8. 8. © 2017 Delphix. All Rights Reserved. Private and Confidential. 8 Why Should We Master Tools? • We Should be part of the development process from the beginning • We perform code reviews • We check in and release code • We write code, too. • We create automation scripts. • We monitor and optimize
  9. 9. © 2017 Delphix. All Rights Reserved. Private and Confidential. Tools of the Development • Oracle SQL Developer • DBArtisan • Redgate SQL Toolbelt and SQL Prompt • SQL Automation PL/SQL Developer
  10. 10. © 2017 Delphix. All Rights Reserved. Private and Confidential. 10 DevOps Terms • Build Automation • Configuration Management • Continuous Delivery/Rapid Deploy • Functional Testing/Unit Testing/Test Driven Development • Monitoring/Alerting • Packages/Containers/Virtualization • Release Coordination/Orchestration https://xebialabs.com/glossary/
  11. 11. © 2017 Delphix. All Rights Reserved. Private and Confidential. 11 Method to this Madness • Identify Category/Methodology • Identify tools • Some tools will bridge more than one category or methodology • Dig into unique or interesting specifics about the tool
  12. 12. © 2017 Delphix. All Rights Reserved. Private and Confidential. 12 Build Automation • Maven • Gradle • Packer • Cmake • ANT DBAs are a natural fit, due to their history with scripting and automation experience for backups, monitoring and alerting. Build links: Maven, Gradle, Make, Packer, MSBuild
  13. 13. © 2017 Delphix. All Rights Reserved. Private and Confidential. 13 MS Build MS Build 15 came out in March 2017 and part of Visual Studio • No longer installed in the Global Assembly cache. • XML based • Project file made up of items, properties and tasks- and of course, targets.
  14. 14. © 2017 Delphix. All Rights Reserved. Private and Confidential. Continuous Delivery DBAs are much slower to embrace continuous delivery. The risk is high, with it’s “fail first, fail fast” approach. Yet we are its greatest asset.
  15. 15. © 2017 Delphix. All Rights Reserved. Private and Confidential. 15 Jenkins for CI/CD Jenkins is an open source automation server- • Easiest executed via a jetty servlet • Has an easy to access URL, (commonly host:8080) • Automation is done via individual projects. • Has numerous plugins to support environments • Supports multiple scripting languages, including shell, batch commands and Ant scripts. • Supports robust revision control and compatible with version control systems like CVS and Subversion. • Supports notifications like qmail and sendmail.
  16. 16. © 2017 Delphix. All Rights Reserved. Private and Confidential. 16 Jenkins Groovy Script Example // prepare variables for subscript execution: String subscript = "cdmserver-monitor-instance-status.groovy"; println "executing subscript ${subscript}" // variables to be passed to the subscript status = (action == "start" ? "started" : "stopped") def s = ScriptlerConfiguration.getConfiguration().getScriptById(subscript) File scriptSrc = new File(ScriptlerManagment.getScriptDirectory(), s.getScriptPath());
  17. 17. © 2017 Delphix. All Rights Reserved. Private and Confidential. 17 Jenkins Plugin for Delphix https://github.com/delphix/delphix-jenkins-plugin
  18. 18. © 2017 Delphix. All Rights Reserved. Private and Confidential. 18 Configuration Management Concept & Development Testing & Verification Operations & Support Deployment to Production
  19. 19. © 2017 Delphix. All Rights Reserved. Private and Confidential. 19 Ansible • Learn about Playbooks • Powerful collection of steps, (aka plays) • Plays consist of a list of tasks • Handlers allow for remote operations • YAML Syntax • Security configurations based on hosts and users • Ability to run as secondary users, (privilege escalation) • Huge support community and best practices
  20. 20. © 2017 Delphix. All Rights Reserved. Private and Confidential. 20 Example of an Ansible Call - name: Transfer and execute a script. hosts: server remote_user: test_user sudo: yes tasks: - name: Transfer the script copy: src=test.sh dest=/home/test_user mode=0777 - name: Execute the script command: sh /home/test_user/test.sh
  21. 21. © 2017 Delphix. All Rights Reserved. Private and Confidential. 21 Puppet • Review, automate, deploy and invest in all your software. • Control and enforce consistency across environments and platforms. • Share, test and enforce changes on-premise and in the cloud. Similar to Puppet • Chef • Jenkins
  22. 22. © 2017 Delphix. All Rights Reserved. Private and Confidential. 22 Rapid Deploy Software Content Enablement Service Need or Focus Area All Tiers of Specific Environment
  23. 23. © 2017 Delphix. All Rights Reserved. Private and Confidential. 23 Release Coordination/Orchestration A demand for release tools and orchestration was at times a direct result to DBAs hesitation to risk production with a release. This helped automate and bypass the DBA to get code and features into production. • Docker Hub • Artifactory • Zephyr • Octopus Deploy Deployment Links- RapidDeploy, ElasticBox, JuJu
  24. 24. © 2017 Delphix. All Rights Reserved. Private and Confidential. 24 Octopus Deploy • Categorized as Continuous Integration Tool • Release and deploy • Promote to subsequent environments • Cross tier/application, etc.
  25. 25. © 2017 Delphix. All Rights Reserved. Private and Confidential. Automation and Orchestration, DB Style • Idera SQL Admin Toolset • Redgate DML Automation • Automic • CA Automation
  26. 26. © 2017 Delphix. All Rights Reserved. Private and Confidential. Source Control and Repositories • Git • Bitbucket • GitLab • Subversion • Mercurial
  27. 27. © 2017 Delphix. All Rights Reserved. Private and Confidential. 27 GitHub Git is an open-source version control system that was started by Linus Trovalds, (the creator of Linux) • If you aren’t using it now, start. Yes, as a DBA… • Great option to highlight your scripting work GitHub Alternatives • Subversion • CVS • Mercurial
  28. 28. © 2017 Delphix. All Rights Reserved. Private and Confidential. 28 Creating a GitHub Repository Project
  29. 29. © 2017 Delphix. All Rights Reserved. Private and Confidential. 29 Github Repository Example
  30. 30. © 2017 Delphix. All Rights Reserved. Private and Confidential. 30 Source/Version Control, but Data? We all know about version control with code, but how are we, as DBAs doing version control with data? This is a concept that hasn’t been discussed much and DataOps is the new way of bridging some of this challenge.
  31. 31. © 2017 Delphix. All Rights Reserved. Private and Confidential. 31 The Goals of All Tools are the Same • Shorter Delivery Cycles • Higher Success Rate on Releases • Shorten Time to Address Issues • Continued Improvement in Communication • Increased Revenue
  32. 32. © 2017 Delphix. All Rights Reserved. Private and Confidential. 32 Let’s Talk Data Gravity and It’s Impact • Data is HEAVY. • Relational databases are heavily weighted and can be very impacting to DevOps implementations.
  33. 33. © 2017 Delphix. All Rights Reserved. Private and Confidential. 33 Oh DBA, to Embrace Virtualization… A technical approach in which users and applications do not use physical machines, but simulated systems running on actual, “real” hardware. Virtualization can be used to eliminate resource usage and enable savings for databases, network, file and application management, along with server infrastructure.
  34. 34. © 2017 Delphix. All Rights Reserved. Private and Confidential. Older Cloning Methods
  35. 35. © 2017 Delphix. All Rights Reserved. Private and Confidential. Virtualize Compress and Dedup
  36. 36. © 2017 Delphix. All Rights Reserved. Private and Confidential. 36 Why Database Virtualization is Different • Many tools take considerable time to recover. • DevOps automate an “undo” for development and testing that includes data rewind. • Include a self-service tool that will allow for rewind without DBA intervention. • Allows for data version control and DataOps, the next step in DevOps
  37. 37. © 2017 Delphix. All Rights Reserved. Private and Confidential. 37 Virtualization Tools • Delphix • Veritas Velocity • Oracle Thin Cloning Features • Actifio
  38. 38. © 2017 Delphix. All Rights Reserved. Private and Confidential. ▶▶▶ Virtualize and Deployed▶ ▶ ▶ 600GB Virtualization 101 via Delphix Storage Pool for Delphix QA DEV PATCH TEST PRODUCTION Database/App Tier 1 TB 1 TB Read From Production Each Virtual Database takes up around 5-10Gb upon creation, (dependent upon parameters) TEST Read AND Write
  39. 39. © 2017 Delphix. All Rights Reserved. Private and Confidential. Virtualized Database, (VDB)
  40. 40. © 2017 Delphix. All Rights Reserved. Private and Confidential. Data Virtualization: Space Savings Delphix Virtualization EngineValidated Sync Target Dozen Virtual Database Copies, (VDBs)
  41. 41. © 2017 Delphix. All Rights Reserved. Private and Confidential. 41 The Future is Now- Containers/Data Pods Containers offer the ability to isolate application code and/or the whole infrastructure stack into a package able entity to ease deployment, even from the same kernel. This is a powerful tool for DevOps to ease deployment for complex tiers, applications and multiple data stores. Data Pods is the next step, where a pod is created from virtualized environments.
  42. 42. © 2017 Delphix. All Rights Reserved. Private and Confidential. 42 Docker • Virtualization is important, this is OS level • Resource level isolation at the kernel level • Technology such as OverlayFS, allows for containers, which removes overhead of other virtualization technologies. Docker Alternatives: • Open Container Initiative, (OCI) • Kubernetes • CoreOS • Apache Mesos
  43. 43. © 2017 Delphix. All Rights Reserved. Private and Confidential. Group Environments into Data Pods
  44. 44. © 2017 Delphix. All Rights Reserved. Private and Confidential. Why Create Data Pods • Development is done by project or feature vs. tier or product. • With the introduction to the cloud, the business requires those that can do more at a global scale. • Ease of management and maintenance • Provide more value to the business
  45. 45. © 2017 Delphix. All Rights Reserved. Private and Confidential. 45 Data Pods Database Unstructured Data Application
  46. 46. © 2017 Delphix. All Rights Reserved. Private and Confidential. Data Pods for Ease of Delivery Create “Container” Rsync (UNIX/Linux) Robocopy (Windows) Reporting NFS iSCSI Development NFS iSCSI Testing NFS iSCSI Delphix Virtualization Engine Same Storage Size as Production Deployed using Jenkins
  47. 47. © 2017 Delphix. All Rights Reserved. Private and Confidential. 47 How Does This Improve Agile? Finance HR CRM
  48. 48. © 2017 Delphix. All Rights Reserved. Private and Confidential. 48 Timeline in a Project Waterfall- 5 Releases January February March April Agile with Virtualization- 12 Releases January February March April Database Refresh Development Production Release Testing Cycle The Goal of Agile Data Holdup to Development Cycle
  49. 49. © 2017 Delphix. All Rights Reserved. Private and Confidential. Data Versioning • Delphix Jet Stream • Redgate SQL Data Compare • Oracle SQL Developer
  50. 50. © 2017 Delphix. All Rights Reserved. Private and Confidential. 50 Branching and Bookmarking • The ability to mark each iteration of development with a bookmark • Simplify to lock and deliver while testing a consistent image via a virtual database, (VDB) • If a something goes wrong, the ability to “bookmark”, (and subsequent snapshot) to deliver to development to address.
  51. 51. © 2017 Delphix. All Rights Reserved. Private and Confidential. 51 DevOps and Source Control
  52. 52. © 2017 Delphix. All Rights Reserved. Private and Confidential. Administration Ease
  53. 53. © 2017 Delphix. All Rights Reserved. Private and Confidential. Self-Service Release 1.2 Release 1.3 Branch 1.3 Branch 1.1
  54. 54. © 2017 Delphix. All Rights Reserved. Private and Confidential. Methodologies and Collaboration
  55. 55. © 2017 Delphix. All Rights Reserved. Private and Confidential. 55 Scrum https://www.scrumalliance.org/community/articles/2014/april/devops-and-agile
  56. 56. © 2017 Delphix. All Rights Reserved. Private and Confidential. 56 Lean/Kanban
  57. 57. © 2017 Delphix. All Rights Reserved. Private and Confidential. 57 Extreme Programming
  58. 58. © 2017 Delphix. All Rights Reserved. Private and Confidential. 58 Crystal Alistair Cockburn said, “Crystal is a family of human-powered, adaptive, ultra light, ‘stretch-to-fit’ software development methodologies.” • People are the most important aspect of Crystal. • Development is self-sufficient and self-organizing, (very different from Scrum) • Very lightweight • Very little documentation, overhead management or reporting.
  59. 59. © 2017 Delphix. All Rights Reserved. Private and Confidential. 59 Feature Driven Development
  60. 60. © 2017 Delphix. All Rights Reserved. Private and Confidential. 60 Collaboration Practices can Mutate • Methods- Scrum, KanBan, Extreme Programming, Continuous Delivery • Official hybrid approaches like Scrumban and KanPlan • Numerous other combinations created from any of the originals.
  61. 61. © 2017 Delphix. All Rights Reserved. Private and Confidential. • All IaaS solutions provide encryption in-flight and encryption at- rest • But encryption doesn’t protect data as much as it needs to be . • Europe already requires data masking, not just data encryption for any confindential data, (GDPR): • http://ec.europa.eu/justice/data-protection/article- 29/documentation/opinion-recommendation/files/2014/wp216_en.pdf Confidential data
  62. 62. © 2017 Delphix. All Rights Reserved. Private and Confidential. • Encryption is reversible data obfuscation, which is very different from masking data. • Data masking is non-reversible. • It solves the issue at the data level. • Is authentication and authorization in non-production in compliance with security goals? • All organizations will soon need to review if critical data in non- production environments be accessible to developers, testers and users. Confidential data
  63. 63. © 2017 Delphix. All Rights Reserved. Private and Confidential. Encryption and Masking • Optim • Delphix Masking • Dataveil • CA Test Data Manager • Oracle Encryption and Masking
  64. 64. © 2017 Delphix. All Rights Reserved. Private and Confidential. • Masking personally-identifiable, (PII, HIPPA, PCI, etc.) information renders it useless from a security standpoint • Resolves both the technical and personal responsibility issue. • The data can be masked before it moves to non-production, removing unnecessary risk. Why Masking is Part of the Answer
  65. 65. © 2017 Delphix. All Rights Reserved. Private and Confidential. Confidential data Production Non-Production Exposure
  66. 66. © 2017 Delphix. All Rights Reserved. Private and Confidential. Confidential data Production Non-Production Exposure Encryption Masking Solution
  67. 67. © 2017 Delphix. All Rights Reserved. Private and Confidential. Auditing Solutions • Idera SQL Secure and SQL Compliance Manager • Redgate SQL Source Control • Oracle Auditing Features
  68. 68. © 2017 Delphix. All Rights Reserved. Private and Confidential. Data Causes Friction • If those managing the data sources aren’t included and working well with those needing that data to produce features and products, friction is the result. • DBAs are consistently working with archaic utilities that guarantee outcome, rarely speed of results. A DBA is only as good as their last backup.
  69. 69. © 2017 Delphix. All Rights Reserved. Private and Confidential.
  70. 70. © 2017 Delphix. All Rights Reserved. Private and Confidential. What is Data Gravity? Data Gravity is the ability for data to attract applications, services, etc. As with the laws of physical gravity, data, due to its mass, will attract all else that has less mass.
  71. 71. © 2017 Delphix. All Rights Reserved. Private and Confidential. When you consider… An estimate of a 10% increase in data accessibility will result in more than $65 million in additional net income, you can see why data gravity is a big deal.
  72. 72. © 2017 Delphix. All Rights Reserved. Private and Confidential. How much data are we talking about? • More data has been created in just the last two years than the previous history of humanity • 1.7MB of new information created every second per human on the planet. • Introduction of big data often has same development pain points.
  73. 73. © 2017 Delphix. All Rights Reserved. Private and Confidential.
  74. 74. © 2017 Delphix. All Rights Reserved. Private and Confidential. 74 Know Your Tools Incredible variation of tools and knowing tool classifications. Use the right tool for the job and recognize that each tools has its own “flavor.”
  75. 75. © 2017 Delphix. All Rights Reserved. Private and Confidential. Remove the Roadblocks • Remove the data as the roadblock. • Use virtualization and containers to simplify. • Self-service portals for developers and testers to refresh and develop they way they do in an agile environment. • Tools with an interface made for Agile and Development data at its focus.
  76. 76. © 2017 Delphix. All Rights Reserved. Private and Confidential. 76 Review- DBA Focus Areas • Update skills, (scripting, monitoring, automating) to remove bottlenecks • Enhance monitoring utilities that take infrastructure and development tools into overall view • Recognize a need to become more skilled in scripting and become knowledgeable of DevOps tools. • Data is EVERYWHERE! Recognize the integral part Database Administrators play in DevOps and learn more about the next generation- DataOps! Don’t fear the future, embrace it. DBA DevOpsSkills
  77. 77. © 2017 Delphix. All Rights Reserved. Private and Confidential. 77 The Periodic Table of DevOps Tools https://xebialabs.com/periodic-table-of-devops-tools/
  78. 78. © 2017 Delphix. All Rights Reserved. Private and Confidential. 78 Read- The Phoenix Project and the DevOps Handbook! Delphix with SQL Server- the Basics https://docs.delphix.com/docs/delphix-administration/sql-server-environments-and-data- sources/managing-sql-server-environments/overview-of-setting-up-sql-server- environments Delphix Upgrade Workflow: https://community.delphix.com/delphix/topics/tip-of-the-day- upgrading-a-sql-server-dsource Upgrading the Dsource after an Upgrade: https://docs.delphix.com/docs/delphix- administration/sql-server-environments-and-data-sources/virtualizing-databases-using- delphix-with-sql-server/managing-sql-server-dsources/additional-dsource-topics/upgrading- a-dsource-after-a-sql-server-upgrade Delphix in the Cloud https://www.delphix.com/solutions/cloud-migration-virtual-data References and Tips
  79. 79. © 2017 Delphix. All Rights Reserved. Private and Confidential.© 2017 Delphix. All Rights Reserved. Private and Confidential. Kellyn Pot’Vin-Gorman Technical Intelligence Manager kellyn@delphix.com http://dbakevlar.com

×