Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

SplDevOps: Making Splunk Development a Breeze With a Deep Dive on DevOps' Containerization, Version Control and Automation


Published on

As Splunk scales, it grows with more Splunk engineers, developers and users. Maintaining proper knowledge object development, deployment changes and best practices can become a daunting task where fear-driven development takes its toll. In this session we present our enhancement of Splunk’s scalability in terms of software management, continuous integration and continuous delivery (CI/CD) by providing a framework which consists of DevOps tooling in combination with our Splunk expertise. Specifically, we are able to maintain a proper Splunk development cycle by using Docker containers, configuration and secret management with Ansible and version control with Git (VCS), all achieved by taking advantage of Splunk's ".conf" versatility. Our result is a CI/CD development-to-testing-to-production framework that complements Splunk’s scalability with modern DevOps culture and facilitates a smoother yet moderated development experience.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

SplDevOps: Making Splunk Development a Breeze With a Deep Dive on DevOps' Containerization, Version Control and Automation

  1. 1. © 2018 SPLUNK INC.© 2018 SPLUNK INC. SplDevOps Making Splunk Development a Breeze with a Deep Dive on DevOps, Containerization, Version Control & Automation Harry McLaren, Ilias Diamantakos, Tomasz Dziwok October 2018 | Version 1.3
  2. 2. © 2018 SPLUNK INC. During the course of this presentation, we may make forward-looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward-looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release. Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2018 Splunk Inc. All rights reserved. Forward-Looking Statements
  3. 3. v © 2018 SPLUNK INC. HARRY MCLAREN Splunk Enablement Lead, Managing Consultant ILIAS DIAMANTAKOS Splunk Engineer, Associate Consultant Who Are We? cyberharibu ilias-diamantakos
  4. 4. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Best Security Company of the Year Employer: ECS (UK Splunk Elite Partner)
  5. 5. © 2018 SPLUNK INC.© 2018 SPLUNK INC. What’s It All About? ▶ Customer Challenges ▶ What Do We Want? ▶ Our Idea to Deploy Splunk ▶ Technical Deep Dive ▶ Project Roadmap ▶ Key Takeaways ~40mins
  6. 6. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Customer Challenges “The expansion of Splunk has increased operational complexity, as we manage it manually and can’t keep on top of project change requests.” – High-Street Retailer “We require a full route- to-live to maintain system integrity and can’t deploy changes fast enough in our current setup.” – National Bank “Multiple developers within the same DEV environment, causes repeated configuration conflicts and delays to planned changes.” – National Building Society
  7. 7. © 2018 SPLUNK INC.© 2018 SPLUNK INC. What Do We Want? Enterprises Want to Respond Quickly, Safely & With Less Risk Rapid Changes to Splunk Software 01 Orchestrated Deployment 02 Fragile Route-to-Live  Fail Safe, Fast Backout 03 Development at Scale • Enterprise Scale Development • Synchronous Changes / Multiple Admins & Developers • Splunk Defined via Code • Familiar Approach (AKA: DevOps/Agile) Reduction in Custom Config • Every ’Custom’ Configuration Introduces Disparity • Inconsistent Dev, Test, Pre-Prod, Prod • Testing is “Best Endeavors” • Increased Risk, Changes Batched
  8. 8. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Splunk for Agility Supporting Agile Methodology by Default Schema at Read, Supporting Multiple Use Cases Analytic Tools Exposed to UI, Empowering Users to Experiment Plain Text Configuration Files, Documented & Supported Splunk API is Enumerated, Dev Licenses, Labs Encouraged SPL Web UI Plain Text Config Open API Monitor InvestigateIntelligence
  9. 9. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Our Idea “SplDevOps” Became the Solution Version Control  Git[Lab] Utilized  Multiple Projects/Branches  Key Releases Tagged Full Route-to-Live  Multi-Stage Environments  Dev > Pre-Prod > Prod • Automated Testing Agile Development  Short Sprints  Test Driven Development  Issue Management & Feature Backlog Configuration Management  Orchestrated Deployment  Centralized Config  Ansible used via SSH
  10. 10. © 2018 SPLUNK INC. Project: Internal Monitoring Ask: Deploy Splunk Internally for SecOps & ITOps Use Cases
  11. 11. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Automation Engine Containerization Version Control Brief Background Let’s talk tools!
  12. 12. © 2018 SPLUNK INC. What Tool Fits Where? a new Splunk infrastructure the DevOps way! ▶ Identical environments & route-to-live • Development, Pre-production, Production ▶ Eliminate fear driven development • It’s ok to make mistakes! ▶ Minimize direct production changes • Always go through route-to-live • Transparent change control ▶ Modern means of disaster recovery ▶ Security driven ← Ansible + Git + Docker + Python ← Docker + Git ← GitLab ← Ansible (IaC) ← Ansible Vault
  13. 13. © 2018 SPLUNK INC. How We Wanted It To Look Spoiler Alert: This is also the end result dev pre-prod prod IX: Splunk Indexer SH: Splunk Search Head DS: Splunk Deployment Server AS: Ansible Server
  14. 14. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Multiple Repositories Of /opt/splunk/etc for each instance Ansible & DS IX splunk_ix SH splunk_sh Syslog Collector syslog ansible splunk_ds
  15. 15. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Git Workflow aka “the change process”
  16. 16. © 2018 SPLUNK INC. Everything starts from our DevEnv So let’s spin one up
  17. 17. © 2018 SPLUNK INC. What’s going on in the background
  18. 18. © 2018 SPLUNK INC. What’s going on in the background
  19. 19. © 2018 SPLUNK INC. How It Looks
  20. 20. © 2018 SPLUNK INC. How It Looks
  21. 21. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Let’s Share Secrets No really, we are sharing! ▶ How to version sensitive information • Encryption ▶ How to decrypt automatically • Ansible Vault ▶ How to store Ansible Vault Password • More encryption
  22. 22. © 2018 SPLUNK INC. Let’s Decrypt One password to rule them all Ansible Server
  23. 23. © 2018 SPLUNK INC. Use Case Scenario Demo time
  24. 24. © 2018 SPLUNK INC.© 2018 SPLUNK INC. How it should have been done Integrating with our change process
  25. 25. © 2018 SPLUNK INC.© 2018 SPLUNK INC. How it should have been done Integrating with our change process
  26. 26. © 2018 SPLUNK INC. How it gets deployed
  27. 27. © 2018 SPLUNK INC. How it gets deployed
  28. 28. © 2018 SPLUNK INC. How it gets deployed
  29. 29. © 2018 SPLUNK INC. Lessons We Learned Not everything was easy… ▶ Multiple repositories • What goes where? • Many lines of history ▶ Identical code for different environments • There are always exceptions (Eventgen, production API calls) ▶ Data for different environments • Production data is sensitive ▶ Automated deployment of code • When do you restart?
  30. 30. © 2018 SPLUNK INC. Deployment Results Did it work!?
  31. 31. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Full Route-to-Live Implemented in Production Users & Admins Educated & Empowered Everything Under Version Control Promoting Changes in ~ 5mins (Dev>Prod) Foundations Built for Future Development End Result Prototype Success, Production Rollout
  32. 32. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Adaptable Framework Expressed in Software (Python + Git + Ansible) Environment Agnostic & Scales to Clustered Deployments& Hybrid Cloud Architecture User Friendly & End-to-End Integrated with Issue/Change Management Roadmap Introducing “Splunk Compiler” (v2.0+)
  33. 33. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Splunk Supports Experimentation by Default Agile/DevOps Methodologies are Compatible Doesn’t Require Automation Expertise Version Control BEFORE Software Orchestration Key Takeaways Remember Four Things…
  34. 34. © 2018 SPLUNK INC. Don't forget to rate this session in the .conf18 mobile app Thank You