Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DCSF19 CMD and Conquer: Containerizing the Monolith

98 views

Published on

Tony Lee & Nelson Wang, Splunk
Modern microservice-oriented software architectures evangelize the principles of infrastructure-as-code and declarative directives to manage and run applications. At Splunk, we wanted to marry these ideals with the majestic monolith, Splunk Enterprise, to simplify the use of our product through containerization. Without rearchitecting the entire product from the ground-up, which can be a costly investment, we focused on incorporating a flexible configuration management layer on top of the core application. This has enabled us to make running Splunk in Docker act and behave as a true microservice, greatly reducing the friction of migrating towards more container-native software.

We not only concentrated on making our open-source Docker image initiative user-friendly and production-ready, but we also wanted to seamlessly integrate it back into our internal engineering process. Join us for this session as we discuss migrating a traditional application into a microservice ecosystem, developing a containerization strategy for both external customer usage and internal development, as well as learning about our internal container platform at scale.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

DCSF19 CMD and Conquer: Containerizing the Monolith

  1. 1. CMD and Conquer PACKAGING THE MONOLITH
  2. 2. Software Engineer, Splunk Tony Lee Senior Software Engineer, Splunk Nelson Wang
  3. 3. Why? What? How? Next? Agenda
  4. 4. https://mpora.com/mountaineering-expeditions/famous-explorers-history-introduction
  5. 5. Time Intensive Disruptive Expensive $ monolith | microservice
  6. 6. Our Journey
  7. 7. Taking the First Step
  8. 8. APPLICATION REQUIREMENTS SCHEDULER REQUIREMENTS ACCESSIBILITY ADAPTABILITY AUTONOMY FAULT TOLERANCY SCALABILITYSTATELESS
  9. 9. BASE PLATFORM LAYER SPLUNK LAYER FORWARDER LAYER
  10. 10. BASE PLATFORM LAYER As Splunk evolves and develops, the image should grow with it
  11. 11. • Add file/directory monitoring • Open a TCP/UDP socket & listen for input • Enable receiving of syslog data • Setup HTTP event collection token • Forward data to another service • Setup administrator password • Create user accounts for access control • Install an enterprise license • Connect to an existing license master • Setup indexes • Touch configuration files • Install apps • etc. Usability
  12. 12. • Asynchronous dynamic provisioning performed by Ansible • Eliminate reliance on any third-party service registrar/catalog • Parallelized setup removed ordered dependencies and results in faster time-to-value Micro-Orchestration Layer entrypoint.sh Dockerfile
  13. 13. Why Ansible? Domain-specific, developer-friendly language that enables the dynamic configurability
  14. 14. APPLICATION REQUIREMENTS SCHEDULER REQUIREMENTS ACCESSIBILITY ADAPTABILITY AUTONOMY FAULT TOLERANCY SCALABILITYSTATELESS
  15. 15. • Intermittent service interruption is expected within configuration layer • Health-checks and probes to assist in lifecycle management Fault Tolerance
  16. 16. • Orchestration engine administers the pods/containers • Application-context is shared amongst cluster members via environment variables Distribution of State SPLUNK DEPLOYMENT Container Indexer 2 Container Indexer 1 Container Cluster Master Container Search Head
  17. 17. SPLUNK DEPLOYMENT Container Indexer 2 Container Indexer 1 Container Cluster Master Container Search Head
  18. 18. SPLUNK DEPLOYMENT Container Indexer 2 Container Indexer 1 Container Cluster Master Container Search Head
  19. 19. SPLUNK DEPLOYMENT Container Indexer 2 Container Indexer 1 Container Cluster Master Container Search Head
  20. 20. SPLUNK DEPLOYMENT Container Indexer 2 Container Indexer 1 Container Cluster Master Container Search Head
  21. 21. SPLUNK DEPLOYMENT Container Indexer 2 Container Indexer 1 Container Cluster Master Container Search Head
  22. 22. Dynamic scaling of specific roles How do I add a new indexer? Scalability SPLUNK DEPLOYMENT Container Cluster Master Container Search Head 1 Container Indexer 2 Container Indexer 1 Container Indexer 3
  23. 23. Our Results
  24. 24. External Usage • Customers have been waiting to run Splunk on Docker! • Adoption amongst many members of the open-source community • Examples for both swarm + k8s manifests in our GitHub repo
  25. 25. Internal Usage Support • Cheap, reproducible environments • Support any size topology, any version Open-source images are fed back into our internal deployment provisioning service (orca) used across the company Sales • Rapid, reliable demo instances on-demand • Support for local scenarios for the field Dev/QA • Disposability around testing changes • Accelerates iteration cycles in CI/CD
  26. 26. How? orca leverages Docker Enterprise and Universal Control Plane (UCP) Node Worker Node Worker Node Worker Node Worker swarm mode cluster Node Worker Node Worker Dev/QA/Sales Team Using Swarm/K8s Support Team using Swarm/K8s Ops Team DOCKER ENTERPRISE EDITION
  27. 27. ORCA
  28. 28. Image CI/CD • Support for multiple platforms • Integrated security scanning • Functional tests for various configurations and topologies • Version control, custom dependencies for individual teams
  29. 29. Pipeline GitHub IBM s390x Agent $ build splunk- debian Linux Agent $ build splunk- debian Linux Agent $ push splunk-debian DockerHub Public/Open-source Internal
  30. 30. The Future • Deployment frameworks (helm, operators) • Find ways to make Splunk more accessible and reduce time-to- value • Always consider the future of the monolith!
  31. 31. Summary • Find ways to bridge the gap while migrating monolithic software to microservice architecture • Plan for the future • Dynamic self-orchestration made our software offering more nimble.
  32. 32. Get Hands On! Repositories • https://github.com/splunk/docker-splunk • https://github.com/splunk/splunk-ansible Documentation • https://splunk.github.io/docker-splunk • https://splunk.github.io/splunk-ansible #docker @ http://splunk-usergroups.slack.com/
  33. 33. Thank you! Questions?

×