The DreamPort Splunk Project; How We Use AWS, Terraform, and Ansible to Automate Everything About a Splunk Cluster At DreamPort, we use cloud platforms, infrastructure-as-code tooling, configuration tools, automation software, and container technologies to very quickly design, develop, and prototype projects. This particular talk focuses on the tools used to deploy and configure a Splunk cluster for a particular project we recently ran. We will cover the deployment, configuration, and orchestration of a large 16 node Splunk cluster using tools that are a core set to DreamPort's cloud infrastructure toolbox; AWS, Terraform, Ansible, and Docker. It is recommended that attendees have a general understanding of AWS, Linux, Splunk, and Docker, and know about automation tools such as Terraform and Ansible. Attendees will learn how to use AWS, Terraform, Ansible, and Docker to deploy a large Splunk cluster, how to use Ansible to orchestrate and manage the Splunk cluster, and how to use Ansible to orchestrate and manage the Splunk cluster. ------------------------------------------------- Bill Cawthra is a Principal Cloud Infrastructure Architect for CyberPoint, managing project-related cloud systems and platforms. He works primarily on the AWS platform, using various automation tools to rapidly deploy and manage infrastructure. Bill has over 18 years of experience in computers and technology, working in a range of fields, including construction, DoD, health care, and social media.

1. Using AWS, Terraform, and Ansible for
DreamPort Projects - the Splunk Cluster
How we used (and are still using) tools such as AWS, Terraform, and Ansible to
automate everything about a Splunk cluster.

2. Intro
The Who, the What, the Why, and the How
Hands on Keys – Live Demo
Summary, Questions, Extra Deep Dives
On the Agenda Today...

3. Prerequisites – Terms and Tools
• Basic understanding of AWS and cloud computing platforms
• Aware of configuration management/orchestration tools such as
Terraform and Ansible
• Aware of the concepts of Docker
• Need to have a basic understanding of Splunk and a Splunk cluster
• PLEASE ASK QUESTIONS.

4. The Who – Me, MISI, and DreamPort
• Bill Cawthra - Cloud Infrastructure Architect
• I play with little fluffy clouds all day (AWS, Google Cloud, Azure)
• MISI/DreamPort - Support and help develop various cyber security projects
through collaboration with .gov, private industry, community, and .edu
• DreamPort projects – over 20 projects/AWS environments, usually 30-90
days long (some are notably longer)
• https://misi.tech/#about
• https://dreamport.tech/about-us.php

5. The What and the Why - The Splunk Evaluation
• We wanted to build a Splunk cluster to analyze it's machine learning capabilities.
• The data set was 9 TB of Zeek data
• 20 users accessing this data at a time (so fairly light on the frontend)
• But very intense work done on the backend (indexers)
• Big beefy i3.8xlarge instances… Use the instance-store for fast IO (but ephemeral!
Therefore we used Splunk SmartStore)
• With the help of many people at Splunk (Bryan Pluta, Tyler Muth, Matt Toth, and
others), we came up with a design to fit these requirements
• We are going to use AWS, Terraform, and Ansible as our tools of choice

6. The How - AWS
• Amazon Web Services; provides an on-demand
computing platform
• "Elastic" resources
• Allows us to rapidly scale out and scale down
• Very easy to manage many disparate projects
• Best datacenter money can buy

7. The How - Terraform
• Our infrastructure configuration tool of choice
• This "frames the house"; creating the AWS resources (VPC, security
groups, instances, IAM policies, IAM roles, S3 buckets, etc)
• Enforces configuration from the very start (no GUI. No artisinally
crafted architecture)

8. The How - Ansible - Drywall, Paint,
and Fixtures
• Our automation and configuration management tool of choice
• Handles configuration of systems
• Handles automation tasks (upgrade and reboot of systems… and ingest orchestration!)
• Does everything after the "house is framed"

9. The How - Docker
• Easy binary management (example: to upgrade, just docker pull
splunk:<VERSION>)
• The splunk-docker project makes it very easy to assign roles, access
variables

10. The How - Infrastructure Diagram

11. Before We Go Live
• I will be covering things at a high level
• I will be skipping many things
• Ask questions if you want to see XYZ
• Look at the code on your own too!
• It’s tricky to balance being concise in a talk and detail of the code
• Need to avoid turning this into a code review session…
• If something looks confusing or wrong, I probably made a mistake.

12. Before We Go Live - Resources
• https://github.com/TheDreamPort/splunk-infrastructure (santiized
version of this project)
• Also great references:
• https://splunk.github.io/splunk-ansible/ - Splunk Ansible reference
• https://splunk.github.io/docker-splunk/ - Splunk Docker

13. TO THE TERMINAL AND BROWSER

14. Conclusion
• We automate automate automate
• Which means, we configure/deploy everything programmatically
• Ingest is automated
• Makes it so easy to redo
• Break up the automation into logical pieces
• It is not fun having a single mega-script

15. Extra Notes - Splunk Ingest
• Ingest the 9TB of data in batches (basically did it a month at a time) and
wait for completion
• Limited disk space on the ingesters
• Minimize impact of mistakes
• Had to be very specific on what was ingested; did not want to duplicate
data
• Ingest process would attempt to detect if a file had been ingested
• Had to verify data was properly ingested (document count of files vs
document count in Splunk)

16. Extra Notes - Monitoring and Logging
• Delicious dashboards using Grafana
• Graphs the Prometheus metric data
• Can graph Loki events too (logs)

17. Questions? Comments?