Security Automation Simplified - BSides Austin 2019

Security automation simplified
An intro to DIY security automation
Moses Schwartz, Security Automation Engineer
moses@box.com | @mosesschwartz | github.com/mosesschwartz
BSides Austin 2019

2Security automation simplified
Incident response / security monitoring infrastructure
Before automation
Network
Devices
User
Laptops
Servers
Security
Tools
SIEM
Logs Ticketing
System
Active
Directory
Asset
Mgmt
VirusTotal
Playbook
Docs
???? ????
Alerts

Automation approaches
Building an automation/orchestration solution for every environment is a huge undertaking
Building one for your environment is a much more tractable problem
Centralized
Logic mostly in one place, often with some
sort of workflow or orchestration engine
• Examples: Most commercial offerings,
building everything in a single platform,
this DIY approach
• Downsides: single point of failure, major
effort to build out, major learning curve
Distributed / ad hoc
Bits of automation from many different tools
to make your life better
• Examples: 1:1 pairings using web hooks
between tool APIs
• Downsides: gets complicated, hard to
debug, hard to piece together complex
workflows, tribal knowledge

Build a search query that
matches the condition you
want to alert on
Splunk
Alert development

Create an alert using your search
query
Best practice is to create an App
to contain all of your custom
settings
I like to specify a cron schedule
for maximum flexibility
Ensure your time range matches
the schedule
Splunk
Alert development

Trigger when Number of Results
is greater than 0
Trigger for each result - Splunk
webhooks only include the first
row of data
Add a Webhook Trigger Action
and aim it at your server (we’ll
build this in the next step)
Splunk
Alert development

We will use Flask for super simple API
development
Always include a status/health endpoint
splunk_webhook will write the JSON
payload with indentation to a file
host=“0.0.0.0” exposes this to the
world!
automation_server.py
Receive Splunk webhook payload

It’s JSON from our alert!
Development tip: modify that alert to run
every minute and extend the time range
Don’t run it like this in production - there
are many tutorials on deploying a Flask
app with Nginx or Apache and a WSGI
server
Run the server and check the output

Your code should be under version control,
but your passwords shouldn’t!
A super lightweight approach is to keep
your secrets and settings in a Python file
that is NOT checked in with code (don’t
forget to add this file to your .gitignore)
This file can then be pushed as part of
configuration management or manually
settings.py
Keep secrets out of git!

Let’s create a ticket in Jira
Create your authenticated JIRA object
using the Python library
Use the create_issue method to create
the ticket and set fields
Round two: ticket creation

After the next Splunk
webhook fires, we’ll have a
Jira ticket
Right now the description is
just a JSON blob of the alert
Jira
Issue created

Create a webhook to do
enrichments – start by just
extracting user and MD5 and
commenting on the ticket
Point the URL to your
automation server with a new
endpoint
Filter for Issue created
events that match our project
and alert name
Jira
Webhook configuration

Add a comment to Jira

Returns a JSON object:
ad_lookup.py
Lookup a user in Active Directory
{'entries': [{'attributes': {
'cn': 'Moses Schwartz',
'title': 'Staff Security Engineer',
'company': 'Box, Inc',
'department': 'Security Automation',
'employeeID': '1234',
'l': 'Redwood City',
'streetAddress': '900 Jefferson Avenue',
# ... tons more fields omitted
}}]}

Active Directory lookup enrichment

virustotal.py
Get a file scan report
{'scan_id': 'e3b0c44298fc1c149afbf48996f...',
'sha1': 'da39a3ee5e6b4b0d3255bff9560189...',
'resource': 'd41d8cd98f00b204e980098ecf...',
'scan_date': '2019-03-01 23:35:34',
'permalink': 'https://www.virustotal.com/...
'total': 60,
'positives': 0,
'md5': 'd41d8cd98f00b204e9800998ecf8427e'
{'scans': {'Bkav': {'detected': False,
'version': '1.3.0.9899',
'result': None,
'update': '20190301'}
....

Now with AD and VT enrichments

Incident response / security monitoring infrastructure
With automation
MockScan Splunk
Logs
Automation server
Alerts
Jira
Create ticket
Webhook
Active
Directory
Enrich ticket
VirusTotal

• Search for and link to previous tickets, populate ticket fields, close duplicate tickets
• Run a Splunk search
• Lookup DNS and WHOIS records
• Run Ansible playbooks
• Send a sample to a sandbox
• Upload files to Box
• Isolate hosts and grab memory
• Pull PCAPs
• Flash a light or connect to other smart devices
More things we could automate
Anything you can write a script to do

Input validation, error messages, API documentation
Error handling
If one task fails, they all fail
Authentication
Deployment
Health/status monitoring
Moving toward production
Everything that can break, will break
500
Internal Server Error

Flask extensions such as Flask-RESTPlus
can do a lot of work for you
Provides consistent API interaction and
error messages
Automatically generates Swagger docs!!
Now using Flask-RESTPlus

Swagger is a framework for documenting and
building APIs
Swagger UI allows us to interact with our API
through the browser
We’ll have to add model/schema definitions
to complete the Swagger documentation
Swagger Docs
Automatically generated from code

Specify a model for the endpoint’s
expected input
We don’t need to worry about all of the
webhook fields, just the ones we use
Enable validation to reject bad requests
Input validation

The model is used to automatically build the
example payload
Now we can run these endpoints through the
web interface
Swagger Docs
Model specifies payload format

We weren’t handling errors in our code
before, so we got default web server
HTML error messages
Flask-RESTPlus will return properly
formatted JSON message
Adding input validation prevents the
error in the first places and provides
semi-helpful error messages
API interaction
Error handling / input validation

Enrichments and other tasks should be asynchronous to avoid the scenario where if one fails, they all fail
Our example was synchronous, some other options to run jobs asynchronously are:
• Individual Jira webhooks for each enrichment
• Celery, multiprocessing, asyncio (DIY Python approaches)
• StackStorm, Jenkins, Rundeck (DevOps tools that fit this use case)
• AWS Lambda jobs (hey, we could build this whole thing out of Lambdas)
Add a retry capability to each task: just trying again fixes a surprising number of problems
Error handling, retrying, task execution

Options:
• Include a “secret” in your webhook receiver URL
• Don’t worry and restrict access by IP
• Store access tokens in settings.py
• Rely on .htaccess or similar from your webserver
• Use an extension like Flask-Login
• Roll your own solution
Authentication is nearly meaningless without encryption
Authentication
Almost as complicated as secret management

Get a domain and SSL cert
• IP addresses and HTTP are fine for development, but are not so cool for production use
Use configuration management
• Ansible, Salt, Puppet, Chef, etc.
Consider deployment options
• Containers (Docker)
• Package builders (Pex, Omnibus)
Strive for continuous integration/continuous deployment (CI/CD)
• Automated, repeatable builds / deployments are the dream
Deployment
DevOps for SecOps

Exception logging and alerting are critical: even if our code is perfect, external API lookups will fail
Use a status monitoring tool like Nagios, Icinga, Sensu, Datadog (that’s what the /status endpoint is for)
Forward your server logs to Splunk, and set up alerts on error messages
Monitor for skipped searches in Splunk:
index=_internal source=*scheduler.log status=skipped
Build some status dashboards to monitor API usage and errors
Status and health monitoring
What could go wrong? Splunk. Splunk could go wrong.

Maintaining state is hard: avoid it whenever possible
Try, try again: a surprising number of errors can be resolved
by waiting and retrying
Deploy your code from your git repo even if you don’t have
CI/CD: this minimizes the temptation to make in-place
hotfixes
Consider writing tests: sometimes they’re really handy
Keep your code modular and small
Done is better than perfect
Development tips and lessons learned
Let me be a cautionary tale…
Source: xkcd.com

Management loves colorful numbers
Assign a number of minutes saved per enrichment or action and calculate the total from your logs
Security automation isn’t about replacing people, and it’s not a set-it-and-forget-it solution
• Security automation is different from factory automation – you can’t replace human incident responders
• Automation should become a core part of your process – continually improve
Target 25% effort spent automating
Metrics
Quantify your impact

Security automation is not black magic
Existing tools that aren't marketed toward security can work great in this space
There is so much low hanging fruit
Our job is to make the rest of the team more effective (which is pretty awesome)
This niche is a great path into security from development or into development from security
Takeaways

Join the Austin DFIR Slack group! https://atx-dfir.herokuapp.com
Meetups every ~quarter, next one scheduled for week of April 22
Come work with us! https://www.box.com/careers
Austin:
Security automation, threat analyst, security analyst, red team
Redwood City:
Security automation interns, application security, security engineering
And finally…

Security Automation Simplified - BSides Austin 2019

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Security Automation Simplified - BSides Austin 2019

Similar to Security Automation Simplified - BSides Austin 2019 (20)

Recently uploaded

Recently uploaded (20)

Security Automation Simplified - BSides Austin 2019