SlideShare a Scribd company logo
1 of 17
Download to read offline
Automated Collection &
Enrichment (ACE)
Jared Atkinson & Robby Winchester
Overview
● Concept
● Setup
○ System Requirements
○ Architecture
○ System Management
● Collection
● Automation
● Enrichment
● Data Export
● Future
What is ACE
● An open source solution that enables agentless threat
hunting in an environment
● Allows for scanning of remote Windows and macOS/Linux
systems
○ Only requires Local Admin access
○ macOS requires SSH enabled for a “sudoer” account
● Can perform host scans or be scheduled for routine use in
an enterprise
Why ACE?
● As consultants performing Compromise Assessments, we
rarely have the authority or ability to alter a
customer’s environment to support assessment operations
● No external dependencies or agent to install
● Executes solely in memory with nothing being written to
disk
● JSON formatted output allows for easy ingestion into SIEM
of choice
● It’s free!
Architecture Overview
Scan to Target Systems
Download Script & Return Results
Enriched Data
Target Network
Management
System
Scan Tasking
ACE
ACE Architecture
Enrichments
- Enriches scan data
- Outputs data from ACE
ACE Management
- Hosts users/computers/creds
- Backend management of ACE
Web Server
- Initiates scan activity
- Hosts scanning scripts
- Receives host data
Management
System
- Performs tasking
System Management
● Users - Internal ACE system accounts
○ Create, Enumerate, Remove, Update
● Credentials - Credentials to target systems
○ Add, Enumerate
● Scripts - Scans being run on remote systems
○ Add, Enumerate
● Computers - Computers in target environment
○ AD & Computer List Discovery
System Management
Web Server
Enrichments
ACE Management
Management
System
Add/R
em
ove
AC
E
O
bjects
ACE information
(Users/Credentials/Computers)
Discovery
Automation
● Architecture of ACE allows for easy configuration and use
● Schedule scans
○ Time in future
○ Repeat Scan ‘x’ times at ‘y’ interval
○ Can specify desired scripts/computers per scan
● Take care of collection process behind the scenes
○ Tasking and script execution
○ Upload of completed scan data
Automation
Web Server
Download
Scripts
Scheduling
Results
Enrichments
ACE Management
Management
System
Relay
Tasking
Access ACE Information for Scan
Collection
● Scripts
○ Windows - PowerShell (PS v2 compatible)
○ macOS/Linux - Python (2.7 with no dependencies)
● Downloaded and executed from the Web Server
● Results uploaded to web server via HTTPS POST requests
Collection
Web Server
Download
Scripts
Scan
Taskings
Results
Enrichments
ACE Management
Management
System
Relay
Tasking
Access ACE Information for Scan
Enrichment
● RabbitMQ messaging system used for enrichment
○ Queues and exchanges used to manage the data
○ Uses AMQP 0.9.1 messaging protocol
● Currently supports public API VirusTotal enrichment
● Allows for easy additional custom enrichments
○ Currently all written in C#, however RabbitMQ is supported in Python,
Java, Ruby, PHP, JavaScript, Go, Elixir, Objective-C, Swift, and
Spring AMQP
Enrichment
Web Server
Scan
Results
Enrichments
ACE Management
Management
System
Scan Results for Enrichment
Enriched Data
Pre_Hash
File
SIEM
X
ACE
Exchange
VirusTotal
Hash Match
C
FileWriter
C
SIEM
P
ACE Web
Server
C/P
RabbitMQ Enrichment Process
Data Export
● Enriched data can currently be exported from ACE two ways
○ Pulled from a RabbitMQ Queue straight to a SIEM (such as ELK)
■ Logstash can map to RabbitMQ input
○ Written out to flat files
● All data is formatted as one object per line flat JSON
○ Easily can import into SIEM of choice for analysis
Future Enhancements
● Simplified Deployment - 4 Aug
● Sweep Tracking - ~2 months
● Additional Scanning Scripts - Ongoing
○ Add wider Linux/Unix support
○ Additional Windows scripts
● Additional Protocol Support - Ongoing
● Additional Enrichments - Ongoing
○ IP Reputation
● Web GUI - TBD

More Related Content

What's hot

What's hot (20)

Distributed Kafka Architecture Taboola Scale
Distributed Kafka Architecture Taboola ScaleDistributed Kafka Architecture Taboola Scale
Distributed Kafka Architecture Taboola Scale
 
Prometheus casual talk1
Prometheus casual talk1Prometheus casual talk1
Prometheus casual talk1
 
API Facade Pattern with Apache Synapse
API Facade Pattern with Apache SynapseAPI Facade Pattern with Apache Synapse
API Facade Pattern with Apache Synapse
 
Prometheus
PrometheusPrometheus
Prometheus
 
HTTP Analytics for 6M requests per second using ClickHouse
HTTP Analytics for 6M requests per second using ClickHouseHTTP Analytics for 6M requests per second using ClickHouse
HTTP Analytics for 6M requests per second using ClickHouse
 
How fast is it?
How fast is it?How fast is it?
How fast is it?
 
Monitoring Kafka w/ Prometheus
Monitoring Kafka w/ PrometheusMonitoring Kafka w/ Prometheus
Monitoring Kafka w/ Prometheus
 
Prometheus on AWS
Prometheus on AWSPrometheus on AWS
Prometheus on AWS
 
Administrative techniques to reduce Kafka costs | Anna Kepler, Viasat
Administrative techniques to reduce Kafka costs | Anna Kepler, ViasatAdministrative techniques to reduce Kafka costs | Anna Kepler, Viasat
Administrative techniques to reduce Kafka costs | Anna Kepler, Viasat
 
Apache Kafka : Monitoring vs Alerting
Apache Kafka : Monitoring vs AlertingApache Kafka : Monitoring vs Alerting
Apache Kafka : Monitoring vs Alerting
 
Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day
Hadoop summit - Scaling Uber’s Real-Time Infra for  Trillion Events per DayHadoop summit - Scaling Uber’s Real-Time Infra for  Trillion Events per Day
Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day
 
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at UberDisaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber
 
Consul
ConsulConsul
Consul
 
Flink Forward Berlin 2017: Maciek Próchniak - TouK Nussknacker - creating Fli...
Flink Forward Berlin 2017: Maciek Próchniak - TouK Nussknacker - creating Fli...Flink Forward Berlin 2017: Maciek Próchniak - TouK Nussknacker - creating Fli...
Flink Forward Berlin 2017: Maciek Próchniak - TouK Nussknacker - creating Fli...
 
Monitoring Kubernetes with Prometheus
Monitoring Kubernetes with PrometheusMonitoring Kubernetes with Prometheus
Monitoring Kubernetes with Prometheus
 
Service Discovery with Consul - Arunvel Arunachalam
Service Discovery with Consul  - Arunvel Arunachalam Service Discovery with Consul  - Arunvel Arunachalam
Service Discovery with Consul - Arunvel Arunachalam
 
Integration Of Mulesoft and Apache Active MQ
Integration Of Mulesoft and Apache Active MQIntegration Of Mulesoft and Apache Active MQ
Integration Of Mulesoft and Apache Active MQ
 
Monitoring a Kubernetes-backed microservice architecture with Prometheus
Monitoring a Kubernetes-backed microservice architecture with PrometheusMonitoring a Kubernetes-backed microservice architecture with Prometheus
Monitoring a Kubernetes-backed microservice architecture with Prometheus
 
ICANN DNS Symposium (IDS 2019): RDAP CDN Distribution Experience
ICANN DNS Symposium (IDS 2019): RDAP CDN Distribution ExperienceICANN DNS Symposium (IDS 2019): RDAP CDN Distribution Experience
ICANN DNS Symposium (IDS 2019): RDAP CDN Distribution Experience
 
WSO2Con USA 2015: Deployment Patterns and Capacity Planning
WSO2Con USA 2015: Deployment Patterns and Capacity PlanningWSO2Con USA 2015: Deployment Patterns and Capacity Planning
WSO2Con USA 2015: Deployment Patterns and Capacity Planning
 

Similar to Automated, Collection, and Enrichment (ACE)

Similar to Automated, Collection, and Enrichment (ACE) (20)

Red Hat Forum Tokyo - OpenStack Architecture Design
Red Hat Forum Tokyo - OpenStack Architecture DesignRed Hat Forum Tokyo - OpenStack Architecture Design
Red Hat Forum Tokyo - OpenStack Architecture Design
 
Designing for operability and managability
Designing for operability and managabilityDesigning for operability and managability
Designing for operability and managability
 
Open Audit
Open AuditOpen Audit
Open Audit
 
Scaling 100PB Data Warehouse in Cloud
Scaling 100PB Data Warehouse in CloudScaling 100PB Data Warehouse in Cloud
Scaling 100PB Data Warehouse in Cloud
 
Exploring the Final Frontier of Data Center Orchestration: Network Elements -...
Exploring the Final Frontier of Data Center Orchestration: Network Elements -...Exploring the Final Frontier of Data Center Orchestration: Network Elements -...
Exploring the Final Frontier of Data Center Orchestration: Network Elements -...
 
Infra / Cont delivery - 3rd party automation
Infra / Cont delivery - 3rd party automationInfra / Cont delivery - 3rd party automation
Infra / Cont delivery - 3rd party automation
 
2009-08-24 Managing your Red Hat Enterprise Linux Guests with RHN Satellite
2009-08-24 Managing your Red Hat Enterprise Linux Guests with RHN Satellite2009-08-24 Managing your Red Hat Enterprise Linux Guests with RHN Satellite
2009-08-24 Managing your Red Hat Enterprise Linux Guests with RHN Satellite
 
Bare Metal Provisioning for Big Data - OpenStack最新情報セミナー(2016年12月)
Bare Metal Provisioning for Big Data - OpenStack最新情報セミナー(2016年12月)Bare Metal Provisioning for Big Data - OpenStack最新情報セミナー(2016年12月)
Bare Metal Provisioning for Big Data - OpenStack最新情報セミナー(2016年12月)
 
AWS_Community_Day_2023-Chathra Serasinghe.pptx
AWS_Community_Day_2023-Chathra Serasinghe.pptxAWS_Community_Day_2023-Chathra Serasinghe.pptx
AWS_Community_Day_2023-Chathra Serasinghe.pptx
 
Creating an open source load balancer for S3
Creating an open source load balancer for S3Creating an open source load balancer for S3
Creating an open source load balancer for S3
 
slides (PPT)
slides (PPT)slides (PPT)
slides (PPT)
 
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...How Docker Accelerates Continuous Development at ironSource: Containers #101 ...
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...
 
Silverstripe at scale - design & architecture for silverstripe applications
Silverstripe at scale - design & architecture for silverstripe applicationsSilverstripe at scale - design & architecture for silverstripe applications
Silverstripe at scale - design & architecture for silverstripe applications
 
ENT401 Deep Dive with Amazon EC2 Systems Manager
ENT401 Deep Dive with Amazon EC2 Systems ManagerENT401 Deep Dive with Amazon EC2 Systems Manager
ENT401 Deep Dive with Amazon EC2 Systems Manager
 
Pcp
PcpPcp
Pcp
 
Server Monitoring from the Cloud
Server Monitoring from the CloudServer Monitoring from the Cloud
Server Monitoring from the Cloud
 
EPM Automate - Automating Enterprise Performance Management Cloud Solutions
EPM Automate - Automating Enterprise Performance Management Cloud SolutionsEPM Automate - Automating Enterprise Performance Management Cloud Solutions
EPM Automate - Automating Enterprise Performance Management Cloud Solutions
 
USENIX LISA15: How TubeMogul Handles over One Trillion HTTP Requests a Month
USENIX LISA15: How TubeMogul Handles over One Trillion HTTP Requests a MonthUSENIX LISA15: How TubeMogul Handles over One Trillion HTTP Requests a Month
USENIX LISA15: How TubeMogul Handles over One Trillion HTTP Requests a Month
 
What's new in NGINX Plus R19
What's new in NGINX Plus R19What's new in NGINX Plus R19
What's new in NGINX Plus R19
 
SCCM 2007 Introduction - PICC 2012
SCCM 2007 Introduction - PICC 2012SCCM 2007 Introduction - PICC 2012
SCCM 2007 Introduction - PICC 2012
 

More from Jared Atkinson

More from Jared Atkinson (7)

Red + Blue, How Purple Are You
Red + Blue, How Purple Are YouRed + Blue, How Purple Are You
Red + Blue, How Purple Are You
 
Mapping Detection Coverage
Mapping Detection CoverageMapping Detection Coverage
Mapping Detection Coverage
 
Paranoia 2018: A Process is No One
Paranoia 2018: A Process is No OneParanoia 2018: A Process is No One
Paranoia 2018: A Process is No One
 
Purpose Driven Hunt (DerbyCon 2017)
Purpose Driven Hunt (DerbyCon 2017)Purpose Driven Hunt (DerbyCon 2017)
Purpose Driven Hunt (DerbyCon 2017)
 
BSidesDC - **** it, Do It Live (PowerShell Digital Forensics)
BSidesDC - **** it, Do It Live (PowerShell Digital Forensics)BSidesDC - **** it, Do It Live (PowerShell Digital Forensics)
BSidesDC - **** it, Do It Live (PowerShell Digital Forensics)
 
44CON London 2015: Old Dog, New Tricks: Forensics with PowerShell
44CON London 2015: Old Dog, New Tricks: Forensics with PowerShell44CON London 2015: Old Dog, New Tricks: Forensics with PowerShell
44CON London 2015: Old Dog, New Tricks: Forensics with PowerShell
 
44CON London 2015: NTFS Analysis with PowerForensics
44CON London 2015: NTFS Analysis with PowerForensics44CON London 2015: NTFS Analysis with PowerForensics
44CON London 2015: NTFS Analysis with PowerForensics
 

Recently uploaded

Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Victor Rentea
 
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Victor Rentea
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
panagenda
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
WSO2
 

Recently uploaded (20)

[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf
 
Exploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with MilvusExploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with Milvus
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
Quantum Leap in Next-Generation Computing
Quantum Leap in Next-Generation ComputingQuantum Leap in Next-Generation Computing
Quantum Leap in Next-Generation Computing
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Six Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal OntologySix Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal Ontology
 
API Governance and Monetization - The evolution of API governance
API Governance and Monetization -  The evolution of API governanceAPI Governance and Monetization -  The evolution of API governance
API Governance and Monetization - The evolution of API governance
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
 
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
 
Platformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityPlatformless Horizons for Digital Adaptability
Platformless Horizons for Digital Adaptability
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
 
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 AmsterdamDEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
 
Introduction to Multilingual Retrieval Augmented Generation (RAG)
Introduction to Multilingual Retrieval Augmented Generation (RAG)Introduction to Multilingual Retrieval Augmented Generation (RAG)
Introduction to Multilingual Retrieval Augmented Generation (RAG)
 
Modernizing Legacy Systems Using Ballerina
Modernizing Legacy Systems Using BallerinaModernizing Legacy Systems Using Ballerina
Modernizing Legacy Systems Using Ballerina
 
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Decarbonising Commercial Real Estate: The Role of Operational Performance
Decarbonising Commercial Real Estate: The Role of Operational PerformanceDecarbonising Commercial Real Estate: The Role of Operational Performance
Decarbonising Commercial Real Estate: The Role of Operational Performance
 
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In PakistanCNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistan
 

Automated, Collection, and Enrichment (ACE)