Cloudgrep is a command-line utility designed for searching cloud storage files without the need for indexing, supporting AWS, Azure, and GCP. It allows for fast parallel searches of plaintext, compressed, and binary files, and can run detection rules via regex or YARA. The document includes installation instructions, basic usage examples, and highlights contributions from the community.

1. #BHUSA @BlackHatEvents
cloudgrep
cloudgrep is grep for cloud storage
https://github.com/cado-security/cloudgrep
https://x.com/cadosecurity

2. # BHUSA @BlackHatEvents
Introducing clougrep

3. # BHUSA @BlackHatEvents
So you’ve got an incident in the cloud…
Cloud
(Control Plane)
Forensics
Host Forensics Log Forensics Network Forensics

4. # BHUSA @BlackHatEvents
DFIR in the Cloud == Log Analysis?

5. # BHUSA @BlackHatEvents
Time
● Do you need to ship the logs first?
● Do you need to wait for the logs to be indexed?
● Do you know where the logs are stored?
Cost
● Can you pay to index and store all the data?
● Hot search vs cold data
● Egress fees
Log Analysis in The Cloud -
Challenges

6. # BHUSA @BlackHatEvents
If the data is already in your SIEM and
you're in a position to do this - that's
fantastic. You can stop now :)
But they can also be expensive, and
you need to get the data there.
Modern SIEM’s often use a cheaper
“indexless” approach to search larger
amounts of data at less cost.
Or they even keep logs “in place” to
reduce the cost of moving them.
SIEM Systems

7. # BHUSA @BlackHatEvents
Athena for parsing and searching logs in
S3
CREATE EXTERNAL TABLE IF NOT EXISTS elb_logs_raw_native (
request_timestamp string,
request_ip string,
request_port int,
backend_ip string,
backend_port int, …)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
'serialization.format' = '1','input.regex' = '([^ ]*) ... ([A-Za-z0-9.-]*)$' )
LOCATION 's3://athena-examples/elb/raw/';
SELECT * FROM elb_logs_raw_native WHERE elb_response_code = '200' LIMIT 100;
See also: Automated First-Response in AWS using Sigma and
Athena Invictus Incident Response Invictus Incident Response

8. # BHUSA @BlackHatEvents
● CloudWatch
● CloudSearch
● AWS OpenSearch
● … more
A million ways to search logs in AWS

9. # BHUSA @BlackHatEvents
I just want to grep my cloud files…
grep is a command-line utility for searching plaintext datasets for lines that match a regular
expression. Its name comes from the ed command g/re/p
Mountpoint
https://aws.amazon.com/s3/features/mountpoint/
aws s3 cp s3://bucket/log-folder/ .
--recursive
zgrep "evil" *
mount-s3 DOC-EXAMPLE-BUCKET /path/to/mount
AWS CLI

10. # BHUSA @BlackHatEvents
Introducing clougrep

11. # BHUSA @BlackHatEvents
Overview of cloudgrep

12. # BHUSA @BlackHatEvents
● Simple installation and operation
● No indexing required
● Operates across aws, azure and gcp
● Files are downloaded and searched in parallel for speed
● Low cost operation (see later)
● Able to run detection rules via regex or Yara for file
content
● Can search compressed files
● Can search binary files
Why cloudgrep
MZ@ !L!This program cannot be run in DOS mode.

13. # BHUSA @BlackHatEvents
git clone https://github.com/cado-security/cloudgrep.git
pip3 install -r requirements.txt
AWS
~/.aws/credentials or instance profile
Use a VPC endpoint to S3 on an EC2 to avoid egress charges
Azure
az login
GCP
export
GOOGLE_APPLICATION_CREDENTIALS="/Users/creds.json"
Setup

14. # BHUSA @BlackHatEvents
Demo

15. # BHUSA @BlackHatEvents
Demo
# Basic search
python3 cloudgrep.py -b blackhat-cado-demo -q "ssm.amazonaws.com"
# Debug mode
python3 cloudgrep.py -b blackhat-cado-demo -q "ssm.amazonaws.com" --debug
# Azure
python3 cloudgrep.py -an some_account -cn some_container -q my_search
# GCP
python3 cloudgrep.py -gb my-gcp-bucket -q my_search
# Json parsing and output
python3 cloudgrep.py -b blackhat-cado-demo -q "2020-07-31T23:58:37Z" -lt cloudtrail -jo
# --prefix is a *fast* filter
# Start and end date are a *fast* filter
# --filename is a *slow* filter
python3 cloudgrep.py -b blackhat-cado-demo -q "2024-01-02T02:51:00Z" -s "2023-01-09 20:30:00" -e
"2025-01-09 20:45:00" --file_size 100
# Search a number of queries
python3 cloudgrep.py -b blackhat-cado-demo --file queries.txt
# Search files with a Yara rule
python3 cloudgrep.py -b blackhat-cado-demo --yara yara.rule

16. # BHUSA @BlackHatEvents
Thank-you for all the community contributions!
Particular shout out to Daniel Bohannon, Andi
Ahmeti and the rest of the team @ Permiso for:
- Log parsing support
- JSON output
Also see their tool Cloud Grappler for threat
detection against logs in cloud storage, using
cloudgrep as the search library.
Community Contributions

17. #BHUSA @BlackHatEvents
cloudgrep
cloudgrep is grep for cloud storage
https://github.com/cado-security/cloudgrep
https://x.com/cadosecurity