AEO Training - 2023.pdf

OSSEC Con: Day 3 Agenda
09:30: Pre-class setup time
10:00: Workshop
11:00: Lab Time
12:00: Lunch
13:00: Workshop
14:00: Lab Time
15:00: Workshop
16:00: Lab Time

OSSEC Con: Day 4 Agenda
09:30: Pre-class setup time
10:00: Workshop
11:00: Lab Time
12:00: Lunch
13:00: Workshop
14:00: Lab Time
15:00: Workshop
15:45: Break
16:00: Exam

Download links
git clone
https://github.com/Atomicorp/training
Contains the all the examples used here
for this workshop:

OSSEC 2021 Workshops
● Installation (Server)
● Basic Server Conﬁguration
● Installation (Agents,
Ansible)
● Troubleshooting
● Just FIM
● Agentless / Drift detection
● Advanced: AWS cloudtrail,
Rules/Decoders, Atomic
Inspector

Atomicorp Workshop: AEO Overview
Atomic Enterprise OSSEC is a Host Based Intrusion Detection System
● Log based Intrusion Detection (LIDS)
● File Integrity Monitoring (FIM)
● Vulnerability Scanning
● Compliance Auditing
● Rootkit detection
● Active Response

Examples and Exam data:
git clone https://github.com/atomicorp/training
Atomicorp Workshop: Workshop setup

Threat Landscape
Unstructured Threat - “Background noise”, non-human automated
attacks. Bots, virus, etc
Structured Threat - Organized, human driven, targeted campaigns
Highly Structured Threat - Intelligence agencies, MICE, extra-judicial
access

Atomic
● Single Event
● Ex: Remote attempt to MSSQL
Composite
● Multiple Events
● Ex: Logon failure + Logon Failure + Logon
Failure = 3 Logon failures
Content
● Misuse Detection
● Ex: Virus detected at /path/to/file
Context
● Anomaly Detection
● User logged in from GEO source not seen
before

Log Based Intrusion Detection / FIM
● Inspects log files
● Runs commands (Ex: check disk space)
● Tracks file/registry changes
Pros
● Low(er) false positive Rate
● Not resource intensive
● Encryption is not an issue

Cons / Blind Spots
● if it is not logged, or stored on the filesystem, it is not detectable
● Cannot inspect egress
● Cannot inspect network traﬀic (ie: port scans, etc)
● Custom applications need to have their own rules / decoders
developed when they are introduced/changed

Alert Behavior
When a rule triggers an alert, one or more actions can be configured
● logging (locally, or to one or more remote SIEMʼs)
● sending an email or sms alert
● execute a custom action

Decoders turn this
Jul 20 08:35:07 app3 sshd[22421]: Invalid user flavio from 23.83.239.130 port 42814
into this:
<hostname> app3
<program_name> sshd
<username> flavio
<src_ip> 23.83.239.130

Rules evaluate this message as an Atomic event
<rule id=”123” level=”3”>
<program_name>sshd</program_name>
<match>Invalid user</match>
<description> SSH: Invalid username login attempt>
</rule>

Rules evaluate 123 as a Composite event
<rule id=”456” level=”10” frequency=”3” timeframe=”60” ignore=”60”>
<if_matched_sid>123</if_matched_sid>
<same_source_ip />
<description> Brute Force: multiple attempts from the same source
</description>
</rule>

FTS - First Time Seen
GEO SRC/DST IP - Country/City codes (ex: France / Paris)
same_user - conditional modifier based on the username field
same_source_ip - conditional modifier based on srcip
same_field - Dynamic field identifier.
same_location - same source of data, ie: syscheck, /var/log/messages,
etc
same_id - same id field, usually used with NIDS or WAF components

Atomicorp Workshop: Media Company Case Study
10,000 systems
150 GB/day in log traﬀic
Store for 3 years
Integrating with an external SIEM (Sumo)
Atomicorp Workshop: Media Company Case Study

Atomicorp Workshop: Case Study (The Problem)
Data with No information (no usernames, ips, filenames, errors, etc)
Disk space and Processing Overhead
SIEM charges based on volume
Analysts overwhelmed by data, performing intrusion detection manually

1 server with 1000 agents
Before:
4 million events per hour (1100/sec)
40.48 GB data per day
After:
783,211 events per hour (217/sec)
6.81 GB data per day
81.4% reduction in SIEM traﬀic, savings of over $270,000 per quarter

OSSEC Workshop 1 : Installation
● Server builds, and common
conﬁguration settings
● Agent builds, installation
automation
● Group Management

Atomic OSSEC Workshop 1 : Installation /
Configuration
● Advanced Atomic OSSEC
installation
● Basic Server configuration
○ Email notification
○ FIM configuration
○ SSL certificates

OSSEC Workshop 1: Build a Server
● Server builds, and common issue troubleshooting
● You will need:
○ Rocky Linux 8 or Centos 7
○ Basic linux navigation
○ License Credentials
○ Slack channel access
○ Internet access to the vendor (Rocky, Centos)
○ Internet access to the atomicorp repos
(updates.atomicorp.com)
Atomic OSSEC Workshop 1: Build a Server

Recommended conﬁguration for 1000 endpoints
● 16G ram / 4 Cores
● 250GB disk (resizable)
● Vigorous partitioning is NOT recommended
Minimum Conﬁguration for 10 endpoints
● 4G ram / 2 Cores
● 100GB disk at /var
Atomic OSSEC Workshop 1: Build a Server

OSSEC Workshop 1: Building the Server
● Avoid it if you can! Complicated partitioning WILL increase your
operational costs.
● Plan to grow your partitions
● MOST space will be under /var
○ /var/cache/yum - package updates
○ /var/www/html - archive mirror
○ /var/lib/docker - Atomic Inspector
○ /var/awp - Atomic OSSEC framework
○ /var/ossec - Atomic OSSEC
Atomic OSSEC Workshop 1: Partitioning
EXAM question: Where will Atomic OSSEC use the most
space?

Live Build of a new Rocky Linux 8 system. Highlights:
● Hypervisor is KVM
● Using a network installation method
● Pick Minimal installation
Atomic OSSEC Workshop 1: LAB time

1) Does the environment have internet access? Yes/No
2) Does the internet access require a proxy? Yes/No
3) Does the system have working yum/dnf repositories? Yes/No
4) Are you able to run this as root? Yes/No
if 1 is No, then you need the Offline installer
if 2 is Yes, then you need to declare the Proxy settings
if 1 is No, then you need the Offline installer OR you need to
register the host/fix the repos
if 1 is No, then you need to get root/sudo access
Atomic OSSEC Workshop 1: Installation pre-flight

INSTALLATION LAB

Log in to the system:
Run:
curl -so https://updates.atomicorp.com/installers/awp-hub

Atomicorp Hub Installler
Version: 7.2.0
Usage: awp-hub [options]
Options:
--https-proxy=<URL> will set the https_proxy environment variable
--https-proxy-username=<PROXY_USER> will set the https_proxy_username environment variable
--https-proxy-password=<PROXY_PASS> will set the https_proxy_password environment variable
--beta will install from the beta repository
--standalone local install (no hub)
Atomic OSSEC Workshop 1: Installation pre-ﬂight

INSTALL AEO LAB
Atomic OSSEC Workshop 1: Server Conﬁguration

● You will need:
○ Atomic OSSEC server
○ Network Security policy allowing
■ TCP 22 INBOUND
■ TCP 80/443 INBOUND
■ TCP 30001 INBOUND

OSSEC Workshop 1: Server Components
● ossec-analysisd : IDS analysis (rules/decoders)
● ossec-remoted : Listener for agent trafﬁc
● ossec-syscheckd : FIM daemon
● ossec-logcollectord : Log collector daemon
● ossec-execd : Active Response daemon
● ossec-monitord : Logrotation, cleanup, and reporting daemon
● ossec-maild : Mail User Agent daemon
● ossec-dbd : Database Connector daemon
● ossec-authd: Agent registration daemon
● ossec-clusterd: Clustering daemon
● ossec-integratord: integration daemon
Atomic OSSEC Workshop 1: Server Components

Troubleshooting:
Startup/debug logs here:
/var/ossec/logs/ossec.log
Typos? Invalid conﬁgs? run this:
/var/ossec/bin/ossec-analysisd -t
EXAM QUESTION: Where are the ossec debug logs?

Troubleshooting
● OSSEC Daemons exercise the chroot() function when running.
● Chroot() cannot traverse ﬁlesystems
EXAM QUESTION: Can I assign /var/ossec/log to its own ﬁlesystem?

Troubleshooting
● Alerts are stored at /var/ossec/log/alerts/alerts.json
● Alerts are rotated/compressed nightly to
/var/ossec/log/alerts/YYYY/MON/
EXAM QUESTION: Where are alerts stored?

Configuration Lab for Letsencrypt or Externally
provisioned certificates
OSSEC 2021 Workshop 1: Server Configuration

Highlights
● LetsEncrypt requires internet access
● Requires a valid hostname
● Logs to /var/log/letsencrypt/
● Solves SSL issues for yum, dnf, apt, registration, and
the console
OSSEC 2021 Workshop 2: SSL LetsEncrypt Lab

● You will need:
○ A valid domain name for your server (Letsencrypt)
OR
○ Externally generated certiﬁcate
○ Network Security policy allowing
■ TCP 22 INBOUND
■ TCP 80/443 INBOUND (from letsencrypt)
OSSEC 2021 Workshop 1: SSL Conﬁguration

OSSEC 2021 Workshop 2: SSL LetsEncrypt Lab

LetsEncrypt Lab
OSSEC 2021 Workshop 1: SSL Conﬁguration

Conﬁguration Lab for Email, Grouping, and FIM
conﬁguration

Email Lab

Highlights:
● Email can come from the Hub server OR an
externally conﬁgured MTA
● Logs are at /var/logs/maillog
● Downstream problems:
○ Spamﬁlters
○ Max size limits

Grouping Lab

Highlights
● Groups are for organization
● Groups are for conﬁguration
● Agents join a group AFTER they initialize. Broken
agents are “Unassigned”. Generally this is a 1514 port
issue
EXAM Question: An agent reports as being “Unassigned” in the interface,
what does this mean?

File Integrity Monitoring Conﬁguration Lab

Highlights
● FIM conﬁguration is handled on the server
● Picked up in a few minutes
● Ignores do not support wildcards yet
EXAM Question: Does FIM support ignoring through wildcards?

● Agent installs, and common issue troubleshooting
● You will need:
○ Rocky Linux 8 or Centos 7 agent system
○ Basic linux navigation
○ Security Groups (agent to server)
■ TCP Port 80/443
■ TCP Port 1514
■ TCP Port 1515
■ Access to vendor yum/dnf repos
Atomic OSSEC Workshop 2: Agent installation

● I got that last drawing when I asked an AI to
“describe the connection between the OSSEC agent
and OSSEC server”
● Needless to say it is literally wrong, but
metaphorically correct. Probably.
● Agents talk to servers, by default on UDP port 1514,
but you can use any port TCP or UDP.
EXAM Question: Does the agent connect to the server, or does the server
connect to the agent?

OSSEC Workshop 2: Installation Automation

Atomic OSSEC Workshop 2: Installation Automation

Log in to Agent
curl -so -k https://<hub
ip>/installers/ossec-installer.sh
chmod +x ossec-installer.sh
sudo ./ossec-installer.sh <hub ip>
Atomic OSSEC Workshop 2: Agent Installation

Highlights
● Agents are installed using the package manager
(yum/dnf)
● Logs are at /var/ossec/logs/ossec.log
● Windows installer is at http://<hub
ip>/installers/windows-installer.ps1

Agent Installation Lab

● You will need:
○ Working SSH keys and SSH-Agent
○ SSH Access to the agent
Atomic OSSEC Workshop 2: Agent Installation with
Ansible

● You will need:
○ Working SSH keys and SSH-Agent
○ SSH Access to the agent
● This is available in both the UI and the CLI
Ansible

Ansible

Ansible Agent Installation Lab
OSSEC 2021 Workshop 2: Agent Installation with Ansible

Highlights
● Real-Time
● User tracking depends on auditd working correctly
● Forensic ﬁles are at /var/ossec/queue/diff/local/
● This works on Linux, Windows, and Mac (but not
legacy systems like AIX, Solaris, or HP-UX)
Atomic OSSEC Workshop 3: Just FIM

● Create a new file (/etc/testfile1.txt)
● Observe the alert
● Modify this file
● Delete this file
● (UI) Filter for FIM
● (CLI) Track changes, location of forensic files

FIM Lab

You will need
● A working Atomic OSSEC hub
● A working Agentless device (GNS3 example)
● Working SSH keys to this device
Atomic OSSEC Workshop 4: Agentless

FIM
● Slower
● Cannot make forensic copies
● Cannot track user/parent user
● Useful for devices that cannot run an agent
Drift Detection using Ansible
● Maintains copies of conﬁguration ﬁles as they
change
● Stores these in a git repository

Agentless LAB

Atomic OSSEC Workshop 5: Installation Automation
● Windows using Active Directory,
Powershell, and reboots
● You will need:
○ Active Directory server (win2016)
○ Windows 10 agent
○ Powershell
○ OSSEC Server
○ Webserver
Open Hyperqube environment:
OSSEC 1

OSSEC Workshop 2: Using a GPO
● This installs when the Windows 10 system reboots
● Active Directory GPO configures the system to
○ Copy the powershell installer to the system from share
○ Run the installer as SYSTEM
○ Pass variables to the powershell script for the server IP
● Gotchas:
○ Package signing can break installs over shares
○ Firewalls can break registration
○ Permissions!
Atomic OSSEC Workshop 5: Using a GPO

OSSEC Workshop 2: Using a GPO
● Example uses powershell, this is probably overkill
● This can be used for
○ new installs
○ upgrades
○ re-keying
Atomic OSSEC Workshop 5: Using a GPO

OSSEC Workshop 2: Using a GPO Workflow
● Installs and configures the agent on a host reboot
● Runs 1 time
● copies installer.ps1 from SYSVOL to the system
● Agent runs installer.ps1 locally as SYSTEM user**
● Downloads software to C:ossec-agent-latest.exe
● Installs ossec-agent-latest.exe
● Registers the agent with the hub server with agent-auth.exe
● Configures ossec.conf and starts the agent on the host
**you can change this to a domain admin, etc
Atomic OSSEC Workshop 5: Using a GPO Workﬂow

OSSEC Workshop 2: Using a GPO Step 1
Server manager select Tools
Group Policy Management
Select domain: atomicorp.local
Right click on the domain, create GPO and Link it here
name this: install1
Right click on install1 and select Edit
Atomic OSSEC Workshop 5: Using a GPO Step 1

Select Computer Configuration
Preferences
Windows Settings
Files
Select New->Files
set the Action to Create
set path to source file : Ad-serversysvolatomicorp.localinstaller.ps1
set path to destination on host: C:installer.ps1
click OK

Select Computer Configuration
Preferences
Control Panel Settings
Scheduled Tasks
Right click and select New->Immediate Scheduled Task (At
least windows 7)
Enter name: install-agent
Enter description: OSSEC agent
Select when running task use the following user account:
SYSTEM

OSSEC Workshop 2: Using a GPO Step 2 cont.
Select run whether user is logged on or not
Select Run with highest privileges
Select configure for Windows 7, windows server 2008R2
Atomic OSSEC Workshop 5: Using a GPO Step 2 cont.

Select action tab, and click New
Enter in program/Script: powershell.exe
Enter in Add arguments:
-executionpolicy bypass -file C:installer.ps1 -ossec_exe
http://192.168.1.102/ossec-agent-latest.exe -server_ip
192.168.1.102
Click OK, select the Common tab, and check Apply once and do
not reapply. Click OK

Log in to the OSSEC server, and run:
tail -f /var/log/httpd/*
Log in to the Windows 10 system, and reboot.
You should see the windows 10 system request the ossec
package, and in a few minutes complete the installation

OSSEC Workshop 2: Troubleshooting
Can the new agent read the share drive?
check the win10 system if it copied C:install1.ps1
Did the GPO run?
from the win10 system, run: gpresult /r
Did the agent register?
from the ossec server, run /var/ossec/bin/agent_control -l
Atomic OSSEC Workshop 5: Troubleshooting

OSSEC Workshop 2: Bonus Round Cloud-Init
The Problem:
Dynamic scaling on Amazon (Google, Azure, etc)
OSSEC agent keys have to be unique
Solution:
Cloud-init
Atomic OSSEC Workshop 6: Bonus Round Cloud-Init

Launched in 2008: https://cloud-init.io
Supports more than 20 public cloud providers
Openstack, LXD, KVM, etc
Adds an “init” type API to the operating system for:
per-once: First time the system has ever booted
per-boot: Every time the system boots
per-instance: First time a cloned (dynamic scaling) instance
boots
Available for: Ubuntu, Debian, Redhat, Centos, *BSD, and more

Our action is simple, just rekey the agent:
/var/ossec/bin/agent_auth -m 10.10.10.10
But we need to do this immediately without requiring a human
or external devops action.

What about rc.local?
It would work, however it rc.local happens after the regular
ossec-agent daemon starts
It could result in creating even more keys, given that the
rc.local is set at the master instance level.
We need something smarter

Itʼs this easy:
cat /var/lib/cloud/scripts/per-instance/ossec-agent.sh
#!/bin/sh
/var/ossec/bin/agent-auth -m 10.10.10.10

Atomic OSSEC Workshop 7: Advanced Topics
● Network Troubleshooting
● Central Management with
shared/
● Rootcheck: Malware detection,
Compliance Testing,
Application discovery
● Malware / FIM whitelisting
(filename)

OSSEC Workshop 3: Network Troubleshooting
Atomic OSSEC Workshop 7: Network Troubleshooting

OSSEC Workshop 3: Network troubleshooting
Scenario 1, agent_control reports “Never Connected”
This indicates the TCP Port 1515 (authd) registration completed
successfully, but the agent communication is blocked
● Check the agent to ensure the server ip is correct and the agent is started
● Use a sniffer on the Server to watch for TCP/UDP 1514 traffic from the host:
tshark -i eth0 port 1514
● No traffic means a firewall is blocking TCP/UDP 1514 at some point
EXAM Question: What does “Never Connected” mean when
installing NEW agents

Scenario 2, agent_control reports “Disconnected”
This indicates the UDP Port 1514 had worked in the past, but
the agent communication is blocked
Run: /var/ossec/bin/agent_control -i <ID> to see when the
agent last checked in successfully
Is the agent running?
Is the Server IP correct?
Is a firewall blocking UDP 1514?
Is its key good?

agent_control cheatsheet
“Never Connected” - This means agent registered (TCP 1515) but
has never connected over remoted (UDP 1514)
“Disconnected” - Agent registered (TCP 1515) and had previously
connected over remoted (UDP 1514) but is no longer online
“Active” - Everything is fine!
“Pending” - a transitional state, the agent is in the process of
connecting. This is only an issue if it takes a long period of time

Interna Options : /var/ossec/etc/internal_options.conf
<key>.debug=0
Debug 0 : no debugging
Debug 1: Level 1 debugging
Debug 2: Level 2 debugging
Logs to /var/ossec/logs/ossec.log
EXAM Question: Where do you enable debug logging in OSSEC

Real edge case issues:
● Maximum size of a UDP packet was 400 bytes, this allowed
partial agent logins and manifested as “constant disconnects”.
Cause: Network filters in AWS
● Permission denied errors when updating agent: Cause, /var was
too small, yum could not complete its actions. User had added
/var/ossec after the fact

LAB

OSSEC Workshop: Active Response
● Block source addresses (srcip)
● Disable Accounts (username)
● Malware / FIM whitelisting (filename)
● Self-healing (pin to a rule)
● Reporting (JIRA, slack, etc)
● PaaS API (cloudflare, aws, etc)
● IFTTT
● Amazon Echo / Google Home
● etc!
Atomic OSSEC Workshop 8: Active Response

● ossec-execd runs active response (ossec-agent on windows)
○ Commands live in: /var/ossec/active-response/bin/
○ This daemon forks! Beware! Job control is up to you!
○ Context:
■ srcip
■ username
■ filename
■ or no context at all

● Can run on:
○ where the attack happened
○ a specific system
○ every system
● Configured from the server, but the action has to be on the
agent (except… repeated_oﬀenders...)
● ARs can be in any language (Powershell, bash, python, go, etc)
● Timed, Repeat oﬀenders, or no timer
● Active response can be configured in TWO places
○ /var/ossec/etc/ossec.conf or in a rule

OSSEC Workshop: Active Response Values
● Action (add or delete)
● Username (ex: testguy)
● IP address (ex: 1.2.3.4)
● Alert ID (ex: 1552939106.13039)
● Rule ID (ex: 553)
● Agent (ex: (testagent1.atomicorp.com))
● Location (ex: 10.10.10.10->syscheck)
● Filename (ex: /mnt/test1)
Atomic OSSEC Workshop 8: Active Response Values

● In a ossec.conf
<command>
<name>syscheck-api</name>
<executable>syscheck-api</executable>
<expect>filename</expect>
</command>
<active-response>
<command>syscheck-api</command>
<location>server</location>
<level>5</level>
<rules_group>syscheck</rules_group>
</active-response>

● In a rule:
○ <action> to declare the name of the script
○ <status> to pass the add or delete value
<rule id="601" level="3">
<if_sid>600</if_sid>
<action>firewall-drop.sh</action>
<status>add</status>
<description>Host Blocked by firewall-drop.sh Active Response</description>
<group>active_response,</group>
</rule>

OSSEC Workshop: Active Response Utils
● List: /var/ossec/bin/agent_control -L
Response name: test-all0, command: test-all.sh
Note: 0 indicates the timer, if set. Not set in this example
● Run manually (I use this for testing) Example:
/var/ossec/bin/agent_control -b 1.2.3.4 -f test-all0 -u 000
Atomic OSSEC Workshop 8: Active Response Utils

OSSEC Workshop: Active Response Utils
Debugging Tip: syscheck wont start generating events until
rootcheck finishes its job. Rootcheck can take a while, so turn it oﬀ
for development
Debugging Tip: Not clear if syscheck is running? Tail ossec.log and
look for “Ending syscheck scan”. After this, perform your tests
Atomic OSSEC Workshop 8: Active Response Utils

OSSEC Workshop: Active Response FILENAME
● Simulation and Testing configuration
● Syscheck can take a long time to run, for this workshop we will
set the following to speed things up:
○ <directories realtime="yes" check_all="yes" report_changes="yes">/mnt</directories>
○ disable rootcheck
○ internal_options.conf
■ syscheck.sleep=1
■ syscheck.sleep_after=150
Atomic OSSEC Workshop 8: Active Response FILENAME

cd /root/src/workshop2020/lab03/active-response
/var/ossec/bin/ossec-control stop
cp ossec.conf /var/ossec/etc/
cp internal_options.conf /var/ossec/etc/
cp syscheck-api.sh /var/ossec/active-response/bin/
/var/ossec/bin/ossec-control start

● perform actions against FIM events
● active response configuration key values:
○ <expect>filename</expect>
○ <rules_group>syscheck</rules_group>
This example only logs the script being run. Restart OSSEC and
Create a test file:
date >> /mnt/testfile1

● Update /mnt/testfile1:
date >> /mnt/testfile1
Generates 552 event, and logs:
Tue Mar 10 09:04:59 EDT 2019
/var/ossec/active-response/bin/syscheck_all.sh add - -
1553000699.9105 552 field6(syscheck) Filename: (/mnt/hosts)
field8() field9() field10(add0)

Atomic OSSEC Workshop 9: Rules and Decoders
● Dynamic Decoders
● Atomic Rules
● Composite rules

OSSEC Workshop: Dynamic Decoders
● /var/ossec/bin/ossec-logtest - Command line utility for
developing rules
● /var/ossec/logs/archives/archives.log - Raw log data
pre-processed

● /var/ossec/etc/decoders.d/ - decoders
● /var/ossec/etc/rules.d/ - rules
Each class is separated into its own file, and loaded via
globbing.

Archives.log
● Disabled by default, can be enabled in the UI
● Captures the data as it is received by analysis
● Adds a header, which you must remove

2023 Feb 03 11:01:15 awp-hub-rocky8->/var/log/secure Feb 3 11:01:14
awp-hub-rocky8 sshd[788381]: Accepted publickey for root from 192.168.1.1 port
40320 ssh2: RSA SHA256:w58xBpETtCACkxUS93OAjCtFk5euIny6xXuP0X7vw4E
2023 Feb 03 11:01:15 - Timestamp when it was received by the hub server
awp-hub-rocky8->/var/log/secure - The location field, an agentless syslog event would
include the senders IP address here
Trailing whitespace - there is a trailing whitespace here!
Remove all of this.Do not forget to remove the trailing whitespace!

This is the event after truncating the headers from the event.
Feb 3 11:01:14 awp-hub-rocky8 sshd[788381]: Accepted publickey for root from
192.168.1.1 port 40320 ssh2: RSA
SHA256:w58xBpETtCACkxUS93OAjCtFk5euIny6xXuP0X7vw4E
● Other applications may introduce characters into their event streams that your
terminal or browser may not interpret.
● archives.log is your source of truth for the way the even was really received

Atomicorp Workshop 9: Rules and Decoders

● Decoders turn events into a key->value store
● This is where we use regular expressions to identify srcip for
example
● Key->Valueʼs can be used in rules, like srcip, or username
● Key->Values can also be used in Atomic Inspector for custom
fields

Example:
Paste workshop2023/workshop09/01/event-01.txt into
logtest
This is an event from the kernel security project,
grsecurity.net. It is capturing an exploit attempt against
/usr/local/psa/admin/sbin/filwrpr, it captures the IP
address, and user ID and parent userID

Parent decoder is the generic “kernel”
/var/ossec/etc/decoders.d/50-asl-kernel-decoder.xml
Child decoder is “grsecurity-generic”
A decoder can have multiple children, however a decoder child can not
have a child decoder.
Decoders instantiate “buckets”. This allows a rule group to be
associated with a specific feed of data.

Interactive Lab
workshop2023/workshop09/02/
regex-cheatsheet.txt - OSSEC regular expressions
custom-event1.txt - list of 3 sample events for this lab
Successful login
Login failure
A user action

Interactive Lab
V0.1 (99-khand-custom-decoder.xml.v0.1)
This just captures the program_name field in logtest

Interactive Lab
This just captures the program_name field in logtest
restart logtest each time you make a change!

Interactive Lab
Our child decoder now writes to the key “data”
Note that this has automatically stripped the timestamp,
hostname, and program_name fields.

Interactive Lab
Our child decoder now writes to the keys srcip and “data”
Note that this data just contains the content after the srcip field.

Interactive Lab
Our child decoder now has keys for “srcip” and “user”
Note that user is a special key, and gets recorded as “dstuser”.
This is user/username/dstuser internally.

Interactive Lab (Bonus Round: oﬀset)
● oﬀsets are optional
● They are used to enhance performance, or in rare cases
reduce complicated event data.

Interactive Lab (Bonus)
● Now we create a 2nd child decoder, for our record event
● A decoder cannot use another child as the parent, we can
only go 2 tiers with this design.

The Rule for Rules
● level 0-15. 0 is evaluated first, THEN 15
● Must be unique IDʼs unless the overwrite flag is declared
● Load your rules last! (99-something-rule.xml)
● Level 0 is a “bucket” rule, use this for performance
● Rules generate alerts, if nothing matches a rule, nothing is in
alerts.json**
** but they are in archives.log

The Rule for Rules (Level 0)
● Do not generate alerts
● Intended as your anchor for IDS logic, ideally oﬀ a decoder
● Rules do not need decoders, but its bad practice if you dont
● Should always be Atomic, and never be Composite
● Use rule IDs for custom rules in 200000-220000
● Always load your rules last, as 99-yourname-rule.xml

Interactive Lab
V0.1 (99_custom_khand_rules.xml.v0.1)
● Did you notice our example triggered 2501 on login failures?
● Rules have to be in a group, and a group value should end in
,
● Weʼll put all of khand into rule 210000 as a level 0
● Use both success and failure samples here

Interactive Lab
● This is an atomic rule
● We use if_sid like an if/then statement
● match is a simple expression, and very very fast

Interactive Lab
● This is an atomic rule just like before
● We use if_sid like before, on the same bucket rule, 210000
● Note that I raised the level to a 3, since its a failure and I am
slightly more concerned about khan getting his hands on
the genesis device

Interactive Lab
● We use if_sid like before, on the same bucket rule, 210000
● But then we have another if_sid oﬀ of 210003, this lets us
further refine our logic
● Match is now on an audit event, rather than a login event
● Audit events are level 3

Interactive Lab
● Now we really want to split hairs, its an audit event
● Its engaged
● Its genesis! Level 15 Weʼre gonna get another movie!

Interactive Lab
● This is an composite rule
● composite rules count atomic rules using the key <if_matched_sid>
● frequency = # of events, timeframe = period of time
● ignore indicates “do not report this alert again for X seconds”
● We anchor on login failures from the same source ip

OSSEC Workshop: Rootcheck Lab
restore ossec.conf to re-enable rootcheck
cp /var/ossec/etc/ossec.conf.org /var/ossec/etc/ossec.conf
/var/ossec/bin/ossec-control restart
Atomic OSSEC Workshop 10: Rootcheck Lab

OSSEC Workshop: Rootcheck
What to know:
rootcheck scans the filesystem on a timer.
No really. Rootcheck. Scans. The. Filesystem.
Syscheck (FIM) will not report events until rootcheck has finished
starting
Centrally managed from /var/ossec/etc/shared/<groupname>
Atomic OSSEC Workshop 10: Rootcheck

OSSEC Workshop: Rootcheck Capabilities
Capabilities
● Can look at the content of a file/registry
● Tests for processes
● Examine directories
Compliance: cis_rhel7_linux_rcl.txt
Malware: rootkit_files.txt, rootkit_trojans.txt
Application Inventory: win_applications_rcl.txt
Atomic OSSEC Workshop 10: Rootcheck

OSSEC Workshop: Rootcheck Compliance
edit /var/ossec/etc/shared/cis_rhel7_linux_rcl.txt
File example, detect partitions, 1.1.1
This reads /etc/fstab, and looks for a string
f:/etc/fstab <- FOR this file
!r:/tmp <- regular expression for this value.
This test fails (!) if /tmp is not detected in /etc/fstab
Atomic OSSEC Workshop 10: Rootcheck Compliance

Process lookup example:
Goto 3.2, remove X Windows
f:/usr/lib/systemd/system/default.target
r:Graphical (looking for the string )
OR
p:gdm-x-session; <- this is looking for the running process
Both conditions will flag this event

Gotchas and Advanced Usage
edit /var/ossec/etc/shared/system_audit_rcl.txt
$web_dirs=/var/www,/var/htdocs
d:$web_dirs -> ^.ssh
the above will crawl every directory tree declared in web_dirs
looking for the directory “.ssh”. This IOC detection can be IO intensive
depending on the size or type of directory. Realtime FIM is an alternative

OSSEC Workshop: Rootcheck Malware detction
Simple:
d:$web_dirs -> ^.htaccess -> r:RewriteCond S+HTTP_REFERERS
S+google;
rootkit_trojans, this is performing a binary search
ls !bash|^/bin/sh
Registries, win_malware_rcl.txt
r:HKEY_CURRENT_USERSoftwareMicrosoftWindowsCurrentVersion
Run -> userinit -> r:ntos.exe
Atomic OSSEC Workshop 10: Rootcheck Malware
detection

OSSEC Workshop: Rootcheck Application Inventory
win_applications_rcl.txt
[Remote Access - gotomypc]
f:Program FilesCitrixGoToMyPCg2comm.exe
r:HKLMsoftwaremicrosoftwindowscurrentversionrun ->
gotomypc;
p:r:g2svc.exe
Atomic OSSEC Workshop 10: Rootcheck Application
Inventory
FILE:

OSSEC Workshop: Rootcheck a new compliance
test
Open one of the CIS benchmark PDFʼs from your desktop: Ubuntu
Save yourself some time, copy the debian benchmark to:
cis_ubuntu18_linux_L1_rcl.txt
Atomic OSSEC Workshop 10: Rootcheck a New
Compliance
Test

OSSEC Workshop: Rootcheck v2
/var/ossec/ruleset/sca/cis_rhel7_linux.yml
YAML based
Dynamic JSON fields
Test types:
c: command
f: file
d: directory
Operators:
r: regular expression
n: number compare
Atomic OSSEC Workshop 10: Rootcheck v2

/var/ossec/ruleset/sca/cis_rhel7_linux.yml Cont
Conditions, returns true or false
all- every test matches
any - any test matches
none - no test matches

- id: 5530
title: "Ensure discard services are not enabled"
description: "discard is a network service that simply discards all data it receives. This service is
intended for debugging and testing purposes. It is recommended that this service be disabled."
rationale: "Disabling this service will reduce the remote attack surface of the system."
remediation: "Run the following commands to disable discard-dgram and discard-stream: #
chkconfig discard-dgram oﬀ; # chkconfig discard-stream oﬀ"
compliance:
- cis: ["2.1.3"]
- cis_csc: ["9.1"]
- pci_dss: ["2.2.3"]
- nist_800_53: ["CM.1"]
condition: none
rules:
- 'c:chkconfig --list -> r:^s*t*discard-dgram:s*t*on'

Step 1)
Create the cloudtrail in AWS with the DEFAULT
location, ie: do not customize the S3 path
Step 2)
Ensure the Access Key has the rights to read this key
Step 3)
Add the Access Key in the UI
Step 4)
Add the Configuration to /var/ossec/etc/ossec.conf
Atomic OSSEC Workshop 11: AWS Cloudtrail

Step 1)
Use the group “default” here, otherwise if you use a custom name here
adjust the aws_profile field in the next slide
Step 2)
Enter the Access Key
Step 3)
Enter the Secret Key
Step 4)
Enter the region. Note: This region MUST be the same for the role in
IAM/S3/Cloudtrail, if it is incorrect it will fail

Example of manual configuration:
<wodle name="aws-s3">
<disabled>no</disabled>
<interval>10m</interval>
<run_on_start>yes</run_on_start>
<skip_on_error>yes</skip_on_error>
<bucket type="cloudtrail">
<name>aws-cloudtrail-logs-234176589015-20a92ef4</name>
<aws_profile>default</aws_profile>
</bucket>
</wodle>
aws-cloudtrail-logs-234176589015-20a92ef4 - This is the name generated by AWS, use their defaults!
Dont forget to restart OSSEC!

Troubleshooting
● The roles assigned in AWS are nearly always the issue if there is a failure to
extract the data. Double check the rights for the key, and if necessary create a
new role/key
● S3 bucket alternate paths are not supported, if there was some type of
customization beyond the amazon defaults, then that will prevent the module
from accessing the file
● Cloudtrail is not real-time, records may take up to an hour to be recorded,
depending on the region

AEO Workshop: Lab 1 AEO Navigation
Dashboards
Vulnerability (and compliance!)
Trend charts for vulnerabilities
Impact score
Compliance trends
Top 10 agent data
Events (Log based IDS)
Sorted by severity

AtomicWP
WAF & HIDS Rules: Configure rules for WAF and HIDS engines
File Integrity: Configure FIM for the AEO hub only
Agent management: Configure agent groups, agent FIM and AV
settings
TWAF Configuration: WAF reverse proxy
AWP Configuration: AEO hub settings (email, alerting, etc)
AWP Web Configuration: Role Based Access Control for the AEO
console
AWP Support: Support options, SSH, VPN, Create tickets
SSL Management: Configure SSL certificates on AEO

Access Control
IP Controls: Active response blocking system
Firewall Rules: Advanced firewall editor for the AEO hub firewall
policy
Connections: User and IP connection tracking system for the AEO
hub

Reporting
Event Search: Common event search interface (agent, source, level,
etc)
Hub Status: Local hub status view (updates, vulnerabilities, etc)
Systems scan: AEO Hub scan interface
Compliance Reports: AEO hub compliance reports
Login Failures: Pre-configured Windows login failure report

Integrations
Remote Syslog: inbound and outbound syslog settings, integration
for SEIM and other analytics systems
Remote Archiving: Amazon Glacier support
Cloudflare API: Cloudflare CDN active response integration
Custom Active Responses: Configure custom actions for events
OpenID Connect: Single Sign On (SSO)
Webauthn registration: Hardware token support (yubikey, etc)

Set Default SSL certificates for HTTPS, 30001 and 1515
AtomicWP
SSL Management
Upload as Text OR
Letsencrypt (Internet access required!)
Valid certificates are recommended, but not required
AEO Workshop: Lab 1 AEO Setup

Single Sign On (SSO) support
Uses OpenID Connect, more than 70 providers supported.
(OAUTH2.0, ADFS, Gsuite, Redhat SSO, IBM identity manager, SAML, and
more)
Integrations
OpenID Connect
Name: <user defined>
Provider URL: <from provider>
Client ID: <from provider>
Client Secret: <from provider>
Redirect URL: <from provider>

Support
Grant Atomicorp SSH access, Support VPN, Create Tickets
AtomicWP
AWP Support
Support Key: Installs/Uninstalls Atomicorp support ssh keys
(inbound internet access required)
Remote Support: Sets up VPN to atomicorp to allow remote
support
Submit a ticket: Opens ticket in atomicorp.zendesk.com

Atomic Inspector
● Module for AEO
● Analyst Centric
● Containerized
● Originated as our SaaS
platform

AEO Workshop: ELK Dashboards 01
AEO Workshop: Atomic Inspector

User Persona
● Analysts
● Threat Hunters
● Auditors / Auditee

Supports installations into environments with no internet connectivity
Can be used to consolidate all services into a single port (TCP 443)
● Agent Communication
● AEO hub
● Agent Installation
● Inspector Console

Requirements:
● 16G memory
● 1TB Disk
● 4-8 Cores
● AEO Hub installation
● Docker based, will use /var/ for space.
Based on Opensearch, Inspector requires higher resources than a
standard AEO hub server

curl -so https://updates.atomicorp.com/installers/awp-db

Atomic Inspector Lab

Lab 1: Discovery Search
Searches can be used to create a specific feed of data used in
visualizations and dashboards
Can be set to auto update every X seconds/minutes/hours
Best for exploring what data is available

Navigate to Discover
Ensure Index Pattern is: atomicorp-alerts-3.3*
Create a simple form of:
agent.name
rule.level
rule.id
rule.description

AEO Workshop: Elasticsearch / Kibana Dashboards
Power tip 1
use Search Field Names on the left column to rapidly expand the
name of the field to add
Power tip 2
Select the “>” on an individual event to see all the Fields available in
this type of alert

Click Save
Select Save
Enter in Title: ELK-01
Compare this Saved search to Simple-Search-01

Lab 2: Discovery Search with filters (only show events level 3 and up)
Open saved search ELK-01 from Lab 2
Select Add Filter
select Field, select rule.level
Set Operator to: is between
start: 3 and end to : 15
click save

Power Tip 1 :
If you donʼt know what the field names are for the filter, it will tell
you. Start typing in the field and it will expand the automatically
Power Tip 2:
The time frame is part of the Saved search, ie: if this is saved as a 24
hour search, when used elsewhere it will use this same timeframe

Click Save
Note the “Save as new search option”. This will be new
Select Save as new search (Note: best practice!)
Enter in Title: ELK-02
Compare this Saved search to Simple-Search-02
Note: Always select Save as new search, or you could overwrite an older
search used by other components (visuals, dashboards, etc)

Lab 3: Simple Agent count Pie chart
Select Visualize
Create visualization
Pie
source: (or your saved search)
Simple-Search-01
You should now have a single visual based on “count”

Aggregate “count” by agent
Select “add” under Buckets
Select Split Slices
Select “Terms” under Aggregation
Select “agent.name.keyword” under Field
Set Size to “5”
Count should now be broken out by agent

Add Labels to this chart
Select “Options”
Select “Show labels”
Click “Update”
Agent names should now be shown on each section as well as the
legend on the top right.

Click Save
Note, there wont be a Save as new visual option the first time
Enter in Title: ELK-Visual-03
Compare this visual to Simple-Visual-03

Lab 4: Simple Agent count Pie chart + Level data
This lab adds a pie chart inside a chart, showing data elements as they
apply to the top level data. In this case, our outer ring will be the agent,
the inner ring will be a visual representation of alerts by level per agent.
This is an advanced chart, involving 2 tiers of data, and color
manipulation

Select Add
Select Split slices
Sub-aggregation: Terms (Not: “Significant Terms”)
Field: rule.level
Size: 5
Click: Update
You should now have a new ring on the pie chart, but in the wrong place
(outside) rather than inside.

Changing data positioning, we change places with our two rings:
Note the “=” sign on each of your data buckets
drag the bottom data bucket “Split slices: rule.level: Descending”
above “Split slices agent.name.keyword: Descending”
Click Update
You should now have the agent name ring, with labels, on the outside of
the chart.

Power Tip 1:
Want to just see the data for a single agent? Double click the ring of
the agent name, it will automatically filter this data for
Power Tip 2:
Colors selection is available under the legend, select the data
element and change the color from there.

Click Save
Note, Save as new visual option is now available
Enter in Title: ELK-Visual-04
Compare this visual to Simple-Visual-04

Threat Intelligence
● Locally Stored
● Updated every 24h
● Agentless tracking
● Analysis

Threat Intelligence: Egress point analysis

Firewall / Edge device logging
● Capture outbound (destination IP) traﬀic from Firewalls
● Cross reference this against botnet IP destinations + Port
○ Fortigate UTM also includes URLʼs
○ Cisco ASAs contain basic information
● Advantages: Can cover many endpoints concurrently, cheap
● Disadvantages: Post connection processing, covering devices you
dont care about
Threat Intelligence

Firewall / Edge device logging
Enhancements
● Are all exit points covered?
● Assuming they are covered, are they being logged?
● What else can they log? URLs? Content?
● This can also be augmented with DNS log traﬀic (or disrupted by it)
Threat Intelligence

Threat Intelligence

APIʼs
● Sending
○ realtime (REST, MQ, Syslog)
○ actions
● Receiving
○ agents
○ agentless
● Polling
○ REST, MQ, custom

Events are stored in JSON format
Action streams JSON to destination (REST, MQ, Syslog, etc)
All data
Filtered by severity
Filtered by group
API: Sending data

Legacy Systems
● AIX
● EOL Linux (RHEL5, 32-bit)
● EOL Windows (XP, 2003, etc)
● HP-UX
● Solaris
● VAX

This is not a sales plug for AIX. Its going to sound like it
Atomic OSSEC on Legacy: AIX

At a glance, AIX is “legacy”. Under the hood?
● Continues to see operating system updates. So what?
● They are implementing it on new CPU hardware, like Power10. Still
so what?
● Its available in a cloud environment, and its actually easy to use.
What isnt?
● They are implementing features from linux. Ha ha, like what?
● Like clamav in firewall rules, and DNF for package management. ….
● and golang, and even some kubernetes utilities.

Challenges
● Tend to be critical systems, risk tolerance for change is low
● Almost always custom, complicating installation
● Almost always out of date or missing something
● Security instrumentation in the OS is outdated
● Common services rarely log useful information like source ips or
usernames.
● Does not support real-time FIM**
**AHAFS- may allow for this in the future, but does not appear robust enough at this time

The Good News:
● Yum/DNF support is excellent, if it is installed
● IBM maintains an excellent yum repository
● an Inotify API may be in the future
● IBM Partnerworld has an excellent SaaS for AIX testing and
development
● CIS has modern, maintained, compliance content for AIX
● Atomicorp maintains AIX packages and installers

DEMO

Malware Detection with
OSSEC and Atomic OSSEC

● Rootcheck: OSSEC & Atomic OSSEC
● CDB: OSSEC & Atomic OSSEC
● Clamav: Atomic OSSEC
● SSDeep: Atomic OSSEC
Malware Detection with OSSEC and Atomic OSSEC

Rootcheck
● strings based
● fast, easy to develop signatures
● Cant handle compressed malware
● Not real time
CDB
● Hash based
● Fast
● Polymorphic malware can evade
● Real-Time with FIM

ClamAV
● Multiple signature formats (hash, logical, yara, etc)
● IPS Mode (Block inline with fanotify)
● Client/Server or Standalone designs
SSDeep
● Distance Hash
● Fast, easy to write signatures
● Real-Time with FIM or WAF

RootCheck
● Runs on a timer as part of syscheckd
● Can scan files, directories, or look for processes
● Targets are declared in the definitions, ie /path/to/file

CDB
● Key-Value store of hashes
● Tied to FIM, so alerting is real-time and generic.
● Updates to CDBʼs do not require a restart, can be done inline
● Feeds for malicious malware available from CISA, and other sources
● Bonus round: CDBʼs can be used for other things, like threat intel
data

Clamav
● Runs as a complementary service to OSSEC
● Centrally managed from Atomic OSSEC
● Supports many malware data formats (hash, logical signatures,
yara, etc)
● Atomicorp Linux malware signatures
● Available on Linux, AIX, and windows
● In-line blocking (IPS) with Fanotify on Linux

DEMO

Malware Reverse Engineering
with Ghidra

https://github.com/NationalSecurityAgency/ghidra
● Open Source: Apache 2.0 license
● Desktop Application with “Friendly” UI
● Developed by the NSA Research Directorate
● Modern alternative to IDA Pro
● Excellent tool for investigating malware
Malware Reverse Engineering: Ghidra

A brief detour from Ghidra by request.
● Obfuscated PHP malware
● Delivered via an RFE vulnerability in wordpress
● Attacker deleted this immediately after running
● Artifacts captured by real-time FIM

A moment about my setup:
● Fedora 36/37 Virtual machine desktop
● Im connecting to it via Virt-Manager / Spice
● Private network
○ Iocal DNS server, everything is application based fw
○ SSL wildcard CA on the desktop
○ Transparent proxy with wildcard SSL on 80/443
○ Modified the KVM to not look like a KVM
○ Traﬀic, when enable is rate limited to 64k/s

DEMO

AEO Training - 2023.pdf

More Related Content

Similar to AEO Training - 2023.pdf

Recently uploaded

AEO Training - 2023.pdf