Cloud firewall logging

Overview
● Cloud Firewall (CFW) Logging records the following events:
○ Dropped inbound packets.
○ New TCP connections or UDP sessions.
● It does so in a manner that’s aggregatable for a Triton
deployment.
● Specified in RFD 163, the changes needed were “full stack”
(ick, pardon the cliché).
○ illumos and fwadm(1M), CFW log daemon, *API support.
○ This talk will proceed from the bottom up.

Motivation
● Korean law requires logging on inbound connections.
● Logging per-guest presents some complications:
○ ipfilter’s existing ipmon requires one file descriptor (FD) per
VM if you wish to aggregate.
○ existing ipfilter already does most of the work we need with
its “keep state” option.
● We want to be able to add “log” to a Triton-administered
firewall rule.

Constraints
● As mentioned earlier, we really don’t want one FD per VM.
● Yet we need per-VM attributions of CFW Logging events.
● We need to make sure a post-CFW-Logging platform works
on older Triton deployments.
● Or have newer Triton deployments understand older PIs
won’t allow CFW-Logging.

illumos changes - /dev/ipfev
● All illumos changes are encapsulated as OS-7667.
● Creates a new global-zone-only device: /dev/ipfev
● /dev/ipfev is only openable by one process.
● Its being open is a check for performing event generation.
● Events are 88bytes. Listed below by purpose, not by layout:
○ Type, length, zone DID (unique)
○ IP packet information (src, dst, proto, ports)
○ ipfilter ruleid, CFW rule UUID
● Both Zone DID and rule UUID needed to be added to ipfilter’s
internal state.

illumos changes - ipfilter rules
● ipfilter in SmartOS has two types of filter rules:
○ Traditional root@zone per-netstack ones (incl. global)
○ Global-zone-administered ones for non-global zones.
■ These are the ones we wish to monitor.
■ We now cache the zone DID in these rulebases.
● New ipf(4) extensions (we chose rule tags) needed for:
○ Do we care if this ipfilter rule is CFW-logged?
○ If so, what is the CFW rule UUID?
● CFW-logged rules need to keep state (for new connections)
○ ... keep state set-tag(cfwlog,uuid=a7d57476-4699-4d2a-b4ae-7af857fea3d5)

illumos changes - /dev/ipfev in context
● So every event generated by any VM’s ipfilter gets put on a
global ring buffer.
● The single /dev/ipfev consumer reads off the ring buffer.
COMPUTE NODE
RING BUFFER
VM/Zone VM/Zone VM/ZoneVM/Zone VM/Zone VM/ZoneVM/ZoneVM/Zone
/dev/ipfev
cfwlogd

smartos-live changes
● All smartos-live changes encapsulated as OS-7668.
● New fwadm versioning mechanism. (As of this, Version 2.)
● fwadm(1M) rule json now has new boolean: log.
● Existing log and uuid pass through to new ipf(4) rules
mentioned earlier.

cfwlogd
● Needed a performant userland daemon capable of
connecting to “/dev/ipfev” and sucking down events out of the
ring buffer.
● The daemon needs to translate events from raw bytes
represented by cfwev_t in the kernel into new line separated
JSON formatted logs
● There’s been a recent interest in Rust from multiple
engineering groups within Joyent.

cfwlogd
● Rust has a great FFI interface to interface with OS level
concepts such as issuing ioctls.
● There’s a great serialization/deserialization crate (library) in
rust called serde.
○ Gives us the ability to translate from raw bytes directly to
JSON
● Rust offers safety and performance out of the box!

cfwlogd - gathering info
● We need to be able to get the customer uuid, the zone uuid,
and the zone alias when we only have the zonedid from the
kernel
○ We have something that does this already, it’s called
vminfod and it provides access to this information in real
time via an http stream!
● Match the information from vminfod with the event from the
device ring buffer and use serde to serialize the log out to
disk as newline separated JSON.

cfwlogd - data flow
vminfod
127.0.0.1:9090
/dev/ipfev
cfwlogd
global zone
kernel
Customer 1
Zone 1
Alias: node-app1
Customer 2
Zone 2
Alias: web1
Customer 2
Zone 1
Alias: openvpn1
Customer 1
Zone 2
Alias: ruby-app2
ixgb0 - External Network
● Runs as an agent in the global
zone.
● Drops as many privileges(5) as
it can at startup after opening
/dev/ipfev
● Talks to /dev/ipfev to suck
down events out of the
Kernel’s ring buffer
● Correlates the events with
information gathered from
vminfod in real time

cfwlogd - data flow continued
vminfod
Event Reader Thread
Fanout Thread
/dev/ipfev
Vminfod Watcher Thread Signal Listener Thread
Zone Logger Thread
Zonedid: 1
Customer: 1
Alias: Redis
Zone Logger Thread
Zonedid: 2
Customer: 2
Alias: OpenVPN
Zone Logger Thread
Zonedid: 3
Customer: 3
Alias: Nginx
CN Filesystem: /var/log/firewall/<customer>/<zone uuid>/current.log
Main Thread
SIGHUP
SIGTERM
SIG...
cfwlogd

cfwlogd - setup
● Install/Modify/Check cfwlogd
○ sdcadm post-setup firewall-logger-agent
○ sdcadm experimental update firewall-logger-
agent@<uuid> -C experimental
○ sdcadm health firewall-logger-agent
● If cfwlogd lands on a CN that doesn’t have “/dev/ipfev” it
gracefully exits telling SMF that the service was successful
but leaves no process running.
○ Also useful if a CN needs to get it’s PI rolled back.

cfwlogd - next steps
● Want to incorporate CMON metrics to alert operators that
cfwlogd’s internal queues are maxed out and we are dropping
events. (We try as hard as possible to remain running).
● Want more control over queue size per zone so that one bad
actor doesn’t hurt everyone. (Serializing data and writing to
disk is slower than event generation).

cfwlogd - next steps continued
● Would like to break out the vminfod client into a more generic
crate that other consumers could use.
● Add support for any future event types such as “Ruleset
change”.
● Explore a different model other than a thread per zone once
zone density increases per CN?

Triton changes
● Need to build "something" to take the log files generated by
cwflogd and put them into manta under a location the user
can have access to, maybe use RBAC …
● ... we already got Hermes to do that for Triton services from
the sdc zone.
● Should not add more stuff to the sdc zone. If possible, what
we should do is to remove from there.

Triton changes: logarchiver
● New service/zone pair logarchiver, including its own
Hermes instance, which is able to deploy its own agent
logarchiver-agent that will run in parallel with hermes-
agent.
● sdcadm post-setup logarchiver
● Same configuration than the sdc zone to talk to manta.
● Manta location for log files:
/:customer_login/reports/firewall-
logs/:year/:month/:day/:vm_uuid/:iso8601stamp
.log.gz

Triton changes: Hermes
● Added ability to connect customers’ UUIDs with manta
accounts using MAHI.
● Ability to deploy different agents FMRIs depending on the
zone running hermes-proxy.
● Ability to delete some logs based in the same logsets we use
to upload log files.

Triton changes: log (Boolean)
● fwrules have now a new attribute `log (Boolean)` which
defaults to false when not present.
● AdminUI: Checkbox for log or not fwrules.
● CloudAPI/node-triton: Log column added to rules list. `-l|--
log` option added to rule creation/update.
$ ./bin/triton -i fwrule ls
SHORTID ENABLED GLOBAL LOG RULE
78d77145 true - true FROM any TO vm 3a2b9998... BLOCK tcp PORT 666

Triton changes: firewaller-agent
● ipf*.conf files modifications are not backwards
compatible.
● requires a service that makes sure that each zone gets the
rules which are compatible with the running system:
firewaller-config-migration.
● vmadm and zones services depend on firewaller-
config-migration

Triton changes: firewaller-agent
● There’s a # smartos_ipf_version <version> line into
ipf*.conf files to determine which config is used.
● The configuration supported by a CN is at
/etc/ipf/smartos_version, which consist of a single
line containing a single integer.
● Rules tag fragments added/removed depending on versions
set-tag(uuid=rule.uuid,cfwlog).

Cloud firewall logging

More Related Content

What's hot

Similar to Cloud firewall logging

Recently uploaded

Cloud firewall logging