Cloud Firewall (CFW) Logging also known as RFD 163 is a feature where we will start logging specific kinds of firewall records in a manner that doesn’t require as many per compute node resources. This logging will allow us to pay attention to inbound packets that drop. We want to record new TCP connections or connectionless UDP sessions in a manner that fits in nicely and are “aggregatable” into a proper Triton deployment. To activate this, a user has to opt into logging by marking a firewall rule with the "log" attribute.

1. Cloud Firewall Logging

2. Overview
● Cloud Firewall (CFW) Logging records the following events:
○ Dropped inbound packets.
○ New TCP connections or UDP sessions.
● It does so in a manner that’s aggregatable for a Triton
deployment.
● Specified in RFD 163, the changes needed were “full stack”
(ick, pardon the cliché).
○ illumos and fwadm(1M), CFW log daemon, *API support.
○ This talk will proceed from the bottom up.

3. Motivation
● Korean law requires logging on inbound connections.
● Logging per-guest presents some complications:
○ ipfilter’s existing ipmon requires one file descriptor (FD) per
VM if you wish to aggregate.
○ existing ipfilter already does most of the work we need with
its “keep state” option.
● We want to be able to add “log” to a Triton-administered
firewall rule.

4. Constraints
● As mentioned earlier, we really don’t want one FD per VM.
● Yet we need per-VM attributions of CFW Logging events.
● We need to make sure a post-CFW-Logging platform works
on older Triton deployments.
● Or have newer Triton deployments understand older PIs
won’t allow CFW-Logging.

5. illumos changes - /dev/ipfev
● All illumos changes are encapsulated as OS-7667.
● Creates a new global-zone-only device: /dev/ipfev
● /dev/ipfev is only openable by one process.
● Its being open is a check for performing event generation.
● Events are 88bytes. Listed below by purpose, not by layout:
○ Type, length, zone DID (unique)
○ IP packet information (src, dst, proto, ports)
○ ipfilter ruleid, CFW rule UUID
● Both Zone DID and rule UUID needed to be added to ipfilter’s
internal state.

6. illumos changes - ipfilter rules
● ipfilter in SmartOS has two types of filter rules:
○ Traditional root@zone per-netstack ones (incl. global)
○ Global-zone-administered ones for non-global zones.
■ These are the ones we wish to monitor.
■ We now cache the zone DID in these rulebases.
● New ipf(4) extensions (we chose rule tags) needed for:
○ Do we care if this ipfilter rule is CFW-logged?
○ If so, what is the CFW rule UUID?
● CFW-logged rules need to keep state (for new connections)
○ ... keep state set-tag(cfwlog,uuid=a7d57476-4699-4d2a-b4ae-7af857fea3d5)

7. illumos changes - /dev/ipfev in context
● So every event generated by any VM’s ipfilter gets put on a
global ring buffer.
● The single /dev/ipfev consumer reads off the ring buffer.
COMPUTE NODE
RING BUFFER
VM/Zone VM/Zone VM/ZoneVM/Zone VM/Zone VM/ZoneVM/ZoneVM/Zone
/dev/ipfev
cfwlogd

8. smartos-live changes
● All smartos-live changes encapsulated as OS-7668.
● New fwadm versioning mechanism. (As of this, Version 2.)
● fwadm(1M) rule json now has new boolean: log.
● Existing log and uuid pass through to new ipf(4) rules
mentioned earlier.

9. cfwlogd
● Needed a performant userland daemon capable of
connecting to “/dev/ipfev” and sucking down events out of the
ring buffer.
● The daemon needs to translate events from raw bytes
represented by cfwev_t in the kernel into new line separated
JSON formatted logs
● There’s been a recent interest in Rust from multiple
engineering groups within Joyent.

10. cfwlogd
● Rust has a great FFI interface to interface with OS level
concepts such as issuing ioctls.
● There’s a great serialization/deserialization crate (library) in
rust called serde.
○ Gives us the ability to translate from raw bytes directly to
JSON
● Rust offers safety and performance out of the box!

11. cfwlogd - gathering info
● We need to be able to get the customer uuid, the zone uuid,
and the zone alias when we only have the zonedid from the
kernel
○ We have something that does this already, it’s called
vminfod and it provides access to this information in real
time via an http stream!
● Match the information from vminfod with the event from the
device ring buffer and use serde to serialize the log out to
disk as newline separated JSON.

12. cfwlogd - data flow
vminfod
127.0.0.1:9090
/dev/ipfev
cfwlogd
global zone
kernel
Customer 1
Zone 1
Alias: node-app1
Customer 2
Zone 2
Alias: web1
Customer 2
Zone 1
Alias: openvpn1
Customer 1
Zone 2
Alias: ruby-app2
ixgb0 - External Network
● Runs as an agent in the global
zone.
● Drops as many privileges(5) as
it can at startup after opening
/dev/ipfev
● Talks to /dev/ipfev to suck
down events out of the
Kernel’s ring buffer
● Correlates the events with
information gathered from
vminfod in real time

13. cfwlogd - data flow continued
vminfod
Event Reader Thread
Fanout Thread
/dev/ipfev
Vminfod Watcher Thread Signal Listener Thread
Zone Logger Thread
Zonedid: 1
Customer: 1
Alias: Redis
Zone Logger Thread
Zonedid: 2
Customer: 2
Alias: OpenVPN
Zone Logger Thread
Zonedid: 3
Customer: 3
Alias: Nginx
CN Filesystem: /var/log/firewall/<customer>/<zone uuid>/current.log
Main Thread
SIGHUP
SIGTERM
SIG...
cfwlogd

14. cfwlogd - setup
● Install/Modify/Check cfwlogd
○ sdcadm post-setup firewall-logger-agent
○ sdcadm experimental update firewall-logger-
agent@<uuid> -C experimental
○ sdcadm health firewall-logger-agent
● If cfwlogd lands on a CN that doesn’t have “/dev/ipfev” it
gracefully exits telling SMF that the service was successful
but leaves no process running.
○ Also useful if a CN needs to get it’s PI rolled back.

15. cfwlogd - next steps
● Want to incorporate CMON metrics to alert operators that
cfwlogd’s internal queues are maxed out and we are dropping
events. (We try as hard as possible to remain running).
● Want more control over queue size per zone so that one bad
actor doesn’t hurt everyone. (Serializing data and writing to
disk is slower than event generation).

16. cfwlogd - next steps continued
● Would like to break out the vminfod client into a more generic
crate that other consumers could use.
● Add support for any future event types such as “Ruleset
change”.
● Explore a different model other than a thread per zone once
zone density increases per CN?

17. Triton changes
● Need to build "something" to take the log files generated by
cwflogd and put them into manta under a location the user
can have access to, maybe use RBAC …
● ... we already got Hermes to do that for Triton services from
the sdc zone.
● Should not add more stuff to the sdc zone. If possible, what
we should do is to remove from there.

18. Triton changes: logarchiver
● New service/zone pair logarchiver, including its own
Hermes instance, which is able to deploy its own agent
logarchiver-agent that will run in parallel with hermes-
agent.
● sdcadm post-setup logarchiver
● Same configuration than the sdc zone to talk to manta.
● Manta location for log files:
/:customer_login/reports/firewall-
logs/:year/:month/:day/:vm_uuid/:iso8601stamp
.log.gz

19. Triton changes: Hermes
● Added ability to connect customers’ UUIDs with manta
accounts using MAHI.
● Ability to deploy different agents FMRIs depending on the
zone running hermes-proxy.
● Ability to delete some logs based in the same logsets we use
to upload log files.

20. Triton changes: log (Boolean)
● fwrules have now a new attribute `log (Boolean)` which
defaults to false when not present.
● AdminUI: Checkbox for log or not fwrules.
● CloudAPI/node-triton: Log column added to rules list. `-l|--
log` option added to rule creation/update.
$ ./bin/triton -i fwrule ls
SHORTID ENABLED GLOBAL LOG RULE
78d77145 true - true FROM any TO vm 3a2b9998... BLOCK tcp PORT 666

21. Triton changes: firewaller-agent
● ipf*.conf files modifications are not backwards
compatible.
● requires a service that makes sure that each zone gets the
rules which are compatible with the running system:
firewaller-config-migration.
● vmadm and zones services depend on firewaller-
config-migration

22. Triton changes: firewaller-agent
● There’s a # smartos_ipf_version <version> line into
ipf*.conf files to determine which config is used.
● The configuration supported by a CN is at
/etc/ipf/smartos_version, which consist of a single
line containing a single integer.
● Rules tag fragments added/removed depending on versions
set-tag(uuid=rule.uuid,cfwlog).

23. 23© 2019 Joyent. All rights reserved. Joyent Confidential
Thank you for
watching!