This document summarizes the Linux audit system and proposes improvements. It begins with an overview of auditd and how audit messages are generated and processed in the kernel. Issues with auditd's performance, output format, and filtering are discussed. An alternative approach is proposed that uses libmnl for netlink handling, groups related audit messages into JSON objects, applies Lua-based filtering, and supports multiple output types like ZeroMQ and syslog. Benchmark results show this rewrite reduces CPU usage compared to auditd. The document advocates for continued abstraction and integration of additional data sources while avoiding feature creep.

1. INELUCTABLE MODALITY
OF LINUX AUDIT
A story of hate and delectation
Faster & Smarter IR

2. SUMUS
Mark Ellzey
@strcpy - @threatstack
Software for down there.

3. !! WARNING !!
If you are a current employee of RedHat (specifically on
the auditd codebase), and have an extensive
background in martial arts, please leave the room now.
I would rather not spend the night in a hospital.

4. BLATANT MARKETING SLIDE
The contents within are the result of making this product a reality.

5. • Provides information to a user which, in most
cases, only the kernel is privy to.
• A passive mechanism only; no alteration or
control over the data that is being emitted.
• Simplistic filtering using boolean logic over a linked
list of comparison operations.
• Can easily be extended by other kernel APIs,
including loadable modules.
The Linux Audit System
EXORDIUM

6. WRONG!
Wait for the din of shocked gasps to become quiescent, then proceed
AUDITD & AUDIT ARE MUTUALLY
EXCLUSIVE

7. AUDIT IS NOT MAGICAL
#define SYSCALL_DEFINEx(x, sname, ...)
SYSCALL_METADATA(sname, x, __VA_ARGS__);
!
#define SYSCALL_METADATA(sname, nb, ...)
SYSCALL_TRACE_ENTER_EVENT(sname);
SYSCALL_TRACE_EXIT_EVENT(sname);
!
!
auditsys:
movq %r10,%r9
movq %rdx,%r8
movq %rsi,%rcx
movq %rdi,%rdx
movq %rax,%rsi
call __audit_syscall_entry
sysret_audit:
call __audit_syscall_exit
code is generated for all syscall entry and exit points

8. audit_syscall_entry audit_syscall_exit
• Determines if the syscall
should be audited.
• Initializes underlying
audit_context structure from
the current task_struct.
• Emits several messages with
data associated with the
syscall over the netlink socket.
• The last message is always of
type “AUDIT_EOE”
AUDIT IS NOT MAGICAL
what is the meaning of this?

9. • return status
• execve
• sockaddrs
• fd pairs
• pid / auid / uid / sessionid
• current working directories
• path information
AUDIT IS NOT MAGICAL
some various data which is logged at exit

10. CAN’T STOP HERE,THIS
IS BAT COUNTRY

11. FEAR AND LOATHING IN KERNEL/AUDIT.C
there can be only one
Only one process can read from the audit netlink socket
this is a good thing - the kernel only has to maintain one buffer
Creating a second reader will hijack the first and will not be restored on exit
I get it, otherwise the kernel would be required to keep a backlog stack of processes

12. FEAR AND LOATHING IN KERNEL/AUDIT.C
debugging is impossible
Prior to linux 3.8, when the audit backlog was hit, and audit_log_start was
called during schedule_timeout : a deadlock would occur and fuck you
so if you’re in gdb and hit a breakpoint, it was a raging race to disable audit
I always forgot. The holes in the walls are a testament to that

13. FORMAT DISAPPROBATION :(
The kernel is to blame for this shameful log format!
audit_log_format(ab,"a0=%lx a1=%lx a2=%lx a3=%lx items=%d"
" ppid=%d pid=%d auid=%u uid=%u gid=%u"
" euid=%u suid=%u fsuid=%u"
" egid=%u sgid=%u fsgid=%u tty=%s ses=%u",
context->argv[0], context->argv[1], ...);
b u t w h a t i s t h e a l t e r n a t i v e ?
You’re insane. A JSON encoder in the kernel?
You’re insane.
An overly complex binary
message format?
How about this JSON thing I’ve
been hearing so much about?

14. • Everything that comes from the kernel
is a key value pair, treat it like so.
• Unquoted values are (usually) deemed
as “untrusted” strings, encoded as
ascii-hex.
• The “serial number” is the kernel’s way
of designating multiple messages into
a single group. It is up to the user-land
application to reassemble.
• User-land sourced messages are
always encapsulated in a key of “msg”
• If you were like me, stop bitching and
deal with it.
Just follow these simple rules
FORMAT APPROBATION :)

15. • Performance problems under load.
• Limited output format.
• Difficult to extend.
• Impossible to read.
• Poorly designed (opinion).
• Did I mention performance issues?
• I’ve seen better code in openssl.
THINE ENEMY LIES WITHIN LEGACY
AUDITING AUDITD

16. The mere presence of a comment
containing “Global” is a good sign
that the rest will be, in all probability,
terrible.
Is this what I think it is? Every single
message from the kernel is inserted
into a thread queue.
!
Also: “FIXME” in production code will
always induce cringe.
THINE ENEMY LIES WITHIN LEGACY

17. GOALS
• Lower resource utilization
under high load
• Extend (or create) logging
and filtering capabilities
• Keep some backwards
compatibility with auditd
• Don’t reinvent the wheel, if
the wheel isn’t broken
• Abstract EVERYTHING
• Follow all of the rules in the
next slide.
DEPRECATION

18. RULE ONE THROUGH ∞
An afterthought in auditd.

19. LIBEV FOR NETWORK AND SIGNAL IO
MANUAL NETLINK SOCKET HANDLING
LIBEVENT FOR NETWORK AND SIGNAL IO
LIBMNL FOR NETLINK SOCKET HANDLING
LOW LEVEL PROCESSING
THE AUDITD METHOD
OUR METHOD

20. !! WARNING !!
No statements about how libev is faster than libevent.
These comments are usually some variant of
regurgitated information based on the flawed
performance comparisons found on the libev website.
“It’s only cheating if you do it on purpose.”

21. PROCESSING : LIBMNL
creating the socket and registering with the kernel

22. PROCESSING : RAW NETLINK
creating the socket and registering with the kernel
thereisactuallymore

23. PROCESSING : LIBMNL
receiving a message from the netlink socket
libmnl does all the ugly work

24. PROCESSING : RAW NETLINK
receiving a message from the netlink socket
thereisactuallymore

25. PROCESSING : MESSAGES
post processing runtime grouping
The method used by auditd
requiring an external application to
parse and join multiple messages
using the “serial number” as a
grouping key.
dealing with the raw data
The method used by our
system which appends data
received from the kernel to a list
until the final AUDIT_EOE packet
has been processed.

26. PROCESSING : MESSAGE GROUPS
an abstract example; executing “tail -f tsaudit.log”
- serial=43480, type=SYSCALL, syscall=“sys_execve”, exe=“/usr/bin/tail”
- serial=43480, type=EXECVE, argc=2, a0=“tail”, a1=“tsaudit.log”
- serial=43480, type=CWD, cwd=“/var/log”
- serial=43480, type=PATH, name=“/usr/bin/tail”
- serial=43480, type=EOE
ungrouped
grouped
[
{
“type" : "SYSCALL",
“syscall" : "execve",
“exe" : "/usr/bin/tail"
},
{
“type" : "EXECVE",
“argc" : 2,
“argv” : [ “tail”, “tsaudit.log” ],
},
{
“type" : "CWD",
“cwd" : "/var/log"
},
{
"type": "PATH",
"name": "/usr/bin/tail"
}
]

27. PROCESSING : MESSAGE GROUPSA few more fun examples of grouping.
[
{
"exe": "/bin/cat",
"comm": "cat",
"ses": 10,
"auid": 4294967295,
"pid": 31335,
"ppid": 31334,
"items": 2,
"exit": 0,
"success": "yes",
"syscall": "execve",
"epoch": 1399248110,
"serial": 855516,
"type": "SYSCALL"
},
{
"a1": "eth0.dhclient",
"a0": "cat",
"argc": 2,
"epoch": 1399248110,
"type": "EXECVE"
},
{
"cwd": "/run/resolvconf/interface",
"epoch": 1399248110,
"type": "CWD"
},
{
"name": "/bin/cat",
"epoch": 1399248110,
"type": "PATH"
}
]
[
{
"res": "success",
"terminal": "ssh",
"addr": "192.168.56.1",
"hostname": "babby.local",
"exe": "/usr/sbin/sshd",
"acct": "mthomas",
"op": "PAM:session_open",
"ses": 24,
"auid": 1000,
"uid": 0,
"pid": 10469,
"epoch": 1393886985,
"serial": 3393,
"type": "USER_START"
}
]
[
{
"exe": "/usr/sbin/nginx",
"comm": "nginx",
"ses": 238,
"pid": 966,
"ppid": 965,
"items": 1,
"a3": "fffffffffffffffb",
"a2": 0,
"a1": "800",
"a0": "ee7c05",
"exit": 13,
"success": "yes",
"syscall": "open",
"epoch": 1392316421,
"serial": 301316,
"type": "SYSCALL"
},
{
"cwd": "/",
"type": "CWD"
},
{
"ogid": 0,
"name": "/www/index.html",
"type": "PATH"
}
]
[
{
"exe": "/usr/sbin/nginx",
"comm": "nginx",
"ses": 238,
"pid": 966,
"ppid": 965,
"items": 0,
"a3": "800",
"a2": "7fff8afba6cc",
"a1": "7fff8afba6d0",
"a0": 0,
"exit": 12,
"success": "yes",
"syscall": "accept4",
"epoch": 1392316421,
"serial": 301314,
"type": "SYSCALL"
},
{
"saddr": "192.168.56.1",
"port": 51997,
"prot": "ipv4",
"type": "SOCKADDR"
}
]
/var/resolvconf/interface$ cat eth0.dhclient nginx: int fd = open(“/www/index.html”); // fd == 13 fd = accept(“192.168.56.1:51997”); user “mthomas” started a pam session

28. PARSING
“Every year, one out of ten programmers will commit suicide due to
maintaining parsers written in C”
Every C developer who has had to maintain a parser in C

29. PARSING AUDIT MESSAGES
you be the judge
type=SYSCALL msg=audit(1386803107.182:7960575): arch=c000003e syscall=288 success=yes exit=26 a0=7
a1=7fff986ec590 a2=7fff986ec58c a3=800 items=0 ppid=952 pid=956 auid=4294967295 uid=33 gid=33 euid=33 suid=33
fsuid=33 egid=33 sgid=33 fsgid=33 ses=4294967295 tty=(none) comm="nginx" exe="/usr/sbin/nginx" key=(null)
BRUTE FORCE
~/Code/auditd$ egrep '(strstr|strchr|strtok|strcmp|strcasecmp|strdup|strcat|sprintf|snprintf)' auparse/*.c | wc -l
448
STATE DRIVEN
~/Code/tsaudit$ egrep '(strstr|strchr|strtok|strcmp|strcasecmp|strdup|strcat|sprintf|snprintf)' auparser/*.c | wc -l
4
AUDITD METHOD
OUR METHOD

30. PARSING : BRUTE FORCE
oneofmanyhorriblethingsyouwillencounter

31. PARSING : STATE DRIVEN
switch/casegenerateslookuptables-faster

32. PARSING AUDIT MESSAGES
taking things to the next level
TURBO BOOSTING CONDITIONAL LOGIC
- Generate a perfect hash table
using “gperf”.
- Assign “known” keys and values
to an enumerable type.
- Filter out keys and values which
we deemed unnecessary for
further processing
- Add validation and auto-parsers
for specific key values.
- Determine if the value of a key
can be treated as a different data
type, such as an integer or
boolean.

33. SO WE CAN DO STUFF LIKE THIS
with perfect hash tables - lookups are O(1)

34. AND THIS

35. FILTERING
optionally preprocess data before it is logged
-Load per-instance or per-output LUA script during
startup.
-Convert the grouped messages to a native LUA table.
-Call a pre-defined LUA function.
-A non-zero return will drop the message.
-A zero return will continue processing the message
-If a LUA table is returned, it is converted to JSON and
used as the output.

36. FILTERING
Example : simple boolean filter

37. FILTERING
Example : return a table which is converted to JSON on output
function find_set(k, set)
for _,v in pairs(set) do
if k == v then
return 1
end
end
return 0
end
!
!
function tsaudit_filter(data)
local ret = {}
local comms = { 'irqbalance', 'whoopsie', 'top', 'dhclient' }
!
for k,ent in pairs(data) do
if find_set(ent['comm'], comms) == 1 then
-- if any of the keys are found, return this table which will be
-- transformed into the JSON { "this" : "filtered", "dont" : "log" }
ret = {this="filtered", dont="log" }
end
!
if ent['success'] == 'no' then
-- if the syscall did not succeed, then return 1 which will not generate a
-- log.
ret = 1
end
end
!
return ret
end

38. LOGGING : AUDITD
1. File
2. There is no 2
OUTPUT TYPES
Don’t worry, auditd can be extended with “audispd” plugins!

39. AUDISPD : A MONUMENTAL HACK
For each plugin, audispd will fork,
execve, and send audit messages
to stdout.
A plugin just has to have the ability
to read from stdin.
if this design seems sane to you, keep it a secret, and start sharpening that programming knife
new term : “infinite noose”

40. LOGGING : OUR WAY
1. ZeroMQ
2. Syslog
3. Audit Emulator
4. AIO
5. Raw
6. Nanomsg
jsonpluggable chained inlined
OUTPUT TYPES
as of right now

41. ACCESSION
meticulous attention to abstraction enhances creativity
A fully functional auditd’esque
application can be written in under
50 lines of C.
Introduced many other autonomous
inputs which can be integrated
seamlessly.
• rtnetlink
• netlink connector
• netlink inet diag
• netlink task stats
• pcap data
• userland audit

42. consummation
I am your typical developer. “I can do better than that!”
Usually a dumb statement to make, so was it worth it?
Running apache bench (ab) at 10,000 requests per second
AUDITD 120% CPU
OUR REWRITE 10% CPU
I think that’s better.

43. • Additional methods for grouping related
data, further reducing overhead.
• Add simple analysis and statistical
gathering functionality.
• Continue abstractions to avoid the
potential bloat that comes with feature-
creep
CEREBRATION

44. COME WORK WITH
US
@threatstack