Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.



Published on

Published in: Food, Technology
  • Be the first to comment


  1. 1. INELUCTABLE MODALITY OF LINUX AUDIT A story of hate and delectation Faster & Smarter IR
  2. 2. SUMUS Mark Ellzey @strcpy - @threatstack Software for down there.
  3. 3. !! WARNING !! If you are a current employee of RedHat (specifically on the auditd codebase), and have an extensive background in martial arts, please leave the room now. I would rather not spend the night in a hospital.
  4. 4. BLATANT MARKETING SLIDE The contents within are the result of making this product a reality.
  5. 5. • Provides information to a user which, in most cases, only the kernel is privy to. • A passive mechanism only; no alteration or control over the data that is being emitted. • Simplistic filtering using boolean logic over a linked list of comparison operations. • Can easily be extended by other kernel APIs, including loadable modules. The Linux Audit System EXORDIUM
  6. 6. WRONG! Wait for the din of shocked gasps to become quiescent, then proceed AUDITD & AUDIT ARE MUTUALLY EXCLUSIVE
  7. 7. AUDIT IS NOT MAGICAL #define SYSCALL_DEFINEx(x, sname, ...)      SYSCALL_METADATA(sname, x, __VA_ARGS__); ! #define SYSCALL_METADATA(sname, nb, ...)   SYSCALL_TRACE_ENTER_EVENT(sname);        SYSCALL_TRACE_EXIT_EVENT(sname); ! ! auditsys:   movq %r10,%r9   movq %rdx,%r8   movq %rsi,%rcx   movq %rdi,%rdx   movq %rax,%rsi   call __audit_syscall_entry   sysret_audit:   call __audit_syscall_exit code is generated for all syscall entry and exit points
  8. 8. audit_syscall_entry audit_syscall_exit • Determines if the syscall should be audited. • Initializes underlying audit_context structure from the current task_struct. • Emits several messages with data associated with the syscall over the netlink socket. • The last message is always of type “AUDIT_EOE” AUDIT IS NOT MAGICAL what is the meaning of this?
  9. 9. • return status • execve • sockaddrs • fd pairs • pid / auid / uid / sessionid • current working directories • path information AUDIT IS NOT MAGICAL some various data which is logged at exit
  11. 11. FEAR AND LOATHING IN KERNEL/AUDIT.C there can be only one Only one process can read from the audit netlink socket this is a good thing - the kernel only has to maintain one buffer Creating a second reader will hijack the first and will not be restored on exit I get it, otherwise the kernel would be required to keep a backlog stack of processes
  12. 12. FEAR AND LOATHING IN KERNEL/AUDIT.C debugging is impossible Prior to linux 3.8, when the audit backlog was hit, and audit_log_start was called during schedule_timeout : a deadlock would occur and fuck you so if you’re in gdb and hit a breakpoint, it was a raging race to disable audit I always forgot. The holes in the walls are a testament to that
  13. 13. FORMAT DISAPPROBATION :( The kernel is to blame for this shameful log format! audit_log_format(ab,"a0=%lx a1=%lx a2=%lx a3=%lx items=%d" " ppid=%d pid=%d auid=%u uid=%u gid=%u" " euid=%u suid=%u fsuid=%u"   " egid=%u sgid=%u fsgid=%u tty=%s ses=%u",    context->argv[0], context->argv[1], ...); b u t w h a t i s t h e a l t e r n a t i v e ? You’re insane. A JSON encoder in the kernel? You’re insane. An overly complex binary message format? How about this JSON thing I’ve been hearing so much about?
  14. 14. • Everything that comes from the kernel is a key value pair, treat it like so. • Unquoted values are (usually) deemed as “untrusted” strings, encoded as ascii-hex. • The “serial number” is the kernel’s way of designating multiple messages into a single group. It is up to the user-land application to reassemble. • User-land sourced messages are always encapsulated in a key of “msg” • If you were like me, stop bitching and deal with it. Just follow these simple rules FORMAT APPROBATION :)
  15. 15. • Performance problems under load. • Limited output format. • Difficult to extend. • Impossible to read. • Poorly designed (opinion). • Did I mention performance issues? • I’ve seen better code in openssl. THINE ENEMY LIES WITHIN LEGACY AUDITING AUDITD
  16. 16. The mere presence of a comment containing “Global” is a good sign that the rest will be, in all probability, terrible. Is this what I think it is? Every single message from the kernel is inserted into a thread queue. ! Also: “FIXME” in production code will always induce cringe. THINE ENEMY LIES WITHIN LEGACY
  17. 17. GOALS • Lower resource utilization under high load • Extend (or create) logging and filtering capabilities • Keep some backwards compatibility with auditd • Don’t reinvent the wheel, if the wheel isn’t broken • Abstract EVERYTHING • Follow all of the rules in the next slide. DEPRECATION
  18. 18. RULE ONE THROUGH ∞ An afterthought in auditd.
  20. 20. !! WARNING !! No statements about how libev is faster than libevent. These comments are usually some variant of regurgitated information based on the flawed performance comparisons found on the libev website. “It’s only cheating if you do it on purpose.”
  21. 21. PROCESSING : LIBMNL creating the socket and registering with the kernel
  22. 22. PROCESSING : RAW NETLINK creating the socket and registering with the kernel thereisactuallymore
  23. 23. PROCESSING : LIBMNL receiving a message from the netlink socket libmnl does all the ugly work
  24. 24. PROCESSING : RAW NETLINK receiving a message from the netlink socket thereisactuallymore
  25. 25. PROCESSING : MESSAGES post processing runtime grouping The method used by auditd requiring an external application to parse and join multiple messages using the “serial number” as a grouping key. dealing with the raw data The method used by our system which appends data received from the kernel to a list until the final AUDIT_EOE packet has been processed.
  26. 26. PROCESSING : MESSAGE GROUPS an abstract example; executing “tail -f tsaudit.log” - serial=43480, type=SYSCALL, syscall=“sys_execve”, exe=“/usr/bin/tail” - serial=43480, type=EXECVE, argc=2, a0=“tail”, a1=“tsaudit.log” - serial=43480, type=CWD, cwd=“/var/log” - serial=43480, type=PATH, name=“/usr/bin/tail” - serial=43480, type=EOE ungrouped grouped [ { “type" : "SYSCALL", “syscall" : "execve", “exe" : "/usr/bin/tail" }, { “type" : "EXECVE", “argc" : 2, “argv” : [ “tail”, “tsaudit.log” ], }, { “type" : "CWD", “cwd" : "/var/log" }, { "type": "PATH", "name": "/usr/bin/tail" } ]
  27. 27. PROCESSING : MESSAGE GROUPSA few more fun examples of grouping. [ { "exe": "/bin/cat", "comm": "cat", "ses": 10, "auid": 4294967295, "pid": 31335, "ppid": 31334, "items": 2, "exit": 0, "success": "yes", "syscall": "execve", "epoch": 1399248110, "serial": 855516, "type": "SYSCALL" }, { "a1": "eth0.dhclient", "a0": "cat", "argc": 2, "epoch": 1399248110, "type": "EXECVE" }, { "cwd": "/run/resolvconf/interface", "epoch": 1399248110, "type": "CWD" }, { "name": "/bin/cat", "epoch": 1399248110, "type": "PATH" } ] [ { "res": "success", "terminal": "ssh", "addr": "", "hostname": "babby.local", "exe": "/usr/sbin/sshd", "acct": "mthomas", "op": "PAM:session_open", "ses": 24, "auid": 1000, "uid": 0, "pid": 10469, "epoch": 1393886985, "serial": 3393, "type": "USER_START" } ] [ { "exe": "/usr/sbin/nginx", "comm": "nginx", "ses": 238, "pid": 966, "ppid": 965, "items": 1, "a3": "fffffffffffffffb", "a2": 0, "a1": "800", "a0": "ee7c05", "exit": 13, "success": "yes", "syscall": "open", "epoch": 1392316421, "serial": 301316, "type": "SYSCALL" }, { "cwd": "/", "type": "CWD" }, { "ogid": 0, "name": "/www/index.html", "type": "PATH" } ] [ { "exe": "/usr/sbin/nginx", "comm": "nginx", "ses": 238, "pid": 966, "ppid": 965, "items": 0, "a3": "800", "a2": "7fff8afba6cc", "a1": "7fff8afba6d0", "a0": 0, "exit": 12, "success": "yes", "syscall": "accept4", "epoch": 1392316421, "serial": 301314, "type": "SYSCALL" }, { "saddr": "", "port": 51997, "prot": "ipv4", "type": "SOCKADDR" } ] /var/resolvconf/interface$ cat eth0.dhclient nginx: int fd = open(“/www/index.html”); // fd == 13 fd = accept(“”); user “mthomas” started a pam session
  28. 28. PARSING “Every year, one out of ten programmers will commit suicide due to maintaining parsers written in C” Every C developer who has had to maintain a parser in C
  29. 29. PARSING AUDIT MESSAGES you be the judge type=SYSCALL msg=audit(1386803107.182:7960575): arch=c000003e syscall=288 success=yes exit=26 a0=7 a1=7fff986ec590 a2=7fff986ec58c a3=800 items=0 ppid=952 pid=956 auid=4294967295 uid=33 gid=33 euid=33 suid=33 fsuid=33 egid=33 sgid=33 fsgid=33 ses=4294967295 tty=(none) comm="nginx" exe="/usr/sbin/nginx" key=(null) BRUTE FORCE ~/Code/auditd$ egrep '(strstr|strchr|strtok|strcmp|strcasecmp|strdup|strcat|sprintf|snprintf)' auparse/*.c | wc -l 448 STATE DRIVEN ~/Code/tsaudit$ egrep '(strstr|strchr|strtok|strcmp|strcasecmp|strdup|strcat|sprintf|snprintf)' auparser/*.c | wc -l 4 AUDITD METHOD OUR METHOD
  30. 30. PARSING : BRUTE FORCE oneofmanyhorriblethingsyouwillencounter
  31. 31. PARSING : STATE DRIVEN switch/casegenerateslookuptables-faster
  32. 32. PARSING AUDIT MESSAGES taking things to the next level TURBO BOOSTING CONDITIONAL LOGIC - Generate a perfect hash table using “gperf”. - Assign “known” keys and values to an enumerable type. - Filter out keys and values which we deemed unnecessary for further processing - Add validation and auto-parsers for specific key values. - Determine if the value of a key can be treated as a different data type, such as an integer or boolean.
  33. 33. SO WE CAN DO STUFF LIKE THIS with perfect hash tables - lookups are O(1)
  34. 34. AND THIS
  35. 35. FILTERING optionally preprocess data before it is logged -Load per-instance or per-output LUA script during startup. -Convert the grouped messages to a native LUA table. -Call a pre-defined LUA function. -A non-zero return will drop the message. -A zero return will continue processing the message -If a LUA table is returned, it is converted to JSON and used as the output.
  36. 36. FILTERING Example : simple boolean filter
  37. 37. FILTERING Example : return a table which is converted to JSON on output function find_set(k, set) for _,v in pairs(set) do if k == v then return 1 end end return 0 end ! ! function tsaudit_filter(data) local ret = {} local comms = { 'irqbalance', 'whoopsie', 'top', 'dhclient' } ! for k,ent in pairs(data) do if find_set(ent['comm'], comms) == 1 then -- if any of the keys are found, return this table which will be -- transformed into the JSON { "this" : "filtered", "dont" : "log" } ret = {this="filtered", dont="log" } end ! if ent['success'] == 'no' then -- if the syscall did not succeed, then return 1 which will not generate a -- log. ret = 1 end end ! return ret end
  38. 38. LOGGING : AUDITD 1. File 2. There is no 2 OUTPUT TYPES Don’t worry, auditd can be extended with “audispd” plugins!
  39. 39. AUDISPD : A MONUMENTAL HACK For each plugin, audispd will fork, execve, and send audit messages to stdout. A plugin just has to have the ability to read from stdin. if this design seems sane to you, keep it a secret, and start sharpening that programming knife new term : “infinite noose”
  40. 40. LOGGING : OUR WAY 1. ZeroMQ 2. Syslog 3. Audit Emulator 4. AIO 5. Raw 6. Nanomsg jsonpluggable chained inlined OUTPUT TYPES as of right now
  41. 41. ACCESSION meticulous attention to abstraction enhances creativity A fully functional auditd’esque application can be written in under 50 lines of C. Introduced many other autonomous inputs which can be integrated seamlessly. • rtnetlink • netlink connector • netlink inet diag • netlink task stats • pcap data • userland audit
  42. 42. consummation I am your typical developer. “I can do better than that!” Usually a dumb statement to make, so was it worth it? Running apache bench (ab) at 10,000 requests per second AUDITD 120% CPU OUR REWRITE 10% CPU I think that’s better.
  43. 43. • Additional methods for grouping related data, further reducing overhead. • Add simple analysis and statistical gathering functionality. • Continue abstractions to avoid the potential bloat that comes with feature- creep CEREBRATION
  44. 44. COME WORK WITH US @threatstack