Have you ever heard of FreeBSD? Probably. Have you ever interacted with its kernel? Probably not. In this talk, Gili Yankovitch (nyxsecuritysolutions.com) will talk about the FreeBSD operating system, its network stack and how to write network drivers for it. The talk will cover the following topics: * Kernel/User interation in FreeBSD * The FreeBSD Network Stack * Network Buffers API * L2 and L3 Hooking

1. FreeBSD and Drivers
Gili Yankovitch, Nyx Software Security Solutions

2. Key Points
● What is FreeBSD?
● FreeBSD Drivers: How to create, compile and run a driver.
● Char devices
● Network Hooking
○ L3
○ L2
● Interaction with the network stack

3. What is FreeBSD?
● “FreeBSD is a free UNIX-like operating system descended from Research
Unix via the Berkeley Software Distribution (BSD).” - Wikipedia
● The BSD Project was founded in 1976 by Bill Joy.
● Contained code written by AT&T (Who later sued people related to BSD)
● In 1993 the first FreeBSD distribution was released.
○ Two years AFTER the Linux Kernel was founded.

4. What is FreeBSD?
● https://www.freebsd.org/
● Unlike Linux, it comes with a lot of user mode tools
● Doesn’t come in many flavours (distributions)
● Supported architectures: amd64, i386, ia64, powerpc, powerpc64, sparc64,
mips, armv6, aarch64.
● Unfortunately, lacks a lot of features implemented in Linux.
○ Namespaces, Good L2 hooking (yay -_-) and more…
● It is very unfortunate but there is very little documentation on
FreeBSD on the internet. :(
○ This means that if you are stuck, you need to deal with it on your own.
■ True story.

5. How to get FreeBSD
● Download
○ https://www.freebsd.org/where.html
● Git:
○ https://github.com/freebsd
■ Yes they use GitHub.
● On a FreeBSD system, the sources are usually at:
○ /usr/src/sys/

6. Folder Structure
● In contrast to Linux, FreeBSD has a lot of folders in its root directory.
○ kern/ - Core kernel implementation.
○ libkern/ - Core kernel libraries (printf, uprintf, strcpy etc…).
○ fs/ - File systems implementation.
○ net/ netinet/ - Core net and Inet implementation.
○ sys/ - Include directory. Contains a lot of *.h files.
○ amd64/ arm/ mips/ … - Architecture specific sources.
○ modules/ dev/ - Drivers.
○ ...

7. First And Foremost - Prints!
● There are two types of prints from the kernel:
○ printf
○ uprintf
● Both appear in dmesg
● uprintf prints to your current console
● printf prints to tty0

8. Char Devices

9. Compiling Our First Driver
static int nethook_loader (struct module *m, int what, void *arg)
{
int err = 0;
switch (what)
{
case MOD_LOAD : /* kldload */
uprintf ("Nethook KLD loaded. n");
break;
case MOD_UNLOAD :
uprintf ("Nethook KLD unloaded. n");
break;
}
return err;
}
static moduledata_t nethook_mod =
{
"nethook",
nethook_loader ,
NULL
};
DECLARE_MODULE (nethook, nethook_mod , SI_SUB_KLD , SI_ORDER_ANY );
nethook.c
Don’t return anything different than 0!
Different value will prevent you from unloading the module!!

10. Compiling Our First Driver
● Yep. That simple.
SRCS=nethook.c
KMOD=nethook
.include <bsd.kmod.mk>
Makefile

11. Running the Driver
● Just like Linux, we need to inject it to the Kernel:
$ kldload ./nethook.ko
$ kldunload ./nethook.ko
$ kldstat
● Removing from Kernel:
● Modules list:

12. Creating A Char Device
● This actually has a very good tutorial:
○ https://www.freebsd.org/doc/en/books/arch-handbook/driverbasics-char.html
● But here’s the snippets anyhow:
○ Create a struct with function pointers to read, write, open, close.
■ ioctl seems to fail with this method...
/* Character device entry points */
static struct cdevsw echo_cdevsw = {
.d_version = D_VERSION ,
.d_open = echo_open ,
.d_close = echo_close ,
.d_read = echo_read ,
.d_write = echo_write ,
.d_name = "echo",
};
● Just like Linux...

13. Creating A Char Device
● Then all you need to do is register:
static struct cdev *echo_dev ;
static int nethook_loader (struct module *m, int what, void *arg)
{
...
error = make_dev_p (MAKEDEV_CHECKNAME | MAKEDEV_WAITOK ,
&echo_dev ,
&echo_cdevsw ,
0,
UID_ROOT ,
GID_WHEEL ,
0600,
"echo");
…
}
Char Device Kernel Object
Pointer to File Operations
Owner UID of File System Node
Owner GID of File System Node
File System Node Permissions
Name of File System Node

14. Creating A Char Device
● Read operation
● Note the struct uio:
○ uio_resid - Space left in buffer sent from user mode (read length usually)
○ uio_offset - Current write offset
● uio knows whether it’s a read or a write depending on current action.
○ Here uiomove() writes from kernel buffer to user mode buffer.
static int echo_read (struct cdev *dev __unused , struct uio *uio, int ioflag __unused )
{
size_t amt;
int error;
amt = MIN(uio->uio_resid , uio->uio_offset >= echomsg ->len + 1 ? 0 :
echomsg ->len + 1 - uio->uio_offset );
if ((error = uiomove (echomsg->msg, amt, uio)) != 0)
uprintf("uiomove failed! n");
return (error);
}

15. Creating A Char Device
● Corresponding write().
static int echo_write (struct cdev *dev __unused , struct uio *uio, int ioflag __unused )
{
size_t amt;
int error;
if (uio->uio_offset != 0 && (uio->uio_offset != echomsg ->len))
return (EINVAL);
/* Copy the string in from user memory to kernel memory */
amt = MIN(uio->uio_resid , (BUFFERSIZE - echomsg ->len));
error = uiomove (echomsg->msg + uio->uio_offset , amt, uio);
if (error != 0)
uprintf("Write failed: bad address! n");
return (error);
}

16. Networking
● DISCLAIMER:
○ Before you begin to build your own network driver, be absolutely sure you understand the
below.
● OK lets continue...

17. Network Stack

18. Networking
● Just like Linux has its skb structure, FreeBSD has a basic buffer system
● It’s called: mbuf
● mbufs are buffer chains of size 256
○ Larger buffers are possible in an mbuf cluster but unfortunately usually it’s not the case.
● When you get a packet larger than 256 bytes, you get an mbuf chain
● Mellanox created a module called OFED to help port drivers from Linux to
Freebsd.
○ It’s a great place to start learning about networking in FreeBSD.
○ Unfortunately it lacks a HELL LOT of functionality sometimes needed.

19. Meet struct mbuf
● Yeah I know it’s weird and complicated.
○ Our interest is in m_hdr and in m_dat.M_databuf (Which means a normal packet)
struct mbuf {
struct m_hdr m_hdr ;
union {
struct {
struct pkthdr MH_pkthdr ; /* M_PKTHDR set */
union {
struct m_ext MH_ext ; /* M_EXT set */
char MH_databuf [MHLEN];
} MH_dat;
} MH;
char M_databuf [MLEN]; /* !M_PKTHDR, !M_EXT */
} M_dat;
};
/sys/mbuf.h

20. Meet struct m_hdr
● mh_next - Already mentioned this is an mbuf chain
● mh_nextpkt - mbufs provide us with a linked-list of packets storage place.
● mh_data - Pointer to beginning of data within the data buffer
● mh_len - Length of data in this mbuf
struct m_hdr {
struct mbuf *mh_next; /* next buffer in chain */
struct mbuf *mh_nextpkt ; /* next chain in queue/record */
caddr_t mh_data ; /* location of data */
int32_t mh_len ; /* amount of data in this mbuf */
uint32_t mh_type :8, /* type of data in this mbuf */
mh_flags :24; /* flags; see below */
#if !defined(__LP64__ )
uint32_t mh_pad ; /* pad for 64bit alignment */
#endif
};
/sys/mbuf.h

21. nbuf Structure
mbuf
mh_data
mbuf aaaaa
mh_next
mh_len
mbuf aaaaa
mbuf aaaaa
mh_nextpkt

22. Whatever you do, do NOT access these directly!
● Seriously. For everything you need there’s a function.
● When in doubt, see man mbuf (9).
● mbuf function names are non indicative, so I’ll explain a few here:

23. Allocating and freeing buffers
● m_get(int how, int type) - Allocates a new mbuf and sets its type.
● m_free(struct mbuf *m) - Frees a single mbuf.
● m_freem(struct mbuf *m) - Frees an entire mbuf chain.
● m_dup(struct mbuf *m, int how) - Duplicates an entire mbuf.
● m_copym(struct mbuf *mbuf, int offset, int len, int how) - Copy only a portion
of the mbuf to a new mbuf chain.
● m_copydata(const struct mbuf *mbuf, int offset, int len, caddr_t buf) - Copy
the mbuf data to a different buffer.
● m_length(struct mbuf *m, struct mbuf ** last) - Returns the entire mbuf chain
length (in bytes).

24. Shorten or Lengthen the Buffer
● m_adj(struct mbuf *m, int len) - Shorten the buffer from the beginning.
mbuf aaaaa mbuf
mh_data
mh_next
mbuf
mh_len mh_lenmh_len
mbuf
mh_data
void shorten_my_mbuf (struct mbuf *m)
{
m_adj (m);
}
m
is
stillpointing
to
the
firstm
buf!!

25. Shorten or Lengthen the Buffer
● m_prepend(struct mbuf *m, int len, int how) - Prepend len bytes in te
beginning.
mbuf aaaaa mbuf
mh_data
mh_next
mh_len mh_len
mbuf aaaaa
mh_next
mh_data mh_data
mh_len mh_len
mbuf

26. Accessing data
● Because mbufs are divided to 256-bytes parts, header might fall between two
mbufs.
● Accessing the header linearly might cause an unexpected behaviour.
mbuf aaaaa mbufhea der
Write.. OVERFLOW...
● NEVER access directly, or before using this:
● m_pulldown(struct mbuf *mbuf, int offset, int len, int *offsetp)
mbuf aaaaa mbufhea dermbuf a mbufheader
Might allocate a new mbuf

27. Interfaces
● Interfaces in FreeBSD are represented by struct ifnet
struct ifnet {
struct vnet *if_vnet; /* pointer to network stack instance */
TAILQ_ENTRY (ifnet) if_link ; /* all struct ifnets are chained */
...
char if_xname [IFNAMSIZ ]; /* external name (name + unit) */
...
struct ifaddrhead if_addrhead ; /* linked list of addresses per if */
...
u_short if_index ; /* numeric abbreviation for this if */
int (*if_output ) /* output routine (enqueue) */
(struct ifnet *, struct mbuf *, const struct sockaddr *,
struct route *);
void (*if_input ) /* input routine (from h/w driver) */
(struct ifnet *, struct mbuf *);
…
void (*if_transmit ) /* initiate output routine */
(struct ifnet *, struct mbuf *);
u_int if_fib ; /* interface FIB */
...
};
/net/if_var.h

28. L3 Hooking
● Just like Linux has netfilter, FreeBSD has a framework called pfil
● It enables to create a list of filters for both IN and OUT packets.
● Unlike Linux, pfil allows hooking in only one place for incoming and outgoing
packets.

29. L3 Hooking
● Hooking is easy. Use:
struct pfil_head *pfh_inet ;
/* Initializing L3 Hooking */
if (!(pfh_inet = pfil_head_get (PFIL_TYPE_AF , AF_INET )))
{
uprintf ("Failed getting packet filter head n");
return ESRCH;
}
pfil_add_hook(in_filter, NULL, PFIL_IN | PFIL_WAITOK, pfh_inet);
static int in_filter(void *arg, struct mbuf **m, struct ifnet *ifp, int dir, struct inpcb *inp)
● Then, register the callback:
● Hook signature:

30. The L2-L3 Input Stack Driver
ifp->if_input()
ether_input_internal
BPF
LAGG
ng_ether
Bridging
Hooks
ether_demuxVLAN Handlingvlan_input_p()
IP: ip_input() IPv6 ARP: arpintr() ATALK AARP
pfil_run_hooks()
PFil

31. The L2-L3 Input Stack
Driver
ip_output
pfil_run_hooks()
if_output()
ether_output
Bridge
ng_ether
PFil
Hooks
ifp->if_transmit()

32. L2 Hooking
● Apparently, it’s not as trivial hooking to the network stack in L2
● For example, in order to make Libpcap work, NIC drivers need to explicitly call
Libpcap kernel hooks to redirect L2 flow to it.
● Suggested implementation in user mode:
○ BPF - Explained in previous lectures
○ Libpcap - Explained above
○ Nethook - Memory-mapping based network handling. Exists in both Linux, Windows and
FreeBSD.
● Despite what is said above, you can use netgraph to attach to ng_ether.
○ There is a way to use it more easily. Source code will be uploaded later.

33. ● DDB is the static kernel debugger. You can read about it here:
○ https://www.freebsd.org/cgi/man.cgi?ddb(4)
● Compile kernel with:
○ Options DDB
● Compiling the kernel:
○ Configs are in:
■ amd64/conf/GENERIC
■ Always copy GENERIC to a new file and edit it.
Other useful tips
$ cd /usr/src/
$ make buildkernel KERNCONF=GENERIC.MYCONF && make installkernel KERNCONF=GENERIC.MYCONF && shutdown -r now
● If kernel hangs, useful VirtualBox command (Opens DDB):
$ VBoxManage debugvm <VM Name> injectnmi

34. Questions? :)