- Inodes (inode_t) represent files and directories in the filesystem. They are created and linked into an inode table when files/directories are accessed. Inodes transition between active, LRU, and purge lists over time. - File descriptors (fd_t) represent open files and directories. They are allocated from an fd table when files are opened and destroyed when files are closed. On network disconnects, open files for disconnected clients are cleaned up. - Tools are provided to debug reference count leaks for inodes and file descriptors.

1. Glusterfs Session #5
inode_t, fd_t lifecycles

2. Scope
- Life cycle of inode_t
- Life cycle of fd_t
- Debugging refcount leaks

3. inode-table
- All inodes created in the process are linked into inode-table
- inode_table_new (size_t lru_limit, xlator_t *xl)
- In general top xlators like fuse/server/NFS/gfapi would create the inode tables
to its child(ren).
- Inode table lives as long as the fs it manages is valid.
- Demo: git grep inode_table_new

4. Inode_t lifecycle(Active)
- When an operation happens on a file/directory, lookup operation is performed
before this operation.
- Lookup operation creates the inode_t object for that file/directory and links the
inode into inode-table and keeps it in active_list of inode-table
- Entry creation operation can also lead to the new inode created becoming
‘active’
- Demo: fuse-code-walk-through

5. Inode_t lifecycle(LRU)
- By default Brick has 16384 lru size and fuse has 65536(May vary between
versions, on fuse it used to be infinity, but now it has invalidation TBD)
- When refcount of inode is 0 but the nlookup count is non-zero
- Nlookup is maintained differently between fuse and server. With fuse
nlookup-count will be cleared(using forget call) by kernel-fuse. Where as other
xlators do the forget themselves.
- Demo: inode_unref() and inode_forget() in fuse and server

6. Inode_t lifecycle(PURGE)
- When an entry is deleted/LRU limit is reached, the inode will be put in purge
list to be pruned out of the inode-table. Please note that the inode will be
unhashed before putting it in this list to avoid race conditions.
- All inodes in this list will be destroyed
- Demo: inode_table_prune() -- forget cbk in each xlator

7. Inode_t lifecycle(Invalidations)
- Fuse used to have infinite LRU because fuse-kernel was doing destroy of
inodes using forget.
- But since glusterfs is a user process, if the number of inodes in LRU is huge
and kernel doesn’t call forgets quick enough, it could lead to OOM killers
- In the latest versions LRU is limited to 65536 and fuse-kernel will be notified
to do an invalidation on the inodes that are LRUed out of the list to keep a
check on the memory usage.

8. Fd-table
- Auto expanding fd table is present in top xlators.
- All opened files/directories will have fd entry from this table
- When there are disconnects/graph switches, the fds are closed/migrated
respectively from this table.
- Fdtable is active as long as the graph for that fs is active

9. fd-create(pre-open)
- Fd_t data structure is created with necessary pid, inode
- Fd_no is allocated from table
- Fd-flags are set
- Open fop is wound to the child xlator

10. fd-create(post-open)
- fd_bind() is called on a successful open so that the fd can be put in
inode->fd_list and is maintained as active-fd
- Demo fd_bind()

11. Flush fop(Application-close)
- Close on the fd by application leads to Flush fop in glusterfs. Any cleanup
related to the fd will happen in this phase
- In gluster this can lead to flushing writes in write-behind/locks will be cleared.
- File will be open until fd->refcount is non-zero inside gluster

12. fd-destroy(Post-application-close)
- When the refcount reaches zero, fd_destroy() is called which will call
release/releasdir based on if it is a file/directory respectively.
- The fd will be removed from inode->fd_list as well during this process.
- Fd_table’s entry is relinquished

13. Network disconnect leading to fd-cleanup in server
- When a connection is dropped between a client and server it will lead to
cleanup of resources from that client.
- Fd’s associated with the client will be cleaned up during this phase.
- All locks on the fd will also be cleaned up

14. Fd-reopen in client xlator
- When there is a disconnect and reconnect between a client xlator and brick,
files that were open will be re-opened
- Demo, server: server_connection_cleanup
- Demo, client: client_post_handshake

15. Debugging ref-leaks
- Demo

16. Q & A