The document discusses file systems and the FUSE (Filesystem in Userspace) framework. It provides background on traditional kernel-based file systems and alternatives like synthetic files, 9P, and Hurd translators. FUSE allows implementing fully functional file systems in userspace by communicating with the kernel via a character device file. Examples are given like CephFS, Couchfuse, and etcd-fs to demonstrate how FUSE can be used as a starting point for new file systems.

1. Sabato 24 Ottobre 2015
#LDROMA15
http://lug.uniroma2.it/ld15/

2. All'ombra del
Leviatano

3. https://robertoreale.me/linux-day-2015

4. File e filesystem

5. Die Dataien hat der liebe Gott gemacht, alles andere ist
Menschenwerk.
Source: Leopold Kronecker (apocrifo)

6. file is the new byte

7. All files are created equal.
Source: Anonimo

8. Everything is a file.
Source: Anonimo

9. All file systems are not created equal.
Source: 11th USENIX Symposium on Operating Systems Design and Implementation
(OSDI 14)

10. Fondazioni

11. For most users, the filesystem is the most visible
aspect of an operating system.
Source: Silberschatz & Galvin, Operating System Concepts, 7th ed.

12. The filesystem consists of two distinct parts: a collection
of files, each storing related data, and a directory
structure, which organizes and provides information about
all the files in the system.
Source: Silberschatz & Galvin, Operating System Concepts, 7th ed.

13. The most important job of UNIX is to provide a filesystem.
Source: Ritchie & Thompson, The UNIX TimeSharing System

14. A file contains whatever information the user places on it, for
example symbolic or binary (object) programs.
No particular structuring is expected by the
system.
Source: Ritchie & Thompson, The UNIX TimeSharing System

15. A file does not exist within a particular directory; the directory
entry for a file consists merely of its name and a pointer to the
information actually describing the file.
Source: Ritchie & Thompson, The UNIX TimeSharing System

16. There is a threefold advantage in treating I/O devices this way:
file and device I/O are as similar as possible; file and device
names have the same syntax and meaning, so that a program
expecting a file name as a parameter can be passed a device
name; finally, special files are subject to the same protection
mechanism as regular files.
Source: Ritchie & Thompson, The UNIX TimeSharing System

17. Perhaps paradoxically, the success of UNIX is largely due to
the fact that it was not designed to meet any predefined
objectives.
Source: Ritchie & Thompson, The UNIX TimeSharing System

18. Precisazioni

19. The whole point with "everything is a file" is not that you have
some random filename, but the fact that you can use
common tools to operate on different things.
Source: Linux Torvalds, 8 giugno 2002

20. The UNIX philosophy is often quoted as "everything is a file",
but that really means everything is a stream of
bytes.
Source: Linux Torvalds, 8 marzo 2007

21. It should be just a "read()", and then people can use
general libraries and treat all sources the
same.
Source: Linux Torvalds, 8 marzo 2007

22. Il paradosso

23. everything is a file
ma
il perimetro di cosa è un file non è flessible ad libitum

24. il perimetro di cosa è un file è
fissato dal kernel

25. la sintassi e la semantica del filesystem sono fissate dal
kernel

26. Il Leviatano

27. VFS: oltre 65 mila righe di codice già nel 2008
Source: Galloway et al., Model-Checking the Linux Virtual File System

28. approccio conservativo
kernel-centrico

29. debug difficile

30. l'utente non amministratore semplicemente non può

31. ecce spes eius frustrabitur eum et videntibus cunctis
praecipitabitur
Source: Iob, 41, 1

32. VFS

33. Astrazione
File: modello comune
Strutture dati: superblock, inode, file, dentry
Operazioni
Object-oriented

34. Implementazione
Disk data structures
Memory data structures
Disk space management

35. Precursori
Earlier VFS implementations include Sun's VFS (in SunOS
version 2.0, circa 1985) and IBM and Microsoft's "Installable
File System" for IBM OS/2.
Source: M. Tim Jones, Anatomy of the Linux virtual filesystem switch

36. Altre strade

37. Synthetic Files

38. 9P: Plan 9 Filesystem Protocol

39. puffs: Pass-to-Userspace Framework File System su NetBSD

40. A filesystem is a protocol translator: it interprets incoming
requests and transforms them into a form suitable to store and
retrieve data.
Source: Antti Kantee, Send and Receive of File System Protocols

41. Hurd translators

42. A translator is simply a normal program acting as an object
server and participating in the Hurd's distributed virtual
filesystem.
Source: https://www.gnu.org/software/hurd/hurd/translator.html

43. It is so-called because it typically exports a filesystem
(although need not: cf. auth, proc and pfinet) and thus
translates object invocations into calls appropriate for the
backing store (e.g., ext2 filesystem, nfs server, etc.).
Source: https://www.gnu.org/software/hurd/hurd/translator.html

44. Another way of putting it is that it translates from one
representation of a data structure into another representation,
for example from the on-disk ext2 data layout to a traditional
filesystem hierarchy, or from a XML file to a virtual
hierarchical manifestation.
Source: https://www.gnu.org/software/hurd/hurd/translator.html

45. A translator is usually registered with a specific filesystem
node by using the settrans command.
Source: https://www.gnu.org/software/hurd/hurd/translator.html

46. Translators do not require any special privilege to run. The
privilege they require is simply that to access the indiviudal
resources they use.
Source: https://www.gnu.org/software/hurd/hurd/translator.html

47. FUSE
Filesystem in
Userspace

49. With FUSE it is possible to implement a fully functional
filesystem in a userspace program.
Source: http://fuse.sourceforge.net/

50. Autore Miklos Szeredi
Licenze GPL + LGPL

51. Features include...

52. simple library API

53. simple installation (no need to patch or recompile the kernel)

54. secure implementation

55. userspace-kernel interface is very efficient

56. usable by non privileged users
Source: http://fuse.sourceforge.net/

57. Interazione attraverso un file (ancora!): /dev/fuse.

58. FUSE is a userspace filesystem framework. It consists of a
kernel module (fuse.ko), a userspace library (libfuse.*) and a
mount utility (fusermount).
Source: http://fuse.sourceforge.net/doxygen/index.html

59. One of the most important features of FUSE is allowing
secure, non-privileged mounts. This opens up new possibilities
for the use of filesystems. A good example is sshfs: a secure
network filesystem using the sftp protocol.
Source: http://fuse.sourceforge.net/doxygen/index.html

60. Since the mount() system call is a privileged operation, a
helper program (fusermount) is needed, which is installed
setuid root.
Source: http://fuse.sourceforge.net/doxygen/index.html

61. Vocabolario

62. Userspace filesystem
A filesystem in which data and metadata are provided by an
ordinary userspace process. The filesystem can be accessed
normally through the kernel interface.
Source: http://fuse.sourceforge.net/doxygen/index.html

63. Filesystem daemon
The process(es) providing the data and metadata of the
filesystem.
Source: http://fuse.sourceforge.net/doxygen/index.html

64. Non-privileged mount (or user mount)
A userspace filesystem mounted by a non-privileged (non-
root) user. The filesystem daemon is running with the
privileges of the mounting user.
Source: http://fuse.sourceforge.net/doxygen/index.html

65. Filesystem connection
A connection between the filesystem daemon and the kernel.
The connection exists until either the daemon dies, or the
filesystem is umounted.
Source: http://fuse.sourceforge.net/doxygen/index.html

66. Mount owner
The user who does the mounting.
Source: http://fuse.sourceforge.net/doxygen/index.html

67. User
The user who is performing filesystem operations.
Source: http://fuse.sourceforge.net/doxygen/index.html

68. hello.c

69. /*
FUSE: Filesystem in Userspace
Copyright (C) 2001-2007 Miklos Szeredi <miklos@szeredi.hu>
This program can be distributed under the terms of the GNU GPL.
See the file COPYING.
*/
#define FUSE_USE_VERSION 30
#include <fuse.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <fcntl.h>
static const char *hello_str = "Hello World!n";
static const char *hello_path = "/hello";

70. static int hello_getattr(const char *path, struct stat *stbuf)
{
int res = 0;
memset(stbuf, 0, sizeof(struct stat));
if (strcmp(path, "/") == 0) {
stbuf->st_mode = S_IFDIR | 0755;
stbuf->st_nlink = 2;
} else if (strcmp(path, hello_path) == 0) {
stbuf->st_mode = S_IFREG | 0444;
stbuf->st_nlink = 1;
stbuf->st_size = strlen(hello_str);
} else
res = -ENOENT;
return res;
}

71. static int hello_readdir(const char *path, void *buf, fuse_fill_dir_t filler,
off_t offset, struct fuse_file_info *fi)
{
(void) offset;
(void) fi;
if (strcmp(path, "/") != 0)
return -ENOENT;
filler(buf, ".", NULL, 0);
filler(buf, "..", NULL, 0);
filler(buf, hello_path + 1, NULL, 0);
return 0;
}

72. static int hello_open(const char *path, struct fuse_file_info *fi)
{
if (strcmp(path, hello_path) != 0)
return -ENOENT;
if ((fi->flags & 3) != O_RDONLY)
return -EACCES;
return 0;
}

73. static int hello_read(const char *path, char *buf, size_t size, off_t offset,
struct fuse_file_info *fi)
{
size_t len;
(void) fi;
if(strcmp(path, hello_path) != 0)
return -ENOENT;
len = strlen(hello_str);
if (offset < len) {
if (offset + size > len)
size = len - offset;
memcpy(buf, hello_str + offset, size);
} else
size = 0;
return size;
}

74. static struct fuse_operations hello_oper = {
.getattr = hello_getattr,
.readdir = hello_readdir,
.open = hello_open,
.read = hello_read,
};

75. int main(int argc, char *argv[])
{
return fuse_main(argc, argv, &hello_oper, NULL);
}

76. Bestiario

77. CephFS
FUSE come vivaio, come coltura di giovani filesystem.
CephFS è nel kernel dalla versione 2.6.34.

78. Couchfuse
Couchfuse is a FUSE filesystem that exposes Couchdb
databases as filesystem folder.
Source: http://narkisr.github.io/couch-fuse/

79. elfs
A simple (FUSE) filesystem on top of ELF objects.
Autore: Guillaume Leconte
Source: https://github.com/pozdnychev/elfs

80. $ elfs `which fdup` /tmp/elf
$ ls -l /tmp/elf/
total 0
drw-r--r-- 1 root root 0 Jan 1 1970 header
drw-r--r-- 1 root root 0 Jan 1 1970 libs
drw-r--r-- 1 root root 0 Jan 1 1970 sections

81. estensione ad altri formati binari
astrazione dal formato
interfaccia verso exec()

82. etcd-fs
A replicated filesystem on top of etcd.
Autore: Jonathan Leibiusky
Source: https://github.com/xetorthio/etcd-fs

83. fusepy
Simple ctypes bindings for FUSE.
Autore: Terence Honles
Source: https://github.com/fusepy/fusepy

84. GlusterFS
GlusterFS is a scalable network filesystem. Using common
off-the-shelf hardware, you can create large, distributed
storage solutions for media streaming, data analysis, and other
data- and bandwidth-intensive tasks.
Source: http://www.gluster.org/

85. PNGdrive
PNG meets Steganography meets Fuse: the easiest way to have
plausible deniability.
Source: https://code.google.com/p/pngdrive/

86. WikipediaFS
WikipediaFS is a virtual filesystem which allows users to view
and edit Wikipedia articles as if they were real files on a local
disk drive.
Source: https://en.wikipedia.org/wiki/WikipediaFS

87. Colofóne
Presentazione composta con vim e Hovercraft! su Ubuntu Saucy.
Featuring Google Fonts: Libre Baskerville, Racing Sans One,
Satisfy.

88. exit()

89. Roberto Reale
https://robertoreale.me/linux-day-2015