Goes into details about the prior knowledge required to design a file system. Concludes with a demo that explains how these concepts are used by Second Extended File System (ext2fs) A demo is also provided.
Presented at Florence, Italy on 6th December 2017
Virtual File System in Linux Kernel
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
Goes into details about the prior knowledge required to design a file system. Concludes with a demo that explains how these concepts are used by Second Extended File System (ext2fs) A demo is also provided.
Presented at Florence, Italy on 6th December 2017
Virtual File System in Linux Kernel
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
Page cache mechanism in Linux kernel.
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
FUSE and beyond: bridging filesystems slides by Emmanuel Dreyfuseurobsdcon
Abstract
Everything started with the desire to have a distributed filesystem better than NFS, which would be elastic and replicated with no SPOF. glusterFS was among the filesystems that could do that, but it uses The FUSE interface to the kernel, which was not available on NetBSD.
We will therefore discuss FUSE implementation and its NetBSD counterpart which is called PUFFS. A first but isufficient bridging attmpt called reFUSE was done, but porting real filesystems required the more complete PERFUSE bridge, which we will present here.
In a third part, we will look at the various unexpected roadblocks that had to be handled during PERFUSE implementation, in order to get a fully functionnal glusterFS on NetBSD.
Speaker bio
Emmanuel Dreyfus works as an IT manager at ESPCI ParisTech in France. He has been contributing for 13 years to NetBSD, in areas such as binary compatibility, and FUSE implementation. More at http://hcpnet.free.fr/pubz/
A presentation on the Ext4 file system and the evolution of Ext filesystem in Linux operating system. Linux uses virtual filesystem. The comparison of the ext filesystem generations is provided.
One of the tutorials I gave at University of Wollongong on inodes.
Describes direct, single, double and triple linking, and how that all ties together with addressable space.
Page cache mechanism in Linux kernel.
Note: When you view the the slide deck via web browser, the screenshots may be blurred. You can download and view them offline (Screenshots are clear).
FUSE and beyond: bridging filesystems slides by Emmanuel Dreyfuseurobsdcon
Abstract
Everything started with the desire to have a distributed filesystem better than NFS, which would be elastic and replicated with no SPOF. glusterFS was among the filesystems that could do that, but it uses The FUSE interface to the kernel, which was not available on NetBSD.
We will therefore discuss FUSE implementation and its NetBSD counterpart which is called PUFFS. A first but isufficient bridging attmpt called reFUSE was done, but porting real filesystems required the more complete PERFUSE bridge, which we will present here.
In a third part, we will look at the various unexpected roadblocks that had to be handled during PERFUSE implementation, in order to get a fully functionnal glusterFS on NetBSD.
Speaker bio
Emmanuel Dreyfus works as an IT manager at ESPCI ParisTech in France. He has been contributing for 13 years to NetBSD, in areas such as binary compatibility, and FUSE implementation. More at http://hcpnet.free.fr/pubz/
A presentation on the Ext4 file system and the evolution of Ext filesystem in Linux operating system. Linux uses virtual filesystem. The comparison of the ext filesystem generations is provided.
One of the tutorials I gave at University of Wollongong on inodes.
Describes direct, single, double and triple linking, and how that all ties together with addressable space.
The Linux Kernel Implementation of Pipes and FIFOsDivye Kapoor
A walkthrough of the code structure used in the linux kernel to implement pipes and FIFOs.
This was presented to a Senior level class at the Indian Institute of Technology, Roorkee.
Part 03 File System Implementation in LinuxTushar B Kute
Presentation on "Virtual File System Implementation in Linux".
Presented at Army Institute of Technology, Pune for FDP on "Basics of Linux Kernel Programming". by Tushar B Kute (http://tusharkute.com).
This is a short introduction to some basic concepts about device drivers on Linux, and does not get into the nuts and bolts of it; uses a serial driver as an example, and how does it fit within the TTY layer. Includes code walk-through.
Writing Character driver (loadable module) in linuxRajKumar Rampelli
It covers the step by step approach on how to write a simple loadable character device driver in linux. What are Device files in linux detail. How user application interact with character driver using a device file.
It is the File system that is contained on the same partition on which the "Root directory" is located. It is the File system on which all the other file systems are mounted
A File Structure should be according to a required format that the operating system can understand.
A file has a certain defined structure according to its type.
A text file is a sequence of characters organized into lines.
A source file is a sequence of procedures and functions.
An object file is a sequence of bytes organized into blocks that are understandable by the machine.
File Type
File type refers to the ability of the operating system to distinguish different types of file such as text files source files and binary files etc. Many operating systems support many types of files. Operating system like MS-DOS and UNIX have the following types of files −
Ordinary files
These are the files that contain user information.
These may have text, databases or executable program.
The user can apply various operations on such files like add, modify, delete or even remove the entire file.
Directory files
These files contain list of file names and other information related to these files.
Special files
These files are also known as device files.
These files represent physical device like disks, terminals, printers, networks, tape drive etc.
Have you ever heard of FreeBSD? Probably.
Have you ever interacted with its kernel? Probably not.
In this talk, Gili Yankovitch (nyxsecuritysolutions.com) will talk about the FreeBSD operating system, its network stack and how to write network drivers for it.
The talk will cover the following topics:
* Kernel/User interation in FreeBSD
* The FreeBSD Network Stack
* Network Buffers API
* L2 and L3 Hooking
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Le nuove frontiere dell'AI nell'RPA con UiPath Autopilot™UiPathCommunity
In questo evento online gratuito, organizzato dalla Community Italiana di UiPath, potrai esplorare le nuove funzionalità di Autopilot, il tool che integra l'Intelligenza Artificiale nei processi di sviluppo e utilizzo delle Automazioni.
📕 Vedremo insieme alcuni esempi dell'utilizzo di Autopilot in diversi tool della Suite UiPath:
Autopilot per Studio Web
Autopilot per Studio
Autopilot per Apps
Clipboard AI
GenAI applicata alla Document Understanding
👨🏫👨💻 Speakers:
Stefano Negro, UiPath MVPx3, RPA Tech Lead @ BSP Consultant
Flavio Martinelli, UiPath MVP 2023, Technical Account Manager @UiPath
Andrei Tasca, RPA Solutions Team Lead @NTT Data
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Quantum Computing: Current Landscape and the Future Role of APIs
Linux
1. LINUX FILE SYSTEM
V.N.Murali
CSE-597A Presentation
03/29/2001,
ACKNOWLEDGEMENT:
a)LINUX KERNEL INTERNALS
BECK,BOHM et al
b)THE LINUX KERNEL
DAVID GOSLING.
2. Virtual File System Switch
Supplies applications with the system calls for file system
management.
Specific focus on ext2fs and procfs of LINUX.
Different types of files are possible:regular
files,directories,device files,fifo’s,pipes,symbolic
links,sockets.
Meta data information kept separately.”inodes” are used
for describing files.
6. FEATURES OF UNIX FILE SYSTEMS
Boot Block : Will it always be present?
SuperBlock:Information for managing file systems
InodeBlocks:inode structures for the file system.
DataBlocks:blocks for files,directories..etc
Mounting,Unmounting file systems
7. Representation of File systems
Management structures for file systems is similar to logical structure of
UNIX file system.
VFS responsible for calling file-system specific implementations. How
does VFS know which filesystems are configured?
Void register_filesystem(struct file_system_type *);
Configured at boot-time and possibly loaded as modules as well.
struct file_system_type {
– Const char *name;
– Int fs_flags;
– Struct super_block *(*read_super)(struct super_block*,void *,int);
– #ifdef VERSION > 2.4
– Struct module *owner;
– Struct vfsmount *kern_mnt;
– #endif
– Struct file_system_type *next;
– };
8. Mounting
Any file system in UNIX must be mounted before it can be
accessed,either through mount() or mount_root().
What is mount_root() ?
Representation of a mounted file system : struct
super_block.Limited to NR_SUPER.
It is initialized by the function “read_super” in the VFS.It
interrogates floppy disks,CD-ROM for a change of
media,tests if superblock is present and if so returns it.else
it calls the fs-specific function to create a superblock.
File system specific Read_super() reads its data from the
block device using the LINUX cache
functions,bread,brw_page.Processes may be put to sleep
when reading/writing or mounting file systems.
10. Locking/Unlocking superblock
Lock_super(struct super_block *sb)
{
– If(sb->s_lock) __wait_on_super(sb);
– Sb->s_lock=1;
– }
– unlock_super(struct super_block *sb)
– {
– sb->s_lock=0;
– wake_up(&sb->s_wait);
– }
– The read_super( ) function in the file system specific
implementation should also make the root inode available using
the function iget( );
11. Superblock operations
s_op points to a vector of functions for accessing the file system.
struct super_operations {
void (*read_inode)(struct inode *);
Int (*notify_change)(struct inode *,struct iattr *);
Void (*write_inode)(struct inode *);
Void (*put_inode)(struct inode *);
Void (*put_super)(struct super_block *);
Void (*write_super)(struct super_block *);
Void (*statfs)(struct super_block *,struct statfs *);
Void (*remount_fs)(struct super_block *,int *,char *);
};
These functions serve to translate the specific representation of the
superblock and inode on data media to their general form in memory and
vice-versa.Does every file system have a superblock and inode?
Elaborating a little more…
12. Write_super(sb) : saves info on super block.Used in synchronising the
device.dirty bit is set in buffer.
Put_super(sb) : Called when unmounting file systems.Releases
super_block and buffers used.restores consistency if
necessary.super_block structure can be reused.
Statfs(sb,statfsbuf) : fills the statfs structure with file system
information.
Remount_fs(sb,flags,options) : new attributes for file system and
restoring consistency.
Read_inode(inode) : fills in the inode structure.Marks different file
types by using different inode_operations struct.for example
If(S_ISREG(inode->I_mode)
Inode->I_op=&ext2_file_inode_operations;
Else if(S_ISDIR(inode->I_mode)
Inode->I_op=&ext2_dir_inode_operations;
Else if(S_ISLNK(inode->I-mode)
Inode->I_op=&ext2_symlink_inode_operations;
…….Similiarly for character devices,block devices,fifos,etc
13. Notify_change(inode,attr) : changes made to inode are
acknowledged by this function.It is of interest only in NFS/
Coda etc.Why?Each file on such a system has a local and a
remote inode.The remote file system is informed of the
change using iattr.
Write_inode(inode) : saves inode structure.
Put_inode(inode) : if inode is no longer required.Called
when deleting file and release its blocks.
15. INODE…contd
Struct inode *I_mount;
Struct socket *I_socket;
Reference_counter,flags,lock,bits that denote if this is a pipe,socket.
Union {
Struct pipe_node_info pipe_i;
Struct minix_inode_info minix_i;
….
Void *generic_ip;
}u;
};
Inodes are managed as doubly linked lists as well as a hash table.starts with
first_inode(/ inode).iget() function can be used to get the inode specified by
the superblock.It uses hints to resolve cross mounted file systems as
well.Any access to inode increments a usage counter.Analogous function
iput( ).
16. Inode Operations
Struct inode_operations {
Struct file_operations *default_file_ops;
Int (*create)(struct inode *,const char *,int,int,struct inode **);
Int (*lookup)(struct inode *,const char *,int,struct inode **);
Int (*link)(struct inode *,struct inode *,const char *,int);
Int (*unlink)(struct inode *,const char *,int);
Int (*symlink)(struct inode *,const char *,int);
Int (*mkdir)(struct inode *,const char *,int);
Similiarly we have
rmdir,mknod,rename,readlink,follow_link,bmap,truncate,perm
ission,smap.
NOTE :All these functions are directly called from the
implementation of the corresponding system call.
17. File System Calls.
Create(),lookup(),link(),unlink(),symlink(),mkdir(),rmdir(),
mknod(),rename(),readlink(),follow_link()
Bmap() : used for mmap’ing files .This function is called by the function
generic_mmap() to map a block of data from file to an address in the user
space.
Smap():allows swap files to be created on a UMSDOS file system.
To allow for shared accesses to files,UNIX introduced an extra structure.
Struct file {
Mode_t f_mode;
Loff_t f_pos;
Unsigned short f_flags;
Unsigned short f_count;
Off_t f_reada;/* readahead flag */
Struct file *f_next,*f_prev;
Struct inode *f_inode;
Struct file_operations *f_op;
Void *private_data;
};
18. File Operations .
General interface to work on files,Has functions to
open,close,read,write etc.Why are these isolated from
inode_operations?These operations also need to modify struct
file.Hence.
Struct file_operations {
Struct module *owner;
Int (*lseek)(…)
Read,write,readdir,select,ioctl,mmap,open,release,fsync,fasyn
c,check_media_change,revalidate..
These are also used by sockets/drivers etc..The inode
operations only use the representation of the socket/device .
If any of these functions are not implemented ,it tries to call
default file operations function,failing which an error is
returned.Certain functions only make sense for drivers,like
release,open
19. Opening Files
Open : most complicated system call. Authorization to open
the file is checked,an empty file structure is allocated and
entered in file decsriptor table of process and open_namei() is
used to retreive the inode for the file.Performs a lot of tests.
Name ends in a /.What should be done?
If O_CREAT is set,What function should this call?
What about symbolic links,directories?
Permission is checked with the inode permission list also.
What about character devices and block devices ?
Directory cache originated in ext2fs.Directory names are
cached.It is a 2-level cache with LRU at both levels.Hash key
is formed by the dev number and the inode number and the
name.Exports dcache_add,dcache_lookup functions
20. PROC FILE SYSTEM
Every process has a directory in /proc/pid which contains files relevant
to the process.
VFS function read_super( ) is called by do_mount( ) and in turn calls
proc_read_super( ) for the proc file system.
Struct super_block *proc_read_super(struct super_block *s,void *data,
int silent)
{
Lock_super(s);
Sets some fields in s.
S->s_op=&proc_sops;
Unlock_super(s);
If(!(s->s_mounted=iget(s,PROC_ROOT_INO))) {
S->s_dev=0;/* free the super block */
Return NULL;
}
Parse_options(data,&s->s_mounted->I_uid,s->s_mounted->I_gid);
Return s;
}
21. PROC FILE SYSTEM…contd
Some of the invariable entries in the root dircetory entry are
. , .. ,
loadavg,uptime,meminfo,kmsg,version,pci,cpuinfo,self,iopo
rts,profile.
22. EXT2 File System
Originally used MINIX file system.Serious limitations like partition
size,file name size etc.Proposed ext file system which was even
worse.Led to excessive fragmentation.Remy Card et’al propose ext2fs
which is the LINUX file system.
Influenced by BSD FFS.Every partition is divided into a number of block
groups,corresponding to the cylinder groups of FFS,with each block
group holding a copy of superblock,inode and data blocks.
23. STRUCTURE OF E2FS
What is the basic idea of block groups ?
The physical superblock contains information on the number
of inodes,blocks per block groups,time of last mount,last
write to sb,mount counter,maximum mount
operations.Padded to a size of 1k bytes.
Block group descriptors:Information on block groups.Each
block group is described by a 32 byte desc.Contains block
numbers in the inode bitmap,block bitmap and inode
table,number of free inodes,number of free blocks,number of
directories in this block group.
Number of directories is made use by allocator to spread files
evenly.
Block size=1024bytes => 8192 blocks per block group.
Inode size=128 Bytes.
24. Directories in e2fs
Directories are singly linked lists.
Struct ext2_dir_entry {
Unsigned long inode;
Unsigned short rec_len;
Unsigned short name_len;
Char name[EXT2_NAME_LEN];
};
What happens on a delete?
25. Block allocation in e2fs
Attempts to prevent fragmentation as much as possible.
Target-oriented allocation.
Always look for space in the area of target blocks.If this block
is free use that,otherwise look for a block within 32 blocks from
this.If not found then try to allocate from the same block
group.else go to other block groups.
Pre-allocation:
If a free block is found then upto 8 of the following blocks are
reserved if they are free.When a file is closed,the remaining
blocks that are reserved are released.Also called clustering of
data blocks.
How is target block determined?Let n=logical block #,l=logical
block# of the last block allocated.Then l+1 is a possible target
block
Editor's Notes
Functions are made known to the VFS via the function “register_filesystem”
Sets up the first file system.called by system call setup() after all file systems have been registered.setup() is called after init is created.Why is a system call required here?User mode process.
Some of the operations listed in the superblock may be NULL in which case no action will take place.
Inode operations create.
Keep data blocks close to inode,file inodes close to directory inode.
Don’t push/move the buffers around.Just change the value of the previous entry’s reclen to include this entry as well.Zero the inode number also.It can be reclaimed easily as well.
Don’t push/move the buffers around.Just change the value of the previous entry’s reclen to include this entry as well.Zero the inode number also.It can be reclaimed easily as well.