Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How to design a file system

264 views

Published on

Goes into details about the prior knowledge required to design a file system. Concludes with a demo that explains how these concepts are used by Second Extended File System (ext2fs) A demo is also provided.

Presented at Florence, Italy on 6th December 2017

Published in: Software
  • Be the first to comment

  • Be the first to like this

How to design a file system

  1. 1. How to design a FileSystem A Beginner’s Tale V N Nikhil Anurag
  2. 2. Who are you?
  3. 3. Who are you? ● Curious about File Systems ● Interested in low level systems ● Building an OS from scratch is hard! ● Not sure how to approach such a task ● C/C++ is not your forte
  4. 4. Who are you? What we really want ● Feasible ● Practical ● Yet involves complexity of low level system design
  5. 5. Why not design a File System (FS)?
  6. 6. Who am I?
  7. 7. Who am I? ● Software Engineer (REST & API) ● Tools of trade: Go, Python & Clojure ● Github: https://github.com/last-ent ● Twitter: @last_ent ● Site: www.last-ent.com ● Learns by Doing
  8. 8. Who am I?
  9. 9. Beginner’s Tale
  10. 10. File System & Operating System
  11. 11. File System & Operating System Hard Disk
  12. 12. File System & Operating System OS ● Where does the FS sit? ● How does the OS know about it? ● How is OS stored? Hard Disk
  13. 13. File System & Operating System OS File System Hard Disk
  14. 14. File System & Operating System FS Operations FS Formatted Data File System Process OS (in-memory)
  15. 15. Creating Or Initialization Mounting Or Opening Unmounting Or Closing Reading Writing Deleting File System ✔ ✔ ✔ Files ✔ ✔ ✔ ✔ ✔ ✔ Directories ✔ ✔ ✔ ✔ ✔ ✔ File System & Operating System Operations Type File System Operations
  16. 16. How is a File System mounted?
  17. 17. How can an Operating System load Hard Disk as a File System?
  18. 18. Investigating the Hard Disk
  19. 19. Investigating the Hard Disk /dev/sda - Hard Disk /dev/sdaX - Partitions on the Hard Disk
  20. 20. Unix Philosophy: Everything is a File
  21. 21. Investigating the Hard Disk: less -f /dev/sda1
  22. 22. Investigating the Hard Disk: less -f /dev/sda
  23. 23. Investigating the Hard Disk Conclusion: 1. Linux/Unix-like OSes view all hard disks as files. 2. A Hard Disk can be opened with standard “Open File” commands! 3. A Hard Disk contains a stream of bytes.
  24. 24. How to parse the bytes on the hard disk?
  25. 25. File System Constructs
  26. 26. File System Constructs Block The smallest unit of bytes available to the OS & FS. It is a multiple of disk’s block size. Block Size = n * 512 Bytes Block Group Boot Block Contiguous set of blocks that form a logical group. A Block Group consists of FS metadata & file data. Used by the computer to bootstrap kernel loading process. (Irrelevant for our purposes)
  27. 27. File System Constructs Boot Block Block Group 0 Block Group N Formatted Hard Disk or Partition
  28. 28. File System Constructs: Block Group Superblock Consists of all the metadata required by the OS to load the File System Inode Consists of all the metadata required by File System to load a File or a Directory. Metadata Bitmap Provides a minimap for the state of Blocks or other things.
  29. 29. File System Constructs: Block Group Boot Block Block Group 0 Block Group N Superblock Inode map Inodes Bitmap Data Bitmap Data Blocks Inodes
  30. 30. How does the Operating System identify a File System?
  31. 31. How does the OS identify a FS? ● A Hard Disk or Partition is just a stream of bytes. ● The OS can’t store the partition’s FS type anywhere else.
  32. 32. How does the OS identify a FS? File System Magic Signature ● Unique for each FS ● Stored at a unique place for each FS
  33. 33. File System Constructs: Block Group Boot Block Block Group 0 Block Group N Superblock Inode map Inodes Bitmap Data Bitmap Data Blocks Inodes $FS
  34. 34. We mounted the File System. What’s next?
  35. 35. Look at all the Files & Directories in the File System
  36. 36. Let’s start with Root Directory “/”
  37. 37. Root Directory ● Where is the Root Directory? ● How does the OS know about it. ● How is it represented?
  38. 38. Root Directory Inode Consists of all the metadata required by File System to load a File or a Directory. Inode map Look Up table to quickly reach Inodes based on their Inode# Root Directory is at a predefined position on the File System.
  39. 39. Root Directory ● Root Directory is at a predefined position on the File System. ● It can be looked up in the Inode map. ● An Inode can be interpreted differently based on whether it is a Directory or a File.
  40. 40. Root Directory Inode map Location of Root Directory Root Directory Inode
  41. 41. Directory Inode
  42. 42. Directory Inode Directory Inode Expected ● Directory Name ● Directory Contents Actual ● Directory Name ● Directory Entries ○ Inode ○ Name ○ Type
  43. 43. Root Directory’s name is “/” So we are safe!
  44. 44. Block0 Root Directory: Data Blocks / Inode: _ Block1 BlockN
  45. 45. Block0 Root Directory: Dentries in Block0 / Inode: _ dir1 Inode: #inode1 dirN Inode: #inodeN file1 Inode: #inodeF1 fileN Inode: #inodeFN
  46. 46. File Inode
  47. 47. File Inode File Inode Expected ● File Name ● File Size ● File Contents Actual ● File Name ● File Size ● File’s Data Blocks
  48. 48. Creating a File & Directory Look Up
  49. 49. Creating a File & Directory Look Up Create Directory: /dir1/subdir1 FileName: file.txt / dir0 dir1 subdir1 File Inode Name: file.txt Create Inode for file
  50. 50. Creating a File ● Find Directory Inode ● Create File Inode ○ Write all relevant metadata Don’t add File Content yet!
  51. 51. Case Study: ext2 File System
  52. 52. File System Constructs Block The smallest unit of bytes available to the OS & FS. It is a multiple of disk’s block size. Block Size = n * 512 Bytes Block Group Boot Block Contiguous set of blocks that form a logical group. A Block Group consists of FS metadata & file data. Used by the computer to bootstrap kernel loading process. (Irrelevant for our purposes)
  53. 53. File System Constructs: Block Group Superblock Consists of all the metadata required by the OS to load the File System Inode Consists of all the metadata required by File System to load a File or a Directory. Metadata Bitmap Provides a minimap for the state of Blocks or other things.
  54. 54. ext2 File System: Layout Boot Block Block Group 0 Block Group N Superblock Inode map Inodes Bitmap Data Bitmap Data Blocks Inodes $FS Group Descriptor
  55. 55. ext2 File System: Layout ● Magic Signature: 0xEF53 ● Root Folder Inode# 1 ● Number of Inodes: Size in Bytes ➗ Block Size
  56. 56. Demo
  57. 57. Demo Instructions 1. dd if=/dev/zero of=demo.ex2 bs=1024 count=131072 2. mke2fs ./demo.ex2 3. sudo mount ./demo.ext2 /demo-mnt 4. go run github.com/last-ent/ext2-explorer/main.go 5. Enter file path in browser: /home/entux/Documents/Code/demo/demo.ext2 6. View Ext2 Data 7. sudo umount /demo-mnt
  58. 58. How are File System operations connected to OS?
  59. 59. Operating System & FS Operations OS (in-memory) ext4 Open Close Dentries Inode ... btrfs Virtual File System - Used in Kernel Space FUSE - Used in User Space
  60. 60. Summary ● File System is just a program. ● It is well structured and there is no magic. ● Easy to implement a practical low level System.
  61. 61. References Understanding the Linux Kernel, Third Edition 3rd Edition Practical File System Design Demo Project - https://github.com/last-ent/ext2-explorer

×