Quo vadis Linux File Systems: An  operations point of view on      EXT4 and BTRFS           Udo Seidel
Agenda●   Introduction/motivation●   EXT4 – the new member of the extfs family    ●   Facts, specs    ●   Operational cons...
Me ;-)●   Teacher of mathematics and physics●   PhD in experimental physics●   Started with Linux in 1996●   Linux/UNIX tr...
File systems challenges●   Speed●   Size/growth●   Flexibility●   Sustainability                     LinuxCon Europe 26-28...
Linux file systems●   More than 50 file systems shipped with Linux    kernel    ●   Local    ●   Remote    ●   Cluster    ...
Operational challenges with Linux              file systems●   ReiserFS    ●   sun-setted    ●   Big installation base●   ...
Chances, possibilities, choices for       the next Linux file systems●   New version of the extfs family -> EXT4     ●   M...
th               4 extended file system●   Shipped since 2.6.19●   Stable since 2.6.28●   To overcome limits of EXT3    ● ...
EXT4 - facts●   Max volume size: 1 EB = 1024 PB (EXT3: 16    TB)●   Max file size: 16 TB (EXT3: 2 TB)●   Max length of fil...
EXT4 – operational first glance●   Looks promising●   OPROC/procedure changes?●   Behaviour changes?●   Migration         ...
EXT4 – OPROCs/procedures●   Known tools    ●   mkfs    ●   fsck    ●   tune2fs    ●   e2label    => Easy to re-use or adap...
EXT4 – Behaviour changes ...●   Extents●   Improved block allocation●   Journaling●   File system check                   ...
EXT4 – from blocks to extents●   Common addressing for modern file systems    ●   Overdue    ●   Contiguous area of blocks...
EXT4 – delayed allocation●   Use cache information for placement●   Less fragmentation●   Risk of data loss in early versi...
EXT4 – “clever” allocation●   Support of system call fallocate()    ●   Application reserves blocks ahead    ●   File syst...
EXT4 – consistency via JBD2●   Transactions have checksums●   64 bit ready●   Deactivation possible    ●   Keep other EXT4...
EXT4 – repair●   Improved fsck()    ●   No check of unused blocks        –   information stored in block group header     ...
EXT4 – other news●   Nano second precision time stamps    ●   Unix millennium bug shifted to 2514●   More subdirectories  ...
EXT4 – background for migration●   2 kind of changes compared to EXT3    ●   change of on-disk format:        –   Extents ...
EXT4 – general migration paths●   mkfs() and backup/restore    ●   Clean new file system structure    ●   Only way for fil...
EXT4 – migration strategy●   It depends●   For us: both options    ●   Significant installation base of non-EXT2/3    ●   ...
EXT4 – migration via mkfs()●   For all new deployments●   for existing file systems    ●   On demand    ●   By chance     ...
EXT4 – migration via tune2fs●    Results in mix of EXT2/3 and EXT4 structure●    Access via EXT2/3 driver impossible●    f...
EXT4 – migration hints●   fsck() recommended●   /boot – booting from EXT4 possible?●   Rescue media enabled for EXT4?●   B...
EXT4 – summary●   Good successor of EXT3●   Manages higher amount of data●   Faster    ●   Performance    ●   recovery●   ...
Better/b-tree file system●   Shipped since 2.6.29●   Still experimental    ●   In vanilla kernel    ●   But .... productio...
BTRFS - facts●   Max volume/file size: 16 EB●   Max length of file name: 256 Bytes●   Support of    ●   Extended attribute...
BTRFS – operational first glance●   OPROC/procedure changes?●   Behaviour changes?●   Migration                 LinuxCon E...
BTRFS – OPROCs/procedures●   Known tools    ●   mkfs ... with limitation    ●   fsck ... with limitation    ●   tune2fs ->...
BTRFS – behaviour changes ...●   Device management●   Snapshots●   Compression                  LinuxCon Europe 26-28/10 2...
BTRFS – device management●   Included volume manager    ●   RAID-0, RAID-1, ...    ●   Add/remove devices    ●   ...●   Ne...
BTRFS – snapshots●   Not really new ... but not used so far●   Integration into package management    ●   automatic - no a...
BTRFS – compression●   Transparent●   Flexible:    ●   different algorithms possible    ●   Easy to switch on/off●   Opera...
BTRFS – file system check●   Yes ... there is something ....●   Reliable?●   Use of alternatives?!    ●   Backup/restore  ...
BTRFS – what else●   Support of POSIX ACLs●   Online grow/shrink●   Online add/removal of disks●   SSD-aware●   Management...
BTRFS – migration paths●   mkfs() and backup/restore    OR●   In place from EXT3/4 via tool btrfs-convert    ●   Via libe2...
BTRFS – migration strategy●   In place via btrfs-convert    ●   Fast    ●   EXT3/4 a given in data centre by then    ●   C...
BTRFS – migration hints●   New file system – check everything!●   General knowledge/awareness●   /boot – booting from BTRF...
BTRFS summary●   Still experimental●   Meets standard file systems requirements●   Bridges existing gaps●   Easy migration...
Summary●   Improvement moving to EXT4●   Safe switching to EXT4●   In place migration from EXT3 possible●   Future is BTRF...
References●   http://ext4.wiki.kernel.org●   http://btrfs.wiki.kernel.org                     LinuxCon Europe 26-28/10 201...
Thank you!LinuxCon Europe 26-28/10 2011   42
Upcoming SlideShare
Loading in...5
×

Linuxconeurope2011.ext4btrfs.talk

358
-1

Published on

EXT4 and BTRFS from an operational point of view (autumn/fall 2011)

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
358
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Linuxconeurope2011.ext4btrfs.talk

  1. 1. Quo vadis Linux File Systems: An operations point of view on EXT4 and BTRFS Udo Seidel
  2. 2. Agenda● Introduction/motivation● EXT4 – the new member of the extfs family ● Facts, specs ● Operational considerations● BTRFS – the newbie .. the hope ● Facts, specs ● Operational considerations● Summary LinuxCon Europe 26-28/10 2011 2
  3. 3. Me ;-)● Teacher of mathematics and physics● PhD in experimental physics● Started with Linux in 1996● Linux/UNIX trainer● Solution engineer in HPC and CAx environment● Joined Amadeus in 2006● Head of a international team with people in Erding, Miami and Sydney LinuxCon Europe 26-28/10 2011 3
  4. 4. File systems challenges● Speed● Size/growth● Flexibility● Sustainability LinuxCon Europe 26-28/10 2011 4
  5. 5. Linux file systems● More than 50 file systems shipped with Linux kernel ● Local ● Remote ● Cluster ● ...● A few as standard for root directory ● EXT2, EXT3 ● XFS LinuxCon Europe 26-28/10 2011 5
  6. 6. Operational challenges with Linux file systems● ReiserFS ● sun-setted ● Big installation base● EXT3 ● Stable, robust ● Slow● XFS ● Additional costs?● Changes in recent Enterprise distributions LinuxCon Europe 26-28/10 2011 6
  7. 7. Chances, possibilities, choices for the next Linux file systems● New version of the extfs family -> EXT4 ● Marked as stable ● Shipped with Enterprise distributions● New approach with BTRFS ● Still experimental in vanilla kernel ● Shipped as production ready by some Enterprise distributions LinuxCon Europe 26-28/10 2011 7
  8. 8. th 4 extended file system● Shipped since 2.6.19● Stable since 2.6.28● To overcome limits of EXT3 ● Size ● Performance LinuxCon Europe 26-28/10 2011 8
  9. 9. EXT4 - facts● Max volume size: 1 EB = 1024 PB (EXT3: 16 TB)● Max file size: 16 TB (EXT3: 2 TB)● Max length of file name: 256 Bytes● Support of extended attributes● No encryption● Not really compression● Partially 64bit LinuxCon Europe 26-28/10 2011 9
  10. 10. EXT4 – operational first glance● Looks promising● OPROC/procedure changes?● Behaviour changes?● Migration LinuxCon Europe 26-28/10 2011 10
  11. 11. EXT4 – OPROCs/procedures● Known tools ● mkfs ● fsck ● tune2fs ● e2label => Easy to re-use or adapt of existing :-) LinuxCon Europe 26-28/10 2011 11
  12. 12. EXT4 – Behaviour changes ...● Extents● Improved block allocation● Journaling● File system check LinuxCon Europe 26-28/10 2011 12
  13. 13. EXT4 – from blocks to extents● Common addressing for modern file systems ● Overdue ● Contiguous area of blocks – Less management information needed – Less meta data operations – Less “fragmentation”● Requires change of on-disk format :-( ● Migration paths? LinuxCon Europe 26-28/10 2011 13
  14. 14. EXT4 – delayed allocation● Use cache information for placement● Less fragmentation● Risk of data loss in early versions => improved since 2.6.30● Easy switch on/off● No change of on-disk format! :-) LinuxCon Europe 26-28/10 2011 14
  15. 15. EXT4 – “clever” allocation● Support of system call fallocate() ● Application reserves blocks ahead ● File system ensures disk space availability● Allocation information in extent structure ● Only usable with extents! LinuxCon Europe 26-28/10 2011 15
  16. 16. EXT4 – consistency via JBD2● Transactions have checksums● 64 bit ready● Deactivation possible ● Keep other EXT4 features ● Comparison: EXT3 ... -> EXT2 ● Reduction of file systems in use :-) LinuxCon Europe 26-28/10 2011 16
  17. 17. EXT4 – repair● Improved fsck() ● No check of unused blocks – information stored in block group header – Information secured via checksums – (de)activation possible at any time ● First run possibly as slow like in EXT3 LinuxCon Europe 26-28/10 2011 17
  18. 18. EXT4 – other news● Nano second precision time stamps ● Unix millennium bug shifted to 2514● More subdirectories ● Up to 65000 ● More than 65000 ... with limitation LinuxCon Europe 26-28/10 2011 18
  19. 19. EXT4 – background for migration● 2 kind of changes compared to EXT3 ● change of on-disk format: – Extents – Only enabled for new files via tune2fs – Additional tasks needed ● On-disk format not relevant: – block allocation – Immediately enabled via tune2fs LinuxCon Europe 26-28/10 2011 19
  20. 20. EXT4 – general migration paths● mkfs() and backup/restore ● Clean new file system structure ● Only way for file systems other than EXT2/3 ● Extended outage● Conversion via tune2fs ● Partial only ● Possible for EXT2/3 family only ● Faster/easier LinuxCon Europe 26-28/10 2011 20
  21. 21. EXT4 – migration strategy● It depends● For us: both options ● Significant installation base of non-EXT2/3 ● Big landscape (2500+ Linux servers) LinuxCon Europe 26-28/10 2011 21
  22. 22. EXT4 – migration via mkfs()● For all new deployments● for existing file systems ● On demand ● By chance LinuxCon Europe 26-28/10 2011 22
  23. 23. EXT4 – migration via tune2fs● Results in mix of EXT2/3 and EXT4 structure● Access via EXT2/3 driver impossible● fsck() needed parameter description extent Extent based block allocation flex_bg Flexible placement of meta data uninit_bg Flag uninitialized blocks for faster fsck dir_nlink Infinite number of sub directories extra_isize Timestamps with nano seconds LinuxCon Europe 26-28/10 2011 23
  24. 24. EXT4 – migration hints● fsck() recommended● /boot – booting from EXT4 possible?● Rescue media enabled for EXT4?● Backup/restore of EXT4? LinuxCon Europe 26-28/10 2011 24
  25. 25. EXT4 – summary● Good successor of EXT3● Manages higher amount of data● Faster ● Performance ● recovery● Safer● Sufficient migration options from/to EXT2/3 => GO! LinuxCon Europe 26-28/10 2011 25
  26. 26. Better/b-tree file system● Shipped since 2.6.29● Still experimental ● In vanilla kernel ● But .... production ready in some Enterprise distributions● Supposed to replace EXT3/4● New storage management approach LinuxCon Europe 26-28/10 2011 26
  27. 27. BTRFS - facts● Max volume/file size: 16 EB● Max length of file name: 256 Bytes● Support of ● Extended attributes ● No encryption yet ● Snapshot ● Compression ● Copy-on-Write LinuxCon Europe 26-28/10 2011 27
  28. 28. BTRFS – operational first glance● OPROC/procedure changes?● Behaviour changes?● Migration LinuxCon Europe 26-28/10 2011 28
  29. 29. BTRFS – OPROCs/procedures● Known tools ● mkfs ... with limitation ● fsck ... with limitation ● tune2fs -> nope ● XXlabel -> nope => No easy to re-use or adapt of existing :-( LinuxCon Europe 26-28/10 2011 29
  30. 30. BTRFS – behaviour changes ...● Device management● Snapshots● Compression LinuxCon Europe 26-28/10 2011 30
  31. 31. BTRFS – device management● Included volume manager ● RAID-0, RAID-1, ... ● Add/remove devices ● ...● New tool ● change of OPROCs/procedures :-( ● Change to CLI/GUI or new CLI/GUI tools :-( LinuxCon Europe 26-28/10 2011 31
  32. 32. BTRFS – snapshots● Not really new ... but not used so far● Integration into package management ● automatic - no additional tasks ● Faster roll-back ● Post change tracking● New tool ● New OPROCs anyway :-| ● Minimal human intervention :-) LinuxCon Europe 26-28/10 2011 32
  33. 33. BTRFS – compression● Transparent● Flexible: ● different algorithms possible ● Easy to switch on/off● Operations: ● Minimal changes on admin side ● No changes on other levels :-) LinuxCon Europe 26-28/10 2011 33
  34. 34. BTRFS – file system check● Yes ... there is something ....● Reliable?● Use of alternatives?! ● Backup/restore – Possible longer outage :-( – covered in OPROCs :-) ● Snapshots -> new at all :-( LinuxCon Europe 26-28/10 2011 34
  35. 35. BTRFS – what else● Support of POSIX ACLs● Online grow/shrink● Online add/removal of disks● SSD-aware● Management tool evolution (btrfsctl -> btrfs)● du/df not fully BTRFS-aware LinuxCon Europe 26-28/10 2011 35
  36. 36. BTRFS – migration paths● mkfs() and backup/restore OR● In place from EXT3/4 via tool btrfs-convert ● Via libe2fs ● BTRFS meta data location flexible ● Old EXT3/4 organized in snapshot ● Roll-back possible to date/time of conversion LinuxCon Europe 26-28/10 2011 36
  37. 37. BTRFS – migration strategy● In place via btrfs-convert ● Fast ● EXT3/4 a given in data centre by then ● Change management friendly LinuxCon Europe 26-28/10 2011 37
  38. 38. BTRFS – migration hints● New file system – check everything!● General knowledge/awareness● /boot – booting from BTRFS possible?● Rescue media enabled for BTRFS?● Backup/restore of BTRFS? LinuxCon Europe 26-28/10 2011 38
  39. 39. BTRFS summary● Still experimental● Meets standard file systems requirements● Bridges existing gaps● Easy migration from EXT3/4 possible● New approach to storage management● Prospects ● SSD ● Compression LinuxCon Europe 26-28/10 2011 39
  40. 40. Summary● Improvement moving to EXT4● Safe switching to EXT4● In place migration from EXT3 possible● Future is BTRFS● In place migration from EXT3/4 to BTRFS possible LinuxCon Europe 26-28/10 2011 40
  41. 41. References● http://ext4.wiki.kernel.org● http://btrfs.wiki.kernel.org LinuxCon Europe 26-28/10 2011 41
  42. 42. Thank you!LinuxCon Europe 26-28/10 2011 42
  1. Gostou de algum slide específico?

    Recortar slides é uma maneira fácil de colecionar informações para acessar mais tarde.

×