Performance evaluation of Linux Discard Support

1,315 views
1,102 views

Published on

Basic overview of discard and why we have it. Introduce discard benchmarking tool and show some of its results. What have we done for ext4 and what can you use now.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,315
On SlideShare
0
From Embeds
0
Number of Embeds
18
Actions
Shares
0
Downloads
13
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Performance evaluation of Linux Discard Support

  1. 1. Performance evaluation of Linux DiscardSupport(Overview, benchmark results, current status)Red HatLuk´ˇ Czerner asMay 27, 2011
  2. 2. Copyright © 2011 Luk´ˇ Czerner, Red Hat. asPermission is granted to copy, distribute and/or modify thisdocument under the terms of the GNU Free DocumentationLicense, Version 1.3 or any later version published by the FreeSoftware Foundation; with no Invariant Sections, no Front-CoverTexts, and no Back-Cover Texts. A copy of the license is includedin the COPYING file.
  3. 3. Part IDiscard Background
  4. 4. Agenda1 SSD Description2 Thinly Provisioned Storage3 Introducing Linux Discard Support
  5. 5. SSD DescriptionSolid-State Drive Flash memory block device Wear-leveling needed Firmware = black box
  6. 6. SSD DescriptionATA TRIM Command Helps handle garbage collection overhead Subsequent READ of TRIM’ed blocks 1 Read data should NOT change between READ’s 2 Read data should NOT be retrieved from data previously written to any other LBA. As long as the device has enough free pages to write to we do not necessarily need it. In a nutshell: TRIM command tells the device what LBA’a is not used by the OS anymore.
  7. 7. Thinly Provisioned StorageThin Provisioning Unlike in traditional storage, there is no fixed one-to-one logical bock to physical storage mapping More efficient use of storage space Block reclamation interface needed
  8. 8. Thinly Provisioned StorageSCSI UNMAP / WRITE SAME Storage space reclamation interface Subsequent READ of unmapped blocks 1 Read data should NOT change between READ’s 2 Read data should NOT be retrieved from data previously written to any other LBA. Unlike with SSD’s we can not afford to wait until we run out of space for reclamation.
  9. 9. Introducing Linux Discard SupportLinux Discard Implementation Abstraction for the two underlying specifications: 1 ATA TRIM Command 2 SCSI UNMAP / WRITE SAME API for user-space BLKDISCARD ioctl Added with v2.6.27-rc9-30-gd30a260 API for File Sytems 1 sb issue discard() 2 blkdev issue discard()
  10. 10. Part IIDiscard Performance
  11. 11. Agenda4 Testing Methodology5 Results
  12. 12. Testing MethodologyWhat do we need to find out ? Does discard really work ? Is it reliable ? How fast/slow is it ? Is there any difference between devices from different vendors ? What is the ideal discard size ? SSD performance degradation
  13. 13. Testing MethodologyHow do we test it ? BLKDISCARD ioctl() Automatic discard of different ranges Different discard patterns 1 sequential performance 2 random IO peformance 3 discard already discarded blocks test-discard - discard benchmarking tool http://sourceforge.net/projects/test-discard/ impression - filesystem aging tool
  14. 14. ResultsSequential discard performance 300 1200 Duration Summary Throughput 250 1000 200 800 Throughput [MB/s]Duration [s] 150 600 100 400 50 200 0 0 10 100 1000 Record size [kB]
  15. 15. ResultsDifferent modes comparison 1200 Throughput (sequential) Throughput (random IO) Throughput (discard2) 1000 800Throughput [MB/s] 600 400 200 0 10 100 1000 Record size [kB]
  16. 16. ResultsDifference between various vendors 1200 Throughput Vendor 1 Throughput Vendor 2 Throughput Vendor 3 1000 800Throughput [MB/s] 600 400 200 0 10 100 1000 Record size [kB]
  17. 17. ResultsSSD performance degradation 70 Vendor1 Vendor2 Vendor3 60 Vendor3 discard Write performance [MB/s] 50 40 30 20 10 0 50 100 150 200 250 300 350 400 Filesystem saturation [%]
  18. 18. Part IIIDiscard Support for Linux File systems
  19. 19. Agenda6 Periodic Discard7 Discard Batching8 Different Approach
  20. 20. Periodic DiscardPeriodic discard Easy to implement File system support 1 ext4 (v2.6.27-5185-g8a0aba7) 2 btrfs (since upstream) 3 gfs2 (v2.6.29-9-gf15ab56) 4 fat, swap, nilfs mount -o discard /dev/sdc /mnt/test TRIM is non-queueable command - implications ?
  21. 21. Periodic DiscardBenchmarking periodic discard Expectations ? Testing methodology 1 Metadata intensive load 2 Load with removing files 3 Reasonable file size distribution Discard-kit 1 Using PostMark 2 http://sourceforge.net/projects/test-discard/files/
  22. 22. Periodic DiscardExt4 performance (18% hit) 800 Deleted/s Append/s Read/s 700 Files-created/s Transactions/s Read[B/s] 600 Write[B/s] Opetation/s 500 400 300 200 100 nodiscard discard
  23. 23. Periodic DiscardPerformance with various file systems 750 ext4 700 btrfs 650 gfs2 Transactions/s 600 550 500 450 400 350 no di no di no di sc sc sc di di di ar ar ar sc sc sc d d d ar ar ar d d -18% -7% -63% d
  24. 24. Discard BatchingDiscard Batching - The idea Fine-grained discard is not necessarily needed Small extents are slow With time, freed extents tends to coalesce Disadvantages 1 There is a price for tracking freed extents 2 Discarding already discarded blocks should be easy, but... 3 Daemon (in-kernel, user-space) needed. 4 File system independent solution would most likely be pain to do right (if possible).
  25. 25. Discard BatchingBatched discard support File system specific solution Provide ioctl() interface - FITRIM Do not disturb other ongoing IO too much 1 Prevent allocations while trimming 2 How to handle huge filesystem ? File system support 1 ext4 (v2.6.36-rc6-35-g7360d17) 2 ext3 (v2.6.37-11-g9c52749) 3 xfs (v2.6.37-rc4-63-ga46db60)
  26. 26. Discard BatchingFITRIM ioctl Ioctl with one RW parameter defined in linux/fs.h struct fstrim range { u64 start; u64 len; u64 minlen; } fstrim tool http://sourceforge.net/projects/fstrim/ util-linux-ng Since v2.18-165-gd9e2d0d
  27. 27. Discard BatchingBatched discard benchmark results 6 FITRIM on ext4 BLKDISCARD 5 4 Duration [s] 3 2 1 0 0 20 40 60 80 100 Filesystem saturation [%]
  28. 28. Different ApproachAlternative approach It is always a compromise The future of SSD’s and thinly provisioned LUN’s (???)
  29. 29. Part IVDiscard Support in user-space
  30. 30. Agenda9 e2fsprogs10 Other utilities
  31. 31. e2fsprogsDiscard in e2fsprogs tools Using BLKDISCARD ioctl() mke2fs 1 Refresh SSD’s garbage collector 2 discard zeroes data - significant speed boost 3 mkfs.ext4 -E discard /dev/sdc e2fsck 1 After the last check discard free space 2 Non detected file system errors ? oops 3 fsck.ext4 -E discard /dev/sdc resize2fs 1 Refresh SSD’s garbage collector 2 discard zeroes data - significant speed boost 3 resize2fs -E discard /dev/sdc
  32. 32. e2fsprogsFile system creation 30 nodiscard discard 25 20Duration [s] 15 10 5 0 EXT4 XFS File system
  33. 33. Other utilitiesFstrim tool Very simple tool to invoke FITRIM ioctl on mounted file system Stand-alone tool http://sourceforge.net/projects/fstrim/ Since v2.18-165-gd9e2d0d part of util-linux-ng
  34. 34. Part VSummary
  35. 35. Summary Linux Discard support is a abstraction for underlying specification Exported via BLKDISCARD ioctl to user-space and blkdev issue discard() for filesystems Discard testing kit (Discard-kit) 1 test-discard 2 PostMark Filesystem support 1 Fine grained (online) discard - mount -o discard 2 Batched discard support - fstrim from util-linux-ng Support in user-space utilities 1 Filesystem creation (mkfs) 2 e2fsprogs - mkfs,e2fsck,resize2fs 3 xfsprogs - mkfs 4 fstrim
  36. 36. The end.Thanks for listening.
  37. 37. Useful links http://sourceforge.net/projects/fstrim/ http://sourceforge.net/projects/test-discard/ http://people.redhat.com/lczerner/discard/

×