The document discusses optimizing file systems for efficient workflows, focusing on RAID configurations and file system choices like ext4 and XFS. It emphasizes the importance of cache management, benchmarking tools such as sysbench, and understanding I/O patterns for improved performance. Best practices for setup, configuration, and tuning are provided, along with references to further resources on file systems and MySQL performance.

1. Exploiting Your File System to Build Robust &
Efficient Workflows
Jason Johnson
jajohnson@softlayer.com
Exploiting Your File System to Build Robust &
Efficient Workflows
Jason Johnson
jajohnson@softlayer.com

3. What is /dev/sdc, anyway?

4. The Hard Disk Drive

5. Basic Platter Geometry
Cylinder-Head-Sector (obsolete)
Logical Block Addressing, LBA

6. What is /dev/sdc, anyway?

7. The Disk Array Controller
● Adaptec 5405Z
● PCIe x8
● 1.2 GHz Dual Core RAID on Chip (ROC)
● 128-1024 MB Battery-Backed DDR
● 1-4 GB NAND
● Up to 256 SATA or SAS HDD's
● arcconf

8. Write Caching

9. “...you *must* disable the
individual hard disk write cache in
order to ensure to keep the file
system intact after a power
failure.”
XFS.org FAQ

10. Initial 1MB Sector Alignment
Sector Size Starting Sector Drive Type
512 B 2048 SATA & SAS
2 KB 512 SSD
4 KB 256 Advanced Format & SSD
blockdev --getpbsz /dev/sdc
blockdev --getss /dev/sdc
“Aligning IO on a hard disk RAID”
http://www.mysqlperformanceblog.com/2011/06/09/aligning-io-on-a-hard-disk-raid-the-theory/

11. (s)gdisk

12. Tuning the File System
● Disable Caching
● Tools: sysbench, iozone, iostat, vmstat
● Start Simple
● Apply Increasing Parallel I/O
● ext2, ext3, ext4, xfs, btrfs, zfs?
● Graph Everything

13. arcconf
arcconf
create 1 logicaldrive
stripesize 256
wcache wt
rcache roff
max
0
0 3
0 4
...
0 18

14. sysbench, fileio
sysbench
--num-threads=[8-1024]
--test=fileio
--file-total-size=10G
--file-test-mode=rndwr
--file-fsync-all=on
--file-num=64
--file-block-size=16384
[prepare|run|cleanup]

15. EXT4, mkfs
mkfs.ext4 /dev/sdc1
mke2fs
-b 4096
-O journal_dev
/dev/sdb1 32768
mkfs.ext4
-b 4096
-E stride=4,stripe_width=16
-J device=/dev/sdb1
/dev/sdc1
mount -o
noatime,stripe=16
/dev/sdc1
/mnt/data

16. I/O Requests per Second, EXT4

17. Latency, EXT4

18. XFS, mkfs
mkfs.xfs /dev/sdc1
mkfs.xfs
-d sw=16,su=16k
-l
logdev=/dev/sdb1,
size=128m,
su=256k
/dev/sdc1
mount -o
noatime,
logdev=/dev/sdb1,
logbufs=8,logbsize=256k
/dev/sdc1
/mnt/data

19. I/O Requests per Second, XFS

20. Latency, XFS

21. What are we looking for?
I/O Request per Second... per Drive
&
Reasonable Latency

22. XFS vs. EXT4, Latency

23. XFS vs. EXT4, per Drive

24. Scenario 1, Efficiency
MySQL

25. MySQL Write Pattern

26. MySQL Configuration
System Variable Value
innodb_io_capacity 5000
innodb_thread_concurrency 256
innodb_write_io_threads 192
innodb_read_io_threads 64
innodb_log_file_size 32M
innodb_log_files_in_group 32
innodb_buffer_pool_size 10GB
innodb_buffer_pool_instances 10
“MySQL System Variables”

27. sysbench, mysql
sysbench
--num-threads=[32|64|128|256]
--test=oltp
--oltp-test-mode=nontrx
--oltp-nontrx-mode=insert
--oltp-table-size=100000
--max-requests=10000000
[prepare|run|cleanup]

28. Transactions per Second
+96.28%
+125.37%
+102.29%
+69.43%
15,962.79/s @ 16ms
9,421.29/s @ 27ms

30. inotify
Event Mask Fired when...
IN_ACCESS File was accessed (read)
IN_ATTRIB Metadata changed
IN_CLOSE_WRITE File opened for writing was closed
IN_CLOSE_NOWRITE File not opened for writing was closed
IN_CREATE File/directory created in watched directory
IN_DELETE File/directory deleted from watched directory
IN_DELETE_SELF Watched file/directory was itself deleted
IN_MODIFY File was modified
IN_MOVE_SELF Watched file/directory was itself moved
IN_MOVED_FROM File moved out of watched directory
IN_MOVED_TO File moved into watched directory
IN_OPEN File was opened

32. inotify in [language]
Language Source
Python pip install pyinotify
PHP pecl install inotify
Go go's exp repository
Ruby gem install rb-inotify
C #include <sys/inotify.h>

33. Scenario 2, Robustness
A Custom Message Queue

34. Message Queue Architecture

35. I/O Serialization

36. I/O Serialization
hash(queue_id) % num_threads

37. Message Queue Architecture
64K

38. Summary
● Caching
● File system choice
● Benchmarking w/ sysbench
● Efficiency through proper configuration
● Robustness through cooperation & decoupling
● Discovering & understanding your write pattern
● Benchmark & Graph everything
● Never Assume Anything (atime, stripe width, etc.)

39. Jason Johnson jajohnson@softlayer.com
https://github.com/jasonjohnson
http://www.slideshare.net/jasonajohnson
“A Case for Redundant Arrays of Inexpensive Disks (RAID)”
http://www.cs.cmu.edu/~garth/RAIDpaper/Patterson88.pdf
“Practical File System Design”
http://www.nobius.org/~dbg/practical-file-system-design.pdf
“XFS Papers and Documentation”
http://xfs.org/index.php/XFS_Papers_and_Documentation
“Kernel Documentation on File Systems”
https://www.kernel.org/doc/Documentation/filesystems/
“MySQL Performance Blog”
http://www.mysqlperformanceblog.com/
“MySQL DBA”
http://mysqldba.blogspot.com/
“MySQL Server System Variables”
http://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html