Disk I/O is much slower than memory access. When reading from a file, the OS first checks the page cache in memory, which has nanosecond access times, before needing to physically access the disk with millisecond seek times. When writing to a file, pages are marked as dirty in the cache before being written to disk asynchronously by flush threads. Fsync forces an immediate write to disk, but caches can lie and data may not be safely written. Virtualized environments add additional layers of caching that can affect performance unpredictably. Hardware techniques like SSDs, RAID controllers, and dedicated flash cards can improve I/O speeds.

1. What Every Data Programmer Needs to Know about DisksOSCON Data – July, 2011 - PortlandTed Dziuba@dozbatjdziuba@gmail.comNot proprietary or confidential. In fact, you’re risking a career by listening to me.

2. Who are you and why are you talking?First job: Like college but they pay you to go.A few years ago: Technical troll for The Register.Recently: Co-founder of Milo.com, local shopping engine.Present: Senior Technical Staff for eBay Local

3. The Linux Disk AbstractionVolume/mnt/volumeFile Systemxfs, extBlock DeviceHDD, HW RAID array

4. What happens when you read from a file?f = open(“/home/ted/not_pirated_movie.avi”, “rb”)avi_header = f.read(56)f.close()Diskcontrolleruserbufferpagecacheplatter

5. What happens when you read from a file?DiskcontrolleruserbufferpagecacheMain memory lookup

6. Latency: 100 nanoseconds

7. Throughput: 12GB/sec on good hardwareplatter

8. What happens when you read from a file?DiskcontrolleruserbufferpagecacheNeeds to actuate a physical device

9. Latency: 10 milliseconds

10. Throughput: 768 MB/sec on SATA 3

11. (Faster if you have a lot of money)platter

12. Sidebar: The Horror of a 10ms Seek LatencyA disk read is 100,000 times slower than a memory read.100 nanosecondsTime it takes you to write a really clever tweet10 millisecondsTime it takes to write a novel, working full time

13. What happens when you write to a file?f = open(“/home/ted/nosql_database.csv”, “wb”)f.write(key)f.write(“,”)f.write(value)f.close()Diskcontrolleruserbufferpagecacheplatter

14. What happens when you write to a file?f = open(“/home/ted/nosql_database.csv”, “wb”)f.write(key)f.write(“,”)f.write(value)f.close()DiskcontrolleruserbufferpagecacheplatterYou need to make thispart happenMark the page dirty,call it a day and go have a smoke.

15. Aside: Stick your finger in the Linux Page CachePre-Linux 2.6 used “pdflush”, now per-Backing Device Info (BDI) flush threadsDirty pages: grep –i “dirty” /proc/meminfo/proc/sys/vmLove:dirty_expire_centisecs : flush old dirty pages

16. dirty_ratio: flush after some percent of memory is used

17. dirty_writeback_centisecs: how often to wake up and start flushingClear your page cache: echo 1 > /proc/sys/vm/drop_cachesCrusty sysadmin’s hail-Mary pass: sync; sync; sync

18. Fsync: force a flush to diskf = open(“/home/ted/nosql_database.csv”, “wb”)f.write(key)f.write(“,”)f.write(value)os.fsync(f.fileno())f.close()DiskcontrolleruserbufferpagecacheplatterAlso note, fsync() has a cousin, fdatasync() that does not sync metadata.

19. Aside: point and laugh at MongoDBMongo’s “fsync” command:> db.runCommand({fsync:1,async:true}); wat.Also supports “journaling”, like a WAL in the SQL world, however…It only fsyncs() the journal every 100ms…”for performance”.

20. It’s not enabled by default.Fsync: bitter liesf = open(“/home/ted/nosql_database.csv”, “wb”)f.write(key)f.write(“,”)f.write(value)os.fsync(f.fileno())f.close()DiskcontrolleruserbufferpagecacheplatterDrives will lie to you.

21. Fsync: bitter liesplatter…it’s a cache!DiskcontrollerpagecacheTwo types of caches: writethrough and writeback

22. Writeback is the demon(Just dropped in) to see what condition your caches are inA Typical WorkstationplatterNo controller cacheWriteback cache on diskDiskcontroller

23. (Just dropped in) to see what condition your caches are inA Good ServerplatterWritethrough cacheon controllerWritethrough cache on diskDiskcontroller

24. (Just dropped in) to see what condition your caches are inAn Even Better ServerplatterBattery-backedwritebackcache on controllerWritethrough cache on diskDiskcontroller

25. (Just dropped in) to see what condition your caches are inThe Demon SetupplatterBattery-backed writebackcache orWritethrough cacheWriteback cache on diskDiskcontroller

26. Disks in a virtual environmentThe Trail of Tears to the PlatterHostpagecacheVirtualcontrolleruserbufferpagecachePhysicalcontrollerHypervisorplatter

27. Disks in a virtual environmentWhy EC2 I/O is Slow and UnpredictableShared HardwarePhysical Disk

28. Ethernet Controllers

29. Southbridge

30. How are the caches configured?

31. How big are the caches?

32. How many controllers?

33. How many disks?

34. RAID?Image Credit: ArsTechnica

35. Aside: Amazon EBSMySQLAmazon EBSPlease stop doing this.

36. What’s Killing That Box?ted@u235:~$ iostat -xLinux 2.6.32-24-generic (u235) 07/25/2011 _x86_64_ (8 CPU)avg-cpu: %user %nice %system %iowait %steal %idle 0.15 0.14 0.05 0.00 0.00 99.66Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz%utilsda 0.00 3.27 0.01 2.38 0.58 45.23 19.21 0.24

37. Cool Hardware TricksBeginner Hardware Trick: SSD Drives$2.50/GB vs 7.5c/GB

38. Negligible seek time vs 10ms seek time

39. Not a lot of spaceCool Hardware TricksIntermediate Hardware Trick: RAID ControllersStandard RAID Controller

40. SSD as writeback cache

41. Battery-backed

42. Adaptec “MaxIQ”

43. $1,200Image Credit: Tom’s Hardware

44. Cool Hardware TricksAdvanced Hardware Trick: FusionIOSSD Storage on the Northbridge (PCIe)

45. 6.0 GB/sec throughput. Gigabytes.

46. 30 microsecond latency (30k ns)

47. Roughly $20/GB

48. Top-line card > $100,000 for around 5TBQuestionsQuestions & HecklingThank Youhttp://teddziuba.com/@dozba