Understanding Many-core Scalability of File Systems Changwoo Min et al. Published in USENIX ATC’18

1. Understanding Many-core Scalability
of File Systems
1
Changwoo Min et al.
Published in USENIX ATC’18
Van Giang Nguyen, Duc Canh Le

2. Contents
1
• Motivation
2
• Methodology
3
• Evaluation
4
• Conclusion
2

3. 3
Motivation

4. Motivation 1: Multicore machine’s era
▪ Multicore machine is popular today
▪ Multicore massively increases the computational capacity of
machine
▪ How about I/O system (ex. Storage)?
▪ In general, multicore can benefit from parallel I/O operations, but it
also produces many issues (e.g. data consistency)
4-16 cores 16-32 cores 32-64 cores
4

5. Motivation 2: Single CPU is not enough!
▪ Single CPU is not enough to fully utilize storage device at all
** IBM RamSan-70
* Intel, Performance Benchmarking for PCIe* and NVMe* Enterprise SSDs
5
▪ 00’s era, HDD (100-200 IOPS) → yes
▪ Now: SSD (so far 1M IOPS(**)) → no
▪ Disk is not always the bottleneck of system as before!!
Consider CPU run at 2.6GHz
→ ~10 us per I/O operation
→ ~100K I/O operations per sec (IOPS) / CPU
→ Is this enough?
On average(*):
27K CPU cycles per I/O

6. Motivation 3: Design or Implementation?
▪ How to tackle the problem
▪ New interfaces and system stacks, ex. NVMe can reduce to 10K cycles / IO operation
▪ Increase CPU frequency
▪ File system should be scalable
▪ Are existing file systems scalable?
▪ No in-depth study
▪ Need some insights to design scalable file systems
6

7. Research questions
7
▪ What file system operations are not scalable?
▪ Why they are not scalable?
▪ Is it the problem of implementation or design?

8. 8
Methodology

9. Technical challenges
9
▪ Applications are usually stuck with a few bottlenecks
→ cannot see the next level of bottlenecks before
resolving them
→ difficult to understand overall scalability behavior
▪ How to systematically stress file systems to understand scalability
behavior?

10. FXMARK
▪ Micro-benchmark:
▪ General file system operation
▪ Stress each component of FS
▪ Multi-process test
▪ Application benchmark
▪ Real-life applications
▪ I/O intensive
10

11. Work-flow
11

12. Experimental setup
Five most popular file
systems on Linux
Typical storage
platforms
12

13. 13
Evaluation

14. Expected Result
14

15. Are existing File Systems scalable?
▪ No, in most of benchmarks
15

16. In-depth Analysis
16

17. Evaluation – First scenario
17

18. Evaluation – First scenario
18

19. … only when not being accessed reference counter is working
19
Evaluation – First scenario

20. High contention on a page reference counter → Huge memory stall
20
Evaluation – First scenario

21. 21
Evaluation

22. 22
Evaluation – Second scenario

23. 23
Evaluation – Second scenario

24. 24
Evaluation – Second scenario

25. 25
Evaluation

26. 26
Evaluation – Third scenario

27. 27
Evaluation – Third scenario

28. 28
Evaluation

29. Findings
29
▪ High locality can cause performance collapse
▪ Overwriting could be as expensive as appending
▪ A file cannot be concurrently updated
▪ All directory operations are sequential
▪ Renaming is system-wide sequential
▪ Metadata changes are not scalable
▪ Non-scalability often means wasting CPU cycles
▪ Scalability is not portable See paper for details

30. 30
Evaluation – App Benchmark

31. 31
Evaluation – App Benchmark

32. Conclusion
32
▪ Comprehensive analysis of many-core scalability of five widely-
used file systems using FxMark
▪ Manycore scalability should be of utmost importance in file system
design
▪ New challenges in scalable file system design
▪ Minimizing contention, scalable consistency guarantee, spatial
locality, etc.