The document discusses scaling HDFS to manage billions of files. It describes how HDFS usage has grown from millions of files in 2007 to potentially billions of files in the future. To address this, the speakers propose storing HDFS metadata in a key-value store like LevelDB instead of solely in memory. They evaluate this approach and find comparable performance to HDFS for most operations. Future work includes improving operations like compaction and failure recovery in the new architecture.

1. Scaling HDFS to Manage Billions of Files
Haohui Mai, Jing Zhao
Hortonworks, Inc.

2. About the speakers
• Haohui Mai
• Active committers and PMC in Hadoop
• Ph.D. in Computer Science from UIUC in 2013
• Joined the HDFS team in Hortonworks
• 250+ commits in Hadoop

3. About the speakers
• Jing Zhao
• Active committers and PMC in Hadoop
• Ph.D. in Computer Science from USC in 2012
• HDFS team member in Hortonworks
• 250+ commits in Hadoop

4. Past: the scale of data

5. Past: the scale of data
• In 2007 (PC)
• ~500 GB hard drives
• thousands of files

6. Past: the scale of data
• In 2007 (PC)
• ~500 GB hard drives
• thousands of files
• In 2007 (Hadoop)
• several hundred nodes
• several hundred TBs
• millions of files

7. Past: the scale of data
• In 2007 (Hadoop)
• several hundred nodes
• several hundred TBs
• millions of files

8. Past: the scale of data
• In 2007 (Hadoop)
• several hundred nodes
• several hundred TBs
• millions of files
• In 2015
• 4,000+ nodes (10x)
• 150+ PBs (1000x)
• 400M+ files (100x)

9. Present: a generic storage system

10. Present: a generic storage system
• SQL-On-Hadoop
• Machine learning
• Real-time analytics
• Data streaming
• File archives, NFS…

11. Present: a generic storage system
• SQL-On-Hadoop
• Machine learning
• Real-time analytics
• Data streaming
• File archives, NFS…
• From MR-centric filesystem to a
generic distributed storage system

12. Future: Billions of files in HDFS

13. Future: Billions of files in HDFS
• HDFS clusters continue to grow

14. Future: Billions of files in HDFS
• HDFS clusters continue to grow
• New use cases emerge
• IoT, time series data…

15. Future: Billions of files in HDFS
• HDFS clusters continue to grow
• New use cases emerge
• IoT, time series data…
• Files are natural abstractions of
data
• Few big files → many small
files in HDFS
• Billions of files in a few years

16. NameNode limits the scale

17. NameNode limits the scale
• Master / slave architecture
• All metadata in NN, data
across multiple DNs
• Simple and robust
NN
DN DN DN

18. NameNode limits the scale
• Master / slave architecture
• All metadata in NN, data
across multiple DNs
• Simple and robust
• Does not scale beyond the size
of the NN heap
• 400M files ~ 128G heap
• GC pauses
NN
DN DN DN

19. Next-gen arch: HDFS on top of KV stores

20. Next-gen arch: HDFS on top of KV stores
• Namespace (NS) on top of Key-
Value (KV) stores
• Storing the NS into LevelDB

21. Next-gen arch: HDFS on top of KV stores
• Namespace (NS) on top of Key-
Value (KV) stores
• Storing the NS into LevelDB
• Working set fits in memory,
cold metadata on disks
• Match the usage patterns of
HDFS
Namespace

22. Next-gen arch: HDFS on top of KV stores
• Namespace (NS) on top of Key-
Value (KV) stores
• Storing the NS into LevelDB
• Working set fits in memory,
cold metadata on disks
• Match the usage patterns of
HDFS
• Low adoption cost: fully
compatible
Namespace

23. • Introduction
• Namespace on top of KV stores
• Evaluation
• Future work & conclusions

24. Encoding the NS as KV pairs
• inode_id → flat binary
representation of inode
• avoid serialization costs
• <pid,foo> → the inode id of
the child foo whose parent’s
inode id is pid
struct inode {
uint64_t inode_id;
uint64_t parent_id;
uint32_t flags;
…
};

25. Encoding directories
root
foo
bar

26. Encoding directories
Key Value
1 root
2 foo
3 bar
<1,foo> 2
<2,bar> 3
root
foo
bar

27. Encoding directories
Key Value
1 root
2 foo
3 bar
<1,foo> 2
<2,bar> 3
root
foo
bar

28. Encoding directories
Key Value
1 root
2 foo
3 bar
<1,foo> 2
<2,bar> 3
root
foo
bar

29. Resolving paths
Key Value
1 root
2 foo
3 bar
<1,foo> 2
<2,bar> 3

30. Resolving paths
Key Value
1 root
2 foo
3 bar
<1,foo> 2
<2,bar> 3
/ foo / bar

31. Resolving paths
Key Value
1 root
2 foo
3 bar
<1,foo> 2
<2,bar> 3
/ foo / barID: 1

32. Resolving paths
Key Value
1 root
2 foo
3 bar
<1,foo> 2
<2,bar> 3
/ foo / barID: 1
<1,foo>

33. Resolving paths
Key Value
1 root
2 foo
3 bar
<1,foo> 2
<2,bar> 3
/ foo / bar
<1,foo>
ID: 2

34. Resolving paths
Key Value
1 root
2 foo
3 bar
<1,foo> 2
<2,bar> 3
/ foo / bar
ID: 2
<2,bar>

35. Resolving paths
Key Value
1 root
2 foo
3 bar
<1,foo> 2
<2,bar> 3
/ foo / bar
<2,bar>
ID: 3

36. Listing directories
Key Value
1 root
2 foo
3 bar
<1,foo> 2
<2,bar> 3

37. Listing directories
Key Value
1 root
2 foo
3 bar
<1,foo> 2
<2,bar> 3
$ ls /

38. Listing directories
Key Value
1 root
2 foo
3 bar
<1,foo> 2
<2,bar> 3
$ ls /
ID: 1

39. Listing directories
Key Value
1 root
2 foo
3 bar
<1,foo> 2
<2,bar> 3
$ ls /
ID: 1
<1,*>

40. Listing directories
Key Value
1 root
2 foo
3 bar
<1,foo> 2
<2,bar> 3
$ ls /
foo
ID: 1
<1,*>

41. Integrate with existing HDFS features

42. Integrate with existing HDFS features
• HDFS snapshots
• Metadata only operations
• Append version ids for each key
• Map between snapshot ids and version ids

43. Integrate with existing HDFS features
• HDFS snapshots
• Metadata only operations
• Append version ids for each key
• Map between snapshot ids and version ids
• NameNode High Availability (HA)
• Use edit logs instead of the WAL of the KV stores to persist operations
• Minimal changes in the current HA mechanisms

44. Current status
• Phase I — NS on top of KV interfaces (HDFS-8286)
• NS on top of an in-memory KV store
• Under active development
• Phase II — Partial NS in the memory
• Working set of the NS in the memory, cold metadata on disks
• Scaling the NS beyond the size of heap

45. • Introduction
• Namespace on top of KV stores
• Evaluation
• Future work & conclusions

46. Evaluation
• 4 node clusters, connected with 10GbE networks
• 6-core Intel Xeon E5-2630 @ 2.3GHz * 2, 64G DDR3 @ 1333 MHz
• WDC 1TB disks @ 7200 RPM, 64MB cache
• OpenJDK 1.8.0_25

47. Evaluation (cont.)
Apache Hadoop
2.7.0
2.7.0 InMem LevelDB
NS on top of an in-
memory KV map
NS on top of
LevelDB

48. NNThroughput (write)
• 300,000 operations
• 8 threads
• Comparable performance
• Simpler implementation
on delete()
• Syncing edit logs is the
bottleneck
Throughput(ops/s) 0
125
250
375
500
create mkdirs delete rename
2.7.0 InMem LevelDB

49. NNThroughput (read)
• Read-only operations
Throughput(ops/s)
0
50000
100000
150000
200000
open fileStatus
2.7.0 InMem
LevelDB LevelDB-opt

50. NNThroughput (read)
• Read-only operations
• 1/3 throughput of vanilla
LevelDB v.s. 2.7.0
• Contentions of the global lock
in LevelDB during get()
Throughput(ops/s)
0
50000
100000
150000
200000
open fileStatus
2.7.0 InMem
LevelDB LevelDB-opt

51. NNThroughput (read)
• Read-only operations
• 1/3 throughput of vanilla
LevelDB v.s. 2.7.0
• Contentions of the global lock
in LevelDB during get()
• A lock-free fast path of get() to
recover the performance
(LevelDB-opt)
Throughput(ops/s)
0
50000
100000
150000
200000
open fileStatus
2.7.0 InMem
LevelDB LevelDB-opt

52. YCSB: Throughput
• YCSB against HBase 1.0.1.1
• Enabled short-circuit reads
• 100 threads, 10M records
Throughput(ops/s)
0
30000
60000
90000
120000
A B C F D E
2.7.0 InMem LevelDB

53. Latency(us)
0
2500
5000
7500
10000
A-Read
B-Read
C
-Read
F-RM
W
F-Read
E-Scan
2.7.0 InMem LevelDB
YCSB: Latency

54. Runtime(s)
0
350
700
1050
1400
TeraG
en
TeraSort
TeraValidate
2.7.0 InMem LevelDB
TeraSort
• 10G data per node
• Comparable performance

55. Throughput(op/s)
0
32500
65000
97500
130000
Working set
256M
128M
64M
32M
Impact on working set size
• Throughput of GetFileStatus()
under different sizes of working
sets

56. • Introduction
• Namespace on top of KV stores
• Evaluation
• Future work & conclusions

57. Future work
• Implementation and stabilization
• Operation concerns
• Compaction and fsck
• Failure recovery
• Cold startup

58. Conclusions
• HDFS needs to continue to scale

59. Conclusions
• HDFS needs to continue to scale
• Evolve HDFS towards KV-based architecture

60. Conclusions
• HDFS needs to continue to scale
• Evolve HDFS towards KV-based architecture
• Scaling beyond the size of the NN heap

61. Conclusions
• HDFS needs to continue to scale
• Evolve HDFS towards KV-based architecture
• Scaling beyond the size of the NN heap
• Preliminary evaluation looks promising

62. Acknowledgement
• Xiao Lin, interned with Hortonworks in 2013
• PoC implementation of LevelDB backed namespace
• Zhilei Xu, interned with Hortonworks in 2014
• Integration between various HDFS features and LevelDB
• Performance tuning

63. Thank you!

65. Integrating with LevelDB
Write operations in HDFS
• Updating LevelDB inside the
global lock
• New LevelDB::Write() w/o
blocking calls
• Write edit log
• logSync()

66. Integrating with LevelDB
Write operations in HDFS
• Updating LevelDB inside the
global lock
• New LevelDB::Write() w/o
blocking calls
• Write edit log
• logSync()
Pruning edit logs
• Dump memtable into the disks
• Update MANIFEST
• Prune edit logs