Cross-DC Fault-Tolerant ViewFileSystem @ Twitter

3. ○
○
○
○

4. ○
○
○
○

6. Command Line Interface
hadoop —config /etc/hadoop/revenue-dcA fs -ls /
hadoop —config /etc/hadoop/dw-dcB fs -ls /
……
Directory layout on local file system
/etc/hadoop
revenue-dcA
dw-dcB
core-site.xml, hdfs-site.xml
Log processing
MR
YARN
HDFS
Revenue
MR
YARN
HDFS
Data Warehouse
MR
YARN
HDFS
Log processing
MR
YARN
HDFS
DataCenter A DataCenter B

7. ○
○
○

8. ○
○
○
○
○

9. DW @ dcB
HDFS blocks
Client machine: /etc/hadoop/dw-dcB/hdfs-site.xml, core-site.xml

10. DW @ dcB
HDFS blocks
Client machine:
hadoop -config /etc/hadoop/dw-dcB fs -get /user/bob/fileB
Distributed FileSystem
fs.defaultFS -> hdfs://dw-nn-dcB -> NameNode1, NameNode2
/etc/hadoop/dw-dcB/hdfs-site.xml, core-site.xml

11. DW @ dcB

12. App paths HDFS paths
viewfs://dw-nn-dcB/var hdfs://dw-tmp-dcB/var
viewfs://dw-nn-dcB/user hdfs://dw-user-dcB/user
viewfs://dw-nn-dcB/logs hdfs://dw-log-dcB/logs
mountable.xml
FileSystem
ViewFileSystem (viewfs://)
Distributed FileSystem (hdfs://)
S3 FileSystem

13. Client machine:
hadoop -config /etc/hadoop/dw-dcB fs -get /user/bob/fileB
Distributed FileSystem
dw-user-dcB -> dw-user-NN1-dcB, dw-user-NN2-dcB
ViewFileSystem
/user -> hdfs://dw-user-dcB/user
/etc/hadoop/dw-dcB/hdfs-site.xml, core-site.xml, mountable.xml
DW @ dcB

14. ○
○
Revenue
Revenue
MR
YARN
HDFS
Data Warehouse
Revenue
MR
YARN
HDFS …..
DataCenter A DataCenter B

15. hadoop —config /etc/hadoop/revenue-dcA fs -get /user/bob/fileA

16. hadoop —config /etc/hadoop/revenue-dcA fs -get /user/bob/fileA
hdfs --config /etc/hadoop/dw-dcB fsck -fs hdfs://dw-user-dc2 /

17. hadoop —config /etc/hadoop/revenue-dcA fs -get /user/bob/fileA
hdfs --config /etc/hadoop/dw-dcB fsck -fs hdfs://dw-user-dc2 /
// find all “fileC” files on all clusters
for i in `ls /etc/hadoop`;
do hadoop --config /etc/hadoop/$i fs -ls fileC; done

18. ○

19. ○
Description $sourcePath Result
global namespace viewfs://revenue-nn-dcA/logs/dirB /logs/dirB Source path unresolvable

20. ○
Description $sourcePath Result
global namespace viewfs://revenue-nn-dcA/logs/dirB /logs/dirB Source path unresolvable
Active namenode hdfs://revenue-log-nn1-dcA/logs/dirB /logs/dirB not reliable due to hard
coded namenode

21. ○
Description $sourcePath Result
global namespace viewfs://revenue-nn-dcA/logs/dirB /logs/dirB Source path unresolvable
Active namenode hdfs://revenue-log-nn1-dcA/logs/dirB /logs/dirB not reliable due to hard
coded namenode
hftp & DNS alias hftp://revenue-nn-dcA/logs/dirB hftp reliability and efficiency

22. ○
○
○

26. viewfs://cluster-dc/
user tmp log

27. ClusterA
DC1
root /

30. ○
○

31. scalding> fsShell("-ls /dc*/") // list all clusters
Found N items
...

32. // no need to remember copy(From/To)Local aka get/put
scalding> fsShell("-cp /local/user/cecily/file.txt
/user/cecily/hdfs_file.txt")
res6: Int = 0

38. user user
Forward FileSystem API calls to all target filesystems with
certain policies.

41. c@dc1
FileSystem nfly = FileSystem.get(conf);
out = nfly.create(“/nfly/C/user/cecily/file.txt”);
out.write(....);
out.close() BEGIN
out.close() END
In = nfly.open(“/nfly/C/user/cecily/file.txt”);
c@dc2
/user/cecily/_nfly_tmp_file.txt
/user/cecily/_nfly_tmp_file.txt
close, rename to
/user/cecily/file.txt
close, rename to
/user/cecily/file.txt
client@dc1

42. ○
○
○
○