WebHDFS allows for more efficient data transfers compared to traditional Hadoop methods. It provides a simple, efficient, ubiquitous, parallelizable, bidirectional, and fast way to load data from HDFS into databases and applications versus older methods that required moving data locally first. WebHDFS provides a REST API to access HDFS data via HTTP without needing to move files locally, improving load times.

2. 2
How to turbo charge
your data transfers with
WebHDFS
Andy Done, Data Platform Lead
andy.done@king.com

4. Last time…

5. Since then…

6. 100
40
Hadoop

7. 1
0.5
Storage

8. 15
10
Events

9. 10
4
ExaSol

11. 2.5
6
Load times

12. Problem
WebHDFS
12

13. Old way
WebHDFS

14. Old way
hadoop fs –cat /some/path/* | bulk_load my_table
WebHDFS

15. WebHDFS way
WebHDFS

16. WebHDFS way
IMPORT INTO TABLE my_table FROM
FILE ‘http://namenode/webhdfs/v1/some/path/file_1’
FILE ‘http://namenode/webhdfs/v1/some/path/file_2’
…
FILE ‘http://namenode/webhdfs/v1/some/path/file_n’
WebHDFS

17. WebHDFS benefits
• Simple
• Efficient
• Ubiquitous
• Parallelisable
• Bidirectional
• Fast
WebHDFS

18. 18
Conclusion
WebHDFS

19. Thank you
19

20. We're hiring!
20