WebHDFS at King - May 2014 Hadoop MeetUp

1,034 views

Published on

The latest developments at King on their work with WebHDFS .

Published in: Technology, Sports
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,034
On SlideShare
0
From Embeds
0
Number of Embeds
14
Actions
Shares
0
Downloads
7
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

WebHDFS at King - May 2014 Hadoop MeetUp

  1. 1. 2 How to turbo charge your data transfers with WebHDFS Andy Done, Data Platform Lead andy.done@king.com
  2. 2. Last time…
  3. 3. Since then…
  4. 4. 100 40 Hadoop
  5. 5. 1 0.5 Storage
  6. 6. 15 10 Events
  7. 7. 10 4 ExaSol
  8. 8. 2.5 6 Load times
  9. 9. Problem WebHDFS 12
  10. 10. Old way WebHDFS
  11. 11. Old way hadoop fs –cat /some/path/* | bulk_load my_table WebHDFS
  12. 12. WebHDFS way WebHDFS
  13. 13. WebHDFS way IMPORT INTO TABLE my_table FROM FILE ‘http://namenode/webhdfs/v1/some/path/file_1’ FILE ‘http://namenode/webhdfs/v1/some/path/file_2’ … FILE ‘http://namenode/webhdfs/v1/some/path/file_n’ WebHDFS
  14. 14. WebHDFS benefits •  Simple •  Efficient •  Ubiquitous •  Parallelisable •  Bidirectional •  Fast WebHDFS
  15. 15. 18 Conclusion WebHDFS
  16. 16. Thank you 19
  17. 17. We're hiring! 20

×