How to build system storage bigfile

1. How to transfer
a BigFile
Presenter: Nguy n Văn Ti nễ ế
Barcamp, RMIT, 2013
tipuder@gmail.com

2. Architecture

3. Upload

4. Download

5. Applications

6. Architecture

7. SBFSession
SBFProfiles
SBFMeta
SBFUserItems
IDZen
Frontend
SBFUpload
FilePool SBFWorker
SBFChunks
SBFZenTok
SBFDownload
Java, C++
LibEvent, Poco,
KyotoCabinet

8. Upload

9. SBFSession
SBFProfiles
SBFMeta
SBFUserItems
IDZen
Frontend
SBFUpload
FilePool SBFWorker
SBFChunks
1: reqPageUp
SBFZenTok
2b:genTok 3:tok2a:checkUser
SBFDownload
4:pageUp4:up(tok)
5a:check(tok)
6a:genId()
6b:writeFile(id)
7: putMeta
9:notify
8: putUsersFile
10: readFile
12: putChunks
11: genIds
6c: fileId

10. Download

11. SBFSession
SBFProfiles
SBFMeta
SBFUserItems
IDZen
Frontend
SBFUpload
FilePool SBFWorker
SBFChunks
1: reqPageDown
SBFZenTok
SBFDownload
4:pageDown
2a:checkUser2b:genTok 3:tok
4:down(tok)
5:getMeta
6:getChunks
loop
sendChunks

12. One more thing

13. SBFSession
SBFProfiles
SBFMeta
SBFUserItems
IDZen
Frontend
SBFUpload
FilePool SBFWorker
SBFChunks
SBFZenTok
SBFDownload
SBFChunks
SBFDownload
Node 1
SBFUpload
FilePool SBFWorker
SBFDownload
SBFChunks
Node 2
SBFNodeInfo
SBFBroker

14. Applications

15. Issues
Disk Bottleneck
Chunk Size, Sector Size, Block Size
Speed process file slow when concurrent upload high

16. Bottleneck disk