My presentation for attaining my Master's Degree in Computing Science. Thesis links: http://irs.ub.rug.nl/dbi/522047fa91559 http://www.cs.rug.nl/~aiellom/tesi/vdtil.pdf

1. CloudStorageforstoringsensordata
Jos van der Til
Byzantinefaulttolerant

2. WHATWASTHATFIRSTSLIDE?

3. IKNOWSOMEOFTHESEWORDS!
SENSORDATA

4. IKNOWSOMEOFTHESEWORDS!
SENSORDATAVARIESBYMEASUREMENTINTERVAL

5. IKNOWSOMEOFTHESEWORDS!
SENSORDATAVARIESBYMEASUREMENTINTERVAL
VARIESBYDIMENSIONALITY

6. IKNOWSOMEOFTHESEWORDS!
SENSORDATAVARIESBYMEASUREMENTINTERVAL
VARIESBYDIMENSIONALITY
VARIESBYSIZE

7. IKNOWSOMEOFTHESEWORDS!
SENSORDATAVARIESBYMEASUREMENTINTERVAL
VARIESBYDIMENSIONALITY
VARIESBYSIZE
IMPORTANT:IMAGESANDVIDEOAREALSOSENSORDATA!

8. IKNOWSOMEOFTHESEWORDS!
CLOUDSTORAGE
SENSORDATA

9. IKNOWSOMEOFTHESEWORDS!
CLOUDSTORAGE
SENSORDATA

10. IKNOWSOMEOFTHESEWORDS!
CLOUDSTORAGE
SENSORDATAUNLIMITEDSTORAGE

11. IKNOWSOMEOFTHESEWORDS!
CLOUDSTORAGE
SENSORDATA
ACCESSIBLEFROMANYWHERE
UNLIMITEDSTORAGE

12. IKNOWSOMEOFTHESEWORDS!
CLOUDSTORAGE
SENSORDATA
ACCESSIBLEFROMANYWHERE
ACCESSIBLEANYTIME
UNLIMITEDSTORAGE

13. IKNOWSOMEOFTHESEWORDS!
CLOUDSTORAGE
SENSORDATA
ACCESSIBLEFROMANYWHERE
ACCESSIBLEANYTIME
UNLIMITEDSTORAGE
PAYFORWHATYOUUSE!

14. IKNOWSOMEOFTHESEWORDS!
CLOUDSTORAGE
FAULTTOLERANT
SENSORDATA

15. IKNOWSOMEOFTHESEWORDS!
CLOUDSTORAGE
FAULTTOLERANT
SENSORDATAPROCESSESONLYCRASH

16. IKNOWSOMEOFTHESEWORDS!
CLOUDSTORAGE
FAULTTOLERANT
SENSORDATAPROCESSESONLYCRASH…RIGHT?

17. IKNOWSOMEOFTHESEWORDS!
CLOUDSTORAGE
FAULTTOLERANT
HOWBADCANITGET? SENSORDATA

18. IKNOWSOMEOFTHESEWORDS!
CLOUDSTORAGE
FAULTTOLERANT
BYZANTINE
SENSORDATA

19. HOWDOPROCESSESFAIL?

20. HOWDOPROCESSESFAIL?

21. Fail stop Crash
HOWDOPROCESSESFAIL?

22. Fail stop Crash
Send Omission
Receive
Omission
General
Omission
HOWDOPROCESSESFAIL?

23. Fail stop Crash
Send Omission
Receive
Omission
General
Omission
Arbitrary failures
with message
authentication
HOWDOPROCESSESFAIL?

24. HOWDOPROCESSESFAIL?
Fail stop Crash
Send Omission
Receive
Omission
General
Omission
Arbitrary failures
with message
authentication
Arbitrary
(Byzantine)
failures

25. Storage clouds
Sensor Network
Measurements
Sensor server
StorageLib
Measurements
Storage Lib
Processing server
Writer
Reader

26. HOWDOPROCESSESFAIL?

27. READERSAREPROCESSES
HOWDOPROCESSESFAIL?

28. READERSAREPROCESSES
HOWDOPROCESSESFAIL?
WRITERSAREPROCESSES

29. READERSAREPROCESSES
HOWDOPROCESSESFAIL?
WRITERSAREPROCESSES
buttheyarecool.

30. READERSAREPROCESSES
HOWDOPROCESSESFAIL?
WRITERSAREPROCESSES
buttheyarecool.
canfailwithoutcausingdamage.
areonlyexpectedtofailbycrashing.

31. HOWDOPROCESSESFAIL?
CLOUDPROVIDERSAREPROCESSES

32. HOWDOPROCESSESFAIL?
CLOUDPROVIDERSAREPROCESSES
buttheyareNOTcool.

33. HOWDOPROCESSESFAIL?
CLOUDPROVIDERSAREPROCESSES
buttheyareNOTcool.
canleakyourdata

34. HOWDOPROCESSESFAIL?
CLOUDPROVIDERSAREPROCESSES
buttheyareNOTcool.
cancorruptyourdata

35. HOWDOPROCESSESFAIL?
CLOUDPROVIDERSAREPROCESSES
buttheyareNOTcool.
candeleteyourdata

36. HOWDOPROCESSESFAIL?
CLOUDPROVIDERSAREPROCESSES
buttheyareNOTcool.
canstoprespondingtoyourrequests

37. HOWDOPROCESSESFAIL?
CLOUDPROVIDERSAREPROCESSES
buttheyareNOTcool.
HASFULLCONTROLOVERYOURDATABUTBEHAVESBYZANTINE

38. YOURDATAISSTORED
INAPROCESSTHATCANFAIL
BYZANTINE

39. HOWTOACHIEVE
BYZANTINEFAULTTOLERANCE?

40. DONOTTRUSTASINGLECLOUD!

41. DOTRUSTMULTIPLECLOUDS!

42. UPLOADDATATOALLTHECLOUDS!

43. HOWMANYCLOUDSDOWENEED?
𝑛 ≥ 3𝑓 + 1

44. HOWISDATASTORED?

45. DATAISSTOREDINAQUORUMOFCLOUDPROVIDERS
QUORUMOF2f+1PROVIDERSTOBEEXACT

46. DATAISSTOREDINAQUORUMOFCLOUDPROVIDERS
QUORUMOF2f+1PROVIDERSTOBEEXACT
DATACANNOTBERETRIEVEDBYLESSTHANf+1PROVIDERS

47. DATAISSTOREDINAQUORUMOFCLOUDPROVIDERS
QUORUMOF2f+1PROVIDERSTOBEEXACT
DATACANNOTBERETRIEVEDBYLESSTHANf+1PROVIDERS
ENCRYPTION
SECRETSHARING

48. DATAISSTOREDINAQUORUMOFCLOUDPROVIDERS
QUORUMOF2f+1PROVIDERSTOBEEXACT
DATACANNOTBERETRIEVEDBYLESSTHANf+1PROVIDERS
SHOULDNOTREQUIREnTIMESTHESTORAGESPACE

49. DATAISSTOREDINAQUORUMOFCLOUDPROVIDERS
QUORUMOF2f+1PROVIDERSTOBEEXACT
DATACANNOTBERETRIEVEDBYLESSTHANf+1PROVIDERS
SHOULDNOTREQUIREnTIMESTHESTORAGESPACE
ERASURECODING

50. DATAISSTOREDINAQUORUMOFCLOUDPROVIDERS
QUORUMOF2f+1PROVIDERSTOBEEXACT
DATACANNOTBERETRIEVEDBYLESSTHANf+1PROVIDERS
SHOULDNOTREQUIREnTIMESTHESTORAGESPACE
ERASURECODING
𝑛
𝑓 + 1
−1
=
𝑓 + 1
𝑛
lim
𝑓→∞
𝑓 + 1
𝑛
=
𝑓 + 1
3𝑓 + 1
=
1
3
SPACEEFFICIENCY
MAXIMUMSPACEEFFICIENCY
lim
𝑓→∞
1
𝑛
= 0NOTBADCOMPAREDTO

51. TRADITIONALAPPROACH
DATAISABLOCK!

52. TRADITIONALAPPROACH
DATAISABLOCK!
IWANTTOREADTHISBLOCK

53. TRADITIONALAPPROACH
DATAISABLOCK!
IWANTTOENCRYPTTHISBLOCK

54. TRADITIONALAPPROACH
DATAISABLOCK!
IWANTTOHASHTHISBLOCK

55. TRADITIONALAPPROACH
DATAISABLOCK!
IWANTTOUPLOADTHISBLOCK

56. TRADITIONALAPPROACH
DATAISABLOCK!
IWANTTODOWNLOADTHISBLOCK

57. TRADITIONALAPPROACH
DATAISABLOCK!
BLOCKDOESNOTFITINMEMORY

58. TRADITIONALAPPROACH
DATAISABLOCK!
BLOCKDOESNOTFITINMEMORY
:(

59. NOWWHAT?

60. MYAPPROACH
DATAISASTREAM!

61. MYAPPROACH
DATAISASTREAM…ofblocks!

62. MYAPPROACH
DATAISASTREAM…ofblocks!
Everyblockshouldfitintomemory

63. MYAPPROACH
DATAISASTREAM…ofblocks!
Everyblockshouldfitintomemory
Everyblockisprocessedindependent

64. MYAPPROACH
DATAISASTREAM…ofblocks!
Everyblockshouldfitintomemory
Everyblockisprocessedindependent
Everyblockhasachecksum(thinkBitTorrent)

65. MYAPPROACH
HASMOREOVERHEADINMEMORYANDSTORAGE

66. MYAPPROACH
HASMOREOVERHEADINMEMORYANDSTORAGE

67. MYAPPROACH
HASMOREOVERHEADINMEMORYANDSTORAGE
REQUIRESLESSMEMORYFORPROCESSING

68. MYAPPROACH
HASMOREOVERHEADINMEMORYANDSTORAGE
REQUIRESLESSMEMORYFORPROCESSING
CANFAILFASTER,SAVINGBANDWIDTH

69. WHENISTHISUSED?

70. WHENISTHISUSED?
DATAISPETABYTESCALE

71. DATAISPETABYTESCALE
…ORATLEASTACOUPLEOF100TERABYTESAYEAR
WHENISTHISUSED?

72. DATAISPETABYTESCALE
…ORATLEASTACOUPLEOF100TERABYTESAYEAR
DATAISCONTINOUSLYADDED
WHENISTHISUSED?

73. DATAISPETABYTESCALE
…ORATLEASTACOUPLEOF100TERABYTESAYEAR
DATAISCONTINOUSLYADDED
TIMEBETWEENBATCHESISTOOLONG
WHENISTHISUSED?

74. DATAISPETABYTESCALE
…ORATLEASTACOUPLEOF100TERABYTESAYEAR
DATAISCONTINOUSLYADDED
TIMEBETWEENBATCHESISTOOLONG
KEEPINGTHOUSANDSOFMACHINESRUNNINGISEXPENSIVE
WHENISTHISUSED?

75. DATAISPETABYTESCALE
…ORATLEASTACOUPLEOF100TERABYTESAYEAR
DATAISCONTINOUSLYADDED
TIMEBETWEENBATCHESISTOOLONG
KEEPINGTHOUSANDSOFMACHINESRUNNINGISEXPENSIVE
WHATIFMYHADOOPCLUSTERISDESTROYED?
WHENISTHISUSED?

76. New Data
All data
Streaming cluster
Batch cluster Batch view
Realtime view
Client
Query
Query

77. WHYNOTHADOOP?

78. WHYNOTHADOOP?
HADOOPSTILLHASITSPLACE

79. WHYNOTHADOOP?
HADOOPSTILLHASITSPLACE
JUSTNOTFORSTORAGE

80. New Data
All data
Streaming cluster
Batch cluster Batch view
Realtime view
Client
Query
Query

81. OK…WHYNOTHADOOPSTORAGE?

82. OK…WHYNOTHADOOPSTORAGE?
BATCHLAYERISANEXCELLENTPLACEFORHADOOPPROCESSING

83. OK…WHYNOTHADOOPSTORAGE?
BATCHLAYERISANEXCELLENTPLACEFORHADOOPPROCESSING
BUTHADOOPSTORAGEISEXPENSIVE!

84. OK…WHYNOTHADOOPSTORAGE?
BATCHLAYERISANEXCELLENTPLACEFORHADOOPPROCESSING
BUTHADOOPSTORAGEREQUIRESMAINTENANCE!

85. OK…WHYNOTHADOOPSTORAGE?
BATCHLAYERISANEXCELLENTPLACEFORHADOOPPROCESSING
BUTHADOOPSTORAGEISONLINEEVENWHENIDLE!

86. OK…WHYNOTHADOOPSTORAGE?
BATCHLAYERISANEXCELLENTPLACEFORHADOOPPROCESSING
BUTHADOOPSTORAGECONSUMESLOTSOFENERGY!

87. DOESTHISWORK?

88. PERFORMANCE
Requestsdoneby16threadsconcurrently

89. PERFORMANCE
Requestsdoneby16threadsconcurrently
8corevirtualmachine
atleast4GBRAM(butoften>16GB)

90. PERFORMANCE
Requestsdoneby16threadsconcurrently
8corevirtualmachine
Atleast4GBRAM(butoften>16GB)
f=1,thusn=4

91. PERFORMANCE
Requestsdoneby16threadsconcurrently
8corevirtualmachine
Atleast4GBRAM(butoften>16GB)
f=1,thusn=4
Twoimplementations:
StreamingDepSky-A
StreamingDepSky-CA

92. PERFORMANCE
Throughput downstream(perthread):
Filesize4MB,750KB/second(90th percentile)
Filesize8MB,1MB/second(90th percentile)
Throughput upstream(perthread):
Filesize4MB,1.2MB/second(90thpercentile)
Filesize8MB,1.7MB/second(90thpercentile)

93. 0.9997
0.9998
0.9999
1.0000
0 5 10 15 20 25
log2(Filesize (b))
Successrate HTTP Verb
GET
PUT
DELETE
LIST
Streaming DepSky-A
0.994
0.996
0.998
1.000
0 5 10 15 20 25
log2(Filesize (b))
Successrate
HTTP Verb
GET
PUT
DELETE
LIST
Streaming DepSky-CA
AVAILABILITY

94. thanks!

95. Thesisavailableat:
http://www.cs.rug.nl/~aiellom/tesi/vdtil.pdf