Presentation at the International Industry-Academia Workshop on Cloud Reliability and Resilience. 7-8 November 2016, Berlin, Germany. Organized by EIT Digital and Huawei GRC, Germany. Twitter: @CloudRR2016

1. 1
Dependable Storage and Computing
using Multiple Cloud Providers
Alysson Bessani
www.di.fc.ul.pt/~bessani
International Industry-Academia Workshop on
Cloud Reliability and Resilience
Berlin, 7-8 November 2016

2. 3
Cloud Sec. & Dep.
• Secure and dependable services are a
necessary condition for the long-term
existence of a cloud provider
– Clients need to trust providers
– Providers need to justify this trust
• To stay in the market, a provider needs
to invest on Sec. & Dep., however…

3. 4
Unavailability
Number of nines
99,9% = 8,76 hours 99,99% = 53 minutes

4. 5
Unavailability

5. 6
Data Loss
2011
2015
2016

6. 7
Privacy Issues…

7. 8

8. 9
Important!
To the best of my knowledge, there is no work on the
existing technical literature saying that using the cloud
is less secure than having everything on premises
Cloud services work pretty well…
but they are not perfect!

9. 10
Our view
• The best way to use the cloud in a
dependable and secure way is to not
rely on a single provider
– Don’t base your service on a single
provider, i.e., an application works if the
provider where it is running is correct
– Instead, consider distributed trust: an
application is correct if at most f out-of n
services/providers are faulty

10. 12
Cloud-of-clouds Service
• Two fundamental characteristics
1. No modification on existing cloud services
2. No collaboration between providers

11. 13
Cloud-of-Clouds Storage

12. 14
Cloud Storage Diversity
• 24+ cloud object storage services
– Locations: 68
– Model: Standard - Archival
– Cost per GB/month stored: $0,001 - $0,13
– Cost per GB downloaded: $0,01 - $0,25
– Availability SLA: undefined, 99,9%-100%
– Access control: inexistent, URLs, …, ACLs,
– Standards compliance: 0-33+

13. 15
Fundamentals
• Let’s consider the problem of
implementing a cloud-of-clouds object
storage tolerating one cloud failure
• Assume you cannot know for sure if a
unresponsive cloud is faulty or not

14. 16
Writing Data
One needs to write at least two copies of the data!
What if one of the clouds take too
much to acknowledge the write?
Use three clouds, and wait for at least two acks to complete the write

15. 17
Reading Data
How many clouds you need to access for reading?
One cloud: might access an outdated/empty cloud
Two clouds: always access the last complete write
write
quorum
read
quorum

16. 18
Reading Data II
What if data can be modified or corrupted?
write
quorum
read
quorum

17. 19
Reading Data II
To solve this we need more clouds and bigger quorums
write
quorum
read
quorum

18. 20
DepSky: Dependable
Cloud-of-Clouds Object Storage
[EuroSys’11, ACM Trans. on Storage 2013]
Availability, Integrity and Confidentiality
despite the failure of up to f clouds

19. 21
Challenges for implementing
Updatable Objects
• How to implement an efficient
replication protocol using only
passive storage nodes?
• How to make it affordable?
read
write

20. 22
Cloud A
Cloud B
Cloud C
Cloud D
D
D
D
D
qwjda
sjkhd
ahsd
qwjda
sjkhd
ahsd
qwjda
sjkhd
ahsd
qwjda
sjkhd
ahsd
DepSky Write
WRITE
DATA
D
ACK
D
D
D
D
WRITE
METADATA
qwjda
sjkhd
ahsd
ACK
qwjda
sjkhd
ahsd
qwjda
sjkhd
ahsd
qwjda
sjkhd
ahsd
qwjda
sjkhd
ahsd

21. 23
Cloud A
Cloud B
Cloud C
Cloud D
D
D
D
D
DepSky Read
READ
DATA
D
DATA
D
D
D
D
READ
METADATA
qwjda
sjkhd
ahsd
qwjda
sjkhd
ahsd
qwjda
sjkhd
ahsd
qwjda
sjkhd
ahsd
METADATA
qwjda
sjkhd
ahsd
Data will be
fetched from
other clouds
if needed.
highest version number

22. 24
Cloud A Cloud B Cloud C Cloud D
DepSky Confidentiality and
Storage Efficiency
Data Limitations:
1. Data is accessible
by cloud providers
2. Requires n×|Data|
storage space
Data Data DataData

23. 25
Cloud A Cloud B Cloud C Cloud D
DepSky Confidentiality and
Storage Efficiency
S1 S2 S3 S4
share
K
generated
key
Data
disperse
F1 F2 F3 F4
F1 S1 F2 S2 F3 S3 F4 S4
encrypt
Inverse process for reading
from f+1 shares/fragments.

24. 27
Practical Considerations
• Read/write latency ≈ accessing a single cloud
• Storage costs roughly 50% higher
• DepSky is a Java programming library
http://cloud-of-clouds.github.io/depsky
• It does not support concurrent writers to the
same object without using locks
– Recent extensions for multi-writer storage (2016)
http://github.com/cloud-of-clouds/mwmr-registers
• How to build a complete system around
it?

25. 28
Coordination
Service
SCFS: Shared Cloud-backed File System
Storage
cloudsCloud Storage
Cache
Cache
Cache
Lock
Service
Access
Control
Metadata
Computing
clouds
SCFS
Agent
SCFS
Agent
SCFS
Agent
[USENIX ATC 2014]

26. 29
SCFS Consistency on Close
fd = open(“x”,…);
…
read(fd,…);
write(fd,…);
…
fsync(fd);
…
write(fd,…);
…
close(fd);
Coordination
Service
Cloud(-of-Clouds)
Persistent
Storage
Main Memory
READ
WRITE
Local Storage
If “x” is cached, it
is only validated
1
2
1
2

27. 30
• SCFS can use different backends
– i.e., different cloud storage and a coordination service plugin
• Operation: blocking, non-blocking and non-sharing
ire1 million
B of storage
1KB, assum-
usand tuples
d, requiring a
importantly,
substantially
rvice, allow-
pace file sys-
o connect the
l, the SCFS
mented Java
storage back-
Java mainly
dination and
high latency
aJava-based
meansthat datasecurity isensured even if f out-of 3f + 1
of the cloud providers suffer arbitrary faults, which en-
compasses unavailability and data deletion, corruption or
creation [15]. Although cloud providers havetheir means
to ensurethedependability of their services, therecurring
occurrence of outages, security incidents (with internal or
external origins) and datacorruptions [19, 24] justifies the
need for this sort of backend in several scenarios.
SCFS%
Agent%
SCFS%
Agent%
DS%
BFT$SMaRt*
DepSky*
S3(
EC2(
AWS%Backend% CoC%Backend%
S3(
DS% DS%
DS%DS% RS(
WA(
GS(
S3(
Figure5: SCFSwith Amazon Web Services(AWS) and Cloud-
of-Clouds (CoC) backends.
SCFS Backends

28. 32
Sharing Latency: SCFS vs DropBox
Amazon S3
Google
Storage
Rackspace
Files
Windows
Azure Blob
DATA
DATA
?$
#!
%|
DATA
DATA
SCFS
Dropbox

29. 34
Practical Considerations
• SCFS improves the transparency of cloud-of-
clouds(-backed) storage
• It is implemented as a Linux FUSE FS:
http://cloud-of-clouds.github.io/SCFS
• Limitations:
– Does not work well with big files
– Require computing instances to run the
coordination service (e.g., Zookeeper replicas)
• Can we do better?

30. 35
Private
Cloud
Private
Cloud
Private
Cloud
Charon Cloud-backed File System
Charon CharonCharon
+ big data storage and sharing +multiple storage locations
+ serverless design

31. 37
Charon Architecture
CharonCharon TLS ChannelCache
Site 1 Site 2
Cache
CA
Public
Cloud A
Public
Cloud B
D
B
A
B
d1
D
d2
C
metadata

32. 38
Avoiding write-write Conflicts without
External Coordination
f-tolerant Byzantine lease
lease
release
renew
enqueue,
dequeue, peekAll
put, get, del, cas
atomic
transactions
lease
release
renew Queue
lease
release
renew
Queue
lease
release
renew
Dynamo
DB
lease
release
renew Data Store
f+1
clouds

33. 40
Confidentiality & Storage-Efficiency
Cloud A Cloud B Cloud C Cloud D
S1 S2 S3
secret
sharing
K
randomly
generated
key
Compressed Chunk
erasure code
F1 F2 F3
F1 S1 F2 S2 F3 S3
encrypt
Big File
Chunk 1 Chunk N
compress
…
Optional

34. 41
Processing Writes
CH
file
data
C1
metadata
local
write
file
data
C2
CH
metadata
1 42 3 5 6 7 8
on first write on each write
asynchronous
later
Figure3. Fileupdate steps.

35. 43
Upload/Download Latency
0.6 GB/min1 GB/min
related
work
related
work

36. 45
Cloud-of-clouds Computing
(very briefly)

37. 46
Disaster Recovery/Tolerance
• Services in cloud A can have a backup in cloud B
• For less than €1/month it is possible to keep up to
30GB of data in another cloud w/ a RPO of 3 min
DBMS
Failover
Recovers take less
than 40s per GB

38. 47
Permissioned Blockchains
and Smart Contracts
• Permanent decentralized ledger used for recording
transactions; requires a practical BFT consensus
• Ex.: Hyperledger, Symbiont Assembly, Ethereum, …
operation
result
operation
result
operation
result
operation
result
operation
result

39. 48
Geo-replicated Services
• Largely used in (single domain) internet-scale apps
• Wide-area Crash or Byzantine fault tolerant
replication protocols are needed
Service
Replica
Service
Replica
Service
Replica
Service
Replica

40. 49
Final Remarks
• In 2010
– Paxos and Zookeeper were not very popular
– RAFT didn’t exist
– Bitcoin was a crazy new idea
– Blockchain and smart contracts were mostly ignored
• The world is changing fast, and advanced
distributed applications are here to stay...
• What will happen in 2020?

41. 50
Further reading…
Storage
• Bessani, Correia, Quaresma, André, Sousa. DepSky: Dependable and Secure
Storage in the Cloud of Clouds. ACM Transactions on Storage. 2013.
(preliminary version on ACM EuroSys’11).
• Oliveira, Mendes, Bessani. Exploring Key-Value Stores in Multi-Writer
Byzantine-Resilient Register Emulations. OPODIS’16.
• Mendes, Oliveira, Cogo, Neves, Bessani. Charon: A Dependable Cloud-
Backed System for Storing and Sharing Big Data. Under Submission. 2016.
• Bessani, Mendes, Oliveira, Neves, Correia, Pasin, Verissimo. SCFS: A Shared
Cloud-backed File System. USENIX ATC’14.
Computing
• Sousa, Bessani. Separating the WHEAT from the Chaff: An Empirical Design for Geo-
Replicated State Machines. IEEE SRDS’15.
• Bessani, Sousa, Alchieri. State Machine Replication for the Masses with BFT-SMaRt.
IEEE/IFIP DSN’14.