Structured Data Archiving: Key Issues 9th July 2009 – CloudCamp, London Mark Cusack Principal Architect RainStor

Ideal Use Case: Application Retirement Dev & Test Retirement Mission Critical Production System - On-premise Implementation End-of-Life Life Cycle Usage Cloud Bursting e.g reporting & peak processing

Data Volume Issues How to transfer in terabytes of data? Structured data can be massively compressed before uploading to a storage cloud, e.g. Amazon S3: Network Connection Upload time (1 TB) Compressed (40:1) upload time DSL 166 days 4 days 10 Mbps 13 days 8 hours 1 Gbps Less than a day Less than an hour

How to transfer in terabytes of data?

Structured data can be massively compressed before uploading to a storage cloud, e.g. Amazon S3:

Data Security Issues How to maintain data privacy and integrity? Theoretical solution: homomorphic encryption!? Reality: encrypt network pathways and data rest-points Blind cloud storage Key should be generated by the application owner Data should be encrypted on-premise, prior to transfer to the cloud Tamper-proofing and auditing Keep digests of database files off-cloud Practical in the case of application retirement (read-only)

How to maintain data privacy and integrity?

Theoretical solution: homomorphic encryption!?

Reality: encrypt network pathways and data rest-points

Blind cloud storage

Key should be generated by the application owner

Data should be encrypted on-premise, prior to transfer to the cloud

Tamper-proofing and auditing

Keep digests of database files off-cloud

Practical in the case of application retirement (read-only)

Data Availability Issues How to ensure that the data is always available? Make multiple copies of the data Data compression makes this cost-effective Employ emerging cloud interoperability standards De facto standards: Eucalyptus, Amazon Web Services SNIA eXtensible Access Method (XAM): as an alternative cloud storage interoperability standard

How to ensure that the data is always available?

Make multiple copies of the data

Data compression makes this cost-effective

Employ emerging cloud interoperability standards

De facto standards: Eucalyptus, Amazon Web Services

SNIA eXtensible Access Method (XAM): as an alternative cloud storage interoperability standard

Data Query Issues How to query the data without compromising performance, security and accessibility? Use a compute cloud to query the data E.g. run retired application reports on EC2 against S3 data Query directly against compressed data Problem becomes CPU-bound rather than IO-bound Pass encryption key to the cloud on a per-query basis Provide an ODBC/JDBC interface to compute instance

How to query the data without compromising performance, security and accessibility?

Use a compute cloud to query the data

E.g. run retired application reports on EC2 against S3 data

Query directly against compressed data

Problem becomes CPU-bound rather than IO-bound

Pass encryption key to the cloud on a per-query basis

Provide an ODBC/JDBC interface to compute instance

More Information and Contact Details www.rainstor.com twitter.com/rainstor twitter.com/markcusack [email_address]

www.rainstor.com

twitter.com/rainstor

twitter.com/markcusack

[email_address]