Adam's slides from his talk at the CloudStack European User group meetup, March 13, London. To provide tighter integration between the S3 compatible object store and CloudStack, Cloudian has developed a connector to allow users and their applications to utilize the object store directly from within the CloudStack platform in a single sign-on manner with self-service provisioning. Additionally, CloudStack templates and snapshots are centrally stored within the object store and managed through the CloudStack service. The object store offers protection of these templates and snapshots across data centres using replication or erasure coding.

1. Cloudian HyperStore
Advanced S3 Compatible Storage for CloudStack
Adam Dagnall, SE Director, Northern EMEA
adagnall@cloudian.com

2. 2
Agenda
• High Level Intro & The Importance of S3
• HyperStore Architecture
• HyperStore Features
• CloudStack Integration

3. High Level Introduction

4. Data fuels the transformation to digital enterprise
More connected people, apps, and things generating more data in many forms
Human data
Sensor data
Business data
faster growth
than traditional
business data
10x
90% of the world’s data was created in the last 2 years.

5. Challenge: Capacity and Complexity
Fundamental challenges are growing
2010 2020
Sensors &
Devices
Social
Media
VoIP
Enterprise
Data
Volume in
Exabytes
2018
Los Angeles Denver New York
CLOUD: Move towards multi-cloud
ON-PREM: More capacity & silos
Increasing Storage ComplexityAccelerating Capacity Growth
© 2018, Cloudian, Inc. Confidential. Do Not Distribute

6. S3: de-facto Object Storage API Standard
Amazon’s cloud is 10 times larger than the next 14 competitors combined.
© 2017, Cloudian, Inc.8
Launched in 2006, AWS dominates the public cloud market

7. © 2017, Cloudian, Inc.9
On-Premises Storage is Still Essential
Why:
• Performance
• Security
• Compliance
• Cost of access
APPLICATIONS
ON PREM
STORAGE
DATA CENTER
Challenge: Multiple Storage Environments to Manage

8. © 2017, Cloudian, Inc.10
Meet Cloudian
• On-premises S3-compatible object storage
• Ideal solution for hybrid cloud
• Start small, scale to 100s of petabytes

9. © 2017, Cloudian, Inc. 11
Cloudian HyperStore:
S3 Object Storage
• Highly scalable
• Scale by adding nodes
• None of the limitations associated with Block and File
• Custom metadata
• Stored in Cassandra
• Can be exported / Indexed
• REST based (HTTP / HTTPS) Data Access
• S3 API
• Management API
• NON RAID
• Configurable Replicas
• Configurable Erasure Coding

10. S3 Functionality
S3 is a modern storage protocol combining
• Connectivity
• Access control
• Data management
• Reporting & Notifications
• Advanced Functionality
• Tiering
• Encryption
• Billing
• Monitoring
• Notifications
• Replication
• Time limited access
…. that can be used across the internet.
Standard
Storage
Protocols
S3 API
12

11. © 2017, Cloudian, Inc.13
S3 Compatibility

12. 14
Why S3 Compatibility Matters
1. Cloud storage standard established at Amazon; world’s largest object
storage environment
2. MSPs worldwide adopting S3 for cloud storage services
3. Hundreds of applications now support S3, many more in development
S3 Compatibility = Investment Protection

13. Unstructured Data Consolidation – additional use cases
Backup
File
NAS Offload
Media Asset Mgmt
Sync and Share
AI / Machine Learning
S3 API

14. ©16
Industry’s Most Complete Feature Set
100% Native
S3 API
Multi Tenancy,
QoS, Billing
Policy-based
data protection
One System for
Files & Objects
Policy-based
Tiering to Cloud
Flexible Deployment
Options
Up to 14 nines
data durability
_
2
1
Cloud-like
cost
GB/
mo.

15. HyperStore Architecture

16. 18
HyperStore Node Hardware
CPU HDD Drives Network SSD Drives
• CPUs – CPU Clock speed is preferred over
number of cores, but of course the more cores
the better.
• Memory – More memory is required as a
node has more drives/capacity
• OS/Meta-data disks - More capacity is
required for meta-data storage as a node has
more drives/capacity
• Network Interface – For better performance it
is highly recommended to deploy nodes with 2
x 10GbE interfaces as a minimum, more ports
can be utilized by bonding ports together for
resilience and greater bandwidth capability.
RAM

17. ©19
250TB
500TB
750TB
1PB
Limitlessly Scalable
Grows with your needs
• Infinite capacity
• Never migrate data
again
• Performance increases
as nodes are added
Looks
like
one
device

18. HyperStore High Level Structure
Region
Data Center
Nodes
Data Center
Nodes

19. Support for multiple Topologies and Protection
Schemes
© 2016, Cloudian, Inc. Confidential. Do Not Distribute21
Replication Factor
Clients
Replicated EC
EC Single Site
Distributed EC
DC1
S3 Bucket – RF=3
DC1
S3 Bucket – EC:4+2
Clients
S3 Bucket – RF=2 over EC:4+2
Clients
DC1 – EC:4+2 DC2 – EC:4+2
Replication
Storage Overhead
Raw TBs / RF
Storage Overhead
(k/k+m) x Raw TBs
Storage Overhead
(k/k+m) x Raw TBs / DC RF
DC1 DC3DC2
S3 Bucket – Distributed EC:7+5
ClientsStorage Overhead
(k/k+m) x Raw TBs
Low Latency Network Required

20. Peer-to-peer system = no SPOF
Distributed Everything = Data , Metadata, Configuration
User Defined
Location Affinity
DC1
DC2
Add Node <-> Auto Rebalance
Server <-> vNodes <-> Disks
Distributed & Elastic Geo Cluster

21. 23
Support for Multiple Configurations
Storage MSP
On-Premises
Apps
Multi-Site
Hybrid
Apps
Stand alone, on-premise storage Replicate or migrate to the cloud for capacity, data
protection, content distribution
Fully S3-compatible storage servicesReplicate or distribute data across sites
SITE A
SITE B SITE C

22. This eliminates the traditional bottlenecks
associated with other architectures.
• S3 Service
• HyperStore Service
• Cassandra DB service
• Redis DB service
• Admin service
• Cloudian Management Console (CMC)
• Supporting Services
Key Services Run on ALL nodes
HyperStore Services:
© 2017, Cloudian, Inc.24

23. © 2017, Cloudian, Inc. Confidential. Do Not Distribute25
Start Small, Grow Without Disruption
The Solution for Scalable Storage
Begin with just 3 nodes
• Full data protection
Grow to 100s of Petabytes
• Zero down-time scaling
• Unlock more efficient & redundant policies as system scales
Single Management Environment
• Introduce new Regions & DC’s after initial deployment
• Unlock additional protection polices as system scales

24. HyperStore Features

25. © 2017, Cloudian, Inc.27
Policy Based Data Protection & Consistency Settings
Tunable Consistency Levels
Sync or Async
IO Acknowledgements
Single and Multi Site Support
• Granular Control, set at the bucket level
• Unlock more efficient protection schemes by adding nodes

26. ©28
Hybrid Cloud Tiering
Metadata kept on local platform
APPLICATIONS S3 ON PREM
DATA CENTER
TBs to PBs
Policy-based
• Multicloud storage
• Bimodal access

27. 29
Versioning and WORM support
• Accidental deletion
• Ransomware
• Rouge admins
• Malicious attacks
Enabled at the Bucket Level, versioning can provide a
safeguard against:
When using WORM objects in the bucket cannot
be deleted, overwritten or modified from any
source or by any command until the retention
period has expired.
Once locked, the bucket lock policy cannot be
deleted from the bucket.
The bucket can only be deleted when the retention
period for all objects in the bucket has expired and
the objects have been deleted

28. Hybrid Cloud: Cross Region Replication
Target Use Cases
Secure Vault – Log all changes to separate Cluster
Hyperstore Backup – Backup data & Restore to any point time
DR – Protection from Primary Cluster Loss (eg Disasters)
v2
RESTORE Objects
from Past
To New Bucket (eg Jan 1)
Photo.
jpg
ALL new PUTs
are replicated to Destination Cluster
v1
v2
v3
v1
v2
v3
Source
Cluster
Destination
Cluster
Photo.jpg
Dec1
Jan1
Feb1
DELETEs has NO
Affect on Destination Cluster
v1
v2
v3
v1
v2
v3
Delete Marker
Jan9
Supported Deployments
Hyperstore 6 ->
Hyperstore6
Hyperstore 6 ->
© 2017, Cloudian, Inc.30

29. Built In S3 Client: Bucket & Objects Browser Object Publishing
Object Expiration
Object Lifecycle Management
© 2017, Cloudian, Inc.31

30. 32
Encryption
Object
S3 bucket
SSL
Object
Master key
Data keys
Server side Encryption key (regular SSE)
• Server manages master key and generates
per-object key that’s stored in object metadata.
• Can be integrated with an external key
management system.
Object
Per-object key
Encrypt S3 bucket
SSL
Customer-provided encryption key
(SSE-C)
• Encryption key is never stored.
• Customer must use same key on PUT
and GET.
Object
Encrypted
Object & key S3 bucket
Master key
AWS SDK
Envelope keys
Client-side Encryption
• Client provides and manages master keys.
• With AWS SDK, dynamic “envelope” key
generated and used to encrypt object and key.
• Encrypted envelope key is sent and stored as
object metadata, and checked on retrieval.
KMS Support for SSE
• Supported with Gemalto KeySecure

31. Compression
3 Key Advantages of Utilizing
Cloudian HyperStore Compression
Faster Speed Higher Compression Rate
Choose between 3
different compression
algorithms to match
speed and compression
rate requirements
• Enabled at bucket level
1. Reduced Storage Utilization
2. Reduced Network Utilization
3. Improved Replication Speed
© 2017, Cloudian, Inc.33

32. Multi-Tenancy
© 2017, Cloudian, Inc.34

33. QoS
Requests
per Min
Storage
Bytes
Storage
Objects
Inbound
Bytes/Min
Outbound
Bytes/Min
HyperStore Software Defined Storage
PROGRAMMABLE APIs
USER
GROUP
If Limit is reached, requests are rejected until next windowDefine Limits – Warning and Max Levels
© 2017, Cloudian, Inc.35

34. Smart Support
HyperStore
Telemetrics
Global
Analytics
Proactive
Resolution
Real Time
Monitoring
Cloudian Smart
Support
S3 Operations
Resource Usage
© 2017, Cloudian, Inc.36

35. CloudStack Integration

36. Unified Storage – Multiple Use Cases
© 2017, Cloudian, Inc.38
Backup Snapshots Templates NFSWeb Apps

37. NFS Support
• NFS Gateway support is offered via HyperFile
• Allows access to same object via S3 or NFS
• Suitable use cases :
• VM templates/snapshots storage
• Backup workloads (reading/writing large files)
• Media Repository Sharing
S3
NFS
39

38. CloudStack Integration – SSO
•Integrated Management GUI
•Integrated provisioning
• Store OS images & snapshots
• Single Sign-on
40

39. •Open standards approach – full S3 compatibility
•Store templates, ISO images and snapshots on a multi-tenant hybrid object
storage platform
•Share data between user projects and virtual machine instances
•Built in advanced data protection features including:
•Versioning
•Erasure Coding
•Replication
•DR with multi DC support, multiple regions and tunable consistency settings
41
Key Benefits

40. Questions?