Skylable provides an object storage solution which you can run on your own hardware. It provides data replication, deduplication, client-side encryption, and built-in data integrity checks. Skylable scales easily to hundreds of TBs and it can be deployed in literally 2 minutes by anyone with a minimal experience of Linux administration. The installation and setup process is straightforward and requires very few dependencies. Skylable is fully compatible with Amazon S3 APIs. It also offers a more advanced RESTful protocol called SX, which solves many limitations of the S3 protocol, most importantly the ability to upload/download files in parallel from multiple hosts and true support for resuming transfers. The software is released under the GPL licence, the client library (libsx) is LGPL. This talk is about the ease of use of Skylable, a comparison of S3 vs SX and the real-world advantage of doing deduplication and multiplexed transfers.

1. Luca Gibelli
www.skylable.com

2. Today’s presentation
•The limits of traditional SANs
•The limits of the S3 protocol
•Why SX Cluster is better
•Use cases

3. Find out more: http://www.skylable.com/about
From open-source security …

4. Find out more: http://www.skylable.com/about
… to open-source storage!
SX Cluster,
open-source
distributed object storage
Seed round by
Sunstone Capital

5. Find out more: http://www.skylable.com/about
How did it happen?

6. “We have outgrown
our SAN”
Most pressing issues
“Connectivity between HQ
and satellite offices is poor”
“No vendor offers all
the features we need”
“That new cool project?
Put it on-hold”
“We have
no disaster
recovery plan”
“The new SAN
is over-budget”

7. Storage:
never big enough

8. Is that what we need?
1 step back…

9. The CAP theorem

10. Block storage: CA
Structured
Limited by metadata
Complex
Expensive

11. Filesystem storage: CA
Same story

12. Object-storage: AP
Unstructured
Metadata is in the obj
Application layer
Commodity hardware

13. Object storage, when…
YOU NEED:
Scalability, durability, low price
FOR:
Media files, documents, backups, log files

14. Switching is hard
Block/filesystem storage: just syscalls
Object storage: application layer
Must re-engineer the application!

15. Typicial object storage
makes compromises
PUT, GET, LIST, DELETE
No append
No locking
No file hierarchy (just fakedirs)

16. What is wrong with S3?
S3
Data Upload Storage

17. Good enough?
With SX Cluster
you can do better…

18. SX protocol with
client side
deduplication
Sx
Data Upload Storage

19. SX is the answer!
Q. No append?
A. Client side deduplication

20. SX is the answer!
Q. No append?
A. Client side deduplication
Q. No locking?
A. Revisions

21. SX is the answer!
Q. No append?
A. Client side deduplication
Q. No locking?
A. Revisions
Q. Revisions??
A. Client side deduplication

22. • Complex to install and run
• S3 protocol only
• No de-duplication
Our competitors

23. • 3-steps installation process
• SX protocol
• Native de-duplication and
versioning
We do better!

24. sx@node1 # sxsetup
Enter the cluster name (use the same across all nodes: sx.foo.com
Path to SX storage: /mnt/spare/sx
Maximum size: 1T
Enter the IP address of this node: 192.168.10.1
Is this (192.168.10.1) the first node of a new cluster? (Y/n) <ENTER>
Is this correct? (Y/n) <ENTER>
sx@node1 #
sxsetup on
first node

25. sx@node2 # sxsetup
Enter the cluster name: sx.foo.com
Path to SX storage /mnt/spare/sx
Maximum size: 1T
Enter the IP address of this node: 192.168.10.2
Is this (192.168.10.2) the first node of a new cluster? (Y/n) n
Please provide the IP address of a working node in 'sx.foo.com'.
IP address: 192.168.10.1
Admin key or path to key-file [default=]:
0DPiKuNIrrVmD8IUCuw1hQxNqZcVPDD82Gkq7PMFYpk3qA8ddxxxxxxx
Is this correct? (Y/n) <ENTER>
Server certificate:
SHA1 fingerprint: 198b0ce5161757ea9bc83fc77627eb8c0958d591
Do you trust this SSL certificate? [y/N] y
sx@node2 #
sxsetup on
additional nodes

26. The best of 3 worlds
torrent
rsync
S3 • Firewall friendly
• Never transfer the same
data twice
• Speed grows linearly
with nr. of nodes

27. Use cases
What can you do with SX Cluster?

28. • Drop-in replacement for AWS S3
• Create as many endpoints as you
need and LB between them
• Faster local transfers inside the DC
Use case #1:
S3 compatible storage

29. • Serve data from multiple locations
• Use S3 to upload/update the data,
serve over http/https
• Replicate synchronously to all
locations
Use case #2:
CDN (credil.org, imgy.org)

30. • Hundreds of TBs of new medical
images every month
• Plugs into dcm4che
• Multi-tiering
• Offsite live replica, write once, read
many
Use case #3:
Picture Archiving
Communication System

31. • No middleware!
Scales horizontally
• Multi-platform:
Win/OSX/Linux, Android and
iOS, web
• Client-side encryption
• Incremental transfers,
revisions w/ deduplication
Use case #4:
White label cloud drive

32. • Commercial support
• SX Console
• Real-time metrics
• Multi-tenancy
• LDAP/AD/SAML + 2FA auth
• SX Indexer
SX Enterprise Edition

33. Try Skylable
http://www.skylable.com/products/
Build your own distributed object-storage
• GPL licence
• Deploy with a single command
• Scales horizontally
• Parallel transfers, client-side encryption, and
more…

34. Try Skylable
http://www.skylable.com/products/
SX Cluster
LibreS3
SXDrive
sxfs (beta)

35. Get in touch:
Luca Gibelli
luca@skylable.com / @nervous