Your SlideShare is downloading. ×
0
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Introduce LeoFS
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Introduce LeoFS

4,896

Published on

Published in: Technology, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,896
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
25
Comments
0
Likes
3
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. You can rely on whenever you need. 27th July 2012 Yosuke Hara July 28th, 2012 Yosuke Hara
  • 2. We launched LeoFS as OSS on July 4th. LeoFS - http://www.leofs.org
  • 3. Motivation
  • 4. Motivation ? NFS - Very Expensive 1. Low ROI 2. Possibility of SPOF 3. Storage Expansion is difficult during increasing data
  • 5. Motivation 762B X 3.5 220B
  • 6. Object Storage Farm
  • 7. Object Storage Farm Unstructured data S3FS-C
  • 8. Object Storage Farm 8
  • 9. Overview
  • 10. LeoFS Overview
  • 11. LeoFS Overview Request from Web Application(s) or Browser Load Balancer Gateway Manager HTTP LeoFS-Manager Cluster Request/Response Handling S3-API Management + REST over HTTP w/Object Cache RPC Ring Watcher LeoFS-Gateway (AWS S3-API) Node Watcher SNMP RPCLeoFS-Storage Storage GUI Console Object Storage, Meta data Storage Storage Engine/Router Storage Engine/Router Storage Engine/Router + META Replicator/Recoverer META Object Store META Object Store Object Store
  • 12. LeoFS Overview Request from Web Application(s) or Browser [Load Balancer] LeoFS-Manager S3-API REST over HTTP RPC LeoFS-Gateway SNMP RPCLeoFS-Storage Storage Engine/Router Storage Engine/Router Storage Engine/Router GUI Console META Object Store META Object Store META Object Store
  • 13. LeoFS Overview - Example of System layouts Minimum for Service (App) Dev Manager x 1 Gateway x 1 Storage x 1 50TB Storage System (# of replicas = 3) XEN, VMWare,... 10TB .. 20TB / server Manager x 2 Gateway x 3 .. Storage x 8 - 15 300TB Storage System (# of replicas = 3) XEN, VMWare,... 10TB .. 20TB / server Manager x 2 Gateway x 4 .. Storage x 45 - 90
  • 14. Inside LeoFS
  • 15. LeoFS Architecture HTTP Gateway Object Cache Erlang RPC Erlang RPC Storage Cluster Erlang RPC Process Monitor Manager Cluster
  • 16. LeoFS Architecture - Gateway / Storage Erlang/OTP Runtime Gateway REST over HTTP (S3-API) get put delete head redundant-manager membership (fault-detection) RPC Storage redundant-manager replicator read-repairer RPC RPC membership (fault-detection) queue Storage Engine Object Storage Metadata
  • 17. LeoFS Architecture - Manager Erlang Mnesia RING, Member / Cluster State, Auth Process Monitor Gateway(s) / Storage Cluster
  • 18. LeoFS Gateway
  • 19. LeoFS Gateway From Applications (Clients) S3-API Gateway Object Cache Consistent Hashing - Horizontal Distribution Erlang RPC replicate when using RPC Storage Nodes
  • 20. Storage Engine
  • 21. LeoFS Storage Engine Request From Gateway LeoFS Storage replicator ... repairer Object Storage queue Metadata : Keeps an in-memory index of all data. Object Storage : Log structured (append-only) object store.
  • 22. LeoFS Storage Engine - Retrieve an object from the storage Log Structured File System LeoFS Storage Engine < META DATA > ID Filename Offset Size Checksum Metadata Data Header File Footer Object Container
  • 23. LeoFS Storage Engine - Insert an object into the storage Log Structured File System Add a Metadata Metadata Data Append an object
  • 24. LeoFS Storage Engine - Reduce unnecessary objects from the storage Log Structured File System Compaction NEW Object Container OLD Object Container
  • 25. LeoFS Manager
  • 26. LeoFS Manager For Administrators Manager monitor RING, Node State operate status, suspend, resume, detach, whereis, ...
  • 27. Future Works
  • 28. Future Works - Leo Tamer Log Analyzer / Searcher Operation Center
  • 29. Future Works - Integrated HBase with LeoFS “Cloud DWH” = +
  • 30. Wrap Up
  • 31. LeoFS can realize 3-HIGHs - HIGH Cost performance HIGH Reliability HIGH Scalability
  • 32. We set sail for our goal. LeoFS - http://www.leofs.org

×