Your SlideShare is downloading. ×
  • Like
Introduce LeoFS
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Introduce LeoFS

  • 4,786 views
Published

 

Published in Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
4,786
On SlideShare
0
From Embeds
0
Number of Embeds
3

Actions

Shares
Downloads
21
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. You can rely on whenever you need. 27th July 2012 Yosuke Hara July 28th, 2012 Yosuke Hara
  • 2. We launched LeoFS as OSS on July 4th. LeoFS - http://www.leofs.org
  • 3. Motivation
  • 4. Motivation ? NFS - Very Expensive 1. Low ROI 2. Possibility of SPOF 3. Storage Expansion is difficult during increasing data
  • 5. Motivation 762B X 3.5 220B
  • 6. Object Storage Farm
  • 7. Object Storage Farm Unstructured data S3FS-C
  • 8. Object Storage Farm 8
  • 9. Overview
  • 10. LeoFS Overview
  • 11. LeoFS Overview Request from Web Application(s) or Browser Load Balancer Gateway Manager HTTP LeoFS-Manager Cluster Request/Response Handling S3-API Management + REST over HTTP w/Object Cache RPC Ring Watcher LeoFS-Gateway (AWS S3-API) Node Watcher SNMP RPCLeoFS-Storage Storage GUI Console Object Storage, Meta data Storage Storage Engine/Router Storage Engine/Router Storage Engine/Router + META Replicator/Recoverer META Object Store META Object Store Object Store
  • 12. LeoFS Overview Request from Web Application(s) or Browser [Load Balancer] LeoFS-Manager S3-API REST over HTTP RPC LeoFS-Gateway SNMP RPCLeoFS-Storage Storage Engine/Router Storage Engine/Router Storage Engine/Router GUI Console META Object Store META Object Store META Object Store
  • 13. LeoFS Overview - Example of System layouts Minimum for Service (App) Dev Manager x 1 Gateway x 1 Storage x 1 50TB Storage System (# of replicas = 3) XEN, VMWare,... 10TB .. 20TB / server Manager x 2 Gateway x 3 .. Storage x 8 - 15 300TB Storage System (# of replicas = 3) XEN, VMWare,... 10TB .. 20TB / server Manager x 2 Gateway x 4 .. Storage x 45 - 90
  • 14. Inside LeoFS
  • 15. LeoFS Architecture HTTP Gateway Object Cache Erlang RPC Erlang RPC Storage Cluster Erlang RPC Process Monitor Manager Cluster
  • 16. LeoFS Architecture - Gateway / Storage Erlang/OTP Runtime Gateway REST over HTTP (S3-API) get put delete head redundant-manager membership (fault-detection) RPC Storage redundant-manager replicator read-repairer RPC RPC membership (fault-detection) queue Storage Engine Object Storage Metadata
  • 17. LeoFS Architecture - Manager Erlang Mnesia RING, Member / Cluster State, Auth Process Monitor Gateway(s) / Storage Cluster
  • 18. LeoFS Gateway
  • 19. LeoFS Gateway From Applications (Clients) S3-API Gateway Object Cache Consistent Hashing - Horizontal Distribution Erlang RPC replicate when using RPC Storage Nodes
  • 20. Storage Engine
  • 21. LeoFS Storage Engine Request From Gateway LeoFS Storage replicator ... repairer Object Storage queue Metadata : Keeps an in-memory index of all data. Object Storage : Log structured (append-only) object store.
  • 22. LeoFS Storage Engine - Retrieve an object from the storage Log Structured File System LeoFS Storage Engine < META DATA > ID Filename Offset Size Checksum Metadata Data Header File Footer Object Container
  • 23. LeoFS Storage Engine - Insert an object into the storage Log Structured File System Add a Metadata Metadata Data Append an object
  • 24. LeoFS Storage Engine - Reduce unnecessary objects from the storage Log Structured File System Compaction NEW Object Container OLD Object Container
  • 25. LeoFS Manager
  • 26. LeoFS Manager For Administrators Manager monitor RING, Node State operate status, suspend, resume, detach, whereis, ...
  • 27. Future Works
  • 28. Future Works - Leo Tamer Log Analyzer / Searcher Operation Center
  • 29. Future Works - Integrated HBase with LeoFS “Cloud DWH” = +
  • 30. Wrap Up
  • 31. LeoFS can realize 3-HIGHs - HIGH Cost performance HIGH Reliability HIGH Scalability
  • 32. We set sail for our goal. LeoFS - http://www.leofs.org