Introduce LeoFS

5,620 views

Published on

Published in: Technology, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,620
On SlideShare
0
From Embeds
0
Number of Embeds
3,449
Actions
Shares
0
Downloads
32
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Introduce LeoFS

  1. 1. You can rely on whenever you need. 27th July 2012 Yosuke Hara July 28th, 2012 Yosuke Hara
  2. 2. We launched LeoFS as OSS on July 4th. LeoFS - http://www.leofs.org
  3. 3. Motivation
  4. 4. Motivation ? NFS - Very Expensive 1. Low ROI 2. Possibility of SPOF 3. Storage Expansion is difficult during increasing data
  5. 5. Motivation 762B X 3.5 220B
  6. 6. Object Storage Farm
  7. 7. Object Storage Farm Unstructured data S3FS-C
  8. 8. Object Storage Farm 8
  9. 9. Overview
  10. 10. LeoFS Overview
  11. 11. LeoFS Overview Request from Web Application(s) or Browser Load Balancer Gateway Manager HTTP LeoFS-Manager Cluster Request/Response Handling S3-API Management + REST over HTTP w/Object Cache RPC Ring Watcher LeoFS-Gateway (AWS S3-API) Node Watcher SNMP RPCLeoFS-Storage Storage GUI Console Object Storage, Meta data Storage Storage Engine/Router Storage Engine/Router Storage Engine/Router + META Replicator/Recoverer META Object Store META Object Store Object Store
  12. 12. LeoFS Overview Request from Web Application(s) or Browser [Load Balancer] LeoFS-Manager S3-API REST over HTTP RPC LeoFS-Gateway SNMP RPCLeoFS-Storage Storage Engine/Router Storage Engine/Router Storage Engine/Router GUI Console META Object Store META Object Store META Object Store
  13. 13. LeoFS Overview - Example of System layouts Minimum for Service (App) Dev Manager x 1 Gateway x 1 Storage x 1 50TB Storage System (# of replicas = 3) XEN, VMWare,... 10TB .. 20TB / server Manager x 2 Gateway x 3 .. Storage x 8 - 15 300TB Storage System (# of replicas = 3) XEN, VMWare,... 10TB .. 20TB / server Manager x 2 Gateway x 4 .. Storage x 45 - 90
  14. 14. Inside LeoFS
  15. 15. LeoFS Architecture HTTP Gateway Object Cache Erlang RPC Erlang RPC Storage Cluster Erlang RPC Process Monitor Manager Cluster
  16. 16. LeoFS Architecture - Gateway / Storage Erlang/OTP Runtime Gateway REST over HTTP (S3-API) get put delete head redundant-manager membership (fault-detection) RPC Storage redundant-manager replicator read-repairer RPC RPC membership (fault-detection) queue Storage Engine Object Storage Metadata
  17. 17. LeoFS Architecture - Manager Erlang Mnesia RING, Member / Cluster State, Auth Process Monitor Gateway(s) / Storage Cluster
  18. 18. LeoFS Gateway
  19. 19. LeoFS Gateway From Applications (Clients) S3-API Gateway Object Cache Consistent Hashing - Horizontal Distribution Erlang RPC replicate when using RPC Storage Nodes
  20. 20. Storage Engine
  21. 21. LeoFS Storage Engine Request From Gateway LeoFS Storage replicator ... repairer Object Storage queue Metadata : Keeps an in-memory index of all data. Object Storage : Log structured (append-only) object store.
  22. 22. LeoFS Storage Engine - Retrieve an object from the storage Log Structured File System LeoFS Storage Engine < META DATA > ID Filename Offset Size Checksum Metadata Data Header File Footer Object Container
  23. 23. LeoFS Storage Engine - Insert an object into the storage Log Structured File System Add a Metadata Metadata Data Append an object
  24. 24. LeoFS Storage Engine - Reduce unnecessary objects from the storage Log Structured File System Compaction NEW Object Container OLD Object Container
  25. 25. LeoFS Manager
  26. 26. LeoFS Manager For Administrators Manager monitor RING, Node State operate status, suspend, resume, detach, whereis, ...
  27. 27. Future Works
  28. 28. Future Works - Leo Tamer Log Analyzer / Searcher Operation Center
  29. 29. Future Works - Integrated HBase with LeoFS “Cloud DWH” = +
  30. 30. Wrap Up
  31. 31. LeoFS can realize 3-HIGHs - HIGH Cost performance HIGH Reliability HIGH Scalability
  32. 32. We set sail for our goal. LeoFS - http://www.leofs.org

×