Introduce LeoFS

17,767 views
17,622 views

Published on

Introduce LeoFS - Object Storage System for the Web

Published in: Technology

Introduce LeoFS

  1. 1. You can rely on whenever you need Yosuke Hara Oct 30, 2012Tuesday, October 30, 12
  2. 2. 1. MotivationTuesday, October 30, 12
  3. 3. Need to store and manage huge amount of “media files” at low-cost.Tuesday, October 30, 12
  4. 4. ? 1. Low ROI 2. Possibility of SPOF 3. Storage Expansion is difficult during increasing dataTuesday, October 30, 12
  5. 5. Face the situation X 3.5 *ZDNet: http://www.zdnet.com/blog/btl/amazon-web-services-hosts-more-than-762-billion-objects-in-s3/68308Tuesday, October 30, 12
  6. 6. “Object Storage System”Tuesday, October 30, 12
  7. 7. From Photo Storage To Cloud Storage 5MB 100MB a few GB 1st step as Cloud Storage Specialize in “Photo”Tuesday, October 30, 12
  8. 8. From Photo Storage To Cloud Storage 5MB 100MB a few GB Aim to “DATA-HUB” in the Cloud Handle various unstructured dataTuesday, October 30, 12
  9. 9. S3FS-CTuesday, October 30, 12
  10. 10. Aim ToTuesday, October 30, 12
  11. 11. 2. OverviewTuesday, October 30, 12
  12. 12. Storage Gateway Manager Concurrency Distribution Fault tolerance Using in Telecom, Banking, e-commerce, Instant messaging,...Tuesday, October 30, 12
  13. 13. LeoFS Overview Request from Web Application(s) or Browser Load Balancer Gateway (Stateless Proxy) Manager HTTP Request/Response Handling LeoFS-Manager System Management + S3-API w/Object Cache REST over HTTP Ring Monitor RPC LeoFS-Gateway Node State Monitor SNMP RPC LeoFS-Storage Storage Storage Engine/Router Storage Engine/Router Storage Engine/Router GUI Console Object Storage, Meta data Storage + META Object Store Replicator/Recoverer, Qeueue META Object Store META Object StoreTuesday, October 30, 12
  14. 14. Request from Web Application(s) or Browser Load Balancer LeoFS-Manager S3-API REST over HTTP (80/443) RPC (4369) LeoFS-Gateway (4000,4010,4020) Monitor RPC (4369) (SNMP) LeoFS-Storage (10020, 10021) Storage Engine/Router Storage Engine/Router Storage Engine/Router GUI Console META Object Store META Object Store META Object StoreTuesday, October 30, 12
  15. 15. 1 - Minimum for Development Manager x 1 Gateway x 1 Storage x 1 20 - 50TB Storage System (# of replicas = 3) 10TB .. 20TB / server Manager x 2 Gateway x 3 .. Storage x 8 - 15 50 - 300TB Storage System (# of replicas = 3) 10TB .. 20TB / server Manager x 2 Gateway x 4 .. Storage x 45 - 90 on XEN, VMWareTuesday, October 30, 12
  16. 16. DATA-HUB Media Platform PaaS / IaaS Search / Analysis Application / Log CollectorTuesday, October 30, 12
  17. 17. Tuesday, October 30, 12
  18. 18. “DATA-HUB”Tuesday, October 30, 12
  19. 19. Inside LeoFSTuesday, October 30, 12
  20. 20. LeoFS Architecture HTTP GatewayTuesday, October 30, 12
  21. 21. LeoFS Architecture HTTP Gateway Erlang RPC Object Cache Erlang RPC Storage Cluster Erlang RPC State/Process Monitor Manager ClusterTuesday, October 30, 12
  22. 22. LeoFS Architecture - Gateway / Storage Gateway REST over HTTP (S3-API) get put delete head redundant-manager membership (fault-detection) RPC Storage redundant-manager replicator read-repairer RPC RPC membership (fault-detection) queue Storage Engine Object Storage MetadataTuesday, October 30, 12
  23. 23. LeoFS GatewayTuesday, October 30, 12
  24. 24. “Stateless Proxy” From Applications S3-API Gateway “Cowboy” Object Cache [ LRU, Slab allocator, Skip graph ] Consistent Hashing Horizontal Distribution Replicate when using RPC Storage Nodes *Cowboy: Erlang light-weight HTTP-Server - http://http://www.ninenines.eu/Tuesday, October 30, 12
  25. 25. LeoFS StorageTuesday, October 30, 12
  26. 26. LeoFS Storage Engine Request From Gateway LeoFS Storage ... replicator leo-object-storage repairer queue ... Metadata : Keeps an in-memory index of all data. Object Storage : Log structured (append-only) object store.Tuesday, October 30, 12
  27. 27. LeoFS Storage Engine - Data Structure for Retrieve an File (Object) <Metadata> for Sync Custom Time- {VNodeId, Key} KeySize Meta Size File Size Offset Version Checksum stamp <Object> User-Meta Time- {VNodeId, Actual Checksum KeySize Size DataSize Offset Version User-Meta Footer stamp Key} File Header (Metadata - Fixed length) Body (Variable Length) Footer (8B) <Object Container> Super-block Object-1 Object-2 Object-3 Object-4 Object-5Tuesday, October 30, 12
  28. 28. LeoFS Storage Engine Traditional File System Leo Object Storage directory inode owner info size timestamp blocks directory data inode ID Object filename file inode owner info size timestamp blocks FileTuesday, October 30, 12
  29. 29. LeoFS Storage Engine Leo Object Storage Robustness ID Object High PerformanceTuesday, October 30, 12
  30. 30. LeoFS Storage Engine - Retrieve an object from the storage Storage Engine < META DATA > > < META DATA ID Id Filename Filename Offset, Size Offset Checksum (MD5) Size Version# Metadata Storage Checksum Header File Footer “Object Container”Tuesday, October 30, 12
  31. 31. LeoFS Storage Engine - Insert an object into the storage Storage Engine Insert a metadata Metadata Storage Data Append an object into the object containerTuesday, October 30, 12
  32. 32. LeoFS Storage Engine - Remove unnecessary objects from the storage Storage Engine Compaction NEW Object Container New Metadata OLD Object Container OLD MetadataTuesday, October 30, 12
  33. 33. Large object SupportTuesday, October 30, 12
  34. 34. Large-object Support (LeoFS v0.12) 1. Able to equalize disk usage of each storage node 2. High I/O efficiency chunk-0 chunk-1 chunk-2 Storage Cluster chunk-3 Client Gateway #metadata{ Original fils’s Metadata ... dsize = 10485768, cnumber = 4, csize = 3670016, ... }Tuesday, October 30, 12
  35. 35. LeoFS ManagerTuesday, October 30, 12
  36. 36. LeoFS Manager For Administrators TCP/IP Manager for continue operation Monitor RING, Node State Operate status, suspend, resume, detach, whereis, ...Tuesday, October 30, 12
  37. 37. Request from Web Application(s) or Browser Load Balancer LeoFS-Manager S3-API REST over HTTP (80/443) RPC Verticalized Storage System (4369) LeoFS-Gateway (4000,4010,4020) Monitor RPC (4369) (SNMP) LeoFS-Storage (10020, 10021) Storage Engine/Router Storage Engine/Router Storage Engine/Router GUI Console META Object Store META Object Store META Object StoreTuesday, October 30, 12
  38. 38. 3. Future WorksTuesday, October 30, 12
  39. 39. Improve Performance & Functionality 1. Multi-layer Cache (Using SSD) HIGH-Performance (Low latency) High I/O efficiency Gateway Primary Cache Secondary Cache Storage Reduce traffic between Gateway and StorageTuesday, October 30, 12
  40. 40. Improve Performance & Functionality 2. Multi-Datacenter for Geographical Optimization HIGH-Scalability HIGH-AvailabilityTuesday, October 30, 12
  41. 41. Improve Performance & Functionality 3. LeoFS’s Tools System Console Log Analyzer / SearcherTuesday, October 30, 12
  42. 42. 4. Wrap UpTuesday, October 30, 12
  43. 43. LeoFS will realize 3-HIGHs, soon HIGH Cost Performance HIGH Reliability HIGH ScalabilityTuesday, October 30, 12
  44. 44. Thank you for your time LeoFS - http://www.leofs.orgTuesday, October 30, 12

×