Progress on adapting BlobSeer to WAN scale

  • 414 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
414
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
2
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Towards a Scalable File SystemProgress on adapting BlobSeer to WAN scalefor the HGMDS distributed metadata systemViet-Trung Tran, Gabriel Antoniu, Alexandru Costan (INRIA - Rennes)In collaboration with Kohei Hiraga, Osamu Tatebe (U Tsukuba)FP3C meetingBordeaux, 2 – 3 September 2011
  • 2. Plan1. Background and context2. Goal3. Approach and solution4. Preliminary evaluation5. ConclusionFP3C meeting – Bordeaux, 2-3 September 2011 -2
  • 3. 1BackgroundBlobSeer & HGMDSFP3C meeting – Bordeaux, 2-3 September 2011 -3
  • 4. BlobSeer: A large-scale data managementserviceGeneric data-management platform for huge, unstructured data•  Huge data (TB) : BLOBs•  Highly concurrent, fine-grain access (MB): R/W/A•  Prototype availableKey design features•  Decentralized metadata management•  Beyond MVCC: multiversioning exposed to the user•  Lock-free write access through versioningA back-end for higher-level, sophisticated data management systemsFP3C meeting – Bordeaux, 2-3 September 2011 -4
  • 5. BlobSeer: ArchitectureClients Providers•  Perform fine grain blob accessesProviders•  Store the pages of the blobProvider manager•  Monitors the providers•  Favours data load balancing Provider Clients managerMetadata providers•  Store information about page location VersionVersion manager manager•  Ensures concurrency control Metadata providersFP3C meeting – Bordeaux, 2-3 September 2011 -5
  • 6. HGMDS: A distributed metadatamanagement system for global file systems•  Multi-master file system The  Internet metadata server (MDS). Site A Site B •  Managing inode structure. File system Clients •  High latency networks dontaffect metadata operation HGMD HGMDperformance. S S - Both reading and writing.•  One MDS per site.•  Metadata versioning using mkdir/rmdir/ Propagate create/stat/ updates invector clocks for collision unlink backgrounddetection. Site C •  Automatic collision resolutionby system side.FP3C meeting – Bordeaux, 2-3 September 2011 -6
  • 7. 2GoalA joint architecture integrating BlobSeer and HGMDSFP3C meeting – Bordeaux, 2-3 September 2011 -7
  • 8. Goal BlobSeer HGMDS Data management Metadata management Typically on a single site Global scale, multiple sitesIdea: build a global file system deployed on multiple site by integratingBlobSeer to HGMDSPotential benefits:•  HGMDS: efficient multi-site file metadata management•  BlobSeer: concurrency-optimized access to globally shared dataFP3C meeting – Bordeaux, 2-3 September 2011 -8
  • 9. 3Our approach and solutionFP3C meeting – Bordeaux, 2-3 September 2011 -9
  • 10. Two approachesMultiple BlobSeer instances•  One BlobSeer / siteOne single BlobSeer-WAN over distributed geographicsitesFP3C meeting – Bordeaux, 2-3 September 2011 - 10
  • 11. 1st approach: 1 BlobSeer instance / site ClientFP3C meeting – Bordeaux, 2-3 September 2011 - 11
  • 12. 1st approach: ZoomHigh latency when accessing remote BLOBs:•  Too many remote requests for small metadataEMETTEUR - NOM DE LA PRESENTATION - 12
  • 13. 2nd approach: 1 BlobSeer-WAN instanceover distributed geographic sitesMultiple version managers•  1 version manager/siteMultiple provider managers•  1 provider manager/siteOn each site•  Multiple data providers and metadata servers•  Data providers are under control of local provider managerEMETTEUR - NOM DE LA PRESENTATION - 13
  • 14. Idea: leverage localityfor remote metadata accesses 2Metadata I/O is resolved locallyEMETTEUR - NOM DE LA PRESENTATION - 14
  • 15. 2nd approach: I/O scheme in BlobSeer-WANWriting•  Publish version on local version manager•  Locally write metadata on local metadata servers•  Locally write data on local data providersReading (Read your write in many cases)•  Ask a version to local version manager•  Local metadata accesses•  Access remote/local providers if necessaryFP3C meeting – Bordeaux, 2-3 September 2011 - 15
  • 16. Vector clocks and optimistic metadatareplicationFP3C meeting – Bordeaux, 2-3 September 2011 - 16
  • 17. Expected benefits•  On WAN: BlobSeer coordinates with HGMDS to provide a global versioning file system - Low latency metadata I/O - Eventually consistency model - Load balancing/fault tolerance•  On LAN: - Distributed version management - Load balancing/fault toleranceFP3C meeting – Bordeaux, 2-3 September 2011 - 17
  • 18. 4Preliminary evaluationBlobSeer-WAN on G5KFP3C meeting – Bordeaux, 2-3 September 2011 - 18
  • 19. TestbedUsing 2 sites of G5K•  Rennes: 40 nodes • 30 nodes reserved for BlobSeer services • 10 nodes for clients•  Grenoble: 40 nodes • 30 nodes reserved for BlobSeer services • 10 nodes for clients•  Interconnect network between sites 10 GbpsFP3C meeting – Bordeaux, 2-3 September 2011 - 19
  • 20. Concurrent appending: 512 MB/clientFP3C meeting – Bordeaux, 2-3 September 2011 - 20
  • 21. 5ConclusionOn going workFP3C meeting – Bordeaux, 2-3 September 2011 - 21
  • 22. SummaryDiscussed the integration of BlobSeer and HGMDS:•  BlobSeer-WAN extension is requiredBlobSeer-WAN•  Preliminary results look encouraging•  Performance of BlobSeer-WAN on two sites similar to that of vanilla BlobSeer on a single site•  Prototype available at BlobSeer’s repository/branches/ BlobSeer-WAN-dev/HGMDS•  Implementation almost done•  Works on multi-sites•  Collisions automatically solved by a ruleFP3C meeting – Bordeaux, 2-3 September 2011 - 22
  • 23. Next steps•  A more extensive evaluation for BlobSeer-WAN•  Integrate BlobSeer-WAN to HGMDS•  Preliminary evaluation of HGMDS BlobSeer-WAN on Grid5000 and on the Japanese Clusters•  Submit co-authored paper by Spring 2012•  Next internships: Kohei @Inria RennesFP3C meeting – Bordeaux, 2-3 September 2011 - 23
  • 24. Thank you! FP3C meeting 2 – 3 September 2011