Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Sdc challenges-2012


Published on

Sdc challenges-2012

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Sdc challenges-2012

  1. 1. GlusterFS Challenges and Futures Jeff Darcy Storage Developer Conference September 17, 2012
  2. 2. What Is GlusterFS? ● Just your basic distributed file system ● sharding, replication, striping ● Decentralized ● no central metadata server ● core functionality on clients ● Modular ● “translators”
  3. 3. Translator Stacking FUSE DHT AFR-1 Client-3Client-1 AFR-0 Client-2Client-0 ● One to one, one to many, one to zero (?) ● Rearrange, move from client to server, ...
  4. 4. Distribution (now) Server A Server BServer C File X File Y
  5. 5. Replication (now) ● Based on changelog (“dirty flags”) ● set flags, do operation, clear flags ● use flags to determine repair (“self-heal”) after failure ● Latency sensitive ● 3+ network round trips per user request ● implementation heavily optimized
  6. 6. Challenge: Multitenancy ● Focus of CloudFSHekaFS ● to be merged with GlusterFS soon-ish ● Isolate name and ID spaces ● Encrypt in flight and at rest ● Auth* ● Quota (next slide) ● Performance isolation ● cgroups
  7. 7. Challenge: Distributed Quota ● Can't trust clients to enforce quota ● Can't just divide equally among servers ● unequal usage (e.g. due to explicit placement) ● EDQUOT on one while still space on another ● “Quota rebalancing daemon” ● monitor/adjust continuously ● interesting problem at high scale
  8. 8. Challenge: Better Rebalancing ● Optimal placement vs. minimal data movement ● Different kinds of weighting Node A Node B Node C Node D Node E
  9. 9. Challenge: Replication Latency ● Reaching limits of current approach ● Have to go async? ● but still ordered ● exploit compute/data locality (e.g. Hadoop) ● journaling, conflict resolution
  10. 10. Challenge: Directory Traversal ● Piggyback attrs (and xattrs) on readdir ● Even better: cursor approach ● read everything in opendir ● zero network activity for readdir ● less current, but more consistent
  11. 11. Challenge: Many Small Files ● Prefetch whole directories ● if marked, below size threshold, ... ● Exploit async journal ● only works if compute/data colocated ● Weaken consistency? ● allow create/write/close to be buffered ● directory-level fsync (magic xattr)
  12. 12. Conclusions ● Most of these challenges are not unique to GlusterFS ● Modularity and incremental progress are preferable to monolithic “solve all problems at once” ● GlusterFS provides a good environment in which to experiment with solutions