Your SlideShare is downloading. ×
File Context
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

File Context


Published on

Published in: Technology

  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. 1
    The New File system API:
    FileContext & AbstractFileSystem
    Sanjay Radia
    Cloud Computing
    Yahoo Inc.
  • 2. Agenda
    Overview – old vs new
    The New APIs
    What is next
  • 3. In a Nutshell: Old vs New
    Old: 1 Layer
    New: 2 Layers
    User API
    User API
    FS Impl API
    FS Impl API
  • 4. Motivation (1): 1st class URI file system namespace
    First class URI file system namespace
    Shell offers a URI namespace view: e.g. cp uri1 uri2
    But at user-API layer,
    Create a FileSystem instance for each target scheme-authority
    FileSystem.create(uriPath..) must take path in the FileSystem instance
    Fs = FileSystem.get(Uri1, …)
    Fs.create(pathInUri1, …)
    Original patch for symbolic link depicts the problem, … ) is invalid if UriPath is foreign
    But one can fool the FileSystem into following a symlink to the same UriPath.
    Need a layer that provides first class URI file namespace
  • 5. Motivation (2) : Separate Layers
    Two layers are merged in old FileSystem
    User Api which provides notion of default file system and working dir
    Implementation Api for implementers of the file systems
    Why separate?
    Simpler to implement file systems
    an API for implementing file systems (like VFS in Unix).
    It does not need to deal with Slash, wd, umask, etc
    Each file system instance is limited to its namespace
    User Api layer provides a natural place for
    The context: Slash, wd, umask, ….
    Uri namespace – cuts across the namespace of all file system instances.
    Hence a natural place to implement symbolic links to foreign namespaces
  • 6. Motivation (3): Cleanup API and Semantics
    FileSystem API & some semantics are not very good
    We should have adapted Unix Apis where appropriate
    Ask yourself: are you smarter than Ritche & Thompson and understand the issues well enough to be different
    Semantics: the recursive parent creation
    This convenience can cause accidental creation of parents
    E.g. A problem for speculative executions
    Semantics: Rename method, etc
    Too many overloaded methods … (eg. Create)
    The cache has leaked through: FileSystem.getNewInstance()
    Ugliness: e.g. copyLocal() copy(), …
    FileSystem leaked into Path
    Adding InterruptedException
    Some could have been fixed in the FileSystem class, but was getting messy to provide compatibility in the transition
    A clean break made things much easier
  • 7. Motivation (4): The Config
    The client-side config is too complex:
    Client should only need: Slash, wd, umask; that’s it nothing more.
    But Hadoop needs server-side defaults in client-side config
    An unnecessary burden on the client and admin
    Cluster admin cannot be expected to copy the config to the desktops
    Does not work for a federated environment where a client connects to many file systems each with its own defaults
    Solution: client grabs/uses needed properties from target server
    A transition to this solution from the current config is challenging if one needs to maintain compatibility within the existing APIs
    A common complaint is that Hadoop config is way too complicated
  • 8. 8
    The New File System APIs
    HADOOP-4952, HADOOP-6223
  • 9. First: Some Naming Fundamentals
    Addresses, routes, names are ALL names (identifiers)
    Numbers, or strings, or paths, or addresses or routes are chosen based on the audience or how they are processed
    ALL names are relative to some context
    Even absolute or global names have a context in which they are resolved
    Names appear to be global/absolute because you have simply chosen a frame-of-reference and are excluding the world outside that frame-of-reference.
    When two worlds, that each have “global” names, collide
    names get ambiguous unless you manage the closure/context
    There is always an implicit context – if you make that implicit context to be explicit by naming it, you need a context for the name of the context
    A more local context makes apps portable across similar environments
    A program can move from one Unix machine to another as long the names relative to the machine’s root refer to the “same” objects
    A Unix process’s context: Root and working dir
    plus default default domain, etc.
  • 10. We have URIs, why do we need Slash-relative names?
    Our world:
    a forest of file systems, each referenced by its URI
    Why isn’t the URI namespace good enough?
    The URI’s will bind your application to the very specific servers that provide that URI namepace.
    A application may run on cluster 1 today and be moved to cluster two in the future.
    If you move the data to the second cluster the app should work
    Better to let each cluster have its on default fs (i.e. slash)
    Also need the convenience of working dir
  • 11. Enter FileContext: A focus point on a forest of file systems
    A FileContext is a focus point on a forest of file systems
    In general, it is set for you in your environment (just like your DNS domain)
    It lets you access the common files in your cluster using location independent names
    Your home, tmp, your project’s data,
    You can still access the files in other clusters or file systems
    In Unix you had to mount remote file systems
    But we have URIs which are fully qualified, automatically mounted
    Fully qualified Uri is to Slash-relative-name
    as Slash-relative-names is to wd-relative-name
    … its just contexts ….
  • 12. Examples
    Use default config which has your default FS
    myFC = FileContext.getFileContext();
    Access files in your default file system
    myFC.create(“/foo”, ...);
    myFC.setWorkingDir(“/foo”) (“bar”, ...);
    Access files in other clusters“hdfs://nn3/bar”, ..)
    You can even set your wd to another fs!
    myFC. setWorkingDir(“hdfs://nn3/foo”)
    Variations on getting your context
    A specific URI as the default FS
    myFC = FileContext.getFileContext(URI)
    Local file system as the default FS
    myFC = FileContext.getLocalFSFileContext()
    Use a specific config,
    ignore $HADOOP_CONFIG
    Generally you should not need use a config unless you are doing something special
    configX = someConfigPassedToYou.
    myFC =FileContext.getFileContext(configX);
    //configX not changed but passed down
  • 13. So what is in the FileContext?
    The default file system (Slash) - obtained from config
    A pointer to the file system object is kept
    The working dir (lack of Slash)
    Stored as a path which is prefixed to relative path names
    Umask – obtained from config
    Absolute permissions after applying mask are sent to layer below
    Any other file system accessed are simply created
    0.21 – uses the FileSystem which has a cache
    0.22 – use the new AbstractFileSystem
    Do we need to add a cache? Hadoop-6356
  • 14. HDFS config – client & server side
    Client side config:
    Default file system
    Default values for blocksize, buffersize, replication are obtained at runtime from the specific filesystem in question
    Finally, federation can work
    Server side config:
    What used to be there before (except the above two items)
    + cleaned config variables for SS defaults for blocksize, etc.
  • 15. Abstract File System (0.22)
    User API
    Does not deal with
    FS Impl API
    DelegateTo FileSystem
    RawLocal FileSystem
  • 18. The Jira: Approx 9 months
    FileContext: Three main discussions
    Use Java’s new IO. Main issue is that their default and wd is tied to the file system of the JVM in order to provide compatibility with older APIs
    Config – a simpler per-process environment?
    Differed to Doug’s view that a per-process env is not sufficient for a MT Java application
    Besides, out of scope for this Jira – a radical change
    SS config
    What I would have liked to do, but didn’t
    2 Apis: Java Interface and Abstract class
    Not the Hadoop way!
    For example if we had done this in the old FileSystem, it would have facilitated dynamic class loading for protocol compatibility as an interim solution
  • 19. What is next?
    Adding InterruptedException and declaring the sub-exceptions of IOException
    Apps need to manage an interrupt differently then other exceptions
    IO-streams throw IOInterruptedException
    FileContext – three choices:
    An unchecked exception
    The Cache:
    Do we need it?
    How do we deal with Facebook’s application that forced the cache to leak through
    Other AbstractFileSystem impls (can use the delegator)