NIO.2, the I/O API for the future


Published on

This is my slides from the JavaZone conference. I presented NIO.2 and here are the slides.

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

NIO.2, the I/O API for the future

  1. 1. NIO.2: The I/O API For Future Masoud Kalali [email_address] Twitter: @MasoudKalali
  2. 2. Agenda <ul><li>Why NIO.2 </li></ul><ul><ul><li>What is lacking in the previous version </li></ul></ul><ul><li>File System API: The part deals with files as whole </li></ul><ul><ul><li>Copying a file, </li></ul></ul><ul><ul><li>Reading a file attributes </li></ul></ul><ul><ul><li>Watching a directory for changes </li></ul></ul><ul><ul><li>Etc. </li></ul></ul><ul><li>IO API </li></ul><ul><ul><li>Asynchronous IO </li></ul></ul><ul><ul><li>Other interesting features. </li></ul></ul>
  3. 3. Java I/O history from Java 1.0 to Java 7 I/O in Java 1.1 NIO (JSR-51) in JDK 1.4 NIO.2 (JSR-203) in Java 7
  4. 4. Comparing I/O before Java 7 and Java 7: Basic Features Before Java 7 <ul><li>File System part of the NIO.2 is platform specific </li></ul><ul><li>File tree walk </li></ul><ul><li>File Operations (Copy, Delete, Move, etc.) </li></ul><ul><li>Symbolic links support </li></ul><ul><li>Support for file attributes in NIO.2 </li></ul><ul><li>Support for handling file permissions </li></ul>What NIO.2 provides Java 7 <ul><li>All File System were treated the same </li></ul><ul><li>No API for walking over the file system </li></ul><ul><li>No API for file operations </li></ul><ul><li>No support for symbolic links </li></ul><ul><li>No support for reading and writing attributes </li></ul><ul><li>No support for dealing with permissions </li></ul>Current lacking IO features
  5. 5. Comparing I/O before Java 7 and Java 7: Advanced Features <ul><li>No proper exception handling </li></ul><ul><li>No watch API to handle file system changes </li></ul><ul><li>No SPI for adding support for new file systems </li></ul><ul><li>No Asynchronous I/O support </li></ul>Current lacking IO features Before Java 7 <ul><li>Proper Exception handling </li></ul><ul><li>Support for change notification </li></ul><ul><li>Provider SPI to add new file system support </li></ul><ul><li>Asynchronous I/O </li></ul>What NIO.2 provides Java 7
  6. 6. The NIO.2 Architecture <ul><li>Classes, interfaces reside in java.nio </li></ul><ul><li>New set of main classes, interfaces </li></ul><ul><ul><li>FileSystems factory, FileSystem interface: The file system doorway </li></ul></ul><ul><ul><li>Path : The object representing an existing or non-existing file or directory in the file system. </li></ul></ul><ul><ul><li>FileStore : Represents underlying storage devices, partitions and so on. </li></ul></ul><ul><ul><li>WatchService : Watching a directory or a file in the file system </li></ul></ul><ul><ul><li>Attributes and related classes: Reading and writing attributes </li></ul></ul><ul><ul><li>Permissions and related classes: Reading and writing permissions </li></ul></ul><ul><ul><li>etc </li></ul></ul>A FileSystem A FileSystemsProvider WatchService Attibutes Path FileStore Etc Through FileSystems Factory class
  7. 7. How the NIO.2 works FileSystem fs = FileSystems.getDefault(); Path p = fs.getPath(&quot;/home/masoud/simple.txt&quot;); try { p.deleteIfExists(); } catch (IOException ex) { //handle it } Basically everything start with FileSystems and FileSystem object
  8. 8. Basic File operations with Path <ul><li>Copying and moving Files </li></ul><ul><li>Attributes state after copying / moving; creating files or directories. </li></ul><ul><li>Overwriting the target file </li></ul><ul><li>Move can be atomic </li></ul>FileSystem fs = FileSystems.getDefault(); Path p = fs.getPath(&quot;/home/masoud/simple.txt&quot;); Path newCopy = fs.getPath(&quot;/home/masoud/copy of simple.txt&quot;); p.copyTo(newCopy, StandardCopyOption.COPY_ATTRIBUTES, StandardCopyOption.REPLACE_EXISTING); newCopy.moveTo(fs.getPath(&quot;/home/masoud/moved.txt&quot;), StandardCopyOption.ATOMIC_MOVE);
  9. 9. Symbolic links support <ul><li>What are Symbolic Links and hard links </li></ul><ul><li>Creating and Resolving Symbolic Links </li></ul><ul><li>Creating hard links </li></ul>FileSystem fs = FileSystems.getDefault(); Path actualFile = fs.getPath(&quot;/home/masoud/simple&quot;); //creating a symbolic link Path symLinkFile = fs.getPath(&quot;/home/masoud/simple.sym&quot;); symLinkFile.createSymbolicLink(actualFile); System.out.println(&quot;Getting the link target: &quot;+symLinkFile.readSymbolicLink()); //Creating hard link Path linkFile = fs.getPath(&quot;/home/masoud/;); linkFile.createLink(actualFile);
  10. 10. Walking The File System Tree • Files.walkFileTree * Walks a file tree from a given starting path down to the given depth * Invoke FileVisitor methods for each file/directory public class SimpleVisitor { public static void main(String args[]) { FileSystem fs = FileSystems.getDefault(); Path p = fs.getPath(&quot;/home/masoud/&quot;); MyVisit v = new MyVisit(); Files.walkFileTree(p, EnumSet.allOf(FileVisitOption.class), 2, v); } } class MyVisit extends SimpleFileVisitor<Path> { @Override public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) { System.out.println(file.getName()); return super.visitFile(file, attrs); } } PreVisitDirectory PreVisitDirectoryFailed VisitFile VisitFileFailed postVisitDirectory
  11. 11. File attributes and metadata <ul><li>Reading and Writing File attributes </li></ul><ul><li>Different views for different file systems </li></ul><ul><ul><li>Dos View: DosFileAttributeView </li></ul></ul><ul><ul><li>Posix view: PosixFileAttributeView </li></ul></ul><ul><ul><li>ACL view: AclFileAttributeView </li></ul></ul><ul><ul><li>User defined attributes view: UserDefinedFileAttributeView </li></ul></ul><ul><ul><li>Basic view: BasicFileAttributeView : Common for All FS </li></ul></ul><ul><li>Each view has a corresponding *FileAttributes object holding the attributes for a file system entry. </li></ul><ul><li>Checking for supported attributes using FileSystem.supportedFileAttributeViews() </li></ul>
  12. 12. File attributes and metadata: basic attributes BasicFileAttributes: Common to all FS and contains basic attributes. Creation Time Last Access Time Modification Time Size File Type: isDirectory, IsFile, IsSymbolicLink, IsOther File Key FileSystem fs = FileSystems.getDefault(); Path p = fs.getPath(&quot;/home/masoud/simple&quot;); BasicFileAttributeView bav= p.getFileAttributeView(BasicFileAttributeView.class, LinkOption.NOFOLLOW_LINKS); BasicFileAttributes ba =bav.readAttributes(); System.out.println(p.toString() + &quot; last access: &quot; + ba.lastAccessTime()); System.out.println(p.toString() + &quot; last modified: &quot; + ba.lastModifiedTime());
  13. 13. File attributes and metadata: Platform specific Attributes PosixFileAttributeView Extends BasicFileAttributeView, contain Posix specific attributes. FileSystem fs = FileSystems.getDefault(); Path p = fs.getPath(&quot;/home/masoud/photo.png&quot;); Set<String> supportedViews = fs.supportedFileAttributeViews(); if (supportedViews.contains(&quot;unix&quot;)) { PosixFileAttributes pat = Attributes.readPosixFileAttributes(p, LinkOption.NOFOLLOW_LINKS); System.out.println(; System.out.println(pat.owner(). } All Basic attributes Retrieving group Retrieving owner Retrieving permission basic unix dos acl owner user etc.
  14. 14. <ul><li>Using Attributes utility class methods: </li></ul><ul><ul><li>Attributes.readBasicFileAttributes(...); </li></ul></ul><ul><ul><li>Attributes.readDosFileAttributes(...); </li></ul></ul><ul><ul><li>Attributes.readPosixFileAttributes(...) </li></ul></ul><ul><li>Bulk read using Path.readAttributes(...) method: </li></ul><ul><ul><li>Map<String, ?> att = (Map<String, String>) p.readAttributes(&quot;posix:owner,basic:*&quot;); </li></ul></ul><ul><li>Using Path and AttributeViews: Path.getFileAttributeView(Class<V> type, LinkOption... options) </li></ul>Different ways for retrieving file attributes <ul><li>Comma separated list of </li></ul><ul><li>[View name: Attribute name] </li></ul><ul><li>Default view is Basic View. </li></ul><ul><li>For example: </li></ul><ul><li>basic:lastModifiedTime </li></ul><ul><li>basic:size </li></ul><ul><li>posix:group </li></ul><ul><li>dos:hidden </li></ul><ul><li>posix:* </li></ul><ul><li>fileKey -> with no view name </li></ul>BasicFileAttributeView DosFileAttributeView PosixFileAttributeView
  15. 15. File attributes and metadata: Managing POSIX files permissions <ul><li>PosixFilePermission : A posix permission </li></ul><ul><li>PosixFilePermissions : Utility class to convert permissions from between OS notation and a set of PosixFilePermission </li></ul><ul><li>Attributes : To read file attributes </li></ul><ul><li>PosixFileAttributes: includes posix file permissions </li></ul>FileSystem fs = FileSystems.getDefault(); Path p = fs.getPath(&quot;/home/masoud/;); PosixFileAttributes patts = Attributes.readPosixFileAttributes(p, LinkOption.NOFOLLOW_LINKS); Set<PosixFilePermission> st = patts.permissions(); System.out.println(PosixFilePermissions.toString(st));
  16. 16. //Setting permissions Set<PosixFilePermission> st = PosixFilePermissions.fromString(&quot;rwxrwxrwx&quot;); Attributes.setPosixFilePermissions(p, st2); //Setting the owner username, we should find the user prior to using Attributes utility class UserPrincipal up = s.getUserPrincipalLookupService().lookupPrincipalByName(&quot;masoud&quot;); Attributes.setOwner(p, up); //Another way to set the owner name. FileOwnerAttributeView fo = p.getFileAttributeView(FileOwnerAttributeView.class, LinkOption.NOFOLLOW_LINKS); fo.setOwner(up); FileOwnerView: Deal with file owner attribute Attributes: To write attributes, PosixFileAttributes includes permissions as well. File attributes and metadata: Managing POSIX files permissions
  17. 17. FileStore features The FileStore : Represent the file system underlying storage devices, pools, partitions, and so on. FileSystem fs = FileSystems.getFileSystem(new URI(&quot;File:///&quot;)); Path p = fs.getPath(&quot;/home/masoud&quot;); FileStore fstore = p.getFileStore(); FileStoreSpaceAttributes attrs = Attributes.readFileStoreSpaceAttributes(fstore); long total = attrs.totalSpace() / (1024*1024); long used = (attrs.totalSpace() - attrs.unallocatedSpace()) / (1024*1024); long avail = attrs.usableSpace() /(1024*1024); System.out.println(&quot; device size(MB) used(MB) available(MB) &quot;); System.out.format(&quot;%-20s %12d %12d %12d%n&quot;, fstore, total, used, avail); We can use FileSystem.getFileStores() to get all underlying file stores.
  18. 18. FileSystem fs = FileSystems.getDefault(); Path p = fs.getPath(&quot;/usr/bin&quot;); DirectoryStream.Filter<Path> filter = new DirectoryStream.Filter<Path>() { public boolean accept(Path file) throws IOException { String perm = PosixFilePermissions.toString(Attributes.readPosixFileAttributes(file).permissions()); if (perm.equalsIgnoreCase(&quot;rwxr-xr-x&quot;)) return true; return false; } }; DirectoryStream<Path> ds = p.newDirectoryStream(filter); for (Path p : ds) { System.out.println(p); } Directory content stream <ul><li>Can filter a directory content by file names using glob syntax </li></ul><ul><li>Can filter and walk over a directory content by file attributes. The filter interface has one method we need to implement to accept a file and return it up. </li></ul><ul><li>Scale to very large directories with tens of thousands of entries </li></ul>
  19. 19. File system change notification <ul><li>How does it works: </li></ul><ul><ul><li>Use underlying FS facilities when available. </li></ul></ul><ul><ul><li>Retreat to rudimentary polling if underlying FS does not provide change notification. </li></ul></ul><ul><li>API consists of four elements: </li></ul><ul><li>Watchable: Path implements Watchable interface so the watch service works over it. </li></ul><ul><li>A set of event types: whether we want to receive creation, deletion, … events. In our case we will use StandardWatchEventKind which implements the WatchEvent.Kind<T> . </li></ul><ul><li>The Watcher: This is the entity watching watchable. In our case the watcher ( java.nio.file.WatchService ) watches the File System for changes. </li></ul><ul><li>Watch Key: </li></ul>
  20. 20. Change notification: Sample Code FileSystem fs = FileSystems.getDefault(); WatchService ws = fs.newWatchService(); Path path = fs.getPath(&quot;/home/masoud/Pictures&quot;); path.register(ws, StandardWatchEventKind.ENTRY_CREATE, StandardWatchEventKind.ENTRY_DELETE); WatchKey key = ws.take(); List<WatchEvent<?>> events = key.pollEvents(); for (WatchEvent object : events) { if (object.kind() == StandardWatchEventKind.ENTRY_DELETE) { System.out.println(&quot;Delete: &quot; + path.toRealPath(true)+&quot;/&quot;+ object.context().toString()); } if (object.kind() == StandardWatchEventKind.ENTRY_CREATE) { System.out.println(&quot;Created: &quot; + path.toRealPath(true)+&quot;/&quot;+ object.context().toString()); } }
  21. 21. Provider interface <ul><li>Plug a new FS implementation just by adding the Jar file to the runtime classpath. </li></ul><ul><li>The implementation archive must include the implementation descriptor file (META-INF/services/java.nio.file.spi.FileSystemProvider containing the FQN of the implementation class) </li></ul><ul><li>Custom file systems, zip, jar, in memory, etc... </li></ul><ul><li>We just need to implement or extend some contracts including: </li></ul><ul><ul><ul><ul><ul><li>java.nio.file.spi.FileSystemProvider </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>java.nio.file.Path </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>java.nio.file.DirectoryStream and java.nio.file.DirectoryEntry if we need content iterator. </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>java.nio.file.attribute.BasicFileAttributeView and java.nio.file.attribute.BasicFileAttrbutes if we need attributes support. </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Other interfaces or abstract classes like WatchService can be implemented if required. </li></ul></ul></ul></ul></ul>
  22. 22. Blocking, Non-Blocking and Asynchronous I/O Prior to Java SE 1.4 : Everything was blocking and and use of threads was inevitable. ( which causes scalability issues and performance overhead) Java SE 1.4 : Brings the JSR-51 or NIO with non-blocking IO; Introduction of channels and selectors. Non-Blocking not Asynchronous. Java SE 7 : Addition of asynchronous IO to the pack; included inside java.nio.channels
  23. 23. Asynchronous I/O for sockets and Files <ul><li>What is Asynchronous I/O </li></ul><ul><ul><li>No blocking, no quick pass over the buffer, simple manual check backs or call back handlers (completion handlers). </li></ul></ul><ul><li>Why Asynchronous I/O </li></ul><ul><ul><li>More scalability, benefiting from underlying OS IO features. </li></ul></ul><ul><li>How Asynchronous I/O works </li></ul><ul><ul><li>Using java.util.concurrent.Future : Manual check backs. </li></ul></ul><ul><ul><li>Using java.nio.channels.CompletionHandler : Callback handler for completion or failure of IO operation </li></ul></ul>
  24. 24. Sample code for Asynchronous I/O: Using Future FileSystem fs = FileSystems.getDefault(); Path path = fs.getPath (&quot;/home/masoud/a.png&quot;); AsynchronousFileChannel ch = (path); ByteBuffer buf = ByteBuffer.allocate (2048); Future<Integer> result = (buf,0); while (!result.isDone ()) { System.out.println (&quot;Doing something else&quot;); Thread.sleep (500); } System.out.println (&quot;Bytes we read: &quot;+result.get ()); ch.close (); The Future object allows us to check on the status of the asynchronous job when we want to get back at it.
  25. 25. Sample code for Asynchronous I/O: Using Callback Path path = Paths.get(&quot;/home/masoud/series.txt&quot;); AsynchronousFileChannel ch =; final ByteBuffer buf = ByteBuffer.allocate(2048);, 0, buf, new SimpleHandler()); class SimpleHandler implements CompletionHandler<Integer, ByteBuffer> { public void completed(Integer result, ByteBuffer attachment) { System.out.println(new String(attachment.array())); } public void failed(Throwable th, ByteBuffer attachment) { System.out.println(th.toString()); } } interface CompletionHandler<V,A> Attachment is something to carry a context object into the handler. Value number of bytes we read or write. read(ByteBuffer dst, long position, A attachment, CompletionHandler<Integer,? super A> handler) interface CompletionHandler<V,A> Attachment is something to carry a context object into the handler. Value number of bytes we read or write.
  26. 26. Different types of Asynchronous channels AsynchronousChannel AsynchronousByteChannel AsynchronousFileChannel <ul><li>Asynchronous read/write </li></ul><ul><li>No global file position/offset </li></ul><ul><ul><li>Each read/write specifies position in file </li></ul></ul><ul><ul><li>Access different parts of file concurrently </li></ul></ul><ul><li>Open method specifies options: READ, WRITE, APPEND... </li></ul>AsynchronousSocketChannel <ul><li>Asynchronous connect </li></ul><ul><li>Asynchronous read/write </li></ul><ul><li>Asynchronous scatter/gather (multiple buffers) </li></ul><ul><li>Read/write operations support timeout: failed method invoked with timeout exception </li></ul><ul><li>Implements NetworkChannel for binding, setting socket options, etc. </li></ul>AsynchronousServerSocketChannel <ul><li>Asynchronous accept </li></ul><ul><ul><li>handler invoked when connection accepted </li></ul></ul><ul><ul><li>Result is AsynchronousSocketConnection </li></ul></ul><ul><li>Implements NetworkChannel </li></ul><ul><ul><li>for binding, setting socket options, etc. </li></ul></ul>AsynchronousDatagramChannel <ul><li>Asynchronous read/write (connected) </li></ul><ul><li>Asynchronous receive/send (unconnected) </li></ul><ul><li>Implements NetworkChannel </li></ul><ul><ul><li>for binding, setting socket options, etc. </li></ul></ul><ul><li>Implements MulticastChannel </li></ul><ul><ul><li>to join multicast groups </li></ul></ul>
  27. 27. Other JSR 203 features The method File type MIME detection using Files.probeContentType
  28. 28. More Information OpenJDK New I/O Project – Questions?
  29. 29. Thank you Thank you for joining this session Masoud Kalali Twitter: @MasoudKalali [email_address]