The HDFS Connector API provides operations for working with Hadoop Distributed File System (HDFS) such as making directories, renaming files, listing file statuses, copying files between local and HDFS, setting permissions and owners. It allows configuring HDFS and reading file contents from HDFS paths which are then streamed to the rest of the flow while adding file metadata properties.

1. HDFS Connector API

2. Make Directories
• <hdfs:make-directories>
• Make the given file and all non-existent
parents into directories. Has the semantics of
Unix 'mkdir -p'. Existence of the directory
hierarchy is not an error

3. Make Directories - Attributes
Name Java Type Description
config-ref String Specify which config to use
path String the path to create directories for.
permission String the file system permission to use
when creating the directories, either
in octal or symbolic format (umask).

4. Rename
• <hdfs:rename>
• Renames path target to path destination

5. Rename - XML Sample
• <hdfs:rename source="/tmp/my-dir"
target="/tmp/new-dir" config-ref="hdfs-
conf"/>

6. Rename - Attributes
Name Java Type Description
config-ref String Specify which config to use
source String the source path to be renamed.
target String the target new path after rename

7. Returns
Return Java Type Description
Boolean Boolean true if rename is
successful

8. List Status
• <hdfs:list-status>
• List the statuses of the files/directories in the
given path if the path is a directory

9. List Status - XML Sample
• <hdfs:list-status path="/tmp/my-dir"
filter="^.*/2014/02/$" config-ref="hdfs-
conf"/>

10. List Status - Attributes
Name Java Type Description
config-ref String Specify which config to use
path String the given path
filter String the user supplied path filter

11. List Status - Returns
Return Java Type Description
List FileStatus the statuses of the
files/directories in the given path

12. Glob Status
• <hdfs:glob-status>
• Return all the files that match file pattern and
are not checksum files. Results are sorted by
their names

13. Glob Status - XML Sample
• <hdfs:glob-status pathPattern="/tmp/*/*"
config-ref="hdfs-conf"/>

14. Glob Status - Attributes
Name Java Type Description
config-ref String Specify which config to use
pathPattern String a regular expression specifying the path
pattern.
filter PathFilter the user supplied path filter

15. Glob Status - Returns
Return Java Type Description
List FileStatus an array of paths that
match the path pattern

16. Copy From Local File
• <hdfs:copy-from-local-file>
• Copy the source file on the local disk to the
FileSystem at the given target path, set
deleteSource if the source should be removed

17. Copy From Local File - XML Sample
• <hdfs:copy-from-local-file
deleteSource="true" overwrite="false"
source="/tmp/mulesoft/"
target="/user/mulesoft/" config-ref="hdfs-
conf"/>

18. Copy From Local File -
Name Java Type Description
config-ref String Specify which config to use
deleteSource boolean whether to delete the source.
overwrite boolean whether to overwrite a existing file.
source String the source path on the local disk.
target String the target path on the File System.

19. Copy to Local File
• <hdfs:copy-to-local-file>
• Copy the source file on the FileSystem to local
disk at the given target path, set deleteSource
if the source should be removed
• useRawLocalFileSystem indicates whether to
use RawLocalFileSystem as it is a non CRC File
System

20. Copy to Local File - XML Sample
• <hdfs:copy-to-local-file
deleteSource="false"
useRawLocalFileSystem="false"
source="/tmp/mulesoft/"
target="/user/mulesoft/" config-ref="hdfs-
conf"/>

21. Copy to Local File - Attributes
Name Java Type Description
config-ref String Specify which config to use
deleteSource boolean whether to delete the source.
useRawLocal
FileSystem
boolean whether to use RawLocalFileSystem as local
file system or not.
source String the source path on the File System.
target String the target path on the local disk.

22. Set Permission
• <hdfs:set-permission>
• Set permission of a path (i.e., a file or a
directory)

23. Set Permission - XML Sample
• <!-- Set permission of a path to change. -->
<hdfs:set-permission path="/tmp/my-dir"
permission="511" config-ref="hdfs-conf"/>

24. Set Permission - Attributes
Name Java Type Description
config-ref String Specify which config to use
path String the path of the file or directory to set
permission.
permission String the file system permission to be set

25. Set Owner
• <hdfs:set-owner>
• Set owner of a path (i.e., a file or a directory).
The parameters username and groupname
cannot both be null

26. Set Owner - XML Sample
• <!-- Set owner of a path to change. -->
<hdfs:set-owner path="/tmp/my-dir"
ownername="mulesoft"
groupname="supergroup" config-ref="hdfs-
conf"/>

27. Set Owner - Attributes
Name Java Type Description
config-ref String Specify which config to use
path String the path of the file or directory to set
owner.
ownername String If it is null, the original username
remains unchanged.
groupname String If it is null, the original groupname
remains unchanged.

28. Sources
• Read From Path
• <hdfs:read>
• Read the content of a file designated by its path and
streams it to the rest of the flow, while adding the path
metadata in the following inbound properties:
– HDFSConnector#HDFS_PATH_EXISTS: a boolean set to true if the
path exists
– HDFSConnector#HDFS_CONTENT_SUMMARY: an instance of
ContentSummary if the path exists.
– HDFSConnector#HDFS_FILE_STATUS: an instance of FileStatus if
the path exists.
– HDFSConnector#HDFS_FILE_CHECKSUM: an instance of
FileChecksum if the path exists, is a file and has a checksum.

29. Sources - Attributes
Name Java Type Description
config-ref String Specify which config to use
path String the path of the file to read.
bufferSize int the buffer size to use when reading the file.
sourceCallba
ck
SourceCallb
ack
the SourceCallback used to propagate the
event to the rest of the flow

30. Question and answer