This document describes how the HDFS client uses the NameNode's audit log and DFSInotifyEventInputStream to monitor file system changes since its last poll. The client caches the highest event ID it has seen and periodically polls the NameNode for any events with higher IDs. The NameNode assigns monotonically increasing IDs to each event in its audit log, which also gets replayed as events through the DFSInotifyEventInputStream that clients can read to learn about changes like renames, deletes, metadata updates etc.

2. ●
●
●

3. ●
●
●
●
●
●

5. DFSClient
DFSClient
DFSClient
/user/cmccabe/tpcds_1
/user/cmccabe/reports
/user/awang/reports

6. ●
●
●
●
●

7. ●
●
●
●

8. ●
○
○
●
○

9. ●
●
●

10. ●
●
○
●
○

11. ●
Client
HDFS
create /a
delete /b
...

12. ●
○
○ …
■ …

13. ●
●
○

14. ●
○
○

15. ●
○
●
●

16. NameNodeClient
Any events
after #123?
Event #124,
#125, etc.
Client caches highest
event number that it’s
seen.
Client polls periodically.
NameNode uses
monotonically increasing
64-bit event IDs

17. ●
●
●

18. HDFS Audit Log
123 AddCloseOp(file=/a)
124 DeleteOp(file=/b)
125 SetReplicationOp(file=/a,
rep=5)
126 AddCloseOp(file=/c)
127 RenameOp(src=/c, dst=/d)
128 ConcatOp(dst=/x, src1=/y,
src2=/z)
...
DFSINotifyInputStream
123 CreateEvent
124 UnlinkEvent
125 MetadataUpdateEvent
126 CloseEvent
127 RenameEvent
128 [AppendEvent /x,
UnlinkEvent /y,
UnlinkEvent /z]
...

19. ●
●
○
○
●

20. message RenameEventProto {
required string srcPath = 1;
required string destPath = 2;
required int64 timestamp = 3;
}

21. DFSInotifyEventInputStream stream =
dfs.getInotifyEventStream(prevHighestTxId);
EventBatch batch = stream.take();
long newTxid = batch.getTxid();
switch (batch.getEvents()[0].getEventType()) {
case Event.EventType.RENAME:
…
}
stream.close();

22. ●
○
○
○
●
○

23. ●
○
●
○
○
○

24. ●
●
● …
●

25. ●
●
○

26. ●
●
●

27. ●
●
●
●

28. ●
●
●