S3A is the Hadoop filesystem client for Amazon S3. It allows Hadoop to access data stored in S3 as if it were a normal filesystem. The document discusses issues encountered with S3A such as intermittent failures when copying data from local to S3 due to cached 404 responses from S3, workarounds tried including increasing timeout values, and fixes committed including avoiding HEAD checks and improving retry policies. It also summarizes new features in S3A for Hadoop 3.3 such as delegation tokens, support for object versions and etags, and an S3Guard fsck tool.

1. 404 Not found
Gabor Bota
Sw Engineer at Cloudera
April 2020.
and other S3A bits in hadoop 3.3

2. 404 Coffee - Puerto Rico

3. S3A and Filesystems
HADOOP client
FileSystem API
Spark Hive HBase
HDFS
...
Azure AWS S3
Google cloud
storage

4. S3A, S3Guard
● S3: not a real FS
● List&co consistency: S3Guard
● HBoss
Hadoop
S3A
Client
AWS
DynamoDB
AWS
S3

6. S3Guard no longer experimental
HADOOP-14936 Resolved:
● Rename work
● On-demand ddb tables
● Etags/versions for read inconsistency
● Metadata expiry (OOB deletions and creation)
● Authoritative (exclusive) mode for path
● Fsck
● and many other fixes, improvements.

7. 404 (Not Found) - Bristol, UK

8. What’s not found?
● QE: “Input/output error while copying data from local to S3 folder”
● Issue with command: hdfs dfs -copyFromLocal
● Nightly test failure
● Intermittent failures
● Turning S3Guard on did not help

9. What’s not found?
# hdfs dfs -copyFromLocal /someData s3a://qe-bucket/st-hdfs-190612-085408-
s7z/user/hrt_1/
copyFromLocal: rename `s3a://qe-bucket/SF100_05.txt._COPYING_' to `s3a://qe-
bucket/SF100_05.txt': Input/output error

10. What’s not found?
More logging...
10:18:24,181 HEAD jar1.jar._COPYING (S3AFS.create) -> 404
10:18:24,944 HEAD jar1.jar._COPYING_ (deleteOnExit() registration) -> 404
10:18:25,733 PUT jar1.jar._COPYING_ (put file to S3A bucket) ->200
10:18:26,740 HEAD jar1.jar._COPYING_ (as out of band check at the start of
rename) -> 404
10:18:27,411 HEAD jar1.jar._COPYING_ (as preamble to COPY) -> 404
...repeated until 10:18:38,939 and final failure

11. What happened?
● S3 caches negative response
● It’s in the cache / load balancers for a while
● Every HEAD/GET refreshes the 404 in the cache
● How long - hours? No official answer in docs.
● Based on our tests it’s short amount (minutes)

12. Workaround #1
● Hotfix: increase timeout for operations. (fs.s3a.retry.limit and
fs.s3a.retry.interval)
● Overall performance impacted negatively
● Easy, config level
● Emergency workaround
● Linear waiting
● Stopped the test failing.

13. Workaround #2
HADOOP-16499. S3A retry policy to be exponential
● Add a non-linear policy
● Structured workaround
● Really it’s proportional
○ RetryUpToMaximumCountWithProportionalSleep
● Still one shared retry policy for everything

14. Fixing it
HADOOP-16490. Avoid/handle cached 404s during S3A file creation
● Better exception propagation
● Don’t do HEAD checks when creating files with overwrite=true
● Shell commands to avoid deleteOnExit calls (exists() checks)
● How would you test this?
○ Mocking - simulate getting back 404 util it’s 200.

15. Issues
1. Basic commit protocols (like distcp) use commit with rename in the end. This is
not good for cloud storages.
2. the -d option wasn't used in copyFromLocal, so the file is copied to a temp file
and then renamed.
3. S3Guard is off. So when rename() looks for a file and it isn't found, it doesn't
retry to see if it turns up. We use different policies for retry with S3Guard.

16. Recommendations
● Use -d for copyFromLocal uploads, maybe even -t for more threads on the
(slow) upload
● Use -direct for distcp for cloud storages!
● Spark: S3AStagingCommitter
● Turn S3Guard on.
Follow-up: HADOOP-16484 S3Guard: fail or warn if turned off!

17. 404 Coffee Shop - Wisconsin, US

18. New in S3A Hadoop 3.3
● Delegation Token
○ AWS STS is used to request short-lived session tokens
● Support for etags and versioning
○ Credit for Cerner (Ben Roling)
● OpenSSL support via WildFly
● S3Select (cli tool, api)
● Delete: thundering herd.
○ Problem: Delete 1000 object at a time.
○ Don’t try the same request - split up to smaller page sizes.
● Rename parallelized, failure handling
○ DDB is updated even when failing partially

19. S3Guard FSCK
● CLI command
● hadoop s3guard fsck [-check | -internal] [-fix] (s3a://BUCKET |
s3a://PATH_PREFIX)
● Check: S3->DDB
● Internal: DDB internal consistency of DDB
● Fix: remove orphan entries

20. S3Guard FSCK
● It’s in hadoop 3.3
● Good starting point to get into
S3A and S3Guard
● Contributions welcome

21. ?What’s next
● Directory marker problems: HADOOP-13230
● New stat interface: IOStatistics
● Vectored IO
○ We’d like to do it across the stack

22. Takeaways
● CDP QE is constantly working to find out issues before customers/users do.
● Fast turnaround for updates in upstream
○ Only a few weeks
○ We add incremental PRs instead of huge ones.
● Hadoop 3.3.0 has branched
○ We’ll be releasing soon
○ Come and help test the new release
○ It’s better to find the bugs before we ship

23. Q/A