Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

S3Guard: What's in your consistency model?

1,631 views

Published on

S3Guard: What's in your consistency model?

Published in: Technology
  • Be the first to comment

S3Guard: What's in your consistency model?

  1. 1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved S3Guard: What’s in Your Consistency Model? Mingliang Liu @liuml07 Steve Loughran @steveloughran December 2016
  2. 2. Steve Loughran Hadoop committer & PMC, ASF Member Mingliang Liu Apache Hadoop committer Chris Nauroth, Hadoop committer & PMC, ASF member Rajesh Balamohan Tez Committer & PMC
  3. 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved S3A: Hadoop File System for S3 (EMR: use Amazon's s3:// )
  4. 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Storage Use Evolution HDFS Application HDFS Application GoalEvolution towards cloud storage as the primary Data Lake Input Output Backup Restore Input Output Copy HDFS Application Input Output tmp
  5. 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved org.apache.hadoop.fs.FileSystem hdfs s3awasb adlswift gs Hadoop File System - One Interface Fits All
  6. 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved / work pending part-00 part-01 00 00 00 01 01 01 complete part-01 rename("/work/pending/part-01", "/work/complete") A FileSystem: Directories, Files  Data
  7. 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved S3A: Object Store Pretending A FileSystem  Cloud Object Stores designed for – Scale – Cost – Geographic Distribution – Availability  Cloud apps dedicatedly deal with cloud storage semantics and limitations  Hadoop apps should work on cloud storage transparently – S3A partially adheres to the FileSystem specification
  8. 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved 00 00 00 01 01 s01 s02 s03 s04 hash("/work/pending/part-01") ["s02", "s03", "s04"] 01 01 01 01 hash("/work/pending/part-00") ["s01", "s02", "s04"] hash(name)->blob
  9. 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved What Is The Problem?  Performance – separated from compute – cloud storage not designed for file-like access patterns  Limitations in APIs – delete(path, recursive=true) – rename(source, dest)  Eventual consistency – Create Consistency – Update – Delete – Listing • take time to list created objects • lag in changed metadata about existing objects • lag in observing deleted objects
  10. 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved 00 00 00 01 01 s01 s02 s03 s04 hash("/work/pending/part-01") ["s02", "s03", "s04"] copy("/work/pending/part-01", "/work/complete/part01") 01 01 01 01 delete("/work/pending/part-01") hash("/work/pending/part-00") ["s01", "s02", "s04"] rename(): A Series of Operations on The Client
  11. 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Eventual Consistency From FileSystem’s View  When listing "a directory” – Newly created files may not yet be visible, deleted ones still present  After updating an object – Opening and reading the object may still return the previous data  After deleting an object – Opening the object may succeed, returning the data  While reading an object – If object is updated or deleted during the process
  12. 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved 00 00 00 01 01 s01 s02 s03 s04 01 DELETE /work/pending/part-00 HEAD /work/pending/part-00 GET /work/pending/part-00 200 200 200 Eventually Consistent – Seeing Deleted Data
  13. 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved S3Guard: Fast, Consistent S3 Metadata (EMR: use Amazon's EMRFS)
  14. 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved S3Guard: Fast, Consistent S3 Metadata  Inspired by Apache licensed S3mper project from Netflix  Using DynamoDB as the consistent metadata store  Mutating file system operations – Update both S3 and DynamoDB  Read operations – Return results to callers as sourced from S3 – First check their results against the metadata in DynamoDB – S3A waits and rechecks both S3 and DynamoDB until they agree  Goals – Provide consistent list and get status operations on S3 objects written with S3Guard enabled • listStatus() after put and delete • getFileStatus() after put and delete – Provide tools to manage associated metadata and caching policies. – Configurable error handling when inconsistency is detected – Performance improvements that impact real workloads.
  15. 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved 00 00 00 01 01 s01 s02 s03 s04 01 DELETE part-00 200 HEAD part-00 200 HEAD part-00 404 PUT part-00 200 00 DynamoDB As The Consistent Metadata Store
  16. 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Demo
  17. 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved https://issues.apache.org/jira/browse/HADOOP-13345
  18. 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved© Hortonworks Inc. 2011 – 2016. All Rights Reserved18 Questions?
  19. 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Backup Slides
  20. 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved 00 00 00 01 01 s01 s02 s03 s04 HEAD /work/complete/part-01 PUT /work/complete/part01 x-amz-copy-source: /work/pending/part-01 01 DELETE /work/pending/part-01 PUT /work/pending/part-01 ... DATA ... GET /work/pending/part-01 Content-Length: 1-8192 GET /?prefix=/work&delimiter=/ REST APIs

×