0
Adding ACID Transactions, Inserts,
Updates and Deletes in Apache Hive
Owen O’Malley and Alan Gates
Hortonworks
© Hortonworks Inc. 2014
Adding ACID Updates to Hive
April 2014
Owen O’Malley Alan Gates
owen@hortonworks.com gates@hortonw...
© Hortonworks Inc. 2014
•Hive Only Updates Partitions
–Insert overwrite rewrites an entire partition
–Forces daily or even...
© Hortonworks Inc. 2014
•Hadoop and Hive have always…
–Worked without ACID
–Perceived as tradeoff for performance
•But, yo...
© Hortonworks Inc. 2014
•Updating a Dimension Table
–Changing a customer’s address
•Delete Old Records
–Remove records for...
© Hortonworks Inc. 2014
•HDFS Does Not Allow Arbitrary Writes
–Store changes as delta files
–Stitched together by client o...
© Hortonworks Inc. 2013
Stitching Buckets Together
© Hortonworks Inc. 2014
•Partition locations remain unchanged
–Still warehouse/$db/$tbl/$part
•Bucket Files Structured By ...
© Hortonworks Inc. 2014
•Created new AcidInput/OutputFormat
–Unique key is transaction, bucket, row
•Reader returns most r...
© Hortonworks Inc. 2014
•Need to split buckets for MapReduce
–Need to split base and deltas the same way
–Use key ranges
–...
© Hortonworks Inc. 2014
•Existing lock managers
–In memory - not durable
–ZooKeeper - requires additional components to
in...
© Hortonworks Inc. 2014
•No explicit transactions in first release
–Future releases will have them
•Snapshot isolation
–Re...
© Hortonworks Inc. 2014
•Three types of locks
–shared
–semi-shared (can co-exist with shared, but not
other semi-shared)
–...
© Hortonworks Inc. 2014
•Each transaction (or batch of
transactions in streaming ingest)
creates a new delta
•Too many fil...
© Hortonworks Inc. 2014
•Run when there are 10 or more deltas
(configurable)
•Results in base + 1 delta
Minor Compaction
/...
© Hortonworks Inc. 2014
•Run when deltas are 10% the size of
base (configurable)
•Results in new base
Major Compaction
/hi...
© Hortonworks Inc. 2014
•Metastore thrift server will schedule and
execute compactions
–No need for user to schedule
–User...
© Hortonworks Inc. 2014
•Phase 1, Hive 0.13
–Transaction and new lock manager
–ORC file support
–Automatic and manual comp...
© Hortonworks Inc. 2014
•Only suitable for data warehousing, not
for OLTP
•Table must be bucketed, and (currently)
not sor...
© Hortonworks Inc. 2014
•JIRA:
https://issues.apache.org/jira/browse/HI
VE-5317
•Adds ACID semantics to Hive
•Uses SQL sta...
© Hortonworks Inc. 2013
Thank You!
Questions & Answers
Adding ACID Transactions, Inserts, Updates, and Deletes in Apache Hive
Upcoming SlideShare
Loading in...5
×

Adding ACID Transactions, Inserts, Updates, and Deletes in Apache Hive

3,612

Published on

Published in: Technology
0 Comments
10 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,612
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
135
Comments
0
Likes
10
Embeds 0
No embeds

No notes for slide

Transcript of "Adding ACID Transactions, Inserts, Updates, and Deletes in Apache Hive"

  1. 1. Adding ACID Transactions, Inserts, Updates and Deletes in Apache Hive Owen O’Malley and Alan Gates Hortonworks
  2. 2. © Hortonworks Inc. 2014 Adding ACID Updates to Hive April 2014 Owen O’Malley Alan Gates owen@hortonworks.com gates@hortonworks.com @owen_omalley @alanfgates
  3. 3. © Hortonworks Inc. 2014 •Hive Only Updates Partitions –Insert overwrite rewrites an entire partition –Forces daily or even hourly partitions •What Happens to Concurrent Readers? –Ok for inserts, but overwrite causes races –There is a zookeeper lock manager, but… •No way to delete, update, or insert rows –Makes adhoc work difficult What’s Wrong?
  4. 4. © Hortonworks Inc. 2014 •Hadoop and Hive have always… –Worked without ACID –Perceived as tradeoff for performance •But, your data isn’t static –It changes daily, hourly, or faster –Ad hoc solutions require a lot of work –Managing change makes the user’s life better •Do or Do Not, There is NO Try Why is ACID Critical?
  5. 5. © Hortonworks Inc. 2014 •Updating a Dimension Table –Changing a customer’s address •Delete Old Records –Remove records for compliance •Update/Restate Large Fact Tables –Fix problems after they are in the warehouse •Streaming Data Ingest –A continual stream of data coming in –Typically from Flume or Storm Use Cases
  6. 6. © Hortonworks Inc. 2014 •HDFS Does Not Allow Arbitrary Writes –Store changes as delta files –Stitched together by client on read •Writes get a Transaction ID –Sequentially assigned by Metastore •Reads get Committed Transactions –Provides snapshot consistency –No locks required –Provide a snapshot of data from start of query Design
  7. 7. © Hortonworks Inc. 2013 Stitching Buckets Together
  8. 8. © Hortonworks Inc. 2014 •Partition locations remain unchanged –Still warehouse/$db/$tbl/$part •Bucket Files Structured By Transactions –Base files $part/base_$tid/bucket_* –Delta files $part/delta_$tid_$tid/bucket_* •Minor Compactions merge deltas –Read delta_$tid1_$tid1 .. delta_$tid2_$tid2 –Written as delta_$tid1_$tid2 •Compaction doesn’t disturb readers HDFS Layout
  9. 9. © Hortonworks Inc. 2014 •Created new AcidInput/OutputFormat –Unique key is transaction, bucket, row •Reader returns most recent update •Also Added Raw API for Compactor –Provides previous events as well •ORC implements new API –Extends records with change metadata –Add operation (d, u, i), transaction and key Input and Output Formats
  10. 10. © Hortonworks Inc. 2014 •Need to split buckets for MapReduce –Need to split base and deltas the same way –Use key ranges –Use indexes Distributing the Work
  11. 11. © Hortonworks Inc. 2014 •Existing lock managers –In memory - not durable –ZooKeeper - requires additional components to install, administer, etc. •Locks need to be integrated with transactions –commit/rollback must atomically release locks •We sort of have this database lying around which has ACID characteristics (metastore) •Transactions and locks stored in metastore •Uses metastore DB to provide unique, ascending ids for transactions and locks Transaction Manager
  12. 12. © Hortonworks Inc. 2014 •No explicit transactions in first release –Future releases will have them •Snapshot isolation –Reader will see consistent data for the duration of his/her query –May extend to other isolation levels in the future •Current transactions can be displayed using new SHOW TRANSACTIONS statement Transaction Model
  13. 13. © Hortonworks Inc. 2014 •Three types of locks –shared –semi-shared (can co-exist with shared, but not other semi-shared) –exclusive •Operations require different locks –SELECT, INSERT – shared –UPDATE, DELETE – semi-shared –DROP, INSERT OVERWRITE – exclusive Locking Model
  14. 14. © Hortonworks Inc. 2014 •Each transaction (or batch of transactions in streaming ingest) creates a new delta •Too many files = NameNode  •Need a way to –Collect many deltas into one delta – minor compaction –Rewrite base and delta to new base – major compaction Compactor
  15. 15. © Hortonworks Inc. 2014 •Run when there are 10 or more deltas (configurable) •Results in base + 1 delta Minor Compaction /hive/warehouse/purchaselog/ds=201403311000/base_0028000 /hive/warehouse/purchaselog/ds=201403311000/delta_0028001_0028100 /hive/warehouse/purchaselog/ds=201403311000/delta_0028101_0028200 /hive/warehouse/purchaselog/ds=201403311000/delta_0028201_0028300 /hive/warehouse/purchaselog/ds=201403311000/delta_0028301_0028400 /hive/warehouse/purchaselog/ds=201403311000/delta_0028401_0028500 /hive/warehouse/purchaselog/ds=201403311000/base_0028000 /hive/warehouse/purchaselog/ds=201403311000/delta_0028001_0028500
  16. 16. © Hortonworks Inc. 2014 •Run when deltas are 10% the size of base (configurable) •Results in new base Major Compaction /hive/warehouse/purchaselog/ds=201403311000/base_0028000 /hive/warehouse/purchaselog/ds=201403311000/delta_0028001_0028100 /hive/warehouse/purchaselog/ds=201403311000/delta_0028101_0028200 /hive/warehouse/purchaselog/ds=201403311000/delta_0028201_0028300 /hive/warehouse/purchaselog/ds=201403311000/delta_0028301_0028400 /hive/warehouse/purchaselog/ds=201403311000/delta_0028401_0028500 /hive/warehouse/purchaselog/ds=201403311000/base_0028500
  17. 17. © Hortonworks Inc. 2014 •Metastore thrift server will schedule and execute compactions –No need for user to schedule –User can initiate via new ALTER TABLE COMPACT statement •No locking required, compactions run at same time as select, inserts –Compactor aware readers, does not remove old files until readers have finished with them •Current compactions can be viewed via new SHOW COMPACTIONS statement Compactor Continued
  18. 18. © Hortonworks Inc. 2014 •Phase 1, Hive 0.13 –Transaction and new lock manager –ORC file support –Automatic and manual compaction –Snapshot isolation •Phase 2, Hive 0.14 (we hope) –INSERT … VALUES, UPDATE, DELETE –BEGIN, COMMIT, ROLLBACK •Future (all speculative based on user fedback) –Additional isolation levels such as dirty read or read committed –MERGE Phases of Development
  19. 19. © Hortonworks Inc. 2014 •Only suitable for data warehousing, not for OLTP •Table must be bucketed, and (currently) not sorted –Sorting restriction will be removed in the future Limitations
  20. 20. © Hortonworks Inc. 2014 •JIRA: https://issues.apache.org/jira/browse/HI VE-5317 •Adds ACID semantics to Hive •Uses SQL standard commands –INSERT, UPDATE, DELETE •Provides scalable read and write access Conclusion
  21. 21. © Hortonworks Inc. 2013 Thank You! Questions & Answers
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×