Tips & Tricks
Adding ACID Transactions, Inserts, Updates, and Deletes in Apache Hive
Like this presentation? Why not share!
Adding ACID Transactions, Inserts, ...
by Hadoop Summit
Adding ACID Updates to Hive
by Owen O'Malley
Hive & HBase for Transaction Proces...
Hortonworks Technical Workshop: Int...
Hortonworks Technical Workshop: Rea...
Email sent successfully!
Show related SlideShares at end
Adding ACID Transactions, Inserts, Updates, and Deletes in Apache Hive
Jun 17, 2014
Comment goes here.
12 hours ago
Are you sure you want to
Your message goes here
Be the first to comment
10 months ago
Senior Big Data Architect
1 year ago
Number of Embeds
No notes for slide
Transcript of "Adding ACID Transactions, Inserts, Updates, and Deletes in Apache Hive"
1. © Hortonworks Inc. 2014 Adding ACID Updates to Hive April 2014 Page 1 Owen O’Malley Alan Gates email@example.com firstname.lastname@example.org @owen_omalley @alanfgates
© Hortonworks Inc. 2014 Page 2 •Hive Only Updates Partitions –Insert overwrite rewrites an entire partition –Forces daily or even hourly partitions •What Happens to Concurrent Readers? –Ok for inserts, but overwrite causes races –There is a zookeeper lock manager, but… •No way to delete, update, or insert rows –Makes ad hoc work difficult What’s Wrong?
© Hortonworks Inc. 2014 Page 3 •Hadoop and Hive have always… –Worked without ACID –Perceived as tradeoff for performance •But, your data isn’t static –It changes daily, hourly, or faster –Ad hoc solutions require a lot of work –Managing change makes the user’s life better •Need consistent views of changing data! Why is ACID Critical?
© Hortonworks Inc. 2014 Page 4 •Updating a Dimension Table –Changing a customer’s address •Delete Old Records –Remove records for compliance •Update/Restate Large Fact Tables –Fix problems after they are in the warehouse •Streaming Data Ingest –A continual stream of data coming in –Typically from Flume or Storm Use Cases
© Hortonworks Inc. 2014 Page 5 •Mutiple statement transactions –Group statements that need to work together •Query tables as they appeared in past –Configurable length of history •Row-level lineage –Track users and queries that updated each row Longer Term Use Cases
© Hortonworks Inc. 2014 Page 6 •HDFS Does Not Allow Arbitrary Writes –Store changes as delta files –Stitched together by client on read •Writes get a Transaction ID –Sequentially assigned by Metastore •Reads get Committed Transactions –Provides snapshot consistency –No locks required –Provide a snapshot of data from start of query Design
© Hortonworks Inc. 2013 Stitching Buckets Together Page 7
© Hortonworks Inc. 2014 Page 8 •Partition locations remain unchanged –Still warehouse/$db/$tbl/$part •Bucket Files Structured By Transactions –Base files $part/base_$tid/bucket_* –Delta files $part/delta_$tid_$tid/bucket_* •Minor Compactions merge deltas –Read delta_$tid1_$tid1 .. delta_$tid2_$tid2 –Written as delta_$tid1_$tid2 •Compaction doesn’t disturb readers HDFS Layout
© Hortonworks Inc. 2014 Page 9 •Created new AcidInput/OutputFormat –Unique key is transaction, bucket, row •Reader returns most recent update •Also Added Raw API for Compactor –Provides previous events as well •ORC implements new API –Extends records with change metadata –Add operation (d, u, i), transaction and key Input and Output Formats
© Hortonworks Inc. 2014 Page 10 •Existing lock managers –In memory - not durable –ZooKeeper - requires additional components to install, administer, etc. •Locks need to be integrated with transactions –commit/rollback must atomically release locks •We sort of have this database lying around which has ACID characteristics (metastore) •Transactions and locks stored in metastore •Uses metastore DB to provide unique, ascending ids for transactions and locks Transaction Manager
© Hortonworks Inc. 2014 Page 11 •No explicit transactions in 0.13 –First implementation of INSERT, UPDATE, DELETE will be auto-commit –Will then add BEGIN, COMMIT, ROLLBACK •Snapshot isolation –Reader will see consistent data for the duration of his/her query –May extend to other isolation levels in the future •Current transactions can be displayed using new SHOW TRANSACTIONS statement Transaction Model
© Hortonworks Inc. 2014 Page 12 •Three types of locks –shared –semi-shared (can co-exist with shared, but not other semi-shared) –exclusive •Operations require different locks –SELECT, INSERT – shared –UPDATE, DELETE – semi-shared –DROP, INSERT OVERWRITE – exclusive Locking Model
© Hortonworks Inc. 2014 Page 13 •Each transaction (or batch of transactions in streaming ingest) creates a new delta file •Too many files = NameNode •Need a way to –Collect many deltas into one delta – minor compaction –Rewrite base and delta to new base – major compaction Compactor
© Hortonworks Inc. 2014 Page 14 •Run when there are 10 or more deltas (configurable) •Results in base + 1 delta Minor Compaction /hive/warehouse/purchaselog/ds=201403311000/base_0028000 /hive/warehouse/purchaselog/ds=201403311000/delta_0028001_0028100 /hive/warehouse/purchaselog/ds=201403311000/delta_0028101_0028200 /hive/warehouse/purchaselog/ds=201403311000/delta_0028201_0028300 /hive/warehouse/purchaselog/ds=201403311000/delta_0028301_0028400 /hive/warehouse/purchaselog/ds=201403311000/delta_0028401_0028500 /hive/warehouse/purchaselog/ds=201403311000/base_0028000 /hive/warehouse/purchaselog/ds=201403311000/delta_0028001_0028500
© Hortonworks Inc. 2014 Page 15 •Run when deltas are 10% the size of base (configurable) •Results in new base Major Compaction /hive/warehouse/purchaselog/ds=201403311000/base_0028000 /hive/warehouse/purchaselog/ds=201403311000/delta_0028001_0028100 /hive/warehouse/purchaselog/ds=201403311000/delta_0028101_0028200 /hive/warehouse/purchaselog/ds=201403311000/delta_0028201_0028300 /hive/warehouse/purchaselog/ds=201403311000/delta_0028301_0028400 /hive/warehouse/purchaselog/ds=201403311000/delta_0028401_0028500 /hive/warehouse/purchaselog/ds=201403311000/base_0028500
© Hortonworks Inc. 2014 Page 16 •Metastore thrift server will schedule and execute compactions –No need for user to schedule –User can initiate via new ALTER TABLE COMPACT statement •No locking required, compactions run at same time as select, inserts –Compactor aware of readers, does not remove old files until readers have finished with them •Current compactions can be viewed via new SHOW COMPACTIONS statement Compactor Continued
© Hortonworks Inc. 2014 Page 17 •Data is flowing in from generators in a stream •Without this, you have to add it to Hive in batches, often every hour –Thus your users have to wait an hour before they can see their data •New interface in hive.hcatalog.streaming lets applications write small batches of records and commit them –Users can now see data within a few seconds of it arriving from the data generators •Available for Apache Flume in HDP 2.1 –Working on Apache Storm integration Application, Streaming Ingest
© Hortonworks Inc. 2014 Page 18 Streaming Ingest Illustrated Flume Agent HDFS
© Hortonworks Inc. 2014 Page 19 Streaming Ingest Illustrated Flume Agent HDFS while (…) write(); commit(); Commit can be time based or size based, up to writer commit() flushes to disk and sends commit to metastore
© Hortonworks Inc. 2014 Page 20 Streaming Ingest Illustrated Flume Agent HDFS while (…) write(); commit(); Next write() appends to the same file
© Hortonworks Inc. 2014 Page 21 Streaming Ingest Illustrated Flume Agent HDFS while (…) write(); commit(); Reader Task Reader uses txnid to determine which records to read
© Hortonworks Inc. 2014 Page 22 • Phase 1, Hive 0.13 –Transaction and new lock manager –ORC file support –Automatic and manual compaction –Snapshot isolation –Streaming ingest via Flume • Phase 2, Hive 0.14 (we hope) –INSERT … VALUES, UPDATE, DELETE –BEGIN, COMMIT, ROLLBACK • Future (all speculative based on user feedback) –Versioned or point in time queries –Additional isolation levels such as dirty read or read committed –MERGE Phases of Development
© Hortonworks Inc. 2014 Page 23 •Only suitable for data warehousing, not for OLTP •Table must be bucketed, and (currently) not sorted –Sorting restriction will be removed in the future Limitations
© Hortonworks Inc. 2014 Page 24 •Good –Handles compactions for us –Already has similar data model with LSM •Bad –No cross row transactions –Would require us to write a transaction manager over HBase, doable, but not less work –Hfile is column family based rather than columnar –HBase focused on point lookups and range scans –Warehousing tends to require full scans Why Not HBase?
© Hortonworks Inc. 2014 Page 25 •JIRA: https://issues.apache.org/jira/browse/HI VE-5317 •Adds ACID semantics to Hive •Uses SQL standard commands –INSERT, UPDATE, DELETE •Provides scalable read and write access Conclusion
© Hortonworks Inc. 2013 Thank You! Questions & Answers Page 26