Adding ACID Transactions, Inserts, Updates, and Deletes in Apache Hive

© Hortonworks Inc. 2014
Adding ACID Updates to Hive
April 2014
Page 1
Owen O’Malley Alan Gates
owen@hortonworks.com gates@hortonworks.com
@owen_omalley @alanfgates

Page 2
•Hive Only Updates Partitions
–Insert overwrite rewrites an entire partition
–Forces daily or even hourly partitions
•What Happens to Concurrent Readers?
–Ok for inserts, but overwrite causes races
–There is a zookeeper lock manager, but…
•No way to delete, update, or insert rows
–Makes ad hoc work difficult
What’s Wrong?

Page 3
•Hadoop and Hive have always…
–Worked without ACID
–Perceived as tradeoff for performance
•But, your data isn’t static
–It changes daily, hourly, or faster
–Ad hoc solutions require a lot of work
–Managing change makes the user’s life better
•Need consistent views of changing data!
Why is ACID Critical?

Page 4
•Updating a Dimension Table
–Changing a customer’s address
•Delete Old Records
–Remove records for compliance
•Update/Restate Large Fact Tables
–Fix problems after they are in the warehouse
•Streaming Data Ingest
–A continual stream of data coming in
–Typically from Flume or Storm
Use Cases

Page 5
•Mutiple statement transactions
–Group statements that need to work together
•Query tables as they appeared in past
–Configurable length of history
•Row-level lineage
–Track users and queries that updated each row
Longer Term Use Cases

Page 6
•HDFS Does Not Allow Arbitrary Writes
–Store changes as delta files
–Stitched together by client on read
•Writes get a Transaction ID
–Sequentially assigned by Metastore
•Reads get Committed Transactions
–Provides snapshot consistency
–No locks required
–Provide a snapshot of data from start of query
Design

Stitching Buckets Together
Page 7

Page 8
•Partition locations remain unchanged
–Still warehouse/$db/$tbl/$part
•Bucket Files Structured By Transactions
–Base files $part/base_$tid/bucket_*
–Delta files $part/delta_$tid_$tid/bucket_*
•Minor Compactions merge deltas
–Read delta_$tid1_$tid1 .. delta_$tid2_$tid2
–Written as delta_$tid1_$tid2
•Compaction doesn’t disturb readers
HDFS Layout

Page 9
•Created new AcidInput/OutputFormat
–Unique key is transaction, bucket, row
•Reader returns most recent update
•Also Added Raw API for Compactor
–Provides previous events as well
•ORC implements new API
–Extends records with change metadata
–Add operation (d, u, i), transaction and key
Input and Output Formats

Page 10
•Existing lock managers
–In memory - not durable
–ZooKeeper - requires additional components to
install, administer, etc.
•Locks need to be integrated with
transactions
–commit/rollback must atomically release locks
•We sort of have this database lying around
which has ACID characteristics (metastore)
•Transactions and locks stored in metastore
•Uses metastore DB to provide unique,
ascending ids for transactions and locks
Transaction Manager

Page 11
•No explicit transactions in 0.13
–First implementation of INSERT, UPDATE,
DELETE will be auto-commit
–Will then add BEGIN, COMMIT, ROLLBACK
•Snapshot isolation
–Reader will see consistent data for the duration of
his/her query
–May extend to other isolation levels in the future
•Current transactions can be displayed
using new SHOW TRANSACTIONS
statement
Transaction Model

Page 12
•Three types of locks
–shared
–semi-shared (can co-exist with shared, but not
other semi-shared)
–exclusive
•Operations require different locks
–SELECT, INSERT – shared
–UPDATE, DELETE – semi-shared
–DROP, INSERT OVERWRITE – exclusive
Locking Model

Page 13
•Each transaction (or batch of
transactions in streaming ingest)
creates a new delta file
•Too many files = NameNode 
•Need a way to
–Collect many deltas into one delta – minor
compaction
–Rewrite base and delta to new base – major
compaction
Compactor

Page 14
•Run when there are 10 or more deltas
(configurable)
•Results in base + 1 delta
Minor Compaction
/hive/warehouse/purchaselog/ds=201403311000/base_0028000
/hive/warehouse/purchaselog/ds=201403311000/delta_0028001_0028100

Page 15
•Run when deltas are 10% the size of
base (configurable)
•Results in new base
Major Compaction

Page 16
•Metastore thrift server will schedule and
execute compactions
–No need for user to schedule
–User can initiate via new ALTER TABLE
COMPACT statement
•No locking required, compactions run at
same time as select, inserts
–Compactor aware of readers, does not remove old
files until readers have finished with them
•Current compactions can be viewed via
new SHOW COMPACTIONS statement
Compactor Continued

Page 17
•Data is flowing in from generators in a stream
•Without this, you have to add it to Hive in
batches, often every hour
–Thus your users have to wait an hour before they can
see their data
•New interface in hive.hcatalog.streaming lets
applications write small batches of records and
commit them
–Users can now see data within a few seconds of it
arriving from the data generators
•Available for Apache Flume in HDP 2.1
–Working on Apache Storm integration
Application, Streaming Ingest

Page 18
Streaming Ingest Illustrated
Flume
Agent
HDFS

Page 19
Flume
Agent
HDFS
while (…)
write();
commit();
Commit can be
time based or size
based, up to writer
commit() flushes to
disk and sends
commit to metastore

Page 20
Flume
Agent
HDFS
while (…)
write();
commit();
Next write() appends
to the same file

Page 21
Flume
Agent
HDFS
while (…)
write();
commit();
Reader
Task
Reader uses txnid to
determine which records
to read

Page 22
• Phase 1, Hive 0.13
–Transaction and new lock manager
–ORC file support
–Automatic and manual compaction
–Snapshot isolation
–Streaming ingest via Flume
• Phase 2, Hive 0.14 (we hope)
–INSERT … VALUES, UPDATE, DELETE
–BEGIN, COMMIT, ROLLBACK
• Future (all speculative based on user feedback)
–Versioned or point in time queries
–Additional isolation levels such as dirty read or read
committed
–MERGE
Phases of Development

Page 23
•Only suitable for data warehousing, not
for OLTP
•Table must be bucketed, and (currently)
not sorted
–Sorting restriction will be removed in the future
Limitations

Page 24
•Good
–Handles compactions for us
–Already has similar data model with LSM
•Bad
–No cross row transactions
–Would require us to write a transaction manager over
HBase, doable, but not less work
–Hfile is column family based rather than columnar
–HBase focused on point lookups and range scans
–Warehousing tends to require full scans
Why Not HBase?

Page 25
•JIRA:
https://issues.apache.org/jira/browse/HI
VE-5317
•Adds ACID semantics to Hive
•Uses SQL standard commands
–INSERT, UPDATE, DELETE
•Provides scalable read and write access
Conclusion

Thank You!
Questions & Answers
Page 26

Adding ACID Transactions, Inserts, Updates, and Deletes in Apache Hive

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Adding ACID Transactions, Inserts, Updates, and Deletes in Apache Hive

Similar to Adding ACID Transactions, Inserts, Updates, and Deletes in Apache Hive (20)

More from DataWorks Summit

More from DataWorks Summit (20)

Recently uploaded

Recently uploaded (20)

Adding ACID Transactions, Inserts, Updates, and Deletes in Apache Hive