Trafodion Distributed Transaction Management
•
2 likes•850 views
Trafodion brings a completely distributed scalable transaction management implementation integrated into HBase. It does not suffer from the scale and performance limitations of other transaction managers on HBase. This presentation reviews the elegant architecture and how this architecture is leveraged to provide full ACID SQL transactional capabilities across multiple rows, tables, statements, and region servers. It discusses the life of a transaction from BEGIN WORK, to updates, to ABORT WORK, to COMMIT WORK, and then discusses recovery and high availability capabilities provided. An accompanying white paper goes into depth explaining this animated presentation in more detail. Given the increasing interest for transaction managers on Hadoop, or to provide transactional capabilities for NoSQL users when needed, the Trafodion community can certainly open up this Distributed Transaction Management support to be leveraged by implementations other than Trafodion.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (15)
Similar to Trafodion Distributed Transaction Management
Similar to Trafodion Distributed Transaction Management (20)
More from Rohit Jain
Recently uploaded
Recently uploaded (20)
Trafodion Distributed Transaction Management
- 1. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Trafodion Distributed Transaction Management April 2015
- 2. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Agenda Trafodion Distributed Transaction Management • Capabilities • Architecture • Life of a transaction – BEGIN WORK – get / put / delete – COMMIT WORK • Prepare phase 1 • Commit phase 2 – ABORT WORK – Transaction Recovery • Control Point processing • Performance • Futures
- 3. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. • Multi-version Concurrency Control (MVCC) non-locking optimistic concurrency control algorithm – Reads are never blocked – Transactions do not see changes made by other active transactions – Conflict checking done at commit time; transaction aborts if it made changes in conflict with already committed transaction – Applications need to retry transactions which abort due to conflicts • BEGIN WORK, COMMIT WORK, ROLLBACK WORK, SET TRANSACTION – If BEGIN WORK is not used to initiate a transaction, SQL defaults to AUTOCOMMIT with HBase transactional support • READ COMMITTED as the transactional isolation level, which is also the default – Stronger than ANSI SQL READ COMMITTED in that it prohibits lost updates but allows non-repeatable reads and read skew • Multiple SQL processes participating in the same transaction concurrently – Such as multiple parallel insert processes launched by the transaction initiating process e.g. for a large parallel INSERT…SELECT operation • Recovery after region server, transaction manager, or node failure • Transactional integrity across HBase region splits and rebalancing of region Capabilities Trafodion Distributed Transaction Management
- 4. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4 Node n SQL Process Transaction Manager Library Resource Manager Library SQL Process Transaction Manager Library Resource Manager Library SQL Process Transaction Manager Library Resource Manager Library Transaction Manager HBase trx Region Server HBase Region Server TLOG HBase RegionHBase RegionTrx Region Endpoint coproc Node 2 SQL Process Transaction Manager Library Resource Manager Library SQL Process Transaction Manager Library Resource Manager Library SQL Process Transaction Manager Library Resource Manager Library Transaction Manager HBase trx Region Server HBase Region Server TLOG HBase RegionHBase RegionTrx Region Endpoint coproc Scalable Architecture Trafodion Distributed Transaction Management ... Implemented using HBase 0.98 coprocessors Node 1 SQL Process Transaction Manager Library Resource Manager Library SQL Process Transaction Manager Library Resource Manager Library SQL Process Transaction Manager Library Resource Manager Library Transaction Manager HBase trx Region Server HBase Region Server TLOG HBase RegionHBase RegionTrx Region Endpoint coproc
- 5. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.5 Peeling the onion – Component Architecture Trafodion Distributed Transaction Management SQL Transaction initiating process Transaction Manager Library HBase RegionServer TLO G register(region, tx1) begin/commit/abort(tx1 ) get/put/ delete(tx1 ) commitRequest/commit/abort(tx1) Trx Region Endpoint Trx Region Endpoint Trx Region Endpoint C++ Jav a HLOG Transaction Manager Log Meta info/state about transaction (more than one region in a txn) HBase table(s) HBase Write Ahead Log Region level transaction info Each region server has its own HLOG Transaction Manager DTM Core Branch I/F Transaction Management JNI TransactionManager TrxRegionEndpoint coprocessor client open/perform/ close scanner(tx1) Jav a transactional aggregation(tx1) Coprocessors implemented TrxRegionEndpoint implements SQL transactions TrxRegionObserver implements recovery process SQL JNI Resource Manager Library TrxRegionEndpoint coprocessor client Transaction alTable Transaction alScanner Transactional operations get/put/delete TrxRegionEndpoi nt coprocessor client
- 6. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.6 begintransaction request BEGIN WORK – BEGINTRANSACTION Trafodion Distributed Transaction Management SQL Transaction initiating process Transaction Manager Library HBase RegionServer Transaction Manager DTM Core Branch I/F Transaction Management JNI TransactionManager TrxRegionEndpoint coprocessor client BEGINTRANSACT ION or BEGINTX Trx Region Endpoint Trx Region Endpoint Trx Region Endpoint new transaction object allocated Assign new transid TransactionManager. beginTransaction(transid) new TransactionState transid transid is allocated by TM for distributed support 1 2 3 4 5 6 Txn Object Txn State Resource Manager Library TrxRegionEndpoint coprocessor client Transaction alTable Transaction alScanner Transactional operations get/put/delete TrxRegionEndpoi nt coprocessor client
- 7. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7 Transaction Manager DTM Core Branch I/F Transaction Management JNI TransactionManager TrxRegionEndpoint coprocessor client Txn Object Txn State Transaction Manager DTM Core Branch I/F Transaction Management JNI TransactionManager TrxRegionEndpoint coprocessor client SQL Transaction initiating process Resource Manager Library TrxRegionEndpoint coprocessor client Transaction alTable Transaction alScanner Transactional operations get/put/delete TrxRegionEndpoi nt coprocessor client get / put / delete Trafodion Distributed Transaction Management Transaction Manager Library HBase RegionServer Trx Region Endpoint Trx Region Endpoint Trx Region Endpoint TransactionalTable.put(TransactionState, region) If region not in participatingRegions list of TransactionState add it & initiate registration TransactionManager.register Region which adds it to the TransactionState.participatin gRegions list 1 8 3 7 10 SQL Transaction initiating process Transaction Manager Library HBase RegionServer Trx Region Endpoint Trx Region Endpoint Trx Region Endpoint TrxRegionEndpoint.beginTransaction(transid) for region executed if transaction not initiated for the region call register (region, transid) 4 put(transid, region) Look up TransactionState by transid – create object if not there 2 TransactionStateObjAdds participant to the participating processes list of the Transaction Object 6 5 If region is not registered for this transaction, calls HBaseTxn.registerRegi on 9 TrxRegionEndpoint.put(transid) for region 11 Update region transaction object context to include puts, deletes Resource Manager Library TrxRegionEndpoint coprocessor client Transaction alTable Transaction alScanner Transactional operations get/put/delete TrxRegionEndpoi nt coprocessor client
- 8. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.8 Node 2 Node 3 Node 4Node 1 get / put / delete – multiple participating processes Trafodion Distributed Transaction Management Transaction Manager SQL Transaction initiating processResource Manager Library Transaction Manager Library HBase RegionServer Trx Region Endpoint Trx Region Endpoint Trx Region Endpoint Transaction Manager SQL Transaction initiating processResource Manager Library Transaction Manager Library HBase RegionServer Trx Region Endpoint Trx Region Endpoint Trx Region Endpoint Transaction Manager SQL Transaction initiating processResource Manager Library Transaction Manager Library HBase RegionServer Trx Region Endpoint Trx Region Endpoint Trx Region Endpoint Transaction Manager SQL Transaction initiating processResource Manager Library Transaction Manager Library HBase RegionServer Trx Region Endpoint Trx Region Endpoint Trx Region Endpoint Single process updates multiple regions Multiple processes update multiple regions
- 9. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.9 Transaction Manager DTM Core Branch I/F Transaction Management JNI TransactionManager TrxRegionEndpoint coprocessor client Txn Object Txn State endtransaction COMMIT WORK – ENDTRANSACTION – Phase 1, prepare Trafodion Distributed Transaction Management SQL Transaction initiating process Transaction Manager Library HBase RegionServer Trx Region Endpoint Trx Region Endpoint Trx Region Endpoint prepare_branches HBaseTxn.prepareComm it TransactionManage r. prepareCommit 1 2 3 4 5 ENDTRANSACTIO N SQL Transaction initiating process Transaction Manager Library HBase RegionServer Trx Region Endpoint Trx Region Endpoint Trx Region Endpoint TrxRegionEndpoint.commitRequest(transid) against each region in participatingRegions list, sent in parallel 6 Region endpoint coprocessor checks for conflicts / concurrency issue If conflict it votes abort else checks for writes in region If no writes, votes read-only & is excluded from phase 2 HLOG If writes performed, context for transid, including all updates, is forced written to HLOG 7 Abort if conflict encountered 9 8 Votes abort, read-only, or commit to transaction manager HLOG Resource Manager Library TrxRegionEndpoint coprocessor client Transaction alTable Transaction alScanner Transactional operations get/put/delete TrxRegionEndpoi nt coprocessor client Resource Manager Library TrxRegionEndpoint coprocessor client Transaction alTable Transaction alScanner Transactional operations get/put/delete TrxRegionEndpoi nt coprocessor client Transaction Manager DTM Core Branch I/F Transaction Management JNI TransactionManager TrxRegionEndpoint coprocessor client
- 10. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.10 Transaction Manager DTM Core Branch I/F Transaction Management JNI TransactionManager TrxRegionEndpoint coprocessor client Txn Object Txn State COMMIT WORK – ENDTRANSACTION – Phase 2, commit Trafodion Distributed Transaction Management SQL Transaction initiating process Transaction Manager Library HBase RegionServer Trx Region Endpoint Trx Region Endpoint Trx Region Endpoint commit_branche s TransactionManag er. doCommit commit 2 3 1 5 SQL Transaction initiating process Transaction Manager Library HBase RegionServer Trx Region Endpoint Trx Region Endpoint Trx Region Endpoint TrxRegionEndpoint.commit(transid) against each region in participatingRegions list, sent in parallel If at least one region voted commit & the rest voted commit or read-only TLO G commit_branches HBaseTxn.doCom mit Forces write of committed trans state record 4 Unforced write of forgotten trans state record 9 doCommit forgets transaction & returns control to TM 1 0 All puts and deletes are written to HBase tables from in-memory transaction state HBase WAL writes are disabled since these were already recorded during prepare 7 8 HLOG Context for transid as committed txn is written unforced to HLOG 6 HLOG Used for control point / aging process discussed later Records hashed key, ASN, txn id & state, and tables in txn Reduces recovery time Resource Manager Library TrxRegionEndpoint coprocessor client Transaction alTable Transaction alScanner Transactional operations get/put/delete TrxRegionEndpoi nt coprocessor client Resource Manager Library TrxRegionEndpoint coprocessor client Transaction alTable Transaction alScanner Transactional operations get/put/delete TrxRegionEndpoi nt coprocessor client Transaction Manager DTM Core Branch I/F Transaction Management JNI TransactionManager TrxRegionEndpoint coprocessor client
- 11. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.11 Transaction Manager DTM Core Branch I/F Transaction Management JNI TransactionManager TrxRegionEndpoint coprocessor client Txn Object Txn State HLOG HLOG ABORT WORK – ABORTTRANSACTION Trafodion Distributed Transaction Management SQL Transaction initiating process Transaction Manager Library HBase RegionServer Trx Region Endpoint Trx Region Endpoint Trx Region Endpoint rollback_branche s TransactionManag er. abort Abort 3 4 2 6 SQL Transaction initiating process Transaction Manager Library HBase RegionServer Trx Region Endpoint Trx Region Endpoint Trx Region Endpoint TrxRegionEndpoint.abortTransaction(transid) against each region in participatingRegions list, sent in parallel If writes associated with txn, unforced write of abort to HLOG 8 rollback_branche s HBaseTxn.abort does not write trans record If forgotten records not written, presume abort 5 9 Abort forgets transaction & returns control to TM 10 1 TM does forget processing No writes to log HRegion.abort sets status of txn to abort Removes txn from commit list if previously voted commit Remove txn from pending list 7 TLO G Reduces recovery time Resource Manager Library TrxRegionEndpoint coprocessor client Transaction alTable Transaction alScanner Transactional operations get/put/delete TrxRegionEndpoi nt coprocessor client Resource Manager Library TrxRegionEndpoint coprocessor client Transaction alTable Transaction alScanner Transactional operations get/put/delete TrxRegionEndpoi nt coprocessor client Transaction Manager DTM Core Branch I/F Transaction Management JNI TransactionManager TrxRegionEndpoint coprocessor client
- 12. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.13 Transaction Manager DTM Core Branch I/F Transaction Management JNI TransactionManager TrxRegionEndpoint coprocessor client Transaction Manager DTM Core Branch I/F Transaction Management JNI TransactionManager TrxRegionEndpoint coprocessor client Update txn state obj from last state of transid, & participating regions in TLOG Transaction Recovery Trafodion Distributed Transaction Management SQL Transaction initiating process Transaction Manager Library HBase RegionServer SQL Transaction initiating process Transaction Manager Library HBase RegionServer ... 1 When failed server(s) are reintegrated, Region Server reads HLOG. If its empty, indicating clean shutdown, it resumes txn processing. Otherwise it initiates recovery for regions. HLOG HLOG 3 Each Region with in-doubt transactions posts need for recovery to bulletin of TM that originated the transaction if not yet posted 1 6 Collates list of in-doubt transactions from all regions and creates empty transaction state object for each txn 7 Recovery thread re-drives phase 2 Commit / Abort will be done as normal for in- doubt txns 8 9 Zookeeper Bulletin Board for TM 4 TM builds in-doubt region list and initiates recovery for those regions 5 TM disables transactions during recovery TLO G Table names mapped to regions Accommodates region splits & moves Txn StateTxn StateTxn State Resource Manager Library TrxRegionEndpoint coprocessor client Transaction alTable Transaction alScanner Transactional operations get/put/delete TrxRegionEndpoi nt coprocessor client In-doubt transaction 1. No prepare record abort transaction 2. Commit record re-drive commit in redo processing 3. Abort record do nothing 4. Prepare record needs to know commit or abort? Resource Manager Library TrxRegionEndpoint coprocessor client Transaction alTable Transaction alScanner Transactional operations get/put/delete TrxRegionEndpoi nt coprocessor client Trx Region Observer Trx Region Observer Trx Region Observer Trx Region Observer Trx Region Observer Trx Region Observer Trx Region Observer coprocessor for all Regions builds in-doubt txn list In-doubt regions deregister from bulletin board Trx Region Endpoint Trx Region Endpoint Trx Region Endpoint Trx Region Endpoint Trx Region Endpoint Trx Region Endpoint 2 Gather in-doubt transactions from in-doubt regions 10
- 13. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.14 Transaction Manager DTM Core Branch I/F Transaction Management JNI TransactionManager TrxRegionEndpoint coprocessor clientTLO G Trafodion Distributed Transaction Management Used to re-drive transaction recovery • After server or cluster failure to ensure data consistency given in-flight operations at time of failure • Control points written to Control Point table at a configurable 1interval, currently set at 2 minutes • Re-drives of transaction commits (redo) driven by recovery needs of regions Used for aging • TLOG entries prior to 2 control points can be aged out and only two entries maintained in control point table • Commit records rewritten to log, along with writing txn state of all running transactions, until all commits received and forgotten record written Transaction Recovery – Control Point processing SQL Transaction initiating process Transaction Manager Library CP Forced write of key & 2ASN for txn state record written to CP Unforced grouped writes of txn record for each running txn 2monotonically increasing value indicating the audit write sequence number1control point interval is represented by two consecutive control point records where the first record defines the ASN from the start of the interval and the second defines the ASN at the end of the interval Txn StateTxn StateTxn State Txn ObjectTxn ObjectTxn Object
- 14. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.15 Performance objective: overhead within 20% Trafodion Distributed Transaction Management Order Entry: multi-statement transactional workload • 5 transaction types (New Orders, Payments, Order Status, Deliver, and Stock Level checks • On average has about 20 statements per transaction 0 128 256 384 512 640 768 896 1,024 Throughput(TPM) Concurrency (Streams) OrderEntry Traf 1.1 Autcommit Meets current objective! With max variance at 11.3%
- 15. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.16 Trafodion Distributed Transaction Management Optimizations Single operation / region transactionss Transaction flow Isolation support * Stateless Stateful Concurrency Control – Repeatable read – Snapshot isolation – First writer instead of first committer wins Serialized snapshot isolation *DDL transaction protection Row Locking paradigm for certain tables Tables involved in long running transactions Pessimistic locking for highly concurrent operations High Availability Flexible cluster growth – add nodes on the fly Handle region splits/rebalancing during transactions Better integration with HBase clusters Handle complex failure scenarios Manageability Transaction management & metrics via REST APIs Futures *Technology preview in Trafodion R1.1
- 16. © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Thank you