Omid: Efficient Transaction Management and Incremental Processing for HBase (Hadoop Summit Europe 2013)

1,604 views
1,496 views

Published on

http://yahoo.github.com/omid

Many data processing tasks can be thought as small mutations to a large database triggered by events. Contrary to batch processing, the incremental processing model achieves very low delay from the reception of an event to the application of the mutation. In the absence of transactions support, developers have to use ad-hoc mechanisms to ensure atomic execution of mutations despite failures and concurrent accesses to the database by other clients. Most NoSQL data stores, like HBase, BigTable and Cassandra, lack support of transactions, which makes them unsuitable for a whole range of applications. In this talk we present Omid, an open source tool for transactional support and incremental processing on top of HBase (https://github.com/yahoo/omid). Due to the centralized nature of its client-replicated status oracle, Omid (i) avoids distributed locks, (ii) scales up to 60,000 TPS and a thousand clients, (iii) requires no changes to HBase, and (vi) adds a negligible overhead to data servers.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,604
On SlideShare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
34
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Omid: Efficient Transaction Management and Incremental Processing for HBase (Hadoop Summit Europe 2013)

  1. 1. OmidEfficient Transaction Management and Incremental Processing for HBase Daniel Gómez Ferro 21/03/2013 Copyright © 2013 Yahoo! All rights reserved. No reproduction or distribution allowed without express written permission.
  2. 2. About me § Research engineer at Yahoo! § Main Omid developer § Apache S4 committer § Contributor to: • ZooKeeper • HBaseOmid - Yahoo! Research 2
  3. 3. Outline § Motivation § Goal: transactions on HBase § Architecture § Examples § Future workOmid - Yahoo! Research 3
  4. 4. Motivation § Traditional DBMS: § NoSQL stores: • SQL • SQL • Transactions • Transactions • Scalable • Scalable § Some use cases require • Scalability • Consistent updatesOmid - Yahoo! Research 4
  5. 5. Motivation § Requested feature • Support for transactions in HBase is a common request § Percolator • Google implemented transactions on BigTable § Incremental Processing • MapReduce latencies measured in hours • Reduce latency to seconds • Requires transactionsOmid - Yahoo! Research 5
  6. 6. Incremental Processing Offline Online MapReduce Incremental Processing Fault tolerance Fault tolerance Scalability Scalability Shared state State0 State1 Shared State Using Percolator, Google dramatically reduced crawl-to-index timeOmid - Yahoo! Research 6
  7. 7. Transactions on HBase
  8. 8. Omid § ‘Hope’ in Persian § Optimistic Concurrency Control system • Lock free • Aborts transactions at commit time § Implements Snapshot Isolation § Low overhead § http://yahoo.github.com/omidOmid - Yahoo! Research 8
  9. 9. Snapshot Isolation § Transactions read from their own snapshot § Snapshot: • Immutable • Identified by creation time • Values committed before creation time § Transactions conflict if two conditions apply: A)  Overlap in time B)  Write to the same rowOmid - Yahoo! Research 9
  10. 10. Transaction conflictsId: Time overlap: Write set:A r1 r3B r2 r4C r3 r4D r1 r2 r3 r4 Time § Transaction B overlaps with A and C • B ∩ A = ∅ • B ∩ C = { r4 } • B and C conflict § Transaction D doesn’t conflictOmid - Yahoo! Research 10
  11. 11. Omid Architecture
  12. 12. Architecture 1.  Centralized server 2.  Transactional metadata replicated to clients 3.  Store transaction ids in HBase timestamps Status Oracle (SO) (1) (2) S O Omid Omid Omid Clients HBase Clients Clients HBase HBase HBase Region Region Region Region Servers Servers Servers Servers (3)Omid - Yahoo! Research 12
  13. 13. Status Oracle § Limited memory • Evict old metadata • … maintaining consistency guarantees! § Keep LowWatermark • Updated on metadata eviction • Abort transactions older than LowWatermark • Necessary for Garbage CollectionOmid - Yahoo! Research 13
  14. 14. Transactional Metadata Status Oracle StartTS CommitTS Row LastCommitTS T10 T20 r1 T20 … … … … LowWatermark Aborted T4 T2 T18 …Omid - Yahoo! Research 14
  15. 15. Metadata Replication § Transactional metadata is replicated to clients § Status Oracle not in critical path • Minimize overhead § Clients guarantee Snapshot Isolation • Ignore data not in their snapshot • Talk directly to HBase • Conflicts resolved by the Status Oracle § Expensive, but scalable up to 1000 clientsOmid - Yahoo! Research 15
  16. 16. Architecture recap Status Oracle StartTS CommitTS Row LastCommitTS T10 T20 r1 T20 … … … … LowWatermark Aborted T4 T2 T18 … S O Omid Omid HBase Omid Clients HBase HBase HBase Clients Region Region Region Clients Region Servers Servers Servers ServersOmid - Yahoo! Research 16
  17. 17. Architecture + Metadata Conflict detection Status Oracle (not replicated) ST CT R LC HBase values tagged 10 20 r1 20 with Start TimeStamp LowW Abort … … Client ST CT HBase 10 20 row TS val LowW Abort r1 10 hi … …Omid - Yahoo! Research 17
  18. 18. Comparison with PercolatorStatus Omid PercolatorOracle TS ServerHBase Clients BigTable ClientsOmid - Yahoo! Research 18
  19. 19. Simple API § TransactionManager: Transaction begin(); void commit(Transaction t) throws RollbackException; void rollback(Transaction t); § TTable: Result get(Transaction t, Get g); void put(Transaction t, Put p); ResultScanner getScanner(Transaction t, Scan s);Omid - Yahoo! Research 19
  20. 20. Example Transaction
  21. 21. Transaction Example >> | Status Oracle ST CT R LC 10 20 r1 20 LowW Abort … … Client HBase ST CT 10 20 row TS val r1 10 hi LowW Abort … …Omid - Yahoo! Research 21
  22. 22. Transaction Example >> t = TM.begin() Transaction { ST: 25 } Status Oracle >> | ST CT R LC 10 20 r1 20 LowW Abort … … Client t ST: 25 HBase ST CT 10 20 row TS val r1 10 hi LowW Abort … …Omid - Yahoo! Research 22
  23. 23. Transaction Example >> t = TM.begin() Transaction { ST: 25 } Status Oracle >> TTable.get(t, ‘r1’) ‘hi’ ST CT R LC >> | 10 20 r1 20 LowW Abort … … Client t ST: 25 HBase ST CT 10 20 row TS val r1 10 hi LowW Abort … …Omid - Yahoo! Research 23
  24. 24. Transaction Example >> t = TM.begin() Transaction { ST: 25 } Status Oracle >> TTable.get(t, ‘r1’) ‘hi’ ST CT R LC >> TTable.put(t, ‘r1’, ‘bye’) 10 20 r1 20 >> | LowW Abort … … Client t ST: 25 HBase ST CT R: {r1} 10 20 row TS val r1 10 hi LowW Abort r1 25 bye … …Omid - Yahoo! Research 24
  25. 25. Transaction Example >> TTable.get(t, ‘r1’) ‘hi’ Status Oracle >> TTable.put(t, ‘r1’, ‘bye’) >> TM.commit(t) ST CT R LC Committed{ CT: 30 } 25 30 r1 30 >> | LowW Abort 20 … Client t ST: 25 HBase ST CT R: {r1} 10 20 row TS val r1 10 hi LowW Abort r1 25 bye … …Omid - Yahoo! Research 25
  26. 26. Transaction Example >> t_old Transaction{ ST: 15 } Status Oracle >> TT.put(t_old, ‘r1’, ‘yo’) >> TM.commit(t_old) ST CT R LC Aborted 25 30 r1 30 >> | LowW Abort … 15 Client t_old ST: 15 HBase ST CT R: {r1} 10 20 row TS val r1 15 yo LowW Abort r1 25 bye … …Omid - Yahoo! Research 26
  27. 27. Centralized ApproachOmid - Yahoo! Research 27
  28. 28. Omid Performance § Centralized server = bottleneck? • Focus on good performance • In our experiments, it never became the bottleneck 30 2 rows 25 4 rows 8 rows 512 rows/t 20 16 rows 32 rows 128 rows 1K TPSLatency in ms 512 rows 10 ms 15 10 2 rows/t 5 100K TPS 0 5 ms 1K 10K 100K Throughput in TPS Omid - Yahoo! Research 28
  29. 29. Fault Tolerance § Omid uses BookKeeper for fault tolerance • BookKeeper is a distributed Write Ahead Log § Recovery: reads log’s tail (bounded time) Downtime < 25 s 18000 Throughput in TPS 15000 12000 9000 6000 3000 0 5900 5950 6000 6050 6100 Time in sOmid - Yahoo! Research 29
  30. 30. Example Use CaseNews Recommendation System
  31. 31. News Recommendation System § Model-based recommendations • Similar users belong to a cluster • Each cluster has a trained model § New article • Evaluate against clusters • Recommend to users § User actions • Train model • Split clusterOmid - Yahoo! Research 31
  32. 32. Transactions Advantages § All operations are consistent • Updates and queries § Needed for a recommendation system? • We avoid corner cases • Implementing existing algorithms is straightforward § Problems we solve • Two operations concurrently splitting a cluster • Query while cluster splits • …Omid - Yahoo! Research 32
  33. 33. Example overviewArticles & Updates QueriesUser actions Index Data Index Wor Store kersOmid - Yahoo! Research 33
  34. 34. Other use cases § Update indexes incrementally • Original Percolator use case § Generic graph processing § Adapt existing transactional solutions to HBaseOmid - Yahoo! Research 34
  35. 35. Future Work
  36. 36. Current Challenges § Load isn’t distributed automatically § Complex logic requires big transactions • More likely to conflict • More expensive to retry § API is low level for data processingOmid - Yahoo! Research 36
  37. 37. Future work § Framework for Incremental Processing • Simpler API • Trigger-based, easy to decompose operations • Auto scalable § Improve user friendliness • Debug tools, metrics, etc § Hot failoverOmid - Yahoo! Research 37
  38. 38. Questions? All time contributors Ben Reed Flavio Junqueira Francisco Pérez-Sorrosal Try it! Ivan Kelly yahoo.github.com/omid Matthieu Morel Maysam Yabandeh Partially supported byCumuloNimbo project (ICT-257993)funded by the European Commission
  39. 39. BackupOmid - Yahoo! Research 39
  40. 40. Commit Algorithm § Receive commit request for txn_i § Set CommitTimestamp(txn_i) <= new timestamp § For each modified row: • if LastCommit(row) > StartTimestamp(txn_i) => abort § For each modified row: • Set LastCommit(row) <= CommitTimestamp(txn_i)Omid - Yahoo! Research 40

×