Omid powering transactions in Apache Phoenix.

1. Omid: Scalable an d Highly Available
Transaction Processing for Phoenix
Ohad Shacham, Edward Bortnikov ⎪ PhoenixCon, Jun 13, 2017

2. Let’s Get Started …
2
Our Yahoo Journey with Transactions over HBase
Omid for Users: Semantics, API, Integration with Phoenix
Omid for Programmers: Architecture and Use Cases
Omid, Advanced: Scalability, HA, Low-Latency

3. Transaction Processing in NoSQL @Yahoo
3
Motivation: Data Pipelines (Search, Mail, etc.)
Stream Processing a Popular Pattern
Compute Tasks process Data Items that arrive in the Real Time
Intermediate Artifacts stored in NoSQL (KV-)Storage
Extensive Use of Hadoop Technologies (Storm, HBase)
Scale: Thousands of Hadoop Nodes

4. Content Indexing for Search
Crawl Docproc
Link
Analysis Stream
Crawl
schedule
Content
Queue
Links
STORM
HBase

5. Zooming in on Tasks
Document processing
Read page content from the store
Compute search index features
Update computed features
Link processing
Read outgoing links for a page
Update reference for all linked-to pages
begin
begin
commit
commit

6. Transaction Processing: ACID 101
6
Multiple data accesses in a single logical operation
Atomic
“All or nothing” – no partial effect observable
Consistent
The DB transitions from one valid state to another
Isolated
Appear to execute in isolation
Durable
Committed data cannot disappear

7. Omid (‫)امید‬
7
2011
Incepted
@Yahoo Research
“Omid1”
2014
Large-Scale
Deployment
@Yahoo
2014/5
Major Re-Design
for Scalability & HA
“Omid2”
2016
Apache
Incubator
2017
Prototype
Integration
with Phoenix
Transaction Processing Service for Apache HBase

8. Contributors
8
Ohad Shacham
Yahoo Research
Francisco
Perez Sorrosal
Yahoo
Edward Bortnikov
Yahoo Research
Eshcar Hillel
Yahoo Research
Idit Keidar
Yahoo, Technion
Ivan Kelly
Midokura
Sameer Paranjpye
Databricks
Matthieu Morel
Skyscanner
Igor Katkov
Atlassian
Yonatan Gottesman
Yahoo Research

9. Omid 101
9
Client Library + Runtime Service
Database Agnostic (can work with other backends)
Snapshot Isolation consistency
Very Scalable (>380K peak tps) and Highly Available

10. Omid Programming Example
10
TransactionManager tm = HBaseTransactionManager.newInstance();
TTable txTable = new TTable("MY_TX_TABLE”);
Transaction tx = tm.begin(); // Control path
Put row1 = new Put(Bytes.toBytes("EXAMPLE_ROW1"));
row1.add(family, qualifier, Bytes.toBytes("val1"));
txTable.put(tx, row1); // Data path
Put row2 = new Put(Bytes.toBytes("EXAMPLE_ROW2"));
row2.add(family, qualifier, Bytes.toBytes("val2"));
txTable.put(tx, row2); // Data path
tm.commit(tx); // Control path

11. Snapshot Isolation (SI) Semantics
Distinct read (snapshot) and write (commit) points
No write-write conflicts allowed

12. Tephra: Sibling Technology
12
Transaction Processing technology for HBase
SI Semantics. Design Similar to Omid1
Apache Incubator since 2016
Integrated with Phoenix to provide ACID semantics (BETA)
Implements some Phoenix-specific scenarios

13. Phoenix-Omid Integration
13
Work in Progress under JIRA PHOENIX-3623
Backward Compatible – Configurable TP Provider Choice
Current Options: Tephra and Omid
How?
Internal Transaction Abstraction Layer (TAL) API
Multiple Implementations, Configurable Instantiation

14. Transaction Processing, Refactored
14
Transaction
Abstraction Layer
Tephra
Client
Omid
Client
Phoenix
Phoenix
Tephra
Client
Refactor

15. How Omid Works
Client
Begin/Commit
Data
Data
Data
Commit
Table
Persist
Commit
Verify commit
Read/Write
Conflict
Detection
15
Transaction
Manager
(TSO)
Lock-Free SI Implementation. Exploits Built-in MVCC.

16. Transacti
on
Manager
Client
Begin
Data
Data
Data
Commit
Table
t1
Write (k1, v1, t1)
Write (k2, v2, t1)
Read (k’, last committed t’ < t1)
(k1, v1, t1)
(k2, v2, t1)
Execution Example
tr = t1
Transaction
Manager
16

17. Client
Commit: t1, {k1, k2}
Data
Data
Data
Commit
Table
t2
(k1, v1, t1)
(k2, v2, t1)
Write (t1, t2)
(t1, t2)
Execution Example
tr = t1
tc = t2
17
Transaction
Manager

18. Client
Data
Data
Data
Commit
Table
Read (k1, t3)
(k1, v1, t1)
(k2, v2, t1)
(t1, t2)
Read (t1)
Execution Example
tr = t3
18
Bottleneck!
Transaction
Manager

19. Client
Data
Data
Data
Commit
Table
t2
(t1, t2)
(k1,v1,t1,t2)
(k2,v2,t1,t2)
Delete(t1)
Post-Commit Timestamp Replication
tr = t1
tc = t2
Update
commit
cells
19
Transaction
Manager

20. Data
Data
Data
Commit
Table
Read (k1, t3)
Using Commit Cells
Client
tr = t3
20
Transaction
Manager
(k1,v1,t1,t2)
(k2,v2,t1,t2)

21. Phoenix – New Scenarios for Omid
21
Secondary Indexes
On-the-Fly Index Creation
Atomic Updates
Query by Secondary Key
Extended Snapshot Isolation
Read-Your-Own-Writes Queries

22. On-the-Fly Secondary Index Creation
22
CREATE INDEX (CI) in parallel with writes to the base table
How? Distinguish between the pre-CI and post-CI data
CREATE INDEX command issue time defines a timestamp
1. All data committed before snapshot: scanned, bulk-inserted into index
2. All data generated after snapshot: triggers random update of index
3. All transactions in flight at snapshot time: aborted (FENCE)

23. Secondary Index: Creation and Maintenance
23
T1
T2
T3
CREATE INDEX started
T4
CREATE INDEX complete
T5
T6
Bulk-
Inserted
into index
Abort
(enforced
upon
commit)
Added by a
coprocessor
Added by a
coprocessor
Index
update
(stored
procedure)

24. Extended Snapshot Isolation
24
CREATE TABLE T (ID INT);
BEGIN;
1: INSERT INTO T
SELECT ID+10 FROM T;
2: INSERT INTO T
SELECT ID+100 FROM T;
COMMIT;
Traditional SI: Read-Your-Writes
Challenge:
Circular Dependency
(Statement in Infinite Loop)
Solution: Moving Snapshot
(series of checkpoint snapshots)

25. Moving Snapshot Implementation
25
Checkpoint for
Statement 1
Checkpoint for
Statement 2
Writes by
Statement 1
Timestamps allocated by TM in blocks.
Client promotes the checkpoint.

26. Omid Scalability
26
Extremely lean Client-Transaction Manager protocol
Omid1, Tephra replicate the entire state to client side upon BEGIN
Aggressive batching of writes to CT in Transaction Manager
Concurrent conflict detection (experimental)
HA algorithm incurs zero overhead in the mainstream

27. 0
50
100
150
200
250
300
350
400
450
500
550
Omid1
Omid1 Non Durable
Omid
Omid Non Durable
Tps*103
Throughput Benchmark
YCSB workload driver
12-core Transaction Manager
1G network

28. 0
500
1000
1500
2000
2500
document inversion
duplicate detection
out-link processing
in-link processing
stream to runtime
TaskLatency(ms)
Commit + CT update
Begin
Compute
Read
Update
Overhead in Production: Web Search Indexing

29. Low-Latency Omid (Experimental)
29
Original Design: Throughput-Oriented Applications in Mind
Sometimes, this comes at the expense of latency
Example: writes to Commit Table batched at the Transaction Manager
Key: Dissolve the Transaction Manager I/O Bottleneck
Distribute the Commit Table and the Writes to it
How?
The client, rather than the TM, persists the Commit Timestamp (CTS)
CTS embedded in the first row written by the transaction

30. Benchmark: Single-Write Transaction Workload
0
10
20
30
40
50
60
70
80
0
50
100
150
200
250
300
Omid
Low latency
Throughput (tps * 103)
Latency(msec)

31. Summary
31
Scalable, Highly Available Open Source Transaction Processing
Battle-Tested, Ready for Public Cloud
Integration with Apache Phoenix Underway (GA in 2017)

32. Thanks to Our Partners for Being Awesome
32

33. Backup
33

34. Architecture, Recapped
Client
Begin/Commit
Data
Data
Data
Commit
Table
Persist
Commit
Verify commit
Read/Write
SPoF
34
Transaction
Manager
(TSO)

35. HA: Primary-Backup Transaction Manager
Client
Data
Data
Data
Commit
Table
35
Transaction
Manager
(TSO)
Transaction
Manager
Recovery
state (ZK)
Primary
Backup

36. Split Brain
Client
Commit
Table
36
Transaction
Manager
(TSO)
Transaction
Manager
Primary
Backup
Race
Conditions
Violate SI
Take I:
Fence CT upon
every write (slow!)

37. HA Algorithm – Key Ideas
37
Old and New Primaries may write conflicting commit records
No Locks!
Client detects inconsistencies, invalidates problematic records
Lease-Based Leader Election
Optimization: Local lease check before/after writing to CT
Zero Overhead in Non-Recovery Scenarios