Presentation at OOPSLA'16.

1. Hybrid STM/HTM for Nested
Transactions on OpenJDK
Keith Chapman
Purdue U
Tony Hosking
ANU/Data61, Purdue U
Eliot Moss
UMass

2. Motivation
STM has been around for ages
• But STM is slow
Commodity hardware for transactions available now
• But HTM approaches are only best effort
Out goal: Accelerate STM with HTM when possible

3. Nested Transactions
Allow composition of transactions
Two flavors
• Closed Nested Transactions
• HTM inside STM not useful; must do all the same work
• Open Nested Transactions
• HTM inside STM avoids most TM work – HTM well-suited!

4. XJ: Transactional Java
XJ Language
• Supports flat and closed/open/boosted nested transactions
The implementation supports hybrid transactions
• HTM support via Intel TSX
• HTM and STM can co-exist
Uses an Optimistic-reads / Pessimistic-writes protocol

5. TM Metadata
Txn Log
0 0Version # 0 0 0Txn ID 1
One metadata word for every object

6. STM Transaction Protocol (Read)
T1 Log
0 00 0

7. STM Transaction Protocol (Read)
T1 Log
0 00 0
Check Mode

8. STM Transaction Protocol (Read)
T1 Log
0 0
0 0
T1 Read

9. STM Transaction Protocol (Read)
T1 Log
0 0
0 0
T1 Validate

10. STM Transaction Protocol (Read)
T1 Log
0 0
T1 Commit

11. 0 0
STM Transaction Protocol (Write)
T1 Log

12. 0 0
STM Transaction Protocol (Write)
T1 Log
Check Mode

13. T1 1
0 0
STM Transaction Protocol (Write)
T1 Log
T1 Write

14. T1 1
0 0
STM Transaction Protocol (Write)
T1 Log
T1 Write

15. T1 1
0 0
STM Transaction Protocol (Write)
T1 Log
T1 Write

16. T1 11 0
STM Transaction Protocol (Write)
T1 Log
T1 Commit

17. STM Conflict Detection
T1 Log
0 00 0
T2 Log
0 0

18. STM Conflict Detection
T1 Log
0 0
0 0
T2 Log
0 0
T1 Read

19. STM Conflict Detection
T1 Log
0 0
0 0
T2 Log
0 0
T2 Write
T2 1

20. STM Conflict Detection
T1 Log
0 0
0 0
T2 Log
0 0
T1 Validate
T2 1

21. STM Conflict Detection
T1 Log
0 0
T2 Log
0 0
T1 Abort
T2 1

22. Hybrid Transaction Protocol
STM – HTM conflicts detected by lock word accesses
• Explicit XABORT if locked by another transaction
• HTM reads – Read the metadata word
• STM writes modify the metadata word
• Causes HTM to abort
• HTM writes – Increment version number
• Causes STM read invalidation / HTM abort

23. Abstract Locking
&
Undo Operations

24. Abstract Locks for STM
Open / Boosted Atomic Method
Body
Acquire abstract locks
Release abstract locks
If top level transaction
Open / Boosted Atomic Method
Body
Acquire abstract locks
Log abstract locks
If nested transaction
Log undo operations

25. Abstract Locks for Hybrid TM
Open / Boosted Atomic Method
Body
Validate abstract locks
Open / Boosted Atomic Method
Body
Acquire abstract locks
Log abstract locks
If top level is HTM
Log undo operations

26. Why Validation Works
• HTM vs STM
• If abstract locks conflict they must touch some same
physical words in the abstract locking data structure
— otherwise they could not detect the conflict
Open / Boosted Atomic Method
Body
Validate abstract locks

27. Why Validation Works
• HTM vs STM
• If abstract locks conflict they must touch some same
physical words in the abstract locking data structure
— otherwise they could not detect the conflict
• HTM vs HTM
• No conflict in the locking data structure because all
accesses to it are reads
• Any real conflicts that exist will occur on the actual
data structure
Open / Boosted Atomic Method
Body
Validate abstract locks

28. STM and HTM Methods
STM needs logging HTM doesn't
Different actions during read/write
Different actions for abstract locks
HTM should fall back to STM

29. STM and HTM Methods
STM needs logging HTM doesn't
Different actions during read/write
Different actions for abstract locks
HTM should fall back to STM
Maintain separate HTM and STM versions of methods

30. STM Method Variants
Original
(Non-txnal)

31. STM Method Variants
Original
(Non-txnal)
Can call
A B

32. STM Method Variants
Original
(Non-txnal)
Top-level Txn
STM
Can call
A B

33. STM Method Variants
Original
(Non-txnal)
Top-level Txn
STM
Can call
A B

34. STM Method Variants
STM
Transactionalized
Original
(Non-txnal)
Top-level Txn
STM
Can call
A B

35. STM Method Variants
STM
Transactionalized
Original
(Non-txnal)
Top-level Txn
STM
Can call
A B

36. STM Method Variants
STM
Transactionalized
Original
(Non-txnal)
New nested Txn
STM
Top-level Txn
STM
Can call
A B

37. STM Method Variants
STM
Transactionalized
Original
(Non-txnal)
New nested Txn
STM
Top-level Txn
STM
Can call
A B

38. STM Method Variants
STM
Transactionalized
Original
(Non-txnal)
New nested Txn
STM
Top-level Txn
STM
Can call
A B

39. STM Method Variants
STM
Transactionalized
Original
(Non-txnal)
New nested Txn
STM
Top-level Txn
STM
Can call
A B

40. Hybrid TM Method Variants
Original
(Non-txnal)

41. Hybrid TM Method Variants
Original
(Non-txnal)Can call
A B

42. Hybrid TM Method Variants
Router
Original
(Non-txnal)Can call
A B

43. Hybrid TM Method Variants
Router
Original
(Non-txnal)Can call
A B

44. Hybrid TM Method Variants
Top-level Txn
STM
Router
Original
(Non-txnal)Can call
A B

45. Hybrid TM Method Variants
Top-level Txn
HTM
Top-level Txn
STM
Router
Original
(Non-txnal)Can call
A B

46. Hybrid TM Method Variants
HTM
Transactionalized
Top-level Txn
HTM
Top-level Txn
STM
Router
Original
(Non-txnal)Can call
A B

47. Hybrid TM Method Variants
HTM
Transactionalized
Top-level Txn
HTM
Top-level Txn
STM
Router
Original
(Non-txnal)Can call
A B

48. Hybrid TM Method Variants
Optimized
New nested Txn
HTM
HTM
Transactionalized
Top-level Txn
HTM
Top-level Txn
STM
Router
Original
(Non-txnal)Can call
A B

49. Hybrid TM Method Variants
Optimized
New nested Txn
HTM
HTM
Transactionalized
Top-level Txn
HTM
Top-level Txn
STM
Router
Original
(Non-txnal)Can call
A B

50. Hybrid TM Method Variants
Optimized
New nested Txn
HTM
HTM
Transactionalized
Top-level Txn
HTM
Top-level Txn
STM
Router
Original
(Non-txnal)Can call
A B

51. Hybrid TM Method Variants
Optimized
New nested Txn
HTM
HTM
Transactionalized
Top-level Txn
HTM
STM
Transactionalized
Top-level Txn
STM
Router
Original
(Non-txnal)Can call
A B

52. Hybrid TM Method Variants
Optimized
New nested Txn
HTM
HTM
Transactionalized
Top-level Txn
HTM
STM
Transactionalized
Top-level Txn
STM
Router
Original
(Non-txnal)Can call
A B

53. Hybrid TM Method Variants
Optimized
New nested Txn
HTM
HTM
Transactionalized
Top-level Txn
HTM
STM
Transactionalized
Top-level Txn
STM
New nested
Txn STM
Router
Original
(Non-txnal)Can call
A B
Parent is closed
A B

54. Hybrid TM Method Variants
Optimized
New nested Txn
HTM
HTM
Transactionalized
Top-level Txn
HTM
STM
Transactionalized
Top-level Txn
STM
New nested
Txn STM
Router
Original
(Non-txnal)Can call
A B
Parent is closed
A B

55. Hybrid TM Method Variants
Optimized
New nested Txn
HTM
HTM
Transactionalized
Top-level Txn
HTM
STM
Transactionalized
Top-level Txn
STM
New nested
Txn STM
Router
Original
(Non-txnal)Can call
A B
Parent is closed
A B

56. Hybrid TM Method Variants
Optimized
New nested Txn
HTM
HTM
Transactionalized
Top-level Txn
HTM
STM
Transactionalized
Top-level Txn
STM
New nested
Txn STM
Router Nested Router
Original
(Non-txnal)Can call
A B
Parent is closed
A B
Parent is open
A B

57. Hybrid TM Method Variants
Optimized
New nested Txn
HTM
HTM
Transactionalized
Top-level Txn
HTM
STM
Transactionalized
Top-level Txn
STM
New nested
Txn STM
Router Nested Router
Original
(Non-txnal)Can call
A B
Parent is closed
A B
Parent is open
A B

58. Hybrid TM Method Variants
Optimized
New nested Txn
HTM
HTM
Transactionalized
Top-level Txn
HTM
New nested
Txn HTM
STM
Transactionalized
Top-level Txn
STM
New nested
Txn STM
Router Nested Router
Original
(Non-txnal)Can call
A B
Parent is closed
A B
Parent is open
A B

59. Hybrid TM Method Variants
Optimized
New nested Txn
HTM
HTM
Transactionalized
Top-level Txn
HTM
New nested
Txn HTM
STM
Transactionalized
Top-level Txn
STM
New nested
Txn STM
Router Nested Router
Original
(Non-txnal)Can call
A B
Parent is closed
A B
Parent is open
A B

60. Hybrid TM Method Variants
Optimized
New nested Txn
HTM
HTM
Transactionalized
Top-level Txn
HTM
New nested
Txn HTM
STM
Transactionalized
Top-level Txn
STM
New nested
Txn STM
Router Nested Router
Original
(Non-txnal)Can call
A B
Parent is closed
A B
Parent is open
A B

61. XJ System Architecture

62. XJ System Architecture
XJ source code

63. XJ System Architecture
XJ source code XJ Compiler
standard Java
bytecode
compile

64. XJ System Architecture
XJ source code XJ Compiler
standard Java
bytecode XJ Rewriter
bytecode
+ run-time calls
compile load

65. XJ System Architecture
XJ source code
XJ run-time
library
XJ Compiler
standard Java
bytecode XJ Rewriter
bytecode
+ run-time calls
HTM-enabled
JVM
compile load run

66. XJ System Architecture
HTM 4-5 times faster than STM
XJ source code
XJ run-time
library
XJ Compiler
standard Java
bytecode XJ Rewriter
bytecode
+ run-time calls
HTM-enabled
JVM
compile load run

67. VM Modifications
Kept to a minimum
Modifications done on OpenJDK:
• Native methods to begin, end, and abort a HW transaction
• Made them intrinsic to the HotSpot C1/C2 compilers
Had to go through several hoops to get HTM to work with HotSpot’s
optimizing compilers

68. Results

69. Synchrobench
Micro-benchmarks to evaluate synchronization performance on various
data structures
Added ability to run multiple operations within a single transaction
(group size)
Included XJ versions of the benchmarks
• TransactionalFriendlyTreeSet
48-way, Intel Xeon E5-2690 v3 machine with 2 sockets of 12
hyperthreaded cores

70. Throughput(Normalized)
0
0.2
0.4
0.6
0.8
1
Threads
1 2 4 8 12 16 20 24 28 32 36 40 44 48
Throughput(Normalized)
0
0.2
0.4
0.6
0.8
1
Threads
1 2 4 8 12 16 20 24 28 32 36 40 44 48
Open nestedClosed nested
Group size 1
5% Updates

71. Throughput(Normalized)
0
0.2
0.4
0.6
0.8
1
Threads
1 2 4 8 12 16 20 24 28 32 36 40 44 48
Throughput(Normalized)
0
0.2
0.4
0.6
0.8
1
Threads
1 2 4 8 12 16 20 24 28 32 36 40 44 48
Group size 1
Group size 2
Open nestedClosed nested 5% Updates

72. Throughput(Normalized)
0
0.2
0.4
0.6
0.8
1
Threads
1 2 4 8 12 16 20 24 28 32 36 40 44 48
Throughput(Normalized)
0
0.2
0.4
0.6
0.8
1
Threads
1 2 4 8 12 16 20 24 28 32 36 40 44 48
Group size 1
Group size 2
Group size 4
Open nestedClosed nested 5% Updates

73. Throughput(Normalized)
0
0.2
0.4
0.6
0.8
1
Threads
1 2 4 8 12 16 20 24 28 32 36 40 44 48
Throughput(Normalized)
0
0.2
0.4
0.6
0.8
1
Threads
1 2 4 8 12 16 20 24 28 32 36 40 44 48
Group size 1
Group size 2
Group size 4
Group size 8
Open nestedClosed nested 5% Updates

74. Throughput(Normalized)
0
0.2
0.4
0.6
0.8
1
Threads
1 2 4 8 12 16 20 24 28 32 36 40 44 48
Throughput(Normalized)
0
0.2
0.4
0.6
0.8
1
Threads
1 2 4 8 12 16 20 24 28 32 36 40 44 48
Group size 1
Group size 2
Group size 4
Group size 8
Group size 16
Open nestedClosed nested 5% Updates

75. Throughput(Normalized)
0
0.2
0.4
0.6
0.8
1
Threads
1 2 4 8 12 16 20 24 28 32 36 40 44 48
Throughput(Normalized)
0
0.2
0.4
0.6
0.8
1
Threads
1 2 4 8 12 16 20 24 28 32 36 40 44 48
Group size 1
Group size 2
Group size 4
Group size 8
Group size 16
Group size 32
Open nestedClosed nested 5% Updates

76. 0
50
100
150
200
1
2
4
8
12
16
20
24
28
32
36
40
44
48
1
2
4
8
12
16
20
24
28
32
36
40
44
48
1
2
4
8
12
16
20
24
28
32
36
40
44
48
Group size 1 Group size 2 Group size 4
committedopsandabortedtxns(10
6
) open htm commits
closed htm commits
open stm commits
closed stm commits
htm aborts

77. Conclusions
STM and HTM can co-exist for nested transactions in Java
• Closed nesting — Similar to previous schemes
• Open nesting — Novel validation mechanism
• Implemented in OpenJDK on Intel TSX — Artifact evaluated

78. Conclusions
STM and HTM can co-exist for nested transactions in Java
• Closed nesting — Similar to previous schemes
• Open nesting — Novel validation mechanism
• Implemented in OpenJDK on Intel TSX — Artifact evaluated

79. Conclusions
STM and HTM can co-exist for nested transactions in Java
• Closed nesting — Similar to previous schemes
• Open nesting — Novel validation mechanism
• Implemented in OpenJDK on Intel TSX — Artifact evaluated
When it works, HTM is ~4-5× faster than STM

80. Conclusions
STM and HTM can co-exist for nested transactions in Java
• Closed nesting — Similar to previous schemes
• Open nesting — Novel validation mechanism
• Implemented in OpenJDK on Intel TSX — Artifact evaluated
When it works, HTM is ~4-5× faster than STM
Open nesting increases the envelope of effectiveness for HTM

81. Conclusions
STM and HTM can co-exist for nested transactions in Java
• Closed nesting — Similar to previous schemes
• Open nesting — Novel validation mechanism
• Implemented in OpenJDK on Intel TSX — Artifact evaluated
When it works, HTM is ~4-5× faster than STM
Open nesting increases the envelope of effectiveness for HTM
Production VM would need deeper modification