This document discusses latch contention issues that can occur in the BizTalk MessageBox database when processing high volumes of business transactions. It notes that while adding more BizTalk servers initially improved throughput, latency significantly increased. The root cause is explained as latch convoys occurring on indexes in the database under heavy concurrent workloads. Specific wait types and sessions experiencing latch waits are shown. The contention is seen on tables with high insert, select, update and delete rates against small tables. Workarounds suggested minimizing load on the MessageBox through application design changes.

1. Ricardo Torre
Product Strategist – BizTalk360

17. Primary
(default file group)
Misc Data Misc Indexes Predicate Data
Predicate
Indexes
Message Data
Message
Indexes
For more information: BizTalk Server MessageBox Database Filegroups SQL Script

22. 2 BizTalk Servers:
• 0.5 Business Transactions / second
• 6.3 seconds avg. latency (5 sec wait time)
4 BizTalk Servers:
• 0.69 Business Transactions / second
• 11.2 seconds avg. latency
BUT…

24. When a new row is inserted into an index, SQL Server will
use the following algorithm to execute the modification:
1.Record a log entry that row has been modified.
2.Traverse the B-tree to locate the correct page to hold the new record.
3.“Latch” (lock) the page with PAGELATCH_EX, preventing others from modifying it.
4.Add the row to the page and, if needed, mark the page as dirty.
5.“Unlatch” the page
Eventually this can lead to “latch convoys” in highly
concurrent workloads

25. Session
ID Latch Type Database SchemaObject Name Index Name Wait Type
241
PAGELATCH_E
X
BizTalkMsgBoxD
b dbo Spool IX_Spool 109
273
PAGELATCH_E
X
BizTalkMsgBoxD
b dbo Spool IX_Spool 63
348
PAGELATCH_E
X
BizTalkMsgBoxD
b dbo Spool IX_Spool 63
177
PAGELATCH_E
X
BizTalkMsgBoxD
b dbo Spool IX_Spool 62
302
PAGELATCH_E
X
BizTalkMsgBoxD
b dbo Spool IX_Spool 62
311
PAGELATCH_E
X
BizTalkMsgBoxD
b dbo Spool IX_Spool 62
108
PAGELATCH_E
X
BizTalkMsgBoxD
b dbo Spool IX_Spool 47
110
PAGELATCH_E
X
BizTalkMsgBoxD
b dbo Spool IX_Spool 47

26. Contention on Small Number of Pages in Heavily Accessed Tables
(Select, Update, Delete, Insert all running concurrently on multiple threads)
High rate of Insert/Select/Update/Delete access patterns against very small tables
We have observed latch contention in MessageBox schema on 16 Cores and above
This problem is now being reported by production deployments, waits 100+ms
Impact:
Latch waits > 50-100ms+
Observed waits on internal SQL Structure
• ACCESS_METHODS_HOBT_VIRTUAL_ROOT
Workaround
Minimize load on MessageBox
• App design to avoid persistence points
• Use inline sends if necessary
Multi-MessageBox may alleviate this problem somewhat

28. Q & A