BTUG – BizTalk Tracking
Toon Vanhoutte – Integration Architect
Agenda
Configuration

Best Practices

Tools

Under the hood
Demo

Visualization

2
Demo

Setup
Specify unique reference for messages #search

Specify simulated response times of backend systems #sla

Speci...
Configuration

4
Configuration
|

Port Tracking Configuration

Only track what’s really required #performance

5
Configuration
|

Pipeline Tracking Configuration

Bug BTS2010 CU2  CU5 #solved #CU6 #kb2728652

6
Configuration
|

Orchestration Tracking Configuration

Avoid Shape start and end
#performance #orchestrationdebug

7
Configuration
|

Property Schema Tracking Configuration

Make context properties searchable #troubleshooting
Global + spec...
Configuration
|

Disable Global BizTalk Tracking

9
Under the hood

10
Under the hood
|

BizTalk Database Structure
Host Instance processes message
SQL Agent

11
Under the hood
|

BizTalk Database Structure
Host Instance processes message
SQL Agent

Message stored in MessageBox

12
Under the hood
|

BizTalk Database Structure
Host Instance processes message
SQL Agent

Message stored in MessageBox
TDDS ...
Under the hood
|

BizTalk Database Structure
Host Instance processes message
SQL Agent

Message stored in MessageBox
TDDS ...
Under the hood
|

Store messages in MessageBox database
|

Physical runtime message bodies are stored in:
|
|
|
|

Spool
M...
Under the hood
|

Process messages in MessageBox database
|

Per Host, these tables are created:
|

|

|
|

HostNameQ: Mai...
Under the hood
|

Store messages in MessageBox database

#probts2006

17
Under the hood
|

Store tracking data in MessageBox database
|

#probts2006

Host Instance serializes data and stores it i...
Under the hood
|

Copy Events from MessageBox to DTA database
|

Tracked Message Events copied by TDDS Service:
|
|

Track...
Under the hood
|

Copy Events from MessageBox to DTA database
BizTalk MessageBox
TrackingData_0_0
TrackingData_0_1
Trackin...
Under the hood
|

Copy Events from MessageBox to DTA database
BizTalk MessageBox
TrackingData_x_x

TDDS

BAM Primary Impor...
Under the hood
|

Copy Bodies from MessageBox to DTA database
|

Tracked Message Bodies copied by SQL Agent Job:
|
|

Trac...
Under the hood
|

Copy Bodies from MessageBox to DTA database
BizTalk MessageBox
Spool
MessageParts
Parts
Fragments

SQL
A...
Under the hood
|

Remove data from MessageBox
|

This is performed by SQL Agent Jobs:
|

MessageBox_Message_ManageRefCount...
Under the hood
|

Remove data from DTA
|

|

Enable + Configure the DTA Purge and Archive Job #avoidlargeDTA

Remove data ...
Under the hood
|

BizTalkDTADb Database model
|

No foreign keys #performance
|
|

|

Designed for fast insert
Results in ...
Under the hood
|

BizTalkDTADb Database model
|

No foreign keys #performance
|
|

|

Designed for fast insert
Results in ...
Under the hood
|

Create a chain of messages
Under the hood
dta_MessageInOutEvents
 Service Instance ID of messages
|
Create a chain
 Message Instance ID

SI = 1
MI ...
Tools & SDK

30
Tools & SDK
|

BizTalk Administration Console #technical #notsexy

31
Tools & SDK
|

BizTalk MessageBox Viewer #troubleshooting

32
Tools & SDK
|

BizTalk Terminator #troubleshooting

Import MBV scripts

33
Tools & SDK
|

Performance Counters #troubleshooting
|
|

Throttling state  find reason on MSDN
TDDS

34
Tools & SDK
|

Retrieve tracked message body #3ways
|

Microsoft.BizTalk.Operations.dll

35
Visualization

36
Visualization
|

Bad practice
|

Develop a dashboard directly on BizTalkDTADb #performance
|
|
|

No control on user queri...
Visualization
|

Bad practice
|

Develop custom tracking mechanism

38
Visualization
|

Good practice
|

Use default BizTalk Tracking #CID

39
Visualization
|

Good practice
|

Use BizTalk Tracking #CID

40
Visualization
|

Business Activity Monitoring? #notideal
|

Development impact
|
|

|

SQL Server Analysis Services
|

|
|...
Visualization
|

Use case #1: Message Search
|

BizTalk Administration Console

|

Codit Integration Dashboard
42
Visualization

Demo

|

Use case #1: Message Search
|

Search on context properties:
|
|
|
|
|

Application
Flow
Sender
Re...
Demo

|

Visualization

Use case #1: Message Search

44
Visualization
|

Use case #2: Operational reports
|
|
|

#managedservices

Message Load
Message Size
Database Size

45
Visualization
|

Use case #3: Pro-Active Monitoring
|

#managedservices

Custom SCOM alert: monitoring non-events #vbs
|

...
Visualization
|

Use case #4: SLA Reporting
|

Processing times of web services
|
|
|

Total processing time
Processing ti...
Demo

|

Visualization

Use case #4: SLA Reporting
GetCustomer
GetCustomerItemList

sp_CRM_GetCustomer.WCF
ORCH

rp_Master...
Visualization

Demo

Use case #4: SLA Reporting
BizTalk Processing Time = Total Processing Time – SAP Processing Time
Same...
Visualization

Demo

|

Use case #4: SLA Reporting
|

SQL Agent Job
|

BizTalkDTADB  Custom database #chaining

50
Demo

|

Visualization

Use case #4: SLA Reporting

51
Demo

|

Visualization

Use case #4: SLA Reporting

52
Visualization
|

Use case #5: Process Overview
|

Visualization of related integration flows

53
Demo

|

Visualization

Use case #5: Process Overview
|

Example: EDI Order Processing

54
Demo

|

Visualization

Use case #5: Process Overview

55
Visualization
|

Use case #6: Dashboards
|
|
|
|
|
|
|

BizTalk 360: Operational focus
AIMS for BizTalk: Behavior / trend ...
Visualization

Demo

|

Use case #6: Dashboards
|

Codit Integration Dashboard
|
|
|

Friendly Naming of technical artifac...
Demo

|

Visualization

Use case #6: Dashboards
|

Codit Integration Dashboard: resubmit

58
Questions?
You

BTUG
Microsoft

59
Upcoming SlideShare
Loading in …5
×

20131028 BTUG.be - BizTalk Tracking

1,633 views

Published on

Toon Vanhoutte s

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,633
On SlideShare
0
From Embeds
0
Number of Embeds
263
Actions
Shares
0
Downloads
26
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

20131028 BTUG.be - BizTalk Tracking

  1. 1. BTUG – BizTalk Tracking Toon Vanhoutte – Integration Architect
  2. 2. Agenda Configuration Best Practices Tools Under the hood Demo Visualization 2
  3. 3. Demo Setup Specify unique reference for messages #search Specify simulated response times of backend systems #sla Specify customer and transporter names #inspiration 3
  4. 4. Configuration 4
  5. 5. Configuration | Port Tracking Configuration Only track what’s really required #performance 5
  6. 6. Configuration | Pipeline Tracking Configuration Bug BTS2010 CU2  CU5 #solved #CU6 #kb2728652 6
  7. 7. Configuration | Orchestration Tracking Configuration Avoid Shape start and end #performance #orchestrationdebug 7
  8. 8. Configuration | Property Schema Tracking Configuration Make context properties searchable #troubleshooting Global + specific property schema #generic #consistent 8
  9. 9. Configuration | Disable Global BizTalk Tracking 9
  10. 10. Under the hood 10
  11. 11. Under the hood | BizTalk Database Structure Host Instance processes message SQL Agent 11
  12. 12. Under the hood | BizTalk Database Structure Host Instance processes message SQL Agent Message stored in MessageBox 12
  13. 13. Under the hood | BizTalk Database Structure Host Instance processes message SQL Agent Message stored in MessageBox TDDS copies events to DTA + BAM 13
  14. 14. Under the hood | BizTalk Database Structure Host Instance processes message SQL Agent Message stored in MessageBox TDDS copies events to DTA + BAM SQL Agent copies bodies to DTA 14
  15. 15. Under the hood | Store messages in MessageBox database | Physical runtime message bodies are stored in: | | | | Spool MessageParts Parts Fragments Fragmentation controlled by large message size 15
  16. 16. Under the hood | Process messages in MessageBox database | Per Host, these tables are created: | | | | HostNameQ: Main queue: each host instance polls this queue table to collect messages that are pending processing HostNameQ_Suspended: Suspended queue: queue table that holds supended messages HostNameQ_Scheduled: Scheduled queue: not used InstanceStateMessageReferences_Host: State queue: holds list of messages that have been processed but are required later Stores only reference to physical message in Spool table 16
  17. 17. Under the hood | Store messages in MessageBox database #probts2006 17
  18. 18. Under the hood | Store tracking data in MessageBox database | #probts2006 Host Instance serializes data and stores it in the MessageBox | Message events: | | | TrackingData_0_x [x: 0 - 3] TrackingData_1_x [x: 0 - 3] Message bodies: | | | | Spool MessageParts Parts Fragments 18
  19. 19. Under the hood | Copy Events from MessageBox to DTA database | Tracked Message Events copied by TDDS Service: | | Tracking Data Decode Service Runs in each host instance with tracking enabled Create a dedicated tracking host 19
  20. 20. Under the hood | Copy Events from MessageBox to DTA database BizTalk MessageBox TrackingData_0_0 TrackingData_0_1 TrackingData_0_2 TrackingData_0_3 TrackingData_1_0 TrackingData_1_1 TrackingData_1_2 TrackingData_1_3 TDDS BAM Primary Import Activity_Active Activity_ActiveRelationships Activity_Completed Activity_CompletedRelantionships Activity_Continuations BizTalk DTA Database Dta_MessageInOutEvents Dta_ServiceInstances ... 20
  21. 21. Under the hood | Copy Events from MessageBox to DTA database BizTalk MessageBox TrackingData_x_x TDDS BAM Primary Import TDDS_StreamStatus TDDS_FailedTrackingData BizTalk DTA Database TDDS_StreamStatus TDDS_FailedTrackingData TDDS_StreamStatus contains last copied sequence n° TDDS moves events with > sequence n° from TrackingData_x_x to DTA TDDS_FailedTrackingData contains tracking failures 21
  22. 22. Under the hood | Copy Bodies from MessageBox to DTA database | Tracked Message Bodies copied by SQL Agent Job: | | TrackedMessages_Copy_BizTalkMsgBoxDb Scheduled by default each minute (development: 10 seconds) Monitor the SQL Agent Jobs #SCOM #blog 22
  23. 23. Under the hood | Copy Bodies from MessageBox to DTA database BizTalk MessageBox Spool MessageParts Parts Fragments SQL Agent BizTalk DTA Database Tracking_Spoolx Tracking_Partsx Tracking_Fragmentsx [x: 1 - 2]: Always one online / one offline 23
  24. 24. Under the hood | Remove data from MessageBox | This is performed by SQL Agent Jobs: | MessageBox_Message_ManageRefCountLog_BizTalkMsgBoxDb: | | | Runs continuously (due to former SQL limitation of 1 min job schedule) This jobs checks weather a message is still referenced by instances MessageBox_Message_Cleanup_BizTalkMsgBoxDb | | | Is disabled (keep it like this) It’s called from with the infinite loop of the ManageRefCount Job This job deletes physical messages from MessageBox that have no outstanding references anymore 24
  25. 25. Under the hood | Remove data from DTA | | Enable + Configure the DTA Purge and Archive Job #avoidlargeDTA Remove data from BAM | SQL Agent Job BAM_DM_Activity moves data to BAMArchive 25
  26. 26. Under the hood | BizTalkDTADb Database model | No foreign keys #performance | | | Designed for fast insert Results in slower query No single chain ID | Difficult to reproduce the “route” within BizTalk 26
  27. 27. Under the hood | BizTalkDTADb Database model | No foreign keys #performance | | | Designed for fast insert Results in slower query No single chain ID | Difficult to reproduce the route within BizTalk 27
  28. 28. Under the hood | Create a chain of messages
  29. 29. Under the hood dta_MessageInOutEvents  Service Instance ID of messages | Create a chain  Message Instance ID SI = 1 MI = A SI = 1 MI = B SI = 2 MI = B SI = 2 MI = C SI = 3 MI = C SI = 3 MI = D 29
  30. 30. Tools & SDK 30
  31. 31. Tools & SDK | BizTalk Administration Console #technical #notsexy 31
  32. 32. Tools & SDK | BizTalk MessageBox Viewer #troubleshooting 32
  33. 33. Tools & SDK | BizTalk Terminator #troubleshooting Import MBV scripts 33
  34. 34. Tools & SDK | Performance Counters #troubleshooting | | Throttling state  find reason on MSDN TDDS 34
  35. 35. Tools & SDK | Retrieve tracked message body #3ways | Microsoft.BizTalk.Operations.dll 35
  36. 36. Visualization 36
  37. 37. Visualization | Bad practice | Develop a dashboard directly on BizTalkDTADb #performance | | | No control on user queries Impacts directly the BizTalk runtime Bad dashboard performance 37
  38. 38. Visualization | Bad practice | Develop custom tracking mechanism 38
  39. 39. Visualization | Good practice | Use default BizTalk Tracking #CID 39
  40. 40. Visualization | Good practice | Use BizTalk Tracking #CID 40
  41. 41. Visualization | Business Activity Monitoring? #notideal | Development impact | | | SQL Server Analysis Services | | | | | Tracking Profile Editor is limited BAM API is rather complex 2008 R2 required, when using SQL 2012 Deprecated BAM portal Impact on operations Not ideal OLAP cube structure Termination of activities not straightforward 41
  42. 42. Visualization | Use case #1: Message Search | BizTalk Administration Console | Codit Integration Dashboard 42
  43. 43. Visualization Demo | Use case #1: Message Search | Search on context properties: | | | | | Application Flow Sender Receiver Reference 43
  44. 44. Demo | Visualization Use case #1: Message Search 44
  45. 45. Visualization | Use case #2: Operational reports | | | #managedservices Message Load Message Size Database Size 45
  46. 46. Visualization | Use case #3: Pro-Active Monitoring | #managedservices Custom SCOM alert: monitoring non-events #vbs | E.g.: Just-In-Time ordering | | | Daily check on expected amount of received orders Query on BizTAlkDTADb Automated email alert / support ticket 46
  47. 47. Visualization | Use case #4: SLA Reporting | Processing times of web services | | | Total processing time Processing time of every backend system Processing time in BizTalk 47
  48. 48. Demo | Visualization Use case #4: SLA Reporting GetCustomer GetCustomerItemList sp_CRM_GetCustomer.WCF ORCH rp_MasterDataService.WCF GetItem GetEmployee sp_SAP_GetItem.WCF sp_AX_GetEmployee.WCF 48
  49. 49. Visualization Demo Use case #4: SLA Reporting BizTalk Processing Time = Total Processing Time – SAP Processing Time Same MessageID 1 2 3 rp_MasterDataService.WCF 8 4 sp_SAP_GetItem.WCF 7 6 Same MessageID 5 SAP Processing Time Total Processing Time | 49
  50. 50. Visualization Demo | Use case #4: SLA Reporting | SQL Agent Job | BizTalkDTADB  Custom database #chaining 50
  51. 51. Demo | Visualization Use case #4: SLA Reporting 51
  52. 52. Demo | Visualization Use case #4: SLA Reporting 52
  53. 53. Visualization | Use case #5: Process Overview | Visualization of related integration flows 53
  54. 54. Demo | Visualization Use case #5: Process Overview | Example: EDI Order Processing 54
  55. 55. Demo | Visualization Use case #5: Process Overview 55
  56. 56. Visualization | Use case #6: Dashboards | | | | | | | BizTalk 360: Operational focus AIMS for BizTalk: Behavior / trend analysis Codit Integration Dashboard: Advanced message handling Minotaur Frends Helium Moesion … 56
  57. 57. Visualization Demo | Use case #6: Dashboards | Codit Integration Dashboard | | | Friendly Naming of technical artifacts Message Search Message Resume / Resubmit | | Editing Multi selection 57
  58. 58. Demo | Visualization Use case #6: Dashboards | Codit Integration Dashboard: resubmit 58
  59. 59. Questions? You BTUG Microsoft 59

×