BizTalk Tracking: best practices & troubleshooting

How it works

The BizTalk tracking service works with 2 separate artifacts:

• The TDDS service (the BizTalk tracking host instance)
• The SQL Agent job “TrackedMessages_Copy_BizTalkMsgBoxDb”

Both services have their own particular tasks for moving the tracking data to the BizTalkDTADb database.

The TDDS service is responsible for moving all the tracking events from the BizTalkMsgBoxDb to the BizTalkDtaDb and/or the BAMPrimaryImport database. The TDDS service is running in the host instances that have the “Allow Host Tracking” enabled.

The SQL job is responsible for moving the actual body contents from the BizTalkMsgBoxDb to the BizTalkDtaDb.

Best Practices

Configure a dedicated tracking host.

• Do not bind this host instance to any other BizTalk artifact (receive location / send port / orchestration).
• Create only 1 dedicated host per message box database (so in most cases, only one host should be configured for tracking).

Only activate tracking where it is necessary.

This is a very important point that is overlooked a lot during the development of a flow. To avoid any performance problems, the BizTalkDtaDb database needs to be as small as possible. Only activate tracking where it is absolutely  necessary.

• Global tracking: If you don’t need any tracking at all, you can disable global tracking. Tracking can be very useful to detect bottlenecks / performance issues for a certain application, so I wouldn’t recommend disabling tracking on a global level.
• Pipeline tracking: If it’s not necessary to track any events for a particular pipeline, disable the pipeline tracking. Note: you won’t see any events for that pipeline, even if the tracking on the ports are enabled!
• Port tracking: You can define tracking for a certain port. It’s important to think about which ports you want to track. Too much tracking can lead to a large tracking database.

Disable orchestration start and end shape tracking.

• This tracking is only useful for the orchestration debugger, which is rarely used on a production environment. This can have a large impact on the performance if there are a lot of long-running orchestrations. When this tracking is enabled, the events are saved in the “dta_debug_trace” table in the BizTalkDtaDb, it is recommended that this table doesn’t contain more than 1 million rows.

Track the promoted context properties that need to be searchable.

• If you need to be able to search for messages based on some context properties, make sure to enable the tracking on the property schema.

Configure the DTA purge & archive job.

• To prevent the tracking database for growing infinitely, you need to configure this job properly.

Troubleshooting the tracking service

There can be several reasons why the tracking isn’t working as expected.

• The SQL agent jobs or tracking host instance aren’t working properly
• The tracking isn’t configured properly
• Certain issues with the BizTalk runtime

The first things you need to check when you experience issues, are the BizTalk host instances and the BizTalk SQL jobs.
If the host instances are not started or the SQL job “TrackedMessages_Copy_BizTalkMsgBoxDb” doesn’t complete successfully, the tracking simply won’t work.

If those services are running ok, then you should check if the tracking is configured properly. If the tracking is not enabled for certain ports or pipelines, it’s normal that you can’t access the tracking information for those ports. Enabling the tracking should fix the issue (only for new messages).

If the above isn’t the reason why the tracking isn’t working, it’s an indication that there is something wrong in the BizTalk environment. These are the tools you can use to further troubleshoot the issues:

• Event Viewer: In most of the cases, checking the application event log of the servers is very helpful for troubleshooting the issues. Check if you see errors related to “BAM Eventbus service” or “BizTalk Server” that can help you detect the problem.
• Performance Counters: Using perfmon, you can check the performance counters for the “BizTalk:TDDS” service. It can be useful to verify if there are any failed TDDS batches
• Table “TDDS_FailedTrackingData” that is located in the BizTalkDtaDb database. If there are recent entries in those tables, it means that there were some issues with moving the tracked events/data. Check the “errmsg” column for the reason why it failed.
• TDDS tracing: You can enable TDDS tracing by adding the entry below in the btsntsvc.exe.config (or btsntsvc64.exe.config if you are running a 64 bit tracking host) file that is located in the BizTalk installation folder. After adding the entry, restart the tracking host instance. This will show you clear error messages in the log file if there are failures in the TDDS service.

<system.diagnostics>
 <switches>
  <add name=”Microsoft.BizTalk.Bam.EventBus” value=”1″ />
 </switches>
 <trace autoflush=”true” indentsize=”4″>
  <listeners>
   <add name=”Text” type=”System.Diagnostics.TextWriterTraceListener” initializeData=”c:tdds.log“/>
  </listeners> 
 </trace>
</system.diagnostics>

Out of sequence tracking streams

I would like to elaborate one particular issue we encountered at a customer.

The tracking of their BizTalk 2009 environment suddenly stopped working. There were no errors in the event log and no clear indications of issues with BizTalk. The tracking data remained in the BizTalkMsgBoxDb and wasn’t moved to the BizTalkDtaDb.

This can have a large impact, since the MessageBox database keeps growing. This can cause a lot of performance issues and eventually database size throtthling!

On this particular environment, these problems occurred after having some disk space issues on the SQL server. When the disk space issues were resolved, they noticed that the tracking of new messages weren’t visible in the BizTalk Administration console anymore but the messages were being processed correctly.

We enabled the TDDS tracing and noticed following entries:

Microsoft.BizTalk.Bam.EventBus.TDDSDecoder Run Info 1754188 records are missing from the sequence number range for this batch.
Microsoft.BizTalk.Bam.EventBus.TDDSDecoder Run Info Record with sequence number 9780138 is not in the batch. Trying to read it again..
Microsoft.BizTalk.Bam.EventBus.TDDSDecoder Run Info 1757776 records are missing from the sequence number range for this batch.
Microsoft.BizTalk.Bam.EventBus.TDDSDecoder Run Info Record with sequence number 9760524 is not in the batch. Trying to read it again..
Microsoft.BizTalk.Bam.EventBus.TDDSDecoder Run Warning Record with sequence number 9780138 was not found.
Microsoft.BizTalk.Bam.EventBus.TDDSDecoder Run Info Record with sequence number 9780139 is not in the batch. Trying to read it again..
Microsoft.BizTalk.Bam.EventBus.TDDSDecoder Run Warning Record with sequence number 7956152 was not found.
 
This error occurs when the tracking streams are out of sync.

Thanks to the blog post by Michael Stephenson we were able to resolve this problem.

The tracking streams were out of sync (possibly due to the disk space problems on the database server). Running the update query fixed the issue. Please note that it isn’t recommended to run this query on a production environment!

I hope you found this blog post useful and that you can apply some of this information in your BizTalk implementation. If you have any remarks / questions, feel free to leave a comment.