wiki

Codit Wiki

Loading information... Please wait.

Codit Blog

Posted on Monday, February 1, 2016 5:00 PM

Toon Vanhoutte by Toon Vanhoutte

This blog post series aims to provide an introduction to AS4, a recent B2B messaging standard. It provides the basic insights on how AS4 establishes a reliable and secure message exchange between trading partners. This is a jump start to get to know the ins and outs of the standard, as the specifications are quite complex to understand without any prior introduction.

The content of this blog post series has also been presented during an Integration Monday  session of the Integration User Group, within the Microsoft community. Check out the video, which includes some nice demos of the interoperability of Microsoft .NET with AS4.

What is AS4?

AS4 defines a standardized, secure and reliable exchange of messages, containing one or multiple payloads. These are the key features of the messaging standard:

  • Interoperable: AS4 is defined as an OASIS standard. It is built on top of existing standards, which have proven interoperability in the past: MIME, SOAP and WS-Security.
  • Secure: AS4 uses a subset of the WS-Security features in order to assure message non repudiation and data confidentiality.
  • Reliable: AS4 guarantees once-and-only-once delivery, via the exchange of acknowledgements and additional requirements on both send and receive side.
  • Payload   agnostic: AS4 can exchange any kind of payloads (XML, JSON, HL7, EDI, binary …) and it supports also multiple payloads being sent in one AS4 message.

A Short History

AS4 originates from ebXML (Electronic Business XML), which enables enterprises to conduct business over the internet. Here’s a short overview of how AS4 was originated:

  • 2002 – ebMS 2.0 : ebXML Messaging Services version 2.0 defines a communications-protocol neutral method for exchanging electronic business messages, using a flexible enveloping technique.

  • 2007 – ebMS 3.0 Core Features : ebXML Messaging Services version 3.0 core specification defines message packaging and provides several options to deal with message exchange patterns, error handling, security and reliability.

  • 2010 – ebMS 3.0 Advanced Features : ebXML Messaging Services version 3.0 advanced features adds some extra features and scenarios to ebMS 3.0. The most important are: multi-hop messaging, message bundling and a split/join mechanism for large messages.

  • 2013 – AS4 Profile of ebMS 3.0 : this AS4 profile has the success of AS2 in mind and applies the same “just enough” design principles in order to narrow down the options of ebMS 3.0, which results in a simplified usage profile of ebMS 3.0. AS4 defines 3 possible conformances clauses: minimal client (minimal feature set), light client and the eb handler (richest feature set).

This history makes AS4 a rather difficult standard to read, as it is spread across multiple specifications.

Where is AS4 applied?

Even though it is a rather recent standard, AS4 gains more and more interest from organizations that want to expand their B2B/B2C capabilities. In Europe, eSENS e-delivery promotes and funds the usage of AS4 for all cross-border communication between member states. This results in many large-scale projects that are looking into AS4. Also outside the European borders, AS4 is adopted. Here’s a non-exhaustive list of known projects / organizations that are applying AS4, spread over different sectors, with implementation levels going from design phase, over piloting, until production ready.

  • Australia:  Superstream  Pensions
  • Europe:  PEPPOL  (Pan-European Public Procurement Online)
  • Europe:  ENTSOG  (European Network of Transmission System Operators for Gas)
  • Europe:  EESSI  (Electronic Exchange of Social Security Information)
  • Europe:  e-CODEX  (e-Justice Communication via Online Data Exchange)
  • Japan:  JEITA  (Japan Electronics and Information Technology Industries Association)
  • World Wide:  IATA  (International Air Transport Association)

Read more on AS4 in the coming days as this blog series will progress...

Categories: Technology
Tags: AS4
written by: Toon Vanhoutte

Posted on Monday, January 25, 2016 10:46 AM

Jonas Van der Biest by Jonas Van der Biest

This blog post describes how you can automatically update your Swagger definition in Azure API Management with TFS Build.

We would like to automate the import Swagger API functionality in Windows Azure API Management management portal. You could to this manually using the following screen:

There are Azure Commandlets for Azure Management API but there is currently a bug. When using the "Import-AzureApiManagementApi" command it is not possible to replace the API definition in Azure API Management  (this is possible in the portal, but not in the PowerShell cmdlet).

But no worries, we can still use REST and invoke the REST API from powershell (as supported in PS 5.0) ourselves.

Enable Rest API

First we will enable REST API on Azure Management Service. Go to API Management, navigate to Security and select "Enable API Management REST API".

Generate an Access Token

Next, generate an Access Token and copy the complete string so we can use it later. Also copy the name of your management API tenant (in upper left corner).

Create the following PowerShell build script and do a check-in on source control

Add a build step to your build definition

Next, we will configure our build and add an extra deploystep to deploy to Azure Management API.

Select a PowerShell script by navigating to Utility

Drag the task just after the Azure Web App Deployment. You might have noticed that there is a sleep of 15 seconds in the PowerShell script. This is because our website will need some time to startup. We could always move the API deployment step a bit further in the build chain but we don't want to have our API and Azure Management API out of sync that long.

Select the script from source control and enter the parameters.

 

The PowerShell parameters are:

-link "link to swagger url" -tenantName "api mgmt tenant name" -sas "shared access key"

And we should get a successful build after queueing a new one.

Wonderful, one step less to worry about when deploying!!

Categories: Azure
written by: Jonas Van der Biest

Posted on Thursday, January 21, 2016 1:16 PM

Falco Lannoo by Falco Lannoo

As part of the Managed Services Team in Codit, I have come across some various BizTalk tracking issues and bad configurations. I wrote this blog post to summarize the best-practices for tracking in BizTalk and to help you troubleshoot tracking issues.

How it works

The BizTalk tracking service works with 2 separate artifacts:

  • The TDDS service (the BizTalk tracking host instance)
  • The SQL Agent job “TrackedMessages_Copy_BizTalkMsgBoxDb”

Both services have their own particular tasks for moving the tracking data to the BizTalkDTADb database.

The TDDS service is responsible for moving all the tracking events from the BizTalkMsgBoxDb to the BizTalkDtaDb and/or the BAMPrimaryImport database. The TDDS service is running in the host instances that have the “Allow Host Tracking” enabled.

The SQL job is responsible for moving the actual body contents from the BizTalkMsgBoxDb to the BizTalkDtaDb.

Best Practices

Configure a dedicated tracking host.

  • Do not bind this host instance to any other BizTalk artifact (receive location / send port / orchestration).
  • Create only 1 dedicated host per message box database (so in most cases, only one host should be configured for tracking).

Only activate tracking where it is necessary.

This is a very important point that is overlooked a lot during the development of a flow. To avoid any performance problems, the BizTalkDtaDb database needs to be as small as possible. Only activate tracking where it is absolutely  necessary.

  • Global tracking: If you don’t need any tracking at all, you can disable global tracking. Tracking can be very useful to detect bottlenecks / performance issues for a certain application, so I wouldn’t recommend disabling tracking on a global level.
  • Pipeline tracking: If it’s not necessary to track any events for a particular pipeline, disable the pipeline tracking. Note: you won’t see any events for that pipeline, even if the tracking on the ports are enabled!
  • Port tracking: You can define tracking for a certain port. It’s important to think about which ports you want to track. Too much tracking can lead to a large tracking database.

Disable orchestration start and end shape tracking.

  • This tracking is only useful for the orchestration debugger, which is rarely used on a production environment. This can have a large impact on the performance if there are a lot of long-running orchestrations. When this tracking is enabled, the events are saved in the “dta_debug_trace” table in the BizTalkDtaDb, it is recommended that this table doesn’t contain more than 1 million rows.

Track the promoted context properties that need to be searchable.

  • If you need to be able to search for messages based on some context properties, make sure to enable the tracking on the property schema.

Configure the DTA purge & archive job.

  • To prevent the tracking database for growing infinitely, you need to configure this job properly.

Troubleshooting the tracking service

There can be several reasons why the tracking isn’t working as expected.

  • The SQL agent jobs or tracking host instance aren’t working properly
  • The tracking isn’t configured properly
  • Certain issues with the BizTalk runtime

The first things you need to check when you experience issues, are the BizTalk host instances and the BizTalk SQL jobs.
If the host instances are not started or the SQL job “TrackedMessages_Copy_BizTalkMsgBoxDb” doesn’t complete successfully, the tracking simply won’t work.

If those services are running ok, then you should check if the tracking is configured properly. If the tracking is not enabled for certain ports or pipelines, it’s normal that you can’t access the tracking information for those ports. Enabling the tracking should fix the issue (only for new messages).

If the above isn’t the reason why the tracking isn’t working, it’s an indication that there is something wrong in the BizTalk environment. These are the tools you can use to further troubleshoot the issues:

  • Event Viewer: In most of the cases, checking the application event log of the servers is very helpful for troubleshooting the issues. Check if you see errors related to “BAM Eventbus service” or “BizTalk Server” that can help you detect the problem.
  • Performance Counters: Using perfmon, you can check the performance counters for the “BizTalk:TDDS” service. It can be useful to verify if there are any failed TDDS batches
  • Table “TDDS_FailedTrackingData” that is located in the BizTalkDtaDb database. If there are recent entries in those tables, it means that there were some issues with moving the tracked events/data. Check the “errmsg” column for the reason why it failed.
  • TDDS tracing: You can enable TDDS tracing by adding the entry below in the btsntsvc.exe.config (or btsntsvc64.exe.config if you are running a 64 bit tracking host) file that is located in the BizTalk installation folder. After adding the entry, restart the tracking host instance. This will show you clear error messages in the log file if there are failures in the TDDS service.

<system.diagnostics>
 <switches>
  <add name="Microsoft.BizTalk.Bam.EventBus" value="1" />
 </switches>
 <trace autoflush="true" indentsize="4">
  <listeners>
   <add name="Text" type="System.Diagnostics.TextWriterTraceListener" initializeData="c:\tdds.log"/>
  </listeners> 
 </trace>
</system.diagnostics>

Out of sequence tracking streams

I would like to elaborate one particular issue we encountered at a customer.

The tracking of their BizTalk 2009 environment suddenly stopped working. There were no errors in the event log and no clear indications of issues with BizTalk. The tracking data remained in the BizTalkMsgBoxDb and wasn’t moved to the BizTalkDtaDb.

This can have a large impact, since the MessageBox database keeps growing. This can cause a lot of performance issues and eventually database size throtthling!

On this particular environment, these problems occurred after having some disk space issues on the SQL server. When the disk space issues were resolved, they noticed that the tracking of new messages weren’t visible in the BizTalk Administration console anymore but the messages were being processed correctly.

We enabled the TDDS tracing and noticed following entries:

Microsoft.BizTalk.Bam.EventBus.TDDSDecoder Run Info 1754188 records are missing from the sequence number range for this batch.
Microsoft.BizTalk.Bam.EventBus.TDDSDecoder Run Info Record with sequence number 9780138 is not in the batch. Trying to read it again..
Microsoft.BizTalk.Bam.EventBus.TDDSDecoder Run Info 1757776 records are missing from the sequence number range for this batch.
Microsoft.BizTalk.Bam.EventBus.TDDSDecoder Run Info Record with sequence number 9760524 is not in the batch. Trying to read it again..
Microsoft.BizTalk.Bam.EventBus.TDDSDecoder Run Warning Record with sequence number 9780138 was not found.
Microsoft.BizTalk.Bam.EventBus.TDDSDecoder Run Info Record with sequence number 9780139 is not in the batch. Trying to read it again..
Microsoft.BizTalk.Bam.EventBus.TDDSDecoder Run Warning Record with sequence number 7956152 was not found.
 
This error occurs when the tracking streams are out of sync.

Thanks to the blog post by Michael Stephenson we were able to resolve this problem.

The tracking streams were out of sync (possibly due to the disk space problems on the database server). Running the update query fixed the issue. Please note that it isn’t recommended to run this query on a production environment!

I hope you found this blog post useful and that you can apply some of this information in your BizTalk implementation. If you have any remarks / questions, feel free to leave a comment.

Categories: BizTalk
written by: Falco Lannoo

Posted on Wednesday, January 13, 2016 7:34 AM

Luis Delgado by Luis Delgado

In the IT world, it is typical to find job titles such as consultant, senior consultant, architect, solutions architect, etc. But are you really an architect if you don't possess business know-how? Read my opinion in this blog.

The line between a consultant and an architect is a blurry one, with more senior consultants being able to deliver architect work and architects with deeper knowledge being able to do implementation work as well. All too often, however, I've seen people being promoted to the title of architect based on their elevated level of technical prowess, but lacking key business know-how that architects need to possess.

Like with a traditional architect (you know, the kind of people that design buildings), IT architects are supposed to produce the blueprints for an IT solution: a holistic, big-picture, 30,000ft view of the solution and its components which, as a functional system, are supposed to facilitate specific organizational objectives. However, the architect should produce a solution that supports such objectives at the lowest possible cost per user. Lacking essential financial acumen or business insights, it is the second part of the previous sentence that is too often neglected.

Companies are not paying top dollar to get golden engineering solutions. Companies are paying top dollar to get even more dollars back, in the form of return on investment (ROI). Architects and senior consultants over-engineering their solutions towards technical perfection are doing their clients a disfavor by delivering solutions that are typically unnecessarily complex and expensive to maintain. Note that "at the lowest possible cost per user" does not mean that the solution needs to be cheap. All it means is that, if your organizational needs would be perfectly served by a Toyota, there is no need to build (and pay for) a Lexus.

People delivering architectural work must be trained into the essential principles of financial investment. They should also be comfortable in estimating cost and revenue/savings across time, calculate a project's Net Present Value, ROI, IRR and optimize the solution design to maximize those financial indicators. Clients must also be willing to share their cost structures with their advisers, otherwise estimating these financials is close to impossible.

Categories: Architecture
written by: Luis Delgado

Posted on Thursday, January 7, 2016 10:55 AM

Glenn Colpaert by Glenn Colpaert

With the release of cumulative update 2 for BizTalk 2013 R2 Microsoft released a potential issue/bug in the mapping components of BizTalk Server.

Status

The Microsoft Product Group has released a new package to overcome this issue, you can download it from following location: https://support.microsoft.com/en-us/kb/3119352

Details

It is advised to not install BizTalk Server 2013 R2 CU2, which was released a few weeks ago, as it contains a potential issue for all mappings due to the change introduced by the following KB: "Backward compatibility option to use XslTransform instead of XslCompileTransform is available at map level".

A more in-depth description of the problem can be found on the following blog: http://www.bizzybloggers.com/2016/01/biztalk-2013-r2-cu2-failure-has-occurred-while-loading-a-type/

All credits to Steven Van Eycken for the detailed description of the issue.

Regards,

Glenn

Categories: BizTalk
written by: Glenn Colpaert