wiki

Codit Wiki

Loading information... Please wait.

Codit Blog

Posted on Wednesday, May 24, 2017 12:00 PM

Toon Vanhoutte by Toon Vanhoutte

This blog post covers several ways to optimize the monitoring experience of your Logic Apps. Logic Apps provide two ways to add functional data to your workflow runs: tracked properties and outputs. This functional data can be used to improve operational experience, to find a specific run based on business data and to serve as a base for reporting. Both options are discussed in depth and compared.

Introduction

Tracked Properties

As the documentation states: Tracked properties can be added onto actions in the workflow definition to track inputs or outputs in diagnostics data. This can be useful if you wish to track data like an "order ID" in your telemetry

Tracked properties can be added in code view to your Logic Apps actions in this way:

Outputs

As the documentation states: Outputs specify information that can be returned from a workflow run. For example, if you have a specific status or value that you want to track for each run, you can include that data in the run outputs. The data appears in the Management REST API for that run, and in the management UI for that run in the Azure portal. You can also flow these outputs to other external systems like PowerBI for creating dashboards. Outputs are not used to respond to incoming requests on the Service REST API

Outputs can be added in code view to your Logic Apps workflow in this way. Remark that you can also specify their data type:

Runtime Behaviour

Do runtime exceptions influence tracking?

What happens if my Logic App fails? Are the business properties still tracked or are they only available in a happy scenario? After some testing, I can conclude the following:

  • Tracked properties are only tracked if the appropriate action is executed.
  • Outputs are only tracked if the whole Logic App completes successfully.

When using tracked properties, my advice is to assign them to the first action (not supported on triggers) of your Logic App, so they are certainly tracked.

Do tracking exceptions influence runtime?

Another important aspect on tracking / monitoring is the fact that it should never influence the runtime behaviour. It's unacceptable that a Logic App run fails, because a specific data field to be tracked is missing. Let's have a look what happens in case we try to track a property that does not exist, e.g.: "Reference": "@{triggerBody()['xxx']}"

In both cases the Logic App ends in a failed state, however there are some differences:

Tracked Properties

  • The action that is configured with tracked properties fails.
  • There is a TrackedPropertiesEvaluationFailed exception.

Outputs

  • All actions completed successfully, despite the fact that the Logic App run failed.
  • In the Run Details, an exception is shown for the specific output. Depending on the scenario, I've encountered two exceptions:

    >  The template language expression 'triggerBody()['xxx']' cannot be evaluated because property 'xxx' doesn't exist, available properties are 'OrderId, Customer, Product, Quantity'.

    > The provided value for the workflow parameter 'Reference' at line '1' and column '57' is not valid.

Below you can find a screen capture of the workflow details.

This is not the desired behaviour. Luckily, we can easily avoid this. Follow this advice and you'll be fine:

  • For strings: Use the question mark operator to reference potential null properties of an object without a runtime error. In case of a null reference, an empty string will be tracked. Example: "Reference": "@{triggerBody()?['xxx']}"

  • For integers: Use the int function to convert the property to an integer, in combination with the coalesce function which returns the first non-null object in the arguments passed. Example: "Quantity": "@int(coalesce(triggerBody()?['xxx'], 0))"

Azure Portal

The first place where issues are investigated is the Azure Portal. So it would be very helpful to have these business properties displayed over there. Tracked properties are nowhere to be found in the Azure Portal, but fortunately the outputs are displayed nicely in the Run Details section. There is however no option to search for a specific Logic App run, based on these outputs.

Operations Management Suite

In this post, I won't cover the Logic Apps integration with Operations Management Suite (OMS) in depth, as there are already good resources available on the web. If you are new to this topic, be sure to check out this blog post and this webcast.

Important take-aways on Logic Apps integration with OMS:

  • Only tracked properties are available in OMS, there's no trace of outputs within OMS.
  • OMS can be used to easily search a specific run, based on business properties
    > The status field is the status of the particular action, not of the complete run
    > The resource runId can be used to find back the run details in the Azure Portal

  • OMS can be used to easily create reports on the tracked business properties
  • Think about data retention. The free pricing tier gives you data retention of 7 days.

Management API

Another way to search for particular Logic Apps runs, is by using the Azure Service Management API. This might come in handy if you want to develop your own dashboard on top of Logic Apps.
The documentation clearly describes operations available to retrieve historical Logic Apps run in a programmatic way. This can be done easily with the Microsoft.Azure.Management.Logic library. Outputs are easier to access than tracked properties, as they reside on the WorkflowRun level. The following code snippet can serve as a starting point:

Unfortunately, the filter operations that are supported are limited to the ones available in the Azure Portal:

This means you can only find specific Logic Apps runs - based on outputs or tracked properties - if you navigate through all of them, which is not feasible from a performance perspective.

Conclusion

Comparison

Below you can find a comparison table of the investigated features. My current advice is to use both of them, as you'll get the most in return. 

Feature Tracked Properties Outputs
Tracking Level Action Logic App Run
Runtime behaviour    If action executes If Logic App succeeds 
Azure Portal No integration Visible in run details
OMS Full integration No integration
Management API No search No search

Feedback to Product Team

Monitoring is one of areas that needs some improvements to provide all we need for operating and monitoring an integration environment. Here are my feature requests in order of priority:

  • Ensure outputs are also monitored in case a workflow run fails. Vote here.
  • Provide OMS integration for outputs. Vote here.
  • Extend the search filter in the Management API. Vote here.
  • Provide a unique URL to directly redirect to a specific Logic App run.

Hope this post clarifies the monitoring capabilities of Logic Apps. 

Feel free to vote for the mentioned feature requests!
Toon

Categories: Azure
Tags: Logic Apps
written by: Toon Vanhoutte

Posted on Friday, May 19, 2017 11:29 AM

Toon Vanhoutte by Toon Vanhoutte

Receiving data from a SQL table and processing it towards other back-end systems; It's a very common use case in integration. Logic Apps has all the required functionality in its toolbox to fulfill this integration need. This blog post explains how you can do this in a reliable fashion, in case you're dealing with mission critical interfaces where no data loss is accepted.

Scenario

Let's discuss the scenario briefly.  We need to consume data from the following table.  All orders with the status New must be processed!

The table can be created with the following SQL statement:

First Attempt

Solution

To receive the data, I prefer to create a stored procedure. This avoids maintaining potentially complex SQL queries within your Logic App. The following stored procedure selects the first order with status New and updates its status to Processed in the same statement. Remark that it also returns the @@ROWCOUNT, as this will come in handy in the next steps.

The Logic App fires with a Recurrence trigger.  The stored procedure gets executed and via the ReturnCode we can easily determine whether it returned an order or not.  In case an order is retrieved, its further processing can be performed, which will not be covered in this post.

Evaluation

If you have a BizTalk background, this is a similar approach on using a polling SQL receive location. One very important difference: the BizTalk receive adapter executes the stored procedure within the same distributed transaction as it persists the data in the MessageBox, whereas Logic Apps is completely built on API's that have no notion of MSDTC at all.

In failure situations, when a database shuts down or the network connection drops, it could be that the order is already marked as Processed, but it never reaches the Logic App. Depending on the returned error code, your Logic App will end up in a Failed state without clear description or the Logic App will retry automatically (for error codes 429 and 5xx). In both situations you're facing data loss, which is not acceptable for our scenario.

Second attempt

Solution

We need to come up with a reliable way of receiving the data. Therefore, I suggest to implement a similar pattern as the Azure Service Bus Peek-Lock. Data is received in 2 phases:

  1. You mark the data as Peeked, which means it has been assigned to a receiving process
  2. You mark the data as Completed, which means it has been received by the receiving process

Next to these two explicit processing steps, there must be a background task which reprocesses messages that have the Peeked status for a too long duration. This makes our solution more resilient.

Let's create the first stored procedure that marks the order as Peeked.

The second stored procedure accepts the OrderId and marks the order as Completed.

The third stored procedure should be executed by a background process, as it sets the status back to New for all orders that have the Peeked status for more than 1 hour.

Let's consume now the two stored procedures from within our Logic App.  First we Peek for a new order and when we received it, the order gets Completed.  The OrderId is retrieved via this expression: @body('Execute_PeekNewOrder_stored_procedure')?['ResultSets']['Table1'][0]['Id']

The background task could be executed by a SQL Agent Job (SQL Server only) or by another Logic App that is fired every hour.

Evaluation

Happy with the result? Not a 100%! What if something goes wrong during further downstream processing of the order? The only way to reprocess the message is by changing its status in the origin database, which can be a quite cumbersome experience for operators. Why can't we just resume the Logic App in case of an issue?

Third Attempt

Solution

As explained over here, Logic Apps has an extremely powerful mechanism of resubmitting workflows. Because Logic Apps has - at the time of writing - no triggers for SQL Server, a resubmit of the Recurrence trigger is quite useless. Therefore I only want to complete my order when I'm sure that I'll be able to resubmit it if something fails during its further processing. This can be achieved by splitting the Logic App in two separate workflows.

The first Logic App peeks for the order and parses the result into a JSON representation. This JSON is passed to the next Logic App.

The second Logic App gets invoked by the first one.  This Logic App completes the order first and performs afterwards the further processing.  In case something goes wrong, a resubmit of the second Logic App can be initiated.

Evaluation

Very happy with the result as:

  • The data is received from the SQL table in a reliable fashion
  • The data can be resumed in case further processing fails

Conclusion

Don't forget that every action is HTTP based, which can have an impact on reliability. Consider a two-phased approach for receiving data, in case you cannot afford message loss. The same principle can also by applied on receiving files: read the file content in one action and delete the file in another action. Always think upfront about resume / resubmit scenarios. Triggers are better suited for resubmit than actions, so if there are triggers available: always use them!

This may sound overkill to you, as these considerations will require some additional effort. My advice is to determine first if your business scenario must cover such edge case failure situations. If yes, this post can be a starting point for you final solution design.

Liked this post? Feel free to share with others!
Toon

 

 

Categories: Azure
written by: Toon Vanhoutte

Posted on Friday, May 5, 2017 4:56 PM

Massimo Crippa by Massimo Crippa

The Analytics module in Azure API Management provides insights about the health and usage levels of your APIs, to identify key trends that impact the business. Analytics also provides a number of filtering and sorting options, to better understand who is using what, but what if I want more? For example, how about drill down reports or getting mobile access?

I am a big fan of Power BI so, let's combine the power of Azure Functions and the simplicity of the APIM REST APIs, to flow the analytics data to Power BI.

The picture below displays my scenario: Azure functions connect and combine APIs from different Azure services (AAD, APIM, Storage) to create a smooth and lightweight integration.

It's a serverless architecture which means that we don't have to worry about the infrastructure so we can focus on the business logic, having rapid iterations and a faster time to market.

The APIM analytics (aggregated data) can be read by calling the report REST API. This information can then be written to Azure Tables and automatically synchronized with Power BI. 

 

Function

The Azure function:

  1. Is triggered via HTTP POST. It accepts a body parameter with the report name (byApi, byGeo, byOperation, byProduct, bySubscrition, byUser) and the day to export.

  2. Calls the AAD token endpoint using the resource owner password flow to get the access token to authorize the ARM call.

  3. Calls the APIM rest API (https://management.azure.com/subscriptions/9124e1d1-c144-1ec2-7cb2-bef226961d93/resourceGroups/rg-apim/providers/Microsoft.ApiManagement/service/apim-codit-dev/reports/bySubscription?api-version=2016-07-07&$filter=timestamp%20ge%20datetime'2017-02-15T00:00:00'%20and%20timestamp%20le%20datetime'2017-02-16T00:00:00')

  4. Iterate through the JTokens in the response body to build a collection of IEnumerable<DynamicTableEntity> that is passed to the CloudTable.ExecuteBatch to persist the data in the azure storage. 

Because I am using a second function to extract and load (to azure storage) additional APIM tables (e.g. apis, products, users etc..), I found this article very useful on reusing code in different Azure functions.

I created a logic app to trigger the functions multiple times, one per report to be exported. The code can support any new aggregation or additional fields added in the future without any modification.

Power BI

Using Power BI desktop I put together some visualizations and pushed them to the Power BI service. The report dataset is synced with the Azure tables one time per day, which is configurable. Here below, you can see screens from my mobile phone (left) and the desktop experience (right).

Conclusion

Even though the same result can be achieved using other Azure services like Webjobs or Data Factory, Azure functions provide multiple benefits like a simple programming model, the abstraction of servers and the possibility to use a simple editor to build, test and monitor the code without leaving your browser. That's a perfect fit for quick development cycle, faster adaptation that gains business advantages, isn't it?

Cheers,

Massimo

Categories: Azure
written by: Massimo Crippa

Posted on Thursday, April 27, 2017 5:22 PM

Toon Vanhoutte by Toon Vanhoutte

This blog post dives into the details of how you can achieve batching with Logic Apps. Batching is still a highly demanded feature for a middle-ware layer. It's mostly introduced to reduce the performance impact on the target system or for functional purposes. Let's have a closer look.

Scenario

For this blog post, I decided to try to batch the following XML message.  As Logic Apps supports JSON natively, we can assume that a similar setup will work quite easily for JSON messages.  Remark that the XML snippet below contains an XML declaration, so pure string appending won't work.  Also namespaces are included.

Requirements

I came up with the following requirements for my batching solution:

  • External message store: in integration I like to avoid long-running workflow instances at all time. Therefore I prefer messages to be stored somewhere out-of-the-process, waiting to be batched, instead of keeping them active in a singleton workflow instance (e.g. BizTalk sequential convoy).

  • Message and metadata together: I want to avoid to store the message in a specific place and the metadata in another one. Keep them together, to simplify development and maintenance.

  • Native Logic Apps integration: preferably I can leverage an Azure service, that has native and smooth integration with Azure Logic Apps. It must ensure we can reliably assign messages to a specific batch and we must be able to remove them easily from the message store.

  • Multiple batch release triggers: I want to support multiple ways to decide when a batch can be released.
    > # Messages: send out batches containing each X messages
    > Time: send out a batch at a specific time of the day
    > External Trigger: release the batch when an external trigger is receive

Solution

After some analysis, I was convinced that Azure Service Bus queues are a good fit:

  • External message store: the messages can be queued for a long time in an Azure Service Bus queue.

  • Message and metadata together: the message is placed together with its properties on the queue. Each batch configuration can have its own queue assigned.

  • Native Logic Apps integration: there is a Service Bus connector to receive multiple messages inside one Logic App instance. With the peak-lock pattern, you can reliably assign messages to a batch and remove them from the queue.

  • Multiple batch release triggers:
    > # Messages: In the Service Bus connector, you can choose how many messages you want to receive in one Logic App instance

    > Time
    : Service Bus has a great property ScheduledEnqueueTimeUtc, which ensures that a message becomes only visible on the queue from a specific moment in time. This is a great way to schedule messages to be releases at a specific time, without the need for an external scheduler.

    > External Trigger
    : The Logic App can be easily instantiated via the native HTTP Request trigger

 

Implementation

Batching Store

The goal of this workflow is to put the message on a specific queue for batching purpose.  This Logic App is very straightforward to implement. Add a Request trigger to receive the messages that need to be batched and use the Send Message Service Bus connector to send the message to a specific queue.

In case you want to release the batch only at a specific moment in time, you must provide a value for the ScheduledEnqueueTimeUtc property in the advanced settings.

Batching Release

This is the more complex part of the solution. The first challenge is to receive for example 3 messages in one Logic App instance. My first attempt failed, because there is apparently a different behaviour in the Service Bus receive trigger and action:

  • When one or more messages arrive in a queue: this trigger receives messages in a batch from a Service Bus queue, but it creates for every message a specific Logic App instance. This is not desired for our scenario, but can be very useful in high throughput scenarios.

  • Get messages from a queue: this action can receive multiple messages in batch from a Service Bus queue. This results in an array of Service Bus messages, inside one Logic App instance. This is the result that we want for this batching exercise!

Let's use the peak-lock pattern to ensure reliability and receive 3 messages in one batch:

As a result, we get this JSON array back from the Service Bus connector:

The challenge is to parse this array, decode the base64 content in the ContentData and create a valid XML batch message from it.  I tried several complex Logic App expressions, but realized soon that Azure Functions is better suited to take care of this complicated parsing.  I created the following Azure Fuction, as a Generic Webhook C# type:

Let's consume this function now from within our Logic App.  There is seamless integration with Logic Apps, which is really great!


As an output of the GetBatchMessage Azure Funtion, I get the following XML :-)

Large Messages

This solution is very nice, but what with large messages? Recently, I wrote a Service Bus connector that uses the claim check pattern, which exchanges large payloads via Blob Storage. In this batching scenario we can also leverage this functionality. When I have open sourced this project, I'll update this blog with a working example.  Stay tuned for more!

Conclusion

This is a great and flexible way to perform batching within Logic Apps. It really demonstrates the power of the Better Together story with Azure Logic Apps, Service Bus and Functions. I'm sure this is not the only way to perform batching in Logic Apps, so do not hesitate to share your solution for this common integration challenge in the comments section below!

I hope this gave you some fresh insights in the capabilities of Azure Logic Apps!
Toon

Categories: Azure
Tags: Logic Apps
written by: Toon Vanhoutte

Posted on Thursday, April 6, 2017 4:16 PM

Toon Vanhoutte by Toon Vanhoutte

"Logic Apps is not a fit for real enterprise integration!". This is a statement I often heard throughout the last year. In this blog, I'll summarize the main reasons why people tend to make such statements. Even though their reasoning is completely true, I can't disagree more with their final conclusion.

Scenario

Let's get our hands dirty and try to develop the following old-school integration scenario: receive a flat file from disk, archive it, convert it to XML, transform the result and finally write the XML message again to disk.

Based on some documentation on the internet, I got the following scenario working in minutes!  The Logic App kicks off with a Reccurrence trigger. It loops over all files in the input folder, reads their content and deletes them.  The flat file content gets archived.  Next, we decode it and perform a transformation towards the desired format.  The result gets written to the file system again, where we check if we need to perform a Create file or an Update file action.

Issues

When this Logic Apps gets analysed by people with a background of enterprise integration, you'll get the following remarks:

We cannot reuse this development effort in similar integrations!
As an example: sending a file will typically be a combination of multiple Logic App actions working together: inject values into the file name and decide whether an update or create should happen. It's not desired to redevelop this functionality repeatedly for every Logic App that requires it! This will cost us too much time and we will end-up with different flavours of this functionality, which is not maintainable.

This workflow is too tightly coupled with its receiving transport protocol!
If we want to change the receiving transport protocol from FILE to FTP, we almost need to redevelop the Logic App from scratch again, as the subsequent actions have a hard dependency on it. We are even not sure if the FTP connector has the same actions available as the FILE connector. We have also integrations that are using SFTP in production, but are tested with FTP in the other environments. With this design, we cannot switch easily between environments.

There is no option to resume this workflow if it fails during processing!
For our solution, the resubmit functionality of Logic Apps is useless. The Logic App is initiated by a simple Reccurrence trigger. If we initiate a resubmit, it behaves exactly the same as if a new trigger is fired. If something goes wrong inside the Logic App, it will take a huge (manual) effort to get that message reprocessed again. In case of enterprise integration, we're processing thousands of files, so it will be a cumbersome experience for our operations personnel.

This is spaghetti interfacing. There's no loose coupling or pub/sub involved!
This is real point-to-point interfacing! If we design our new integration platform in this way, we will end up with unmanageable spaghetti interfacing. Production proven integration patterns like loosely coupling and publish / subscribe are nowhere to be found. A redeployment is required in case another backend system is interested in this dataset.

This workflow handles multiple files, which results in difficult troubleshooting!
This specific Logic App loops over every single file in the input folder. Logic Apps for each statements are - by default - executed in parallel, so an issue with one file will not block the processing of the other files. In case there is a problem with one file, the Logic App will end up in a Failed state. To resolve the issue, an operator will need to scroll through each iteration until the faulted message is found. Again, a manual intervention will be required to resume this file.

Conclusion

Do I agree with the statements above? Of course I do, I wrote them myself! However, this is exercise should not result in concluding that Logic Apps can't handle the job! Contrary, this is intended to create awareness about potential pitfalls when doing enterprise integration within Logic Apps. Understanding these pitfalls is a first step in designing Logic Apps solutions for robust enterprise integration!

Want to learn more? Check out my Integration Monday session on this matter!

Hope to see you there!
Toon

Categories: Azure
Tags: EAI, Logic Apps
written by: Toon Vanhoutte