Codit Wiki

Loading information... Please wait.

Codit Blog

Posted on Monday, March 2, 2015 4:58 PM

Sriram Narayanan by Sriram Narayanan

Microsoft recently announced the general availability of its cloud-based Azure Machine Learning service for Predictive Analytics. This post is a quick read about Azure Machine Learning's features, what it means to Data Scientists & Developers and some important Microsoft market news surrounding the topic.

Microsoft has unleashed its Azure Machine Learning as a general availability from 18-02-2015 onwards. Microsoft’s driving factor behind Azure Machine Learning is “to offer a fully-managed cloud service that will enable data scientists and developers to efficiently embed predictive analytics into their applications, helping organizations use massive amounts of data and bring all the benefits of the cloud to machine learning”. 

A 7 Point Features Overview :

  1. Azure Machine Learning is a  fully managed service - which means you do not need to buy any hardware nor manage VMs manually
  2. A browser-based machine learning IDE that enables quick creation and automation of  machine learning workflows
  3. Built-in ML libraries that can be drag/dropped to jump-start your predictive analytics solutions
  4. Features a library of sample experiments, R and Python packages and best-in-class algorithms from Microsoft businesses
  5. Supports R and Python custom code with shared workspaces
  6. Creation and discovery of web services, train/retrain your models through APIs, scale web services on demand and configure diagnostics for service monitoring and debugging
  7. Publish, consume, monetize and brand Machine Learning web services that expose trained models directly using the Azure Machine Learning Marketplace 

4 Advantages for Data Scientists?

  • No setup time, ready to start and no more computing resource limitations
  • Azure marketplace to utilize existing models or publish/monetize your new models
  • Supports familiar R, Python languages and reuse of existing scripts
  • Simple and easy configurations for modeling and deployment 

4 What's in it for IT Developers?

  • Brings prediction capabilities to the masses and available to non-experts
  • Opens up the possibility to utilize ML models in day-to-day IT  applications and infrastructure
  • Predictive models would be a big plus to interpret the huge data that would result from the Internet Of Things(IOT)
  • Ready to use ML APIs , sample experiments to learn from, Machine Learning blogs and forums to support 

4 Initial Challenges :

  • IT Pros: Support for familiar languages like C# would make it more quickly explorable
  • IT Pros: “Drag/Drop” and “Quick Start” examples might help in designing basic models but the selection of algorithms, methods and tuning the models for real-world examples would require statistics, science and mathematical knowledge
  • Data Scientists : Spreading the word to non-Microsoft line of scientists/researchers, adapting and shifting to the new working model from familiar environments like Matlab
  • Data Scientists : Most of the algorithms and methods are available considering its initial launch but still lacks some methods, algorithms that the data scientists might need in their experiments (feedback from a Data Scientist) 

A 2 point recent Microsoft’s Analytics Market Facts: 

  • Microsoft acquired Revolution Analytics in January 2015, a leading commercial provider of software and services using R language for statistical computing and predictive analytics to power its analytics vision
  • Gartner’s Magic Quadrant for Advanced Analytics Platforms for 2015 states Azure Machine Learning(preview during the evaluation) as one of Microsoft’s strength placing it as a visionary in 2015 from a niche player in 2014 

A 1 point action :

Go and try it!  Microsoft Azure Machine Learning free trial. 

Please post your comments, questions or feedback.

Categories: Azure Machine Learning
written by: Sriram Narayanan

Posted on Friday, February 20, 2015 12:20 PM

Massimo Crippa by Massimo Crippa

Good news everyone! The Nevatech team have just released a major new version of their SOA and API Management solution, Sentinet 4.0.
This release comes up with many improvements, performance enhancements and fixes. You can find all the details in the release notes, but here I want to have a look at what we consider to be the coolest new features in this release.

Repository object dependencies

The dependencies feature allows Sentinet users to track Repository objects dependencies on each other, and provides Sentinet users with impact and change analysis. 
In the picture below I collected the dependencies of three different objects (Backend service, Virtual Service and Access Rule). Clicking on the shapes in the graph you can then navigate to the object for further analysis.

Sentinet 4.0 also enables to configure declarative dependencies. For example you can document a "virtual services chain" editing the dependency diagram and adding objects and relations with a drag and drop.

Message Processing pipeline

Sentinet 4.0 adds the capability to configure any virtual services with built-in and custom message processing components. The Message Processing Pipeline graphical Designer significantly enhance Sentinet product capability to implement runtime messages processing, messages transformations and other custom behaviors.

Message Processing Pipeline may consist of any number of Message Processing components that can be executed based on multiple flexible conditional criteria.

The Message Processing Pipeline can be extended with custom components. Development and deployment of the custom Message Processing components is much easier than extending Sentinet with custom WCF endpoint and service behaviors. All WCF extensibility points still remain available in the product.

Schema first design.

The bigger design-time governance improvements is the "Schema-First" design option. This feature allows the design and management of the physical services structure (interfaces, operations, data schemas etc.) without re-uploading new WSDL documents and without forcing to create a new service version.


As simple as it is. Any change of the Repository is audited with all the details of the change initiated by a Sentinet user.


No configuration data will be lost during upgrade to the Sentinet 4.0. All the Sentinet physical and virtual services and other associated objects will be available immediately after the upgrade.

Enjoy this new major release, feel free to comment below and let us know what are your thoughts on it!




Categories: Sentinet
written by: Massimo Crippa

Posted on Wednesday, January 28, 2015 3:40 PM

Glenn Colpaert by Glenn Colpaert

In this blog post I will show you how to create a scheduling job in Windows Azure Scheduler. I will also talk about some scenarios where you can use the Azure Scheduler in Hybrid setup with BizTalk.


As Integration developers we sometimes need to implement interfaces that need to trigger specific processes or resources at certain times or schedules.
As you all know, the problem with BizTalk is that there are no out of the box solutions to develop these kind of trigger/scheduling scenarios.

There are already couple of good blog post out there that take you around the possibilities of having these kind of triggers setup on-premise.
You could for example use SQL Agent Job Scheduling, Windows Task Scheduling or the community developed Scheduled Task Adapter.
Be sure to take a look at this good overview by Michael Stephenson on following location:

The intention of this blogpost is to put another option out there: using the Microsoft Azure Scheduler to get those triggers into your BizTalk/Integration platform.

About the Azure Scheduler

The Azure Scheduler lets you create jobs in the cloud that invoke services (inside and outside Azure), it’s great for recurring actions and the jobs can run on any schedule. The Azure scheduler is a reliable cloud service and is designed to operate even in the face of network or machine failures.

More information on the Azure Scheduler can be found on following location:

The Concept/Scenario

The concept I’m going to describe below is to create a canvas or scenario where the Azure Scheduler can be used in combination with BizTalk Server.

Inside our datacenter we have a SQL Server that contains all the synchronized master data from our SAP system this master data is used for daily and monthly reporting.
Next to that we have a SAP system that is used for our day to day businesses: bookings, invoices, planning…
Every night, except during the weekend our master data needs to be synchronized with the data from our SAP system.

For this synchronization we want to use a reliable scheduler option that is easy and fast to implement: the Azure Scheduler.

The Implementation

In this implementation description I’m not going to talk about the BizTalk implementation of this scenario, but rather focus on the setup necessary to get the trigger inside the BizTalk environment to trigger our synchronization flow. From a BizTalk implementation view we need a HTTP endpoint exposed on our environment. This HTTP endpoint will be triggered by the Azure Scheduler.

In this example I am hosting my endpoint on following location:
If you need more information on how to create a HTTP endpoint inside BizTalk you can check following link:

Once we have our HTTP endpoint and our flow ready all we need to do is setup the Azure Scheduler to send a trigger to our endpoint and we’re good to go. In the next few steps I will show you how easy it is to create this kind of behavior.

The Azure Scheduler

Once you're logged in into the Azure Portal choose New - App Services - Scheduler - Custom Create.

Once you clicked custom create following screen will popup.
In this screen you will need to specify your Region and the Job Collection you want to have your Scheduling Job associated with.

If you do not have any Job Collection yet, you can create a new one by specifying a specific name.

Click on the arrow to continue creating the scheduling job.

In the next screen you can specify the specific Job Action. After you specified an appropriate name for your Job Action you can choose an Action Type.

In our case we selected the HTTP action, but other options are: HTTPS and Storage Queues.  Specify POST as your Method and fill in the BizTalk HTTP endpoint.

Next to that you can also specify the body content, in our case we added the <AzureMasterDataTrigger /> tag. This is to identify the trigger inside BizTalk.

Click on the arrow to continue creating the scheduling job.

The final screen will allow you to create a schedule on which your job will run. In our case we will implement a recurring job that will run every week on every week day.

Next to that you can choose a starting time and the data/time until when to run this job.

Click on finish you complete this job creation.

Based on your configuration and setup the job will start running or will start running in the future. Inside the Azure Portal you can see a history of when the Job ran and what status it has.

You can see an example of the tracking history below. More details are available inside the Azure Portal once you’ve created the job.


As there is no out of the box solution for triggering in BizTalk we sometimes need to be creative as developers.

Using the Azure Scheduler can be an option to bring triggering options to BizTalk. It’s fairly simple and creating and configuring a Scheduling Job literally takes a couple of minutes.

In this blogpost I showed you how to create a Scheduling Job and in what kind of scenario’s you can use the Azure Scheduler in combination with BizTalk.

Happy Triggering!

Glenn Colpaert





Categories: Azure BizTalk
written by: Glenn Colpaert

Posted on Wednesday, January 21, 2015 2:19 PM

Sam Vanhoutte by Sam Vanhoutte

Azure Stream Analytics is a very interesting service on the Azure platform that allows developers and data analysts to get interesting information or events out of a stream of incoming events. The service is part of the typical IoT value chain and is positioned in the transformation part. This post describes how to get started.

Azure Stream Analytics is a very interesting service on the Azure platform that allows developers and data analysts to get interesting information or events out of a stream of incoming events.  The service is part of the typical IoT value chain and is positioned in the transformation part.  The following picture is taken from a Microsoft presentation on IoT and shows Stream Analytics, connected with Event Hubs.


This post explains how you can start using Azure Stream Analytics and covers some troubleshooting tips that should save you some time.  My scenario will explain how you can send data through the Azure Event Hubs and how you can apply the standing queries of Azure Stream Analytics to get some added value out of the incoming stream.

In my scenarios and this post, I will solely focus on Azure Event Hubs as the input for my jobs.  You can use blobs as well, but I believe Event Hubs is the closest match with the integration world.

Typical scenarios for Complex Event Processing

I tried to describe some of the typical scenarios where Stream analytics can be used.  But on the high level, you can say that any scenario that needs to get intelligence out of incoming streams by correlating events should be a good fit for ASA (the abbreviation of Azure Stream Analytics).

Noise reduction

When talking with customers on frequency of incoming events (for example sensor data), we see often that there's a tendency to send data in a higher frequency that mostly needed.  While this has some advantages (better accuracy), this is not always needed.  The higher the frequency of ingested events, the higher the ingestion costs.  (potential extra data transfer costs on the device side and higher number of incoming events).  Typically the ingestion costs are not that high (when using Azure Event Hubs, for example you pay around 2 cents for 1mio events), but the long term storage cost is always higher (as the data mostly increases each month and you pay for the same data every month again). 

That's why it would be good to reduce the incoming stream with the high frequency into an aggregated (averages/sums)  output stream to the long term storage.  Then still you benefit from the higher accuracy, but you save on the expensive cold storage.

Enrich events with reference data

It is possible to join the incoming data with reference tables/data.  This allows you to build in decision logic. For example, you could have a table with all the devices and their region or customer.  And you could use that extra information to join with the telemetry information. (using the DeviceId) and aggregate per region, customer etc.

Leverage multiple outputs (pub-sub style)

When using Azure Event Hubs, it is perfectly possible to create multiple ASA jobs that share the same EventHub as their input.  This allows to get multiple results, calculations or events out of the incoming stream.  For example, one job could extract data for archiving, while another job just looks for anomalies, or event detection.

Detect special events, anomalies or trends

The canonical example, used in complex event processing, is the TollBooth or traffic speed scenario.  Every car passing a toll booth or the traffic camera, results in a new event.  But the added value is in detection special cases in the stream of events. 

  • Maybe it's needed to automatically trigger an event if the duration of one car from one booth to another is shorter than the allowed time, because he is speeding.  This event can then be processed by the backend.
  • The same stream can also be used to detect traffic jams, if the average time between booths decreased in a window of time.  Here it is important to take the average values, in order to avoid the extremes of cars that are very slow, or cars that speed.
  • And at the same time, license plates could be matched against the reference data of suspected cars.  Cars that are being sought after because they are stolen, or because they are driven by suspects of crime.

Getting started with Azure Stream Analytics

We will start with a very basic scenario, where we will send events to an Event Hub and we will execute standing queries against that stream, using ASA. This is a quick step-by-step guide to create an Analysis job.

Sending data to Azure Event Hubs

In order to create an event hub, there are a lot of blog posts and samples that explain this.

The sending of data is probably the most important part, as it defines the data structure that will be processed by the ASA job.  Therefore, it is important that the event that is sent to the Event Hub is using supported structures.  At the end of this post, I have some troubleshooting tips and some limitations that I encountered.

The following code snippet sends my telemetry-event to the Event Hub, serializing it as JSON.

As you can see, we are defining a TelemetryInfo object that we send multiple times (in parallel threads) to the event hub.  We just have 10 devices here to simulate data entry.  These json-serialized objects get UTF8-encoded and are sent in the EventData object of EventHub SDK.

Creating the Analysis job

In order to create the job, you have to make sure the subscription is enabled for Azure Stream Analytics.  To do this, log on to with the subscription owner and click on preview features to enable ASA.

Log on to the Azure portal and click the ASA icon.  There, you can create a new job.  The job has to have a name, a region and a monitoring account.  (there is one monitoring account for every region).

Once this is done, it is required to specify one or more inputs, the actual query and the output to which the job has to write.

Create the inputs

Select the inputs tab in your job.

Create a new input and define the input as Data Stream (constant inflow of data).  Select Event Hub in the second window of the wizard, as that's the input we want to work with.  In the 3rd part of the wizard, we have to select the event hub we are sending our events to.  Ideally this is in the same subscription.

In the last part, we have to specify the encoding and serialization.  We will use json as UTF8

Create the output

One job can have only one output at this moment, which is unfortunate.  If you also would love to see this changed, don't hesitate to vote on my suggestion on uservoice.

In this step, we will create a blob output that takes our stream events and outputs it in a CSV file. 

For that, we click on the Output tab and create a new one.  To start, we select Blob storage.  As you can see, we can also output our data to SQL Azure, or to another Event Hub.  In the wizard, we have to provide our blob settings and the serialization/encoding, as you can see in the following screenshots.

Creating the query

The scenario we are now implementing is really like the "hello world" example of Stream Analytics.  We just take all the incoming data from the event hub and will output it to the blob storage.

Therefore, we just create the following query: SELECT * FROM TelemetryStream, where TelemetryStream is the name of our input.  

Starting and testing the job

Now we have all things configured, we can start the job.  And if everything goes well, we can run the producer (see above) and we should see that the data is getting written to our blob container, based on the properties we specified.  This allows us to easily get data and get a basic test on which we can start defining our query to the level we need.  If you have issues, please refer to the troubleshooting section of this post.

Testing queries

The query windows also allows users to test their queries against sample files, which is a big improvement since a few months.  The only challenge one has is to get a good sample file.  For that, I mostly change the output of my job to JSON/UTF and apply the SELECT * query again, resulting in a perfect good test file.  I then take that json file to upload it in the test wizard.  If the data is valid, you can easily test the query and see the output of the query in the bottom of the query screen.

Writing Analytics queries

As you can see in the source code (and in the output on our blob), the data we are sending contains the following fields:

DeviceId, ErrorReading, MainValue, Type, EventProcessedUtcTime, PartitionId, EventEnqueuedUtcTime.  The last 3 fields are added by ASA and can be used in the query as well.

Now we will dive a little deeper in the query language of ASA, that really feels like plain old SQL. 

Time-based aggregation

One of the most common aggregation methods that is needed (explained in the noise reduction scenario) is the grouping of events by a window of time.  ASA provides 3 types of windows, of which the tumbling window (fixed time intervals) is the most common.  The details of these windows are explained well in the documentation.  The other windowing methods are Hopping (overlapping time windows) and Sliding (flexible windowing) windows.

If we now want to aggregate the data from our sample, we can easily update the query to aggregate the sensor data per minute interval, by writing the following query:

The documentation outlines good examples and other functions.

Using reference data

It is also possible to add reference data to your query, so that this data can be used to enrich the query or take extra filters or decisions.  For that, it is needed to add a new input to the query of type 'Reference data' and browse to an existing blob (I always use CSV files for this).  In this sample, I am uploading the SensorList.csv that you can view on our github account. 

Now, you can use this data to make SQL-like joins and enrich your query.  I am using the SensorList.csv that is part of the Gist I created for this blog:

 And this allows me to write the following join-query that adds the name of the sensor to the output on my blob.

Troubleshooting and diagnostics

I had some issues in the past days, trying to get things work.  I hope this post helps others in avoiding the same issues.  With this, I also want to thank the ASA team for the help they gave in fixing this.

For logging information, you should fall back on the Operation Logs of Azure that you can get to, through the dashboard of your ASA job.  But (it's a preview, don't forget that!) there were some errors that were not visible in the logs and that required support from the team.  I'll try to list some of the limitations that I encountered here.

Data Conversion Errors

When you are getting Data Conversion errors in the log, the chance is big that you are sending an unsupported object type in your event.  I had sent booleans and Arrays in my event and they were causing issues. 

I created the ToJson() method in my sender (see above), where I am json-serializing a dynamic object with only the allowed types.  This still allows my local application to work with all properties, I'm just making sure to remove or change the incorrect data types, before sending them (by using the dynamic object).

However, the intention is that the value of the unsupported fields will be set to NULL in an update that is expected shortly.  That way, you can still process the events, even if they contain invalid data types.

There is also the possibility to declare the data structure and data types in the query.  That way, you describe the data types of the different fields (without creating something specific)

The following query is an example of that

The details of the issue and the resolution I had, can be found on the msdn forum.

Next blog post

In a next blog post, I'm planning to explore the following things:

  • Output data to SQL Azure
  • ASA - Event Hub specifics
  • Dynamic reference data
  • Combining multiple input streams


Categories: Azure IoT Troubleshooting
written by: Sam Vanhoutte

Posted on Monday, January 12, 2015 11:28 AM

Tom Kerkhove by Tom Kerkhove

Automatically expiring messages in Service Bus is a nice feature but you have to be aware that it does not "automatically" move expired messages to the dead-letter queue. In some scenarios this can cause reliability loss.

Note: This content was originally released on my personal blog

Lately my team and I have been using Service Bus for Windows Server in our projects. We are hosting it on our on-premise infrastructure.
Besides the name, everything is actually pretty much the same as Azure Service Bus, except for the fact that you’re in charge of the installation & administration. Next to that, you’re stuck with an “old” version of Azure Service Bus - Currently v2.1 - which has a more limited set of features than the latest and greatest Azure Service Bus. f.e. No auto forwarding from deadletter queue to another entity.

Our scenario was built around the concept of queues and messages that should only live for a certain amount of time, also called time-to-live (TTL).
Once a message has exceeded its time-to-live it needs to expire and be moved to the deadletter queue. This is a common messaging scenario.

Getting started with Message expiration & dead-lettering

When you create a queue in Service Bus, you are actually creating two queues – Your requested queue called ‘YourQueueName’ and a dead-letter queue. Once a message is expired, it will automatically be moved to this dead-letter queue if you enable this.

Dead-lettering scenario

Enabling expiration of messages on your queue

Enabling this functionality on your queue is very easy - You create a QueueDescription based on your queue name and you enable EnableDeadLetteringOnMessageExpiration, that's it!

But hold on, it currently uses the default Time-to-Live for each message in the queue which is 10675199 days and thus almost infinite. You can specify this time window yourself by changing the DefaultMessageTimeToLive-property.

Specifying Time-to-Live to a message

You can also assign a Time-to-Live window to your individual messages. This gives you more control on expiring individual messages.

You can achieve this by assigning a window to TimeToLive property of your message.

Queue Time-to-Live vs Message Time-to-Live

As we've just seen we can specify the Time-to-Live on a queue and on a message, but you can also combine them. This allows you to define a default on the queue-level and exceptionally assign one to a message.

Bare in mind that the shortest Time-to-Live windows will be applied, whether it is your queue's default or the message's Time-to-live!

Here is a small example - Imagine you're in a scenario where you send different types of messages to one queue but certain messages need to expire sooner than the others.

Processing expired messages from the dead-letter queue

As mentioned before all your expired messages will be moved to the dead-letter queue of your queue and sit there until you process them.

Processing them is the same as processing messages from your normal queues except that the name, also called entity path, is different and applies the following pattern


If you are using the Azure SDK, you can even generate the name for your dead-letter queue with one line of code -

Note - It is important to know that there could also be other messages in the DLQ f.e. poison messages but this is out of scope of this article.

How does Service Bus expiration work?

Now that we've seen how we can enable expiration and assign the time your messages should be alive, we can take a look at how it works based on a fictional scenario.

We have a back-end system that is sending invoices to our client application over a Service Bus queue. This allows the customer to determine when he wants to receive the next invoice but to make sure that each invoice is payed in the required amount of time we want to expire the invoice and receive it back at the back-end.

Periodic Receive-scenario

As I was building this scenario I noticed that Service Bus will not monitor your queue expired messages, instead it waits until you perform a Receive. Then it will move all expired messages at that time to your dead-letter queue. Important to know that a Peek is not sufficient as messages will not expire!

In our scenario this means that when the customer doesn't request a new invoice for a very long time the messages will not expire and thus not be processed by the backend.

Although this is a big problem in our scenario since we are relying on Service Bus, it also pretty much makes sense as well - This avoids that Service Bus is permanently monitoring our messages which results in a higher load and lower scalability.

Implementing the "Monitoring" pattern

We had to redesign our scenario because of this behavior. Since we couldn't rely on the automatically expiration of messages, we stepped away from TTL and thanks to Clemens Vasters we implemented a pattern I call the "Monitoring" pattern.

Instead of only using one queue we are now using two queues - Our InvoiceQueue and a MonitorQueue.

Every time we send a message to our InvoiceQueue we will send a dummy message to the Monitoring-queue. That dummy message will contain metadata about the original invoice and will also have a ScheduledEnqueueTimeUtc based on the current UTC time with addition of the required TTL timespan. This will enqueue our dummy message but it will not be visible for the receivers until the specified time.

Once our backend system picks up one of the dummy messages from the MonitorQueue it will use the metadata to look if the message still exists on the InvoiceQueue.
If it is still present, it will remove it from the InvoiceQueue and perform the required logic because the customer failed to process it in time. If the message is already gone it will just remove it from the MonitorQueue and move on to the next one.

Alternative Monitoring pattern

One small side note is that we used Service Bus sessioning on the messages in the InvoiceQueue, this allows the backend system to retrieve the session Id from the dummy message its metadata and request that specific session on the InvoiceQueue.

Don't believe me? Try it yourself!

I've prepared a sample application that shows you the behavior with Azure Service Bus and Service Bus for Windows Server.

You can download it here.


Automatically expiring messages in Service Bus is a nice feature but you have to be aware that it does not "automatically" move expired messages to the dead-letter queue. Once you know a Receive() is required you can build your solution around this and increase your expired messages reliability.

Thanks for reading,


Categories: Azure Service Bus
written by: Tom Kerkhove