wiki

Codit Wiki

Loading information... Please wait.

Codit Blog

Sending large messages to AppFabric session-enabled queues

 

Posted on Sunday, June 5, 2011 11:19 PM

Sam Vanhoutte by Sam Vanhoutte

Sending large messages to AppFabric session-enabled queues]]>
We will demonstrate how to send large messages in an atomic batch of chunks over a queue.

In the recent CTP of Windows Azure AppFabric, we can see a lot of new rich-messaging features.  In my previous blog post, I blogged about the publish-subscribe features.  Another interesting feature is the use of sessions when sending messages to a queue.  And that’s what this post is about.

We will demonstrate how to send large messages in an atomic batch of chunks over a queue.

The new queues

The new AppFabric Service Bus queues are much richer in functionality and features than the V1-message buffers or the Windows Azure storage queues.  A list of the biggest differences:

  • Reliable & durable storage (no limit on the TTL-time to live).
  • The maximum size of a queue is 1GB. (100Mb in the CTP version).
  • A message can measure up to 256KB.
  • Different messaging API’s are available: REST, .NET client and WCF bindings.
  • Transactional support in sending messages to a queue.
  • De-duplication of messages.
  • Deferring of messages.

As you can see, the limit of a message size is 256Kb, which is much more than the storage queue limit, but can still be not enough in certain cases.  And that is where sessions come in the picture.

Session concept

The concept of sessions allows receiving messages that belong to a certain logical group all by the same receiver.  This is done by specifying a SessionId on a message.  Receivers can listen on a specific session, or can lock the session for their usage on a first come first served basis.

Sessions can also be used to implement Request-Reply patterns over a queue.  But more on that can be expected in a future blog post.

 

Creating a session-enabled queue

By default, queues are created session-less.  The usage of sessions can only be used on queues that have session-support enabled.  The following code extract shows how to create a queue with sessions enabled.  (notice the usage of the ServiceBusNamespaceClient object.  This is the object you’ll always use in administrative operations.

sbClient.CreateQueue("qName", new QueueDescription { RequiresSession = true });

In this way, the queue will now be created and be able to handle sessions.

 

Sending messages to the queue

The only specific thing that is needed to send messages in a session to a queue, is to define the SessionId property on the specific message.  In this case, I also write a value to the message properties to indicate that the last message of a session is being sent.

var message = BrokeredMessage.CreateMessage(msgContent);
message.SessionId = "MySessionId"; 
message.MessageId = "MyMessageId"; 
if (isLastMessage) { message.Properties["LastMessageInSession"] = true; }

 

Receiving messages from a session

The specific thing on receiving messages in a session, is to use a SessionReceiver.  This receiver will make sure that all messages it receives will belong to the same session.  It is possible to listen on a specific session (by passing in the session name) and to specify a session timeout.

QueueClient queueClient = msgFactory.CreateQueueClient("LargeFileQueue");
SessionReceiver sessionReceiver = queueClient.AcceptSessionReceiver();
while (sessionReceiver.TryReceive(TimeSpan.FromSeconds(10), out receivedMessage))
{
	Console.WriteLine("Message received in session " + receivedMessage.SessionId);
	if (receivedMessage.Properties.ContainsKey("LastMessageInSession"))
	{     
		Console.WriteLine("Last message of session received");
		break; 
	} 
}

In the above mentioned sample, I am looping until I receive the ‘last message in the session’.

The sample Sending large messages in chunks to a session

In the sample that I upload, I send large messages in chunks to a session-enable queue.  Using multiple receivers, I am guaranteed that each file will only be received by exactly one receiver.

 

Functionality

The sample contains a sender where you can specify a directory where all messages in that directory will be picked up and submitted (in parallel) to a session-enabled queue, if the checkbox is checked.  Otherwise they will be sent to a session-less queue. (to demonstrate that messages will be received in random order).

The receiver is a console app that listens for incoming messages based on the session.  You can startup multiple receivers, to indicate that each receiver will receive messages to its own session.

sessioncapture

These are the most important design steps I took:

 

Sending messages

  • In the sample, I am using a Parallel.ForEach to loop over a bunch of files and send them in parallel.  This way, I am sure that messages won’t arrive in sequence on the queue.
  • Each large message is being sent in a transaction to the queue. 
  • On every message, I use the file name as the session id.
  • On the last message in the batch, I write the ‘LastMessageInSession’ property to make this visible to the receiver.

Receiving messages

  • I am using the PeekLock receive method to make sure I only remove the messages from the queue, when the full session has been received.
  • I receive the messages in a TransactionScope.  This makes sure that the session on itself is being rollbacked, in case of an exception.

The code of this sample can be found here.

Conclusions

This post was a new example of another great messaging feature that comes with the AppFabric Service Bus Enhancements.  We explored the sessions to group messages to the same receiver.  This scenario in this post demonstrated that we can split large messages in smaller chunks but still are able to handle them in one atomic batch.

Sam Vanhoutte

written by: Sam Vanhoutte
Please see: http://msdn.microsoft.com/en-us/library/windowsazure/hh767287(v=vs.103).aspx
Kurt Friedrich