We were using Arcus, a library for building applications that use Azure Event Grid, which provides tooling for writing integration tests.

By using Azure Relay Hybrid Connection it subscribes for new events emitted to Azure Event Grid and then flows them back into the tests to verify them.

This flow was fine when I was working all by myself but once I started creating PRs I noticed that the outcome became unreliable.

For every pull request we automatically run 2 builds:

1. One CI that verifies that the codebase is ok
2. One build that builds, tests and pushes a preview package to MyGet.org.

This means that for every pull request the tests were running in parallel creating 2 simultaneous consumers which is exactly where the problem lies.

By design Azure Relay will randomly distribute incoming client connections across all active clients meaning that events from test run A will end up in test run B and vice versa. Our tests were fine, but the events were not routed correctly.

We could merge these builds together but if you have scheduled builds to run your integration tests or somebody running locally against the same testing infrastructure this will only get worse.

A more scalable approach

In order to tackle this approach we came up with a new testing infrastructure that is more robust and allows you to run the tests in parallel across multiple machines and multiple tests in parallel:

As you can see we got rid of Azure Relay and replaced it with a Logic App that subscribes for all new events. For every event, it will simply queue a new message on an Azure Service Bus Topic allowing us to fan out across multiple subscriptions.

With our newly introduced ServiceBusEventConsumerHost, you can now consume messages from an Azure Service Bus Topic where it will create a dedicated subscription every time you start the host.

This allows you to fully scope one subscription per test run. This will guarantee that every test run will get all events during that run, even when running tests in parallel.

Want to get started? Read our documentation and provision your testing infrastructure and you’re good to go!

Thanks for reading,

Tom.