Ever since Azure Event Grid was launched I have been super excited about it. At its launch, I wrote about what was and how it’s different from Service Bus Topics.
Later on, I wrote about why Azure Event Grid is important and how Event Grid is the heart of Azure by allowing customers, and Microsoft themselves, to build reactive applications to build innovative platforms.
But there was one problem – The adoption of Azure services was slowly increasing, very slowly.
In 2019, I’ve decided to no longer wait and try to help increase the adoption by providing feedback to product groups about why it would be great to have events and why Event Grid is different.
While some Azure services started to see the power of Azure, such as Azure Key Vault, not all of them did – Either they haven’t heard about Azure Event Grid before, or they didn’t see the point because they already do XYZ.
That didn’t work out.
Today, we’ll talk about those XYZ scenarios and why Event Grid is different. Hopefully, this helps to put a different perspective on Azure Event Grid.
How Event Grid works
Before we start, let’s have a look at how Event Grid works.
Event publishers will emit events to a topic without knowing who the consumers are and, frankly, they don’t care. On the other hand, event subscribers can create a subscription on that topic and determine what events they are interested in. As part of that, they can define certain filter criteria such as the even type, what subject it applies to or based on data in the payload.
Once a subscription is being created, Azure Event Grid will start pushing events to your configured destination allowing you to process them. If you are using a webhook, you’ll need to first validate that you own that domain.
Azure has a small list of Azure services that are already a publisher to which you can subscribe, my most favorite example is Azure Key Vault which provides insights on certificates that are about to expire, secrets that have been updated, etc.
By subscribing to those events you can build automated certificate renewal processes and automatically evict cached secrets.
I highly recommend watching Sam Vanhoutte his “From poll to push” talk from IglooConf which provides a nice overview of Azure Event Grid.
Publishing your custom events
Don’t forget that you can become a publisher as well. By creating an Event Grid topic you can publish & subscribe to your internal events.
This is a great way to collaborate with other teams and integrate in a decoupled fashion.
A while ago, we’ve built a fully reactive ingestion pipeline with serverless technologies like Azure Functions, Data Factory & Logic Apps and were using Azure Event Grid to keep other teams up-to-date, but also react on events that were occurring on their end.
If you are building multi-tenant applications you can even publish events to a topic where the events are partition on a tenant level with Azure Event Grid Domains, super powerful.
Diagnostic Logs
Diagnostic logs are a great way to gain insights on what is going on and are available for all Azure services, but you have to opt-in for it.
As a customer, you define the kind of information that you are interested in, such as logs and metrics, and where it should be stored ranging from a storage account to other things like Event Grid.
They are a great way to gain a deep understanding of what is going while troubleshooting an issue.
How is it different from Event Grid?
This scenario seems to be the most opposite from Event Grid leverages:
- It’s intensive – You have to constantly pull the feed to determine if there are changes. This also means that you need compute to process all of this which costs money.
- It’s complex – Diagnostic logs tend to have a lot of data in them which you have to plow through, crunch it and try to figure out what kind of data you need. All of this requires a lot of compute & expertise.
- It’s not filtered –
Unfortunately, I cannot remember which Azure service gave me the feedback that we can use diagnostic logs instead of Event Grid, but it’s for totally different purposes and makes no sense to set all of this up.
Alerts
Alerts are super important. They are a way to react to things that are going on, without you having to constantly monitor your infrastructure. An alert can be scoped to a single Azure resource or a group of resources.
When creating an alert, it is up to you to define the criteria of what is considered. In Azure, you can create alerts based on a variety of things such as metrics, queries, security violations, Azure consumption and more.
How is it different from Event Grid?
Both mechanisms are using a push-based mechanism for which we need to subscribe – Either by creating an alert that defines the criteria (filter) or by creating a new subscription.
However, the big difference is in how they are being processed. Events typically contain contextual information about what happened, while an alert only provides information about the alert criteria which can also be based on aggregated data.
The issue with alerts is that it’s hard to decide on them as you need to do additional lookups rather than interpret the self-contained event.
Lastly, alerts are not always based on near-real-time information which makes it no longer reactive. In Azure Monitor, there is typically a delay of 5 minutes before the metrics show up which can be a very long time depending on your application.
Example
Imagine you’d want to start a process when an Azure Function has triggered. As of today, you can create an alert for every function invocation based on their metric, but what function was it? We don’t know, so we have to query the Azure Function API first, and then proceed.
With an event, however, the function name would be in the subject which we’ve filtered on and in the payload so we can immediately take action.
Change feed
A change feed provides a full track record of what has changed by using a log-based data store.
Since it’s a cursor-based approach, you can replay all of the incremental changes in the data stream and act accordingly. However, that means that you’ll have to keep track of where you’ve stopped processing (and thus where your cursor was) so you don’t have to re-process the whole stream next time.
If you’re interested in learning more about logs-based data stores, I highly recommend reading “I ♥ Logs”.
How is it different from Event Grid?
One of the main benefits of Event Grid is that it’s push-based, meaning that we can sit and wait until somebody calls us.
With a change feed, however, the consumption model is not so trivial:
- It’s intensive – You have to constantly pull the feed to determine if there are changes. This also means that you need compute to process all of this which costs money.
- It’s not stateless – You have to keep track of what you have processed before and make sure all your replicas are up-to-date.
Changefeeds are great because they provide the ** same kind of data** that an event would contain and they provide a full history of what occurred but reactive workloads typically don’t care about history only what just happened.
Every tool has its use-cases and that’s where change feed and Event Grid is different, so we should be forced to use change feed.
That’s something the Azure Storage team does very well – They provide Event Grid events for the reactive workloads to create on things, such as new blobs uploaded, while they also provide a change feed to keep track of what happened when.
You will not start an Azure Data Factory pipeline because you’ve been polling the change feed all day long, but you would if an event triggers it.
Example
A change feed is a great way of going through the full history and takes action on that. For example, you can re-process full history because a bug in your system corrupted your application.
If you want to send a welcome email to your new employee because a new document was created you can subscribe to just those events.
Webhooks
Webhooks are an easy way for customers to subscribe to changes and let the service push your updates.
How is it different from Event Grid?
It sounds very similar, doesn’t it?! That’s because Event Grid is a superset of webhooks.
While webhooks are being pushed to subscribers, Event Grid allows you to have an overview of all your active subscriptions without the publisher having to worry about who’s interested and doing their bookkeeping.
Another downside is that typical webhook providers do not always support filtering, certainly not with the same granularity as Event Grid does.
That said, I don’t think it’s that much of a problem but I’d say Event Grid for 3rd parties would be great to have as we can re-use the same approach as other events. This is something I’ve talked about in my “Azure Event Grid, the heart of Azure” blog post.
I’ve touched upon this topic in my “Adventures of building a (multi-tenant) PaaS platform” talk at IglooConf 2020 if you are interested in learning more.
Conclusion
Azure Event Grid is the foundation for building reactive & serverless applications, and hopefully, 2020 is the year where we’ll see a tremendous increase of first-class Azure publishers.
Azure services that emit Event Grid events are not only pleasing their customer, they are also helping themselves and other Azure services! For example, Azure Service Bus emits events when new messages are available on a queue while it was empty before. Azure Logic Apps uses this to stop pulling an empty queue until there are new messages again.
On the other hand, the same service can also use their events to feed their change feed by simply subscribing to their events and appending them to their change feed.
Sean Feldman started an awesome initiative with Azure Event Wishlist where you can open an issue for a certain event that you’d like an Azure service to emit. Have some good ideas? Don’t hesitate!
Event Grid is powerful, it is time to let it shine.
Today
Thanks for reading,
Tom.
Subscribe to our RSS feed