wiki

Codit Wiki

Loading information... Please wait.

Codit Blog

Posted on Thursday, August 24, 2017 7:35 AM

Toon Vanhoutte by Toon Vanhoutte

I've always been intrigued by agile development and scrum methodology. Unfortunately, I never had the opportunity to work within a real agile organization. Because I strongly believe in scrum and its fundamental key principles, I've tried to apply an agile mindset on integration projects; even within very waterfall oriented organizations and projects. I'm not an expert at all on scrum methodology, I've just adopted it in a very pragmatic way.

Please do not hesitate to share your vision in the comments section below, even if it conflicts with my statements!

Important note: this post is not intended to state that when you do scrum, you should make your own interpretation of it. It's to explain how you can benefit from agile / scrum principles on integration projects, that are not using the scrum methodology at all. It's a subtle, but important difference!

1. Prototype in an early stage 

I'm working for more than 10 years on integration projects and every new assignment comes with its specific challenges: a new type of application to integrate with, a new protocol that is not supported out-of-the-box and specific non-functional requirements that you never faced before. Challenges can become risks if you do not tackle them soon. It's important to list them and to perform a short risk assessment.
 
Plan proof of concepts (PoC) to overcome these challenges. Schedule these prototyping exercises early in the project, as they might influence overall planning (e.g. extra development required) and budget (e.g. purchase of third party tool or plug-in). Perform them in an isolated sandbox environment (e.g. cloud), so you do not lose time with organizational procedures and administration overhead. A PoC must have a clear scope and success criteria defined. Real life examples where we introduced a PoC: validate performance characteristics of the BizTalk MLLP adapter, determine the best design to integrate with the brand-new Dynamics 365 for Operations (AX), test the feature set of specific Logic Apps connectors against the requirements…

2. Create a Definition of Ready 

A Definition of Ready is a kind of a prerequisite list that the development team and product owner agree on. This list contains the essential information that is required in order to kick off the development of a specific backlog item. It's important to agree on a complete, but not too extended Definition of Ready. Typical items that are listed on an integration focused Definition of Ready are: samples files, data contracts, transformation analysis, single point of contact of relying backend application.

This is a very important aspect in large integration projects. You want to avoid that your development team is constantly blocked by unclear dependencies, but on the other hand it's not advised to postpone development constantly as this imposes a risk. It's a difficult balancing exercise that requires a pragmatic approach and a decent level of flexibility.
 
It's important to liberate your development team from the task of gathering these prerequisites, so they can focus on delivering business value. In large integration projects, it's a full-time occupation to chase responsibles from the impacted teams to get the required specs or dependencies. The person taking up this responsibility has a crucial role in the success of the project. Excellent communication and people skills are a must.

3. Strive for a self-organized team

"The team lead gives direct orders to each individual team member". Get rid of this old-fashioned idea of "team work". First, the development team must be involved in estimating the effort for backlog items. In that way, you get a realistic view on the expected development progress and you get the team motivated to meet their estimates. Secondly, it's highly advised to encourage the team to become self-organized. This means they decide on how they organize themselves to get the maximum out of the team, to deliver high quality and to meet the expectations. In the beginning you need to guide them towards that direction, but it's amazing how quick they adapt to that vision.

Trust is the basis of this kind of collaboration between the team lead (or product owner) and the team. I must admit that it wasn't easy for me in the beginning, as my natural flavour is to be in control. However, the advantages are incredible: team members become highly involved, take responsibility, are better motivated and show real dedication to the project.

One might think you lose control, but nothing is less true. Depending on the development progress, you can shift the product backlog in collaboration with your stakeholders. It's also good to schedule regular demo sessions (with or without the customer) to provide your feedback to the development team.

Each team member has its own role and responsibilities within the team, even though no one ever told them to do so. Replacing one member within the team, always has a drastic impact on the team performance and behaviour. It's like the team loses part of its DNA and needs some time to adjust to the new situation. I'm blessed that I was always able to work together with highly motivated colleagues, but I can imagine it's a hell of a job to strive for a self-organized team that includes some unmotivated individuals.

4. Bridge the gap between teams

The agile vision encourages cross-functional teams, consisting of e.g. business analysts, developers and testers. Preferably one person within the team, can take multiple roles. However, if we face reality, many large organizations still have the mindset of teams per expertise (HR, Finance, .NET, Integration, Java, Testing…). Often there is no good interaction amongst these teams and they are even physically separated.

If you are part of the middleware team, you're stuck between the two teams: the ones who manage the source application and those who are developing the target system. Try to convince them to create cross-functional project teams, that are preferably working at the same place. If this is not an option, you can aim at least for a daily stand-up meeting with the most important key-players (the main analysts and developers) involved. Avoid at all time that communication always goes via a management layer, as this is time consuming and a lot of context is lost. As a last resort, you can just go on a daily basis to the floor where the team is situated and discuss the most urgent topics.

Throughout many integration projects, I've seen the importance of people and communication skills. These soft skills are a must to bridge the gap between different teams. Working full time behind your laptop on your own island, is not the key to success within integration. Collaborate on all levels and cross teams!

5. Leverage the power of mocking

In an ideal scenario, all backend services and modules we need to integrate with are already up and running. However, if we face reality, this is almost never the case. In a waterfall approach, integration would be typically scheduled in the last phase of the project, assuming all required prerequisites are ready at that moment in time. This puts a big risk on the integration layer. According to the scrum and agile principles, this must be avoided at all time.
 
This introduces a challenge for the development team. Developers need to make an abstraction of external systems their solution relies on. They must get familiar with dependency injection and / or mocking frameworks that simulate back-end applications. These techniques allow to start development of the integration layer with less prerequisites and ensure a fast delivery once the depending backend applications are ready. A great mocking framework for BizTalk Server is Transmock, definitely worth checking out if you face problems with mocking. Interesting blogs about this framework can be found here and here, I've also demonstrated its value in this presentation.

6. Introduce spikes to check connectivity

Integration is all about connecting backend systems seamlessly with each other. The setup of a new connection with a backend system can often be a real hassle: exceptions need to be made in the corporate firewall, permissions must be granted on test environments, security should be configured correctly, valid test data sets must be available, etc...
 
In many organizations, these responsibilities are spread across multiple teams and the procedures to request such changes can cause a lot of administrative and time consuming overhead. In order to avoid your development team is being blocked by such organizational waste, it is advised to put these connectivity setups early on the product backlog as "spikes". When the real development work starts in a later iteration, the connectivity setup has already been given a green light.

7. Focus first on end-to-end

This flowchart explains in depth the rules you can apply to split user stories. Integration scenarios match the best with Workflow Steps. This advice is really helpful: "Can you take a thin slice through the workflow first and enhance it with more stories later?". The first focus should be to get it work end-to-end, so that at least some data is exchanged between source and target application. This can be with a temporary data contract, within a simplified security model, without more advanced features like caching, sequence controlling, duplicate detection, batching, etc…
 
As a real-life example, we recently had the request to expose an internal API that must consume an external API to calculate distances. There were some additional requirements: the responses from the external API must be stored for a period of 1 month, to save transaction costs of the external API; authentication must be performed with the identity of the requesting legal entity, so this can be billed separately; both sync and asynchronous internal API must be exposed. The responsibility of the product owner is to find the Minimal Valuable Product (MVP). In this case, it was a synchronous internal API, without caching and with one fixed identity for the whole organization. During later phases, this API was enhanced with caching, with a dynamic identity and an async interface.
 
In some projects, requirements are written in stone upfront and are not a subject of negotiation: the interface can only be released in production if all requirements are met. In such cases, it's also a good exercise to find the MVP required for acceptance testing. In that way, you can release faster internally, which results in faster feedback from internal testing.

8. Put common non-functionals on the Definition of Done

In middleware solutions, there are often requirements on high performance, high throughput and large message handling. Most of these requirements can be tackled by applying best practices in your development: use a streaming design in order to avoid loading messages entirely in memory, reduce the number of persistence points, cache configuration values wherever it's applicable, etc…
 
It's a good practice to put such development principles on the Definition of Done, to ensure an overall quality of your product. Code reviews should check whether these best practices are applied. Only in case that specific measures need to be taken to meet exceptional performance criteria, it's advised to list these requirements explicitly as user stories on the product backlog.

"Done" also means: it's tested and can be shipped at any moment. Agree on the required level of test automation: is unit testing (white box) sufficient, do you fully rely on manual acceptance testing or is a minimal level of automated system testing (black box) required? Involve the customer in this decision, as this impacts the team composition, quality and budget. It's also a common practice to ensure automated deployment is in place, so you can release quickly, with a minimal impact. Fantastic to see that team members are challenging each other, during the daily stand-up, to verify if the Definition of Done has been respected.

9. Aim for early acceptance (testing)

In quite a lot of ERP implementations, go-live is performed in some big phases, preceded by several months of development. Mostly, acceptance testing is planned at the same pace. This means that flows being developed at the beginning of the development stage, will remain several months untouched until acceptance testing is executed. One important advice here: acceptance testing should follow the iterative development approach and not the slow-paced go-live schedule.
 
One of the base principles of an agile approach is to get fast feedback: fail fast and cheap. Early acceptance testing will ensure your integrations will be evaluated by the end users against the requirements. If possible, also involve operations in this acceptance process: they will be able to provide feedback on the monitoring, alerting, troubleshooting capabilities of your integration solution. This feedback is very useful to optimize the integration flows and to take into account these lessons learned for the subsequent development efforts. This approach can avoid a lot of refactoring afterwards…
 
Testing is not the only way to get feedback. Try to schedule demos on a regular basis, to verify if you are heading in the right direction. It's very important to adapt the demo to your stakeholders. A demo for operations can be done with technical tools, while explaining all details about reliability and security. When presenting to functional key users, keep the focus on the business process and the added value that integration brings. Try to include both source and target application, so they can witness the end result without exactly knowing what is under the hood. If you can demonstrate that you create a customer in one application and this get synchronised into two other applications within 10 seconds, you have them on your side!

10. Adapt to improve

Continuous improvement is a key to success. This improvement must be reflected on two levels: your product and your team. Let's first consider improvements on the product, of which you have two types. You have optimizations that are derived from direct feedback from your stakeholders. They provide immediate value to your product, which is in this case your integration project. These can be placed on the backlog. Secondly, there are adaptations that result in indirect value, such as refactoring. Refactoring is intended to stabilize the product, to improve its maintenance and to prepare it for change. It's advised to only refactor a codebase that is thoroughly tested, to ensure you do not introduce regression bugs.
 
Next to this, it's even more important to challenge the way the team is working and collaborating. Recurring retrospectives are the starting point, but they must result in real actions. Let the development team decide on the subjects they want to improve. Sometimes these could be quick wins: making some working agreements about collaboration, communication, code review, etc… Other actions might take more time: improve the development experience, extend the unit testing platform, optimize the ALM approach. All these actions result in better collaboration, higher productivity and faster release cycles.

I find it quite challenging to deal with such indirect improvements. I used to place them also on the backlog, whilst the team decides on their priority. We mixed them with backlog items that result in direct business value, in a 90% (direct value) / 10% (indirect value) proportion. The drawback of this approach is that not everyone is involved in indirect improvements. Another way to tackle this is reserving 1 day, every two weeks, that is dedicated for such improvements. In that way the whole team is involved in this process, which encourages the idea of having a self-organized development team.

Hope you've enjoyed this one!

Toon

Categories: Technology
Tags: Integration
written by: Toon Vanhoutte

Posted on Tuesday, August 22, 2017 4:45 PM

Tom Kerkhove by Tom Kerkhove

In this third and final article on Azure Event Grid, we'll have a look at how this relates to Azure Service Bus Topics and why they are still relevant.

No, Service Bus Topics are not dead.

I don't think that Azure Service Bus Topics are going away any time soon. Although Azure Event Grid also leverages publish-subscribe capabilities and uses the concept of "Topics", they are not the same.

Here is why:

  • Message Exchange Patterns - Azure Event Grid uses a push-push-model where all the events are pushed directly to the Event Handlers. Azure Service Bus Topics, however, are using a pulling model where the Message Processor will actively check the topic subscription if there are any new messages available. This means that the Message Processor is capable of controlling when and how many message it wants to process and thus controls the load it will handle. With Azure Event Grid you don't, so make sure your handlers can process this.

  • Differences in velocity - Since Azure Service Bus Topics are using a pull-mechanism the Message Processor is in charge of getting new messages. The advantage here is that it has full control on the pace it is processing messages. That said, if they can't keep up with the ingestion throughput the messages will only pile up until the size of the topic has been met. With Azure Event Grid however you are no longer in charge since they will push your messages to the Event Handlers. This means that your Event Handler needs to be capable to handle the load and provide some throttling to protect itself from crashing, Event Grid will retry the delivery anyway.

  • Throughput - Azure Event Grid promises 10 million events per second, per region. This is far more than what Azure Service Bus can handle, unless you distribute it across multiple Service Bus Namespaces for which they have a soft limit of 100 per subscription, that's not event close.

  • Message & Event Sizes - Azure Service Bus supports message sizes up to 256 kb for Basic/Standard or even 1MB for Premium. While I couldn't find an official limitation on the event size, my guess is that this will be similar to or less than Service Bus Basic given the throughput they promise. Of course, there is still the Claims Check pattern to bypass these limitations.

In summary, I think they have their own use-cases where I see Service Bus Topics more for a fan-out transactional processing, but with a smaller throughput where Azure Event Grid is more used as an eventing infrastructure that provides higher velocity for near-real-time processing.

As with every technology you need to compare both and see which one is best for your scenario.

Thanks for reading,

Tom Kerkhove.

Posted on Tuesday, August 22, 2017 4:40 PM

Tom Kerkhove by Tom Kerkhove

In this second article on Azure Event Grid, we'll have a look at what I'd like to see being added to Azure Event Grid.

With a nice foundation of Event Publishers & Handlers, we can expect a vast amount of new ones being added that will be available out of the box in the future.

Currently Azure Event Grid only supports events in JSON, but in the future they will support other data formats such as binary.

Let's have a look at what I'd like to see being added.

High-Level Event Grid Monitoring

I'm curious to see what the operations & monitoring story will be for Azure Event Grid.

In the following sections I will refer to other Azure services/technologies that provide similar features but I'd love to see Azure Event Grid expose those capabilities out of the box instead of integrating with Azure Application Insights for example.

This would allow us to have one centralized monitoring experience for everything related to Azure Event Grid instead of having this in another service (aka "Dependency"), since this service is running as an infrastructure service supporting other services.

High-Level Component Overview

Personally, I think it would be great to have a dashboard that shows me all the Event Publishers, Topics, Subscriptions & Handlers that are connected with each other.

My vision on this is comparable with the monitoring experience that Azure Data Factory provides:

While this is only the pipeline overview, it clearly indicates how each data set, service and pipeline are connected which each other. If you go to the Monitoring-dasbhoard it also provides you an overview of all processed data slices.

It would be nice to have a similar experience for Azure Event Grid where every failed event is listed so we can view the body of the event. This also enables us to troubleshoot why the event failed and if it's related to the content of the event or not. That said, since Azure Event Grid is a high-volume service I'm not hoping for this one. However, it would be nice to have as a premium feature at least.

Another interesting feature to have is a real-time sense of the throughput of all the events in the grid, something similar to Netflix' Vizceral (GitHub).

Performance metrics per Event Handler

Next to the high-level component overview it would be great to have some integrated performance gauges.

These gauges allow us to gain insights on the processing performance of Event Handlers allowing us to pinpoint scalability problems.

This can be comparable to what Azure Application Insights "Application Map" offers where you can see the amount of requests, success rate and failures:

Adding metadata to events

Introducing a metadata-node in the event payload would be great as well. This would allow us to specify additional context information about the event, while it's not business specific information.

By doing this; we can add telemetry information, such as correlation ids, allowing us to correlate all our telemetry across all the Event Publishers, Topics, Handlers and downstream systems.

Taking it a step further, it would be nice to use "Application Insights Analytics" (aka Kusto). This provides us to search for these events and correlate the route it took in Azure Event Grid.

Integration with Azure Data Factory

Thé Event Handler I'm looking forward to is Azure Data Factory. As of today, Azure Data Factory only supports a slicing model where it will trigger your pipeline every 1 hour, day, week, etc while in several scenarios this is not the best use-case.

It would be good if we could use Azure Event Grid to forward events for newly uploaded blobs and send that to the Data Factory Handler to trigger your pipeline. This can not only make the data processing flow feel more natural and also the performance could increase while we divide the processing in smaller pieces instead of running one big pipeline.

Summary

While Azure Event Grid is still in preview, it's always good to think about ways it can be improved and how we will operate this service. We've talked about a few features I'd like to see being added which are mainly focused to monitoring the whole infrastructure and how we can correlate this back to our other telemetry.

My biggest wish is having a high-level overview of the Event Grid components and how they are connected (which Azure Functions also lacks).

My second request would be to have an out of the box centralized monitoring experience and not being forced to use Azure Application Insights. This would mean that we are fully depending on Application Insights which adds an unnecessary dependency; which is also not that cheap, certainly not with the telemetry this service will generate.

Does this mean that I don't want to have integration with Azure Application Insights? No! Just not as the built-in way to operate Azure Event Grid.

This is of course early thinking, my vision on this can change once I use this more.

Thanks for reading,

Tom Kerkhove.

Posted on Monday, August 21, 2017 10:47 AM

Tom Kerkhove by Tom Kerkhove

Azure Event Grid is here - In this first article we'll have a look at what it is, dive into the details and discuss certain new scenarios.

Last week Microsoft announced Azure Event Grid (Preview), an event-driven service that allows you to stitch together all your components and design event-driven architectures.

Next to the built-in support for several Azure services you can also provide your own custom topics and custom webhooks that fix your needs.

By using a combination of filters and multicasting, you can create a flexible event routing mechanism that fits your needs by for example sending event A to one handler, while event B is being multicasted to multiple handlers. Read more about this here.

Azure resources can act as Event Publishers where they send a variety of events to Event Grid. By using Event Subscriptions you can then subscribe to those events and send them to an Event Handler.

The main scenarios for Azure Event Grid are serverless architectures, automation for IT/operations and integration:

  • Serverless Architectures - Trigger a Logic App when a new blob is uploaded
  • Operations - Listen & react on what happens in your subscription by subscribing to Azure Subscription changes
  • Integration - Extend existing workflows by triggering a Logic App once there is a new record in your database
  • Custom - Create your own by using application topics (aka custom topics)

The pricing for Azure Event Grid is fairly simple - You pay $0.60 per million operations and you get the first 100k operations per month for free. Operations are defined as event ingress, advanced match, delivery attempt, and management calls. Currently you only pay $0.30 since it's in public preview, more information on the pricing page.

Basically you can see Azure Event Grid as an extension service that allows you to integrate Azure Services with each other more closely while you also have the flexibility to plug in your own custom topics.

Let's have a closer look at what it has to offer.

Diving into Azure Event Grid

Event Handling at Scale

Azure Event Grid is designed as an high scalable eventing backplane which comes with some serious performance targets:

  • Guaranteed sub-second end-to-end latency (99th percentile)
  • 99.99% availability
  • 10 million events per second, per region
  • 100 million subscriptions per region
  • 50 ms publisher latency for batches of 1M

These are very big numbers which also indirectly have impact on the way we design our custom event handlers. They will need to be scalable and protect themselves from being overwhelmed and should come with a throttling mechanism.

But then again, designing for the cloud typically means that each component should be highly scalable & resilient so this should not be an exception.

Durable Message Delivery

Every event will be pushed to the required Event Handler based on the configured routing. For this, Azure Event Grid provides durable message delivery with an at-least-once delivery.

By using retries with exponential backoff, Event Grid keeps on sending events to the Event Handler until it acknowledges the request with either an HTTP 200 OK or HTTP 202 Accepted.

The Event Handler needs to be capable of processing the event in less than one minute, otherwise Event Grid will consider it as failed and retry it. This means that all Event Handlers should be idempotent to avoid creating invalid state in your system.

However, if your Event Handler is unable to process the event in time and Event Grid has been retrying for up to 24h, 2h in public preview, it will expire the event and stop retrying.

In summary, Event Grid guarantees an at-least-once delivery for all your events but you as an Event Handler are still in charge of being capable of processing the event in time. This also means that it should be able to preserve performance when they are dealing with load spikes.

It is also interesting to see what really happens with the expired events. Do they really just go away or will there be a fallback event stream to which they are forwarded for later processing? In general, I think expiration of events will work but in certain scenarios I see a case where having the fallback event stream is a valuable asset for mission critical event-driven flows.

You can read more on durable message delivery here.

How about security?

Azure Event Grid offers a variety of security controls on all levels:

  • Managing security on the Event Grid resource itself is done with Role-based Access Control (RBAC). It allows you to define granular control to the correct people. It's a good practice to use the least-priviledge principle, but that is applicable to all Azure resources. More information here.
  • Webhook Validation - Each newly registered webhook needs to be validated by Azure Event Grid first. This is to prove that you have ownership over the endpoint. The service will send a validation token to the webhook, which the webhook implementer needs to send back as a validation. It's important to note that only HTTPS webhooks are supported. More information here.
  • Event Subscription uses Role-based Access Control (RBAC) on the Event Grid resource where the person creating a new subscription needs to have the Microsoft.EventGrid/EventSubscriptions/Write permissions.
  • Publishers need to use SAS Tokens or key authentication when they want to publish an event to a topic. SAS tokens allow you to scope the access you grant to a certain resource in Event Grid for a certain amount of time. This is similar to the approach Azure Storage & Azure Service Bus use.

The current security model looks fine to me, although it would be nice if there would be a concept of SAS tokens with a stored access policysimilar to Azure Storage. This would allow us to issue tokens for a certain entity, while still having the capability to revoke access in case we need this, i.e. when a token was compromised.

An alternative to SAS stored access policies would be to be able to create multiple authorization rulessimilar to Azure Service Bus, where we can use the key approach for authentication while still having the capability to have more granular control over whom uses what key and being able to revoke it for one publisher only, instead of revoking it for all publishers.

You can read more on security & authentication here.

Imagine the possibilities

Integration with other Azure services

As of today there are only a few Azure services that integrate with Azure Event Grid but there are a lot of them coming.

Here are a couple of them that I would love to use:

  • Use API Management as a public-facing endpoint where all events are transformed and sent over to Azure Event Grid. This would allow us to use API Management as a webhook proxy between the 3rd party and Azure Event Grid. More on this later in the post
  • Streamlined event processing for Application Insights custom events where it acts as an Event Publisher. By doing this we can push them to our data store so that we can use it in our Power BI reporting, instead of having to export all telemetry and setting up a processing pipeline for that, as described here
  • Real-time auditing & change notifications for Azure Key Vault
    • Publish events when a new version of a Key or Secret was added to notify dependent processes about this so they can fetch the latest version
    • Real-time auditing by subscribing to changes on the access policies
  • Sending events when alerts in Azure Monitor are triggered would be very useful. In the past I've written about how using webhooks for processing alerts instead of emails are more interesting as you can trigger an automation workflow such as Logic Apps. If an alert would send an event to Azure Event Grid we can take it even a step further and create dedicated handlers per alert or alert group. You can already achieve this with Logic Apps & Service Bus Topics as of today but with Event Grid this comes out of the box and makes it more easy to create certain routings
  • Trigger an Azure Data Factory when an event occurs, i.e. when a blob was added to an Azure Storage container
  • Send an event when Azure Traffic Manager detects a probe that is unhealthy

New way of handling webhook events?

When we want to provide 3rd parties to send notifications to a webhook we need to provide a public endpoint which they can call. Typically, these just take the event and queue them for later processing allowing the 3rd party to move on as we handle the event at our own pace.

The "problem" here is that we still need to host an API middleware somewhere; be it an Azure Function, Web App, Api App, etc; that just handles this message. Even if you use Azure API Management, you still need to have the middleware running behind the API Management proxy since you can't push directly to a topic.

Wouldn't it be nice if we can get rid of that host and let API Management push the requests directly to Azure Event Grid so that it can fan-out all the external notifications to the required processors?

That said, this assumes that you don't do any validation or other business logic before the webhook middleware pushes to the topic for processing. If you need this capability, you will have to stick with hosting your own middleware I'm afraid.

Unified integration between APIs

Currently when you are using webhooks inside your infrastructure the Event Publishers are often calling webhooks directly creating a spaghetti infrastructure. This is not manageable since each Event Publisher needs to have the routing logic inside their own component.

By using Azure Event Grid we can route all the events through Azure Event Grid and use it as an event broker, or routing hub if you will, and thus decoupling Event Publisher from the corresponding Event Handlers.

By doing this we can easily change the way we route events to new Event Handlers by simply changing the routing, not the routing logic in the Event Publishers.

Depending on the monitoring Azure Event Grid will provide, it can also provide a more generic approach in how we monitor all the event handling instead of using the monitoring on each component. More on this in my next blog.

Depending on the load, you can of course also use Azure Service Bus Topics but all depends on the load you are expecting. As always, it depends on the scenario; to pick which technology is best for the scenario.

Summary

Azure Event Grid is a unique service that has been added to Microsoft Azure and brings a lot to the table. It promises big performance targets and will enable new scenarios, certainly in the serverless landscape.

I'm curious to see how the service will evolve and what publishers & handlers will be coming soon. Personally, I think it's a big announcement and will give it some more thinking on how we can use it when building platforms on Microsoft Azure.

Want to learn more yourself? Here's a good Cloud Cover episode that will give you a high-level overview of Azure Event Grid or read about the concepts of Event Grid.

What features would you like to see being added to the service? In what scenarios do you see Event Grid as a good fit? Feel free to mention them in the comments!

Thanks for reading,

Tom Kerkhove.

Posted on Wednesday, August 16, 2017 11:05 AM

Toon Vanhoutte by Toon Vanhoutte

Accidental removal of important Azure resources is something to be avoided at all time. For sure if it occurs with statefull Azure services, that are hosting very important information. Common examples of such Azure services are Azure Storage Accounts and Azure KeyVaults. This blog post describes three steps to minimize the risk of such unwanted removals, however additional measures might be needed, depending on your use case.

Prevent accidental removal

Role Based Access Control

Azure services offer real fine-grained access control configuration. You should take advantage of it, in order to govern access to your Azure environment. Resources in your production environment should only be deployed by automated processes (e.g. VSTS). Assign the least required privileges to the developers, operators and administrators within your subscription. Read only access in production should be a default setting. In case of incidents, you can always temporary elevate permissions if needed. Read here more about this subject.

Naming Conventions

It's a good practice to include the environment within the name of your Azure resources. Via this simple naming convention, you create awareness on what environment one is working. Combined with this naming convention, it's also advised to have a different subscription for prod and non-prod resources.

Resource Locks

An additional measure you can take, is applying resource locks on important resources. A resource lock can set two different lock levels:

  • CanNotDelete means authorized users can still read and modify a resource, but they can't delete the resource.
  • ReadOnly means authorized users can read a resource, but they can't delete or update the resource. Applying this lock is similar to restricting all authorized users to the permissions granted by the Reader role.

These locks can only be created or removed by a Resource Owner or a User Access Administrator. This prevents that e.g. Contributors delete the resource by accident.

Locks can be deployed through PowerShell, Azure CLI, REST API and ARM templates. This is an example of how to deploy a lock through an ARM template:

Be aware that locks can also be configured on subscription and resource group level. When you apply a lock at a parent scope, all resources within that scope inherit the same lock. Even resources you add later inherit the lock from the parent.

Conclusion

The Microsoft Azure platform gives you all the access control tools you need, but it's your (or your service provider's) responsibility to use them the right way. One should be very cautious when giving / getting full control on a production environment, because we're all human and we all make mistakes.

Categories: Azure
Tags: Logic Apps
written by: Toon Vanhoutte