Integrate 2019 - Day 1 Recap

We are very proud to once again have one of the biggest delegations at Integrate, representing the biggest integration company! This blog post was put together by each and every one of our colleagues attending Integrate 2019.

Integrate 2019 – Welcome Speech – Saravana Kumar

Saravana welcoming everyone to the 8th edition of Integrate. With 500 attendees this is the biggest one yet.

Keynote – John Fancy

Microsoft views Integration as a key enabler for Digital Transformation and Innovation and wants to democratize integration for organizations. First, it was shown that Gartner agrees that Microsoft lived up to its vision in 2015, where they intended to be an iPaaS leader, democratize integration and build out a rich eco-system. And that vision has been achieved:

• Microsoft got recognized as iPaaS leaders in 2018 and 2019
• More than 300 connectors are available (1st & 3rd party, some of which are open sourced)
• Microsoft Flow democratizes automation and Logic Apps democratizes integration

The biggest element of the session was spent by 3 interesting customers each demonstrating how they focused on one of the three keywords of the integration vision: Transform, Innovate & Democratize. Because, according to Gartner: “The more you innovate, the more you need to integrate”.

Mrs Ramak Robinson, who is Area Architect at H&M explained how they are transforming their business and are leveraging Azure Integration Services for that. With their cloud-first transition, they notice that most applications or systems will be replaced, while some of them might be re-architected to be cloud-ready.

Mr Daniel Ferreira, who is cyber security data scientist at Shell demonstrated how their cyber security center has built innovative solutions to cope with some of the operational challenges they face (such as lack of resources, speed of detection or triaging, etc). They leveraged chatbots (through a Teams channel) and Machine Learning to handle security issues in an AI-driven, but user-centric chat-interface.

Mr Bibhor Mathur, who is Lead Architect at Anglo-Gulf bank, showed how they managed to build a disruptive trade-finance bank in 6 months, that runs on 1% of the traditional operating cost. The cloud services really helped them in democratizing this ambitious project, through faster integration in a secure-first approach.

Logic Apps update – Kevin Lam

The number of Logic App users in Azure is growing at stupendous speed. The session started with presenting these spectacular numbers. Due to the extensive use of Logic Apps, there was a lot of curiosity and excitement in the audience too. Microsoft is working hard to make Logic App development lot easier so that non-integration IT specialist can also take up the development.

Many new offerings, Few interesting once caught my attention are:

• Parameterized templates: Currently you can quickly develop a Logic App. But turning it into a valid ARM templates sometimes takes more time than the development itself. Although this feature wasn’t shown, but you can download Parameterized template for quick preparation of Arm template.

• Static results: This could be useful for unit testing. Instead of calling an external system, a default response can be defined. What’s neat is that this is visible by a special icon in the designer, so you don’t forget to remove it before going live ????
• Resubmit from action: is also a great new feature. Currently, you can only resubmit the entire Logic Apps, but sometimes it is undesirable if an action is called again when resubmitting. Hence, it saves the developer taking care of this.
• Many more could be viewed in the following screenshot. There is a lot coming up in the Integration Service Environment (ISE) as well:

Some of the new features ISE are:

• New connectors: 38 New connectors are added in the last 6 months to provide more out of the box connection possibilities. Connectors added include BlockChain, Servicebus with Claimcheck, DB2 New SMTP.
• Connector Manager for easier management of the available connectors
• Cheaper Dev Environment to reduce the costs.
• Increased Timeout from 2 to 4 mins to provide more flexibility for connections.
• Increased processing power by 10 additional scale units.
• Logic App lifetime increased to 365 days, so processes can stay alive longer.
• BYO SSL Cert is bring your own SSL certificate for extensive security.

API Management: Update – Vlad Vinogradsky

After a very brief overview of APIM (in case anyone in the audience wasn’t familiar with APIM yet) Vlad then covered some of the most recently released features in API Management.

As a quick recap, some of these updates include Policy updates, support was added for encrypted tokens. You can add the signing and encryption keys and you can output the decrypted token to a variable for use later in the policy

For Managed identity, support was added for managed identity. New policy added to allow you to use the managed identity to access a backend service.

Support was added for “Bring your own [Redis] cache”. All existing policies should work as-is, there is, however, a new attribute to express a cache preference, internal or external.

As for Subscriptions, the scope of subscription and keys have been updated. One can create a key for one (or more) API’s, no need for a product or subscription anymore. Tracing can be enabled on key level.

Observability was also covered by Vlad. He reviewed recent changes on configuring logging to Application Insights and Azure Monitor in the Azure Portal.

APIM now has much better Function App integration. In the portal a function can be published as an API from with the Functions area of the Azure Portal.

The biggest new feature announced was the new Self-hosted APIM Gateway that is due soon.

The self-hosted APIM Gateway component will be made available to be deployed on-premises. It’s packaged as a docker container for Linux and Kubernetes is recommended for hosting the gateway. Vlad gave a great in-depth demo displaying these features.

The Value of Hybrid Integration – Paul Larsen

In this session, Paul announced the next version of BizTalk Server.
BizTalk Server vNext, now officially named, 2020, planned to be available Q4 2019 and offering new functions to further enable Hybrid Integration within new scenario’s.

Much of the release comes down to product alignment with Windows Server and SQL Server, and new functions previously released through feature packs for BizTalk 2016 Enterprise Edition; amongst which are improved scheduling options and Office365 adapters. HIS2020 follows a same tech alignment update.

Regarding to Hybrid Integration some new features are:

• Publishing orchestrations to API Management
• EventHubs adapter
• LogicApps adapter
• Blob storage adapter.
• Sending tracking data to EventHubs and Application Insight

For the future Microsoft has set its mind on supporting Azure SQL and SQL Managed instances. But not in this upcoming release.

New is the prolonged migration path, with migration from 2010 and later to be supported, to also allow for an Admin Anywhere scenario to play out where most admin operations starting in this release will be available as a REST api.

Then, moving back decades into the past, we actually see plans for a LogicApps screen capture and screen navigation adapter for IBM 3270. It also comes with a designer. Talk about taking Hybrid to the Max!

Basically, Microsoft is investing huge in enabling IBM based legacy integration from Azure. Paul even suggests that the future is in directly connecting, bypassing the On-premises Data Gateway. A next strategical step away from the need for BizTalk Server and HIS as it seems.

Event hubs update – Dan Rosanova

Dan Rosanova talked about how Event Hubs and Kafka is different from queues. Event hubs are not queues. Old approach server side cursor competing consumer.

Event Hubs and Kafka is different, records a forward only stream which can be replayed over again. Like a cassette type left and right channels.

Event hubs has partitions like a cassette has tracks.

Event producers write to the hub, event receivers read from the hubs.

Also the Kafka/Event Hub conceptual Architecture:

The cool things which we get from Event Hubs:

One of the most important features in Event Hub is capture.

How Capture Works?

Using Capture:

Where we can use Event hubs?

• Golden Copy
• Leveraging Batch Processing

Where does Kafka Fit?

• Change Data Capture
  • Oracle Golden Gate
  • Attunity
  • HVR
• Logging of telementry
  • Splunk
  • IBM Qradar
• Building applications with event sourcing.
• Feeding Big Data Pipelines.

What Microsoft offers in Event Hubs for Apache Kafka?

What did Microsoft do differently from Kafka?

• Utilizing external storage
• Caching on brokers with gate to prevent dirty reads
• Always replicated before ACK
• Built internally on Service Fabric

Event Hubs Dedicated Capacity?

• Single tenant Event Hubs instance
• Still the same PaaS service
• Pay only for hours
• Start at 1 capacity unit

Five nines

• Highly availability
• Success rate is equivalent to 99.9995, achieved by tuning load balancing
• Algorithms

Azure Metadata service

• Scheduled events from windows VM’s
• Give your application time to prepare shutdowns
• Provide information about upcoming maintenance events

Summary

• There is no scale that Microsoft cannot do
• Event hubs is already most available messaging platform on cloud
• Event hubs is extremely affordable
• Already powers much of Microsoft cloud
• Try out Capture

Service Bus update – Ashish Chhabria

Of the three real messaging services, Service Bus is probably the one that is used mostly in our traditional integration scenarios. Also with our Invictus methodology, this is the main messaging engine that we propose. It takes care of reliable publish-subscribe principles that are so important in our vision of loose-coupled integrations.

In this session, Ashish focused on 3 buckets in which he positioned new Service Bus features:

Business continuity: Service Bus is a core (ring zero) services and will always be available in every (new) data center of Azure. High availability is available through the Availability zones, the Geo-disaster recovery that synchronizes metadata between two different namespaces. An important announcement was that full data copy will follow for this feature!

Enterprise features: support for Managed Service identity and RBAC will become generally available by July. And Virtual network service endpoints (and firewalls) will allow organizations to limit access to a service bus namespace from specific IP-addresses or virtual networks.

Clients & connectors: more and better support from Logic Apps connectors (lock renewal & managed sessions, for example) and better event grid integration (queues as event handlers!) show the adoption by other Azure Integration Services. And the Service Bus Data explorer tool in the portal allows customers to have better insights in the messages (including dead letter messages) for service bus entities. (think service bus explorer in the portal)

The main conclusion of this session is that, when you need these enterprise-grade features, you will need Service Bus premium. And the good news for us (and Microsoft obviously) is that seamless upgrade from Standard to Premium will be coming through the portal soon, as we saw in a recorded demo.

How Microsoft IT does Integration – Mike Bizub

Mike Bizub started with an overview of the Microsoft IT Integration Landscape. Mike gave further insights into how Microsoft implemented their integration solutions (using a combination of Logic Apps and BizTalk server) in-house.

He talked about how Microsoft is enabling the business Processes by starting with a recap of the B2B approach with the different trading partners. Mike described Microsoft IT’s approach for moving to the cloud and breaking down all their current processes in granular steps that could make up a “Composable Architecture” solution. A routing slip pattern driven with metadata allows messages to flows through all the implemented integrations “paths” and these paths could be updated at run time by adding additional steps into the paths.

Mike spoke briefly about how their exception handling is implemented and how Microsoft uses the exceptions raised in their Logic Apps in “Hot Paths” and “Warm Paths” to feed backend processes to deal with these exceptions. For example, exceptions are published to event hubs and processes consume the messages and raise internal service desk tickets.

Mike’s colleague took over for the last part of the presentation to show the AIS migration tool which Microsoft IT has developed to simplify and speed up migrating configuration (such as trading partner agreements, etc.) from BizTalk server to an Azure Integration Account. These tools save hours in effort in migrating configurations (which might have taken 5 days before could be migrated in less than a day). These tools created by Microsoft have been open sourced and can be found on GitHub at the following link: http://aka.ms/Integrate2019

Enterprise Logic Apps – Divya Swarnkar

Divya starts out by stating that companies now-a-days are challenged to ‘Innovate First’ to keep an edge on the competition. This requires change, and change should come incrementally. In most cases reusing existing processes hosted in ERP systems in a new way can be a valid route to start innovation.

To illustrate this she takes the Contoso Grocery Store posing that malfunctioning cooling units cause loss of value (spoiled goods).

To stay ahead of this, the scenario describes an IoT (temperature sensor) trigger that causes a maintenance order in SAP allowing intervention before loss of value occurs.

Using LogicApps and the new SAP connector she then demonstrates this scenario with the help of a simulated sensor device. She confirms that the SAP connector has entered GA status. The adapter provides native data encryption to hide traffic results from monitoring when required.

As the SAP adapter depends on the use of the Azure Data Gateway, it is good to hear that a single gateway can soon be reused across subscriptions preventing unnecessary redundancy of resources.

Other improvements include:

• Integration Account now supporting 1000 EDI artefacts
• AS2v2 is supported
• Increased Monitoring to show, a.o., Batch Trigger and whether the run was a first time activation or resubmitted

Although this scenario can also be demonstrated with the use of Dynamics365 it did give a good insight into the possibilities of leveraging existing processes to gain an edge.

Highlighting other LogicApps features for Enterprise Integration, the future roadmap includes an ISE SAP adapter and support for sFTP. Also mentioned are support for Oracle EBS, ODBC and a dev/SKU for Integration Accounts along with other features mentioned earlier in the session of Kevin Lam this morning.

Serverless real use cases and best practices – Thiago Almeida

In this sessions, Thiago Almeida gave an overview of some of the challenges where Serverless technologies helped solving real world solutions at customers. In several scenarios, Thiago combined the power of Azure Functions, Logic Apps and other services that are truly PaaS oriented.

The common definition of Serverless he used was a good one to remember: “Serverless is a scalable and reliable event-based compute experience for code and workflows that accelerates the lifecycle of applications while hiding the infrastructure and providing auto-scale and micro-billing.”

In his customer-oriented role, Thiago has mostly seen the serverless advantages in the following scenarios: Web & Mobile API’s, IoT and stream processing, Chatbots, File processing, Scheduled (CRON) jobs and job automation.

Several specific scenarios and solutions were demonstrated and also the integration (where PaaS meets IaaS) between the Serverless and the Infrastructure world was shown with VNET integration, etc.

Interesting resources for our readers:

• Azure solution blueprints : https://docs.microsoft.com/en-us/azure/governance/blueprints/overview
• Python functions with Machine learning: https://github.com/Azure-Samples?utf8=%E2%9C%93&q=functions-python&type=&language=

Microsoft Flow – Kent Weare & Sarah Critchley

Last session of the day was an overview of Microsoft Flow giving by Kent Weare and Sarah Critchley. The session started with the quote: “Business models must continually evolve to technology-driven market changes to stay competitive, organizations need the ability to update products and processes rapidly”. With this in mind, the session about Microsoft flow started.

They pointed to the Total Economical impact of Microsoft Flow. And went more in detail about the 6 unique capabilities of flow.

Flows help people and teams to work smarter by automating process across their apps and data.

Unique capabilities of Flow:

• Approvals
  • Every company needs approval processes, Microsoft Flow to create easily approval flows.
• Business Process Flow
• Mobile
• DLP, Data Loss Prevention Policies
  • Unified Admin Center
  • Admin Analytics
  • Management Connectors
  • Templates
• Geo Triggering
  • Flows can be triggered when you’re in a certain region.
• Solution packaging
  • A demo was given on how a solution can be exported from one to another environment. After the import we saw that the solution was put in the disabled state, also a warning was visible so that after importing people are pushed to smoke test and validate all configurations.

Thank you for reading our blog post, feel free to comment or give us feedback in person.

Bart Defoort, Carlo Garcia-Mier, Charles Storm, Erwin Kramer, Francis Defauw, Gajanan Kotgire, Jonathan Gurevich, Kapil Baj, Martin Peters, Miguel Keijers, Peter Brouwer, Sam Vanhoutte, Serge Verborgh, Sjoerd van Lochem, Steef-Jan Wiggers and Tim Lewis