A Brief History of Cloud-Based Integration in Microsoft Azure

By Rob Callaway

Mission Briefing

In conversations with students and other integration specialists, I’m discovering more and more how confused some people are about the evolution of cloud-based integration technologies. I suspect that cloud-based integration is going to be big business in the coming years, but this confusion will be an impediment to us all.

To address this I want to write a less technical, very casual, blog post explaining where we are today (November of 2015), and generally how we got here. I’ll try to refrain from passing judgement on the technologies that came before and I’ll avoid theorizing on what may come in the future. I simply want to give a timeline that anyone can use to understand this evolution, along with a high-level description of each technology.

I’ll only speak to Microsoft technologies because that’s where my expertise lies, but it’s worth acknowledging that there are alternatives in the marketplace.

If you’d like a more technical write-up of these technologies and how to use them, Richard Seroter has a good article on his blog that can be found here.

On the First Day, Microsoft Created Azure

Way, way back in October of 2008 Microsoft unveiled Windows Azure (although it wouldn’t be until February of 2010 that Azure went “live”). On that first day, Azure wasn’t nearly the monster it has become.

It provided a service platform for .NET services, SQL Services, and Live Services. Many people were still very skeptical about “the cloud” (if they even knew what that meant). As an industry we were entering a brave new world with many possibilities.

From an integration perspective, Windows Azure .NET Services offered Service Bus as a secure, standards-based messaging infrastructure.

What’s the Deal with Service Bus?

Over the years, Service Bus has been rebranded several times but the core concepts have stayed the same: reduce the barriers for building composite applications, even when their components have to communicate across organizational boundaries. Initially, Service Bus offered Topics/Subscriptions and Queues as a means for systems and services to exchange data reliably through the cloud.

Service Bus Queues are just like any other queueing technology. We have a queue to which any number of clients can post messages. These messages can be received from the queue later by some process. Transactional delivery, message expiry, and ordered delivery are all built-in features.

Sample Service Bus queue

Sample Service Bus queue

I like to call Topics/Subscriptions “smart queues.” We have concepts similar to queues with the addition of message routing logic. That is, within a Topic I can define one or more Subscription(s). Each Subscription is used to identify messages that meet certain conditions and “grab” them. Clients don’t pick up messages from the Topic, but rather from a Subscription within the Topic. A single message can be routed to multiple Subscriptions once published to the Topic.

Sample Service Bus Topic and Subscriptions

Sample Service Bus Topic and Subscriptions

If you have a BizTalk Server background, you can essentially think of each Service Bus Topic as a MessageBox database.

Interacting with Service Bus is easy to do across a variety of clients using the .NET or REST APIs. With the ability to connect on-premises applications to cloud-based systems and services, or even connect cloud services to each other, Service Bus offered the first real “integration” features to Azure.

Since its release, Service Bus has grown to include other messaging features such as Relays, Event Hubs, and Notification Hubs, but at its heart it has remained the same and continues to provide a rock-solid foundation for exchanging messages between systems in a reliable and programmable way. In June of 2015, Service Bus processed over 1 trillion (1,000,000,000,000) messages! (Starts at 1:20)

What About VETRO?

As integration specialists we know that integration problems are more complex than simply grabbing some data from System A and dumping it in System B.

Message transport is important but it’s not the full story. For us, and the integration applications we build, VETRO (Validate, Enrich, Transform, Route, and Operate) is a way of life. I want to validate my input data. I may need to enrich the data with alternate values or contextual information. I’ll most likely need to transform the data from one format or schema to another. Identifying and routing the message to the correct destination is certainly a requirement. Any integration solution that fails to deliver all of these capabilities probably won’t interest me much.

VETRO Diagram

VETRO Diagram

So, in a world where Service Bus is the only integration tool available to me, do I have VETRO? Not really.

I have a powerful, scalable, reliable, messaging infrastructure that I can use to transport messages, but I cannot transform that data, nor can I manipulate that data in a meaningful way, so I need something more.

I need something that works in conjunction with this messaging engine.

You Got Your BizTalk in My Cloud!

Microsoft’s first attempt at providing a more traditional integration platform that provided VETRO-esque capabilities was Microsoft Azure BizTalk Services (MABS) (to confuse things further, this was originally branded as Windows Azure BizTalk Services, or WABS). You’ll notice that Azure itself has changed its name from Windows Azure to Microsoft Azure, but I digress.

MABS was announced publicly at TechEd 2013.

Despite the name, Microsoft Azure BizTalk Services DOES NOT have a common code-base with Microsoft BizTalk Server (on second thought, perhaps the EDI pieces share some code with BizTalk Server, but that’s about all). In the MABS world we could create itineraries. These itineraries contained connections to source and destination systems (on-premises & cloud) and bridges. Bridges were processing pipelines made up of stages. Each stage could be configured to provide a particular type of VETRO function. For example, the Enrich stage could be used to add properties to the context of the message travelling through the bridge/itinerary.

Stages of a MABS Bridges

Stages of a MABS Bridges

Complex integration solutions could be built by chaining multiple bridges together using a single itinerary.

MABS message flow

MABS message flow

MABS was our first real shot at building full integration solutions in the cloud, and it was pretty good, but Microsoft wasn’t fully satisfied, and the industry was changing the approach for service-based architectures. Now we want Microservices (more on that in the next section).

The MABS architecture had some shortcomings of its own. For example, there was little or no ability to incorporate custom components into the bridges, and a lack of connectors to source and destination systems.

Give Me Those Sweet, Sweet Microservices

Over the past couple of years the trending design architecture has been Microservices. For those of you who aren’t already familiar with it, or don’t want to read pages of theory, it boils down to this:

“Architect the application by applying the Scale Cube (specifically y-axis scaling) and functionally decompose the application into a set of collaborating services. Each service implements a set of narrowly related functions. For example, an application might consist of services such as the order management service, the customer management service etc.

Services communicate using either synchronous protocols such as HTTP/REST or asynchronous protocols such as AMQP.

Services are developed and deployed independently of one another.

Each service has its own database in order to be decoupled from other services. When necessary, consistency is between databases is maintained using either database replication mechanisms or application-level events.”

So the shot-callers at Microsoft see this growing trend and want to ensure that the Azure platform is suited to enable this type of application design. At the same time, MABS has been in the wild for just over a year and the team needs to address the issues that exist there. MABS Itineraries are deployed as one big chunk of code, and that does not align well to the Microservices way of doing things. Therefore, need something new but familiar!

App Service, and API Apps, and Logic Apps, Oh My!

Azure App Service is a cloud platform for building powerful web and mobile apps that connect to data anywhere, in the cloud or on-premises. Under the App Service umbrella we have Web Apps, Mobile Apps, API Apps, and Logic Apps.

Azure App Service

Azure App Service

I don’t want to get into Web and Mobile Apps. I want to get into API Apps and Logic Apps.

API Apps and logic Apps were publicly unveiled in March of 2015, and are currently still in preview.

API Apps provide capabilities for developing, deploying, publishing, consuming, and managing RESTful web APIs. The simple, less sales-pitch sounding version of that is that I can put RESTful services in the Azure cloud so I can easily use them in other Azure App Service-hosted things, or call the API (you know, since it’s an HTTP service) from anywhere else. Not only is the service hosted in Azure and infinitely scalable, but Azure App Service also provides security and client consumption features.

So, API Apps are HTTP / RESTful services running in the cloud. These API Apps are intended to enable a Microservices architecture. Microsoft offers a bunch of API Apps in Azure App Service already and I have the ability to create my own if I want. Furthermore, to address the integration needs that exist in our application designs, there is a special set of BizTalk API Apps that provide MABS/BizTalk Server style functionality (i.e., VETRO).

What are API Apps?

What are API Apps?

This is all pretty cool, but I want more. That’s where Logic Apps come in.

Logic Apps are cloud-hosted workflows made up of API Apps. I can use Logic Apps to design workflows that start from a trigger and then execute a series of steps, each invoking an API App whilst the Logic App run-time deals with pesky things like authentication, checkpoints, and durable execution. Plus it has a cool rocket ship logo.

What are Logic Apps?

What are Logic Apps?

Putting the Pieces Together

What does all this mean? How can I use these Azure technologies together to build awesome things today?

Service Bus review

Service Bus review

Service Bus provides an awesome way to get messages from one place to another using either Queues or Topics/Subscriptions.

API Apps are cloud-hosted services that do work for me. For example, hit a SaaS provider or talk to an on-premises system (we call these connectors), transform data, change an XML payload to JSON, etc.

Logic Apps are workflows composed of multiple API Apps. So I can create a composite process from a series of Microservices.

Logic App review

Logic App review

But if I were building an entire integration solution, breaking the process across multiple Logic Apps might make great sense. So I use Service Bus to connect the two workflows to each other in a loosely-coupled way.

Logic Apps and Service Bus working together

Logic Apps and Service Bus working together

And as my integration solution becomes more sophisticated, perhaps I have need for more Logic Apps to manage each “step” in the process. I further use the power of Topics to control the workflow to which a message is delivered.

More Logic Apps and Service Bus Topics provide a sophisticated integration solution

More Logic Apps and Service Bus Topics provide a sophisticated integration solution

In the purest of integration terms, each Logic App serves as its own VETRO (or subset of VETRO features) component. Decomposing a process into several different Logic Apps and then connecting them to each other using Service Bus gives us the ability to create durable, long-running composite processes that remain loosely-coupled.

Doing VERTO using Service Bus and Logic Apps

Doing VERTO using Service Bus and Logic Apps

Summary

Today Microsoft Azure offers the most complete story to date for cloud-based integration, and it’s a story that is only getting better and better. The Azure App Service team and the BizTalk Server team are working together to deliver amazing integration technologies. As an integration specialist, you may have been able to ignore the cloud for the past few years, but in the coming years you won’t be able to get away with it.

We’ve all endeavored to eliminate those nasty data islands. We’ve worked to tear down the walls dividing our systems. Today, a new generation of technologies is emerging to solve the problems of the future. We need people like you, the seasoned integration professional, to help direct the technology, and lead the developers using it.

If any of this has gotten you at all excited to dig in and start building great things, you might want to check out QuickLearn Training’s 5-day instructor-led course detailing how to create complete integration solutions using the technologies discussed in this article. Please come join us in class so we can work together to build magical things.

Aligning Microsoft Azure BizTalk Services Development with Enterprise Integration Patterns

By Nick Hauenstein

Earlier this week, I started a series of discussions with the development team here at QuickLearn Training. These discussion included John Callaway, Rob Callaway, and Paul Pemberton, and centered around Best Practices for development of Microsoft Azure BizTalk Services integrations. The topic arose as we were working on our upcoming Microsoft Azure BizTalk Services course.

Best practices are always a strange topic to address for a technology that is relatively young, and at the same time rapidly evolving. However, the question comes up in both classes and consulting engagements – no matter what the technology.

With regards to BizTalk Server, we have years’ worth of real-world experience from which to pull both personally, and from our customers’ narratives. In addition, there are extensive writings in books, blog posts, white papers, and technical documentation. The same can’t be said, yet, of BizTalk Services.

This then represents an attempt at importing some known-good ideas and ideals into the world of MABS so that value can be realized. It is written not to codify practices and anoint them as “Best” without being proven, but instead to start a dialog on how best to use MABS both in present form, and with additional functionality that is yet to come.

NOTE: I am actively building out integrations to explore these ideas more fully, and expect to find that at least one thing I mention here either won’t work, or isn’t the best after all.

A Language of Patterns

For a brief period of my BizTalk life, I worked on the team that built the Enterprise Service Bus Toolkit 2.0. My job was to document not only the functionality as it was built, as well as sample applications, but also how existing Integration Patterns and SOA Patterns could be implemented using the Toolkit.

I explicitly recall that this last point was especially emphasized by the leader of that small team, Dmitri Ossipov. He wanted to communicate how integration patterns documented by Gregor Hophe and SOA patterns documented by Thomas Erl could be implemented using the ESB Toolkit.

Why would we spend time linking product documentation to patterns? Because Dmitri understood something important. He understood that buzzword compliance wasn’t enough to drive adoption of the platform (i.e., being able to say “this thing uses an ESB under the covers,” or nowadays, “this is cloud-based integration,” or “hybrid integration” [isn’t all integration a hybrid of something?]). Instead adoption would be driven as developers could see it solving a problem, and solving it elegantly – where each Itinerary Service, out-of-the-box or custom, implemented a specific integration pattern and composing them could solve any integration challenge.

So which problems were the best to try and solve? Probably the most common ones (i.e., the ones that are so common as to have vendor-agnostic industry-wide documented patterns for overcoming them). So what does that have to do with MABS? Everything.

The problem space hasn’t changed. The cloud has been quite disruptive in the overall industry – likely for the best. It has leveled the playing field to the point that the established players can be publicly challenged by marketing teams of smaller players that are brand-new to the scene because the scene itself is still new. At the same time, the art and science of integrating systems is the same.

The patterns for approaching this brave new world are remixes and riffs on the patterns that we’ve already had. As a result I’m going back to the fundamentals, and using it to understand MABS.

When I’m starting on a new integration with BizTalk Server, I first sit down and think of that integration in terms of the problems I’m trying to solve and well known patterns that can be used to overcome those problems. A nice visual example of such a pattern is reproduced here (from Gregor Hohpe’s list of Enterprise Integration Patterns):

Here we’re seeing the Scatter-Gather pattern, which is actually a composite of both the Recipient List pattern (1 message being sent to multiple recipients) and the aggregation pattern (multiple messages being aggregated into a single message). This is being presented in the light of a fictional RFQ scenario.

To see it further broken down, we could pull in just the Recipient List Pattern:

Or we could pull in just the Aggregator Pattern:

Once we’ve determined the patterns involved, and how they’re composed, it’s a fairly straight-forward exercise to envision the BizTalk components that would be involved and/or absolutely necessary in order to implement the solution.

For the purposes of this post, I’m going to see how specific patterns might be implemented in Microsoft BizTalk Server, as well as Microsoft Azure BizTalk Services, and Microsoft Azure Service Bus. Specifically, I will be examining the Canonical Data Model pattern, Normalizer pattern, Content-Based Router pattern, Publish-Subscribe Channel pattern, and even the Dynamic Router pattern.

This is not to say that these are somehow the only patterns possible in MABS, but will instead form a nice basis for discussion. Let’s not get ahead of ourselves though, as there’s still some groundwork to cover.

Seeing the Itinerary Through the Bridges

So many times as I’m reading through content discussing Microsoft Azure BizTalk Services (blogs, code samples, documentation, tutorials, books), I see lots of emphasis placed on a single bridge, but rarely on the itinerary as a whole that contains it. Essentially I’m seeing a lot of this:

image

What are we looking at here? A really boring itinerary that does not showcase the value of MABS, and makes it seem impotent in the process.

Inside the bridge though, there’s a lot going on: message format translation, message validation, message enrichment, message data model transformation, with extensibility points to boot – all put together in a pipes and filters style.

But what if I want to build something bigger than data-in/data-out where not everything has a one-to-one relationship?

Bridges Revisited In Light of BizTalk Server

In our BizTalk Server classes, we ultimately caution people against building these direct point-to-point integrations. Instead, we try to leverage the Canonical Data Model pattern when possible – with the Canonical Data Model being a Canonical Schema.

What does that look like in BizTalk Server? First of all, I personally like to extend a tad beyond this base pattern and mix in a little bit of the Datatype Channel pattern as well.

With that in mind, we would start with a Receive Port for each type of message (e.g., Order, Status Request), and a Receive Location for each source system and data type we expect to see (e.g., FTP with Flat-file, vs. WCF-BasicHttp with XML).

From there, we have a canonical schema for each of those types of messages. An internal Order message for example, or internal Status Request message. At the end of the Receive Port, the appropriate transform would be selected so that a message conforming to the canonical schema is published to the Message Box database.

image

At the Message Box, another pattern is at work, whether or not we ever wanted it. BizTalk Server is doing publish subscribe routing through the Message Box based on properties stored in the context of the message.

At this point each recipient of the message has their own Send Port through which a final transformation, to THEIR order format is performed along with a final pipeline before it is transmitted through the adapter.

What is all of this doing for us? It’s separating concerns and providing a loosely-coupled design.

I can have multiple sources for a message, and easily add a new source. To add a source is a matter of adding a Receive Location and a Map that translates the incoming message to internal message formats. To add a destination is just as simple and loosely coupled. It really is a beautiful place to live.

The Tightly Coupled Nightmare We’re Building for Ourselves

So why then, when presented with the same integration problems, do we use MABS like this? Is it just that what’s being published widely isn’t representative of the real-world live production solutions that are out there right now? Or is it that we’re blinded and can’t see the itinerary as a whole because we’re caught up with the bridge? I honestly don’t know the answer to this question. All I know for sure is that single bridge itineraries are not the answer for everything.

Why? Because this is a fairly tightly coupled design, and something that we might want to reconsider.

image

So let’s do just that. Let’s start though by following a message as it would flow through this system, while comparing how BizTalk Server would handle the same.

The whole thing starts with the Channel Adapter pattern.

In BizTalk Server, that’s the Adapter configured within each Receive Location that indicates from where the message will be transported. In MABS, it’s our Source.

Next we go through some filters:

In BizTalk Server, this is our Pipeline inside the Receive Location. In MABS, it’s most of our Bridge. WCF is also rocking this pattern (potentially earlier in the process) in the form of the channel stack, and both BizTalk Server and MABS give you hooks into deep customization of that WCF channel stack if you can’t get enough opportunities to touch the message.

For the BizTalk implementation of the Canonical Data Model pattern, we would then go through the Message Translator:

This is actually where things get interesting. In BizTalk Server, this is the Map that executes after the Pipeline, within the context of the Receive Port. For MABS, we’re still in the middle of the Bridge at this point. In fact for MABS, it’s one of the “Filters” to borrow a term from the pattern above, alongside other patterns such as message enrichment.

After this point, BizTalk enters a publish and subscribe step, and our simple, single-bridge, MABS itinerary breaks down. We won’t have another opportunity to prepare the message for its destination beyond our Route Actions – unless we’ve already prematurely made the leap and readied the message for only a single destination, ever.

Why does it break down? Because we’ve artificially limited ourselves to a single bridge. We’ve seen that bridges are highly capable, and have a lot going on, so we try to get it all done up-front. But is that a Best Practice even for BizTalk? Do we do all of our processing on receive just because pipelines are extensible and can host any custom code our hearts desire? Thankfully the model of BizTalk Server, centered around the Message Box, forces us to be a little bit smarter than that.

Standard best practices for BizTalk are helping us “Minimize dependencies when integrating applications that use different data formats.” Shouldn’t best practices for MABS allow us to do the same?

Crossing the Next Bridge

Consider the following Itinerary:

image

Here we have a situation where as I’m receiving my message, I don’t yet know the destination. In fact, I may have to perform all of the work in my first bridge before I know that destination. After I’m done with that first bridge though, I will have a standard internal representation of a “Trade” in this case.

From there, I am using Route Filters to determine my destination, and bridges to ensure the destination gets a message it can deal with.

More specifically, there are really four stages of the overall execution:

image

  1. The first stage is my source and first bridge. These are playing a role similar to that of a Receive Port in BizTalk Server, ending in a Map. I’m not necessarily taking advantage of every stage within the bridge, but I am certainly doing some message transformation, and maybe even enrichment. The main goal of this first stage is to get the message translated FROM whatever it looked like in the source system TO our internal representation
  2. The second stage of execution is between bridges. These are my routes, each with route filters that determine the next stage of the execution. They are playing a role similar to the Filters on my Send Ports in BizTalk, and yet at the same time they’re doing something completely different. Why? Because only one bridge will ever receive the message – even if both filters match, one will have priority. This is not true publish-subscribe, it is instead more akin to a Content-based Router of sorts. We’ll address in the next section how to get publish and subscribe happening for more even more loosely-coupled execution.
  3. The third stage is my second bridge, route actions, and my destination. These are playing a role similar to that of a Send Port in BizTalk Server beginning with a Map. I’m using this bridge to take our internal representation of the message and get it ready for the destination system.
  4. Sometimes we need dynamic routing to some degree even when we know the destination. That’s what might be happening through Route Actions in step 4. In this specific case, that’s not a true statement. However, if my destination were SFTP, for example, this would be my opportunity to set a destination file name based on some value inside the message.

True Publish-Subscribe In MABS

What if I were to tell you that MABS does publish-subscribe better than the Message Box database? Well it’s not true, so you shouldn’t believe it – like most click baiting headlines. In fact, MABS doesn’t do publish-subscribe at all. Rather, it relies on Microsoft Azure Service Bus to provide publish-subscribe capabilities – interestingly, a technology that we can also take advantage of through adapters first made available in BizTalk Server 2013.

So how do we do it? We publish to a Service Bus Topic, and receive through Service Bus Subscriptions (Queues that subscribe to messages published to Topics based on Filters). Consider the following itinerary:

image

We no longer have connections between the three seemingly independent sections of this itinerary. In fact, if we didn’t need to share any resources between them, they could be built as separate itineraries, in separate Visual Studio projects.

This is not new stuff, in fact Richard Seroter mused about the possibilities that Service Bus would provide in MABS all the way back in February.

Dynamic Sends and Receives

Another interesting scenario enabled here is the idea of a Dynamic Receive. What would that look like? Let’s say I have a subscription that I want to toggle or reconfigure at runtime. Maybe all trades for a specific company should result in a special alert message routed somewhere, based on conditions that are not known at design-time, and could change at any time. Let’s also say that I don’t want to re-deploy my existing bridge.

I can simply add another subscription to the topic for all trades related to that company, and deploy a new itinerary that handles the special alert message. But that doesn’t sound really dynamic in nature. In fact, it’s not bringing anything new to the table that BizTalk Server doesn’t do.

On the other hand, what if I had this alerting itinerary already built and deployed, and I want to dynamically route messages to it with conditions that change constantly at runtime (outside the scope of the messages themselves) – e.g., toggle a subscription for a given trade based on its current Ask price and the symbol in the trade message?

In that case, my special alert itinerary might be fed by a queue that receives messages forwarded from a subscription generated on the fly at runtime (because Service Bus can do that!), maybe even within a custom messaging inspector if I go fully insane.

Pain Points and Opportunities For Improvement

Ultimately we’re so close, and yet still so far away from having an ideal development experience with MABS. I’m happy that we have process visibility at design-time. Being able to visualize the whole message flow can be invaluable. On the other hand, I’m not happy with the cost at which this comes: sacrificing the ability to easily partition my resources on Visual Studio Project boundaries.

Separate Projects Per Interface

Ideally I would like to have a project per interface that I’m integrating with, and one common project that contains common schemas. I would also like to be able to split up the whole itinerary into smaller pieces (i.e., files) without losing the ability to visualize the itinerary as a whole.

Instance Subscription Equivalent for Two-Way Sources

In BizTalk Server, our two-way receives turn into two one-way exchanges as messages are published to the Message Box. There’s some magic built-in for correlation, but for the most part we aren’t forced into doing everything that follows in a purely request-reply message exchange pattern. Currently Microsoft Azure BizTalk Services does not offer the same capability. As a result, the nice publish/subscribe example itinerary above is only possible in the case of a one-way pattern.

Best Practices

So what are some practices that we can derive from this whole discussion? Here’s an incomplete list that I’m toying with at the moment:

  1. Build your cloud-based integrations with integration patterns in mind – don’t ignore the fundamentals, or throw away the lessons of the past
  2. Allow bridges to specialize on one portion of the process flow – don’t try to build your entire integration using a single bridge
  3. Use the canonical data model pattern when integrating between multiple source and destination systems when the list of those systems in subject to change
  4. Leverage the publish-subscribe capabilities of Service Bus when needed, don’t rely only on routes within MABS
  5. Push down to on-premise BizTalk Server to handle patterns that aren’t yet possible in MABS.
    • Mediating mismatched of message exchange patterns, through BizTalk Server’s instance subscriptions for two-way receives (e.g., incoming synchronous request/response interface provided against asynchronous one-way request with one-way callback)
    • Handling aggregation of messages, which will require the use of Orchestration
    • Handling the O in the VETRO  pattern (Validate, Enrich, Transform, Route, Operate), which again will require the use of Orchestration
    • Handling the execution of Business Rules within BizTalk Server on-premise

Looking Forward

I’m really looking forward to seeing how Workflow is going to fit into the picture. We’ll have to wait until Integrate 2014 in December to know for sure, but if we keep the fundamentals in mind, it shouldn’t be hard to figure out how they will fit into a concept of MABS itineraries rooted in the integration patterns that make them up.

I truly hope that you have enjoyed this post, and I thank you for reading to the end, if indeed you have and not simply skipped to the end to find out if I ever stopped typing. Remember that I want this to be a dialog, both retrospectively in terms of what is working and what isn’t working with your MABS integrations, as well as a forward-looking dialog as to what practices will likely emerge as generally applicable and ultimately most beneficial.