BizTalk Server 2016 Feature Pack 1

By Rob Callaway

Ooh, shiny!

On April 26th 2017, Microsoft released Feature Pack 1 (FP1) for BizTalk Server 2016 and it’s been a while since I was this excited for a BizTalk Server release. Yeah, I just said that. I’m more excited for this feature pack than I was for BizTalk Server 2016 or even 2013 and 2013 R2, and here’s why… this is the first ever Feature Pack for any release of BizTalk Server, and is setting a precedent that we have never seen before in the 16+ years of the product.

So, what is a feature pack?

A feature pack is a release of new non-breaking features for the product. These are not bug fixes or anything like that (those are distributed quarterly through Cumulative Updates). These are brand-new features that extend the product in new ways and help customers get the most out of their BizTalk Server investment.

The product team has confirmed that other feature packs are in the works, but they have not publicly confirmed when we can expect them. In discussions I’ve had with Tord Glad Nordahl (a program manager at Microsoft and longtime lover of BizTalk Server), he said:

“If it takes 6 months to build new features there will be another feature pack in 6 months, and if it takes 2 months there will be a new one in 2 months.”

My takeaway is that the team’s goal is to offer real answers to problems that customers face in the timeliest manner possible.

I’m excited that the BizTalk Engineering team at Microsoft will be releasing new features on a more regular cadence. It’s awesome that they are building new features to address long-overlooked issues and not making us wait another 2 years to get our hands on them.

FP1 is available to customers with Software Assurance who are using the Developer or Enterprise editions of BizTalk Server 2016.

FP1 introduces some innovative new features to BizTalk Server and addresses some longstanding concerns that many customers have had. The new features break down into three categories.

Deployment

Anyone who’s worked with BizTalk Server knows that the deployment/ALM story has left something to be desired. For years, the “official” deployment story has been to deploy applications using BizTalk MSI Packages. Although the BizTalk MSIs are pretty easy to use and they work well for simple applications, they tend to be inflexible and break with the complexities of a real application in the real world.

For example, to create a BizTalk MSI Package I have to deploy all my assets to the BizTalk Server Management database and then I can generate an MSI with those assets. It sounds easy and it is. The issue is that if I introduce a new assembly, or port, or anything at all, I have to add that new resource to the BizTalkMgmtDb and then generate a new MSI. In the modern world of DevOps and continuous integration/deployment, the standard MSI-based deployment is pretty cumbersome and most teams wanting to adopt those types of strategies need a different answer.

In the past, those teams have used community-designed tools such as the BizTalk Deployment Framework to automate building an MSI from source code repositories (like most modern ALM solutions) and therefore eliminating the need to deploy to a BizTalk Server system to create the deployment package. Feature Pack 1 for BizTalk Server 2016 introduces two new features that will serve as a foundation for more sophisticated deployment strategies in the future (which my inside sources at Microsoft have confirmed are coming in later feature packs).

  • Deploy with VSTS – Enable Continuous Integration to automatically deploy and update applications using Visual Studio Team Services (VSTS).
    For anyone who has used the build/release features of TFS or VSTS, this will be immediately familiar. This is a deployment task that you add to your release pipelines to deploy new or redeploy/update existing BizTalk Server applications.
    If you haven’t used the build/release features of TFS or VSTS check out this post where I explain how to use those features to enable continuous release for a Logic App.
  • New management APIs – Manage your environment remotely using the new REST APIs with full Swagger support.
    Imagine having RESTful web APIs for updating, adding, or querying the status of your BizTalk Server applications and their resources… now stop imagining it because it’s a real thing!

Analytics

The tracking capabilities in BizTalk Server are extensive, but the configuration is often unintuitive, and no one likes digging through the BizTalkDTADb for the instance data they need.

FP1 enables you to send your tracking data to Azure Application Insights and feed operational data (subscriptions, batching status, message instance counts, etc.) to Power BI.

  • Application Insights – Tap into the power of Azure for tracking valuable application performance, usage, diagnostics, and availability.
    Enabling this is super easy: after creating an Application Insights instance, in the BizTalk Settings Dashboard there’s a new section for enabling analytics.

    Enabling Application Insights for the BizTalk Server group as introduced in BizTalk Server 2016 Feature Pack 1

    Once you’ve enabled analytics and provided your App Insights instrumentation key, in your ports and orchestrations you will have a new setting to output the tracking data to App Insights.

    Enabling Application Insights for a BizTalk Server orchestration as introduced in BizTalk Server 2016 Feature Pack 1

  • Leverage operational data – View operational data from anywhere and with any device using Power BI.
    This operational data is the same kind of information that you’d typically view using the BizTalk Group Hub (suspended instances, subscriptions, tracked events, etc.). If you can build a query for it using the Group Hub, you can output that to Power BI… but why would you?
    Because Power BI gives you the ability to view that data from anywhere (without having direct access to the BizTalk Group), and the tools in Power BI make querying that data surprisingly easy. With Power BI, you can ask questions in plain English and have MDX-style queries created for you in the background. FP1 comes with a pre-built Power BI template, but you of course have the ability to build your own. I’m interested to see what the BizTalk community can create using these tools.

Runtime

If I’m being completely honest, the two features in this runtime category weren’t really on my radar at all until Tord Glad Nordahl stopped by one of my classes last month and discussed them with the students. But now that I’ve seen them, I’m excited for the potential and happy that customers with these requirements are getting some much needed love.

  • Support for Always Encrypted – Use the WCF-SQL adapter to connect to SQL Server secure Always Encrypted columns.
    Basically, SQL Server 2016 introduced a feature that enables client applications to read/write encrypted data within a SQL table without actually providing the encryption keys to SQL Server. This gives a new level of data security since the owners of the secure data (i.e., the client applications) can see it, but the manager of the data (i.e., SQL Server) cannot.
    This ensures that on-premises or cloud database administrators or other high-privileged (but unauthorized) users cannot access the sensitive data.
    With Feature Pack 1 of BizTalk Server 2016, the WCF-SQL adapter now offers an Always Encrypted property where you can simply enable or disable the feature as your needs dictate.

    WCF-SQL Adapter Always Encrypted property as introduced in BizTalk Server 2016 Feature Pack 1

  • Advanced Scheduling – Set up advanced schedules for BizTalk receive locations.
    The Schedule page of receive locations has additional options for shifting time zones and setting up recurrence schedules.

    BizTalk Server receive location advanced scheduling options as introduced in BizTalk Server 2016 Feature Pack 1

As always, the QuickLearn Training team is already looking for the best ways to incorporate these new features into our courses, but until we do you should grab the Feature Pack for yourself and give these new features a spin for yourself. While you’re at it go to the BizTalk Server User Voice page and vote for the features that you’d like to see in the next feature pack, or if you have an original idea for a feature add it there and see how love it gets.

BizTalk Server 2016 New Features: Shared Access Signature Support for Relay Adapters

By Nick Hauenstein

At the end of last week, a few of us from QuickLearn Training hosted a webinar with an overview of a few of the new features in BizTalk Server 2016. This post serves as a proper write-up of the feature that I shared and demonstrated – Shared Access Signature Support for Relay Adapters. If you missed it, we’ve made the full recording available on YouTube here. We’ve also clipped out just the section on Shared Access Signature Support for Relay Adapters over here – which might be good to watch before reading through this post.

While that feature is not the most flashy or even the most prominent on the What’s New in BizTalk Server 2016 page within the MSDN documentation, it should come as a nice relief for developers who want to host a service in BizTalk Server while exposing it to consumers in the cloud — with the least amount of overhead possible.

Shared Access Signature (SAS) Support for Relay Adapters

Configuring SAS Security for the WCF-BasicHttpRelay Adapter

You can now use SAS authentication with the following adapters:

  • WCF-BasicHttpRelay
  • WCF-NetTcpRelay
  • WCF-BasicHttp*
  • WCF-WebHttp*

* = SAS for these adapters is used only when sending messages as a client (the adapters can still be used as receive adapters, just not to host Azure Relay enabled endpoints)

Why Use SAS Instead of ACS?

Before BizTalk Server 2016, our only security option for the BasicHttpRelay and NetTcpRelay adapters was the Microsoft Azure Access Control Service (ACS).

One of the main scenarios that the Access Control Service was designed for was Federated Identity. For simpler scenarios, wherein I don’t need claims mapping, or even the concept of a user, using ACS adds potentially unnecessary overhead to (1) the deployed resources (inasmuch as you must setup an ACS namespace alongside the resources you’re securing), and (2) the runtime communications.

Shared Access Signatures were designed more for fine-grained and time-limited authority delegation over resources. The holder of a key could sign and distribute small string-based tokens that define a resource a client could access and timeframe within which they were allowed to access the resource.

image

Hosting a Relay Secured by Shared Access Signatures

In order to expose a BizTalk hosted service in the cloud via Azure Relay, you must first create a namespace for the relay – a place for the cloud endpoint to be hosted. It’s at the namespace level that you can generate keys used for signing SAS tokens that allow BizTalk server to host a new relay, and tokens that allow clients to send messages to any of that namespace’s relays.

The generated keys are associated with policies that have certain associated claims / rights that each is allowed to delegate.

Shared Access Policies for the Azure Relay Namespace

In the example above, using the key associated with the biztalkhost policy, I would be able to sign tokens that allow applications to listen at a relay endpoint within the namespace, but I would not be able to sign tokens allowing applications to Send messages to the same relays.

Clicking a policy reveals its keys. Each policy has 2 keys that can be independently refreshed, allowing you to roll over to new keys while giving a grace period in which the older keys are still valid.

Shared Access Policy Keys

Either one of these keys can be provided in the BizTalk Server WCF-BasicHttpRelay adapter configuration to host a new relay.

Configuring the Security Settings for the WCF-BasicHttpRelay Adapter

When configuring the WCF-BasicHttpRelay adapter, rather than providing a pre-signed token with a pre-determined expiration date, you provide the key directly. The adapter can then sign its own tokens that will be used to authorize access to the Relay namespace and listen for incoming connections. This is configured on the Security tab of the adapter properties.

WCF-BasicHttpRelay Shared Access Signature Configuration

If you would like to require clients to authenticate with the relay before they’re allowed to send messages, you can set the Relay client authentication type to RelayAccessToken:

Enabling client authentication for relay endpoints

From there it’s a matter of choosing your service endpoint, and then you’re on your way to a functioning Relay:

Relay endpoint

Once you Enable the Receive Location, you should be able to see a new WCF Relay with the same name appear in the Azure Portal for your Relay namespace. If not, check your configuration and try again.

image

Most importantly, your clients can update their endpoint addresses to call your new service in the cloud.

The Larger Picture: BizTalk Hybrid Cloud APIs

The Larger Picture: BizTalk Hybrid Cloud APIs

One thing to note about this setup, however, is that the WCF-BasicHttpRelay adapter is actually not running in the Isolated Host. In other words, rather than running as part of a site in IIS, it’s running in-process within the BizTalk Server Host Instance itself. While that provides far less complexity, it also sacrifices the ability to run the request through additional processing before it hits BizTalk Server (e.g., rate limiting, blacklisting, caching, URL rewriting, etc…). If I were hosting the service on-premises I would have this ability right out of the box. So what would I do in the cloud?

Using API Management with BizTalk Server

In the cloud, we have the ability to layer on other Azure services beyond just using the Azure Relay capability. One such service that might solve our dilemma described in the previous section would be Azure API Management.

Rather than having our clients call the relay directly (and thus having all message processing done by BizTalk Server), we can provide API Management itself a token to access to our BizTalk Hosted service. The end users of the service wouldn’t know the relay address directly, or have the required credentials to access it. Instead they would direct all of their calls to an endpoint in API Management.

image

API Management, like IIS, and like BizTalk Server, provide robust and customizable request and response pipelines. In the case of API Management, the definitions of what happens in these pipelines are called “policies.” There are both inbound policies and outbound policies. These policies can be configured for a whole service at a time, and/or only for specific operations. They enable patterns like translation, transformation, caching, and rewriting.

In my case, I’ve designed a quick and dirty policy that replaces the headers of an inbound message so that it goes from being a simple GET request to being a POST request with a SOAP message body. It enables caching, and at a base level implements rate-limiting for inbound requests. On the outbound side it translates the SOAP response to a JSON payload — effectively exposing our on-premises BizTalk Server hosted SOAP service as a cloud-accessible RESTful API.

So what does it look like in action? Below, you can see the submission of a request from the client’s perspective:

BizTalk API from the client's perspective

How does BizTalk Server see the input message? It sees something like this (note that the adapter has stripped away the SOAP envelope at this point in processing):

Request message from BizTalk Server

What about on the outbound side? What did BizTalk Server send back through the relay? It sent an XML message resembling the following:

Response message from BizTalk Server

If you’re really keen to dig into the technical details of the policy configuration that made this possible, they’re all here in their terrifying glory (click to open in a new window, and read slowly from top to bottom):

API Management REST to SOAP policy definition

The token was generated with a quick and dirty purpose-built simple console app (the best kind).

Tips, Tricks, and Stumbling Blocks

Within the API Management policy shown above, you may have noticed the CDATA sections. This is mandatory where used. You’ll end up with some sad results if you don’t remember to escape any XML input you have, or the security token itself which includes unescaped XML entities.

Another interesting thing with the policy above is that the WCF-BasicHttpRelay adapter might choke while creating a BizTalk message out of the SOAP message constructed via the policy above (which includes heaps of whitespace so as to be human readable), failing with the following message The adapter WCF-BasicHttpRelay raised an error message. Details “System.InvalidOperationException: Text cannot be written outside the root element.

This can be fixed quite easily by adjusting the adapter properties so that they’re looking for the message body with the expression set to “*”.

image

Questions and Final Thoughts

During the webinar the following questions came up:

  • One audience member asked, “Is https supported?”
    • A: Yes, for both the relay itself and the API management endpoint.
  • Another audience member inquired, “Maximum size is 256KB; I was able to get a response about 800 KB; Is that because BizTalk and Azure apply the compression technology and after compression the 800KB response shrinks to about 56KB?”
    • A: The size limit mentioned applies to brokered messages within Service Bus (i.e., those you would receive using the SB-Messaging adapter). Azure Relay is a separate service that is not storing the message for any period of time – messages are streamed to the service host. Which means if BizTalk Server disconnects, the communication is terminated, but on the plus side you’re not having to worry about how much space you’re allowed to use per message in the cloud. There’s a nice article comparing the two communication styles over here.

I hope this has been both helpful and informative. Be sure to keep watching for more of QuickLearn Training’s coverage of New Features in BizTalk Server 2016, and our upcoming BizTalk Server 2016 training courses.