BizTalk Server 2013 Support for RESTful Services (Part 1/5)

By Nick Hauenstein

This post is the thirteenth in a weekly series intended to briefly spotlight those things that you need to know about new features in BizTalk Server 2013. It is also the first part of a five-part series on REST support in BizTalk Server 2013.

BizTalk Server 2013 is trying to make BizTalk Server not only the application/integration server for the enterprise, but also for the Cloud. Here we’re referring specifically to those Software-as-a-Service solutions provided by 3rd party organizations in an elastic fashion – usually exposing an API in the form of REST endpoints.

In this post, we will begin to explore the REST support in BizTalk Server 2013 in an overview fashion. We will then continue our exploration with a series of posts that cover common scenarios (Receiving data, POST-ing data, JSON en/decoding, and finally OAuth authentication).

How Do We Do REST in BizTalk Server 2013?

We do REST through BizTalk’s new WCF-WebHttp adapter. This adapter really is just another WCF adapter (like the SFTP adapter). It can be used on the Send or Receive side, in both one-way and two-way configurations.

image

In this case though, it is really clear that WCF is the underlying technology. Like all other service operations within BizTalk, we are exposed to operation mapping (the mapping of operations within orchestrations to actions in the service). However with the operation mappings in the WCF-WebHttp adapter, we don’t specify WCF actions so much as resources that we want to request (since it’s all just plain-old HTTP on the other end):

image

Do I Need a Dynamic Send Port To Do a GET?

The operation mapping shown above begs the question of how a GET request like this would work. Here we’re showing some sort of weather service that is providing temperature readings on a postal code basis. In order to make a request for the readings, one would have to access a URL that would change on a per postal code basis (i.e., the endpoint address changes).

Typically to accomplish something like sending to a dynamic address, one would use a dynamic send port. In the case of the WebHttp adapter, however, we get to specify variables within the path (which is shown in template form). In the screenshot above, we used the {postalCode} variable. Clicking the Edit button under Variable Mapping will lead us to the following screen:

image

Here we can see that in order to specify those variables, we simply pull from the context of the message traveling through the port. In this case we are grabbing the value for the {postalCode} part of the path from the Context Property named ZipCode.

Since most of the time that we are using the GET method, we don’t want to include a message (really it’s a request for a resource, and the dynamic part is the URL, not the body of the request), we can suppress the body using the Suppress Body for Verbs, and specifying the methods for which we want to not include a request body.

image

What Don’t I Get?

So far looking through the settings, it looks like we have a pretty good story around creating RESTful requests. However, we don’t have everything out of the box that we need to consume every single RESTful API out there.

You will still have to provide things like the following:

  • Encoding/decoding support (e.g., for JSON encoded messages)
  • WCF extensions for performing authentication (e.g., OAuth)
  • Schemas representing the requests/responses

These are some of the things that we will be covering within this series, so stay tuned over the next few weeks!

Do I Need BizTalk Server 2013 To Make this Happen?

While the adapter is new in BizTalk Server 2013, the underlying mechanism that makes it work is the WCF WebHttpBinding, which was around during the BizTalk Server 2010 timeframe. I’ve actually had people complain at me while explaining the WCF-WebHttp adapter as a new feature because of this.

Does that mean you can get it all up and running using something like the WCF-Custom adapter? Yes, and no. You can technically make it function with some additional legwork (e.g., as seen here), but you’re not going to get as smooth as an experience as you will see with BizTalk Server 2013 – it really feels cobbled together without having BizTalk Server 2013 at your disposal.

Coming Up Next

Next week I will be talking through the receive side of this story as we see how to host a RESTful endpoint in BizTalk Server 2013. Until then, take care (and take our classes)! Smile

Exploring the Out-of-the-box Support for SFTP in BizTalk Server 2013

By Nick Hauenstein

This post is the tenth in a weekly series intended to briefly spotlight those things that you need to know about new features in BizTalk Server 2013.

This week we’re going to examine the new SFTP adapter included with BizTalk Server 2013. You read that correctly, we finally have an SFTP adapter out of the box with BizTalk Server 2013!

Background Information

In the 2010 release, we saw the addition of the FTPS adapter (or rather the enhancement of the existing FTP adapter to support SSL), which was great for those that wanted to be able to transfer files between systems in an encrypted fashion. Since the SSL support was built into the FTP adapter, we were also able to take advantage of other enhancements (e.g., read-only FTP mode) at the same time.

The SFTP adapter is designed to support a completely different scenario. Here we’re not doing regular FTP with SSL encryption; instead we’re using an SSH tunnel to access a remote file system (with a protocol that is most definitely not FTP).

As a result, SFTP functionality isn’t something that could be easily hacked onto an existing adapter. Instead, there is a brand new adapter built from the ground up for inclusion in BizTalk Server 2013.

Putting it to the Test

While I have a few Linux boxes at home of various flavors, and even a few running on EC2, I have not yet spun-up a non-windows VM on WIndows Azure (it just doesn’t feel right). Thus, I felt that this would be a fitting opportunity to do so.

Unlike with the Windows Virtual Machines, the Quick Create experience does not allow you to specify a custom user name, so the user name I will be using (along with everyone else on the service) is azureuser. As part of the provisioning process, Windows Azure will configure sshd and generate an RSA 2048-bit key pair. After provisioning, the key thumbprint will be visible in the Windows Azure Management Portal:

image

From there I fired up PuTTY to login to the VM. The only details that you will need are the host name (your-vm-name-here.cloudapp.net), the user name (azureuser), and the password that you chose when creating the VM.

From there, I used nano1 to create a quick text file for testing purposes:

image

Great. Now I have a file. Let’s see if BizTalk 2013 can now access and do something meaningful with it.

Creating a Receive Location

For the purposes of this blog post, all I really want to do is get the file into BizTalk, examine the properties that the adapter is promoting, see how it responds in read-only situations, and then route it out to a file location to get it out of my message box and placed somewhere meaningful.

To that end, I created a one-way receive location and selected the SFTP adapter. Upon configuring the adapter, I was greeted with the following properties:

image

It is interesting to see that Microsoft ditched their regular language used to refer to public key fingerprints (the “thumbprint”), and went with the more widely used terminology. Other than that, there is not a lot of surprises here for those who have worked with the plain-old FTP adapter.

A few notable things are missing here that I would have expected, though. I would have liked to see integration with SSO (i.e., an SSO Affiliate property like we have in the FTP adapter), and I would like to have seen some read-only support as well (i.e., a Delete After Download property, and an Enable Timestamp comparison property)2.

Filling in the property grid with the information from my Azure VM, I ended up with:

image

In the Security section, I changed the AcceptAnySSHServerHostKey because I wanted to be certain that it was indeed my VM that was accepting the connection (and I knew the fingerprint to expect). I set the SSHServerHostKeyFingerPrint property to the finger print listed on the Window Azure Management Portal (after adding a colon ‘:’ every two characters to put it in a more standard format).

Hitting a Wall

Well, it always happens at some point in the day – a road block that leaves me scratching my head until after a cup of coffee (hopefully it’s not just me, eh?).

In this case, the road block was an error message in the Event Log that took the following form (reproduced for those exercising Bing-fu3 to resolve a problem):

The Messaging Engine failed to add a receive location “OrdersIn_SFTP” with URL “sftp://my-server-url-is-not-your-url.cloudapp.net:22/home/azureuser/*.txt” to the adapter “SFTP”. Reason: “Microsoft.BizTalk.Adapter.SftpInvoker.SftpException: Open SFTP connection error.
   at Microsoft.BizTalk.Adapter.SftpInvoker.SftpInvoker.Open()
   at Microsoft.BizTalk.Adapter.Sftp.SftpConnection.OpenUnderlyingConnection(SftpConnectionProperties connectionProperties)
   at Microsoft.BizTalk.Adapters.CommonHelpers.Connection`3.ConnectionPool`3.GetConnection(T1 connectionProperties, TimeSpan timeout)
   at Microsoft.BizTalk.Adapters.CommonHelpers.Connection`3.GetConnection(T connectionProperties, TimeSpan timeout)
   at Microsoft.BizTalk.Adapter.Sftp.SftpRLConfig.ValidateConfiguration(SftpReceivePropertyBag receivePropertyBag)
   at Microsoft.BizTalk.Adapter.Sftp.SftpRLConfig.CreateBinding(RHConfig rhConfig)
   at Microsoft.BizTalk.Adapter.Wcf.Runtime.BtsServiceHostBase.InitializeRuntime()
   at System.ServiceModel.ServiceHostBase.OnBeginOpen()
   at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfReceiveEndpoint.Enable()
   at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfReceiveEndpoint..ctor(BizTalkEndpointContext endpointContext, IBTTransportProxy transportProxy, ControlledTermination control)
   at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfReceiver`2.AddReceiveEndpoint(String url, IPropertyBag adapterConfig, IPropertyBag bizTalkConfig)”.

Say what? An error that goes all the way down to a WcfReceiver class? This just got crazy.

After spending some time changing settings to see which configuration option might be the culprit, I found that setting the AcceptAnySSHServerHotKey to True resulted in the error disappearing and the Receive Location staying alive. Unfortunately, I actually do care about that setting being set to False – I don’t want someone to impersonate my box and get away with it!

So then, what must we do here? Well, it turns out if one will simply RTFM, they will find happiness. So I decided to do that and found this article on MSDN which indicated that I had specified the host key fingerprint in the wrong format. It must include the type of key, the size of the key (in bits) and then the fingerprint.

Azure VMs use a 2048-bit RSA key, so that will need to be reflected in that setting. After some adjustments, my properties were updated to look like this:

image

Suspending the File for Inspection

Since I really wanted to get to the bottom of what this adapter could do for me, I decided to not create a Send Port initially (but rather to just have it suspend the message so that I could inspect the context, and then decide what to do with it later – if anything).

So I enabled the receive location and waited. The file appeared in the message box within the 5 second polling interval as configured, and yielded the following look into its context:

image

From the image above, you will find that there are quite a few properties from the WCF-properties namespace. In addition, you will see that the ReceivedFileName property from both the file-properties namespace, and the new sftp-properties namespace (dated 2012) has been written to the context (but not promoted). Beyond that, there wasn’t really anything super interesting :-/.

Checking on the server side of things, the file was indeed removed as expected:

image

What about Read Only?

I decided to disable the receive location for a few minutes so that I could setup a test wherein there was a file that a user would not have permissions to delete (this also required adding another user which I affectionately named nickh):

image

After I setup a new locked down directory, with a new locked down file, preventing all but read-only access to a new locked down user, I decided to update the Receive Location properties to see how BizTalk would react to the situation.

Guess what? The file isn’t deleted even after leaving the Receive Location running for some time, and the Event Log isn’t full of errors either. However, my output directory (yes, I did finally setup a Send Port to route out these suckers), is full:

image

Now about the Event Log, it is definitely not full of errors, but it is fairly full of warnings at this point:

image

The error reads:

The adapter “SFTP” raised an error message. Details “System.Exception: Message: Delete file error.. Sftp Error Code: ‘3221488222’. Sftp Error message: ‘The system cannot find the file specified.
‘. —> Microsoft.BizTalk.Adapter.SftpInvoker.SftpException: Delete file error.
   at Microsoft.BizTalk.Adapter.SftpInvoker.SftpInvoker.DeleteRemoteFile(String filename)
   at Microsoft.BizTalk.Adapter.Sftp.SftpFileReceiver.DownloadComplete(String fileName, Boolean deleteFile)
   — End of inner exception stack trace —
   at Microsoft.BizTalk.Adapter.Sftp.SftpFileReceiver.DownloadComplete(String fileName, Boolean deleteFile)
   at Microsoft.BizTalk.Adapter.Sftp.SftpRequestContext.Reply(Message message, TimeSpan timeout)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.Reply(MessageRpc& rpc)”.

Essentially, it will fail to delete the file, but still process the contents. If I need to receive the same nightly batch (with the same file name every day), that’s not a problem – I can set my polling interval to 1 day, or even mess with service windows to help out. But if the nightly batch has a new name every night (and the others remain in the directory), my life will be somewhat painful as I work to deal with the duplicate receives.

AN ASIDE: Peeling Back the Layers

After seeing WCF in the error message I received at the beginning of this exploration, I had to do some digging around to satisfy my curiosity.

Snooping around a bit inside the BizTalk Server 2013 installation direction, you will find how it all fits together. There’s an sshlib.dll and sshmessages.dll inside the Bins32 & Bins64 directories – these appear to be unmanaged code that implements the raw connection. Alongside those, we see a managed library named Microsoft.Biztalk.Adapter.SftpInvoker.dll. Finally, at the root of the installation directory, there is a Microsoft.BizTalk.Adapter.Sftp.dll. Visual Studio’s object browser reveals that this is, indeed, like the Service Bus adapters, yet another new WCF adapter (likely using a custom binding that somehow relies on the aforementioned components):image

Unfortunately, it does not appear to publicly expose whatever binding it is using, so it is unlikely that the implementation here will be re-usable outside of BizTalk.

Bringing it to a Close

I hope this article helped you out in one way or another. Maybe you found out that there is an SFTP adapter out of the box in BizTalk Server 2013 and that was big news for you. Maybe you were curious about how it would handle read-only situations, and now you have some idea of that behavior. Or maybe you were struggling with the configuration, and something here has shed light on an issue you were encountering. Either way, best of luck to you on your integration projects.

If you want to learn more about the new features in BizTalk Server 2013, or even want to learn BizTalk Server for the first time, check out one of our BizTalk Server 2013 Developer Immersion of BizTalk Server 2013 Developer Deep Dive classes running now from your home, at your workplace, or right here in Kirkland, WA.

1I choose to be Switzerland in the great editor war – though you will find on my VM that vi is installed while emacs is not.

2Listen, I’m not going to be that guy and not tell you that you can still achieve your goal here. Check out the AfterGetActions enum in this bad boy. That adapter still exists, and still lets you do what you want to do.

3Yes, that is me shamelessly selling out. They give you a free Redbox rental after 110 rewards points. I’m a sucker for points and free movies.