Exploring the Out-of-the-box Support for SFTP in BizTalk Server 2013

By Nick Hauenstein

This post is the tenth in a weekly series intended to briefly spotlight those things that you need to know about new features in BizTalk Server 2013.

This week we’re going to examine the new SFTP adapter included with BizTalk Server 2013. You read that correctly, we finally have an SFTP adapter out of the box with BizTalk Server 2013!

Background Information

In the 2010 release, we saw the addition of the FTPS adapter (or rather the enhancement of the existing FTP adapter to support SSL), which was great for those that wanted to be able to transfer files between systems in an encrypted fashion. Since the SSL support was built into the FTP adapter, we were also able to take advantage of other enhancements (e.g., read-only FTP mode) at the same time.

The SFTP adapter is designed to support a completely different scenario. Here we’re not doing regular FTP with SSL encryption; instead we’re using an SSH tunnel to access a remote file system (with a protocol that is most definitely not FTP).

As a result, SFTP functionality isn’t something that could be easily hacked onto an existing adapter. Instead, there is a brand new adapter built from the ground up for inclusion in BizTalk Server 2013.

Putting it to the Test

While I have a few Linux boxes at home of various flavors, and even a few running on EC2, I have not yet spun-up a non-windows VM on WIndows Azure (it just doesn’t feel right). Thus, I felt that this would be a fitting opportunity to do so.

Unlike with the Windows Virtual Machines, the Quick Create experience does not allow you to specify a custom user name, so the user name I will be using (along with everyone else on the service) is azureuser. As part of the provisioning process, Windows Azure will configure sshd and generate an RSA 2048-bit key pair. After provisioning, the key thumbprint will be visible in the Windows Azure Management Portal:

image

From there I fired up PuTTY to login to the VM. The only details that you will need are the host name (your-vm-name-here.cloudapp.net), the user name (azureuser), and the password that you chose when creating the VM.

From there, I used nano1 to create a quick text file for testing purposes:

image

Great. Now I have a file. Let’s see if BizTalk 2013 can now access and do something meaningful with it.

Creating a Receive Location

For the purposes of this blog post, all I really want to do is get the file into BizTalk, examine the properties that the adapter is promoting, see how it responds in read-only situations, and then route it out to a file location to get it out of my message box and placed somewhere meaningful.

To that end, I created a one-way receive location and selected the SFTP adapter. Upon configuring the adapter, I was greeted with the following properties:

image

It is interesting to see that Microsoft ditched their regular language used to refer to public key fingerprints (the “thumbprint”), and went with the more widely used terminology. Other than that, there is not a lot of surprises here for those who have worked with the plain-old FTP adapter.

A few notable things are missing here that I would have expected, though. I would have liked to see integration with SSO (i.e., an SSO Affiliate property like we have in the FTP adapter), and I would like to have seen some read-only support as well (i.e., a Delete After Download property, and an Enable Timestamp comparison property)2.

Filling in the property grid with the information from my Azure VM, I ended up with:

image

In the Security section, I changed the AcceptAnySSHServerHostKey because I wanted to be certain that it was indeed my VM that was accepting the connection (and I knew the fingerprint to expect). I set the SSHServerHostKeyFingerPrint property to the finger print listed on the Window Azure Management Portal (after adding a colon ‘:’ every two characters to put it in a more standard format).

Hitting a Wall

Well, it always happens at some point in the day – a road block that leaves me scratching my head until after a cup of coffee (hopefully it’s not just me, eh?).

In this case, the road block was an error message in the Event Log that took the following form (reproduced for those exercising Bing-fu3 to resolve a problem):

The Messaging Engine failed to add a receive location “OrdersIn_SFTP” with URL “sftp://my-server-url-is-not-your-url.cloudapp.net:22/home/azureuser/*.txt” to the adapter “SFTP”. Reason: “Microsoft.BizTalk.Adapter.SftpInvoker.SftpException: Open SFTP connection error.
   at Microsoft.BizTalk.Adapter.SftpInvoker.SftpInvoker.Open()
   at Microsoft.BizTalk.Adapter.Sftp.SftpConnection.OpenUnderlyingConnection(SftpConnectionProperties connectionProperties)
   at Microsoft.BizTalk.Adapters.CommonHelpers.Connection`3.ConnectionPool`3.GetConnection(T1 connectionProperties, TimeSpan timeout)
   at Microsoft.BizTalk.Adapters.CommonHelpers.Connection`3.GetConnection(T connectionProperties, TimeSpan timeout)
   at Microsoft.BizTalk.Adapter.Sftp.SftpRLConfig.ValidateConfiguration(SftpReceivePropertyBag receivePropertyBag)
   at Microsoft.BizTalk.Adapter.Sftp.SftpRLConfig.CreateBinding(RHConfig rhConfig)
   at Microsoft.BizTalk.Adapter.Wcf.Runtime.BtsServiceHostBase.InitializeRuntime()
   at System.ServiceModel.ServiceHostBase.OnBeginOpen()
   at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfReceiveEndpoint.Enable()
   at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfReceiveEndpoint..ctor(BizTalkEndpointContext endpointContext, IBTTransportProxy transportProxy, ControlledTermination control)
   at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfReceiver`2.AddReceiveEndpoint(String url, IPropertyBag adapterConfig, IPropertyBag bizTalkConfig)”.

Say what? An error that goes all the way down to a WcfReceiver class? This just got crazy.

After spending some time changing settings to see which configuration option might be the culprit, I found that setting the AcceptAnySSHServerHotKey to True resulted in the error disappearing and the Receive Location staying alive. Unfortunately, I actually do care about that setting being set to False – I don’t want someone to impersonate my box and get away with it!

So then, what must we do here? Well, it turns out if one will simply RTFM, they will find happiness. So I decided to do that and found this article on MSDN which indicated that I had specified the host key fingerprint in the wrong format. It must include the type of key, the size of the key (in bits) and then the fingerprint.

Azure VMs use a 2048-bit RSA key, so that will need to be reflected in that setting. After some adjustments, my properties were updated to look like this:

image

Suspending the File for Inspection

Since I really wanted to get to the bottom of what this adapter could do for me, I decided to not create a Send Port initially (but rather to just have it suspend the message so that I could inspect the context, and then decide what to do with it later – if anything).

So I enabled the receive location and waited. The file appeared in the message box within the 5 second polling interval as configured, and yielded the following look into its context:

image

From the image above, you will find that there are quite a few properties from the WCF-properties namespace. In addition, you will see that the ReceivedFileName property from both the file-properties namespace, and the new sftp-properties namespace (dated 2012) has been written to the context (but not promoted). Beyond that, there wasn’t really anything super interesting :-/.

Checking on the server side of things, the file was indeed removed as expected:

image

What about Read Only?

I decided to disable the receive location for a few minutes so that I could setup a test wherein there was a file that a user would not have permissions to delete (this also required adding another user which I affectionately named nickh):

image

After I setup a new locked down directory, with a new locked down file, preventing all but read-only access to a new locked down user, I decided to update the Receive Location properties to see how BizTalk would react to the situation.

Guess what? The file isn’t deleted even after leaving the Receive Location running for some time, and the Event Log isn’t full of errors either. However, my output directory (yes, I did finally setup a Send Port to route out these suckers), is full:

image

Now about the Event Log, it is definitely not full of errors, but it is fairly full of warnings at this point:

image

The error reads:

The adapter “SFTP” raised an error message. Details “System.Exception: Message: Delete file error.. Sftp Error Code: ‘3221488222’. Sftp Error message: ‘The system cannot find the file specified.
‘. —> Microsoft.BizTalk.Adapter.SftpInvoker.SftpException: Delete file error.
   at Microsoft.BizTalk.Adapter.SftpInvoker.SftpInvoker.DeleteRemoteFile(String filename)
   at Microsoft.BizTalk.Adapter.Sftp.SftpFileReceiver.DownloadComplete(String fileName, Boolean deleteFile)
   — End of inner exception stack trace —
   at Microsoft.BizTalk.Adapter.Sftp.SftpFileReceiver.DownloadComplete(String fileName, Boolean deleteFile)
   at Microsoft.BizTalk.Adapter.Sftp.SftpRequestContext.Reply(Message message, TimeSpan timeout)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.Reply(MessageRpc& rpc)”.

Essentially, it will fail to delete the file, but still process the contents. If I need to receive the same nightly batch (with the same file name every day), that’s not a problem – I can set my polling interval to 1 day, or even mess with service windows to help out. But if the nightly batch has a new name every night (and the others remain in the directory), my life will be somewhat painful as I work to deal with the duplicate receives.

AN ASIDE: Peeling Back the Layers

After seeing WCF in the error message I received at the beginning of this exploration, I had to do some digging around to satisfy my curiosity.

Snooping around a bit inside the BizTalk Server 2013 installation direction, you will find how it all fits together. There’s an sshlib.dll and sshmessages.dll inside the Bins32 & Bins64 directories – these appear to be unmanaged code that implements the raw connection. Alongside those, we see a managed library named Microsoft.Biztalk.Adapter.SftpInvoker.dll. Finally, at the root of the installation directory, there is a Microsoft.BizTalk.Adapter.Sftp.dll. Visual Studio’s object browser reveals that this is, indeed, like the Service Bus adapters, yet another new WCF adapter (likely using a custom binding that somehow relies on the aforementioned components):image

Unfortunately, it does not appear to publicly expose whatever binding it is using, so it is unlikely that the implementation here will be re-usable outside of BizTalk.

Bringing it to a Close

I hope this article helped you out in one way or another. Maybe you found out that there is an SFTP adapter out of the box in BizTalk Server 2013 and that was big news for you. Maybe you were curious about how it would handle read-only situations, and now you have some idea of that behavior. Or maybe you were struggling with the configuration, and something here has shed light on an issue you were encountering. Either way, best of luck to you on your integration projects.

If you want to learn more about the new features in BizTalk Server 2013, or even want to learn BizTalk Server for the first time, check out one of our BizTalk Server 2013 Developer Immersion of BizTalk Server 2013 Developer Deep Dive classes running now from your home, at your workplace, or right here in Kirkland, WA.

1I choose to be Switzerland in the great editor war – though you will find on my VM that vi is installed while emacs is not.

2Listen, I’m not going to be that guy and not tell you that you can still achieve your goal here. Check out the AfterGetActions enum in this bad boy. That adapter still exists, and still lets you do what you want to do.

3Yes, that is me shamelessly selling out. They give you a free Redbox rental after 110 rewards points. I’m a sucker for points and free movies.

A Brief Tour of the BizTalk Server 2013 Evaluation Windows Azure Virtual Machine

By Nick Hauenstein

This post is the sixth in a weekly series intended to briefly spotlight those things that you need to know about new features in BizTalk Server 2013.

In this blog post, I’m going to take a little bit of a departure (for this week only) from the tone of my earlier posts – which focused on scenarios you will encounter after using the product – and look instead at things that would be of interest to people who are new to product, or are still evaluating BizTalk Server 2013 as an option for enterprise integration.

By far the quickest way to get started with BizTalk Server 2013 is to be handed a machine that has it pre-installed. With the Infrastructure-as-a-Service offering in BizTalk Server 2013, we have just that. In fact, we have it both for production as well as evaluation use.

Just Give Me a VM

While attending “//build/” this week (quick shout-out to those also here), I spent some time in the big room next to the Windows Azure booths. They had 3 challenges for attendees that involved completing certain tasks using the platform, and were offering a prize for each challenge completed. They were designed to sound somewhat difficult, yet prove to the participant just how easy life can be with Windows Azure. The first challenge was to provision a virtual machine running Windows Server 2012 — a trivial task (like it should be).

Well guess what? Provisioning a BizTalk Server 2013 instance is just as easy. You can start by logging into the Windows Azure Management Portal. From there, I can show you all the requisite clicks in a single screenshot:

ProvisionVMClicks 

Coffee Break #1

That should kick-off the provisioning process for the Virtual Machine, and also starts off your first coffee break within this process.

provisioning

Get Connected

So you have a virtual machine running BizTalk Server in the cloud, now what? Well you probably want to get connected, and remote in so that you can make use of it. That can be done by clicking the Connect button in the toolbar while the virtual machine is highlighted in the list of Virtual Machines.

ConnectVM

This will cause a .rdp file to download with all of the settings (DNS name/port) required to make an Remote Desktop Protocol (RDP) connection, and fire up the Microsoft Terminal Services Client (still named mstsc.exe) to make it all happen. Make sure your internal firewall allows this, or you will not have much fun after this point.

You will be connecting using the user name that you entered while creating the virtual machine (not your internal domain account). The virtual machine starts out in workgroup mode, and will not be joined to a domain.

Make IE Usable

I think we all know that secure != usable, and sometimes there is an inverse relationship between the two. You will find this to be the case with the virtual machine that you have just provisioned. It is configured to be a headless server by default, and to have a really small attack surface. That’s fantastic, and I love security by default. However, for our purposes of evaluating the product, and doing some development within the Virtual Machine, it’s probably not our first choice.

As a result, you will probably want to head straight over to Server Manager first thing (which will open up automatically). From there, you will click Local Server, and then proceed to toggle IE Enhanced Security Configuration to the Off state:

IE_ESC

What will this do for us? It will allow us to use the Internet in a more reasonable fashion. But it will also potentially open you up to make poor choices. Do this with care:

IE_ESC_P2

Write Some Code

If you’re evaluating BizTalk Server 2013, it’s likely that you will want to have a development environment available to you – Visual Studio 2012 would be our environment of choice in this case. If we want to do anything interesting with BAM activities and views, we will also need Microsoft Office installed.

So you will probably want to get those installed. With double-click mounting of disk images, it shouldn’t be a problem to install once you have the download links to the ISO files. Where do those live? Right here (but your best bet is to go to the product sites directly and walk through the registration process which will give you a longer evaluation period without persistent nag screens):

Configure This Beast

At this point you will have quite a few tools installed, but a BizTalk Group hasn’t yet been setup. Luckily, since you’re using the Evaluation image, you already have SQL Server installed and ready to serve. At this point you could run through the configuration wizard, or you could go with some happy defaults and just get it done.

So let’s go with the happy defaults. Go ahead and download this file.

Open it up, and then replace the following values, and then save it out to D:config.xml:

Special String Replace With
@@ADMINUSER@@ Name of user that you created when you created the virtual machine
@@MACHINENAME@@ Name of your virtual machine
@@PASSWORD@@ Password for the user that you created when you created the virtual machine

Once you have that all put together, you can automate the configuration by executing the following commands at the command prompt:

cd “C:Program Files (x86)Microsoft BizTalk Server 2013”
.Configuration.exe /s “D:config.xml” /l “D:config.log”

This should bring you around to Coffee Break #2

What About the Adapter Pack / ESB Toolkit?

At this point you have a usable installation of BizTalk which will give you hours of fun. But what if you want to do something with one of the adapters in the Adapter Pack, or even install the ESB Toolkit? In that case, you will find the binaries for the BizTalk 2013 Evaluation edition at: C:BizTalkServer2013Evaluation

Now What?

Now that you have a fully configured BizTalk environment. Kick the tires, play around, look through this blog for some of the capabilities you can experiment with. After that, get some training to find out what BizTalk Server is really capable of. I recommend you check out one of our upcoming BizTalk Server 2013 classes. We have classes for Administrators and classes for Developers now available on the QuickLearn Training calendar.

That’s all from me at //build/, see you next week!