This week we’re going to examine the new SFTP adapter included with BizTalk Server 2013. You read that correctly, we finally have an SFTP adapter out of the box with BizTalk Server 2013!
In the 2010 release, we saw the addition of the FTPS adapter (or rather the enhancement of the existing FTP adapter to support SSL), which was great for those that wanted to be able to transfer files between systems in an encrypted fashion. Since the SSL support was built into the FTP adapter, we were also able to take advantage of other enhancements (e.g., read-only FTP mode) at the same time.
The SFTP adapter is designed to support a completely different scenario. Here we’re not doing regular FTP with SSL encryption; instead we’re using an SSH tunnel to access a remote file system (with a protocol that is most definitely not FTP).
As a result, SFTP functionality isn’t something that could be easily hacked onto an existing adapter. Instead, there is a brand new adapter built from the ground up for inclusion in BizTalk Server 2013.
Putting it to the Test
While I have a few Linux boxes at home of various flavors, and even a few running on EC2, I have not yet spun-up a non-windows VM on WIndows Azure (it just doesn’t feel right). Thus, I felt that this would be a fitting opportunity to do so.
Unlike with the Windows Virtual Machines, the Quick Create experience does not allow you to specify a custom user name, so the user name I will be using (along with everyone else on the service) is azureuser. As part of the provisioning process, Windows Azure will configure sshd and generate an RSA 2048-bit key pair. After provisioning, the key thumbprint will be visible in the Windows Azure Management Portal:
From there I fired up PuTTY to login to the VM. The only details that you will need are the host name (your-vm-name-here.cloudapp.net), the user name (azureuser), and the password that you chose when creating the VM.
From there, I used nano1 to create a quick text file for testing purposes:
Great. Now I have a file. Let’s see if BizTalk 2013 can now access and do something meaningful with it.
Creating a Receive Location
For the purposes of this blog post, all I really want to do is get the file into BizTalk, examine the properties that the adapter is promoting, see how it responds in read-only situations, and then route it out to a file location to get it out of my message box and placed somewhere meaningful.
To that end, I created a one-way receive location and selected the SFTP adapter. Upon configuring the adapter, I was greeted with the following properties:
It is interesting to see that Microsoft ditched their regular language used to refer to public key fingerprints (the “thumbprint”), and went with the more widely used terminology. Other than that, there is not a lot of surprises here for those who have worked with the plain-old FTP adapter.
A few notable things are missing here that I would have expected, though. I would have liked to see integration with SSO (i.e., an SSO Affiliate property like we have in the FTP adapter), and I would like to have seen some read-only support as well (i.e., a Delete After Download property, and an Enable Timestamp comparison property)2.
Filling in the property grid with the information from my Azure VM, I ended up with:
In the Security section, I changed the AcceptAnySSHServerHostKey because I wanted to be certain that it was indeed my VM that was accepting the connection (and I knew the fingerprint to expect). I set the SSHServerHostKeyFingerPrint property to the finger print listed on the Window Azure Management Portal (after adding a colon ‘:’ every two characters to put it in a more standard format).
Hitting a Wall
Well, it always happens at some point in the day – a road block that leaves me scratching my head until after a cup of coffee (hopefully it’s not just me, eh?).
In this case, the road block was an error message in the Event Log that took the following form (reproduced for those exercising Bing-fu3 to resolve a problem):
The Messaging Engine failed to add a receive location “OrdersIn_SFTP” with URL “sftp://my-server-url-is-not-your-url.cloudapp.net:22/home/azureuser/*.txt” to the adapter “SFTP”. Reason: “Microsoft.BizTalk.Adapter.SftpInvoker.SftpException: Open SFTP connection error.
at Microsoft.BizTalk.Adapter.Sftp.SftpConnection.OpenUnderlyingConnection(SftpConnectionProperties connectionProperties)
at Microsoft.BizTalk.Adapters.CommonHelpers.Connection`3.ConnectionPool`3.GetConnection(T1 connectionProperties, TimeSpan timeout)
at Microsoft.BizTalk.Adapters.CommonHelpers.Connection`3.GetConnection(T connectionProperties, TimeSpan timeout)
at Microsoft.BizTalk.Adapter.Sftp.SftpRLConfig.ValidateConfiguration(SftpReceivePropertyBag receivePropertyBag)
at Microsoft.BizTalk.Adapter.Sftp.SftpRLConfig.CreateBinding(RHConfig rhConfig)
at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfReceiveEndpoint..ctor(BizTalkEndpointContext endpointContext, IBTTransportProxy transportProxy, ControlledTermination control)
at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfReceiver`2.AddReceiveEndpoint(String url, IPropertyBag adapterConfig, IPropertyBag bizTalkConfig)”.
Say what? An error that goes all the way down to a WcfReceiver class? This just got crazy.
After spending some time changing settings to see which configuration option might be the culprit, I found that setting the AcceptAnySSHServerHotKey to True resulted in the error disappearing and the Receive Location staying alive. Unfortunately, I actually do care about that setting being set to False – I don’t want someone to impersonate my box and get away with it!
So then, what must we do here? Well, it turns out if one will simply RTFM, they will find happiness. So I decided to do that and found this article on MSDN which indicated that I had specified the host key fingerprint in the wrong format. It must include the type of key, the size of the key (in bits) and then the fingerprint.
Azure VMs use a 2048-bit RSA key, so that will need to be reflected in that setting. After some adjustments, my properties were updated to look like this:
Suspending the File for Inspection
Since I really wanted to get to the bottom of what this adapter could do for me, I decided to not create a Send Port initially (but rather to just have it suspend the message so that I could inspect the context, and then decide what to do with it later – if anything).
So I enabled the receive location and waited. The file appeared in the message box within the 5 second polling interval as configured, and yielded the following look into its context:
From the image above, you will find that there are quite a few properties from the WCF-properties namespace. In addition, you will see that the ReceivedFileName property from both the file-properties namespace, and the new sftp-properties namespace (dated 2012) has been written to the context (but not promoted). Beyond that, there wasn’t really anything super interesting :-/.
Checking on the server side of things, the file was indeed removed as expected:
What about Read Only?
I decided to disable the receive location for a few minutes so that I could setup a test wherein there was a file that a user would not have permissions to delete (this also required adding another user which I affectionately named nickh):
After I setup a new locked down directory, with a new locked down file, preventing all but read-only access to a new locked down user, I decided to update the Receive Location properties to see how BizTalk would react to the situation.
Guess what? The file isn’t deleted even after leaving the Receive Location running for some time, and the Event Log isn’t full of errors either. However, my output directory (yes, I did finally setup a Send Port to route out these suckers), is full:
Now about the Event Log, it is definitely not full of errors, but it is fairly full of warnings at this point:
The error reads:
The adapter “SFTP” raised an error message. Details “System.Exception: Message: Delete file error.. Sftp Error Code: ‘3221488222’. Sftp Error message: ‘The system cannot find the file specified.
‘. —> Microsoft.BizTalk.Adapter.SftpInvoker.SftpException: Delete file error.
at Microsoft.BizTalk.Adapter.SftpInvoker.SftpInvoker.DeleteRemoteFile(String filename)
at Microsoft.BizTalk.Adapter.Sftp.SftpFileReceiver.DownloadComplete(String fileName, Boolean deleteFile)
— End of inner exception stack trace —
at Microsoft.BizTalk.Adapter.Sftp.SftpFileReceiver.DownloadComplete(String fileName, Boolean deleteFile)
at Microsoft.BizTalk.Adapter.Sftp.SftpRequestContext.Reply(Message message, TimeSpan timeout)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.Reply(MessageRpc& rpc)”.
Essentially, it will fail to delete the file, but still process the contents. If I need to receive the same nightly batch (with the same file name every day), that’s not a problem – I can set my polling interval to 1 day, or even mess with service windows to help out. But if the nightly batch has a new name every night (and the others remain in the directory), my life will be somewhat painful as I work to deal with the duplicate receives.
AN ASIDE: Peeling Back the Layers
After seeing WCF in the error message I received at the beginning of this exploration, I had to do some digging around to satisfy my curiosity.
Snooping around a bit inside the BizTalk Server 2013 installation direction, you will find how it all fits together. There’s an sshlib.dll and sshmessages.dll inside the Bins32 & Bins64 directories – these appear to be unmanaged code that implements the raw connection. Alongside those, we see a managed library named Microsoft.Biztalk.Adapter.SftpInvoker.dll. Finally, at the root of the installation directory, there is a Microsoft.BizTalk.Adapter.Sftp.dll. Visual Studio’s object browser reveals that this is, indeed, like the Service Bus adapters, yet another new WCF adapter (likely using a custom binding that somehow relies on the aforementioned components):
Unfortunately, it does not appear to publicly expose whatever binding it is using, so it is unlikely that the implementation here will be re-usable outside of BizTalk.
Bringing it to a Close
I hope this article helped you out in one way or another. Maybe you found out that there is an SFTP adapter out of the box in BizTalk Server 2013 and that was big news for you. Maybe you were curious about how it would handle read-only situations, and now you have some idea of that behavior. Or maybe you were struggling with the configuration, and something here has shed light on an issue you were encountering. Either way, best of luck to you on your integration projects.
If you want to learn more about the new features in BizTalk Server 2013, or even want to learn BizTalk Server for the first time, check out one of our BizTalk Server 2013 Developer Immersion of BizTalk Server 2013 Developer Deep Dive classes running now from your home, at your workplace, or right here in Kirkland, WA.
1I choose to be Switzerland in the great editor war – though you will find on my VM that vi is installed while emacs is not.
2Listen, I’m not going to be that guy and not tell you that you can still achieve your goal here. Check out the AfterGetActions enum in this bad boy. That adapter still exists, and still lets you do what you want to do.
3Yes, that is me shamelessly selling out. They give you a free Redbox rental after 110 rewards points. I’m a sucker for points and free movies.