Exploring the Out-of-the-box Support for SFTP in BizTalk Server 2013

By Nick Hauenstein

This post is the tenth in a weekly series intended to briefly spotlight those things that you need to know about new features in BizTalk Server 2013.

This week we’re going to examine the new SFTP adapter included with BizTalk Server 2013. You read that correctly, we finally have an SFTP adapter out of the box with BizTalk Server 2013!

Background Information

In the 2010 release, we saw the addition of the FTPS adapter (or rather the enhancement of the existing FTP adapter to support SSL), which was great for those that wanted to be able to transfer files between systems in an encrypted fashion. Since the SSL support was built into the FTP adapter, we were also able to take advantage of other enhancements (e.g., read-only FTP mode) at the same time.

The SFTP adapter is designed to support a completely different scenario. Here we’re not doing regular FTP with SSL encryption; instead we’re using an SSH tunnel to access a remote file system (with a protocol that is most definitely not FTP).

As a result, SFTP functionality isn’t something that could be easily hacked onto an existing adapter. Instead, there is a brand new adapter built from the ground up for inclusion in BizTalk Server 2013.

Putting it to the Test

While I have a few Linux boxes at home of various flavors, and even a few running on EC2, I have not yet spun-up a non-windows VM on WIndows Azure (it just doesn’t feel right). Thus, I felt that this would be a fitting opportunity to do so.

Unlike with the Windows Virtual Machines, the Quick Create experience does not allow you to specify a custom user name, so the user name I will be using (along with everyone else on the service) is azureuser. As part of the provisioning process, Windows Azure will configure sshd and generate an RSA 2048-bit key pair. After provisioning, the key thumbprint will be visible in the Windows Azure Management Portal:

image

From there I fired up PuTTY to login to the VM. The only details that you will need are the host name (your-vm-name-here.cloudapp.net), the user name (azureuser), and the password that you chose when creating the VM.

From there, I used nano1 to create a quick text file for testing purposes:

image

Great. Now I have a file. Let’s see if BizTalk 2013 can now access and do something meaningful with it.

Creating a Receive Location

For the purposes of this blog post, all I really want to do is get the file into BizTalk, examine the properties that the adapter is promoting, see how it responds in read-only situations, and then route it out to a file location to get it out of my message box and placed somewhere meaningful.

To that end, I created a one-way receive location and selected the SFTP adapter. Upon configuring the adapter, I was greeted with the following properties:

image

It is interesting to see that Microsoft ditched their regular language used to refer to public key fingerprints (the “thumbprint”), and went with the more widely used terminology. Other than that, there is not a lot of surprises here for those who have worked with the plain-old FTP adapter.

A few notable things are missing here that I would have expected, though. I would have liked to see integration with SSO (i.e., an SSO Affiliate property like we have in the FTP adapter), and I would like to have seen some read-only support as well (i.e., a Delete After Download property, and an Enable Timestamp comparison property)2.

Filling in the property grid with the information from my Azure VM, I ended up with:

image

In the Security section, I changed the AcceptAnySSHServerHostKey because I wanted to be certain that it was indeed my VM that was accepting the connection (and I knew the fingerprint to expect). I set the SSHServerHostKeyFingerPrint property to the finger print listed on the Window Azure Management Portal (after adding a colon ‘:’ every two characters to put it in a more standard format).

Hitting a Wall

Well, it always happens at some point in the day – a road block that leaves me scratching my head until after a cup of coffee (hopefully it’s not just me, eh?).

In this case, the road block was an error message in the Event Log that took the following form (reproduced for those exercising Bing-fu3 to resolve a problem):

The Messaging Engine failed to add a receive location “OrdersIn_SFTP” with URL “sftp://my-server-url-is-not-your-url.cloudapp.net:22/home/azureuser/*.txt” to the adapter “SFTP”. Reason: “Microsoft.BizTalk.Adapter.SftpInvoker.SftpException: Open SFTP connection error.
   at Microsoft.BizTalk.Adapter.SftpInvoker.SftpInvoker.Open()
   at Microsoft.BizTalk.Adapter.Sftp.SftpConnection.OpenUnderlyingConnection(SftpConnectionProperties connectionProperties)
   at Microsoft.BizTalk.Adapters.CommonHelpers.Connection`3.ConnectionPool`3.GetConnection(T1 connectionProperties, TimeSpan timeout)
   at Microsoft.BizTalk.Adapters.CommonHelpers.Connection`3.GetConnection(T connectionProperties, TimeSpan timeout)
   at Microsoft.BizTalk.Adapter.Sftp.SftpRLConfig.ValidateConfiguration(SftpReceivePropertyBag receivePropertyBag)
   at Microsoft.BizTalk.Adapter.Sftp.SftpRLConfig.CreateBinding(RHConfig rhConfig)
   at Microsoft.BizTalk.Adapter.Wcf.Runtime.BtsServiceHostBase.InitializeRuntime()
   at System.ServiceModel.ServiceHostBase.OnBeginOpen()
   at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfReceiveEndpoint.Enable()
   at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfReceiveEndpoint..ctor(BizTalkEndpointContext endpointContext, IBTTransportProxy transportProxy, ControlledTermination control)
   at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfReceiver`2.AddReceiveEndpoint(String url, IPropertyBag adapterConfig, IPropertyBag bizTalkConfig)”.

Say what? An error that goes all the way down to a WcfReceiver class? This just got crazy.

After spending some time changing settings to see which configuration option might be the culprit, I found that setting the AcceptAnySSHServerHotKey to True resulted in the error disappearing and the Receive Location staying alive. Unfortunately, I actually do care about that setting being set to False – I don’t want someone to impersonate my box and get away with it!

So then, what must we do here? Well, it turns out if one will simply RTFM, they will find happiness. So I decided to do that and found this article on MSDN which indicated that I had specified the host key fingerprint in the wrong format. It must include the type of key, the size of the key (in bits) and then the fingerprint.

Azure VMs use a 2048-bit RSA key, so that will need to be reflected in that setting. After some adjustments, my properties were updated to look like this:

image

Suspending the File for Inspection

Since I really wanted to get to the bottom of what this adapter could do for me, I decided to not create a Send Port initially (but rather to just have it suspend the message so that I could inspect the context, and then decide what to do with it later – if anything).

So I enabled the receive location and waited. The file appeared in the message box within the 5 second polling interval as configured, and yielded the following look into its context:

image

From the image above, you will find that there are quite a few properties from the WCF-properties namespace. In addition, you will see that the ReceivedFileName property from both the file-properties namespace, and the new sftp-properties namespace (dated 2012) has been written to the context (but not promoted). Beyond that, there wasn’t really anything super interesting :-/.

Checking on the server side of things, the file was indeed removed as expected:

image

What about Read Only?

I decided to disable the receive location for a few minutes so that I could setup a test wherein there was a file that a user would not have permissions to delete (this also required adding another user which I affectionately named nickh):

image

After I setup a new locked down directory, with a new locked down file, preventing all but read-only access to a new locked down user, I decided to update the Receive Location properties to see how BizTalk would react to the situation.

Guess what? The file isn’t deleted even after leaving the Receive Location running for some time, and the Event Log isn’t full of errors either. However, my output directory (yes, I did finally setup a Send Port to route out these suckers), is full:

image

Now about the Event Log, it is definitely not full of errors, but it is fairly full of warnings at this point:

image

The error reads:

The adapter “SFTP” raised an error message. Details “System.Exception: Message: Delete file error.. Sftp Error Code: ‘3221488222’. Sftp Error message: ‘The system cannot find the file specified.
‘. —> Microsoft.BizTalk.Adapter.SftpInvoker.SftpException: Delete file error.
   at Microsoft.BizTalk.Adapter.SftpInvoker.SftpInvoker.DeleteRemoteFile(String filename)
   at Microsoft.BizTalk.Adapter.Sftp.SftpFileReceiver.DownloadComplete(String fileName, Boolean deleteFile)
   — End of inner exception stack trace —
   at Microsoft.BizTalk.Adapter.Sftp.SftpFileReceiver.DownloadComplete(String fileName, Boolean deleteFile)
   at Microsoft.BizTalk.Adapter.Sftp.SftpRequestContext.Reply(Message message, TimeSpan timeout)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.Reply(MessageRpc& rpc)”.

Essentially, it will fail to delete the file, but still process the contents. If I need to receive the same nightly batch (with the same file name every day), that’s not a problem – I can set my polling interval to 1 day, or even mess with service windows to help out. But if the nightly batch has a new name every night (and the others remain in the directory), my life will be somewhat painful as I work to deal with the duplicate receives.

AN ASIDE: Peeling Back the Layers

After seeing WCF in the error message I received at the beginning of this exploration, I had to do some digging around to satisfy my curiosity.

Snooping around a bit inside the BizTalk Server 2013 installation direction, you will find how it all fits together. There’s an sshlib.dll and sshmessages.dll inside the Bins32 & Bins64 directories – these appear to be unmanaged code that implements the raw connection. Alongside those, we see a managed library named Microsoft.Biztalk.Adapter.SftpInvoker.dll. Finally, at the root of the installation directory, there is a Microsoft.BizTalk.Adapter.Sftp.dll. Visual Studio’s object browser reveals that this is, indeed, like the Service Bus adapters, yet another new WCF adapter (likely using a custom binding that somehow relies on the aforementioned components):image

Unfortunately, it does not appear to publicly expose whatever binding it is using, so it is unlikely that the implementation here will be re-usable outside of BizTalk.

Bringing it to a Close

I hope this article helped you out in one way or another. Maybe you found out that there is an SFTP adapter out of the box in BizTalk Server 2013 and that was big news for you. Maybe you were curious about how it would handle read-only situations, and now you have some idea of that behavior. Or maybe you were struggling with the configuration, and something here has shed light on an issue you were encountering. Either way, best of luck to you on your integration projects.

If you want to learn more about the new features in BizTalk Server 2013, or even want to learn BizTalk Server for the first time, check out one of our BizTalk Server 2013 Developer Immersion of BizTalk Server 2013 Developer Deep Dive classes running now from your home, at your workplace, or right here in Kirkland, WA.

1I choose to be Switzerland in the great editor war – though you will find on my VM that vi is installed while emacs is not.

2Listen, I’m not going to be that guy and not tell you that you can still achieve your goal here. Check out the AfterGetActions enum in this bad boy. That adapter still exists, and still lets you do what you want to do.

3Yes, that is me shamelessly selling out. They give you a free Redbox rental after 110 rewards points. I’m a sucker for points and free movies.

Automating and Managing BizTalk Server 2013 with PowerShell

By Nick Hauenstein

This post is the ninth in a weekly series intended to briefly spotlight those things that you need to know about new features in BizTalk Server 2013.

A lot of people want to be able to manage BizTalk Server from the command line or from within scripts – it is enterprise level server software after all. In the search for the right tool for the job, it seems most reach for the old stand-by BtsTask utility. Some come to realize that BizTalk Server also has pretty decent WMI support (wrapped around the Microsoft.BizTalk.ExplorerOM.dll assembly). Even fewer realize that BizTalk Server 2013 provides a nice PowerShell alternative (out-of-the-box) to that WMI interface that can be used to fully control your BizTalk installation. In this blog post, we will examine how to make use of that new functionality and we will see how we might also accomplish the same things in BizTalk Server 2010.

NOTE: The focus of this week’s post is as of yet undocumented in the official product documentation 🙁

If you spend enough time rummaging through the installation directory for BizTalk Server, you will inevitably eventually locate the figurative gold mine of tools that live in the %PROGRAMFILES(X86)%Microsoft BizTalk Server 2013SDKUtilities folder:

image

One of those folders recently caught my eye, as it was absent from my BizTalk Server 2010 installations – specifically the PowerShell folder.

image

Inside, one finds that this folder contains the soafactory PowerShell Provider for BizTalk. This is actually a project that wasn’t initiated by Microsoft, has some history before BizTalk Server 2013, but has now found its way to be bundled alongside the other Utilities in the SDK – and I couldn’t be happier about that.

Getting Started

In order to get started, fire-up the readme.txt, and follow along with the instructions listed:

  1. Open a command prompt as an administrator
  2. Change the directory to the folder containing the binaries
  3. Run the following command to install/register the dlls:
    %windir%Microsoft.NETFrameworkv4.0.30319InstallUtil.exe .BizTalkFactory.PowerShell.Extensions.dll
  4. Open a powershell console as admin in x86 mode and set execution policy as remote-signed:
    Set-ExecutionPolicy -ExecutionPolicy RemoteSigned
  5. Add the PS snap-in using the usual approach:
    Add-PSSnapIn -Name BiztalkFactory.PowerShell.Extensions

What Do I Get?

Any time that you add a PowerShell Snap-in, there is actually a relatively easy way to find out what new powers (in the form of cmdlets) that it gave you. If you’re not familiar fully with PowerShell, each cmdlet returns objects – not just strings of text that spill forth from the console. As a result, we can use a cmdlet called Get-Command to get a list of commands that are available, and then filter that list based on a specific property that each item in the list may have. In this case, the property that we want to filter on is the ModuleName (to see a listing of all the properties we could filter on use Get-Member | Get-Command).

I know some of you right now are saying, “Just tell me what to type so it will tell me what it gave me already!” So, here you go (here I’m using BizTalk* in place of typing the full name of the snap-in, so that you don’t have to type as much):

Get-Command | where { $_.ModuleName -like "BizTalk*" }

As you can see, you now have quite a bit of control of your BizTalk group at your finger tips:image

So how do you know how to use each of these cmdlets, and what it does? In the days before PowerShell, you may have typed something like this to see a given command’s (an executable of some kind) help screen and usage:

commandname.exe -?

Given that cmdlets aren’t actually independent .exe files, that’s not going to do you much good. Instead, PowerShell provides you a common way to access the documentation for any cmdlet through the use of the Get-Help cmdlet. Here’s an example:

Get-Help Enable-ReceiveLocation

Unfortunately, you may notice rather quickly that these particular cmdlets are lacking in the documentation department:

image

Traversing a BizTalk Group

Another interesting thing about PowerShell is the concept of providers themselves. They have the ability to provide visibility into the worlds of specific technologies. To be more specific, consider your hard drive. We think nothing of opening a command prompt and being able to type something like dir to see into that world and retrieve a listing of files. In PowerShell, dir is actually a helpful alias for the Get-ChildItem cmdlet (which is technically, again, returning a listing of objects).

If you happen to get a listing of PowerShell providers using the Get-PSProvider command, you will find that the snap-in we added also added a new world for us to explore:

image

In order to get to the world of BizTalk and start looking around, we can do things as simple as typing and executing the following:

cd BizTalk:
dir

image

I can then continue to naturally interact with what I’m presented, and navigate into the Applications “directory”, and then into a specific application I may have installed:

cd Applications dir

cd ‘.Which Runs Fastest’

image

image

Interacting with an Item

So that’s cool that we can see a listing of items in a given scope within our farm, but how can we actually interact with it? For that, PowerShell gives us the Get-Item cmdlet. As an example, consider my installation wherein I have the Which Runs Fastest application. I can back up to BizTalk:Applications and then perform a Get-Item on that application. Likely when I do that, I want to store it in some sort of variable so that I can then interact with that. So I am going to do that with the following:

$myApp = Get-Item '.Which Runs Fastest'

From there, I can investigate both the properties and methods (operations) that I can use to interact with that application using the Get-Member cmdlet:

image

So let’s try to make use of the Orchestrations property shown to give us a listing of Orchestrations alongside their ports (some prior investigation into that property using Get-Member was done for this one):

$myApp.Orchestrations | Format-Table FullName,Ports

image

Here we told PowerShell to provide us with a table formatted list by enumerating the values returned by that property, and then for the objects returned to show us the value of the FullName and Ports properties on those objects.

The sky is the limit though. You have full and scriptable access to the world of BizTalk, and the power of PowerShell at your disposal!

Take it to the Next Level

If you’re hungry for more and want to see more about how you can use PowerShell to manage your BizTalk environments, come join us for one of our upcoming BizTalk 2013 Administrator Immersion classes and you won’t regret it 🙂