Category Archives: Azure

Azure, BizTalk, Blog , , , , , , 5 Comments

Exploring the Out-of-the-box Support for SFTP in BizTalk Server 2013

By Nick Hauenstein

This post is the tenth in a weekly series intended to briefly spotlight those things that you need to know about new features in BizTalk Server 2013.

This week we’re going to examine the new SFTP adapter included with BizTalk Server 2013. You read that correctly, we finally have an SFTP adapter out of the box with BizTalk Server 2013!

Background Information

In the 2010 release, we saw the addition of the FTPS adapter (or rather the enhancement of the existing FTP adapter to support SSL), which was great for those that wanted to be able to transfer files between systems in an encrypted fashion. Since the SSL support was built into the FTP adapter, we were also able to take advantage of other enhancements (e.g., read-only FTP mode) at the same time.

The SFTP adapter is designed to support a completely different scenario. Here we’re not doing regular FTP with SSL encryption; instead we’re using an SSH tunnel to access a remote file system (with a protocol that is most definitely not FTP).

As a result, SFTP functionality isn’t something that could be easily hacked onto an existing adapter. Instead, there is a brand new adapter built from the ground up for inclusion in BizTalk Server 2013.

Putting it to the Test

While I have a few Linux boxes at home of various flavors, and even a few running on EC2, I have not yet spun-up a non-windows VM on WIndows Azure (it just doesn’t feel right). Thus, I felt that this would be a fitting opportunity to do so.

Unlike with the Windows Virtual Machines, the Quick Create experience does not allow you to specify a custom user name, so the user name I will be using (along with everyone else on the service) is azureuser. As part of the provisioning process, Windows Azure will configure sshd and generate an RSA 2048-bit key pair. After provisioning, the key thumbprint will be visible in the Windows Azure Management Portal:

image

From there I fired up PuTTY to login to the VM. The only details that you will need are the host name (your-vm-name-here.cloudapp.net), the user name (azureuser), and the password that you chose when creating the VM.

From there, I used nano1 to create a quick text file for testing purposes:

image

Great. Now I have a file. Let’s see if BizTalk 2013 can now access and do something meaningful with it.

Creating a Receive Location

For the purposes of this blog post, all I really want to do is get the file into BizTalk, examine the properties that the adapter is promoting, see how it responds in read-only situations, and then route it out to a file location to get it out of my message box and placed somewhere meaningful.

To that end, I created a one-way receive location and selected the SFTP adapter. Upon configuring the adapter, I was greeted with the following properties:

image

It is interesting to see that Microsoft ditched their regular language used to refer to public key fingerprints (the “thumbprint”), and went with the more widely used terminology. Other than that, there is not a lot of surprises here for those who have worked with the plain-old FTP adapter.

A few notable things are missing here that I would have expected, though. I would have liked to see integration with SSO (i.e., an SSO Affiliate property like we have in the FTP adapter), and I would like to have seen some read-only support as well (i.e., a Delete After Download property, and an Enable Timestamp comparison property)2.

Filling in the property grid with the information from my Azure VM, I ended up with:

image

In the Security section, I changed the AcceptAnySSHServerHostKey because I wanted to be certain that it was indeed my VM that was accepting the connection (and I knew the fingerprint to expect). I set the SSHServerHostKeyFingerPrint property to the finger print listed on the Window Azure Management Portal (after adding a colon ‘:’ every two characters to put it in a more standard format).

Hitting a Wall

Well, it always happens at some point in the day – a road block that leaves me scratching my head until after a cup of coffee (hopefully it’s not just me, eh?).

In this case, the road block was an error message in the Event Log that took the following form (reproduced for those exercising Bing-fu3 to resolve a problem):

The Messaging Engine failed to add a receive location “OrdersIn_SFTP” with URL “sftp://my-server-url-is-not-your-url.cloudapp.net:22/home/azureuser/*.txt” to the adapter “SFTP”. Reason: “Microsoft.BizTalk.Adapter.SftpInvoker.SftpException: Open SFTP connection error.
   at Microsoft.BizTalk.Adapter.SftpInvoker.SftpInvoker.Open()
   at Microsoft.BizTalk.Adapter.Sftp.SftpConnection.OpenUnderlyingConnection(SftpConnectionProperties connectionProperties)
   at Microsoft.BizTalk.Adapters.CommonHelpers.Connection`3.ConnectionPool`3.GetConnection(T1 connectionProperties, TimeSpan timeout)
   at Microsoft.BizTalk.Adapters.CommonHelpers.Connection`3.GetConnection(T connectionProperties, TimeSpan timeout)
   at Microsoft.BizTalk.Adapter.Sftp.SftpRLConfig.ValidateConfiguration(SftpReceivePropertyBag receivePropertyBag)
   at Microsoft.BizTalk.Adapter.Sftp.SftpRLConfig.CreateBinding(RHConfig rhConfig)
   at Microsoft.BizTalk.Adapter.Wcf.Runtime.BtsServiceHostBase.InitializeRuntime()
   at System.ServiceModel.ServiceHostBase.OnBeginOpen()
   at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfReceiveEndpoint.Enable()
   at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfReceiveEndpoint..ctor(BizTalkEndpointContext endpointContext, IBTTransportProxy transportProxy, ControlledTermination control)
   at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfReceiver`2.AddReceiveEndpoint(String url, IPropertyBag adapterConfig, IPropertyBag bizTalkConfig)”.

Say what? An error that goes all the way down to a WcfReceiver class? This just got crazy.

After spending some time changing settings to see which configuration option might be the culprit, I found that setting the AcceptAnySSHServerHotKey to True resulted in the error disappearing and the Receive Location staying alive. Unfortunately, I actually do care about that setting being set to False – I don’t want someone to impersonate my box and get away with it!

So then, what must we do here? Well, it turns out if one will simply RTFM, they will find happiness. So I decided to do that and found this article on MSDN which indicated that I had specified the host key fingerprint in the wrong format. It must include the type of key, the size of the key (in bits) and then the fingerprint.

Azure VMs use a 2048-bit RSA key, so that will need to be reflected in that setting. After some adjustments, my properties were updated to look like this:

image

Suspending the File for Inspection

Since I really wanted to get to the bottom of what this adapter could do for me, I decided to not create a Send Port initially (but rather to just have it suspend the message so that I could inspect the context, and then decide what to do with it later – if anything).

So I enabled the receive location and waited. The file appeared in the message box within the 5 second polling interval as configured, and yielded the following look into its context:

image

From the image above, you will find that there are quite a few properties from the WCF-properties namespace. In addition, you will see that the ReceivedFileName property from both the file-properties namespace, and the new sftp-properties namespace (dated 2012) has been written to the context (but not promoted). Beyond that, there wasn’t really anything super interesting :-/.

Checking on the server side of things, the file was indeed removed as expected:

image

What about Read Only?

I decided to disable the receive location for a few minutes so that I could setup a test wherein there was a file that a user would not have permissions to delete (this also required adding another user which I affectionately named nickh):

image

After I setup a new locked down directory, with a new locked down file, preventing all but read-only access to a new locked down user, I decided to update the Receive Location properties to see how BizTalk would react to the situation.

Guess what? The file isn’t deleted even after leaving the Receive Location running for some time, and the Event Log isn’t full of errors either. However, my output directory (yes, I did finally setup a Send Port to route out these suckers), is full:

image

Now about the Event Log, it is definitely not full of errors, but it is fairly full of warnings at this point:

image

The error reads:

The adapter “SFTP” raised an error message. Details “System.Exception: Message: Delete file error.. Sftp Error Code: ‘3221488222’. Sftp Error message: ‘The system cannot find the file specified.
‘. —> Microsoft.BizTalk.Adapter.SftpInvoker.SftpException: Delete file error.
   at Microsoft.BizTalk.Adapter.SftpInvoker.SftpInvoker.DeleteRemoteFile(String filename)
   at Microsoft.BizTalk.Adapter.Sftp.SftpFileReceiver.DownloadComplete(String fileName, Boolean deleteFile)
   — End of inner exception stack trace —
   at Microsoft.BizTalk.Adapter.Sftp.SftpFileReceiver.DownloadComplete(String fileName, Boolean deleteFile)
   at Microsoft.BizTalk.Adapter.Sftp.SftpRequestContext.Reply(Message message, TimeSpan timeout)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.Reply(MessageRpc& rpc)”.

Essentially, it will fail to delete the file, but still process the contents. If I need to receive the same nightly batch (with the same file name every day), that’s not a problem – I can set my polling interval to 1 day, or even mess with service windows to help out. But if the nightly batch has a new name every night (and the others remain in the directory), my life will be somewhat painful as I work to deal with the duplicate receives.

AN ASIDE: Peeling Back the Layers

After seeing WCF in the error message I received at the beginning of this exploration, I had to do some digging around to satisfy my curiosity.

Snooping around a bit inside the BizTalk Server 2013 installation direction, you will find how it all fits together. There’s an sshlib.dll and sshmessages.dll inside the Bins32 & Bins64 directories – these appear to be unmanaged code that implements the raw connection. Alongside those, we see a managed library named Microsoft.Biztalk.Adapter.SftpInvoker.dll. Finally, at the root of the installation directory, there is a Microsoft.BizTalk.Adapter.Sftp.dll. Visual Studio’s object browser reveals that this is, indeed, like the Service Bus adapters, yet another new WCF adapter (likely using a custom binding that somehow relies on the aforementioned components):image

Unfortunately, it does not appear to publicly expose whatever binding it is using, so it is unlikely that the implementation here will be re-usable outside of BizTalk.

Bringing it to a Close

I hope this article helped you out in one way or another. Maybe you found out that there is an SFTP adapter out of the box in BizTalk Server 2013 and that was big news for you. Maybe you were curious about how it would handle read-only situations, and now you have some idea of that behavior. Or maybe you were struggling with the configuration, and something here has shed light on an issue you were encountering. Either way, best of luck to you on your integration projects.

If you want to learn more about the new features in BizTalk Server 2013, or even want to learn BizTalk Server for the first time, check out one of our BizTalk Server 2013 Developer Immersion of BizTalk Server 2013 Developer Deep Dive classes running now from your home, at your workplace, or right here in Kirkland, WA.

1I choose to be Switzerland in the great editor war – though you will find on my VM that vi is installed while emacs is not.

2Listen, I’m not going to be that guy and not tell you that you can still achieve your goal here. Check out the AfterGetActions enum in this bad boy. That adapter still exists, and still lets you do what you want to do.

3Yes, that is me shamelessly selling out. They give you a free Redbox rental after 110 rewards points. I’m a sucker for points and free movies.

Azure, BizTalk, Blog , , , 4 Comments

Publishing a WCF Service via Service Bus Relay in BizTalk Server 2013 (Part 1)

By Nick Hauenstein

This post is the seventh in a weekly series intended to briefly spotlight those things that you need to know about new features in BizTalk Server 2013.

This post will focus on a feature that is technically present in BizTalk Server 2010, but was not installed by default. This feature was a subtle enhancement to the BizTalk WCF Service Publishing Wizard – namely the ability to publish a Windows Azure Service Bus Relay Endpoint without leaving the wizard (and also without using one of the new adapters that has Relay in its title).

Even if you didn’t know that this feature existed, you will find it the first time that you attempt to publish any BizTalk artifact (schema/orchestration) as a WCF service in BizTalk Server 2013. It takes the form of the following screen within the BizTalk WCF Service Publishing Wizard, which appears immediately after choosing to publish a service (rather than simply a metadata description):

publish_sbendpoint

If you check this box and proceed through the wizard (regardless of adapter selected on the first screen), you will see the following screen at the end of the typical wizard:

publish_sbendpoint_part2

The wizard assumes that you have already gone to the Windows Azure Management Portal and created a Service Bus namespace (effectively reserving yourself a sub-domain of servicebus.windows.net, and setting up an access control namespace for claims resolution and authorization purposes). If you’ve never done that before, it can be done through the New menu as shown below:

servicebus_namespace

The next page of the wizard will require information necessary for BizTalk to authenticate with Windows Azure and prove that it is indeed the destination endpoint for the relay. This information is also obtainable only from the Windows Azure Management Portal. In order to access this information, you will need to click on your newly created namespace in the list of Service Bus namespaces, and then click the Connection Information button at the bottom of the page:

connection_info_button

This will bring up the following listing, of which you really only need to worry about the Default Issuer (Issuer Name in the wizard), and Default Key (Issuer Key in the wizard):

connection_info_listing 

Once you gather this information, you’re ready to have a copy/paste party and fill out the last page of the wizard (unless you really want to live life in hard mode, don’t enable client authentication for metadata exchange – if you feel the need to do that, you may as well pass around the raw WSDL to whomever needs it and just forget about a MEX endpoint):

publish_sbendpoint_part3

Once you’re up to this point, you can sprint to the finish by clicking Next, Create (after reviewing the wonderful WSDL and making sure it’s something that you can be proud of), and then finally Finish.

Tackling Common Issues

Outside of the typical issues one might encounter hosting a WCF Service in a BizTalk Isolated Host (e.g., App Pool Identity needs permissions to Message Box database, correct .NET Framework version needs to be selected for App Pool, Receive Location must be started), you may also encounter a new one:

image 

Again, for those using Google-fu in attempt to resolve an error, the error message reads:

Invalid element in configuration. The extension name ‘transportClientEndpointBehavior’ is not registered in the collection at system.serviceModel/extensions/behaviorExtensions

And it’s highlighting the portion of the configuration file that includes the key for Service Bus authentication.

Before getting right to the resolution, let’s recap what we’re looking at. Going through the wizard, I had selected to host a WCF-WSHttp endpoint internally in a BizTalk Isolated Host (i.e., running in an IIS App Pool rather than a BizTalk Host Instance process). I then indicated that I wanted to also expose the service through a NetTcp relay endpoint hosted externally (on Windows Azure).

My local IIS instance has been provided all the configuration information that it needs to coordinate with Windows Azure and make the relay live, but it currently doesn’t know what to do with it – which is why I have the error, and why my relay won’t be alive yet.

In reality, this all could have been avoided by reading. Specifically reading the last page of the BizTalk WCF Service Publishing Wizard, which should tell you that you need to install the AppFabric 1.0 SDK before any of the relays will work. Again, this is functionality that was technically available in the previous generation, hence the older SDK version number.

How to Know That It’s Working

If you have made it this far successfully, hitting the local service endpoint should give you a page that looks something like this:

workingservice

If you go to the Windows Azure Management Portal and dig into the namespace you created, you should see something like this:

relay_azure

It’s Not Real Until I Can Consume It

If you’re anything like me, this is an unsatisfactory ending point – the service doesn’t really exist until I can consume it from some client. To make it as fair as possible, I am hosting the service inside a fairly locked down Windows Azure Virtual Machine, and I will be consuming it from my laptop connected ultimately via a microwave connection somewhere in the shadow of Mt. Pilchuk.

To consume this beast, we need to know our namespace (e.g., unique-name.servicebus.windows.net), and the mex endpoint exposed via relay (seen in the screenshot above). From there I can construct the address I would use in the Add Service Reference dialog in Visual Studio. In this case, that URL will be: https://unique-name.servicebus.windows.net/GetItemServiceDescription/GetItemService.svc_mex

Upon consuming the service, you may notice that the WSDL importer chokes hard on the WSDL it’s finding. It tries to build up a custom binding to make the call, and it’s finding that it doesn’t really know anything about some of the binding elements required:

choked_importer

Again for those searching for this specific error, the message reads:

WsdlImporter encountered unrecognized policy assertions in ServiceDescription ‘YOUR NAMESPACE HERE’

In reality, we don’t need to build a custom binding to make this all happen. Instead, we can use the netTcpRelayBinding to make the connection. An easy way to make that happen is to install the Windows Azure Service Bus NuGet package to our project. We can start out by using the Manage NuGet Packages context menu item in Solution Explorer:

manage_nuget_packages

Then search for the Windows Azure Service Bus package, and click the Install button:

manage_nuget_packages_part2

This should update the App.config file of your application to include the following WCF extensions:

appconfig

From there, you will want to update your endpoint to reference the netTcpRelayBinding (instead of the custom binding that the WSDL importer failed to generate properly):

[sourcecode language=”xml”]
<client>
<endpoint address="sb://YOUR-NAMESPACE.servicebus.windows.net/YOUR-SERVICE-LOCATION/YOUR-SERVICE-HERE.svc"
behaviorConfiguration="sharedSecretClientCredentials" binding="netTcpRelayBinding"
contract="RelayedItemService.ItemService" name="RelayEndpoint" />
</client>
[/sourcecode]

You will also notice above that we have assigned a behaviorConfiguration — one that currently does not yet exist. So next, we will need to add an endpoint behavior (inside the system.serviceModel section of the App.config) to perform client authentication (if you don’t want to re-use the same credentials as before, make that visit over to the ACS Management Portal that the Connection Information page is begging you to do):

[sourcecode language=”xml”]
<behaviors>
<endpointBehaviors>
<behavior name="sharedSecretClientCredentials">
<transportClientEndpointBehavior>
<tokenProvider>
<sharedSecret issuerName="owner" issuerSecret="YOUR SHARED SECRET VALUE HERE" />
</tokenProvider>
</transportClientEndpointBehavior>
</behavior>
</endpointBehaviors>
</behaviors>
[/sourcecode]

This is something that has changed more recently. In fact, I couldn’t find a single place documenting what this behavior should actually look like right now. Hopefully that will change.

Invoking the Service

Once we get through all the configuration craziness, we’re ready to make quick work of calling the service with two lines of code (backed by a billion lines of configuration):

[sourcecode language=”csharp”]
YourServiceDotNetNamespace.YourServiceClient client = new YourServiceDotNetNamespace.YourServiceClient("RelayEndpoint");
client.GetItem(new YourServiceRequest() { Id = "test" });
[/sourcecode]

That’s all it takes.

Final Thoughts

I hope you found some of this helpful. Like I said, this is all technically possible in BizTalk Server 2010, but required additional installation. In BizTalk Server 2013, it is there by default and ready to go (with one tiny SDK install) out of the box.

If you enjoyed this week’s post, then stay tuned. Next week, we will be accomplishing the same tasks with a service hosted in-process in BizTalk Server 2013 using one of the new WCF Relay adapters.