How to Build A Polling Trigger API App

By Nick Hauenstein

In my first article recapping the BizTalk Summit 2015, I said I would revisit the topic of triggers for those of you wanting to build out custom API Apps that implemented either of those patterns.

After working through the official docs, and reviewing the code presented at the BizTalk Summit 2015 in anticipation of our upcoming Cloud-Based Integration Using Azure App Service class, I decided to take a little bit of a different direction.

Rather than do a write-up both here, and in hands-on-lab form within the class of writing a bunch of custom Swashbuckle operation filters for generating the appropriate metadata for a Logic App to properly consume an API App that’s trying to be a trigger, I decided to write up a library that just made it a little bit easier to create custom API Apps that are consumable from the Logic App designer.

How did I go about doing that? Well Sameer Chabungbam had mentioned in his presentation that a more elegant solution might be to use attributes for friendly names and the like. So I went in that direction, and made attributes for all of the stuff that the Logic App designer needs to have in a certain way (really anything that involved vendor extensions in the generated swagger). What do we do with all of those attributes? Well, we uh, read them in custom operation / schema filters of course! So yes, I did have to write some custom filters after all, but now you don’t have to!

Announcing QuickLearn’s T-Rex Metadata Library

PackageIconI rolled all of the code into a library that I’ve named the T-Rex Metadata Library1. The library is available as a NuGet package as well that you can add directly to your API App projects within Visual Studio 2013.

So how can we use that to make custom triggers? I’m glad you asked. Let’s get right into it.

Creating Custom Polling Triggers

The easiest kind of trigger to implement is a polling trigger. A polling trigger is polled by a Logic App at a set polling interval, and is asked for data. When the trigger has data available, it is supposed to return a 200 OK status with the data contained in the response body. When there is not data available, a 202 Accepted status should be returned with an empty response body.

You can find an example of a polling trigger over here. This polling trigger takes in a single configuration parameter named “divisor” and when polled will return data if and only if the current minute is evenly divisible by the divisor specified (kind of silly, I know).

So, how do I separate sample from reality and actually build one?

Steps for Creating a Polling Trigger

  1. Create a new project in Visual Studio using the Web Application template
  2. Choose API App (Preview) as the type of Web Application you are creating
  3. Add the TRex NuGet package to your project
  4. In the SwaggerConfig.cs file, add a using directive for TRex.Metadata
  5. In the SwaggerConfig.cs file, just after the line showing how to use the  c.SingleApiVersion add a line that reads c.ReleaseTheTRex();
  6. Add using directives for the Microsoft.Azure.AppService.ApiApps.Service, and TRex.Metadata namespaces.
    • Microsoft.Azure.AppService.ApiApps.Service provides the EventWaitPoll andEventTriggered extension methods
    • TRex.Metadata provides the T-Rex Metadata attribute, and the Trigger attribute
  7. Create an action that returns an HttpResponseMessage
  8. Decorate the action with the HttpGet attribute
  9. Decorate the action with the Metadata attribute and provide a friendly name, and description, for your polling action
  10. Decorate the action with the Trigger attribute passing the argument TriggerType.Poll to the constructor, as well as the type of model that will be sent when data is available (e.g.,typeof(MyModelClassHere))
  11. Make sure the action has a string parameter named triggerState
    • This is a value that you can populate and pass back whenever polling data is returned to the Logic App, and the Logic App will send it back to you on the next poll (e.g., to let you know that it is finished with the last item sent)
    • You do not need to decorate this parameter with any attributes. T-Rex looks for this property by name and automatically applies the correct metadata (friendly name, description, visibility, and default value)
  12. Optionally, add any other parameters that controls how it should poll (e.g., file name mask, warning temperature, target heart rate, etc…)
    • Decorate these parameters with the Metadata attribute to control their friendly names, descriptions, and visibility settings
  13. Make sure that the action returns the value generated by calling Request.EventWaitPoll when no data is available
    • You can also provide a hint to the Logic App as to a proper polling interval for the next request (if you anticipate data available at a certain time)
    • You can also provide a triggerState value that you want the Logic App to send to you on the next poll
  14. Make sure that the action returns the value generated by calling Request.EventTriggered when data is available
    • The first argument should be the data to be returned to the Logic App, followed by the newtriggerState value that you want to receive on the next poll, and optionally a new polling interval for the next request (if you anticipate data available at a certain time, or more likely that you know more data is immediately available and there isn’t a need to wait).

After you publish your API App, you should be able to use it as a trigger in a Logic App. Here’s what the sample looks like after being deployed:

Polling Trigger Sample

Next Up

In my next post, I’ll turn my focus to custom push triggers. For now though, I need to rest a little bit to get ready for 2 days of teaching followed immediately by Build 2015! It’s going to a long, yet quite fun week this week!

Until next week, take care!

1It was named this so that it would comply with the latest trends of fanciful library names, and so that I could justify naming a method ReleaseTheTRex. If you believe the name too unprofessional, let me know and I may add a TRex.Enterprise.Extensions namespace that includes a more professional sounding method name which simply calls into the former.

BizTalk Summit 2015 – London: Day 2

By Nick Hauenstein

After a relaxing 9-hour flight on a surprisingly empty plane, and a  week to re-adjust to the UTC-0700 time zone, I am finally back home in the Pacific Northwest of the United States and ready to record all of the happenings of Day 2 of the BizTalk Summit in London. This post has been delayed as I have been without my Surface Pro since arriving home — the power adapter was destroyed in transit.

If you haven’t already read codit’s write-up of day 2, I highly recommend giving that a read as well so that you can get a few different perspectives.

Hybrid Integration with BizTalk Server 2013 R2

Steef-Jan Wiggers BizTalk Summit 2015 London

The morning started off with back-to-back sessions from fellow Integration MVPs, beginning with Steef-Jan Wiggers who demonstrated how BizTalk Server 2013 R2 can be used to implement hybrid integration solutions (without relying on things that are currently in preview).

To get a flavor of his demo, you can check out his sample code here, and a wiki write-up over here. Ultimately, his session was a refreshing look at how we can deal with present challenges using presently available software.

I stole this image from the BizTalk 360 photo stream on Facebook because it looked really serious and epic, until you catch the background. It just goes to show that a good sense of humor really is something to be treasured when dealing with the insane world of integration.

From 17 Seconds to Sub-Second Processing with BizTalk Server

Johan Hedberg BizTalk Summit London 2015

Johan Hedberg took the stage next to walk through how he was able to take one integration from 17s all the way down to 0.95s, and how one might apply the same optimizations.

The optimizations that he applied were:

  • Reduce MsgBox hops (Call Orchestration vs. Send shape where possible) to provide a 7s improvement
  • Consider your Level/Layer of re-use (Avoid calling BizTalk process through external interface, and round-trip through send/receive port) to provide a 5s improvement
  • Use Caching to provide a 0.6s improvement
  • Optimize your logical flow (respond to caller as soon as you can, potentially before all work is done) to provide a 0.6s improvement
  • Consider your Host Settings (specifically reducing the polling interval from 500ms to 50ms) to provide a 1.6s improvement
  • Inline Sends (using code to send the message, rather than routing to send port) to provide a 0.3s improvement [Although, I’m personally not sure that it’s worth the cost of giving up everything a Send Port gives you most of the time, so use with caution)
  • Optimize instrumentation (find out where time is being spent). In this case, he mentioned that he discovered a database call being made where indexes weren’t being fully utilized. No sense doing a table scan where an index scan will do. Use the index Luke! This provided a 0.9s improvement
  • Optimize persistence points, providing a 0.05s improvement at this point.

I really enjoyed his talk, mainly because I’ve been through these things while beating integrations into shape to eek out the last little bit of raw performance, and it was nice to see the process all laid out and broken down in terms of numbers gained at each step.

If you want to see the talk for yourself, he gave the talk again as part of an #IntegrationMonday talk.

Sandro Pereira Annoys Tord

Sandro Pereira Annoys Tord

Sandro was up next with a list of tips for both BizTalk Server Administrators and BizTalk Server Developers. Here’s the list:

Tip #1 – Trying to annoy Tord: PowerShell!

Sandro started by weaving a tale involving famed commercial actor turned BizTalk Administrator Tord Glad Nordahl. In this tale, a developer unwittingly annoyed their BizTalk Administrator with mindless EventLog entries written out for every single event and action that occurred during normal processing. Rather than requiring code to change, or bad habits to die, the smart BizTalk Administrator used PowerShell to clean up and redirect the entries to a dedicated log. The script behind the legend is now available for download.

Tip #2 – What RosettaNet, ESB, and UDDI have in common?

They’re backed by databases that aren’t backed up by default. The second tip is then to ensure that these are included in your BizTalk backups.

Tip #3 – BizTalk MarkLog Tables

By default, every 15 minutes a string is stored in the table. Microsoft Terminator tool helps curb the growth within these tables, but requires stopping your entire environment. The solution here is a custom stored procedure to handle all of the clean-up work

Tip #4 – WCF-SAP Adapter supports 64-bit?

Yes. Just make sure that you install all the pre-reqs correctly, and you’ll be set.

Tip #5 – Take control of your environment: Tracking Data

If, as an administrator, you’re receiving MSI packages from your developers with full tracking enabled all over the place within the application, don’t fret, but instead take control of your environment by automating the tracking settings (again using PowerShell).

Tip #6,7 – Single Pipeline for Debatch/XML Validation/FF Encode

To combine tips 6 and 7, if you’re doing XML Validation, XML Debatching, FF Decoding, etc… You may not need to create a pipeline per-message type. Instead creating a generic pipeline and setting the properties in the BizTalk Server Administration console (rather than inside Visual Studio) can save you some heartache and suffering.

Tip #8 – Request-Response CBR with LOB operations

You don’t need orchestrations to use LOB Adapters (but you do need to promote BTS.Operation property for action mapping on the send side). One thing that I liked here, was that rather than creating a pipeline component that by code or configuration “hard-coded” an action in for the BTS.Operation, the solution presented guessed it from the root node of the message – pretty smart.

This is actually one of the things that I used to demo in the BizTalk Server Developer Immersion class if the question was ever posed, and one of the things that is now directly included as part of the core materials for the BizTalk Server Developer Deep Dive class. Yes, I am being shameless with those plugs, because I love our new Deep Dive class.

This is one of those things that provides a real Take the Red Pill moment when you show it to a BizTalk Developer who has been doing it the hard way for a few years.

Tip #9 – Creating custom pipeline components

Here is where my notes start to break down. I typed “custom pipeline components” but what follows are notes about custom functoid development. So either I missed a tip, or I mistyped. I will retain it as written here, and leave it up for you to determine.

So what was the tip in my notes? If you’re building out a custom functoid (see what I mean?), create custom inline functoids whenever possible. This will avoid the need to deploy a bunch of runtime artifacts, since the script itself is inlined in the map. Beautiful, and excellent tip with which to end the talk.

Tord Will Not Be Shaken… By Power BI

Tord will not be shaken by Power BI

Tord was next to the stage for his 30 minute tour of Power BI. I think he introduced it in the right way – at least I felt like it addressed where I was at with the technology. I’ve always been scared of BI, probably unnecessarily – ever since that defining moment as a young developer being exposed to the horror that was a raw, naked, MDX query. When I saw Power BI demoed for the first time, it looked amazing and absolutely incredible – with MDX queries hidden behind plain English, pretty graphs leaping off of dash boards. It looked great. Because it looked great, and computers aren’t really magic, that must mean that there is hard work and lots of technologies to learn behind the scenes, right?

That was the angle that Tord took, as even he was looking at it skeptically, thinking there was going to be a lot of stuff to learn just to get started. However, during his talk, he proved that this wasn’t the case, and showed just how easy it really can be to get up and running with Power BI.

So where’s the BizTalk tie-in? He was using Power BI against data gathered by BAM, because it really is just data in a database at the end of the day. A really beautiful way to tie it all together (and certainly more modern looking than the BAM portal).

Since I’ve bulleted pointed everything else, here’s some bullet points for you on Power BI (really selling points when it comes down to it):

  • Free access to Power BI designer
  • Easy to use web interface
  • Works on any platform
  • Loved by management
  • Like Azure Services, it’s cheap (10 USD per month, per user)

On the Origin of Microservices – Charles Young Connects Past to Present

Charles Young - The Donald Knuth of EAI

For me it’s a personal treat to be able to hear Charles Young speak, and luckily I got the opportunity at the 2015 BizTalk Summit in London. He’s the Donald Knuth of EAI, and always serves to bring the larger picture to developers (myself included) who tend to have myopic tendencies driven by the lust for shiny things.

So how would he introduce himself? “I am an EAI dinosaur. I’ve been doing it for years, and years, and years, and we either evolve, or obviously we atrophy or whatever”. Of course, we all know what whatever is. Poor dinos (except for the raptors in Jurassic Park – they had it coming).

His talk walked through the evolution from n-tier architectures of the 90’s to Alistair Cockburn’s Hexagonal Architecture (to which BizTalk Server closely aligns, and shares terminology with, though pre-dating the architecture), all the way through Microservices Architecture.

He saw as driving forces of this evolution the aspiration for:

  • Simplicity in addressing complex requirements
  • Velocity (speed of delivering solutions)
  • Evolution (see above)
  • Democratization

On Microservices, “[they are] essentially based very consciously on hexagonal architecture. It’s the idea of taking your application domain and ensuring that the way you implement services within the application domain are very fine grained […]. Microservice principles are based on the notion that the services that we tended to build in the middle tier have tended to be monolithic, which isn’t quite exactly what is being built, but in terms of how it is packaged and hosted, maybe. There’s a lot of chunky services that cover lots of concerns which can be problematic. The idea is to decompose the monolith into fine grained services […]. [It is] the new new fine grained SOA approach for a modern audience. But just because Microservices is the new buzzword doesn’t mean that we should leave our brain at the door and forget what we know about building complex integrations”

As he expanded out to iPaaS and Microservices, he mentioned that he has a problem with most of the iPaaS platforms today (e.g., MABS), where the full integration / mediation workflow is emitted up to the cloud as a monolithic thing which makes no proper separation of concerns (e.g., Itinerary containing a bridge with VETR flow). However, with App Service, he sees a change of direction in which we don’t have to use Logic Apps, for example, to use API Apps – the capabilities are no longer tightly coupled to the workflow.

I must admit, his session was so packed full of essential and incredible content, that I was unable to take notes fast enough to capture it all. He doesn’t waste a lot of words, and still succeeds to use lots of them.

If you want a better view into his brain right now, as he’s processing the change in the industry, read through his latest series of posts on architecture (Part 1, Part 2, Part 3).

Announcing The Migration Factory

The Migration Factory

Next up was Jon Fancy and Dan Probert, to completely blow my mind. At first, it seemed that their talk was going to be providing tips to migrate BizTalk solutions to Azure App Service. Instead they announced that they had created an automated service to do just that.

I thought it was a joke. I thought they were going to have a fake announcement, and then resolve it with, “of course, automated migration will never be possible, so here’s a checklist that you will need to manually go through”. But this was not the case. Indeed, they were announcing, and DEMONSTRATED a working migration service that was able to take a BizTalk Server application and migrate it over to Logic App form on Azure App Service. Good show!

I was so thrown off by this that I couldn’t get my camera up fast enough and ended up with some wonderful candid pictures of my shirt, my shoes, and the floor.

Best of all, you can sign-up today. I’m still in shock over this one.

API Management with Kent Weare and Tomasso Groenendijk

The next set of talks were a coordinated effort between Kent Weare and Tomasso Groenendijk.

Kent introduced the topic of APIs by showing their power in modern business (e.g., Uber – the largest hired ride company that owns no cars, Facebook – the largest media company that produces no content) to enable the exchange of data which is where the real value lies. He likened APIs to a doorway for your business, which with the help of API Management gets a bouncer. Why a bouncer? Because they handle similar concerns:

  • Authentication / authorization
  • Policy enforcement
  • Analytics
  • Engagement

So what’s the Red Pill moment that API Management delivers to a BizTalk Server developer who is keen on exposing BizTalk endpoints directly to end consumers with whatever format the end consumer wants? Of course, we still have to think about security, enrollment, governance, visibility, etc…

 What if I told you, you don't have to involve the entirety of IT to get this setup?

Kent followed through with a nice end to end demo of API Management being applied to a car insurance purchasing scenario with a shiny Windows Store app front-end.

Next up was Tomasso.

Tomasso showing off BizTalk 360's ESB Management Portal

I always get excited when I see Tomasso, because I know that I’m in the same room as one of the few human beings that shares the same passion for the ESB Toolkit as I. He’s also probably the only person I know who has ever implemented IExtenderStyle (so rare, it’s not even Google-able, though Bing makes an effort) – that shows that he really takes pride in his work.

He demonstrated what it looks like to put API management in front of the ESB Toolkit on-premises, which adds the benefit of message repair thanks to the ESB Exception Management Framework. For an added bonus he demonstrated the message repair being done through the BizTalk360 ESB Portal.

Power BI to replacing the BAM Portal, and BizTalk360 to replacing the ESB Portal demoed in the same day to the same audience? Yes! I’m really looking forward to seeing more modern looking front-ends for BizTalk installations in the wild as a result.

Nino Brings it Home

Nino's Evolution

Nino closed out the conference with his talk on his own personal journey as a developer that led to the thing that is currently eating up his free development time – JitGate.

So how did he get to that point? He started out riding the roller coaster of integration technologies through to the present era (shedding hair and resolve along the way), until he had a realization – what he really wanted out of integration he got from file-based integration:

  • Simple to manage
  • Fast to use
  • Polymorphic
  • Adaptable
  • Serializable
  • Fully extensible
  • Persistent
  • Multi-platform
  • Scalable
  • Reliable

From there he set out to use the tools provided by Azure to build out an integration framework that gave him those same benefits. The result was JitGate – Just-in-time integration.

So does this mean he’s announcing an App Service killer? Not necessarily. It means that Azure is an enabler. It has the power as a platform to do integration, and to handle insane loads. You don’t have to use Microsoft’s solution for doing it, you can use the same technologies that they’re using and build your own that is just as capable.

In a way, this is similar to what we had with BizTalk Server – built on top of .NET since 2004. .NET and SQL were always available to any developer who wanted to build something similar to BizTalk.

I don’t want to minimize what Nino has built here though. It is truly impressive and something that sadly, I can’t really capture here. It’s one of those things that you have to see demoed. Stay tuned to his blog for a post with full details and a video of it in action.

What’s Next?

Well it looks like Microsoft has announced Azure Service Fabric, which appears to be a re-branding of Azure Pack and promises to bring the world of Azure App Service on-premises. There are certainly exciting times ahead.

I’ll be at Build 2015 next week in San Francisco, CA to learn all that I can about that, and then the following week will be back in Kirkland, WA teaching the first run of our Cloud-Based Integration Using Azure App Service class.

I hope to see some of you along the way, and if you’re going to the BizTalk Boot Camp 2015 event, y’all better tweet everything so I can get a glimpse of it too!