Integrate 2014 – Final Thoughts

By Nick Hauenstein

The last day at Integrate 2014 started off early with Microsoft IT demonstrating the benefits of their early investment in BizTalk Services to the bottom line, and then transitioned into presentations by Microsoft Integration MVPs Michael Stephenson and Kent Weare discussing the cloud in practice, and  how to choose an integration platform respectively.

Those last two talks were especially good, and I would recommend giving them a watch once the videos are posted on the Integrate 2014 event channel on the Channel 9 site.

Integration Current/Futures q+A Panel

At this point, I’m going to stray from the style of my previous posts on Integration 2014. The reason for that is that I want to take a little bit of time to clarify some things that I have posted, as well as to correct factual errors – given that we’re all learning this stuff at the same time right now. Don’t get me wrong, I do not at all want to discount the excellent sessions for the day from Integration MVPs and partners, I just believe it more important right now to make sure that I don’t leave errors on the table and propagate misunderstanding.

It seemed like throughout the conference, the whole BizTalk team, but Guru especially, was constantly fielding questions and correcting misunderstandings about the new microservices strategy. To that end, he organized an informal ad-hoc panel of fellow team members and Microsoft representatives to put everything out on the table and to answer any question that was kicking around in the minds of attendees about all of the new stuff for the week.

I’m going to let an abbreviated transcript (the best I could manage without recording the audio) do the talking here.

Microservices is not the name of the product, it’s a way you can build Stuff

Q – We heard about microservices on Wednesday, how come you (to Kannan from MS IT) are going live with MABS, when you know that there are changes coming down the pipeline?

A – (Vivek Dali): “A lot of people walked away with microservices is the name of the product, and it’s not the name of the product, it’s an architectural pattern that creates the foundation for building BizTalk on top of. There is no product called Microservices that will replace BizTalk Services. BizTalk Services v1.0 had certain functionality, it had B2B, EAI, and there was big demand for orchestration and rules engine, and what we’re doing is adding that functionality. It does not mean that BizTalk Service v1.0 is dead and we have to re-do it. MS IT is actually one of the biggest customers of BizTalk [Services], and we’re telling them to stay in it, and we’re committing to support them as well as move them as we introduce new functionality over […]. The next step in how MABS is evolving is to a microservices architecture.”

Microsoft Is CommitteD To a Solid Core For Long-Term Cloud Integration

Q – (Michael Stephenson): I’m kind of one of the people that gives Guru quite the hard time […] About this time last year, I did a really deep dive with you on MABS 1.0 because we were considering when we would use it, what it offered, what the use cases were. At the time, I decided that it wasn’t ready for us yet […]. When we did the SDR earlier this last year, it was quite different at that time […] We were giving the team a lot of feedback on isolation and ease of deployment, and personally my opinion is that I really like the stuff shown this week, you really fielded that feedback. What I’ve seen from where we were a year ago, and from that SDR, personally I’m really pleased.

A) Don’t worry about coming around and telling us what we’re doing wrong — we do value that feedback. We will commit to come back to you as often as we can […].

(Vivek): Here’s how I think about the product: there’s a few fundamental things that we HAVE to get right, and then there’s a feature list. I’m not worried about the feature list right now, I’m worried about what we NEED to get right to last for the next ten years. Don’t worry about how we’ll take the feedback, send us your emails, we value that feedback.

BizTalk Services Isn’t Going Away, It’s Being Aligned to a Microservices Architecture

Q: I had a conversation with Guru outside, which I think is worthwhile sharing with everybody […] I was really confused at the beginning of the session as to how microservices fits in with where we are with BizTalk Server and with MABS 1.0 and where that brings us moving forward. How do the pipelines and bridges map to where we’re going. I was really excited about the workflow concept, but I couldn’t see that link between the workflow and the microservices.

A – (Guru): The flow was that you had to have a receive port, and a pipeline, and you would persist it in a message box for somebody to subscribe, and that subscriber could be a workflow and a downstream system. That was server, that continues and that has been there for 10+ years.

Then there’s a pattern of I’m receiving something, transforming, and sending it somewhere, and in services that was one entity — and we called that a bridge. It consisted of a receiving endpoint a sequence of activities and then routing it. This concept was a bridge. If you look at it as executing a sequence of activities, then what you have is a workflow.

The difference between what we were doing then and what we’re doing now is that we’re exposing the pieces of that workflow for external pieces to access. [Paraphrased]

How do we extend those workflow capabilities outside of just a BizTalk Server application? (microservices) [Paraphrased]

I’m (Nick) going to inject this slide from Sameer’s first day presentation where he compared/contrasted EAI in MABS with the microservices model for the same, as it’s incredibly relevant to demonstrating the evolution of MABS:

Sameer compares MABS with microservices architecture for the same

You Don’t Have to Wait for vNext in 2015 to Upgrade Your BizTalk Environment

Q: We’ve got a small but critical BizTalk 2006 installation that we’re upgrading now, or in the very near future. And I was wondering if we should upgrade it to 2013 R2, or should we upgrade it to the next release, and when is the next release?

A – (Guru): This is a scenario where we’re starting from 2006? I would strongly encourage you to move to 2013 R2, not just for the support lifecycle. One for lifecycle, and the other for compatibility with the latest Windows, SQL, SharePoint etc…

Then, look at what the application is doing. Is it something that needs to be on-prem, or is it something that is adaptable to the cloud architecture, or even if that application is something that could be managed in the cloud? There’s nothing that is keeping you from migrating to 2013 R2 today.

To further drive home Guru’s point here, I’m (Nick) personally going to add in a slide that was shown on the first day, showing the huge investments the BizTalk team has been making into BizTalk Server over the last few versions. Quite a few people see it as not a lot of change on the surface, but this really goes to show just how much change is really there (heck, it took me ~100 pages worth of content just to lightly touch on the changes in 2013, and I’m still working on 2013 R2):

How BizTalk Server 2006 R2 stacks up to BizTalk Server 2013 R2

MABS 1.0 is Production Ready and Already Doing Great Things, You Should Feel Confident to Use It

Q: How do we reassure our customers that are moving to cloud based integration now, and are seeing MABS now, and are seeing the same tweets about the next version? Migration tools aren’t the full answer because there’s still a cost in doing a migration, so how do we convince customers to use MABS now?

A – (Guru): MABS 1.0 primary market has been EDI because that was the first workload that we targeted. That’s something that is complete in all aspects. So if you’re looking at a customer that is looking to use MABS for EDI, then I strongly encourage that because there’s nothing that changes between using EDI in MABS and whatever future implementation we have [Heavily paraphrased]

(Vivek): Remember MS IT is one of the biggest customers, and it’s not like we’re telling them a different thing than we’re telling you […]. Joking aside, the stuff they’re running is serious stuff, and we don’t want to take a risk, and if there’s not faith in that technology, I don’t want them to take a dependency on it.

Azure Resource MAnager Isn’t The New Workflow – But the Engine that it uses Is

Q: How will the Azure Resource manage fit into this picture?

A – (Vivek): [How] Azure Resource Manager fit in? Azure Resource Manager is a product that has a purpose of installing resources on Azure. It is built on an engine that can execute actions in a long-running fashion, and wait for the results to come, it does parallels. Azure Resource Manager has a purpose and it will be its own thing, but we’re using the engine. We picked that engine because it’s already running at a massive scale and it was built thinking about how the workload will evolve eventually. It already knows how to talk to different services. We share technologies, but those are two different products.

Microservices ALM Is partially there, and is On the Radar, But Is Still A Challenge

Q: What is the ALM story?

A: Support for CI for example? The workflow is a part that we’re still trying to figure out. For the microservices technology part of it, the host that we run on already supports it. One other feedback that came was “how do I do this for an entire workflow” and we’ll go figure that out.

componentizing Early Will Pay Dividends Moving Forward

Q: (Last question) As teams continue to design to the existing platform, we understand the messaging of don’t worry about microservices quite yet. As we design systems going forward, is there a better way to do it, keeping in mind how that will fit into the microservices world? For example, componentizing things more, deciding when to use what kind of adapter. What are things that we can do to ensure a clean migration

A – (Vivek): I think there are two kinds of decisions. One are the business decisions (do we need to have it on premise, etc…) What stays on Hybrid vs what goes on cloud. We want you to make the decision based on business, we will have technology everywhere.

There are patterns that you can benefit from. I think componentizing [will be good]. There are design principles that are just common patterns that you should follow (e.g., how you take dependencies).

So that’s where we are in terms of hearing things direct from the source at this point. Certainly a lot of information to take in, but I’m really happy to see that the team building the product realizes that, and is actively working on clearing up misconceptions and clarifying the vision of for the microservices ecosystem.

Three Shout-Outs

Before I wrap this up, I want to give 3 shout outs right now in terms of the content that I more or less glossed over and/or omitted right now.

  • Stott Creations is doing great things, and I have to hand it to the HIS team for being so intimately involved in not only helping a customer, but helping a partner look good while helping that customer. In addition to that – the Informix adapter looks awesome, and I’m really digging the schema generation from the custom SQL query; that was a really nice touch.

Paul Larsen Presents Features of the BizTalk Adapter for Informix

  • Sam Vanhoutte’s session touched on a not too often discussed tension between what the cloud actually brings in terms of development challenges, and what customers are trying to get out of it. While he was presenting in terms of how Codit addresses these customer asks by dealing with the constant change and risk on their customers’ behalf, these are all still valid points in general. I think he did a great job at summing it up nicely in these two slides:

Challenges - Constant change / Multi-tenancy / Roadmap / DR Planningimage

  • Last, but certainly not least, I want to give a shout out and huge thanks to Saravana and  the BizTalk 360 team for making the event happen. Also they really took one for the team here today as well, as Richard Broida pointed out – ensuring that everyone would have time to share on a jam packed day. The execution was spot-on for a really first class event.

To Microsoft: Remember That BizTalk Server Connects Customers To The Cloud

As a final thought from the Integrate 2014 event: We’re constantly seeing Microsoft bang the drum of “Azure, azure, azure, cloud, cloud, cloud…” Personally, I love it, I fell in love with Azure in October of 2008 when Steve Marx showed up on stage at PDC and laid it all out. However, what we can’t forget, and what Microsoft needs to remember is that any customer bringing their applications to the cloud is doing integration – and Microsoft’s flagship product for doing that integration, better than any other, is BizTalk Server.

BizTalk Server is how you get customers connected to the cloud – not in a wonky disruptive way – but in a way that doesn’t necessarily require that other systems bend to either how the cloud works, or how BizTalk works.

It’s a Good Day To Be a BizTalk Dev

These are good times to be a developer, and great times to be connected into the BizTalk Community as a whole. The next year is going to open up a lot of interesting opportunities, as well as empower customers to take control of their data (wherever it lives) and make it work for them.

I’m out for now. If you were at the conference, and you want to stick around town a little bit longer, I will be teaching the BizTalk Server Developer Deep Dive class over at QuickLearn Training headquarters in Kirkland, WA this coming week. I’d love to continue the discussion there. That being said, you can always connect live to our classes from anywhere in the world as well! Winking smile

BizTalk Microservices: A Whole New World – Or Is It?

By Nick Hauenstein

NOTE: I’m still processing all of the information I took in from Day 1 at the Integrate 2014 conference here in Redmond, WA. This post represents a summary of the first day with some thoughts injected along the way.

This morning was a morning of great changes at Integrate 2014. It kicked off with Scott Guthrie presenting the keynote session without his characteristic red shirt – a strange omen indeed. He brought the latest news from the world of Azure and touted the benefits of the cloud alongside the overall strategy and roadmap.

ScottGu Blue Shirt

After his presentation concluded, Bill Staples (unfortunate owner of the bills@microsoft.com email address) took the stage and presented the new vision for BizTalk Services.

Introducing Azure BizTalk Microservices

NOTE: Since there are a lot of places linking directly to this post, I have made factual changes in light of new information found here.

Microsoft Azure BizTalk Services, still very much in its 1.0 iteration is undergoing a fundamental change. Instead of providing the idea of a bridge tied together with other bridges in an itinerary, the actual bridge stages themselves – the raw patterns – are being extracted out and exposed as Azure BizTalk Microservices aligned with a microservices style architecture.

In reality, this re-imagination of BizTalk Services won’t really be a separate Azure offering – in fact, it’s more like the BizTalk capabilities are being exposed as first class capabilities within the core Azure Platform. Every developer that leverages Azure in any way could choose to pull in (and pay for) only the specific capabilities BizTalk Microservices they need – at the same time that same developer has a framework that allows them to build their own microservices and deploy them to a platform that enables automatic scaling & load balancing, and also provides monetization opportunities.

Bill Staples Presents Azure BizTalk Microservices Microservices

The following BizTalk features were presented as candidates for implementation in the form of microservices.

  • Validation
  • Batching/Debatching
  • Format Conversion (XML, JSON, FlatFile) – i.e., Translation
  • Extract
  • Transform
  • Mediation Patterns (Request Response / One Way)
  • Business Rules
  • Trading Partner Management
  • AS2 / X12 / EDIFACT

It definitely all sounds familiar. I remember a certain talk with Tony Meleg at the helm presenting a similar concept a few years back. This time, it looks like it has legs in a big way – inasmuch as it actually exists, even if only in part – with a public preview coming in Q1 2015.

So What Are Microservices Anyway?

Microservice architecture isn’t a new thing in general. Netflix is known for a very successful implementation of the pattern. No one has commented to me about the previous link regarding Netflix’s implementation. Read it, understand it, and then you can have a prophetic voice in the future as you are able to anticipate specific challenges that can come up when using this architecture – although Microsoft’s adoption of Azure Websites as the hosting container can alleviate some of these concerns outright.  Martin Fowler says this as his introduction to the subject:

The term “Microservice Architecture” has sprung up over the last few years to describe a particular way of designing software applications as suites of independently deployable services. While there is no precise definition of this architectural style, there are certain common characteristics around organization around business capability, automated deployment, intelligence in the endpoints, and decentralized control of languages and data.

Fowler further features a sidebar that distances microservice architecture from SOA in a sort of pedantic manner – that honestly, I’m not sure adds value. There are definitely shades of SOA there, and that’s not a bad thing. It also adds value to understand the need for different types of services and to have an ontology and taxonomy for services (I’m sure my former ESB students have all read Shy’s article, since I’ve cited it to death over the years).

Yeah, But How Are They Implemented?

Right now, it looks like microservices are going to simply be code written in any language* that exposes a RESTful endpoint that provides a small capability. They will be hosted in an automatically scaled and load balanced execution container (not in the Docker sense, but instead Azure Websites rebranded) on Azure. They can further be monetized (e.g., you pay me to use my custom microservice), and tied together to form a larger workflow.

Azure BizTalk Microservices Workflow Running in the Azure Portal

Yes, I did just use the W word, but it’s NOT the WF word surprisingly. XAML has no part in the workflows of BIzTalk vFuture. Instead, we have declarative JSON workflows seemingly based on those found in Azure Resource Manager. That is, the share the same engine that Azure Resource Manager uses under the covers, because that engine was already built for cloud scale and has certain other characteristics that made it a good candidate for microservice composition and managing long running processes. They can be composed in the browser, and as shown in the capture above, they can also be monitored in the browser as they execute live.

Workflow Designer

The workflow engine calls each service along the path, records execution details, and then moves along to the next service with the data required for execution (which can include the output of any previous step):

JSON Workflows -- Check the Namespace, we have Resource Manager in play

Further, the workflow engine has the following characteristics:

  • Supports sequential or conditional control flow
  • Supports long-running workflows
  • Can start, pause, resume, or cancel workflow instances
  • Provides message assurance
  • Logs rich tracking data

I’m really keen on seeing how long-running workflow is a thing when we’re chaining RESTful calls (certainly we don’t hold the backchannel open for a month while waiting for something to happen) – but I may be missing something obvious here, since I just drank from the fire hose that is knowledge.

What does the Designer Look Like?

The designer supports the idea of pre-baked workflow templates for common integrations :

  • SurveyMonkey to Salesforce
  • Copy Dropbox files to Office 365
  • “When Facebook profile picture…”
  • Add new leads to newsletter – Salesforce + Mailchimp
  • Alert on Tweet Workflow – Twitter + Email
  • Download photos you’re tagged in – Facebook + Dropbox
  • Tweet new RSS articles
  • Twitter + Salesforce (?)

However, it also provides for custom workflows built from BizTalk Microservice activities microservices composed within a browser-based UI. It was presented as if it were going to be a tool that Business Analysts would ultimately use, but I’m not sure if that’s going to be the case up front, or even down the line.

Workflow Designer in BizTalk vFuture

These workflows will be triggered by something. Triggers shown in the UI and/or on slides included (but weren’t necessarily limited to):

  • FTP File Added
  • Any tweet
  • A lot of tweets
  • Recurring schedule
  • On-demand schedule
  • Any Facebook post
  • A lot of Facebook posts

In terms of the microservices actually seen in the UI, we saw (and likely more if those presenting were to have scrolled down):

  • Validate
  • Rules
  • Custom filter
  • Send an Email
  • SendResponse
  • SurveyMonkey
  • Custom API
  • Custom map
  • Create digest
  • Create multi-item digest
  • XML Transform
  • XML Validate
  • Flat File Decode
  • XML XPath Extract
  • Delete FTP File
  • Send To Azure Table
  • Add Salesforce leads

The tool is definitely pretty, and it was prominently featured in demos for talks throughout the day – even though quite a few pieces of functionality were shown in the form of PowerPoint Storyboards.

So How Do We Do Map EAI Concepts To This New Stuff?

image

Well, we have special entities within this world called Connectors. They are our interface to the outside world. Everything else within the world of the original MABS 1.0 and even BizTalk Server is seen as simply a capability that could be provided by a microservice.

So That’s the Cloud, What’s on-Prem?

In the future – not yet, but at some point – we will see this functionality integrated into the Azure Pack alongside all of the other Azure goodness that it already brings to your private cloud. But remember, this is all still in the very beginning stages. We’ve yet to hear much about how really critical concerns like debugging, unit testing, or even team development, differencing / branching / merging / source control in general are going to be handled in a world where you’re building declarative stuff in a browser window.

So that’s all fine and good for the future, but what about my BizTalk Server 2013 R2 stuff that I have right now? Well keep doing great things with that, because BizTalk Server isn’t going away. There’s still going to be a new major version coming every 2 years with minor versions every other year, and cumulative updates every 3 months.

image

What about my investments in Azure BizTalk Services 1.0? Well it’s not like Microsoft is going to pull the plug on all of your great work that you’re actively paying them to host. That’s monies they are still happy to take from you in exchange for providing you a great service, and they will continue to do so under SLA – it’s a beautiful thing.

Also, if you’re moving to the new way of doing things, your schemas and maps remain unchanged, they will move forward in every way. However, you will see a new web-based mapping tool (which I simply lack the energy at the moment to discuss further for this post).

However, future investment in that model is highly unlikely based on everything announced today. I’m going to let this statement stand, because it was opinion at the time I wrote it. That being said, read this post before formulating your own.

The Old New Thing

I hate to keep coming back to patterns here, but I find myself in the same place. I will soon have yet another option available within the Microsoft stack for solving integration challenges (however, this time it’s not a separate offering, it is part of the core stack). At the same time, the problems being solved are the same, and we still can apply lessons learned moving forward. Also, integration problems are presenting themselves to a larger degree in a world of devices, APIs everywhere, and widely adopted SaaS solutions.

It’s an exciting time to be a BizTalk Developer – because after today, every developer became a BizTalk Developer – it’s part of the core Azure stack, every piece of it. For those that have been around the block a few times, the wisdom is there to do the right things with it. For those who haven’t, a whole new world has just opened up.

That’s all for now. I need some sleep before day 2. 🙂

* With regards to the any language comment – that was the statement, but there was a slide at one point that called out very specifically “.NET, Java, node.js, PHP” as potential technologies there, so take it with a grain of salt. It looks like the reason for that is we’re hosting our microservices in a Azure Websites hosting container that has been bebranded.

** Still waiting for some additional clarification on this point, I will update if my understanding changes. Updated in red.