Titled "BizTalk Blogs", I wasn’t quite sure what my blog was doing there. It’s not that I never write about BizTalk – every once in a while I even find something nice to say about it My quick post on BizTalk and Performance is one such example. But, let’s face it, a lot of the work I do is to provide BizTalk-like features like routing, transaction-management, and choreography (orchestration) without the actual product.

Apparently, I’m not the only non-BizTalk-only blogger there.

Including such names as Christian Weyer and Michelle Leroux Bustamante , there is a veritable who’s who in the Microsoft Connected Systems ecosystem and, quite frankly, I’m surprised the bouncer let me in the door.

So, this post is for my readers who, like me, have pretty much ignored anything looking like BizTalk for the past few years. Don’t let the name fool you. BizTalk Blogs is a valuable resource even for people who don’t care about BizTalk – and hey, you might even like what you start hearing about the future directions Microsoft is taking it.

But that’s enough of that. We’ll be back with your regularly scheduled BizTalk bashing right after this break…

My question is about distributed topology.

The EAI-hub-and-spoke model is all about the central server. It’s useful sometimes, but there are a lot of reasons why I’m not gung-ho on using a hub as the center of the integration universe.

The ESB distributed model puts code on the endpoints. That code solves some of the messaging problems that apps face, so that apps don’t have to face them. It also solves some of the messaging problems that enterprises face, so that enterprises don’t have to face them. (I need both).

Those problems include simple coding and deployment model, pub-sub routing, reliable transport, simple transformation, and orchestration. I wonder which of these you can do in your tool, and which you are planning to do.

I’m also interested in management. How do you insure that the endpoints are correctly configured? Do you have a central configuration store? How do you propagate changes from the center to the messaging endpoints?

And here’s my response:

The important parts of NServiceBus that are independent of the distributed topology are the API and the connection to long-running workflow. This code is indeed on the endpoint. However, if you wanted to you could easily connect to something like BizTalk and do whatever you wanted there. This general idea though is to support the ESB distributed model since there’s no such things as a centralized ESB.

In terms of the capabilities you’ve mentioned, I’ve seen developers pick up the coding model in a day or two. The deployment model is just a bunch of DLLs you deploy with each endpoint. Dependency Injection is supported by www.SpringFramework.net but you can replace that with something else easily as another implementation of the ObjectBuilder interfaces.

Currently pub/sub routing is supported over regular point-to-point transports in a transport agnostic way. You also have the ability to have subscriptions be persisted so that even if a server restarts (and clients don’t, and can’t know about that) all the subscriptions will be remembered.

The reliable transport that is currently supported is MSMQ, with the option of defining per-message type if you want durable messaging (using the [Recoverable] attribute).

In terms of orchestration you get a nice model for long-running workflow that gets kicked off by messages decorated with the [StartsWorkflow] attribute, and messages that implement the IWorkflowMessage interface get automatically routed to the persistent workflow instance. You have the ability to change the storage of workflow instances easily as well. Workflows are simple classes which are easily unit-testable in that they expose a “void Handle(T message);” method for every message type (T) that is involved in the workflow.

I haven’t done anything in terms of simple transformation yet but am currently looking for the right place in the message processing pipeline to put it. I also haven’t done anything yet in terms of management.

What is currently being done management-wise on the projects that use it are the commercial options for managing configuration files in distributed environments coupled with the regular ability to restart windows services and IIS applications. I haven’t seen anything lacking in that solution yet.

If you have any questions, please don’t hesitate to send them my way – NServiceBus@UdiDahan.com.

Q: Could you tell you your thoughts or preference for a distributed or centralized ESB?

DON: there is no such thing as a centralized ESB.

This is the problem with a lot of the products that call themselves ESBs. They are centralized brokers which may be clustered for availability. But they are in no way an implementation of the Bus Architectural Pattern. Please check this before cutting a check to your vendor.

Also, understand that if you do security related things in your ESB, possibly as a part of your routing rules, that if the security infrastructure is centralized that means your ESB is too. Even if it really was distributed to begin with.

Buyer beware.

I ran into another BizTalk optimization story online that once again points out that disk IO is a good first place to look, as I wrote about in my Database performance optimization article.

In closing, you need to be aware of the full environment. For instance, in BizTalk, it’s not just about messages per second. Session contention (multiple parties hitting the same session) can just lock things up tighter than, well, a tight thing. Designing for these things “up front” can save you very costly rewrites later.

How do I store some state about the current request so that I can use it later during the same service operation?

One analysis I read came at it from a technological angle – how to do this with WCF. I want to take a look at two other angles here.

The first has to do with one interpretation of the question – in the course of handling that request, there are numerous objects involved. How can we make it so that all of them have access to the request data? From a technological perspective, the answer is simple – make it thread-static (in .net this is done by applying the ThreadStaticAttribute to it), or store it in some thread-local-storage. From a design perspective, though, things aren’t all that clear. Which property of which class contains the request data, so that we can mark it with such an attribute, or under what key is the data stored in the thread-local-storage?

What I usually do is use my “IBus” interface, which exposes a “MessageBeingHandled” thread-static property. Any object that needs state about the current request makes sure to get an instance of “IBus” injected. The classic example of objects that need this data include message handlers (implementing the “IMessageHandler” interface). For more information about this design, take a look at this.

The second interpretation looks at having that request data available to the service on subsequent invocations. Personally, I don’t like the idea of having this data in-memory on the object that serviced the original request. One reason I don’t like it is that it creates an affinity between the client and the specific server handling its requests. Those of you who know me already are expecting this…

What if the server restarts?

Will that client’s state be lost? Well, not if we persisted it somewhere durable instead of just in memory. Will we stop servicing requests from that client until the original server becomes available again? Well, if the data was durably persisted, then any server could pick it up. And this is exactly what BizTalk does. You don’t want to implement BizTalk again, do you?

I can tell that some of you are surprised to hear me say this. Such a small requirement, and already we need BizTalk? Did Udi really say that?

Well, there is another, simpler way. If what you have is some kind of back-and-forth between the client and the service, you could use the Message History pattern and pack up the previous request data into the messages being sent. Although we’re increasing the message size, we’ve made it so that any server can handle any request and have access to all the previous data without creating some sort of durable contention area within the service like a database. Another option is to look at long-running workflow to model these interactions.

Finally, when it comes to ultra-scalable systems, I strongly suggest keeping the network dumb and pushing the smarts out to the edges – the clients. If you don’t need to have one client pick up where another client left off, this could be the ultimate solution. It combines with the Message History pattern and ends up sending only the data necessary on subsequent requests, thus keeping message size to a minimum. Also, your service doesn’t have to handle the state any more making it capable of handling more concurrent clients.

State management is the heart of any distributed systems development effort. Unfortunately, there aren’t any easy answers to it, but it’s important not to gloss over it if you want to have any hope of scalability in the future. Patterns help, but eventually we have to make the tradeoffs ourselves. Just don’t go running to one product or another in the hopes that it will make everything magically better.

From his post:

“The 2.0 framework doesn’t really give you a massive amount of really new really cool features. There is one-click, which we probably wouldn’t use anyway, generics (which are useful as a time saver, but don’t really do much that you can’t do without them), and some other little features… but the best part about the 2.0 framework is probably the IDE/dev experience, not what customers get out of it.”

Granted, One-Click is nothing to write home about, but the runtime level additions, they just totally changed the way I write code. I’m talking about generics, anonymous methods, delegate inference, and the rest. The rest of 2.0, like the enhancements of the provider model in ASP.Net, well you would have developed the same kind of framework yourself if you were doing serious web development.

The whole 3.0 story, I’ve got to tell you, I’m pretty underwhelmed. Everybody seems to be jumping up and down about WPF, and yes, it’s new and shiny, but there still the clunkety Windows message pump in the background. No real changes in how you’re going to write multi-threaded UIs, which seem to be the real future given the rise in multi-core processing. The visual aspects of client side code in the systems I write run at around 5% of the overall effort. So the UI will look better, I dunno, 4D buttons and stuff, sorry for not falling over with enthusiasm.

And then there’s WCF. Ah, wait, no publish/subscribe. Bummer, most of my systems being asynchronous in nature are built on the pub/sub model. An OO interface for interprocess communication? Who wants it – I need a message-based interface.

Don’t forget WF – what was that for again? The main place where WF can fit my needs is for handling long-running workflows between systems, since I don’t use Biztalk. But the performance of WF doesn’t seem to fit this environment, it seems to be more suited for human workflow times.

If anything, I’d have to say that .Net 2.0 was a relatively big deal. 3.0 will probably be just as important with the runtime level enhancements like lambda expressions, extension methods, anonymous types, and implicitly typed variables. All the rest of the hyped up stuff in 3.0, I don’t really expect it to change anything in how I work today.

This is a call to the architecture folks at Microsoft: If something is in the middle of all communication, it is a broker. It cannot be a bus, by definition. And don’t even get me started on the single point of failure issue.

I don’t have a problem with people marketing BizTalk as the replacement of 42, however, when the technical guys start talking, well, I expect them to tell the truth.

“… Finally on performance; if you’re truly seeing 1 transaction per second then something isn’t right. The SOAP adapter and heavy use of orchestration will knock your transaction rates down. Depending on what you’re doing you should be able to get that up over a hundred Tx/sec. on a well tuned BizTalk server farm.”

Isn’t one of the major selling points of BizTalk the whole webservices thing (SOAP) and the ability to express complex workflow (orchestration)? If it can’t do these things at a high transaction rate, then why do I need it? If the logic is simple, why don’t I just hand code it? I could probably get over 100 Tx/s on a single server, let alone “a well tuned … server farm”.