What’s New

The quickly approaching next release of NServiceBus will be version 2.0 and is a big step from 1.9. After 2 betas and 2 release candidates, this version has had a longer stabilization period than any of the versions so far (1.4-1.9). Many of my clients are already using it in production and are very pleased with it. I’ve heard similar reports from others in the community (now with over 500 members in the discussion group). There have been almost 10,000 downloads since the version 1.9 release and in every country I visit I meet people using NServiceBus in new and interesting applications.

With my appearance on Hanselminutes, many in the mainstream .NET industry have started taking a look at NServiceBus. That, and the fact that Microsoft’s Oslo technology has now taken a very data-driven turn (rather than its original service-oriented direction).

Interestingly enough, I’ve been hearing more and more reports about people using NServiceBus as a developer-friendly API on top of other technologies. This includes BizTalk and even Neuron. I never thought that people would take the pluggability of NServiceBus that far.

So, what is NServiceBus?

Well, it’s a service bus, y’know, like an ESB – just an open-source one.

All kidding aside, in a nutshell, it gives you an easy way to integrate transactional messaging into your applications.

One of the reasons why you might want to do that is so that you don’t lose messages containing valuable data when IIS recycles your AppDomain, every 15-20 minutes (as I wrote about in this MSDN magazine article).

There are many other nice things in there, like the ability to unit test your service layers and long-running processes but you can read more about that here…

Documentation

One of the biggest differences to NServiceBus in this release is documentation.

A lot of work has gone into the NServiceBus.com site to help developers hit the ground running with NServiceBus, including the more advanced aspects of transparent scale-out with the distributor and multi-site communications.

There is still work to be done in this area but feedback so far has been extremely positive (except for some grumblings from certain old-timers saying that if they could figure it out by themselves, well, you know the rest).

In Closing

If you’re building a distributed enterprise .NET system, take 5 minutes, download it, and see transactional publish/subscribe messaging working on your machine without any big heavy-weight middleware.

www.NServiceBus.com

Take a listen.

“A common complaint is that the first call on a client object takes some disproportionately large amount of time, usually ten seconds or more, while successive calls are instantaneous. There are many reasons why this might happen so there’s no generic resolution for this problem.” — Nicholas Allen

The thing is that there IS a generic solution to this problem.

It’s queued messaging.

The only thing is that you have to give up talking to your services as if they were regular objects – calling methods on them and expecting a response. In other words, designing a distributed systems isn’t like designing a regular OO system just with some WCF sprinkled on top.

Even when trying to do fire and forget messaging on top of WCF (void method calls with the OneWay attribute), the underlying channel can still block your thread, as Nick mentioned.

A queue isn’t an implementation detail.
It’s the primary architectural abstraction of a distributed system.

Disclaimer

Before giving some numbers, please keep in mind that high performance of system infrastructure does not necessarily by itself mean that the system above it is running that fast. For instance, you may have server heartbeats running really quickly but the time it takes to save a purchase order borders on a minute. So, please, take all benchmarks with a grain of salt, or two, or a whole shaker-full.

While I’m not at liberty to say on which specific domain/company these numbers were measured, I can say that we had the full gamut of “stateless services”, statefull services (sagas), number crunching, large data sets, many users, complex visualization, etc. Also, this wasn’t the largest installation of nServiceBus that I’m aware of, but its the one I have the most specific numbers for.

Setup

OK, so using the default nServiceBus distribution using MSMQ, on servers where the queue files themselves were on separate SCSI RAID disks, we were pumping around 1000 durable, transactionally processed messages per second, per server. That means that similar to the Neuron case, no messages would be lost in the case of a single fault per server per window (time to replace a failed disk set at 3 hours from failure, through detection, to replacement per site – but that’s more an operational staffing concern, not the technology itself).

So, that’s 3.6 million messages per hour per server, at full load. We had a total of 98 servers doing these kinds of processing, not including web servers, databases, etc. Keep in mind that web servers would be communicating with other servers using nServiceBus, but that would maybe be an unfair comparison to the Neuron numbers.

Server Breakdown

Anyway, the 48 number crunching servers (blade centers) we had were at full load, so we were pumping more than 170 million messages there. Keep in mind that those servers had a really fast backbone so weren’t held up by IO. Your environment may be different.

Another 30 (regular pizza boxes) were doing our sagas. Saga state was stored in a distributed in-memory “cache”, so once again IO wasn’t an issue for processing those messages. We were at about 70% utilization there, coming to just over 100 million messages an hour.

The last 20 were clustered boxes (fairly expensive) that handled the various nServiceBus distributor and timeout manager processes were at full load since they handled control messages for all the servers as well as dynamically routing the load. However, on those boxes we used much higher performance disks for the messages, since they had to feed everything else, capable of doing, on average, around 5000 messages a second. That adds up to 360 million messages an hour.

Unnecessary Durability

Later, we moved a bunch of messages that didn’t need all that durability and transactionality off the disks, pushing the total throughput over 1 billion messages an hour. That was about 100 million per hour durable, 900 million per hour non-durable. You can guess that we were left with plenty of IO to spare at that point while we weren’t yet pushing the limit of our memory.

One thing that’s important to understand is the size of the messages that didn’t require durability was less than 1MB, with most weighing in under 10KB. Also, since most of those messages were published, less state management was required around them, enabling us to further improve performance.

Summary

NServiceBus didn’t give us all that by itself. It was the result of skilled architects, developers, and operations staff working together for many iterations, deploying, monitoring, re-designing, etc. You need to understand your technology, your hardware, and your specific performance, availability, and fault-tolerance requirements if you want to get anywhere.

There’s no magic.

I didn’t see the number or kinds of servers involved in the Neuron case study so this wasn’t ever really a comparison. Nor or we talking about the same system here.

So, please, don’t base your decisions on arbitrary numbers. Spend some time setting up a scaled down version of your target architecture with all the relevant technologies and measure. Be aware that you want high performance end to end, not just of the messaging part. At times, it makes sense to actively throw away messages (of the non-durable, published kind) to help a server come online faster especially after a restart.

Thus ends the tale of another “benchmark”.

Hi Udi, I think  I see the point about the domain-driven approach but I’m wondering what my messages will look like. If it’s this:

IAppointment InsertInterview(Guid recruiterId, Guid applicantId, Guid appointmentId); OR

IRecuiter UpdateRecuiter(IRecuiter recruiter); (passing in an operation flag attached to the IRecuiter object) OR

IRecuiter UpdateRecuiter(IRecuiter recruiter); (setting a state flag on the relevant object and have the business object check the flag and behave according the state change)

Hope I’m not way off

Sean

Well, Sean, first of all – messages don’t look like functions. They’re a lot more like structures – data transfer objects. In this case, you’d probably be looking at a ScheduleInterviewMessage that had the relevant fields. It would look something like this:

   1:  using System;

   2:  using NServiceBus;

   3:  using System.Xml.Serialization;

   4:   

   5:  namespace Messages

   6:  {

   7:      [Serializable]

   8:      [Recoverable]

   9:      [TimeToBeReceived("0:01:00.000")]

  10:      public class ScheduleInterviewMessage : IMessage

  11:      {

  12:          public Guid InterviewerId;

  13:          public Guid CandidateId;

  14:          public DateTime RequestedTime;

  15:   

  16:          [XmlAnyElement]

  17:          public object extra;

  18:      }

  19:  }

Before we go on, I want to explain what we see. The “recoverable” attribute is the way we indicate to the infrastructure that these messages should not be lost in case a server fails, there are network problems, etc. In essence, it does durable, store-and-forward messaging. This will create an environment in which, in the case of network problems, these messages will be written to disk. That’s a good thing, since once connectivity comes back or the server boots back up, the messages will still be around and can be sent.

Now these messages are fairly small, so even at a relatively high load, we shouldn’t be chewing through too much of our expensive, small, high performance local disks. However, if these messages were bigger, we may fill up our disks before connectivity comes back, and we all know what happens to Windows boxes when there’s no room on the file system left:

In order to prevent our system from Denial-of-Servicing itself we need to make those messages clean themselves up. That’s what the “TimeToBeReceived” attribute is for. The amount of time that if a message had not yet been received by the other side that it will be deleted. This could be that the message even made it to the other machine, but the process handling those messages was down. You wouldn’t want to be filling the other side’s disk either causing them to crash, would you? This protects both parties.

The way to figure out how long to set is by looking at the smallest amount of durable storage you have available at your nodes, divide that by the size of the average message, and then again by the rate you need to process messages – and leave yourself at least 100% spare.

In other words, to build a robust system you not only will need to deal with lost messages, but you will be actively throwing messages away.

Finally, that last “XmlAnyElement” attribute is there for versioning. As we version our system and schema, we’ll be adding fields to the message. However, an old client may be talking to a new server, or vice versa. Since we wouldn’t want data to get lost just because of serialization. In a future post, I’ll show how to set up a message handler pipeline exactly for these issues.

Now that we’ve covered all the intricacies around messaging, we can see how the code that handles that above message looks:

   1:  using System;

   2:  using Messages;

   3:  using NServiceBus;

   4:  using NHibernate;

   5:   

   6:  namespace Server

   7:  {

   8:      public class ScheduleInterviewMessageHandler :

   9:                   BaseMessageHandler<ScheduleInterviewMessage>

  10:      {

  11:          public override void Handle(ScheduleInterviewMessage message)

  12:          {

  13:              using (ISession session = SessionFactory.OpenSession())

  14:              using (ITransaction tx = session.BeginTransaction())

  15:              {

  16:                  ICandidateInterviewer interviewer = session.Get<ICandidateInterviewer>(

  17:                          message.InterviewerId);

  18:                  ICandidate candidate = session.Get<ICandidate>(

  19:                          message.CandidateId);

  20:   

  21:                  interviewer.ScheduleInterviewWith(candidate)

  22:                          .At(message.RequestedTime);

  23:   

  24:                  tx.Commit();

  25:              }

  26:   

  27:              // publish new appointment data

  28:          }

  29:      }

  30:  }

If you’ve read this far and have more questions, please feel free to send them my way. If you’re at a more time-critical part of your project and need an answer quickly, we can set up a skype call. This has been working quite well for many of my overseas clients (shout out to the guys in Ireland and Florida).

Until next time

By the way, if you’re looking for more Q&A style info, check out the Ask Udi podcast. If you have a pressing question and need a shorter turn around time than the month or so it usually takes me for the podcast, send me an email to OnlineConsultation@UdiDahan.com.

Number 1

Ron: Hey, welcome back to ARCast Rapid Response. This is your host Ron Jacobs and today I’m looking at the MSDN architecture forum where I see this message from “theking2.” Yeah? OK, so “king,” he says, he’s building a distributed architecture that has a number of external systems. These external systems interface through a telnet connection and so they accept commands and return results as ACKS or NACKS.

Typically these systems have limited resources for the number of simultaneous sessions you can open, so, five to fifty depending on the system. What he did to get around this, was, he created some Enterprise Services objects and some pooled objects that set up these connections and then he has some Web services. The Web services are going to receive an incoming message. They’re going to call these pooled COM+ objects and they’re going to make the telnet calls to the external systems. Sounds interesting.

He says, after a year of production it has become apparent that some of the external systems are not performing very well. He says the bulk of the requests, but not all, to the external systems can be done asynchronously. So, he’s opting for a message queue-based solution using pseudosynchronous calls whenever a direct response is needed.

So, the question is, at what layer would message queuing make most sense?

So, should the clients, this Web service that receives the message — should it do a queue? Put a message in the queue and then the COM+ objects would pop off or they have some central Web services that would pop it. So, the central Web services or these Enterprise Service objects? Or maybe just a communication at the top of the telnet. He says this is the first time when he’s using message queuing.

On the line with me I have Udi Dahan, the Software Simplist from Israel.

Udi, this is a very interesting application and my first gut reaction is, does it really matter where you put the queuing?

Udi Dahan: Well, actually I took a look at it as well and I’d have to say that it does because the problem that he’s trying to solve isn’t that clear. We know that there is some sort of performance problem but we’re not quite sure where it is. We know that there are long and varying latencies in the responses but we’re not really quite sure why.

While we know that their external system is a bit slow but our choice of where to put the queue will probably have an impact, obviously on the development model of the clients and the Web services as well as how those external systems would work. So, I’d have to say that choosing the correct place to put the queue is important.

Ron: Well, let me interject something here because what you said just made me think. Now, if the problem is that these external systems are slow and limited number of connections, the first question we ought to ask is, does queuing help this situation at all?

Udi: Well, that’s probably a good first step. I mean every single time someone comes with a solution and then says, “OK, what’s the problem,” it’s always a good thing to check that solution first.

It looks like the problem that he has here has to deal with or the reason that he wants to use a queue is to do some kind of load leveling. He’s getting too many requests or at too high a rate from his clients and external Web services and external web applications more than his back end systems can use. So, using a queue as a load leveling mechanism is definitely the right way to go. So, from that perspective I think that putting a queue somewhere in there is a good idea.

Ron: OK. So then if you put a queue, it seems to me that it’s not going to make that much difference which layer you put the queue, would it?

Udi: Well, it might for the main reason that you really have to look at where his bottleneck is and that’s his back end systems. The bottleneck also has to do with the number of connections that can be opened and the number of sessions that can be opened. The place that I’d be looking at doing that is probably between those pooled COM+ objects and his central Web services for the main reason that that really gives a nice encapsulation in terms of the Web services towards both his organization’s internal services if they are other Web services, web applications or clients and everybody else that’s going on out there while keeping that abstraction out of the way.

So, the choice of using pooled COM objects is one of the ways he does the load leveling now. One of the problems he has is that it doesn’t seem to be doing that much for him because the switches and knobs that are available in COM+ in order to do that load leveling aren’t that great. What I’d be looking at in his situation is to put a queue in there but on the back side of that queue, not talking directly to the external system but doing something with WCF.

WCF has an incredible amount of switches and knobs in order to do the load throttling and the number of threads that are open. He could also do that on a large number of URIs in order to sort of split up the load from that perspective allowing him to cache results quite a bit better. So, that’s where I’d be looking at too. Just throw away those COM+ objects, put WCF in there, use the MSMQ binding and start configuring things from there.

Ron: There’s a lot of stuff in the message, but I think his core concern is performance. He mentions pseudosynchronous calls. I think by that he means, a message comes in to the web service, he’s going to drop something on the queue and then hold that message response until he gets a response back from a queue. So, it’s sort of synchronous but sort of not synchronous. So, in effect he’s kind of waiting on a queue instead of waiting on the pooled object to make this outbound telnet call.

I could agree if you said, “Well, look, our big problem is that we keep getting time outs because when we go to get a COM+ object from the pool, COM+ waits for a while and then it says, “Hey, there’s no object available’ and it returns an error,” then the queue is definitely going to help that problem. But in terms of the sheer through put or performance of the system, this is not going to help at all. It’s going to still be the same performance.

Now if you said, “Oh look, we can do some of this work kind of at a later point in time, ” well queuing doesn’t allow you to time shift the work. Right? So, if you said, “Look we can rethink this solution.” So you get a message in, we stuff something in a queue that we’ll deal with later, and then very quickly return a response like some kind of a number like, hey “your transaction number blah, blah, blah, will be processed later, it’s queued for processing, ” whatever.

I mean that introduces a lot of complexity in the system but it clearly would provide better response at the Web service layer. What do you think?

Udi: Well I think that at the most basic level, his throughput is dictated by his back end systems. From what he seems to be describing, every single request that is going through there, has to hit that back end system. If he has a limited number of back end systems that are supporting a limited number of connections, that’s going to limit his throughput no matter what technology he puts in front of that. So that’s at the core level. You just can’t get away from that.

The one thing that I would agree with you in your description there is the choice of using those COM+ objects. I mean COM+ was a great technology when it came out. The problem occurs, of course, when we start getting into larger and larger delays around the response time and we start getting all sorts of time out exceptions and things like that. So in that respect, I definitely say you know, take a step back from there.

But in terms of everything that he has around there, the queue isn’t going to make the back end system run any faster. What it will do is definitely complicate his system because he’s taking something that used to be synchronous and making it asynchronous. Writing Web services in order to handle that, I mean just adding a bunch of threads in order to listen to queues is not going to make things any simpler.

However, what it might do is to improve the resource usage of those Web services, OK? So instead of having those Web services have a bunch of threads open, waiting for the response coming back from those COM+ pooled objects, those threads could be relinquished and really just be triggered back up when a response comes back from the queue.

So I don’t see an improvement in the kind of solution that MSMQ or queuing would put in there in terms of the latency — how long it would take for a response to get back. However, I do see an improvement in terms of the resource usage of all the other players in the system.

Ron: I would agree with that. I would just say though that if you make the Web server that is hosting these Web services more resource efficient, maybe all you’re going to do is enable it to get more requests in queue the more quickly. Ultimately, this solution I think is going to solve a lot of problems related to time outs and server busy errors and that sort of thing, thread contentions, but not likely to increase overall performance.

But I definitely agree though. I would move this solution forward to WCF. I used to be on the COM+ team. COM+ was rolled into WCF so that it would have similar capabilities for pooling, instancing behavior, transactional support, those sorts of things. I would definitely move that forward into WCF.

OK! So great answer, Udi. Thank you so much for being on this ARCast Rapid Response.

Number 2

Ron: Hey this is Ron Jacobs back with another ARCast.TV Rapid Response. Today I’m joined by Udi Dahan, the Software Simplist from Israel.

Udi, I’m looking at the MSDN Architecture Forum and here’s a question from “blast.” Blast says he’s looking for where to put business rules. He’s developing a WinForm application. He uses data sets as the data layer, he says. He’s thinking about business rules and where to put them.

He says obviously, the more organized and centralized business rules are, the better. He’s tempted to put the business rules in the UI layer especially with the type data set. It makes a lot of sense there but not all rules belong on the client. He says some rules belong on the server, perhaps in a trigger.

So he’s asking where do you put your rules? How do you think about this problem, Udi?

Udi: Well, it looks like what he’s doing here is developing a two-tier client that is using WinForms and using datasets and speaking directly to the database. That in essence is part of his problem in that in terms of performance, he’d like to run more rules in the UI layer so that the user won’t be sending garbage to the database.

He also understands that because he’s building a multi-user system, there is a limited capability, in terms of concurrency, of actually having all the rules run correctly in order to make sure that everything is correct. So, his choice of an architecture, working two-tier is the main problem of why he has to fragment his business rules.

If he were to move towards a three-tier solution, that is put an application server between his smart client and the database, it would be a lot easier to put those business rules there. Now, once the business rules are out of the database, because again, we don’t have to deal with the concurrency issues once we have an application server and we’re using transactions there and we don’t have any disconnected problems, then what we can do is use those same DLLs, that same CLR code that runs the business rules, and deploy it client-side and use it there.

So, in terms of deployment, what we’d have is we’d have the same rules, both running client-side and server-side, whereas from a development perspective, we’d have them organized and centralized. That’s the way that I’d go about it.

Ron: Yeah, you know, I think conceptually I agree with you that a multi-tier solution would be a very good idea here. What I would probably think about conceptually, is breaking down rules into things that really ought to happen on the client-side. In particular, rules related to validation of data, so you know that you’ve got good and complete data before you ship it off to the server-side. Oftentimes you have to do that anyway because you have a button that shouldn’t be enabled until the data is valid, or something like that.

Udi: Absolutely.

Ron: Of course, we all know that if you have middle-tier web services, you must do validation both on the client-side and the server-side, because you must ensure that the valid data is received on the web server. So I agree with you that creating an assembly that you deploy on both sides is a good idea.

I would just expand on what you said a little bit and think about maybe on the server-side using a workflow foundation and business rules and workflow as a way to handle a lot of the heavier lifting, server-side validations and business rules that might require maybe sifting through more data or whatever kind of things, but server-side business rules that are more oriented towards business logic, and even if you have very, very data-intensive roles, then maybe some of those might even happen in the database. Don’t you agree?

Udi: Oh, absolutely. Absolutely. That’s something that I think often gets swept under the rug too much. Things like unique constraints and things like that are kinds of business rules. They protect the referential integrity and if we look at the alternatives, sometimes getting 10 million rows out of the database, in order to do some sort of unique email validation upon them, that’s just going to kill your performance.

There are certain things that it just makes sense to do them in the database, it’s just the best way to do it. The hard part, from a development perspective, is maintaining the coherence of your business rules. When you say, “OK, I want a single perspective, what are all the rules in my system?”

Even though we might try to keep it all CLR based, some of the things like unique constraints, like referential integrity, will be in the database. So, what I sometimes suggest to do is to have a separate solution, in terms of your development team, where you put all your business rules.

This includes both the SQL statements for defining your unique constraints and your referential integrity. Also put in that validation logic, your workflow that you’re going to be running server-side. If it’s AJAX controls and regular expressions that you’re going to be doing client-side in order to validate that data, absolutely make sure you have, from a development perspective, one place where you can go where you can see everything, because if you don’t do that [inaudible] can be running, and when things stop working, you won’t know how to debug it.

Ron: All right. Well, excellent answer. Udi, thank you so much.

Since you’re concerned that maybe your shipping company’s servers may be down for some kind of planned (or unplanned) maintenance just as you’re trying to fulfill orders, you use a durable messaging solution there. What happens is that messages get written to disk on your end, and later the messaging tries to transfer the messages until it succeeds. So what’s wrong with that?

Well, let’s say that the shipping company’s servers went up in smoke (true story – broken down air conditioners + poor ventilation, you get the picture). Those servers aren’t going to be coming back online any second now. So, you have all these order messages buffering on your disk. Taking into account all the data, meta-data, XML, SOAP, encryption and everything, we may get up to 1MB per message.

And now’s holiday season and your company’s selling hand over fist, hundreds of orders per second from all over the world. So that means we’re eating up 100MB of disk per second, that’s 6GB a minute, and in under an hour of our shipping company’s servers going down – so do ours.

Durable messaging – yay? We don’t want to lose those orders, right? In short, durable messaging is an important part of the solution, but it’s not the whole solution.

[Continued next time...]

If you’re impatient and just want the solution, yes, nServiceBus give you all the tools you need.

Our long time listener Bill asks:

Hi Udi,

I have a question around processing of messages in proper order. When leveraging multiple threads to process messages in a message queue, it is possible for the second message in the queue to get processed before the first – especially if the first message is considerably larger than the second. I have taken a lot of care to make sure that messages are sent in the correct order, only to find that the receiving system can process them out of order anyway.

Consider a Policy Created notification, which must come before a Policy Approved notification. If both messages are sitting in the queue when the receiving service starts up, the approval message can be processed before the creation message. How can I make sure that message ordering is respected by the receiving system? I am using WCF/MSMQ as the underlying transport by the way. The only way I have found so far is to limit the receiving service to a single thread, which is by no means desirable.

Best Regards,

Bill

Download

Download via the Dr. Dobb’s site

Or download directly here.

Additional References

• Blog post: In-Order Messaging a Myth?
• Blog post: Handling Messages out of Order

Want more?

Check out the “Ask Udi” archives.

Got a question?

Send Udi your question to answer on the show.

In Gavin Terrill’s news post on InfoQ, Big Architecture Up Front – A Case of Premature Scalaculation? he covers one of the questions so many of my clients deal with:

“How hard will it be to make it scale later?”

This is a valid question, especially for companies/products just beginning their lifecycle. When the product/web site isn’t bringing in any revenue yet, how much money should we spend on getting it ready for that future success?

The answer to that question lies in treating capacity and scalability differently (source).

What I mean by that is designing for scalability, yet separating out all technological aspects of the scaling from the core solution. That way, you can start with simple, low capacity technologies that won’t be too expensive. As you grow, upgrade that infrastructure and plug it in to your solution. Arnon’s recent post on Tier Splitting touches on a project we worked on together where we designed it in a way that we could scale down to a single process on a single machine and scale out to a server farm, all without changing the core system.

Let me take the design of nServiceBus as an example:

One primary property of scalable systems is the explicit treatment of all IO/communication. This can be seen in the one-way messaging exposed by the Bus object. There is no immediately evident way to do synchronous RPC-style request/response. This design decision is taken up front. However, the way that messages are passed around is abstracted behind an ITransport interface. You can deploy the first version of your system on MSMQ, and as load increases, switch to a more performant solution like RV or MQ, just by changing configuration. WCF does this kind of abstraction as well.

Another important element of the scalability of a system is how workflow instances are persisted. This behaviour is also abstracted behind an interface – IWorkflowPersister. Start out persisting workflows to a database. As you grow, swap that out of a replicated in-memory cache. In any case, the interaction between workflow and messaging at the logical level is set up front. All the pieces of the design are there. Up front. Helping you design your core application in a way that won’t limit your scalability in the future.

This is plain Separation-of-Concerns; code that works with your specific ESB kept out of your business logic.

Design first, scale with technology later.

[Full disclosure: I'm an editor for InfoQ]

I have a question around processing of messages in proper order.  When leveraging multiple threads to process messages in a message queue, it is possible for the second message in the queue to get processed before the first – especially if the first message is considerably larger than the second.  I have taken a lot of care to make sure that messages are sent in the correct order, only to find that the receiving system can process them out of order anyway.

Consider a Policy Created notification, which must come before a Policy Approved notification.  If both messages are sitting in the queue when the receiving service starts up, the approval message can be processed before the creation message. How can I make sure that message ordering is respected by the receiving system?  I am using WCF/MSMQ as the underlying transport by the way.  The only way I have found so far is to limit the receiving service to a single thread, which is by no means desirable.

Well, the solution is really quite simple (at first).

If you’ve received a message that you think has arrived out of order, just call:

this.bus.HandleCurrentMessageLater();

and that will put the message back at the end of the queue.

Once you start considering the fact that you don’t know when the first message is supposed to arrive, you might turn to using a workflow to handle the logic. The workflow would store the policy id, and then allow for N round-trips, before it decided that something bad had happened (like the Policy Created message getting lost), and then it could forward that to an operator, or possibly contact the first system and ask for a replay of the policy created message – or whatever automated fault resolution protocol you like.

In other words, message ordering is probably more trouble than its worth.

Here are some choice tidbits:

“For any developer who architects using SOA principles, on the basis for using NServiceBus over some other methodology or some other so-called enterprise service bus:”

NServiceBus makes it difficult to work in ways that will hurt your scalability. Since the asynchronous messaging patterns are brought to the fore, developers will, by default, avoid the temporal coupling so prevalent in most Web Service implementations. Other methodologies make so many other options just as easy to use that developers can make mistakes that will hurt their scalability and availability and only find out about those mistakes in production .

Another thing NServiceBus does differently is to totally isolate all workflow code from all technologies. This makes it easy to unit-test workflow classes, making it possible to iterate quickly on the development of key business processes. These portable .NET POJOs give developers the flexibility to host their workflows in whatever runtime they want.

“Having noticed NServiceBus is built with MSMQ in mind or as an option, when asked why this choice of direction: ”

The core of NServiceBus is not dependent on MSMQ. The extensibility points of NServiceBus allow you to plug in your own communication transport, subscription storage, and workflow persister. I’ve already implemented a transport over MSMQ, and another over WCF’s NetTCP binding. Developers can use those as-is or write their own. Many of the SOA products currently out there are much more tied to HTTP, so this is something of a break from the common practice.

I chose to use MSMQ since it is one of the two main Microsoft communications technologies that enables parties to communicate in a disconnected fashion (SQL Server Service Broker is the other). MSMQ has a much more accessible API in that it is directly available as a part of the .NET Framework whereas Service Broker currently isn’t. I consider disconnected communications a critical part of any SOA infrastructure since the Tenet of Service Autonomy does not allow us to assume that the service with which we wish to communicate is currently available.

Is there anything you’d like to know about NServiceBus? Drop me a comment below.

Thanks.

I’ve also put in a bit of effort in setting up a bunch of configuration files allowing you to try the various transports for the example code. Please go through the Readme as it contains the detailed instructions. There are currently config files setting up non-WCF MSMQ, NetTCP, Basic HTTP, and WS Dual.

If you do go the MSMQ route, the example will run the long-running workflow code. The reason the example code won’t do that for the WCF cases is that I haven’t figured out yet the best way for a service to send a message back to itself, especially for the connected bindings. Any thoughts on this would be most appreciated.

Also, I’m open to hearing what directions you think the next version should go in. I’m currently thinking about including message transformation. Either that are doing a Multicast bus over a UDP transport.

This is a non-backwards-compatible update. In order to propertly integrate with the WCF philosophy I had to change the ITransport interface for Unicast transports. The meaning of this change is that transports are now expected to do their own threading and transaction management, while the bus object now joins the context that comes from the transport.

Download here.

My question is about distributed topology.

The EAI-hub-and-spoke model is all about the central server. It’s useful sometimes, but there are a lot of reasons why I’m not gung-ho on using a hub as the center of the integration universe.

The ESB distributed model puts code on the endpoints. That code solves some of the messaging problems that apps face, so that apps don’t have to face them. It also solves some of the messaging problems that enterprises face, so that enterprises don’t have to face them. (I need both).

Those problems include simple coding and deployment model, pub-sub routing, reliable transport, simple transformation, and orchestration. I wonder which of these you can do in your tool, and which you are planning to do.

I’m also interested in management. How do you insure that the endpoints are correctly configured? Do you have a central configuration store? How do you propagate changes from the center to the messaging endpoints?

And here’s my response:

The important parts of NServiceBus that are independent of the distributed topology are the API and the connection to long-running workflow. This code is indeed on the endpoint. However, if you wanted to you could easily connect to something like BizTalk and do whatever you wanted there. This general idea though is to support the ESB distributed model since there’s no such things as a centralized ESB.

In terms of the capabilities you’ve mentioned, I’ve seen developers pick up the coding model in a day or two. The deployment model is just a bunch of DLLs you deploy with each endpoint. Dependency Injection is supported by www.SpringFramework.net but you can replace that with something else easily as another implementation of the ObjectBuilder interfaces.

Currently pub/sub routing is supported over regular point-to-point transports in a transport agnostic way. You also have the ability to have subscriptions be persisted so that even if a server restarts (and clients don’t, and can’t know about that) all the subscriptions will be remembered.

The reliable transport that is currently supported is MSMQ, with the option of defining per-message type if you want durable messaging (using the [Recoverable] attribute).

In terms of orchestration you get a nice model for long-running workflow that gets kicked off by messages decorated with the [StartsWorkflow] attribute, and messages that implement the IWorkflowMessage interface get automatically routed to the persistent workflow instance. You have the ability to change the storage of workflow instances easily as well. Workflows are simple classes which are easily unit-testable in that they expose a “void Handle(T message);” method for every message type (T) that is involved in the workflow.

I haven’t done anything in terms of simple transformation yet but am currently looking for the right place in the message processing pipeline to put it. I also haven’t done anything yet in terms of management.

What is currently being done management-wise on the projects that use it are the commercial options for managing configuration files in distributed environments coupled with the regular ability to restart windows services and IIS applications. I haven’t seen anything lacking in that solution yet.

If you have any questions, please don’t hesitate to send them my way – NServiceBus@UdiDahan.com.

First of all, there were two major forces that drove me to write NServiceBus. First of all I wanted to formalize the way Service Layer classes were written when using asynchronous messaging (regardless of Web Services). And second I wanted to formalize a communications API which supported pub/sub so that I could freely move between transports – whether they inherently supported pub/sub or not (the fact that Indigo did not falls under this).

From there it was filling in the details. Until finally I made the step to formalize the way long-running workflow connects to asynchronous messaging. That’s the third pillar as far as I’m concerned about ESBs. Using MSMQ I do enable Shared Subscriptions like Sonic does.

I try to stay away from the vendor side of ESB definitions and just deal with what capabilities such a library should provide. As such, it is inherently distributed – running on each server/client.

This update includes support for long-running workflows and includes persistent timer functionality. Workflow persistence is abstracted behind an IWorkflowPersister. The current drop includes an in-memory persister, but you could easily implement it over a database, or a distributed in-memory datagrid / space.

Other important details. To have a message kick off a workflow, just put the StartsWorkflow attribute on it. To define a message type as being involved in workflows, have it implement IWorkflowMessage instead of just IMessage. If you do these two things, the infrastructure will be able to generically dispatch messages to your workflow; you will not have to write any message handlers for your workflow, just the workflow class itself. This is done by implementing IWorkflow<T> for every message type received.

Persistent timers are activated from within your workflow like so:

this.reminder.ExpireIn(TimeSpan.FromSeconds(5), this, state);

As always, dependency injection is used – via my simple ObjectBuilder. The implementation provided delegates to Spring, but, once again, that can be easily substituted with whatever you want.

Download (509 KB) (sometimes problematic with IE, try FireFox before emailing me).

Finally, I’m leaning towards changing the name to NServiceBus. Anyone against, please leave a comment. Questions and comments are more than welcome.

Next step: implementing a WCF transport.

• You get all the source code. Everything.
• Extensible enough to work entirely without MSMQ.

Download (sometimes problematic with IE, try FireFox before emailing me).

Next steps:

• Creating a WCF transport implementation and host
• Persistent timers to support long-running workflows
• Turn into a full-blown open source project

And to wrap it up, a question to those who plan to use it:

NBus.com is taken. I’m thinking of changing the project name to something that has an available domain. Vote for what you want in the comments. Feel free to suggest whatever you want, but here are some ideas:

• NMessageBus
• NServiceBus

You get this simple yet rich API:

Currently, I have only MSMQ implementations for the transport of messages, the storage of subscription information, and for an error queue. This coming release will make it possible for you to customize each implementation. The release after that will be focusing on WCF implementations.

I’m also going to start releasing some more structured guidance on working with this infrastructure – getting started, configuration, etc. If I can this anywhere near what Jeremy has done with StructureMap or his MVP series, that would be an enormous achievement.

Also, now would be a great time to start telling me about additional things you’d like to see in the API, or other transports you’d like to see supported.

This is going to be a huge effort so any support that you can offer would be more than welcome.

Check out the ongoing discussion around the bus as well.

This implementation makes use of MSMQ so make sure you read the “readme.txt” for information on how to install it and what queues you need to define.

In order to have the “client” process send the “server” process a message, just press ‘Enter’ in its console.

Try to ignore the use of callbacks and subscriptions at first so you can hone in on the basic messaging semantics.

Questions and comments are more than welcome. Please send them to:

Messaging@UdiDahan.com.

Also, if you’re in the giving spirit and feel like donating something back, please do so. It will make it that much easier for me to continue supporting your use of this code.

Let’s take a rather simple case, but definitely a common one when we start versioning our services – that of the receipt of a malformed message. It could be that we were unable to deserialize the data or something more complicated, but at the most basic level, there’s nothing the service can do with the message. This is often expressed in terms of some sort of exception that reaches all the way down to the messaging infrastructure. If we were using the queue in a transaction, we’d see a rollback and the message return to the queue, causing the service to loop endlessly on that message. This sort of failure case is best handled by having the messaging infrastructure just move the message to an error queue.

Other kinds of failure conditions include having our DB transactions fail because they were chosen as the victim of a deadlock. Matts does a good job of describing why these things occur and how to handle and minimize their occurrence. The appropriate system level error handling is to just retry the transaction a bit later. Once again, if we were using our queues within the context of a transaction scope that included that of the database, the message would return to the queue when that deadlock exception bubbled through. This is exactly the behavior that we want in this case, although having the message go to the end of the queue instead of the front would also be just fine.

Other kinds of failure conditions we might run into include things like unique-constraint violations, attempting to perform actions on entities that no longer exist, and other activities where trying them over again probably won’t yield any different behavior. In these cases, if an exception were to cause the perpetrating message to return to the input queue we’d get exactly the wrong behavior. At the very least we would want to maybe log the exception information, but putting the message in the error queue doesn’t seem right. I mean, there was nothing wrong with the message, it’s just that the application state had moved on or that some other logic failed. One of the reasons for putting the message in the error queue is so that a person can manually fix what’s wrong with the message and send it back to the input queue for reprocessing. In these failure conditions, that hardly seems likely.

What I usually suggest for handling failures when messaging is involved goes something like this. First of all, have an error queue/topic – they’re exceedingly useful. Second, don’t automatically roll messages back. If the service wants to retry, they should do so after handling all the other messages already in the queue – unless (!) you’ve got message ordering issues like Thomas mentioned. This means having the service request the messaging infrastructure send the message back to itself. When it comes to exceptions around the logic handling the message, just write them to a log and be done with it. The service logic will probably log anything else that’s of particular interest to it. This leaves us with the malformed message scenario. In that case, the communications layer can know that the problem occurred before any service logic (like deserialization exceptions) and therefore can (rightly) pass the message along to the error queue.

Of course, for the above guidance to be valid, we need to educate service logic developers as to when it is appropriate for them to throw exceptions and when they actually have to send the message back themselves. It is often the case that by taking on certain constraints, we can greatly simplify the handling of scenarios which are difficult in the general case.

Anyway, when it comes to scaling out queues help a lot. Although not explicitly mentioned in the above post, having multiple machines feeding off of the same queue is the key to scalability and is known as the Competing Consumer pattern. The added benefit of such a design is that you get availability without any additional work, given that you have more than one consuming machine per queue.

One thing to keep in mind about the Microsoft platform today is that MSMQ does not currently support remote, transactional receives (Update MSMQ 4 released as a part of Vista / Server 2008 now supports this, yet people in the know have told me to avoid this feature). What this means is that, in the above design, you cannot make sure that if one of the servers fails while processing a message from the queue, that that message will return to the queue. For some kinds of messages this isn’t a big deal (like stock prices), but in other cases (like money transfers) this isn’t acceptable.

So, bottom line is that queues and other asynchronous transports (JMSs and topics) enable robust systems to be built using proven patterns, but be aware of any limitations of the technology and what ramifications they may have.

Resources

1. Article on Microsoft’s solution for ESBs
2. Patterns & Anti-Patterns for SOA webcast
2. Ask Udi a question on SOA
3. Subscribe to the Ask Udi podcast

Get it via the Dr. Dobb’s site here.

Or download directly here.

Want more? Go to the “Ask Udi” archives.

“Udi,

The problem we’ve encountered is how do we make it easy for our ops team to just add/remove servers from the cluster, and have the load redistribute itself properly. MSMQ tied to virtual IPs by itself doesn’t seem smart enough. We’re toying with both HW load balancing, and/or some NLB/MOM/AppCenter interactions (with programming layers to make it work.)”

Eric, the current version of MSMQ lacks the remote transactional receive capability – meaning that if your “application” is trying to perform a “Receive” operation on a queue that is located on a remote machine in a transactional context, it will not work. The next version of MSMQ is supposed to “fix” this.

Why would you want such a feature? Well, if you put a single queue on a clustered server (for availability) and had many other servers using that queue as their input queue, you’d have the ability to add servers at runtime to handle increases in load. However, since the handling of a message from a queue often requires a transaction, the current version of MSMQ won’t support this out of the box.

What you could do is have some sort of dispatcher application that would send messages to the other servers. When a server would be ready to receive a message (not under heavy load), it could send the dispatcher a message saying just that. The dispatcher would store the return address (ResponseQueue) so that when a message would arrive, it could send that message to the server.

This scenario would have each server have its own input queue and be aware of the dispatcher. The dispatcher would not have to know about the other servers directly – it would just store the return addresses of the messages it receives. The result of this behavior would cause the messages to be sent transactionally to only one server, which would, in turn, transactionally receive the message from its local queue and process it.

I hope that answers your question Eric. If you’d like more information, feel free to follow up.

Download the MSMQ implementation.

Service-Layer Separation of Concerns achieved by using Bus API.

How the Bus API handles the connection between messages and transactions

How information from one request can be used elsewhere

You’d think that with Indigo/WCF/WinCom out now, people like me who develop distributed systems would be overjoyed. Finally, a single technology stack to deal with, right?

Well, maybe if the API wasn’t so cumbersome.

I know that Indigo tries to be almost everything to all people through its pervasive extensibility model, but I’d really like to have my (rather simple) needs handled well, without having to special case a lot of the low-level mechanisms.

The thing is, I’ve already found a much better API. The reason its turned out so simple is the result of accepting a certain way of working – the SOA way. Once you give up the desire to tightly couple, the fog clears. You start to see the following simple elements combining in well defined ways…

The only data that goes between services are messages. In other words, any class that implements the IMessage marker interface. If you want XML or SOAP serialization, that class should probably be serializable.

The thing you use to send message implements the IBus interface. IBus exposes the following methods:

void Send(IMessage message);
void Send(IMessage message, CompletionCallback callback, object cookie);
void Reply(IMessage message);
void Reply(IMessage message, int errorCode);

void Start();

void LoadTypesFromAssembly(Assembly a);

** void CompletionCallback(int errorCode, object cookie);

There is one more interface to mention, and that is IMessageHandler where T : IMessage. It has a single method – void Handle(T message);

Before we get too mired down in the details, you’ll probably notice that the “Send” methods don’t take a destination parameter. That’s because there’s no way in hell that application code can/should know where a message should go. Here’s the reasoning:

A message schema, a group of classes that implement IMessage, aren’t just available. An autonomous service “published” that schema, in essence saying “this is how you speak to me”. The corrollary of that statement is that an instance of one of those classes should always be sent to an endpoint of that autonomous service. The implementation of the bus, probably with a config file, will know which endpoints are for which message type.

Another thing you’ll probably notice is that there’s no “Receive” method. What happens is this:

Since the bus implementation will deal with the underlying technology, it will know when a message has arrived. It now needs to inform the application of this. While it could expose a MessageReceived event, what we’d really like it to do is to dispatch that message to the appropriate message handler. The bus could easily reflect on the generic type parameter of the message handler in order to build a map from message types to message handler types.

When we start to think about message handlers, they really are one shot deals. There’s no reason to keep them in memory when they’re not handling messages. So, the bus would do per-message instantiation. Since the communication is based one asynchronous messages, there really is no meaning to the term “session”.

And about the messages – there really isn’t any need to decorate their fields with attributes or anything, is there? All the data is passed, always.

The “Reply” method is there so that a message handler, as a part of its logic, can send a message back to the client that sent it the request. Alternatively, it could just “Send” one of its own messages (like CustomersUpdatedMessage) to all clients subscribed.

How would we know that a client subscribed? Simple – it has a message handler for a message defined as belonging to someone else (the autonomous service). The client bus could then send its own specific subscription messages to the bus running on the service, which would then save that subscription.

I could go on, but I think that you can see that Indigo wasn’t designed to make this kind of scenario easy. Of course it’s possible to implement this on top of, and inside of Indigo, but, to put it mildly, its a bitch. The hard part is getting rid of the need to use all those ugly attributes.

The real thing that gets under my skin is having to explain to technical business types why we aren’t using Indigo either as an API or implementation. It’s just too much complexity! Look at all the examples out there on the blogosphere, did you see how much techno-crap you have to write to get something working. And the resulting application level code is mired in technologically specific details. Yuck. I barely have enough time to develop the system itself without wasting any on debugging infrastructure code just to find out I forgot yet another attribute (YAA). How’s that for a replacement for WCF?

Just to fill in the description of the API, the LoadTypesFromAssembly method scans a given assembly and loads and maps all the types that implement IMessage and IMessageHandler. The Start method tells the bus to start listening for messages – in essence spawning the appropriate number of worker threads. For client side buses, you probably only need a single worker thread. Service buses will probably have more.

The Send method that accepts the callback and cookie is used so that when the bus receives a response message (possibly identified by a correlationId field, but that’s an implementation detail), it would activate the callback passed before. The cookie is there so that the initiator can differentiate between different messages that it sent. For instance, if we had two CustomerDetailsForms open, and from each we initiated an update, when we get two responses back, we need to know which response goes with which form. If we were to pass the form object in as a cookie, which we’d get back in the callback, that would be just dandy. The errorCode passed in the reply would also be passed to the callback.

This asynchronous request/response pattern, combined with pub/sub – event-based communication is at the heart of loosely coupled systems. The API I described does a fine job of enabling it in a technology neutral fashion. Indigo does a much poorer job, in my opinion.

So, I’m afraid that I’ll have to keep using my MSMQ implementation – it seems that Indigo never really wanted to be a bus anyway. Although it does a fine job of uniting the technology stacks that were previously out there, WCF/YAA appears to have a long way to go.

BTW, my podcast on ESBs is up on Dr. Dobbs now.