The problem is that, as software developers, we’re all too quick to accept them at face value. We don’t question the requirements – in all fairness, it was never our job to do so. We were the ones that implemented them, preferably quickly.

For example

Let’s say we get the requirement the following requirements:

1. If the order was already shipped, don’t let the user cancel the order.
2. If the order was already cancelled, don’t let the user ship the order.

The race condition here is when we have two users who are looking at the same order, which is neither cancelled nor shipped yet, and each submits a command – one to ship the order, the other to cancel it.

In these cases, the code is simple – just an if statement before performing the relevant command.

So what’s the problem

A microsecond difference in timing shouldn’t make a difference to core business behaviors. Which means that we’ve actually got here is a bug in the requirements. Users are actually dictating solutions here rather than requirements.

Let’s ask our stakeholders, “why shouldn’t we let users cancel a shipped order? I mean, the users don’t want the products.”

And the stakeholders would respond with something like, “well, we don’t want to refund the user’s money then. Or, at least, not all their money. Well, maybe if they return the products in their original packaging, *then* we could give a full refund.”

And as we drilled deeper, “when do refunds need to be given? Right away, in the same transaction?”

The stakeholders would explain, “no, refunds don’t need to be given right away.”

It turns out we were missing the concept of a refund, as well as assuming that all things needed to be processed and enforced immediately. Once we dug into the requirements, we found that there is actually plenty of time to allow both transactions to go through. We just need to add some checks during shipping’s long-running process to see if the order was cancelled, and then to cut the process short.

So is everything a long-running process then?

That’s actually a fair question – long-running processes are a lot more common than at first appears.

What we’re seeing is that cancellation is now a command that has no reason to fail – just like CQRS tells us. When this command is performed, it publishes the OrderCancelled event, which the billing service subscribes to.

Billing then starts a long-running process (a saga, in NServiceBus lingo), also listening to events from the shipping process, ultimately making a decision when a refund should be given, and for how much.

Deeper business analysis

As we discuss matters more with our business stakeholders, we hear that most orders are actually cancelled within an hour of being submitted. It is quite rare for orders to be cancelled days later.

In which case, we could look at modeling the acceptance of an order as a long-running process itself.

When a user places an order, we don’t immediately publish an event indicating the acceptance of an order, instead a saga is kicked off – which opens up a timeout for an hour later. If a cancellation command arrives during that period of time, the user gets a full refund (seeing as we didn’t charge anything since billing didn’t get the accepted event to begin with), and the saga just shuts itself down. If the timeout occurs an hour later, and the saga didn’t get a cancel command, then the order is actually accepted and the event is published.

Yes, sagas are everywhere, once you learn to see with business eyes, and no race conditions are left.

In closing

Any time you see requirements that indicate a race condition, dig deeper.

What you’re likely to find are some additional business concepts as well as the introduction of time and the creation of long-running business processes. The implementation at that point will pivot from being trivial if-statements to being richer sagas.

Keep an eye out.

The .NET Service Bus was designed to bridge internet communications using the cloud to enable a variety of devices to communicate using a WCF-remote-procedure-call type of API. NServicebus was designed to simplify the design of on-premise distributed systems using reliable messaging.

Still, people seem to want a kind of comparison, so here’s a quick one off-the-top-of-my-head:

This is not meant to be an exhaustive list, or even an accurate representation of the relative strengths and weaknesses of the technologies because, as I said, they’re designed for different purposes.

One could even plug the .NET Service Bus into NServiceBus instead of its MSMQ transport to get broad reach where required, and then switching back to the on-premise strengths of MSMQ. The pluggability of NServiceBus makes it easy to swap out almost all implementation components like the subscription storage, transport, authorization mechanisms, containers, etc.

For more information on the .NET Service Bus see the Azure AppFabric page. Juval Lowey also has a nice article on it up on MSDN magazine here.

For more information on NServiceBus see the NServiceBus site. Also take a look at the pages giving you comparisons to WCF and BizTalk.

Mistakes Made

In the previous version of NServiceBus (1.9) we over aggresively merged 3rd party libraries like Castle into the NServiceBus binaries. That in itself, wasn’t the big problem. The problem was that we didn’t know about internalizing the libraries we merged.

This resulted in all the 3rd party types being exposed when a developer used NServiceBus. For those who happened to use the same version of those libraries as that which was merged, it wasn’t a problem. Unfortunately, developers who used libraries like Castle tended to be very particular about which version they used – which resulted in version conflicts, also known as “versioning hell”.

Internalization Challenges

As we moved to NServiceBus 2.0, we were a lot more careful about which libraries were merged and made sure to internalize them as much as possible. Yes, you read that right, “as much as possible”. You can’t always internalize all the types of a given library.

For example, NServiceBus internally uses NHibernate to persist the state of long-running processes (called sagas) and we want this implementation detail to not conflict with anything that developers want to do on top of NServiceBus. The only thing is that for NHibernate to work, it needs certain types to be exposed to the configuration environment, specifically all the types in the NHibernate.Cfg.MappingSchema namespace.

Extensibility Challenges

Once you take on an external dependency, you want to configure it to “just work” so that developers that aren’t familiar with it don’t have to learn yet another library to use your framework. So far, we’ve been able to do that quite successfully with NServiceBus. The challenge comes when developers want to customize the behavior of those 3rd party libraries, wanting to call into their APIs.

You see, once you internalize those libraries and their types, developers can’t access them. This leads to all sorts of tricky extensibility problems especially for different kinds of developers. Some are happy enough to configure things from the outside using XML – like using hbm.xml files to describe how their sagas get persisted, while others really want to use Fluent NHibernate, which is fully internalized.

Finding a Balance

What we’ve currently got for NServiceBus to try to keep everyone happy is a progressive exposure model. From the developer who’s downloading NServiceBus for the first time and wants everything to just work without changing anything, our API unfolds in multiple dimensions to allow for the highest level of extensibility and pluggability. That being said, the Fluent NHibernate issue mentioned above isn’t solvable at just an API level.

In order to address the class of developer that wants full control, we’ve got a “core only” build that doesn’t merge any assemblies into it. This class of developer usually doesn’t have a problem with referencing some more assemblies as long as they retain full control of the behaviors they want.

It ain’t easy keeping everybody happy, or, at least, not unhappy.

In Closing

I would agree that more OSS frameworks should merge and internalize 3rd party libraries that don’t need to be exposed to developers – it shortens the learning curve and increases adoption. But walk this path cautiously, it’s hard striking a balance that will work all users of your framework and it takes quite some time to get it right.

And one last thing, please, PLEASE, take care of maintaining binary compatibility from one version to the next. I know it’s a pain – we’ve been doing it with NServiceBus for the past 3 years, but your users will thank you for it.

Some background

As a cohesive framework, NServiceBus makes it quite easy for developers to pick and choose which settings they want turned on and off. Being built as a loosely-coupled set of components that don’t know about each other has always kept the internal complexity low. But as the NServiceBus API has been evolving over the years, and the functionality offered has increased, some interesting challenges have popped up as the codebase has been refactored.

The challenge

The UnicastBus class has grown too large and it’s time to refactor something out. Coincidentally, users have been asking for a better “header” story for messages – the ability to specify static headers that will be appended to all messages being sent (useful for things like security tokens), as well as per message headers. So, we want to refactor all the header management out to its own component independent of the UnicastBus class.

So, here’s the issue. So far, users have specified “.UnicastBus()” as a part of the fluent code-configuration, and shouldn’t have to change that – they shouldn’t need to know that header management is now a separate component. But then how can the new component bootstrap itself into the startup, such that it gets all the dependency injection facilities of the rest of the framework? Remember that the component doesn’t know which container technology is being used (since the user can swap it out) or when the container has been set.

The solution

The only part of the framework that knows about when all DI configuration is set is the configuration component, thus it will have to be the one that invokes the new component (without knowing about it). Introduce an interface (say INeedInitialization) and scan all the types loaded looking for classes which implement that type, register them into the container, and invoke them. Have the new component implement that interface, and in its initialization have it hook into the events and/or pipelines of other parts of the system.

Other uses

One historically problematic area in NServiceBus has been people forgetting to call “.LoadMessageHandlers()”. This can now be wired in automatically by a class in the UnicastBus component via the same mechanism.

A new feature coming in the next version is the “data bus”, a component which will allow sending large quantities of data through the bus without going through the messaging pipelines. This will help people get around the 4MB limit of MSMQ and, even more importantly, the much smaller 8KB limit of Azure. We will be able to introduce the functionality transparently with the same mechanism.

As an extension point, developers can now enrich the NServiceBus framework with their own capabilities and make those available via the contrib project to the community at large. This is better than the IWantToRunAtStartup interface that was only available for those using the generic host (which excluded web apps) and gives a consistent extensibility story for all uses.

Summary

Extensibility has always been a challenge when writing object-oriented code and dependency injection techniques have helped, but sometimes you need a bit more to take things to the next level while maintaining a backwards-compatible API.

Like I said, not a ground-shaking topic but something quite necessary in creating loosely-coupled frameworks and applications. Once you know it’s there, it isn’t really a big deal. If you didn’t know to do it, you may have been contorting your codebase in all kinds of ways to try to achieve similar things.

If you want to take a look at the code, you can find the SVN repository here: https://nservicebus.svn.sourceforge.net/svnroot/nservicebus/trunk/

After I had finished speaking at QCon London last March, I sat down for a short interview with the guys from InfoQ chatting about topics from CQRS, to DDD, to NServiceBus. I’m happy to say that the interview is now online with a full (and mostly accurate) transcript as well as with an MP3 download link.

Get it here: Udi Dahan on CQRS, DDD and NServiceBus

Currently, the video is being graciously hosted by Jon Torresdal on his blog, so let’s hope that the bandwidth holds up.

Get it here.

While Mule has many features in terms of connectivity and integration, NServiceBus provides platform interop only. One could say that this is a product of the different backgrounds I and Ross come from.

On the other hand, the saga capabilities in NServiceBus for handling long-running processes are considered to be BPM functionality on the Java side of the industry, and as such, Mule does not have them.

In terms of other enterprise features like management and monitoring, Mule is more mature, but NServiceBus holds its own in terms of high availability and actually surpasses Mule with the grid and scale-out capabilities of its distributor.

Anyway, I think it’s about time each of these parts was explicitly described so that companies already invested in Java ESB tools will know what they’re getting with NServiceBus.

Until then, I hope this podcast describing the full spectrum of NServiceBus, from top level SOA services to in-the-weeds transaction management, will provide more information about what it is and why you might want to use it:

Deep Fried Bytes, episode 49 – Getting the right message about NServiceBus with Udi Dahan.

Comments most welcome.

NServiceBus 2.0 RTM is now generally available.

There were some small tweaks after the RC2 but I’m happy to say that, all in all, this was a very quiet stabilization period. Key customers have reported very high levels of satisfaction with the NServiceBus stability, scalability, and simplicity.

For example

Conduit.com has very happily doubled their user base with NServiceBus to 100 million.

#1 on American Banker’s 100 top financial tech companies, Fiserv has been quietly employing NServiceBus across many of their core services.

The Irish SaaS company Candidate Manager have been running the automated HR processes of geographically distributed giants like Hilton International on NServiceBus without missing a beat.

Community

Now with an active community of over 600 members, new users are quickly brought up to speed by the veterans and the much improved documentation on the site make adopting NServiceBus simpler than ever.

Training on NServiceBus has also ramped up nicely with the April course in Philadelphia now sold out. The next course will be in London in May via Skills Matter.

More courses are being planned in:

• Toronto, Canada Aug 9-13
• Calgary, Canada in September
• Johannesburg, South Africa in October
• Sydney, Australia in November
• Seattle WA, USA in December

Give it a try

You’ll have publish/subscribe messaging working in under 5 minutes with a simple F5 on the PubSub sample.

Get it here.

What’s New

The quickly approaching next release of NServiceBus will be version 2.0 and is a big step from 1.9. After 2 betas and 2 release candidates, this version has had a longer stabilization period than any of the versions so far (1.4-1.9). Many of my clients are already using it in production and are very pleased with it. I’ve heard similar reports from others in the community (now with over 500 members in the discussion group). There have been almost 10,000 downloads since the version 1.9 release and in every country I visit I meet people using NServiceBus in new and interesting applications.

With my appearance on Hanselminutes, many in the mainstream .NET industry have started taking a look at NServiceBus. That, and the fact that Microsoft’s Oslo technology has now taken a very data-driven turn (rather than its original service-oriented direction).

Interestingly enough, I’ve been hearing more and more reports about people using NServiceBus as a developer-friendly API on top of other technologies. This includes BizTalk and even Neuron. I never thought that people would take the pluggability of NServiceBus that far.

So, what is NServiceBus?

Well, it’s a service bus, y’know, like an ESB – just an open-source one.

All kidding aside, in a nutshell, it gives you an easy way to integrate transactional messaging into your applications.

One of the reasons why you might want to do that is so that you don’t lose messages containing valuable data when IIS recycles your AppDomain, every 15-20 minutes (as I wrote about in this MSDN magazine article).

There are many other nice things in there, like the ability to unit test your service layers and long-running processes but you can read more about that here…

Documentation

One of the biggest differences to NServiceBus in this release is documentation.

A lot of work has gone into the NServiceBus.com site to help developers hit the ground running with NServiceBus, including the more advanced aspects of transparent scale-out with the distributor and multi-site communications.

There is still work to be done in this area but feedback so far has been extremely positive (except for some grumblings from certain old-timers saying that if they could figure it out by themselves, well, you know the rest).

In Closing

If you’re building a distributed enterprise .NET system, take 5 minutes, download it, and see transactional publish/subscribe messaging working on your machine without any big heavy-weight middleware.

www.NServiceBus.com

Take a listen.

The thing is, when looking at this in light of the full software development lifecycle, there are signs that the waters run deeper than we might have originally thought.

Let’s take things one step at a time though…

What is it?

Wikipedia tells us:

“Convention over Configuration (aka Coding by convention) is a software design paradigm which seeks to decrease the number of decisions that developers need to make, gaining simplicity, but not necessarily losing flexibility. The phrase essentially means a developer only needs to specify unconventional aspects of the application.”

What this means is that frameworks built in this way have default implementations that can be swapped out if needed. So far so good.

For example…

In NServiceBus, there is an abstraction for how subscription data is stored and multiple implementations – one in-memory, another using a durable MSMQ queue, and a third which uses a database. The convention for that part of the system is that the MSMQ implementation will be used, unless something else is specified.

Developers wishing to specify a different implementation can specify the desired implementation in the container – either one that comes out of the box, or their own implementation of ISubscriptionStorage.

Things get more interesting when we consider the full lifecycle.

Lifecycle effects

When developers are in the early phases of writing a new service, they want to focus primarily on what the service does – its logic. They don’t want to muck around with MSMQ queues for storing subscriptions and would much rather use the in-memory storage.

As the service takes shape and the developers want to run the full service on their machine, possibly testing basic fault-tolerance behaviors – kill one service, see that the others get a timeout, bring the service back up, wanting it to maintain all the previous subscriptions.

Moving on from there, our developers want to take the same system they just tested on their machine and move it into a staging environment. There, they don’t want to use the MSMQ implementation for subscription storage, but rather the database implementation – as will be used in the production environment.

While it may not sound like a big deal – changing the code which specifies which implementation to use when moving from one environment to another, consider that on top of just subscription storage, there is logging (output to console, file, db?), saga persistence (in-memory, file-based DB, relational DB), and more.

It’s actually quite likely that something will get missed as we move the system between environments. Can there be a better way?

What if…

What if there was some way for the developer to express their intent to the system, and the system could change its conventions, without the developer having to change any code or configuration files?

You might compare this (in concept) to debug builds and release builds. Same code, same config, but the runtime behaves different between the two.

As I mulled over how we could capture that intent without any code or config changes, the solution that I kept coming to seemed too trivial at first, so I dismissed it. Yet, it was the simplest one that would work for console and WinForms applications, as well as windows services – command line arguments. The only thing is that I don’t think those are available for web applications.

But since we’re still in “what if” land, and I’m more thinking out loud here than providing workable solutions for tomorrow morning, let’s “what if” command line arguments worked for web apps too.

Command-Line Intent

Going back to our original scenario, when developers are working on the logic of the service, they run it using the generic NServiceBus host process, passing it the command line parameter /lite (or whatever). The host then automatically configures all the in-memory implementations.

As the system progresses, when the developer wants to run everything on their machine, they run the processes with /integration. The host then configures the appropriate implementations (MSMQ for subscription storage, SQLite for saga persistence, etc.

When the developers want to run the system in production, they could specify /production (or maybe that could be the default?), and the database backed implementations would be configured.

Imagine…

Imagine being able to move that fluidly from one environment to another. Not needing to pore over configuration files or startup script code which configures a zillion implementation details. Not needing to worry that as you moved the system to staging something would break.

Imagine short, frictionless iterations even for large scale systems.

Imagine – lifecycle-aware frameworks making all this imagination a reality.

In Closing

We’re not there yet – but we’re not that far either. The generic host we’re providing with NServiceBus 2.0 is now being extended to support exactly these scenarios.

It’s my hope that as more of us think about this challenge, we’ll come up with better solutions and more intelligent frameworks. Just as convention came to our rescue before, breaking us out of the pain of endless XML configuration, I hope this new family of lifecycle-aware frameworks will make the friction of moving a system through dev, test, staging, and production a thing of the past.

A worthy problem for us all to solve, don’t you think?

Any ideas on how to make it a reality?
Send them in – leave a comment below.

“A common complaint is that the first call on a client object takes some disproportionately large amount of time, usually ten seconds or more, while successive calls are instantaneous. There are many reasons why this might happen so there’s no generic resolution for this problem.” — Nicholas Allen

The thing is that there IS a generic solution to this problem.

It’s queued messaging.

The only thing is that you have to give up talking to your services as if they were regular objects – calling methods on them and expecting a response. In other words, designing a distributed systems isn’t like designing a regular OO system just with some WCF sprinkled on top.

Even when trying to do fire and forget messaging on top of WCF (void method calls with the OneWay attribute), the underlying channel can still block your thread, as Nick mentioned.

A queue isn’t an implementation detail.
It’s the primary architectural abstraction of a distributed system.

The common e-commerce example

We accept orders, bill the customer, and then ship them the product.

Fairly straight-forward.

Since each part of that process can be quite complex, let’s have each step be handled by a service:

Sales, Billing, and Shipping. Each of these services will publish an event when it’s done its part. Sales will publish OrderAccepted containing all the order information – order Id, customer Id, products, quantities, etc. Billing will publish CustomerBilledForOrder containing the customer Id, order Id, etc. And Shipping will publish OrderShippedToCustomer with its data.

So far, so good. EDA and SOA seem to be providing us some value.

Where’s the saga?

Well, let’s consider the behavior of the Shipping service. It shouldn’t ship the order to the customer until it has received the CustomerBilledForOrder event as well as the OrderAccepted event. In other words, Shipping needs to hold on to the state that came in the first event until the second event comes in. And this is exactly what sagas are for.

Let’s take a look at the saga code that implements this. In order to simplify the sample a bit, I’ll be omitting the product quantities.

   1:      public class ShippingSaga : Saga<ShippingSagaData>,

   2:          ISagaStartedBy<OrderAccepted>,

   3:          ISagaStartedBy<CustomerBilledForOrder>

   4:      {

   5:          public void Handle(OrderAccepted message)

   6:          {

   7:              this.Data.ProductIdsInOrder = message.ProductIdsInOrder;

   8:          }

   9:   

  10:          public void Handle(CustomerBilledForOrder message)

  11:          {

  12:               this.Bus.Send<ShipOrderToCustomer>(

  13:                  (m =>

  14:                  {

  15:                      m.CustomerId = message.CustomerId;

  16:                      m.OrderId = message.OrderId;

  17:                      m.ProductIdsInOrder = this.Data.ProductIdsInOrder;

  18:                  }

  19:                  ));

  20:   

  21:              this.MarkAsComplete();

  22:          }

  23:   

  24:          public override void Timeout(object state)

  25:          {

  26:              

  27:          }

  28:      }

First of all, this looks fairly simple and straightforward, which is good.

It’s also wrong, which is not so good.

One problem we have here is that events may arrive out of order – first CustomerBilledForOrder, and only then OrderAccepted. What would happen in the above saga in that case? Well, we wouldn’t end up shipping the products to the customer, and customers tend not to like that (for some reason).

There’s also another problem here. See if you can spot it as I go through the explanation of ISagaStartedBy<T>.

Saga start up and correlation

The “ISagaStartedBy<T>” that is implemented for both messages indicates to the infrastructure (NServiceBus) that when a message of that type arrives, if an existing saga instance cannot be found, that a new instance should be started up. Makes sense, doesn’t it? For a given order, when the OrderAccepted event arrives first, Shipping doesn’t currently have any sagas handling it, so it starts up a new one. After that, when the CustomerBilledForOrder event arrives for that same order, the event should be handled by the saga instance that handled the first event – not by a new one.

I’ll repeat the important part: “the event should be handled by the saga instance that handled the first event”.

Since the only information we stored in the saga was the list of products, how would we be able to look up that saga instance when the next event came in containing an order Id, but no saga Id?

OK, so we need to store the order Id from the first event so that when the second event comes along we’ll be able to find the saga based on that order Id. Not too complicated, but something to keep in mind.

Let’s look at the updated code:

   1:      public class ShippingSaga : Saga<ShippingSagaData>,

   2:          ISagaStartedBy<OrderAccepted>,

   3:          ISagaStartedBy<CustomerBilledForOrder>

   4:      {

   5:          public void Handle(CustomerBilledForOrder message)

   6:          {

   7:              this.Data.CustomerHasBeenBilled = true;

   8:   

   9:              this.Data.CustomerId = message.CustomerId;

  10:              this.Data.OrderId = message.OrderId;

  11:   

  12:              this.CompleteIfPossible();

  13:          }

  14:   

  15:          public void Handle(OrderAccepted message)

  16:          {

  17:              this.Data.ProductIdsInOrder = message.ProductIdsInOrder;

  18:   

  19:              this.Data.CustomerId = message.CustomerId;

  20:              this.Data.OrderId = message.OrderId;

  21:   

  22:              this.CompleteIfPossible();

  23:          }

  24:   

  25:          private void CompleteIfPossible()

  26:          {

  27:              if (this.Data.ProductIdsInOrder != null && this.Data.CustomerHasBeenBilled)

  28:              {

  29:                  this.Bus.Send<ShipOrderToCustomer>(

  30:                     (m =>

  31:                     {

  32:                         m.CustomerId = this.Data.CustomerId;

  33:                         m.OrderId = this.Data.OrderId;

  34:                         m.ProductIdsInOrder = this.Data.ProductIdsInOrder;

  35:                     }

  36:                     ));

  37:                  this.MarkAsComplete();

  38:              }

  39:          }

  40:      }

And that brings us to…

Saga persistence

We already saw why Shipping needs to be able to look up its internal sagas using data from the events, but what that means is that simple blob-type persistence of those sagas is out. NServiceBus comes with an NHibernate-based saga persister for exactly this reason, though any persistence mechanism which allows you to query on something other than saga Id would work just as well.

Let’s take a quick look at the saga data that we’ll be storing and see how simple it is:

   1:      public class ShippingSagaData : ISagaEntity

   2:      {

   3:          public virtual Guid Id { get; set; }

   4:          public virtual string Originator { get; set; }

   5:          public virtual Guid OrderId { get; set; }

   6:          public virtual Guid CustomerId { get; set; }

   7:          public virtual List<Guid> ProductIdsInOrder { get; set; }

   8:          public virtual bool CustomerHasBeenBilled { get; set; }

   9:      }

You might have noticed the “Originator” property in there and wondered what it is for. First of all, the ISagaEntity interface requires the two properties Id and Originator. Originator is used to store the return address of the message that started the saga. Id is for what you think it’s for. In this saga, we don’t need to send any messages back to whoever started the saga, but in many others we do. In those cases, we’ll often be handling a message from some other endpoint when we want to possibly report some status back to the client that started the process. By storing that client’s address the first time, we can then “ReplyToOriginator” at any point in the process.

The manufacturing sample that comes with NServiceBus shows how this works.

Saga Lookup

Earlier, we saw the need to search for sagas based on order Id. The way to hook into the infrastructure and perform these lookups is by implementing “IFindSagas<T>.Using<M>” where T is the type of the saga data and M is the type of message. In our example, doing this using NHibernate would look like this:

   1:      public class ShippingSagaFinder : 

   2:          IFindSagas<ShippingSagaData>.Using<OrderAccepted>,

   3:          IFindSagas<ShippingSagaData>.Using<CustomerBilledForOrder>

   4:      {

   5:          public ShippingSagaData FindBy(CustomerBilledForOrder message)

   6:          {

   7:              return FindBy(message.OrderId)

   8:          }

   9:   

  10:          public ShippingSagaData FindBy(OrderAccepted message)

  11:          {

  12:              return FindBy(message.OrderId)

  13:          }

  14:   

  15:          private ShippingSagaData FindBy(Guid orderId)

  16:          {

  17:              return sessionFactory.GetCurrentSession().CreateCriteria(typeof(ShippingSagaData))

  18:                  .Add(Expression.Eq("OrderId", orderId))

  19:                  .UniqueResult<ShippingSagaData>();

  20:          }

  21:   

  22:          private ISessionFactory sessionFactory;

  23:   

  24:          public virtual ISessionFactory SessionFactory

  25:          {

  26:              get { return sessionFactory; }

  27:              set { sessionFactory = value; }

  28:          }

  29:      }

For a performance boost, we’d probably index our saga data by order Id.

On concurrency

Another important note is that for this saga, if both messages were handled in parallel on different machines, the saga could get stuck. The persistence mechanism here needs to prevent this. When using NHibernate over a database with the appropriate isolation level (Repeatable Read – the default in NServiceBus), this “just works”. If/When implementing your own saga persistence mechanism, it is important to understand the kind of concurrency your business logic can live with.

Take a look at Ayende’s example for mobile phone billing to get a feeling for what that’s like.

Summary

In almost any event-driven architecture, you’ll have services correlating multiple events in order to make decisions. The saga pattern is a great fit there, and not at all difficult to implement. You do need to take into account that events may arrive out of order and implement the saga logic accordingly, but it’s really not that big a deal. Do take the time to think through what data will need to be stored in order for the saga to be fault-tolerant, as well as a persistence mechanism that will allow you to look up that data based on event data.

If you feel like giving this approach a try, but don’t have an environment handy for this, download NServiceBus and take a look at the samples. It’s really quick and easy to get set up.

Very little has changed, so the version 1.9 story described here is still accurate.

Just last week one of my clients went live with a rollout to one of the world’s biggest names in the hospitality industry and things are looking good. Since stability is such a big deal to them (and many of my other clients), they’ve rolled out on nServiceBus 1.8 but now are ready to make the move to 1.9. Hopefully we’ll be able to get a case study out of them

For more information, go to the NServiceBus site.

What is it?

From the NServiceBus.com site:

“NServiceBus is a powerful, yet lightweight, open source messaging framework
for designing distributed .NET enterprise systems. Entirely pluggable yet simple to use, NServiceBus gives programmers a head-start on developing robust, scalable,
and maintainable service-layers and long-running business processes.”

One of the developers who downloaded nServiceBus, Jürgen, sent me this in an email:

“I took the samples, swapped in my own code, and had a machines subscribing, publishing, messaging, in like 15 minutes.

I’ve got to tell you – from reading the architecture stuff on your blog I always thought this stuff was going to be hard. I know you always say its supposed to be simple but I never really believed it. I mean, seriously, if I had to do this stuff with web services or WCF – well, I wouldn’t know where to begin.”

(His English totally surprised me – not what you’d expect from someone called Jürgen. Born in Sweden but grew up in the US.)

BTW, Sam’s showed where to begin with WCF: basic pub/sub without durability is 480 LOC.

What’s different – Containers

People who looked at earlier versions of nServiceBus will see many incremental improvements that smooth over previously rough parts of the framework.

Chris Patterson, co-founder of MassTransit mentioned one such area in his blog post:

“Both Dru and I needed a framework for asynchronous messaging to address some work-related application requirements. While MSMQ is provided out of the box [on Windows], it doesn’t directly encourage some good distributed application practices such a loose coupling. Our goal was to abstract the messaging aspects so the services could be built to deal with plain old objects (POCOs) instead of lower level transport messages.

Originally, we both looked at NServiceBus as a way to make this happen. I’ve followed Udi’s blog for a while and have really gained a lot of knowledge from his posts and presentations. However, our lack of experience in Spring.NET, along with a general lack of understanding of all the complexity of such a framework led us down the path of building our own framework.”

Version 1.9 takes a totally different approach (than v1.6/1.7 at which I believe Chris was looking back then) to dependency injection frameworks (like Spring.NET) and decreases their footprint. Developers don’t need to know anything about Spring, Castle, or any other container to get started, but always have the ability to configure it however they want and even swap in their container of choice.

What’s different – DLL Footprint

Another difference is the number of assemblies that come with nServiceBus. Nathan Stults had this to say about version 1.8:

“NServiceBus (for valid architectural reasons) is split up into eleven thousand, two hundred and nine different DLL’s. That may be an exaggeration, but many of the dll’s have only one or two code files in them, with only a few lines of code each.”

NServiceBus has always supported swapping out various technological implementations and will continue do so. For that reason, the development environment is organized into 79 projects whose dependencies are managed very strictly. As of version 1.9, most of these assemblies including many supporting libraries like Spring and Castle have been merged into NServiceBus.dll. There’s also NServiceBus.Testing.dll which supports fluent-unit-testing entire business processes.

There is one non-optional external dependency which hasn’t been merged and that is log4net – although if you configure in a Common.Logging provider to your own logging infrastructure, you can do without it as well. With log4net, the minimum deployment footprint is 2 assemblies. The other dependency which is optional is NHibernate. The reason for leaving these out is that many teams depend on specific versions of those assemblies.

In short – you reference ONE assembly. Just one.

What else?

There is quite a lot in there. Ayende’s put out several posts describing those features and similarities to the bus he’s working on:

• Managing Timeouts
• Long-running processes
• Load Balancing

Ayende’s description of the NServiceBus load-balancing capability was:

“The distributor section of NServiceBus is a thing of beauty.”

Performance

You may be surprised by the kind of performance we’re able to wring out of “basic” MSMQ. Keep in mind, though, that you can swap in your own transport – there are already some others out there on the Contrib site including ActiveMQ and shared memory.

The most thorough performance numbers I’ve seen on nServiceBus have been written up here:

“total throughput over 1 billion messages an hour. That was about 100 million per hour durable, 900 million per hour non-durable”

This was on an 3 blade centers (48 blades), 30 pizza boxes (1U), and 20 clusters, version 1.8.

On Gojko’s blog there is a video of talk on nServiceBus where Dave de Florinier mentioned throughputs of 600,000 messages per hour (no mention of supporting hardware) and I think it was version 1.7 or 1.8 of nServiceBus.

Recently, Raymond Lewallen posted his numbers for 1.9 to the discussion group:

“We load tested our full duplex and pub/sub scenarios the other day at about 500 messages per second with no hiccups at all.” (save yourself the math, its 1.8M/hr)

The really interesting numbers are coming in from one of my clients (on 1.9) where the focus is on business process (saga) throughput where they’re seeing millions completing each day on top of IO intensive technical processes. I’m excited.

Looking forward

After working with clients using nServiceBus over 4 years now (even before it was open source or had a real name), it just keeps getting better. It’s also really great to see more open source projects coming on the scene – MassTransit (now a year old), and Rhino Service Bus (young, but in production with NH-Prof).

The mutual stealing cross-pollination is increasing the pace all around.

I intend to take up TopShelf (which came out of MassTransit) as the generic service host. That combined with the robust distributor hosting model will make “grid-friendly” nServiceBus endpoints much easier. Ayende’s posts about automatically creating queues and other “first-time-developer” features have really worked to decrease the nServiceBus learning curve – and it shows.

Documentation is a notorious problem in open-source projects and nServiceBus hasn’t escaped it. API and internal documentation is only now getting close to 100% and the samples now give developers a good start on using it. Those developers looking at swapping out certain bits of functionality have a harder time, but that’s slowly improving as well. The discussion group is the best place to get those hard-to-find answers now with a community over 200 strong.

The difficulty developers have in adopting nServiceBus is giving up “request/response” thinking. This requires an adjustment to a system’s architecture and is what most of this blog has been about. Conversely, if you have been following this blog and this thinking resonates with you, you’ll find it very simple and straight-forward. The overview page on NServiceBus.com also gives a good description of messaging basics – stuff that I’ve kind of glossed over on this blog so far.

Go on then. Take it for a spin. Write a review. Sam Gentile had this to say about v1.8:

“The bottom line is: I like what I see. Although it’s a framework, not an ESB product like Neuron, it’s a powerful framework that takes the right approach on SOA and enforces a paradigm of reliable one-way, *non-blocking* calls.”

Can’t wait to hear the response to 1.9.

In regular client-server development, the server is responsible for providing the client with all CRUD (create, read, update, and delete) capabilities. However, when users look at data they do not often require it to be up to date to the second (given that they often look at the same screen for several seconds to minutes at a time). As such, retrieving data from the same table as that being used for highly consistent transaction processing creates contention resulting in poor performance for all CRUD actions under higher load.

A Scalable Solution

One of the common answers to this question is for the server/service to publish a message when data changes (say, as the result of processing a message) and for clients to subscribe to these messages. When such a notification arrives at a client, the client would cache the data it needs. Then, when the user wants to see a grid of data, that data is already on the client. Of course, this solution doesn’t work so well for older client machines (like some point of service devices) or if there are millions of rows of data.

The thing is that this solution is one implementation of a more general pattern – command query separation (CQS).

Command Query Separation

Wikipedia describes CQS as a pattern where "… every method should either be a command that performs an action, or a query that returns data to the caller, but not both. More formally, methods should return a value only if they are referentially transparent and hence possess no side effects."

Martin Fowler is less strict about the use of CQS allowing for exceptions: "Popping a stack is a good example of a modifier that modifies state. Meyer correctly says that you can avoid having this method, but it is a useful idiom. So I prefer to follow this principle when I can, but I’m prepared to break it to get my pop."

So, how does separating commands from queries and SOA help at all in getting data to and from a UI? The answer is based on Pat Helland’s thinking as described in his article Data on the Inside vs. Data on the Outside.

Services Cross Boxes

The biggest lie around SOA is that services run.

Let that sink in a second.

Sure services have runnable components, but that’s not why they’re important.

I’ll skip the books of background and cut to the chase:

Services communicate with each other using publish/subscribe and one-way messaging. Services have components inside them. Inside a service, these components can communicate with each using synchronous RPC, or any other mechanism. Also, these components can reside on different machines.

This is broader than just scaling out a service. There can be service components running on the client as well as the server.

SOA & CQS

Combining these two concepts together, here’s what comes out:

In this solution there are two services that span both client and server – one in charge of commands (create, update, delete), the other in charge of queries (read). These services communicate only via messages – one cannot access the database of the other.

The command service publishes messages about changes to data, to which the query service subscribes. When the query service receives such notifications, it saves the data in its own data store which may well have a different schema (optimized for queries like a star schema).

The client component which is in charge of showing grids of data to the user behaves the same as it would in a regular layered/tiered architecture, using synchronous blocking request/response to get its data – SOA doesn’t change that.

Composite Applications

Although the client side components of both the command and query services are hosted in the same process, they are very much independent of each other. That being said, from an interoperability perspective (the one that most people attribute to SOA), all of the client-side components will likely be developed using the same technology – although there are already ways to host Java code in .NET and vice-versa.

Of course, once we talk about web UI’s things are a bit different – but still similar. While web-server-side there may be a level of independence, for browser side inter-component communications we’re still likely to target javascript. There, I’ve managed to say something technical supporting mashups and SOA without lying through my teeth.

On the Microsoft side with the recent release of the Composite Application Guidance & Library (pronounced "prism") I hope that more of these principles will be reaching the "smart client". The command pattern is especially critical in maintaining the separation while enabling communication to still occur so I’m glad that, as one of the Prism advisors, I was able to simplify that part (Glenn still has nightmares about that rooftop conversation).

Publish / Subscribe

In the "scalable solution" section up top I mentioned how publish/subscribe to the smart client is really just one implementation of CQS and SOA. So, how different is it really?

Well, there will probably be a different technology mapping. Instead of a star-schema OLAP product, we might simply store the published data in memory on the client. That is, if you designed your components to be technology agnostic.

In terms of the use of nServiceBus, the same component is going to be subscribing to the same type of message – all that’s different is that now every client will be having data pushed to them rather than this occurring server-side only.

You could have the same code deployed differently in the same system – stronger clients subscribing themselves, weaker ones using a remote server. Web servers would probably be considered stronger clients. This kind of flexible deployment has proven to be extremely valuable for my larger clients. The added benefit of enabling users to work (view data) even while offline (somewhere there’s no WIFI) is just icing on the cake.

A Word of Warning

Once the client starts receiving notifications, and handling those on a background thread (as it should) the code becomes susceptible to deadlocks and data races. Juval does a good job of outlining some of those with respect to the use of WCF. Prism doesn’t provide any assurances in this area either.

Summary

NServiceBus is not designed to be used for any and all types of communication in a given architecture. In the examples above, nServiceBus handles the publish/subscribe but leaves the synchronous RPC to existing solutions like WCF. Not only that, but synchronous RPC does have its place in architecture, just not across service boundaries. In all cases, data is served to users from a store different from that which transaction processing logic uses.

Command Query Separation is not only a good idea at the method/class level but has advantages at the SOA/System level as well – yet another good idea from 20 years ago that services build upon. Making use of CQS requires understanding your data and its uses – SOA builds on that by looking into data volatility and the freshness business requirements around it.

Finally, designing the components of your services in such a way that their dependency on technology is limited buys a lot of flexibility in terms of deployment and, consequently, significant performance and scalability gains.

Simple, it is. Easy, it is not.

Reliability

Locking in SqlWorkflowPersistenceService (via Ron Jacobs) where, if you want predictable persistence (MS: ‘none of our customers asked for this to be easy’), you need to use a custom activity (which Ron was kind enough to supply).

“Given what I learned today I’d have to say that I’d be very careful about using workflows with an optimistic locking.  Detecting these types of situations is not that simple.”

Let’s think about that. If we’re doing pessimistic locking, we get into the problem of, if a host restarts (as the result of a critical windows patch or some other unexpected occurrence), that the workflow won’t be able to be handled by any other host in the meantime (you didn’t care so much about your SLA, did you?).

Luckily, someone’s come up with a hack that works around this robustness problem in Scalable Workflow Persistence and Ownership.

“So this code will attempt to load workflow instances with expired locks every second. Is it a hack? Yes. But without one of two things in the SqlWorkflowPersistenceService its the sort of code you have to write to pick up unlocked workflow instances robustly.”

This will seriously churn the table used to store your workflows, decreasing performance of workflows that haven’t timed out. Oh well.

Testability

Implementing WCF Services without Referencing WCF (via Mark Seemann):

“More than a year ago, I wrote my first post on unit testing WCF services. One of my points back then was that you have to be careful that the service implementation doesn’t use any of the services provided by the WCF runtime environment (if you want to keep the service testable). As soon as you invoke something like OperationContext.Current, your code is not going to work in a unit testing scenario, but only when hosted by WCF.”

After pointing out some of the more basic difficulties in testability a straightforward WCF implementation brings, Mark turns the heat up in his follow-up post, Modifying Behavior of WCF-Free Service Implementations:

“Perhaps you need to control the service’s ConcurrencyMode, or perhaps you need to set UseSynchronizationContext. These options are typically controlled by the ServiceBehaviorAttribute. You may also want to provide an IInstanceProvider via a custom attribute that implements IContractBehavior. However, you can’t set these attributes on the service implementation itself, since it mustn’t have a reference to System.ServiceModel.”

Wow – all the things required to make a WCF service scalable and thread-safe make it difficult to test. In the end, we’re beginning to see how many hoops we have to go through in order to get separation of concerns, but until we can take all this and get it out of our application code, it’s an untenable solution. I hope Mark will continue with this series, if only so I can take the framework that might grow out of it and use it as a generic WCF transport for NServiceBus.

Comparison

After the Neuron-NServiceBus comparison that Sam and I had, we talked some more. After going through some of the rational and thinking, Sam even put nServiceBus into his WCF-Neuron comparison talk. Sam had this to say about nServiceBus:

“The bottom line is: I like what I see. Although it’s a framework, not an ESB product like Neuron, it’s a powerful framework that takes the right approach on SOA and enforces a paradigm of reliable one-way, *non-blocking* calls. That is the point of the talk tonight overall; we need to get away from the stack world of synchronous RPC calls to true asynchronous non-blocking message based SOA systems.”

The main concern I have with a WCF+WF based solution is that developers need to know a lot in order to make it testable, scalable, and robust. In nServiceBus, that’s baked into the design. It would be extremely difficult for a developer writing application logic to interfere with when persistence needs to happen, or the concurrency strategy of long-running workflows. The fact that message handlers in the service layer don’t need concurrency modes, instance providers, or any of that junk make them testable by default.

Think about that a second.

Scary, isn’t it?

At TechEd Israel I had a discussion with Manu on this very issue, just under a different hat:

What’s the difference between a short-running workflow and a long-running one?

Manu suggested that we look at the actual time that things ran to differentiate between them. I asserted that if any external communication was involved in some part of state-management logic, that logic should automatically be treated as long-running.

Manu’s reasoning was that the complexity involved in writing long-running workflows was not justified for things that ran quickly, even if there was communication involved. Many developers don’t think twice about synchronously calling some web services in the middle of their database transaction logic. In the many Microsoft presentations I’ve been at on WF, not once has it been mentioned that state machines should be used when external communication is involved.

The problem that I have with this guidance is how do you know how quickly a remote call will return?

Do you just run it all locally on your machine, measure, and if it doesn’t take more than a second or so, then you’re OK?

The fact of the matter is that we can never know what the response time of a remote call will be. Maybe the remote machine is down. Maybe the remote process is down. Maybe someone changed the firewall settings and now we’re doing 10KB/s instead of 10MB/s. Maybe the local service is down and we’re communicating with the backup on the other side of the Pacific Ocean.

But the thing is, Manu’s right.

Writing long-running workflows (with WF) is more complex than is justified. My guess is that since WF wasn’t specifically designed for long-running workflows only, that this complexity crept in.

Sagas in nServiceBus were specifically designed for long-running workflows only.

Maybe that’s what kept them simple.

Since all external communication is done via one-way, non-blocking messaging only, each step of a saga runs as quick as if no communication were done at all. This keeps the time the transaction in charge of handling a message is open as short as possible. That, in turn, leads to the database being able to support more concurrent users.

In short, sagas are both more scalable and more robust.

No need to worry about garbaging-up your database.

I’ll be giving my Avoid a Failed SOA talk (again). Apparently, I’m not the only one who’s been seeing one SOA project fail after another. Luckily, I’ve lived through enough of them to figure out what sort of things empirically have lead to failure, and now I’m telling the tale. One of the big reasons, by the way, is calling everything in the system a service (and no, adding a prefix doesn’t change anything – entity services, process services, etc).

I’m also going to be speaking about core design principles in my Intentions & Interfaces talk (which was quite a hit at QCon London). The interesting thing I’ve discovered over the years about design is that generics and dependency injection, when used together, can be used to create extensible systems with very little complexity. Not only that, but that this pattern is useful for all parts of a system, from communication, through data access, all the way to custom validation. As such, it can keep the complexity of the various technology stacks out of your core business logic, giving it a longer lease on life.

Finally, there’s also going to be a half-day tutorial on nServiceBus. I don’t want to go on-and-on about it here, but I can say that people who have attended this tutorial have come to me later telling me how they feel that they’ve had their eyes opened. I try to give just enough theory so that attendees can understand why nServiceBus exists, because after that using nServiceBus is pretty straight forward. This will be the first time I’m doing this in half-day format, so you’ll be getting the bottom-line, distilled version of this regularly full-day tutorial.

You can find the list of all my talks here, and for being the loyal reader you are, you get $100 bucks off the VIP price when you register using the code 8ASPK.

Hope to see you there.

Disclaimer

Before giving some numbers, please keep in mind that high performance of system infrastructure does not necessarily by itself mean that the system above it is running that fast. For instance, you may have server heartbeats running really quickly but the time it takes to save a purchase order borders on a minute. So, please, take all benchmarks with a grain of salt, or two, or a whole shaker-full.

While I’m not at liberty to say on which specific domain/company these numbers were measured, I can say that we had the full gamut of “stateless services”, statefull services (sagas), number crunching, large data sets, many users, complex visualization, etc. Also, this wasn’t the largest installation of nServiceBus that I’m aware of, but its the one I have the most specific numbers for.

Setup

OK, so using the default nServiceBus distribution using MSMQ, on servers where the queue files themselves were on separate SCSI RAID disks, we were pumping around 1000 durable, transactionally processed messages per second, per server. That means that similar to the Neuron case, no messages would be lost in the case of a single fault per server per window (time to replace a failed disk set at 3 hours from failure, through detection, to replacement per site – but that’s more an operational staffing concern, not the technology itself).

So, that’s 3.6 million messages per hour per server, at full load. We had a total of 98 servers doing these kinds of processing, not including web servers, databases, etc. Keep in mind that web servers would be communicating with other servers using nServiceBus, but that would maybe be an unfair comparison to the Neuron numbers.

Server Breakdown

Anyway, the 48 number crunching servers (blade centers) we had were at full load, so we were pumping more than 170 million messages there. Keep in mind that those servers had a really fast backbone so weren’t held up by IO. Your environment may be different.

Another 30 (regular pizza boxes) were doing our sagas. Saga state was stored in a distributed in-memory “cache”, so once again IO wasn’t an issue for processing those messages. We were at about 70% utilization there, coming to just over 100 million messages an hour.

The last 20 were clustered boxes (fairly expensive) that handled the various nServiceBus distributor and timeout manager processes were at full load since they handled control messages for all the servers as well as dynamically routing the load. However, on those boxes we used much higher performance disks for the messages, since they had to feed everything else, capable of doing, on average, around 5000 messages a second. That adds up to 360 million messages an hour.

Unnecessary Durability

Later, we moved a bunch of messages that didn’t need all that durability and transactionality off the disks, pushing the total throughput over 1 billion messages an hour. That was about 100 million per hour durable, 900 million per hour non-durable. You can guess that we were left with plenty of IO to spare at that point while we weren’t yet pushing the limit of our memory.

One thing that’s important to understand is the size of the messages that didn’t require durability was less than 1MB, with most weighing in under 10KB. Also, since most of those messages were published, less state management was required around them, enabling us to further improve performance.

Summary

NServiceBus didn’t give us all that by itself. It was the result of skilled architects, developers, and operations staff working together for many iterations, deploying, monitoring, re-designing, etc. You need to understand your technology, your hardware, and your specific performance, availability, and fault-tolerance requirements if you want to get anywhere.

There’s no magic.

I didn’t see the number or kinds of servers involved in the Neuron case study so this wasn’t ever really a comparison. Nor or we talking about the same system here.

So, please, don’t base your decisions on arbitrary numbers. Spend some time setting up a scaled down version of your target architecture with all the relevant technologies and measure. Be aware that you want high performance end to end, not just of the messaging part. At times, it makes sense to actively throw away messages (of the non-durable, published kind) to help a server come online faster especially after a restart.

Thus ends the tale of another “benchmark”.

Here’s a taste:

UI Thread and Concurrency Management

Whenever you use hosting on the UI thread, deadlocks are possible. For example, the following setup is guaranteed to result with a deadlock: A Windows Forms application is hosting a service with UseSynchronizationContext set to true, and UI thread affinity is established. The Windows Forms application then calls the service over one of its endpoints. The call to the service blocks the UI thread, while WCF posts a message to the UI thread to invoke the service. That message is never processed, because of the blocking UI thread—hence, the deadlock.

Another possible case for a deadlock occurs when a Windows Forms application is hosting a service with UseSynchronizationContext set to true and UI thread affinity is established. The service receives a call from a remote client. That call is marshaled to the UI thread and is eventually executed on that thread. If the service is allowed to call out to another service, that can result in a deadlock if the callout causality tries somehow to update the UI or call back to the service’s endpoint, because all of the service instances that are associated with any endpoint (regardless of the service-instancing mode) share the same UI thread.

Similarly, you risk a deadlock if the service is configured for reentrancy and it calls back to its client. You risk a deadlock if the callback causality tries to update the UI or enter the service, because that reentrance must be marshaled to the blocked UI thread.

Actually, I have difficulty believing that Juval would go so far as to suggest that even the forms should be services, but he does:

Form as a Service

The main motivation for hosting a WCF service on the UI thread is if the service must update the UI or the form. The problem is always: How does the service reach out and obtain a reference to the form? While the techniques and ideas that appear thus far in the listings certainly work, it would be simpler yet if the form were the service and hosted itself. For this to work, the form (or any window) must be a singleton service. The reason is that singleton is the only instancing mode that enables you to provide WCF with a live instance to host. In addition, you would not want a per-call form that exists only during a client call (which is usually very brief), nor would you want a per-session form that only a single client can establish a session with and update.

When a form is also a service, having that form as a singleton service is the best instancing mode all around.

I think that this article serves as a great treatise leading to only one conclusion – you’d have to be crazy to try to do this without some higher level framework, preferably with a different low-level framework too . Sucks Microsoft didn’t put one out – nor is there a pending beta, CTP, or even word about some project with a codename handling this. From what I know about Prism, it doesn’t intend to handle this issue either.

One thing that isn’t covered in the article is that if you do choose not to tie the client-side service to the UI thread, you open yourself up to race conditions. Reasons you’d want to handle messages on a different thread center around UI responsiveness. I’ve written about these things before:

• Object Builder – the place to fix system-wide threading bugs

• Eureka! AOP is the final piece of the multi-threaded smart client puzzle

• What Makes Smart Clients Safe?

The more I read things like this, the more I feel that I have to get going with my nServiceBus based solution. I’m fairly swamped as it is, so if anyone is interested in helping get this project off the ground, I’d be most grateful – as I think anyone else that had to build a smart client would.

SendLocal

First of all, the effect of calling SendLocal on IBus takes all the logical messages passed in (params IMessage[] messages), wraps them in a single TransportMessage, and puts that physical message at the end of the local queue. This call is equivalent to calling “Send(TransportMessage m, string destination);” on ITransport when passing in transport.Address as the parameter of destination.

There are numerous advantages to having this method, but one is the most important.

When client send a service a set of messages using “void Send(params IMessage[] messages);”, the client is requesting that the server treat this batch of messages as a unit of work. Under certain conditions, the service may choose to ignore the clients wishes – not least of which because the client has sent a ton of messages and the service doesn’t want ACID transactions to last a long time as they hurt throughput. In this case the server would use an intercepting message handler to go over those messages and call SendLocal for each. In other words, the server can set up units of work as it sees fit – taking into account client preference as well.

Other advantages include the ability to break apart complex or long-running logic into an “internal pipeline”. The Timeout Manager also makes use of this facility for “holding onto” messages until some condition occurs.

Return(errorCode)

The reason that integers are used as error codes is just so that you can push enums through them. This is the simplest way to get errors back to the client. More importantly, we take into account who on the client would be interested in this data.

Clients are often built using MVC with an additional Service Agent layer. Service Agents deal with translating the intent of Controllers into messages. Controllers don’t know about messaging, nor should they. However, they need to know when something fails with calls they initiated. As such, they are the final consumer of these error-code-enums, and integers are used to express them; that way Controllers don’t need to take a dependency on nServiceBus.

DoNotContinueDispatchingCurrentMessageToHandlers

This method on bus is used by intercepting message handlers in order to instruct the bus not to pass the current message on to subsequent handlers in the pipeline. This is often used by authentication and authorization handlers when those checks fail. This is what makes the message handling pipeline possible.

BuildAndDispatch

This method is defined on IBuilder and is used by the bus when dispatching messages to handlers. The reason that this exists instead of just having the bus ask the builder to create the handler and dispatch the call itself has to do with client-side threading. You can find the full explanation here – Object Builder, the place to fix system-wide threading bugs.

Summary

NServiceBus has grown over the years in environments where I’ve had the luxury of deciding most, if not all of the design of the systems involved. As such, it has taken on just the responsibilities needed from infrastructure in order to develop robust, flexible, and scalable systems. Check out the nServiceBus site.

Note to those who didn’t attend the session: you didn’t hear me say it, so I’ll repeat it: I hate WSDL almost as much as I hate Las Vegas. Ask me why sometime, or if I get enough of a critical mass of questions, I’ll blog it. If you’ve seen me do talks on Web Services, though, you’ve probably heard the rant: WSDL creates tightly-coupled endpoints precisely where loose coupling is necessary, WSDL encourages schema definitions that are inflexible and unevolvable, and WSDL intrinsically assumes a synchronous client-server invocation model that doesn’t really meet the scalability or feature needs of the modern enterprise. And that’s just for starters.

I hate WSDL.

I still hate Vegas more, though.

Web Services, and WSDL by connection have taken hold of the industry like cancer – inhibiting the minds of otherwise intelligent developers and architects. Whenever I get the “Web Services Question” (Does X support Web Services – where X is some design pattern, tool, and sometimes nServiceBus), I have to suppress an urge to groan – I’ve got the question that many times. The other day I was at a client and Sam, their head architect asked me that question. I gave my stock response:

“When you say ‘Web Services’, are you referring to SOAP or WSDL, and is HTTP a necessary component too?”

See how good I got at the suppressing thing?

Sam conceded that Web Services over TCP is OK too, so I pressed on with:

“What about UDP? FTP? MSMQ? Is it still ‘Web Services’ then? Is the rule then that ‘Web Services’ == SOAP?”

At that point, Sam was beginning to get a little flustered.

“And what’s so great about SOAP? Is it the interoperability? Because that’s just because it’s based on XSD.”

He didn’t know how to reply. Instead, he walked away from the whiteboard and sat down. I didn’t let up:

“And what if we want to do something other than Request/Response? How about one request with many responses? How about many requests and one response? And why does this decision need to be rigid? Shouldn’t we just be able to decide programmatically how many responses we want to return? Wouldn’t that flexibility be better than creating huge response structures for web methods to return?”

Sam made his last stand:

“Look, we can’t go and do something different from the rest of the industry. Everybody else is doing Web Services. It’s not like the technology doesn’t work.”

I gave way, a little:

“If you want, we can offer two interfaces. One, the flexible, robust, scalable XSD over messaging based solution. The second, an icky, synchronous Web Services facade which calls into our first interface.

I’m not saying that the technology doesn’t work – but both of us know that every problem has multiple solutions, some are fragile and error prone like WS, others are more elegant and have decades of knowledge behind them like messaging.

But we can do both if you like. How’s that?”

And it was agreed. The entire system would be built on one-way messaging patterns using XSD in cases where interoperability was required. And WS would be layered on, like a tiny little pig on top of a gigantic lipstick … thing – hmm, that metaphor isn’t really working – well, you get the idea.

I hate WSDL. Never been to Vegas, though.

So, I’ve started thinking what else should be included so that you get the most out of them.

Here’s what I’ve currently got in mind:

• All the powerpoint presentations
• Full source and samples of nServiceBus so you can run the code as I talk about it
• 4 hours of online consultation so you can get up to speed quickly in applying these principles and practices to your project

If you have anything else in mind that you think will help, please drop me a comment below.

So far, 32 people have expressed interest in getting this and it looks like I should be able to handle up to about 40 in a timely manner with my current setup. I hadn’t originally thought about corporate licensing, but since there have been some requests (so that all employees can use the information freely, get copies of the DVD’s, etc), I’ll be doing that too. If you’ve already left a comment about your interest in the DVD’s, I’ll assume you want the personal option. If you want to change that to the corporate option, please leave a comment either here or on the previous post.

Pricing

Ayende mentioned that the guys from Dot Net Rocks are selling Sharepoint training for about $700 per developer for a day of training. I really don’t know how much to charge for this – the guys in Australia paid quite a bit and I wouldn’t want them to feel – well, like you feel when you find out that the guy sitting next to you on the plane paid half what you did for his ticket; especially given that they’ve done so much of the production work on the DVD’s (Simon, Brad – you guys are my heroes. I really couldn’t do this without you).

So, I figure that I’ll hear what you guys think that the package is worth first. Leave me comment, or send me an email if you’d like more anonymity.

I’d also be interested in hearing what kind of domain you’re thinking about applying this stuff to – I might be able to put you in touch with people already applying SOA and nServiceBus in those areas so you can learn from each other as well.

Looking forward to hearing from you.

NServiceBus Tutorial

I gave a full day tutorial on nServiceBus and we had a full house! The tutorial was about 90% how to think about distributed systems, and 10% mapping those concepts onto nServiceBus. I made an effort to cram about 3 days of a 5 day training course I give clients into one day, but I think I was only about 85% successful. People didn’t have the time needed to let things really sink in and ask questions, but the lively forums and skype conversations available will probably do the trick.

Jim Webber after looking at the unit testing features of nServiceBus had this to say:

“Oh my God – you’ve created testable middleware! It’ll never catch on. The vendors won’t have it.”

To which I replied that several vendors were already coming on board with their own implementations of transports and saga persistence. I have absolutely no intention, desire, or (quite frankly) the ability to write an enterprise-class middleware runtime. All I hope to do with nServiceBus is to make it so that developers use what’s out there in one, middleware-product-agnostic way that will make their code more robust and flexible.

MEST & Mark – REST & Stefan

It was also great finally meeting the head MESTian, Mark Little, who also happens to work for Redhat as SOA Technical Development Manager and Director of Standards in the JBoss division. It was interesting to see the difference between how I went about messaging in nServiceBus (full peer-to-peer including pub/sub) whereas most of the Java world has the messaging infrastructure handled by something database-like in a deployment/networking kind of perspective. If that’s the way things are done, then I can definitely appreciate the advantages of Space-Based Architectures.

And I even got to steal Stefan Tilkov’s RESTful ear for an hour or so before I had to jet back home. It looks like we MESTians and RESTians can be one big happy family. I’m guessing that our despise of WS connects us all at a deeper level

Core Design Principles

I also gave a talk about core design principles, “Intentions & Interfaces – making patterns concrete”, and it went over very well especially considering that that was the first time that I gave that talk. You can find the slides here. From the feedback I heard after the talk, I think many people were surprised how many different parts of a system can be designed this way, and how flexible it is without making the code any more complex. The message was this:

Make Roles Explicit

Despite its simplicity, that leads to IEntity, IValidator<T> where T : IEntity, (which I wrote about a year ago – generic validation) and with a bit of Service Locator capabilities, you can add a line of code to your infrastructure that will validate all entities before they’re sent from the client to the server.

It leads to IFetchingStrategy<T> for improved database loading performance (also a year old – better DDD implementation and the NHibernate implementation).

It’s also how nServiceBus does message handling – IMessage, IMessageHandler<T> where T: IMessage, ISaga<T> where T : IMessage.

San Francisco?

Just a quick shout to my readers in the San Francisco area, if you’d be interested in hearing these talks/tutorials, give the organizers of QCon a shout and they’ll bring me out. That’s actually what got me to London – one of the attendees of a talk I gave at Oredev in Sweden last November missed my tutorial there so he put in a request and that did it. (Thanks Jan, I appreciate it!)

If you’re in a different part of the world and you’d like to have me give one of these talks, or other ones (I have a fair amount of material on Domain Models/DDD and Occasionally Connected Smart Clients), I’d be happy to make the trip and see you there as well.