NServiceBus 2.0 RTM is now generally available.

There were some small tweaks after the RC2 but I’m happy to say that, all in all, this was a very quiet stabilization period. Key customers have reported very high levels of satisfaction with the NServiceBus stability, scalability, and simplicity.

For example

Conduit.com has very happily doubled their user base with NServiceBus to 100 million.

#1 on American Banker’s 100 top financial tech companies, Fiserv has been quietly employing NServiceBus across many of their core services.

The Irish SaaS company Candidate Manager have been running the automated HR processes of geographically distributed giants like Hilton International on NServiceBus without missing a beat.

Community

Now with an active community of over 600 members, new users are quickly brought up to speed by the veterans and the much improved documentation on the site make adopting NServiceBus simpler than ever.

Training on NServiceBus has also ramped up nicely with the April course in Philadelphia now sold out. The next course will be in London in May via Skills Matter.

More courses are being planned in:

• Toronto, Canada Aug 9-13
• Calgary, Canada in September
• Johannesburg, South Africa in October
• Sydney, Australia in November
• Seattle WA, USA in December

Give it a try

You’ll have publish/subscribe messaging working in under 5 minutes with a simple F5 on the PubSub sample.

Get it here.

What’s New

The quickly approaching next release of NServiceBus will be version 2.0 and is a big step from 1.9. After 2 betas and 2 release candidates, this version has had a longer stabilization period than any of the versions so far (1.4-1.9). Many of my clients are already using it in production and are very pleased with it. I’ve heard similar reports from others in the community (now with over 500 members in the discussion group). There have been almost 10,000 downloads since the version 1.9 release and in every country I visit I meet people using NServiceBus in new and interesting applications.

With my appearance on Hanselminutes, many in the mainstream .NET industry have started taking a look at NServiceBus. That, and the fact that Microsoft’s Oslo technology has now taken a very data-driven turn (rather than its original service-oriented direction).

Interestingly enough, I’ve been hearing more and more reports about people using NServiceBus as a developer-friendly API on top of other technologies. This includes BizTalk and even Neuron. I never thought that people would take the pluggability of NServiceBus that far.

So, what is NServiceBus?

Well, it’s a service bus, y’know, like an ESB – just an open-source one.

All kidding aside, in a nutshell, it gives you an easy way to integrate transactional messaging into your applications.

One of the reasons why you might want to do that is so that you don’t lose messages containing valuable data when IIS recycles your AppDomain, every 15-20 minutes (as I wrote about in this MSDN magazine article).

There are many other nice things in there, like the ability to unit test your service layers and long-running processes but you can read more about that here…

Documentation

One of the biggest differences to NServiceBus in this release is documentation.

A lot of work has gone into the NServiceBus.com site to help developers hit the ground running with NServiceBus, including the more advanced aspects of transparent scale-out with the distributor and multi-site communications.

There is still work to be done in this area but feedback so far has been extremely positive (except for some grumblings from certain old-timers saying that if they could figure it out by themselves, well, you know the rest).

In Closing

If you’re building a distributed enterprise .NET system, take 5 minutes, download it, and see transactional publish/subscribe messaging working on your machine without any big heavy-weight middleware.

www.NServiceBus.com

Take a listen.

The thing is, when looking at this in light of the full software development lifecycle, there are signs that the waters run deeper than we might have originally thought.

Let’s take things one step at a time though…

What is it?

Wikipedia tells us:

“Convention over Configuration (aka Coding by convention) is a software design paradigm which seeks to decrease the number of decisions that developers need to make, gaining simplicity, but not necessarily losing flexibility. The phrase essentially means a developer only needs to specify unconventional aspects of the application.”

What this means is that frameworks built in this way have default implementations that can be swapped out if needed. So far so good.

For example…

In NServiceBus, there is an abstraction for how subscription data is stored and multiple implementations – one in-memory, another using a durable MSMQ queue, and a third which uses a database. The convention for that part of the system is that the MSMQ implementation will be used, unless something else is specified.

Developers wishing to specify a different implementation can specify the desired implementation in the container – either one that comes out of the box, or their own implementation of ISubscriptionStorage.

Things get more interesting when we consider the full lifecycle.

Lifecycle effects

When developers are in the early phases of writing a new service, they want to focus primarily on what the service does – its logic. They don’t want to muck around with MSMQ queues for storing subscriptions and would much rather use the in-memory storage.

As the service takes shape and the developers want to run the full service on their machine, possibly testing basic fault-tolerance behaviors – kill one service, see that the others get a timeout, bring the service back up, wanting it to maintain all the previous subscriptions.

Moving on from there, our developers want to take the same system they just tested on their machine and move it into a staging environment. There, they don’t want to use the MSMQ implementation for subscription storage, but rather the database implementation – as will be used in the production environment.

While it may not sound like a big deal – changing the code which specifies which implementation to use when moving from one environment to another, consider that on top of just subscription storage, there is logging (output to console, file, db?), saga persistence (in-memory, file-based DB, relational DB), and more.

It’s actually quite likely that something will get missed as we move the system between environments. Can there be a better way?

What if…

What if there was some way for the developer to express their intent to the system, and the system could change its conventions, without the developer having to change any code or configuration files?

You might compare this (in concept) to debug builds and release builds. Same code, same config, but the runtime behaves different between the two.

As I mulled over how we could capture that intent without any code or config changes, the solution that I kept coming to seemed too trivial at first, so I dismissed it. Yet, it was the simplest one that would work for console and WinForms applications, as well as windows services – command line arguments. The only thing is that I don’t think those are available for web applications.

But since we’re still in “what if” land, and I’m more thinking out loud here than providing workable solutions for tomorrow morning, let’s “what if” command line arguments worked for web apps too.

Command-Line Intent

Going back to our original scenario, when developers are working on the logic of the service, they run it using the generic NServiceBus host process, passing it the command line parameter /lite (or whatever). The host then automatically configures all the in-memory implementations.

As the system progresses, when the developer wants to run everything on their machine, they run the processes with /integration. The host then configures the appropriate implementations (MSMQ for subscription storage, SQLite for saga persistence, etc.

When the developers want to run the system in production, they could specify /production (or maybe that could be the default?), and the database backed implementations would be configured.

Imagine…

Imagine being able to move that fluidly from one environment to another. Not needing to pore over configuration files or startup script code which configures a zillion implementation details. Not needing to worry that as you moved the system to staging something would break.

Imagine short, frictionless iterations even for large scale systems.

Imagine – lifecycle-aware frameworks making all this imagination a reality.

In Closing

We’re not there yet – but we’re not that far either. The generic host we’re providing with NServiceBus 2.0 is now being extended to support exactly these scenarios.

It’s my hope that as more of us think about this challenge, we’ll come up with better solutions and more intelligent frameworks. Just as convention came to our rescue before, breaking us out of the pain of endless XML configuration, I hope this new family of lifecycle-aware frameworks will make the friction of moving a system through dev, test, staging, and production a thing of the past.

A worthy problem for us all to solve, don’t you think?

Any ideas on how to make it a reality?
Send them in – leave a comment below.

“A common complaint is that the first call on a client object takes some disproportionately large amount of time, usually ten seconds or more, while successive calls are instantaneous. There are many reasons why this might happen so there’s no generic resolution for this problem.” — Nicholas Allen

The thing is that there IS a generic solution to this problem.

It’s queued messaging.

The only thing is that you have to give up talking to your services as if they were regular objects – calling methods on them and expecting a response. In other words, designing a distributed systems isn’t like designing a regular OO system just with some WCF sprinkled on top.

Even when trying to do fire and forget messaging on top of WCF (void method calls with the OneWay attribute), the underlying channel can still block your thread, as Nick mentioned.

A queue isn’t an implementation detail.
It’s the primary architectural abstraction of a distributed system.

The common e-commerce example

We accept orders, bill the customer, and then ship them the product.

Fairly straight-forward.

Since each part of that process can be quite complex, let’s have each step be handled by a service:

Sales, Billing, and Shipping. Each of these services will publish an event when it’s done its part. Sales will publish OrderAccepted containing all the order information – order Id, customer Id, products, quantities, etc. Billing will publish CustomerBilledForOrder containing the customer Id, order Id, etc. And Shipping will publish OrderShippedToCustomer with its data.

So far, so good. EDA and SOA seem to be providing us some value.

Where’s the saga?

Well, let’s consider the behavior of the Shipping service. It shouldn’t ship the order to the customer until it has received the CustomerBilledForOrder event as well as the OrderAccepted event. In other words, Shipping needs to hold on to the state that came in the first event until the second event comes in. And this is exactly what sagas are for.

Let’s take a look at the saga code that implements this. In order to simplify the sample a bit, I’ll be omitting the product quantities.

   1:      public class ShippingSaga : Saga<ShippingSagaData>,

   2:          ISagaStartedBy<OrderAccepted>,

   3:          ISagaStartedBy<CustomerBilledForOrder>

   4:      {

   5:          public void Handle(OrderAccepted message)

   6:          {

   7:              this.Data.ProductIdsInOrder = message.ProductIdsInOrder;

   8:          }

   9:   

  10:          public void Handle(CustomerBilledForOrder message)

  11:          {

  12:               this.Bus.Send<ShipOrderToCustomer>(

  13:                  (m =>

  14:                  {

  15:                      m.CustomerId = message.CustomerId;

  16:                      m.OrderId = message.OrderId;

  17:                      m.ProductIdsInOrder = this.Data.ProductIdsInOrder;

  18:                  }

  19:                  ));

  20:   

  21:              this.MarkAsComplete();

  22:          }

  23:   

  24:          public override void Timeout(object state)

  25:          {

  26:              

  27:          }

  28:      }

First of all, this looks fairly simple and straightforward, which is good.

It’s also wrong, which is not so good.

One problem we have here is that events may arrive out of order – first CustomerBilledForOrder, and only then OrderAccepted. What would happen in the above saga in that case? Well, we wouldn’t end up shipping the products to the customer, and customers tend not to like that (for some reason).

There’s also another problem here. See if you can spot it as I go through the explanation of ISagaStartedBy<T>.

Saga start up and correlation

The “ISagaStartedBy<T>” that is implemented for both messages indicates to the infrastructure (NServiceBus) that when a message of that type arrives, if an existing saga instance cannot be found, that a new instance should be started up. Makes sense, doesn’t it? For a given order, when the OrderAccepted event arrives first, Shipping doesn’t currently have any sagas handling it, so it starts up a new one. After that, when the CustomerBilledForOrder event arrives for that same order, the event should be handled by the saga instance that handled the first event – not by a new one.

I’ll repeat the important part: “the event should be handled by the saga instance that handled the first event”.

Since the only information we stored in the saga was the list of products, how would we be able to look up that saga instance when the next event came in containing an order Id, but no saga Id?

OK, so we need to store the order Id from the first event so that when the second event comes along we’ll be able to find the saga based on that order Id. Not too complicated, but something to keep in mind.

Let’s look at the updated code:

   1:      public class ShippingSaga : Saga<ShippingSagaData>,

   2:          ISagaStartedBy<OrderAccepted>,

   3:          ISagaStartedBy<CustomerBilledForOrder>

   4:      {

   5:          public void Handle(CustomerBilledForOrder message)

   6:          {

   7:              this.Data.CustomerHasBeenBilled = true;

   8:   

   9:              this.Data.CustomerId = message.CustomerId;

  10:              this.Data.OrderId = message.OrderId;

  11:   

  12:              this.CompleteIfPossible();

  13:          }

  14:   

  15:          public void Handle(OrderAccepted message)

  16:          {

  17:              this.Data.ProductIdsInOrder = message.ProductIdsInOrder;

  18:   

  19:              this.Data.CustomerId = message.CustomerId;

  20:              this.Data.OrderId = message.OrderId;

  21:   

  22:              this.CompleteIfPossible();

  23:          }

  24:   

  25:          private void CompleteIfPossible()

  26:          {

  27:              if (this.Data.ProductIdsInOrder != null && this.Data.CustomerHasBeenBilled)

  28:              {

  29:                  this.Bus.Send<ShipOrderToCustomer>(

  30:                     (m =>

  31:                     {

  32:                         m.CustomerId = this.Data.CustomerId;

  33:                         m.OrderId = this.Data.OrderId;

  34:                         m.ProductIdsInOrder = this.Data.ProductIdsInOrder;

  35:                     }

  36:                     ));

  37:                  this.MarkAsComplete();

  38:              }

  39:          }

  40:      }

And that brings us to…

Saga persistence

We already saw why Shipping needs to be able to look up its internal sagas using data from the events, but what that means is that simple blob-type persistence of those sagas is out. NServiceBus comes with an NHibernate-based saga persister for exactly this reason, though any persistence mechanism which allows you to query on something other than saga Id would work just as well.

Let’s take a quick look at the saga data that we’ll be storing and see how simple it is:

   1:      public class ShippingSagaData : ISagaEntity

   2:      {

   3:          public virtual Guid Id { get; set; }

   4:          public virtual string Originator { get; set; }

   5:          public virtual Guid OrderId { get; set; }

   6:          public virtual Guid CustomerId { get; set; }

   7:          public virtual List<Guid> ProductIdsInOrder { get; set; }

   8:          public virtual bool CustomerHasBeenBilled { get; set; }

   9:      }

You might have noticed the “Originator” property in there and wondered what it is for. First of all, the ISagaEntity interface requires the two properties Id and Originator. Originator is used to store the return address of the message that started the saga. Id is for what you think it’s for. In this saga, we don’t need to send any messages back to whoever started the saga, but in many others we do. In those cases, we’ll often be handling a message from some other endpoint when we want to possibly report some status back to the client that started the process. By storing that client’s address the first time, we can then “ReplyToOriginator” at any point in the process.

The manufacturing sample that comes with NServiceBus shows how this works.

Saga Lookup

Earlier, we saw the need to search for sagas based on order Id. The way to hook into the infrastructure and perform these lookups is by implementing “IFindSagas<T>.Using<M>” where T is the type of the saga data and M is the type of message. In our example, doing this using NHibernate would look like this:

   1:      public class ShippingSagaFinder : 

   2:          IFindSagas<ShippingSagaData>.Using<OrderAccepted>,

   3:          IFindSagas<ShippingSagaData>.Using<CustomerBilledForOrder>

   4:      {

   5:          public ShippingSagaData FindBy(CustomerBilledForOrder message)

   6:          {

   7:              return FindBy(message.OrderId)

   8:          }

   9:   

  10:          public ShippingSagaData FindBy(OrderAccepted message)

  11:          {

  12:              return FindBy(message.OrderId)

  13:          }

  14:   

  15:          private ShippingSagaData FindBy(Guid orderId)

  16:          {

  17:              return sessionFactory.GetCurrentSession().CreateCriteria(typeof(ShippingSagaData))

  18:                  .Add(Expression.Eq("OrderId", orderId))

  19:                  .UniqueResult<ShippingSagaData>();

  20:          }

  21:   

  22:          private ISessionFactory sessionFactory;

  23:   

  24:          public virtual ISessionFactory SessionFactory

  25:          {

  26:              get { return sessionFactory; }

  27:              set { sessionFactory = value; }

  28:          }

  29:      }

For a performance boost, we’d probably index our saga data by order Id.

On concurrency

Another important note is that for this saga, if both messages were handled in parallel on different machines, the saga could get stuck. The persistence mechanism here needs to prevent this. When using NHibernate over a database with the appropriate isolation level (Repeatable Read – the default in NServiceBus), this “just works”. If/When implementing your own saga persistence mechanism, it is important to understand the kind of concurrency your business logic can live with.

Take a look at Ayende’s example for mobile phone billing to get a feeling for what that’s like.

Summary

In almost any event-driven architecture, you’ll have services correlating multiple events in order to make decisions. The saga pattern is a great fit there, and not at all difficult to implement. You do need to take into account that events may arrive out of order and implement the saga logic accordingly, but it’s really not that big a deal. Do take the time to think through what data will need to be stored in order for the saga to be fault-tolerant, as well as a persistence mechanism that will allow you to look up that data based on event data.

If you feel like giving this approach a try, but don’t have an environment handy for this, download NServiceBus and take a look at the samples. It’s really quick and easy to get set up.

Very little has changed, so the version 1.9 story described here is still accurate.

Just last week one of my clients went live with a rollout to one of the world’s biggest names in the hospitality industry and things are looking good. Since stability is such a big deal to them (and many of my other clients), they’ve rolled out on nServiceBus 1.8 but now are ready to make the move to 1.9. Hopefully we’ll be able to get a case study out of them

For more information, go to the NServiceBus site.

What is it?

From the NServiceBus.com site:

“NServiceBus is a powerful, yet lightweight, open source messaging framework
for designing distributed .NET enterprise systems. Entirely pluggable yet simple to use, NServiceBus gives programmers a head-start on developing robust, scalable,
and maintainable service-layers and long-running business processes.”

One of the developers who downloaded nServiceBus, Jürgen, sent me this in an email:

“I took the samples, swapped in my own code, and had a machines subscribing, publishing, messaging, in like 15 minutes.

I’ve got to tell you – from reading the architecture stuff on your blog I always thought this stuff was going to be hard. I know you always say its supposed to be simple but I never really believed it. I mean, seriously, if I had to do this stuff with web services or WCF – well, I wouldn’t know where to begin.”

(His English totally surprised me – not what you’d expect from someone called Jürgen. Born in Sweden but grew up in the US.)

BTW, Sam’s showed where to begin with WCF: basic pub/sub without durability is 480 LOC.

What’s different – Containers

People who looked at earlier versions of nServiceBus will see many incremental improvements that smooth over previously rough parts of the framework.

Chris Patterson, co-founder of MassTransit mentioned one such area in his blog post:

“Both Dru and I needed a framework for asynchronous messaging to address some work-related application requirements. While MSMQ is provided out of the box [on Windows], it doesn’t directly encourage some good distributed application practices such a loose coupling. Our goal was to abstract the messaging aspects so the services could be built to deal with plain old objects (POCOs) instead of lower level transport messages.

Originally, we both looked at NServiceBus as a way to make this happen. I’ve followed Udi’s blog for a while and have really gained a lot of knowledge from his posts and presentations. However, our lack of experience in Spring.NET, along with a general lack of understanding of all the complexity of such a framework led us down the path of building our own framework.”

Version 1.9 takes a totally different approach (than v1.6/1.7 at which I believe Chris was looking back then) to dependency injection frameworks (like Spring.NET) and decreases their footprint. Developers don’t need to know anything about Spring, Castle, or any other container to get started, but always have the ability to configure it however they want and even swap in their container of choice.

What’s different – DLL Footprint

Another difference is the number of assemblies that come with nServiceBus. Nathan Stults had this to say about version 1.8:

“NServiceBus (for valid architectural reasons) is split up into eleven thousand, two hundred and nine different DLL’s. That may be an exaggeration, but many of the dll’s have only one or two code files in them, with only a few lines of code each.”

NServiceBus has always supported swapping out various technological implementations and will continue do so. For that reason, the development environment is organized into 79 projects whose dependencies are managed very strictly. As of version 1.9, most of these assemblies including many supporting libraries like Spring and Castle have been merged into NServiceBus.dll. There’s also NServiceBus.Testing.dll which supports fluent-unit-testing entire business processes.

There is one non-optional external dependency which hasn’t been merged and that is log4net – although if you configure in a Common.Logging provider to your own logging infrastructure, you can do without it as well. With log4net, the minimum deployment footprint is 2 assemblies. The other dependency which is optional is NHibernate. The reason for leaving these out is that many teams depend on specific versions of those assemblies.

In short – you reference ONE assembly. Just one.

What else?

There is quite a lot in there. Ayende’s put out several posts describing those features and similarities to the bus he’s working on:

• Managing Timeouts
• Long-running processes
• Load Balancing

Ayende’s description of the NServiceBus load-balancing capability was:

“The distributor section of NServiceBus is a thing of beauty.”

Performance

You may be surprised by the kind of performance we’re able to wring out of “basic” MSMQ. Keep in mind, though, that you can swap in your own transport – there are already some others out there on the Contrib site including ActiveMQ and shared memory.

The most thorough performance numbers I’ve seen on nServiceBus have been written up here:

“total throughput over 1 billion messages an hour. That was about 100 million per hour durable, 900 million per hour non-durable”

This was on an 3 blade centers (48 blades), 30 pizza boxes (1U), and 20 clusters, version 1.8.

On Gojko’s blog there is a video of talk on nServiceBus where Dave de Florinier mentioned throughputs of 600,000 messages per hour (no mention of supporting hardware) and I think it was version 1.7 or 1.8 of nServiceBus.

Recently, Raymond Lewallen posted his numbers for 1.9 to the discussion group:

“We load tested our full duplex and pub/sub scenarios the other day at about 500 messages per second with no hiccups at all.” (save yourself the math, its 1.8M/hr)

The really interesting numbers are coming in from one of my clients (on 1.9) where the focus is on business process (saga) throughput where they’re seeing millions completing each day on top of IO intensive technical processes. I’m excited.

Looking forward

After working with clients using nServiceBus over 4 years now (even before it was open source or had a real name), it just keeps getting better. It’s also really great to see more open source projects coming on the scene – MassTransit (now a year old), and Rhino Service Bus (young, but in production with NH-Prof).

The mutual stealing cross-pollination is increasing the pace all around.

I intend to take up TopShelf (which came out of MassTransit) as the generic service host. That combined with the robust distributor hosting model will make “grid-friendly” nServiceBus endpoints much easier. Ayende’s posts about automatically creating queues and other “first-time-developer” features have really worked to decrease the nServiceBus learning curve – and it shows.

Documentation is a notorious problem in open-source projects and nServiceBus hasn’t escaped it. API and internal documentation is only now getting close to 100% and the samples now give developers a good start on using it. Those developers looking at swapping out certain bits of functionality have a harder time, but that’s slowly improving as well. The discussion group is the best place to get those hard-to-find answers now with a community over 200 strong.

The difficulty developers have in adopting nServiceBus is giving up “request/response” thinking. This requires an adjustment to a system’s architecture and is what most of this blog has been about. Conversely, if you have been following this blog and this thinking resonates with you, you’ll find it very simple and straight-forward. The overview page on NServiceBus.com also gives a good description of messaging basics – stuff that I’ve kind of glossed over on this blog so far.

Go on then. Take it for a spin. Write a review. Sam Gentile had this to say about v1.8:

“The bottom line is: I like what I see. Although it’s a framework, not an ESB product like Neuron, it’s a powerful framework that takes the right approach on SOA and enforces a paradigm of reliable one-way, *non-blocking* calls.”

Can’t wait to hear the response to 1.9.

In regular client-server development, the server is responsible for providing the client with all CRUD (create, read, update, and delete) capabilities. However, when users look at data they do not often require it to be up to date to the second (given that they often look at the same screen for several seconds to minutes at a time). As such, retrieving data from the same table as that being used for highly consistent transaction processing creates contention resulting in poor performance for all CRUD actions under higher load.

A Scalable Solution

One of the common answers to this question is for the server/service to publish a message when data changes (say, as the result of processing a message) and for clients to subscribe to these messages. When such a notification arrives at a client, the client would cache the data it needs. Then, when the user wants to see a grid of data, that data is already on the client. Of course, this solution doesn’t work so well for older client machines (like some point of service devices) or if there are millions of rows of data.

The thing is that this solution is one implementation of a more general pattern – command query separation (CQS).

Command Query Separation

Wikipedia describes CQS as a pattern where "… every method should either be a command that performs an action, or a query that returns data to the caller, but not both. More formally, methods should return a value only if they are referentially transparent and hence possess no side effects."

Martin Fowler is less strict about the use of CQS allowing for exceptions: "Popping a stack is a good example of a modifier that modifies state. Meyer correctly says that you can avoid having this method, but it is a useful idiom. So I prefer to follow this principle when I can, but I’m prepared to break it to get my pop."

So, how does separating commands from queries and SOA help at all in getting data to and from a UI? The answer is based on Pat Helland’s thinking as described in his article Data on the Inside vs. Data on the Outside.

Services Cross Boxes

The biggest lie around SOA is that services run.

Let that sink in a second.

Sure services have runnable components, but that’s not why they’re important.

I’ll skip the books of background and cut to the chase:

Services communicate with each other using publish/subscribe and one-way messaging. Services have components inside them. Inside a service, these components can communicate with each using synchronous RPC, or any other mechanism. Also, these components can reside on different machines.

This is broader than just scaling out a service. There can be service components running on the client as well as the server.

SOA & CQS

Combining these two concepts together, here’s what comes out:

In this solution there are two services that span both client and server – one in charge of commands (create, update, delete), the other in charge of queries (read). These services communicate only via messages – one cannot access the database of the other.

The command service publishes messages about changes to data, to which the query service subscribes. When the query service receives such notifications, it saves the data in its own data store which may well have a different schema (optimized for queries like a star schema).

The client component which is in charge of showing grids of data to the user behaves the same as it would in a regular layered/tiered architecture, using synchronous blocking request/response to get its data – SOA doesn’t change that.

Composite Applications

Although the client side components of both the command and query services are hosted in the same process, they are very much independent of each other. That being said, from an interoperability perspective (the one that most people attribute to SOA), all of the client-side components will likely be developed using the same technology – although there are already ways to host Java code in .NET and vice-versa.

Of course, once we talk about web UI’s things are a bit different – but still similar. While web-server-side there may be a level of independence, for browser side inter-component communications we’re still likely to target javascript. There, I’ve managed to say something technical supporting mashups and SOA without lying through my teeth.

On the Microsoft side with the recent release of the Composite Application Guidance & Library (pronounced "prism") I hope that more of these principles will be reaching the "smart client". The command pattern is especially critical in maintaining the separation while enabling communication to still occur so I’m glad that, as one of the Prism advisors, I was able to simplify that part (Glenn still has nightmares about that rooftop conversation).

Publish / Subscribe

In the "scalable solution" section up top I mentioned how publish/subscribe to the smart client is really just one implementation of CQS and SOA. So, how different is it really?

Well, there will probably be a different technology mapping. Instead of a star-schema OLAP product, we might simply store the published data in memory on the client. That is, if you designed your components to be technology agnostic.

In terms of the use of nServiceBus, the same component is going to be subscribing to the same type of message – all that’s different is that now every client will be having data pushed to them rather than this occurring server-side only.

You could have the same code deployed differently in the same system – stronger clients subscribing themselves, weaker ones using a remote server. Web servers would probably be considered stronger clients. This kind of flexible deployment has proven to be extremely valuable for my larger clients. The added benefit of enabling users to work (view data) even while offline (somewhere there’s no WIFI) is just icing on the cake.

A Word of Warning

Once the client starts receiving notifications, and handling those on a background thread (as it should) the code becomes susceptible to deadlocks and data races. Juval does a good job of outlining some of those with respect to the use of WCF. Prism doesn’t provide any assurances in this area either.

Summary

NServiceBus is not designed to be used for any and all types of communication in a given architecture. In the examples above, nServiceBus handles the publish/subscribe but leaves the synchronous RPC to existing solutions like WCF. Not only that, but synchronous RPC does have its place in architecture, just not across service boundaries. In all cases, data is served to users from a store different from that which transaction processing logic uses.

Command Query Separation is not only a good idea at the method/class level but has advantages at the SOA/System level as well – yet another good idea from 20 years ago that services build upon. Making use of CQS requires understanding your data and its uses – SOA builds on that by looking into data volatility and the freshness business requirements around it.

Finally, designing the components of your services in such a way that their dependency on technology is limited buys a lot of flexibility in terms of deployment and, consequently, significant performance and scalability gains.

Simple, it is. Easy, it is not.

Reliability

Locking in SqlWorkflowPersistenceService (via Ron Jacobs) where, if you want predictable persistence (MS: ‘none of our customers asked for this to be easy’), you need to use a custom activity (which Ron was kind enough to supply).

“Given what I learned today I’d have to say that I’d be very careful about using workflows with an optimistic locking.  Detecting these types of situations is not that simple.”

Let’s think about that. If we’re doing pessimistic locking, we get into the problem of, if a host restarts (as the result of a critical windows patch or some other unexpected occurrence), that the workflow won’t be able to be handled by any other host in the meantime (you didn’t care so much about your SLA, did you?).

Luckily, someone’s come up with a hack that works around this robustness problem in Scalable Workflow Persistence and Ownership.

“So this code will attempt to load workflow instances with expired locks every second. Is it a hack? Yes. But without one of two things in the SqlWorkflowPersistenceService its the sort of code you have to write to pick up unlocked workflow instances robustly.”

This will seriously churn the table used to store your workflows, decreasing performance of workflows that haven’t timed out. Oh well.

Testability

Implementing WCF Services without Referencing WCF (via Mark Seemann):

“More than a year ago, I wrote my first post on unit testing WCF services. One of my points back then was that you have to be careful that the service implementation doesn’t use any of the services provided by the WCF runtime environment (if you want to keep the service testable). As soon as you invoke something like OperationContext.Current, your code is not going to work in a unit testing scenario, but only when hosted by WCF.”

After pointing out some of the more basic difficulties in testability a straightforward WCF implementation brings, Mark turns the heat up in his follow-up post, Modifying Behavior of WCF-Free Service Implementations:

“Perhaps you need to control the service’s ConcurrencyMode, or perhaps you need to set UseSynchronizationContext. These options are typically controlled by the ServiceBehaviorAttribute. You may also want to provide an IInstanceProvider via a custom attribute that implements IContractBehavior. However, you can’t set these attributes on the service implementation itself, since it mustn’t have a reference to System.ServiceModel.”

Wow – all the things required to make a WCF service scalable and thread-safe make it difficult to test. In the end, we’re beginning to see how many hoops we have to go through in order to get separation of concerns, but until we can take all this and get it out of our application code, it’s an untenable solution. I hope Mark will continue with this series, if only so I can take the framework that might grow out of it and use it as a generic WCF transport for NServiceBus.

Comparison

After the Neuron-NServiceBus comparison that Sam and I had, we talked some more. After going through some of the rational and thinking, Sam even put nServiceBus into his WCF-Neuron comparison talk. Sam had this to say about nServiceBus:

“The bottom line is: I like what I see. Although it’s a framework, not an ESB product like Neuron, it’s a powerful framework that takes the right approach on SOA and enforces a paradigm of reliable one-way, *non-blocking* calls. That is the point of the talk tonight overall; we need to get away from the stack world of synchronous RPC calls to true asynchronous non-blocking message based SOA systems.”

The main concern I have with a WCF+WF based solution is that developers need to know a lot in order to make it testable, scalable, and robust. In nServiceBus, that’s baked into the design. It would be extremely difficult for a developer writing application logic to interfere with when persistence needs to happen, or the concurrency strategy of long-running workflows. The fact that message handlers in the service layer don’t need concurrency modes, instance providers, or any of that junk make them testable by default.

Think about that a second.

Scary, isn’t it?

At TechEd Israel I had a discussion with Manu on this very issue, just under a different hat:

What’s the difference between a short-running workflow and a long-running one?

Manu suggested that we look at the actual time that things ran to differentiate between them. I asserted that if any external communication was involved in some part of state-management logic, that logic should automatically be treated as long-running.

Manu’s reasoning was that the complexity involved in writing long-running workflows was not justified for things that ran quickly, even if there was communication involved. Many developers don’t think twice about synchronously calling some web services in the middle of their database transaction logic. In the many Microsoft presentations I’ve been at on WF, not once has it been mentioned that state machines should be used when external communication is involved.

The problem that I have with this guidance is how do you know how quickly a remote call will return?

Do you just run it all locally on your machine, measure, and if it doesn’t take more than a second or so, then you’re OK?

The fact of the matter is that we can never know what the response time of a remote call will be. Maybe the remote machine is down. Maybe the remote process is down. Maybe someone changed the firewall settings and now we’re doing 10KB/s instead of 10MB/s. Maybe the local service is down and we’re communicating with the backup on the other side of the Pacific Ocean.

But the thing is, Manu’s right.

Writing long-running workflows (with WF) is more complex than is justified. My guess is that since WF wasn’t specifically designed for long-running workflows only, that this complexity crept in.

Sagas in nServiceBus were specifically designed for long-running workflows only.

Maybe that’s what kept them simple.

Since all external communication is done via one-way, non-blocking messaging only, each step of a saga runs as quick as if no communication were done at all. This keeps the time the transaction in charge of handling a message is open as short as possible. That, in turn, leads to the database being able to support more concurrent users.

In short, sagas are both more scalable and more robust.

No need to worry about garbaging-up your database.

I’ll be giving my Avoid a Failed SOA talk (again). Apparently, I’m not the only one who’s been seeing one SOA project fail after another. Luckily, I’ve lived through enough of them to figure out what sort of things empirically have lead to failure, and now I’m telling the tale. One of the big reasons, by the way, is calling everything in the system a service (and no, adding a prefix doesn’t change anything – entity services, process services, etc).

I’m also going to be speaking about core design principles in my Intentions & Interfaces talk (which was quite a hit at QCon London). The interesting thing I’ve discovered over the years about design is that generics and dependency injection, when used together, can be used to create extensible systems with very little complexity. Not only that, but that this pattern is useful for all parts of a system, from communication, through data access, all the way to custom validation. As such, it can keep the complexity of the various technology stacks out of your core business logic, giving it a longer lease on life.

Finally, there’s also going to be a half-day tutorial on nServiceBus. I don’t want to go on-and-on about it here, but I can say that people who have attended this tutorial have come to me later telling me how they feel that they’ve had their eyes opened. I try to give just enough theory so that attendees can understand why nServiceBus exists, because after that using nServiceBus is pretty straight forward. This will be the first time I’m doing this in half-day format, so you’ll be getting the bottom-line, distilled version of this regularly full-day tutorial.

You can find the list of all my talks here, and for being the loyal reader you are, you get $100 bucks off the VIP price when you register using the code 8ASPK.

Hope to see you there.

Disclaimer

Before giving some numbers, please keep in mind that high performance of system infrastructure does not necessarily by itself mean that the system above it is running that fast. For instance, you may have server heartbeats running really quickly but the time it takes to save a purchase order borders on a minute. So, please, take all benchmarks with a grain of salt, or two, or a whole shaker-full.

While I’m not at liberty to say on which specific domain/company these numbers were measured, I can say that we had the full gamut of “stateless services”, statefull services (sagas), number crunching, large data sets, many users, complex visualization, etc. Also, this wasn’t the largest installation of nServiceBus that I’m aware of, but its the one I have the most specific numbers for.

Setup

OK, so using the default nServiceBus distribution using MSMQ, on servers where the queue files themselves were on separate SCSI RAID disks, we were pumping around 1000 durable, transactionally processed messages per second, per server. That means that similar to the Neuron case, no messages would be lost in the case of a single fault per server per window (time to replace a failed disk set at 3 hours from failure, through detection, to replacement per site – but that’s more an operational staffing concern, not the technology itself).

So, that’s 3.6 million messages per hour per server, at full load. We had a total of 98 servers doing these kinds of processing, not including web servers, databases, etc. Keep in mind that web servers would be communicating with other servers using nServiceBus, but that would maybe be an unfair comparison to the Neuron numbers.

Server Breakdown

Anyway, the 48 number crunching servers (blade centers) we had were at full load, so we were pumping more than 170 million messages there. Keep in mind that those servers had a really fast backbone so weren’t held up by IO. Your environment may be different.

Another 30 (regular pizza boxes) were doing our sagas. Saga state was stored in a distributed in-memory “cache”, so once again IO wasn’t an issue for processing those messages. We were at about 70% utilization there, coming to just over 100 million messages an hour.

The last 20 were clustered boxes (fairly expensive) that handled the various nServiceBus distributor and timeout manager processes were at full load since they handled control messages for all the servers as well as dynamically routing the load. However, on those boxes we used much higher performance disks for the messages, since they had to feed everything else, capable of doing, on average, around 5000 messages a second. That adds up to 360 million messages an hour.

Unnecessary Durability

Later, we moved a bunch of messages that didn’t need all that durability and transactionality off the disks, pushing the total throughput over 1 billion messages an hour. That was about 100 million per hour durable, 900 million per hour non-durable. You can guess that we were left with plenty of IO to spare at that point while we weren’t yet pushing the limit of our memory.

One thing that’s important to understand is the size of the messages that didn’t require durability was less than 1MB, with most weighing in under 10KB. Also, since most of those messages were published, less state management was required around them, enabling us to further improve performance.

Summary

NServiceBus didn’t give us all that by itself. It was the result of skilled architects, developers, and operations staff working together for many iterations, deploying, monitoring, re-designing, etc. You need to understand your technology, your hardware, and your specific performance, availability, and fault-tolerance requirements if you want to get anywhere.

There’s no magic.

I didn’t see the number or kinds of servers involved in the Neuron case study so this wasn’t ever really a comparison. Nor or we talking about the same system here.

So, please, don’t base your decisions on arbitrary numbers. Spend some time setting up a scaled down version of your target architecture with all the relevant technologies and measure. Be aware that you want high performance end to end, not just of the messaging part. At times, it makes sense to actively throw away messages (of the non-durable, published kind) to help a server come online faster especially after a restart.

Thus ends the tale of another “benchmark”.

Here’s a taste:

UI Thread and Concurrency Management

Whenever you use hosting on the UI thread, deadlocks are possible. For example, the following setup is guaranteed to result with a deadlock: A Windows Forms application is hosting a service with UseSynchronizationContext set to true, and UI thread affinity is established. The Windows Forms application then calls the service over one of its endpoints. The call to the service blocks the UI thread, while WCF posts a message to the UI thread to invoke the service. That message is never processed, because of the blocking UI thread—hence, the deadlock.

Another possible case for a deadlock occurs when a Windows Forms application is hosting a service with UseSynchronizationContext set to true and UI thread affinity is established. The service receives a call from a remote client. That call is marshaled to the UI thread and is eventually executed on that thread. If the service is allowed to call out to another service, that can result in a deadlock if the callout causality tries somehow to update the UI or call back to the service’s endpoint, because all of the service instances that are associated with any endpoint (regardless of the service-instancing mode) share the same UI thread.

Similarly, you risk a deadlock if the service is configured for reentrancy and it calls back to its client. You risk a deadlock if the callback causality tries to update the UI or enter the service, because that reentrance must be marshaled to the blocked UI thread.

Actually, I have difficulty believing that Juval would go so far as to suggest that even the forms should be services, but he does:

Form as a Service

The main motivation for hosting a WCF service on the UI thread is if the service must update the UI or the form. The problem is always: How does the service reach out and obtain a reference to the form? While the techniques and ideas that appear thus far in the listings certainly work, it would be simpler yet if the form were the service and hosted itself. For this to work, the form (or any window) must be a singleton service. The reason is that singleton is the only instancing mode that enables you to provide WCF with a live instance to host. In addition, you would not want a per-call form that exists only during a client call (which is usually very brief), nor would you want a per-session form that only a single client can establish a session with and update.

When a form is also a service, having that form as a singleton service is the best instancing mode all around.

I think that this article serves as a great treatise leading to only one conclusion – you’d have to be crazy to try to do this without some higher level framework, preferably with a different low-level framework too . Sucks Microsoft didn’t put one out – nor is there a pending beta, CTP, or even word about some project with a codename handling this. From what I know about Prism, it doesn’t intend to handle this issue either.

One thing that isn’t covered in the article is that if you do choose not to tie the client-side service to the UI thread, you open yourself up to race conditions. Reasons you’d want to handle messages on a different thread center around UI responsiveness. I’ve written about these things before:

• Object Builder – the place to fix system-wide threading bugs

• Eureka! AOP is the final piece of the multi-threaded smart client puzzle

• What Makes Smart Clients Safe?

The more I read things like this, the more I feel that I have to get going with my nServiceBus based solution. I’m fairly swamped as it is, so if anyone is interested in helping get this project off the ground, I’d be most grateful – as I think anyone else that had to build a smart client would.

SendLocal

First of all, the effect of calling SendLocal on IBus takes all the logical messages passed in (params IMessage[] messages), wraps them in a single TransportMessage, and puts that physical message at the end of the local queue. This call is equivalent to calling “Send(TransportMessage m, string destination);” on ITransport when passing in transport.Address as the parameter of destination.

There are numerous advantages to having this method, but one is the most important.

When client send a service a set of messages using “void Send(params IMessage[] messages);”, the client is requesting that the server treat this batch of messages as a unit of work. Under certain conditions, the service may choose to ignore the clients wishes – not least of which because the client has sent a ton of messages and the service doesn’t want ACID transactions to last a long time as they hurt throughput. In this case the server would use an intercepting message handler to go over those messages and call SendLocal for each. In other words, the server can set up units of work as it sees fit – taking into account client preference as well.

Other advantages include the ability to break apart complex or long-running logic into an “internal pipeline”. The Timeout Manager also makes use of this facility for “holding onto” messages until some condition occurs.

Return(errorCode)

The reason that integers are used as error codes is just so that you can push enums through them. This is the simplest way to get errors back to the client. More importantly, we take into account who on the client would be interested in this data.

Clients are often built using MVC with an additional Service Agent layer. Service Agents deal with translating the intent of Controllers into messages. Controllers don’t know about messaging, nor should they. However, they need to know when something fails with calls they initiated. As such, they are the final consumer of these error-code-enums, and integers are used to express them; that way Controllers don’t need to take a dependency on nServiceBus.

DoNotContinueDispatchingCurrentMessageToHandlers

This method on bus is used by intercepting message handlers in order to instruct the bus not to pass the current message on to subsequent handlers in the pipeline. This is often used by authentication and authorization handlers when those checks fail. This is what makes the message handling pipeline possible.

BuildAndDispatch

This method is defined on IBuilder and is used by the bus when dispatching messages to handlers. The reason that this exists instead of just having the bus ask the builder to create the handler and dispatch the call itself has to do with client-side threading. You can find the full explanation here – Object Builder, the place to fix system-wide threading bugs.

Summary

NServiceBus has grown over the years in environments where I’ve had the luxury of deciding most, if not all of the design of the systems involved. As such, it has taken on just the responsibilities needed from infrastructure in order to develop robust, flexible, and scalable systems. Check out the nServiceBus site.

Note to those who didn’t attend the session: you didn’t hear me say it, so I’ll repeat it: I hate WSDL almost as much as I hate Las Vegas. Ask me why sometime, or if I get enough of a critical mass of questions, I’ll blog it. If you’ve seen me do talks on Web Services, though, you’ve probably heard the rant: WSDL creates tightly-coupled endpoints precisely where loose coupling is necessary, WSDL encourages schema definitions that are inflexible and unevolvable, and WSDL intrinsically assumes a synchronous client-server invocation model that doesn’t really meet the scalability or feature needs of the modern enterprise. And that’s just for starters.

I hate WSDL.

I still hate Vegas more, though.

Web Services, and WSDL by connection have taken hold of the industry like cancer – inhibiting the minds of otherwise intelligent developers and architects. Whenever I get the “Web Services Question” (Does X support Web Services – where X is some design pattern, tool, and sometimes nServiceBus), I have to suppress an urge to groan – I’ve got the question that many times. The other day I was at a client and Sam, their head architect asked me that question. I gave my stock response:

“When you say ‘Web Services’, are you referring to SOAP or WSDL, and is HTTP a necessary component too?”

See how good I got at the suppressing thing?

Sam conceded that Web Services over TCP is OK too, so I pressed on with:

“What about UDP? FTP? MSMQ? Is it still ‘Web Services’ then? Is the rule then that ‘Web Services’ == SOAP?”

At that point, Sam was beginning to get a little flustered.

“And what’s so great about SOAP? Is it the interoperability? Because that’s just because it’s based on XSD.”

He didn’t know how to reply. Instead, he walked away from the whiteboard and sat down. I didn’t let up:

“And what if we want to do something other than Request/Response? How about one request with many responses? How about many requests and one response? And why does this decision need to be rigid? Shouldn’t we just be able to decide programmatically how many responses we want to return? Wouldn’t that flexibility be better than creating huge response structures for web methods to return?”

Sam made his last stand:

“Look, we can’t go and do something different from the rest of the industry. Everybody else is doing Web Services. It’s not like the technology doesn’t work.”

I gave way, a little:

“If you want, we can offer two interfaces. One, the flexible, robust, scalable XSD over messaging based solution. The second, an icky, synchronous Web Services facade which calls into our first interface.

I’m not saying that the technology doesn’t work – but both of us know that every problem has multiple solutions, some are fragile and error prone like WS, others are more elegant and have decades of knowledge behind them like messaging.

But we can do both if you like. How’s that?”

And it was agreed. The entire system would be built on one-way messaging patterns using XSD in cases where interoperability was required. And WS would be layered on, like a tiny little pig on top of a gigantic lipstick … thing – hmm, that metaphor isn’t really working – well, you get the idea.

I hate WSDL. Never been to Vegas, though.

So, I’ve started thinking what else should be included so that you get the most out of them.

Here’s what I’ve currently got in mind:

• All the powerpoint presentations
• Full source and samples of nServiceBus so you can run the code as I talk about it
• 4 hours of online consultation so you can get up to speed quickly in applying these principles and practices to your project

If you have anything else in mind that you think will help, please drop me a comment below.

So far, 32 people have expressed interest in getting this and it looks like I should be able to handle up to about 40 in a timely manner with my current setup. I hadn’t originally thought about corporate licensing, but since there have been some requests (so that all employees can use the information freely, get copies of the DVD’s, etc), I’ll be doing that too. If you’ve already left a comment about your interest in the DVD’s, I’ll assume you want the personal option. If you want to change that to the corporate option, please leave a comment either here or on the previous post.

Pricing

Ayende mentioned that the guys from Dot Net Rocks are selling Sharepoint training for about $700 per developer for a day of training. I really don’t know how much to charge for this – the guys in Australia paid quite a bit and I wouldn’t want them to feel – well, like you feel when you find out that the guy sitting next to you on the plane paid half what you did for his ticket; especially given that they’ve done so much of the production work on the DVD’s (Simon, Brad – you guys are my heroes. I really couldn’t do this without you).

So, I figure that I’ll hear what you guys think that the package is worth first. Leave me comment, or send me an email if you’d like more anonymity.

I’d also be interested in hearing what kind of domain you’re thinking about applying this stuff to – I might be able to put you in touch with people already applying SOA and nServiceBus in those areas so you can learn from each other as well.

Looking forward to hearing from you.

NServiceBus Tutorial

I gave a full day tutorial on nServiceBus and we had a full house! The tutorial was about 90% how to think about distributed systems, and 10% mapping those concepts onto nServiceBus. I made an effort to cram about 3 days of a 5 day training course I give clients into one day, but I think I was only about 85% successful. People didn’t have the time needed to let things really sink in and ask questions, but the lively forums and skype conversations available will probably do the trick.

Jim Webber after looking at the unit testing features of nServiceBus had this to say:

“Oh my God – you’ve created testable middleware! It’ll never catch on. The vendors won’t have it.”

To which I replied that several vendors were already coming on board with their own implementations of transports and saga persistence. I have absolutely no intention, desire, or (quite frankly) the ability to write an enterprise-class middleware runtime. All I hope to do with nServiceBus is to make it so that developers use what’s out there in one, middleware-product-agnostic way that will make their code more robust and flexible.

MEST & Mark – REST & Stefan

It was also great finally meeting the head MESTian, Mark Little, who also happens to work for Redhat as SOA Technical Development Manager and Director of Standards in the JBoss division. It was interesting to see the difference between how I went about messaging in nServiceBus (full peer-to-peer including pub/sub) whereas most of the Java world has the messaging infrastructure handled by something database-like in a deployment/networking kind of perspective. If that’s the way things are done, then I can definitely appreciate the advantages of Space-Based Architectures.

And I even got to steal Stefan Tilkov’s RESTful ear for an hour or so before I had to jet back home. It looks like we MESTians and RESTians can be one big happy family. I’m guessing that our despise of WS connects us all at a deeper level

Core Design Principles

I also gave a talk about core design principles, “Intentions & Interfaces – making patterns concrete”, and it went over very well especially considering that that was the first time that I gave that talk. You can find the slides here. From the feedback I heard after the talk, I think many people were surprised how many different parts of a system can be designed this way, and how flexible it is without making the code any more complex. The message was this:

Make Roles Explicit

Despite its simplicity, that leads to IEntity, IValidator<T> where T : IEntity, (which I wrote about a year ago – generic validation) and with a bit of Service Locator capabilities, you can add a line of code to your infrastructure that will validate all entities before they’re sent from the client to the server.

It leads to IFetchingStrategy<T> for improved database loading performance (also a year old – better DDD implementation and the NHibernate implementation).

It’s also how nServiceBus does message handling – IMessage, IMessageHandler<T> where T: IMessage, ISaga<T> where T : IMessage.

San Francisco?

Just a quick shout to my readers in the San Francisco area, if you’d be interested in hearing these talks/tutorials, give the organizers of QCon a shout and they’ll bring me out. That’s actually what got me to London – one of the attendees of a talk I gave at Oredev in Sweden last November missed my tutorial there so he put in a request and that did it. (Thanks Jan, I appreciate it!)

If you’re in a different part of the world and you’d like to have me give one of these talks, or other ones (I have a fair amount of material on Domain Models/DDD and Occasionally Connected Smart Clients), I’d be happy to make the trip and see you there as well.

It’s not “done” yet, but I’m pretty sure it’s past time that nServiceBus had its own site separate from this blog. I’m still working out the DNS and other domain forwarding and hosting stuff, but we’re live.

There is some information on the “overview” page about one-way messaging, store-and-forard, and why those patterns were chosen for nServiceBus.

Check it out.

When I was at TechEd Barcelona last November (07), the morning after I flew in I experienced “the fish bowl” and Virtual TechEd for the first time. Anyway, after a short chat – and quite to my surprise, my interviewer, Paul Foster, decided that we should talk about nServiceBus.

So here it is. The Microsoft/Marketing friendly description of what nServiceBus is and how nicely it plays with things like WCF and WF. Always be a gracious guest. Don’t bite the hand that feeds you. But a nibble here and there – well, that you can get away with

Download:

Hi Udi, I think  I see the point about the domain-driven approach but I’m wondering what my messages will look like. If it’s this:

IAppointment InsertInterview(Guid recruiterId, Guid applicantId, Guid appointmentId); OR

IRecuiter UpdateRecuiter(IRecuiter recruiter); (passing in an operation flag attached to the IRecuiter object) OR

IRecuiter UpdateRecuiter(IRecuiter recruiter); (setting a state flag on the relevant object and have the business object check the flag and behave according the state change)

Hope I’m not way off

Sean

Well, Sean, first of all – messages don’t look like functions. They’re a lot more like structures – data transfer objects. In this case, you’d probably be looking at a ScheduleInterviewMessage that had the relevant fields. It would look something like this:

   1:  using System;

   2:  using NServiceBus;

   3:  using System.Xml.Serialization;

   4:   

   5:  namespace Messages

   6:  {

   7:      [Serializable]

   8:      [Recoverable]

   9:      [TimeToBeReceived("0:01:00.000")]

  10:      public class ScheduleInterviewMessage : IMessage

  11:      {

  12:          public Guid InterviewerId;

  13:          public Guid CandidateId;

  14:          public DateTime RequestedTime;

  15:   

  16:          [XmlAnyElement]

  17:          public object extra;

  18:      }

  19:  }

Before we go on, I want to explain what we see. The “recoverable” attribute is the way we indicate to the infrastructure that these messages should not be lost in case a server fails, there are network problems, etc. In essence, it does durable, store-and-forward messaging. This will create an environment in which, in the case of network problems, these messages will be written to disk. That’s a good thing, since once connectivity comes back or the server boots back up, the messages will still be around and can be sent.

Now these messages are fairly small, so even at a relatively high load, we shouldn’t be chewing through too much of our expensive, small, high performance local disks. However, if these messages were bigger, we may fill up our disks before connectivity comes back, and we all know what happens to Windows boxes when there’s no room on the file system left:

In order to prevent our system from Denial-of-Servicing itself we need to make those messages clean themselves up. That’s what the “TimeToBeReceived” attribute is for. The amount of time that if a message had not yet been received by the other side that it will be deleted. This could be that the message even made it to the other machine, but the process handling those messages was down. You wouldn’t want to be filling the other side’s disk either causing them to crash, would you? This protects both parties.

The way to figure out how long to set is by looking at the smallest amount of durable storage you have available at your nodes, divide that by the size of the average message, and then again by the rate you need to process messages – and leave yourself at least 100% spare.

In other words, to build a robust system you not only will need to deal with lost messages, but you will be actively throwing messages away.

Finally, that last “XmlAnyElement” attribute is there for versioning. As we version our system and schema, we’ll be adding fields to the message. However, an old client may be talking to a new server, or vice versa. Since we wouldn’t want data to get lost just because of serialization. In a future post, I’ll show how to set up a message handler pipeline exactly for these issues.

Now that we’ve covered all the intricacies around messaging, we can see how the code that handles that above message looks:

   1:  using System;

   2:  using Messages;

   3:  using NServiceBus;

   4:  using NHibernate;

   5:   

   6:  namespace Server

   7:  {

   8:      public class ScheduleInterviewMessageHandler :

   9:                   BaseMessageHandler<ScheduleInterviewMessage>

  10:      {

  11:          public override void Handle(ScheduleInterviewMessage message)

  12:          {

  13:              using (ISession session = SessionFactory.OpenSession())

  14:              using (ITransaction tx = session.BeginTransaction())

  15:              {

  16:                  ICandidateInterviewer interviewer = session.Get<ICandidateInterviewer>(

  17:                          message.InterviewerId);

  18:                  ICandidate candidate = session.Get<ICandidate>(

  19:                          message.CandidateId);

  20:   

  21:                  interviewer.ScheduleInterviewWith(candidate)

  22:                          .At(message.RequestedTime);

  23:   

  24:                  tx.Commit();

  25:              }

  26:   

  27:              // publish new appointment data

  28:          }

  29:      }

  30:  }

If you’ve read this far and have more questions, please feel free to send them my way. If you’re at a more time-critical part of your project and need an answer quickly, we can set up a skype call. This has been working quite well for many of my overseas clients (shout out to the guys in Ireland and Florida).

Until next time

Some important features:

• TimeoutMessages now handled by separate endpoint
  • allows for scaling out timeout handling – can use a distributor behind that endpoint
  • improves performance of other endpoints – less messages to handle
• When sagas complete, TimeoutMessages for those sagas get cleared as well
  • improves throughput when sagas that can take a long time complete quickly
  • this is done by sending a TimeoutMessage with a Clear flag
• Testing environment for sagas (example available)
  • Reminder class removed to make testing more explicit
  • Timeout Manager address now configured like all other messages
• Change number of worker threads for an endpoint dynamically
  • After finding a bottleneck (most messages in queue), can increase the number of resources allocated to that endpoint by sending a message
  • Enables writing scripts which monitor all endpoints and use rules to allocate resources at runtime – like a grid
  • Simple UI available for monitoring queues and allocating threads

• Configure queues by using transport independent format:  queueName@MachineName
• SagaMessageHandler designed to be inherited from and overridden for simpler business process handling – see saga sample message handlers
• One Message Handler class can now handle more than one type of message without needing to handle IMessage
• Even when configured incorrectly, messages don’t get lost – they’re forwarded to an error queue, which is how admins will know to fix the config, and then can replay those messages again.

You might enjoy reading this as a comparison for how WF tries to support multiple hosts running the same process.

Here’s the download link again – download here.

I was wondering if any of my loyal subscribers would be interested in getting a set of DVDs containing Udi talking for hours and hours about how to identify services, map out cross-service business processes, zero in on business fracture points to further decompose services into business components, and decompose those into autonomous components by analyzing non-functional message properties,  summing up with using all that information for choosing cost-effective technologies for each autonomous component.

In other words, get 5 days of training you can pause, think about, and replay as many times as you need. There’s something for almost every phase of an enterprise project, from top level architecture, through coding, testing, to deployment tips and monitoring tactics, so you can pick up what you need – right when you need it.

Please bear with me as I get the processes in place for getting this out.

I’m wondering – how valuable do you think it would be to have weekly live online Q&A sessions as opposed to the more asynchronous (and scalable) simple forum thing?

Just so I can see what I need to be preparing myself for, please leave a comment below expressing your interest. If you also know someone else who might benefit from this, drop them a link. The last thing I want to have happen is for this to take months and months to get out because I didn’t prepare things in advance that I could have.

And a big thanks to Simon and the gang in Australia for helping make this happen. It was a great two weeks and I thank you for that.

Here’s the core list from Joe Armstrong’s book, Programming Erlang:

“In Erlang:

• Creating and destroying processes is very fast.
• Sending messages between processes is very fast.
• Processess behave the same way on all operating systems.
• We can have very large numbers of processes.
• Processes share no memory and are completely independent.
• The only way for processes to interact is through message passing.”

In nServiceBus, we don’t create or destroy processes – that’s a Windows issue. Instead, we just do messaging with endpoints. If there’s a process behind that endpoint, and it responds, then other interesting things can occur.

In the continued list:

• Message passing is asynchronous.
• Processes can monitor each other.
• It is possible to selectively receive messages.
• Remote processes appear largely the same as local processes.

All of this is part of the design philosophy of nServiceBus. While I have yet to see a carrier-grade implementation of nServiceBus, we are enjoying very impressive system-wide uptimes in production. Oh, and the programming model is still plain-old .NET, so you don’t have to learn any new languages or environments (even though I think that you might learn something – I know I did).