What’s RavenDB?

For those of you who haven’t heard about RavenDB yet – it’s a transactional document database for .NET, and it’s been/being developed by my good friend and partner in crime Ayende Rahien (a.k.a Oren Eini). Although Ayende is known for his “not invented here” tendencies, when it comes to transactional document databases, well, you’d have been hard pressed to find one – especially with a decent .NET API.

NServiceBus Storage

In NServiceBus we have a variety of storage needs – from things like durable subscriptions so that you get fault-tolerant pub/sub, through persistence of long-running workflow (a.k.a saga) state, and durable timeouts so that your time-bound long-running processes never get stuck. None of these actually require relations – we were just using relational databases for storage because it was the easy answer everyone was going with.

Relational vs. Document

And these relational databases came with some downsides – developers had to “beg” their DBA to create the needed tables in the production databases, when that central database was down it prevented otherwise autonomous publishers and subscribers from going about their business, and it was very difficult to version the long-running workflows especially when newer versions required different state.

By moving to an embedded and transactional document DB, no longer do you need to bargain with the DBA, by keeping the storage together with the processing nodes you are back in parallel processing bliss, and the schema-less nature of the storage makes versioning those long-running workflows much easier.

Licensing

And I’m also happy to announce that an agreement has been reached with Hibernating Rhinos (Ayende’s company) so you won’t need to license RavenDB separately to get all of these benefits. RavenDB will be bundled with NServiceBus so licensing NServiceBus covers them both (for the storage needs mentioned above).

If you’re also thinking about using RavenDB as the storage for the rest of your system, you’ll be able to get a nice discount on the license when purchasing it together with NServiceBus. This will be going into effect with the release of NServiceBus 3.0 this October.

Companies Using NServiceBus

Interestingly enough, there’s been a big uptick in companies using NServiceBus with the introduction of licensing. Most of these companies are not the kind of big, stand-up-and-take-notice names that everybody likes to have on their roster.

What makes NServiceBus particularly attractive is that you can get a lot done without requiring some kind of dedicated BizTalk/WebSphere/Tibco expert. This has brought down the barrier for thousands of developers who just want to get on with the business of getting their app to market.

And when it comes to the big names, well, once they see how much faster they can get stuff done with NServiceBus as well as how robust and scalable it is in production, they don’t want any of their competitors to know about it!

Anyway, I’m glad to say that two companies have stepped forwards: Rackspace and Reuters. Hopefully we’ll get confirmation from one of the big banks soon that we can go public with them too.

Exciting times ahead.

The question

The main touted benefit of these workflow-centric architectures is that we don’t have to change the code of the system in order to change its behavior resulting in ultimate flexibility!

Some of you may have already gone down this path and are shaking your heads remembering how your particular road to hell was paved with the exact same good intentions.

Let me explain why these things tend to go horribly wrong.

What’s behind the curtain

It starts with the very nature of workflow – a flow chart, is procedural in nature. First do this, then that, if this, then that, etc. As we’ve experienced first hand in our industry, procedural programming is fine for smaller problems but isn’t powerful enough to handle larger problems. That’s why we’ve come up with object-oriented programming.

I have yet to see an object-oriented workflow drag-and-drop engine. Yes, it works great for simple demo-ware apps. But if you try to through your most complex and volatile business logic at it, it will become a big tangled ball of spaghetti – just like if you were using text rather than pictures to code it.

And that’s one of the fundamental fallacies about these tools – you are still writing code. The fact that it doesn’t look like the rest of your code doesn’t change that fact. Changing the definition of your workflow in the tool IS changing your code.

On productivity

Sometimes people mention how much more productive it would be to use these tools than to write the code “by hand”. Occasionally I hear about an attempt to have “the business” use these tools to change the workflows themselves – without the involvement of developers (”imagine how much faster we could go without those pesky developers!”).

For those of us who have experienced this first-hand, we know that’s all wrong.

If “the business” is changing the workflows without developer involvement, invariably something breaks, and then they don’t know what to do. They haven’t been trained to think the way that developers have – they don’t really know how to debug. So the developers are brought back in anyway and from that point on, the business is once again giving requirements and the devs are the one implementing it.

Now when it comes to developer productivity, I can tell you that the keyboard is at least 10x more productive than the mouse. I can bang out an if statement in code much faster than draggy-dropping a diamond on the canvas, and two other activities for each side of the clause.

On maintainability

Sometimes the visualization of the workflow is presented as being much more maintainable than “regular code”.

When these workflows get to be to big/nested/reused, it ends up looking like the wiring diagram of an Intel chip (or worse). Check out the following diagram taken from the DailyWTF on a customer friendly system:

The bigger these get, the less maintainable they are.

Now, some would push back on this saying that a method with 10,000 lines of code in it may be just as bad, if not worse. The thing is that these workflow tools guide developers down a path where it is very likely to end up with big, monolithic, procedural, nested code. When working in real code, we know we need to take responsibility for the cleanliness of our code using object-orientation, patterns, etc and refactoring things when they get too messy.

Here is where I’d bring up the SOA/pub-sub approach as an alternative – there is no longer this idea of a centralized anything. You have small pieces of code, each encapsulating a single business responsibility, working in concert with each other – reacting to each others events.

Productivity take 2: testing and version control

If you’re going to take your most complex and volatile business logic and put it into these workflow tools, have you thought about how your going to test it? How do you know that it works correctly? It tends to be VERY difficult to unit-test these kinds of workflows.

When a developer is implementing a change request, how do they know what other workflows might have been broken? Do they have to manually go through each and every scenario in the system to find out? How’s that for productivity?

Assuming something did break and the developer wants to see a diff – what’s different in the new workflow from the old one, what would that look like? When working with a team, the ability to diff and merge code is at the base of the overall team productivity.

What would happen to your team if you couldn’t diff or merge code anymore?
In this day and age, it should be considered irresponsible to develop without these version control basics.

In closing

There are some cases where these tools might make sense, but those tend to be much more rare than you’d expect (and there are usually better alternatives anyway). Regardless, the architectural analysis should start without the assumption of centralized workflow, database, or centralized anything for that matter.

If someone tries to push one of these tools/architectures on you, don’t walk away – run!

The ugly truth

You don’t need an ESB for integration.

There. I’ve said it.

Most of the ESB products on the market, in focusing on integration, are addressing the wrong problem.

What integration is about

There are three primary components to integration – data format translation, protocol bridging, and logic. Let’s take the Agile “simplest solution that could possibly work” motto and apply it to these elements:

1. Data format translation
   
   • Use something like MapForce (from Altova, the guys behind XMLSpy).
   • At under $1200 for a dev license and handling mapping to/from XML, EDI, flat files, and relational databases, you can host the resulting mapping in any endpoint, as XSLT or even Java/C# – no need to have “the bus” do this for you.
2. Protocol bridging
   
   • Use something like /n software
   • At about $1500 for a dev license, you get solutions for FTP, HTTP, SMTP, POP, IMAP, LDAP, DNS, RSS, SMS, Jabber, SOAP, WebDav, etc and, again – you can host the DLLs in any endpoint you want. Don’t need expensive buses for this
3. Logic
   
   This is all you – no technology can do this for you. If anything, the most important thing is to get all the ugly mapping and protocol bridging stuff out of the way and let you focus on your logic. This is Single Responsibility Principle (un)common sense.

Digging deeper into the logic

The interesting thing about most of the protocol stuff mentioned above is that they’re inherently unreliable and also their performance at runtime is unknowable due to the total load on the target server.

We wouldn’t want any of the above situations to cause our integration to “get stuck”. As such, it is best we think of our integration logic as a long-running process that manages other endpoints which do the actual protocol bridging and data transformations.

This is one of the areas where NServiceBus, with its sagas can actually help a lot. Just like the other pieces of integration mentioned above, these sagas can run on any endpoint.

You could alternatively look at other technologies like BizTalk and Business Process Execution Language (BPEL) engines, though many of those are designed to be physically centralized (just like many of the ESBs out there).

By the way, if you do want to use NServiceBus together with BizTalk, Michael Stephenson has published some great white papers on getting the two to work together. His latest is about integrating NServiceBus into BizTalk’s RFID processes – check it out.

Putting it all together

When we take all of these pieces and look at them in a cohesive architecture, here’s what it can look like:

As you can see, integration is physically distributed across multiple endpoints.

Not only that, but the integration logic is kept separate from the protocol bridging and data transformation, enabling independent versioning of each. Just as important, it makes it much easier to unit test that the integration logic is correct as we don’t have to simulate the target technologies.

Costs

As you can see, you can get integration capabilities just as powerful as if you went with something like BizTalk, but without creating a single point of failure in your architecture. In terms of costs, it’s also quite a bit cheaper.

For a high availability BizTalk deployment, you’ll be paying over $40,000 per CPU for the Enterprise Edition, not including the extra $7000 per CPU for SQL Server Standard Edition (clustered). For a clustered 4-CPU mid-size deployment, you’d be in the area of $200,000.

For the distributed integration solution above, you’d be paying around $2700 in dev licenses (for /n software and Altova MapForce), and $500 per core for NServiceBus Standard Edition. The reason there’s per-core licensing for NServiceBus is that in a virtualized environment, you’ll be provisioning virtual cores to your virtual machines. No reason to pay for a quad-core CPU when all you’re using is a single core. You can also use any number of cores running NServiceBus Express Edition at no cost, so a mid-size deployment with say 8 cores running Standard Edition, and another 24 cores running Express Edition would cost (with the volume discount) an additional $3800 – a total cost of $6,500.

BizTalk: $200,000       NServiceBus: $6,500

‘Nuff said.

In closing

We’ve been bitten by centralized architectures before.

By having our integration distributed, we can version, upgrade, and scale each piece independently.

You’ve seen how we can use simple, lightweight, and inexpensive technologies to create distributed integration solutions just as powerful and robust as the centralized ESB vendor-offerings out there, but at a tiny fraction of the cost.

Next time the topic of integration is brought up, you’ll know not to be suckered in by re-branded EAI brokers.

Learn more about NServiceBus.

Brokers

Message brokers, more broadly known and used on the Java platform, don’t come with this constraint. For example, when using ActiveMQ, you can have any number of endpoints come to the broker and publish a message under a given topic.

So where’s the problem?

It’s all about accountability.

Let’s say you’ve subscribed to a given topic, and have received two events – one telling you that the price of bananas next week will be $1/kg and another telling you that it’ll be $2/kg.

Which one is right?

Especially given that those events may have been published by any other endpoint via the broker.

Is it first one wins? Last one wins? How about first one sent vs. first one received? Ditto for last. As a subscriber, can you really be held accountable for having the logic to choose the right one? Shouldn’t this responsibility have fallen to the publishing side?

This is one of the big drawbacks of the broker, hub and spoke architecture. No responsibility. No single source of truth – unless everybody’s going to some central database, in which case – what’s the point of all this messaging anyway?

Buses

The Bus Architectural Style is all about accountability. If you are going to publish an event, you are accountable for the correctness of the data in that event – there is no central database that a subscriber can go to “just in case”. And the only way that you can be held accountable, is if you have full responsibility – ergo, you’re the only one who can publish that type of event.

If you say bananas are going to cost $1/kg next week, that’s that. Subscribers will not hear from anybody else on that topic.

Now, this is not to say that you can’t have more than one physical publishing endpoint.

You see, buses differentiate between the logical and the physical. Brokers tend to assume that the physical hub-and-spoke topology is also the logical.

In a bus, while there can only be one logical endpoint publishing a given type of event, that endpoint can be physically scaled out across multiple machines. It is the responsibility of the bus to provide infrastructure facilities to allow for that to happen in such a way that to subscribers, it still appears as if there is really only one publishing endpoint.

The same is true about the subscriber – one logical subscribing endpoint may be scaled out across multiple machines.

Product Mix-ups

Unfortunately, there are many broker-style technologies out there that are being marketed under the banner of the Enterprise Service Bus. While some products have the ability to be deployed in both a centralized and distributed fashion (sometimes called “federated” or “embedded” mode), many do not enforce the “single publishing endpoint per event-type” rule.

Without this constraint, it is just too easy to make mistakes.

NServiceBus

By enforcing this constraint, we see the same kind of question appear on the discussion group time and time again:

“I have an Audit event that I’d like all of my machines to publish, and have one machine subscribe to them all, but NServiceBus won’t let me. How do I make NServiceBus support this scenario?”

And the answer is the same every time:

“You should have all the machines Send the Audit message (configured to go to the single machine handling that message), and not Publish. It is not an event until its been handled by the endpoint responsible for it.”

The semantics of the message matter a lot.

When looking at Service-Oriented Architecture, these messages are the contract and, as any lawyer will tell you, contracts need to be explicit and the intentions really need to be spelled out – otherwise the contract is practically worthless.

In closing

Friction is sometimes a good thing – it prevents us from making mistakes. It keeps cars on the road. And because that’s not enough friction, we introduce curbs as well.

If you’re looking for a service bus technology for your next project, check that it’ll give you the friction that you need to keep everybody safe. Really check what it is that the vendors are offering you – more often than not, it’s some ESB lipstick on a broker pig.

To learn more about how NServiceBus supports this kind of publish/subscribe, click here.

This will actually be part 2 of his previous presentation – you can find the recording of part 1 here.

This time Andreas will be showing publish/subscribe communication as well as the use of sagas – get the full details here.

Messaging Basics

First of all, when building a system using messaging, you don’t have methods that are invoked on some remote object (a.k.a “service”) to which you pass parameters. Instead, you use some generic piece of infrastructure (in the world of Java, this is most commonly a Message Broker) to send a message where a message can be thought of as a serializable class. Here’s an example of a message:

public class UserCreated : IMessage
{
    public Guid UserId { get; set; }
    public string Name { get; set; }
}

This message would be published using NServiceBus like this:

bus.Publish<UserCreated>( m =>
{
    m.UserId = Guid.NewGuid();
    m.Name = "John Smith";
});

This can be contrasted with RPC models like WCF where you need to define a “service” that has methods on it, where those methods accepts parameters. Sometimes developers try to make this more generic by having a single “service” with one method on it named something like “Process” where the parameter it accepts is of the type “object”, or they introduce generics like this: Process<T>(T message);

This is where Polymorphic Message Dispatch comes in:

Polymorphic Message Dispatch

While you can pull of the WCF generics thing, one thing that is more difficult (without writing your own dispatch model) is to have a pipeline of classes which can be invoked based on their relationship to the type passed in. Using NServiceBus, both of the following message handlers will be invoked when UserCreated arrives:

public class Persistence : IHandleMessages<UserCreated>
{
    public void Handle(UserCreated message) { }
}

public class Audit : IHandleMessages<IMessage>
{
    public void Handle(IMessage message) { }
}

Now some might say that WCF, BizTalk, and the .NET Service Bus allow you to do auditing in their own internal pipeline, and that’s true. The place where this becomes more powerful is when you need to build V2 of your system, and the publisher now publishes a slightly different event – that a user was created as a part of a campaign, requiring the subscriber to register statistics about the campaign. Of course, this event also means that a user was created. Here’s how you’d do that with NServiceBus:

public class UserCreatedFromCampaign : UserCreated
{
    public Guid CampaignId { get; set; }
}

//publisher code
bus.Publish<UserCreatedFromCampaign>( m =>
{
    m.UserId = Guid.NewGuid();
    m.Name = "John Smith";
    m.CampaignId = theCampaignId;
}

//subscriber code
public class Statistics : IHandleMessages<UserCreatedFromCampaign>
{
    public void Handle(UserCreatedFromCampaign message) { }
}

The important part is what you don’t see – since UserCreatedFromCampaign inherits from UserCrated, the Persistence handler we had from V1 will also be invoked, and so will the Audit handler of course. You don’t have to make your new code call the old code like you would in a method based dispatch model. This makes sure that the coupling in your service layer code remains constant over time as you grow the functionality of your system.

This was one of the main benefits mentioned by Rackspace in their use of NServiceBus (here):

“The main benefit NServiceBus has brought us so far is developer scalability due to lower coupling and higher consistency in our code.”

But, when looking at the above scenario, we can obviously expect that all sorts of things can happen in relation to campaigns – it is a separate concern, and thus should be handled by a separate subscriber. And this bring us to…

Polymorphic Message Routing

The challenge that we have here is that we no longer have a hierarchy where something clearly belongs on top of something else. We have users created and activities happening related to campaigns – that may happen in any combination. By having separate subscribers, we could then introduce new handlers/subscribers to our environment without touching or taking down any of the other subscribers. Here’s what the subscribers would look like:

public class Persistence : IHandleMessages<UserCreated>
{
    public void Handle(UserCreated message) { }
}

public class Statistics : IHandleMessages<CampaignActivityOccurred>
{
    public void Handle(CampaignActivityOccurred message) { }
}

But if each of the above messages were a class, how could we define a message which inherited from both?

Before answering that, we need to understand why the publisher wouldn’t just publish both of the above messages. You see, the publisher can’t make any assumptions about its subscribers – it could be that one of them has logic that correlates across both of these messages that could end up counting the occurrence as happening twice rather than once, possibly charging the account associated with the campaign twice. Publishing two messages results in two transactions when there really should have been one.

So, here’s how to define messages so that we can have multiple inheritance:

public interface UserCreated : IMessage
{
    Guid UserId { get; set; }
    string Name { get; set; }
}

public interface CampaignActivityOccurred : IMessage
{
    Guid CampaignId { get; set; }
    Guid ActivityId { get; set; }
}

public interface UserCreatedFromCampaign
                 : UserCreated,
                   CampaignActivityOccurred
{
}

And when the publisher publishes UserCreatedFromCampaign, the event would be routed to both the UserCreated subscriber and the CampaignActivityOccurred subscriber. The power of this approach is felt as we handle new requirements around purchases made related to a campaign. Now we can have another event which inherits from CampaignActivityOccurred and not have to worry since the existing subscriber will be routed those messages automatically.

Since WCF doesn’t have publish/subscribe capabilities, we might as well move along.

Not to throw a burning match on an ocean of oil, but REST doesn’t really support this either.

Not Content-Based Routing

This may sound like the content-based router pattern from EIP (CBR), but it’s not. The important difference is that there isn’t some part of the routing that depends on the structure of the messages. The major drawback of CBR is that it creates a central place in your system that needs to be changed any time *syntactic* changes happen to message structure *in addition to* to changes in the subscribers.

Now, this is where the BizTalk guys would say that “that’s why we can do message transformations”, and then the subscribers wouldn’t need to be changed. However, can we really know when getting a requirement that the change is syntactic and not semantic? I mean, it’s quite common that changes to message structures happen together with changes to processing logic.

You may be beginning to get the feeling that more and more logic is being sucked out of the subscribers into some monolithic black hole that is likely going to be unmaintainable and quite slow.

This is one of the main differences between using a bus and a broker – a bus supports the correct distribution of logic keeping the system loosely coupled; brokers are useful integration engines when you absolutely can’t change the applications being integrated. Enterprise Application Integration (EAI) brokers don’t usually make good Enterprise Service Bus (ESB) technology.

In Closing

NServiceBus has all sorts of features you didn’t know you needed until you saw what life could be like when you had them. Most of these features don’t have snazzy drag-and-drop demos that make people ooh-and-aah and TechEds and PDCs, but they’re really necessary to avoid finding yourself in yet another big-ball-of-mud code base telling your manager/customer (again) that it would be faster to rebuild the system from scratch than to implement that new requirement in the old one.

Take NServiceBus for a spin and see for yourself.

Yes, there’s a new logo, and the website’s been redesigned.
It’s been a long time coming – the previous version (2.0) was released in March.

I’m really quite excited about this version as it rolls up all the bug fixes and enhancements that customers have asked for as they ran version 2.0 under the most severe types of production environments. The next thing that is a big deal that many have been asking for is a licensed version of NServiceBus – that is, the ability to purchase a commercial license and receive support.

We all know how managers like having a throat to choke.

And now they’ll have one – NServiceBus Ltd is the company that will be providing licensing, services, and support for all customers’ NServiceBus needs. After more than 33,000 downloads and over 1000 developers in the community, the demand has really grown. Who would’ve thought all this would happen when I started NServiceBus 4 years ago (before it even had a name).

Why NServiceBus is better than WCF for your distributed systems

This question comes up repeatedly for people hearing about NServiceBus for the first time.

The answer is simple – reliability.

A system built with NServiceBus is so much more reliable to all kinds of production conditions than WCF that it’s hardly a fair comparison at all. While WCF can be configured to provide something kind of close to the same level of reliability, you need to do a fair amount of spelunking through the various options of netMsmqBinding to get it right.

The second reason to use NServiceBus instead of WCF is publish/subscribe.

The ability to make use of events and the observer pattern not just to achieve loose coupling within a single process, but across many processes, machines, and sites. Can you imagine going back to programming without events? Shudder. But that’s exactly what it’s like to use WCF in your distributed system. NServiceBus brings you the best of object-oriented programming but in a distributed and reliable infrastructure.

Don’t wait any longer

Take NServiceBus for a spin.

But things may look a bit different after you do…

http://www.NServiceBus.com

And have a happy New Year.

“SOA is a practice that focuses on modeling the entities, and relationships between entities, that comprise the business as a set of services. This can be done on a small or large scale. Typically, the relationships in this model represent consumer/provider relationships.”

I have some serious concerns about the ramifications of this definition/description.

First of all, when reading “entities”, many people will interpret that to mean the entities found in Entity Relationship Diagrams [ERD] or in Object Oriented Analysis & Design [OOAD]. In both, these entities are identified as the “nouns” of the domain. Examples of these ERD/OOAD-type entities include things like Customer, Order, and Product.

These are almost always the wrong place to start for identifying services in SOA.

Second, on the consumer/provider relationship: on the one had, this fits very well with how web services can consume (or call) other web services. However, the downsides of using web services as services in SOA is becoming well enough known that even in the same post we see this warning:

“Web Services is not SOA, it is merely a standardized approach to accessing functionality on remote systems.”

But the question remains, if a producer/consumer relationship is OK for SOA-type services, why doesn’t that hold for web services? And the answer is… it depends on the type of producer/consumer relationship. The typical relationship is one of synchronous calls from consumer to producer, this is not OK for SOA-type services either.

You see, this synchronous producer/consumer implies a model where services are not able to fulfill their objectives without calling other services. In order for us to achieve the IT/Business alignment promised by SOA, we need services which are autonomous, ie. able to fulfill their objectives without that kind of external help.

Instead, we need to look for a more loosely coupled producer/consumer relationship – like publish/subscribe, where the producer emits events, and the consumer subscribes and handles those events. The reason that this kind of relationship doesn’t hurt autonomy is that it disconnects services on the dimension of time. In order for a service to be able to make a decision autonomously without synchronously calling any other service, using only information provided by events it received in the past, it must be strongly aligned with the business.

Most projects which bandy about the SOA acronym aren’t actually made up of services – they’re made up of XML over HTTP functions calling other XML over HTTP functions, eventually calling XML over HTTP databases. You can layer as much XML and HTTP as you want on top of it, but at the end of the day, most projects are just functions calling functions calling databases – in other words, procedural programming in the large, and no amount of SOAP will wash away the stink.

Here’s a different definition of services for SOA that may communicate a bit better what it’s all about:

A service is the technical authority for a specific business capability.
Any piece of data or rule must be owned by only one service.

What this means is that even when services are publishing and subscribing to each other’s events, we always know what the authoritative source of truth is for every piece of data and rule.

Also, when looking at services from the lense of business capabilities, what we see is that many user interfaces present information belonging to different capabilities – a product’s price alongside whether or not it’s in stock. In order for us to comply with the above definition of services, this leads us to an understanding that such user interfaces are actually a mashup – with each service having the fragment of the UI dealing with its particular data.

Ultimately, process boundaries like web apps, back-end, batch-processing are very poor indicators of service boundaries. We’d expect to see multiple business capabilities manifested in each of those processes.

I know that this may be more confusing than the traditional web services approach but, to paraphrase Donald Rumsfeld, it is better to know that you don’t know, than to not know that you don’t know

The problem is that, as software developers, we’re all too quick to accept them at face value. We don’t question the requirements – in all fairness, it was never our job to do so. We were the ones that implemented them, preferably quickly.

For example

Let’s say we get the requirement the following requirements:

1. If the order was already shipped, don’t let the user cancel the order.
2. If the order was already cancelled, don’t let the user ship the order.

The race condition here is when we have two users who are looking at the same order, which is neither cancelled nor shipped yet, and each submits a command – one to ship the order, the other to cancel it.

In these cases, the code is simple – just an if statement before performing the relevant command.

So what’s the problem

A microsecond difference in timing shouldn’t make a difference to core business behaviors. Which means that we’ve actually got here is a bug in the requirements. Users are actually dictating solutions here rather than requirements.

Let’s ask our stakeholders, “why shouldn’t we let users cancel a shipped order? I mean, the users don’t want the products.”

And the stakeholders would respond with something like, “well, we don’t want to refund the user’s money then. Or, at least, not all their money. Well, maybe if they return the products in their original packaging, *then* we could give a full refund.”

And as we drilled deeper, “when do refunds need to be given? Right away, in the same transaction?”

The stakeholders would explain, “no, refunds don’t need to be given right away.”

It turns out we were missing the concept of a refund, as well as assuming that all things needed to be processed and enforced immediately. Once we dug into the requirements, we found that there is actually plenty of time to allow both transactions to go through. We just need to add some checks during shipping’s long-running process to see if the order was cancelled, and then to cut the process short.

So is everything a long-running process then?

That’s actually a fair question – long-running processes are a lot more common than at first appears.

What we’re seeing is that cancellation is now a command that has no reason to fail – just like CQRS tells us. When this command is performed, it publishes the OrderCancelled event, which the billing service subscribes to.

Billing then starts a long-running process (a saga, in NServiceBus lingo), also listening to events from the shipping process, ultimately making a decision when a refund should be given, and for how much.

Deeper business analysis

As we discuss matters more with our business stakeholders, we hear that most orders are actually cancelled within an hour of being submitted. It is quite rare for orders to be cancelled days later.

In which case, we could look at modeling the acceptance of an order as a long-running process itself.

When a user places an order, we don’t immediately publish an event indicating the acceptance of an order, instead a saga is kicked off – which opens up a timeout for an hour later. If a cancellation command arrives during that period of time, the user gets a full refund (seeing as we didn’t charge anything since billing didn’t get the accepted event to begin with), and the saga just shuts itself down. If the timeout occurs an hour later, and the saga didn’t get a cancel command, then the order is actually accepted and the event is published.

Yes, sagas are everywhere, once you learn to see with business eyes, and no race conditions are left.

In closing

Any time you see requirements that indicate a race condition, dig deeper.

What you’re likely to find are some additional business concepts as well as the introduction of time and the creation of long-running business processes. The implementation at that point will pivot from being trivial if-statements to being richer sagas.

Keep an eye out.

Currently, the video is being graciously hosted by Jon Torresdal on his blog, so let’s hope that the bandwidth holds up.

Get it here.

While Mule has many features in terms of connectivity and integration, NServiceBus provides platform interop only. One could say that this is a product of the different backgrounds I and Ross come from.

On the other hand, the saga capabilities in NServiceBus for handling long-running processes are considered to be BPM functionality on the Java side of the industry, and as such, Mule does not have them.

In terms of other enterprise features like management and monitoring, Mule is more mature, but NServiceBus holds its own in terms of high availability and actually surpasses Mule with the grid and scale-out capabilities of its distributor.

Anyway, I think it’s about time each of these parts was explicitly described so that companies already invested in Java ESB tools will know what they’re getting with NServiceBus.

Until then, I hope this podcast describing the full spectrum of NServiceBus, from top level SOA services to in-the-weeds transaction management, will provide more information about what it is and why you might want to use it:

Deep Fried Bytes, episode 49 – Getting the right message about NServiceBus with Udi Dahan.

Comments most welcome.

The recording of the talk is online here.

There is one important thing that I didn’t have enough time to cover, but I want you to keep in mind as you’re watching this. It is that CQRS is applicable only *within* the context of a single service/BC – NOT across or between them.

Let me know what you think.

What’s New

The quickly approaching next release of NServiceBus will be version 2.0 and is a big step from 1.9. After 2 betas and 2 release candidates, this version has had a longer stabilization period than any of the versions so far (1.4-1.9). Many of my clients are already using it in production and are very pleased with it. I’ve heard similar reports from others in the community (now with over 500 members in the discussion group). There have been almost 10,000 downloads since the version 1.9 release and in every country I visit I meet people using NServiceBus in new and interesting applications.

With my appearance on Hanselminutes, many in the mainstream .NET industry have started taking a look at NServiceBus. That, and the fact that Microsoft’s Oslo technology has now taken a very data-driven turn (rather than its original service-oriented direction).

Interestingly enough, I’ve been hearing more and more reports about people using NServiceBus as a developer-friendly API on top of other technologies. This includes BizTalk and even Neuron. I never thought that people would take the pluggability of NServiceBus that far.

So, what is NServiceBus?

Well, it’s a service bus, y’know, like an ESB – just an open-source one.

All kidding aside, in a nutshell, it gives you an easy way to integrate transactional messaging into your applications.

One of the reasons why you might want to do that is so that you don’t lose messages containing valuable data when IIS recycles your AppDomain, every 15-20 minutes (as I wrote about in this MSDN magazine article).

There are many other nice things in there, like the ability to unit test your service layers and long-running processes but you can read more about that here…

Documentation

One of the biggest differences to NServiceBus in this release is documentation.

A lot of work has gone into the NServiceBus.com site to help developers hit the ground running with NServiceBus, including the more advanced aspects of transparent scale-out with the distributor and multi-site communications.

There is still work to be done in this area but feedback so far has been extremely positive (except for some grumblings from certain old-timers saying that if they could figure it out by themselves, well, you know the rest).

In Closing

If you’re building a distributed enterprise .NET system, take 5 minutes, download it, and see transactional publish/subscribe messaging working on your machine without any big heavy-weight middleware.

www.NServiceBus.com

Download as PDF – this is quite a long post.

Why CQRS

Before describing the details of CQRS we need to understand the two main driving forces behind it: collaboration and staleness.

Collaboration refers to circumstances under which multiple actors will be using/modifying the same set of data – whether or not the intention of the actors is actually to collaborate with each other. There are often rules which indicate which user can perform which kind of modification and modifications that may have been acceptable in one case may not be acceptable in others. We’ll give some examples shortly. Actors can be human like normal users, or automated like software.

Staleness refers to the fact that in a collaborative environment, once data has been shown to a user, that same data may have been changed by another actor – it is stale. Almost any system which makes use of a cache is serving stale data – often for performance reasons. What this means is that we cannot entirely trust our users decisions, as they could have been made based on out-of-date information.

Standard layered architectures don’t explicitly deal with either of these issues. While putting everything in the same database may be one step in the direction of handling collaboration, staleness is usually exacerbated in those architectures by the use of caches as a performance-improving afterthought.

A picture for reference

I’ve given some talks about CQRS using this diagram to explain it:

The boxes named AC are Autonomous Components. We’ll describe what makes them autonomous when discussing commands. But before we go into the complicated parts, let’s start with queries:

Queries

If the data we’re going to be showing users is stale anyway, is it really necessary to go to the master database and get it from there? Why transform those 3rd normal form structures to domain objects if we just want data – not any rule-preserving behaviors? Why transform those domain objects to DTOs to transfer them across a wire, and who said that wire has to be exactly there? Why transform those DTOs to view model objects?

In short, it looks like we’re doing a heck of a lot of unnecessary work based on the assumption that reusing code that has already been written will be easier than just solving the problem at hand. Let’s try a different approach:

How about we create an additional data store whose data can be a bit out of sync with the master database – I mean, the data we’re showing the user is stale anyway, so why not reflect in the data store itself. We’ll come up with an approach later to keep this data store more or less in sync.

Now, what would be the correct structure for this data store? How about just like the view model? One table for each view. Then our client could simply SELECT * FROM MyViewTable (or possibly pass in an ID in a where clause), and bind the result to the screen. That would be just as simple as can be. You could wrap that up with a thin facade if you feel the need, or with stored procedures, or using AutoMapper which can simply map from a data reader to your view model class. The thing is that the view model structures are already wire-friendly, so you don’t need to transform them to anything else.

You could even consider taking that data store and putting it in your web tier. It’s just as secure as an in-memory cache in your web tier. Give your web servers SELECT only permissions on those tables and you should be fine.

Query Data Storage

While you can use a regular database as your query data store it isn’t the only option. Consider that the query schema is in essence identical to your view model. You don’t have any relationships between your various view model classes, so you shouldn’t need any relationships between the tables in the query data store.

So do you actually need a relational database?

The answer is no, but for all practical purposes and due to organizational inertia, it is probably your best choice (for now).

Scaling Queries

Since your queries are now being performed off of a separate data store than your master database, and there is no assumption that the data that’s being served is 100% up to date, you can easily add more instances of these stores without worrying that they don’t contain the exact same data. The same mechanism that updates one instance can be used for many instances, as we’ll see later.

This gives you cheap horizontal scaling for your queries. Also, since your not doing nearly as much transformation, the latency per query goes down as well. Simple code is fast code.

Data modifications

Since our users are making decisions based on stale data, we need to be more discerning about which things we let through. Here’s a scenario explaining why:

Let’s say we have a customer service representative who is one the phone with a customer. This user is looking at the customer’s details on the screen and wants to make them a ‘preferred’ customer, as well as modifying their address, changing their title from Ms to Mrs, changing their last name, and indicating that they’re now married. What the user doesn’t know is that after opening the screen, an event arrived from the billing department indicating that this same customer doesn’t pay their bills – they’re delinquent. At this point, our user submits their changes.

Should we accept their changes?

Well, we should accept some of them, but not the change to ‘preferred’, since the customer is delinquent. But writing those kinds of checks is a pain – we need to do a diff on the data, infer what the changes mean, which ones are related to each other (name change, title change) and which are separate, identify which data to check against – not just compared to the data the user retrieved, but compared to the current state in the database, and then reject or accept.

Unfortunately for our users, we tend to reject the whole thing if any part of it is off. At that point, our users have to refresh their screen to get the up-to-date data, and retype in all the previous changes, hoping that this time we won’t yell at them because of an optimistic concurrency conflict.

As we get larger entities with more fields on them, we also get more actors working with those same entities, and the higher the likelihood that something will touch some attribute of them at any given time, increasing the number of concurrency conflicts.

If only there was some way for our users to provide us with the right level of granularity and intent when modifying data. That’s what commands are all about.

Commands

A core element of CQRS is rethinking the design of the user interface to enable us to capture our users’ intent such that making a customer preferred is a different unit of work for the user than indicating that the customer has moved or that they’ve gotten married. Using an Excel-like UI for data changes doesn’t capture intent, as we saw above.

We could even consider allowing our users to submit a new command even before they’ve received confirmation on the previous one. We could have a little widget on the side showing the user their pending commands, checking them off asynchronously as we receive confirmation from the server, or marking them with an X if they fail. The user could then double-click that failed task to find information about what happened.

Note that the client sends commands to the server – it doesn’t publish them. Publishing is reserved for events which state a fact – that something has happened, and that the publisher has no concern about what receivers of that event do with it.

Commands and Validation

In thinking through what could make a command fail, one topic that comes up is validation. Validation is different from business rules in that it states a context-independent fact about a command. Either a command is valid, or it isn’t. Business rules on the other hand are context dependent.

In the example we saw before, the data our customer service rep submitted was valid, it was only due to the billing event arriving earlier which required the command to be rejected. Had that billing event not arrived, the data would have been accepted.

Even though a command may be valid, there still may be reasons to reject it.

As such, validation can be performed on the client, checking that all fields required for that command are there, number and date ranges are OK, that kind of thing. The server would still validate all commands that arrive, not trusting clients to do the validation.

Rethinking UIs and commands in light of validation

The client can make of the query data store when validating commands. For example, before submitting a command that the customer has moved, we can check that the street name exists in the query data store.

At that point, we may rethink the UI and have an auto-completing text box for the street name, thus ensuring that the street name we’ll pass in the command will be valid. But why not take things a step further? Why not pass in the street ID instead of its name? Have the command represent the street not as a string, but as an ID (int, guid, whatever).

On the server side, the only reason that such a command would fail would be due to concurrency – that someone had deleted that street and that that hadn’t been reflected in the query store yet; a fairly exceptional set of circumstances.

Reasons valid commands fail and what to do about it

So we’ve got a well-behaved client that is sending valid commands, yet the server still decides to reject them. Often the circumstances for the rejection are related to other actors changing state relevant to the processing of that command.

In the CRM example above, it is only because the billing event arrived first. But “first” could be a millisecond before our command. What if our user pressed the button a millisecond earlier? Should that actually change the business outcome? Shouldn’t we expect our system to behave the same when observed from the outside?

So, if the billing event arrived second, shouldn’t that revert preferred customers to regular ones? Not only that, but shouldn’t the customer be notified of this, like by sending them an email? In which case, why not have this be the behavior for the case where the billing event arrives first? And if we’ve already got a notification model set up, do we really need to return an error to the customer service rep? I mean, it’s not like they can do anything about it other than notifying the customer.

So, if we’re not returning errors to the client (who is already sending us valid commands), maybe all we need to do on the client when sending a command is to tell the user “thank you, you will receive confirmation via email shortly”. We don’t even need the UI widget showing pending commands.

Commands and Autonomy

What we see is that in this model, commands don’t need to be processed immediately – they can be queued. How fast they get processed is a question of Service-Level Agreement (SLA) and not architecturally significant. This is one of the things that makes that node that processes commands autonomous from a runtime perspective – we don’t require an always-on connection to the client.

Also, we shouldn’t need to access the query store to process commands – any state that is needed should be managed by the autonomous component – that’s part of the meaning of autonomy.

Another part is the issue of failed message processing due to the database being down or hitting a deadlock. There is no reason that such errors should be returned to the client – we can just rollback and try again. When an administrator brings the database back up, all the message waiting in the queue will then be processed successfully and our users receive confirmation.

The system as a whole is quite a bit more robust to any error conditions.

Also, since we don’t have queries going through this database any more, the database itself is able to keep more rows/pages in memory which serve commands, improving performance. When both commands and queries were being served off of the same tables, the database server was always juggling rows between the two.

Autonomous Components

While in the picture above we see all commands going to the same AC, we could logically have each command processed by a different AC, each with it’s own queue. That would give us visibility into which queue was the longest, letting us see very easily which part of the system was the bottleneck. While this is interesting for developers, it is critical for system administrators.

Since commands wait in queues, we can now add more processing nodes behind those queues (using the distributor with NServiceBus) so that we’re only scaling the part of the system that’s slow. No need to waste servers on any other requests.

Service Layers

Our command processing objects in the various autonomous components actually make up our service layer. The reason you don’t see this layer explicitly represented in CQRS is that it isn’t really there, at least not as an identifiable logical collection of related objects – here’s why:

In the layered architecture (AKA 3-Tier) approach, there is no statement about dependencies between objects within a layer, or rather it is implied to be allowed. However, when taking a command-oriented view on the service layer, what we see are objects handling different types of commands. Each command is independent of the other, so why should we allow the objects which handle them to depend on each other?

Dependencies are things which should be avoided, unless there is good reason for them.

Keeping the command handling objects independent of each other will allow us to more easily version our system, one command at a time, not needing even to bring down the entire system, given that the new version is backwards compatible with the previous one.

Therefore, keep each command handler in its own VS project, or possibly even in its own solution, thus guiding developers away from introducing dependencies in the name of reuse (it’s a fallacy). If you do decide as a deployment concern, that you want to put them all in the same process feeding off of the same queue, you can ILMerge those assemblies and host them together, but understand that you will be undoing much of the benefits of your autonomous components.

Whither the domain model?

Although in the diagram above you can see the domain model beside the command-processing autonomous components, it’s actually an implementation detail. There is nothing that states that all commands must be processed by the same domain model. Arguably, you could have some commands be processed by transaction script, others using table module (AKA active record), as well as those using the domain model. Event-sourcing is another possible implementation.

Another thing to understand about the domain model is that it now isn’t used to serve queries. So the question is, why do you need to have so many relationships between entities in your domain model?

(You may want to take a second to let that sink in.)

Do we really need a collection of orders on the customer entity? In what command would we need to navigate that collection? In fact, what kind of command would need any one-to-many relationship? And if that’s the case for one-to-many, many-to-many would definitely be out as well. I mean, most commands only contain one or two IDs in them anyway.

Any aggregate operations that may have been calculated by looping over child entities could be pre-calculated and stored as properties on the parent entity. Following this process across all the entities in our domain would result in isolated entities needing nothing more than a couple of properties for the IDs of their related entities – “children” holding the parent ID, like in databases.

In this form, commands could be entirely processed by a single entity – viola, an aggregate root that is a consistency boundary.

Persistence for command processing

Given that the database used for command processing is not used for querying, and that most (if not all) commands contain the IDs of the rows they’re going to affect, do we really need to have a column for every single domain object property? What if we just serialized the domain entity and put it into a single column, and had another column containing the ID? This sounds quite similar to key-value storage that is available in the various cloud providers. In which case, would you really need an object-relational mapper to persist to this kind of storage?

You could also pull out an additional property per piece of data where you’d want the “database” to enforce uniqueness.

I’m not suggesting that you do this in all cases – rather just trying to get you to rethink some basic assumptions.

Let me reiterate

How you process the commands is an implementation detail of CQRS.

Keeping the query store in sync

After the command-processing autonomous component has decided to accept a command, modifying its persistent store as needed, it publishes an event notifying the world about it. This event often is the “past tense” of the command submitted:

MakeCustomerPerferredCommand -> CustomerHasBeenMadePerferredEvent

The publishing of the event is done transactionally together with the processing of the command and the changes to its database. That way, any kind of failure on commit will result in the event not being sent. This is something that should be handled by default by your message bus, and if you’re using MSMQ as your underlying transport, requires the use of transactional queues.

The autonomous component which processes those events and updates the query data store is fairly simple, translating from the event structure to the persistent view model structure. I suggest having an event handler per view model class (AKA per table).

Here’s the picture of all the pieces again:

Bounded Contexts

While CQRS touches on many pieces of software architecture, it is still not at the top of the food chain. CQRS if used is employed within a bounded context (DDD) or a business component (SOA) – a cohesive piece of the problem domain. The events published by one BC are subscribed to by other BCs, each updating their query and command data stores as needed.

UI’s from the CQRS found in each BC can be “mashed up” in a single application, providing users a single composite view on all parts of the problem domain. Composite UI frameworks are very useful for these cases.

Summary

CQRS is about coming up with an appropriate architecture for multi-user collaborative applications. It explicitly takes into account factors like data staleness and volatility and exploits those characteristics for creating simpler and more scalable constructs.

One cannot truly enjoy the benefits of CQRS without considering the user-interface, making it capture user intent explicitly. When taking into account client-side validation, command structures may be somewhat adjusted. Thinking through the order in which commands and events are processed can lead to notification patterns which make returning errors unnecessary.

While the result of applying CQRS to a given project is a more maintainable and performant code base, this simplicity and scalability require understanding the detailed business requirements and are not the result of any technical “best practice”. If anything, we can see a plethora of approaches to apparently similar problems being used together – data readers and domain models, one-way messaging and synchronous calls.

Although this blog post is over 3000 words (a record for this blog), I know that it doesn’t go into enough depth on the topic (it takes about 3 days out of the 5 of my Advanced Distributed Systems Design course to cover everything in enough depth). Still, I hope it has given you the understanding of why CQRS is the way it is and possibly opened your eyes to other ways of looking at the design of distributed systems.

Questions and comments are most welcome.

My latest article has been published in issue 21 of the Microsoft Architecture Journal:

EDA: SOA Through The Looking Glass

Please leave questions and comments here.

I’ve gotten some questions about my DDD presentation there based on Aaron Jensen’s pictures:

Yes – I talk with my hands. All the time.

That slide is quite an important one – I talked about it for at least 2 hours.

Here it is again, this time in full:

You may notice that the nice clean layered abstraction that the industry has gotten so comfortable with doesn’t quite sit right when looking at it from this perspective. The reason for that is that this perspective takes into account physical distribution while layers don’t.

I’ll have some more posts on this topic as well as giving a session in TechEd Europe this November.

Oh – and please do feel free to already send your questions in.

Take a listen.

The common e-commerce example

We accept orders, bill the customer, and then ship them the product.

Fairly straight-forward.

Since each part of that process can be quite complex, let’s have each step be handled by a service:

Sales, Billing, and Shipping. Each of these services will publish an event when it’s done its part. Sales will publish OrderAccepted containing all the order information – order Id, customer Id, products, quantities, etc. Billing will publish CustomerBilledForOrder containing the customer Id, order Id, etc. And Shipping will publish OrderShippedToCustomer with its data.

So far, so good. EDA and SOA seem to be providing us some value.

Where’s the saga?

Well, let’s consider the behavior of the Shipping service. It shouldn’t ship the order to the customer until it has received the CustomerBilledForOrder event as well as the OrderAccepted event. In other words, Shipping needs to hold on to the state that came in the first event until the second event comes in. And this is exactly what sagas are for.

Let’s take a look at the saga code that implements this. In order to simplify the sample a bit, I’ll be omitting the product quantities.

   1:      public class ShippingSaga : Saga<ShippingSagaData>,

   2:          ISagaStartedBy<OrderAccepted>,

   3:          ISagaStartedBy<CustomerBilledForOrder>

   4:      {

   5:          public void Handle(OrderAccepted message)

   6:          {

   7:              this.Data.ProductIdsInOrder = message.ProductIdsInOrder;

   8:          }

   9:   

  10:          public void Handle(CustomerBilledForOrder message)

  11:          {

  12:               this.Bus.Send<ShipOrderToCustomer>(

  13:                  (m =>

  14:                  {

  15:                      m.CustomerId = message.CustomerId;

  16:                      m.OrderId = message.OrderId;

  17:                      m.ProductIdsInOrder = this.Data.ProductIdsInOrder;

  18:                  }

  19:                  ));

  20:   

  21:              this.MarkAsComplete();

  22:          }

  23:   

  24:          public override void Timeout(object state)

  25:          {

  26:              

  27:          }

  28:      }

First of all, this looks fairly simple and straightforward, which is good.

It’s also wrong, which is not so good.

One problem we have here is that events may arrive out of order – first CustomerBilledForOrder, and only then OrderAccepted. What would happen in the above saga in that case? Well, we wouldn’t end up shipping the products to the customer, and customers tend not to like that (for some reason).

There’s also another problem here. See if you can spot it as I go through the explanation of ISagaStartedBy<T>.

Saga start up and correlation

The “ISagaStartedBy<T>” that is implemented for both messages indicates to the infrastructure (NServiceBus) that when a message of that type arrives, if an existing saga instance cannot be found, that a new instance should be started up. Makes sense, doesn’t it? For a given order, when the OrderAccepted event arrives first, Shipping doesn’t currently have any sagas handling it, so it starts up a new one. After that, when the CustomerBilledForOrder event arrives for that same order, the event should be handled by the saga instance that handled the first event – not by a new one.

I’ll repeat the important part: “the event should be handled by the saga instance that handled the first event”.

Since the only information we stored in the saga was the list of products, how would we be able to look up that saga instance when the next event came in containing an order Id, but no saga Id?

OK, so we need to store the order Id from the first event so that when the second event comes along we’ll be able to find the saga based on that order Id. Not too complicated, but something to keep in mind.

Let’s look at the updated code:

   1:      public class ShippingSaga : Saga<ShippingSagaData>,

   2:          ISagaStartedBy<OrderAccepted>,

   3:          ISagaStartedBy<CustomerBilledForOrder>

   4:      {

   5:          public void Handle(CustomerBilledForOrder message)

   6:          {

   7:              this.Data.CustomerHasBeenBilled = true;

   8:   

   9:              this.Data.CustomerId = message.CustomerId;

  10:              this.Data.OrderId = message.OrderId;

  11:   

  12:              this.CompleteIfPossible();

  13:          }

  14:   

  15:          public void Handle(OrderAccepted message)

  16:          {

  17:              this.Data.ProductIdsInOrder = message.ProductIdsInOrder;

  18:   

  19:              this.Data.CustomerId = message.CustomerId;

  20:              this.Data.OrderId = message.OrderId;

  21:   

  22:              this.CompleteIfPossible();

  23:          }

  24:   

  25:          private void CompleteIfPossible()

  26:          {

  27:              if (this.Data.ProductIdsInOrder != null && this.Data.CustomerHasBeenBilled)

  28:              {

  29:                  this.Bus.Send<ShipOrderToCustomer>(

  30:                     (m =>

  31:                     {

  32:                         m.CustomerId = this.Data.CustomerId;

  33:                         m.OrderId = this.Data.OrderId;

  34:                         m.ProductIdsInOrder = this.Data.ProductIdsInOrder;

  35:                     }

  36:                     ));

  37:                  this.MarkAsComplete();

  38:              }

  39:          }

  40:      }

And that brings us to…

Saga persistence

We already saw why Shipping needs to be able to look up its internal sagas using data from the events, but what that means is that simple blob-type persistence of those sagas is out. NServiceBus comes with an NHibernate-based saga persister for exactly this reason, though any persistence mechanism which allows you to query on something other than saga Id would work just as well.

Let’s take a quick look at the saga data that we’ll be storing and see how simple it is:

   1:      public class ShippingSagaData : ISagaEntity

   2:      {

   3:          public virtual Guid Id { get; set; }

   4:          public virtual string Originator { get; set; }

   5:          public virtual Guid OrderId { get; set; }

   6:          public virtual Guid CustomerId { get; set; }

   7:          public virtual List<Guid> ProductIdsInOrder { get; set; }

   8:          public virtual bool CustomerHasBeenBilled { get; set; }

   9:      }

You might have noticed the “Originator” property in there and wondered what it is for. First of all, the ISagaEntity interface requires the two properties Id and Originator. Originator is used to store the return address of the message that started the saga. Id is for what you think it’s for. In this saga, we don’t need to send any messages back to whoever started the saga, but in many others we do. In those cases, we’ll often be handling a message from some other endpoint when we want to possibly report some status back to the client that started the process. By storing that client’s address the first time, we can then “ReplyToOriginator” at any point in the process.

The manufacturing sample that comes with NServiceBus shows how this works.

Saga Lookup

Earlier, we saw the need to search for sagas based on order Id. The way to hook into the infrastructure and perform these lookups is by implementing “IFindSagas<T>.Using<M>” where T is the type of the saga data and M is the type of message. In our example, doing this using NHibernate would look like this:

   1:      public class ShippingSagaFinder : 

   2:          IFindSagas<ShippingSagaData>.Using<OrderAccepted>,

   3:          IFindSagas<ShippingSagaData>.Using<CustomerBilledForOrder>

   4:      {

   5:          public ShippingSagaData FindBy(CustomerBilledForOrder message)

   6:          {

   7:              return FindBy(message.OrderId)

   8:          }

   9:   

  10:          public ShippingSagaData FindBy(OrderAccepted message)

  11:          {

  12:              return FindBy(message.OrderId)

  13:          }

  14:   

  15:          private ShippingSagaData FindBy(Guid orderId)

  16:          {

  17:              return sessionFactory.GetCurrentSession().CreateCriteria(typeof(ShippingSagaData))

  18:                  .Add(Expression.Eq("OrderId", orderId))

  19:                  .UniqueResult<ShippingSagaData>();

  20:          }

  21:   

  22:          private ISessionFactory sessionFactory;

  23:   

  24:          public virtual ISessionFactory SessionFactory

  25:          {

  26:              get { return sessionFactory; }

  27:              set { sessionFactory = value; }

  28:          }

  29:      }

For a performance boost, we’d probably index our saga data by order Id.

On concurrency

Another important note is that for this saga, if both messages were handled in parallel on different machines, the saga could get stuck. The persistence mechanism here needs to prevent this. When using NHibernate over a database with the appropriate isolation level (Repeatable Read – the default in NServiceBus), this “just works”. If/When implementing your own saga persistence mechanism, it is important to understand the kind of concurrency your business logic can live with.

Take a look at Ayende’s example for mobile phone billing to get a feeling for what that’s like.

Summary

In almost any event-driven architecture, you’ll have services correlating multiple events in order to make decisions. The saga pattern is a great fit there, and not at all difficult to implement. You do need to take into account that events may arrive out of order and implement the saga logic accordingly, but it’s really not that big a deal. Do take the time to think through what data will need to be stored in order for the saga to be fault-tolerant, as well as a persistence mechanism that will allow you to look up that data based on event data.

If you feel like giving this approach a try, but don’t have an environment handy for this, download NServiceBus and take a look at the samples. It’s really quick and easy to get set up.

My article on “optimizing a large-scale Software+Services application” has been published in the April edition of MSDN Magazine.

Here’s a short excerpt:

“We had to juggle occasional connectivity, data synchronization, and publish/subscribe all at the same time. We learned that we couldn’t solve all problems either client-side or server-side, but rather that an integrated approach was needed since any changes on one side needed corresponding changes on the other side.”

Continue reading…

Greg Young asserts:

“I believe that this shows there to be a rather negligible cost associated with the use of such a model. There is however a small cost, this cost however I believe only exists when one looks at the system in isolation.”

Ayende adds his perspective:

“The cost of messaging, and a very real one, comes when you need to understand the system. In a system where message exchange is the form of communication, it can be significantly harder to understand what is going on.”

Of course, both these intelligent fellows are right. The reason for the apparent disparity in viewpoints has to do with which part of the following graph you look at. Ayende zooms in on the left side:

As systems get larger, though, the only way to understand them is by working at higher levels of abstraction. That’s where messaging really shines, as the incremental complexity remains the same by maintaining the same modularity as before:

In Ayende’s post, he follows the design I described a while back on using messaging for user management and login for a high-scale web scenario. In his comments, he agrees with the above stating:

“I certainly think that a similar solution using RPC would be much more complex and likely more brittle.”

I feel quite conservative in saying the most enterprise solutions fall on the right side of the intersection in the graph.

That being said, don’t underestimate the learning curve developers go through with messaging. While the mechanics are similar, the mindset is very different. Think about it like this:

You’ve driven a car for years in the US. It’s practically second nature. Then you fly to the UK, rent a car, and all of a sudden, your brain is in meltdown. (or vice versa for those going from the UK to the US)

Summary

If you are going down the messaging route, please be aware that there are shades of gray there as well. You don’t have to implement your user management and login the way I outlined in my post if you don’t require such high levels of scalability, but even lower levels of scalability can benefit from messaging.

Just as there isn’t a single correct design for non-messaging solutions, the same is true for those using messaging. Finding the right balance is tricky, and critical.

When the code is simple in every part of the system, and the asynchronous interactions are what provide for the necessary complexity the problem domain requires, that’s when you know you’ve got it just right.

The question of how web-based (or 3rd party) consumers can work with pub/sub based services comes up a lot.

Many developers are used to implementing web services exposing methods on them like GetAllCustomers.

When moving to pub/sub and other more loosely coupled messaging patterns, developers look to implement the same pattern, opting for something like duplex GetCustomersRequest and GetCustomersResponse. The reasoning is simple and straightforward – it is difficult to push data over the web to consumers.

However, there are still ways to disconnect the preparation of the data from its usage thus gaining many of the advantages of pub/sub.

By employing REST principles and modelling our customer list as an explicit resource, web-based consumers would simply perform regular HTTP GET operations on the URI to get the list of customers.

The resource itself could be a simple XML file – it wouldn’t need to be dynamic at all.

You can get all the scalability benefits of pub/sub for web based consumers. All you need is a bit of REST

Also, “passing reference data in a message makes the message-consuming systems dependent on the knowledge and availability of actual persistent data that is stored “somewhere”. This data must separately be accessed for the sake of understanding the event that is represented by the message.”

The discussion of self-contained events can be compared to integration databases vs application databases.

Centralized Integration – Pros & Cons

If everything in a system can access a central datastore, it is enough for one party to publish an event containing only the ID of an entity that that party previously entered/updated. Upon receiving that event, a subscriber would go to the central datastore and get the fields its interested in for that ID. The advantage of this approach is that the minimal amount of data necessary crosses the network, as subscribers only retrieve the fields that interest them. Martin Fowler describes the disadvantages as:

“An integration database needs a schema that takes all its client applications into account. The resulting schema is either more general, more complex or both. The database usually is controlled by a separate group to the applications and database changes are more complex because they have to be negotiated between the database group and the various applications.”

This is far from being aligned with the principle of autonomy so important to SOA. In that respect, the architectural principle of self-contained messages points us away from those problems and towards more autonomous services.

However, once we have these autonomous business services in place, we may find that we don’t need 100% fully self-contained messages anymore.

A Real-World Example

Let’s say we have 3 business services, Sales, Fulfillment, and Billing.

Sales publishes an OrderAccepted event when it accepts an order. That event contains all the order information.

Both Fulfillment and Billing are subscribed to this event, and thus receive it.

Fulfillment does not ship products to the customer until the customer has been billed, so it just stores the order information internally, and is done.

Billing starts the process of billing the customer for their order, possibly joining several orders into a single bill. After completing this process, it publishes a CustomerBilled event containing all billing information, as well as the IDs of the orders in that bill. It does not put all the order information in that event, as it is not the authoritative owner of that data.

When Fulfillment receives the CustomerBilled event, it uses the IDs of the orders contained in the event to find the order information it previously stored internally. It does not need to call the Sales service for this information or contact some central Master Data Management system. It uses the data it has, and goes about fulfilling the orders and shipping the products to the customer, finally publishing its own OrderShipped event.

Notice, as well, that in the original OrderAccepted event there were the IDs of products the customer ordered. These product IDs originated from another service, Merchandising, responsible for the product catalog. The same thing can be said for the customer ID originating from another service – Customer Care.

The Issue of Time

One could argue that since subscribers use previously cached data when processing new events, that data might not be up to date. Also, we may have race conditions between our services. In the above example, if Billing was extremely fast and more highly available than Fulfillment. Billing could have received the OrderAccepted event, processed it, and published the CustomerBilled event before Fulfillment had received the OrderAccepted event. In short, the CustomerBilled and OrderAccepted messages could be out of order in Fulfillment’s queue.

What would Fulfillment do when trying to process the CustomerBilled message when it doesn’t have the order information?

Well, it knows that the world is parallel and non-sequential, so it does NOT return/log an error, but rather puts that message in the back of the queue to be processed again later (or maybe in some other temporary holding area). This enables the OrderAccepted message to be processed before the CustomerBilled message is retried. When the retry occurs, well, everything’s OK – it’s worked itself out over time.

In the case where we retry again and again and things don’t work themselves out (maybe the OrderAccepted event was lost), we move that message off to a different queue for something else to resolve the conflict (maybe a person, maybe software). If/when the conflict is resolved (got the Sales system / messaging system to replay the OrderAccepted event), the conflict resolver returns the CustomerBilled message to the queue, and now everything works just fine.

As all of this is occurring, the only thing that’s visible to external parties is that it happens to be taking longer than usual for the OrderShipped event to be published. In other words, time is the only difference.

Summary

The problem of non-self-contained events is mitigated first and foremost by business services in SOA, and the apparent issue of time-synchronization by business logic inside these services.

Don’t be afraid to put IDs in your messages and events.

Do be afraid of using those IDs to access datastores shared by multiple “services”.

Using IDs to correlated current events to data from previous events is not only OK, it’s to be expected.

The architectural principle of fully self-contained messages steers us away from the problems of Integration Databases and towards Application Databases, autonomous services, and a better SOA implementation. From there, following the principle of autonomy from a business perspective, will lead us to services not publishing data in their messages that is owned by other services, taking us the next step of our journey to SOA.

Related Content

[Podcast] Message Ordering – Is it cost effective?

Don’t EDA between existing systems

[Podcast] Handling dependencies between subscribers in SOA

Durable Messaging

In order to prevent a message from being lost when it is sent from a publisher to a subscriber, the message is written to disk on the publisher side, and then forwarded to the subscriber, where it is also written to disk. This store-and-forward mechanism enables our systems to gracefully recover from either side being temporarily unavailable.

In my MSDN article on this topic, I outlined some problems with this approach. These problems are exacerbated for publishers. Imagine a publisher with 40 subscribers, publishing 10 messages a second, each containing 1MB of XML. If 10 of the subscribers are unavailable, that’s 100MB of data being written to the publisher’s disk every second, 6GB every minute. That’s liable to bring down a publisher before an administrator brews a cup of coffee.

Publishers have no choice but to throw away messages after a certain period of time.

Publisher Contracts

The whole issue of contracts and schema is considered one of the better understand parts of SOA. Unfortunately, the operational aspects of service contracts is hardly ever taken into account.

On top of the schema of the messages a service publishers, additional information is needed in the contract:

1. How big will this message be?
2. How often will it be published?
3. How long will this message be stored if a subscriber is unavailable?

This first two pieces of information are important for subscribers to do load and capacity planning. The last one is the most important as it dictates the required availability and fault-tolerance characteristic of subscribers.

For Example

In the canonical retail scenario, when our sales service accepts an order, it publishes an order accepted event. Other services subscribed to this event include shipping, billing, and business intelligence.

While shipping and billing are highly available and able to keep up with the rate at which orders are accepted, the business intelligence service is not. BI has two main parts to it – a nightly batch that does the number crunching, and a UI for reporting off of the results of that number crunching. Some even do the reporting in a semi-offline fashion, emailing reports back to the user when they’re ready.

Furthermore, nobody’s going to invest in servers for making BI highly available.

And wasn’t the whole point of this publish/subscribe messaging to keep our services autonomous? That not all services have to have the same level uptime?

Houston, do we have a problem.?

Data Freshness

There is a glimmer of light in all this doom and gloom.

Not all services have the same data freshness requirements.

The business intelligence service above doesn’t need to know about orders the second they’re accepted. A daily roll-up would be fine, and an hourly roll-up bring us that much closer to “real time business intelligence”.

So, while BI is ready to accept the sales message schema, it would like a slightly different contract around it – less messages per unit of time, more data in each message.

From the operational perspective of the sales service, it would be cost effective to have less “online” subscribers. It could even take things a few steps further. Instead of using the regular messaging backbone for transmitting these hourly messages, it could use FTP. The data could even be zipped to take up even less space. Since the total data size is less than the corresponding online stream, is stored on cheaper, large storage, and the number of subscribers for this zipped, hourly update is fairly small, these messages can be kept around far longer.

If you’ve heard about consumer-driven contracts, this is it.

Note that we’re still talking about the same logical message schema.

Summary

It’s not that lost notifications aren’t a problem.

It’s that they feed the design process in such a way that the resulting service ecosystem is set up in such a way that notifications won’t get lost. I know that that sounds kind of recursive, but that’s how it works. Either subscribers take care of their SLA allowing them to process the online stream of events, or they should subscribe to a different pipe (which will have different SLA requirements, but maybe they can deal with those).

It make sense to have multiple pipes for the same logical schema.

It’s practically a necessity to make pub/sub a feasible solution.

Related Content

MSDN article on messaging and lost messages

Durable messaging dilemmas

Additional logic required for service autonomy

More in depth example on events and pub/sub between services

Consumer-Driven Contracts

Dennis asked this:

There are different design choices in a SOA, even when you already have identified the services. I have a simple example that I would like to share:

Imagine a order-to-cash process. One part of that process is to register an order. Suppose we have two services, Order Service and Inventory Service. The task is to register the order and make a corresponding reservation of the stock level. I would be pleased to have the groups view on the following 3 design options (A, B, C):

A.
1. The “process/application” sends a message (sync or async) to “registerOrder” on the Order Service.
2. The “process/application” sends another message (sync or async) to “reserveStock” on the the Inventory Service.

B.
1. The “process/application” sends a message (sync or async) to “registerOrder” on the Order Service.
2. The Order Service sends a message (sync or async) to “reserveStock” on the the Inventory Service.

C.
1. The “process/application” sends a message (sync or async) to “registerOrder” on the Order Service.
2. The Order Service publishes an “orderReceived” event.
3. The Inventory Service subscribes to the “orderReceived” event .

On the whole “already identified the services” thing – naming a service doesn’t mean much. It’s all about allocating responsibility, and until that’s been done, those “services” don’t give us very much information.

Business Services

If we were to view this example in light of business services, and look at the business events that make up this process, maybe we’d get a different perspective.

Three business services: Sales, Inventory, and Shipping.

In Sales, many applications and people may operate, including the person and the application he used to submit the order. When the order is submitted and goes through all the internal validation stuff, Sales raises an OrderTentativelyAccepted event.

Inventory and Orders

Inventory, which is subscribed to this event, checks if it has everything in stock for the order. For every item in the order on stock, it allocates that stock to the order and publishes the InventoryAllocatedToOrder event for it. For items/quantities not in stock, it starts a long running process which watches for inventory changes.

When an InventoryChanged event occurs, it matches that against orders requiring allocation – if it finds one that requires stock, based on some logic to choose which order gets precedence, it publishes the InventoryAllocatedToOrder event.

Sales, which is subscribed to the InventoryAllocatedToOrder event, upon receiving all events pertaining to the order tentatively accepted, will publish an OrderAccepted event.

Orders and Shipping

When Inventory receives the OrderAccepted event, it generates the pick list to bring all the stock from the warehouses to the loading docks, finally publishing the PickListGenerated event containing target docks.

When Shipping receives the PickListGenerated event, it starts the yard management necessary to bring the needed kinds of trucks to the docks.

What else is possible

I could go on, talking about things like the maximum amount of time stock of various kinds can wait to be loaded on trucks, subscribing to earlier events to employ all kinds of optimization and prediction algorithms, having a Customer Care service notifying the customer about what’s going on with their order (probably different for different kinds of customers and preferred communication definitions). Obviously, we’d need a Billing service to handle the various kinds of billing procedures, whether or not the customer has credit, pays upon delivery, etc.

It turns out that many business domains map very well to this join of SOA and EDA.

What an ESB is for

When we have these kinds of business services primarily publishing events and subscribing to those of other services, you don’t need much else from your “enterprise service bus”. All sorts of transformation, routing, and orchestration capabilities don’t come into play at all.

In all truthfullness, those bits of functionality are really just a historical artifact of their broker heritage.

Don’t get me wrong, sometimes a broker is a nice thing to have – behind a service boundary in order to perform some complex integration between existing legacy applications.

Just keep that stuff in its place – not between services.

Complex Event Processing

We can look at how Sales transitions an order from being tentatively accepted to being accepted as requiring event correlation around InventoryAllocatedToOrder events. This isn’t exactly “complex” in its own right. If there were some kind of CEP engine that did this for us out of the box, it might be a possible technology choice for implementing this logic within our service.

As we add more concerns, like time, we may find new ways to make use of this engine. For instance, if the time to provide the order to the customer is approaching, we may choose to split the order into two – accepting one for which we have all the stock allocated, and leaving the second as tentatively accepted.

Summary

While it is difficult to move forward on service responsibility without discussing the events it raises and those it subscribe to, the whole issue of CEP can be postponed for a while.

Although there aren’t many who would say that EDA is necessary for driving down coupling in SOA, or that SOA won’t likely provide much value without EDA, or that SOA is necessary for providing the right boundaries for EDA, it’s been my experience that that is exactly the case.

CEP, while being a challenging engineering field, and managing the technical risks around it necessary for a project to succeed in some circumstances, and really shines when used under the SOA/EDA umbrella, it should not be taken by itself and used at the topmost architectural levels.

Related Content

SOA and Enterprise Processes

How client interaction fits with SOA

Time and SOA

Durable Messaging for Fault-Tolerant Services

And if you’re wondering about how to handle all that complexity inside services (different kinds of billing, periodic tests for electronics inventory, etc), you might like listening to this podcast about business components.

Watch that first step

Although sometimes described as the first step of an organization moving to SOA, web-service-izing everything results in synchronous, blocking, request/response interaction between services. The problem being that if one service were to become unavailable, all consumers of that service would not be able to perform any work. With the deep service “call stacks” this architectural style condones, the availability and performance of the entire organization will be dictated by the weakest link.

So, while I’d agree that many organizations do need to take this step, I’d caution against going into production at this step.

Pub/Sub Considered Helpful

When services interact with each other using publish/subscribe semantics we don’t have that technical problem of blocking. Subscribers cache the data published to them (either in memory or durably depending on their fault-tolerance requirements) thus enabling them to function and process requests even if the publisher is unavailable.

Consider the following scenario:

Let’s say we have an e-commerce site, a part of our Sales service responsible for selling products. Another service, let’s call it merchandising, is responsible for the catalog of products, and how much each product costs. Sales is subscribed to price update events published by Merchandising and saves (caches) those prices in its own database. When a customer orders some products on the site, Sales does not need to call Merchandising to get the price of the product and just uses the previously saved (cached) price. Thus, even if Merchandising is unavailable, Sales is able to accept orders. This is a big win as our merchandising application is not nearly as robust as our sales systems.

Yet, there are scenarios where data freshness requirements prevent this.

Too Much of a Good Thing?

Technically, the above story is accurate. There is nothing technically preventing Sales from accepting orders. Yet consider a scenario where Merchandising is down or unavailable for an extended period of time. While this may not be entirely likely for two servers in the same data center, consider physical kiosks which customers can use to buy products. Those kiosks may not receive updates for days. Should they accept orders?

That’s really a question to the business. If pricing data is stale for a time period greater than X, do not sell that item. The value of X may even be different for different kinds of products. Keep in mind that this issue only arose since we architected our services to be fully autonomous. In a synchronous systems architecture, this issue would not come up. As such, it is our responsibility as architects to go digging for these requirements as well as explaining to the business what the tradeoffs are.

In order to have more up to date data, we need to invest in more available hardware, networks, and infrastructure. This needs to be balanced against the predicted increase in revenue that more up to date (read higher) prices would give us.

You Can Get What You Pay For

Beyond the additional cost of writing that additional logic, and the perceived increased complexity, another difference to note between this architectural style and the synchronous/traditional one is that it puts control of spending back in the hands of business.

In a synchronous architecture, in order to achieve required performance and availability, all systems need to be performant requiring across the board investments in servers, networks, and storage. Without investing everywhere, the weakest link is liable to undo all other investments. In other words, your developers have made your investment choices for you. Scary, isn’t it.

A more prudent investment strategy would prefer spending on services that give the biggest bang for the buck, better known as return on investment. A pub/sub based architecture allows investing in data-freshness where it makes the most sense. For example, in sales of high profit products to strategic customers rather than inventory management of raw materials for products slated to be decommissioned.

That sounds a lot like IT-Business Alignment.

Maybe there’s something to this SOA thing after all…

Read more about:

7 Questions for Service Selection

7 Questions around data freshness 

Event-Driven Architecture and Legacy Applications

Autonomous Services and Enterprise Entity Aggregation

Or listen to a podcast describing Business Components, the connection of pub/sub and SOA.