One of the things that surprises some developers that I talk to is that you don’t always get consistency even with end-to-end synchronous communication and a single database. This goes beyond things like isolation levels that some developers are aware of and is particularly significant in multi-user collaborative domains.

The problem

Let’s start with an image to describe the scenario:

The main issue we have here is that the values transaction 2 gets for A and B are those from T0 – before either transaction 1 or 3 completed. The reason this is an issue is that these old values (usually together with some message data) are used to calculate what the new state of C should be.

Traditional optimistic concurrency techniques won’t detect any problem if we don’t touch A or B in transaction 2.

In short, systems today are causing inconsistency.

Some solutions

1. Don’t have transactions which operate on multiple entities (which probably isn’t possible for some of your most important business logic).

2. Turn on multi-version concurrency control – this is called snapshot isolation in MS Sql Server.

Yes, you need to turn it on. It’s off by default.

The good news is that this will stop the writing of inconsistent data to your database.
The bad news is that it will probably cause your system many more exceptions when going to persist.

For those of you who are using transaction messaging with automatic retrying, this will end up as “just” a performance problem (unless you follow the recommendations below). For those of you who are using regular web/wcf services (over tcp/http), you’re “cross cutting” exception management will likely end up discarding all the data submitted in those requests (but since that’s what you’re doing when you run into deadlocks this shouldn’t be news to you).

The solution to the performance issues

Eventual consistency.

Funny isn’t it – all those people who were afraid of eventual consistency got inconsistency instead.

Also, it’s not enough to just have eventual consistency (like between the command and query sides of CQRS). You need to drastically decrease the size of your entities. And the best way of doing that is to partition those entities across multiple business services (also known in DDD lingo as Bounded Contexts) each with its own database.

This is yet another reason why I say that CQRS shouldn’t be the top level architectural breakdown. Very useful within a given business service, yes – though sometimes as small as just some sagas.

Next steps

It may seem unusual that the title of this post implies that SOA is the solution, yet the content clearly states that traditional HTTP-based web services are a problem. Even REST wouldn’t change matters as it doesn’t influence how transactions are managed against a database.

The SOA solution I’m talking about here is the one I’ve spent the last several years blogging about. It’s a different style of SOA which has services stretch up to contain parts of the UI as well as down to contain parts of the database, resulting in a composite UI and multiple databases. This is a drastically different approach than much of the literature on the topic – especially Thomas Erl’s books.

Unfortunately there isn’t a book out there with all of this in it (that I’ve found), and I’m afraid that with my schedule (and family) writing a book is pretty much out of the question. Let’s face it – I’m barely finding time to blog.

The one thing I’m trying to do more of is provide training on these topics. I’ve just finished a course in London, doing another this week in Aarhus Denmark, and another next month in San Francisco (which is now sold out). The next openings this year will be in Stockholm, London; Sydney Australia and Austin Texas will be coming in January of next year. I’ll be coming over to the US more next year so if you missed San Francisco, keep an eye out.

I wish there was more I could do, but I’m only one guy.

Hmm, maybe it’s time to change that.

It isn’t uncommon for me to have a client or student at one of my courses ask me about some kind of workflow tool. This could be Microsoft Workflow Foundation, BizTalk, K2, or some kind of BPEL/orchestration engine. The question usually revolves around using this tool for all workflows in the system as opposed to the SOA-EDA-style publish/subscribe approach I espouse.

The question

The main touted benefit of these workflow-centric architectures is that we don’t have to change the code of the system in order to change its behavior resulting in ultimate flexibility!

Some of you may have already gone down this path and are shaking your heads remembering how your particular road to hell was paved with the exact same good intentions.

Let me explain why these things tend to go horribly wrong.

What’s behind the curtain

It starts with the very nature of workflow – a flow chart, is procedural in nature. First do this, then that, if this, then that, etc. As we’ve experienced first hand in our industry, procedural programming is fine for smaller problems but isn’t powerful enough to handle larger problems. That’s why we’ve come up with object-oriented programming.

I have yet to see an object-oriented workflow drag-and-drop engine. Yes, it works great for simple demo-ware apps. But if you try to through your most complex and volatile business logic at it, it will become a big tangled ball of spaghetti – just like if you were using text rather than pictures to code it.

And that’s one of the fundamental fallacies about these tools – you are still writing code. The fact that it doesn’t look like the rest of your code doesn’t change that fact. Changing the definition of your workflow in the tool IS changing your code.

On productivity

Sometimes people mention how much more productive it would be to use these tools than to write the code “by hand”. Occasionally I hear about an attempt to have “the business” use these tools to change the workflows themselves – without the involvement of developers (”imagine how much faster we could go without those pesky developers!”).

For those of us who have experienced this first-hand, we know that’s all wrong.

If “the business” is changing the workflows without developer involvement, invariably something breaks, and then they don’t know what to do. They haven’t been trained to think the way that developers have – they don’t really know how to debug. So the developers are brought back in anyway and from that point on, the business is once again giving requirements and the devs are the one implementing it.

Now when it comes to developer productivity, I can tell you that the keyboard is at least 10x more productive than the mouse. I can bang out an if statement in code much faster than draggy-dropping a diamond on the canvas, and two other activities for each side of the clause.

On maintainability

Sometimes the visualization of the workflow is presented as being much more maintainable than “regular code”.

When these workflows get to be to big/nested/reused, it ends up looking like the wiring diagram of an Intel chip (or worse). Check out the following diagram taken from the DailyWTF on a customer friendly system:

The bigger these get, the less maintainable they are.

Now, some would push back on this saying that a method with 10,000 lines of code in it may be just as bad, if not worse. The thing is that these workflow tools guide developers down a path where it is very likely to end up with big, monolithic, procedural, nested code. When working in real code, we know we need to take responsibility for the cleanliness of our code using object-orientation, patterns, etc and refactoring things when they get too messy.

Here is where I’d bring up the SOA/pub-sub approach as an alternative – there is no longer this idea of a centralized anything. You have small pieces of code, each encapsulating a single business responsibility, working in concert with each other – reacting to each others events.

Productivity take 2: testing and version control

If you’re going to take your most complex and volatile business logic and put it into these workflow tools, have you thought about how your going to test it? How do you know that it works correctly? It tends to be VERY difficult to unit-test these kinds of workflows.

When a developer is implementing a change request, how do they know what other workflows might have been broken? Do they have to manually go through each and every scenario in the system to find out? How’s that for productivity?

Assuming something did break and the developer wants to see a diff – what’s different in the new workflow from the old one, what would that look like? When working with a team, the ability to diff and merge code is at the base of the overall team productivity.

What would happen to your team if you couldn’t diff or merge code anymore?
In this day and age, it should be considered irresponsible to develop without these version control basics.

In closing

There are some cases where these tools might make sense, but those tend to be much more rare than you’d expect (and there are usually better alternatives anyway). Regardless, the architectural analysis should start without the assumption of centralized workflow, database, or centralized anything for that matter.

If someone tries to push one of these tools/architectures on you, don’t walk away – run!

One of the things I cover early on in my course is the problem with traditional layered architecture driving people to create a business logic layer made up of a bunch of inter-related entities. I see this happening a lot, even though nowadays people are calling that bunch of inter-related entities a “domain model”.

Let me just say this upfront – most inter-related entity models are NOT a domain model.
Here’s why: most transactions don’t respect entity boundaries.

That being said, you don’t always need a domain model.
The domain model pattern’s context is “if you have complicated and everchanging business rules” – right there on page 119 of Patterns of Enterprise Application Architecture.

Persisting the customer’s first name, last name, and middle initial – and later reading and showing that data does not sound either complicated or that it is really going to change that much.

Then there are things like credit limits, that may be on the customer entity as well. It is likely that there are business requirements that expect that value to be consistent with the total value of unpaid orders – data that comes from other entities.

The problem that is created is one of throughput.

Since databases lock an entire row/entity at a time, if one transaction is changing the customer’s first name, the database would block another transaction that tried to change the same customer’s credit limit.

The bigger your entities, the more transactions will likely need to operate on them in parallel, the slower your system will get as the number of transactions increases. This feeds back in on itself as often those blocked transactions will have operated already on some other entity, leaving those locked for longer periods of times, blocking even more transactions.

And the absurd thing is that the business never demanded that the customer’s first name be consistent with the credit limit.

What if we didn’t have a single Customer entity?

What if we had one that contained first name, last name, middle initial and another that contained things like credit limit, status, and risk rating. These entities would be correlated by the same ID, but could be stored in separate tables in the database. That would do away with much of the cascading locking effects drastically improving our throughput as load increases.

And you know what? That division would still respect the 3rd normal form.

Which of these entities do you think would be classified by the business under the “complicated and everchanging rules” category?

And for those entities that are just about data persistence – do you think it’s justified to use 3 tiers? Do we really need a view model which we transform to data transfer objects which we transform to domain objects which we transform to relational tables and then all the way back? Wouldn’t some simpler 2-tier programming suffice – dare I say datasets? Ruby on Rails?

Are we ready to leave behind the assumption that all elements of a given layer must be built the same way?

Not in the business world anyway.

The problem is that, as software developers, we’re all too quick to accept them at face value. We don’t question the requirements – in all fairness, it was never our job to do so. We were the ones that implemented them, preferably quickly.

For example

Let’s say we get the requirement the following requirements:

1. If the order was already shipped, don’t let the user cancel the order.
2. If the order was already cancelled, don’t let the user ship the order.

The race condition here is when we have two users who are looking at the same order, which is neither cancelled nor shipped yet, and each submits a command – one to ship the order, the other to cancel it.

In these cases, the code is simple – just an if statement before performing the relevant command.

So what’s the problem

A microsecond difference in timing shouldn’t make a difference to core business behaviors. Which means that we’ve actually got here is a bug in the requirements. Users are actually dictating solutions here rather than requirements.

Let’s ask our stakeholders, “why shouldn’t we let users cancel a shipped order? I mean, the users don’t want the products.”

And the stakeholders would respond with something like, “well, we don’t want to refund the user’s money then. Or, at least, not all their money. Well, maybe if they return the products in their original packaging, *then* we could give a full refund.”

And as we drilled deeper, “when do refunds need to be given? Right away, in the same transaction?”

The stakeholders would explain, “no, refunds don’t need to be given right away.”

It turns out we were missing the concept of a refund, as well as assuming that all things needed to be processed and enforced immediately. Once we dug into the requirements, we found that there is actually plenty of time to allow both transactions to go through. We just need to add some checks during shipping’s long-running process to see if the order was cancelled, and then to cut the process short.

So is everything a long-running process then?

That’s actually a fair question – long-running processes are a lot more common than at first appears.

What we’re seeing is that cancellation is now a command that has no reason to fail – just like CQRS tells us. When this command is performed, it publishes the OrderCancelled event, which the billing service subscribes to.

Billing then starts a long-running process (a saga, in NServiceBus lingo), also listening to events from the shipping process, ultimately making a decision when a refund should be given, and for how much.

Deeper business analysis

As we discuss matters more with our business stakeholders, we hear that most orders are actually cancelled within an hour of being submitted. It is quite rare for orders to be cancelled days later.

In which case, we could look at modeling the acceptance of an order as a long-running process itself.

When a user places an order, we don’t immediately publish an event indicating the acceptance of an order, instead a saga is kicked off – which opens up a timeout for an hour later. If a cancellation command arrives during that period of time, the user gets a full refund (seeing as we didn’t charge anything since billing didn’t get the accepted event to begin with), and the saga just shuts itself down. If the timeout occurs an hour later, and the saga didn’t get a cancel command, then the order is actually accepted and the event is published.

Yes, sagas are everywhere, once you learn to see with business eyes, and no race conditions are left.

In closing

Any time you see requirements that indicate a race condition, dig deeper.

What you’re likely to find are some additional business concepts as well as the introduction of time and the creation of long-running business processes. The implementation at that point will pivot from being trivial if-statements to being richer sagas.

Keep an eye out.

After listening how the community has interpreted Command-Query Responsibility Segregation I think that the time has come for some clarification. Some have been tying it together to Event Sourcing. Most have been overlaying their previous layered architecture assumptions on it. Here I hope to identify CQRS itself, and describe in which places it can connect to other patterns.

Download as PDF – this is quite a long post.

Why CQRS

Before describing the details of CQRS we need to understand the two main driving forces behind it: collaboration and staleness.

Collaboration refers to circumstances under which multiple actors will be using/modifying the same set of data – whether or not the intention of the actors is actually to collaborate with each other. There are often rules which indicate which user can perform which kind of modification and modifications that may have been acceptable in one case may not be acceptable in others. We’ll give some examples shortly. Actors can be human like normal users, or automated like software.

Staleness refers to the fact that in a collaborative environment, once data has been shown to a user, that same data may have been changed by another actor – it is stale. Almost any system which makes use of a cache is serving stale data – often for performance reasons. What this means is that we cannot entirely trust our users decisions, as they could have been made based on out-of-date information.

Standard layered architectures don’t explicitly deal with either of these issues. While putting everything in the same database may be one step in the direction of handling collaboration, staleness is usually exacerbated in those architectures by the use of caches as a performance-improving afterthought.

A picture for reference

I’ve given some talks about CQRS using this diagram to explain it:

The boxes named AC are Autonomous Components. We’ll describe what makes them autonomous when discussing commands. But before we go into the complicated parts, let’s start with queries:

Queries

If the data we’re going to be showing users is stale anyway, is it really necessary to go to the master database and get it from there? Why transform those 3rd normal form structures to domain objects if we just want data – not any rule-preserving behaviors? Why transform those domain objects to DTOs to transfer them across a wire, and who said that wire has to be exactly there? Why transform those DTOs to view model objects?

In short, it looks like we’re doing a heck of a lot of unnecessary work based on the assumption that reusing code that has already been written will be easier than just solving the problem at hand. Let’s try a different approach:

How about we create an additional data store whose data can be a bit out of sync with the master database – I mean, the data we’re showing the user is stale anyway, so why not reflect in the data store itself. We’ll come up with an approach later to keep this data store more or less in sync.

Now, what would be the correct structure for this data store? How about just like the view model? One table for each view. Then our client could simply SELECT * FROM MyViewTable (or possibly pass in an ID in a where clause), and bind the result to the screen. That would be just as simple as can be. You could wrap that up with a thin facade if you feel the need, or with stored procedures, or using AutoMapper which can simply map from a data reader to your view model class. The thing is that the view model structures are already wire-friendly, so you don’t need to transform them to anything else.

You could even consider taking that data store and putting it in your web tier. It’s just as secure as an in-memory cache in your web tier. Give your web servers SELECT only permissions on those tables and you should be fine.

Query Data Storage

While you can use a regular database as your query data store it isn’t the only option. Consider that the query schema is in essence identical to your view model. You don’t have any relationships between your various view model classes, so you shouldn’t need any relationships between the tables in the query data store.

So do you actually need a relational database?

The answer is no, but for all practical purposes and due to organizational inertia, it is probably your best choice (for now).

Scaling Queries

Since your queries are now being performed off of a separate data store than your master database, and there is no assumption that the data that’s being served is 100% up to date, you can easily add more instances of these stores without worrying that they don’t contain the exact same data. The same mechanism that updates one instance can be used for many instances, as we’ll see later.

This gives you cheap horizontal scaling for your queries. Also, since your not doing nearly as much transformation, the latency per query goes down as well. Simple code is fast code.

Data modifications

Since our users are making decisions based on stale data, we need to be more discerning about which things we let through. Here’s a scenario explaining why:

Let’s say we have a customer service representative who is one the phone with a customer. This user is looking at the customer’s details on the screen and wants to make them a ‘preferred’ customer, as well as modifying their address, changing their title from Ms to Mrs, changing their last name, and indicating that they’re now married. What the user doesn’t know is that after opening the screen, an event arrived from the billing department indicating that this same customer doesn’t pay their bills – they’re delinquent. At this point, our user submits their changes.

Should we accept their changes?

Well, we should accept some of them, but not the change to ‘preferred’, since the customer is delinquent. But writing those kinds of checks is a pain – we need to do a diff on the data, infer what the changes mean, which ones are related to each other (name change, title change) and which are separate, identify which data to check against – not just compared to the data the user retrieved, but compared to the current state in the database, and then reject or accept.

Unfortunately for our users, we tend to reject the whole thing if any part of it is off. At that point, our users have to refresh their screen to get the up-to-date data, and retype in all the previous changes, hoping that this time we won’t yell at them because of an optimistic concurrency conflict.

As we get larger entities with more fields on them, we also get more actors working with those same entities, and the higher the likelihood that something will touch some attribute of them at any given time, increasing the number of concurrency conflicts.

If only there was some way for our users to provide us with the right level of granularity and intent when modifying data. That’s what commands are all about.

Commands

A core element of CQRS is rethinking the design of the user interface to enable us to capture our users’ intent such that making a customer preferred is a different unit of work for the user than indicating that the customer has moved or that they’ve gotten married. Using an Excel-like UI for data changes doesn’t capture intent, as we saw above.

We could even consider allowing our users to submit a new command even before they’ve received confirmation on the previous one. We could have a little widget on the side showing the user their pending commands, checking them off asynchronously as we receive confirmation from the server, or marking them with an X if they fail. The user could then double-click that failed task to find information about what happened.

Note that the client sends commands to the server – it doesn’t publish them. Publishing is reserved for events which state a fact – that something has happened, and that the publisher has no concern about what receivers of that event do with it.

Commands and Validation

In thinking through what could make a command fail, one topic that comes up is validation. Validation is different from business rules in that it states a context-independent fact about a command. Either a command is valid, or it isn’t. Business rules on the other hand are context dependent.

In the example we saw before, the data our customer service rep submitted was valid, it was only due to the billing event arriving earlier which required the command to be rejected. Had that billing event not arrived, the data would have been accepted.

Even though a command may be valid, there still may be reasons to reject it.

As such, validation can be performed on the client, checking that all fields required for that command are there, number and date ranges are OK, that kind of thing. The server would still validate all commands that arrive, not trusting clients to do the validation.

Rethinking UIs and commands in light of validation

The client can make of the query data store when validating commands. For example, before submitting a command that the customer has moved, we can check that the street name exists in the query data store.

At that point, we may rethink the UI and have an auto-completing text box for the street name, thus ensuring that the street name we’ll pass in the command will be valid. But why not take things a step further? Why not pass in the street ID instead of its name? Have the command represent the street not as a string, but as an ID (int, guid, whatever).

On the server side, the only reason that such a command would fail would be due to concurrency – that someone had deleted that street and that that hadn’t been reflected in the query store yet; a fairly exceptional set of circumstances.

Reasons valid commands fail and what to do about it

So we’ve got a well-behaved client that is sending valid commands, yet the server still decides to reject them. Often the circumstances for the rejection are related to other actors changing state relevant to the processing of that command.

In the CRM example above, it is only because the billing event arrived first. But “first” could be a millisecond before our command. What if our user pressed the button a millisecond earlier? Should that actually change the business outcome? Shouldn’t we expect our system to behave the same when observed from the outside?

So, if the billing event arrived second, shouldn’t that revert preferred customers to regular ones? Not only that, but shouldn’t the customer be notified of this, like by sending them an email? In which case, why not have this be the behavior for the case where the billing event arrives first? And if we’ve already got a notification model set up, do we really need to return an error to the customer service rep? I mean, it’s not like they can do anything about it other than notifying the customer.

So, if we’re not returning errors to the client (who is already sending us valid commands), maybe all we need to do on the client when sending a command is to tell the user “thank you, you will receive confirmation via email shortly”. We don’t even need the UI widget showing pending commands.

Commands and Autonomy

What we see is that in this model, commands don’t need to be processed immediately – they can be queued. How fast they get processed is a question of Service-Level Agreement (SLA) and not architecturally significant. This is one of the things that makes that node that processes commands autonomous from a runtime perspective – we don’t require an always-on connection to the client.

Also, we shouldn’t need to access the query store to process commands – any state that is needed should be managed by the autonomous component – that’s part of the meaning of autonomy.

Another part is the issue of failed message processing due to the database being down or hitting a deadlock. There is no reason that such errors should be returned to the client – we can just rollback and try again. When an administrator brings the database back up, all the message waiting in the queue will then be processed successfully and our users receive confirmation.

The system as a whole is quite a bit more robust to any error conditions.

Also, since we don’t have queries going through this database any more, the database itself is able to keep more rows/pages in memory which serve commands, improving performance. When both commands and queries were being served off of the same tables, the database server was always juggling rows between the two.

Autonomous Components

While in the picture above we see all commands going to the same AC, we could logically have each command processed by a different AC, each with it’s own queue. That would give us visibility into which queue was the longest, letting us see very easily which part of the system was the bottleneck. While this is interesting for developers, it is critical for system administrators.

Since commands wait in queues, we can now add more processing nodes behind those queues (using the distributor with NServiceBus) so that we’re only scaling the part of the system that’s slow. No need to waste servers on any other requests.

Service Layers

Our command processing objects in the various autonomous components actually make up our service layer. The reason you don’t see this layer explicitly represented in CQRS is that it isn’t really there, at least not as an identifiable logical collection of related objects – here’s why:

In the layered architecture (AKA 3-Tier) approach, there is no statement about dependencies between objects within a layer, or rather it is implied to be allowed. However, when taking a command-oriented view on the service layer, what we see are objects handling different types of commands. Each command is independent of the other, so why should we allow the objects which handle them to depend on each other?

Dependencies are things which should be avoided, unless there is good reason for them.

Keeping the command handling objects independent of each other will allow us to more easily version our system, one command at a time, not needing even to bring down the entire system, given that the new version is backwards compatible with the previous one.

Therefore, keep each command handler in its own VS project, or possibly even in its own solution, thus guiding developers away from introducing dependencies in the name of reuse (it’s a fallacy). If you do decide as a deployment concern, that you want to put them all in the same process feeding off of the same queue, you can ILMerge those assemblies and host them together, but understand that you will be undoing much of the benefits of your autonomous components.

Whither the domain model?

Although in the diagram above you can see the domain model beside the command-processing autonomous components, it’s actually an implementation detail. There is nothing that states that all commands must be processed by the same domain model. Arguably, you could have some commands be processed by transaction script, others using table module (AKA active record), as well as those using the domain model. Event-sourcing is another possible implementation.

Another thing to understand about the domain model is that it now isn’t used to serve queries. So the question is, why do you need to have so many relationships between entities in your domain model?

(You may want to take a second to let that sink in.)

Do we really need a collection of orders on the customer entity? In what command would we need to navigate that collection? In fact, what kind of command would need any one-to-many relationship? And if that’s the case for one-to-many, many-to-many would definitely be out as well. I mean, most commands only contain one or two IDs in them anyway.

Any aggregate operations that may have been calculated by looping over child entities could be pre-calculated and stored as properties on the parent entity. Following this process across all the entities in our domain would result in isolated entities needing nothing more than a couple of properties for the IDs of their related entities – “children” holding the parent ID, like in databases.

In this form, commands could be entirely processed by a single entity – viola, an aggregate root that is a consistency boundary.

Persistence for command processing

Given that the database used for command processing is not used for querying, and that most (if not all) commands contain the IDs of the rows they’re going to affect, do we really need to have a column for every single domain object property? What if we just serialized the domain entity and put it into a single column, and had another column containing the ID? This sounds quite similar to key-value storage that is available in the various cloud providers. In which case, would you really need an object-relational mapper to persist to this kind of storage?

You could also pull out an additional property per piece of data where you’d want the “database” to enforce uniqueness.

I’m not suggesting that you do this in all cases – rather just trying to get you to rethink some basic assumptions.

Let me reiterate

How you process the commands is an implementation detail of CQRS.

Keeping the query store in sync

After the command-processing autonomous component has decided to accept a command, modifying its persistent store as needed, it publishes an event notifying the world about it. This event often is the “past tense” of the command submitted:

MakeCustomerPerferredCommand -> CustomerHasBeenMadePerferredEvent

The publishing of the event is done transactionally together with the processing of the command and the changes to its database. That way, any kind of failure on commit will result in the event not being sent. This is something that should be handled by default by your message bus, and if you’re using MSMQ as your underlying transport, requires the use of transactional queues.

The autonomous component which processes those events and updates the query data store is fairly simple, translating from the event structure to the persistent view model structure. I suggest having an event handler per view model class (AKA per table).

Here’s the picture of all the pieces again:

Bounded Contexts

While CQRS touches on many pieces of software architecture, it is still not at the top of the food chain. CQRS if used is employed within a bounded context (DDD) or a business component (SOA) – a cohesive piece of the problem domain. The events published by one BC are subscribed to by other BCs, each updating their query and command data stores as needed.

UI’s from the CQRS found in each BC can be “mashed up” in a single application, providing users a single composite view on all parts of the problem domain. Composite UI frameworks are very useful for these cases.

Summary

CQRS is about coming up with an appropriate architecture for multi-user collaborative applications. It explicitly takes into account factors like data staleness and volatility and exploits those characteristics for creating simpler and more scalable constructs.

One cannot truly enjoy the benefits of CQRS without considering the user-interface, making it capture user intent explicitly. When taking into account client-side validation, command structures may be somewhat adjusted. Thinking through the order in which commands and events are processed can lead to notification patterns which make returning errors unnecessary.

While the result of applying CQRS to a given project is a more maintainable and performant code base, this simplicity and scalability require understanding the detailed business requirements and are not the result of any technical “best practice”. If anything, we can see a plethora of approaches to apparently similar problems being used together – data readers and domain models, one-way messaging and synchronous calls.

Although this blog post is over 3000 words (a record for this blog), I know that it doesn’t go into enough depth on the topic (it takes about 3 days out of the 5 of my Advanced Distributed Systems Design course to cover everything in enough depth). Still, I hope it has given you the understanding of why CQRS is the way it is and possibly opened your eyes to other ways of looking at the design of distributed systems.

Questions and comments are most welcome.

So, I’ve gotten back from a most enjoyable couple of days in Sweden where I gave two half-day tutorials, the first being the SOA and UI composition talk I gave at the European Virtual ALT.NET meeting (which you can find online here) and the other on DDD in enterprise apps (the first time I’ve done this talk).

I’ve gotten some questions about my DDD presentation there based on Aaron Jensen’s pictures:

Yes – I talk with my hands. All the time.

That slide is quite an important one – I talked about it for at least 2 hours.

Here it is again, this time in full:

You may notice that the nice clean layered abstraction that the industry has gotten so comfortable with doesn’t quite sit right when looking at it from this perspective. The reason for that is that this perspective takes into account physical distribution while layers don’t.

I’ll have some more posts on this topic as well as giving a session in TechEd Europe this November.

Oh – and please do feel free to already send your questions in.

After reading Ayende’s post advocating against “soft deletes” I felt that I should add a bit more to the topic as there were some important business semantics missing. As developers discuss the pertinence of using an IsDeleted column in the database to mark deletion, and the way this relates to reporting and auditing concerns is weighed, the core domain concepts rarely get a mention. Let’s first understand the business scenarios we’re modeling, the why behind them, before delving into the how of implementation.

The real world doesn’t cascade

Let’s say our marketing department decides to delete an item from the catalog. Should all previous orders containing that item just disappear? And cascading farther, should all invoices for those orders be deleted as well? Going on, would we have to redo the company’s profit and loss statements?

Heaven forbid.

So, is Ayende wrong? Do we really need soft deletes after all?

On the one hand, we don’t want to leave our database in an inconsistent state with invoices pointing to non-existent orders, but on the other hand, our users did ask us to delete an entity.

Or did they?

When all you have is a hammer…

We’ve been exposing users to entity-based interfaces with “create, read, update, delete” semantics in them for so long that they have started presenting us requirements using that same language, even though it’s an extremely poor fit.

Instead of accepting “delete” as a normal user action, let’s go into why users “delete” stuff, and what they actually intend to do.

The guys in marketing can’t actually make all physical instances of a product disappear – nor would they want to. In talking with these users, we might discover that their intent is quite different:

“What I mean by ‘delete’ is that the product should be discontinued. We don’t want to sell this line of product anymore. We want to get rid of the inventory we have, but not order any more from our supplier. The product shouldn’t appear any more when customers do a product search or category listing, but the guys in the warehouse will still need to manage these items in the interim. It’s much shorter to just say ‘delete’ though.”

There seem to be quite a few interesting business rules and processes there, but nothing that looks like it could be solved by a single database column.

Model the task, not the data

Looking back at the story our friend from marketing told us, his intent is to discontinue the product – not to delete it in any technical sense of the word. As such, we probably should provide a more explicit representation of this task in the user interface than just selecting a row in some grid and clicking the ‘delete’ button (and “Are you sure?” isn’t it).

As we broaden our perspective to more parts of the system, we see this same pattern repeating:

Orders aren’t deleted – they’re cancelled. There may also be fees incurred if the order is canceled too late.

Employees aren’t deleted – they’re fired (or possibly retired). A compensation package often needs to be handled.

Jobs aren’t deleted – they’re filled (or their requisition is revoked).

In all cases, the thing we should focus on is the task the user wishes to perform, rather than on the technical action to be performed on one entity or another. In almost all cases, more than one entity needs to be considered.

Statuses

In all the examples above, what we see is a replacement of the technical action ‘delete’ with a relevant business action. At the entity level, instead of having a (hidden) technical WasDeleted status, we see an explicit business status that users need to be aware of.

The manager of the warehouse needs to know that a product is discontinued so that they don’t order any more stock from the supplier. In today’s world of retail with Vendor Managed Inventory, this often happens together with a modification to an agreement with the vendor, or possibly a cancellation of that agreement.

This isn’t just a case of transactional or reporting boundaries – users in different contexts need to see different things at different times as the status changes to reflect the entity’s place in the business lifecycle. Customers shouldn’t see discontinued products at all. Warehouse workers should, that is, until the corresponding Stock Keeping Unit (SKU) has been revoked (another status) after we’ve sold all the inventory we wanted (and maybe returned the rest back to the supplier).

Rules and Validation

When looking at the world through over-simplified-delete-glasses, we may consider the logic dictating when we can delete to be quite simple: do some role-based-security checks, check that the entity exists, delete. Piece of cake.

The real world is a bigger, more complicated cake.

Let’s consider deleting an order, or rather, canceling it. On top of the regular security checks, we’ve got some rules to consider:

If the order has already been delivered, check if the customer isn’t happy with what they got, and go about returning the order.

If the order contained products “made to order”, charge the customer for a portion (or all) of the order (based on other rules).

And more…

Deciding what the next status should be may very well depend on the current business status of the entity. Deciding if that change of state is allowed is context and time specific – at one point in time the task may have been allowed, but later not. The logic here is not necessarily entirely related to the entity being “deleted” – there may be other entities which need to be checked, and whose status may also need to be changed as well.

Summary

I know that some of you are thinking, “my system isn’t that complex – we can just delete and be done with it”.

My question to you would be, have you asked your users why they’re deleting things? Have you asked them about additional statuses and rules dictating how entities move as groups between them? You don’t want the success of your project to be undermined by that kind of unfounded assumption, do you?

The reason we’re given budgets to build business applications is because of the richness in business rules and statuses that ultimately provide value to users and a competitive advantage to the business. If that value wasn’t there, wouldn’t we be serving our users better by just giving them Microsoft Access?

In closing, given that you’re not giving your users MS Access, don’t think about deleting entities. Look for the reason why. Understand the different statuses that entities move between. Ask which users need to care about which status. I know it doesn’t show up as nicely on your resume as “3 years WXF”, but “saved the company $4 million in wasted inventory” does speak volumes.

One last sentence: Don’t delete. Just don’t.

For reporting, that is.

And doesn’t handle concurrency!

Unless you don’t expose setters.

I guess it depends, doesn’t it?

Well, that was Ted’s assertion in his recent Pragmatic Architecture column on data access.

But, “it depends” doesn’t get the system built, does it?

So, here are some rules for using o/r mapping that will get you 99% of the way there.

Yes, you heard me.

Rules.

They do not depend.

If you’re doing something significantly bigger than enterprise-scale development, and you are already doing this, and it isn’t enough, give me a call. Here we go.

1. No reporting.
   

   I mean it. Don’t report off of live data.
   This isn’t just a o/r mapping thing.
   Users can tolerate some, if not quite a lot of latency.

   And it’s not like objects are even used. It’s just rolled up data. Not a single behaviour for miles.

2. Don’t expose setters
   

   You want multiple users sharing and collaborating on data, right? Then don’t force them to either overwrite each others data, or throw away their own. There is one simple way to avoid that: Get an object, call a method. Once the object has the most up to date data, pass all the client data in via a method call. The object will decide if its valid, from a business perspective as well, and then update the appropriate fields.

   Now your DBAs can vertically partition tables accordingly, and improve throughput. After that, you can increase the isolation level, to improve safety, without hurting throughput.

   This will also keep your logic encapsulated, bringing you closer to a true Domain Model.

   If your O/R mapping tool requires you to have setters on your domain classes, hide those from your service layer behind an interface.

3. Grids are like reports.
   

   No o/r mapping required there either. While you probably won’t be showing grids of yesterday’s data to users in an interactive environment, it’s still just data – no behaviour.

   However, users should NOT update data in those grids. This gets back to rule 2. Have users select a specific task they want to perform, pop open a window, and have them do it there. Change customer address. Discount order. You get the picture. That way you’ll know what method to call on those objects you designed in rule 2.

Before wrapping up, one small thing.

You can use an O/R mapping tool to do reporting, just, for the love of Bill, don’t use the same classes you designed for your OLTP domain model. But, just because you can, doesn’t necessarily mean you should. Datasets datatables are probably just as viable a solution.

“So, which services do I need?”

This innocuous question comes up a lot. Usually I get this question after a short problem domain description. One of these came up on the nServiceBus discussion groups. Ayende took it and ran with it turning it into a nice blog post, An exercise in designing SOA systems. I’ve been meaning to write something myself. Bill put up a response already in his Service Granularity Example. So, I’m late to the party, again, but here we go.

It’s almost impossible to know, right away, which services are appropriate.

So, I’m going to focus more on the process of getting there, rather than describing the solution itself.

The domain deals with a placement agency placing physicians in positions at hospitals.

1. So, what does it actually do?

In Ayende’s post, he describes several services, but I’d rather look at them as use cases: registering an open position, registering a candidate, verifying their credentials, etc. It’s worth going through this requirements process. It doesn’t necessarily translate immediately to services, but there’s value in it.

2. What does it do it to?

We should also be looking at the data model, an entity relationship diagram (ERD) , where we see that we may have placed a certain physician at a number of positions. It’s also important for us to know about under which circumstances a physician finished their employment at a previous position before, say, trying to place them at a position in the same hospital or chain of hospitals. Don’t go thinking that this what the database schema will look like, it’s all about understanding connections between various bits of data.

3. When does that happen?

The next step is to map the uses cases above to the entities in the ERD, which entity is used in which use case. It’s also important to differentiate between entities (or even more importantly, specific fields of entities) that are used in a read-only fashion within a given use case. For instance, when registering a new position, we’ll want to check that against other open positions in the same hospital so we don’t end up registering the same position twice. Also, we might want to suggest verified physicians whose credentials match the position’s requirements. Data we wouldn’t be interested in might be which other physicians we placed at that hospital.

4. What just happened?

Another valuable perspective on the problem domain is the business process view – what are the interesting business events in the system and how they unfold over time. For instance, physician registered, position opened, physician’s credentials verified, and physician placed in position (or position filled by physician) are events that describe a different business perspective than use cases.

5. How do I decide?

Once we know what events there are, we can start looking at what kind of decisions we might want to make when those events occur and what data we’d need to make those decisions. These decisions may be as simple as updating a database or sending an email to a user. They also may include more advanced logic like when the profitability of an agreement with a specific hospital chain changes, prefer placing physicians in positions in that chain over others.

6. How do I deal with all this information?

After we have all of this information, we can start looking for cohesive bunching across all of these axes using these rules:

• Data that is modified by a use case gets published as an event.
• Data that is required by a use case for read-only purposes, arrives as the result of subscribing to some event.

Look for rules that differentiate behaviour based on the properties of data. Look for a correlation to some business concept. For instance, physicians probably won’t be changing their specialization, and open positions often deal with a certain specialization. Therefore, specific data instances tied to two different specializations can be said to be loosely coupled.

7. Which property slices across the domain?

Even though the ERD may not have made it clear, and the use cases didn’t show any particular break-down, nor did the events call out this point, the key to finding the way a business domain decomposes into services lies in decoupling specific data instances.

Actually, at this point we can clump autonomous components (mere technical bits) that handle a single message, into more granular business components.

If you think about it, it makes a lot of sense. The kind of credential checking you’d do for physicians specializing in brain surgery would likely be different than for general practitioners. The kind of information you’d store would, therefore, also be different.

But, which services do I need?

Quite frankly, I don’t have enough information to know.

But if we had continued this conversation, going through issues like transactional consistency, availability requirements, and other non-functional issues we could have  gotten there.

If there’s one thing that I hope you got out of this, it’s that the questions are what’s important. The iterative process of looking at the problem domain from various perspectives, incorporating the new-found knowledge, and asking more questions is what leads us to a solution. But we don’t stop there. We keep looking for characteristics which split services apart into business components, and for consistency requirements that brings autonomous components together into services.

It’s not easy, but by focusing on these simple questions, you can get to a coherent service oriented architecture.

Update: The new and improved solution is now available: Domain Events, Take 2.

Most people getting started with DDD and the Domain Model pattern get stuck on this. For a while I tried answering this on the discussion groups, but here we have a nice example that I can point to next time.

The underlying problem I’ve noticed over the past few years is that developers are still thinking in terms of querying when they need more data. When moving to the Domain Model pattern, you have to “simply” represent the domain concepts in code – in other words, see things you aren’t used to seeing. I’ll highlight that part in the question below so that you can see where I’m going to go with this in my answer:

I have an instance where I believe I need access to a service or repository from my entity to evaluate a business rule but I’m using NHibernate for persistence so I don’t have a real good way to inject services into my entity. Can I get some viewpoints on just passing the services to my entity vs. using a facade?

Let me explain my problem to provide more context to the problem.

The core domain revolves around renting video games. I am working on a new feature to allow customers to trade in old video games. Customers can trade in multiple games at a time so we have a TradeInCart entity that works similar to most shopping carts that everybody is familiar with. However there are several rules that limit the items that can be placed into the TradeInCart. The core rules are:

1. Only 3 games of the same title can be added to the cart.
2. The total number of items in the cart cannot exceed 10.
3. No games can be added to the cart that the customer had previously reported lost with regards to their rental membership.
    a. If an attempt is made to add a previously reported lost game, then we need to log a BadQueueStatusAddAttempt to the persistence store.

So the first 2 rules are easily handled internally by the cart through an Add operation. Sample cart interface is below.

   1:  class TradeInCart{

   2:      Account Account{get;}

   3:      LineItem Add(Game game);

   4:      ValidationResult CanAdd(Game game);

   5:      IList<LineItems> LineItems{get;}

   6:  }

However the #3 rule is much more complicated and can’t be handled internally by the cart, so I have to depend on external services. Splitting up the validation logic for a cart add operation doesn’t seem very appealing to me at all. So I have the option of passing in a repository to get the previously reported lost games and a service to log bad attempts. This makes my cart interface ugly real quick.

   1:  class TradeInCart{

   2:      Account Account{get;}

   3:      LineItem Add(

   4:          Game game, 

   5:          IRepository<QueueHistory> repository, 

   6:          LoggingService service);

   7:   

   8:      ValidationResult CanAdd(

   9:          Game game, 

  10:          IRepository<QueueHistory> repository, 

  11:          LoggingService service);

  12:   

  13:      IList<LineItems> LineItems{get;}

  14:  }

The alternative option is to have a TradeInCartFacade that handles the validations and adding the items to the cart. The façade can have the repository and services injected though DI which is nice, but the big negative is that the cart ends up totally anemic.

Any thought on this would be greatly appreciated.

Thanks,
Jesse

As I highlighted above, the thing that will help you with your business rules is to introduce the Customer object (that you probably already have) with the property GamesReportedLost (an IList<Game>). Your TradeInCart would have a reference to the Customer object and could then check the rule in the Add method.

Before I go into the code, it looks like your Account object might be used the same way, but your description of the domain doesn’t mention accounts, so I’m going to assume that that’s unrelated for now:

   1:  public class Customer{

   2:   

   3:      /* other properties and methods */

   4:   

   5:      private IList<Game> gamesReportedLost;

   6:      public virtual IList<Game> GamesReportedLost 

   7:      { 

   8:          get

   9:          {

  10:              return gamesReportedLost;

  11:          }

  12:          set

  13:          {

  14:              gamesReportedLost = value;

  15:          }

  16:      }

  17:  }

Keep in mind that the GamesReportedLost is a persistent property of Customer. Every time a customer reports a game lost, this list needs to be kept up to date. Here’s the TradeInCart now:

   1:  public class TradeInCart

   2:  {

   3:      /* other properties and methods */

   4:   

   5:      private Customer customer;

   6:      public virtual Customer Customer

   7:      { 

   8:          get { return customer; }

   9:          set { customer = value; }

  10:      }

  11:   

  12:      private IList<LineItem> lineItems;

  13:      public virtual IList<LineItem> LineItems

  14:      {

  15:          get { return lineItems; }

  16:          set { lineItems = value; }

  17:      }

  18:   

  19:      public void Add(Game game)

  20:      {

  21:          if (lineItems.Count >= CONSTANTS.MaxItemsPerCart)

  22:          {

  23:              FailureEvents.RaiseCartIsFullEvent();

  24:              return;

  25:          }

  26:   

  27:          if (NumberOfGameAlreadyInCart(game) >=

  28:              CONSTANTS.MaxNumberOfSameGamePerCart)

  29:          {

  30:              FailureEvents

  31:                .RaiseMaxNumberOfSameGamePerCartReachedEvent();

  32:              return;

  33:          }

  34:   

  35:          if (customer.GamesReportedLost.Contains(game))

  36:              FailureEvents.RaiseGameReportedLostEvent();

  37:          else

  38:              this.lineItems.Add(new LineItem(game));

  39:      }

  40:   

  41:      private int NumberOfGameAlreadyInCart(Game game)

  42:      {

  43:          int result = 0;

  44:   

  45:          foreach(LineItem li in this.lineItems)

  46:              if (li.Game == game)

  47:                  result++;

  48:   

  49:          return result;

  50:      }

  51:  }

  52:   

  53:  public static class FailureEvents

  54:  {

  55:      public static event EventHandler GameReportedLost;

  56:      public static void RaiseGameReportedLostEvent()

  57:      {

  58:           if (GameReportedLost != null)

  59:               GameReportedLost(null, null);

  60:      }

  61:   

  62:      public static event EventHandler CartIsFull;

  63:      public static void RaiseCartIsFullEvent()

  64:      {

  65:           if (CartIsFull != null)

  66:               CartIsFull(null, null);

  67:      }

  68:   

  69:      public static event EventHandler MaxNumberOfSameGamePerCartReached;

  70:      public static void RaiseMaxNumberOfSameGamePerCartReachedEvent()

  71:      {

  72:           if (MaxNumberOfSameGamePerCartReached != null)

  73:               MaxNumberOfSameGamePerCartReached(null, null);

  74:      }

  75:  }

Your service layer class that calls the Add method of TradeInCart would first subscribe to the relevant events in FailureEvents. If one of those events is raised, it would do the necessary logging, external system calls, etc.

As you can see, the API of TradeInCart doesn’t need to make use of any external repositories, nor do you need to inject any other external dependencies in.

One thing I didn’t do in the above code to keep it “short” is to define the relevant custom EventArgs for bubbling up the information as to which game was reported lost or already have 3 of those in the cart. That is something that definitely should be done so that the service layer can pass this information back to the client.

Here’s a look at Service Layer code:

   1:  public class AddGameToCartMessageHandler :

   2:      BaseMessageHandler<AddGameToCartMessage>

   3:  {

   4:      public override void Handle(AddGameToCartMessage m)

   5:      {

   6:          using (ISession session = SessionFactory.OpenSession())

   7:          using (ITransaction tx = session.BeginTransaction())

   8:          {

   9:              TradeInCart cart = session.Get<TradeInCart>(m.CartId);

  10:              Game g = session.Get<Game>(m.GameId);

  11:   

  12:              Domain.FailureEvents.GameReportedLost +=

  13:                gameReportedLost;

  14:              Domain.FailureEvents.CartIsFull +=

  15:                cartIsFull;

  16:              Domain.FailureEvents.MaxNumberOfSameGamePerCartReached +=

  17:                maxNumberOfSameGamePerCartReached;

  18:   

  19:              cart.Add(g);

  20:   

  21:              Domain.FailureEvents.GameReportedLost -=

  22:                gameReportedLost;

  23:              Domain.FailureEvents.CartIsFull -=

  24:                cartIsFull;

  25:              Domain.FailureEvents.MaxNumberOfSameGamePerCartReached -=

  26:                maxNumberOfSameGamePerCartReached;

  27:   

  28:              tx.Commit();

  29:          }

  30:      }

  31:   

  32:      private EventHandler gameReportedLost = delegate { 

  33:            Bus.Return((int)ErrorCodes.GameReportedLost);

  34:          };

  35:   

  36:      private EventHandler cartIsFull = delegate { 

  37:            Bus.Return((int)ErrorCodes.CartIsFull);

  38:          };

  39:   

  40:      private EventHandler maxNumberOfSameGamePerCartReached = delegate { 

  41:            Bus.Return((int)ErrorCodes.MaxNumberOfSameGamePerCartReached);

  42:          };

  43:      }

  44:  }

It’s important to remember to clean up your event subscriptions so that your Service Layer objects get garbage collected. This is one of the primary causes of memory leaks when using static events in your Domain Model. I’m hoping to find ways to use lambdas to decrease this repetitive coding pattern. You might be thinking to yourself that non-static events on your Domain Model objects would be easier, since those objects would get collected, freeing up the service layer objects for collection as well. There’s just on small problem:

The problem is that if an event is raised by a child (or grandchild object), the service layer object may not even know that that grandchild was involved and, as such, would not have subscribed to that event. The only way the service layer could work was by knowing how the Domain Model worked internally – in essence, breaking encapsulation.

If you’re thinking that using exceptions would be better, you’d be right in thinking that that won’t break encapsulation, and that you wouldn’t need all that subscribe/unsubscribe code in the service layer. The only problem is that the Domain Model needs to know that the service layer had a default catch clause so that it wouldn’t blow up. Otherwise, the service layer (or WCF, or nServiceBus) may end up flagging that message as a poison message (Read more about poison messages). You’d also have to be extremely careful about in which environments you used your Domain Model – in other words, your reuse is shot.

Conclusion

I never said it would be easy

However, the solution is simple (not complex). The same patterns occur over and over. The design is consistent. By focusing on the dependencies we now have a domain model that is reusable across many environments (server, client, sql clr, silverlight). The domain model is also testable without resorting to any fancy mock objects.

One closing comment – while I do my best to write code that is consistent with production quality environments, this code is more about demonstrating design principles. As such, I focus more on the self-documenting aspects of the code and have elided many production concerns.

Do you have a better solution?

Something that I haven’t considered?

Do me a favour – leave me a comment. Tell me what you think.