Update – clarification post here.

I was on a call recently with the Advisory Board for the Microsoft Patterns & Practices (P&P) CQRS Journey project where they were showing the current state of their development. Towards the middle of the call, I mentioned that I found there to be too many concerns in one place and that I had expected there to be a division into multiple sub-domains/bounded contexts/business components (BCs). The answer was that they hadn’t gotten to the other areas yet and that’s why at that point in time there was only one BC.

The conversation got a bit derailed at that point, and I was asked how I would do it (though not quite as politely), ultimately leading to my tweeting this:

MS P&P CQRS project asked me to show how I would do the conf mgmt domain My Way. Anyone want to help me show them how to do it right?

— UdiDahan (@UdiDahan) March 21, 2012

I think I got over 50 people who wanted in on this, while some of them urged me to work with P&P rather than separately. I think I’ll do both, hopefully resulting in two implementations that can be compared – one based on Azure (done by P&P) and the other based on NServiceBus (done by my guys). Who do you think is more worried

But first things first

The fundamental flaw that I see happening with many software projects (including the P&P CQRS effort) is that not enough time is spent to understand the underlying business objectives – the thinking behind the use cases / user stories. Developers assume behavior is “like” that of another/similar domain – when the difference in the details matter a lot. That often leads to software boundaries that aren’t properly aligned with those of the business.

The effects of this lack of alignment may be felt only much later in the project, when we get a requirement that just doesn’t fit the architecture we’ve set up. I’ve blogged about the symptoms of this problem about 2 years ago in my post Non-functional architectural woes.

We need to get into the nitty-gritty of our problem domain to find out what makes it special.

Not all e-commerce is equal

Anytime somebody is going to make a purchase online, developers immediately create some kind of “order” entity with a bunch of “order lines”, just like they read about in all the blog posts and books. Then, all sorts of other behavior are shoe-horned around those entities and… voila, a working system.

The domain of conferences is different – we don’t actually ship products when people register so payment concerns are very different. If our company is purchasing 5 tickets to the conference, the number of people (and which specific people) that eventually go to the conference may be very different than the people we had originally registered – there doesn’t tend to be that kind of volatility in traditional B2C retail (like selling books to people online).

It’s also quite likely that if a company is sending many people to a conference that they wouldn’t be paying by credit card – invoicing and payment may happen much later. That is no reason to block registration from completing.

Not all registration systems are equal

I understand how people can look at systems like TicketMaster and use that as a model for this system but, once again, the differences in the domain matter.

First of all, most people don’t purchase movie tickets weeks in advance – conference tickets do go on sale that far in advance. Second, if the movie you want to go to is sold out this week, no big deal, you’ll see it next week – conferences are more of a one-time/yearly deal. Third, you usually go to the movies with family/friends – if you can’t get tickets for everyone, you’ll go next week. When it comes to conferences, there is no “next week”, so whoever can go, does. Also, attendees going to a conference together are usually coworkers, not family – there are less qualms about leaving someone behind.

This is already leading us to a model where we should not view a group registration as a single success or failure affair. This will have an impact on the commands, events, and transactions that flow through our system.

In any case where people are reserving something far in advance, there is a high likelihood of cancellations. This is similar to the domain of hotels/hospitality where you can cancel your reservation up to N days before your arrival at no charge. This also tends to influence the payment structure – we’d rather not have to return people’s money as there can be per-transaction charges for that, instead delaying payment can make sense.

Similar to how hotels overbook by a certain amount (to offset cancellations), our conference might look at doing something similar. The difference is that in the case of a hotel, the guest will likely just book a room in a different hotel in the case the first hotel was fully booked. This probably won’t happen with a conference.

For that reason, we want to remember who wanted to come to our conference even when we thought we were full. You see, our best chance of filling a seat that opened up due to a cancellation is by a person who wanted to register before. What we need here is a waiting list – something that doesn’t make the same kind of sense for hotels or airlines (although airlines do use waiting lists, just that that is usually exposed to travel agents and not to travelers booking online).

First-come, first-served – fairness

The traditional developer thinking about systems is rooted in synchronous and sequential processes. In attempting to give a good user experience, developers want to give the user final confirmation as quickly as possible – whether that’s success or failure.

This results in a first-come, first-served user interaction model – whichever user registers in our conference management system first, the better the chance they’ll get what they want. That sounds like a pretty fair system, the only thing is that fairness was not a requirement.

In the real world, if people are standing in line for tickets, they’d get really upset if the tellers decided somewhat arbitrarily to serve people in the back of the line before those in the front. The great thing about online systems is that nobody can see the “virtual line” – the system can be as unfair as we like and there isn’t a real way for the users to know that this is happening.

Why be unfair?

While conferences, theaters, and airlines all want to have all seats filled, the difference between the ongoing models of airlines and theaters and the once-a-year model of the conference influence how sales are done. Some companies send a lot of employees to our conference so we want to give them preference in registration. This is area that we have the most leverage over – when it comes to the masses who arrive in ones and twos, there’s not very much we can do. It makes sense to bend over backwards for a large group, but not for a small one. A commitment from a large company tends to mean more than that from a small one.

If Boeing has already registered 70 people to your conference and now wants to send 5 more, are you really going to tell them “sorry, we’re fully booked”, or are you going to do everything in your power to keep them happy so that next year they’ll want to keep working with you? Wouldn’t it be nice if you could “unregister” some people to make room for the Beoing guys.

Now, you can’t necessarily do this up until the last minute, but potentially 2 weeks (or whatever) before the event could be reasonable, leaving people the ability to cancel flights and hotels without charges (assuming we tell them during registration that they should buy refundable plane tickets).

The easiest way to “unregister” someone is to not tell them that their registration was confirmed. In short, 2 weeks before the start of the event we finalize all registrations deciding (based on our internal priority) who gets in and who doesn’t. We may have logic that decides to immediately finalize registration from Boeing (and other select customers) without waiting until 2 weeks before the event.

Just don’t look TOO unfair

Appearance is everything. Perception matters. You don’t want to get a reputation for being unfair.

So when we open registration, we can allow the first N people to bypass our waiting list and get accepted right away (payment still needing to be handled later). At that point, you can start moving new registrations through the waiting list.

The thing is, nobody knows that you aren’t actually full at that point

Influences on architecture

I hope you’re getting the impression that this collection of scenarios is going to have a big impact on the design. It indicates to us which parts of the business need to be 100% consistent with each other and which parts can be eventually consistent – ultimately defining where one bounded context stops and another begins. This has a direct impact on the events that we’d end up with – who would publish what, and how many others would subscribe to it.

I know some people will look at the above scenarios and say “but what if the requirements were different?”. The thing is that not all requirements are created equal. In working with our business stakeholders, we need to identify which elements are stable and which are potentially volatile and, yes, that’ll be different in each project. We want to align the main boundaries of our software with the stable business elements.

And don’t even try to create a system so flexible that it could handle any new requirement without any architectural changes – down that path lies madness. User-defined custom fields used in user-defined custom workflows, all of it appearing in reports with sorting, filtering, and grouping. You might as well give your users Visual Studio.

Back to P&P

I don’t know if P&P will adopt this set of requirements for their CQRS Journey. The thing here is that we can see the collaborative nature of the domain quite clearly – multiple actors working in parallel where the decisions of one affect the outcomes of another.

The requirements that I’ve seen being handled in the CQRS Journey so far don’t seem complicated enough to justify anything more than a 2-tier architecture – it’s feeling somewhat over-engineered right now. I know that people in the community see other benefits to CQRS but I’ll have to put up a separate blog post describing why there are other better solutions than CQRS most of the time.

Anyway, I’m willing to see how things progress and tweak these requirements (up to a point) so that both the NServiceBus solution and the Azure solution are addressing the same problem.

In closing

Occasionally I hear people still raising the agile mantra against Big Design/Requirements Up Front. The thing is that Agile Manifesto never said to intentionally bury your head in the sand with regards to the purpose of the system. It was a push-back against spending months in analysis without anything but documents coming out, but the goal was to reach a middle ground. Nobody ever said “no design up-front” or “no requirements up front”.

I’m going to try to work with both P&P and the alumni of my Advanced Distributed Systems Design course to come up with simplest possible solution that addresses the requirements (functional and non-functional).

Hope you’ll find this journey interesting.

Update – clarification post here.

Recently I’ve started talking more about modeling and its relation to the real world.

Here’s where it all starts from:

Don’t try to model the real world, it doesn’t exist.

I know that that sounds like a very Matrix-y kind of statement, so let me explain.

The “Real” World

The problem with the “real” world is that you are limited by the laws of physics. The thing is that somewhere along the history of software development, we got this idea that if only the structure of our software represented physical reality, then our software would be maintainable, flexible, robust, … in short, good.

The thing is that a single physical entity can have multiple meanings to various stakeholders.

Let’s look at something simple, like a glass:

From a developer’s perspective we might call that a Product and not think very much more of it. We’d be happy that we could come up with a single abstraction that allowed us to model all the different kinds of products the same way.

Yet, in talking with our business stakeholders, one might call it inventory, another might call it a liability (think of breakage requiring insurance), and another call it merchandise. The important thing to note is that the data relevant to each of those meanings is so different from one stakeholder to another.

And that brings me to “customer”

One of my least favorite entities – a lingering symptom of the Northwind disease.

When someone walks into your store for the first time (whether that store is physical or virtual), are they a customer? Even if they haven’t ever bought anything? Even if you don’t know their name? Are they even a User then? I mean, it’s not like we’re going to force people to login (or create an account) just to browse the site, right? A term like Visitor, Prospect, or Lead sounds like it would describe this type of concept better.

After wandering around your store for a while, they come up to you and ask for help finding something. If this pattern repeated itself over and over again for the same category of item, would that be meaningful to the business? Don’t you think that should be modeled? I hope your answers are yes, and yes. This is the domain of merchandising, and seems more related to Visitors than to Customers.

Let’s say your selling to other businesses rather than to consumers. In the B2B space, it is common not to receive payment for goods or services for some time – you might have heard terms like Net30, which means you will be paid up to 30 days later (in some cases, this may be from the end of the month of the invoice rather than the date of the invoice).

If you talk to the business folks in charge of these scenarios, you’ll hear them talk about Accounts Payable and Accounts Receivable. Yep, they are the accountants. If you were to go about building a DDD Ubiquitous Language, it sounds like the term Account would be a better choice than Customer. The thing is that accountants use the same language regardless of how quickly an account is settled – like if payment is done by credit card at the time of purchase.

There is no Customer.
There is no Product.

The same goes for so many other problem domains.

I know it feels counter-intuitive to not have a single class representing a single physical thing. It feels like it’s the exact opposite of Domain-Driven Design. It feels anti-object-oriented. But remember, most stakeholders you talk to don’t focus on the physical elements either.

The one thing left to be modeled from “reality”

And that’s identity.

It would be most accurate to say that the physical thing you perceive is nothing more than identity serving to correlate all the separate business concerns to each other. It’s this ID that ties the Visitor on the site, to the Account, to the Addressee (for shipment).

These IDs are needed primarily for reporting and UI reasons – it isn’t likely to have a business action operate on entities correlated this way in the same transaction.

Nouns, Verbs, and Reality

In building your ubiquitous language, look past the nouns and verbs visible on the surface.

Watch out for statements like “in reality…” and “in the real world…” as they are really just one person’s interpretation of their perception of reality. Not one of us is able to see reality clearly – it’s all just perceptions. Recognize that, like models, all perceptions are wrong, but some may be useful.

Model the perceptions – at least you can have first hand experience of those.

Forget about reality – all that exists is perceptions.

In closing

Transcend the physical.

In software there is no gravity, no mass, and as many dimensions as you choose to create.

Break free of the Matrix.

You are the god of your software.

One of the things that surprises some developers that I talk to is that you don’t always get consistency even with end-to-end synchronous communication and a single database. This goes beyond things like isolation levels that some developers are aware of and is particularly significant in multi-user collaborative domains.

The problem

Let’s start with an image to describe the scenario:

The main issue we have here is that the values transaction 2 gets for A and B are those from T0 – before either transaction 1 or 3 completed. The reason this is an issue is that these old values (usually together with some message data) are used to calculate what the new state of C should be.

Traditional optimistic concurrency techniques won’t detect any problem if we don’t touch A or B in transaction 2.

In short, systems today are causing inconsistency.

Some solutions

1. Don’t have transactions which operate on multiple entities (which probably isn’t possible for some of your most important business logic).

2. Turn on multi-version concurrency control – this is called snapshot isolation in MS Sql Server.

Yes, you need to turn it on. It’s off by default.

The good news is that this will stop the writing of inconsistent data to your database.
The bad news is that it will probably cause your system many more exceptions when going to persist.

For those of you who are using transaction messaging with automatic retrying, this will end up as “just” a performance problem (unless you follow the recommendations below). For those of you who are using regular web/wcf services (over tcp/http), you’re “cross cutting” exception management will likely end up discarding all the data submitted in those requests (but since that’s what you’re doing when you run into deadlocks this shouldn’t be news to you).

The solution to the performance issues

Eventual consistency.

Funny isn’t it – all those people who were afraid of eventual consistency got inconsistency instead.

Also, it’s not enough to just have eventual consistency (like between the command and query sides of CQRS). You need to drastically decrease the size of your entities. And the best way of doing that is to partition those entities across multiple business services (also known in DDD lingo as Bounded Contexts) each with its own database.

This is yet another reason why I say that CQRS shouldn’t be the top level architectural breakdown. Very useful within a given business service, yes – though sometimes as small as just some sagas.

Next steps

It may seem unusual that the title of this post implies that SOA is the solution, yet the content clearly states that traditional HTTP-based web services are a problem. Even REST wouldn’t change matters as it doesn’t influence how transactions are managed against a database.

The SOA solution I’m talking about here is the one I’ve spent the last several years blogging about. It’s a different style of SOA which has services stretch up to contain parts of the UI as well as down to contain parts of the database, resulting in a composite UI and multiple databases. This is a drastically different approach than much of the literature on the topic – especially Thomas Erl’s books.

Unfortunately there isn’t a book out there with all of this in it (that I’ve found), and I’m afraid that with my schedule (and family) writing a book is pretty much out of the question. Let’s face it – I’m barely finding time to blog.

The one thing I’m trying to do more of is provide training on these topics. I’ve just finished a course in London, doing another this week in Aarhus Denmark, and another next month in San Francisco (which is now sold out). The next openings this year will be in Stockholm, London; Sydney Australia and Austin Texas will be coming in January of next year. I’ll be coming over to the US more next year so if you missed San Francisco, keep an eye out.

I wish there was more I could do, but I’m only one guy.

Hmm, maybe it’s time to change that.

It isn’t uncommon for me to have a client or student at one of my courses ask me about some kind of workflow tool. This could be Microsoft Workflow Foundation, BizTalk, K2, or some kind of BPEL/orchestration engine. The question usually revolves around using this tool for all workflows in the system as opposed to the SOA-EDA-style publish/subscribe approach I espouse.

The question

The main touted benefit of these workflow-centric architectures is that we don’t have to change the code of the system in order to change its behavior resulting in ultimate flexibility!

Some of you may have already gone down this path and are shaking your heads remembering how your particular road to hell was paved with the exact same good intentions.

Let me explain why these things tend to go horribly wrong.

What’s behind the curtain

It starts with the very nature of workflow – a flow chart, is procedural in nature. First do this, then that, if this, then that, etc. As we’ve experienced first hand in our industry, procedural programming is fine for smaller problems but isn’t powerful enough to handle larger problems. That’s why we’ve come up with object-oriented programming.

I have yet to see an object-oriented workflow drag-and-drop engine. Yes, it works great for simple demo-ware apps. But if you try to through your most complex and volatile business logic at it, it will become a big tangled ball of spaghetti – just like if you were using text rather than pictures to code it.

And that’s one of the fundamental fallacies about these tools – you are still writing code. The fact that it doesn’t look like the rest of your code doesn’t change that fact. Changing the definition of your workflow in the tool IS changing your code.

On productivity

Sometimes people mention how much more productive it would be to use these tools than to write the code “by hand”. Occasionally I hear about an attempt to have “the business” use these tools to change the workflows themselves – without the involvement of developers (”imagine how much faster we could go without those pesky developers!”).

For those of us who have experienced this first-hand, we know that’s all wrong.

If “the business” is changing the workflows without developer involvement, invariably something breaks, and then they don’t know what to do. They haven’t been trained to think the way that developers have – they don’t really know how to debug. So the developers are brought back in anyway and from that point on, the business is once again giving requirements and the devs are the one implementing it.

Now when it comes to developer productivity, I can tell you that the keyboard is at least 10x more productive than the mouse. I can bang out an if statement in code much faster than draggy-dropping a diamond on the canvas, and two other activities for each side of the clause.

On maintainability

Sometimes the visualization of the workflow is presented as being much more maintainable than “regular code”.

When these workflows get to be to big/nested/reused, it ends up looking like the wiring diagram of an Intel chip (or worse). Check out the following diagram taken from the DailyWTF on a customer friendly system:

The bigger these get, the less maintainable they are.

Now, some would push back on this saying that a method with 10,000 lines of code in it may be just as bad, if not worse. The thing is that these workflow tools guide developers down a path where it is very likely to end up with big, monolithic, procedural, nested code. When working in real code, we know we need to take responsibility for the cleanliness of our code using object-orientation, patterns, etc and refactoring things when they get too messy.

Here is where I’d bring up the SOA/pub-sub approach as an alternative – there is no longer this idea of a centralized anything. You have small pieces of code, each encapsulating a single business responsibility, working in concert with each other – reacting to each others events.

Productivity take 2: testing and version control

If you’re going to take your most complex and volatile business logic and put it into these workflow tools, have you thought about how your going to test it? How do you know that it works correctly? It tends to be VERY difficult to unit-test these kinds of workflows.

When a developer is implementing a change request, how do they know what other workflows might have been broken? Do they have to manually go through each and every scenario in the system to find out? How’s that for productivity?

Assuming something did break and the developer wants to see a diff – what’s different in the new workflow from the old one, what would that look like? When working with a team, the ability to diff and merge code is at the base of the overall team productivity.

What would happen to your team if you couldn’t diff or merge code anymore?
In this day and age, it should be considered irresponsible to develop without these version control basics.

In closing

There are some cases where these tools might make sense, but those tend to be much more rare than you’d expect (and there are usually better alternatives anyway). Regardless, the architectural analysis should start without the assumption of centralized workflow, database, or centralized anything for that matter.

If someone tries to push one of these tools/architectures on you, don’t walk away – run!

One of the things I cover early on in my course is the problem with traditional layered architecture driving people to create a business logic layer made up of a bunch of inter-related entities. I see this happening a lot, even though nowadays people are calling that bunch of inter-related entities a “domain model”.

Let me just say this upfront – most inter-related entity models are NOT a domain model.
Here’s why: most transactions don’t respect entity boundaries.

That being said, you don’t always need a domain model.
The domain model pattern’s context is “if you have complicated and everchanging business rules” – right there on page 119 of Patterns of Enterprise Application Architecture.

Persisting the customer’s first name, last name, and middle initial – and later reading and showing that data does not sound either complicated or that it is really going to change that much.

Then there are things like credit limits, that may be on the customer entity as well. It is likely that there are business requirements that expect that value to be consistent with the total value of unpaid orders – data that comes from other entities.

The problem that is created is one of throughput.

Since databases lock an entire row/entity at a time, if one transaction is changing the customer’s first name, the database would block another transaction that tried to change the same customer’s credit limit.

The bigger your entities, the more transactions will likely need to operate on them in parallel, the slower your system will get as the number of transactions increases. This feeds back in on itself as often those blocked transactions will have operated already on some other entity, leaving those locked for longer periods of times, blocking even more transactions.

And the absurd thing is that the business never demanded that the customer’s first name be consistent with the credit limit.

What if we didn’t have a single Customer entity?

What if we had one that contained first name, last name, middle initial and another that contained things like credit limit, status, and risk rating. These entities would be correlated by the same ID, but could be stored in separate tables in the database. That would do away with much of the cascading locking effects drastically improving our throughput as load increases.

And you know what? That division would still respect the 3rd normal form.

Which of these entities do you think would be classified by the business under the “complicated and everchanging rules” category?

And for those entities that are just about data persistence – do you think it’s justified to use 3 tiers? Do we really need a view model which we transform to data transfer objects which we transform to domain objects which we transform to relational tables and then all the way back? Wouldn’t some simpler 2-tier programming suffice – dare I say datasets? Ruby on Rails?

Are we ready to leave behind the assumption that all elements of a given layer must be built the same way?

Not in the business world anyway.

The problem is that, as software developers, we’re all too quick to accept them at face value. We don’t question the requirements – in all fairness, it was never our job to do so. We were the ones that implemented them, preferably quickly.

For example

Let’s say we get the requirement the following requirements:

1. If the order was already shipped, don’t let the user cancel the order.
2. If the order was already cancelled, don’t let the user ship the order.

The race condition here is when we have two users who are looking at the same order, which is neither cancelled nor shipped yet, and each submits a command – one to ship the order, the other to cancel it.

In these cases, the code is simple – just an if statement before performing the relevant command.

So what’s the problem

A microsecond difference in timing shouldn’t make a difference to core business behaviors. Which means that we’ve actually got here is a bug in the requirements. Users are actually dictating solutions here rather than requirements.

Let’s ask our stakeholders, “why shouldn’t we let users cancel a shipped order? I mean, the users don’t want the products.”

And the stakeholders would respond with something like, “well, we don’t want to refund the user’s money then. Or, at least, not all their money. Well, maybe if they return the products in their original packaging, *then* we could give a full refund.”

And as we drilled deeper, “when do refunds need to be given? Right away, in the same transaction?”

The stakeholders would explain, “no, refunds don’t need to be given right away.”

It turns out we were missing the concept of a refund, as well as assuming that all things needed to be processed and enforced immediately. Once we dug into the requirements, we found that there is actually plenty of time to allow both transactions to go through. We just need to add some checks during shipping’s long-running process to see if the order was cancelled, and then to cut the process short.

So is everything a long-running process then?

That’s actually a fair question – long-running processes are a lot more common than at first appears.

What we’re seeing is that cancellation is now a command that has no reason to fail – just like CQRS tells us. When this command is performed, it publishes the OrderCancelled event, which the billing service subscribes to.

Billing then starts a long-running process (a saga, in NServiceBus lingo), also listening to events from the shipping process, ultimately making a decision when a refund should be given, and for how much.

Deeper business analysis

As we discuss matters more with our business stakeholders, we hear that most orders are actually cancelled within an hour of being submitted. It is quite rare for orders to be cancelled days later.

In which case, we could look at modeling the acceptance of an order as a long-running process itself.

When a user places an order, we don’t immediately publish an event indicating the acceptance of an order, instead a saga is kicked off – which opens up a timeout for an hour later. If a cancellation command arrives during that period of time, the user gets a full refund (seeing as we didn’t charge anything since billing didn’t get the accepted event to begin with), and the saga just shuts itself down. If the timeout occurs an hour later, and the saga didn’t get a cancel command, then the order is actually accepted and the event is published.

Yes, sagas are everywhere, once you learn to see with business eyes, and no race conditions are left.

In closing

Any time you see requirements that indicate a race condition, dig deeper.

What you’re likely to find are some additional business concepts as well as the introduction of time and the creation of long-running business processes. The implementation at that point will pivot from being trivial if-statements to being richer sagas.

Keep an eye out.

After listening how the community has interpreted Command-Query Responsibility Segregation I think that the time has come for some clarification. Some have been tying it together to Event Sourcing. Most have been overlaying their previous layered architecture assumptions on it. Here I hope to identify CQRS itself, and describe in which places it can connect to other patterns.

Download as PDF – this is quite a long post.

Why CQRS

Before describing the details of CQRS we need to understand the two main driving forces behind it: collaboration and staleness.

Collaboration refers to circumstances under which multiple actors will be using/modifying the same set of data – whether or not the intention of the actors is actually to collaborate with each other. There are often rules which indicate which user can perform which kind of modification and modifications that may have been acceptable in one case may not be acceptable in others. We’ll give some examples shortly. Actors can be human like normal users, or automated like software.

Staleness refers to the fact that in a collaborative environment, once data has been shown to a user, that same data may have been changed by another actor – it is stale. Almost any system which makes use of a cache is serving stale data – often for performance reasons. What this means is that we cannot entirely trust our users decisions, as they could have been made based on out-of-date information.

Standard layered architectures don’t explicitly deal with either of these issues. While putting everything in the same database may be one step in the direction of handling collaboration, staleness is usually exacerbated in those architectures by the use of caches as a performance-improving afterthought.

A picture for reference

I’ve given some talks about CQRS using this diagram to explain it:

The boxes named AC are Autonomous Components. We’ll describe what makes them autonomous when discussing commands. But before we go into the complicated parts, let’s start with queries:

Queries

If the data we’re going to be showing users is stale anyway, is it really necessary to go to the master database and get it from there? Why transform those 3rd normal form structures to domain objects if we just want data – not any rule-preserving behaviors? Why transform those domain objects to DTOs to transfer them across a wire, and who said that wire has to be exactly there? Why transform those DTOs to view model objects?

In short, it looks like we’re doing a heck of a lot of unnecessary work based on the assumption that reusing code that has already been written will be easier than just solving the problem at hand. Let’s try a different approach:

How about we create an additional data store whose data can be a bit out of sync with the master database – I mean, the data we’re showing the user is stale anyway, so why not reflect in the data store itself. We’ll come up with an approach later to keep this data store more or less in sync.

Now, what would be the correct structure for this data store? How about just like the view model? One table for each view. Then our client could simply SELECT * FROM MyViewTable (or possibly pass in an ID in a where clause), and bind the result to the screen. That would be just as simple as can be. You could wrap that up with a thin facade if you feel the need, or with stored procedures, or using AutoMapper which can simply map from a data reader to your view model class. The thing is that the view model structures are already wire-friendly, so you don’t need to transform them to anything else.

You could even consider taking that data store and putting it in your web tier. It’s just as secure as an in-memory cache in your web tier. Give your web servers SELECT only permissions on those tables and you should be fine.

Query Data Storage

While you can use a regular database as your query data store it isn’t the only option. Consider that the query schema is in essence identical to your view model. You don’t have any relationships between your various view model classes, so you shouldn’t need any relationships between the tables in the query data store.

So do you actually need a relational database?

The answer is no, but for all practical purposes and due to organizational inertia, it is probably your best choice (for now).

Scaling Queries

Since your queries are now being performed off of a separate data store than your master database, and there is no assumption that the data that’s being served is 100% up to date, you can easily add more instances of these stores without worrying that they don’t contain the exact same data. The same mechanism that updates one instance can be used for many instances, as we’ll see later.

This gives you cheap horizontal scaling for your queries. Also, since your not doing nearly as much transformation, the latency per query goes down as well. Simple code is fast code.

Data modifications

Since our users are making decisions based on stale data, we need to be more discerning about which things we let through. Here’s a scenario explaining why:

Let’s say we have a customer service representative who is one the phone with a customer. This user is looking at the customer’s details on the screen and wants to make them a ‘preferred’ customer, as well as modifying their address, changing their title from Ms to Mrs, changing their last name, and indicating that they’re now married. What the user doesn’t know is that after opening the screen, an event arrived from the billing department indicating that this same customer doesn’t pay their bills – they’re delinquent. At this point, our user submits their changes.

Should we accept their changes?

Well, we should accept some of them, but not the change to ‘preferred’, since the customer is delinquent. But writing those kinds of checks is a pain – we need to do a diff on the data, infer what the changes mean, which ones are related to each other (name change, title change) and which are separate, identify which data to check against – not just compared to the data the user retrieved, but compared to the current state in the database, and then reject or accept.

Unfortunately for our users, we tend to reject the whole thing if any part of it is off. At that point, our users have to refresh their screen to get the up-to-date data, and retype in all the previous changes, hoping that this time we won’t yell at them because of an optimistic concurrency conflict.

As we get larger entities with more fields on them, we also get more actors working with those same entities, and the higher the likelihood that something will touch some attribute of them at any given time, increasing the number of concurrency conflicts.

If only there was some way for our users to provide us with the right level of granularity and intent when modifying data. That’s what commands are all about.

Commands

A core element of CQRS is rethinking the design of the user interface to enable us to capture our users’ intent such that making a customer preferred is a different unit of work for the user than indicating that the customer has moved or that they’ve gotten married. Using an Excel-like UI for data changes doesn’t capture intent, as we saw above.

We could even consider allowing our users to submit a new command even before they’ve received confirmation on the previous one. We could have a little widget on the side showing the user their pending commands, checking them off asynchronously as we receive confirmation from the server, or marking them with an X if they fail. The user could then double-click that failed task to find information about what happened.

Note that the client sends commands to the server – it doesn’t publish them. Publishing is reserved for events which state a fact – that something has happened, and that the publisher has no concern about what receivers of that event do with it.

Commands and Validation

In thinking through what could make a command fail, one topic that comes up is validation. Validation is different from business rules in that it states a context-independent fact about a command. Either a command is valid, or it isn’t. Business rules on the other hand are context dependent.

In the example we saw before, the data our customer service rep submitted was valid, it was only due to the billing event arriving earlier which required the command to be rejected. Had that billing event not arrived, the data would have been accepted.

Even though a command may be valid, there still may be reasons to reject it.

As such, validation can be performed on the client, checking that all fields required for that command are there, number and date ranges are OK, that kind of thing. The server would still validate all commands that arrive, not trusting clients to do the validation.

Rethinking UIs and commands in light of validation

The client can make of the query data store when validating commands. For example, before submitting a command that the customer has moved, we can check that the street name exists in the query data store.

At that point, we may rethink the UI and have an auto-completing text box for the street name, thus ensuring that the street name we’ll pass in the command will be valid. But why not take things a step further? Why not pass in the street ID instead of its name? Have the command represent the street not as a string, but as an ID (int, guid, whatever).

On the server side, the only reason that such a command would fail would be due to concurrency – that someone had deleted that street and that that hadn’t been reflected in the query store yet; a fairly exceptional set of circumstances.

Reasons valid commands fail and what to do about it

So we’ve got a well-behaved client that is sending valid commands, yet the server still decides to reject them. Often the circumstances for the rejection are related to other actors changing state relevant to the processing of that command.

In the CRM example above, it is only because the billing event arrived first. But “first” could be a millisecond before our command. What if our user pressed the button a millisecond earlier? Should that actually change the business outcome? Shouldn’t we expect our system to behave the same when observed from the outside?

So, if the billing event arrived second, shouldn’t that revert preferred customers to regular ones? Not only that, but shouldn’t the customer be notified of this, like by sending them an email? In which case, why not have this be the behavior for the case where the billing event arrives first? And if we’ve already got a notification model set up, do we really need to return an error to the customer service rep? I mean, it’s not like they can do anything about it other than notifying the customer.

So, if we’re not returning errors to the client (who is already sending us valid commands), maybe all we need to do on the client when sending a command is to tell the user “thank you, you will receive confirmation via email shortly”. We don’t even need the UI widget showing pending commands.

Commands and Autonomy

What we see is that in this model, commands don’t need to be processed immediately – they can be queued. How fast they get processed is a question of Service-Level Agreement (SLA) and not architecturally significant. This is one of the things that makes that node that processes commands autonomous from a runtime perspective – we don’t require an always-on connection to the client.

Also, we shouldn’t need to access the query store to process commands – any state that is needed should be managed by the autonomous component – that’s part of the meaning of autonomy.

Another part is the issue of failed message processing due to the database being down or hitting a deadlock. There is no reason that such errors should be returned to the client – we can just rollback and try again. When an administrator brings the database back up, all the message waiting in the queue will then be processed successfully and our users receive confirmation.

The system as a whole is quite a bit more robust to any error conditions.

Also, since we don’t have queries going through this database any more, the database itself is able to keep more rows/pages in memory which serve commands, improving performance. When both commands and queries were being served off of the same tables, the database server was always juggling rows between the two.

Autonomous Components

While in the picture above we see all commands going to the same AC, we could logically have each command processed by a different AC, each with it’s own queue. That would give us visibility into which queue was the longest, letting us see very easily which part of the system was the bottleneck. While this is interesting for developers, it is critical for system administrators.

Since commands wait in queues, we can now add more processing nodes behind those queues (using the distributor with NServiceBus) so that we’re only scaling the part of the system that’s slow. No need to waste servers on any other requests.

Service Layers

Our command processing objects in the various autonomous components actually make up our service layer. The reason you don’t see this layer explicitly represented in CQRS is that it isn’t really there, at least not as an identifiable logical collection of related objects – here’s why:

In the layered architecture (AKA 3-Tier) approach, there is no statement about dependencies between objects within a layer, or rather it is implied to be allowed. However, when taking a command-oriented view on the service layer, what we see are objects handling different types of commands. Each command is independent of the other, so why should we allow the objects which handle them to depend on each other?

Dependencies are things which should be avoided, unless there is good reason for them.

Keeping the command handling objects independent of each other will allow us to more easily version our system, one command at a time, not needing even to bring down the entire system, given that the new version is backwards compatible with the previous one.

Therefore, keep each command handler in its own VS project, or possibly even in its own solution, thus guiding developers away from introducing dependencies in the name of reuse (it’s a fallacy). If you do decide as a deployment concern, that you want to put them all in the same process feeding off of the same queue, you can ILMerge those assemblies and host them together, but understand that you will be undoing much of the benefits of your autonomous components.

Whither the domain model?

Although in the diagram above you can see the domain model beside the command-processing autonomous components, it’s actually an implementation detail. There is nothing that states that all commands must be processed by the same domain model. Arguably, you could have some commands be processed by transaction script, others using table module (AKA active record), as well as those using the domain model. Event-sourcing is another possible implementation.

Another thing to understand about the domain model is that it now isn’t used to serve queries. So the question is, why do you need to have so many relationships between entities in your domain model?

(You may want to take a second to let that sink in.)

Do we really need a collection of orders on the customer entity? In what command would we need to navigate that collection? In fact, what kind of command would need any one-to-many relationship? And if that’s the case for one-to-many, many-to-many would definitely be out as well. I mean, most commands only contain one or two IDs in them anyway.

Any aggregate operations that may have been calculated by looping over child entities could be pre-calculated and stored as properties on the parent entity. Following this process across all the entities in our domain would result in isolated entities needing nothing more than a couple of properties for the IDs of their related entities – “children” holding the parent ID, like in databases.

In this form, commands could be entirely processed by a single entity – viola, an aggregate root that is a consistency boundary.

Persistence for command processing

Given that the database used for command processing is not used for querying, and that most (if not all) commands contain the IDs of the rows they’re going to affect, do we really need to have a column for every single domain object property? What if we just serialized the domain entity and put it into a single column, and had another column containing the ID? This sounds quite similar to key-value storage that is available in the various cloud providers. In which case, would you really need an object-relational mapper to persist to this kind of storage?

You could also pull out an additional property per piece of data where you’d want the “database” to enforce uniqueness.

I’m not suggesting that you do this in all cases – rather just trying to get you to rethink some basic assumptions.

Let me reiterate

How you process the commands is an implementation detail of CQRS.

Keeping the query store in sync

After the command-processing autonomous component has decided to accept a command, modifying its persistent store as needed, it publishes an event notifying the world about it. This event often is the “past tense” of the command submitted:

MakeCustomerPerferredCommand -> CustomerHasBeenMadePerferredEvent

The publishing of the event is done transactionally together with the processing of the command and the changes to its database. That way, any kind of failure on commit will result in the event not being sent. This is something that should be handled by default by your message bus, and if you’re using MSMQ as your underlying transport, requires the use of transactional queues.

The autonomous component which processes those events and updates the query data store is fairly simple, translating from the event structure to the persistent view model structure. I suggest having an event handler per view model class (AKA per table).

Here’s the picture of all the pieces again:

Bounded Contexts

While CQRS touches on many pieces of software architecture, it is still not at the top of the food chain. CQRS if used is employed within a bounded context (DDD) or a business component (SOA) – a cohesive piece of the problem domain. The events published by one BC are subscribed to by other BCs, each updating their query and command data stores as needed.

UI’s from the CQRS found in each BC can be “mashed up” in a single application, providing users a single composite view on all parts of the problem domain. Composite UI frameworks are very useful for these cases.

Summary

CQRS is about coming up with an appropriate architecture for multi-user collaborative applications. It explicitly takes into account factors like data staleness and volatility and exploits those characteristics for creating simpler and more scalable constructs.

One cannot truly enjoy the benefits of CQRS without considering the user-interface, making it capture user intent explicitly. When taking into account client-side validation, command structures may be somewhat adjusted. Thinking through the order in which commands and events are processed can lead to notification patterns which make returning errors unnecessary.

While the result of applying CQRS to a given project is a more maintainable and performant code base, this simplicity and scalability require understanding the detailed business requirements and are not the result of any technical “best practice”. If anything, we can see a plethora of approaches to apparently similar problems being used together – data readers and domain models, one-way messaging and synchronous calls.

Although this blog post is over 3000 words (a record for this blog), I know that it doesn’t go into enough depth on the topic (it takes about 3 days out of the 5 of my Advanced Distributed Systems Design course to cover everything in enough depth). Still, I hope it has given you the understanding of why CQRS is the way it is and possibly opened your eyes to other ways of looking at the design of distributed systems.

Questions and comments are most welcome.

So, I’ve gotten back from a most enjoyable couple of days in Sweden where I gave two half-day tutorials, the first being the SOA and UI composition talk I gave at the European Virtual ALT.NET meeting (which you can find online here) and the other on DDD in enterprise apps (the first time I’ve done this talk).

I’ve gotten some questions about my DDD presentation there based on Aaron Jensen’s pictures:

Yes – I talk with my hands. All the time.

That slide is quite an important one – I talked about it for at least 2 hours.

Here it is again, this time in full:

You may notice that the nice clean layered abstraction that the industry has gotten so comfortable with doesn’t quite sit right when looking at it from this perspective. The reason for that is that this perspective takes into account physical distribution while layers don’t.

I’ll have some more posts on this topic as well as giving a session in TechEd Europe this November.

Oh – and please do feel free to already send your questions in.

After reading Ayende’s post advocating against “soft deletes” I felt that I should add a bit more to the topic as there were some important business semantics missing. As developers discuss the pertinence of using an IsDeleted column in the database to mark deletion, and the way this relates to reporting and auditing concerns is weighed, the core domain concepts rarely get a mention. Let’s first understand the business scenarios we’re modeling, the why behind them, before delving into the how of implementation.

The real world doesn’t cascade

Let’s say our marketing department decides to delete an item from the catalog. Should all previous orders containing that item just disappear? And cascading farther, should all invoices for those orders be deleted as well? Going on, would we have to redo the company’s profit and loss statements?

Heaven forbid.

So, is Ayende wrong? Do we really need soft deletes after all?

On the one hand, we don’t want to leave our database in an inconsistent state with invoices pointing to non-existent orders, but on the other hand, our users did ask us to delete an entity.

Or did they?

When all you have is a hammer…

We’ve been exposing users to entity-based interfaces with “create, read, update, delete” semantics in them for so long that they have started presenting us requirements using that same language, even though it’s an extremely poor fit.

Instead of accepting “delete” as a normal user action, let’s go into why users “delete” stuff, and what they actually intend to do.

The guys in marketing can’t actually make all physical instances of a product disappear – nor would they want to. In talking with these users, we might discover that their intent is quite different:

“What I mean by ‘delete’ is that the product should be discontinued. We don’t want to sell this line of product anymore. We want to get rid of the inventory we have, but not order any more from our supplier. The product shouldn’t appear any more when customers do a product search or category listing, but the guys in the warehouse will still need to manage these items in the interim. It’s much shorter to just say ‘delete’ though.”

There seem to be quite a few interesting business rules and processes there, but nothing that looks like it could be solved by a single database column.

Model the task, not the data

Looking back at the story our friend from marketing told us, his intent is to discontinue the product – not to delete it in any technical sense of the word. As such, we probably should provide a more explicit representation of this task in the user interface than just selecting a row in some grid and clicking the ‘delete’ button (and “Are you sure?” isn’t it).

As we broaden our perspective to more parts of the system, we see this same pattern repeating:

Orders aren’t deleted – they’re cancelled. There may also be fees incurred if the order is canceled too late.

Employees aren’t deleted – they’re fired (or possibly retired). A compensation package often needs to be handled.

Jobs aren’t deleted – they’re filled (or their requisition is revoked).

In all cases, the thing we should focus on is the task the user wishes to perform, rather than on the technical action to be performed on one entity or another. In almost all cases, more than one entity needs to be considered.

Statuses

In all the examples above, what we see is a replacement of the technical action ‘delete’ with a relevant business action. At the entity level, instead of having a (hidden) technical WasDeleted status, we see an explicit business status that users need to be aware of.

The manager of the warehouse needs to know that a product is discontinued so that they don’t order any more stock from the supplier. In today’s world of retail with Vendor Managed Inventory, this often happens together with a modification to an agreement with the vendor, or possibly a cancellation of that agreement.

This isn’t just a case of transactional or reporting boundaries – users in different contexts need to see different things at different times as the status changes to reflect the entity’s place in the business lifecycle. Customers shouldn’t see discontinued products at all. Warehouse workers should, that is, until the corresponding Stock Keeping Unit (SKU) has been revoked (another status) after we’ve sold all the inventory we wanted (and maybe returned the rest back to the supplier).

Rules and Validation

When looking at the world through over-simplified-delete-glasses, we may consider the logic dictating when we can delete to be quite simple: do some role-based-security checks, check that the entity exists, delete. Piece of cake.

The real world is a bigger, more complicated cake.

Let’s consider deleting an order, or rather, canceling it. On top of the regular security checks, we’ve got some rules to consider:

If the order has already been delivered, check if the customer isn’t happy with what they got, and go about returning the order.

If the order contained products “made to order”, charge the customer for a portion (or all) of the order (based on other rules).

And more…

Deciding what the next status should be may very well depend on the current business status of the entity. Deciding if that change of state is allowed is context and time specific – at one point in time the task may have been allowed, but later not. The logic here is not necessarily entirely related to the entity being “deleted” – there may be other entities which need to be checked, and whose status may also need to be changed as well.

Summary

I know that some of you are thinking, “my system isn’t that complex – we can just delete and be done with it”.

My question to you would be, have you asked your users why they’re deleting things? Have you asked them about additional statuses and rules dictating how entities move as groups between them? You don’t want the success of your project to be undermined by that kind of unfounded assumption, do you?

The reason we’re given budgets to build business applications is because of the richness in business rules and statuses that ultimately provide value to users and a competitive advantage to the business. If that value wasn’t there, wouldn’t we be serving our users better by just giving them Microsoft Access?

In closing, given that you’re not giving your users MS Access, don’t think about deleting entities. Look for the reason why. Understand the different statuses that entities move between. Ask which users need to care about which status. I know it doesn’t show up as nicely on your resume as “3 years WXF”, but “saved the company $4 million in wasted inventory” does speak volumes.

One last sentence: Don’t delete. Just don’t.

For reporting, that is.

And doesn’t handle concurrency!

Unless you don’t expose setters.

I guess it depends, doesn’t it?

Well, that was Ted’s assertion in his recent Pragmatic Architecture column on data access.

But, “it depends” doesn’t get the system built, does it?

So, here are some rules for using o/r mapping that will get you 99% of the way there.

Yes, you heard me.

Rules.

They do not depend.

If you’re doing something significantly bigger than enterprise-scale development, and you are already doing this, and it isn’t enough, give me a call. Here we go.

1. No reporting.
   

   I mean it. Don’t report off of live data.
   This isn’t just a o/r mapping thing.
   Users can tolerate some, if not quite a lot of latency.

   And it’s not like objects are even used. It’s just rolled up data. Not a single behaviour for miles.

2. Don’t expose setters
   

   You want multiple users sharing and collaborating on data, right? Then don’t force them to either overwrite each others data, or throw away their own. There is one simple way to avoid that: Get an object, call a method. Once the object has the most up to date data, pass all the client data in via a method call. The object will decide if its valid, from a business perspective as well, and then update the appropriate fields.

   Now your DBAs can vertically partition tables accordingly, and improve throughput. After that, you can increase the isolation level, to improve safety, without hurting throughput.

   This will also keep your logic encapsulated, bringing you closer to a true Domain Model.

   If your O/R mapping tool requires you to have setters on your domain classes, hide those from your service layer behind an interface.

3. Grids are like reports.
   

   No o/r mapping required there either. While you probably won’t be showing grids of yesterday’s data to users in an interactive environment, it’s still just data – no behaviour.

   However, users should NOT update data in those grids. This gets back to rule 2. Have users select a specific task they want to perform, pop open a window, and have them do it there. Change customer address. Discount order. You get the picture. That way you’ll know what method to call on those objects you designed in rule 2.

Before wrapping up, one small thing.

You can use an O/R mapping tool to do reporting, just, for the love of Bill, don’t use the same classes you designed for your OLTP domain model. But, just because you can, doesn’t necessarily mean you should. Datasets datatables are probably just as viable a solution.