Reliability

Locking in SqlWorkflowPersistenceService (via Ron Jacobs) where, if you want predictable persistence (MS: ‘none of our customers asked for this to be easy’), you need to use a custom activity (which Ron was kind enough to supply).

“Given what I learned today I’d have to say that I’d be very careful about using workflows with an optimistic locking.  Detecting these types of situations is not that simple.”

Let’s think about that. If we’re doing pessimistic locking, we get into the problem of, if a host restarts (as the result of a critical windows patch or some other unexpected occurrence), that the workflow won’t be able to be handled by any other host in the meantime (you didn’t care so much about your SLA, did you?).

Luckily, someone’s come up with a hack that works around this robustness problem in Scalable Workflow Persistence and Ownership.

“So this code will attempt to load workflow instances with expired locks every second. Is it a hack? Yes. But without one of two things in the SqlWorkflowPersistenceService its the sort of code you have to write to pick up unlocked workflow instances robustly.”

This will seriously churn the table used to store your workflows, decreasing performance of workflows that haven’t timed out. Oh well.

Testability

Implementing WCF Services without Referencing WCF (via Mark Seemann):

“More than a year ago, I wrote my first post on unit testing WCF services. One of my points back then was that you have to be careful that the service implementation doesn’t use any of the services provided by the WCF runtime environment (if you want to keep the service testable). As soon as you invoke something like OperationContext.Current, your code is not going to work in a unit testing scenario, but only when hosted by WCF.”

After pointing out some of the more basic difficulties in testability a straightforward WCF implementation brings, Mark turns the heat up in his follow-up post, Modifying Behavior of WCF-Free Service Implementations:

“Perhaps you need to control the service’s ConcurrencyMode, or perhaps you need to set UseSynchronizationContext. These options are typically controlled by the ServiceBehaviorAttribute. You may also want to provide an IInstanceProvider via a custom attribute that implements IContractBehavior. However, you can’t set these attributes on the service implementation itself, since it mustn’t have a reference to System.ServiceModel.”

Wow – all the things required to make a WCF service scalable and thread-safe make it difficult to test. In the end, we’re beginning to see how many hoops we have to go through in order to get separation of concerns, but until we can take all this and get it out of our application code, it’s an untenable solution. I hope Mark will continue with this series, if only so I can take the framework that might grow out of it and use it as a generic WCF transport for NServiceBus.

Comparison

After the Neuron-NServiceBus comparison that Sam and I had, we talked some more. After going through some of the rational and thinking, Sam even put nServiceBus into his WCF-Neuron comparison talk. Sam had this to say about nServiceBus:

“The bottom line is: I like what I see. Although it’s a framework, not an ESB product like Neuron, it’s a powerful framework that takes the right approach on SOA and enforces a paradigm of reliable one-way, *non-blocking* calls. That is the point of the talk tonight overall; we need to get away from the stack world of synchronous RPC calls to true asynchronous non-blocking message based SOA systems.”

The main concern I have with a WCF+WF based solution is that developers need to know a lot in order to make it testable, scalable, and robust. In nServiceBus, that’s baked into the design. It would be extremely difficult for a developer writing application logic to interfere with when persistence needs to happen, or the concurrency strategy of long-running workflows. The fact that message handlers in the service layer don’t need concurrency modes, instance providers, or any of that junk make them testable by default.

Think about that a second.

Scary, isn’t it?

At TechEd Israel I had a discussion with Manu on this very issue, just under a different hat:

What’s the difference between a short-running workflow and a long-running one?

Manu suggested that we look at the actual time that things ran to differentiate between them. I asserted that if any external communication was involved in some part of state-management logic, that logic should automatically be treated as long-running.

Manu’s reasoning was that the complexity involved in writing long-running workflows was not justified for things that ran quickly, even if there was communication involved. Many developers don’t think twice about synchronously calling some web services in the middle of their database transaction logic. In the many Microsoft presentations I’ve been at on WF, not once has it been mentioned that state machines should be used when external communication is involved.

The problem that I have with this guidance is how do you know how quickly a remote call will return?

Do you just run it all locally on your machine, measure, and if it doesn’t take more than a second or so, then you’re OK?

The fact of the matter is that we can never know what the response time of a remote call will be. Maybe the remote machine is down. Maybe the remote process is down. Maybe someone changed the firewall settings and now we’re doing 10KB/s instead of 10MB/s. Maybe the local service is down and we’re communicating with the backup on the other side of the Pacific Ocean.

But the thing is, Manu’s right.

Writing long-running workflows (with WF) is more complex than is justified. My guess is that since WF wasn’t specifically designed for long-running workflows only, that this complexity crept in.

Sagas in nServiceBus were specifically designed for long-running workflows only.

Maybe that’s what kept them simple.

Since all external communication is done via one-way, non-blocking messaging only, each step of a saga runs as quick as if no communication were done at all. This keeps the time the transaction in charge of handling a message is open as short as possible. That, in turn, leads to the database being able to support more concurrent users.

In short, sagas are both more scalable and more robust.

No need to worry about garbaging-up your database.

Well, because Microsoft (or insert vendor of your choice here – they’re all equally guilty of this) tend to deprecate (as in kill) the technology they evangelised just last year/month/week.

Microsoft Sql Server Notification Services are the latest victim.

I hope you don’t have any application code tied to that technology.

Not that it’s the only one.

Workflow Foundation’s warts have started coming out from behind the shiny veneer. It turns out that the threading model is… problematic and requires all sorts of workarounds. Hope those are stable. It’s not like they could have known that we need a high performance way to run our business logic out of the box. I hope you don’t have to change your application code (sorry, pictures diagrams) when you get blocked threads when trying to cancel irrelevant workflows (customer no longer does business with us – cancel order processing workflows).

I forgot to mention that the solution above is for single-box parallelism – if you want true scale-out, you need to go back to solution that “require talented software developer use of call-external-method and handle-external-event activities along with the CLR thread-pool“. That’s OK – I have yet to meet a team/company who attests that they have below average developers.

I apologize for the somewhat sarcastic tone of this post.

It’s just that I’m sick of Microsoft handing developers razor-sharp knives, pointy end forward, and after the developer loses a couple of fingers mentions “oh yah, watch out for these pointy, sharp bits”.

To the developers out there – maybe we need kevlar suits before handling these hazardous materials.

To Microsoft – you think that this doesn’t alienate your customers?

We’re all in the same boat together.

I’m hoping that ALT.NET can help.

When I was at TechEd Barcelona last November (07), the morning after I flew in I experienced “the fish bowl” and Virtual TechEd for the first time. Anyway, after a short chat – and quite to my surprise, my interviewer, Paul Foster, decided that we should talk about nServiceBus.

So here it is. The Microsoft/Marketing friendly description of what nServiceBus is and how nicely it plays with things like WCF and WF. Always be a gracious guest. Don’t bite the hand that feeds you. But a nibble here and there – well, that you can get away with

Download:

Not so.

The other day I sat down to implement and test a non-trivial business process, and the testing was far from easy. Now as developers go, I’m not great, or an expert on unit testing or TDD, but I’m above average. It should not have been this hard. And I tried doing it with Rhino.Mocks, TypeMock, and finally Moq. It seemed like I was in a no-mans-land, between trying to do state-based testing, and setting expectations on the messages being sent (as well as correct values in those messages), nothing flowed.

Until I finally stopped trying to figure out how to test, and focused on what needed to be tested. I mean, it’s not like I was trying to build a generic mocking framework like Daniel.

Here’s an example business process, or actually, part of one, and then we’ll see how that can be tested. By the way, there will be a post coming soon which describes how we go about analysing a system, coming up with these message types, and how these sagas come into being, so stay tuned. Either that, or just come to my tutorial at QCon.

On with the process:

1. When we receive a CreateOrderMessage, whose “Completed” flag is true, we’ll send 2 AuthorizationRequestMessages to internal systems (for managers to authorize the order), one OrderStatusUpdatedMessage to the caller with a status “Received”, and a TimeoutMessage to the TimeoutManager requesting to be notified – so that the process doesn’t get stuck if one or both messages don’t get a response.

2. When we receive the first AuthorizationResponseMessage, we notify the initiator of the Order by sending them a OrderStatusUpdatedMessage with a status “Authorized1”.

3. When we get “timed out” from the TimeoutManager, we check if at least one AuthorizationResponseMessage has arrived, and if so, publish an OrderAcceptedMessage, and notify the initator (again via the OrderStatusUpdatedMessage) this time with a status of “Accepted”.

And here’s the test:

    public class OrderSagaTests
    {
        private OrderSaga orderSaga = null;
        private string timeoutAddress;
        private Saga Saga;     

        [SetUp]
        public void Setup()
        {
            timeoutAddress = "timeout";
            Saga = Saga.Test(out orderSaga, timeoutAddress);
        }     

        [Test]
        public void OrderProcessingShouldCompleteAfterOneAuthorizationAndOneTimeout()
        {
            Guid externalOrderId = Guid.NewGuid();
            Guid customerId = Guid.NewGuid();
            string clientAddress = "client";     

            CreateOrderMessage createOrderMsg = new CreateOrderMessage();
            createOrderMsg.OrderId = externalOrderId;
            createOrderMsg.CustomerId = customerId;
            createOrderMsg.Products = new List<Guid>(new Guid[] { Guid.NewGuid() });
            createOrderMsg.Amounts = new List<float>(new float[] { 10.0F });
            createOrderMsg.Completed = true;     

            TimeoutMessage timeoutMessage = null;     

            Saga.WhenReceivesMessageFrom(clientAddress)
                .ExpectSend<AuthorizeOrderRequestMessage>(
                    delegate(AuthorizeOrderRequestMessage m)
                    {
                        return m.SagaId == orderSaga.Id;
                    })
                .ExpectSend<AuthorizeOrderRequestMessage>(
                    delegate(AuthorizeOrderRequestMessage m)
                    {
                        return m.SagaId == orderSaga.Id;
                    })
                .ExpectSendToDestination<OrderStatusUpdatedMessage>(
                    delegate(string destination, OrderStatusUpdatedMessage m)
                    {
                        return m.OrderId == externalOrderId && destination == clientAddress;
                    })
                .ExpectSendToDestination<TimeoutMessage>(
                    delegate(string destination, TimeoutMessage m)
                    {
                        timeoutMessage = m;
                        return m.SagaId == orderSaga.Id && destination == timeoutAddress;
                    })
                .When(delegate { orderSaga.Handle(createOrderMsg); });     

            Assert.IsFalse(orderSaga.Completed);     

            AuthorizeOrderResponseMessage response = new AuthorizeOrderResponseMessage();
            response.ManagerId = Guid.NewGuid();
            response.Authorized = true;
            response.SagaId = orderSaga.Id;     

            Saga.ExpectSendToDestination<OrderStatusUpdatedMessage>(
                    delegate(string destination, OrderStatusUpdatedMessage m)
                    {
                        return (destination == clientAddress &&
                                m.OrderId == externalOrderId &&
                                m.Status == OrderStatus.Authorized1);
                    })
                .When(delegate { orderSaga.Handle(response); });     

            Assert.IsFalse(orderSaga.Completed);     

            Saga.ExpectSendToDestination<OrderStatusUpdatedMessage>(
                    delegate(string destination, OrderStatusUpdatedMessage m)
                    {
                        return (destination == clientAddress &&
                                m.OrderId == externalOrderId &&
                                m.Status == OrderStatus.Accepted);
                    })
                .ExpectPublish<OrderAcceptedMessage>(
                    delegate(OrderAcceptedMessage m)
                    {
                        return (m.CustomerId == customerId);
                    })
                .When(delegate { orderSaga.Timeout(timeoutMessage.State); });     

            Assert.IsTrue(orderSaga.Completed);
        }
    }

You might notice that this style is a bit similar to the fluent testing found in Rhino Mocks. That’s not coincidence. It actually makes use of Rhino Mocks internally. The thing that I discovered was that in order to test these sagas, you don’t need to actually see a mocking framework. All you should have to do is express how messages get sent, and under what criteria those messages are valid.

If you’re wondering what the OrderSaga looks like, you can find the code right here. It’s not a complete business process implementation, but its enough to understand how one would look like:

using System;
using System.Collections.Generic;
using ExternalOrderMessages;
using NServiceBus.Saga;
using NServiceBus;
using InternalOrderMessages;     

namespace ProcessingLogic
{
    [Serializable]
    public class OrderSaga : ISaga<CreateOrderMessage>,
        ISaga<AuthorizeOrderResponseMessage>,
        ISaga<CancelOrderMessage>
    {
        #region config info     

        [NonSerialized]
        private IBus bus;
        public IBus Bus
        {
            set { this.bus = value; }
        }     

        [NonSerialized]
        private Reminder reminder;
        public Reminder Reminder
        {
            set { this.reminder = value; }
        }     

        #endregion     

        private Guid id;
        private bool completed;
        public string clientAddress;
        public Guid externalOrderId;
        public int numberOfPendingAuthorizations = 2;
        public List<CreateOrderMessage> orderItems = new List<CreateOrderMessage>();     

        public void Handle(CreateOrderMessage message)
        {
            this.clientAddress = this.bus.SourceOfMessageBeingHandled;
            this.externalOrderId = message.OrderId;     

            this.orderItems.Add(message);     

            if (message.Completed)
            {
                for (int i = 0; i < this.numberOfPendingAuthorizations; i++)
                {
                    AuthorizeOrderRequestMessage req = new AuthorizeOrderRequestMessage();
                    req.SagaId = this.id;
                    req.OrderData = orderItems;     

                    this.bus.Send(req);
                }
            }     

            this.SendUpdate(OrderStatus.Recieved);     

            this.reminder.ExpireIn(message.ProvideBy - DateTime.Now, this, null);
        }     

        public void Timeout(object state)
        {
            if (this.numberOfPendingAuthorizations <= 1)
                this.Complete();
        }     

        public Guid Id
        {
            get { return id; }
            set { id = value; }
        }     

        public bool Completed
        {
            get { return completed; }
        }     

        public void Handle(AuthorizeOrderResponseMessage message)
        {
            if (message.Authorized)
            {
                this.numberOfPendingAuthorizations--;     

                if (this.numberOfPendingAuthorizations == 1)
                    this.SendUpdate(OrderStatus.Authorized1);
                else
                {
                    this.SendUpdate(OrderStatus.Authorized2);
                    this.Complete();
                }
            }
            else
            {
                this.SendUpdate(OrderStatus.Rejected);
                this.Complete();
            }
        }     

        public void Handle(CancelOrderMessage message)
        {     

        }     

        private void SendUpdate(OrderStatus status)
        {
            OrderStatusUpdatedMessage update = new OrderStatusUpdatedMessage();
            update.OrderId = this.externalOrderId;
            update.Status = status;     

            this.bus.Send(this.clientAddress, update);
        }     

        private void Complete()
        {
            this.completed = true;     

            this.SendUpdate(OrderStatus.Accepted);     

            OrderAcceptedMessage accepted = new OrderAcceptedMessage();
            accepted.Products = new List<Guid>(this.orderItems.Count);
            accepted.Amounts = new List<float>(this.orderItems.Count);     

            this.orderItems.ForEach(delegate(CreateOrderMessage m)
                                        {
                                            accepted.Products.AddRange(m.Products);
                                            accepted.Amounts.AddRange(m.Amounts);
                                            accepted.CustomerId = m.CustomerId;
                                        });     

            this.bus.Publish(accepted);
        }
    }
}

All this code is online in the subversion repository under /Samples/Saga.

Questions, comments, and general thoughts are always appreciated.

The (pdf) presentation is here: Asynchronous Systems Architecture for the Web

The code sample is here: Asynchronous User Management Code Sample

In the sample, you can see the use of sagas to manage the user registration process; store user email and hashed password, send a confirmation “email”, when the user clicks the “link”, the web server will take the saga id found in the url, and send a message with that id. This will cause the saga to complete and the user to be written to the “database”.

Since I didn’t have an email component on my laptop, and I’m guessing you don’t either, the saga just writes the url to the console. Copy and paste it from there into the browser, and you’re good to go.

A Word on TimeoutExceptions

One other thing that I want to call to your attention. When stepping-through the code in the debugger, you’re liable to spend more time than the Transaction Coordinator likes, which will cause it to rollback and try the message again. This is supposed to happen and occurs by design.

When you’re actually working with a database in a high performance environment, there will be cases where one transaction locks a page of a table and may cause other transactions to either timeout or be chosen as victims and just tossed. The behavior that best handles this scenario is just to retry the transaction.

However, you don’t have to write ugly code that checks for the specific error codes of each specific database for your code to work properly. The infrastructure will automatically do that for you – just let the exception happen. No need to write any try-catch code.

The sample is built on the newly released version of nServiceBus (1.6.1) but already contains all the binaries so you don’t have to set anything up yourself.

What’s coming for nServiceBus

We’re working towards a 2.0 release in the June-July timeframe which, beyond having the necessary documentation, web site, samples and everything any self-respecting open-source project has, is going to have some amazing grid-style features that will make all the message-priority & dynamic-routing stuff look “so last year”. Stay tuned.

“What’s wrong with Workflow Foundation? Why aren’t you using it like you use WCF?”

There’s nothing wrong with Workflow Foundation. The thing is that nServiceBus doesn’t really need workflow in the general sense of the term. An older term that’s been used in the DBMS community might make more sense – “long-lived transactions”. You see, nServiceBus requires state management over many messaging interactions, and thus, some kind of long-lived transaction to maintain consistency.

In 1987, a different model was introduced for handling these scenarios – the Saga [GARCIA-MOLINA 87], and was further expanded in 1992 [CHRYSANTHIS 92] to allow the saga to commit if a non-vital subset of the sub-transactions abort. This is what is used in nServiceBus.

When used in distributed manner on top of one-way messaging, this results in a solution where each service runs its own “mini-workflow”, and coordinates its actions with other services via messages. This integration style is different from the traditional broker, man-in-the-middle approach found in products like Biztalk; and is known as “choreography”, and is in the process of standardization in the WS-* specs, known as WS-Choreography.

So, the bottom line is that the source, examples, and documentation of nServiceBus is being moved in this direction. The source and examples on the sourceforge site have already been brought forward. This is a breaking change.

I hope that this will make it clearer that nServiceBus is a higher-level set of abstractions than WCF/WF – limiting the generality found in these frameworks to enable one cohesive way of working that will result in a solution that is both scalable and robust. On the other hand, nServiceBus is not a fully integrated product like Biztalk and intends to tackle different kinds of problems.

Bigger than a WCF/WF, smaller than a breadbox Biztalk.

I have a question around processing of messages in proper order.  When leveraging multiple threads to process messages in a message queue, it is possible for the second message in the queue to get processed before the first – especially if the first message is considerably larger than the second.  I have taken a lot of care to make sure that messages are sent in the correct order, only to find that the receiving system can process them out of order anyway.

Consider a Policy Created notification, which must come before a Policy Approved notification.  If both messages are sitting in the queue when the receiving service starts up, the approval message can be processed before the creation message. How can I make sure that message ordering is respected by the receiving system?  I am using WCF/MSMQ as the underlying transport by the way.  The only way I have found so far is to limit the receiving service to a single thread, which is by no means desirable.

Well, the solution is really quite simple (at first).

If you’ve received a message that you think has arrived out of order, just call:

this.bus.HandleCurrentMessageLater();

and that will put the message back at the end of the queue.

Once you start considering the fact that you don’t know when the first message is supposed to arrive, you might turn to using a workflow to handle the logic. The workflow would store the policy id, and then allow for N round-trips, before it decided that something bad had happened (like the Policy Created message getting lost), and then it could forward that to an operator, or possibly contact the first system and ask for a replay of the policy created message – or whatever automated fault resolution protocol you like.

In other words, message ordering is probably more trouble than its worth.

The Setup

Just so that the example is in itself secure, we’ll assume that the password is one-way hashed before being stored. Also, given a reasonable network infrastructure our web servers will be isolated in the DMZ and will have to access some application server which, in turn, will communicate with the DB. There’s also a good chance for something like round-robin load-balancing between web servers, especially for things like user login.

Before diving into the meat of it, I wanted to preface with a few words. One of the commonalities I’ve found when people dismiss asynchrony is that they don’t consider a real deployment environment, or scaling up a solution to multiple servers, farms, or datacenters.

The Synchronous Solution

In the synchronous solution, each one of our web servers will be contacting the app server for each user login request. In other words, the load on the app server and, consequently, on the database server will be proportional to the number of logins. One property of this load is its data locality, or rather, the lack of it. Given that user U logged in, the DB won’t necessarily gain any performance benefits by loading all username/password data into memory for the same page as user U. Another property is that this data is very non-volatile – it doesn’t change that often.

I won’t go to far into the synchronous solution since its been analysed numerous times before. The bottom line is that the database is the bottleneck. You could use sharding solutions. Many of the large sites have numerous read-only databases for this kind of data, with one master for updates – replicating out to the read-only replicas. That’s great if you’re using a nice cheap database like mySql (of LAMP), not so nice if you’re running Oracle or MS Sql Server.

Regardless of what you’re doing in your data tier, you’re there. Wouldn’t it be nice to close the loop in the web servers? Even if you are using Apache, that’s going to be less iron, electricity, and cooling all around. That’s what the asynchronous solution is all about – capitalizing on the low cost of memory to save on other things.

The Asynchronous Solution

In the asynchronous solution, we cache username/hashed-password pairs in memory on our web servers, and authenticate against that. Let’s analyse how much memory that takes:

Usernames are usually 12 characters or less, but let’s take an average of 32 to be sure. Using Unicode we get to 64 bytes for the username. Hashed passwords can run between 256 and 512 bits depending on the algorithm, divide by 8 and you have 64 bytes. That’s about 128 bytes altogether. So we can safely cache 8 million of these with 1GB of memory per web server. If you’ve got a million users, first of all, good for you Second, that’s just 128 MB of memory – relatively nothing even for a cheap 2GB web server.

Also, consider the fact that when registering a new user we can check if such a username is already taken at the web server level. That doesn’t mean it won’t be checked again in the DB to account for concurrency issues, but that the load on the DB is further reduced. Other things to notice include no read-only replicas and no replication. Simple. Our web servers are the "replicas".

The Authentication Service

What makes it all work is the "Authentication Service" on the app server. This was always there in the synchronous solution. It is what used to field all the login requests from the web servers, and, of course, allowed them to register new users and all the regular stuff. The difference is that now it publishes a message when a new user is registered (or rather, is validated – all a part of the internal long-running workflow). It also allows subscribers to receive the list of all username/hashed-password pairs. It’s also quite likely that it would keep the same data in memory too.

The same message can be used to publish both single updates, and returning the full list when using NServiceBus. Let’s define the message:

And the message that the web server sends when it wants the full list:

And the code that the web server runs on startup looks like this (assuming constructor injection):

And the code that runs in the Authentication Service when the GetAllUsernamesMessage is received:

And the class on the web server that handles a UsernameInUseMessage when it arrives:

When the app server sends the full list, multiple objects of the type UsernameInUseMessage are sent in one physical message to that web server. However, the bus object that runs on the web server dispatches each of these logical messages one at a time to the message handler above.

So, when it comes time to actually authenticate a user, this the web page (or controller, if you’re doing MVC) would call:

When registering a new user, the web server would of course first check its cache, and then send a RegisterUserMessage that contained the username and the hashed password.

When the RegisterUserMessage arrives at the app server, a new long-running workflow is kicked off to handle the process:

That UsernameInUseMessage would eventually arrive at all the web servers subscribed.

Performance/Security Trade-Offs

When looking deeper into this workflow we realize that it could be implemented as two separate message handlers, and have the email address take the place of the workflow Id. The problem with this alternate, better performing solution has to do with security. By removing the dependence on the workflow Id, we’ve in essence stated that we’re willing to receive a UserValidatedMessage without having previously received the RegisterUserMessage.

Since the processing of the UserValidatedMessage is relatively expensive – writing to the DB and publishing messages to all web servers, a malicious user could perform a denial of service (DOS) attack without that many messages, thus flying under the radar of many detection systems. Spoofing a guid that would result in a valid workflow instance is much more difficult. Also, since workflow instances would probably be stored in some in-memory, replicated data grid the relative cost of a lookup would be quite small – small enough to avoid a DOS until a detection system picked it up.

Improved Bandwidth & Latency

The bottom line is that you’re getting much more out of your web tier this way, rather than hammering your data tier and having to scale it out much sooner. Also, notice that there is much less network traffic this way. Not such a big deal for usernames and passwords, but other scenarios built in the same way may need more data. Of course, the time it takes us to log a user in is much shorter as well since we don’t have to cross back and forth from the web server (in the DMZ) to the app server, to the db server.

The important thing to remember in this solution is doing pub/sub. NServiceBus merely provides a simple API for designing the system around pub/sub. And publishing is where you get the serious scalability. As you get more users, you’ll obviously need to get more web servers. The thing is that you probably won’t need more database servers just to handle logins. In this case, you also get lower latency per request since all work needed to be done can be done locally on the server that received the request.

ETags make it even better

For the more advanced crowd, I’ll wrap it up with the ETags. Since web servers do go down, and the cache will be cleared, what we can do is to write that cache to disk (probably in a background thread), and "tag" it with something that the server gave us along with the last UsernameInUseMessage we received. That way, when the web server comes back up, it can send that ETag along with its GetAllUsernamesMessage so that the app server will only send the changes that occurred since. This drives down network usage even more at the insignificant cost of some disk space on the web servers.

And in closing…

Even if you don’t have anything more than a single physical server today, and it acts as your web server and database server, this solution won’t slow things down. If anything, it’ll speed it up. Regardless, you’re much better prepared to scale out than before – no need to rip and replace your entire architecture just as you get 8 million Facebook users banging down your front door.

So, go check out NServiceBus and get the most out of your iron.

Suresh asks:

Hi Udi,

I’ve been reading about the coming “durable services” that will be available with the next version of WCF. I also have been listening to your podcasts and reading your blog posts about NServiceBus where you talk about long-running workflows. It sounds like both of these technologies are trying to solve the same problem.

Do durable services do away with long-running workflow? If so, does that mean we don’t need Workflow Foundation either? If not, what is the connection between them.

If you could shed some light on the matter that would be great.

Suresh

Download

Download via the Dr. Dobb’s site

Or download directly here

Additional References

• Blog post on Workflow Services in Orcas
• Blog post on Orcas Durable Services
• Blog post on .NET Framework 3.5 Beta1 is Live on the Web

Want more?

Check out the “Ask Udi” archives.

Got a question?

Send Udi your question to answer on the show.

I’ve also put in a bit of effort in setting up a bunch of configuration files allowing you to try the various transports for the example code. Please go through the Readme as it contains the detailed instructions. There are currently config files setting up non-WCF MSMQ, NetTCP, Basic HTTP, and WS Dual.

If you do go the MSMQ route, the example will run the long-running workflow code. The reason the example code won’t do that for the WCF cases is that I haven’t figured out yet the best way for a service to send a message back to itself, especially for the connected bindings. Any thoughts on this would be most appreciated.

Also, I’m open to hearing what directions you think the next version should go in. I’m currently thinking about including message transformation. Either that are doing a Multicast bus over a UDP transport.

This is a non-backwards-compatible update. In order to propertly integrate with the WCF philosophy I had to change the ITransport interface for Unicast transports. The meaning of this change is that transports are now expected to do their own threading and transaction management, while the bus object now joins the context that comes from the transport.

Download here.

My question is about distributed topology.

The EAI-hub-and-spoke model is all about the central server. It’s useful sometimes, but there are a lot of reasons why I’m not gung-ho on using a hub as the center of the integration universe.

The ESB distributed model puts code on the endpoints. That code solves some of the messaging problems that apps face, so that apps don’t have to face them. It also solves some of the messaging problems that enterprises face, so that enterprises don’t have to face them. (I need both).

Those problems include simple coding and deployment model, pub-sub routing, reliable transport, simple transformation, and orchestration. I wonder which of these you can do in your tool, and which you are planning to do.

I’m also interested in management. How do you insure that the endpoints are correctly configured? Do you have a central configuration store? How do you propagate changes from the center to the messaging endpoints?

And here’s my response:

The important parts of NServiceBus that are independent of the distributed topology are the API and the connection to long-running workflow. This code is indeed on the endpoint. However, if you wanted to you could easily connect to something like BizTalk and do whatever you wanted there. This general idea though is to support the ESB distributed model since there’s no such things as a centralized ESB.

In terms of the capabilities you’ve mentioned, I’ve seen developers pick up the coding model in a day or two. The deployment model is just a bunch of DLLs you deploy with each endpoint. Dependency Injection is supported by www.SpringFramework.net but you can replace that with something else easily as another implementation of the ObjectBuilder interfaces.

Currently pub/sub routing is supported over regular point-to-point transports in a transport agnostic way. You also have the ability to have subscriptions be persisted so that even if a server restarts (and clients don’t, and can’t know about that) all the subscriptions will be remembered.

The reliable transport that is currently supported is MSMQ, with the option of defining per-message type if you want durable messaging (using the [Recoverable] attribute).

In terms of orchestration you get a nice model for long-running workflow that gets kicked off by messages decorated with the [StartsWorkflow] attribute, and messages that implement the IWorkflowMessage interface get automatically routed to the persistent workflow instance. You have the ability to change the storage of workflow instances easily as well. Workflows are simple classes which are easily unit-testable in that they expose a “void Handle(T message);” method for every message type (T) that is involved in the workflow.

I haven’t done anything in terms of simple transformation yet but am currently looking for the right place in the message processing pipeline to put it. I also haven’t done anything yet in terms of management.

What is currently being done management-wise on the projects that use it are the commercial options for managing configuration files in distributed environments coupled with the regular ability to restart windows services and IIS applications. I haven’t seen anything lacking in that solution yet.

If you have any questions, please don’t hesitate to send them my way – NServiceBus@UdiDahan.com.

First of all, there were two major forces that drove me to write NServiceBus. First of all I wanted to formalize the way Service Layer classes were written when using asynchronous messaging (regardless of Web Services). And second I wanted to formalize a communications API which supported pub/sub so that I could freely move between transports – whether they inherently supported pub/sub or not (the fact that Indigo did not falls under this).

From there it was filling in the details. Until finally I made the step to formalize the way long-running workflow connects to asynchronous messaging. That’s the third pillar as far as I’m concerned about ESBs. Using MSMQ I do enable Shared Subscriptions like Sonic does.

I try to stay away from the vendor side of ESB definitions and just deal with what capabilities such a library should provide. As such, it is inherently distributed – running on each server/client.

Let’s say I have a system to handle a call center for financial services or telco. When a customer calls on the phone and asks to be enrolled in “Heavily Advertised Program ABC,” there may need to be three or four systems that interact to make that real.

…

Harry asks me to consider using a ‘logical name’ of the receiver. The sender contacts a logical end point, the addressing infrastructure turns that into a physical end point, and we still have decoupling.

Honestly, I like it but I think it is insufficient. What if we need to contact 20 downstream systems in a complex workflow, but I don’t want a single “orchestration coordinator” to be a bottleneck (or single point of failure). I don’t want to hand the orchestration off from my app to a central orchestration hub.

Let me propose a different approach.

When we use SOA/EDA (the same thing as far as I’m concerned), the top-level building block used is the Service. A service may make use of a number of existing systems to perform its work. The business-level events that we publish (and subscribe to) are done by the service, not the existing systems.

If there’s any orchestration/workflow that needs to be done as a result of a service receiving an event, it is done entirely internal to that service. Inter-service orchestrations don’t really exist, as in there is no orchestration coordinator that is not in a service. And the orchestration coordinators within a service don’t touch other services’ back-end systems – if anything, they publish other business level events.

Be aware: when just starting out on an SOA, you’ll find that multiple Services make use of the same backend systems. This may be necessary, but not a desirable state to stay in for too long since it embodies the most insidious and invisible kind of inter-service coupling there is.

I want to go back to Nick’s original question:

So what if no one picks the message up? Is that an error?

The answer is mu.

If a service publishes a business-event (message) and no other services currently care, that’s fine. It’s not an error. Actually, you’d probably have some kind of infrastructure “queue” where messages that haven’t been received more than X time get sent to, so that the event isn’t “lost”. On the other hand, within a service – if an existing system sends out a message that needs to arrive at another system, and that message doesn’t arrive or isn’t picked up “in time”, that is an error.

This is one of the advantages SOA brings to the table in terms of EDA (again, the same as far as I’m concerned). You get simple messaging semantics between services, while within the “sphere of control” of a service you need, and more importantly can do more complex messaging and orchestration.

Bottom line: you need higher abstractions than your existing systems to employ EDA effectively.

You might also want to check out my podcast on this topic: SOA, ESB, and Events.

This update includes support for long-running workflows and includes persistent timer functionality. Workflow persistence is abstracted behind an IWorkflowPersister. The current drop includes an in-memory persister, but you could easily implement it over a database, or a distributed in-memory datagrid / space.

Other important details. To have a message kick off a workflow, just put the StartsWorkflow attribute on it. To define a message type as being involved in workflows, have it implement IWorkflowMessage instead of just IMessage. If you do these two things, the infrastructure will be able to generically dispatch messages to your workflow; you will not have to write any message handlers for your workflow, just the workflow class itself. This is done by implementing IWorkflow<T> for every message type received.

Persistent timers are activated from within your workflow like so:

this.reminder.ExpireIn(TimeSpan.FromSeconds(5), this, state);

As always, dependency injection is used – via my simple ObjectBuilder. The implementation provided delegates to Spring, but, once again, that can be easily substituted with whatever you want.

Download (509 KB) (sometimes problematic with IE, try FireFox before emailing me).

Finally, I’m leaning towards changing the name to NServiceBus. Anyone against, please leave a comment. Questions and comments are more than welcome.

Next step: implementing a WCF transport.

Before going too far ahead, you might want to take a look at my previous post “Space-Based Architectural Thinking, or listen to my podcast Space-Based Architecture for the Web. There’s also a 30 minute webcast online describing SBA more fully here. I’m also going to try to stay away from things concerning Jini this time after already discussing the connection between Jini and SOA, and the tradeoffs between two general approaches: Tasks and Spaces vs Message and Handlers.

OK, so the issue of state-management is a big one. Everybody wants to work stateless, because it scales. The only problem is that the business processes that we are automating are long running, meaning that there are external systems or people involved. This makes these processes inherently stateful. So, we need a way to scale statefully – SBA gives us that. For some background on the “Shared Nothing Architecture”, I suggest reading this post on inter-process SOA and this one as well.

Availability also has to be handled, not only in terms of having enough servers online to handle the required load but in having all the data required to process each request be accessible. This has often been handled by the database using ACID transactions – durability being that which solved availability issues, but also hurting latency the most. The problem with saving the state of our long-running business processes/workflows in the database is the load and the responsiveness requirements. In many verticals – telcos, financial, and defense to name a few, we need millisecond level latency on each stage of the workflow. This is what leads SBA to the in-memory, replicated data grid.

Note that SBA only intends to take these workflows out of the database, and not anything else – especially not Master Data. The lifetime of these workflows is incredibly short compared to that of master data like customers and products. It will have much different backup strategies as well. In terms of load, these workflows will be heavy on reads and writes together in the same transactions, but quite low in terms of just reads. If we have workflows that perform work in parallel, we easily end up with concurrency requirements that make DBAs cringe under the barrage of short transactions.

If you’re worried that Workflow Foundation (WF) won’t scale because of the above, you needn’t be. You can (more or less easily) replace the persistence mechanism of WF with your own, saving your workflow instances to an in-memory replicated data grid.

By enabling the objects in the grid to call back into logic on your servers, you have, in essence, done messaging and more. The added benefit that SBA receives from this is a unification of technology between caching and messaging. This translates directly to savings when it comes time to cluster each of those technology’s environments.

Finally, if we can find an attribute in the incoming stream of messages that creates a nice even distribution, we can then partition our load between our servers by that key. This will work up to the point where the load per key increases beyond a single server’s capacity, and then we have to look at re-partitioning, a non-trivial problem. However, if we put objects in our grid that represent the master data, and tie them to our workflow instances with both of those tied to the key of our load, a smart infrastructure can make sure all that data is already resident on the server that is handling that piece of the load. That decreases latency even more since we no longer have to pay network roundtrips to collect all the data needed before we can process it. That’s a substantial advantage for the above verticals.

But all of this has nothing to do with SOA.

Sure, it’ll change how we implement our Services internally, but it has no impact on their interfaces or the top-level service decomposition. In the Java community, the word “service” is often used to describe the logic of a system. Great significance is placed on keeping these “services” simple, as in Plain-Old Java Objects. The fact of the matter is that the logic of the system should be simple and independent of other concerns like data access and communcations (a la Web Services), but that does not make it a service, not in the SOA sense.

For more information on what Services in SOA are like, check out this podcast on Business and Autonomous Components in SOA. Actually, SBA will probably have the biggest impact on the way autonomous components will handle service-level agreements.

So, it appears that even with SOA, SBA has its place. The former dealing with business level agility, the latter dealing with all the technical aspects of supporting that agility. If you’re tasked with the designing the architecture of a scalable, available, high-throughput, low-latency distributed system, I’d strongly advise you to look at SBA – the technical value is overwhelming. Even if you don’t utilize all elements of SBA and choose the Master Worker Pattern instead of load partitioning, you’ll find the technologies supporting SBA to be quite flexible in that respect.

Will Space-Based Architectures be a part of your future? I don’t know for sure, but they’re a most welcome part of my present.

And you’re absolutely right.

The fact of the matter is, you don’t have to do anything special to deal with failures in workflow. It all can be handled in terms of messaging, since the only interesting kind of workflow is the “long-running” kind. Long running workflows are those that may involve calling out to other services/systems, or involve humans in the process. Conversely, “short-running” workflows involve receiving a single message, handling it by calling code on the same server, in the same process, and on the same thread, and possibly returning a response. Simple request/response interaction is short-running. Things you might think to use BizTalk for would probably be long-running.

Take a look at this post which shows that long-running workflow is just regular messaging with a sprinkle of state-management.

Anyway, in my article describing failure handling for messaging, these workflow failures fall under the following category and solution:

Other kinds of failure conditions include having our DB transactions fail because they were chosen as the victim of a deadlock. The appropriate system level error handling is to just retry the transaction a bit later. Once again, if we were using our queues within the context of a transaction scope that included that of the database, the message would return to the queue when that deadlock exception bubbled through. This is exactly the behavior that we want in this case, although having the message go to the end of the queue instead of the front would also be just fine.

Other kinds of failure conditions we might run into include things like unique-constraint violations, attempting to perform actions on entities that no longer exist, and other activities where trying them over again probably won’t yield any different behavior. In these cases, if an exception were to cause the perpetrating message to return to the input queue we’d get exactly the wrong behavior. At the very least we would want to maybe log the exception information

So, in some cases, it makes perfect sense for the workflow to let the exception bubble through. Not so that the workflow will be terminated, but rather that the message which triggered it would be tried again. You can configure Workflow Foundation to have this behavior as well, or so I’ve been told.

Now you’re happy, the problem’s been solved for you – by design. No need to go mucking around in the nooks and crannys of FaultHandlers, CallExternalMethodActivity, and hacking the InsertWorkflow stored procedure.

Clean, Correct, and Simple – courtesy of The Software Simplist.

Additional References:

• Ron Jacobs interviews me on SOA and Workflow
• A conversation on Workflow Foundation and Service Orientation

I would suggest that a service, when composable, (a) provides information in a manner that can be readily reused without requiring multiple calls to other services for interpretation, and/or (b) provides functionality that can be executed to produce specific semantics in an encapsulated manner without requiring references to multiple other services and without unknown or undesirable side effects.

I woud state that a service that does not comply with the above “suggestions” is not a service in the SOA sense. I don’t care much for the “composable” or “reusable” qualifiers that are being placed on the term “service”. I believe that it creates fractured definitions that don’t improve understanding.

Nick continues with assertions about the connection between SOA and Canonical Data Models:

I argue, passionately, that a service that does not leverage a Canonical data model (explicit or implicit) is not useful outside a small handful of very specific situations. Such a limited service provides some small benefits, but probably not more than a COM+ component would, and certainly not enough to justify all this interest in SOA. We get no business agility from this kind of service. Therefore, as an Enterprise Architect, you can create it, but I will not use it, nor will I look kindly on another application that does. Poor integration is just barely a half step up from no integration. Some would argue it is a step down.

Just to be clear, a Canonical Data Model is something that exists without dependence on any service. That is why I don’t agree with its use in SOA. Each service has its own message schema which makes use of its own data schema if you will. A data schema is the definition of structures that are used in more than one message type. The service controlls both of these schemas and decides when and how to version them, which versions to support on which endpoints, etc. It’s all part of the service’s autonomy. I’m not quite sure what an implicit canonical data model is.

I’m also unclear as to the “benefit” a service provides. Services are the way we model our business domain – they are therefore a part of our business. In my previous post on intermediation and SOA I gave an example of a Purchasing Service. This service does not really exist to provide something external to it with a “benefit”, it’s an inherent part of the structure of the business, much like the Purchasing Department of the company is. It participates in business processes but is not ruled by them. I really don’t see how you could compare that to any kind of technological component. In that respect, you don’t “use” a service. Neither do you really “integrate” them. Each service decides which processes it needs to be a part of, simply by choosing which events (of the EDA kind) it subscribes to. This leads to a kind of “distributed integration” that maintains service autonomy and has no single point of versioning or failure.

One of the comments quoted includes the statement “Intermediation greatly complicates any message exchange pattern other than request-response and pub-sub”. Like I described in my previous post, intermediation, if it even does occur, occurs behind service boundaries. Between services, you have the basic message exchange patterns of duplex request/response (one way messaging each time, with correlation between them) and pub/sub. You really don’t need anything else, and yesterday’s middleware already handled all of that for us. Most of the advanced ESB functionality isn’t needed between services.

The next example given discusses a banking scenario (well, more of an HR one really), which brings Nick to the following conclusion:

This process is neither pub-sub nor request-response. It is a long-running orchestration with compensating events. The process is NOT complicated by the ability to intermediate.

In fact, I would argue that nearly all valuable long running transactions MUST have the ability to intermediate in order to allow them to be composed, and recomposed, and orchestrated.

Long running anything, is a series of message exchanges that are tied together in some way. This could be as simple as having all the messages contain the same process ID in their body. “Compensating events” are just the result of business logic being run in a service, possibly changing its own state (updating those backend systems and applications), and sending out other messages. If there’s any intermediation of the kind Nick describes, it occurs within a service between its backend systems and applications. He also implicitly states that there is value in composing/orchestrating these “long running transactions”, apparently without having the service be involved. I don’t see it. Not the way I do my services. Once again, each service is responsible for itself – its part of the global “long running transaction”, ie business process. If one service were to choose to implement its cross-application workflows in a hard-coded manner, that should have no impact on any other services, nor should they be aware of it. That service may have been implemented poorly and as a result have difficulty responding to changing business conditions quickly, but that’s its own business. That implementation could easily be upgraded in the future, without changing the overall Service-Oriented Architecture.

Nick’s conclusion follows:

In conclusion, I don’t say that intermediation is a requirement of a service oriented architecture. But I do say that intermediation is a requirement of a service oriented architecture designed to deliver composability, and therefore, business agility.

Mine is a little different. When your SOA is made up of autonomous business-level services, you will have business agility. The implementations of some services may benefit through the use of intermediation, but it is not a top level concern.

I know that WF was designed to be general purpose, but I don’t like general purpose tools, not for the projects I work on. I don’t like that it has its own threading model and that I have to override it. I don’t like that it has its own activation model and that I have to override it too. These are architectural concerns that need to be in sync with the entire project – I can’t have every part of the system doing its own thing.

Although not surprised, I was a bit saddened when I was sitting through the Iron Architect contest and each participant described how they would implement the business logic of the system with WF. Personally, I think that that’s a bad choice. Object-oriented domain models do an excellent job of modelling complex business logic. The procedure-oriented nature of workflows predisposes them to the historic failures that method has brought in the past.

One area where I can’t think why WF wouldn’t do well in is for web-based page navigation. I’m sure there are others, but for the systems I work on there are usually better alternatives in those areas. When it comes to products, on the other hand, WF might be a smash. I’m pretty sure that the Sharepoint guys are betting the farm on WF for the human workflow there. Again, since I’m a project guy and not a product guy I don’t think I can do WF justice there.

One area I’m really interested in is long-running workflows and their use in SOA. WF has a very important, and non-trivial thing called Persistent Timers in this area. This is incredibly important if you need your workflows to continue to be correct even in the event of a server restart, I know I do. WF supports a number of scenarios in terms of communication that currently don’t interest me but I am looking forward to seeing how their going to support WCF’s duplex communication patterns.

All in all, WF is an intersting bit of technology that I will continue watching as it evolves.

[Salt and Pepper, Peanut Butter and Jelly, Toast and Vegemite (if your down under) - these are all things that go together. But what goes with SOA you ask? Why Workflow of course! On this episode we are joined by Microsoft Archtiect MVP Udi Dahan who gives us his insight into putting these two technologies together.]

I just can’t seem to shut up about these things. It’s gotten so bad that I’ll be inflicting the same session I gave in TechEd Developers Europe on our local Israeli crowd at the end of this month. You can find out what other horrors Microsoft has in store here.

From his post:

“The 2.0 framework doesn’t really give you a massive amount of really new really cool features. There is one-click, which we probably wouldn’t use anyway, generics (which are useful as a time saver, but don’t really do much that you can’t do without them), and some other little features… but the best part about the 2.0 framework is probably the IDE/dev experience, not what customers get out of it.”

Granted, One-Click is nothing to write home about, but the runtime level additions, they just totally changed the way I write code. I’m talking about generics, anonymous methods, delegate inference, and the rest. The rest of 2.0, like the enhancements of the provider model in ASP.Net, well you would have developed the same kind of framework yourself if you were doing serious web development.

The whole 3.0 story, I’ve got to tell you, I’m pretty underwhelmed. Everybody seems to be jumping up and down about WPF, and yes, it’s new and shiny, but there still the clunkety Windows message pump in the background. No real changes in how you’re going to write multi-threaded UIs, which seem to be the real future given the rise in multi-core processing. The visual aspects of client side code in the systems I write run at around 5% of the overall effort. So the UI will look better, I dunno, 4D buttons and stuff, sorry for not falling over with enthusiasm.

And then there’s WCF. Ah, wait, no publish/subscribe. Bummer, most of my systems being asynchronous in nature are built on the pub/sub model. An OO interface for interprocess communication? Who wants it – I need a message-based interface.

Don’t forget WF – what was that for again? The main place where WF can fit my needs is for handling long-running workflows between systems, since I don’t use Biztalk. But the performance of WF doesn’t seem to fit this environment, it seems to be more suited for human workflow times.

If anything, I’d have to say that .Net 2.0 was a relatively big deal. 3.0 will probably be just as important with the runtime level enhancements like lambda expressions, extension methods, anonymous types, and implicitly typed variables. All the rest of the hyped up stuff in 3.0, I don’t really expect it to change anything in how I work today.