Earlier this month at TechEd North America I gave a fairly new presentation that was only delivered once before (at the Connected Systems User Group in London) and I’m happy to say is now online for your viewing pleasure.

High Availability – A Contrarian View

Comments? Thoughts? Let me know.

Durable Messaging

In order to prevent a message from being lost when it is sent from a publisher to a subscriber, the message is written to disk on the publisher side, and then forwarded to the subscriber, where it is also written to disk. This store-and-forward mechanism enables our systems to gracefully recover from either side being temporarily unavailable.

In my MSDN article on this topic, I outlined some problems with this approach. These problems are exacerbated for publishers. Imagine a publisher with 40 subscribers, publishing 10 messages a second, each containing 1MB of XML. If 10 of the subscribers are unavailable, that’s 100MB of data being written to the publisher’s disk every second, 6GB every minute. That’s liable to bring down a publisher before an administrator brews a cup of coffee.

Publishers have no choice but to throw away messages after a certain period of time.

Publisher Contracts

The whole issue of contracts and schema is considered one of the better understand parts of SOA. Unfortunately, the operational aspects of service contracts is hardly ever taken into account.

On top of the schema of the messages a service publishers, additional information is needed in the contract:

1. How big will this message be?
2. How often will it be published?
3. How long will this message be stored if a subscriber is unavailable?

This first two pieces of information are important for subscribers to do load and capacity planning. The last one is the most important as it dictates the required availability and fault-tolerance characteristic of subscribers.

For Example

In the canonical retail scenario, when our sales service accepts an order, it publishes an order accepted event. Other services subscribed to this event include shipping, billing, and business intelligence.

While shipping and billing are highly available and able to keep up with the rate at which orders are accepted, the business intelligence service is not. BI has two main parts to it – a nightly batch that does the number crunching, and a UI for reporting off of the results of that number crunching. Some even do the reporting in a semi-offline fashion, emailing reports back to the user when they’re ready.

Furthermore, nobody’s going to invest in servers for making BI highly available.

And wasn’t the whole point of this publish/subscribe messaging to keep our services autonomous? That not all services have to have the same level uptime?

Houston, do we have a problem.?

Data Freshness

There is a glimmer of light in all this doom and gloom.

Not all services have the same data freshness requirements.

The business intelligence service above doesn’t need to know about orders the second they’re accepted. A daily roll-up would be fine, and an hourly roll-up bring us that much closer to “real time business intelligence”.

So, while BI is ready to accept the sales message schema, it would like a slightly different contract around it – less messages per unit of time, more data in each message.

From the operational perspective of the sales service, it would be cost effective to have less “online” subscribers. It could even take things a few steps further. Instead of using the regular messaging backbone for transmitting these hourly messages, it could use FTP. The data could even be zipped to take up even less space. Since the total data size is less than the corresponding online stream, is stored on cheaper, large storage, and the number of subscribers for this zipped, hourly update is fairly small, these messages can be kept around far longer.

If you’ve heard about consumer-driven contracts, this is it.

Note that we’re still talking about the same logical message schema.

Summary

It’s not that lost notifications aren’t a problem.

It’s that they feed the design process in such a way that the resulting service ecosystem is set up in such a way that notifications won’t get lost. I know that that sounds kind of recursive, but that’s how it works. Either subscribers take care of their SLA allowing them to process the online stream of events, or they should subscribe to a different pipe (which will have different SLA requirements, but maybe they can deal with those).

It make sense to have multiple pipes for the same logical schema.

It’s practically a necessity to make pub/sub a feasible solution.

Related Content

MSDN article on messaging and lost messages

Durable messaging dilemmas

Additional logic required for service autonomy

More in depth example on events and pub/sub between services

Consumer-Driven Contracts

You can find it here.
The slides can be found here.

I also gave this talk at TechEd Barcelona and wanted to thank the attendee who posted this comment:

“You’ve done it again. Everytime I attend a session of yours I leave the room with new insights and inspiration on how to improve my software…”

You made my day.

Libor asks:

“Would you recommend using durable messaging for systems where there are similar requirements with respect to data reliability as you had – ie. not losing any messages? If so, then why didn’t the final version of your solution use it? If not, can you explain why?”

The answer is, as always, it depends, but here’s on what it depends:

When designing a system, we need to take a good, hard look at how we manage state, and what properties that state has. In a system of reasonable size we can expect various families of state with respect to their business value, data volatility, and fault-tolerance window. Each family needs to be treated differently. While durable messaging may be suitable for one, it may be overkill or underkill for another.

So, here’s what we’re going to be looking at:

1. Business Value
2. Data Volatility
3. Fault-Tolerance Window

Business Value

When talking about business value, I want to talk about what it means “not losing any messages”. The question is under what conditions will the messages not be lost, or rather, what are the threshold conditions where messages may start getting lost. If all our datacenters are nuked, we will lose data. It’s likely the business is OK with that (as much as can be expected under those circumstances). If a single server goes down, it’s likely the business would not be OK with losing messages containing financial data. However if a message requesting the health of a server were to get lost under those same conditions, that would probably be alright. In other words, what does that message represent in business terms.

Data Volatility

Data volatility also has an impact. Let’s say that we’re building a financial trading system. The time that it takes us to respond to an event (message) that the cost of a certain financial instrument has changed, and the message that we send requesting to buy that security is critical. Let’s say that has to be done in under 10ms. Now, some failure has occurred preventing our message from reaching its destination for 20ms. What should we do with that message? Should we keep it around, making sure it doesn’t get lost? Not in this domain. On the contrary, that message should be thrown away as its “business lifetime” has been exceeded. Furthermore, even during that original period of 10ms, the use of durable messaging may make it close to impossible to maintain our response times.

Fault-Tolerance Window

These two topics feed into the third and more architectural one – fault-tolerance window: what period of time do we require fault tolerance, and with respect to how many (and what kind of) faults? This will lead us into an analysis of to how many machines do we need to copy a message before we release the calling thread. We’d also look at in which datacenters those machines reside. This will also impact (or be impacted by) the kinds of links we have to these datacenters if we want to maintain response times. These numbers will need to change when the system identifies a disaster – degrading itself to a lower level of fault-tolerance after a hurricane knocks out a datacenter, and returning to normal once it comes back up.

Re-Evaluating Durable Messaging

Durable messaging may be used at various points in each part of the solution, but we need to look at message size, the rate those messages are being written to disk, how fast the disk is, how much available disk we have (so we don’t make things worse in the case of degraded service), etc. Companies like Amazon also take into account disk failure rates, replacement rates (disks aren’t replaced immediately you know), and many other factors when making these decisions

Summary

Our job as architects when designing the system is to find that cost-benefit balance for the various parts of the system according to these very applicative parameters. No, it’s not easy. No, cloud computing will not magically solve all of this for us. But, we are getting more technical tools to work with, operations staff is getting better at working with us in the design phase, and our thought processes more rigorous in dealing with the scary conditions of the real world.

To your question, Libor, as to why we didn’t eventually use durable messaging in our solution, the answer is that we solved the overall state management problem by setting up an applicative protocol with our partners which was resilient in the face of faults by using idempotent messages that could be resent as many times as necessary. You can read more about it here. This solution isn’t viable for other kinds of interactions but was just what we needed to get the job done.

Hope that helps.

Reliability

Locking in SqlWorkflowPersistenceService (via Ron Jacobs) where, if you want predictable persistence (MS: ‘none of our customers asked for this to be easy’), you need to use a custom activity (which Ron was kind enough to supply).

“Given what I learned today I’d have to say that I’d be very careful about using workflows with an optimistic locking.  Detecting these types of situations is not that simple.”

Let’s think about that. If we’re doing pessimistic locking, we get into the problem of, if a host restarts (as the result of a critical windows patch or some other unexpected occurrence), that the workflow won’t be able to be handled by any other host in the meantime (you didn’t care so much about your SLA, did you?).

Luckily, someone’s come up with a hack that works around this robustness problem in Scalable Workflow Persistence and Ownership.

“So this code will attempt to load workflow instances with expired locks every second. Is it a hack? Yes. But without one of two things in the SqlWorkflowPersistenceService its the sort of code you have to write to pick up unlocked workflow instances robustly.”

This will seriously churn the table used to store your workflows, decreasing performance of workflows that haven’t timed out. Oh well.

Testability

Implementing WCF Services without Referencing WCF (via Mark Seemann):

“More than a year ago, I wrote my first post on unit testing WCF services. One of my points back then was that you have to be careful that the service implementation doesn’t use any of the services provided by the WCF runtime environment (if you want to keep the service testable). As soon as you invoke something like OperationContext.Current, your code is not going to work in a unit testing scenario, but only when hosted by WCF.”

After pointing out some of the more basic difficulties in testability a straightforward WCF implementation brings, Mark turns the heat up in his follow-up post, Modifying Behavior of WCF-Free Service Implementations:

“Perhaps you need to control the service’s ConcurrencyMode, or perhaps you need to set UseSynchronizationContext. These options are typically controlled by the ServiceBehaviorAttribute. You may also want to provide an IInstanceProvider via a custom attribute that implements IContractBehavior. However, you can’t set these attributes on the service implementation itself, since it mustn’t have a reference to System.ServiceModel.”

Wow – all the things required to make a WCF service scalable and thread-safe make it difficult to test. In the end, we’re beginning to see how many hoops we have to go through in order to get separation of concerns, but until we can take all this and get it out of our application code, it’s an untenable solution. I hope Mark will continue with this series, if only so I can take the framework that might grow out of it and use it as a generic WCF transport for NServiceBus.

Comparison

After the Neuron-NServiceBus comparison that Sam and I had, we talked some more. After going through some of the rational and thinking, Sam even put nServiceBus into his WCF-Neuron comparison talk. Sam had this to say about nServiceBus:

“The bottom line is: I like what I see. Although it’s a framework, not an ESB product like Neuron, it’s a powerful framework that takes the right approach on SOA and enforces a paradigm of reliable one-way, *non-blocking* calls. That is the point of the talk tonight overall; we need to get away from the stack world of synchronous RPC calls to true asynchronous non-blocking message based SOA systems.”

The main concern I have with a WCF+WF based solution is that developers need to know a lot in order to make it testable, scalable, and robust. In nServiceBus, that’s baked into the design. It would be extremely difficult for a developer writing application logic to interfere with when persistence needs to happen, or the concurrency strategy of long-running workflows. The fact that message handlers in the service layer don’t need concurrency modes, instance providers, or any of that junk make them testable by default.

Hi Udi, I think  I see the point about the domain-driven approach but I’m wondering what my messages will look like. If it’s this:

IAppointment InsertInterview(Guid recruiterId, Guid applicantId, Guid appointmentId); OR

IRecuiter UpdateRecuiter(IRecuiter recruiter); (passing in an operation flag attached to the IRecuiter object) OR

IRecuiter UpdateRecuiter(IRecuiter recruiter); (setting a state flag on the relevant object and have the business object check the flag and behave according the state change)

Hope I’m not way off

Sean

Well, Sean, first of all – messages don’t look like functions. They’re a lot more like structures – data transfer objects. In this case, you’d probably be looking at a ScheduleInterviewMessage that had the relevant fields. It would look something like this:

   1:  using System;

   2:  using NServiceBus;

   3:  using System.Xml.Serialization;

   4:   

   5:  namespace Messages

   6:  {

   7:      [Serializable]

   8:      [Recoverable]

   9:      [TimeToBeReceived("0:01:00.000")]

  10:      public class ScheduleInterviewMessage : IMessage

  11:      {

  12:          public Guid InterviewerId;

  13:          public Guid CandidateId;

  14:          public DateTime RequestedTime;

  15:   

  16:          [XmlAnyElement]

  17:          public object extra;

  18:      }

  19:  }

Before we go on, I want to explain what we see. The “recoverable” attribute is the way we indicate to the infrastructure that these messages should not be lost in case a server fails, there are network problems, etc. In essence, it does durable, store-and-forward messaging. This will create an environment in which, in the case of network problems, these messages will be written to disk. That’s a good thing, since once connectivity comes back or the server boots back up, the messages will still be around and can be sent.

Now these messages are fairly small, so even at a relatively high load, we shouldn’t be chewing through too much of our expensive, small, high performance local disks. However, if these messages were bigger, we may fill up our disks before connectivity comes back, and we all know what happens to Windows boxes when there’s no room on the file system left:

In order to prevent our system from Denial-of-Servicing itself we need to make those messages clean themselves up. That’s what the “TimeToBeReceived” attribute is for. The amount of time that if a message had not yet been received by the other side that it will be deleted. This could be that the message even made it to the other machine, but the process handling those messages was down. You wouldn’t want to be filling the other side’s disk either causing them to crash, would you? This protects both parties.

The way to figure out how long to set is by looking at the smallest amount of durable storage you have available at your nodes, divide that by the size of the average message, and then again by the rate you need to process messages – and leave yourself at least 100% spare.

In other words, to build a robust system you not only will need to deal with lost messages, but you will be actively throwing messages away.

Finally, that last “XmlAnyElement” attribute is there for versioning. As we version our system and schema, we’ll be adding fields to the message. However, an old client may be talking to a new server, or vice versa. Since we wouldn’t want data to get lost just because of serialization. In a future post, I’ll show how to set up a message handler pipeline exactly for these issues.

Now that we’ve covered all the intricacies around messaging, we can see how the code that handles that above message looks:

   1:  using System;

   2:  using Messages;

   3:  using NServiceBus;

   4:  using NHibernate;

   5:   

   6:  namespace Server

   7:  {

   8:      public class ScheduleInterviewMessageHandler :

   9:                   BaseMessageHandler<ScheduleInterviewMessage>

  10:      {

  11:          public override void Handle(ScheduleInterviewMessage message)

  12:          {

  13:              using (ISession session = SessionFactory.OpenSession())

  14:              using (ITransaction tx = session.BeginTransaction())

  15:              {

  16:                  ICandidateInterviewer interviewer = session.Get<ICandidateInterviewer>(

  17:                          message.InterviewerId);

  18:                  ICandidate candidate = session.Get<ICandidate>(

  19:                          message.CandidateId);

  20:   

  21:                  interviewer.ScheduleInterviewWith(candidate)

  22:                          .At(message.RequestedTime);

  23:   

  24:                  tx.Commit();

  25:              }

  26:   

  27:              // publish new appointment data

  28:          }

  29:      }

  30:  }

If you’ve read this far and have more questions, please feel free to send them my way. If you’re at a more time-critical part of your project and need an answer quickly, we can set up a skype call. This has been working quite well for many of my overseas clients (shout out to the guys in Ireland and Florida).

Until next time