Some interesting numbers were stated in the talk.

• Tens of thousands to hundreds of thousands of requests per second
• Over 3 thousand web servers
• Over a thousand mid-tier servers

No wonder most big web sites don’t run .NET. The Windows licenses would put them out of business.

Well, that is if you follow those same architectural practices.

I’ve written in the past of alternative architectural approaches that can scale to those levels at easily an order of magnitude less hardware (I think it’s closer to two OOMs) – here’s one of them on the topic of weather:

Building Super-Scalable Web Systems with REST.

By the way, the client quoted in that post is now well above 60 million users with only small incremental increases in hardware. Oh, and their running everything on Windows and .NET. The question is not “can it scale”, but rather “how much will it cost to scale”.

Architecture pays itself back faster than ever in the Web 2.0 world.

My article on “optimizing a large-scale Software+Services application” has been published in the April edition of MSDN Magazine.

Here’s a short excerpt:

“We had to juggle occasional connectivity, data synchronization, and publish/subscribe all at the same time. We learned that we couldn’t solve all problems either client-side or server-side, but rather that an integrated approach was needed since any changes on one side needed corresponding changes on the other side.”

Continue reading…

On Web 2.0 Mashups

One would think that sites like Weather.com and friends would be the first choice for implementing such a feature. Only thing is that they were strongly against being mashed-up Web 2.0 style on the client – they had enough scalability problems of their own. Interestingly enough (or not), these partners were quite happy to publish their weather data to us and let us handle the whole scalability issue.

Implementation 1.0

The current implementation was fairly straightforward – client issues a regular web service request to the GetWeather webmethod, the server uses the user’s IP address to find out their location, then use that location to find the weather for that location in the database, and return that to the user. Standard fare for most dynamic data and the way most everybody would tell you to do it.

Only thing is that it scales like a dog.

Add Some Caching

The first thing you do when you have scalability problems and the database is the bottleneck is to cache, well, that’s what everybody says (same everybody as above).

The thing is that holding all the weather of the entire globe in memory, well, takes a lot of memory. More than is reasonable. In which case, there’s a fairly decent chance that a given request can’t be served from the cache, resulting in a query to the database, an update to the cache, which bumps out something else, in short, not a very good hit rate.

Not much bang for the buck.

If you have a single datacenter, having a caching tier that stores this data is possible, but costly. If you want a highly available, business continuity supportable, multi-datacenter infrastructure, the costs add up quite a bit quicker – to the point of not being cost effective (”You need HOW much money for weather?! We’ve got dozens more features like that in the pipe!”)

What we can do is to tell the client we’re responding to that they can cache the result, but that isn’t close to being enough for us to scale.

Look at the Data, Leverage the Internet

When you find yourself in this sort of situation, there’s really only one thing to do:

Leverage the Internet

The internet is the biggest, most scalable data serving infrastructure that mankind was lucky enough to have happen to it. However, in order to leverage it – you need to understand your data and how your users use it, and finally align yourself with the way the internet works.

Let’s say we have 1,000 users in London. All of them are going to have the same weather. If all these users come to our site in the period of a few hours and ask for the weather, they all are going to get the exact same data. The thing is that the response semantics of the GetWeather webmethod must prevent intermediaries from caching so that users in Dublin and Glasgow don’t get London weather (although at times I bet they’d like to).

REST Helps You Leverage the Internet

Rather than thinking of getting the weather as an operation/webmethod, we can represent the various locations weather data as explicit web resources, each with its own URI. Thus, the weather in London would be http://weather.myclient.com/UK/London.

If we were able to make our clients in London perform an HTTP GET on http://weather.myclient.com/UK/London then we could return headers in the HTTP response telling the intermediaries that they can cache the response for an hour, or however long we want.

That way, after the first user in London gets the weather from our servers, all the other 999 users will be getting the same data served to them from one of the intermediaries. Instead of getting hammered by millions of requests a day, the internet would shoulder easily 90% of that load making it much easier to scale. Thanks Al.

This isn’t a “cheap trick”. While being straight forward for something like weather, understanding the nature of your data and intelligently mapping that to a URI space is critical to building a scalable system, and reaping the benefits of REST.

What’s left?

The only thing that’s left is to get the client to know which URI to call. A simple matter, really.

When the user logs in, we perform the IP to location lookup and then write a cookie to the client with their location (UK/London). That cookie then stays with the user saving us from having to perform that IP to location lookup all the time. On subsequent logins, if the cookie is already there, we don’t do the lookup.

BTW, we also show the user “you’re in London, aren’t you?” with the link allowing the user to change their location, which we then update the cookie with and change the URI we get the weather from.

In Closing

While web services are great for getting a system up and running quickly and interoperably, scalability often suffers. Not so much as to be in your face, but after you’ve gone quite a ways and invested a fair amount of development in it, you find it standing between you and the scalability you seek.

Moving to REST is not about turning on the “make it restful” switch in your technology stack (ASP.NET MVC and WCF, I’m talking to you). Just like with databases there is no “make it go fast” switch – you really do need to understand your data, the various users access patterns, and the volatility of the data so that you can map it to the “right” resources and URIs.

If you do walk the RESTful path, you’ll find that the scalability that was once so distant is now within your grasp.

First of all, I wanted to address the relationship between various topics related to scalability:

And on the connection between scalability and throughput:

The important message here is that the scalability of a system is a cost function that gives throughput as a function of recurring costs and one time costs – servers and other hardware, and the join of buy & build:

Did you write your own locking/transaction mechanism on top of an open source distributed cache or did you buy a license for a space-based technology?

Also, don’t forget that people need to administer all the servers that you have. Those people cost money (easily100K per year). Maybe, because you haven’t invested in management or monitoring tools you need one person for every two servers. This will influence the breakdown of up front costs and recurring costs. Also, the level of availability you require will impact this as well.

In my experience, architects don’t consider often enough the operations environment in their "scalability calculations".

What this means is that there’s no such thing as technically "not being able to scale".

Rather, that the cost (up front + recurring) of supporting higher throughput grows faster than the function of revenue per user/request/whatever.

Sometimes, the solution is just to find ways to make more money per customer.

For more technical solutions, take a look at the difference between capacity and scalability and how the competing consumer pattern helps scale out.

Scalability, it’s all about the money.

–

Oh, I almost forgot, I also had a great conversation with Carl and Richard about scaling web sites that’s now up on the .NET Rocks site. Enjoy.

Libor asks:

“Would you recommend using durable messaging for systems where there are similar requirements with respect to data reliability as you had – ie. not losing any messages? If so, then why didn’t the final version of your solution use it? If not, can you explain why?”

The answer is, as always, it depends, but here’s on what it depends:

When designing a system, we need to take a good, hard look at how we manage state, and what properties that state has. In a system of reasonable size we can expect various families of state with respect to their business value, data volatility, and fault-tolerance window. Each family needs to be treated differently. While durable messaging may be suitable for one, it may be overkill or underkill for another.

So, here’s what we’re going to be looking at:

1. Business Value
2. Data Volatility
3. Fault-Tolerance Window

Business Value

When talking about business value, I want to talk about what it means “not losing any messages”. The question is under what conditions will the messages not be lost, or rather, what are the threshold conditions where messages may start getting lost. If all our datacenters are nuked, we will lose data. It’s likely the business is OK with that (as much as can be expected under those circumstances). If a single server goes down, it’s likely the business would not be OK with losing messages containing financial data. However if a message requesting the health of a server were to get lost under those same conditions, that would probably be alright. In other words, what does that message represent in business terms.

Data Volatility

Data volatility also has an impact. Let’s say that we’re building a financial trading system. The time that it takes us to respond to an event (message) that the cost of a certain financial instrument has changed, and the message that we send requesting to buy that security is critical. Let’s say that has to be done in under 10ms. Now, some failure has occurred preventing our message from reaching its destination for 20ms. What should we do with that message? Should we keep it around, making sure it doesn’t get lost? Not in this domain. On the contrary, that message should be thrown away as its “business lifetime” has been exceeded. Furthermore, even during that original period of 10ms, the use of durable messaging may make it close to impossible to maintain our response times.

Fault-Tolerance Window

These two topics feed into the third and more architectural one – fault-tolerance window: what period of time do we require fault tolerance, and with respect to how many (and what kind of) faults? This will lead us into an analysis of to how many machines do we need to copy a message before we release the calling thread. We’d also look at in which datacenters those machines reside. This will also impact (or be impacted by) the kinds of links we have to these datacenters if we want to maintain response times. These numbers will need to change when the system identifies a disaster – degrading itself to a lower level of fault-tolerance after a hurricane knocks out a datacenter, and returning to normal once it comes back up.

Re-Evaluating Durable Messaging

Durable messaging may be used at various points in each part of the solution, but we need to look at message size, the rate those messages are being written to disk, how fast the disk is, how much available disk we have (so we don’t make things worse in the case of degraded service), etc. Companies like Amazon also take into account disk failure rates, replacement rates (disks aren’t replaced immediately you know), and many other factors when making these decisions

Summary

Our job as architects when designing the system is to find that cost-benefit balance for the various parts of the system according to these very applicative parameters. No, it’s not easy. No, cloud computing will not magically solve all of this for us. But, we are getting more technical tools to work with, operations staff is getting better at working with us in the design phase, and our thought processes more rigorous in dealing with the scary conditions of the real world.

To your question, Libor, as to why we didn’t eventually use durable messaging in our solution, the answer is that we solved the overall state management problem by setting up an applicative protocol with our partners which was resilient in the face of faults by using idempotent messages that could be resent as many times as necessary. You can read more about it here. This solution isn’t viable for other kinds of interactions but was just what we needed to get the job done.

Hope that helps.

Disclaimer

Before giving some numbers, please keep in mind that high performance of system infrastructure does not necessarily by itself mean that the system above it is running that fast. For instance, you may have server heartbeats running really quickly but the time it takes to save a purchase order borders on a minute. So, please, take all benchmarks with a grain of salt, or two, or a whole shaker-full.

While I’m not at liberty to say on which specific domain/company these numbers were measured, I can say that we had the full gamut of “stateless services”, statefull services (sagas), number crunching, large data sets, many users, complex visualization, etc. Also, this wasn’t the largest installation of nServiceBus that I’m aware of, but its the one I have the most specific numbers for.

Setup

OK, so using the default nServiceBus distribution using MSMQ, on servers where the queue files themselves were on separate SCSI RAID disks, we were pumping around 1000 durable, transactionally processed messages per second, per server. That means that similar to the Neuron case, no messages would be lost in the case of a single fault per server per window (time to replace a failed disk set at 3 hours from failure, through detection, to replacement per site – but that’s more an operational staffing concern, not the technology itself).

So, that’s 3.6 million messages per hour per server, at full load. We had a total of 98 servers doing these kinds of processing, not including web servers, databases, etc. Keep in mind that web servers would be communicating with other servers using nServiceBus, but that would maybe be an unfair comparison to the Neuron numbers.

Server Breakdown

Anyway, the 48 number crunching servers (blade centers) we had were at full load, so we were pumping more than 170 million messages there. Keep in mind that those servers had a really fast backbone so weren’t held up by IO. Your environment may be different.

Another 30 (regular pizza boxes) were doing our sagas. Saga state was stored in a distributed in-memory “cache”, so once again IO wasn’t an issue for processing those messages. We were at about 70% utilization there, coming to just over 100 million messages an hour.

The last 20 were clustered boxes (fairly expensive) that handled the various nServiceBus distributor and timeout manager processes were at full load since they handled control messages for all the servers as well as dynamically routing the load. However, on those boxes we used much higher performance disks for the messages, since they had to feed everything else, capable of doing, on average, around 5000 messages a second. That adds up to 360 million messages an hour.

Unnecessary Durability

Later, we moved a bunch of messages that didn’t need all that durability and transactionality off the disks, pushing the total throughput over 1 billion messages an hour. That was about 100 million per hour durable, 900 million per hour non-durable. You can guess that we were left with plenty of IO to spare at that point while we weren’t yet pushing the limit of our memory.

One thing that’s important to understand is the size of the messages that didn’t require durability was less than 1MB, with most weighing in under 10KB. Also, since most of those messages were published, less state management was required around them, enabling us to further improve performance.

Summary

NServiceBus didn’t give us all that by itself. It was the result of skilled architects, developers, and operations staff working together for many iterations, deploying, monitoring, re-designing, etc. You need to understand your technology, your hardware, and your specific performance, availability, and fault-tolerance requirements if you want to get anywhere.

There’s no magic.

I didn’t see the number or kinds of servers involved in the Neuron case study so this wasn’t ever really a comparison. Nor or we talking about the same system here.

So, please, don’t base your decisions on arbitrary numbers. Spend some time setting up a scaled down version of your target architecture with all the relevant technologies and measure. Be aware that you want high performance end to end, not just of the messaging part. At times, it makes sense to actively throw away messages (of the non-durable, published kind) to help a server come online faster especially after a restart.

Thus ends the tale of another “benchmark”.

Spectacular Scalability with Smart Service Contracts

In this article, I attempt to debunk some of the myths around stateless-ness as the key to scalability.

Here’s how it starts:

It was a sunny day in June 2005 and our spirits were high as we watched the new ordering system we’d worked on for the past 2 years go live in our production environment. Our partners began sending us orders and our monitoring system showed us that everything looked good. After an hour or so, our COO sent out an email to our strategic partners letting them know that they should send their orders to the new system. 5 minutes later, one server went down. A minute after that, 2 more went down. Partners started calling in. We knew that we wouldn’t be seeing any of that sun for a while.

The system that was supposed to increase the profitability of orders from strategic partners crumbled. The then seething COO emailed the strategic partners again, this time to ask them to return to the old system. The weird thing was that although we had servers to spare, just a few orders from a strategic customer could bring a server to its knees. The system could scale to large numbers of regular partners, but couldn’t handle even a few strategic partners.

This is the story of what we did wrong, what we did to fix it, and how it all worked out.

Continue reading…

First of all let me say that Juval is indeed a master presenter. The “looks like a class, walks like a class, quacks like a class” bit was excellent. I could tell that most people didn’t notice the speedy hands quickly deleting all attributes from the classes before the “looks like a class…” bit. At times, I got flashbacks from the Wizard of Oz – “pay no attention to the man behind the curtain”. If all attributes in WCF only went on the interfaces, then this might actually fly, but we all know that that’s not the case.

One of the interesting comparisons Juval made with WCF was the introduction of .NET. Few people in the audience seemed to remember (or maybe were just professionally younger than .NET’s 8 years), but when it came out .NET was marketed as being mainly about XML Web Services. Juval stated that this was done to play down the fact that .NET made the previous Windows programming technologies obsolete. He then drew the same conclusion about WCF – that it’s as much .NET 3.0 as .NET was the next version of MFC; besides being written in a language that resembles the previous technology, it’s really all different. I don’t think that anyone would argue the difference, but is it really a “plain .NET” killer?

The answer seemed to come around the overhead of WCF – yet Juval deftly deflected that issue with a demo showing WCF doing 200 calls a second. And everybody just bought it – I was shocked. That’s 5ms per call. If you actually take Juval’s advice and use WCF on all your classes, you’ve bought yourself one hell of a performance nightmare. Say you have around 20 of your objects involved in a sequence to handle a user action – not that many actually. With a 5ms lag per object interaction, that user action is going to take 100ms – not including any database or webservice stuff you might be doing. If you do that in a server environment, you’ll be doing roughly 10 concurrent users per core. And that’s not even doing any heavy calculations or anything. Moderately sized systems are running upwards of 1000 concurrent users – if they needed 100 cores (or dozens of servers) for that, I’m guessing that they’d be out of business.

Let’s cut this short – WCF everywhere doesn’t scale, doesn’t perform, isn’t maintainable, or testable either. In other words – don’t do it. I know Juval is a brilliant guy, and an amazing presenter – but I don’t believe he would be employing this with his own clients. This actually bears repeating. WCF is a fine technology for your application’s boundaries, but don’t be pushing it in.

Don’t do it.

Multiple Threads – Why?

One of the properties of smart clients is that they should be able to work offline. Sometimes that means an explicit action of taking data and bringing it to the local machine so that the user can work on it, and other times it just has to do with the fact that wireless connectivity can be flaky. More interesting scenarios include the submission of batch jobs and receiving notification on when they complete. The bottom line is that the user should be able to continue doing their interactive work uninterrupted as all this is occurring.

While the user is disconnected, obviously the data they are working on is local – the client is not calling the server to perform the work on its behalf. However, in the flaky connectivity scenario, this happens all the time. In other words, the design should be the same for handling all scenarios. One thread for interacting with the user, and (at least) on other thread for handling the issues of connectivity. The one thing that is clear is that we are going to have data on the client with which the user interacts on one thread, and which the background thread will also be updating as notifications arrive from the server.

Local Data and Multi-Threading – a Recipe for Disaster

When multiple threads are working with the same data, unless some specific code is in that object to make it thread-safe, there’s a good chance that object will end up in an inconsistent state. This can be catastrophic if we’re talking about air traffic control systems, dispensing medication, factory floor automation, etc. The problem is that we can’t just lock down the entire system whenever something needs to be done. While in some cases I’ve seen projects make each object thread-safe, implementing some base class for handling locking, that doesn’t work between objects and results in deadlocks.

Just as an example of the multi-object problem, consider the doctor prescribing medication for a patient. Now, the data about a patient in a smart client is not all on the Patient Object – there are lists of connected test results, medication already being taken, which nurses and doctors have treated this patient so far, etc. As a test result gets pushed to the client from the server, a new doctor unfamiliar with the history of the patient orders the same test again. These two objects under Patient – TestRequest and TestResult are being acted upon by different threads. One of the goals of the system was to eliminate duplicate testing for patients – cited as costing the hospital chain millions of dollars a year. Good thing we addressed the cross-object multi-threaded locking thing properly

Always Switch from Background to Foreground?

One solution I’ve seen used successfully is to always change threads from background to foreground whenever an update comes in from the server. This works when you don’t have many updates or the work performed on the client as a result of an update doesn’t take long. Always keep in mind, though, the flaky connectivity scenario. What tends to happen is that server notifications bunch up and then hit the client all together. When this occurs, clients developed this way are rendered unusable.

It would seem like this solution isn’t valid because of the above, but don’t dismiss it so quickly. This is an easy solution to implement that may address your specific environment and, as such, be cost effective. A large part of the work I’ve been doing is to make the more complex environments just as easy to develop as these simpler ones.

Infrastructure-level, Safe, Multi-Object Locking

Luckily for us developers, in the .net framework there is a class that handles this for us – well, 2 actually: ContextBoundObject and SynchronizationAttribute. When using these two classes, we can create something known as a Synchronization Domain which acts as a global lock for all objects belonging to the synchronization domain. What this means is that if the user thread is trying to add a test request object while the background thread is already creating a test result object, the user thread will block automatically until the background thread completes its work.

There is only one teensy-weensy problem – ContextBoundObjects are really heavy-weight. The last thing you want is having millions of these running around in your client – you’ll end up with a multi-threading safe unusable UI. Also, the creation of an object inheriting from ContextBoundObject takes quite a bit longer than a plain-old.net object. In other words, technology by itself will not solve our problems – we need some patterns for the correct use of the technology so that we can maintain a reasonable level of performance while taking care of safety.

MVC and Threading – Controllers

Regardless of which flavour of MVC you prefer (I’m in the Supervising Controller camp for smart clients), the logic controlling what goes on in the client is found in the controllers. What this means is that actions from the user as well as background notifications will need to go through these controllers. It is important that these controllers be thread-safe since they are state-full – managing which windows are open, which step in a given process a user is currently doing, etc.

The characteristics of these controller objects which make them best suited to inherit from ContextBoundObject are that there are only a handful of these objects at any point in time and that they are created at startup – they’re singletons (in the “only-one-of-them” sense of the word).

The only special thing that controllers need to do in terms of threading is to dispatch calls to view objects on the foreground thread, even if the thread currently running is the background thread. For example, popping up a “toast” that a test result has arrived when a notification from the server comes in.

These elements – inheriting from ContextBoundObject, use of the SynchronizationAttribute, and thread-switching can be pulled up in to a BaseController class:

using System;
using System.ComponentModel;
using System.Runtime.Remoting.Contexts;

namespace ControllerFramework
{
    [Synchronization(SynchronizationAttribute.REQUIRED)]
    public class BaseController : ContextBoundObject
    {
        protected ISynchronizeInvoke invoker;
        public ISynchronizeInvoke Invoker
        {
            get { return invoker; }
            set { invoker = value; }
        }

        protected void MarshalToUiThread(Delegate toCall, params object[] parameters)
        {
            if (this.invoker == null)
                return;

            if (this.invoker.InvokeRequired)
                invoker.BeginInvoke(toCall, parameters);
            else
                toCall.DynamicInvoke(parameters);
        }
    }
}

Well, I think that this is long enough for a single blog post. In the next instalment of this series I’ll be talking about how model objects and views fit into the multi-threaded smart client. After that, we’ll be seeing how service agents, messaging, and service contract design need to be done in this style. While all this blogging will be going on, I’ll be getting a software factory up that will tie all these patterns and frameworks together so that all developers will be able to write thread-safe, high-performance smart clients without needing a doctorate in computer science – not that I have one

Questions? Comments? Thoughts?

If you’re interested in getting a kick-start with nServiceBus, I’ll be presenting a full day tutorial on it at QCon in London on March 11th. Here are the full details :

Title:

Build Scalable, Maintainable, Distributed Enterprise .NET Solutions with nServiceBus

Description:

Despite the recent flood of technologies and releases, distributed enterprise .net solution development remains as hard as ever.

WCF and WF provide valuable runtime components, yet still leave open the risk of developers using the wrong combination of options and ending up with an unscalable solution.

In this tutorial, developers will learn the specific patterns necessary to achieve scalability, as well as use supporting open-source frameworks to enable parallel development (and debugging!) of service interfaces, layers, and domain models.

After delving in to asynchronous message design, long-running workflow state management, and transaction-boundary placement developers will be able to design, debug, and deploy their specific distributed systems.

—

Hope to see you there.

The important thing to note is that if we just automatically return the message to the queue, we may get “stuck” with that message if the first PolicyCreatedMessage never arrived. This opens us up to a Denial-of-Service attack by quite simply flooding us with a bunch of messages that never get cleaned up.

Anyway, the general idea is to first try the regular happy path, and only if we see that prerequisite data isn’t available, do we see if another thread may be working on that data. This is done by decreasing the isolation level of our transaction from the regular ReadCommitted to ReadUncommitted. This will enable our thread to see if some other thread inserted the policy in to the Policies table but hasn’t committed its transaction yet.

The next step will be how we take this code and make it generic, so that we don’t have write the same code over and over again for the different kinds of message handlers we have.

But that will have to wait until the next installment

When I asked if anybody else saw it, one of the developers said, “Yeah, that happens sometimes – but it fixes itself right after that.”

“What if the user sends a command to the server making use of that location?”, the PM asked. “Could that cause them to collide?”

You could hear a pin drop.

After everyone got passed the preliminary shock, we got down to work. I asked if I could look at the logs, but after more than an hour, I found nothing. No reason to explain the strange behavior. I suggested doing some more instrumentation so that whenever a location changed on the client-side entities, we’d write that to the log.

After that, we ran the system again in the lab under the expected load (several hundred things moving every second, and the user doing the expected activity) and didn’t notice anything. An intern “volunteered” to keep working the system while the rest of us went to lunch. When we came back, he told us that everything seemed to be working OK.

These Heisenbugs are the things that keep me up at night.

“Watching the system changes its behavior”, one of the older devs nodded his head sagely.

Just as we were about to leave the lab another one of the developers gave a shout, “It did it again!”. We quickly stopped the system. Opened the (rather huge) log files and looked for the latest entries.

There it was.

A context switch between setting the latitude and longitude of an entity.

That should not have happened. Not that context switches don’t happen, but rather that it should have been impossible by design. We had made use of synchronization domains and the appropriate patterns so that two threads could never concurrently be working on the same instance of an entity. The synchronization features baked in to nServiceBus had taken care of everything up to that point.

Before getting into the threading solution, I want to address a specific alternate patch that was deployed in the meantime:

The solution for the long/lat problem was simple – just make Location a value object and use a single setter for it rather than one for Latitide and another for Longitude. We were still worried about other bits of data that were correllated in the domain – things that couldn’t be solved the same way.

After getting 3 grizzled C++ veterans in the room, we did a code walkthrough of the threading model of nServiceBus. We went through the nitty gritty details of synchronization domains, how the Bus object was outside of the domain, why that was important for user experience, how the message handlers couldn’t be ContextBoundObjects because of the performance impact of creating and destroying them at a high rate, why they couldn’t just be singletons, why they still had to run in the synchronization domain, so that the UI thread couldn’t work on the same (or related) objects at the same time, etc, etc.

And then it hit me.

The bus was communicated directly with the message handlers.

After the Object Builder created the message handler, the bus dispatched the message to the handler directly. And since the bus was outside the synchronization domain, then the thread calling into the handler wouldn’t have locked the domain, leaving the UI thread open to go in and touch those very same objects.

They say that really understanding the problem is 90% of the solution. I’m hoping to meet them some day, because they’re really smart.

All that we needed to do was have the Object Builder dispatch the message to the handler instead of the bus – since the builder was configured to be in the synchronization domain (on the client side). Something as simple as just adding the method:

So, instead of the bus using this code:

It would do:

[Just FYI, this is now up on the sourceforge site]

We redeployed the system to the lab, ran all the functional, stress, load, etc tests and everything appeared to be stable. The system has been under scrutiny for the past 4 days by batteries of testers instructed specifically to look for those strage kinds of behavior. Other developers are running scripts on the log files looking for other kinds of context switches that may have been missed by the testers. I am happy to report that they haven’t found anything.

Not that this means that the problem isn’t there. We really can’t be sure. However, the PM has decided that we are stable enough to go into pilot mode – deploying into production beside the current system; having users work on both systems at the same time. I’m optimistic.

I’m personally involved in two more production-projects that are making use of nServiceBus in similarly high-end situations and we’ve never had these threading problems – now two years running.

That was an interesting week.

The Setup

Just so that the example is in itself secure, we’ll assume that the password is one-way hashed before being stored. Also, given a reasonable network infrastructure our web servers will be isolated in the DMZ and will have to access some application server which, in turn, will communicate with the DB. There’s also a good chance for something like round-robin load-balancing between web servers, especially for things like user login.

Before diving into the meat of it, I wanted to preface with a few words. One of the commonalities I’ve found when people dismiss asynchrony is that they don’t consider a real deployment environment, or scaling up a solution to multiple servers, farms, or datacenters.

The Synchronous Solution

In the synchronous solution, each one of our web servers will be contacting the app server for each user login request. In other words, the load on the app server and, consequently, on the database server will be proportional to the number of logins. One property of this load is its data locality, or rather, the lack of it. Given that user U logged in, the DB won’t necessarily gain any performance benefits by loading all username/password data into memory for the same page as user U. Another property is that this data is very non-volatile – it doesn’t change that often.

I won’t go to far into the synchronous solution since its been analysed numerous times before. The bottom line is that the database is the bottleneck. You could use sharding solutions. Many of the large sites have numerous read-only databases for this kind of data, with one master for updates – replicating out to the read-only replicas. That’s great if you’re using a nice cheap database like mySql (of LAMP), not so nice if you’re running Oracle or MS Sql Server.

Regardless of what you’re doing in your data tier, you’re there. Wouldn’t it be nice to close the loop in the web servers? Even if you are using Apache, that’s going to be less iron, electricity, and cooling all around. That’s what the asynchronous solution is all about – capitalizing on the low cost of memory to save on other things.

The Asynchronous Solution

In the asynchronous solution, we cache username/hashed-password pairs in memory on our web servers, and authenticate against that. Let’s analyse how much memory that takes:

Usernames are usually 12 characters or less, but let’s take an average of 32 to be sure. Using Unicode we get to 64 bytes for the username. Hashed passwords can run between 256 and 512 bits depending on the algorithm, divide by 8 and you have 64 bytes. That’s about 128 bytes altogether. So we can safely cache 8 million of these with 1GB of memory per web server. If you’ve got a million users, first of all, good for you Second, that’s just 128 MB of memory – relatively nothing even for a cheap 2GB web server.

Also, consider the fact that when registering a new user we can check if such a username is already taken at the web server level. That doesn’t mean it won’t be checked again in the DB to account for concurrency issues, but that the load on the DB is further reduced. Other things to notice include no read-only replicas and no replication. Simple. Our web servers are the "replicas".

The Authentication Service

What makes it all work is the "Authentication Service" on the app server. This was always there in the synchronous solution. It is what used to field all the login requests from the web servers, and, of course, allowed them to register new users and all the regular stuff. The difference is that now it publishes a message when a new user is registered (or rather, is validated – all a part of the internal long-running workflow). It also allows subscribers to receive the list of all username/hashed-password pairs. It’s also quite likely that it would keep the same data in memory too.

The same message can be used to publish both single updates, and returning the full list when using NServiceBus. Let’s define the message:

And the message that the web server sends when it wants the full list:

And the code that the web server runs on startup looks like this (assuming constructor injection):

And the code that runs in the Authentication Service when the GetAllUsernamesMessage is received:

And the class on the web server that handles a UsernameInUseMessage when it arrives:

When the app server sends the full list, multiple objects of the type UsernameInUseMessage are sent in one physical message to that web server. However, the bus object that runs on the web server dispatches each of these logical messages one at a time to the message handler above.

So, when it comes time to actually authenticate a user, this the web page (or controller, if you’re doing MVC) would call:

When registering a new user, the web server would of course first check its cache, and then send a RegisterUserMessage that contained the username and the hashed password.

When the RegisterUserMessage arrives at the app server, a new long-running workflow is kicked off to handle the process:

That UsernameInUseMessage would eventually arrive at all the web servers subscribed.

Performance/Security Trade-Offs

When looking deeper into this workflow we realize that it could be implemented as two separate message handlers, and have the email address take the place of the workflow Id. The problem with this alternate, better performing solution has to do with security. By removing the dependence on the workflow Id, we’ve in essence stated that we’re willing to receive a UserValidatedMessage without having previously received the RegisterUserMessage.

Since the processing of the UserValidatedMessage is relatively expensive – writing to the DB and publishing messages to all web servers, a malicious user could perform a denial of service (DOS) attack without that many messages, thus flying under the radar of many detection systems. Spoofing a guid that would result in a valid workflow instance is much more difficult. Also, since workflow instances would probably be stored in some in-memory, replicated data grid the relative cost of a lookup would be quite small – small enough to avoid a DOS until a detection system picked it up.

Improved Bandwidth & Latency

The bottom line is that you’re getting much more out of your web tier this way, rather than hammering your data tier and having to scale it out much sooner. Also, notice that there is much less network traffic this way. Not such a big deal for usernames and passwords, but other scenarios built in the same way may need more data. Of course, the time it takes us to log a user in is much shorter as well since we don’t have to cross back and forth from the web server (in the DMZ) to the app server, to the db server.

The important thing to remember in this solution is doing pub/sub. NServiceBus merely provides a simple API for designing the system around pub/sub. And publishing is where you get the serious scalability. As you get more users, you’ll obviously need to get more web servers. The thing is that you probably won’t need more database servers just to handle logins. In this case, you also get lower latency per request since all work needed to be done can be done locally on the server that received the request.

ETags make it even better

For the more advanced crowd, I’ll wrap it up with the ETags. Since web servers do go down, and the cache will be cleared, what we can do is to write that cache to disk (probably in a background thread), and "tag" it with something that the server gave us along with the last UsernameInUseMessage we received. That way, when the web server comes back up, it can send that ETag along with its GetAllUsernamesMessage so that the app server will only send the changes that occurred since. This drives down network usage even more at the insignificant cost of some disk space on the web servers.

And in closing…

Even if you don’t have anything more than a single physical server today, and it acts as your web server and database server, this solution won’t slow things down. If anything, it’ll speed it up. Regardless, you’re much better prepared to scale out than before – no need to rip and replace your entire architecture just as you get 8 million Facebook users banging down your front door.

So, go check out NServiceBus and get the most out of your iron.

Neil asks:

Hi Udi,

We’re building a smart client application that uses WCF for full-duplex communications with our server. This is the asynchronous communication you talk about in your podcast. The smart-client is based on the MVC pattern, where model objects raise events when they’re changed so that the views can update themselves.

What’s started happening recently is that the smart-client has been freezing-up on us intermittently. We don’t know how to debug this and are wondering if its an architectural problem.

Any help you can give would be most appreciated.

Neil

Download

Download via the Dr. Dobb’s site

Or download directly here

Additional References

• Blog post on Occasionally Connected Systems Architecture
• Blog post on ObjectBuilder synchronization features needed for pub/sub-ing Smart Clients

Want more?

Check out the “Ask Udi” archives.

Got a question?

Send Udi your question to answer on the show.

So, I’ve implemented the pattern in NHibernate, adding the following method to ISession:

As well as adding the following interface to the NHibernate package:

All this enables you to have a stronger separation between your service layer classes and your domain model class, as well as for you to express each service-level use case as a domain concept – an interface.

Once you have such an interface, you can create a fetching strategy for that use case and define exactly how deep of an object graph you want to load so that you only hit the DB once for that use case.

The nice thing is that its all configured with Spring. In other words, if you the entry for your fetching strategy class exists, you get the improved performance, if it doesn’t, you don’t. All without touching your service layer classes.

Just as an example, when I’m in the use case modeled by “ICustomer”, I want to get all the customer’s orders, and their orderlines. This would be done by having a class like this:

And the configuration would look like this (as a part of the regular spring template):

If you want to take a look at the full solution, you can find it here. For some reason, the combined file was too big for the upload on my blog so it’s split into two. Unzip both packages into the same directory. You’ll find a file called “db_scripts.sql” which contains the schema for the DB. Don’t forget to update your connection string in the “hibernate.cfg.xml”. If you’re looking for the changes I made to the NHibernate source, you can find it in the “Updated NHibernate Files” directory. The only real change is to the “SessionImpl.cs” file.

Relevant NHibernate and Spring binaries.

Source code of example.

BTW, there is some intelligent thread-safe caching going on in SessionImpl now so that you get a much smaller performance hit (in terms of code that uses reflection) on subsequent usages of the same interfaces.

Let me know what you think.

It seems to me that nowadays we work more and more with data streams and not data sets. On a transaction-to-transaction basis, I think it’s an uncommon application that uses more data than can fit into several gigabytes of RAM (obvious exception: multimedia data).

While data stream processing is the heartbeat of many verticals, I’m seeing another trend there as well – the use of historical data as a part of that data stream processing. Some people have begun calling this Complex Event-Stream Processing (CEP), and the analysts are already beginning to eat it up. Regardless, the problem is that it is difficult to hold all historical data in memory so that when events arrive we can process them quickly.

So, my bottom line is that we’re being hit on multiple fronts – both the rate at which we need to process events and the amount of data required to process each event. Multiple cores help a bit, but probably not enough to discount scaling up to more machines. All this at the end of the day points out that we should not treat multiple cores any differently than multiple machines.

So, we either need languages to handle this (Erlang for one) or possibly frameworks (NServiceBus is my contribution). All I know is that Layered (Tiered) Architectures won’t cut it.

Before going too far ahead, you might want to take a look at my previous post “Space-Based Architectural Thinking, or listen to my podcast Space-Based Architecture for the Web. There’s also a 30 minute webcast online describing SBA more fully here. I’m also going to try to stay away from things concerning Jini this time after already discussing the connection between Jini and SOA, and the tradeoffs between two general approaches: Tasks and Spaces vs Message and Handlers.

OK, so the issue of state-management is a big one. Everybody wants to work stateless, because it scales. The only problem is that the business processes that we are automating are long running, meaning that there are external systems or people involved. This makes these processes inherently stateful. So, we need a way to scale statefully – SBA gives us that. For some background on the “Shared Nothing Architecture”, I suggest reading this post on inter-process SOA and this one as well.

Availability also has to be handled, not only in terms of having enough servers online to handle the required load but in having all the data required to process each request be accessible. This has often been handled by the database using ACID transactions – durability being that which solved availability issues, but also hurting latency the most. The problem with saving the state of our long-running business processes/workflows in the database is the load and the responsiveness requirements. In many verticals – telcos, financial, and defense to name a few, we need millisecond level latency on each stage of the workflow. This is what leads SBA to the in-memory, replicated data grid.

Note that SBA only intends to take these workflows out of the database, and not anything else – especially not Master Data. The lifetime of these workflows is incredibly short compared to that of master data like customers and products. It will have much different backup strategies as well. In terms of load, these workflows will be heavy on reads and writes together in the same transactions, but quite low in terms of just reads. If we have workflows that perform work in parallel, we easily end up with concurrency requirements that make DBAs cringe under the barrage of short transactions.

If you’re worried that Workflow Foundation (WF) won’t scale because of the above, you needn’t be. You can (more or less easily) replace the persistence mechanism of WF with your own, saving your workflow instances to an in-memory replicated data grid.

By enabling the objects in the grid to call back into logic on your servers, you have, in essence, done messaging and more. The added benefit that SBA receives from this is a unification of technology between caching and messaging. This translates directly to savings when it comes time to cluster each of those technology’s environments.

Finally, if we can find an attribute in the incoming stream of messages that creates a nice even distribution, we can then partition our load between our servers by that key. This will work up to the point where the load per key increases beyond a single server’s capacity, and then we have to look at re-partitioning, a non-trivial problem. However, if we put objects in our grid that represent the master data, and tie them to our workflow instances with both of those tied to the key of our load, a smart infrastructure can make sure all that data is already resident on the server that is handling that piece of the load. That decreases latency even more since we no longer have to pay network roundtrips to collect all the data needed before we can process it. That’s a substantial advantage for the above verticals.

But all of this has nothing to do with SOA.

Sure, it’ll change how we implement our Services internally, but it has no impact on their interfaces or the top-level service decomposition. In the Java community, the word “service” is often used to describe the logic of a system. Great significance is placed on keeping these “services” simple, as in Plain-Old Java Objects. The fact of the matter is that the logic of the system should be simple and independent of other concerns like data access and communcations (a la Web Services), but that does not make it a service, not in the SOA sense.

For more information on what Services in SOA are like, check out this podcast on Business and Autonomous Components in SOA. Actually, SBA will probably have the biggest impact on the way autonomous components will handle service-level agreements.

So, it appears that even with SOA, SBA has its place. The former dealing with business level agility, the latter dealing with all the technical aspects of supporting that agility. If you’re tasked with the designing the architecture of a scalable, available, high-throughput, low-latency distributed system, I’d strongly advise you to look at SBA – the technical value is overwhelming. Even if you don’t utilize all elements of SBA and choose the Master Worker Pattern instead of load partitioning, you’ll find the technologies supporting SBA to be quite flexible in that respect.

Will Space-Based Architectures be a part of your future? I don’t know for sure, but they’re a most welcome part of my present.

Ayende and I had an email conversation that started with me asking what would happen if I added an Order to a Customer’s “Orders” collection, when that collection was lazy loaded. My question was whether the addition of an element would result in NHibernate hitting the database to fill that collection. His answer was a simple “yes”. In the case where a customer can have many (millions) of Orders, that’s just not a feasible solution. The technical solution was simple – just define the Orders collection on the Customer as “inverse=true”, and then to save a new Order, just write:

Although it works, it’s not “DDD compliant”

In Ayende’s post Architecting for Performance he quoted a part of our email conversation. The conclusion I reached was that in order to design performant domain models, you need to know the kinds of data volumes you’re dealing with. It affects both internals and the API of the model – when can you assume cascade, and when not. It’s important to make these kinds of things explicit in the Domain Model’s API.

How do you make “transparent persistence” explicit?

The problem occurs around “transparent persistence”. If we were to assume that the Customer object added the Order object to its Orders collection, then we wouldn’t have to explicitly save orders it creates, so we would write service layer code like this:

On the other hand, if we designed our Domain Model around the million orders constraint, we would need to explicitly save the order, so we would write service layer code like this:

But the question remains, how do we communicate these guidelines to service layer developers from the Domain Model? There are a number of ways, but it’s important to decide on one and use it consistently. Performance and correctness require it.

Solution 1: Explicitness via Return Type

The first way is a little subtle, but you can do it with the return type of the “CreateOrder” method call. In the case where the Domain Model wishes to communicate that it handles transparent persistence by itself, have the method return “void”. Where the Domain Model wishes to communicate that it will not handle transparent persistence, have the method return the Order object created.

Another way to communicate the fact that an Order has been created that needs to be saved is with events. There are two sub-ways to do so:

Solution 2: Explicitness via Events on Domain Objects

The first is to just define the event on the customer object and have the service layer subscribe to it. It’s pretty clear that when the service layer receives a “OrderCreatedThatRequiresSaving” event, it should save the order passed in the event arguments.

The second realizes that the call to the customer object may come from some other domain object and that the service layer doesn’t necessarily know what can happen as the result of calling some method on the aggregate root. The change of state as the result of that method call may permeate the entire object graph. If each object in the graph raises its own events, its calling object will have to propagate that event to its parent – resulting in defining the same events in multiple places, and each object being aware of all things possible with its great-grandchild objects. That is clearly bad.

What [ThreadStatic] is for

So, the solution is to use thread-static events.

[Sidebar] Thread-static events are just static events defined on a static class, where each event has the ThreadStaticAttribute applied to it. This attribute is important for server-side scenarios where multiple threads will be running through the Domain Model at the same time. The easiest thread-safe way to use static data is to apply the ThreadStaticAttribute.

Solution 3: Explicitness via Static Events

Each object raises the appropriate static event according to its logic. In our example, Customer would call:

And the service layer would write:

The advantage of this solution is that it requires minimal knowledge of the Domain Model for the Service Layer to correctly work with it. It also communicates that anything that doesn’t raise an event will be persisted transparently behind the appropriate root object.

Statics and Testability

I know that many of you are wondering if I am really advocating the use of statics. The problem with most static classes is that they hurt testability because they are difficult to mock out. Often statics are used as Facades to hide some technological implementation detail. In this case, the static class is an inherent part of the Domain Model and does not serve as a Facade for anything.

When it comes to testing the Domain Model, we don’t have to mock anything out since the Domain Model is independent of all other concerns. This leaves us with unit testing at the single Domain Class level, which is pretty useless unless we’re TDD-ing the design of the Domain Model, in which case we’ll still be fiddling around with a bunch of classes at a time. Domain Models are best tested using State-Based Testing; get the objects into a given state, call a method on one of them, assert the resulting state. The static events don’t impede that kind of testing at all.

What if we used Injection instead of Statics?

Also, you’ll find that each Service Layer class will need to subscribe to all the Domain Model’s events, something that is easily handled by a base class. I will state that I have tried doing this without a static class, and injecting that singleton object into the Service Layer classes, and in that setter having them subscribe to its events. This was also pulled into a base class. The main difference was that the Dependency Injection solution required injecting that object into Domain Objects as well. Personally, I’m against injection for domain objects. So all in all, the static solution comes with less overhead than that based on injection.

Summary

In summary, beyond the “technical basics” of being aware of your data volumes and designing your Domain Model to handle each use case performantly, I’ve found these techniques useful for designing its API as well as communicating my intent around persistence transparency. So give it a try. I’d be grateful to hear your thoughts on the matter as well as what else you’ve found that works.

Related posts:

• Fetching Strategy Design – showing how to separate the concern of eager loading from both your Domain Model and your Service Layer.
• Better Domain-Driven Design Implementation – showing the basics of how valuable interfaces between your Domain Model and the Service Layer can be.
• Lazy Loading, and how messaging fixes everything again – describing the advantage of O/R mapping your message classes as well.
• Query Objects vs Methods on a Repository – discussing the scalability (in terms of number of developers and queries) of Query Objects.

I ran into another BizTalk optimization story online that once again points out that disk IO is a good first place to look, as I wrote about in my Database performance optimization article.

In closing, you need to be aware of the full environment. For instance, in BizTalk, it’s not just about messages per second. Session contention (multiple parties hitting the same session) can just lock things up tighter than, well, a tight thing. Designing for these things “up front” can save you very costly rewrites later.

However, in the truly large scale scenarios, that isn’t enough.

On of my customers is having to scale up from 500 concurrent users to 50,000. You need to get seriously close to the metal to handle that. Here’s a great post on the kind of hardware storage considerations I went through that I went through there. Of course, sometimes a database is just the wrong hammer for your screw – sometimes what you really need is a space.

Summary: I was horrified. My beautiful tiered architecture was buckling under the stress tests – at one tenth of the expected load of the system. And this was quality hardware, not the regular junk we scrapped together in most test labs, after all, this was the company’s flagship project. And I began seeing my once-bright career flashing before my eyes. The only redeeming thing about this situation was that I had learned the hard way to do performance tests early in the project, and that there might still be enough time to save the situation. A couple months later things were back on track but I’ll never forget the feeling I had back then – “All the big vendors are pushing this exact architecture. How could this be happening?” Since then, I’ve learned that tiers and networks are not to be taken lightly in any architecture and this article chronicles my journey, and how I got to a robust and scalable architecture.

Continue reading.

“… Finally on performance; if you’re truly seeing 1 transaction per second then something isn’t right. The SOAP adapter and heavy use of orchestration will knock your transaction rates down. Depending on what you’re doing you should be able to get that up over a hundred Tx/sec. on a well tuned BizTalk server farm.”

Isn’t one of the major selling points of BizTalk the whole webservices thing (SOAP) and the ability to express complex workflow (orchestration)? If it can’t do these things at a high transaction rate, then why do I need it? If the logic is simple, why don’t I just hand code it? I could probably get over 100 Tx/s on a single server, let alone “a well tuned … server farm”.