My article on “optimizing a large-scale Software+Services application” has been published in the April edition of MSDN Magazine.

Here’s a short excerpt:

“We had to juggle occasional connectivity, data synchronization, and publish/subscribe all at the same time. We learned that we couldn’t solve all problems either client-side or server-side, but rather that an integrated approach was needed since any changes on one side needed corresponding changes on the other side.”

Continue reading…

In regular client-server development, the server is responsible for providing the client with all CRUD (create, read, update, and delete) capabilities. However, when users look at data they do not often require it to be up to date to the second (given that they often look at the same screen for several seconds to minutes at a time). As such, retrieving data from the same table as that being used for highly consistent transaction processing creates contention resulting in poor performance for all CRUD actions under higher load.

A Scalable Solution

One of the common answers to this question is for the server/service to publish a message when data changes (say, as the result of processing a message) and for clients to subscribe to these messages. When such a notification arrives at a client, the client would cache the data it needs. Then, when the user wants to see a grid of data, that data is already on the client. Of course, this solution doesn’t work so well for older client machines (like some point of service devices) or if there are millions of rows of data.

The thing is that this solution is one implementation of a more general pattern – command query separation (CQS).

Command Query Separation

Wikipedia describes CQS as a pattern where "… every method should either be a command that performs an action, or a query that returns data to the caller, but not both. More formally, methods should return a value only if they are referentially transparent and hence possess no side effects."

Martin Fowler is less strict about the use of CQS allowing for exceptions: "Popping a stack is a good example of a modifier that modifies state. Meyer correctly says that you can avoid having this method, but it is a useful idiom. So I prefer to follow this principle when I can, but I’m prepared to break it to get my pop."

So, how does separating commands from queries and SOA help at all in getting data to and from a UI? The answer is based on Pat Helland’s thinking as described in his article Data on the Inside vs. Data on the Outside.

Services Cross Boxes

The biggest lie around SOA is that services run.

Let that sink in a second.

Sure services have runnable components, but that’s not why they’re important.

I’ll skip the books of background and cut to the chase:

Services communicate with each other using publish/subscribe and one-way messaging. Services have components inside them. Inside a service, these components can communicate with each using synchronous RPC, or any other mechanism. Also, these components can reside on different machines.

This is broader than just scaling out a service. There can be service components running on the client as well as the server.

SOA & CQS

Combining these two concepts together, here’s what comes out:

In this solution there are two services that span both client and server – one in charge of commands (create, update, delete), the other in charge of queries (read). These services communicate only via messages – one cannot access the database of the other.

The command service publishes messages about changes to data, to which the query service subscribes. When the query service receives such notifications, it saves the data in its own data store which may well have a different schema (optimized for queries like a star schema).

The client component which is in charge of showing grids of data to the user behaves the same as it would in a regular layered/tiered architecture, using synchronous blocking request/response to get its data – SOA doesn’t change that.

Composite Applications

Although the client side components of both the command and query services are hosted in the same process, they are very much independent of each other. That being said, from an interoperability perspective (the one that most people attribute to SOA), all of the client-side components will likely be developed using the same technology – although there are already ways to host Java code in .NET and vice-versa.

Of course, once we talk about web UI’s things are a bit different – but still similar. While web-server-side there may be a level of independence, for browser side inter-component communications we’re still likely to target javascript. There, I’ve managed to say something technical supporting mashups and SOA without lying through my teeth.

On the Microsoft side with the recent release of the Composite Application Guidance & Library (pronounced "prism") I hope that more of these principles will be reaching the "smart client". The command pattern is especially critical in maintaining the separation while enabling communication to still occur so I’m glad that, as one of the Prism advisors, I was able to simplify that part (Glenn still has nightmares about that rooftop conversation).

Publish / Subscribe

In the "scalable solution" section up top I mentioned how publish/subscribe to the smart client is really just one implementation of CQS and SOA. So, how different is it really?

Well, there will probably be a different technology mapping. Instead of a star-schema OLAP product, we might simply store the published data in memory on the client. That is, if you designed your components to be technology agnostic.

In terms of the use of nServiceBus, the same component is going to be subscribing to the same type of message – all that’s different is that now every client will be having data pushed to them rather than this occurring server-side only.

You could have the same code deployed differently in the same system – stronger clients subscribing themselves, weaker ones using a remote server. Web servers would probably be considered stronger clients. This kind of flexible deployment has proven to be extremely valuable for my larger clients. The added benefit of enabling users to work (view data) even while offline (somewhere there’s no WIFI) is just icing on the cake.

A Word of Warning

Once the client starts receiving notifications, and handling those on a background thread (as it should) the code becomes susceptible to deadlocks and data races. Juval does a good job of outlining some of those with respect to the use of WCF. Prism doesn’t provide any assurances in this area either.

Summary

NServiceBus is not designed to be used for any and all types of communication in a given architecture. In the examples above, nServiceBus handles the publish/subscribe but leaves the synchronous RPC to existing solutions like WCF. Not only that, but synchronous RPC does have its place in architecture, just not across service boundaries. In all cases, data is served to users from a store different from that which transaction processing logic uses.

Command Query Separation is not only a good idea at the method/class level but has advantages at the SOA/System level as well – yet another good idea from 20 years ago that services build upon. Making use of CQS requires understanding your data and its uses – SOA builds on that by looking into data volatility and the freshness business requirements around it.

Finally, designing the components of your services in such a way that their dependency on technology is limited buys a lot of flexibility in terms of deployment and, consequently, significant performance and scalability gains.

Simple, it is. Easy, it is not.

One of the topics not covered by Prism is occasional connectivity, and I would like to say a word or two about that. First of all, if you’re building a standalone client (one that doesn’t communicate with anything), then there’s a good chance that Prism isn’t for you, although you could be composing other standalone client modules. So, if your client isn’t communicating with anything, well, then this post probably won’t interest you that much. Let’s start with…

Physics

Networks fail. Period.

This means that your client machine will not always be connected to other servers.

Also, servers fail – critical Windows patches and just regular power outages.

Ergo, your “smart” client will be occasionally connected, whether you planned for it or not.

And please don’t take this post as a “dumping on Prism” post – it isn’t intended that way. Rather, it is about how you should think about designing modules in Prism, and why.

Modules and Connectivity

Consider the case where we have two modules being composed in a single client. Each module communicates with a different server. Let’s call these modules Ma and Mb, and the servers Sa and Sb respectively. Now, let’s discuss what occurs given that the modules weren’t designed with occasional connectivity in mind.

User clicks something in Mb which requires communication.

Mb tries to call Sb, say, over HTTP, using a regular web service invocation.

The calling thread, in this case, the one used for user interaction, is blocked waiting for a response from Sb.

Sometime in this call, Sb fails, connectivity goes down, whatever.

30 seconds after the call, the HTTP connection times out.

If something important were happening in Ma at the same time, the user couldn’t even see it, let alone do anything about it since the user interaction thread is stuck. This is a serious concern for the financial services domain, but in many others as well.

You mean there’s more?

I can go on, but I think that that’s enough to paint the picture that if you are building a smart client, there are a lot more things to think about than just learning Prism. That’s my main concern after witnessing what happened around the CAB. Given the learning curve around these frameworks many developers don’t seek to deepen their understanding beyond just becoming proficient with them. This isn’t just centered on the developers, evangelists in Microsoft tend to paint the picture this way:

Once you understand X (CAB, Prism, BizTalk, whatever), all your problems are solved.

That’s not to say there aren’t good things in those technologies, but that’s just it, they’re just tools. Silver hammers and “laser” guided saws do not a master carpenter make. There’s actually a pretty good chance the regular guy will saw their arm off.

Help

I do hope more “instruction manuals” will be coming out of Microsoft on these topics. That’s not to say there aren’t any. Specifically on the topic of occasional connectivity, there is Chapter 4 of the Smart Client Architecture & Design Guide. Unfortunately, it doesn’t say anything about how that connects with the MVC/MVP being used client side (the bits affected by Prism). Chapter 6 of the same guide deals with the client-side threading, but doesn’t address issues like:

• Which model object instance are views bound to.
• Do other threads have access to that object at the same time.
• Which controller/presenter is responsible for giving that object to the view.
• Do they need to clone it.
• How deep should the clone be.
• How do various controllers/presenters (which may be showing the same object in different views at the same time) communicate changes to their various independent clones.

I haven’t yet documented all the patterns that answer these questions, but until I do (or Microsoft does), let me offer these few resources which I’ve put out over the years:

• Occasionally Connected Systems Architecture: The Client
• Occasionally Connected Systems Architecture: Concurrency
• Threading issues when smart clients reconnect
• [Podcast] Occasionally Connected Smart Clients and ADO.NET Sync Services

There’s also some more links under the Smart Client link of my “First time here?” page.

Also, please join me in asking Microsoft for an update to these guides – comments below or your own blog posts would be great.

Here’s a taste:

UI Thread and Concurrency Management

Whenever you use hosting on the UI thread, deadlocks are possible. For example, the following setup is guaranteed to result with a deadlock: A Windows Forms application is hosting a service with UseSynchronizationContext set to true, and UI thread affinity is established. The Windows Forms application then calls the service over one of its endpoints. The call to the service blocks the UI thread, while WCF posts a message to the UI thread to invoke the service. That message is never processed, because of the blocking UI thread—hence, the deadlock.

Another possible case for a deadlock occurs when a Windows Forms application is hosting a service with UseSynchronizationContext set to true and UI thread affinity is established. The service receives a call from a remote client. That call is marshaled to the UI thread and is eventually executed on that thread. If the service is allowed to call out to another service, that can result in a deadlock if the callout causality tries somehow to update the UI or call back to the service’s endpoint, because all of the service instances that are associated with any endpoint (regardless of the service-instancing mode) share the same UI thread.

Similarly, you risk a deadlock if the service is configured for reentrancy and it calls back to its client. You risk a deadlock if the callback causality tries to update the UI or enter the service, because that reentrance must be marshaled to the blocked UI thread.

Actually, I have difficulty believing that Juval would go so far as to suggest that even the forms should be services, but he does:

Form as a Service

The main motivation for hosting a WCF service on the UI thread is if the service must update the UI or the form. The problem is always: How does the service reach out and obtain a reference to the form? While the techniques and ideas that appear thus far in the listings certainly work, it would be simpler yet if the form were the service and hosted itself. For this to work, the form (or any window) must be a singleton service. The reason is that singleton is the only instancing mode that enables you to provide WCF with a live instance to host. In addition, you would not want a per-call form that exists only during a client call (which is usually very brief), nor would you want a per-session form that only a single client can establish a session with and update.

When a form is also a service, having that form as a singleton service is the best instancing mode all around.

I think that this article serves as a great treatise leading to only one conclusion – you’d have to be crazy to try to do this without some higher level framework, preferably with a different low-level framework too . Sucks Microsoft didn’t put one out – nor is there a pending beta, CTP, or even word about some project with a codename handling this. From what I know about Prism, it doesn’t intend to handle this issue either.

One thing that isn’t covered in the article is that if you do choose not to tie the client-side service to the UI thread, you open yourself up to race conditions. Reasons you’d want to handle messages on a different thread center around UI responsiveness. I’ve written about these things before:

• Object Builder – the place to fix system-wide threading bugs

• Eureka! AOP is the final piece of the multi-threaded smart client puzzle

• What Makes Smart Clients Safe?

The more I read things like this, the more I feel that I have to get going with my nServiceBus based solution. I’m fairly swamped as it is, so if anyone is interested in helping get this project off the ground, I’d be most grateful – as I think anyone else that had to build a smart client would.

After the grand release of WPF, and the industry’s collective shrug and back to business, Microsoft stirs the pot again.

I’ve read Brian’s authoritative description, even done some of the examples he’s outlined, and let me tell you, tech won’t help.

Again.

I don’t know about differentiation, but empowerment is important. Make the system work for the user. What an idea.

Let the user specify minimal inventory levels for products, and rules for replenishing. Give them a dashboard where they can see the status – what’s back ordered, orders expected to be authorized in the next relevant time interval, recent supplier timeliness, etc. If you’re thinking that this ties in to Event Driven Architecture, you’re right. Let the user subscribe to business events that interest them, pop up a messenger-like notification when they occur. Let the user click on the data in there, and have the rest of the UI synchronize around that.

Purchasing has struck a new deal with a supplier? Let me know. Click. Products they supply open in one view. Suppliers providing the same or similar products open in another. Contact info in another. Replenishment times in another.

Everything synchronized. Everything in real time.

Let them seamlessly work connected and disconnected.

You do that with regular boring winforms, and they’ll kiss the ground you walk on.

Differentiated user experience? Shrug.

Multiple Threads – Why?

One of the properties of smart clients is that they should be able to work offline. Sometimes that means an explicit action of taking data and bringing it to the local machine so that the user can work on it, and other times it just has to do with the fact that wireless connectivity can be flaky. More interesting scenarios include the submission of batch jobs and receiving notification on when they complete. The bottom line is that the user should be able to continue doing their interactive work uninterrupted as all this is occurring.

While the user is disconnected, obviously the data they are working on is local – the client is not calling the server to perform the work on its behalf. However, in the flaky connectivity scenario, this happens all the time. In other words, the design should be the same for handling all scenarios. One thread for interacting with the user, and (at least) on other thread for handling the issues of connectivity. The one thing that is clear is that we are going to have data on the client with which the user interacts on one thread, and which the background thread will also be updating as notifications arrive from the server.

Local Data and Multi-Threading – a Recipe for Disaster

When multiple threads are working with the same data, unless some specific code is in that object to make it thread-safe, there’s a good chance that object will end up in an inconsistent state. This can be catastrophic if we’re talking about air traffic control systems, dispensing medication, factory floor automation, etc. The problem is that we can’t just lock down the entire system whenever something needs to be done. While in some cases I’ve seen projects make each object thread-safe, implementing some base class for handling locking, that doesn’t work between objects and results in deadlocks.

Just as an example of the multi-object problem, consider the doctor prescribing medication for a patient. Now, the data about a patient in a smart client is not all on the Patient Object – there are lists of connected test results, medication already being taken, which nurses and doctors have treated this patient so far, etc. As a test result gets pushed to the client from the server, a new doctor unfamiliar with the history of the patient orders the same test again. These two objects under Patient – TestRequest and TestResult are being acted upon by different threads. One of the goals of the system was to eliminate duplicate testing for patients – cited as costing the hospital chain millions of dollars a year. Good thing we addressed the cross-object multi-threaded locking thing properly

Always Switch from Background to Foreground?

One solution I’ve seen used successfully is to always change threads from background to foreground whenever an update comes in from the server. This works when you don’t have many updates or the work performed on the client as a result of an update doesn’t take long. Always keep in mind, though, the flaky connectivity scenario. What tends to happen is that server notifications bunch up and then hit the client all together. When this occurs, clients developed this way are rendered unusable.

It would seem like this solution isn’t valid because of the above, but don’t dismiss it so quickly. This is an easy solution to implement that may address your specific environment and, as such, be cost effective. A large part of the work I’ve been doing is to make the more complex environments just as easy to develop as these simpler ones.

Infrastructure-level, Safe, Multi-Object Locking

Luckily for us developers, in the .net framework there is a class that handles this for us – well, 2 actually: ContextBoundObject and SynchronizationAttribute. When using these two classes, we can create something known as a Synchronization Domain which acts as a global lock for all objects belonging to the synchronization domain. What this means is that if the user thread is trying to add a test request object while the background thread is already creating a test result object, the user thread will block automatically until the background thread completes its work.

There is only one teensy-weensy problem – ContextBoundObjects are really heavy-weight. The last thing you want is having millions of these running around in your client – you’ll end up with a multi-threading safe unusable UI. Also, the creation of an object inheriting from ContextBoundObject takes quite a bit longer than a plain-old.net object. In other words, technology by itself will not solve our problems – we need some patterns for the correct use of the technology so that we can maintain a reasonable level of performance while taking care of safety.

MVC and Threading – Controllers

Regardless of which flavour of MVC you prefer (I’m in the Supervising Controller camp for smart clients), the logic controlling what goes on in the client is found in the controllers. What this means is that actions from the user as well as background notifications will need to go through these controllers. It is important that these controllers be thread-safe since they are state-full – managing which windows are open, which step in a given process a user is currently doing, etc.

The characteristics of these controller objects which make them best suited to inherit from ContextBoundObject are that there are only a handful of these objects at any point in time and that they are created at startup – they’re singletons (in the “only-one-of-them” sense of the word).

The only special thing that controllers need to do in terms of threading is to dispatch calls to view objects on the foreground thread, even if the thread currently running is the background thread. For example, popping up a “toast” that a test result has arrived when a notification from the server comes in.

These elements – inheriting from ContextBoundObject, use of the SynchronizationAttribute, and thread-switching can be pulled up in to a BaseController class:

using System;
using System.ComponentModel;
using System.Runtime.Remoting.Contexts;

namespace ControllerFramework
{
    [Synchronization(SynchronizationAttribute.REQUIRED)]
    public class BaseController : ContextBoundObject
    {
        protected ISynchronizeInvoke invoker;
        public ISynchronizeInvoke Invoker
        {
            get { return invoker; }
            set { invoker = value; }
        }

        protected void MarshalToUiThread(Delegate toCall, params object[] parameters)
        {
            if (this.invoker == null)
                return;

            if (this.invoker.InvokeRequired)
                invoker.BeginInvoke(toCall, parameters);
            else
                toCall.DynamicInvoke(parameters);
        }
    }
}

Well, I think that this is long enough for a single blog post. In the next instalment of this series I’ll be talking about how model objects and views fit into the multi-threaded smart client. After that, we’ll be seeing how service agents, messaging, and service contract design need to be done in this style. While all this blogging will be going on, I’ll be getting a software factory up that will tie all these patterns and frameworks together so that all developers will be able to write thread-safe, high-performance smart clients without needing a doctorate in computer science – not that I have one

Questions? Comments? Thoughts?

You see, the main problem that I hadn’t been able to solve was that in order for the code to be thread-safe, you had to make sure that no code in the views would/could change entity data. One solution is not to use data-binding, which sucks, but isn’t enough to be sure. Another solution is to have all supervising-controllers clone an entity before they give it to a view. Even if you could possibly code review every line of those classes, the new guy (or old guy who forgot) will, by accident, write one new line of code that could pass an entity to a view without cloning it first. That’s not a very sustainable solution.

This thing has been bothering me for a couple of months now and I hadn’t found a way around it. Until yesterday, like I said. I was talking to somebody about threading stuff, and somehow my unconscience lobbed me this thought about AOP. Now I’m not the sharpest pencil in the pack, but I know to listen when my unconscience “speaks”.

So I set about going over what I knew about AOP – interceptors, advisors, advice, introductions, etc, etc. And then it dawned on me. I could intercept all calls to any object that implemented IView, check the parameters of those calls, and if they implemented IEntity, to clone them before passing them through.

<Homer-style WOOHOO />

The great thing is that developers don’t need to remember to clone entities – it happens automatically. The even greater thing is that this will lead developers to writing the correct kind of interaction between their views and supervising controllers.

Together with nServiceBus, this is going to make the extremely difficult problem of writing thread-safe smart clients possible.

I’ve never made use of AOP in a framework before so I’d like to get the broader community’s feedback on this before incorporating this in production. I’ve spoken with some serious AOP folks who have allayed most of my uncertainties, but I’d like to hear more. Anyway, here’s the proof of concept (that makes use of Spring).

If this turns out to be a viable solution, I think we’ll have a solid environment for building a software factory on top of. That is something that I’m really excited about. In this multi-core future (present) that is upon us, multi-threading on the client is pretty much a necessity. We need a way to get things safe and stable by default without requiring a member of the CLR team to hold our hand.

Anybody who’s interested in helping, drop a comment below.

When I asked if anybody else saw it, one of the developers said, “Yeah, that happens sometimes – but it fixes itself right after that.”

“What if the user sends a command to the server making use of that location?”, the PM asked. “Could that cause them to collide?”

You could hear a pin drop.

After everyone got passed the preliminary shock, we got down to work. I asked if I could look at the logs, but after more than an hour, I found nothing. No reason to explain the strange behavior. I suggested doing some more instrumentation so that whenever a location changed on the client-side entities, we’d write that to the log.

After that, we ran the system again in the lab under the expected load (several hundred things moving every second, and the user doing the expected activity) and didn’t notice anything. An intern “volunteered” to keep working the system while the rest of us went to lunch. When we came back, he told us that everything seemed to be working OK.

These Heisenbugs are the things that keep me up at night.

“Watching the system changes its behavior”, one of the older devs nodded his head sagely.

Just as we were about to leave the lab another one of the developers gave a shout, “It did it again!”. We quickly stopped the system. Opened the (rather huge) log files and looked for the latest entries.

There it was.

A context switch between setting the latitude and longitude of an entity.

That should not have happened. Not that context switches don’t happen, but rather that it should have been impossible by design. We had made use of synchronization domains and the appropriate patterns so that two threads could never concurrently be working on the same instance of an entity. The synchronization features baked in to nServiceBus had taken care of everything up to that point.

Before getting into the threading solution, I want to address a specific alternate patch that was deployed in the meantime:

The solution for the long/lat problem was simple – just make Location a value object and use a single setter for it rather than one for Latitide and another for Longitude. We were still worried about other bits of data that were correllated in the domain – things that couldn’t be solved the same way.

After getting 3 grizzled C++ veterans in the room, we did a code walkthrough of the threading model of nServiceBus. We went through the nitty gritty details of synchronization domains, how the Bus object was outside of the domain, why that was important for user experience, how the message handlers couldn’t be ContextBoundObjects because of the performance impact of creating and destroying them at a high rate, why they couldn’t just be singletons, why they still had to run in the synchronization domain, so that the UI thread couldn’t work on the same (or related) objects at the same time, etc, etc.

And then it hit me.

The bus was communicated directly with the message handlers.

After the Object Builder created the message handler, the bus dispatched the message to the handler directly. And since the bus was outside the synchronization domain, then the thread calling into the handler wouldn’t have locked the domain, leaving the UI thread open to go in and touch those very same objects.

They say that really understanding the problem is 90% of the solution. I’m hoping to meet them some day, because they’re really smart.

All that we needed to do was have the Object Builder dispatch the message to the handler instead of the bus – since the builder was configured to be in the synchronization domain (on the client side). Something as simple as just adding the method:

So, instead of the bus using this code:

It would do:

[Just FYI, this is now up on the sourceforge site]

We redeployed the system to the lab, ran all the functional, stress, load, etc tests and everything appeared to be stable. The system has been under scrutiny for the past 4 days by batteries of testers instructed specifically to look for those strage kinds of behavior. Other developers are running scripts on the log files looking for other kinds of context switches that may have been missed by the testers. I am happy to report that they haven’t found anything.

Not that this means that the problem isn’t there. We really can’t be sure. However, the PM has decided that we are stable enough to go into pilot mode – deploying into production beside the current system; having users work on both systems at the same time. I’m optimistic.

I’m personally involved in two more production-projects that are making use of nServiceBus in similarly high-end situations and we’ve never had these threading problems – now two years running.

That was an interesting week.

Occasionally Connected Systems Architecture: Concurrency

When designing systems that support users who need to work offline, one of the challenges developers face is how to deal with concurrency problems when these users re-connect and synchronize their data with the server. Explore patterns for conflict identification and resolution in the first of this series on Occasionally Connected Systems Architecture.

Continue reading.

Neil asks:

Hi Udi,

We’re building a smart client application that uses WCF for full-duplex communications with our server. This is the asynchronous communication you talk about in your podcast. The smart-client is based on the MVC pattern, where model objects raise events when they’re changed so that the views can update themselves.

What’s started happening recently is that the smart-client has been freezing-up on us intermittently. We don’t know how to debug this and are wondering if its an architectural problem.

Any help you can give would be most appreciated.

Neil

Download

Download via the Dr. Dobb’s site

Or download directly here

Additional References

• Blog post on Occasionally Connected Systems Architecture
• Blog post on ObjectBuilder synchronization features needed for pub/sub-ing Smart Clients

Want more?

Check out the “Ask Udi” archives.

Got a question?

Send Udi your question to answer on the show.

The first is to insulate the framework and application code that I write from the choice of one dependency injection technology or another. I want the ability to switch easily from one to the other – not so much that projects go back and forth. Updating those config files is definitely not easy. However, it allows me to have “portable” framework code that is applicable to all the projects I consult on, regardless of their choice of technology.

The second has to do with NServiceBus specifically. In order to make use of duplex communication on smart clients, you need a background thread. That thread will be updating the same (model) objects as the UI thread. That means we need synchronization. I prefer to use .NET’s built-in synchronization domains in order to solve this rather thorny problem.

The only thing is that message handlers need to be in the synchronization domain so that they can easily update those objects. However, the Bus object must not be in the synchronization domain so that if we’ve received a large update from the server, we won’t be locking out the UI thread from interacting with data on the client.

Since the bus makes use of a dependency injection framework to create message handlers, this was the best place to put the code which causes message handlers to run within the synchronization domain.

Be aware that in order to enjoy this feature, you need to split up those large server updates into multiple, logical objects (that implement IMessage), but you can still publish them all in one go using the method:

void Publish(params IMessage[] messages);

And, of course, you need to set the JoinSynchronizationDomain property of the Object Builder.

I’ll have a podcast coming out on this topic soon.

You can get the code here:

Object Builder.zip

But you’ll have to get the Spring Framework code from the official site. Make sure you download RC 1.1. Then, take the binaries and copy them to the “BIN” folder of the Object Builder solution. If you’re looking to save on some “weight”, you only need “Spring.Core.dll”, “Common.Logging.dll” and “antlr.runtime.dll” for the solution to compile. You will need one of the logging implementations DLLs to get anything written to a log, obviously.

Download via the Dr. Dobb’s site.

Or download directly here.

And here’s the question that gets us started:

Hello there,

We are developing an application in .NET Smart Client application which connects to a WCF service and then to the data access layer. The issue we have here is on the type of data which can be passed between the client side and web service.

1. Can it be a dataset (No!), custom entity or XML or anything else ?

Which is the best one ?

2. Ofcourse , a dataset can’t be used coz it can’t be used in non-.NET environment?

3. If we go for custom-entity or XML how do we ensure the changes happen to the data during client-side actions ? For example, binding the XML or custom entity to a data grid can be done. But if user makes any changes to the same, how do we identify ?

4. If we use custom entities / XM L, the same can not be passed to the Data Access Application Block which invokes the data objects.

Note : All the above are based on assumption without using a translator in between any of the layers.

Please provide a better solution if you have come across any.

Additional References:

• Podcast on using DataSets with Web Services
• Blog post on tracking changes to objects on the client side
• Looking for more of the “Ask Udi” podcast?

What I find promising: an Acropolis part can essentially consume a SOA service, allowing the composition of process and activity services to be as simple as snapping parts onto a surface.

Since services which conform to SOA principles will expose duplex/asynchronous APIs, this will almost necessarily make the client-side interaction multi-threaded. I do, and have done quite a lot of work on multi-threaded, thread-safe, smart clients. They’re hard. Here’s a post describing how hard. There are other solutions as well, but very little of it has to do with technology.

Since both the CAB and Acropolis do very little in handling, or even describing in their docs these multi-threading issues, I don’t think that they’re going to make it simple or easy to work with SOA-style services. This is not to say they don’t provide any value. On the contrary, I think that they bring the mainstream many steps forward. However, there is still a long way to go.

The answer to this is that this callback is designed for Controller classes in Smart Clients, at least when employing an MVC style. You see, as the Controller class is the one who decided to initiate the action on the remote service by calling into a Service Agent class, it will also contain the logic for what to do when a result arrives. Therefore, it needs to pass a callback to the Service Agent class, possibly also passing in some other state (for instance, a reference to the form for which it is making this call, if there can be multiple instances of the same type of form open).

The only dependency the Controller class has is on the fact that there is asynchronous communication happening – made clear by the tiny assembly containing only the callback definition. The Controller does not need to know anything about ESBs and such, and thus does not need a reference to the IBus interface project.

As always, when working asynchronously you need to be aware that these callbacks will be coming in on a thread other than the UI thread. This means that if your Controller class has any state (ie. member variables), you need to protect that.

On kind of input validation is something you do as close to the user as possible for performance reasons. This includes all sorts of smart stuff you can do with JavaScript in Web scenarios. When in a Smart Client environment, you usually have greater capabilities.

When I look at the issue of validation, I see that it centers around the entity. Sometimes, it is also affected by other things, like what process are we in (as described in the comments on Ayende’s post).

So, we can model the thing that validates an entity with an interface, say, IValidator<T> where T : IEntity. This interface will have one main method: bool IsValid(T entity); and one main property: string ErrorDescription { get; }

What this allows us to do is to separate out different validation concerns into different classes, yet have all of them implement the same interface.

The next thing we’ll need is to be able to get an instance for each of the classes that is a validator for a specific kind of entity. For instance, when the NewCustomerView raises an event proclaiming that it has a Customer object ready to be saved, the Controller will want to find all classes that implement IValidator<Customer> so that it can run all the validation rules.

Luckily, the generics patch I put out for the Spring.Net Framework allows us to do this in one simple line of code:

IList<IValidator<Customer>> validators = spring.GetObjectsOfType(typeof(IValidator<Customer>));

and quite simply perform the validation as follows:

foreach(IValidator<Customer> v in validators)
  if (!v.IsValid(myCustomer))
    // notify user with v.ErrorDescription, write to log, whatever

Now, when using this in Smart Client scenarios, you will often have views that allow the user to enter in a single entity which you will then want to validate. If you have those views implement a generic interface like: IEntityView<T> where T : IEntity, then you could have a single base class implement a “Validate” method, which would perform the work above generically like so:

foreach(IValidator<T> v in spring.GetObjectsOfType(typeof(IValidator<T>)))
  if (!v.IsValid(this.Entity))
    // notify user with v.ErrorDescription

and just have your specific view call that method on the button click.

This enables all entity views to activate all the relevant custom validation logic without being tied to it. It also enables you to extend your system by adding new classes implementing the IValidator<T> interface, and have them automatically run without even changing a config file. How’s that for loose coupling?

Finally, on the issue of tying validation rules to specific processes, this can be done by extending the interface to: IValidator<T, P> where T : IEntity, P : IProcess; You then model each process with a marker interface. You then have your specific validation classes implement the above generic interface for each specific process interface. For instance, say we have a validation rule that needs to run for processes P1 (marked by IP1), and P2 (marked by IP2), but not P3 (marked by IP3), which validates entities of type E. This would be done by defining the class like so:

public class MyValidator : IValidator<E, IP1>, IValidator<E, IP2> {}

and the Controller class that would request validators for process 2 would just call:

spring.GetObjectsOfType(typeof(IValidator<T, IP2>));

That’s it. The basic principles are simple, but, as you can see, can create very powerful structures. I’ve got to say that this exemplifies one of the reasons why I love generics so much. When used with Dependency Injection, and/or Delegates, and/or Anonymous Methods, you get such a power of expression just by defining an interface. This is one of things that make coding fun for me. Or maybe I’m just wierd that way

The answer is simple.

You don’t have to.

Or, if you think you do, you’re probably using them wrong.

An O/R mapper should NOT live in its own tier.

Object/Relational Mappers never stand alone. They are used to provide persistence for something else – either for the Domain Model Pattern or for the Active Record pattern.

In terms of scalability, again these patterns usually don’t stand by themselves, but rather are hosted by something else. When used on the client side, say in a smart client application, then scalability isn’t often considered. There are some kinds of smart clients where hitting the database on most user interactions will bring the system to its knees, but that’s again an issue of database scalability. The common solution is some kind of client-side caching.

Anyway, the two main parameters you need to look at for your common and high-load scenarios when using an O/R mapper are these:

• How many times do you hit the DB per business action.
• How many objects/records/rows/columns do you bring into memory per business action.

You should be looking at appropriate uses of Lazy Loading for both of them.

Finally, keep in mind that O/R mappers are only part of your solution. Measure and optimize wisely.

Hi Udi,

I’m enjoying the recent discussion on Entity Framework, disconnected clients, Unit of Work, and messaging. A few weeks ago I wrote a note to self to “Ask Udi” about the new ADO.NET Sync Services coming out of Microsoft, and this seems to be the perfect timing.

It really seems that Sync Services is meant to address the disconnected problem raised by Andres, but it appears to be primarily targeted at enabling DataSet-centric, grid-oriented UIs, not task-oriented UIs. They are spinning Sync Services as being an “SOA” way of doing data synchronization because “it uses WCF” and ” supports n-tier”. But under the hood it’s all about DataSet-style deltas. See Steve Lasker’s first Q&A for the SOA spin, and the rest of his blog for more info.

I’m curious what your thoughts are on the subject. Do you see Sync Services as a valid service-oriented solution for occasionally-connected clients, or is it just another attempt by the ADO.NET team to keep us all hooked on DataSet-driven development?

Oran

Get it via the Dr. Dobb’s site here.

Or download directly here.

Additional References

• Occasionally Connected Systems Architecture
• Entity Framework: Disconnected Problems & Solutions
• Tasks, Messages, & Transactions–the holy trinity

Want more? Go to the “Ask Udi” archives.

The lowly GUID

One of the simpler patterns that, when viable, keeps basic things simple is that of Client-side Identifier Generation -  if client-side code is capable of creating entities and giving them an identifier which will not be changed at the time those entities are persisted. The simplest implementation of this pattern is to use a Globally Unique Identifier (GUID) to identify entities in the systems. By having client-side code creating objects whose identities will be correct even before saving them, we are saved the chore of refreshing all the objects the client created when it eventually gets connected to the server again and can save those objects.

One requirement that Roger mentions moves the architecture more towards an “N-Tier” style and further from the classic “Client-Server” style:

“Locally cached lookup (catalog or historical) data, which can be very large and thus must merge changes rather than require full-table refreshes”

Messaging & Pub/Sub

By employing the publish/subscribe pattern between clients and an application server, as well as durable subscriptions when clients get connected they get pushed only the updates since the last time they were connected. This pattern improves scalability in that it prevents the scenario of many client connecting and performing costly queries against the database at the same time. Another difference between this (messaging-based) pattern and the more common database-level synchronization is that it maintains the discrete units-of-work (pattern) that were performed. While this isn’t very important for updates being pushed from the server to the client, it is absolutely critical for changes the client wants to send the server. These are the relevant architectural characteristics:

·         “Automatic or semi-automatic push of cached data to the server (two-tier) or data service (n-tier) when connected to the network”

·         “A process to enable the client to resolve data update and deletion concurrency conflicts; alternatively, to notify the client of the action taken by a business rule”

·         “A process to enable the client to resolve insert and other conflicts with multiple rows of child tables, such as order or medication line items”

Be neither overly optimistic nor overly pessimistic

One of the greatest challenges of the business logic in these systems is the handling of concurrency conflicts – multiple users changing the same set of data independent of each other and now trying to “upload” those changes to the server. This is handled on the server-side by the techniques I detailed in “Realistic Concurrency”, but it is critical to have the client pass the Units-of-Work it performed to the server for that to work. By modelling these units-of-work as messages that the client sends to the server, buffering them when there’s no connectivity, we are able to employ these Connected-Systems patterns as well.

Caching

Another interesting question is how the data is cached on the client. Before we get into the solutions, I think that we need to divide this question into two parts. Given a connected system, what is the best way to structure data on the client side; and does the fact that the client is occasionally connected change anything? Ayende suggest separate data structures for the client and server. I strongly agree – regardless of connectivity. I suppose that this is one of the reasons that I don’t particularly like using datasets on the client – it is so strongly tied to persistence concerns that don’t worry me on the client.

A different model for the client

As for the second question – does connectivity change the client-side model? I have found that it does, but that it’s much more an issue of the kind of work that can be performed disconnected. If we choose to employ asynchronous messaging, we may need to handle cases where, due to message-design considerations, the client can get published child objects before parents. It is fairly straight-forward to handle this by simply storing both the parent ID and a reference to the parent object in child objects. In order to keep this data correct, it wise to update objects only by calling business-level methods on them rather then calling multiple setter-methods one after another.

One other issue is where these objects are stored on the client-side. In connected scenarios, this issue doesn’t exist until caching is involved. What I often use is a derivative of the Repository pattern. This repository enables searching for objects on the client-side, persists itself to some durable storage (like the file system) when the client application is shut down, and loads those objects from the storage on startup.

Wrapping up

While this isn’t an exhaustive treatment of all issues related to Occasionally Connected Systems Architecture, I hope that it serves as a good baseline for discussing techniques and technologies.

Related content

Podcast on Occasionally Connected Smart Clients
Podcast on passing data between layers in occasionally connected systems

“In fact, without a proper service oriented architecture and messages that are constructed based on tasks instead of CRUD operations, you will end up in the same boat as if you hadn’t implemented this approach.”

Let me be the first to agree.

In no way did I intend to discuss SOA related issues in my previous posts. Since I talk so much about SOA, and the use of the Bus Pattern and Asynchronous Messaging, I wanted to try to keep the discussion on OCC.

The main thing that I liked about Jesse’s response was that he agreed that my solution “solves some transaction issues” which is what I was aiming for. I’m going to be doing a separate series of posts on message design for SOA later, and would be happy to hear of anyone’s experiences in that area.

Andres Aguiar points out an enormous change to the behavior of data access in tiered architectures that is coming with the ADO.NET Orcas Entity Framework. He sums it up nicely, “Basically, there is no disconnected mode.” He then goes on to talk about the ramifications of this:

“This basically means that if you plan to build a multi-tier application with ADO.NET Orcas in the middle tier, you will need to hack your own change tracking mechanism in the client, send the whole changeset, and apply it in the middle tier. From this point of view, it’s a huge step backwards, as that’s something we already have with DataSets today.”

I do quite a bit of work on large-scale distributed systems, some that could even be called “multi-tier”, but I really don’t like that term. I’ve got to tell you, I’m not that worried. I could say that it’s because I’m using an object/relational mapping tool, but then again, the Entity Framework is supposed to be that too. It also might be because I don’t use DataSets that much today anyway, but that begs Andres’ question – did I develop my own change-tracking mechanism on the client?

The answer lies in the second part of Andres’ question, I don’t “send the whole changeset, and apply it in the middle tier”. I have an entire podcast up on why I don’t think that that’s a good idea, specifically around Web Services and SOA, but it’s broadly applicable. However, the question remains, what do we do about the client side?

Just over a year ago, me and Arjen Poutsma (of Spring fame) had a discussion about this exact topic at the Software Architecture Workshop in beautiful Cortina, Italy. Here’s the important part:

I think it started by me saying that services should not expose CRUD style operations. Arjen countered by mentioning that most user interfaces in line-of-business applications exposed the same model to the user by having them fill out data in grids. My retort to that was that while humans can get used to almost anything as long as it’s consistent, that doesn’t mean that it is a good solution. In the systems that I work on there is usually an HCI (human-computer interaction) person on the project who designs the UI, mostly around the tasks they perform. These tasks often corresponded very well to the coarse-grained messages we employed in terms of SOA. We finally agreed that the successor of SOA would be TOA (Task-Oriented Architecture) in its aggregation of client-side aspects to the already server-centric principles of SOA.

As a concrete example, instead of updating customer data in a grid showing everything laid out flat, users would activate tasks like “Change address”, or “Update marital status”. The form shown would contain only the relevant data to be updated – no need to perform “change tracking”, it is perfectly clear what data needs to be sent. Furthermore, the intent behind the changed data would be sent as well – the server wouldn’t apply the data generically “in the middle tier”. Rather, the current customer data would be retrieved as a part of the customer object, and the appropriate method called upon it (ChangeAddress, or UpdateMaritalStatus) with the relevant data.

I actually view this change in the Entity Framework as a step forward, it’s like taking off the training wheels. I’m positive that most developers will be able to make the change and move to these more advanced architectures.