There’s this belief that if we just reused more code, everything would be better.

Some even go so far as saying that the whole point of object-orientation was reuse – it wasn’t, encapsulation was the big thing. After that component-orientation was the thing that was supposed to make reuse happen. Apparently that didn’t pan out so well either because here we are now pinning our reuseful hopes on service-orientation.

Entire books of patterns have been written on how to achieve reuse with the orientation of the day.
Services have been classified every which way in trying to achieve this, from entity services and activity services, through process services and orchestration services. Composing services has been touted as the key to reusing, and creating reusable services.

I might as well let you in on the dirty-little secret:

Reuse is a fallacy

Before running too far ahead, let’s go back to what the actual goal of reuse was: getting done faster.

That’s it.

It’s a fine goal to have.

And here’s how reuse fits in to the picture:

If we were to write all the code of a system, we’d write a certain amount of code.
If we could reuse some code from somewhere else that was written before, we could write less code.
The more code we can reuse, the less code we write.
The less code we write, the sooner we’ll be done!

However, the above logical progression is based on another couple of fallacies:

Fallacy: All code takes the same amount of time to write

Fallacy: Writing code is the primary activity in getting a system done

Anyone who’s actually written some code that’s gone into production knows this.

There’s the time it takes us to understand what the system should do.
Multiply that by the time it takes the users to understand what the system should do
Then there’s the integrating that code with all the other code, databases, configuration, web services, etc.
Debugging. Deploying. Debugging. Rebugging. Meetings. Etc.

Writing code is actually the least of our worries.
We actually spend less time writing code than…

Rebugging code

Also known as bug regressions.

This is where we fix one piece of code, and in the process break another piece of code.
It’s not like we do it on purpose. It’s all those dependencies between the various bits of code.
The more dependencies there are, the more likely something’s gonna break.
Especially when we have all sorts of hidden dependencies,
like when other code uses stuff we put in the database without asking us what it means,
or, heaven forbid, changing it without telling us.

These debugging/rebugging cycles can make stabilizing a system take a long time.

So, how does reuse help/hinder with that?

Here’s how:

Dependencies multiply by reuse

It’s to be expected. If you wrote the code all in one place, there are no dependencies. By reusing code, you’ve created a dependency. The more you reuse, the more dependencies you have. The more dependencies, the more rebugging.

Of course, we need to keep in mind the difference between…

Reuse & Use

Your code uses the runtime API (JDK, .NET BCL, etc).
Likewise other frameworks like (N)Hibernate, Spring, WCF, etc.

Reuse happens when you extend and override existing behaviors within other code.
This is most often done by inheritance in OO languages.

Interestingly enough, by the above generally accepted definition, most web services “reuse” is actually really use.

Let’s take a look at the characteristics of the code we’re using and reusing to see where we get the greatest value:

The value of (re)use

If we were to (re)use a piece of code in only one part of our system, it would be safe to say that we would get less value than if we could (re)use it in more places. For example, we could say that for many web applications, the web framework we use provides more value than a given encryption algorithm that we may use in only a few places.

So, what characterizes the code we use in many places?

Well, it’s very generic.

Actually, the more generic a piece of code, the less likely it is that we’ll be changing something in it when fixing a bug in the system.

That’s important.

However, when looking at the kind of code we reuse, and the reasons around it, we tend to see very non-generic code – something that deals with the domain-specific behaviors of the system. Thus, the likelihood of a bug fix needing to touch that code is higher than in the generic/use-not-reuse case, often much higher.

How it all fits together

Goal: Getting done faster
Via: Spending less time debugging/rebugging/stabilizing
Via: Having less dependencies reasonably requiring a bug fix to touch the dependent side
Via: Not reusing non-generic code

This doesn’t mean you shouldn’t use generic code / frameworks where applicable – absolutely, you should.
Just watch the number of kind of dependencies you introduce.

Back to services

So, if we follow the above advice with services, we wouldn’t want domain specific services reusing each other.
If we could get away with it, we probably wouldn’t even want them using each other either.

As use and reuse go down, we can see that service autonomy goes up. And vice-versa.
Luckily, we have service interaction mechanisms from Event-Driven Architecture that enable use without breaking autonomy.
Autonomy is actually very similar to the principle of encapsulation that drove object-orientation in the first place.
Interesting, isn’t it?

In summary

We all want to get done faster.

Way back when, someone told us reuse was the way to do that.

They were wrong.

Reuse may make sense in the most tightly coupled pieces of code you have, but not very much anywhere else.

When designing services in your SOA, stay away from reuse, and minimize use (with EDA patterns).

The next time someone pulls the “reuse excuse”, you’ll be ready.

Further Reading

• Additional logic required for service autonomy
• Self-contained events & SOA
• Autonomous Services and Enterprise Entity Aggregation [MS Architecture Journal]
• Does SOA mean the end of OO? [Podcast]

Value Objects Don’t Count

While the canonical example presented is Customer -> Address, and has a good treatment here for nHibernate, it isn’t architecturally representative.

Addresses are value objects. What this means is that if we have to instance of the Address class, and they both have the same business data, they are semantically equivalent. Customers, on the other had, are not value objects – they’re entities. If we have two customers with the same business data (both of them called Bob Smith), that does not mean they are semantically equivalent – they are not the same person.

All Entities

Therefore, for our purposes here we’ll use something different. Say we have an entity called Job which is something that a company wants to hire for. It has a title, description, skill level, and a bunch of other data. Say we also have another entity called Job Board which is where the company posts jobs so that applicants can see them, like Monster.com. A job board has a name, description, web site, referral fee, and a bunch of other data.

A job can be posted to multiple job boards. And a job board can have multiple jobs posted. A regular many to many relationship. At this point, we’re not even going to complicate the association.

This is simply represented in the DB with an association table containing two columns for each of the entity tables’ ids.

In the domain model, developers can also represent this with the Job class containing a list of JobBoard instances, and the JobBoard class containing a list of jobs.

It’s intuitive. Simple. Easy to implement. And wrong.

In order to make intelligent DDD choices, we’re going to first take what may seem to be a tangential course, but I assure you that your aggregate roots depend on it.

Moving forward with our example

Let’s say the user picks a job, and then ticks off the job boards where they want the job posted, and clicks submit.

For simplicity’s sake, at this point, let’s ignore the communication with the actual job sites, assuming that if we can get the association into the DB, magic will happen later causing the job to appear on all the sites.

Our well-intentioned developer takes the job ID, and all the job board IDs, opens a transaction, gets the job object, gets the job board objects, adds all the job board objects to the job, and commits, as follows:

   1:          public void PostJobToBoards(Guid jobId, params Guid[] boardIds)

   2:          {

   3:              using (ISession s = this.SessionFactory.OpenSession())

   4:              using (ITransaction tx = s.BeginTransaction())

   5:              {

   6:                  var job = s.Get<Job>(jobId);

   7:                  var boards = new List<JobBoard>();

   8:   

   9:                  foreach(Guid id in boardIds)

  10:                      boards.Add(s.Get<JobBoard>(id));

  11:   

  12:                  job.PostTo(boards);

  13:   

  14:                  tx.Commit();

  15:              }

  16:          }

In this code, Job is our aggregate root. You can see that is the case since Job is the entry point that the service layer code uses to interact with the domain model. Soon we’ll see why this is wrong.

** Notice that in this service layer code, our well-intentioned developer is following the rule that while you can get as many objects as you like, you are only allowed one method call on one domain object. The code called in line 12 is what you’d pretty much expect:

   1:          public void PostTo(IList<JobBoard> boards)

   2:          {

   3:              foreach(JobBoard jb in boards)

   4:              {

   5:                  this.JobBoards.Add(jb);

   6:                  jb.Jobs.Add(this);

   7:              }

   8:          }

Only that as we were committing, someone deleted one of the job boards just then. Or that someone updated the job board causing a concurrency conflict. Or anything that would cause one single association to not be created.

That would cause the whole transaction to fail and all changes to roll back.

Rightly so, thinks our well-intentioned developer.

But users don’t think like well-intentioned developers.

Partial Failures

If I were to go to the grocery store with the list my wife gave me, finding that they’re out of hazelnuts (the last item on the list), would NOT buy all the other groceries and go home empty handed, what do you think would happen?

Right. That’s how users look at us developers. Before running off and writing a bunch of code, we need to understand the business semantics of users actions, including asking about partial failures.

The list isn’t a unit of work that needs to succeed or rollback atomically. It’s actually many units of work. I mean, I wouldn’t want my wife to send me to the store 10 times to buy 10 items, so the list is really just a kind of user shortcut. Therefore, in the job board scenario, each job to job board connection is its own transaction.

This is more common than you might think.

Once you go looking for cases where the domain is forgiving of partial failures, you may start seeing more and more of them.

Aggregate Roots

In the original transaction where we tried to connect many job boards to a single job, we saw that the single job is the aggregate root. However, once we have multiple transactions, each connecting one job and one job board, the job board is just as likely an aggregate root as the job.

We can do   jobBoard.Post(job);    or     job.PostTo(jobBoard);

But we need just a bit more analysis to come to the right decision.

While we could just leave the bi-directional/circular dependency between them, it would be preferable if we could make it uni-directional instead. To do that, we need to understand their relationship:

If there was no such thing as “job”, would there be meaning to “job board” ? Probably not.

If there was no such thing as “job board”, would there be meaning to “job” ? Probably. Yes. Our company can handle the hiring process of a job regardless of whether the candidate came in through Monster.com or not.

From this we understand that the uni-directional relationship can be modelled as one-to-many from job board to job. The Job class would no longer have a collection of Job Board objects. In fact, it could even be in an assembly separate from Job Board and not reference Job Board in any way. Job Board, on the other hand, would still have a collection of Job objects.

Going back to the code above we see that the right choice is   jobBoard.Post(job);   

Job Board is the aggregate root in this case. Also, the many-to-many mapping has now dissolved, leaving behind a single one-to-many mapping.

Let that sink in a second.

But Wait…

While the GUI showing which jobs are posted on a given job board are well served by the above decision (simply traversing the object graph from Job Board to its collection of Jobs), that’s not the whole story. Another GUI needs to show administrative users which Job Boards a given Job has been posted to. Since we no longer have the domain-level connection, we can’t traverse myJob.JobBoards.

Our only option is to perform a query. That’s not so bad, but not as pretty as object traversal.

The real benefit is in chopping apart the Gordian M-to-N mapping knot and getting a cleaner, more well factored domain model.

That gives us much greater leverage for bigger, system-level decomposition.

We’re now all set to move up to a pub/sub solution between these aggregate roots, effectively upgrading them to Bounded Contexts. From there, we can move to full-blown internet-scale caching with REST for extra scalability on showing a job board with all its jobs.

In Closing

We often look at many-to-many relationships just like any other relationship. And from a purely technical perspective, we’re not wrong. However, the business reality around these relationships is often very different – forgiving of partial failures, to the point of actually requiring them.

Since the business folks who provide us with requirements rarely think of failure scenarios, they don’t specify that “between these two entities here, I don’t want transactional atomicity” (rolling our technical eyes – the idiots [sarcasm, just to make sure you don't misread me]).

Yet, if we were to spell out what the system will do under failure conditions when transactionally atomic, those same business folks will be rolling our eyes back to us.

What I’ve found surprises some DDD practitioners is how critical this issue really is to arriving at the correct aggregate roots and bounded contexts.

It’s also simple, and practical, so you won’t be offending the YAGNI police.

Related Content

From CRUD to Domain-Driven Fluency

[Podcast] Domain Models, SOA, and The Single Version of the Truth

Also, “passing reference data in a message makes the message-consuming systems dependent on the knowledge and availability of actual persistent data that is stored “somewhere”. This data must separately be accessed for the sake of understanding the event that is represented by the message.”

The discussion of self-contained events can be compared to integration databases vs application databases.

Centralized Integration – Pros & Cons

If everything in a system can access a central datastore, it is enough for one party to publish an event containing only the ID of an entity that that party previously entered/updated. Upon receiving that event, a subscriber would go to the central datastore and get the fields its interested in for that ID. The advantage of this approach is that the minimal amount of data necessary crosses the network, as subscribers only retrieve the fields that interest them. Martin Fowler describes the disadvantages as:

“An integration database needs a schema that takes all its client applications into account. The resulting schema is either more general, more complex or both. The database usually is controlled by a separate group to the applications and database changes are more complex because they have to be negotiated between the database group and the various applications.”

This is far from being aligned with the principle of autonomy so important to SOA. In that respect, the architectural principle of self-contained messages points us away from those problems and towards more autonomous services.

However, once we have these autonomous business services in place, we may find that we don’t need 100% fully self-contained messages anymore.

A Real-World Example

Let’s say we have 3 business services, Sales, Fulfillment, and Billing.

Sales publishes an OrderAccepted event when it accepts an order. That event contains all the order information.

Both Fulfillment and Billing are subscribed to this event, and thus receive it.

Fulfillment does not ship products to the customer until the customer has been billed, so it just stores the order information internally, and is done.

Billing starts the process of billing the customer for their order, possibly joining several orders into a single bill. After completing this process, it publishes a CustomerBilled event containing all billing information, as well as the IDs of the orders in that bill. It does not put all the order information in that event, as it is not the authoritative owner of that data.

When Fulfillment receives the CustomerBilled event, it uses the IDs of the orders contained in the event to find the order information it previously stored internally. It does not need to call the Sales service for this information or contact some central Master Data Management system. It uses the data it has, and goes about fulfilling the orders and shipping the products to the customer, finally publishing its own OrderShipped event.

Notice, as well, that in the original OrderAccepted event there were the IDs of products the customer ordered. These product IDs originated from another service, Merchandising, responsible for the product catalog. The same thing can be said for the customer ID originating from another service – Customer Care.

The Issue of Time

One could argue that since subscribers use previously cached data when processing new events, that data might not be up to date. Also, we may have race conditions between our services. In the above example, if Billing was extremely fast and more highly available than Fulfillment. Billing could have received the OrderAccepted event, processed it, and published the CustomerBilled event before Fulfillment had received the OrderAccepted event. In short, the CustomerBilled and OrderAccepted messages could be out of order in Fulfillment’s queue.

What would Fulfillment do when trying to process the CustomerBilled message when it doesn’t have the order information?

Well, it knows that the world is parallel and non-sequential, so it does NOT return/log an error, but rather puts that message in the back of the queue to be processed again later (or maybe in some other temporary holding area). This enables the OrderAccepted message to be processed before the CustomerBilled message is retried. When the retry occurs, well, everything’s OK – it’s worked itself out over time.

In the case where we retry again and again and things don’t work themselves out (maybe the OrderAccepted event was lost), we move that message off to a different queue for something else to resolve the conflict (maybe a person, maybe software). If/when the conflict is resolved (got the Sales system / messaging system to replay the OrderAccepted event), the conflict resolver returns the CustomerBilled message to the queue, and now everything works just fine.

As all of this is occurring, the only thing that’s visible to external parties is that it happens to be taking longer than usual for the OrderShipped event to be published. In other words, time is the only difference.

Summary

The problem of non-self-contained events is mitigated first and foremost by business services in SOA, and the apparent issue of time-synchronization by business logic inside these services.

Don’t be afraid to put IDs in your messages and events.

Do be afraid of using those IDs to access datastores shared by multiple “services”.

Using IDs to correlated current events to data from previous events is not only OK, it’s to be expected.

The architectural principle of fully self-contained messages steers us away from the problems of Integration Databases and towards Application Databases, autonomous services, and a better SOA implementation. From there, following the principle of autonomy from a business perspective, will lead us to services not publishing data in their messages that is owned by other services, taking us the next step of our journey to SOA.

Related Content

[Podcast] Message Ordering – Is it cost effective?

Don’t EDA between existing systems

[Podcast] Handling dependencies between subscribers in SOA

Check it out.

Some of this stuff is a bit counter-intuitive (and not readily supported by the tools available in Visual Studio) so please, do feel free to ask questions (in the comments below).

One of the topics not covered by Prism is occasional connectivity, and I would like to say a word or two about that. First of all, if you’re building a standalone client (one that doesn’t communicate with anything), then there’s a good chance that Prism isn’t for you, although you could be composing other standalone client modules. So, if your client isn’t communicating with anything, well, then this post probably won’t interest you that much. Let’s start with…

Physics

Networks fail. Period.

This means that your client machine will not always be connected to other servers.

Also, servers fail – critical Windows patches and just regular power outages.

Ergo, your “smart” client will be occasionally connected, whether you planned for it or not.

And please don’t take this post as a “dumping on Prism” post – it isn’t intended that way. Rather, it is about how you should think about designing modules in Prism, and why.

Modules and Connectivity

Consider the case where we have two modules being composed in a single client. Each module communicates with a different server. Let’s call these modules Ma and Mb, and the servers Sa and Sb respectively. Now, let’s discuss what occurs given that the modules weren’t designed with occasional connectivity in mind.

User clicks something in Mb which requires communication.

Mb tries to call Sb, say, over HTTP, using a regular web service invocation.

The calling thread, in this case, the one used for user interaction, is blocked waiting for a response from Sb.

Sometime in this call, Sb fails, connectivity goes down, whatever.

30 seconds after the call, the HTTP connection times out.

If something important were happening in Ma at the same time, the user couldn’t even see it, let alone do anything about it since the user interaction thread is stuck. This is a serious concern for the financial services domain, but in many others as well.

You mean there’s more?

I can go on, but I think that that’s enough to paint the picture that if you are building a smart client, there are a lot more things to think about than just learning Prism. That’s my main concern after witnessing what happened around the CAB. Given the learning curve around these frameworks many developers don’t seek to deepen their understanding beyond just becoming proficient with them. This isn’t just centered on the developers, evangelists in Microsoft tend to paint the picture this way:

Once you understand X (CAB, Prism, BizTalk, whatever), all your problems are solved.

That’s not to say there aren’t good things in those technologies, but that’s just it, they’re just tools. Silver hammers and “laser” guided saws do not a master carpenter make. There’s actually a pretty good chance the regular guy will saw their arm off.

Help

I do hope more “instruction manuals” will be coming out of Microsoft on these topics. That’s not to say there aren’t any. Specifically on the topic of occasional connectivity, there is Chapter 4 of the Smart Client Architecture & Design Guide. Unfortunately, it doesn’t say anything about how that connects with the MVC/MVP being used client side (the bits affected by Prism). Chapter 6 of the same guide deals with the client-side threading, but doesn’t address issues like:

• Which model object instance are views bound to.
• Do other threads have access to that object at the same time.
• Which controller/presenter is responsible for giving that object to the view.
• Do they need to clone it.
• How deep should the clone be.
• How do various controllers/presenters (which may be showing the same object in different views at the same time) communicate changes to their various independent clones.

I haven’t yet documented all the patterns that answer these questions, but until I do (or Microsoft does), let me offer these few resources which I’ve put out over the years:

• Occasionally Connected Systems Architecture: The Client
• Occasionally Connected Systems Architecture: Concurrency
• Threading issues when smart clients reconnect
• [Podcast] Occasionally Connected Smart Clients and ADO.NET Sync Services

There’s also some more links under the Smart Client link of my “First time here?” page.

Also, please join me in asking Microsoft for an update to these guides – comments below or your own blog posts would be great.

Download

Download via the Dr. Dobb’s site

Or download directly here.

Additional References

• Podcast: “REST vs Web Services Debate”
• REST Definition
• Roy Fielding’s Thesis: “Architectural Styles and the Design ofNetwork-based Software Architectures”
• ETag Definition

Want more?

Check out the “Ask Udi” archives.

Got a question?

Send Udi your question to answer on the show.

Symon asks:

Hi Udi,

I don’t know if this is really an NServiceBus question or a more general architecture question, so if you want to treat it as an Ask Udi question feel free.   My question is about handling message prioritization between services in an SOA architecture using an ESB-  in this case NServiceBus.

We’ve been running along nicely getting the hang of using the NServiceBus but have just hit a point in our application where it’s become clear that we’re going to need to prioritize messages between services according to an arbitrary set of rules.  To that end we’ve created a “controller” that determines a message’s priority according to a set of rules and that controller should forward the message to other services for handling according to that priority.  Where we’re stuck is figuring out the best way to handle the prioritization of the forwarded message.

We touched on the idea of having several endpoints representing low, medium or high priority and shunting the message to the appropriate endpoint, however since the receiving service consists of multiple workers located on other machines we need to have a dispatcher for each of these endpoints.  Since we’re using MSMQ this seems to be causing an explosion of queues and the solution seems to require a lot of overhead in terms of set up, but it *does* look like it might be flexible as long as we come up with a smart dispatcher.

Is this an “ideal” way to handle message prioritization?  I know there is a notion of priority in MSMQ messages, but this isn’t exposed in NServiceBus as far as I can tell.

Any suggestions?

Cheers,

Symon

Download

Download via the Dr. Dobb’s site

Or download directly here.

Additional References

• Podcast: “Shared Subscriptions between Autonomous Components”
• Podcast: “Space-Based Architecture for the Web”
• Info: “Master-Worker Pattern”

Want more?

Check out the “Ask Udi” archives.

Got a question?

Send Udi your question to answer on the show.

Our long time listener Bill asks:

Hi Udi,

I have a question around processing of messages in proper order. When leveraging multiple threads to process messages in a message queue, it is possible for the second message in the queue to get processed before the first – especially if the first message is considerably larger than the second. I have taken a lot of care to make sure that messages are sent in the correct order, only to find that the receiving system can process them out of order anyway.

Consider a Policy Created notification, which must come before a Policy Approved notification. If both messages are sitting in the queue when the receiving service starts up, the approval message can be processed before the creation message. How can I make sure that message ordering is respected by the receiving system? I am using WCF/MSMQ as the underlying transport by the way. The only way I have found so far is to limit the receiving service to a single thread, which is by no means desirable.

Best Regards,

Bill

Download

Download via the Dr. Dobb’s site

Or download directly here.

Additional References

• Blog post: In-Order Messaging a Myth?
• Blog post: Handling Messages out of Order

Want more?

Check out the “Ask Udi” archives.

Got a question?

Send Udi your question to answer on the show.

After blogging since 2003, with over 500 posts I’ve pulled together the “best of the best” for you in the top 3 areas in which my clients use my expertise. Click to skip the introductions.
[ SOA - Domain Models - Smart Clients ]

Service-Oriented Architecture

Before you roll your eyes at me, believe me I do know that SOA has been hyped to death and that you’re probably sick of it. Me too. I’m interested in actually implementing SOA, not just talking about it – it’s actually what I do for a living; from high-level architecture, to detailed design, to choosing frameworks and technologies, and actually building the thing.

What I’ve got online includes podcasts on SOA, webcasts, books I’ve contributed to, full-lenth articles, and open-source frameworks I’ve written to help developers do the right thing by default. So, if you’d like to hear more about the practical aspects of doing SOA, you can jump to the relevant part of this page by clicking here.

Domain Models

You won’t find much of the more theoretical aspects of the domain model pattern or other patterns of application architecture – rather you’ll find presentations I’ve given on the subject, samples, and even an extension to NHibernate. All of these are derived from the work I’ve done around using these patterns – not the least of which is actually making them performant.

Also on the topic of Domain Models and Data Access, you’ll be able to find information on writing highly concurrent systems. This includes an object-oriented development style for service layers that yeild the best multi-user throughput. Jump to the Domain Model section here.

Smart / Rich / Occasionally Connected Client

I know that Microsoft has been pretty active here, putting out the Composite User Interface Application Block (CAB) and the Smart Client Software Factory (SCSF). The only thing is that those are both complex and not thread-safe – I know, I’ve been burned.

So, here you can find the set of patterns, technologies, and frameworks that make for a thread-safe, multi-threaded client that a team can develop in parallel. Jump to the Smart Client section here.

Doing SOA

If you’re tasked with architecting, designing, or implement a Service-Oriented Architecture then the resources below should help you get to the bottom line of all the theory and get you going in terms of actual development.

Podcasts:

• Does SOA mean the end of OO?
• Migrating from N-Tier to SOA
• Does an SOA require an ESB?

If you’d like to more of these kinds of podcasts, check out the “Ask Udi” podcast.

I’ve also be interviewed on SOA on several other podcasts:

• .NET Rocks interview on SOA
• Channel 9’s Ron Jacobs interview on SOA and Workflow

Webcasts:

• SOA Distilled for UML China [PDF] [Audio - 90MB via EMule]
• Why you can’t do SOA without Message [TechEd USA 2005]

Books:

• Applying Domain-Driven Design and Patterns [Chapter] An introduction to SOA
• SOA Expertise – Methods, Concept, & Practice [Article] The ESB and Your SOA—A Great Team

Full-length Articles:

• Autonomous Services and Enterprise Entity Aggregation [MS Architecture Journal]
• SOA Support, Therapy, and Treatment [IBM DevX Portal]

Open Source Frameworks:

• nServiceBus – communications framework for making enterprise .NET systems easier to build

Using Domain Models

If you’re looking for how to implement business logic and data access in your application, then you’ve come to the right place. Here you’ll find the patterns, design, and code that will make your business logic cohesive, your data access simpler, and keep everything performing at its peak.

Introductory info:

• What is a domain model?
• Where should (and more importantly, shouldn’t) you use a domain model
• Don’t tie your architecture to a specific object/relational mapping product [Developer.com]

Make it perform:

• Fetching Strategy Design – UML diagrams describing performance enhancing patterns
• Samples + NHibernate modifications for Fetching Strategies
• Presentation on High-Performance, Persistent Domain Models [TechEd Europe 2007]

Concurrency for multi-user throughput:

• Concurrency for Occasionally Connected Systems [Developer.com]
• Podcast on DataSets and Web Services [ Ask Udi podcast ]
• Realistic Concurrency (difference between optimistic and pessimistic)

Multi-core efficient Smart Clients

If you’re tasked with building a desktop application that communicates with a server, needs to be able to work offline, makes use of multiple cores for a better user experience, and can be developed without having to remember 5 books worth of theory, then this is for you.

Patterns:

• Sample implementation of Model-View-Presenter pattern
• Unit-testing still critical to project success even when using the relevant patterns
• Generic validation using the service locator pattern

Occasional Connectivity:

• Podcast on occasionally connected smart clients (and why ADO.NET Sync Services doesn’t help)
• Passing data from the UI to SOA Services [Podcast]
• Threading issues when smart clients reconnect [Developer.com article]

Thread-Safety = Just-plain Safe

• Explanation of threading concerns when using dependency injection and asynchronous messaging
• Followup on dependency injection, threading capabilities
• Thread-safe, Asynchronous Smart Clients [podcast]
• Prototype of Aspect-Oriented Programming to make views thread-safe by default

Ketan asks:

Hello Udi,

I am Ketan, working as Analyst Programmer in India. I want some information regarding SOA Architecture. I have visited several blogs and sites and I do have enough knowledge regarding technical terms of SOA. I have worked a lot in 3-tier mechanism a lot and now want to switch on SOA Architecture in Visual Studio 2005.

I read your blog post How to migrate to SOA and liked the contents and am interested in doing it. Can you please be more precise about how to migrate from 3-tier to SOA? Actually, I have read whole content of above link and you have explained enough. But, still I want you to keep me out from dark. You have described it functionally, but I want some technical description of this process. Please help me in this.

If you can provide me some example/application (in which SOA Architecture has been implemented), then it will be very helpful to me.

Thanks in advance. Waiting for your favorable reply.

Ketan

Download

Download via the Dr. Dobb’s site

Or download directly here.

Additional References

• Blog post on migrating from layers to services
• Podcast on “Does SOA mean the end of OO”?

Want more?

Check out the “Ask Udi” archives.

Got a question?

Send Udi your question to answer on the show.

Jarrod asks:

Hi Udi,

I have some questions around versioning and SOA:

Over the years I’ve been fairly adamant about the practice of versioning interfaces should a method change be needed.

Hence, IDog and IDog2

Generally I would also say that a method addition would need a new interface as much as a change to an existing method or property (sometimes requiring completely deprecating the old interface).

With SOA being used more and contract first designs coming down the pipe…an addition of a method or operation to a contract in SOA does not mean its a breaking change.

So, do you continue to create a new interface and version the name within an SOA environment? Even if its just one method that is being added?

A service contract, in general…with just a single method in my opinion is a bad design. Sure, there are exceptions to that but I can just see changes made to the service contracts over the course of a few years with many 1-2 method interfaces.

Day 1 : 12 methods on interface IDog

6 months later : 1 new method, create IDog2 which implements IDog

15 months later : 2 new methods, create IDog3 which implements IDog2

Some can argue that if you’re having to add methods like this, the original design was flawed. That may be…or you could just be in a volatile business environment in which SOA must adapt.

So my main questions are:

1) Do you think its the best practice to continue to version interfaces for non-breaking method additions in an SOA environment in which you also have control of the consumers (internal consumers)?

2) Do you think its the best practice to continue to version interfaces for non-breaking method additions in an SOA environment in which you do not have control of the consumers (external consumers)?

3) If the answer to the above is yes, is a single method on a new interface acceptable?

4) If you know of a problem in an existing interface on a method and have control of the consumers – do you “fix” the method and redeploy to the consumers? Or do you deprecate the entire interface and begin anew?

I’m interested in seeing your responses

Jarrod

Download

Download via the Dr. Dobb’s site

Or download directly here

Additional References

• Podcast on Autonomous Services and Pub/Sub
• Blog post on “The Perils of WSDL first”

Want more?

Check out the “Ask Udi” archives.

Got a question?

Send Udi your question to answer on the show.

Suresh asks:

Hi Udi,

I’ve been reading about the coming “durable services” that will be available with the next version of WCF. I also have been listening to your podcasts and reading your blog posts about NServiceBus where you talk about long-running workflows. It sounds like both of these technologies are trying to solve the same problem.

Do durable services do away with long-running workflow? If so, does that mean we don’t need Workflow Foundation either? If not, what is the connection between them.

If you could shed some light on the matter that would be great.

Suresh

Download

Download via the Dr. Dobb’s site

Or download directly here

Additional References

• Blog post on Workflow Services in Orcas
• Blog post on Orcas Durable Services
• Blog post on .NET Framework 3.5 Beta1 is Live on the Web

Want more?

Check out the “Ask Udi” archives.

Got a question?

Send Udi your question to answer on the show.

Neil asks:

Hi Udi,

We’re building a smart client application that uses WCF for full-duplex communications with our server. This is the asynchronous communication you talk about in your podcast. The smart-client is based on the MVC pattern, where model objects raise events when they’re changed so that the views can update themselves.

What’s started happening recently is that the smart-client has been freezing-up on us intermittently. We don’t know how to debug this and are wondering if its an architectural problem.

Any help you can give would be most appreciated.

Neil

Download

Download via the Dr. Dobb’s site

Or download directly here

Additional References

• Blog post on Occasionally Connected Systems Architecture
• Blog post on ObjectBuilder synchronization features needed for pub/sub-ing Smart Clients

Want more?

Check out the “Ask Udi” archives.

Got a question?

Send Udi your question to answer on the show.

Rishi asks:

Hi Udi,
First of all, thanks for all the posts and info you share, it is very insightful compared to loads n’ loads of marketing bull and vendor whitepapers. Ok, I have two questions for you.

1. I want to create a SOA-based LOB application/platform and I generally understand the ‘tenets of services’ and reasons for the same. However, as you may well know, there is a lot of interdependency between business constructs (such as a customer with an order, or a delivery with a product construct). Now, should we share these constructs within a tightly-coupled and domain-based “system” which exposes a number of services to the outside world and use the constructs directly inside the so-called “system” boundary. Or on the opposite spectrum have a number of autonomous and independent services that expose, own, and share certain business constructs or at least part of the data that represents them as a whole. For example, an order service will need to interact with a customer construct, which primarily/essentially is (or should be?) owned by a customer profile service – in this context, with clear, direct, and immediate inter-dependency what is preferable? Where do we draw the enclosing line?

2. In a line of business application, reporting and shaping data are a given necessity. Now, with autonomous and independent services which define business constructs in their contextual ways, how do we practically shape, combine, and filter data across various services? We certainly can’t do joins over multiple services in a practical way? And as some suggest, if we are to maintain duplicate or master data elsewhere, it just seems very messy to me – just consider having 3 versions of an order services, with 2 versions of support services, and ‘n’ number of other services to comb data from. Further, for reporting and also analytical purposes we really need to have business data structured in a well-defined manner, which for all purposes needs to be the “single version of the truth”. I can’t see granular services, with all its tenets, sitting nicely with such other requirements of the business – and yet I am not even asking them to be real-time. How do we meet such business requirements in your view with SOA?

Hope the above makes sense!

Cheers,
Rishi

PS: I use the word business construct purposefully, and it does not directly attribute to a well-defined programmable entity (in object-oriented way, that is).

Download via the Dr. Dobb’s site.

Or download directly here.

Additional References

• Ask Udi Podcast #000–Does SOA mean the end of OO?
• Ask Udi Podcast #009–Master Data Management and SOA
• Ask Udi Podcast #016–How does Extract, Transform, Load fit with SOA?

Want more?

Check out the “Ask Udi” archives.

Got a question?

Send Udi your question to answer on the show.

Our long-time listener Bill asks the following:

Hi Udi,

Thank you for your podcast clarifying the concept of autonomous components. It certainly cleared up a lot in terms of how you define an autonomous component and helped position the concept better in my mind.

I am considering building our Policy Administration service with 3 autonomous components as you previously suggested – one for each insurance product family. As you previously mentioned, these autonomous components could either share a database instance or have separate database instances, even though they may share schema (being a part of the same service).

What concerns me however is that there is some data that must be shared between these autonomous components. A perfect example is Region. If a new Region is defined somewhere in the enterprise, a RegionCreatedNotification is published onto the bus. If I have 3 autonomous components sharing the same database, is it best I just choose one to be the subscriber for this event? Or should I create a new separate queue to receive these notifications and put a message handler for it in a separate MessageHandlers assembly? Or should I subscribe all 3 autonomous components and have them each check if the Region has already been created in the database before inserting a new one?

Option (1) seems presumptuous because it assumes all 3 autonomous components are sharing the same database. Option (2) seems a bit strange because it is like creating a 4th autonomous component only to receive the RegionCreatedNotification messages. And Option (3) seems wasteful since there is redundant code and the system is doing extra unnecessary lookups. At the moment it looks like Option (2) is my best bet, but I’m very interested to get your take on it.

Another question is whether you would say that autonomous components should be hosted in different processes. Since I don’t need to run these autonomous components on separate servers at this stage, it would seem wasteful to create a service host for each component. I could always create separate service hosts at a later time if I needed to host one or more autonomous components on other servers.

And the final question I have is in terms of solution structure. I was contemplating the following structure:

ProductFamily1
———————-
Company.PolicyAdministration.ProductFamily1.Domain
Company.PolicyAdministration.ProductFamily1.MessageHandlers
Company.PolicyAdministration.ProductFamily1.Messages
Company.PolicyAdministration.ProductFamily1.Persistence
Company.PolicyAdministration.ProductFamily1.Persistence.Implementation
Company.PolicyAdministration.ProductFamily1.Etc…

ProductFamily2
———————-
Company.PolicyAdministration.ProductFamily2.Domain
Company.PolicyAdministration.ProductFamily2.MessageHandlers
Company.PolicyAdministration.ProductFamily2.Messages
Company.PolicyAdministration.ProductFamily2.Persistence
Company.PolicyAdministration.ProductFamily2.Persistence.Implementation
Company.PolicyAdministration.ProductFamily2.Etc…

ProductFamily3
———————-
Company.PolicyAdministration.ProductFamily3.Domain
Company.PolicyAdministration.ProductFamily3.MessageHandlers
Company.PolicyAdministration.ProductFamily3.Messages
Company.PolicyAdministration.ProductFamily3.Persistence
Company.PolicyAdministration.ProductFamily3.Persistence.Implementation
Company.PolicyAdministration.ProductFamily3.Etc…

Common
———————-
Company.PolicyAdministration.Common.Domain
Company.PolicyAdministration.Common.Persistence
Company.PolicyAdministration.Common.Messages
Company.PolicyAdministration.Common.ServiceHost
Company.PolicyAdministration.Common.Etc…

Do you have any thoughts on the above structure?

Once again thank you very much for your extremely valuable advice!

Best Regards,
Bill

Download via the Dr. Dobbs’ site.

Or download directly here.

Additional References:

• Ask Udi Podcast #017 – Using Autonomous Components for SLAs in SOA
• Ask Udi Podcast #020 – How to structure .NET solutions and components

Want More?

Check out the “Ask Udi” archives.

Got a question?

Send Udi your question and have him answer it on the show: podcast@UdiDahan.com.

ESB’s are a lot about messaging and therefore a better name might be “Enterprise Data Bus”. It’s the asynchronous messaging that needs such an infrastructure with persistency and mediation facilities. All the WS-* standards are about messaging as well, leveraging the message itself to tell the infrastructure how it has to be handled.

I think WS-* will make it possible to have the ESB evolve from a vendor-product to a concept implemented in the operating systems an network devices that understand WS-*. Then you can leave the prefix “Enterprise” and we will be ready for an univeral asynchronous data bus over the Internet (or any other network you like). This will help breaking the current “services centric” idea of SOA into a “messages centric” perspective.

What are your thoughts?

Download via the Dr. Dobb’s site

Or download directly here.

Additional References:

• Podcast on if you need an ESB for SOA
• Podcast on Enterprise Service Discovery
• Blog post on ESBs as a global dataspace
• Blog post about what the brave new world of WS-* brings us

Want more? Check out the “Ask Udi” archives.

Got a question? Have Udi answer it on the podcast – podcast@UdiDahan.com

Download via the Dr. Dobb’s site

Or download directly here.

Additional References:

• IASA home page
• IASA Skills Library
• How to start your own IASA chapter
• Join the IASA Training Program

Want more?

Check out the “Ask Udi” archives

Ask Udi a question for the podcast…

And the question that this answers is:

Hi Udi,

We’ve been having some discussions about how to implement our latest project using SOA and this is what we came up with:

Every activity is a method, which is broken into a class ( Think separation of concerns ), so we get high reusability.
On top of those classes we have decided to put a WCF layer, so you can expose the method as a web method. If an activity needs data, then it will access a entity service via WCF. To make the activities useful for the business we put them in a class which we have called processes ( a process layer). A process contains one or more activities and is able to call other processes and activities aren’t allowed to call processes or other activities. On top of each process we have decided to put a WCF so the UI can access them.

So it’s pretty close to what you wrote about in the Microsoft Architecture Journal except that we don’t have direct call to the entity services, we wrap it up in an activity before the call, which is wrapped in a WCF-host. Much like the definitions in Ontology and Taxonomy of Services in a Service-Oriented Architecture

I would love to hear your comments and thoughts about this architecture.

With thanks, Ingo

Download via the Dr. Dobbs’ site.

Or download directly here.

Additional References:

• Ontology and Taxonomy of Services in a Service-Oriented Architecture
• Udi’s Architecture Journal article on Autonomous Services
  
• Podcast on how to structure .NET solutions and components
• Blog post covering a discussion on Entity Services

Want More?

Check out the “Ask Udi” archives.

Got a question?

Send Udi your question and have him answer it on the show: podcast@UdiDahan.com.

Download via the Dr. Dobb’s site.

Or download directly here.

And here’s the question that gets us started:

Hello there,

We are developing an application in .NET Smart Client application which connects to a WCF service and then to the data access layer. The issue we have here is on the type of data which can be passed between the client side and web service.

1. Can it be a dataset (No!), custom entity or XML or anything else ?

Which is the best one ?

2. Ofcourse , a dataset can’t be used coz it can’t be used in non-.NET environment?

3. If we go for custom-entity or XML how do we ensure the changes happen to the data during client-side actions ? For example, binding the XML or custom entity to a data grid can be done. But if user makes any changes to the same, how do we identify ?

4. If we use custom entities / XM L, the same can not be passed to the Data Access Application Block which invokes the data objects.

Note : All the above are based on assumption without using a translator in between any of the layers.

Please provide a better solution if you have come across any.

Additional References:

• Podcast on using DataSets with Web Services
• Blog post on tracking changes to objects on the client side
• Looking for more of the “Ask Udi” podcast?

Download via the Dr. Dobb’s site

Or download directly here.

Additional References:

• OSGi
• MuleSource, the company supporting Mule
• Microsoft BizTalk Internet Services Bus
• ALT.NET, a community around the solution, not the tools

Download via the Dr. Dobb’s site.

Or download directly here.

Here’s the original question:

Hi Udi,

This is Kumar, i am new to .NET and related technologies. Actually i am designing an application with few a components like a Socket Server(a service) which received input from payment terminal for loyalty application, which should communicate with Web Services deployed in an IIS application server, then the web services should communicate with an Oracle Database.

What are the patterns and best practices to use for the communication between the socket server and application server, and how do I scale for multiple clients in socketserver, and what are the security things that i need to consider.

Should i need to use .NET Remoting instead Web Services for my application server layer, and which is best for my socket server?

Thank you for your support.

Kumar

Additional References

• MSDN WCF performance whitepaper
• Microsoft SQL Server Service Broker
• Threat modeling, the STRIDE approach
• Microsoft Patterns & Practices group; Threat modeling for web applications Ebook

Download via the Dr. Dobbs’ site.

Or download directly here.

And here’s this week’s question:

Hi Udi,

Thanks again for your continued assistance. I was very much interested by your advice to consolidate each of the services related to each product family into a single service, but as autonomous components.

From your description of autonomous components from a prior podcast, it seems that they are much the same as services – in that they communicate only via loosely coupled messaging, and can have their own databases. Would you say that the main difference between autonomous components is that different autonomous components within a service may in fact share business logic and databases? If so, it would seem that combining these services into a single service with 3 autonomous components would be a matter of definition, rather than an architectural shift. Any information you could provide to clarify this distinction would be fantastic.

Something else that’s been playing on my mind of late – is whether or not you would consider a topic as having to belong to a specific service. That is, would you say it is bad practice to have multiple services publish on a common topic? I suppose if we have multiple services publishing on a common topic, then they should be defined as autonomous components, belonging to a single larger service – in which case that common topic would belong to that new service.

As usual your advice is always extremely helpful. Please keep those podcasts coming!

Best Regards,
Bill

Additional References

• Podcast on Message Schemas between multiple Publishers and Subscribers
• Podcast on Business and Autonomous Components in SOA

Download via the Dr. Dobbs’ site.

Or download directly here.

And the original question was:

Udi:

I have a question regarding publishing events that relate to data changes. I found the article you wrote in the Arch Journal #8 very helpful. I think striving for autonomy is very important. The scenario I was thinking about is how can you ensure synchronization across subscribers of a particular event.

For example, System A publishes an event when customer information is updated. There are several systems that subscribe to this event. Two of the systems, System B and System C, need to be sync regarding customer information. System B uses operations from System C using the customer data. Using your example, System B has a process that runs for all “Preferred Customers”, and it uses processes on System C. However, System C may not have process the event to and may have a customer as preferred.

I have several thoughts, but would like to get your thoughts on this scenario. Are there any best practices or patterns?

Phil

Additional References:

• Autonomous Services and Enterprise Entity Aggregation (Article in Microsoft Architecture Journal)
• Podcast on Autonomous Services and Pub/Sub

You can find more episodes like this in the Ask Udi archives.

Hi Udi,

I just went through your Blog which talks about not creating entity services, but instead creating a business service. I understood why you would want it to be a business service, although I am still struggling to fully define what a business service is.

Currently what we have requested is to have a single service, extract, transform and load master data. From your explanations it seems like this is a business service and not just an entity one. What one of the sales guys from an SOA company is saying that this is very possible and would be easy to do.. What I wanted to get your perspective is if this is the right thing to do. What we requested is below:

1. To create a business service that will extract, transform and load data (One service for user, one for customer and one for product?)
2. This service will then be called by the different applications we have (3 Java applications, 1 .Net Application and 1 Siebel Application as well as a host of VB/Excel Applications)
3. The service will be calling on different backend sources for data from SAP systems, Access DBs, Excel Files, Web Pages & Oracle Systems

They say that creating this would produce a.) a reusable service and b.) cost savings. Although I am still apprehensive because it seems as you mentioned in your podcast, each system would have slightly different set of rules for the data entity (and btw you are right).. would this still matter? The thing we want to eliminate is the replication of extracting and loading, although each system would transform the data in its own specific way..

Appreciate any perspective that you might have.

Thanks,

Jayan

Get it via the Dr. Dobb’s site here.

Or download directly here.

Additional References

Podcast on Master Data Management and SOA
Podcast on Business and Autonomous Components in SOA