You can find it here.
The slides can be found here.

I also gave this talk at TechEd Barcelona and wanted to thank the attendee who posted this comment:

“You’ve done it again. Everytime I attend a session of yours I leave the room with new insights and inspiration on how to improve my software…”

You made my day.

First of all, I wanted to address the relationship between various topics related to scalability:

And on the connection between scalability and throughput:

The important message here is that the scalability of a system is a cost function that gives throughput as a function of recurring costs and one time costs – servers and other hardware, and the join of buy & build:

Did you write your own locking/transaction mechanism on top of an open source distributed cache or did you buy a license for a space-based technology?

Also, don’t forget that people need to administer all the servers that you have. Those people cost money (easily100K per year). Maybe, because you haven’t invested in management or monitoring tools you need one person for every two servers. This will influence the breakdown of up front costs and recurring costs. Also, the level of availability you require will impact this as well.

In my experience, architects don’t consider often enough the operations environment in their "scalability calculations".

What this means is that there’s no such thing as technically "not being able to scale".

Rather, that the cost (up front + recurring) of supporting higher throughput grows faster than the function of revenue per user/request/whatever.

Sometimes, the solution is just to find ways to make more money per customer.

For more technical solutions, take a look at the difference between capacity and scalability and how the competing consumer pattern helps scale out.

Scalability, it’s all about the money.

–

Oh, I almost forgot, I also had a great conversation with Carl and Richard about scaling web sites that’s now up on the .NET Rocks site. Enjoy.

Libor asks:

“Would you recommend using durable messaging for systems where there are similar requirements with respect to data reliability as you had – ie. not losing any messages? If so, then why didn’t the final version of your solution use it? If not, can you explain why?”

The answer is, as always, it depends, but here’s on what it depends:

When designing a system, we need to take a good, hard look at how we manage state, and what properties that state has. In a system of reasonable size we can expect various families of state with respect to their business value, data volatility, and fault-tolerance window. Each family needs to be treated differently. While durable messaging may be suitable for one, it may be overkill or underkill for another.

So, here’s what we’re going to be looking at:

1. Business Value
2. Data Volatility
3. Fault-Tolerance Window

Business Value

When talking about business value, I want to talk about what it means “not losing any messages”. The question is under what conditions will the messages not be lost, or rather, what are the threshold conditions where messages may start getting lost. If all our datacenters are nuked, we will lose data. It’s likely the business is OK with that (as much as can be expected under those circumstances). If a single server goes down, it’s likely the business would not be OK with losing messages containing financial data. However if a message requesting the health of a server were to get lost under those same conditions, that would probably be alright. In other words, what does that message represent in business terms.

Data Volatility

Data volatility also has an impact. Let’s say that we’re building a financial trading system. The time that it takes us to respond to an event (message) that the cost of a certain financial instrument has changed, and the message that we send requesting to buy that security is critical. Let’s say that has to be done in under 10ms. Now, some failure has occurred preventing our message from reaching its destination for 20ms. What should we do with that message? Should we keep it around, making sure it doesn’t get lost? Not in this domain. On the contrary, that message should be thrown away as its “business lifetime” has been exceeded. Furthermore, even during that original period of 10ms, the use of durable messaging may make it close to impossible to maintain our response times.

Fault-Tolerance Window

These two topics feed into the third and more architectural one – fault-tolerance window: what period of time do we require fault tolerance, and with respect to how many (and what kind of) faults? This will lead us into an analysis of to how many machines do we need to copy a message before we release the calling thread. We’d also look at in which datacenters those machines reside. This will also impact (or be impacted by) the kinds of links we have to these datacenters if we want to maintain response times. These numbers will need to change when the system identifies a disaster – degrading itself to a lower level of fault-tolerance after a hurricane knocks out a datacenter, and returning to normal once it comes back up.

Re-Evaluating Durable Messaging

Durable messaging may be used at various points in each part of the solution, but we need to look at message size, the rate those messages are being written to disk, how fast the disk is, how much available disk we have (so we don’t make things worse in the case of degraded service), etc. Companies like Amazon also take into account disk failure rates, replacement rates (disks aren’t replaced immediately you know), and many other factors when making these decisions

Summary

Our job as architects when designing the system is to find that cost-benefit balance for the various parts of the system according to these very applicative parameters. No, it’s not easy. No, cloud computing will not magically solve all of this for us. But, we are getting more technical tools to work with, operations staff is getting better at working with us in the design phase, and our thought processes more rigorous in dealing with the scary conditions of the real world.

To your question, Libor, as to why we didn’t eventually use durable messaging in our solution, the answer is that we solved the overall state management problem by setting up an applicative protocol with our partners which was resilient in the face of faults by using idempotent messages that could be resent as many times as necessary. You can read more about it here. This solution isn’t viable for other kinds of interactions but was just what we needed to get the job done.

Hope that helps.

Hi Udi, I think  I see the point about the domain-driven approach but I’m wondering what my messages will look like. If it’s this:

IAppointment InsertInterview(Guid recruiterId, Guid applicantId, Guid appointmentId); OR

IRecuiter UpdateRecuiter(IRecuiter recruiter); (passing in an operation flag attached to the IRecuiter object) OR

IRecuiter UpdateRecuiter(IRecuiter recruiter); (setting a state flag on the relevant object and have the business object check the flag and behave according the state change)

Hope I’m not way off

Sean

Well, Sean, first of all – messages don’t look like functions. They’re a lot more like structures – data transfer objects. In this case, you’d probably be looking at a ScheduleInterviewMessage that had the relevant fields. It would look something like this:

   1:  using System;

   2:  using NServiceBus;

   3:  using System.Xml.Serialization;

   4:   

   5:  namespace Messages

   6:  {

   7:      [Serializable]

   8:      [Recoverable]

   9:      [TimeToBeReceived("0:01:00.000")]

  10:      public class ScheduleInterviewMessage : IMessage

  11:      {

  12:          public Guid InterviewerId;

  13:          public Guid CandidateId;

  14:          public DateTime RequestedTime;

  15:   

  16:          [XmlAnyElement]

  17:          public object extra;

  18:      }

  19:  }

Before we go on, I want to explain what we see. The “recoverable” attribute is the way we indicate to the infrastructure that these messages should not be lost in case a server fails, there are network problems, etc. In essence, it does durable, store-and-forward messaging. This will create an environment in which, in the case of network problems, these messages will be written to disk. That’s a good thing, since once connectivity comes back or the server boots back up, the messages will still be around and can be sent.

Now these messages are fairly small, so even at a relatively high load, we shouldn’t be chewing through too much of our expensive, small, high performance local disks. However, if these messages were bigger, we may fill up our disks before connectivity comes back, and we all know what happens to Windows boxes when there’s no room on the file system left:

In order to prevent our system from Denial-of-Servicing itself we need to make those messages clean themselves up. That’s what the “TimeToBeReceived” attribute is for. The amount of time that if a message had not yet been received by the other side that it will be deleted. This could be that the message even made it to the other machine, but the process handling those messages was down. You wouldn’t want to be filling the other side’s disk either causing them to crash, would you? This protects both parties.

The way to figure out how long to set is by looking at the smallest amount of durable storage you have available at your nodes, divide that by the size of the average message, and then again by the rate you need to process messages – and leave yourself at least 100% spare.

In other words, to build a robust system you not only will need to deal with lost messages, but you will be actively throwing messages away.

Finally, that last “XmlAnyElement” attribute is there for versioning. As we version our system and schema, we’ll be adding fields to the message. However, an old client may be talking to a new server, or vice versa. Since we wouldn’t want data to get lost just because of serialization. In a future post, I’ll show how to set up a message handler pipeline exactly for these issues.

Now that we’ve covered all the intricacies around messaging, we can see how the code that handles that above message looks:

   1:  using System;

   2:  using Messages;

   3:  using NServiceBus;

   4:  using NHibernate;

   5:   

   6:  namespace Server

   7:  {

   8:      public class ScheduleInterviewMessageHandler :

   9:                   BaseMessageHandler<ScheduleInterviewMessage>

  10:      {

  11:          public override void Handle(ScheduleInterviewMessage message)

  12:          {

  13:              using (ISession session = SessionFactory.OpenSession())

  14:              using (ITransaction tx = session.BeginTransaction())

  15:              {

  16:                  ICandidateInterviewer interviewer = session.Get<ICandidateInterviewer>(

  17:                          message.InterviewerId);

  18:                  ICandidate candidate = session.Get<ICandidate>(

  19:                          message.CandidateId);

  20:   

  21:                  interviewer.ScheduleInterviewWith(candidate)

  22:                          .At(message.RequestedTime);

  23:   

  24:                  tx.Commit();

  25:              }

  26:   

  27:              // publish new appointment data

  28:          }

  29:      }

  30:  }

If you’ve read this far and have more questions, please feel free to send them my way. If you’re at a more time-critical part of your project and need an answer quickly, we can set up a skype call. This has been working quite well for many of my overseas clients (shout out to the guys in Ireland and Florida).

Until next time

Since you’re concerned that maybe your shipping company’s servers may be down for some kind of planned (or unplanned) maintenance just as you’re trying to fulfill orders, you use a durable messaging solution there. What happens is that messages get written to disk on your end, and later the messaging tries to transfer the messages until it succeeds. So what’s wrong with that?

Well, let’s say that the shipping company’s servers went up in smoke (true story – broken down air conditioners + poor ventilation, you get the picture). Those servers aren’t going to be coming back online any second now. So, you have all these order messages buffering on your disk. Taking into account all the data, meta-data, XML, SOAP, encryption and everything, we may get up to 1MB per message.

And now’s holiday season and your company’s selling hand over fist, hundreds of orders per second from all over the world. So that means we’re eating up 100MB of disk per second, that’s 6GB a minute, and in under an hour of our shipping company’s servers going down – so do ours.

Durable messaging – yay? We don’t want to lose those orders, right? In short, durable messaging is an important part of the solution, but it’s not the whole solution.

[Continued next time...]

If you’re impatient and just want the solution, yes, nServiceBus give you all the tools you need.

The important thing to note is that if we just automatically return the message to the queue, we may get “stuck” with that message if the first PolicyCreatedMessage never arrived. This opens us up to a Denial-of-Service attack by quite simply flooding us with a bunch of messages that never get cleaned up.

Anyway, the general idea is to first try the regular happy path, and only if we see that prerequisite data isn’t available, do we see if another thread may be working on that data. This is done by decreasing the isolation level of our transaction from the regular ReadCommitted to ReadUncommitted. This will enable our thread to see if some other thread inserted the policy in to the Policies table but hasn’t committed its transaction yet.

The next step will be how we take this code and make it generic, so that we don’t have write the same code over and over again for the different kinds of message handlers we have.

But that will have to wait until the next installment

The Setup

Just so that the example is in itself secure, we’ll assume that the password is one-way hashed before being stored. Also, given a reasonable network infrastructure our web servers will be isolated in the DMZ and will have to access some application server which, in turn, will communicate with the DB. There’s also a good chance for something like round-robin load-balancing between web servers, especially for things like user login.

Before diving into the meat of it, I wanted to preface with a few words. One of the commonalities I’ve found when people dismiss asynchrony is that they don’t consider a real deployment environment, or scaling up a solution to multiple servers, farms, or datacenters.

The Synchronous Solution

In the synchronous solution, each one of our web servers will be contacting the app server for each user login request. In other words, the load on the app server and, consequently, on the database server will be proportional to the number of logins. One property of this load is its data locality, or rather, the lack of it. Given that user U logged in, the DB won’t necessarily gain any performance benefits by loading all username/password data into memory for the same page as user U. Another property is that this data is very non-volatile – it doesn’t change that often.

I won’t go to far into the synchronous solution since its been analysed numerous times before. The bottom line is that the database is the bottleneck. You could use sharding solutions. Many of the large sites have numerous read-only databases for this kind of data, with one master for updates – replicating out to the read-only replicas. That’s great if you’re using a nice cheap database like mySql (of LAMP), not so nice if you’re running Oracle or MS Sql Server.

Regardless of what you’re doing in your data tier, you’re there. Wouldn’t it be nice to close the loop in the web servers? Even if you are using Apache, that’s going to be less iron, electricity, and cooling all around. That’s what the asynchronous solution is all about – capitalizing on the low cost of memory to save on other things.

The Asynchronous Solution

In the asynchronous solution, we cache username/hashed-password pairs in memory on our web servers, and authenticate against that. Let’s analyse how much memory that takes:

Usernames are usually 12 characters or less, but let’s take an average of 32 to be sure. Using Unicode we get to 64 bytes for the username. Hashed passwords can run between 256 and 512 bits depending on the algorithm, divide by 8 and you have 64 bytes. That’s about 128 bytes altogether. So we can safely cache 8 million of these with 1GB of memory per web server. If you’ve got a million users, first of all, good for you Second, that’s just 128 MB of memory – relatively nothing even for a cheap 2GB web server.

Also, consider the fact that when registering a new user we can check if such a username is already taken at the web server level. That doesn’t mean it won’t be checked again in the DB to account for concurrency issues, but that the load on the DB is further reduced. Other things to notice include no read-only replicas and no replication. Simple. Our web servers are the "replicas".

The Authentication Service

What makes it all work is the "Authentication Service" on the app server. This was always there in the synchronous solution. It is what used to field all the login requests from the web servers, and, of course, allowed them to register new users and all the regular stuff. The difference is that now it publishes a message when a new user is registered (or rather, is validated – all a part of the internal long-running workflow). It also allows subscribers to receive the list of all username/hashed-password pairs. It’s also quite likely that it would keep the same data in memory too.

The same message can be used to publish both single updates, and returning the full list when using NServiceBus. Let’s define the message:

And the message that the web server sends when it wants the full list:

And the code that the web server runs on startup looks like this (assuming constructor injection):

And the code that runs in the Authentication Service when the GetAllUsernamesMessage is received:

And the class on the web server that handles a UsernameInUseMessage when it arrives:

When the app server sends the full list, multiple objects of the type UsernameInUseMessage are sent in one physical message to that web server. However, the bus object that runs on the web server dispatches each of these logical messages one at a time to the message handler above.

So, when it comes time to actually authenticate a user, this the web page (or controller, if you’re doing MVC) would call:

When registering a new user, the web server would of course first check its cache, and then send a RegisterUserMessage that contained the username and the hashed password.

When the RegisterUserMessage arrives at the app server, a new long-running workflow is kicked off to handle the process:

That UsernameInUseMessage would eventually arrive at all the web servers subscribed.

Performance/Security Trade-Offs

When looking deeper into this workflow we realize that it could be implemented as two separate message handlers, and have the email address take the place of the workflow Id. The problem with this alternate, better performing solution has to do with security. By removing the dependence on the workflow Id, we’ve in essence stated that we’re willing to receive a UserValidatedMessage without having previously received the RegisterUserMessage.

Since the processing of the UserValidatedMessage is relatively expensive – writing to the DB and publishing messages to all web servers, a malicious user could perform a denial of service (DOS) attack without that many messages, thus flying under the radar of many detection systems. Spoofing a guid that would result in a valid workflow instance is much more difficult. Also, since workflow instances would probably be stored in some in-memory, replicated data grid the relative cost of a lookup would be quite small – small enough to avoid a DOS until a detection system picked it up.

Improved Bandwidth & Latency

The bottom line is that you’re getting much more out of your web tier this way, rather than hammering your data tier and having to scale it out much sooner. Also, notice that there is much less network traffic this way. Not such a big deal for usernames and passwords, but other scenarios built in the same way may need more data. Of course, the time it takes us to log a user in is much shorter as well since we don’t have to cross back and forth from the web server (in the DMZ) to the app server, to the db server.

The important thing to remember in this solution is doing pub/sub. NServiceBus merely provides a simple API for designing the system around pub/sub. And publishing is where you get the serious scalability. As you get more users, you’ll obviously need to get more web servers. The thing is that you probably won’t need more database servers just to handle logins. In this case, you also get lower latency per request since all work needed to be done can be done locally on the server that received the request.

ETags make it even better

For the more advanced crowd, I’ll wrap it up with the ETags. Since web servers do go down, and the cache will be cleared, what we can do is to write that cache to disk (probably in a background thread), and "tag" it with something that the server gave us along with the last UsernameInUseMessage we received. That way, when the web server comes back up, it can send that ETag along with its GetAllUsernamesMessage so that the app server will only send the changes that occurred since. This drives down network usage even more at the insignificant cost of some disk space on the web servers.

And in closing…

Even if you don’t have anything more than a single physical server today, and it acts as your web server and database server, this solution won’t slow things down. If anything, it’ll speed it up. Regardless, you’re much better prepared to scale out than before – no need to rip and replace your entire architecture just as you get 8 million Facebook users banging down your front door.

So, go check out NServiceBus and get the most out of your iron.

I don’t have that today. All I have are patterns which, when strictly applied, bring me close. This is a real challenge with geographically dispersed teams. Its even hard with everyone sitting in the same room.

That is why this call is more directed at the platform and tool vendors than anyone else. Stop changing paradigms all the time. Stick with one or two, make them rock solid. Make it simple to get it right.

We need non-stop software.

Let’s start talking about how to get there from here.

1989 also provided us with one of our first opportunities to present Erlang to the world outside Ericsson. This was when we presented a paper at the SETSS conference in Bournemouth. This conference was interesting not so much for the paper but for the discussions we had in the meetings and for the contacts we made with people from Bellcore. It was during this conference that we realised that the work we were doing on Erlang was very different from a lot of mainstream work in telecommunications programming. Our major concern at the time was with detecting and recovering from errors. I remember Mike, Robert and I having great fun asking the same question over and over again: “what happens if it fails?”— the answer we got was almost always a variant on “our model assumes no failures.”We seemed to be the only people in the world designing a system that could recover from software failures…

Those of you who’ve had the chance to hear me speak about technology and architecture know that one of my favorite ways to win an argument is by saying, “yeah, but what if the server restarts?”

Almost 20 years later, the more things change, the more they stay the same.

Q: Could you tell you your thoughts or preference for a distributed or centralized ESB?

DON: there is no such thing as a centralized ESB.

This is the problem with a lot of the products that call themselves ESBs. They are centralized brokers which may be clustered for availability. But they are in no way an implementation of the Bus Architectural Pattern. Please check this before cutting a check to your vendor.

Also, understand that if you do security related things in your ESB, possibly as a part of your routing rules, that if the security infrastructure is centralized that means your ESB is too. Even if it really was distributed to begin with.

Buyer beware.

Before going too far ahead, you might want to take a look at my previous post “Space-Based Architectural Thinking, or listen to my podcast Space-Based Architecture for the Web. There’s also a 30 minute webcast online describing SBA more fully here. I’m also going to try to stay away from things concerning Jini this time after already discussing the connection between Jini and SOA, and the tradeoffs between two general approaches: Tasks and Spaces vs Message and Handlers.

OK, so the issue of state-management is a big one. Everybody wants to work stateless, because it scales. The only problem is that the business processes that we are automating are long running, meaning that there are external systems or people involved. This makes these processes inherently stateful. So, we need a way to scale statefully – SBA gives us that. For some background on the “Shared Nothing Architecture”, I suggest reading this post on inter-process SOA and this one as well.

Availability also has to be handled, not only in terms of having enough servers online to handle the required load but in having all the data required to process each request be accessible. This has often been handled by the database using ACID transactions – durability being that which solved availability issues, but also hurting latency the most. The problem with saving the state of our long-running business processes/workflows in the database is the load and the responsiveness requirements. In many verticals – telcos, financial, and defense to name a few, we need millisecond level latency on each stage of the workflow. This is what leads SBA to the in-memory, replicated data grid.

Note that SBA only intends to take these workflows out of the database, and not anything else – especially not Master Data. The lifetime of these workflows is incredibly short compared to that of master data like customers and products. It will have much different backup strategies as well. In terms of load, these workflows will be heavy on reads and writes together in the same transactions, but quite low in terms of just reads. If we have workflows that perform work in parallel, we easily end up with concurrency requirements that make DBAs cringe under the barrage of short transactions.

If you’re worried that Workflow Foundation (WF) won’t scale because of the above, you needn’t be. You can (more or less easily) replace the persistence mechanism of WF with your own, saving your workflow instances to an in-memory replicated data grid.

By enabling the objects in the grid to call back into logic on your servers, you have, in essence, done messaging and more. The added benefit that SBA receives from this is a unification of technology between caching and messaging. This translates directly to savings when it comes time to cluster each of those technology’s environments.

Finally, if we can find an attribute in the incoming stream of messages that creates a nice even distribution, we can then partition our load between our servers by that key. This will work up to the point where the load per key increases beyond a single server’s capacity, and then we have to look at re-partitioning, a non-trivial problem. However, if we put objects in our grid that represent the master data, and tie them to our workflow instances with both of those tied to the key of our load, a smart infrastructure can make sure all that data is already resident on the server that is handling that piece of the load. That decreases latency even more since we no longer have to pay network roundtrips to collect all the data needed before we can process it. That’s a substantial advantage for the above verticals.

But all of this has nothing to do with SOA.

Sure, it’ll change how we implement our Services internally, but it has no impact on their interfaces or the top-level service decomposition. In the Java community, the word “service” is often used to describe the logic of a system. Great significance is placed on keeping these “services” simple, as in Plain-Old Java Objects. The fact of the matter is that the logic of the system should be simple and independent of other concerns like data access and communcations (a la Web Services), but that does not make it a service, not in the SOA sense.

For more information on what Services in SOA are like, check out this podcast on Business and Autonomous Components in SOA. Actually, SBA will probably have the biggest impact on the way autonomous components will handle service-level agreements.

So, it appears that even with SOA, SBA has its place. The former dealing with business level agility, the latter dealing with all the technical aspects of supporting that agility. If you’re tasked with the designing the architecture of a scalable, available, high-throughput, low-latency distributed system, I’d strongly advise you to look at SBA – the technical value is overwhelming. Even if you don’t utilize all elements of SBA and choose the Master Worker Pattern instead of load partitioning, you’ll find the technologies supporting SBA to be quite flexible in that respect.

Will Space-Based Architectures be a part of your future? I don’t know for sure, but they’re a most welcome part of my present.

In his previous post Is it service-oriented if the message cannot be intermediated, Nick defines intermediability as “SOA should give us the ability to intercept a message going from point A to point B, and react to that message without informing either end of that pipe.”. I’ll respond to this in due course.

Anyway, he continues on by saying “SOA [is] an architecture for Enterprise Application Integration.”

I can’t agree with that statement. The main reason is that EAI puts the application in the center, and that integrating existing applications one of the primary purposes of it. It is my assertion that in order to solve many of the problems that we are having today, we need to take a broader, business based view of the enterprise and model that with services. A service may be implemented with one or more applications. However, my experience has been that these services tend to use parts of existing applications, with multiple services using different parts of the same application. The reason for this is that the applications we have today, especially the ERP monoliths, do a lot, and at the same time, not everything. This is part of the reality that EAI tried to solve, but then got mired down in cross system hell. You just can’t solve poor business decomposition in the technology domain.

The value of putting services at the fore makes it possible to gradually phase out and evolve legacy applications, and migrate costly mainframe apps bit by bit without having these changes ripple out and break other services. The same is true for those systems’ data – backup strategies are defined at the service level, impacted primarily by their Service-Level Agreements.

While I whole-heartedly agree with what Nick has to say in terms of OO intermediation of the Dependency Injection variety, and that scaling up those same concepts in terms of messaging is the right way to go, I take issue with orchestration in the intermediation area. These “tactical changes” need to be done in the context of the top, business-level service strategy. That means that all logic belongs within a service. The “network” between services is just that, a “dumb” network – no business logic of any kind, just technological capabilities like knowing which physical server to route messages to.

In this spirit, I’d like to suggest an alternative solution to the example Nick gives. Here’s the scenario:

Let’s say that system 1 generates an invoice. It sends an event to the world saying “invoice here” and system 2 captures that message. System 2 asks for details about the invoice… perhaps it will place the information on a web site for internal support teams.

Let’s say that we are moving to a CRM solution in our internal support groups. We want to create the information in the CRM system related to the invoices that specific customers have been issued. We need to integrate these two systems. The existing web app needs to have a link to the CRM system’s data, to allow the user to move across easily.

And here is the solution he prescribes:

We can intercept the request for further information from the web app to the publisher. When the publisher responds with information about the invoice, we can insert the invoice in the CRM system, add a link to the CRM record for that invoice to the data structure, and resume our response to the web app. Assuming that our canonical schema has a field for ‘foreign key’, we have just integrated our CRM and web information portal… without changing either one.

Without getting into the business-level analysis of what the correct service decomposition might be, here’s what I suggest (although all of these “systems” might just end up within the same service, or having parts of them being used by multiple services).

First of all, have all information about the invoice available via the message only. This could be done by actually putting all the invoice data in the message, or by placing a URI instead where other systems can HTTP GET it from – REST style. This decreases coupling between the publisher and its subscribers. However, we haven’t solved the problem of our web apps getting access to the relevant data in the CRM system.

The solution presents itself at the business level. The invoice is not “complete” without the appropriate CRM data. Therefore, it does not make sense for a service to publish it that way. Let’s call this service the Purchasing Service. It would handle the workflow of receiving the first system’s event, adding the invoice to the CRM system, and taking the resulting full invoice data and publishing that. All external systems like the web apps would see just the final event. Orchestration, if there even is such a thing, occurs within the service boundary. This technological level intermedation isn’t even a blip at the business level. We can also imagine other services, say a Sales Service, that would use the CRM system as well.

In summary, when moving to SOA, intermediation provides many technological benefits in getting data and behavior to work across existing systems and applications, however it’s laregly a NO-OP at the service level. After phasing out many of those existing applications behind the service boundaries, the same service-level interactions would persist. Your Service-Oriented Architecture would not be any different. That’s the technical agility aspect of SOA.

What they say about the connection to SOA, though, requires some clarification. Here’s the quote:

There is a lot of talk going on about synergy between Grid Computing and SOA. It is however driven primarily by implementation concerns at this point rather than by any deeper considerations. Clearly, Grid Computing can deliver unchanged value without SOA, yet WS-* based implementation (such as Globus) can be beneficial in some cases (highly distributed heterogeneous environments that should only exist in unfortunate legacy-support situations).

The main thing that I want to call out is that “grids” don’t cross service boundaries – not at the logical level anyway. Although, even if you did share a single grid infrastructure between services implementations, you may have some problems maintaining service-level agreements, autonomy may be put in danger.

Just something to keep in mind.

A “JavaSpace” is really just a space, which is, at the end of the day, a queryable distributed in-memory hashtable. Something many of us are already doing for caching. The reason you shouldn’t be doing this yourself is simple. While keeping a single hashtable in memory on a single computer and synchronizing it against changes to your database is simple, doing that in a highly available manner across multiple servers is not. Vendors providing solutions in this space include:

• Gigaspaces
• IBM ObjectGrid
• Alachisoft’s NCache
• GemStone
• Tangosol’s Coherence

But there are others as well. Bottom line: don’t develop one of your own. Do a proof of concept with your short list of vendors and go from there.

The article sums it up nicely like this:

Although JavaSpaces servers are not trivial to set up, they are much easier than any other type of grid computing server. Furthermore, the simplicity of the interface makes the learning curve easier. The greatest advantage of the JavaSpaces approach is the ease with which additional workers can be added to the grid.

It should be clear from the example that there is a lot of extra communication traffic in a JavaSpaces solution so the only reason to use JavaSpaces or any other form of grid computing in support of a Web service is a requirement for computing power or special resources that are not feasible to supply on the server directly.

I have this to add to it. Whereas most traditional systems keep the idea of message-based communication and data caching separate, spaces allow you to kill two birds with one stone. Even if you don’t go the whole Space-Based Architecture route, you’ll find that spaces will fit nicely in your distributed architecture toolkit – I know I did.

However, in the truly large scale scenarios, that isn’t enough.

On of my customers is having to scale up from 500 concurrent users to 50,000. You need to get seriously close to the metal to handle that. Here’s a great post on the kind of hardware storage considerations I went through that I went through there. Of course, sometimes a database is just the wrong hammer for your screw – sometimes what you really need is a space.

“Udi”, he says, “I just heard that for us to realize the potential of our SOA, we should be using an ESB.”

I’m sure he said a lot more than that, but that first sentence was enough for me to tune out. I managed to get through the conversation without catching a case of acronym-itis, but my train of thought was broken. I wasted one Google search to find out that ESB meant “Enterprise Service Bus,” got fed up, and went to Starbucks. Of all the overused terms in software, “Enterprise” is by far the most annoying—although “Service” seems to be moving up in the world.

I’ve been developing loosely-coupled systems for awhile now, using all kinds of technologies, and never thought that I was doing anything particularly ground-breaking. So when the CIO came down and introduced me to the army of high-priced consultants that were going to help us redesign our software to become more “service-oriented” I really was interested in seeing what would be different. Soon after, I realized that the vendor-driven SOA meant nothing more than Web Services, XML, and loose coupling—with the mindset of loose coupling being the most important. I’d been “service-oriented” all this time and didn’t even know it. Oh, and it turns out that we didn’t have to redesign anything.

Imagine my utter joy in hearing that the reason our project was in trouble was that we didn’t have an ESB. And all this time I thought it was because the requirements were changing every two weeks.

It was about a month after that that our project manager got promoted for doing such a great SOA implementation and was now in charge of making the whole company, oops, sorry—enterprise, service oriented. I was made “acting project manager” and managed to do one really smart thing pretty quick—get our software through testing and deployed within the month, before too many new requests came in (I think it was possible because most of the stakeholders were on holiday). The system wasn’t that complex; we had the standard HR, accounting, inventory, etc., functionality split up into the same high-level components. Each top-level component had its own server-cluster and database. We pulled the data from each of the database to the data warehouse using classic ETL. Data flowed between the components primarily using publish-subscribe semantics while the client-side software just request-responded what it needed from each component.

We kept on working, pushing out features as fast as we could, until one morning I found the sysadmin at my door.

“We had a problem with some of the disks on the accounting cluster, so we installed it on a different cluster and brought the first one down. We tried a simple test to make sure everything was OK, and it wasn’t. A bunch of things don’t seem to be working.”

After looking around for a bit I found out that the sysadmins had forgotten to update the config files on the servers and the clients. We restarted the server components and they worked fine, but we couldn’t really go around restarting and changing config files on all the clients. Luckily, the sysadmins had it set up so that every client on our domain that logged in to the network could be sent a script to run, so pushing out the new config files was easy. As for the restarting part, we called up the help desk and told them that if (when) someone called about why their software wasn’t working properly, just to tell them to close it and run it again—which apparently is their standard first suggestion anyway.

Well, things lurched along like that for a while as we put out more and more functionality—put up a Web front-end, tied in some business partners, etc. I thought I had everything under control until our COO charged in, with the CIO in tow.

“Our business partners haven’t been able to send us orders for almost a week,” he fumed. “What did you do?!”

When money talks, you’d better believe everybody listens. After some seriously hectic hours of peering through diffs between the deployed source and the previously deployed source, we were getting nowhere. Somebody, I don’t remember who, had the common sense to get the sysadmins in there too. It turned out to have been the same problem as before, but this time with the inventory cluster. So we used the same solution. The problem was that our business partners’ software didn’t get the updated config files. While I was pondering how we could get the same system to work with external partners, my boss called me in for an urgent meeting.

“I just got a call from Jim (the CIO) and he wants you and me to help him explain what happened to the CEO.”

I started to get that sinking feeling, the kind where you know things are going from bad to worse. That afternoon, we all filed in to the chief’s office, bracing ourselves for the worst. He got directly to the point.

“If anything like this happens again, you three are fired. Now get the hell out of my office.”

How’s that for motivation?

And to top it all off, before he hurried off to another meeting, Jim asks us “You guys know about our first audit for Sarbanes-Oxley in three months, right? I don’t want to see any more screw-ups, and this SOX stuff is getting people anxious. We need full audit trails on everything.”

It looks like Moore’s law will continue indefinitely: You will need to handle twice as much crap today as you did 18 months ago.

It was at about this point where I realized that I needed help. I called up one of my old partners in crime, Clem, who had been doing large-scale distributed systems development for awhile. I told him my sorry tale and asked if he could give me a hand. Unfortunately, he was in the middle of some serious crunch time, but he left me with these pearls of wisdom:

“Udi, it’s all in the message. Forget about remote method invocations and pub-subbing events—down on the wire it’s all just messages. The trick is to think of your system as passing messages at the application level as well.

Asynchronous message passing over queues. It’s really quite simple.

Once you’ve packaged everything into the message, that message can be dynamically routed anywhere, and so can its responses. The application doesn’t need to bind against any specific endpoint—it just drops a message addressed to some logical location. Infrastructure can make sure that messages get to the logical recipient, even if they change physical locations.

That infrastructure is what brings about the “Bus” architectural style between your distributed components.”

Luckily I was writing down what he said, because I had to re-read it at least a dozen times for it to sink in. Flashback to that original conversation with the CIO—so that’s what ESBs are for! Well, you wouldn’t have guessed it with all the hype going on—IT/Business Alignment, like that’s going to happen any time soon.

After talking with some ESB vendors, I understood some nuances in what Clem told me. The message passing at the application level is really passing logical messages—a message is an object just like any other. The transformation that logical message undergoes in order to be sent across the wire is something else entirely. We can transform our message to and from XML, binary, text based key-value pairs—anything we need. Finally, the transport used to pass that wire-representation between machines is an infrastructure detail that is also independent of the logical message.

Once my mind warped itself around asynchronous messaging, the whole SOA thing became clear. The top-level components we were developing were providing top-level services—requests would queue up like people would at the teller in a bank. A component could send out the exact same message either as a broadcast or a unicast, where the recipients would be able to use the same semantics either way. Exposing a method, subscribing to an event, and handling a message were all the same, both internally and externally.

I can’t explain how much this simplified my view of the distributed world. It kind of felt like dominos—as one thing fell into place, it knocked down something else. I was finally beginning to understand what needed to be changed in our system—and all it took was a multi-million dollar mistake and nearly getting fired.

Needless to say, the whole SOX thing caused all hell to break loose. Our team wasn’t compliant, but then neither was any other team. The same goes for most of the company’s software. But, the reassuring thing for me, was knowing where I was going with our system. It took some time—we redesigned most of the communication paths, found a vendor whose product met our needs (at the right price), and several months later, we rolled out the new version. I wouldn’t say that the rollout was flawless, but I will tell you this—when the sysadmins moved a service from one cluster to another, no config files needed to be pushed out in order for things to work, and, more importantly, no orders were lost. I even got promoted from “Acting Project Manager” to “Project Manager”

Anyway, when it comes to scaling out queues help a lot. Although not explicitly mentioned in the above post, having multiple machines feeding off of the same queue is the key to scalability and is known as the Competing Consumer pattern. The added benefit of such a design is that you get availability without any additional work, given that you have more than one consuming machine per queue.

One thing to keep in mind about the Microsoft platform today is that MSMQ does not currently support remote, transactional receives (Update MSMQ 4 released as a part of Vista / Server 2008 now supports this, yet people in the know have told me to avoid this feature). What this means is that, in the above design, you cannot make sure that if one of the servers fails while processing a message from the queue, that that message will return to the queue. For some kinds of messages this isn’t a big deal (like stock prices), but in other cases (like money transfers) this isn’t acceptable.

So, bottom line is that queues and other asynchronous transports (JMSs and topics) enable robust systems to be built using proven patterns, but be aware of any limitations of the technology and what ramifications they may have.

The tenets of Service Orientation as put forth by Microsoft include one about “autonomy”. The tenet states that “Services should be autonomous”. After some digging, I found out that the intent of the authors was “The teams that develop different services should not be dependent on each other”, or in shortened form “Autonomous Teams”. This revelation was surprising to me since the real meaning of the tenet was less profound than what I had imagined – autonomous computing.

The idea of autonomous computing has been around for some time and presents a view of the world in which computing units cooperate to achieve global goals yet are not dependent even on the existence of other computing units to function. (If you’re envisioning tiny robots playing soccer, you’re not far off.)

So when I first saw the autonomy tenet, I was thinking of autonomous services: services so loosely coupled that the correct functioning of a service would not be dependent on the correct functioning of other cooperating services. Services loosely coupled in time as well as in code. Obviously this would mean that if Service A needed to cooperate with Service B, and Service B was not even available, Service A would continue to function, and live up to its service-level-agreement. But before we start drifting off into the outer reaches of business-IT alignment, let’s bring this down to earth.

Before we get into a detailed analysis of the how, let’s first agree on the why. Despite being a historical trend, architectures these days tend to be more loosely coupled than before. Loose coupling being a good thing that enables us to better manage the complexity inherent in large software projects. The practical test of loose coupling in a system is changing the public interface of a class and seeing how much of the system doesn’t compile any more (dynamic languages aside). Service Orientation brings us tenets that, when followed, lead to more loosely coupled architectures than if we actively did not follow them. I think that we can agree then that if we could somehow achieve the loose coupling in time mentioned above, without paying an arm and a leg, that would move our architectures another step forward.

Looking back on the evolution of the field of distributed computing, we can see that, over time, less and less things are being assumed. It is now well understood that anything that goes over the network takes much longer than those calls that stay on the same machine, yet once systems were built that abstracted the network communication into looking just like local calls. The performance of those systems was matched only by their lifetime. With the advent of autonomous computing, the assumption that the called service is available and will respond in a timely fashion is called into question. In the real world, servers crash and network equipment goes up in smoke – we can no longer take for granted that communication will always be available, and that its quality will be good enough. In essence, this marks the end of synchronous RPC/RMI. The following code just won’t cut it in this brave new world:

localhost.service1 s1 = new localhost.service1();
orderReply = s1.HandleOrder(orderRequest);

If the service is unavailable, what will happen to our order request? Will it just get lost?
If the service takes a long time to respond, will our server tie up resources for the same amount of time? If this happens under peak load, might it cause our server to crash?

Performing the above code on a different thread won’t make any difference, autonomous services means the end of Request/Response as we know it.

“No Request/Response between services?!”, you ask incredulously.

The simple answer is “yes”, but there is another level of meaning to it. If you have two software entities that between them you just HAVE to have request/response communication, then they should be in the same service. This is where the real architectural guidance comes in.

In component-orientation and object-orientation, the division of the solution into the right number of parts, with each part having the right amount of responsibility was a kind of black magic passed from master to apprentice. Getting the boundaries right was paramount, but difficult. A number of litmus tests are used to catch the gross errors, and the rest is just gut. So too, the request/response test helps us catch gross errors in service boundary demarcation.

The interesting thing that happens after separating our services out this way is that we often end up with services that mirror the way the business side is structured. Voila, business-IT alignment with your hands closed and one eye tied behind your back! Well, it’s one step in the right direction anyway.

This leaves us with the original types of one-way communication (fire-and-forget, pub-sub, etc) and with one kind of two-way communication: duplex. Duplex is really just two one-way communications (A to B, then B to A) that are correlated. First, I send a message to you, mark it with an id number, and save that number. At some future point in time, you get the message, process it, and send a message back with its own id number. But, you’ll have to put my original id number on the message too, so that I’ll know that your message is a response to mine. At some even more distant point in the future, I get a message from you, look at it, and see that it is the long-awaited response to the request I sent way back when.

If I had to sum up the difference autonomous services bring to the styles of communication used between services, I’d say this: You get a message, look at it, and figure out what it means and what you should do. This isn’t an infrastructure issue. There application level timeouts to deal with (If I don’t get a response back in 3 days, then notify the supervisor), and long-running workflows to manage (next whitepaper ).

If there is one thing to pay attention to in this whole “autonomous services paradigm” it is that the focus has shifted from between services to within a single service. In parting, I want to let you know that systems can be, and are being, built this way. It works. It better than works. Systems created this way are more robust to failures (seeing as they’re designed for failures makes it less impressive) and easier to manage. Give it a try. You didn’t really think that SOA would fizzle away into a bunch of WS specs, did you?