Also, “passing reference data in a message makes the message-consuming systems dependent on the knowledge and availability of actual persistent data that is stored “somewhere”. This data must separately be accessed for the sake of understanding the event that is represented by the message.”

The discussion of self-contained events can be compared to integration databases vs application databases.

Centralized Integration – Pros & Cons

If everything in a system can access a central datastore, it is enough for one party to publish an event containing only the ID of an entity that that party previously entered/updated. Upon receiving that event, a subscriber would go to the central datastore and get the fields its interested in for that ID. The advantage of this approach is that the minimal amount of data necessary crosses the network, as subscribers only retrieve the fields that interest them. Martin Fowler describes the disadvantages as:

“An integration database needs a schema that takes all its client applications into account. The resulting schema is either more general, more complex or both. The database usually is controlled by a separate group to the applications and database changes are more complex because they have to be negotiated between the database group and the various applications.”

This is far from being aligned with the principle of autonomy so important to SOA. In that respect, the architectural principle of self-contained messages points us away from those problems and towards more autonomous services.

However, once we have these autonomous business services in place, we may find that we don’t need 100% fully self-contained messages anymore.

A Real-World Example

Let’s say we have 3 business services, Sales, Fulfillment, and Billing.

Sales publishes an OrderAccepted event when it accepts an order. That event contains all the order information.

Both Fulfillment and Billing are subscribed to this event, and thus receive it.

Fulfillment does not ship products to the customer until the customer has been billed, so it just stores the order information internally, and is done.

Billing starts the process of billing the customer for their order, possibly joining several orders into a single bill. After completing this process, it publishes a CustomerBilled event containing all billing information, as well as the IDs of the orders in that bill. It does not put all the order information in that event, as it is not the authoritative owner of that data.

When Fulfillment receives the CustomerBilled event, it uses the IDs of the orders contained in the event to find the order information it previously stored internally. It does not need to call the Sales service for this information or contact some central Master Data Management system. It uses the data it has, and goes about fulfilling the orders and shipping the products to the customer, finally publishing its own OrderShipped event.

Notice, as well, that in the original OrderAccepted event there were the IDs of products the customer ordered. These product IDs originated from another service, Merchandising, responsible for the product catalog. The same thing can be said for the customer ID originating from another service – Customer Care.

The Issue of Time

One could argue that since subscribers use previously cached data when processing new events, that data might not be up to date. Also, we may have race conditions between our services. In the above example, if Billing was extremely fast and more highly available than Fulfillment. Billing could have received the OrderAccepted event, processed it, and published the CustomerBilled event before Fulfillment had received the OrderAccepted event. In short, the CustomerBilled and OrderAccepted messages could be out of order in Fulfillment’s queue.

What would Fulfillment do when trying to process the CustomerBilled message when it doesn’t have the order information?

Well, it knows that the world is parallel and non-sequential, so it does NOT return/log an error, but rather puts that message in the back of the queue to be processed again later (or maybe in some other temporary holding area). This enables the OrderAccepted message to be processed before the CustomerBilled message is retried. When the retry occurs, well, everything’s OK – it’s worked itself out over time.

In the case where we retry again and again and things don’t work themselves out (maybe the OrderAccepted event was lost), we move that message off to a different queue for something else to resolve the conflict (maybe a person, maybe software). If/when the conflict is resolved (got the Sales system / messaging system to replay the OrderAccepted event), the conflict resolver returns the CustomerBilled message to the queue, and now everything works just fine.

As all of this is occurring, the only thing that’s visible to external parties is that it happens to be taking longer than usual for the OrderShipped event to be published. In other words, time is the only difference.

Summary

The problem of non-self-contained events is mitigated first and foremost by business services in SOA, and the apparent issue of time-synchronization by business logic inside these services.

Don’t be afraid to put IDs in your messages and events.

Do be afraid of using those IDs to access datastores shared by multiple “services”.

Using IDs to correlated current events to data from previous events is not only OK, it’s to be expected.

The architectural principle of fully self-contained messages steers us away from the problems of Integration Databases and towards Application Databases, autonomous services, and a better SOA implementation. From there, following the principle of autonomy from a business perspective, will lead us to services not publishing data in their messages that is owned by other services, taking us the next step of our journey to SOA.

Related Content

[Podcast] Message Ordering – Is it cost effective?

Don’t EDA between existing systems

[Podcast] Handling dependencies between subscribers in SOA

In his post, Steve provides concrete guidance on how to look at data:

“The point about these bits of data is that they are about recording what has happened. Where this approach falls down is when you try and apply that approach to what is happening.”

One of the core ways that I suggest you avoid falling down the Data Services rabbit hole is to keep the context of time in mind as you analyse the data your services use and are responsible for. Ask yourself questions like:

1. Who creates this specific data element – not just this kind of data?
2. Can we partition these creators based on some property of the data?
3. When do they create it?
4. At what point can others access this data?
5. Do others need to use this data in their own business processes?
6. If so, how up-to-date does the data need to be? Up to the minute? Up to the millisecond?
7. Can we avoid transactions between those who create data and those who use it while maintaining business correctness?

In a follow up post, I’ll be analysing how we can identify services and business components in a domain by using these questions. More importantly, we’ll see how the message contracts of our services can be driven out by answering them.

Stay tuned.

Typically, in such an environment [SOA], services tend to be parts of multiple systems. For example, consider a Products service! It might start its life as part of a sales system. Later it might be involved in a purchasing system, a product development system, a marketing system, a warehousing system, and perhaps in several other systems too. This process may take years, and it really never ends. The service is the same, but its responsibilities and its external exposure are increased with each system it’s enrolled in.

One of the core tenets of SOA that all vendors and analysts agree upon is that there should be loose coupling between services. If you were to design such a product service, it’s clear that changing part of its interface could break almost every system in the enterprise. That doesn’t sound like loose coupling to me.

If there’s one place that is the source of loose coupling – it’s the business. Warehousing is viewed by the business as being fairly independent of Marketing. While Sales might make use of data created in Product Development, business wouldn’t want any problems in IT related to Product Development to inhibit Sales ability to accept orders. That is another kind of loose coupling – the ability of one service to make use of “not-accurate-up-to-the-millisecond” data created by another service. That’s known as loose “temporal” coupling, as in loose coupling in the dimension of time.

Loosely-Coupled Services

So, in the example described we’d see the following services:

• Sales
• Purchasing
• Product Development
• Marketing
• Warehousing / Inventory

Product data would flow between the services but each would have a very different internal view of it.

• Product Development would be more interested in managing the schedule and risk around a product’s development.
• Marketing would probably be more focused on its relation to competing products and pricing.
• Purchasing would be maintaining data as to which suppliers are being used to supply raw materials for the production of the product.
• Sales would be looking at actually accepting orders and giving discounts.
• Warehousing would be focused on the storage and transportation aspects of the product.

As you can see, there is very little overlap in the data between these services even on something similar like product data. The logic of each service around the management of its data would be even more different. This leads to services with a high level of autonomy.

There Be Dragons…

Without starting at this business-level loose coupling, I doubt that any technical effort will succeed. That is to say every time I’ve seen this style implemented it has failed, but that’s no proof. Conversely, every time that we did start our SOA efforts by identifying the clear business fracture lines, we were able to maintain loose coupling all the way down. That is not to say that it always will succeed, but the logic is sound.

I suppose that the difference between my view on SOA and Sundblad’s stems from the fact that they put systems at a higher level of abstraction than services, and I put services on top. Regardless, I do agree with their views about architecture and engineering and consider them quite valuable.

After all, you don’t have to go too far down the SOA path before you realize that unless you build reliable, asynchronous, loosely-coupled services, your SOA architecture is going to have serious reliability problems and Service Broker brought reliable, transactional messaging to a whole new level of reliability and efficiency. What I found was a bunch of architects trying to figure out how to use WS-Transactions to build tightly coupled services to replace their tightly coupled objects.

So far, I’m thrilled that he is pushing this message in, and out of, Microsoft.

He then goes on to debunk the idea of Entity Services:

I next ran into a lot of people architecting SOA systems to provide a common services interface to a lot of diverse back-end systems. I’ve talked to people who had over 100 systems that handle customer data for example. If you build a perfect customer service to wrap all these systems with a common schema for the customer record you have a single view of the customer right? The first time your user tries to change the phone number for Acme Rockets Inc. and gets back 80 records which may or may not be for the same customer, the single view of the customer loses some of its appeal.

I’m loving it.

He then suggests a solution:

That’s how I got interested in Master Data Management. I really believe that accurate, unambiguous clean data is a prerequisite to an SOA project.

Hmm, from what I’ve seen of MDM, it puts a different spin on all the other things he said above.

Other than that, I’m totally with him. A deep understanding of data is necessary to get a good service decomposition. Without understanding the transactional nature of that data, you just might end up back tightly coupled and with a monolithic web service mess.

Whether or not you use Microsoft’s SQL Server Service Broker, Roger is definitely a guy worth listening to.

Get it via the Dr. Dobb’s site here.

Or download directly here.

Want more? Go to the “Ask Udi” archives.

Summary: Enterprises today depend on heterogeneous systems and applications to function. Each of these systems manages its own data and often doesn’t explicitly expose it for external consumption. Many of these systems depend on the same basic concepts like customer and employee and, as a result, these entities have been defined in multiple places in slightly different ways. Entity aggregation embodies the business need to get a 360-degree view of those entities in one place. However, this business need is only one symptom of the larger issue: business/IT alignment. Service-oriented architectures (SOAs) have been hailed as the glue that would bring IT closer to business, yet the hype is already fading. We’ll take a look at concrete ways that autonomous services can be used to transform the way we develop systems to more closely match business processes and solve immediate entity aggregation needs.

Continue reading.

Get it via the Dr. Dobb’s site here.

Or download directly here.

Want more? Go to the “Ask Udi” archives.