At that point, we create rollups of the data which can be stored for longer periods of time.

Before the computers took over we had paper baskets and shredders in our businesses. Since the basic business requirements are mainly unchanged we also need the digital equivalent of them today. I don’t want to filter my valuable information out of a trash mountain.

This observation does not mute your main point: Make sure you understand what your user wants/needs and *then* implement the correct logic for this. However, sometimes that can be a plain DELETE.

Saved my day!!!!

Thanks,
Leo!

I absolutely agree that trying to solve this problem at the database level is very complex – especially when that database is used by many different applications.

The recommendation of not deleting is for a database that belongs in its entirety to a single application – what Martin Fowler calls an “application database”, as opposed to an “integration database”. In this kind of case, it is less likely to see that same kind of explosion that you’ve experienced.

Hope that makes sense.

I have to agree with this, as I see this problem occur frequently. In particular, I have seen it in relation to a data source that is central to most of the LOB applications my company writes (the security master for a trading firm). Because the data is so central, we cannot have one person or a group of people in charge of all access to that data; they would simply be overwhelmed. So everyone must have access to that data. But not everyone has the time to become an expert in the security master. And because the designer of the security master subscribes to the “never delete” policy, the number and combinations of statuses has exploded, and is continually increasing. It is very difficult to accurately write a query that will actually return the data that you think you are asking for.

Even something as simple as finding the ticker for a stock is non-trivial, because all of the previous tickers for the stock are still in the table (with various statuses depending on the reason for the change), AND the current ticker might have a status that indicates that it has been scheduled to be changed (but has not been changed yet). While there are some corner cases where seeing the entire ticker history for a stock might be useful, the vast majority of our operational systems don’t care; they just want to know what the ticker is right now. But these systems must deal with the increased complexity for the sake of maintaining a history that is rarely if ever useful.

That sums it up very well.

First, you are compelled to use surrogate keys as natural keys are out of the question because of the need to add a “state” column. Second, the only way to protect against referencing an inactive value at the database level is to use triggers. Third, we lose our database level referential integrity that tells us that an about to be deleted value has been referenced. If we simply change the state, it does nothing to compel us to correct the child references to use an active parent value or to change the state of those child records. Lastly, you have to code against phantom values:

For example, suppose we have a status table with a name column and a series of attributes. We cannot put our unique constraint on name alone; it must be on the state and the name. For each addition, we must check whether we have ever had the given name and if so, “reactivate” it. If we simply reactivate, the new status will contain the attributes of the previously deleted instance unless we guard against that.

It is more error prone because it is more work to implement correctly especially in reporting as all the report designers have to know to account for the state column.

If we go the route of a mirror table contained deleted values, that has its own set of issues. None of these problems are insurmountable, but they do add work, which means cost which means a cost/benefit problem for the stakeholder.

IMO, soft deletes must be a conscious, well thought out decision as it adds cost to the project which can be substantial depending where the soft deletes are needed. It is not just a “tiny” bit more complex. Every query has to account for the state and typically those errors happen in reporting.

Unfortunately, “error-prone” is no so far off. It probably depends on how you structure your code, but adding an extra status to some central widely used entity can ripple changes to many places: you can have a bunch of reports that need to be changes, querying in the UI needs to be changed, the new status has to be added as an extra parameter to UI search windows, you may need to add extra language translation for that status and so on.

In short, adding a simple status can have wide spread consequences.

The SQL etc. might be rather trivial to fix, but the biggest problem is to locate all the places that need fixing – and *that* is error prone.

Neither is a big test suite going to help you: all your test will run fine since none of them are aware of the new status. Adding new tests helps of course, but figuring out which to add is also a problem.

This is *not* to say that I prefer a technical delete over true bussiness logic. I just want to illustrate that it might be a bit more expensive than exptected.

If you have a good solution then please share it

The question of business status is more important than the answer. What I mean by that is that as long as the business people are making the tradeoffs between complexity/cost and value in the statuses, then we as developers have done our job.

It’s when we assume that deleting (whether soft or not) is a good enough technical solution – without vetting that with the business, that we overstep our responsibility.

To your point #1, we don’t bypass RI – we actually model the business life cycle of important entities. And to point #2, correspondingly, we show users what they need to see – adding a single where clause might make it a *tiny* bit more complex, but I’d hardly go so far as to say “error-prone”.

Thanks for your comments.

The way I see it, if you (as suggested) chose to replace physical deletes
with modelling the business rules underlying this operation, you will still
be faced with the two main problems associated with soft deletes.

1. You bypass the referential integrity of the RDMS. (delete is replaced with status)
2. You end up with more complicated and error-prone queries.

It’s a tradeoff, but if you can’t come up with an argument of why you should model
the business rules underlying the delete operation – maybe avoiding the two
problems above and keeping the model simple is the best choice. In my experience,
when you introduce new entity states, you quickly end up in a situation where users cannot explain the difference between two states.

You’re absolutely right – sometimes MS Access really is the right tool for the job .

See my answer (12) to Andy. If the user mistakenly added an entity, then we could call entity.AddedByMistake(); which may result in a delete.

Other actions are just as easy to revert when you have the original context.

Hope that answers your questions.

Well, the concern here is with something you want to revert. Revert an action completely which was by mistake (different than status increments). And there comes physical and its counterpart ‘IsDeleted’ can play a big role. Such entries are useless by any meanse, and can create mess in the system – so the best option is to throw them out – obviously with care.

Your thoughts?

Taking up a GUID in the case of a mistake is not such a big deal. However, since we can set up a preliminary state for data as its being typed in (where most mistakes happen), data in that state doesn’t actually count towards reserving a given name/code/resource. When entities in that state are “published” or “authorized” is when we check for uniqueness and “reserve” that name/code/resource.

As for the archival, I acceded that point above in my comments to Andrig.

Thanks for your comments

It is a general organizational pattern with developers to accept “requirements” at face value and then go develop them exactly as specified. We need to invest more in human-to-human interaction in order to solve/prevent the software problems we’ve been seeing as an industry.

You bring up a valid point that when a given graph of entities is in a certain state for a certain period of time, it can be in the business interest to archive it away from the main database for performance reasons.