I hear this too often: “X sounds like a great pattern, but it’s overkill for small applications”. Many patterns have been subjected to this including (but not limited to): SOA, DDD, CQRS, ORM, etc. Often the statement is made by a person without experience in the given pattern (though possibly experienced in other patterns). Let’s take a look at the second part – the “small application”, and ask:

What makes an app small?

Or inversely, what makes an app warrant the “enterprise” moniker?

If there’s one thing that the history of our industry has shown repeatedly, it’s that developers aren’t particularly accurate with their estimates. Like, orders-of-magnitude inaccurate. Knowing this, it’s surprising that the “small app” argument seems to win so many arguments. The same goes for justifications in the form of “we’ve got to have an X, this is a BIG project”.

So, what makes an app small?

Is it a small number of lines of code? Well, what if those lines of code are keeping planes in the air?

Is it a small number of developers? Same as above. Actually, history has shown that some of the most valuable bits of code written were done by small numbers of developers.

Is it that it will only be installed on a single machine?

Is it…

What could it be?

The real issue

The small app argument is a diversionary tactic.

Loosely translated, it means “I’m comfortable where I am and I don’t want to change”.

Moving on…

The real story of size

Once we actually look at the specific context of an app, we tend to see that someone cares a great deal about it, enough to finance its custom development – rather than buying an off-the-shelf alternative. The expected lifetime of business use is easily 3-5 years, if not 7-10, during which many enhancements will likely be requested. Thus, some non-functional properties of the code matter – at the very least maintainability.

In which case, if the given pattern or approach does significantly improve the desired non-functional properties of the app, it only makes sense to use it.

There is one class of software that might possibly be treated as “small” – the one-off script that’s written to automate some IT task. And even then, so many of these scripts end up living longer than the apps themselves that they should be engineered at the same level of quality.

In closing

Don’t counter a “small app” argument with psychology.
It will only make matters worse.

Instead, rephrase the issue around the lifetime of business use.

I’ve found that there are precious few cases where the harsh light of reality doesn’t help the appropriate decisions be made. If indeed this is a small-lifetime-app, just drag-and-drop until you’re done. Otherwise, the time it takes to understand and evaluate the applicability of the given patterns will definitely pay itself back many times over the life of the app.

And managers, keep your ears open for it. The technical risks behind that statement are icebergs waiting to sink your project.

* with thanks to Mike Nichols for pushing my buttons.

“We need to rewrite the system.”

Thus begins the story of yet another developer trying to convince their manager to adopt test-driven development (or any other methodology or technology). There’s a good chance this developer’s been reading all sorts of stuff on blogs (like those linked here) that have convinced him that salvation lies that way.

Don’t get me wrong.

There’s a good chance the developer’s right.

It’s just that that’s besides the point.

Developers and Managers

There’s a difference between how developers view a practice and how a manager (defined for the purposes of this post as someone in charge of delivering something) view that same practice. From a developer perspective, Ian’s point about unit testing is spot on:

“The problem is that the most important step is not doing it right, but doing it at all.”

Yet, as Ian himself points out in the title, this is a learning issue. If you want to learn to swim, there’s no replacement for jumping in the pool.

The manager’s perspective is a bit different.

Yes, we want our developers to improve their skill set. Yes, we understand that unit testing will ultimately improve quality. Yes, we know developers need to practice these skills as a part of their job. But, and it’s a big ole’ but, when it comes time to sink or swim, and we’ve got a deadline, those desires need to be balanced with delivering. Accounts of unit testing adoption efforts resulting in more (test) code to support with little apparent improvement in quality in the short term, well, they scare us. Arnon’s post gives more links supporting that feeling.

What’s a Unit Test anyway?

Is it any class that happens to have a TestFixture attribute on it?

If we are to “decouple” unit testing from good design, as Roy has described, that’s a not-improbable outcome. If the design of the system is such as there aren’t any real “units”, what exactly are we testing? Regardless of static or dynamic typing, replaceability of code, and other technological things, does the fact that all TestMethods in that TestFixture complete successfully mean anything? In other words, what did the test test?

It is clear that these tests cost something.

It’s more code to write. It’s more code to maintain.

The question is, what value are we getting from these “unit tests that any developer without design skills can write”?

The manager in me doesn’t like this return on investment.

By the way, TDD is as much the evolution of unit testing as the screw driver is the evolution of the hammer. But that’ll have to wait for a different post.

What’s Design Got To Do With It?

If you’re looking for the technical ability to write a test fixture and replace calls to other classes, then design has nothing to it.

If tests are to be valuable – design has everything to do with it.

The difficulty our developer is having unit-testing the system is a symptom of design problems. There’s a good chance that’s why he suggested a rewrite.

By the way, please do a search & replace in your vocabulary on the word “rewrite” with the word “redesign”. The code’s syntax isn’t the problem – it’s not the “m_”, camel case, or anything like that. It’s not that if the code was rewritten under the same design that all problems will go away.

Redesign, or do nothing.

The community’s been discussing the issues of coupling, interfaces, mocking, and tools at length in the context of testability. I won’t reiterate the debate here but I’ll tell you this:

If logic is duplicated, if the code is tightly coupled, if there is no separation of concerns, the unit tests will be useless – even if they “test” the class in isolation.

Cut the coverage crap

Metrics lie.

The fact that there’s a bunch of other code which calls 100% of the system’s code and doesn’t contain false assertions doesn’t mean that the code is high quality or doesn’t contain bugs.

In a well designed system, most “logic” will be contained in two “layers” – the controllers and the domain model. These classes should be independent of any and all technological concerns. You can and should get high unit test coverage on classes in these layers. Shoot for 100%, it’s worth it.

Testing domain models is all about asserting state. While using setters to get the domain objects into a necessary initial state is OK, setters should not be used beyond that. Testing controllers is primarily about interactions – mocks will probably be needed for views and service agents. Commands do not need to be mocked out.

Most other layers have some dependence on technology that makes unit tests relatively less valuable. Coverage on these layers is most meaningless. My guidance is to take the effort that would have been spent on unit testing these other layers and invest it all in automated integration tests. You’re likely to get a much higher return on investment. Much higher.

Much.

Everybody’s Right

Developers aren’t just born knowing good design, testing, or anything else. Universities, colleges, and most vendors do little to change that state of affairs. Books help, a bit, but when learning to swim, you’ve got to get your feet wet, and on the job training is, by and large, all there is. As such, lowering the barrier to entry is important.

Keeping in mind the Dreyfus model of knowledge acquisition, it’s not about “dumbing down” software development, it’s about bringing novices up to speed:

“In the beginning [novices] learn to recognize objective facts and features, relevant to the skill. Characteristic of relevant elements are that they can be recognized context-free, i.e. without reference to the overall situation. The novice acquire basic rules to follow, acting upon those facts and features. The rules are also context-free, i.e. no notice is taken to the surroundings. On account of this the novice feels very little responsibility for the result.” (emphasis mine)

Managers are ultimately responsible for the result.

Managers shouldn’t necessarily sacrifice their projects on this altar of learning. Organizations need to find ways for developers to safely practice these techniques as a part of developing their “human resources”. First of all, this needs to be communicated to everyone – that the organization understands the importance of these techniques, the desires of developers to adopt them, and the projects that need to be delivered.

Some projects may be allocated additional non-functional requirements: the software will be developed test-first, there will be at least 80% unit test coverage, etc. It can make sense to have developers spend some time on these projects after finishing one more delivery focused project and before going onto another one. As more developers become proficient with unit testing and design, the delivery focused projects can start to benefit from these skills.

It’s a gradual process.

The Important Bit

No matter how you go about unit testing, do periodic test reviews.

Just like code reviews.

That’s it.

Related Posts

Business Process Verification

Self documenting and Test-Driven Alien Artifacts

SOA Testing

“I am human, therefore I make mistakes. If I make mistakes, then I cannot assume that I will write code that has no mistakes. If I cannot write code that has no mistakes, then I must assume that mistakes are rampant within the code. If mistakes are rampant within the code, then I must find them. But because I make mistakes, then I must also assume that I make mistakes trying to identify the mistakes in the code. Therefore, I will seek the best support I can find in helping me find the mistakes in my code.”

Source: Ted Neward’s post: Welcome To The Shitty Code Support Group.

And I consider myself to be a top 10% kind-of-guy.

Imagine the kind of crap all those other people are putting out. Not you, the reader, of course. The fact that you’re reading my writing means you’re a top 10% kind of person too  

Having good design helps.

Helps decrease the impact mistakes in one part of the system have on other parts.

That way, when I’m working on one part of the system, I only have to deal with my crap. And that’s a good thing. Because I’ve gotten pretty good at fixing my crap over the years.

–

This movement is definitely a good thing, over all. Moving to a failure-oriented mindset will help us put the appropriate tools and processes in place to making more robust systems, maybe even Non-Stop Software.

After reading Derek Hatchard’s post, The Art and War of Estimating and Scheduling Software, I wanted to follow up on my previous post on the topic, Don’t Trust Developers with Project Management. The problem lies with individualistic thinking.

Developers, and managers too for that matter, by and large are concerned with “productivity”. Developers want the latest tools and technologies so that they can churn out more code faster. Managers create schedules trying to get the maximum efficiency out of each one of their developers. They consider resource utilization and other terms that sound manager-ish.

Fact is, on medium to large sized projects, if you look at the studies you’ll find that developer productivity when measured as total lines of (non-blank) code of the system in production divided by the total number of developer days comes in roughly at 6. Maybe 7.

7 lines of code a day.

Let that sink in for a second.

I can hear the managers screaming already. OMFG, what were they doing all day long?! It takes, what, 10 minutes to put out 7 lines of code? An hour even, if it’s complicated recursive code and stuff. And they say they don’t like us micro-managing them?! Now we know why. It’s because they’re goofing off all day long.

Well, managers, that’s not really the way it goes. You see, you have to take into account the time it took to learn the technology, tools, frameworks, etc. Add to that the time of understanding the requirements, which is really sitting through boring meetings that don’t explain much. Finally, our poor developer actually gets to implement the requirement. Maybe run the system a couple of times, trying out the feature they implemented, and checking the code in.

Well, that’s actually the easy part. Now comes the part which kills most of the time. After a bunch of features have been developed by the team, the testers start banging away at it and find a bunch of bugs. Now the developer has to reverse-engineer some bizarre system behavior and figure out which part of the system is to blame. That involves usually some educated guessing (unless they’ve just joined the team and have been put in the bug-fixer role to “learn the system”, in which case it is thoroughly UNeducated guessing). They change some code, run the system, which looks like its been fixed, check the new code in, and close the bug.

But the bugs keep coming. And as the project progresses towards production, more and more of the developers time is spent looking through code and changing existing code, that actually writing new code.

And the larger the system, the more bugs. And I don’t mean that the number of bugs linearly increases with lines of code, or number of features. It’s probably closer to exponential. If it’s a mission critical system, the performance bugs will be taking an order of magnitude more time to fix than other bugs.

So, as you can see, getting a system into production is a team effort. It includes the developers and testers, of course, but also management, and the customer, and how they manage scope. This is kind of a “duh” statement, but we’re getting to the punch-line.

If getting a system into production involves the entire team, isn’t that obviously true for each feature too?

In which case, why are we asking just the developers to estimate the time it takes to get a feature “done”? Why are we trying so hard to measure their productivity?

I know why. It’s so we can get rid of the less productive ones and give bonuses to the more productive ones!

Back to the main issue. I don’t “trust” developer estimates because I need to see the team’s capability to put features in production. The involves all aspects, and often many team members, in some cases multiple developers going through the same code. This involves all overhead and cross team communication, sick days, etc. It’s also why I try to get multiple data points over time to understand the team’s velocity.

While I care about the quality of my developers, and testers, and everybody on my team and would like them to be able to estimate their work as best they can, I’ve got a project to put into production. And the best way I’ll know when it’ll go into production is by having data that’ll enable me to state to my management:

“Our team is finishing 20 feature-units a month, we’ve got 200 feature-units to go, so we’ll be done in around 10 months.”

If I’m busy micro-measuring each developers estimates, I won’t have the time to see the forest. By first taking a harsh look at the reality of what the team can do, I can start looking for ways to make it better. Maybe the bottleneck is between analysts and developers, maybe we’re seeing the same bugs regressing many times, but until we know where we are, we can’t run controlled experiments to see what makes us better.

Focusing on the individual developer, getting them the latest and greatest tools may be great for their morale, but it probably won’t make a bit of difference to their actual productivity.

Next time – what to do when management asks you what it’ll take to be done sooner.

Jeremy’s trying to answer some hard questions about agile. I wanted to tackle the issue of fixed-bids, since most of my clients work on those kinds of projects and I managed those projects full-time before becoming a consultant.

So, here’s the thing.

The only way to win on fixed-bid projects, is to bid low, and then rack up the change-requests. This is why people spend so much time documenting requirements, and then getting the client to sign off. It’s so they can prove that something is an actual change request, and thus they don’t have to do it. So, if the client wants to do whatever, they have to pay more money.

The problem is that it pisses off the client.

There’s another, subtler problem. It’s that clients get wise to this game, and front-load every possible requirement requesting total flexibility in everything.

This leads to another problem. We can’t bid low anymore.

Which leads to another problem. The client doesn’t have the budget to pay for the longer list of requirements.

Which leads us back to square one.

Fixed bids are a lose-lose proposition.

You see, if you bid rationally, taking into account the fact that some requirements will change, others will appear mid-way through, and so on, you’re bid will be significantly higher than the other guy who low-balled it. That means that the client will have a very hard time explaining to his management why he wants you to do the project.

So, the only way to win is for the client to realize this and game the system. This is sometimes a fine-line, possibly bordering on illegal when it comes to government contracts.

Once you have a client who understands that the fixed-bid is not in their interest, they will work collaboratively with you to get a reasonable system out the door within the given budget. There will be a lot of give-and-take but it can work. After a system goes into production successfully, it’s a lot easier to get management buy-in for the next version.

Fact is, upper management doesn’t really know all the specific requirements. So, if you don’t do them all, you’re OK, and so is your client.

In these circumstances, agile development is not only possible, but likely.

I know that it’s not really fixed-price, fixed-time, fixed-scope this way. But that’s what makes it successful

What with all the warm and fuzzy feelings around trusting developers (here, here, and here) I just have to tone it down a bit. The title takes it a bit far – but less than you might think. Just today I had a talk with one of the team leads on the project I’m consulting on. It boiled down to this:

Developers don’t know how to estimate.

Or more specifically, the variance in the actual completion time of a feature from the estimate given by a developer increases probably exponentially with time.

For example, if the estimate is a day, you can expect it to be finished in around a day. If the estimate is a week (5 works days), it will probably vary between 4-10 work days. If the estimate is a month, in all actuallity the developer probably doesn’t know enough to say but will answer when pressed.

This is why Ron (the team lead) asked me if I wasn’t worried I was putting myself in a lose-lose situation by changing the project structure. There were two “teams” when I came in – developers and testers. All the team leads had “committed” to “finishing” the project in 6 months. When I originally proposed the change to more feature-driven teams, mixing skilled and newer developers and testers together on the same team, came the cry:

“It’ll take us twice as long this way.”

“It’s so much less efficient than before.”

And on, and on. What was funny to me was that 3 of the “6″ months were already gone and not a single feature worked. We were half “gone”, and nowhere near half done.

The thing is that Ron was sure I was cooking my own goose with upper management. What he, and most other developers don’t know is that upper management has gotten used to the state of things. If developers say one month, management has seen enough history to know that it’ll really be 3-4 months. So when I come in and do things different from what developers are used to, upper management is thrilled – that’s why they brought me in the first place.

The difference is that by working based on features, and measuring project progress by feature-units completed per iteration, I drive down variability. This creates solid data about progress saying when we’ll be done (more or less). This is quite different from the “normal” course of many projects:

“OK, so it’s been 8 months now on your 6 month project. When will you guys be finished already?”

Without the data, your only strategy is hope: “Umm, I hope the developers will be done this week(?)”

Don’t get me wrong. I trust my developers and testers deeply. But it’s not their job to know how to estimate and manage projects. PMs who take developers estimates as is and stake the project on that being correct are setting themselves up for failure – with only themselves to blame.

Now back to your nice warm-and-fuzzy blogging…

How much, and what kind of documentation do we need to create even if we have “self-documenting code”? Or is that kind of code enough all by itself? I for one yearn for the day where the code really will be enough, and I think that Scott and Ayende do too. First of all, I think that as time progresses, the size of systems that the average developer works on is increasing substantially. And the larger a system is, the more we find a kind of code/design dialect that developers use to talk about that system. I think that these dialects are significantly different between framework/open source code, and application code. So, when a new developer comes in, do we tell them to “just read the code”? Or if the application has gone into production some time ago and now needs to be enhanced but the original team is long gone, what can be done?

What I usually suggest (and practice) is to have some kind of documentation explaining “language” of the system – what things go together, how, and why; and just as importantly, what things must be kept apart. This can be a Software Architecture and Design document, or even videos of the design sessions where things were first discussed. The relevant requirements should be a part of this as well. We need to know that the reason asynchronous messaging was used was for the strenuous scalability requirements, otherwise we might start adding the high-productivity Visual Studio Web Services we like so much.

Continuous integration is a boon to projects who use them. But developers who are only familiar with building solutions in Visual Studio may not be used to working that way, keeping code checked out for weeks at a time.

Test-Driven Development and Architecture will help those who know about it. We should probably write something up about how the system should be enhanced. Side-effect or not, developers and testers will need to know how we test the system – what tools are used in which way and at which stage of the development lifecycle. We could reference existing methodologies as well as in what ways we deviate from them and why.

What sustainable business value do we provide by leaving behind us alien artifacts and practices? I’m not saying not to use state-of-the-art designs, techniques, and practices. On the contrary, I think that they are the key to sustainable business value. Necessary, but not sufficient. Documentation, of whatever flavor you find most suitable to each specific thing you are documenting, should be considered by the team as a whole, and the conclusions presented to stakeholders. The famous Chrysler C3 project’s long-term business value is shaky as the result of those stakeholders decision that documentation wasn’t necessary – in no way was the Agile process faulty in that respect.

Finally, you may be surprised the holes you find in your own thinking when forced to write down and explain to someone just coming in “how we do things around here”, I know I was. I can tell you that in the cases where I did that exercise, several stupid, costly mistakes were avoided. Fleshing out your thinking isn’t necessarily big design up front (BDUF), it’s just smart.

Over on the Real-World Software Architecture blog I just read a great post, “I want what I want when I say I want it”. Does this sound like one of your customers/managers?

I wonder if these people go into Burger King, order a # 5, and finish the order by saying, and I would like that done in 60 seconds. Then do they proceed to ask every 10 seconds if the order is done? When they hit the 60 second mark do they start screaming at the top of their lungs “I told you I wanted that order in 60 seconds. If you don’t have it to me in the next 15 seconds I want my money back and the order for free.”?

I recently had the pleasure of working with a customer who “got it”. First of all, they understood that they couldn’t really set scope, time, and cost. Or rather, if they did, then quality – the main property that they had the least power over, would suffer. This could manifest itself in terms of an ugly or unusable user interface, or poor scalability – resulting in sky-rocketing operations costs. In short, they understood that in order to get what they really wanted, they had to give up “control”, or rather the illusion of control. By working collaboratively with the development team, they were able to get a good looking and workable prototype out in time for a tradeshow, as well as do a load test early in the project to see that the architecture was scalable.

Risk is an inherent part of all projects, to a lesser or greater degree. We need to manage risk by including risk-reduction activities (like load tests) into the project schedule. Some customers don’t understand that. It is therefore our responsibility to explain that, although this isn’t a value-creating activity, it is still necessary – or rather what cost could manifest later as a result of ignoring the risk (sky-rocketing operations costs).

But at the end of the day, the buck doesn’t stop here, I’m afraid. It’s not the development team’s money. If the customer chooses to make decisions that increase project risk, it is their choice. It is our responsibility to make them aware of the ramifications of their choices. It is good practice to document these decisions so that if (when) they come back to bite us, we can show that we communicated the possible outcomes ahead of time. I can tell you that it is a huge relief when you have an email record to counter “This system is slow. I thought you knew how to make a system that could handle this many users! Now you’re going to tell me that you have to take down the whole system to fix it – do you know how much that costs?!”

I don’t know why customers behave these ways, I just know that they do. I also know that I have to protect myself and my team from it. But, so as not to end on a doom-and-gloom note, I have found that by communicating more with my customers, I find out all sorts of interesting things. For instance, it’s not that they don’t want us to load test, it’s just that they don’t want us to do it now – they need to get something ready for the tradeshow. They just didn’t think it was important to tell us that.

So, reach out. I know I was surprised by the amount of illogical behaviour that disappeared once I started communicating, despite the customer’s apparent inapproachability. It just might save your project, or your career.

Responding to change – it’s one of the primary agile values. This topic came up over lunch with Sébastien in a rather round about way. Sébastien mentioned that, in French, when his clients communicate with him (and he with them) using the informal address “tu” he finds himself more obliged to comply with their requests than when using the more formal “vous”. The agilists quickly chimed in on the importance of responding to change but my fixed-price background wouldn’t let that go.

For starters, let me say that I’m all for giving the greatest value possible to a client. However, when I’ve got a schedule that can’t slip and can’t recruit a larger team (for whatever reason) and my client asks me if we could also handle some new XYZ scenario, I sometimes answer “no”. My agile friends were even more astonished when I told them that quite a few of my customers respect me for that answer. Just so you understand, the long version of that “no” is:

“Since we can’t slip the schedule any, and we’ve already done away with all the features that could have been cut, and we can’t bring in any new people, I really can’t promise you that if we try to do that additional work the project will stay on track.”

With clients with whom I’ve worked for quite a while, I sometimes use the more glib “what? You mean, for free?”, and we usually have a good laugh over it.

Anyway, bottom line is that change costs. Somebody has to pay, sometimes a little, sometimes a lot. If you’re a week or two from shipping, dropping a feature to “make room” for the new feature might not help. So much work may have already been done that to take the old feature out might be more work than leaving it in.

One big thing that I will give to agile over the fixed-price projects is that by forcing customers to define up-front all requirements, and “threatening” with a high cost of subsequent change, customers are made to put in many requirements that they think that they just might need. This is scope creep even before any work’s been done. By allowing the customer to change their mind during the project you are, in one feel swoop, decreasing (often drastically) that scope of the project making it cheaper and easier to give high value.

You’d be surprised by the number of times that clients are just trying to figure out if the cost of additional feature XYZ now is outweighed by its value. Giving an unqualified “yes” short-circuits the necessary discussions around risk that keep the business in the driver’s seat on the project. Sometimes “no” really is the best answer.

After all contestants presented their solutions in the Iron Architect contest at TechEd Developers in Barcelona, and the judges left the room to deliberate, I put a question to them:

“When can you have your solution in production and how much would it cost?”

You could see the shock on their faces. Architects weren’t supposed to answer these kinds of questions, were they? That’s what project managers are for, right? Eventually, they settled on one month for the schedule question, with only one developer. That’s one month in terms of calendar time. I was shocked by that.

Unless you’re Superman, in the span of one month you cannot learn an existing system, figure out all the new requirements, design, develop, test, and deploy, debug, etc, etc, and keep stakeholders happy and in the loop throughout the whole process. And it doesn’t matter how many developers you have.

And on that point, like you’re really going to be able to find a developer who’s any good at the drop of a hat, get them to leave whatever they’re doing for a month, for a short term project like that.

This topic came up in the speaker dinner when I was talking to Pooya and Jurgen (one of the judges). Architects need to be the glue that connect everything together – business, management, technical, test, operations, etc. They need to fully understand project lifecycle issues. When business comes along with a new requirement, the architect is the one who says if it can be just slipped in or needs to be separately budgeted – not the PM. The PM may have the final call, but the architect is the one who provides the information about the ramifications of changes to the project.

Money? Schedule? All in a day’s work for an architect.