So, don’t try to change the organization. It’s too big anyway.
Instead, focus on changing one person at a time – that’s hard enough.

Don’t necessarily take the “one person as a time” too literally, though.
You don’t need to completely and utterly have one person won over before starting on the next.

Understand that for someone to change, that may require them admitting (either implicitly or explicitly) that the way they were doing things before was wrong. In some organizations, this can be suicide. Even if it isn’t, psychologically speaking, there are a huge number of barriers to overcome.

So, if at all possible, massage the situation in such a way that it’ll sound like they were right all along, and no-one really understood. It’s easy for someone to play along with the “misunderstood genius” story.

Next time – how to do just that.

Stay tuned.

Since that post I’ve blogged about many techniques and approaches to identify better boundaries (like with SOA and DDD) and I’m seeing more and more developers starting to apply them.

This post will be slightly different.

You see, occasionally we technical people will get requirements that can’t easily be broken down by functional boundaries. Sometimes the business calls this a “platform” – here’s an example:

“We want a flexible, customizable workflow-driven platform that allows end-users to add their own columns to any screen able to support massive datasets for large enterprise customers that will also be intuitive and easy to use for our SaaS push to small and medium-sized businesses (SMBs) – oh, and then it needs to be multi-tenant too. Did I mention that we promised this would be ready for our most important client by the end of the year?”

There is only one reasonable answer to the above:

“I know you want it but, I’m very sorry, you can’t have that. It isn’t possible to do that with one system.”

It’s really rare for a technical person to say something like that. The simple reason is that in software, we believe that almost everything is fundamentally possible – given enough time/money/resources. So, when someone in business comes to us with the requirements above, we say “It’s *possible*”, loosely translated to “We can’t prove that that’s impossible”.

There are many reasons why you can’t be SaaS and Enterprise at the same time – not all of them dealing with software. The marketing, sales, and support stories of each of those markets is VERY different. In the enterprise you’re usually working with professional services people that customize the generic product – which then becomes a backwards-compatibility requirement for new development. This will hinder the development team’s ability to roll out the new shiny features needed to remain competitive in the SaaS space.

In the cases where I’ve been brought in to help clients with these kinds of systems, I try to work my way up to the person in management who is in charge of the project/product – often the CEO. Then, in the nicest way possible, I explain that really the only way to have your cake and eat it too is to create 2 companies – each one focused on its own space – Saas and Enterprise; each one with its own development team, feature set, release schedule, etc.

There’s a reason that there’s an SAP *and* a Salesforce – it’s because no one can be all things to all people. It’s hard enough to become a market leader in just one space. Trying to do both is VERY expensive, and increases the chance of the project failing from the average 60-70% in the industry to probably about 99.9%.

Hope that will save you some grief.

Unfortunately, there is one tiny little mistake that I see almost everywhere that gets in the way, and that’s going to be the topic of this post.

The Problem

Let’s say you have a standard web app environment – some web servers, application servers, and a database server. Your web servers need to send messages to the application servers. So far, so good.

In your test environment, you have an application server called AS_01_Test, and your web servers are configured to send it messages. However, in your staging environment the application server fulfilling that same role is called AS_01_Stage. This creates a configuration problem – you need to change the config of your web servers as you move the web app from Test to Staging.

I’ve seen companies doing all sorts of creative things to get around this problem – some of them involve putting all configuration settings in a database so that they can be centrally managed and visualized. I’d like to suggest an alternative approach.

What if…

What if server names were the same across all environments?

Well, you wouldn’t need to change configuration as you moved the system between environments. That’s a good thing.

But how can that be? Wouldn’t there be a conflict if there were two machines with the same name?

The answer is that there wouldn’t be a conflict if the machines were on different networks. Not all machines have to be on the same network. We can set up as many networks / virtual networks as we like. And it is clear that we don’t need machines in one environment / network to talk to machines in another environment. I mean, under no circumstances would we want web servers in our test environment to talk to application servers in the production environment.

These separate networks provide much needed isolation, beyond solving the server naming problem.

In closing

It’s really a tiny thing when you think about – multiple networks. But that’s exactly why software developers overlook it so often – because it’s not a “software solution” to the configuration problem we perceive as a “software problem”.

I wrote about related multi-environment configuration issues in this earlier post: Convention over Configuration – The Next Generation

I’m happy to say that this functionality is now in NServiceBus called “profiles” and you can read more about how they work here.

How are you handling the flow of moving software through to production? Leave your comments below.

One of the things that I’ve been trying to do with my presentations around the world on CQRS was to explain the why behind it, just as much as the what. The problem with the format of these presentations is that they’re designed to communicate a fairly closed message: here’s the problem, here’s how that problem manifests itself, here’s a solution.

In this post, I’m going to try to go deeper.

The hitchhiker’s guide to the galaxy

In this most excellent book, one of the things that struck me was the theme that made it’s way through the whole book – starting with the answer to life, the universe, and everything: 42. By the time you get to the end of the book, you find out that the real question to life, the universe, and everything is “what do you get when you multiply 6 by 9″. And that’s how the book leaves it.

To us engineers, we can’t just accept the fact that the book would say that 6*9 = 42 when we know it’s 54. After bashing our heads on the rigid rules of math, we realize that not all math problems are necessarily in base 10, and that if we switch to base 13, the number 42 is 4*13 + 2 = 54. So, the book was right – but that’s not the point.

What’s the point?

The hitchhiker’s guide is an example of a teaching technique which presents an apparent paradox, leaving the student to dig up unspoken and unthought assumptions in order to resolve it. Key to this technique are rigid rules which do not allow any compromise or shortcuts on the student’s part.

The purpose of this technique is not for the student to learn the answer, but to gain deeper understanding, which in turn changes the way they go about thinking about problems in the future.

So, when given the problem 4*5, we do not just immediately answer 20, instead we clarify in which numeric base the question is being phrased, and only then go to solve the problem. In base 13, the answer would be 17. In hex, the answer would be 14.

The externally visible change is that we know which questions to ask in order to arrive at the right answer – not that we know the answer ahead of time.

Making an “ass” out of “u” and “me”

Let’s start at the end – one of the unspoken assumptions that has been causing problems:

All businesses can be treated the same from the perspective of software.

In our previous example, we assumed that all math problems use base 10. It turns out that different bases are useful for different domains (like base 2 for computers). We can say similar things about degrees and radians in geometry. The more we look at the real world, the more we see this repeating itself. There’s no reason that software should be any different.

Base 10 is not a ubiquitous best practice. We shouldn’t be surprised that there really aren’t best practices for software either.

Here’s another problematic assumption:

“The business” can (and do) tell us what they need in a way we can understand.

So many software fads have been built on the quicksand of this assumption. OOAD – on verbs and nouns. 4GL and other visual tools that “the business” will use directly. SOA – on IT business alignment. I expect we haven’t seen the end of this.

Some of you may be wondering why this is false, others are sagely nodding their heads in agreement.

The myth of “the business”

Unless you have a single user, who is also the CEO paying for the development, there is no “the”. It’s an amalgam of people with different backgrounds, skills, and goals – there is no homogeneity. Even if no software was involved, many business organizations are dysfunctional with conflicting goals, policies, and politics.

To some extent, we technical people have hidden ourselves away in IT to avoid the scary world of business whose rules we don’t understand. With the rise in importance of information to the world, we’ve been pulled back – being forced to talk to people, and not just computers. Luckily, we’ve been able to create a buffer to insulate ourselves – we’ve taken the less successful technical people from our heard and nominated them “business analysts”. No, not all companies do it this way, but we do need to take a minute to reflect on how information flows between the business Mars into and out of the IT Venus.

On human communication

Even if we made this insulation layer more permeable, allowing and encouraging more technical people and business people to cross its boundary, we still need to deal with the problem of two humans communicating with each other. There are enough books that have been written on this topic, so I won’t go into that beyond recommending (strongly) to technical people to read (some of) them.

Rather, I’d like to focus on the environment in which these discussions take place. IT has been around long enough, and users have used computers long enough, that a certain amount of tainting has taken place. If the world was a trial, the evidence would have been thrown out as untrustworthy.

When users tell you what they want, they’re usually framing that with respect to the current system that they’re using. “Like the old system – but faster, and with better search, and more information on that screen, and…”

At this point, business analysts write down and formalize these “requirements” into some IT-sanctioned structure (use cases, user stories, whatever), at which point developers are told to build it. Users only know what they didn’t want when developers deliver exactly what was asked.

How can that be?

These are not the “requirements” you are looking for

Users ultimately dictate solutions to us, as a delta from the previous set of solutions we’ve delivered them. That’s just human psychology – writer’s block when looking at a blank page, as compared to the ease with which we provide “constructive criticism” on somebody else’s work.

We need to get the real requirements. We need to probe beyond the veneer:

• Why do you need this additional screen?
• What real-world trigger will cause you to open it?
• Is there more than one trigger?
• How are they different?
• etc, etc, etc…

This is real work – different work than programming. It requires different skills. And that’s not even getting into the political navigation between competing organizational forces.

But let’s say that you don’t have (enough) people with these skills in your organization. What then?

Enter CQRS

CQRS gives us a set of questions to ask, and some rigid rules that our answers must conform to. If our answers don’t fit, we need to go back to the drawing board and move things around and/or go back to “the business” and seek deeper understanding there.

For each screen/task/piece of data:

Will multiple users be collaborating on data related to this task?
Look at every shred of raw data, not just at the entity level.
Are there business consistency requirements around groups of raw data?

If “the business” answers no – ask them if they see that answer changing, and if so, in what time frame, and why. What changing conditions in the business environment would cause that to change – what other parts of the system would need to be re-examined under those conditions.

After understanding all that and you find a true single-user-only-thing, then you can use standard “CRUD” techniques and technologies. There are no inherent time-propagation problems in a single-user environment – so eventual consistency is beyond pointless, it actually makes matters worse.

On the other hand, if the business-data-space is collaborative, the inherent time-propagation of information between actors means they will be making decisions on data that isn’t up-to-the-millisecond-accurate anyway. This is physics, gravity – you can’t fight it (and win).

The rule for collaboration

Actors must be able submit one-way commands that will fail only under exceptional business circumstances.

The challenge we have is how to achieve the real business objectives uncovered in our previous “requirements excavation” activities and follow this rule at the same time. This will likely involve a different user-system interaction than those implemented in the past. UI design is part of the solution domain – it shouldn’t be dictated by the business (otherwise it’s like someone asking you to run a marathon, but also dictating how you do so, like by tying your shoelaces together).

Many of the technical patterns I described in my previous blog post describe the tools involved. BTW, hackers can be considered “exceptional actors” – the business actually wants their commands to fail.

In Summary

The hard and fast rule of CQRS about one-way commands is relevant for collaborative domains only. This domain has inherent eventual consistency – in the real world. Taking that and baking it into our solution domain is how we align with the business.

The process we go through, until ultimately arriving at one-way-almost-always-successful-commands is business analysis. Rejecting pre-formulated solutions, truly understanding the business drivers, and then representing those as directly as possible in our solution domain – that’s our job.

After doing this enough times and/or in more than one business domain, we may gain the insight that there is no cookie-cutter, one-size-fits-all, best-practice solution architecture for everything. Each problem domain is distinct and different – and we need to understand the details, because they should shape the resulting software structure.

The next time the business tell us to implement 42, we’ll use CQRS along with other questioning techniques until we can get “6 x 9″ out of them, learning from the exercise what are the significant and stable parts of the business – ultimately helping us to “build the right system, and to build the system right”.

Don’t Panic

Java takes the enterprise

When Java originally came out, it was an interesting language that you could use to write applets – code that would run the same everywhere, in the browser, on the desktop, etc. SUN was the keeper of Java. And then came this concept of a container – the thing that would run your Java code, which then grew to handle things like transactions, and became the Enterprise Java Bean – EJB, and that came out of IBM, with SUN adopting it later.

The adoption of Java at that point was important enough that the specs were opened up, and many EJB technologies blossomed. With backing from big companies already inside the enterprise, the only possible fight came from COBOL around Y2K, but that was a dying gasp. Microsoft wasn’t in the game as Windows NT wasn’t competition for UNIX or mainframes.

Multi-vendor as a way of life

With multiple big and medium-sized vendors offering similar, competing, and complementary technologies, all tied together by the promise of backwards compatibility in Java and the specs demanding interoperability, customers could safely go for best-of-breed solutions. This forced technological decision makers to truly evaluate the offerings on merits, not just lineage.

Many attribute the rise of OSS in Java to the fact that the existing containers were so heavyweight. I believe that was a secondary effect. As a result of the fact that the industry had embraced and internalized the values of thinking and choosing for itself, it was willing to look at alternatives with much humbler lineage, ultimately using them on their merits. It was the culture.

The Microsoft ecosystem

This culture was practically nonexistent on the Microsoft side of the border. As the only vendor, Microsoft was put on a pedestal – it was the best, period. The industry hungrily looked to Seattle not only for technology, but also for guidance and leadership. If a developer could get a job at Microsoft, they were “hot stuff”, the best of the best. This isn’t a bad thing – it was just a thing.

This enabled technological decision makers on the Microsoft side to have much shorter thought and decision processes than their counterparts on the Java side.

All of these things got baked into the culture.

About Microsoft

Like all that were ever on a pedestal, the fall was a matter of time. Expectations being that high, it was inevitable. You can’t make all people happy all the time, and the conditions in the industry were changing, and the company had to change to remain competitive.

Let me say this clearly: Microsoft was not at fault.

Sure, it’s easy to say in retrospect they should have communicated more clearly about this, or built that technology differently. If you haven’t yet worked in a big company, you may not know this, but big companies aren’t just bigger small companies. It’s a hodge-podge of competing agenda, initiatives, politics, people, and power. There’s a saying that things only get done despite the organization’s best efforts.

For a company Microsoft’s size, what they manage to get done is incredible.

On acquisitions and OSS

Microsoft has come under fire over the years for offering their own implementations of open source technologies- as if the vendors on the Java side didn’t do this. The Java world was ultra competitive, the big vendors would eat promising upstarts in order to win back lost contracts to key customers. This made the technological decision makers broaden their thought processes to include risk management as a part of managing their technological portfolio. To a large extent, this actually justifies the existence of a C-level role related to technology – the CIO.

Chief Officers of Information and Technology

I found it interesting to see the difference in age, experience, background, and thought processes between people holding the CIO title at organizations that were Microsoft-centric and those with a more heterogeneous technology investment. This was likely influenced to a large extent by the history of technological evolution, age and size of organizations with the resulting culture and hiring practices, among other things. This pattern continued with the CTO as well.

Obviously one wouldn’t expect the same thought processes in the CTO of a 20 person IT shop and the CTO of Ford Motor Company (for example). They shouldn’t be the same.

It appeared that as Microsoft became more focused on innovation they started listening more to the technology leaders of smaller companies, not a bad thing by itself. Choosing A means not choosing B, and in order to stay competitive, a choice must be made.

Fast forward

I think that what happened was necessary, and will be good for the industry and Microsoft. Technological decision making in companies that were traditionally Microsoft-centric has evolved. This has clarified Microsoft’s role as a platform vendor who can be trusted, and whose tools can be used or not used as the situation dictates, with comparable commercial and OSS tooling evaluated on the same criteria.

Just as IBM reinvented itself and now occupies a sustainable role in a combined commercial and OSS ecosystem of platforms, tools, and services, Microsoft appears to have made several big strides partnering with the community in much more productive ways, yet with more strides to be made as well.

A challenge to OSS on the Microsoft platform

In this new and more mature environment, OSS can’t remain the same either. Some code a developer whipped up in their free time and put in an online accessible repository with a decent license just won’t cut it any more.

In my previous post I called out the Linq2Sql support story – the same goes for OSS. Active development is required, and so is support, and so is documentation. The commitment needs to be much more serious.

Also, until usage reaches some critical mass, it is unlikely that a single developer or even a small group of committers will be able to do it without the help of the community. Really the only alternative is for there to be some commercial story that can fund it – support, consulting, training, commercial add-ons, etc. A combination of community (both dev and use) and commercial cash-flow is probably most sustainable.

If you are running an OSS project, understand that these criteria will be used to evaluate it.

In closing

I think I’ve managed to alienate previous supporters from all sides.

I believe we are entering some interesting times, where not only are vendors and OSS projects being evaluated differently than in the past, but that traditional architectural paradigms are changing as well.

Regardless of what the answers are, I’m happy that more of us are asking more questions. Some questions are the right questions, some are the wrong ones, and sometimes we just ask at the wrong time, but as an industry I think that we’re getting better.

Thanks for reading.

Although I have been privy over the years to see behind the veil at Redmond, I will be focusing primarily on the externally visible actions of Microsoft and how the industry has reacted on average – specifically in the enterprise space, where I spend most of my time.

What managers are concerned about

When there will be problem with this technology, who will support us?
How will that change when the author of the technology moves on or loses interest?
How long can we expect to be using this technology?
How long will it take to learn it? When will we recoup our investment?

Traditionally, for companies on the Microsoft platform, choosing Microsoft as their primary technology vendor has been the safe bet. As a large company, Microsoft can afford to employ support engineers. These engineers are different from the ones who wrote the technology to begin with. Also, Microsoft has 10 year support guarantees. As a result, companies expected to use the technology for at least that long. With their size, Microsoft also had the ability to create copious amounts of documentation easing learning.

To turn the common phrase, nobody got fired for choosing Microsoft.

Open source seemed like risky business, at the time.

What changed?

It started with the Composite Application Block – CAB.

This technology was put out by the Patterns and Practices (P&P) group at Microsoft. After it came out, the gears of big marketing machines at Microsoft got to work, telling the industry about this great new thing at conferences, and the Microsoft Consulting Services (MCS) folks started using it with clients. CIOs at large companies sent developers to training on CAB by the dozen.

And then CAB was dead. Sorry. Microsoft prefers the word “done” to dead.
Just like Don Box said that COM wasn’t dead, it was done.

“What about that 10 year support thing?”, asked the CIOs (among many others).

You don’t understand, said Microsoft. You see, P&P are not a product group in Microsoft. They don’t have the resources to provide that kind of support. And the rest of the company isn’t obliged to support what they put out – since they’re not a product group.

You could literally hear the collective jaw of the Microsoft part of the industry drop.

This wasn’t supposed to happen (like housing prices in the US).

And then came .NET 3.0, and things seemed to go back to normal.

There were lots of wonderful things that came with it – like LINQ, and…

Linq to Sql

This was BIG.
Finally – after ObjectSpaces was promised at PDC ‘03 and later shelved, then WinFS (same story), it had arrived.
The object-relational mapper (ORM) from Microsoft.

The marketing machine went into high gear – it was the v3 promise. Linq2Sql (L2S) came from a product group. This was serious. It was shown at conferences and user groups all over the world. Developers were sent to be trained on it.

And then Entity Framework (EF) was announced – and L2S was done.

CIOs were rubbing their eyes in disbelief. “But this is from a product group – you have to support it, right?”

Well, said Microsoft, yes. We will support it with our support org. It’s just that we won’t continue to develop new features for it in the product org. We actually have a different sub-org that will be working on EF and they’re under a different division (SQL Server) than the sub-org that made L2S.

Jaws were hitting the pavement. All that investment erased. Just like that. It turns out that support without active development doesn’t mean very much, no matter how many support engineers are involved.

But the industry picked itself back up, and got back to work on the new foundational pieces that came with .NET 3.0. If Microsoft is calling it “Foundation”, that must mean they’re committed to it.

And then came Workflow Foundation

Workflow Foundation (WF) was a dream-come-true. At last, Model-Driven Development (MDD) had come to the Microsoft platform. Drag-and-drop on a whole new level. Imagine the reuse. Imagine the maintainability. Programming at higher levels of abstraction. A marketers dream. This was the culmination of previous efforts of Whitehorse in VS2005 and the Software Factories Initiative – or so we were told.

And then came .NET 3.5 – and the new WF wasn’t backwards compatible with the old one.
And then it happened again with .NET 3.5 SP1, and again with 4.0 (no more state machine – no wait, it’s back again, but not in the box).

All those companies that had long-running workflows in production needed to manually migrate them each time.

But this had become old news to the folks using Microsoft in the enterprise.
With Microsoft – you really can’t be sure any more.

What about open source?

Well, it was beginning to look more and more stable in comparison.
Especially the larger, more established projects. Those with active development.
Log4Net. NHibernate. Castle. etc.

There was also the fact that most enterprises were heterogeneous anyway – doing both Java and .NET development. Open source tools and frameworks were common in the Java space, politically greasing the wheels for .NET OSS in those organization.

Boasting features and capabilities several years ahead of what was coming out of Microsoft, more companies gave them a chance, and were pleasantly surprised. And the virtuous cycle of OSS gained speed. With more use, they become even more stable and got even more features, driving yet more use. Blog posts about them bloomed all over the web. User group presentations were given. At a presentation I gave at TechEd Europe 2006, I used NHibernate in my demo.

OSS had crossed the chasm. No, not everybody used it, or knew of it, but a critical mass of the industry had grown to depend on it.

Microsoft’s actions over the years had done more for OSS adoption than I think many would have imagined.

On Pub/Sub, Messaging, and SOA

Interestingly enough, when the first version of WCF was still in the oven (then called Indigo) there were discussions on whether it would support publish/subscribe messaging. Here we are, 3 versions later, and still no pub/sub, but the discussions continue

Message brokers were always important for enterprises in the Java space – IBM MQ; Tibco RV; Sonic; companies were paying millions for this stuff. Microsoft had MSMQ – finally at v3 with XP and Server 2003, but still with insignificant penetration to the market. BizTalk did have a good run, though, but not so much as a message broker, more as an integration and orchestration engine, unfortunately coming late to the Enterprise Application Integration (EAI) party.

Later, the SQL Server guys came with Service Broker – messaging in the database. But you could see that their heart wasn’t in it. The API was clunky. It still didn’t have pub/sub. There was no binding available for WCF.

After some time making noise in the SOA space with Oslo, that changed as well. Oslo is now Sql Server Modeling.

I imagine that some of the adoption pick-up with NServiceBus can be attributed to the vacuum Microsoft left behind when exiting the messaging/pub-sub/soa space. The way NServiceBus aligns with the principles found in the corresponding Java technologies makes it very palatable to enterprises working with both platforms.

The fact that there’s active development, a vibrant and growing community, and even training available definitely contribute as well.

In closing

I don’t fault Microsoft for any of this. There are a million things that they could have done. Choosing to do one thing means choosing not to do many others. The decisions they made were done with the best intentions. Hindsight is 20/20 of-course.

And that’s just it – we do need to take a look back.

If you’re a manager making a technology related decision, or are working with managers in those positions, knowing the history of today’s technology can give you a more accurate representation of the risk involved in each choice. Also, understanding the vector that Microsoft has decided to take in various areas is critical, especially if you find out that your architectural choices aren’t quite aligned with some of those vectors.

In this post, I’ve tried not to take a stance on whether a certain approach (ORM, Pub/Sub, etc) is good or bad, or even getting into which cases it’s appropriate or not – just to describe Microsoft’s externally visible behavior in that space.

I hope that this short history lesson can help your organization make the right technology decisions in the future for its specific context. Your comments and thoughts are most welcome, as always.

Before getting into that, I want to start with a slightly broader scope of discussion.

You see, I get asked about “best practices” on all sorts of things. And I try not to be the kind of consultant that responds with “it depends”, but the context of the question often makes the answer irrelevant. And the unspoken context of a best-practice question is:

Given infinite time and budget

The biggest problem that I see with well-intentioned, best-practices-following developers and architects is that they don’t ask the question “is this the right thing for us to be focusing on right now?” Understandably, that is a difficult question to answer – but it needs to be asked, since you don’t have infinite time or budget to do everything according to best practices (assuming those even exist).

About testing

The biggest issue I have with the “design for testability” topic is the extremely narrow view it takes of the word “testability”, usually in the form of more code written by a developer which invokes the production code of the system, also known as “unit tests”.

There are many different kinds of testing – unit, integration, functional, load, performance, exploratory, etc… where some may be automated and others not. Should we not discuss what “design for testability” means for not-just-unit-testing?

And what’s the point of testing anyway?

It’s not to find bugs.

Research has shown that testing (of all kinds) is not the most effective way of finding bugs. I don’t have the reference handy but I’m pretty sure that it’s from Alistair Cockburn’s work. Code reviews are (on average) about 60% more effective.

Don’t get me wrong – testing can provide indications that the software has bugs in it, but not necessarily where in the code those bugs are.

The purpose of testing is to provide quantitative and qualitative information about the system that can help various stakeholders in their decision-making processes. The relevance of that information indicates the quality of the testing. Here are some examples:

• The system supports 100 concurrent users, with the expected user-type distribution (X% role A, Y% role B, etc), performing expected use-case distributions, and collaboration scenarios.
• Time to proficiency for new users in role A is expected to be 3 days
• Alternate #2 of use case #12 fails on step #3

As you can see, the relevance of the above information is dependent on what decisions the various stakeholders need to make. The bullet on load can help us decide if more machines are needed or if developers need to tune the performance of the systems. The bullet on time to proficiency can help us decide if larger investment in usability is required. Information like the last bullet can be used in conjunction with the first two to decide on the timing and type of a release.

The timeliness of this relevant information is critical to the success of a project.

Choosing which and how much of the various testing activities to perform when is something that needs to be revisited several times throughout the lifetime of a project, taking into account the current risks (threats and probabilities) and time and resource investment to mitigate them.

Let me reiterate – we’re not going to have enough time to do everything.

On iterations

If the only part of your organization that is doing iterations are your developers, you’re not agile.

In order to capitalize on the information that testers are providing, you need them in your iterations.

The same goes for the other roles involved in the project – business analysts, DBAs, sysadmins, etc.

I know that 99% of organizations aren’t structured in a way to do this.

I never said doing this would be easy.

On design

Figuring out what kind of design and how much to do when is just as important, and just as hard. Design for testability is one part of that, but not the only one, or necessarily the most important one at any point of time.

Within that design for testability topic is the “design for unit-testing” sub-topic which seems to be the popular one. Before getting into the design aspects of it, let’s take a closer look at the unit-testing side of things.

On unit-testing

The assumption is that having more unit tests will lead to a code-base with less bugs, thus requiring shorter time to get the system into production, which will pay back the time it took to write those unit tests to begin with.

In practice, what tends to happen is that as development progresses, testing code breaks as the structure of the production code changes. Now one of two things happens – either the testing code is removed or rewritten. In either case, we didn’t get the return on investment we expected on the first bit of testing code. Unfortunately, rare is the case where the relevant people in the organization understand why, resulting in the same situation repeating itself over and over again.

Those projects would have been better off without unit testing, though the organization as a whole might have used those experiences to learn and improve. It’s been my experience that if the organization wasn’t conscious enough in the context of the project to notice the situation, it is unlikely to do so at higher levels.

On fragile unit tests

The reason that a unit test ends up being rewritten (or removed) is that its code was coupled to the production code in such a way that it broke when the production code changed. This tendency to break (fragility) is a critical property of a unit test. A fragile unit test will slow down a developer doing work on some existing code – it actually makes the system less maintainable.

For a unit test code to be stable (not fragile) it needs to be coupled to stable properties of the production code. The question of whether the production code is designed in such a way that it has stable properties – is a design question. Is it a unit? If not, you will not be able to write a unit-test against it.

And anyway, who said that every class is a unit, or should be a unit? Domain models (when done right) are good examples of a unit, yet the classes that make them up may not be units. Unit-testing should only be attempted with things which are units.

I think too much weight is put on whether a dependency of a class is a concrete or interface type, and not nearly enough on the nature of the dependency. I wouldn’t blame the hammer for pounding my thumb, and by the same token I think that blame should not be directed towards tools like those from TypeMock.

On tools

There is so much more depth to both design and testability that needs to be more broadly understood. No tool has yet been created to handle either design or testing in such a way that humans can give up responsibility for the outcome.

Over the years I’ve noticed that tools are most significant when used by skilled practitioners, which makes sense in retrospect. Giving a novice carpenter a laser-guided saw probably won’t significantly change the outcome of their work. Ultimately, the skilled practitioners are the ones that create tools – not the novices. And no tool, no matter how advanced, will make a novice perform at levels like the skilled practitioner.

In the case of a project too big for a single skilled practitioner to complete in the time required (or at all), the balance of importance shifts away from tools to the project management topics described above.

In summary

I hope that this post has shed some light on the context in which decisions with respect to testing need to be made. Design is one activity that can support certain kinds of testing, but not the only one, or even the most important one for the given type of testing necessary at that time in the project.

Design is hard. Project management is hard. Testing is hard.

Getting the right mix of people that together have enough experience and skills in these activities isn’t easy.

Don’t expect that sprinkling some interfaces in your code base will be enough.
That doesn’t count much in the way of design, just as writing code in a testing namespace doesn’t count much in the way of testability.

Looking forward to hearing your comments.

What makes an app small?

Or inversely, what makes an app warrant the “enterprise” moniker?

If there’s one thing that the history of our industry has shown repeatedly, it’s that developers aren’t particularly accurate with their estimates. Like, orders-of-magnitude inaccurate. Knowing this, it’s surprising that the “small app” argument seems to win so many arguments. The same goes for justifications in the form of “we’ve got to have an X, this is a BIG project”.

So, what makes an app small?

Is it a small number of lines of code? Well, what if those lines of code are keeping planes in the air?

Is it a small number of developers? Same as above. Actually, history has shown that some of the most valuable bits of code written were done by small numbers of developers.

Is it that it will only be installed on a single machine?

Is it…

What could it be?

The real issue

The small app argument is a diversionary tactic.

Loosely translated, it means “I’m comfortable where I am and I don’t want to change”.

Moving on…

The real story of size

Once we actually look at the specific context of an app, we tend to see that someone cares a great deal about it, enough to finance its custom development – rather than buying an off-the-shelf alternative. The expected lifetime of business use is easily 3-5 years, if not 7-10, during which many enhancements will likely be requested. Thus, some non-functional properties of the code matter – at the very least maintainability.

In which case, if the given pattern or approach does significantly improve the desired non-functional properties of the app, it only makes sense to use it.

There is one class of software that might possibly be treated as “small” – the one-off script that’s written to automate some IT task. And even then, so many of these scripts end up living longer than the apps themselves that they should be engineered at the same level of quality.

In closing

Don’t counter a “small app” argument with psychology.
It will only make matters worse.

Instead, rephrase the issue around the lifetime of business use.

I’ve found that there are precious few cases where the harsh light of reality doesn’t help the appropriate decisions be made. If indeed this is a small-lifetime-app, just drag-and-drop until you’re done. Otherwise, the time it takes to understand and evaluate the applicability of the given patterns will definitely pay itself back many times over the life of the app.

And managers, keep your ears open for it. The technical risks behind that statement are icebergs waiting to sink your project.

* with thanks to Mike Nichols for pushing my buttons.

Thus begins the story of yet another developer trying to convince their manager to adopt test-driven development (or any other methodology or technology). There’s a good chance this developer’s been reading all sorts of stuff on blogs (like those linked here) that have convinced him that salvation lies that way.

Don’t get me wrong.

There’s a good chance the developer’s right.

It’s just that that’s besides the point.

Developers and Managers

There’s a difference between how developers view a practice and how a manager (defined for the purposes of this post as someone in charge of delivering something) view that same practice. From a developer perspective, Ian’s point about unit testing is spot on:

“The problem is that the most important step is not doing it right, but doing it at all.”

Yet, as Ian himself points out in the title, this is a learning issue. If you want to learn to swim, there’s no replacement for jumping in the pool.

The manager’s perspective is a bit different.

Yes, we want our developers to improve their skill set. Yes, we understand that unit testing will ultimately improve quality. Yes, we know developers need to practice these skills as a part of their job. But, and it’s a big ole’ but, when it comes time to sink or swim, and we’ve got a deadline, those desires need to be balanced with delivering. Accounts of unit testing adoption efforts resulting in more (test) code to support with little apparent improvement in quality in the short term, well, they scare us. Arnon’s post gives more links supporting that feeling.

What’s a Unit Test anyway?

Is it any class that happens to have a TestFixture attribute on it?

If we are to “decouple” unit testing from good design, as Roy has described, that’s a not-improbable outcome. If the design of the system is such as there aren’t any real “units”, what exactly are we testing? Regardless of static or dynamic typing, replaceability of code, and other technological things, does the fact that all TestMethods in that TestFixture complete successfully mean anything? In other words, what did the test test?

It is clear that these tests cost something.

It’s more code to write. It’s more code to maintain.

The question is, what value are we getting from these “unit tests that any developer without design skills can write”?

The manager in me doesn’t like this return on investment.

By the way, TDD is as much the evolution of unit testing as the screw driver is the evolution of the hammer. But that’ll have to wait for a different post.

What’s Design Got To Do With It?

If you’re looking for the technical ability to write a test fixture and replace calls to other classes, then design has nothing to it.

If tests are to be valuable – design has everything to do with it.

The difficulty our developer is having unit-testing the system is a symptom of design problems. There’s a good chance that’s why he suggested a rewrite.

By the way, please do a search & replace in your vocabulary on the word “rewrite” with the word “redesign”. The code’s syntax isn’t the problem – it’s not the “m_”, camel case, or anything like that. It’s not that if the code was rewritten under the same design that all problems will go away.

Redesign, or do nothing.

The community’s been discussing the issues of coupling, interfaces, mocking, and tools at length in the context of testability. I won’t reiterate the debate here but I’ll tell you this:

If logic is duplicated, if the code is tightly coupled, if there is no separation of concerns, the unit tests will be useless – even if they “test” the class in isolation.

Cut the coverage crap

Metrics lie.

The fact that there’s a bunch of other code which calls 100% of the system’s code and doesn’t contain false assertions doesn’t mean that the code is high quality or doesn’t contain bugs.

In a well designed system, most “logic” will be contained in two “layers” – the controllers and the domain model. These classes should be independent of any and all technological concerns. You can and should get high unit test coverage on classes in these layers. Shoot for 100%, it’s worth it.

Testing domain models is all about asserting state. While using setters to get the domain objects into a necessary initial state is OK, setters should not be used beyond that. Testing controllers is primarily about interactions – mocks will probably be needed for views and service agents. Commands do not need to be mocked out.

Most other layers have some dependence on technology that makes unit tests relatively less valuable. Coverage on these layers is most meaningless. My guidance is to take the effort that would have been spent on unit testing these other layers and invest it all in automated integration tests. You’re likely to get a much higher return on investment. Much higher.

Much.

Everybody’s Right

Developers aren’t just born knowing good design, testing, or anything else. Universities, colleges, and most vendors do little to change that state of affairs. Books help, a bit, but when learning to swim, you’ve got to get your feet wet, and on the job training is, by and large, all there is. As such, lowering the barrier to entry is important.

Keeping in mind the Dreyfus model of knowledge acquisition, it’s not about “dumbing down” software development, it’s about bringing novices up to speed:

“In the beginning [novices] learn to recognize objective facts and features, relevant to the skill. Characteristic of relevant elements are that they can be recognized context-free, i.e. without reference to the overall situation. The novice acquire basic rules to follow, acting upon those facts and features. The rules are also context-free, i.e. no notice is taken to the surroundings. On account of this the novice feels very little responsibility for the result.” (emphasis mine)

Managers are ultimately responsible for the result.

Managers shouldn’t necessarily sacrifice their projects on this altar of learning. Organizations need to find ways for developers to safely practice these techniques as a part of developing their “human resources”. First of all, this needs to be communicated to everyone – that the organization understands the importance of these techniques, the desires of developers to adopt them, and the projects that need to be delivered.

Some projects may be allocated additional non-functional requirements: the software will be developed test-first, there will be at least 80% unit test coverage, etc. It can make sense to have developers spend some time on these projects after finishing one more delivery focused project and before going onto another one. As more developers become proficient with unit testing and design, the delivery focused projects can start to benefit from these skills.

It’s a gradual process.

The Important Bit

No matter how you go about unit testing, do periodic test reviews.

Just like code reviews.

That’s it.

Related Posts

Business Process Verification

Self documenting and Test-Driven Alien Artifacts

SOA Testing

Source: Ted Neward’s post: Welcome To The Shitty Code Support Group.

And I consider myself to be a top 10% kind-of-guy.

Imagine the kind of crap all those other people are putting out. Not you, the reader, of course. The fact that you’re reading my writing means you’re a top 10% kind of person too  

Having good design helps.

Helps decrease the impact mistakes in one part of the system have on other parts.

That way, when I’m working on one part of the system, I only have to deal with my crap. And that’s a good thing. Because I’ve gotten pretty good at fixing my crap over the years.

–

This movement is definitely a good thing, over all. Moving to a failure-oriented mindset will help us put the appropriate tools and processes in place to making more robust systems, maybe even Non-Stop Software.

Developers, and managers too for that matter, by and large are concerned with “productivity”. Developers want the latest tools and technologies so that they can churn out more code faster. Managers create schedules trying to get the maximum efficiency out of each one of their developers. They consider resource utilization and other terms that sound manager-ish.

Fact is, on medium to large sized projects, if you look at the studies you’ll find that developer productivity when measured as total lines of (non-blank) code of the system in production divided by the total number of developer days comes in roughly at 6. Maybe 7.

7 lines of code a day.

Let that sink in for a second.

I can hear the managers screaming already. OMFG, what were they doing all day long?! It takes, what, 10 minutes to put out 7 lines of code? An hour even, if it’s complicated recursive code and stuff. And they say they don’t like us micro-managing them?! Now we know why. It’s because they’re goofing off all day long.

Well, managers, that’s not really the way it goes. You see, you have to take into account the time it took to learn the technology, tools, frameworks, etc. Add to that the time of understanding the requirements, which is really sitting through boring meetings that don’t explain much. Finally, our poor developer actually gets to implement the requirement. Maybe run the system a couple of times, trying out the feature they implemented, and checking the code in.

Well, that’s actually the easy part. Now comes the part which kills most of the time. After a bunch of features have been developed by the team, the testers start banging away at it and find a bunch of bugs. Now the developer has to reverse-engineer some bizarre system behavior and figure out which part of the system is to blame. That involves usually some educated guessing (unless they’ve just joined the team and have been put in the bug-fixer role to “learn the system”, in which case it is thoroughly UNeducated guessing). They change some code, run the system, which looks like its been fixed, check the new code in, and close the bug.

But the bugs keep coming. And as the project progresses towards production, more and more of the developers time is spent looking through code and changing existing code, that actually writing new code.

And the larger the system, the more bugs. And I don’t mean that the number of bugs linearly increases with lines of code, or number of features. It’s probably closer to exponential. If it’s a mission critical system, the performance bugs will be taking an order of magnitude more time to fix than other bugs.

So, as you can see, getting a system into production is a team effort. It includes the developers and testers, of course, but also management, and the customer, and how they manage scope. This is kind of a “duh” statement, but we’re getting to the punch-line.

If getting a system into production involves the entire team, isn’t that obviously true for each feature too?

In which case, why are we asking just the developers to estimate the time it takes to get a feature “done”? Why are we trying so hard to measure their productivity?

I know why. It’s so we can get rid of the less productive ones and give bonuses to the more productive ones!

Back to the main issue. I don’t “trust” developer estimates because I need to see the team’s capability to put features in production. The involves all aspects, and often many team members, in some cases multiple developers going through the same code. This involves all overhead and cross team communication, sick days, etc. It’s also why I try to get multiple data points over time to understand the team’s velocity.

While I care about the quality of my developers, and testers, and everybody on my team and would like them to be able to estimate their work as best they can, I’ve got a project to put into production. And the best way I’ll know when it’ll go into production is by having data that’ll enable me to state to my management:

“Our team is finishing 20 feature-units a month, we’ve got 200 feature-units to go, so we’ll be done in around 10 months.”

If I’m busy micro-measuring each developers estimates, I won’t have the time to see the forest. By first taking a harsh look at the reality of what the team can do, I can start looking for ways to make it better. Maybe the bottleneck is between analysts and developers, maybe we’re seeing the same bugs regressing many times, but until we know where we are, we can’t run controlled experiments to see what makes us better.

Focusing on the individual developer, getting them the latest and greatest tools may be great for their morale, but it probably won’t make a bit of difference to their actual productivity.

Next time – what to do when management asks you what it’ll take to be done sooner.

So, here’s the thing.

The only way to win on fixed-bid projects, is to bid low, and then rack up the change-requests. This is why people spend so much time documenting requirements, and then getting the client to sign off. It’s so they can prove that something is an actual change request, and thus they don’t have to do it. So, if the client wants to do whatever, they have to pay more money.

The problem is that it pisses off the client.

There’s another, subtler problem. It’s that clients get wise to this game, and front-load every possible requirement requesting total flexibility in everything.

This leads to another problem. We can’t bid low anymore.

Which leads to another problem. The client doesn’t have the budget to pay for the longer list of requirements.

Which leads us back to square one.

Fixed bids are a lose-lose proposition.

You see, if you bid rationally, taking into account the fact that some requirements will change, others will appear mid-way through, and so on, you’re bid will be significantly higher than the other guy who low-balled it. That means that the client will have a very hard time explaining to his management why he wants you to do the project.

So, the only way to win is for the client to realize this and game the system. This is sometimes a fine-line, possibly bordering on illegal when it comes to government contracts.

Once you have a client who understands that the fixed-bid is not in their interest, they will work collaboratively with you to get a reasonable system out the door within the given budget. There will be a lot of give-and-take but it can work. After a system goes into production successfully, it’s a lot easier to get management buy-in for the next version.

Fact is, upper management doesn’t really know all the specific requirements. So, if you don’t do them all, you’re OK, and so is your client.

In these circumstances, agile development is not only possible, but likely.

I know that it’s not really fixed-price, fixed-time, fixed-scope this way. But that’s what makes it successful

Developers don’t know how to estimate.

Or more specifically, the variance in the actual completion time of a feature from the estimate given by a developer increases probably exponentially with time.

For example, if the estimate is a day, you can expect it to be finished in around a day. If the estimate is a week (5 works days), it will probably vary between 4-10 work days. If the estimate is a month, in all actuallity the developer probably doesn’t know enough to say but will answer when pressed.

This is why Ron (the team lead) asked me if I wasn’t worried I was putting myself in a lose-lose situation by changing the project structure. There were two “teams” when I came in – developers and testers. All the team leads had “committed” to “finishing” the project in 6 months. When I originally proposed the change to more feature-driven teams, mixing skilled and newer developers and testers together on the same team, came the cry:

“It’ll take us twice as long this way.”

“It’s so much less efficient than before.”

And on, and on. What was funny to me was that 3 of the “6″ months were already gone and not a single feature worked. We were half “gone”, and nowhere near half done.

The thing is that Ron was sure I was cooking my own goose with upper management. What he, and most other developers don’t know is that upper management has gotten used to the state of things. If developers say one month, management has seen enough history to know that it’ll really be 3-4 months. So when I come in and do things different from what developers are used to, upper management is thrilled – that’s why they brought me in the first place.

The difference is that by working based on features, and measuring project progress by feature-units completed per iteration, I drive down variability. This creates solid data about progress saying when we’ll be done (more or less). This is quite different from the “normal” course of many projects:

“OK, so it’s been 8 months now on your 6 month project. When will you guys be finished already?”

Without the data, your only strategy is hope: “Umm, I hope the developers will be done this week(?)”

Don’t get me wrong. I trust my developers and testers deeply. But it’s not their job to know how to estimate and manage projects. PMs who take developers estimates as is and stake the project on that being correct are setting themselves up for failure – with only themselves to blame.

Now back to your nice warm-and-fuzzy blogging…

How much, and what kind of documentation do we need to create even if we have “self-documenting code”? Or is that kind of code enough all by itself? I for one yearn for the day where the code really will be enough, and I think that Scott and Ayende do too. First of all, I think that as time progresses, the size of systems that the average developer works on is increasing substantially. And the larger a system is, the more we find a kind of code/design dialect that developers use to talk about that system. I think that these dialects are significantly different between framework/open source code, and application code. So, when a new developer comes in, do we tell them to “just read the code”? Or if the application has gone into production some time ago and now needs to be enhanced but the original team is long gone, what can be done?

What I usually suggest (and practice) is to have some kind of documentation explaining “language” of the system – what things go together, how, and why; and just as importantly, what things must be kept apart. This can be a Software Architecture and Design document, or even videos of the design sessions where things were first discussed. The relevant requirements should be a part of this as well. We need to know that the reason asynchronous messaging was used was for the strenuous scalability requirements, otherwise we might start adding the high-productivity Visual Studio Web Services we like so much.

Continuous integration is a boon to projects who use them. But developers who are only familiar with building solutions in Visual Studio may not be used to working that way, keeping code checked out for weeks at a time.

Test-Driven Development and Architecture will help those who know about it. We should probably write something up about how the system should be enhanced. Side-effect or not, developers and testers will need to know how we test the system – what tools are used in which way and at which stage of the development lifecycle. We could reference existing methodologies as well as in what ways we deviate from them and why.

What sustainable business value do we provide by leaving behind us alien artifacts and practices? I’m not saying not to use state-of-the-art designs, techniques, and practices. On the contrary, I think that they are the key to sustainable business value. Necessary, but not sufficient. Documentation, of whatever flavor you find most suitable to each specific thing you are documenting, should be considered by the team as a whole, and the conclusions presented to stakeholders. The famous Chrysler C3 project’s long-term business value is shaky as the result of those stakeholders decision that documentation wasn’t necessary – in no way was the Agile process faulty in that respect.

Finally, you may be surprised the holes you find in your own thinking when forced to write down and explain to someone just coming in “how we do things around here”, I know I was. I can tell you that in the cases where I did that exercise, several stupid, costly mistakes were avoided. Fleshing out your thinking isn’t necessarily big design up front (BDUF), it’s just smart.

I wonder if these people go into Burger King, order a # 5, and finish the order by saying, and I would like that done in 60 seconds. Then do they proceed to ask every 10 seconds if the order is done? When they hit the 60 second mark do they start screaming at the top of their lungs “I told you I wanted that order in 60 seconds. If you don’t have it to me in the next 15 seconds I want my money back and the order for free.”?

I recently had the pleasure of working with a customer who “got it”. First of all, they understood that they couldn’t really set scope, time, and cost. Or rather, if they did, then quality – the main property that they had the least power over, would suffer. This could manifest itself in terms of an ugly or unusable user interface, or poor scalability – resulting in sky-rocketing operations costs. In short, they understood that in order to get what they really wanted, they had to give up “control”, or rather the illusion of control. By working collaboratively with the development team, they were able to get a good looking and workable prototype out in time for a tradeshow, as well as do a load test early in the project to see that the architecture was scalable.

Risk is an inherent part of all projects, to a lesser or greater degree. We need to manage risk by including risk-reduction activities (like load tests) into the project schedule. Some customers don’t understand that. It is therefore our responsibility to explain that, although this isn’t a value-creating activity, it is still necessary – or rather what cost could manifest later as a result of ignoring the risk (sky-rocketing operations costs).

But at the end of the day, the buck doesn’t stop here, I’m afraid. It’s not the development team’s money. If the customer chooses to make decisions that increase project risk, it is their choice. It is our responsibility to make them aware of the ramifications of their choices. It is good practice to document these decisions so that if (when) they come back to bite us, we can show that we communicated the possible outcomes ahead of time. I can tell you that it is a huge relief when you have an email record to counter “This system is slow. I thought you knew how to make a system that could handle this many users! Now you’re going to tell me that you have to take down the whole system to fix it – do you know how much that costs?!”

I don’t know why customers behave these ways, I just know that they do. I also know that I have to protect myself and my team from it. But, so as not to end on a doom-and-gloom note, I have found that by communicating more with my customers, I find out all sorts of interesting things. For instance, it’s not that they don’t want us to load test, it’s just that they don’t want us to do it now – they need to get something ready for the tradeshow. They just didn’t think it was important to tell us that.

So, reach out. I know I was surprised by the amount of illogical behaviour that disappeared once I started communicating, despite the customer’s apparent inapproachability. It just might save your project, or your career.

For starters, let me say that I’m all for giving the greatest value possible to a client. However, when I’ve got a schedule that can’t slip and can’t recruit a larger team (for whatever reason) and my client asks me if we could also handle some new XYZ scenario, I sometimes answer “no”. My agile friends were even more astonished when I told them that quite a few of my customers respect me for that answer. Just so you understand, the long version of that “no” is:

“Since we can’t slip the schedule any, and we’ve already done away with all the features that could have been cut, and we can’t bring in any new people, I really can’t promise you that if we try to do that additional work the project will stay on track.”

With clients with whom I’ve worked for quite a while, I sometimes use the more glib “what? You mean, for free?”, and we usually have a good laugh over it.

Anyway, bottom line is that change costs. Somebody has to pay, sometimes a little, sometimes a lot. If you’re a week or two from shipping, dropping a feature to “make room” for the new feature might not help. So much work may have already been done that to take the old feature out might be more work than leaving it in.

One big thing that I will give to agile over the fixed-price projects is that by forcing customers to define up-front all requirements, and “threatening” with a high cost of subsequent change, customers are made to put in many requirements that they think that they just might need. This is scope creep even before any work’s been done. By allowing the customer to change their mind during the project you are, in one feel swoop, decreasing (often drastically) that scope of the project making it cheaper and easier to give high value.

You’d be surprised by the number of times that clients are just trying to figure out if the cost of additional feature XYZ now is outweighed by its value. Giving an unqualified “yes” short-circuits the necessary discussions around risk that keep the business in the driver’s seat on the project. Sometimes “no” really is the best answer.

“When can you have your solution in production and how much would it cost?”

You could see the shock on their faces. Architects weren’t supposed to answer these kinds of questions, were they? That’s what project managers are for, right? Eventually, they settled on one month for the schedule question, with only one developer. That’s one month in terms of calendar time. I was shocked by that.

Unless you’re Superman, in the span of one month you cannot learn an existing system, figure out all the new requirements, design, develop, test, and deploy, debug, etc, etc, and keep stakeholders happy and in the loop throughout the whole process. And it doesn’t matter how many developers you have.

And on that point, like you’re really going to be able to find a developer who’s any good at the drop of a hat, get them to leave whatever they’re doing for a month, for a short term project like that.

This topic came up in the speaker dinner when I was talking to Pooya and Jurgen (one of the judges). Architects need to be the glue that connect everything together – business, management, technical, test, operations, etc. They need to fully understand project lifecycle issues. When business comes along with a new requirement, the architect is the one who says if it can be just slipped in or needs to be separately budgeted – not the PM. The PM may have the final call, but the architect is the one who provides the information about the ramifications of changes to the project.

Money? Schedule? All in a day’s work for an architect.

“Patrick”, I said to the project manager, “you know how we have our iterations set up so that each iteration we work on an entire feature, end to end, and keep the number of features down?” Patrick nodded.

“And Daniel”, I said to our test lead, “you know how you report defects on a feature by feature basis, correlated with the version of the system being tested?” Daniel, too, nodded.

“And Beth”, I said to our SCM/integrator, “you know how each check-in gets the relevant project’s version number to increase, gets built, unit tests run, and, if everything’s OK, the DLL gets put out?” Beth was quick to jump in, “and don’t forget that the version number is taken from the DLL and applied as a label to the corresponding source files.”

“So, what’s the point of all this ceremony?” I asked, turning to the entire group. I got some blank stares. “Why are we doing things like this, and not differently?” Alex spoke up. “This all has to do with the DLL thing I asked you before, doesn’t it?”

I won’t draw this story out too much, so let’s just cut to the chase.

When a tester logs a defect, that defect is tied to a feature and a system wide version number. From that version number, we can find all the specific version numbers of the DLLs that comprise that system wide version number. We can also see which DLLs have changed from the previous iteration. From there, we can find the tree of DLLs which are somehow dependent on those few that have changed. Only a few change because we keep our iterations small. From the specific version of a DLL, we can go into our source control repository and, from the label, find the exact source which comprised that DLL. Those are the source files that a programmer has to sift through in order to locate the origin of the defect.

Obviously, if there is less source to go through, the programmer will, on average, find the origin of the defect faster. And, as we all know, it usually takes longer to find the bug than to actually fix it.

So, by splitting up different classes and interfaces into different DLLs, we can manage dependencies in the system so that less source needs to be examined. For instance, in the case of the interfaces mentioned above, IMapEntity depends on IEntity. However, there are multiple classes that depend on IEntity (like views and presenters/controllers) which don’t really care above IMapEntity. The opposite isn’t true, though. The renderer and layer classes are dependent both on IEntity and IMapEntity. If these two interfaces were in the same DLL, and for a new feature we needed to add the “AlphaBlend” property to IMapEntity, when a bug would be found in that iteration, all the view and presenter classes would have to be examined as well. This is because the version number of the IEntity/IMapEntity DLL would have increased, and all the dependent DLLs could have been adversely affected by that change. Had these two interfaces been separated into different DLLs, this alpha blend change would have left an entire tree of DLLs uninvolved. This means less source for a programmer to look through, which means less time to get the feature working.

While I’m not sure the statement “1 wrong DLL = 3 months gone” is necessarily true, I can’t say it isn’t either. All these tiny little things add up to either huge productivity gains, or losses. Especially when you consider larger, more complex projects. A good design, by itself, can’t save you from the failure conditions described above. You need all the practices in place, working in harmony, and people paying attention to make the project succeed.

That’s if the politics don’t nail you first

I don’t think that I need to get into the advantages of agile methods, there is already so much out there. However, after this somewhat superficial first counter, I thought I’d counter the counter.

There is method to the madness. No, wait, strike that. There is madness to the method. No, that isn’t right either. Screw it.

Ask a pro golfer to document, in detail, every single step of everything he does to win a golf tournament. Take that huge document, and give it to someone who never played golf before. Chances are you won’t get spectacular results. Chances are you’ll get pretty crappy results. On the other hand, give that document to someone who’s been golfing for a while, and they’ll probably pick up a couple of pointers that will improve their game. Over time, that document will become more valuable to that golfer as they begin to understand the intent of what was written.

Same document, different audience, wildly different results. Interesting, isn’t it?

You see, that “amateur” golfer didn’t follow the document to the letter. Rather, he picked out the practices that he understood and could implement and used those. Over time, and with more practice, our amateur golfer understood more and implemented more. This is the best place of the bell-curve to be on – ever increasing ROI.

I think that maybe we got this whole process thing backwards from the start. Maybe we shouldn’t even have tried to do the whole thing from day one. Is it any wonder that some practices were so poorly implemented that they did more harm than good? If someone who never played golf before tried to swing a club, swivel their hips, lift their heel, and follow through all together the very first time, there’s a good chance the club would smack them in the ass, probably quite painfully.

After you do one minimalist design document, you figure out what things you don’t have to waste your time on and focus on what’s important. The second time, maybe you include a sequence diagram or two because they really explain how the messages flow between the servers. The third time, maybe you write down some architectural principles so that new people coming into the project can figure out why one design was used instead of another. As time progresses, you find that – guess what, you’re doing most of the things found in the design document template. Only difference is you understand why you need them, and how each part hooks into everything else.

It’s hard to tell from a cursory glance the difference between a good design document and a poor one. They both follow the same template, they both have class diagrams, sequence diagrams, etc. The difference is what happens to that document after its “finished”. One is shoved in the drawer and never seen again. The other keeps popping up in all sorts of discussions – “well, that implementation seems alright, but didn’t we say that we don’t want this class to depend on that one?”

I guess all this sounds a little zen, but the journey of a thousand miles…, well you know the rest.

–

Udi:
You said “I understand the 2 hours of overtime, although I’m not sure I agree with it. But any more than that, and you’ve got the first indication of a project in trouble.” Could you clarify that for me? I’m not sure I understand.

Johanna:
Sure. Two hours of overtime *could* be slight mis-estimation. But it’s
more likely to be a symptom of trying to do the minimum necessary to
“complete” the task rather than what the task requires. Example: making
sure the code compiles, rather than doing any peer review or testing.
People can do this with “minimum” overtime, but still not be completing
the work.

Udi:
I understand now. I wasn’t using “2 hours” as a hard and fast rule that the project is in trouble, rather that I want to know when things are slipping as soon as possible. If it’s mis-estimation, that’s also something I want to know – especially what other estimates we should be revisiting. On the other hand, it could be just a small slip that means nothing.

And finally, I want my programmers to know that they’re not alone against the bleak realities of project development. That there is someone who will actively fight scope-creep, and look out for their interests. I need my programmers to be happy, creative individuals and I don’t want them doing overtime if they don’t have to, especially not to impress me.

I think that this is one aspect of creating a culture of dealing with the real issues on the project. There is so much “filler” work and rework done on projects and none of it actually brings the project any closer to completion.
–

Lasse Koskela responds with the flip side of the coin:

While it’s very true that programmers doing overtime is a good indicator of timely delivery being at risk, most projects–I’d dare to speculate–are in fact not aware of the need for overtime until very late in the project when it’s already {drum roll, please} too late.

I apparently have had very different experiences. In the projects I’ve been involved with, problems usually stemmed from bad management decisions and politics. When these projects got behind schedule, management played the overtime joker-card without ever considering that there could be real issues that need to be resolved. Programmers would toil nights and weekends, but lets face it, the problems couldn’t be resolved on their level.

On the edge of that very same coin, Digg.com gave a different interpretation to my post:

A rather simplistic approach by the Author but he feels that programmers can contribute only to a certain extent in the project and cannot be held responsible for the failure of a project. Some may agree but some don’t.

As to my simplistic approach, well what else would you expect from The Software Simplist ?

But seriously, programmers can only do so much – especially with changing requirements, no prioritization, no time for testing or quality, politics, and on and on… If you look at the underlying principles found in so many of the Agile methodologies you’ll find: just let the programmers do their job.

When it comes time to hold someone responsible for a project failure, should the poor programmer who was told to work on wildly different things each day take the heat? If we’re talking about project retrospectives, shouldn’t we be focusing on what lessons can be learned from one failure to prevent another?

In my opinion, the person that needs to be held accountable (check out Kent Beck’s discussion on the topic of accountability in this audio presentation at ITConversations) if a project fails is the project manager. He should be holding the development and testing teams accountable for project progress and quality, up to the point the project fails.

A common saying in the Agile camps says:
People trump process. Politics trumps them both.

So true.

I’m spending part of my time these days as chief architect on the development of a mission planning system. Anyway, this project was running into some budgetary constraints for the current fiscal year. The problem was that the impact would be felt like aftershocks in the coming years. There were three critical modules that were hit by these constraints.

When asked to describe how we were going to develop the system, I began talking in an “implement-by-slice” language and putting dates beside each slice, finally showing what could be completed by the target date (quite a bit less than many stakeholders had in mind). One of the dev leads that was with me wrote it up, and sent it out with the appropriate explanations and commentary.

The next day I was busy in meetings all day long with our local Microsoft branch, who are absolutely amazing, by the way. The following morning I got a call from the project lead: “We’ve got a green light for all the modules – budget, staff, everything.” I didn’t connect the dots, I mean, I just wrote up a fairly standard development plan, no reason for that to have any serious effect, right?

Well, it turns out that this bit of process, implement-by-slice, was new to many of the stakeholders, and when they saw that the system would be stable at numerous points of the project lifecycle, it blew them away. This down-to-earth plan became a framework for concrete discussions about priorities and budgets. It forced everyone to stop daydreaming about what would, someday, be possible, and focus on what needed to be done at every step of the way.

–

Anyway, this isn’t a commercial for implement-by-slice. Rather, consider it a way to regain focus; while many of us talk about how process doesn’t matter much, I think that that may not be so true. While it may be well backed in terms of the development team, in the larger context of the entire project, process – in terms of the kinds of documents that get circulated, can make a big difference. Lets not fall into the “knee-jerk” trap (via Don Box).

Whenever I recruit somebody, on their first day I explain to them my project philosophy: Programmers don’t make projects fail, even if they don’t work overtime.

If you look at the Chaos report published yearly, I doubt you’ll ever see the lack of overtime on the top-ten list of reasons projects fail. So why is it that whenever a project looks like its getting into trouble to managers pull the overtime card? Could it just be a case of “well, what else could I do?” God, I hope not.

My philosophy is followed with an operative task: If you ever have to do more than 2 hours of overtime in a week, let me know.

Programmers are on the front line. They’re the first to feel that things are beginning to get off track. Even without talking to their manager, they usually try to get things back on track by working more hours.

Overtime is a symptom of much larger problems. It could be that requirements that appeared simple to implement are much more complex (the most insidious kind of scope-creep). It could be that interfaces to external systems we need to interact with aren’t stabilizing in time. It could be anything. Once you see overtime on your radar, it means that you need to start looking for exactly that “anything”.

I’ve been through several such projects where all the pressure was put on the programmers to bring the project to completion. In each case, they did not have the power to do so simply because the problems that needed to be dealt with weren’t technical. The classic saying of “people trump process, and politics trumps them both” rings so true. If you’re the manager, you need to step up and handle the politics or the project’s doomed.

Don’t take the cowardly path – “I pushed the team as hard as I could, literally 24×7, what else could I do?” Unfortunately, many programmers-turned-managers really don’t know any other way – this is what their managers did when projects got into trouble, so they do the same.

Not to end on a doom-and-gloom note, you’d be surprised how quickly you can identify real issues by checking overtime. Of course, the faster you identify these issues, the sooner you can resolve them.

Now reading:

All I can say is, if you’re managing people, you should read it.