Why development schedules are so hard to predict in the beginning?

If you have worked in application development for any number of years, you've probably noticed that it is oftentimes very difficult to predict how long it will take to complete a given product/project. Especially when the project has just begun, the estimates can vary widely, especially if the project is about to take several months of more to complete.

This problem has been naturally studied, and more than one study indicates exactly what can be felt empirically. Officially, this effect can be called the "cone of uncertainty", although other names have also been used. For instance, one recent study is available here.

This study indicates that in the beginning of requirements specification phase, the estimate of work required for a project can vary as much as 4 X. This means that the same project could be estimated to take two months, but also eight months. That's a large different, right?

Indeed It is. As a professional in the development field, everyone should of course focus on creating better estimates. But why it is so hard to predict how long a project will take? Because there are too many moving parts, and also because most projects are unique in some respects. They might be using new technology, new tools, new development methodologies, or might address a completely new business need.

How could you improve your predictions, then? Shortly put, the best way to do so is to specify the application in more detail. Of course, this will take time, but this time is wisely spent. It is foolish to think that by starting development as soon as possible will lead to projects that finish as fast as possible. If you are using agile methods in your development process, this should ring a bell: being agile does not mean that you rush into coding before you know what should be coded.

From the tools side, you can use for example information from previous projects to better estimate how long the new project might take. For instance in Visual Studio Team System (or should we say Team Foundation Server nowadays), you can easily get reports of previous projects.