Project Management and Collaboration

Why Do IT Projects Fail? The Curse of Optimism

The recent history of computing is littered with the corpses of failed IT contracts. Most countries have their own white elephants in this respect, the hulking £12bn ($20.4bn) of the UK's NHS electronic records project being near the front of the herd in recent years. Civil engineering projects and one-offs like the Olympics or other major sporting events can also sweep through cost projections with impunity, of course, but if you want your project to not merely double or triple its budget but come in an order of magnitude higher, you really have to be in IT.

Why? What is it about IT projects that makes them so prone to flying massively over budget? Tempting as it is to point the finger at corrupt consultancy firms siphoning off public funds with impunity, the real cause is usually more subtle: the result of a trait of human nature.

We are optimists. We believe that things will work out for the best. This is what has kept the human race going in the face of overwhelming adversity for long periods of history. The alternative to our inherent optimism is to curl up into a foetal ball and wait for the worst to happen. We have a name for that frame of mind – depression – and it's regarded as a mental illness. As a species, our outlook is optimistic, even if we don't always realise it.

So when faced with a large (non-IT) project with a multi-billion pound/dollar price-tag, the default position is to knuckle down and get on with it, in the hope and belief that things will work out. And in a surprising number of cases, they do. Optimism can drive us forward, keep us searching for solutions to problems when the pessimists would already have given up and be crying into their beer.

Problems arise when optimism meets technology, though, because technology doesn't think like we do and computers do not work the way most non-IT people (including project managers) think they do. End-users assume a 1:1 relationship between the way they think and the way their computer operates, and they get frustrated when something happens to break that relationship. Human-machine interfaces have come a long way in recent years, making that relationship seem ever more solid, but behind the scenes it just doesn't work like that.

It takes a long time to learn to code properly, and a big part of that experience is learning the translation interface between human expectations and the hard logic of computer machine code, regardless which high-level language is being used for development. Good programmers can make computers do amazing things. The trick is to make them only do the right amazing things, to collapse the waveform of infinite possibilities into the narrow, tightly-defined space of human expectation. As the old computing rhyme ends, "It never does quite what I want, but only what I tell it."

Even the best project managers often fail to grasp this fact. The token IT person on the board of any project panel will be well aware that they're often cast in the role of Cassandra, chided for over-projecting time and cost requirements, told they're not being team-players when they point out politely that actually it might take a little longer than six minutes to create a client-server architecture for the organisation's worldwide operations. It's a thankless role, even when you're later proven correct.

Optimism bias in major projects could even be getting worse. Alice Marwick, who researches online identity at Fordham University in New York City, has found that one effect of the all-pervading influence of social media is that people create their own neutered, work-safe personas, remaining relentlessly upbeat about all aspects of their work, regardless how badly wrong it might be going. In a world where employment is more precarious than ever, that might make sense for the individual but it bodes badly for the IT projects of today and tomorrow.

Organisations and governments are beginning to learn. Optimism bias is now a recognised problem around the world and new procurement rules are being drawn up to fight it. In some countries the tender documents that bidders must complete in order to apply to manage major projects now include mandatory sections on optimism bias. Some have it built into their procurement spreadsheets.

While pessimism may never be welcome on the project panel, a large dose of realism is a vital requirement if an IT project is to come in on time, to plan and within budget. As awareness of optimism bias spreads around the globe, perhaps we can be cautiously optimistic that the worst of the failed IT projects are behind us.


Freelance technology journalist Alex Cruickshank grew up in England and emigrated to New Zealand several years ago, where he runs his own writing business called Ministry of Prose.


« Social Media Censorship in Vietnam


News Roundup: Gulliver Internets, Quitting Facebook and Bitcoin Rock »
Alex Cruickshank

Alex Cruickshank has been writing about technology and business since 1994. He has lived in various far-flung places around the world and is now based in Berlin.  

  • Mail