Bent Flyvbjerg

By Bent Flyvbjerg

Do managers and planners learn from past mistakes?

Most theories of learning assume that decision makers learn from past mistakes. But is this actually true? As rightly observed by Donald Schön, ‘A theory about learning must deal with performance that improves over time. Performance that deteriorates, regresses, or merely swings from one mode of action to another does not qualify as learning.’*

If we look at a set of decisions or projects with defined costs and benefits, then for learning to take place, these decisions and projects must show improvement over time in terms of reduced cost and benefit risks and thus better realization of planned costs and benefits, other things being equal. This might not happen in the short term, but it should happen in the medium and long run if learning takes place.

If the data show such improvement, they support that learning is happening. If the data show no improvement, one would have to conclude that no learning takes place and decision makers are not learning from past experience.

If Hirschman’s theory of learning were right, then cost overrun would come down over time and benefit overrun would go up
Albert O. Hirschman’s principle of the Hiding Hand is a famous example of the type of theory Donald Schön talks about, but many similar theories of learning exist. The principle of the Hiding Hand makes two fundamental claims about human behaviour in the planning of future ventures:

  1. Planners typically underestimate costs and difficulties during the planning of projects, and this is good because it gets projects started. Furthermore, such underestimates are compensated for during project delivery, where higher-than-estimated costs and difficulties are typically outweighed by even higher-than-estimated project benefits and problem-solving abilities, i.e., benefits and problem-solving abilities were initially underestimated to an even higher degree than costs and difficulties.
  2. Cost risk and benefit risk will tend to fall over time because planners learn from project to project, resulting in lower underestimates over time on the part of planners.

Here the second claim is of particular interest. Hirschman sees the Hiding Hand ‘essentially’ as a ‘mechanism through which decision makers learn to take risks’ and that, therefore, ‘the faster the learning the better‘ (Hirschman 1967a: 28; emphasis in original). Specifically, learning via the Hiding Hand makes decision makers better able to reduce risks over time as they find out how to distinguish between acceptable and non-acceptable risks. If the data show such reduction, they support the Hiding Hand on this point. If the data show no reduction, one would have to conclude that no learning takes place and that the Hiding Hand must be rejected.

Not only do the data not support Hirschman’s main claim, they show the exact opposite to be true.

Figure 1 shows the historical trend in cost and benefit risks, measured as cost and benefit overrun in a set of more than a thousand infrastructure projects. The data cover projects from across the world opened to service in the period from 1927 to 2011.

Figure 1

Figure 1: Box plots of cost risk and benefit risk 1920-2-11: cost risk measured as cost overrun (N=1271), benefit risk as benefit overrun (N=625).

If Hirschman’s theory of learning were right, then cost overrun would come down over time and benefit overrun would go up, both indicating reduced risks and better project performance. Eyeballing Figure 1, no clear historical trends seem apparent. For cost overrun, Bayesian tests corroborate this impression as no significant relationship is found between cost overrun and time when using all 1271 projects with information for cost overrun and opening year available. When we analyse 327 projects with available information for cost overrun, benefit overrun, and opening year, there appears to be a significant reduction in cost overrun over time of about 0.5 percent per year. The latter dataset includes data from 1952 to 2011.

For benefit overrun, the tests show a statistically highly significant historical trend of declining performance of 0.5 percent per year, which is the opposite of what Hirschman’s theory predicts. This is when using the 625 projects with information about benefit overrun and opening year. When using the 327 projects, the significance disappears and we witness no movement over time, which again runs counter to the theory. Moreover, the intercept is significantly less than one at 0.86, indicating that instead of benefit overruns we actually find consistent benefit shortfalls over time, opposite the theory.

For benefit overrun minus cost overrun, there is no significant movement over time, but the intercept stays highly significantly negative, indicating that benefit overrun is consistently less than cost overrun, once more counter to the theory. This is for the sample of 327 projects. For the period after 1980, the intercept is highly significantly negative at -1.031 but there is a significant movement over time: benefit overrun minus cost overrun is getting larger by 0.014 per year. This is not a percentage increase, but a nominal yearly increase. However the increase is so small that, starting at the negative intercept, it would take 74 years for benefit overrun minus cost overrun to finally become positive, i.e., before Hirschman’s principle would become true that higher-than-estimated costs are outweighed by even higher-than-estimated benefits. Finally, the positive trend is not supported by the larger and thus more informative samples of 1271 projects (cost overrun) and 625 projects (benefit overrun), respectively.

Decision makers apparently do not learn from past mistakes to make better decisions over time.

In sum, the statistical tests of the Hiding Hand carried out above falsify Hirschman’s ideas at an overwhelmingly high level of statistical significance. This conclusion is robust across both conventional and Bayesian testing, and across different samples. Moreover, not only do the data not support Hirschman’s main claim – that higher-than-estimated project costs will typically be outweighed by even higher-than-estimated benefits – the data show the exact opposite to be true: The typical (average) project is impeded by a twofold setback of higher-than-estimated costs and lower-than-estimated benefits. This undermines project viability in a majority of cases instead of saving projects by the creative benefit-generation predicted by the Hiding Hand.

In other words, Hirschman’s Benevolent Hiding Hand is dominated by what Cass Sunstein has called its “evil twin,” the Malevolent Hiding Hand. And not only do the data show this dominance to be remarkably consistent across all project types and geographies studied, it is also consistent over time, which indicates that no learning is taking place. Decision makers apparently do not learn from past mistakes to make better decisions over time.
* For the full study, including references and notes, see ‘The Fallacy of Beneficial Ignorance: A Test of Hirschman’s Hiding Hand‘ by Bent Flyvbjerg, see esp. pp. 20-22.

First published on August 18, 2016, LinkedIn.

Back to top of article

Share this post:

follow us in feedly

Leave a Reply

Your email address will not be published. Required fields are marked *