Optimism Bias: what you need to know and how you can fight it

In recent years, bias in the workplace, both subconscious and overt, has come under intense scrutiny. Both industry and government have dedicated significant amounts of time and resource to approaching and confronting these biases. It is highly likely that you are already aware of some of the more apparent biases, such as racial, cultural and ageist bias, and that your workplace has taken steps to educate, inform and help team members to address and hopefully overcome these.

However, in this article series, we will be looking at some of the lesser-known workplace subconscious biases: what they are; how they affect you and your colleagues, and what you can do to tackle them. In this first article, we examine the negatives and - perhaps surprisingly - the positives of a bias that affects everyone nearly every day: optimism bias.

Here at BMT, we work hard to counter optimism bias, to ensure that we can grow and develop at every opportunity. Just some of our methods to stay ahead of the game in this field include: regularly reviewing the latest social psychological works and theories; learning from projects and programmes, and inviting SME guest speakers to increase our awareness of the multiple nuances within the field of biases. 

Optimism Bias: what is it?

Optimism bias is a lesser known, perhaps under-appreciated bias and is a cause for concern largely due to its pervasive nature. Optimism bias is a cognitive bias that prompts people to believe that they are more likely to succeed, or that they are less at risk of failure than they really are (National Audit Office, 2013). Optimism bias can take many forms and, in addition to affecting our workplace actions and attitudes, can influence our everyday thoughts and behaviours. Examples of this include, but are by no means limited to, team leaders underestimating time necessary for large infrastructure projects, or individuals believing they have performed better prior to review than they have (Behavioural Insights Team, 2017). In short, there is a common tendency for people to under appreciate risks and to exaggerate their chances of success.

Project work, particularly large infrastructure projects, are vulnerable to optimism bias. These biases manifest themselves in many ways throughout a project’s lifecycle. This is most appreciable during the planning phase, manifesting in consistent over-estimation of success and benefit realisation whilst simultaneously under-estimating time and cost resources. These miscalculations frequently lead to project over-runs and over-spends, with little redundancy available due to these failings having been all but inbuilt during the project planning phase. This is commonly termed ‘the planning fallacy’. (Behavioural Insights Team, 2017) 

Optimism Bias: how does it affect our decision making?

As stated, optimism bias is particularly threatening, but not limited to, large infrastructure projects. This is perhaps best exemplified from an opening statement in the Over-Optimism in Government Projects Report by the National Audit Office.

“The tendency to be overly optimistic leads public bodies to underestimate the delivery challenges of what are often complex projects. They may have financial and timing constraints, have multiple stakeholders, be interlinked or relate to other major projects, and be dependent on organisational or citizen behavioural changes.” (National Audit Office, 2013)

The reasons for this occurring are multiple and not always clearly understood. Over optimism is not necessarily about poor process reasoning and there are many notable behavioural factors.  A key recurring factor includes the desire of individuals to enhance, as well as protect their own prospects, both as a benefit for themselves and to secure greater support for a project. This is further exacerbated by the pressures derived from short term budgetary and political cycles, both of which enhance the risk of over optimism. Crucially this can lead to project plans that are ultimately damned to over promise and under deliver (Kahenman, Daniel and Tversky, Amos, 1977).

Over-optimism can be further exaggerated due to an imbalance of incentivisation, with the weighting towards over-optimism greatly outweighing the weak disincentives. Task owners and decision makers are likely to seek short-term recognition and reward, reinforced by the fact that they may not be in the same role once the project is under way, thus removing accountability.

Although the negative results of such over optimism may be evident in hindsight, when analysed the issue becomes particularly striking. This is highlighted by a report from the Journal of the American Planning Association. This documented the forecast cost, demand and impact of major projects, and found that they have been consistently inaccurate for decades, for example:

“Over the last 70 years in large US transport infrastructure projects inaccuracy in costing was found to be 44.7% for rail, 33.8% for bridges and tunnels and 20.4% for roads.” (Flyvbjerg et al, 2006)

It is not only large government infrastructure projects that are vulnerable to optimism bias; it affects nearly every aspect of commercial life. Start-ups and new businesses are also likely to be subject to this. A report commissioned by the Department of Business Innovation and Skills highlighted that around one third of new business managers estimated their probability of failure as 0%, and that 81% of those interviewed perceived their odds of success as 7 out of 10. This contrasts starkly with reality, with some reports estimating that around 80% of new businesses fail within the first 18 months. This finding suggests that optimism bias is widespread throughout business leaders. (Behavioural Insights Team, 2017)

Ultimately it has been widely documented that human judgement is predominantly optimistic due to inherent overconfidence and insufficient regard for available information:

“People will underestimate the costs, completion times, and risks of planned actions, whereas they will overestimate the benefits of the same actions.” (Flyvbjerg et al, 2006)

This planning fallacy is often the result of t people rarely accounting for their own past experiences of similar tasks, focussing instead on the hoped-for future outcomes. This has further repercussions in that decision makers tend to take an inside view of a project, focussing on the anticipated result, rather than accounting for outcomes of previous projects. Furthermore, patterns of thoughts for project planning tend to be anchored on the future outcomes and scenarios of success, rather than past results, leading to overly optimistic predictions. (Behavioural Insights Team, 2017)

Optimism Bias: are there any positives?

In academic circles there has been a longstanding debate regarding the perceived irrationality of cognitive biases (of which optimism bias is one), and whether they can lead to inherently poor decision making with negative planning connotations. However, it is necessary to account for the fact that cognitive biases have developed for a reason, and that they can create highly efficient decision-making tools. It is hypothesised that such biases have evolved to enable us to make choices and judgements that are “mostly right, most of the time.” (Behavioural Insights Team 2017). These tools evolved in an environment where cognitive attention is at a premium, enabling us to remove uncertainty and navigate complexity without overwhelming limited available processing power.

This has had the dual effect of increasing cognitive efficiency and boosting our psyche to give us the confidence necessary to make decisions in a rapidly changing and potentially hostile world. Furthermore, it is important to account for the fact that these biases do behave in a very predictable manner, thus once we are consciously aware of pervasive nature of optimism bias we can account for it in both project planning and our everyday lives. (Behavioural Insights Team (2017)

Optimism Bias: how can you overcome it?

As stipulated, it is highly likely that subconscious biases such as optimism are simply an inherent part of human decision and logic making, producing many more negatives and positives than have already been mentioned here. However, it is imperative to remember that optimism bias, and the groupthink that goes hand in hand with it, acts in a highly predictable manner. Here are just are a few of the methods we have identified to help you tackle optimism bias:  

• Strong project teams comprised of individuals with varied experiences and diverse backgrounds. This can help produce a more efficient and cohesive team with a reduced vulnerability to optimism bias. A greater range of backgrounds can mean the team has encountered optimism bias in more forms beforehand, thus knowing more of the warning signs earlier on. Teams with a more varied composition may be less likely to be vulnerable to groupthink, with each member’s differing skillset helping to counter the circular nature of planning fallacy and groupthink.
• Where appropriate, team members should be encouraged to discuss issues within the team. Internal experts should be consulted, where appropriate, to discuss problems and challenge core assumptions. It is important to specifically inform individuals that it is encouraged to challenge the consensus, thereby avoiding the groupthink that optimism bias can prompt.
• Ongoing critical evaluation of a project is vital. We recommend using an evaluation model such as the 6 hats model (de Bono 1985). We will be looking at the strengths and weaknesses of several of these models in a future article. Incorporating an evaluation model into your ongoing review process will help to ensure that your team has the bandwidth and freedom to evaluate the project with a critical eye.
• It has been consistently proven that groupthink and optimism bias can lead to severe underestimation regarding forecast budget needs. Therefore, it is recommended that budgeting is approached with a 3-tier format: best case; realistic case, and worst case. This should help to ensure that projects are correctly budgeted and outside or internal changes/ interruptions are accounted for.

(Behavioural Insights Team 2017)

These are a few of the many ways you and your colleagues can work to counter optimism bias and the group conformity that results from it.

It is important to note that the UK government, and the Ministry of Defence in particular, acknowledge and appreciate many of the risks and concerns surrounding optimism bias. Furthermore, the term optimism bias is a recognised financial correction applied in the business case as a counter to such a bias. Further detail from a UK government standpoint can be found in the HM Treasury, Green Book, Central Government Guidance on Appraisal and Evaluation, Annex A5 in particular focusing on “Uncertainty, Optimism Bias and Risk.”

Optimism Bias: conclusions

Optimism bias is a highly prevalent, but well known, issue. It is important to remember that although you are likely to be subject to this bias in future matters of both work and personal life, simple awareness of its existence can be a steppingstone to overcoming it. It is also useful to remember that it is not necessarily a negative trait, and it can help you navigate the intense and, at times, contradictory complexities of life. Ultimately everyone will at times be subject to their own optimism bias and no-one is entirely immune to it, however, it can be a positive and beneficial learning tool. In short, your greatest tool to fight optimism bias is to be aware of its existence, report honestly, question yourself, and review and check your own predictions and plans against previous work.

Read the other articles in our bias series:

1. Confirmation Bias
2. Hindsight Bias