The pitfalls of forecasting

Forecasting is about predicting the future in order to take advantage of emerging opportunities or avoiding the dangers of being blindsided by nasty surprises. Ever since the Oracle of Delphi uttered poetic prophecies in ancient Greece political, military and business leaders have been trying to divine the future so as to guard against the absurdities of fate or to celebrate in anticipation of their victories on the battlefield or in electoral contests.
Nowadays we also have the frenzied proclamations of television pundits and politicians about one or the other expected happening depending on their interpretation and/or their prejudices about a subject.
No doubt there are professions where forecasting is not only accurate but highly necessary. Disregarding a medical doctor’s prognosis of a disease or an engineer’s estimate of the load bearing capacity of a suspension bridge would not only be foolish but also downright dangerous. But what about the economic and political forecasts that we read or hear of in the media usually citing some highly respected think tank or international financial organisation? How much weight should we place on them?
To review the track record of forecasters we need to refer to an expert who has made a habit of studying ‘experts’. The expert in question is Philip Tetlock, a psychologist who is a professor at the University of Pennsylvania. Professor Tetlock spent over 25 years compiling and analysing the predictions of 284 professionals and academics who he had recruited for his study. About a quarter of his subjects were economists. These people were among the leading professionals in their fields and their professions included “commenting or offering advice on political and economic trends.”
By 2003 Professor Tetlock had accumulated over 82,000 forecasts which gave him a large enough database to compare the results of experts’ forecasts against those using simple statistical procedures such as extending trend lines or forecasts made by non-experts. His surprising finding was that experts proved no better at making predictions than well-read non-experts and that both groups fared worse than simple rules. Thus well published professors did no better than journalists and those with PhDs did not outperform those with only an undergraduate degree. Ironically, he also found an inverse relationship between the reputation of the person and the overall accuracy of their predictions: to wit, the greater the reputation, the worse the prediction error rate. The message from Prof Tetlock’s monumental work is that experts may be good for viewership ratings on television but take their predictions with a grain of salt.
One explanation that has been put forward for the experts’ poor forecasting record is the proposition that the models they use rely overwhelmingly on past experience and therefore persistently underestimate the probability of the occurrence of ‘black swan’ events that occur rarely but are not outside the realm of possibility. The name is derived from the once widely held belief that all swans are white which was proved incorrect in the 17th century whence black swans were first sighted after the colonisation of the Australian continent.
This idea has been popularised by the author and finance expert Nassim Nicholas Taleb in his book titled The Black Swan. In the Taleb formulation a black swan has three characteristics: (1) it is an ‘outlier’ – something quite out of the realm of past experience and data; (2) it has an extreme impact; and (3) people concoct after-the-fact explanations for the occurrence of the event – this largely to protect their own egos in the face of glaring prediction errors.
Black swans can be beneficial as well as destructive in their impact. Consider the case of the British author J K Rowling and her books about a scrawny and bespectacled boy wizard, Harry Potter, that have sold over 450 million copies worldwide. Rowling’s first manuscript was rejected by 12 publishers (presumably all with in-house experts capable of spotting a potential bestseller) in 1996 before it was accepted for publication for which she received an advance of £1500. On the other side there was the bankruptcy of Lehman Brothers, a venerable name in American investment banking, in 2008 that wrought havoc in the world’s financial markets wiping out trillions in the market value of equities. These are examples of black swans that can astonish us by their suddenness and their impact.
So how can one guard against black swans? The answer is you can’t. By definition they fall in the realm of the unpredictable – the ‘unknown unknowns’. Major inflection points such as the oil price hike by the Organization of Petroleum Exporting Countries (OPEC) in 1973 or the ‘Great Recession’ of 2008 in the developed countries upended virtually all economic forecasts. In the realm of politics, a good example is the First World War starting in 1914, a war that none of the great powers of that time expected and wanted but nonetheless stumbled into with catastrophic consequences.
Armed with spreadsheet software, the task of the analyst in preparing economic and financial forecasts has become much easier and inordinately less onerous than in the days before the invention of the personal computer. Reliance on spreadsheets has also led to questionable outcomes with analysts projecting revenues, costs, profit margins etc. five or ten or even twenty years in the future.
One reason of course is to appear thorough and professional as if providing a bigger and more complicated spreadsheet conveys the impression that painstaking research has been undertaken and that therefore the forecast can be relied on. Another is that a financial institution may require a long range plan for the repayment of a term loan and the analyst is compelled to provide cash flow projections for the duration of the loan to show that repayment by the borrower is possible.
The problem is that projections oftentimes treated as sacrosanct are at best educated guesses. Nobody knows what, say, oil prices are going to be one year from now let alone five years from now or what the stock market will do or whether a new competitor will suddenly spring up using a completely new technology and render existing technologies obsolete. For the latter consider the case of the Apple iPhone which, after being introduced in 2007, wiped the floor clean with the then market leaders, Blackberry and Nokia, virtually eliminating the latter’s market presence.
Of course forecasts will continue to be made since businesses want to appear systematic rather than ad hoc in their approach. What kind of expert is more often right than wrong? According to Professor Tetlock’s findings, the most important factor that explained experts’ accuracy was not how much education or experience they had but how they thought. He used a metaphor about the fox and the hedgehog described in an essay by the philosopher Isaiah Berlin to explain the difference in the thinking process. According to Berlin: “The fox knows many things, but the hedgehog knows one big thing”.
What Professor Tetlock found was that prediction failures were far more frequent for the ‘hedgehogs’ than for ‘foxes’. The ‘hedgehogs’ tended to be single-minded and focused on one big idea and were far more likely to discard evidence not in line with their thesis. The ‘foxes’, on the other hand, were far more eclectic in their thinking, adaptable to new evidence, tolerant of ambiguity and more self-critical. The media however loves ‘hedgehogs’ because they tend to be more articulate and persuasive and also more entertaining.

The writer is Group Director for Business Development at the Jang Group.
Email: iqbal.hussain@janggroup. com.pk