Venice astrological circle; credit Wikimedia
Pseudoscience and forecasting
Suppose, during a nightly TV weather broadcast, that a reporter presented forecasts by persons, with no credentials in mathematical meteorology, who based their analysis on eyeballing a few charts and graphs. If anyone took such amateur forecasts seriously, when a severe storm was approaching, rather than relying on the consensus of qualified scientists assisted by state-of-the-art supercomputer models, they would risk disaster.
Or suppose that someone suggested that “biorhythms” (a person’s presumed daily, monthly and yearly cycles that supposedly start in sync at birth) could be used to predict the performance of athletes in sporting events. For someone with even modest scientific training, it is clear that even if there were such cycles, in reality they would quickly fall out of sync due to any of a hundred factors, and the resulting athletic performance would be utterly unpredictable from superficial information such as the date of the athlete’s birth. (The present author has personally heard such predictions on radio/TV broadcasts; see also this book.)
Finally, suppose that someone offered predictions of stock market prices based on astrological signs, or in other words on the absurd notion that market prices (which are based on a confluence of thousands of factors, and are negotiated electronically, on behalf of millions of investors around the world, on computers in closed rooms!) can be predicted based on the positions of the Sun, Moon and planets among constellations in the night sky (whose positions and courses were set in place millions if not billions of years ago!). Surely no person with even a modicum of scientific training would place any credence in such predictions. (Sadly, there are some that do proffer such predictions; see, for example, here and here.)
The dismal record of market forecasters
Given these considerations, why are are so many investors willing to accept the advice of market forecasters, most of whom employ only mathematically unsophisticated tools, particularly when their record of predictions is so uniformly poor?
Jeff Sommer, a financial writer for the New York Times, recently summarized the dismal record of 2020 stock market forecasters as follows: “[A]s far as predicting the future goes, Wall Street’s record is remarkable for its ineptitude.”
Sommer noted that in December 2019, the median consensus of an ensemble of prominent Wall Street analysts was that the S&P 500 index would rise 2.7% in 2020. The result (as of 20 Dec 2020) is up 15% — a forecasting error of 12 percentage points. But that is only part of the story. In April, after the market crash in March, the revised consensus of a Bloomberg survey of analysts was that the market would fall 11% overall for the calendar year. So the final result (up 15%, as of 20 Dec 2020) is really off by 26 percentage points.
So is 2020 an outlier? Not really. In an earlier column, Somer mentioned the analysis of Paul Hickey of Bespoke Investment Group. According to Hickey’s analysis of market forecasts since 2000 (see also Hickey’s technical article):
- On average, the median December forecast was that the S&P 500 would rise 9.8% in the next calendar year; the actual average rise was 5.5%.
- The average gap between the median forecast and the actual S&P 500 was 4.31 percentage points, i.e., an error of 44%.
- Each year, the median forecast was for a rise in the S&P 500 index; but it fell in six years, i.e., it was fundamentally wrong in direction 30% of the time.
Hickey observed that these forecasts were off by the most when they would have mattered the most. For example, in 2008 the median forecast was for a rise of 11.1% The actual performance? A fall of 38.5%, i.e., a whopping error of 49.6 percentage points.
Kaissar’s analysis of market forecasters
Somer and Hickey are hardly alone in noting the dismal record of market forecasters. Nir Kaissar analyzed a set of predictions by market forecasters over a 17-year period from 1999 through 2016. He found that although there was a modest correlation between the average forecast and the year-end price of the S&P 500 index for the given year, these predictions were surprisingly unreliable during major shifts in the market.
For example, Kaissar found that the strategists overestimated the S&P 500’s year-end price by 26.2 percent on average during the three recession years 2000 through 2002, yet they underestimated the index’s level by 10.6 percent for the initial recovery year 2003. A similar phenomenon was seen in 2008, when the strategists in his study overestimated the S&P 500’s year-end level by a whopping 64.3 percent in 2008, but then underestimated the index by 10.9 percent for the first half of 2009.
In other words, as Kaissar lamented (independently of Hickey), the forecasts were least useful when they mattered most.
Our analysis of market forecasters
In 2017, the present author, together with three other colleagues, completed an analysis of the records of prominent U.S. market forecasters. For this study, we expanded on a 2013 study conducted by the CXO Advisory Group, which ranked 68 forecasters.
For our study, we further analyzed these 68 forecasters based on two additional factors: (a) the time frame of the forecast (forecasts are categorized as up to one month, up to three months, up to nine months or beyond nine months); and (b) the importance and specificity of the forecast (a forecast that states “the market will be volatile in the next few days” is not a very specific forecast, but the forecast “the market will experience a correction in the next few days” is more specific and thus more important).
The results of our analysis are available in this technical paper. We found, perhaps not too surprisingly, that most of these forecasts did not perform significantly differently than a chance forecast. A few did quite well. The top-ranking forecaster was 78.7% accurate by our metric, followed by scores of 72.5%, 71.8% and 70.5%. A total of 11 of the 68 had accuracy scores exceeding 60%. But at the other end of our ranking, two had accuracy scores near 17%; three others had scores 25% or lower. A total of 18 had accuracy scores less than 40%. Note that significantly more scored less than 40% than scored higher than 60%.
Among the 68 forecasters in our study were 27 who acknowledge, according to publicly available statements or other sources, that they employ technical analysis in their work (some of the others might also; there is no way to tell for sure). So how well did these 27 technical analysts do? Their average prediction score was 44.1% — in other words, less than even chance, and in fact slightly less than the average of all 68 forecasters in our study.
In other words, there is no evidence whatsoever in these data that outdated and mathematically unsophisticated techniques such as technical analysis are effective in predicting today’s high-tech markets. If anything, our results must be on the optimistic side, because of the well-known survivorship bias phenomenon — very likely numerous unsuccessful technical analysis forecasters have dropped out of the business, and thus are absent from our tables. See also this earlier Mathematical Investor blog on the failure of technical analysis.
What should investors do?
One should not be too surprised at these findings. For one thing, they are a straightforward implication of the efficient market hypothesis (see also this report and this interview): in an era where markets are dominated by large, mathematically sophisticated, big-data-mining players, it is impossible for relatively unsophisticated forecasters and investors to consistently beat the markets. True, the efficient market hypothesis has weaknesses, notably the fact that psychological factors often come into play in financial markets. But to assert that one can beat the market based on market psychology requires that one has a superior grasp of mass psychology effects in financial markets, which can hardly be assumed by individual forecasters, let alone by individual investors.
So what do writers such as Sommer recommend for ordinary retail investors? Sommer suggests that the majority of individual investors would do well to simply follow the advice of Vanguard Funds founder Jack Bogle, Berkshire Hathaway founder Warren Buffett and Dimensional Fund Advisors co-founder David Booth: invest in one or a handful of low-cost index funds, selected according to a sober analysis of appropriate risk and time frame, perhaps with the assistance of a qualified professional, and, most importantly, hold these investments for the long term.
As David Booth commented, “We don’t try to forecast the future. … We have no ability to do it. Nor does anyone else.”