Optimistic and pessimistic forecasters
Many investors, individual as well as institutional, rely on market experts and forecasters when making investment decisions. Needless to say, some of these forecasts tend to be more accurate than others. How can one decide which of these forecasts, if any, to take seriously?
Some of these forecasts are optimistic. For example, on 3 January 2015 Thomas Lee predicted that the S&P 500 index would be at 2325 one year hence. (The S&P 500 ranged between 1867 and 2122 during this period, closing at 2012 on 4 January 2016, well short of the goal.)
Some are pessimistic. In July 2013, Jerry Burnham predicted that the Dow Jones Industrial Average (DJIA) would drop to 5,000 before it topped 20,000. He repeated this forecast on the PBS News Hour in May 2014. (The DJIA exceeded 20,000 on 25 January 2017, having never dropped below 14,700 during the period 1 July 2013 through 25 January 2017.)
Ranking the forecasters
There have been several previous analyses of forecaster accuracy, both in academic literature and also in the financial press.
As a single example, recently Nir Kaissar analyzed a set of predictions by market forecasters over a 17-year period from 1999 through 2016. He found that although there was a reasonably high correlation between the average forecast and the year-end price of the S&P 500 index for the given year, these predictions were surprisingly unreliable during major shifts in the market.
For example, Kaissar found that the strategists overestimated the S&P 500’s year-end price by 26.2 percent on average during the three recession years 2000 through 2002, yet they underestimated the index’s level by 10.6 percent for the initial recovery year 2003. A similar phenomenon was seen in 2008, when strategists overestimated the S&P 500’s year-end level by a whopping 64.3 percent in 2008, but then underestimated the index by 10.9 percent for the first half of 2009.
In other words, as Kaissar lamented, the forecasts were least useful when they mattered most.
Our analysis of market forecasters
Two of the present bloggers, together with Amir Salehipour of the University of Newcastle, have completed a new study of market forecasters. For this study, we expanded on a 2013 study conducted by the CXO Advisory Group, which ranked 68 forecasters.
For our study, we further analyzed these 68 forecasters based on two additional factors:
- The time frame of the forecast. Forecasts are categorized as up to one month, up to three months, up to nine months or beyond nine months (e.g., two or three years).
- The importance and specificity of the forecast. For example, a forecast that states “the market will be volatile in the next few days” is not a very specific forecast, but the forecast “the market will experience a correction” is more specific and thus more important.
The results of our analysis are available in a preprint manuscript. We found, perhaps not too surprisingly, that most of these forecasts did not perform significantly different than a chance forecast.
However, a few did remarkably well. The top-ranking forecaster was 78.7% accurate by our metric. The next three had 72.5%, 71.8% and 70.5% accuracy scores. A total of 11 of the 68 had accuracy scores exceeding 60%. At the other end of our ranking, two had accuracy scores near 17%; three others had scores 25% or lower. A total of 18 had accuracy scores less than 40%.
Full details are presented in our preprint manuscript.
Conclusions
One question that remains from this study is to what extent our analysis, or those of any other similar study, are biased by the simple fact that unsuccessful forecasters tend not to remain in the business for a long period of time. Thus long-term accuracy scores and rankings (the only ones that are statistically significant) necessarily omit those forecasters who have dropped out. We cannot answer this question; we merely list it as a concern. But it does mean that all rankings and scorings may tend to be optimistic and must be read carefully.
As we have pointed out in several previous blogs (see, for example, Blog A), we should not be surprised that even the best professionals in the business have difficulty consistently beating the market over the long term. Since markets by definition incorporate the collective judgments of many thousands of participants worldwide, including organizations who employ sophisticated mathematical algorithms, powerful computer systems and high-frequency trading facilities, it follows that most major markets are reduced to time series that exhibit many of the statistical characteristics of a random walk. And one key characteristic of a random walk is unpredictability — the impossibility of prediction, based on the past history of the time series, of the future course the time series will take.
Obviously a few market professionals have indeed been able to beat the market averages over a sustained period of time. Warren Buffett is frequently cited as an example, and there is no doubt his record is distinguished — over 20% compounded per annum return for 40 years running.
On the other hand, almost all of his outsize gains were made in the 1970s and 1980s. If we look at the record, say, of Buffett’s Berkshire Hathaway-B stock from its inception in May 1996 through March 2017, the average compounded annual gain was 9.9%, which is greater than the S&P500 index (8.3%), but not dramatically so. What’s more, over the past nine years BRK-B’s return has been only 6.2%, underperforming the S&P500 (7.1%) by nearly one percentage point. So was Buffett’s performance in the 1970s and 1980s merely a statistical outlier?
We do not suggest that value analyses and forecasting are vain. After all, detailed analyses of fundamental value and prospects for the future for individual securities are essential for a well-functioning marketplace, in our own day as in years past. But forecasters, fund managers and other major market players do need to be rigorously and impartially evaluated from time to time. And all investors, big and small, should be advised not to rely on market forecasters and fund managers whose record is poor or whose record has not been rigorously vetted through careful statistical analysis.
[Added 28 July 2017:] A very nice synopsis of our study is provided by Larry Swedroe in an ETF.com article.