CXO’s evaluation of forecasters
The CXO Advisory Group of Manassas, Virginia, which offers “objective research and reviews to aid investing decisions,” has just published an interesting evaluation of equity market forecasts. Titled Guru Grades, the column concludes that the class did not do very well!
CXO evaluated a total of 6582 forecasts for the U.S. stock market, published by 68 different market experts, over the years 2005 through 2012. These forecasts were limited to publicly published predictions (i.e., not in private newsletters), and excluded forecasts that were judged too vague or too complex be reliably evaluated.
The results were not encouraging. Accuracy scores of individual experts ranged from 20% to 72%, with an average score of 47.4%. The average over all forecasts rather than over all experts was essentially the same — 46.9%. Additional details of the study’s methodology, and a complete set of results, are available at the CXO website. See also this Forbes report.
The present authors did some additional analysis on the data provided at the CXO website. In particular, we computed Z-scores for each expert’s raw score, based on a 50-50 binomial distribution (i.e., a distribution typical of random coin tosses). In other words, for each expert we calculated Z = (F – 0.5) / Sqrt(N/4), where F is the fraction of correct predictions and N is the number of predictions. The 68 resulting Z-scores ranged from -3.61 to 3.65 and were almost perfectly balanced around zero. In other words, the resulting set of Z-scores look very much like those of coin-toss experiments.
Other evaluations
This is not the first analysis of forecaster performance. Mark Hulbert, in a Wall Street Journal column, tracks the analysis of more than 200 investment advisors. He found that the 20 market-timing advisers with the best records during the 2000-2006 period lost, on average, 26% during the 2007-2009 bear market, which was not significantly different from other advisers monitored by Hulbert.
In another column, Hulbert reported that of the 51 advisors (out of over 200) who beat the market over the ten-year period ending 30 April 2012 (as measured by the Wilshire 5000 total market index), just 11 outperformed the market in the next 12-month period.
Analysis
The above evaluations are certainly subject to further review and analysis. In each case, subjective judgment was involved, and some might fault the approaches that were used in the evaluations. Perhaps some of forecasters would have done better if compared over different time periods. But even with these caveats, these evaluations certainly do not paint a picture of clear-cut, positive performance.
A few of these advisors did very well — the one who achieved a Z-score of 3.65 (with 80 of his 120 forecasts correct) might well have a rational and effective methodology. But in most other cases it is hard to see anything more than raw chance in action.
Explanations?
How is it possible that more than half of the most celebrated fundamental and technical analysts in the U.S. financial have a negative expected return over the 2005-2012 time frame? Should this not be a zero-sum game? In fact, one would expect that they should have a highly positive expected performance, because of:
- Guru effect: 401k holders, retail investors and others buying the recommended securities after reading their advice.
- Transaction costs and market impact were not taken into account.
- Survivorship bias: The comparisons above did not take into account the numerous analysts who have been forced out of business due to a poor track record.
Part of the problem, as Mark Hulbert suggests, may be the rise of computer trading systems running sophisticated quantitative algorithms. As Terrance Odean of the University of California Berkeley explains, “Now it’s a supercomputer you’re competing with.” Or as Hulbert observes,
Short-term trading has become so dominated by Wall Street’s computers that individuals — and professional managers — almost certainly will lose out to them over time.
But however you cut it, these results say “caveat emptor.” Perhaps you might as well ask your aunt or your daughter’s soccer coach and save on the transaction costs.