How well does a “robot AI” predict the Japanese stock market?

A recent Bloomberg article reported on the work of Junsuke Senoguchi, who has developed a “robot” artificial intelligence-powered computer program that forecasts the Japanese stock market, in particular the Nikkei-225 index.

Senoguchi, who currently works for Mitsubishi UFJ Morgan Stanley Securities in Tokyo and who has previously worked for Lehman Brothers and also the Bank of Japan, has a Ph.D. in artificial intelligence (AI), and his new investment program employs AI techniques. Senoguchi is delighted when it is working well, “because I feel I can predict the future“.

While we certainly wish Mr. Senoguchi well in his efforts, we need to check his results to date. Bloomberg reports that in the four year period from March 10, 2012 through January 10, 2016, his program has predicted the next month’s movement of the Nikkei-225 index, up or down, 32 times out of 47, or 68% of the time. How does this compare with a “random” strategy?

The Halloween indicator and the January barometer

In previous columns, we discussed the Halloween indicator and the January barometer.

The January barometer is the claim that whether the S&P500 index goes up or down for the year is well predicted by whether it goes up or down in January. This has been correct 62 of the last 85 years, or 73% of the time. This may sound significantly better than a coin-flip, but we must keep in mind that the proper comparison is with a weighted coin-flip, since the S&P500 has gone up 63 out of these 85 years. If one compares the January barometer with a coin-flip weighted to turn up heads 74.12% of the time, then it is almost as good as the January barometer.

Analysis of Senoguchi’s results

So let us take a closer look at Senoguchi’s results. First of all, note that in 47 trials, a 50-50 coin flip will be correct on average 23.5 times. The standard deviation here is 3.4, so Senoguchi’s 32 successes out of 47 is a roughly two-sigma result, which corresponds to only a 5% chance that it is due merely to random chance.

But, as we noted above, it is really not appropriate to compare with a 50-50 coin flip, since in this case the Nikkei-225 index has increased 27 out of the past 47 months since March 2012. If we then consider a weighted coin with probability of heads = 27/47 = 0.574468…, then we obtain the result that a coin-toss experiment with 47 trials will be correct slightly more often — 24 times.

But even this experiment, one might argue, is not quite appropriate, since beginning in March 2012 one would not know the performance of the Nikkei-225 over the next four years.

So let us consider the following additional experiment: each month (on the tenth day), beginning in March 2012, look back over the previous four years and tally the fraction of times the Nikkei-225 has gone up in those 48 months, then toss a coin with this weighted probability as the guess of the market’s behavior in the next month. We have done this with a computer program, based on historical market data. On average, our program is correct approximately 24 times on average out of 47, the same as above.

One way or the other, Senoguchi’s result is a roughly two-sigma result. While promising, in our view it is not yet time to break out the finest champagne.

Conclusion

As we noted in our blog on the January barometer, insomuch as models of this type rely on market indices such as the S&P500, the Nikkei-225, the FTSE100 or the DAX-30, particularly if only taken at an interval of months or years, it is very difficult to avoid backtest overfitting — there simply is not nearly enough data to make any statistically meaningful statements.

Data over a longer time horizon — 25, 50 or more years into the past — is certainly available, but it is of highly questionable relevance today. This is especially true given the rise of computerized trading in recent years, which means that any trends or patterns that may exist in data are spotted (and neutralized) by sophisticated computer programs long before human analysts see them.

In short, while we wish Mr. Senoguchi well, it is premature to say whether his system has any fundamental, persistent “skill,” particularly in today’s highly mathematical, computerized market. There are no slick tricks to wealth, and no substitute for patient, long-term, low-cost investing.

Comments are closed.