Economics, finance and pseudoscience

Beware snake oil salesmen!

Economics

Bloomberg columnist Mohamed El-Erian recently lamented that the discipline of economics “is divorced from real-world relevance and has lost credibility.” Among the problems he mentions currently afflicting the field are the following:

• The proliferation of simplifying assumptions that lead to an “overreliance on excessively abstract estimation techniques and approaches.”
• Insufficient consideration of the possibility that financial dislocations can disrupt the economy.
• Poor and grudging adoption of important insights from behavioral science and other disciplines.
• An oversimplification of uncertainty.
• An overemphasis of equilibrium conditions and mean reversion, and an underemphasis on structural changes and tipping points.

Separately, a 2015 study by researchers at the U.S. Federal Reserve System attempted to replicate 67 papers involving empirical studies that had been recently published in 13 highly regarded journals in the field of economics. Even though they made diligent efforts, and worked with the respective authors of these papers to reproduce their findings, they reported success in only 29 of the 67 cases.

Finance

Readers of our earlier blogs (see, for example, A, B, C and D) will quickly recognize that the field of finance is afflicted with a very similar set of ills:

• An overreliance on theoretical models that yield impressive-looking academic papers and pretty mathematics, but which all too often do not work in the real world.
• A reluctance to incorporate techniques and methodologies from other fields, including rigorous statistical methods, large-scale data analysis, machine learning and high-performance computing.
• Rampant backtest overfitting, much of it rooted in using computer programs to explore millions of alternate configurations of model parameters, yet failing to disclose these explorations either in published journals or to prospective customers.
• A reluctance to recognize that many financial strategies have run their course and are no longer yielding statistically significant above-market returns (if they ever did!).

Many of the above difficulties are rooted in the failure to employ, or even to fully appreciate the need to employ, the full power of modern rigorous statistical analysis, or, more generally, to incorporate rigorous standards of reproducibility that have already been adopted in most other fields of pure and applied science. The results of these lapses are entirely predictable: investment strategies that look great on paper, but which produce disappointing or even disastrous results in practice.

Health and medicine

As Harvard social scientist Steven Pinker observed in his new book Enlightenment Now: The Case for Reason, Science, Humanism, and Progress, as recently as the late 1880s and early 1900s, pseudoscience reigned in the practice of health and medicine, with appalling consequences. In 1880, worldwide life expectancy was only 29; today it is 71. In that same year, in the relatively prosperous nations of Western Europe, infant mortality was 25%; today it is a fraction of one percent. As recently as the early 1900s, epidemics repeatedly ravaged populations around the world, with many millions of victims; today most of these diseases have been eradicated. Similarly, deaths from famine and malnourishment have plunged from 1400 deaths per 100,000 world population in 1870 to virtually zero today.

Drugs available in the late 1880s and early 1900s were almost completely ineffective, and in many cases downright dangerous — death and birth defects were common as a result of adulterated or mislabeled food and drugs. Testing of drugs, if it was performed at all, was spotty and lacked even a modicum of objective methodology and statistical rigor. Unsubstantiated promises and claims were the rule rather than the exception.

By contrast, today rigorous scientific methods are employed throughout the field of health and medicine. What is more, the same scientific and technological advances that make your smartphone possible have also enabled hundreds of amazing life-saving drugs and medical technologies — everything from gene therapy and DNA sequencing to MRI systems. And it is standard practice for personal physicians to discuss the latest double-blind scientific studies with their patients.

New standards for published research

In the wake of a number of high-profile instances where researchers were not able to reproduce previous scientific studies, several prominent scientific journals have instituted even stricter standards than before. The prestigious journal Science, published by the American Association for the Advancement of Science, now requires, among other things, that all data pre-processing and post-processing steps involved in a submitted manuscript be meticulously documented, that the number of sampled units upon which each statistic is based must be declared, that the results of each statistical test be reported in full, and that all steps to ensure objectivity be fully explained and documented. See Science editorial policies for details.

Science and pseudoscience in finance

It is sad but true that scientifically speaking, many financial firms and even academic journals in the field today are arguably at the same level as the “miracle remedy” salesmen of 100 years ago. In all too many cases, they are allowed to present false discoveries as if they were scientific. This is how the fraud works:

1. A researcher runs simulations on 1,000,000 different variations of his or her model or strategy (this is called multiple testing). By sheer luck, one seems to perform well. He or she writes a paper and submits it to a journal. Consciously or not, the author neglects to mention the 999,999 simulations that failed or gave less-than-optimal results (this is called selection bias).
2. The journal receives the paper, but because the reviewers do not know how many trials were behind the claimed results, they cannot discount the effect of selection bias, much less determine whether the claimed underlying principle is true.
3. The financial peer-review process fails, because there are no independent datasets on which the investment strategy can be tested out-of-sample. It may take decades to collect the evidence needed to disprove this false discovery.
4. The professor receives his or her reward: publication, tenure, promotion, or maybe even a prestigious award!
5. A financial firm reads the paper, then structures a financial product around it. Investors are told, “This is a scientific product, supported by journal publications from award-winning authors.” So investors buy the product, often paying hefty fees.
6. The strategy’s performance disappoints; investors lose money, or earn significantly less than the market averages.
7. The financial firm tells investors, “You must trust science. Give it another 10 years; it will eventually work.”
8. Years later, when the strategy still doesn’t work, the firm claims that the market has arbitraged away that opportunity, but “here is a new scientific product that will surely work!”
9. The outcome: Consumers and other investors in the strategy don’t earn enough to retire, but the financial firm and its agents enjoy their healthy profits, all at the expense of the general public.

Why is selection bias a fraud? Because authors and firms have misguided the public, by presenting as scientific a discovery that they knew (or should have known) to be false. They have profited from the public’s trust in science.

What can be done?

The answer is certainly not to shun science. Of course investors should embrace science, but true science rather than pseudoscience! Academic papers such as those described above do not meet the standards of modern rigorous science. Unlike research papers in health and medicine, virtually all papers in the financial field are accepted without careful consideration to the problem of selection bias under multiple-testing, and they are never retracted after it has become clear that they have failed to work. Cynically speaking, if the authors claim to have discovered a way of printing money, then why can’t they print some for themselves?

Yet there is hope. Today, machine learning technologies exist that are able to detect false discoveries in finance. They do so by discounting the probability that the discovery is the result of selection bias. Many authors and financial firms presumably know how to prevent this fraud, however with these new tools hopefully more will now give up their profitable scam voluntarily.

Here is what can be done to stop this fraud:

1. Financial journals must stop publishing unscientific claims. They must control for luck, and ensure that all trials are available for readers. In general, these journals must adopt the same standards of rigor as are now being incorporated by leading journals in other fields.
2. Investors, large and small, need a financial “Food and Drug Administration”. At the least, existing agencies, such as the U.S. Securities and Exchange Commission (SEC), the Commodity Futures Trading Commission (CFTC), the Consumer Financial Protection Bureau (CFPB), as well as similar agencies in other nations, must help stop this fraud:
   • For every financial product or investment advice, financial firms must report the results of all trials, not only the best-looking ones.
   • Each investment product and advice must carry a label, reporting the estimated probability of a false positive, certified by an accredited auditor.
   • Whenever an investment performs significantly below expectations, an SEC, CFTC and/or CFPB investigation must determine whether the probability of a false discovery was correctly estimated and reported.
3. The SEC, CFTC and/or CFPB must set up a database of investment forecasts, so that investors can assess the credibility of gurus and financial firms, based on all outcomes from past predictions. Studies show that most investment gurus underperform dart-throwing monkeys, and the only reason they get away with their failures is because hardly anyone is keeping track.

In short, solutions to this problem do exist. But the entrenched special interests will surely fight them, just as pharmaceutical companies initially opposed regulations by the U.S. Food and Drug Administration (FDA). In the end, this lucrative fraud will only stop if We The People demand regulators to take action.