Backtesting

Backtesting is the practice of applying an investment strategy to historical market data and computing the profits and losses it would have generated. The authors of one of the standard papers on its failure modes describe it in a sentence: "A backtest is a historical simulation of an algorithmic investment strategy." The distinction that matters is between a backtest and a track record. A track record is what an actual portfolio did with actual money under the rules that were actually in force at the time. A backtest is what a rule would have done, worked out after the period is over, by someone who already knows how the period turned out. That knowledge is not a small advantage, and most of what goes wrong with backtesting is a consequence of it.

The central failure is overfitting, and the important finding is how cheaply it happens. In "Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance", published in the Notices of the American Mathematical Society in May 2014, David H. Bailey, Jonathan M. Borwein, Marcos López de Prado and Qiji Jim Zhu computed how many variations of a strategy a researcher can try before the best of them looks good purely by chance. Their answer: "After trying only seven independent strategy configurations, the expected maximum SR IS is 1 for a two-year long backtest, while the expected SR OOS is 0." In plain terms, seven attempts on two years of data are enough for the winner to show a healthy in-sample Sharpe ratio while its genuine expected performance is nothing at all. With five years of data the corresponding figure is forty-five attempts.

The consequence the authors draw is the practical one: "a backtest which does not report the number of trials N used to identify the selected configuration makes it impossible to assess the risk of overfitting." A backtest without that disclosure is not weak evidence. It is evidence that cannot be weighed, because the reader cannot distinguish a rule that survived one honest test from the best of a thousand quiet ones.

Several other biases work in the same direction, all of them inflating the simulated result:

Survivorship bias. A data set assembled from the funds or companies that exist today has already dropped the ones that failed, so a rule tested on it is tested on a filtered history. This has its own page and its own mechanism, and it is the single most common contaminant of long historical data sets.

Look-ahead bias. Using information the strategy could not have had at the moment it supposedly traded. Restated financial statements, index membership assigned to a company retroactively, or a trade executed at a closing price that was not known until the close, all quietly hand the simulated strategy tomorrow's newspaper.

Costs and frictions. Commissions, bid-ask spreads, market impact, borrowing costs on short positions, and taxes are easy to leave out and are exactly the costs that punish an active rule. A strategy can be clearly profitable gross of costs and clearly negative after them.

Regime dependence. However long the sample, it contains one path: one inflation history, one sequence of interest rates, one set of tax rules. A rule fitted to that path is fitted to a sample size of one in the dimensions that usually matter most.

Start-date sensitivity. Moving the start of the test by a year or two can change the headline number substantially, and choosing the start after seeing the results is a form of the same overfitting problem.

There is an asymmetry that makes backtesting worth doing anyway. A backtest can reliably rule a strategy out: a rule that failed under conditions favorable to it, with costs excluded and the researcher's knowledge of the outcome baked in, is not going to do better in the future. What a backtest cannot do is rule a strategy in. Passing is weak evidence; failing is strong evidence.

In United States regulation the results have a name. Under the Securities and Exchange Commission's marketing rule for investment advisers, 17 CFR 275.206(4)-1, "Hypothetical performance means performance results that were not actually achieved by any portfolio of the investment adviser", and the definition expressly includes "Performance that is backtested by the application of a strategy to data from prior time periods when the strategy was not actually used during those time periods". Paragraph (d)(6) then bars an adviser's advertisement from including hypothetical performance unless the adviser adopts and implements policies and procedures reasonably designed to ensure the results are relevant to the likely financial situation and investment objectives of the intended audience, and provides enough information for that audience to understand both the criteria and assumptions used and the risks and limitations of relying on the figures.

A backtest is built in five steps, and a bias can enter at each one.

1. Assemble the historical data. This is where survivorship bias enters, if the data set contains only names that still exist, and where look-ahead bias enters, if the data has been restated since.
2. Define the rule precisely enough to run, including entry, exit, position sizing and rebalancing frequency.
3. Simulate the trades at prices that were actually available at the moment of the simulated decision.
4. Subtract costs: commissions, spreads, market impact, financing, and where the account is taxable, tax.
5. Report the result alongside the number of variations tried, which is the disclosure that makes the rest of it interpretable.

A hypothetical example of step 4, which is the one most often skipped. A rule turns over the whole portfolio once a month, so twelve times a year, and each round trip costs 0.10 percent of the amount traded once spreads and commissions are counted. Trading cost alone is 12 times 0.10 percent, which is 1.2 percentage points a year. On a $250,000 portfolio that is $3,000 in the first year, before any tax on the realized gains that monthly turnover generates in a taxable account.

For the rule to be worth following it must beat the simple alternative by more than 1.2 points a year, every year, and the backtest as usually presented shows the return before that subtraction. This is why turnover is the first thing worth looking for in a backtested strategy: it converts a plausible edge into a required one.

Pros

• Rules a strategy out cheaply. A rule that fails on favorable historical data with costs excluded is unlikely to succeed with real costs and unknown futures.
• Forces a vague idea to be specified exactly, since a rule that cannot be written precisely enough to simulate cannot be followed consistently either.
• Reveals the practical properties an investor has to live with: how often the strategy trades, how large its worst declines were, how long it spent trailing a simple alternative.
• Makes cost sensitivity visible, because turnover and its expense can be computed directly.

Cons

• Cannot rule a strategy in. A good backtest is consistent with genuine merit and with pure chance, and the two look identical on the page.
• Overfits easily, and after very few attempts, so the researcher does not need to be dishonest for the result to be meaningless.
• Depends on data that is often silently filtered by survival or revised after the fact.
• Contains one macroeconomic history, so it cannot test how a rule behaves in conditions the sample does not include.
• Is almost always presented without the one disclosure that would let a reader judge it, namely how many variations were tried.