Survivorship Bias

Survivorship bias is a form of selection bias in which the observations available for study have already been screened by whether they lasted. Because failure removes cases from the record, the remaining cases are not a random sample of what existed, and any average computed from them describes the survivors rather than the population. The distinctive feature, and the reason it is so hard to notice, is that nothing about the analysis is wrong. The returns are computed correctly, the average is the correct average of the numbers present, and the conclusion is a valid inference from the sample. The error is entirely in which rows are in the table.

The clearest illustration comes from outside finance. During the Second World War the Statistical Research Group at Columbia University, working for the National Defense Research Committee, was asked how to allocate armor on military aircraft. Abraham Wald's contribution, a set of memoranda from 1943 later declassified and reprinted by the Center for Naval Analyses under the title "A Method of Estimating Plane Vulnerability Based on Damage of Survivors", worked out how to estimate the vulnerability of each part of an aircraft from the damage pattern on aircraft that came back. The point that makes it famous is the sampling one: the aircraft available for inspection were, by definition, the ones that survived, so a part showing little damage among the survivors was evidence that damage there tended to prevent a return rather than evidence that the part was rarely hit.

In personal finance the same filter operates in several places, and it is worth being able to name them.

Fund performance data. Funds that perform badly do not usually limp along visibly. They are merged into other funds or liquidated, and when that happens they stop reporting. Any table assembled from the funds available in a category today therefore contains no row for a fund that closed three years ago, and the category's apparent record is the record of the funds that lasted long enough to be counted. The filter is not random with respect to performance, which is precisely what makes it a bias rather than noise.

Backtests and stock screens. A historical test run on the companies currently in an index inherits the index's own membership decisions, which removed companies that failed. This is one of the standard contaminants of backtested strategies and it always pushes the simulated result in the same direction.

Manager and strategy track records. A composite built from accounts still with a firm has already dropped the ones that left, and clients who left after a poor stretch are not a random subset.

Success stories. Founders who left school and became wealthy are interviewed; the far larger group who left school and did not are not interviewed, and never were counted. The same structure governs stories about day trading, property investing, concentrated stock positions and early retirement. Each is a sample drawn from the winners' end of a distribution and presented as though it described the strategy.

One important case is often assumed to suffer from the bias and does not, in the way people expect. A broad market index's historical return series includes the performance of its constituents for the period during which they were members, including declines before a failing company was removed. What an index does have is a construction rule, so its historical figures reflect the membership decisions that rule produced. That is a different property, and worth keeping distinct from a data set that simply dropped the failures.

The bias enters in three stages, and only the first one is visible in the finished analysis.

1. A population exists, containing successes and failures.
2. A filter removes some cases, and the filter is correlated with the outcome being studied. Funds close. Firms fail. People stop posting.
3. The analysis is run on what remains, correctly, and reported as though it described the population in stage one.

A hypothetical example with numbers that can be checked by hand. One hundred funds launch in the same category in the same year. Over the following decade, forty are closed or merged into other funds, and, as one would expect, they are disproportionately the poor performers. Suppose the sixty survivors averaged 8 percent a year over the decade, while the forty that closed averaged 3 percent a year over the portion of the decade they existed.

A table of "the funds in this category" compiled at the end of the decade shows sixty funds and an average of 8 percent, because the other forty have no row. The average across all one hundred as they actually performed is 0.60 times 8 percent plus 0.40 times 3 percent, which is 4.8 plus 1.2, or 6.0 percent.

The published figure overstates the category's record by two full percentage points a year, and it does so without a single incorrect calculation. Every number in the table is right. The correction is not to recompute anything; it is to find the missing forty rows.

Pros

• The concept is unusually actionable, because the check is a single question about how the data set was assembled rather than a judgment call.
• It explains a large family of misleading claims, from fund tables to founder interviews, with one mechanism instead of many.
• Research data sets that address it exist and say so in their descriptions, so an analysis can state whether it used one.
• Recognizing it does not require distrusting the source, since the source is usually reporting the numbers accurately.

Cons

• It is close to invisible in a finished analysis, because nothing in the arithmetic is wrong and the missing rows leave no trace.
• Correcting it requires data that has often not been kept, since a closed fund or a failed company stops being tracked precisely when it becomes interesting.
• It is easy to over-apply, and a reader who dismisses every good result as survivorship bias has replaced one error with another.
• It compounds with other biases, particularly availability, because the survivors are also the cases that come to mind.