Backtesting integrity: the eight ways a backtest lies

Look-ahead bias is the use, at a simulated decision point, of any fact that would not have been knowable at that instant. It is the most seductive failure because the code looks correct and the equity curve looks wonderful, and it is only in live trading, where the future is genuinely unavailable, that the edge evaporates.

The textbook version is the same-bar close. A rule computes a signal from a daily candle's closing price and then enters at that same close, but the close is not known until the session ends, so the position could never have been taken at that price. A subtler version is adjusted data leaking a corporate action: a back-adjusted price series encodes a split or bonus that had not yet been announced at the historical date, so the simulation quietly reads a future event. A third is the repainting indicator, one whose past values change as new data arrives, so the signal you see on the chart today was not the signal that existed then.

What makes look-ahead worth measuring rather than merely naming is that the damage is quantifiable on a single trade. Take a rule that fires on a strong close. The signal bar closes at a price; the next session opens somewhere else, usually in the direction of the signal, because everyone else read the same close. The distance between those two prices is not slippage and it is not bad luck. It is the exact amount the simulation stole from the future, and it is booked on every trade the rule ever takes.

The cure is embarrassingly small. Every input used at time t must be provably available by t, which in practice means the signal computed on a bar must be held back before any position can act on it. In a vectorised back-test that is one line: position = signal.shift(1). The signal still comes from the close; the position it produces simply cannot exist until the next bar. Most look-ahead in retail code is that one missing shift, and the version without it will always show the better curve, which is precisely why it survives review.

Survivorship bias is testing a strategy over history using a universe defined by present membership. Today's index lists exclude every constituent that was delisted, merged away or demoted, so a backtest built from them is a backtest run only on the companies that made it to the end. The failures are simply absent from the sample, and their absence makes the past look kinder than it was.

The Indian illustration is exact. Backtest a rule on the current Nifty 500 and you have silently dropped every stock that fell out of the index over your test window. The Nifty 500 is reconstituted semi-annually, with cut-off dates of 31 January and 31 July and changes effective in March and September, and at each review names leave and names enter. A strategy tested only on the residents of the latest list never had to hold the ones that were thrown out, which is precisely the set most likely to have hurt.

The error runs in both directions, and the second direction is the one people miss. Today's list also contains names that joined partway through your window: a company that listed in 2021 and qualified for the index in 2022 is on the list now, and testing it across 2018 to 2020 back-fills a known winner into years it did not occupy. That is not merely survivorship; it is look-ahead wearing a universe as a disguise, because membership today is information from the future. Both errors push the same way, and they compound.

The countermeasure has a name and it is not optional: a point-in-time universe. For each historical date in the test, reconstruct the constituent list that was actually live on that date, including the names that later left, and let the rule trade only what it could have traded then. This is real work, because it needs a history of index changes and price series for delisted names that vendors are least likely to keep. That difficulty is the whole reason the bias is so common: the correct universe is expensive and the wrong one ships with the data.

Overfitting is tuning a strategy's parameters until the past submits. Every free parameter is a dial, and with enough dials any curve can be bent to fit any history, so a beautifully smooth in-sample equity line can be nothing more than a rule that has memorised noise. The failure is not that the numbers are wrong; it is that they describe the past and not the future.

The deeper trap is multiple testing, and it is worth being precise about it, because it is the one failure people believe they are too careful to commit. Suppose you try one hundred parameter variants and report the best. That best result is not an unbiased estimate of anything: it is the maximum of one hundred draws, and the maximum of many noisy draws is high by construction, whether or not any real edge exists. You did not discover the winner. You selected it, and selection alone manufactures an impressive statistic out of pure chance.

That claim is easy to assert and easy to demonstrate, so the figure below demonstrates it rather than illustrating it. It contains one hundred equity curves generated from a coin flip, a process with a true edge of exactly zero, because zero is what was fed in. Nothing in the picture has an edge. The best of the hundred still finishes well clear of the line.

This is the core of a well-established research tradition. David H. Bailey and Marcos Lopez de Prado, with co-authors, formalised it in work on backtest overfitting, the probability of backtest overfitting, and the deflated Sharpe ratio, which adjusts a strategy's headline statistic downward to reflect how many trials were run before the winner was picked. The lesson needs no formula to state: the more variants you test, the higher the score you will find by luck alone, and an honest evaluation has to discount for the breadth of the search.

The number you report is not the number your rule earned. It is the number your search found, and a search that looks at a hundred things will find a hundredth-best thing.

The practical discipline follows directly. Record the number of trials, including the ones you abandoned, because a variant you tried and discarded still counts against you; the count is not how many you kept, it is how many you looked at. Prefer fewer parameters, and prefer parameters with an economic reason to exist over ones that merely improved the fit. And treat the winner of any search as a suspect rather than a discovery, because you have not measured its edge, you have measured the right-hand tail of your own effort. This failure and the last section's cure are the same problem seen twice: information leaking from where it should not, whether from the future or from the search.

A backtest run without costs is a backtest of a market that does not exist. Every real fill pays a stack of frictions, and in India that stack is not trivial: brokerage, the securities transaction tax, exchange transaction charges, the goods and services tax on those charges, stamp duty, the bid-ask spread you cross, and the market impact of your own order pushing the price. A rule that trades often can look excellent gross and be a slow loss net, because each round trip pays the toll and the toll compounds.

Two facts about the Indian stack matter more than the rest, and a worked round trip makes both of them visible. The first is that the securities transaction tax is the heaviest single line by a wide margin: on an equity delivery round trip it is charged at 0.1 percent on the buy and 0.1 percent on the sell, which dwarfs the exchange transaction charge, the SEBI turnover fee and stamp duty combined. The second is that turnover is the multiplier: almost every line is levied as a percentage of value traded, so the toll is not a fixed cost you can amortise, it is a tax on activity that scales one-for-one with how often the rule trades.

Read the table as a rule rather than a number. Rates move with every budget, and the arithmetic above is illustrative, but the shape is stable: one line dominates, that line is charged on turnover, and two of the largest costs never appear on a contract note at all. The two unbilled lines are the ones a backtest is most likely to omit, because there is no invoice to remind you they existed. High-frequency rules are the most exposed, since their edge per trade is smallest relative to a toll that does not shrink. The principle is non-negotiable: model the full cost stack, then model impact on top, before you believe any result. The worked rupee arithmetic for intraday, futures and options belongs to the method guide; what belongs here is the discipline of never reading a gross curve as though it were a net one.

A strategy tested across a single market regime has been tested against a single question, and it will answer that question and no other. If your window is one uninterrupted uptrend, a rule that simply stays long will look like genius, and a rule that shorts will look broken, and neither verdict tells you how either behaves when the regime turns. The past is not one environment; it is a sequence of them, trending and ranging, calm and violent, rising and falling.

A strategy that works in one regime is not wrong. It is conditional, and a conditional strategy whose condition you have not named is an unexploded position.

The fix is to span regimes and bucket the results. Deliberately include trending and mean-reverting periods, high and low volatility, drawdowns as well as rallies, and then report performance separately for each bucket rather than as one blended number that hides where the edge came from. Bucketing is what converts a vague result into a usable one: a rule that earns everything in high-volatility trends and gives it back in quiet ranges is not a failure, it is a rule with an operating envelope, and knowing the envelope is the whole point. The companion discipline is to detect the regime you are in and adapt, which is the subject of the guide on regime filters for trading in India.

This is the failure most specific to Indian data, and the one worth working through in full. NSE historical price series are not adjusted for corporate actions. When a company issues bonus shares or splits its stock, the number of shares rises and the price per share falls proportionally on the ex-date, with no change to the company's value. The adjustment factor is defined and public: for a bonus of ratio A:B it is (A + B) / B, which for a 1:1 bonus equals 2, and the exchange applies it after the close of the last day the stock trades cum-bonus.

Now read that through the eyes of a naive backtest running on raw, unadjusted prices. A stock near ₹800 goes ex on a 1:1 bonus and opens the next session near ₹400. Nothing bad has happened: every holder now owns twice the shares and is exactly as wealthy as before. But the unadjusted series shows a vertical fall of about 50 percent overnight, a cliff. A trend or breakdown rule reads that cliff as a crash and fires a short. A stop-loss on a long position is blown through for a loss that never occurred. The signal is entirely a phantom, an artefact of unadjusted data, and every trade it produces is fictional. The same cliff appears on any split, scaled by its own factor.

Notice the second panel's axis, because it carries a lesson the first panel hides. Once the history is back-adjusted, the real range of that fortnight is about twenty-four rupees wide. The cliff was roughly seventeen times the stock's actual two-week range, which is why no volatility filter, no sanity check on daily returns and no outlier rule will save a backtest that has swallowed one: the artefact is not a mild distortion of the signal, it is overwhelmingly the largest thing in the data. A rule tuned on a series containing a few of these has been tuned mostly on events that did not happen.

A backtest fills orders in an idealised world, and the gap between that world and the order book is a bias all its own. The common fantasies are filling at the exact close or the exact touch of a level, as if your order were always first in the queue; assuming a whole order fills at one price when real depth would give you partial fills at worsening prices; shorting in the cash segment where intraday-only and availability constraints apply; ignoring lot sizes in futures and options, where you cannot trade an arbitrary quantity and rounding to whole lots changes both sizing and cost; and trading through circuit limits, where the simulated price is available but the real one is frozen and no fill exists at any size.

Each fantasy tilts the result the same way, toward fills better than reality, and that one-directional quality is the signature of the whole family. Random error would cancel; these do not. The countermeasure is conservative execution assumptions: fill at the next bar's open rather than the signal bar's close, add slippage that scales with size and thins with liquidity, respect lot sizes and short-selling rules, refuse fills at frozen prices, and never assume you received a price a real queue would have denied you. When in doubt, model the pessimistic fill; a strategy that survives conservative execution has a margin the optimistic version never proved.

The final failure is not in the data or the code; it is in the researcher. You want the curve to work. You have spent hours on the idea, you are attached to it, and that attachment quietly steers a hundred small choices: which window to test, when to stop tuning, which unflattering result to dismiss as an outlier, which bug to fix only when it hurts the curve and ignore when it helps. None of it is dishonest by intent, and all of it inflates the result. This is why the same person who would never fake a number will, unprompted, run the search until the past looks kind.

Nobody fabricates a backtest. They simply stop looking for problems at the exact moment the curve starts to please them, and call that moment the end of the research.

Two habits of thought do most of the damage, and both have proper names. Selection is reporting the one window, one universe or one variant that worked and quietly retiring the rest, which is multiple testing committed by hand rather than by loop. Snooping is subtler: you have already read the history, you know which years were kind, and a rule invented after seeing the answer is fitted to the answer even when no optimiser ran. Every Indian trader designing a rule today already knows what the last several years did. That knowledge is in the room, and it cannot be unlearned, only controlled for.

The discipline that answers both is pre-registration: write down the complete rule set, the parameters, the universe, the cost model and the test windows before you run the backtest, and treat any change made after seeing results as a new hypothesis that needs its own out-of-sample data. Fixing the rule in advance removes the freedom to fit yourself to the answer, and writing it down removes the freedom to misremember what you predicted. That habit, deciding the method before you see the outcome, is exactly what the method we teach is built around, because integrity is a process you install before the test, not a verdict you reach after it.

The catalogue compresses into one table: the failure, the mechanism that drives it, the tell that betrays it, and the countermeasure that answers it. Read it as a diagnostic, not a checklist to skim. Note the column of tells: every one of them is something you can look for in a backtest you have been handed, without access to the code.

Seven of the eight failures share one root: information reaching the rule that the rule should not have had, whether from the future, from the survivors, from the search or from the researcher's memory. That shared root is why the defence is single. You cannot patch each leak individually and hope you found them all. You need a structure in which some data has demonstrably never touched the fitting process, so that whatever leaked cannot have leaked into it.

That structure has two parts. The first is walk-forward: optimise on one window, score once on the next window the fit never saw, then slide forward and repeat. The track you report is the stitched sequence of unseen windows, and the discipline that makes it honest is that you keep the losing windows. A walk-forward from which bad folds have been quietly dropped is just multiple testing with extra steps. The second is a held-out set: one block of history sealed at the start and not opened until every decision, every parameter and every rule change is final. It is opened once. If you look at it, tune, and look again, it is no longer held out; it has become part of the fit, and you have spent the only unbiased measurement you had.

Even this only limits the damage; it does not eliminate it. A walk-forward has its own free choices, the window lengths and the roll frequency, and those can be tuned like anything else. The held-out set is one sample, so passing it is weak evidence rather than proof. What the structure buys is not certainty but a boundary on how much you can have fooled yourself, and that boundary is the most any test on historical data can give you.

This is the page's artefact: a numbered list to run against any backtest, yours or one presented to you, before you let its result influence a single decision. If a line cannot be answered yes, the number the backtest produced is not yet evidence.

Run the whole catalogue and a sober picture emerges, and it is narrower than most people want. A backtest is not a promise and not a proof; it is a single controlled experiment on one historical path, and its worth is exactly the discipline with which it was run. The value is not in the equity curve, which any of the eight failures can inflate at will. The value is in the honesty of the assumptions behind it: the point-in-time universe, the realistic costs, the out-of-sample validation, the regimes spanned, the rule fixed in advance.

Which leads to the one sentence worth carrying away. A backtest can only ever fail to disqualify a strategy. It cannot confirm one. Every failure on this page is a way of manufacturing a pass, and not one of them is a way of manufacturing a fail, so a good result is weak evidence and a bad result is strong evidence. That asymmetry is not pessimism, it is the correct reading of what the instrument does: you are asking whether the past is consistent with an idea, and consistency is a much smaller claim than truth. A rule that survives all eight failures, walk-forward and a sealed hold-out has earned exactly one thing, the right to be tried small with real money, where the future is finally unavailable and the test becomes honest whether you like it or not.

Integrity is upstream of the result. It is installed before the test, in the choices you make about data, universe, costs and the pre-registered rule. It is never recovered afterwards by admiring the curve.

• Backtest overfitting and the deflated Sharpe ratio. David H. Bailey and Marcos Lopez de Prado, The Deflated Sharpe Ratio, Journal of Portfolio Management 40(5), 2014, which corrects a strategy's Sharpe ratio for selection bias under multiple testing. Establishes that picking the best of many trials inflates the statistic by construction. papers.ssrn.com
• The probability of backtest overfitting. David H. Bailey, Jonathan M. Borwein, Marcos Lopez de Prado and Qiji J. Zhu, The Probability of Backtest Overfitting, Journal of Computational Finance 20(4), 2017. Establishes the formal link between the number of configurations searched and the likelihood the winner is a false positive. davidhbailey.com
• Corporate action adjustment and unadjusted NSE data. NSE Clearing, Corporate Actions Adjustment: the adjustment factor for a bonus issue A:B is (A+B)/B, applied after the last cum-basis trading day, and NSE historical price series are not adjusted for corporate actions. Establishes the phantom-cliff mechanism on the ex-date. nseclearing.in
• Index reconstitution and survivorship. NSE Indices, Methodology Document for Equity Indices: the Nifty 500 is reconstituted semi-annually with 31 January and 31 July cut-off dates, effective in March and September. Establishes that today's list is not the historical universe. nseindia.com
• The Indian cost stack and why STT dominates it. NSE first-time-investor reference on SEBI turnover fees, the securities transaction tax and other levies, together with the exchange transaction charge circulars: equity delivery STT 0.1 percent on both the buy and the sell, equity intraday 0.025 percent on the sell only, cash-segment exchange transaction charge about 0.00307 percent, SEBI turnover fee ₹10 per crore, stamp duty on the buy side, and GST 18 percent on brokerage plus exchange charges plus the SEBI fee. Establishes that STT is the heaviest single line and that every material line is levied on turnover. nseindia.com