Guide · Quant tooling · 2026-04-25

Vectorised backtesting of Indian equities in pandas, done correctly

Pandas makes backtesting deceptively easy and honest backtesting genuinely hard. The arithmetic below is ten lines; the correctness around it is the whole job. This walkthrough builds a daily equity backtest the vectorised way and foregrounds the traps that make a good-looking equity curve worthless. The examples use generic column names and a placeholder universe so the pattern is what you take away, not any particular instrument. Where a number appears it is arithmetic you can reproduce, never a claimed result.

1. Loading Indian daily data into a price panel

The National Stock Exchange publishes an end-of-day bhavcopy: a daily CSV with one row per traded security carrying the symbol, the series (regular equity is series EQ), and the open, high, low and close. A backtest wants the opposite shape. You want a price panel: a DataFrame indexed by date, one column per instrument, each cell a close. Getting from stacked daily files to a clean panel is where the first quiet bugs enter.

import pandas as pd
import numpy as np

# Read a folder of daily bhavcopy CSVs into one long, tidy frame.
# Each file is one trading day; keep only regular equity (series EQ).
frames = []
for path in sorted(bhavcopy_paths):            # e.g. a list of daily CSVs
    day = pd.read_csv(path, parse_dates=["DATE"])
    day = day[day["SERIES"] == "EQ"]
    frames.append(day[["DATE", "SYMBOL", "CLOSE"]])

long = pd.concat(frames, ignore_index=True)

# Pivot to a price panel: dates down the index, instruments across columns.
prices = (long
          .pivot(index="DATE", columns="SYMBOL", values="CLOSE")
          .sort_index())

prices.index = pd.DatetimeIndex(prices.index)   # ensure a real datetime index
prices = prices.asfreq("B")                      # business-day calendar

Two decisions are hiding in those last two lines. A DatetimeIndex is what lets pandas align two series by date later; without it, joins fall back to positional and mask errors. And asfreq("B") reindexes onto a business-day calendar, which surfaces missing rows as explicit NaN instead of letting them vanish. Indian markets also close on exchange holidays that are not weekends, so a business-day calendar still contains days the market never traded. Those show up as all-NaN rows, and how you treat them, drop, forward-fill or leave as gaps, is a modelling choice you should make on purpose rather than inherit from a default.

2. The vectorised backtest pattern

With a clean panel, the backtest itself is a short chain of array operations. The idiom is the same whether you run one instrument or a whole panel: prices become returns, a rule becomes a signal, the signal becomes a position, the position earns the next bar's return, and returns compound into equity. No row loop appears anywhere, which is what "vectorised" means and why it runs in milliseconds on years of data.

# A single-instrument backtest on a close series `px` (a column of `prices`).
returns = px.pct_change()                        # asset returns, one per bar

fast = px.rolling(20).mean()                     # a rule: fast over slow
slow = px.rolling(50).mean()
signal = (fast > slow).astype(float)             # 1.0 = long, 0.0 = flat

position = signal.shift(1)                        # act on YESTERDAY's signal
strat = position * returns                        # strategy return per bar
equity = (1.0 + strat.fillna(0)).cumprod()        # compounded equity curve

Read the last three lines carefully, because they are the entire method. position = signal.shift(1) says the exposure you hold today was decided at yesterday's close. strat = position * returns earns today's asset return on that already-decided position. cumprod compounds. The fillna(0) is deliberately placed on the strategy return, not on the position, so days with no position simply earn nothing rather than fabricating a value. On a full panel the identical code runs elementwise across every column at once; you sum or average across columns afterwards to get a portfolio.

3. The lookahead trap, the signature error

Here is the single mistake that produces the most impressive fake backtests. Compute a signal from today's close and multiply it by today's return, with no shift. It looks innocent, and pandas will not complain. But it encodes an impossibility: you are buying at a close you could only know once the bar is over, then collecting the move that already happened during that same bar. The strategy is trading on the answer.

# WRONG: same-bar. The position uses today's signal on today's return.
strat_wrong = signal * returns                    # lookahead: inflated

# RIGHT: the position is yesterday's signal, applied to today's return.
strat_right = signal.shift(1) * returns           # tradable: honest

The wrong line is not a rounding issue. On any signal with real predictive edge it can turn a flat strategy into a soaring one, because on every up day the "position" was set by the very close that made the day up. The tell is a Sharpe that looks too clean and an equity curve with almost no losing stretches. Any time a result looks extraordinary, the first thing to audit is the direction and size of every shift between the signal and the return. And beware the mirror image, shift(-1): pulling a future value into the present row is the same crime committed on purpose.

A related subtlety: even shift(1) assumes you can transact at the same close whose data you used. That is a defensible daily approximation, but if you need to be stricter you shift entries and exits to the next bar's open and price fills there. The principle is unchanged: no line of the return calculation may depend on a bar the position had not been set before.

4. Corporate actions and the phantom crash

A raw close series is blind to what happened to the share itself. When a company does a bonus issue or a stock split, the quoted price steps down mechanically overnight, not because value was lost but because the same value is now spread over more shares. To pct_change that step looks like a savage one-day loss, a phantom crash on a day the holder lost nothing. That single corrupt return then poisons the drawdown, the volatility estimate and the compounded equity for the rest of the run.

The fix is to compute returns from a corporate-action-adjusted series, an adjusted close or a total-return series that already folds splits, bonuses and dividends back into a continuous line, or to apply the adjustment factor to the raw prices before differencing. The mechanics of why the raw and adjusted lines diverge, and how the factor is built, are covered in the backtesting-mistakes walkthrough; the rule for this article is narrow and absolute.

# If you hold an adjustment factor per instrument per day, apply it first.
# `adj_factor` is a same-shaped frame of cumulative split/bonus factors.
adj_prices = prices * adj_factor                  # continuous, action-aware
returns = adj_prices.pct_change()                 # now free of phantom crashes

5. A survivorship-bias-free universe

The most flattering error in equity backtesting is not in the arithmetic at all; it is in the choice of what to test on. Take today's index members and run them back over ten years, and you have quietly deleted every company that was delisted, merged away or dropped from the index in that decade. The survivors are, tautologically, the ones that did not fail. A backtest on survivors inherits a tailwind that no live strategy could have caught, because in real time you did not know which names would survive.

This is not a rounding effect. A 2026 study of India's NIFTY Smallcap 250 by Harjot Singh Ranse measured it directly over 2016 to 2025: testing on survivors only, rather than a point-in-time universe, overstated annual returns by 4.94 percentage points, a 23.3 percent relative inflation, and overstated the Sharpe ratio by 0.097. Over the period, 82.5 percent of the index constituents changed. The pull is stronger in small-caps than large-caps because small-caps churn more, but it points the same way in every market: survivors flatter the past.

The mitigation is a point-in-time membership record: for each historical date, the set of instruments that were genuinely in your universe on that date, including the ones later delisted. In pandas you carry a boolean membership mask aligned to the price panel and mask the position by it, so an instrument earns strategy return only on the dates it was actually a member and actually tradable.

# `membership` is a boolean frame, same shape as `prices`, True where an
# instrument was a point-in-time member (delisted names included, up to delist).
eligible = membership.reindex_like(prices).fillna(False)
position = position.where(eligible, 0.0)          # no exposure off-universe
# Delisted instruments simply stop contributing once membership goes False.

6. Realistic costs: charge the turnover

A gross backtest assumes you trade for free. You do not. The honest way to bolt costs onto a vectorised framework is to charge for turnover: the amount of position you change on each bar. Turnover is position.diff().abs(), and multiplying it by a per-unit-traded cost rate gives the drag for that bar, which you subtract from the strategy return before compounding. One extra term keeps the whole thing vectorised.

cost_rate = 0.0015                                # illustrative round-trip fraction
turnover = position.diff().abs().fillna(0)        # position changed this bar
cost = turnover * cost_rate                        # cost drag per bar

gross = position.shift(1) * returns
net = gross - cost                                 # returns after costs
equity_net = (1.0 + net.fillna(0)).cumprod()

The cost_rate above is a placeholder, not a market figure; set it from the actual round-trip stack you face. The point of putting cost in the loop is the stress test it enables: a rule that turns over its position often can look strong gross and collapse net, and the vectorised form lets you sweep the cost rate upward and watch exactly where the edge disappears. Frequent-trading strategies live or die on this term.

7. The quieter pitfalls: alignment, NaN, and tuning

Three remaining traps do not announce themselves. The first is alignment. Pandas aligns two objects on their index before any arithmetic, so if your price panel and your signal carry different date indexes, the mismatched dates become NaN and any careless fillna converts them into fake flat days. Reindex everything onto one clean trading-day calendar and never let a forward-fill invent a price on a day the exchange was shut.

The second is NaN discipline. Rolling windows produce leading NaNs, the first return is NaN, and off-universe cells are NaN by design. Decide where each becomes a zero and where it must stay absent. A blanket fillna(0) on positions rather than on returns is a classic way to accidentally hold phantom exposure.

The third is in-sample tuning. If you try many parameter sets and keep the best, its performance is inflated by selection alone: the more you test, the higher the best in-sample result you expect from pure noise. The deflated Sharpe ratio of Bailey and Lopez de Prado makes this precise by adjusting the significance bar for the number of trials and the shape of the return distribution. The practical rule is to hold out data your tuning never touches, and to count how many configurations you tried before trusting any single number. The idea, and its cousins, are developed further in the backtesting-mistakes article and put to work in the pairs setting in the cointegration and pairs-trading walkthrough.

The pipeline, and the catalogue of lies

Two tables to keep beside the code. The first is the pipeline as a checklist: each step, its idiomatic pandas, and the trap it hides. The second is the catalogue of ways a backtest inflates, each with the symptom you would see and the fix.

Where this sits in a workflow

A vectorised pandas backtest is the right first instrument for daily, position-based equity research: fast, transparent, and easy to reason about line by line. It stops being enough when execution detail dominates, partial fills, intrabar stops, queue position, path-dependent sizing, at which point an event-driven engine that steps bar by bar earns its extra complexity. But most daily research never needs that, provided the four correctness pillars, the shift, the adjustment, the universe and the costs, are all in place. The tooling is the easy part; the choice of instrument and the account setup that precede it are covered in the guide to starting algorithmic trading in India, and the mechanics of pulling live and historical data through a broker API are worked in the broker-API Python tutorial.

Read plainly, the backtest does not create an edge; it tests whether one survives contact with reality. The arithmetic is trivial and the discipline is not: the position that earns a return has to have been decided before that return existed, the prices have to be action-aware, the universe has to include the failures, and the costs have to be charged. That upstream judgement, deciding what is worth testing and refusing to believe a number you cannot reproduce out of sample, is exactly what the method we teach is built around. A backtest is only as honest as the person reading it. For the broader philosophy of validating a strategy before you risk anything on it, see the backtesting overview.

Frequently asked questions

Where the facts come from

• Survivorship bias, India-specific magnitude. Harjot Singh Ranse, "Survivorship Bias in Emerging Market Small-Cap Indices: Evidence from India's NIFTY Smallcap 250" (2026). Survivor-only testing over 2016 to 2025 overstated annual returns by 4.94 percentage points (23.3 percent relative) and the Sharpe ratio by 0.097, with 82.5 percent constituent turnover. arxiv.org/abs/2603.19380
• Selection bias and backtest overfitting. David H. Bailey and Marcos Lopez de Prado, "The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality" (2014): the reported Sharpe of a strategy chosen from many trials must be deflated for the number of trials and the return distribution's shape. papers.ssrn.com
• The shift convention and pandas mechanics. The pandas documentation for DataFrame.shift, DataFrame.pct_change and index alignment defines the forward and backward shift, the return computation, and the automatic index alignment on which the lookahead discipline and the alignment pitfalls in this guide rest.
• End-of-day Indian equity data. The National Stock Exchange bhavcopy is the daily end-of-day file carrying symbol, series, and open, high, low and close per traded security, filtered to regular equity by the EQ series, and used here as the source shape for the price panel.