Guide · Research method · Updated 2026-09-20

A five per cent test on Indian index returns was measured rejecting six times too often, and it is blind to the change that would actually hurt you

Every figure below was computed in the build script for this page, on 3,385 real NSE sessions from 2013-01-01 to 2026-09-18, with the method and the seeds stated. The test was implemented rather than imported, and it was proven in both directions before anything was built on it.

What the test asks, in plain terms

Take two batches of numbers. For each one, walk along the number line and record the fraction of the batch that falls at or below each point. That running fraction is the batch's distribution function, and it rises from zero to one. Lay the two on top of each other and measure the largest vertical distance between them anywhere. That distance is the statistic. The question the test answers is how often two batches drawn from a single distribution would happen to be that far apart.

The appeal is that it assumes nothing about the shape. A comparison of averages tests the average and a comparison of variability tests the variability, whereas this responds to a change in the middle, the spread, the skew or the shape without being told where to look. That is exactly what you want when someone says a system stopped working, because the statement is a distributional claim and its two rivals are distributional too: either the distribution of outcomes has moved, or it has not and this is the ordinary bad stretch an unchanged distribution delivers from time to time.

The generality is also the source of every difficulty that follows. A test that looks everywhere has to spend its sensitivity everywhere, and it turns out to spend almost none of it in the place a trader cares about most.

Proving the instrument before believing it

A test you have never fed a known answer is not an instrument, it is a function that returns numbers. It has to be proven in both directions, and the second is the one people skip: it must reject when there is genuinely something to find, and it must hold its stated false positive rate when there is not. A test that rejects everything passes the first check perfectly and is useless.

So: fed pairs of samples of 1,686 each, drawn from distributions that genuinely differ, and then fed 10,000 pairs drawn from the same distribution.

The last row is the one that licenses everything else on this page. Against two samples from an identical distribution the test rejected 5.00 per cent of the time at a nominal five per cent, and 0.90 per cent at a nominal one. It is calibrated. Two concrete runs at the same sample size: two samples from the same law gave a statistic of 0.0297 and a p value of 0.45; two samples differing only in spread, by a quarter, gave 0.0973 and a p value below a millionth.

The implementation was also checked against an established one, in a script that does not import this page's build code. Over two hundred random sample pairs the statistic agreed to the last digit, the p values differed by at most about 0.013 in the region that decides a verdict, and the two agreed on the verdict at five per cent in 3,985 of 4,000 pairs. Every failure described below is therefore a property of the data, not of a broken tool.

The power problem, which is the spine of the whole subject

The requirement scales as one over the square of the difference being looked for. A change half as large needs roughly four times the data. That is not a convention, it is the same square root law that governs every average, and it is why the arithmetic runs away so fast.

The size of a change, in the units this test works in, is the largest gap it opens between the two distribution functions. For a shift of the average by a given number of standard deviations, that gap has a closed form and it is smaller than people expect: a shift of a tenth of a standard deviation opens a gap of only 0.0399. The constant relating the gap to the sample needed was measured here by bisection, not assumed, and then validated at two other shifts where the predicted eighty per cent power came out at 82 and 81 per cent.

Read the last two columns together and the difficulty is plain. With 1,686 observations a side, which is every trading day from 2013-01-01 to 2019-11-19, the smallest shift reliably visible is about 0.11 standard deviations a day. Converted into the units people quote, that is a change of roughly 1.8 in the ratio of annualised average to annualised variability. A change that large is not a regime shift, it is a different instrument.

The change that actually ends a system is nothing like that size. Suppose a method averaged 0.05 standard deviations a day and that edge halved. In distributional terms the shift is 0.025 standard deviations, and the table says detecting it takes 34,821 observations a side, or about 142 years of daily data. The trader knows within a year that something is wrong. The test will not know within a lifetime. The general arithmetic of that gap is worked through in how many trades it takes to tell an edge from luck, and this test sits on the unfavourable side of it.

Returns are not independent, and the correlation people check is the wrong one

The p value the test reports is computed from how fast a distribution function settles as observations accumulate, and that rate assumes every observation is a fresh piece of information. Returns are not like that, and the usual defence against the objection is checking the wrong quantity.

On this record the correlation between consecutive daily returns is -0.0044, which is effectively nothing, and that is the number most people look at before declaring the independence assumption safe. The correlation between consecutive absolute returns is 0.232, it is still 0.202 five days out, and 0.124 twenty days out. Large days arrive next to large days. The dependence is in the size of the moves, not their direction, and the size is exactly what a distribution test reads.

Whether that matters is measurable. Below, two kinds of stationary series in which nothing changes at all: the two halves are drawn from one distribution by construction, so every rejection is a false positive and the column should read five per cent throughout.

Correlation between consecutive returns does modest damage: even at 0.30, which no equity index comes close to, the rate reaches only 12.2 per cent. Volatility clustering does severe damage. At the calibration whose absolute return correlations best match this index, 0.205 at one day against the measured 0.232, the nominal five per cent test rejected 31.4 per cent of the time with nothing whatsoever having changed. That is 6.3 times the rate it advertises, and the neighbouring calibrations put the plausible range at roughly 24 to 37 per cent.

The third column is the reason, and it is worth reading before the fourth. The rate is not driven by how strong the clustering is at one day but by how long it persists. The last row has the highest one day correlation in the table and a distinctly lower false positive rate, because by twenty days its clustering has largely decayed. What breaks the test is a series that spends months in a quiet state and months in a violent one, which is a fair description of an equity index.

The same numbers, in a different order

A simulation is a model, and a model can be argued with. The cleanest version of this measurement uses the real series and changes exactly one thing about it.

Take the 3,373 real daily returns. Walk a pair of adjacent windows along the record and test every position, so at one year a side that is 2,874 separate tests. Then shuffle the identical 3,373 numbers into a random order and do the same. The shuffled series has the same values, the same histogram, the same average, the same spread, the same skew and the same worst day. A distribution test cannot see any difference between the two, because everything it looks at is unchanged. Only the sequence differs.

Shuffled, the rate lands between 4.2 and 5.2 per cent against a nominal five, which is the calibration again. In historical order, at one year a side, the test calls the distribution changed on 32.8 per cent of adjacent window pairs, and at two years a side on 52.1 per cent. Ordering is the only thing that changed.

One honesty note, because it matters for what the measurement proves. Part of that excess is genuine slow change in the real distribution rather than dependence alone, and the test cannot separate the two. The simulated rows in the previous table isolate dependence, because there the distribution provably does not move. Both effects are present in the real record, both inflate the rejection rate, and neither is what the reported p value is describing. Whether a series can be treated as having one fixed distribution at all is the subject of a price series has a history, not a distribution.

What the test structurally cannot see

A test that looks at the whole distribution finds the largest gap wherever it happens to be. The consequence is a blind spot with a hard edge, and it sits precisely where risk lives.

There is only so much probability in a tail. If a change touches only the worst two per cent of observations, it can move the distribution function by at most 0.02 at any point, because two per cent is all the mass there is to move. That is a ceiling, not a tendency. Below is the same normal distribution with its worst two per cent of days made progressively worse, tested at 1,686 observations a side.

Read the last column down. It does not fall away gradually, it simply never moves: 5.7 per cent when the tail is a quarter worse and 5.0 per cent when it is five times worse, both indistinguishable from the 5.0 per cent the test rejects at when nothing has changed. The average statistic is pinned at about 0.030 regardless. The test is not weak against this change, it is blind to it.

The threshold arithmetic makes the edge exact. Rejection at five per cent needs a statistic above 1.3581 times the square root of two over the sample size a side, and for that to fall below the 0.02 ceiling the sample must be at least 9,223 observations a side, which is about 37 years of daily data. Measured against a tripled tail, the test rejected 4.8 per cent of the time at 1,686 a side, 52 per cent at 9,223, which is exactly the coin flip the arithmetic predicts when the true gap sits on the threshold, and 100 per cent at 20,000.

This is the single most important limitation on the page, because the change a trader most needs to detect is almost always a tail change. Standard deviation already understates it, as set out in standard deviation is a second moment, and the whole distribution test understates it further.

The real split, and what it does not say

Now the obvious thing to do with the data. Take the 3,373 daily log returns, split them at the midpoint, 2019-11-19, and test 1,686 against 1,686. The statistic is 0.0302 against a threshold of 0.0468, for a p value of 0.42. It does not reject.

Anyone reading that as evidence the two periods were alike would be badly wrong, and the way in which is instructive.

The second half is 25 per cent more variable by standard deviation, its worst day is 2.3 times deeper than anything in the first, and the average of its worst one per cent of days is 1.8 times worse. Those are not small differences to live through. Yet the median absolute deviation rose only 4.8 per cent and the interquartile range only 5.0 per cent.

That is the whole article in one table. The middle of the distribution barely moved and the tail moved enormously, so the standard deviation moved while the distribution function did not. A useful counterfactual: if the second half genuinely were the first half rescaled by 1.250, the largest gap between them would be 0.0538, comfortably above the 0.0468 threshold, and the test would very likely have caught it. The observed statistic was 0.0302. The halves are not a rescaling of one another, and the test read the part that did not change.

Running the same split on two more indices gives the other half of the lesson.

One of three rejected, at a p value of 0.015. Reporting that one and not the other two is the trap, though the usual multiple testing arithmetic is the weaker objection here, because three index return series are nowhere near independent of one another. The stronger objection is the threshold itself. Resampled from that index's own history in 250 day blocks, the honest five per cent level is 0.0599 against an observed 0.0540. Once the dependence the series actually has is allowed for, it does not reject either.

Testing again every month guarantees a rejection

The natural way to use this test is as monitoring: run it each month as new data arrives and act when it fires. That converts a single test into a sequence of them, and a sequence of five per cent tests is not a five per cent procedure.

Both measured columns disagree with the naive bound, in opposite directions, and both reasons are worth knowing. With independent observations the measured rate is lower than the bound, because overlapping windows share most of their data, so consecutive tests are nearly the same test and you get far fewer independent chances than you paid for. With clustered observations it is higher, because each individual test is already rejecting several times too often before any repetition is counted.

Three years of monthly monitoring on a series where nothing changes produced at least one rejection in 93 per cent of runs, with the first false alarm arriving at a median of month 5. The structure of this trap, and what to do about it, is the subject of the best of fifty backtested rules.

What to do instead, measured rather than recommended

None of the above is an argument for abandoning the test. It is an argument against the textbook threshold, which is the part that is wrong. Three changes, each measurable.

Take the threshold from your own series. Resample the record in contiguous blocks, so short range dependence survives and no real change can, split each resampled series in half and test it, and read off the ninety fifth percentile of the statistic. That is the level your data produces when nothing has changed.

The single day row is the check on the method: with blocks of one there is no dependence left and the threshold lands on the textbook value. As the blocks lengthen the honest threshold rises to 1.47 times it. Which block length is right is a judgement about how long the dependence lasts, and reporting the verdict across several rather than picking one is the honest presentation. This is the same resampling logic used for interval estimates in bootstrap intervals on an equity curve.

Remove the clustering first, and accept that it changes the question. Dividing each return by a trailing estimate of volatility cuts the absolute return correlation from 0.232 to 0.111, and the effect on the false alarm rate is real.

At two years a side the rate falls from 52.1 per cent to 18.9 per cent. That is a large improvement and it is not a fix, because the rate is still several times the nominal five. It also narrows what is being asked: a standardised test can no longer see a change in the level of volatility, only a change in the shape that remains once volatility is divided out. That is often the more interesting question, but it is a different one and it should be stated as such.

Decide the split before you look, and test the quantity you care about. A split chosen after seeing where the results turned is guaranteed to produce a gap, and no threshold repairs that. And since the test is blind to the tail, a targeted comparison is worth far more: on the same samples where the whole distribution test rejected a tripled tail 4.8 per cent of the time, a direct comparison of the average of the worst two per cent rejected essentially always. Sensitivity is not free, but it can be aimed.

The question that is usually better

Put the three limits together and what this test can do is narrower than its reputation. It rules out a dramatic break well, because a dramatic break opens a large gap and a large gap shows up in a few hundred observations. It confirms a subtle one poorly, because subtlety is what the sample size cannot reach. And it is worst at the change that matters most, because a tail change cannot open a large gap by construction.

A rejection therefore tells you something worth having: the two periods are further apart than dependence alone explains, which is a reason to look harder. A non rejection tells you very little, and the real split on this page is the demonstration, returning a p value of 0.42 across a boundary with a day more than twice as bad on one side.

Which leads to the reframing. When a system stops working, the instinctive question is has the distribution changed, and at the sample sizes anyone has it is usually unanswerable. The answerable question is was it ever what I thought it was. If a method was established over a period short enough that a distribution test cannot now resolve a change in it, the original evidence was already too thin to support the claim, and nothing broke because nothing was ever demonstrated. That conclusion is reachable today, from the record you already hold.

None of this argues against measuring. It argues for knowing what a measurement can carry, which is the difference between a research process and a ritual, and it is why this is taught as method and as the honest limits of method rather than as a list of tests to run. Running the test is easy. Knowing which of its answers you are entitled to use is the skill, and the related discipline of publishing a number someone else can check is set out in publishing a result someone else can check.