Guide · Research method · Updated 2026-09-23

The extreme observations in market data are usually real, and a rule that deletes them moves every risk estimate in the dangerous direction

Every figure below was computed from the exchange's own daily files, and the method is stated in full at the end so the work can be redone. The page makes one argument and measures it several ways: an extreme observation in market data is guilty only when something outside the series says so, and a rule applied without that evidence changes the answer in a direction the rule chose.

An outlier is a question about the recording, not about the number

Two different things produce an extreme value. A bad tick is a recorded value that does not describe what traded: a mistyped price, a decimal slip, a value carried forward from a stale feed, a print from a different instrument, or two sessions merged into one. An extreme move is a value that describes exactly what traded, however improbable it looked. A daily return of minus 13 per cent looks the same in both cases. The number carries no mark of its origin.

A single series offers one signature. An isolated bad print enters and leaves: the close is wrong on one session and right on the next, so the return series shows a move followed by its mirror image. A real move usually persists. Usually is the problem. A rule that flags any move beyond three standard deviations that is at least half undone the next session flags 12 sessions of this record, and every one was matched by the sector indices. They include the rally of 3 June 2024 after the general election exit polls and the fall of 5.93 per cent on counting day, 4 June 2024, of which 54 per cent came back the next session. By the textbook signature, both were bad ticks.

The series can also hide a real extreme entirely. On 13 March 2020 the market wide circuit breaker halted trading for 45 minutes after the broad index fell 10 per cent in the morning. That day's file records a low 10.79 per cent below the previous close and a close 3.81 per cent above it. In a close to close series the session is an ordinary up day, and the extreme lives only in the low column. Whether an observation is an outlier depends on the resolution of the record as much as on the market.

What separates the two cases is not in the broad index at all. That is the idea the rest of this page tests: corroboration from series that share a cause, rather than the size of the number.

The close is already a cleaned number, and on 3 August 2026 its rule changed

The official close of a stock is not its last trade, and was not before the auction either. Until 31 July 2026 the exchange set the close of every stock in continuous trading as the volume weighted average price of its trades in the last 30 minutes, the method the regulator itself describes in its consultation paper of 12 September 2026. That average is a cleaning rule applied before anyone downloads anything: a stray print at 3:29 pm moves a 30 minute average only in proportion to its share of the half hour's volume. It is one reason bad ticks are rare in a daily close series and common in tick data.

From 3 August 2026 the rule is different for every stock with derivatives. SEBI's circular of 16 January 2026 (SEBI/HO/47/11/11(3)2025-MRD-POD2/I/2765/2026) introduced a closing auction session in the equity cash segment. The exchange's operating circular NSE/CMTR/73362 of 18 March 2026 sets the mechanics: the close is the equilibrium price discovered in the auction, the auction runs inside a band of 3 per cent either side of a reference price, and the reference price is the volume weighted average of trades between 3:00 and 3:15 pm, which becomes the close itself if no equilibrium price is found. Circular NSE/CMTR/75479 of 30 July 2026 made it live from 3 August. Stocks without derivatives keep the 30 minute average.

The change is visible in the exchange's own security file, and it breaks a common cleaning check. The daily bhavcopy reports both the last traded price and the close, and comparing the two is a routine test for a late erratic print.

For the 210 stocks with derivatives the median gap between the last price and the close fell from 13.2 basis points to 0.0. For every other stock nothing moved. The check did not fail on 3 August. It stopped measuring anything for those stocks, silently, on a date no column header records.

Three consequences follow for anyone cleaning a close series. The close of a stock with derivatives now carries a hard clamp of 3 per cent around its reference price, normally the average of the 15 minutes before the auction, which is winsorisation performed by the exchange, by rule, before the data reaches you. An index built from those closes inherits the auction: the operating circular computes the indicative index close during the auction from the constituents' indicative auction prices, under its paragraph 6.5.3. And the definition is moving again. The consultation paper of 12 September 2026 proposes new auction timings, two options for the expiry day settlement price, one of them a blended average across the last 30 minutes of continuous trading and the auction, and an end to the indicative index value during the auction, with comments invited until 3 October 2026. A guide that describes the Indian close as a plain 30 minute average is now describing only the stocks without derivatives.

Most anomalies in a real archive are missing sessions, and the file reports them itself

Before any outlier rule there is a decision about what counts as a session, and it is usually made by a download script. The daily all index file carries the exchange's own points change for each index against the previous session's close, so each file contains a check on its neighbour: the close minus the reported change must equal the previous file's close. Across the 3,371 files a weekday downloader collected for this record, the check failed 26 times, and every failure had a cause that the archive itself could establish.

Fourteen are weekend sessions, the most recent the Sunday session of 1 February 2026, held because the Union Budget was presented that day. A downloader that requests weekdays only never asks for any of them. Twelve more are weekday sessions between 2013 and 2016 for which the archive answers not found, which a downloader that treats not found as a holiday silently skips.

The damage is to individual sessions. In the downloaded series 25 returns span more than one session, and one became a false outlier: 2015-09-07 shows -3.38 per cent, beyond three standard deviations, and is the sum of -2.15 per cent on 2015-09-04 and -1.26 per cent on 2015-09-07, neither of which is. The budget session of 1 February 2020 fell 2.51 per cent and does not exist in the download at all; it is folded into a Monday of -2.13 per cent.

The file also supplies the repair. The next file's close minus its change recovers the missing close, and the method can be validated wherever both numbers exist: on the fourteen special sessions it matched the special file's own close in 14 of 14 cases, to within 0.05 of an index point. The reconciled record holds 3,396 sessions and 3,395 returns, with one two session composite left where two consecutive weekdays are missing. In aggregate the repair is small: annualised volatility moves from 16.20 to 16.13 per cent and excess kurtosis from 17.37 to 17.53. It is the session by session record that changes, and a cleaning rule acts on sessions. Security level data has a third category, the split or bonus that is neither a bad tick nor a move, which the guide to backtest integrity covers.

Ten detectors, tested on the real record and on planted errors

Each detector below is a standard way of finding outliers, run with its textbook threshold on the reconciled record, and each is measured twice. First, what it flags in the real record, which shows no sign of a recording error once reconciled: every session any detector flagged moved the same way as at least 6 of the ten sector indices, and every one beyond three standard deviations as at least 8, so each flag is an observation a deletion rule would destroy rather than an error it would remove. Second, what it catches when an error is known to be there. Every eligible session in turn, 3,321 of them, had its close replaced by one recorded three, five or eight standard deviations too low, correct again the next session. The experiment is exhaustive rather than random, so it needs no seed and carries no sampling error.

Read across a row and each detector's trade is visible. The fixed 5 per cent threshold caught only 43 per cent of prints five standard deviations wide, because a planted error adds to the day's real move and a down print on an up day stays inside the line; the 10 per cent threshold caught none at that size. The whole record standard deviation rule caught 98 per cent and flagged 44 real sessions to do it. The median absolute deviation caught 99 per cent and flagged 98. The cross check caught 98 per cent and flagged 2. Only one family tells an error from a move, because only one looks at evidence the error cannot fake.

The standard deviation rule defeats itself

A standard deviation rule asks whether a value is far from the centre in units of the spread, and it estimates the spread from the same values it is testing. On fat tailed data the extremes inflate the yardstick meant to catch them. The whole record standard deviation of the broad index is 1.016 per cent a day. Clip everything beyond three of those, recompute, and repeat until nothing more goes, and the core that remains has a standard deviation of 0.799 per cent. The 85 sessions removed on the way, 2.5 per cent of the record, had widened the yardstick by 27 per cent. The procedure took 5 passes because each removal shrinks the scale and exposes the next layer, which is why repeated clipping of a fat tailed series does not converge on the errors. It converges on a thinner market.

A window that includes the value being tested makes the problem absolute. In a sample of n observations no single value can sit more than n minus 1, divided by the square root of n, sample standard deviations from the sample mean, a classical inequality. For a 10 session window that ceiling is 2.85, so a three standard deviation rule computed over a trailing window that includes today cannot flag anything, on any data; on this record the largest value it produced was 2.69. For a 20 session window the ceiling is 4.25, so a five standard deviation rule built that way never fires. The usual rolling window functions in dataframe libraries include the current row unless told otherwise, so this is not a contrived construction.

Excluding the tested value fixes the arithmetic and not the masking. A trailing rule compares each session with the 63 before it, and every real shock raises its threshold for the next three months.

The planted errors measure the cost. The trailing rule caught 99 per cent of five standard deviation prints in the calmest third of sessions and 60 per cent in the most turbulent third. Across March and April 2020 it caught 26 per cent, and the centred robust filter, whose window was full of real crash sessions, caught 5 per cent. A detector built on the series' own spread is least able to see an error exactly when an error does most damage to a risk estimate, because that is when the spread is widest. The measurement of how far a standard deviation understates the tail is the same fact seen from the other side.

Remove, winsorise or flag: three different records

Once a session is flagged there are three things to do with it. Removal deletes it, so the path the returns imply skips that move. Winsorisation keeps the session and clamps its size to the rule's own boundary. Flagging changes nothing and keeps the list. Each produces a different record, and the statistics below describe the record, not the market.

The spread is the finding. The same data gives annualised volatility from 12.5 to 16.1 per cent, excess kurtosis from 0.2 to 22.2, and a worst drawdown from 16.0 to 44.0 per cent. None of the rules is unreasonable on its face; each is a standard construction. Removing the 11 sessions beyond 5 per cent roughly halves the worst drawdown, and after the robust rule removes 98 sessions the worst episode in the record is no longer the pandemic crash at all but a decline from 2024-09-26 to 2025-03-04.

The direction is not random, and the mechanism is the skew. The record's largest moves lean down: of the 44 sessions beyond three standard deviations 25 were falls and 19 rises, and of the 98 robust flags 59 were falls and 39 rises. A symmetric rule applied to an asymmetric tail deletes more losses than gains, so the mean rises while the volatility falls, and every risk statistic moves toward calm.

Two rows run the other way, and they matter more than the ones that flatter, because they show that a cleaning rule does not simply lower risk estimates. It replaces them with the rule's own shape. The centred robust filter deleted 94 sessions and raised excess kurtosis from 17.5 to 22.2: its window could see an isolated shock or an ordinary move in a calm fortnight, but not a crisis session surrounded by other crisis sessions, so it removed the shoulders of the distribution and kept the clustered extremes that dominate the tail. Deleting the 12 sessions with a bad print's reversal signature deepened the worst drawdown to 44.0 per cent, because the rebound sessions it deleted, 13 and 20 March 2020 among them, sat inside the worst fall in the record.

Flagging is the only treatment under which a risk estimate remains an estimate of what happened. It costs nothing but discipline: the statistic is computed on the full record, the flagged list is kept beside it, and any sensitivity is reported as a second figure rather than substituted for the first. The worst drawdown is already a noisy sample statistic, and a cleaning rule adds a second source of variation to it that has nothing to do with the market.

A move that appears everywhere is a market move

A bad print is a property of one record. A market move is a property of many. On the real record the evidence is unanimous: all 44 sessions beyond three standard deviations were matched by at least 8 of the ten sector indices moving the same way, 32 of them by all ten, and the median sector move ran between 0.40 and 1.54 times the broad index's. The planted prints were matched by nothing, which is why the cross check caught them.

The converse does not hold. A move that appears alone is suspect, not guilty. Scanning the ten sector indices for sessions beyond six robust standard deviations of their own history, on which the broad index moved less than 1 per cent, finds 10, and every one of them is real.

The public sector bank index rose 29.6 per cent on 25 October 2017, the day after the government announced a ₹2.11 lakh crore recapitalisation plan for public sector banks. The metals index fell 8.1 per cent on 23 May 2022, after export duties on iron ore and steel were notified on 21 May. No other sector corroborates either, because no other sector shared the cause. The corroboration lives in the series that did: the banking index, which includes the largest public sector bank, rose 3.4 per cent that October day, and on every one of the 10 sessions the sector index's own turnover ran at least 2.2 times its recent median. A bad print arrives with ordinary turnover, because nothing traded at it.

The cross check has a threshold too, and it produced two false flags. On 25 and 26 March 2020 every sector rose, but the broad index rose more than twice the median sector because banking led, at 8.0 and 6.1 per cent, and banking weighs heavily in a capitalisation weighted index. The rule as specified, that the median sector must reach half the broad move, flagged both. A version that asks only whether seven of ten sectors moved the same way flags no real session and misses 34 per cent of the planted prints, the sessions on which most sectors happened to drift in the print's direction. Every detector here trades one error for another, and the best of them still needs a person to read its list.

A cleaning rule chosen after the result is a fitted parameter

The statistics table above holds 19 versions of one record. Take the simplest statistic anyone reports, the ratio of mean daily return to its volatility, annualised, for merely holding the index. As recorded it is 0.63. Across the versions it runs from 0.59 to 1.29, and the highest comes from removing the 98 robust flags: a doubling produced by editing 2.9 per cent of the sessions, without one assumption about the market changing. These are gross measurements of a published index, not of anything tradable, and they describe the past only.

A researcher who tries several cleaning rules and keeps the one that makes the result look best has run a search, and the reported figure is the maximum of that search. It is the same defect as choosing the null after seeing the result, and it enters through the gap the measurement of multiple testing describes: once the result is known, whether an outlier day is a data error or a real event is decided by someone with an interest in one answer. No dishonesty is required. The rule that made the result legible is simply the one that gets written up.

The defence is sequence, not virtue. Fix the definition of a session, the detector, its threshold, its window and the treatment before looking at any performance statistic. If the rule changes afterwards, which is sometimes right, report both versions and say which came first.

What a stated cleaning rule contains

The session. Whether weekend special sessions are included, how sessions missing from the source are treated, and how many returns span more than one session. On this record that decision alone changed 25 returns and created one false outlier.

The detector. The statistic, the threshold, the window, and whether the window includes the value it tests. A rule stated without its window cannot be checked, because the window decides what it can see.

The treatment. Removed, winsorised to which boundary, repaired from which related series, or flagged only.

The list. Every affected session, with its size and the evidence for calling it an error: the related series that failed to move, the turnover, the source file. A flag without evidence is an opinion.

Both answers. The headline statistics with the flagged sessions and without them. If the two disagree materially, the conclusion rests on the cleaning rule, and a reader is entitled to know that.

The order. When the rule was fixed relative to the first result, and the data snapshot it was applied to. A result someone else can check needs the file, not a description of the period.

What cleaning is for

Cleaning exists to remove errors of recording. It is not a way to make a distribution resemble the model someone intends to fit to it. In an index record the extreme sessions are not contamination of the sample; they are the part of the sample a risk estimate exists to describe, and on this record deleting 11 of them halves the worst drawdown of 13.7 years. A series cleaned of them describes a milder market than the one that traded, and a rule tested on it has been tested on a market that did not exist.

The work that survives is unglamorous: reconcile the calendar, check each file against its neighbour, test every candidate against series that share its cause, delete only what the evidence convicts, and report the rest flagged. Treating the cleaning rule as a modelling decision, stated in advance and reported both ways, is method rather than a figure to memorise, and it is how this material is taught here.