Parameter Sensitivity: If the Edge Dies at n=21, It Was Never There

A single result is a coordinate, not a measurement

Consider what a strategy result at one setting actually is. You chose a rule with an adjustable number in it, you evaluated the rule at that number against a stretch of history, and you recorded what came out. That is one point. It has no error bar, no context, and no indication of whether it is representative of anything. It is a coordinate in a space you have not looked at.

The reason this matters is that the space around the point is not smooth in the way most people assume. Market history contains behaviour that repeats and behaviour that happened once. A setting that captures repeating behaviour will have neighbours that also capture it, because the repeating behaviour has a rough timescale and any setting near that timescale will detect some of it. A setting that fits behaviour which happened once will have neighbours that do not, because the arrangement was accidental and accidents are not shared between nearby settings. So the shape of the neighbourhood carries information the point cannot.

This is why the practice of quoting a single optimised figure is not merely imprecise but actively misleading. The number a trader reports is almost always the maximum of the search, and the maximum is the setting most contaminated by whatever went right by chance. If you ran a hundred settings and kept the best, you did not measure a strategy. You measured the right hand edge of a distribution, and the distribution is the part you threw away.

Our catalogue of the eight ways a backtest lies treats this under multiple testing, and states the core result: the maximum of many noisy trials is high by construction, so the count of trials has to be recorded and the headline discounted for the width of the search. That is the correct diagnosis and it is where this subject should begin. What that section does not do, because it is a survey of eight failures rather than a study of one, is show you what to do with the search once you have run it. The count of trials tells you how much to distrust the winner. The shape of the results tells you whether there was anything to win.

The distinction is practical. Two traders can each test a hundred settings and report an identical best figure. One of them tested a rule whose good region spans thirty settings; the other tested a rule with one lucky setting in a hundred. Their trial counts are the same, their headline numbers are the same, and their situations are not remotely comparable. Nothing in a trial count distinguishes them. The sensitivity profile does, immediately, and it costs nothing extra to produce because you already ran every setting.

Plateau and spike, and what each one is telling you

Sweep one parameter across a wide range and plot the result against the setting. Two shapes are worth naming.

A plateau is a broad region in which the result changes slowly. Move the lookback from twenty to sixteen or twenty-six and the outcome shifts a little, in proportion to how far you moved. The peak inside a plateau is not special; it is simply the highest point of a gentle mound, and if you re-ran the exercise on a different stretch of history the peak would land somewhere else inside the same mound. A plateau says: there is an effect here whose timescale is roughly this, and a range of settings can see it.

A spike is an isolated setting that is excellent while its surroundings are ordinary or negative. There is no gentle mound. There is one value that worked, sitting in a field that did not. A spike says: something in this specific history lined up with this specific setting. Nothing about the arrangement is a property of the market, so nothing about it will be there next year.

The reason the distinction is decision-relevant rather than merely descriptive is that you have to choose one setting to trade. If the results form a plateau, your choice barely matters, which is another way of saying the strategy does not depend on getting a number right that you have no way of getting right. If the results form a spike, your choice is everything, and you are betting that a value selected from history will remain the correct value in a future that has not been consulted. The first bet is modest. The second is close to a coin flip dressed up as a decision.

Stated that way, the recommendation is almost obvious: do not select the maximum, select the centre of the widest stable region, and accept a lower headline figure in exchange for a result that might survive. That is the conventional advice and it is what this page set out to demonstrate. What the computation actually found is more interesting than the conventional advice, and the section further down reports it without softening.

The setup, stated in full so it can be checked

The worked example uses a simulated daily series rather than a live instrument. A simulated series lets the regimes be stated openly, keeps the whole exercise reproducible, and avoids implying that any particular Indian instrument would have produced any particular outcome. Two design choices in the table below do more work than the rest combined and are worth reading closely.

The first choice worth dwelling on is the removal of net drift. A long-only rule on a rising series makes money by being invested, and the longer the channel the longer it stays invested, so the sweep would show a smooth upward slope that has nothing to do with timing skill at all. Deleting the drift removes that artefact, and it means every rupee visible in the curves further down came from when the rule was in the market rather than from the market going up. This is a stronger test than the usual one, and it is also the only way to make the control series meaningful.

The second choice is that control series. Shuffling the daily returns is the cleanest possible way to build a twin that matches on everything except the property being tested. The mean matches to the last decimal, the standard deviation matches, the individual returns are the identical 3,000 numbers, and the buy and hold result matches exactly, because multiplying a set of numbers together does not depend on their order. What shuffling destroys is the sequence, and the sequence is the only thing a timing rule can use. Whatever a search finds in the shuffled twin is therefore a pure measurement of what your search procedure can manufacture out of nothing.

What the sweep actually looked like

Running every lookback from five days to a hundred and twenty on both series produces two curves. They are drawn on the same axes below, at the same scale, so the comparison is not doing any work through presentation.

Take the structured series first. The best setting was a twenty-four day channel, returning 10.52 percent a year. The median setting across the sweep returned 7.66 percent, the worst setting in the entire range still returned 3.48 percent, and every one of the 116 settings made money. Twenty-four of them landed within a fifth of the peak, and the widest unbroken run of those spans seventeen to thirty-one days. This is a plateau in the sense that matters: no matter where in the range you had chosen, you would have had a workable result, and choosing well would have improved it modestly rather than transformed it.

Now the shuffled twin. The best setting was an eighty-five day channel returning 2.95 percent a year, which on its own would look like a perfectly respectable finding. The median setting returned 1.47 percent. Twenty-two percent of the settings lost money, and the worst lost 3.10 percent a year. The curve crosses zero repeatedly and wanders without direction, because there is no direction in it to find. Every one of those results came from a process that contains, by construction, no relationship between one day and the next.

The single most useful comparison between the two is the neighbourhood, not the peak. Around the structured peak, the band from eighteen to thirty days averaged 9.58 percent, its worst member returned 8.58 percent, and the spread across it was half a percentage point. The neighbourhood retained 91 percent of the peak. Around the shuffled peak, the corresponding band from sixty-four to a hundred and six averaged 1.91 percent, its worst member returned 0.18 percent, which is nothing at all, and the spread across it was 0.80 points on a mean of 1.91. The neighbourhood retained 65 percent of the peak, and the variation inside it was 42 percent of its own level against 6 percent on the structured curve.

That is what the two shapes look like when they are measured rather than described. The structured peak is the top of something. The shuffled peak is the top of nothing, and the giveaway is not the height of the point but the behaviour of everything near it.

One more figure from that sweep deserves emphasis because it reframes the headline. On the structured series, the gap between the best setting and the median setting was 2.86 points, which is 27 percent of the peak. On the shuffled twin the same gap was 1.47 points, which is 50 percent of its peak. On the series with nothing in it, half the reported result was pure selection. On the series with something in it, roughly a quarter was. Neither is zero, which is the point: selection inflates every optimised figure, and it inflates the meaningless ones most.

The check almost everybody runs, and why it does not work

The standard robustness check, when one is run at all, is to look at the settings either side of the chosen one. Twenty-one worked, so try twenty and twenty-two. If they are similar, the result is declared stable. This check is close to worthless, and the sweep above shows exactly why.

On the structured series, the peak stood 0.29 points above the average of its two immediate neighbours. On the shuffled twin, where there is provably nothing to find, the peak stood 0.29 points above the average of its two immediate neighbours. The same number, to two decimal places, in a case with a real effect and a case with none. Averaged across the whole sweep, the step between adjacent settings was 0.26 points on the structured curve and 0.29 on the shuffled one. Local smoothness does not distinguish the two cases at all.

The reason is mechanical once stated. A twenty-day channel and a twenty-one day channel produce nearly the same signals on nearly the same days, so they hold nearly the same positions and earn nearly the same result. Two adjacent settings are not two tests. They are one test, run twice, with a rounding difference. Any curve produced by a smoothing-window parameter will look locally smooth whether or not it means anything, and a check that measures local smoothness is therefore measuring a property of the parameter rather than a property of the strategy.

The correction is to make the perturbation proportional. Moving a twenty-day lookback by one day is a five percent change. Moving a hundred-day lookback by one day is a one percent change. Neither is a test. Moving either by a quarter of its own value is a test, because the resulting rule genuinely trades differently. Every neighbourhood figure on this page is defined that way, as the band within a quarter of the setting in each direction, which is why the structured band runs from eighteen to thirty around a peak of twenty-four and the shuffled band runs from sixty-four to a hundred and six around a peak of eighty-five.

Read the table as a warning about instruments rather than about strategies. Four of the nine diagnostics are the ones people actually run, and all four fail. The five that work share a feature: they look at the whole field rather than at the winner, and they express the winner in units of the field rather than in percent a year. That is the practical lesson from the entire exercise, and it can be applied to any optimisation output you already have sitting in a spreadsheet.

Two parameters, where the difference stops being subtle

Most real rules have more than one number in them. The breakout used here can take a separate lookback for entry and for exit, which is both realistic and a much harder test, because a two-dimensional search offers many more ways to get lucky. Sweeping both from five to a hundred and twenty in steps of five gives 576 combinations on each series.

On the structured series every one of the 576 combinations made money, the worst returning 2.87 percent a year and the median 7.59 percent. The peak, at a fifteen-day entry against a twenty-five day exit, returned 10.44 percent, and its eight immediate neighbours averaged 9.65 percent with the weakest at 8.83 percent. A hundred and six cells, more than a sixth of the surface, sat within a fifth of the peak. Those cells fall into five separate clusters, but eighty-two of them, more than three quarters, sit in a single connected block. Choosing badly within that block costs something. Choosing badly outside it costs a little more. Nothing about the surface is a cliff.

On the shuffled twin, a fifth of the surface loses money, the worst combination losing 4.88 percent a year against a median of 1.35 percent. The peak returned 3.00 percent, and the fifty-six cells within a fifth of it break into ten separate clusters, the largest holding thirty-four. That fragmentation is the visual signature of a search finding accidents rather than structure. If you had run this search, reported the peak, and been asked whether the result was robust, you could truthfully have said that fifty-six of the 576 combinations landed close to it, and you would have been describing scattered noise.

Be precise about the strength of that contrast, because it is a difference of degree rather than of kind. Five clusters against ten, and three quarters of the good cells together against three fifths, is a real gap but not a chasm. The blunter fact from the same two grids is the one to lead with: the entire structured surface made money and its worst corner still returned 2.87 percent, while a fifth of the shuffled surface lost money and its worst corner lost 4.88 percent. In one dimension a plateau and a spike field can look similar enough to argue about. In two, there is simply more evidence, and it points the same way.

The test that decides it: what each way of choosing then delivered

Everything so far describes shapes. The question that matters is whether reading the shape improves the decision, and the only way to answer it is to choose a setting on one stretch of data and then apply it, unchanged, to a stretch the choice never saw. The experiment below does that 600 times on 600 independently generated series: settings are chosen on the first 7.1 years and scored on the following 4.8 years, which the selection procedure has never been shown.

Four ways of choosing were compared. Take the peak of the sweep. Take the centre of the best neighbourhood, defined as the setting whose proportional band has the highest average. Never optimise at all and fix the lookback at forty days. And never choose at all, which is measured as the average of what all 116 settings produced, since that is what you would expect from picking one at random.

Peak-picking promised 8.78 percent a year on the data it could study and delivered 5.86 percent on data it had not seen. Two thirds of the promise survived, which is to say a third of it was never there. The plateau centre promised 8.01 percent and delivered 5.76 percent, retaining 72 percent. The pre-committed forty-day setting promised 6.61 percent and delivered 6.48 percent, retaining 98 percent. Doing nothing at all promised 6.04 percent and delivered 5.53 percent.

Now the result that complicates the argument, and it is the reason this section exists in this form. Selecting the plateau centre did not deliver more than selecting the peak. It delivered 5.76 percent against 5.86 percent, a difference of a tenth of a point in the wrong direction, and it came out ahead in only 41 percent of the 600 runs. The conventional advice, taken at face value as a way to earn more, is not supported by this experiment. Widening the neighbourhood further did not rescue it either: a band of a tenth returned 5.80 percent and a band of two fifths returned 5.64 percent, all of them clustered around the same place.

What the plateau centre did buy is worth having, and it is a different thing from a higher result. It promised 0.77 points less, and since both approaches delivered the same amount, the plateau centre's forecast was simply closer to the truth. A trader who sized a position off the peak figure was planning around a number that was 50 percent too high. A trader who sized off the neighbourhood figure was planning around a number that was 39 percent too high. Neither is good. One is meaningfully less bad, and the difference lands directly in position sizing, in expectations, and in how long someone persists with a system that is merely performing as it should.

The larger and more uncomfortable finding is in the other two rows. A setting fixed in advance at forty days, chosen without reference to any individual run, delivered 6.48 percent against the optimiser's 5.86 percent and beat the optimiser in 61 percent of the 600 runs. The honest caveat is that forty is a round number in the middle of the range and it was not selected in total ignorance of how this process behaves, so it is not a fully blind choice and should not be read as one. The assumption-free comparison is the last row, where no choice is made at all: that delivered 5.53 percent, only 0.33 points below the optimiser, with a materially tighter distribution across the 600 runs and a worse case that was better.

Set those two figures side by side, because together they are the whole argument. Against making no choice at all, the optimiser advertised an improvement of 2.74 points a year. It delivered an improvement of 0.33. Twelve percent of the advertised value of the optimisation survived contact with unseen data, and it was bought with an eighteen percent wider spread of outcomes and a tenth percentile that was worse. That is what parameter optimisation is worth on a process where the underlying effect is genuinely present. On a process where it is not, the next section shows what happens.

One last row from the same experiment is worth reporting because it disposes of any remaining hope. The setting that turned out to be best on the unseen block would have returned 8.73 percent, so there was real value available. That setting looked entirely unremarkable in the search window, scoring 5.92 percent when the peak was scoring 8.78. The optimiser cannot find it, because nothing in the search window points to it.

What an optimiser finds when there is nothing to find

The most valuable single test in this whole exercise costs one line of code and is almost never run. Take your returns, shuffle them into a random order, and run your identical search on the shuffled series. You have not changed the mean, the volatility, the tails or the buy and hold result. You have deleted the only thing a timing rule can use. Whatever your search now reports is the size of the number your procedure fabricates from nothing.

Repeating the full 600-run experiment on shuffled series gives the answer. Peak-picking found a best setting every single time, and those best settings promised an average of 2.57 percent a year on the search window. Applied to the unseen block, they delivered an average loss of 0.69 percent a year and lost money in 61 percent of runs. Selecting the plateau centre instead made no difference here, promising 1.82 percent and delivering a loss of 0.76 percent, which is exactly as it should be: there is no plateau to find, so no procedure for finding one can help.

Sit with the first of those figures. An honest, carefully coded, look-ahead-free optimisation, run on a process containing no exploitable structure whatsoever, produced a headline of 2.57 percent a year, six hundred times out of six hundred. It never once reported that there was nothing there. Optimisers do not have a way of returning that answer. They return the best of what they were given, and the best of a noisy field is always positive-looking.

This is why the shuffle test belongs in every serious validation routine, and why it is more informative than most of the elaborate procedures that get run instead. It gives you a concrete, personal, procedure-specific number: for my rule, my data length and my search width, the noise floor is roughly two and a half percent a year. Any real result has to clear that floor before it is a result at all. And unlike a rule of thumb borrowed from someone else's paper, it is measured on your own setup, which is the only place the number was ever going to be valid.

The selection rule this argues for

The findings above do not add up to the advice this page expected to give. They add up to something narrower and more defensible.

Sweep wide and plot it. Three values are not a sensitivity analysis. Run the full plausible range at a fine step and draw the curve, because a curve you have not drawn is a curve you cannot read, and the shape is the entire content of the exercise. The cost of the extra settings is a few seconds of computer time.

Score the neighbourhood, never the point. Replace the result at each setting with the average result across settings within a quarter of its own value. That single transformation encodes everything this page has demonstrated: a lone spike scores badly because its surroundings drag it down, and the middle of a broad shelf scores well because its surroundings support it. Then read your answer off the transformed curve.

Quote the neighbourhood figure, not the peak. The number that goes into your notes, your position sizing and your expectations is the neighbourhood average. It will be lower than the peak, by 0.77 points on average in the experiment above, and it will be closer to what you actually get. This is the step that survives every objection, because it costs nothing and cannot make the decision worse.

Run the shuffle control. Same returns, random order, identical search. If your search finds two and a half percent a year in the shuffled version, then a real result of three percent is not a result. Very few people run this, and it is the cheapest genuine information available anywhere in strategy validation.

Decide the width threshold before the sweep runs. Write down, while the answer is still unknown, how wide the stable region has to be and how much of the peak it has to retain before you are willing to trade anything inside it. Deciding this afterwards is not a decision, it is a description of what you already found, and it is the point at which the whole exercise turns decorative.

What the rule does not promise is a better return. The experiment says plainly that it will not reliably give you one. What it gives you is a forecast you can size against, a floor you can compare against, and a defensible reason to reject a strategy whose good region turns out to be one setting wide. The most valuable output of a sensitivity study is usually a rejection, and a procedure that never produces one is not a validation procedure.

Where this sits next to walk-forward testing

Sensitivity analysis and walk-forward analysis are frequently discussed as alternatives, as though a trader should pick one. They test different axes of the same object, and passing either one says nothing about the other.

Walk-forward tests across time. It asks whether settings chosen using information available at one date still work at a later date, by re-choosing at every step and scoring only on blocks the choice never saw. Its characteristic finding is degradation: the promise made on data the optimiser could study is larger than the delivery on data it could not.

Sensitivity tests across parameter space. It asks whether the settings around the chosen one behave like it, at a single point in time. Its characteristic finding is fragility: a result that exists only at one coordinate is not a result.

A strategy can pass one and fail the other, and both failure modes are common. A rule can sit on a beautiful plateau and still degrade across time, because the entire plateau was a feature of one market regime that has since ended. A rule can hold up across every walk-forward window and still be balanced on a spike, because the same accidental alignment persisted through the whole sample. Running both is not redundancy. The two tests answer questions that do not overlap, and the arithmetic for each takes an afternoon.

There is also a shared trap, and it is the one that ruins both. Both procedures are worthless if you run them, dislike the answer, adjust the rule, and run them again. At that point the test data has participated in choosing what gets tested, and the protection is spent silently, with no error message. The order of operations in a proper backtest exists mainly to make that failure hard to commit by accident.

What the sweep is actually for

It would be reasonable to finish this page believing that optimisation is pointless. That is not quite the conclusion, and the more precise version is more useful.

An optimised result answers a real question: what was the best available version of this rule, in hindsight, on this exact history. The answer is close to useless as a forecast, but it is not meaningless. It bounds what the rule could possibly have done, and the distance between that bound and the median of the field tells you how much room chance had to operate. A sensitivity sweep answers a different question: is there anything here that a range of settings can see, or is there one accident? That question has a stable answer, because the shape of a field is a more durable property than the height of its highest point.

The portable habit, the one worth carrying to every strategy you look at afterwards, is to stop reading optimisation output as a ranked table and start reading it as a surface. The ranked table puts the most contaminated result at the top and hides the field that produced it. The surface shows you the field, and the field is where all the information is. Once you are reading surfaces, the question of which exact setting to use largely dissolves, which is the strongest possible evidence that it was never the important question.

This applies well beyond an optimiser. When someone shows you a strategy, ask what the results looked like at settings either side of the one they are quoting, and ask for the whole sweep rather than three points. If the answer is a broad region, they are describing something about the market. If the answer is a single value, or if the sweep was never run, you now know how much weight the number can carry. The same question works on your own work, where it is considerably harder to ask honestly. Establishing whether an edge exists at all is a separate discipline again, and it starts from the same place: distrust any single number that has not been asked what its neighbours are doing.

None of this requires advanced mathematics. It requires plotting the whole sweep instead of reading the top row, expressing the winner in units of the field it came from, running the shuffle control that tells you your own noise floor, and writing the acceptance threshold down before the answer is visible. Those are habits rather than techniques, which is both the good news and the reason they are usually skipped. They are also the substance of what serious work on systematic trading consists of, and if the arithmetic on this page was the interesting part rather than the tedious part, that is the method we teach.

Every figure comes from a single deterministic simulation, seeded so that it reproduces identically on each run. The price series is synthetic, built from twelve stated regimes with defined drift and volatility, and it is not a model of any specific security or index. A constant is subtracted from every daily return so that the series has exactly zero net drift across the span, which removes the component of any long-only result that comes from the market rising rather than from the rule's timing. The control series holds the identical 3,000 returns in a random order, verified in code to be the same multiset with the same mean and standard deviation and the same buy and hold total.

Returns are computed on closes, positions are taken from the bar after the signal so that no future information can enter, and 12 basis points per round trip is deducted on every position change. The one-dimensional sweep evaluates all 116 lookbacks from five to a hundred and twenty days. The two-dimensional sweep evaluates all 576 combinations of entry and exit channel from five to a hundred and twenty in steps of five. The selection experiment repeats the whole procedure on 600 independently generated series, choosing on the first 1,800 days and scoring on the remaining 1,200.

All results are illustrative and simulated. They are not a track record, they are not a forecast, and they are not an indication of what any strategy would produce in a live account. The purpose of the exercise is to demonstrate the relationship between the shape of a parameter sweep and the reliability of the setting chosen from it, which is a property of the testing procedure rather than of any particular market.

• Building a trading system →
• Regime filters, and when a rule should stand aside →
• What a breakout actually is →
• Algorithmic trading in India →