Guide · Research method · Updated 2026-09-19

An unseeded result is not a result, it is one draw from a distribution of results

This is the least interesting subject in quantitative work and the one that decides whether any of the rest of it means anything. A number nobody can recreate is not evidence. It is a claim with arithmetic attached.

Reproducing is not replicating, and the difference decides what you need

Two words get used interchangeably and they are not the same requirement. Reproducing a result means taking the same data and the same code and arriving at the same answer. Replicating it means running a fresh study, on your own data and your own implementation, and reaching the same conclusion.

Reproduction is a disclosure problem. Replication is a truth problem. The uncomfortable consequence is that a result can be perfectly reproducible and entirely wrong, because reproducing a mistake reproduces it exactly. Reproducibility is therefore not a claim of correctness. It is the precondition for anyone being able to find out.

Where the randomness actually enters

Most people setting out to make work reproducible think of simulation, and stop there. Randomness enters a quantitative workflow in more places than that, and every one of them moves the answer.

The last two are the ones that surprise people, because no random number generator is involved. Floating point addition is not associative: summing the same values in a different order can give a slightly different total. Usually that is invisible. At a threshold, it is the whole decision.

The measurement

Rather than assert that an unseeded result moves, here is the measurement. A fixed series of 500 observations, a bootstrap of 400 replications, a ninety five per cent interval on the mean. The same code, the same data, twelve times over, each run landing on its own random draw exactly as an uncontrolled run would.

Across all twelve, the lower bound ranged from -0.0600 to -0.0433 and the upper from +0.1177 to +0.1369. The interval's own width varied between 0.1635 and 0.1895. Publishing any one of those twelve as the answer, with no indication that eleven others were equally available, is the ordinary practice.

Running the identical procedure with a seed fixed at a stated value produced identical output on all twelve runs, at -0.0490 to +0.1236. The fix is one line and it is free.

How many replications is enough, measured

A seed makes a result repeatable. It does not make it stable, in the sense of being insensitive to the arbitrary choice of seed. That is a separate question with its own measurable answer: run the whole procedure repeatedly on different draws, and see how much the reported figure moves at each replication count.

The shape of that table is the useful part. Going from 100 to 20,000 replications is two hundred times the computation and cut the wobble by roughly 13 times, because the error falls with the square root of the count. There is no replication number at which the answer stops moving, only one at which it stops moving in the digits you intend to quote. That is the number to find, and it is why the replication count belongs next to the result.

The instrument that agreed with itself perfectly

This article nearly shipped with a fabricated result, and the way it happened is worth the space because it is exactly the failure the subject is about.

The first version built its series like this:

series = [random.Random(11).gauss(0, 1) for _ in range(500)]

That rebuilds the generator on every iteration from the same seed, so it returns five hundred identical values. Every bootstrap sample of a constant series has the same mean, so every interval came out with zero width, and every run agreed with every other run to the last digit.

The output looked like a flawless demonstration of reproducibility. It was a broken instrument reporting nothing at all. What caught it was not the code review but the number: an interval of exactly zero width is not a good result, it is an impossible one.

The fix in the shipped script is two lines, and the second matters more than the first: seed once outside the comprehension, then assert the series is not degenerate before anything is built on it. The control now reports 500 distinct values at a standard deviation of 1.0169, and the assertion fails the build if that ever stops being true.

The general rule this illustrates has nothing to do with this particular bug. A result that looks impossibly clean is a defect report. Perfect agreement, a zero variance, a correlation of exactly one, a p-value of exactly zero: each is far more likely to be an instrument reporting on itself than a discovery.

The data snapshot, which is the hard one

Seeds are easy. The genuinely difficult part of reproducibility in market work is that the data moves underneath you, and naming a period does not pin it down.

The consequence is specific: a period is not a snapshot. Two people who both say they used the last ten years, from the same provider, can hold materially different data. What identifies a snapshot is the source, the period, the adjustment convention and the date it was pulled, and the only fully reliable version is to keep the file.

The minimum disclosure

Everything above reduces to a short list. It is worth applying to your own work first, where it is hardest to pass.

The seventh row is the one that carries the most information and is almost never present, because nobody records it as they go. It is also the only item on the list that cannot be reconstructed afterwards: seeds can be re-run, data can be re-pulled, versions can be looked up, but the count of discarded attempts exists only if somebody wrote it down at the time.

When the data cannot be shared, which is most of the time

The standard advice is to publish the data alongside the result, and for market work that advice is frequently impossible to follow. Exchange and vendor data is licensed, and a licence to use it is not a licence to redistribute it. Anyone working from a paid feed is contractually unable to do the thing reproducibility guides tell them to do.

Treating that as the end of the matter is the common response and it is wrong, because the licence blocks one specific item on the list and leaves the rest untouched. What can be published without redistributing anything is considerable.

The checksum line does more work than it looks. Most failed reproductions in practice are not disagreements about method at all: they are two people holding different data and arguing about the analysis. A hash published next to the result resolves that in seconds, and it discloses nothing about the contents of the file.

The synthetic-series line matters for a different reason. A pipeline that runs on generated data of the right shape is a pipeline somebody can execute, inspect and find a bug in, even if they can never reproduce your exact number. That is a large fraction of the value, and it survives any licence.

The order that makes all of this nearly free

Everything on this page is cheap while the work is happening and expensive or impossible afterwards. That asymmetry, rather than any principle, is the practical argument.

Read the right-hand column and the ordering becomes obvious. The items that cannot be recovered are the ones to set up before the first run, not after the interesting result appears, because by then the thing worth recording has already happened and gone unrecorded.

The last row deserves its own emphasis because it is the only item on the list that is irrecoverable in principle rather than merely in practice. A seed can be regenerated by re-running. A dataset can sometimes be re-purchased. The number of ideas you tried and discarded on the way to this one exists nowhere except in a note you did or did not make at the time, and without it nobody, including you, can say what the surviving result is worth.

What this looks like in practice, and what it costs

The smallest version that genuinely works is a single file that runs end to end and regenerates the result, with the seed at the top, the data source and snapshot date in a comment, and every parameter as a named constant rather than a number buried in the middle of a line. That is not a research infrastructure. It is an afternoon, and it puts a result ahead of most of what gets published.

It is worth being honest that full reproducibility, in the sense of bit-identical results on another machine in five years, is expensive and frequently unattainable. That is not an argument against the attempt. The realistic goal is weaker and far more useful: a reader should be able to see exactly what was done, and could redo it in principle. That is achievable at almost no cost, and it is enough for somebody to find the step they disagree with, which is the entire point.

There is also a self-interested case, and for most people it is the persuasive one. The reader who most often fails to reproduce your work is you, several months later, looking at a number you no longer remember producing. Everything here is cheaper to do while you are doing the work than to reconstruct afterwards, and reconstruction usually turns out to be impossible.