What a reproducible analysis pipeline actually looks like
Reproducibility tends to be treated as a finishing step. In practice it comes down to a few unglamorous decisions made at the very start of a project.
A note from the Neuroprocessing Team.
“Reproducible” is one of those words everyone nods along to and few people set time aside for. It usually surfaces at the end of a project, when a reviewer asks for the code and you lose a weekend reconstructing what you did months earlier. We have come round to the opposite habit: treating reproducibility as a series of small, unremarkable choices made on the first day, each of which quietly repays the effort later. What follows is what that looks like in practice, using a recent project of ours — a motor-skill assessment system — as the running example.
The project is a folder, not a script
The first choice is structure. Raw data goes in one place and is never touched again. Everything after it is produced from that raw data by code, which means any intermediate file can be thrown away and regenerated at will. That one rule — raw is read-only, everything downstream is disposable — quietly removes a whole family of “which version of this spreadsheet is the real one?” problems. Our rough test is simple: if we cannot delete the results folder without a flicker of anxiety, the pipeline is not finished.
In concrete terms it means a data/raw/ directory we treat as untouchable, a
data/processed/ directory we could wipe at any moment, and a numbered
sequence of scripts that turns the first into the second. The numbers are not
decoration; they are the order a stranger should run things in.
The environment is part of the claim
This is the part that tends to get skipped. When you report a result, you are quietly claiming that running this analysis on this data yields this number. The claim only holds if a reader can rebuild the same environment you had. Language version, library versions, the random seed — these are not clerical details, they are part of the scientific statement, and they belong in the record. We keep a pinned environment file beside the code, and would rather over-specify it than let some future library release move a result without anyone noticing.
We began taking this seriously the first time an analysis produced two slightly different effect sizes on two machines. Nothing was strictly wrong; a dependency had simply shifted underneath us. But if a version bump can move your number, then the version is part of your method.
Quality control belongs in the report
In the motor-skill work the data come off custom instrumentation: contact sensors sampled at roughly 3.6 Hz alongside 60 fps video, with limb positions recovered from that video using DeepLabCut. Real acquisition is untidy. Trials fail. Tracking loses a limb for a few frames. A file goes missing.
The tempting response is to clean all of that away quietly and present the survivors. We do the opposite, and count: how many trials went in, how many were set aside, and for exactly which reason. Across the 84-trial dataset the pipeline reports its own retention, so a reader can see what was kept, what was not, and whether those exclusions could plausibly have shaped the result. Quality control you cannot see is really just a decision you are being asked to take on faith. Quality control you write down is evidence.
The subject is the unit of analysis
This one is easy to get wrong in a way that still looks rigorous. Pool every trial from every subject into one heap, run your statistics on that, and you can end up with confident-looking numbers that mostly reflect a handful of talkative participants. Trials from the same person are not independent, and pretending they are inflates your certainty.
So the analysis aggregates to the subject before it does anything else, and models the structure explicitly: linear mixed-effects models for the movement trajectories, reported alongside effect sizes. The question is about people, not about trials, and the statistics have to answer the question that was actually asked. It is slower, and the numbers tend to come out more modest. We take that to be the honest outcome.
None of this depends on a particular tool
You will notice we have not recommended a framework. That is deliberate. Reproducibility is not something you install; it is the habit of leaving a legible trail — raw data left untouched, an environment that can be rebuilt, quality control written down, statistics matched to the question. Swap the tools for whichever ones you prefer. The habits are what carry from one project to the next.
It is also why we work in the open. The code for this particular study will be released once the work is published. Until then, these habits are the part worth sharing, because they travel further than any single result.
If you work on this kind of thing and would like to compare notes, we would genuinely be glad to hear from you: get in touch.