What a reproducible analysis pipeline actually looks like
Reproducibility isn't a badge you add at the end — it's a set of small, boring decisions you make from the first day of a project. Here's how I think about it in practice.
A note from Ali Mirzakhani, founder of Neuroprocessing.
“Reproducible” is one of those words everyone agrees with and few people budget time for. It tends to arrive at the end of a project, when a reviewer asks for the code and you spend a weekend reconstructing what you actually did. I’ve come to think of it the other way around: reproducibility is a set of small, unglamorous decisions you make from the first day, and each one quietly pays you back later. Here is what it looks like when I build an analysis pipeline, using a recent project — a motor-skill assessment system — as the example.
The project is a folder, not a script
The first decision is structure. Raw data goes in one place and is never edited. Everything downstream is derived from it by code, so any processed file can be deleted and regenerated. That single rule — raw is read-only, everything else is disposable — removes a whole category of “wait, which version of the spreadsheet is this?” problems. If I can’t delete my results folder without anxiety, the pipeline isn’t finished.
Concretely, that means: a data/raw/ directory I treat as sacred, a
data/processed/ directory I could wipe at any moment, and a numbered set of
scripts that turn one into the other. The numbering isn’t decoration; it’s the
order a stranger should run them in.
The environment is part of the claim
Here’s the part people skip. When you report a result, you’re implicitly claiming “if you run this analysis on this data, you get this number.” That claim isn’t true unless the reader can build the same environment you had. Python version, library versions, the random seed — these are not IT trivia, they’re part of the scientific statement. A pinned environment file lives in the repository next to the code, and I’d rather over-specify it than let a future NumPy release quietly change a result.
I learned to take this seriously the first time an analysis gave me two slightly different effect sizes on two machines. Nothing was “wrong” — a dependency had moved. But if a version bump can move your number, then the version is part of your method, and it belongs in the record.
Quality control is reported, not hidden
In the motor-skill project, data comes off custom instrumentation: contact sensors sampled at roughly 3.6 Hz and 60 fps video, with markerless kinematics estimated using DeepLabCut. Real acquisition is messy. Trials drop out. Tracking occasionally loses a limb. A frame is missing.
The temptation is to silently clean all of that and present the tidy survivors. The reproducible version does the opposite: it counts. How many trials went in, how many were excluded, and for exactly which reason. Across the 84-trial dataset, the pipeline reports its own retention — so anyone reading can see what was kept, what wasn’t, and whether the exclusions could have shaped the outcome. Quality control you can’t see is just a decision you’re asking people to trust. Quality control you report is evidence.
The subject is the unit of analysis
This one is easy to get wrong in a way that looks rigorous. If you pool every trial from every subject into one big pile and run your statistics on that, you can end up with impressive-looking numbers that mostly reflect a few talkative subjects. Trials within a subject aren’t independent, and treating them as if they are inflates your confidence.
So the pipeline aggregates to the subject first, and models the structure explicitly — linear mixed-effects models for the movement trajectories, with effect sizes reported alongside. The question I’m actually asking is about people, not about trials, and the analysis has to respect that. It’s slower and the numbers are usually more modest. That’s the point.
None of this is about tooling
You’ll notice I haven’t recommended a framework. That’s deliberate. Reproducibility isn’t a library you install; it’s a habit of leaving a clear trail — raw data you never touch, an environment you can rebuild, quality control you report out loud, and statistics that match the question. The tools are interchangeable. The discipline is the thing.
This is also why Neuroprocessing is being built in the open. The code for this particular project will be released when the work is published. Until then, the habits above are the part worth sharing — because they’re the part that travels from one project to the next.
If you do this kind of work and want to compare notes, I’d genuinely like to hear from you: get in touch.