When Stephanie Noble, a new assistant professor of psychology in the Center for Cognitive and Brain Health, says that psychology and neuroscience are facing a “reproducibility crisis,” what she means is that the scientific method itself is at risk. Scientists are publishing results that other scientists may not be able to duplicate.
And if a study cannot be reproduced, how can we trust that its results are accurate?
The reasons for this crisis are complex, but include practices open to interpretation, the quirks of algorithmic software and the general difficulty of establishing methodological standards in “complex and quickly evolving” fields, Noble says.
Noble is seeking to change that. Her research, on “precision neuroscience,” is a kind of meta-research, aiming to help other researchers do their work better. With her work on functional MRI, Noble focuses on “getting better measurements, precision and to address questions and concerns in the field about reproducibility,” she says.
Aiding her endeavors is a transition grant provided by the National Institute of Mental Health. The award supports a researcher’s transition from postdoctoral work through the first three years of their professorship.
Noble will use this ongoing funding to design a “power calculator” tailored to neuroimaging studies.
Scientists use power calculators to determine the sample size of an experiment — most commonly, how many subjects they might need. The larger the study, the higher the precision of the experiment.
Noble notes that “Federal agencies often require these [calculations] when researchers apply for funding, in order to ensure that their planned research will be rigorous.”
But the calculations become more complex as the complexity of the dataset — and of the researchers’ own practices — grows. Functional MRI research involves both “very complex data and very complex inferential procedures,” Noble says.
While other power calculators already exist, they “are really geared towards very simple data and simple inferential procedures,” which leads to downstream problems for researchers.
“How many subjects might I need?” Noble asks. “Can I even get a ballpark? Right now, we don’t really have great tools for that. But the idea is to build this thing into a really simple, easy-to-use web app where somebody can just specify some parameters that describe their study, and then they just click the button.”
Much of Noble’s work is computational, intended to aid her colleagues on the applied side of the field. “Now that we have these large datascets and the computational power to be able to make these more precise for everyday researchers,” she continues, “the idea is to create a tool that anybody can use and that everybody should be using in the [neuroimaging] field to get an estimate of how they should be planning their studies.”
“Reproducibility,” she says, “affects every single researcher.” With a power calculator better tailored to their research, researchers will produce results that can ultimately be confirmed by others in their field — leading to more trustworthy science for everyone.
Noah Lloyd is a Senior Writer for NGN Research. Email him at n.lloyd@northeastern.edu. Follow him on Twitter at @noahghola.