The goal of publishing careful replications—and, humbly, of this newsletter—is to gain a more accurate scientific picture of reality. Adhering closely to original methodologies and reporting all results are key, but often overlooked, steps in this process.
Enter preregistration, a process whereby researchers publish their intended methods before they begin their experiments, enabling the research community to evaluate how and why any subsequent deviations occurred. An even more rigorous approach is the registered report, in which a journal peer-reviews a study’s hypotheses and methods before data are collected and agrees to publish the findings no matter their outcome.
Neuroscience led the way when Cortex became the first journal to accept registered reports in 2013. The next year, the Center for Open Science began offering badges that recognize preregistered papers.
The study we are highlighting in this month’s newsletter examines the extent to which neuroscientists have used preregistrations and registered reports and how successful these have been in improving the rigor of science. Notably, it is itself a registered report—and the published paper details how the team deviated from their registered methods.
Have a notable null result or replication we should cover, or other suggestions for this newsletter? Send your tips to [email protected].
MATCHING METHODS
For researchers using electroencephalography (EEG), the path from hypothesis to findings is winding and full of branching. That makes accurately reported methodologies particularly important—and makes the field a great place to examine the effects of preregistration, says study principal investigator Peter Clayson, assistant professor of psychology at the University of South Florida.
Clayson and his team looked for all preregistered EEG methods and registered reports—as well as those measuring event-related potentials (ERPs), a brain response to specific stimuli—by searching Open Science’s database, as well as journals that use Open Science badges or accept registered reports. They also contacted investigators directly if they could not locate published data.
The team scored each paper based on how completely it reported eight criteria: research question, direction of hypothesis, variables collected, the operationalization of variables, sample size, data collection procedure, data exclusion criteria and statistical methodology.
Out of 385 preregistered studies and registered reports, 92 have so far resulted in published papers, only 15 of which—16 percent—fully adhered to their preregistration details, Clayson’s team found. On average, the criteria reported in the papers matched 60 percent of what was initially registered.
That’s nearly identical to a result Clayson published almost six years ago: Across 150 ERP papers published from 2011 to 2017, researchers included just 63 percent of the information recommended in methodological reporting guidelines—a lack of improvement Clayson calls “disheartening,” considering the advent of preregistration and registered reports.
“Clearly from our study, it’s not the panacea we were hoping it would be,” he says.
The lack of adherence doesn’t imply that researchers are committing fraud or otherwise acting nefariously, Clayson says. Rather, time-crunched researchers likely do not prioritize careful reporting of small deviations—particularly if they suspect those shifts won’t affect the final results.
“We don’t really know the impact of many of these decisions on the statistical analysis,” he says. “It could just be an underappreciation of the importance of these methodological steps.”
The differences may matter to researchers using EEG and ERP findings to identify biomarkers or design a clinical trial, says Emily Jones, professor of translational neurodevelopment at Birkbeck, University of London, who was not involved in the work. For example, an ERP response to faces called the N170 may be a biomarker for autism, Jones’ work has shown.
Potential solutions to the problem exist, Jones says. If a preregistered study turns up a null result, for example, researchers should at least note the results on their registration, even if they don’t publish a full paper.
Journals could also identify peer reviewers who are committed to carefully reviewing the preregistrations, Clayson says. And researchers can make use of available preregistration templates to streamline their writing process.
Preregistration and registered reports are, after all, relatively new to the publishing world.
“We’re looking at work in progress,” Jones says. “Papers like this are super important in terms of checking where we’re up to and where we need to go.”
ET AL.:
- Socioeconomic status does not affect how likely adolescents are to engage in risk-taking behavior, even among those who show heightened activity in the ventral striatum, an area involved in social reward. Developmental Cognitive Neuroscience
- Rats inclined toward impulsivity are no more likely to seek out cocaine than are their more deliberate counterparts. Cocaine use, however, seems to increase impulsive behaviors in the animals, flipping “the widely held view that impulsivity is a vulnerability factor for cocaine use disorder” on its head, the investigators write. eNeuro
- Rare variants in autism-linked genes influence a person’s chances of having the condition equally regardless of their sex. American Journal of Human Genetics
- The National Institute of Neurological Disorders and Stroke seeks to award the Early-Career Rigor Champions Prize—up to $50,000—to early-career scientists who have “promoted or enhanced research rigor practices” beyond the scope of their own work. challenge.gov
- A study of the brain’s ability to rapidly consolidate memories replicated papers from 2011 and 2012 showing that recalling one event tends to activate memories that are similar in content. Nature Neuroscience
- Dogs taught to associate commands with specific locations do no better at remembering the association after sleeping. They’re still good dogs, though. eNeuro
- Antiretroviral therapy does not reduce inflammation in organoids grown from HIV-infected progenitor cells, even though it succeeds at reducing the organoid’s viral load to undetectable levels. Journal of Neuroinflammation
