Time for some soul-searching in science

Lately I’ve been reading too many articles about scientific scandals. Whether it’s the high-profile retraction of a Science paper that supported equality for gays or the jailing of a former Iowa State University researcher who faked data on an HIV vaccine, discredited studies affect the public’s faith in research.

Autism researchers have seen their share of scientific misconduct. Andrew Wakefield’s unethical work linking autism to the MMR vaccine is our most notorious example. The paper, retracted from The Lancet five years ago, still haunts the field.

But I also worry about other, more innocent, scientific missteps that undermine our credibility. For instance, researchers often struggle to replicate the findings of another lab, or even their own lab. Replication is a cornerstone of good science, but there is little incentive for researchers to spend time and money redoing experiments.

The field of autism is somewhat spared, at least in the clinical realm, in that many researchers study the same kinds of phenomena. So replication naturally occurs in the course of doing science. But findings often conflict with each other. Consider research on theory of mind — a person’s ability to attribute thoughts to others. For every study that links autism to deficits in theory of mind, another study fails to find any difference between people with autism and controls.

Why is the field fraught with contradictory findings? The fundamental problem is that most of the studies are seriously flawed.

It is not unusual to see studies carried out in fewer than 20 people with autism with little regard to age, cognitive level, language ability or other co-occurring conditions. Matching such a small sample to a comparison group without autism is utterly inadequate to control for these variables.

Clear caveats:

Our field also has a history of methodological problems. Take neuroimaging studies, which attempt to identify distinctive structural or functional abnormalities in the brains of people with autism. We now know that even the smallest amount of movement in the scanner can spark significant distortions in the data. This problem could help to explain conflicting findings from studies that use diffusion tensor imaging, which measures the movement of water molecules in the brain, to assess brain connectivity in autism.

Now that we have better methods for limiting the impact of movement in the scanner, we should repeat earlier studies to see if the original findings hold up.

As scientists, we accept that we work within a self-correcting system. We know that our studies aren’t perfect or definitive, but we believe they’re worth publishing so that other researchers can consider our findings and perhaps even build on them. We must couch our conclusions and acknowledge the limitations of our work. Journal editors also have a duty to keep us in check.

Some journals are even adopting best practices to avoid publishing research that can’t be replicated. In June, Science released eight guidelines for journals to make research more transparent. It’s time for all of the leading autism journals to get behind these guidelines.

But it’s also important to remember that our fellow scientists aren’t the only ones reading our work. The media is hungry for news about science, and those stories aren’t subject to peer review. We must make sure that our sound bites are balanced and the caveats of our studies are clear. Otherwise, if other researchers can’t replicate our findings, the public is likely to grow skeptical of what science has to offer.

The consequences of this eroding trust in scientific research are all too clear. For example, the proportion of families of children with autism who turn to unproved treatments involving, say diet, supplements or dangerous invasive interventions is estimated at 40 to 95 percent. Unfortunately, I doubt that research debunking these approaches would make much of a difference in this mind-set. Since research into autism treatments is replete with conflicting findings, the term ‘scientific evidence’ has lost its meaning for many of these families.

Where do we go from here? There are no simple solutions. A combination of approaches is necessary to regain public confidence and to educate laypeople about the state of our knowledge. These should start with the way we do science and include policing by journals and proper handling of the media.

One way to accelerate progress might be to bring scientists together in official forums to discuss solutions. As an example of this, four major scientific organizations in the U.K. — the Academy of Medical Sciences, the Wellcome Trust, the Medical Research Council and the Biotechnology and Biological Sciences Research Council — held a symposium earlier this year on the current state of biomedical research.

The report has not yet been released, but we can only hope that this public airing of science’s dirty linen will bring about bold recommendations that put science back on track and help to win back public trust. For those of us in the autism community, this has never been more important.

Helen Tager-Flusberg is professor of psychological and brain sciences at Boston University, where she directs the Center for Autism Research Excellence.