Last year, we completed a project to determine whether psychoeducational therapies for young autistic children are effective. Initially, we were excited: We had been able to track down 150 studies completed over the past five decades comparing treatment and control groups. Locating such a large number of studies would usually allow for a high degree of confidence in our conclusions.
As part of our protocol, we rated and summarized the rigor of the studies and planned to do a statistical analysis of only those that we deemed to be high quality. We quickly became disappointed, however: Across intervention types, there were too few high-quality studies for us to compute reliable statistics.
Inspired by conversations and social-media interactions with fellow autism researcher Michelle Dawson, we conducted two follow-up analyses, scrutinizing the ethical conduct of researchers who authored the studies we had compiled. One focused on the researchers’ disclosure of conflicts of interest (COIs), and the other on their reporting of adverse events. The results only deepened our concerns about how scientists conduct and report their research on autism interventions for young children.
Altogether, our analyses revealed three ethical problems: Most studies are poorly designed, fail to disclose COIs and lack attention to adverse outcomes. Disclosing COIs and taking steps to mitigate the biases they may introduce will lead to better-quality research and instill greater trust from the autism community. Proactively measuring and reporting adverse outcomes will enable families to adequately weigh the potential benefits of interventions against any potential negative consequences.
Conflict resolution:
We found that 70 percent of studies were co-authored by a researcher with a COI, which we defined as any situation in which a person holds a vested interest that might bias a study’s outcome. For example, a researcher who provides an intervention for profit stands to benefit from evidence suggesting the method is effective. That researcher’s interest in fairly evaluating the intervention conflicts with his potential for financial gain.
Having COIs does not necessarily constitute ethical misconduct, but failing to disclose them might — especially if a journal requires it. We found that only 6 percent of the studies with COIs had actually reported them.
One conflict we found especially worrisome was when the investigator was the developer of the intervention. Developers stand to gain from career advancement, training others to use their intervention via for-profit workshops and selling manuals describing the intervention. However, none of the studies we evaluated had disclosed this particular COI, even though most interventions were studied by the same researchers who had developed them.
We surmise that the low quality of autism-intervention research to date is related to this paucity of COI reporting. Biases associated with having COIs may lead autism-intervention researchers to make questionable methodological decisions that compromise research integrity and quality. For instance, a researcher with a COI may choose a flawed study design in which participants are not randomly assigned to treatment and control groups, or may use a measure that is vulnerable to placebo effects. These types of flaws increase the likelihood of getting favorable results, even when the interventions are not actually effective. COIs may also bias researchers’ interpretations of their results, leading them to claim their studies provide strong evidence of effectiveness when in fact the results are mixed.
Additionally, when researchers collectively and routinely fail to disclose COIs, there is no pressure from either the research community or the autism community to reckon with the effects these COIs may have. Essentially, COIs have remained out of sight and out of mind for many researchers — meaning that the problems they introduce are not being addressed.
Unfortunately, it is not always clear what constitutes a COI; there is no exhaustive list of the kinds of relationships that should be considered as such. We therefore urge researchers to think carefully about any potential COIs that they or their team members may have and to disclose them. Such disclosures may compel researchers to design unbiased studies and to be clear about how they prevented their COIs from influencing their research results (such as by excluding themselves from data collection and statistical analysis).
Facing adversity:
Our other follow-up looked at the extent to which autism-intervention researchers measured and reported adverse events, adverse effects and harms. Adverse events include any occurrence of psychological or physical distress; other adverse outcomes that might be reasonably attributed to an intervention are deemed adverse effects. Harms refer to adverse events that remain long after an intervention has ended. For example, if a child develops a fever while receiving an intervention, researchers should document this as an adverse event. If a child breaks her arm on a play structure during an intervention session, this should be reported as an adverse effect. If a child becomes fearful of a particular toy used during an intervention and remains afraid of it several years later, this should be recorded as a harm.
We found that, similar to COIs, adverse events are rarely reported. Only 7 percent of the 150 studies we examined mentioned adverse events. None mentioned harms, nor described systematic procedures for measuring adverse events.
Because autism research has a history of using punishments that induce pain as an intervention strategy — some of the first behavioral therapists gave electric shocks or slaps to children to discourage certain behaviors — we believe it is especially important for autism-intervention researchers to proactively monitor adverse outcomes. Although contemporary interventions tend not to use punishment, unintended negative consequences are still possible.
To get some insight into these unintended consequences, we compiled a list of the adverse effects reported in the literature, along with any reported reasons for a participant’s withdrawal that met the definition of an adverse effect but were not labeled as such. We also scrutinized results indicating that children had less optimal outcomes when they received the treatment of interest than when they were in a control condition.
The adverse effects we identified included major and minor physical trauma, child distress and dislike of the intervention, caregiver anxiety and stress, and poorer outcomes in several areas, such as social communication, restricted and repetitive behavior, language, play and social-emotional behavior. The majority of the participants in the studies we reviewed likely did not experience adverse effects, but our findings still suggest researchers should monitor for and record them.
We were not able to find any data on the potential harms of autism interventions, but many autistic adults have expressed that they experienced harm — even trauma — as a result of the interventions they received as children. Researchers should conduct follow-up evaluations of study participants long after interventions are over, to document possible harms that arise or persist over time.
We hope that bringing these issues to light will lead to greater awareness of these pitfalls, improve researcher practices and ultimately lead to better services to support autistic children and their families.
Kristen Bottema-Beutel is associate professor of teaching, curriculum and society at Boston College in Massachusetts. Shannon Crowley is a doctoral candidate in the curriculum and instruction program at Boston College. Micheal Sandbank is assistant professor of special education at the University of Texas at Austin. Tiffany Woynaroski is assistant professor of hearing and speech sciences at Vanderbilt University Medical Center in Nashville, Tennessee.
