Methodological issues have plagued studies on the effectiveness of early autism interventions for nearly three decades, according to new unpublished research. These problems include multiple types of bias and an overreliance on caregivers to report outcomes.
Researchers presented the findings virtually yesterday at the 2021 International Society for Autism Research annual meeting. (Links to abstracts may work only for registered conference attendees.)
“Poor-quality studies tend to inflate our estimates of intervention effectiveness,” says lead researcher Micheal Sandbank, assistant professor of special education at the University of Texas at Austin.
The new work aimed to evaluate the reliability of studies that examine the effectiveness of early interventions. Conventional wisdom on therapy says “the earlier the better, and the more the better,” Sandbank says.
But the evidence supporting that idea is weak, she and her colleagues found. Several sources of bias threaten to undermine the reliability of many early-intervention studies: selection bias, when experimental groups and control groups are not randomly assigned; detection bias, when the same person administers an intervention and judges its effectiveness; and attrition bias, when participants from a control or experimental group disproportionately drop out over time.
The number of studies at high risk for selection or attrition bias decreased from 1989 to 2017, Sandbank and her colleagues found, drawing on data collected for their 2020 study on early interventions. But both types of bias were still common among the studies they reviewed, and 50 to 75 percent were at high risk for detection bias.
Many studies also leaned heavily on reports from parents or caregivers to assess the effectiveness of an intervention, instead of relying on clinical observations, which studies show are more reliable. Between 25 and 50 percent of the studies from 2011 to 2017 relied on parent reports, down from around 90 percent in 2000.
The team’s 2020 meta-analysis concluded that many early interventions had positive effects. But once they excluded studies that relied on caregiver reports or had a high risk of one or more types of bias, there were often too few studies left to assess an intervention’s effects.
“I really hope sharing these results will inspire people to look towards ways of improving the quality of our collective research,” says Tiffany Woynaroski, assistant professor of hearing and speech sciences at Vanderbilt University in Nashville, Tennessee, who also worked on the analysis.
The researchers plan to update their previous work by including studies published since 2017.
Read more reports from the 2021 International Society for Autism Research annual meeting.
