A 15-item test could strengthen research on early behavioral interventions for autism by enabling head-to-head comparisons of different approaches. In a proof-of-principle study, the measure, known as the Brief Observation of Social Communication Change (BOSCC), picked up social communication improvements among participants in two of the three interventions, using data from previously published trials.
In the past, it has been difficult to compare the effectiveness of different behavioral interventions because studies use a variety of tests to track social communication skills. Tests that assess core traits of autism are often study-specific.
Measures commonly used across studies, such as the Mullen Scales of Early Learning, capture general cognitive functioning. Meanwhile, the Autism Diagnostic Observation Schedule (ADOS), the gold standard diagnostic test for autism, often fails to pick up small but meaningful changes in social communication skills.
By contrast, the BOSCC “is intended to be more sensitive to the subtle changes that we might see in core symptoms of autism” with interventions, says study lead investigator So Hyun “Sophy” Kim, associate professor of clinical and counseling psychology at Korea University.
“The BOSCC was designed to evaluate changes in observable autism features, and this study shows that, generally, its performance matches what the designers had hoped for,” says Melanie Penner, developmental pediatrician and senior clinician scientist at Holland Bloorview Kids Rehabilitation Hospital, who was not involved in the study.
The instrument is a twist on the gold standard, with behaviors coded on a broader scale of 0 to 5—rather than 0 to 2 as on the ADOS—in order to be more sensitive to subtle changes in skills. It is also much shorter, with 15 items that highlight behaviors that are most likely to change with intervention, Kim says.
T
he test involves analysis of 10- to 12-minute videos of child-caregiver interactions for behaviors such as eye contact, facial expressions and gestures. Observers evaluating the videos don’t know whether a given child is in a treatment or control group, nor how long the child has been receiving an intervention.This makes the test more objective than measures that rely on parent reports of children’s behavior, since parents may be invested in the intervention being successful, the researchers say.
“I think it is really beneficial to the field to see researchers focusing on designing measures that reduce bias, and tell us about developmentally important aspects of intervention,” says Kristen Bottema-Beutel, professor of special education at Boston College, who was not involved in the work.
In the new study, researchers gathered 582 videos of 207 caregiver-child dyads from three previous randomized controlled trials testing the effectiveness of Early Social Intervention (ESI), Early Start Denver Model (ESDM), and Joint Attention Symbolic Play Engagement and Regulation (JASPER).
“The authors include the developers of the three studied [interventions], who all contributed their study data to allow this analysis to happen,” Penner says. “This also sends a strong message that scientists need to collaborate to harmonize outcome measurement to allow the whole field to move forward.”
Several past studies have tapped the BOSCC to evaluate individual interventions, but this is the first to assess multiple interventions at once.
Children who received ESI or ESDM showed improvements in BOSCC scores compared with controls from each study, the researchers report in a paper published 1 July in the Journal of the American Academy of Child and Adolescent Psychiatry. However, there were no differences in BOSCC scores between children who received JASPER and controls, even though the previous trial showed it to be effective.
“Our intention was not really to look at the effectiveness of treatment,” Kim says. “We were more focused on really trying to implement a uniform measurement approach.”
T
he BOSCC may not have been able to pick up social communication improvements in the JASPER group because, among other reasons, that intervention is shorter than the other two, the researchers write.“However, this seems a bit contradictory to the authors’ assertion that a strength of the BOSCC is that it is especially sensitive to change,” Bottema-Beutel says.
Even with the BOSCC, there’s a limit to the ability to compare data from studies that differ radically in design—for instance, with distinct participant populations or dosage and duration of treatments, Kim says. But the new test could be used in future studies designed to compare multiple interventions.
The findings of such studies could help clinicians design individualized intervention strategies by building a picture of which interventions work, how and for whom.
The BOSCC does not eliminate all sources of bias, Bottema-Beutel points out, because caregivers are involved in the videotaped interaction sessions. By participating in an intervention, caregivers may learn how to better elicit certain behaviors from their child—which may not generalize to the child’s social interactions with others.
In recent intervention studies, Kim and her colleagues have addressed this issue by evaluating interactions between autistic children and “masked” examiners who don’t know whether the child is participating in an intervention, she reports. But evaluating children’s interactions with caregivers is also important.
Kim and other researchers have developed an additional test, the Measure of NDBI Strategy Implementation: Caregiver Change (MONSI-CC), to assess how participating in these therapies, known as naturalistic developmental behavioral interventions, changes the behavior of caregivers. Their unpublished data shows that changes in caregiver behavior are associated with changes in children’s behavior, she says.
