Autism tests struggle to balance accuracy and speed

Diagnosing autism is a tricky endeavor and, at least in the foreseeable future, based almost entirely on behavioral observations. Two studies published in the past month highlight the challenges in balancing the accuracy of autism diagnosis with cost-effectiveness and speed.

One study, published today in the Archives of General Psychiatry, shows that whether someone is diagnosed with classic autism, Asperger syndrome or pervasive developmental disorder–not otherwise specified (PDD-NOS) varies significantly among different clinical sites, even when they are all relying on the same gold standard tests.

There is also inconsistency among the sites on how to assign a diagnosis of classic autism, according to the study. “There’s probably a lot of disagreement on those borderlands even between having autism and not having autism,” says lead investigator Catherine Lord, director of the Institute for Brain Development at New York-Presbyterian Hospital in New York City.

The results suggest that a truly accurate diagnosis of autism needs more analysis, not less, Lord says.

Another study, published 3 October in Autism Research, reports that two commonly used rapid diagnostic tests, the Social Responsiveness Scale, or SRS, and the Social Communication Questionnaire (SCQ) can together identify whether a child has autism with 80 percent accuracy². But the tests, which rely on parent reports, misdiagnose autism in about 88 percent of individuals who have other developmental disabilities.

Together, these studies suggest that clinicians and researchers are both misdiagnosing individuals with the disorder.

“Lots of people recognize autism when they see it, but actually measuring it, and measuring it reliably, accurately, quickly and cheaply, is proving very difficult,” says Tony Charman, chair of autism education at the University of London’s Institute of Education.

Tried and trusted:

The so-called gold standard tests for diagnosis, the Autism Diagnostic Observation Schedule (ADOS) and the Autism Diagnostic Interview-Revised (ADI-R), are time-consuming and together can cost up to $5,000. Both tests rely on trained clinicians, and the ADOS includes direct interaction between the child and the examiner.

“People say that we have to get the time down for this assessment and that it’s just too long and too hard,” says Lord, who developed the tests. “[They] cannot resist the temptation to say there must be a simple answer.”

There are several screening questionnaires that are inexpensive and can be quickly filled out by parents in a doctor’s office or as part of a research study. But Lord’s new study suggests that without careful analysis, labels are not useful.

The study examined how autism spectrum disorders are diagnosed at the 12 participating sites of the Simons Simplex Collection, or SSC. The collection, funded by SFARI.org’s parent organization, is a database of genetic and clinical information from families that have one child with autism and unaffected parents and siblings.

The researchers originally designed the study to see whether they could benefit from clinicians’ experience in diagnosis, says Lord. Instead, they found that the clinicians at each site seemed to interpret the diagnostic guidelines differently.

For example, each site assigned similar ADOS and ADI-R scores overall, but two of the sites diagnosed fewer than 40 percent of the children as having classic autism, whereas another site assigned the label to every single child.

Although the sites “swore that they had exactly the same procedure,” they also interpreted the spectrum disorders differently, says Lord. One site assigned a diagnosis of Asperger syndrome or PDD-NOS to all individuals with intelligent quotient (IQ) scores above 115, for example, whereas another used an IQ of 70 as the cutoff. Others appeared to be influenced by the age of the children, which is not supposed to be a factor at all. “It turns out each site knew what Asperger’s was, but the way they thought about it was different from everyone else,” says Lord.

Clinical sites often specialize in treating subgroups of children, such as those who also have epilepsy or gastrointestinal issues, notes Sylvie Goldman, a developmental psychologist in neurology and pediatrics at the Albert Einstein College of Medicine in New York. “Sites are not completely neutral,” she says.

It’s possible that these discrepancies in diagnosing Asperger syndrome and PDD-NOS will be resolved when those categories are subsumed into a single diagnosis of autism spectrum disorder. That change is set to take place in 2013, with the publication of the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders.

But this single-category diagnosis will probably require detailed descriptions of how the individual fits into the spectrum, notes Lord. “The problem is, if you have a dimensional diagnosis, you have to measure each one of those dimensions.”

Greater power:

The majority of autism studies don’t use the gold standard tests, but a range of more rapid and inexpensive measures, including the SRS and the SCQ.

Based on the numbers needed to set up autism genetic repositories, rapid screening measures could save the field an enormous amount of money, says John Constantino, professor of psychiatry and pediatrics at Washington University in St. Louis, who developed the SRS.

But studies have suggested that rapid tests could be misdiagnosing individuals with other developmental disorders, according to a meta-analysis published in January³.

The October study in Autism Research examined 333 simplex families who were referred to the SSC. Extensive clinical diagnoses, which included the ADOS and the ADI-R, determined that 57 of the 333 children do not have autism. The SSC researchers found that all but 4 of the 57 are likely to have other developmental disorders, such as attention deficit hyperactivity disorder and intellectual disability.

Based on the cutoff scores that yield the highest accuracy, or ability to identify autism, the SRS diagnosed children with autism with an accuracy of 94 percent, and the SCQ with an accuracy of 80 percent when compared with the clinical diagnoses. By contrast, the ADI-R accurately diagnosed 98 percent of the children, and the ADOS was 100 percent accurate, the study found.

All of the tests were less specific, however, meaning that they did less well at distinguishing autism from other developmental disorders. The SRS diagnosed 33 of 44 children with other developmental disabilities as having autism and the SCQ misdiagnosed 45 of 50 children. The ADI-R also wrongly diagnosed 33 of 48 children with other developmental disabilities as having autism. The ADOS fared best, misdiagnosing 16 of 57 children.

Because each measure is compared to the clinical standard, which includes a combination of the ADOS and the ADI-R, the researchers also looked at the SRS and SCQ in combination. Together, these two measures are 80 percent accurate compared with the standard, but wrongly assigned an autism diagnosis to 88 percent of individuals with other developmental disabilities. By contrast, the ADOS alone is 95 percent accurate.

In the general population, the rapid screens are likely to be even less powerful, says lead investigator Zachary Warren, assistant professor of pediatrics at Vanderbilt University in Nashville, Tennessee. “If you were doing this in clinical populations at large, where [autism] was not disproportionately represented, the total accuracy could be very challenging.”

But the specificity of the SRS isn’t that far off from the ADI-R, points out Constantino.

The study also suggests that it’s harder to rule out a diagnosis of autism than it really is, because the individuals included are all at the border of an autism diagnosis, Constantino says. “It offers minimal insight about the potential validity of rapid phenotyping measures in large unselected populations.”

The study combines the SRS and the SCQ, both of which are based on parent reports. Constantino argues that a better comparison would include SRS reports from both parents and teachers, which studies have shown to increase the screen’s reliability⁴.

Despite these caveats, other researchers say the results confirm what they already knew: Rapid screening measures can’t yet replace the more detailed diagnostic assessments.

“It’s a challenge for autism to move very quickly to large-scale accurate phenotyping at low cost,” says Charman. “It’s still a roadblock in the field.”

References:

1: Warren Z. et al. Autism Res. Epub ahead of print (2011) PubMed

2: Lord C. et al. Arch. Gen. Psychiatry Epub ahead of print (2011)

3: Fernandopulle N. Indian J. Psychol. Med. 33, 5-10 (2011) PubMed

4: Constantino J.N. et al. J. Am. Acad. Child Adolesc. Psychiatry 46, 1668-1676 (2007) PubMed