New study exposes flaws in estimates of autism prevalence

A widely cited 2011 study that pegged autism prevalence in South Korea at a whopping 2.6 percent is inherently flawed because of the assumptions the researchers had to make, suggests a new analysis of the data¹. The findings have implications for estimates of autism prevalence anywhere in the world.

Intense controversy surrounds estimates of autism prevalence, which have been rising steadily over the past 30 years. How researchers collect their data can have a huge impact on the numbers, and may even account for the rapidly rising rates.

The 2011 study garnered attention for its two-phase design, in which researchers screened more than 20,000 children for autism and then clinically assessed a small proportion². They extrapolated these findings to South Korea’s population to arrive at their estimate.

Three-quarters of the children the study tagged as having autism did not have a prior diagnosis and were attending mainstream schools. Flagging children based on signs noted in medical and school records — the standard method in the U.S. — may have missed these children.

But the two-stage design is also flawed. The screening method the researchers used is not perfectly accurate, and the results can be skewed by which parents choose to participate. Taking these limitations into account, the analysis, published 29 June in Autism, says the 2011 study’s range for autism prevalence could have been at least 2 to 5.4 percent.

This numerical spread is too wide to allow researchers to pinpoint any single rate with certainty, says lead researcher Dan Kennedy, assistant professor of psychological and brain sciences at Indiana University in Bloomington.

“The estimate can be misleading if incorrect assumptions are made,” Kennedy says. “If we are overconfident in the assumptions we make, then we end up with an estimate we believe, but probably shouldn’t.”

Double testing:

The findings may have implications beyond South Korea.

Any study of prevalence in a large sample has to rely on some assumptions. But researchers rarely address these caveats because they are under pressure to “go for the punch,” says Catherine Lord, professor of psychology at Weill Cornell Medical College in New York, who was not involved with the work. “In the end, we can’t make conclusions without a ton of assumptions. We really have to be careful about that.”

In the 2011 study, the researchers approached the parents at every school in Ilsan, a district in South Korea, to fill out an autism screening questionnaire. Families of all 294 children in Ilsan’s special education schools participated, but only 63 percent of parents of the 36,592 children at mainstream schools filled out the questionnaire.

“We can’t make conclusions [about autism prevalence] without a ton of assumptions. We really have to be careful about that.”

Based on this initial screen, 1,742 children met the criteria for a follow-up assessment at a clinic. Only 71 percent of the parents of these children agreed to go to a clinic, however, and only 30 percent showed up for their appointment.

This follow-up rate is worryingly low, says Kathleen Merikangas, chief of genetic epidemiology research at the National Institute of Mental Health. “As an epidemiologist, if I do a follow up and I don’t get 60 to 70 percent, people really would feel that my data aren’t valid.”

The low rate is one of the concerns noted in the new analysis, as it can significantly affect the prevalence estimate. The researchers assumed that the children who showed up for the assessment are representative of the larger group of 1,742 flagged in the screen, but this may not be the case. Families of children with autism may be particularly eager to cooperate, inflating the estimate.

Alternatively, a stigma against autism in South Korea may have led families of children with autism to avoid the assessment, artificially lowering the reported prevalence.

Young-Shin Kim, who led the South Korea study, agrees with these concerns. She notes, however, that there were some simpler explanations for the drop-off in participation. For example, many families had trouble traveling to the clinic, she says, and researchers may have better luck visiting families in their homes.

Kim adds that in some obvious ways, the children who participated did seem representative of the larger group. For example, both groups have the same male-female ratio and include children of similar ages.

The screening tool the researchers used comes with its own caveats. Kim and her colleagues had to assume that the tool identifies every child with autism — which is incorrect. To refine the estimate, the researchers should have verified that the children who screened negative for autism do not have the disorder, Kennedy says.

Kim says she did not have the funding for this step.

The two-step method the researchers used was a good choice for South Korea, where autism carries significant stigma. Unlike in the U.S., parents there may not report their concerns to doctors or teachers, says Maureen Durkin, professor of population health sciences and pediatrics at the University of Wisconsin-Madison.

Some experts have criticized the way the U.S carries out its prevalence studies and recommended that the Centers for Disease Control and Prevention also use the two-step method. But the new analysis shows that that method is equally problematic, says Durkin, who estimates prevalence for the state of Wisconsin.

“I’d rather put our efforts into improving the [existing] approach for the U.S. instead of jumping to do something different, which is actually no better, probably a lot worse and would cost a lot more,” says Durkin. “That’s why I really like this [new] study. It thinks things through a little more deeply.”

References:

1: Pantelis P.C and D.P. Kennedy Autism Epub ahead of print (2015) PubMed

2: Kim Y.S. et al. Am. J. Psychiatry 168, 904-912 (2011) PubMed