The U.S. Food and Drug Administration (FDA) has given the okay to an app designed to help clinicians diagnose autism in children, labeling it safe for use.
But it’s still unclear whether the technology is effective, researchers say, raising concerns that it could actually disrupt the standard diagnosis process and, in some cases, lead to delays in starting therapy.
“I think what they’ve done looks beautiful and is an example of what money can buy,” says Catherine Lord, distinguished professor of psychiatry and education at the University of California, Los Angeles, and co-creator of the Autism Diagnostic Observation Schedule, considered the ‘gold standard’ for diagnosing autism. “But it may cause more harm than help.”
The app, which is targeted at primary care physicians and pediatricians, was created by Cognoa, a California-based company founded by autism researcher Dennis Wall. It is part of a machine-learning-based system called Canvas Dx that uses algorithms to evaluate videos and other information about a child, uploaded by their parents and doctor. The company aims to release the app and the rest of the Canvas Dx system in the United States by the end of 2021, says Sharief Taraman, Cognoa’s chief medical officer.
“We believe Canvas Dx will help primary care physicians diagnose autism equitably in young children, regardless of gender, race, ethnicity, socioeconomic status or geography, and hope that use of Canvas Dx may lead to shortening time-to-diagnosis, which can enable initiation of earlier interventions,” Taraman wrote in an email to Spectrum.
In 2018, Cognoa caused confusion among some autism researchers by announcing that the app had been designated an FDA Class II diagnostic medical device, long before the agency made the classification earlier this month. On 2 June, the FDA authorized Cognoa to market the app as a tool to assist in diagnosing autism in children aged 18 months through 5 years whose clinicians or caregivers observe possible autism traits.
The company gained authorization through the FDA’s ‘de novo premarket review pathway,’ established in 2018 for low- to moderate-risk novel medical devices.
“I think it is important for Cognoa marketing purposes not to conflate de novo classification with product approval [from the FDA],” says Matthew Goodwin, director of the computational behavioral science lab at Northeastern University in Boston, Massachusetts.
Trial results:
To use Canvas Dx, a parent or caregiver uploads two videos of their child and fills out a questionnaire in the app. A clinician then meets with the family and fills out a second questionnaire in the app. An algorithm analyzes the data and provides a conclusive ‘yes’ or ‘no’ about whether the child has autism or suggests the child be evaluated for other conditions. The process takes a few weeks, Taraman says.
As part of its safety review, the FDA reviewed data from Cognoa’s clinical trial, which compared the app’s evaluation of autism in 425 children, 153 of them girls, with that of a panel of three experts. None of the children had been formally evaluated for autism before, but their caregivers or pediatricians had expressed concern about their development.
According to the specialists’ evaluations, 122 of the children are autistic, 40 are neurotypical and 263 have a neurodevelopmental condition other than autism.
The app gave a conclusive result for 135 children, or 32 percent of the group. Of the 290 children who received an inconclusive result, 264, or 91 percent, have at least one other neurodevelopmental condition, according to the expert panel.
The app correctly identified autism in 63 of the 122 autistic children, and 1 autistic child received a false negative. For the remaining 58 autistic children, the app suggested they needed further evaluation.
Of the 71 non-autistic children who received a result, 15 were incorrectly identified as autistic. All 15 had a different developmental condition; 5 had been diagnosed as autistic by one of the three experts.
Cognoa plans to publish the results before launching the app, Taraman says.
The technology allows pediatricians to diagnose autism in a subset of children who would otherwise need to wait for a specialist, he says.
“Pediatricians are more than capable of diagnosing, I would say, a good portion of children with autism,” he says. “They just really need to have appropriate tools.”
But Goodwin has a laundry list of concerns: The app missed autism in many cases, which could delay care for autistic children if their parents didn’t seek further evaluations; much of Cognoa’s research on the app has not been published in autism journals, meaning the peer reviewers didn’t have expertise with autism diagnosis; and in his view, the company hasn’t been clear about the study participants’ trait severity, making it difficult to evaluate how effective the app is at distinguishing autism from other conditions, such as attention deficit hyperactivity disorder.
“I have not seen an evidence-based demonstration suggesting that Cognoa is better than gold standard, is less costly than gold standard, is more accurate than gold standard or is more scalable than gold standard,” Goodwin says.
Uncertain future:
The approval raises questions for the field about where technology such as Cognoa’s should fit into the diagnostic process, says Stephen Sheinkopf, associate professor of psychiatry and human behavior at Brown University in Providence, Rhode Island.
“Given the performance that Cognoa has published so far, this seems like a potential second-level screening tool. There are other examples of such tools in the literature and if used appropriately they could help to increase access to care,” he says. “But I would be cautious about the use of this tool as a formal diagnostic test at this time.”
Even though the Cognoa app could in theory help to shorten wait times for diagnosis, it cannot address the underlying problems in access to care, Sheinkopf adds.
“There’s real value for scientifically rigorous and advanced machine-learning approaches,” he says. “Those are going to be very important tools for us, but we also need to think about the basics, and the basics are we need people who are well trained and adequately reimbursed to care for families that we see in clinic.”
Cognoa has not yet set a price for the system but is working to ensure it is covered by health insurance, Taraman says.
Much more data on its effectiveness need to be collected before families should shoulder any costs, Goodwin says. “I don’t want to say this is a bad approach that shouldn’t be done and shut it all down. But it’s way premature to be commercializing for profit, charging families, in the absence of an evidence base.”
