In 2015, Robert Schultz walked into his student’s office to watch an animation of a talking dog. On a split screen, the cartoon dog made facial expressions that mirrored the student’s own. Schultz had a flash of inspiration. He realized the software tools the student had used to take fine-grained measurements of his own expressions and superimpose them on the dog might also be used to diagnose autism: Whereas typical people unconsciously mimic the expressions of conversational partners, autistic people often do not.
Since then, Schultz, director of the Center for Autism Research at the Children’s Hospital of Philadelphia in Pennsylvania, has created a video-camera system that simultaneously records 360 facial signals — such as how the corner of the mouth moves — exchanged between two conversation partners. Algorithms analyze how much these signals are coordinated and calculate the chance that one of the partners has autism. In a preliminary study, the system identified autistic people with about 90 percent accuracy.
Schultz spoke to Spectrum about how wearable sensors, cameras and microphones can fill gaps in autism research.
Spectrum: Why did you get into the area of wearable technology for autism?
Robert Schultz: I’ve been in the field long enough that I’m frustrated. It stems from a dissatisfaction with current measurements of social functioning among individuals with autism. Why we have that dissatisfaction is a deeper question yet: It is the fact that autism is heterogenous. It doesn’t seem to conform to a distinct category. It has amorphous boundaries; and it has lots of comorbidities.
When the light bulb regarding wearables went on one day five years ago, I thought, “This is going to be a huge benefit to the field, and I don’t know why others don’t see it.” Now many people do.
I have done neuroimaging research my entire career, and still do, but in terms of what is most needed in the field now, I think it pales in comparison to the importance of digital phenotyping. We have a lot of granular data in science about brain function and genetics, but behavioral data lacks that kind of precision.
I think of wearables as a component under a rubric of digital phenotyping. The purpose is to digitally measure people. Wearables are one part of that solution because they can provide that granular data we need.
S: What challenges in the field can be addressed with wearables?
RS: There are a lot of deep theoretical questions and important research questions that promote our movement toward other ways of measuring behavior. For example, if you want really large samples, a traditional study running in my lab and many other labs might have a family, a parent with a child or adolescent, or an adult in the lab for as long as seven hours. How do you scale that up to 1,000 or 2,000 individuals? Realistically, you can’t, at least not with current funding models.
We’re always left with a collection of studies that are, at some level, not satisfying. There are issues with replication and differing answers to similar questions. There’s always doubt as to what those findings really mean. It’s hardly the best foundation for developing a science of a neurodevelopmental condition. I think the question of wearables is fundamentally tied to these deeper questions. It allows us to take many more measurements more easily on larger samples.
S: How can wearable technology be used to facilitate larger, more robust studies?
RS: If you had a wearable and it was collecting data periodically or continuously in the wild, you would get a lot of data, essentially for free. It will be highly granular for whatever it’s designed to measure. These signals are time series of data that readily lend themselves to analysis. By contrast, in a lab we collect less granular data, and it may have to be entered in a database, or there may be interpretation of data, first.
Questionnaires, which are so often used in research, are really the psychology of opinion. If you give me a questionnaire, it’s my opinion about that child and this and that attribute. Or if I’m a clinician doing a diagnostic evaluation, I generally have to take all my impressions and summarize them into a score. Wearables give you the raw stuff of behavior. They can help form the foundation for a proper behavioral science.
S: How are wearables and other technology useful in identifying autism?
RS: When you think about autism, it has two fundamental diagnostic criteria. The first is inherently dyadic, when you’re looking social issues. Social issues exist in a context — they exist with other people. Tools such as questionnaires don’t directly acquire data on social behavior in a social context. So we’re big believers in having assessments of two people interacting to help us understand the social deficits, and measuring these interactions with algorithms.
In our work to date, we find in a three-minute conversation, we can be about 90 percent accurate about whether a conversational partner has autism or not, just from looking at how well they coordinate behaviors with the person they’re talking to. It also predicts dimensions, like the social score on the Autism Diagnostic Observation Schedule. We asked a group of experts — clinicians who do autism diagnoses — watching the same three-minute video whether the person of focus had autism. They were only about 80 percent accurate.
We have a National Institutes of Health grant now trying to move the age of initial diagnosis down to 1 year of age. The core symptoms as described in the Diagnostic and Statistical Manual of Mental Disorders aren’t evident in any clear way at 1 year of age, so research still needs to discover the earliest emerging behaviors. These more granular features of the child’s coordination of behavior with someone else are evident, and we can measure them.
S: Is the technology useful in other areas of autism research?
RS: I think wearables are going to be really important day to day as care-management tools. In terms of turning them into therapies, if I’m doing something, a behavior, and I’m in the real world, and I can get feedback on what that behavior is or what a change in behavior is from someone else in real time, that could be used in a novel therapy.
In clinical trials of medications, there is also a lot of energy and excitement about using digital quantitative phenotyping as a more robust way to find a signal. Presuming that the medication is effective, how do you prove that it’s effective without a many-millions-of-dollars treatment study, where you need hundreds and hundreds of people to pick up these signals? How long is it going to be before I see a signal? Are my measurement tools robust to repeated use? That is a very expensive proposition with traditional approaches. It becomes much cheaper and much more granular with a digital measurement.
S: Where do you think the field of wearables is headed?
RS: I am not so sure that the wearables we have now, such as accelerometers or gyroscopes, are the ones that are going to paint a detailed picture of a person. There are yet-to-be-invented wearables that I think would be better. I really think the thing we want for measuring behavior is a microphone and a camera, which could be made wearable. Because with that, you can capture everything.
And I think it would be a disservice to the community if autistic people couldn’t use these technologies on their own. How to make sure they’re interpreting the information they get from wearables accurately is a huge responsibility. That is a challenge that will be hard and long to solve.
