Where communication breaks down for people with autism

T

he quirks in Ramsey Brewer’s conversation are subtle. The 17-year-old repeats himself from time to time and makes small mistakes in the words he uses. For instance, he says he and his best friend look scaringly, not scarily, similar. He also pauses at odd spots, and for a beat or two longer than most people do. When he’s talking, he makes eye contact briefly but then slides his eyes sideways — or closes them. And his comments swerve in unexpected directions: Asked where he goes to school, he says Boston Latin Academy, but then suddenly adds, “I’m not actually from this state,” even though he and his family have lived in Massachusetts for years.

Ramsey knows he regularly misreads other people, but he leaves it to his mother, Kathryn Brewer, to explain how. Once, she says, when she had just climbed some stairs and was short of breath, he thought she had been about to cry. During a visit to the dentist, when Ramsey put on the safety sunglasses, the dental hygienist joked with him: “Hey, you can really pull those off.” Taking her comment literally, he pulled the glasses off his face.

These slight miscues mean that it only takes a few minutes of being in Ramsey’s company to guess that he has autism. Yet he falls within normal ranges on standardized tests of language and cognition and gets high grades at his prestigious mainstream school. It’s that combination of challenges and capabilities that brought him to Ruth Grossman’s Facial Affective and Communicative Expressions (FACE) lab at Emerson College in downtown Boston. There, Ramsey is participating in a series of studies investigating how social communication goes wrong for adolescents like him.

And go wrong it does. Many of the teenagers with autism whom Grossman works with experience what she calls “conversational breakdowns.” In 2014, Grossman showed 87 people videotapes of 9 children with a mild form of autism and 10 without, each speaking a few sentences. The observers didn’t know anything about the children, but it only took one second of tape — video or audio alone, or together — for them to realize there was something different about the children on the spectrum. The results, which have since been validated several times over, inspired Grossman’s subsequent work and her passion for this line of inquiry. “Clearly there is something going on, something very salient and very subtle at the same time,” she says, with an intense energy appropriate for someone who is studying expressiveness. “It happens all the time, and it leads to a reduced willingness of people to engage with people with autism.”

The big question is just what, exactly, those communication quirks are. Ever since Leo Kanner and Hans Asperger first identified autism, language problems have been part of the diagnostic criteria in some form. But historically, those criteria bifurcated people on the spectrum into those with or without severe language deficits, and didn’t consider less obvious problems. The current iteration of the “Diagnostic and Statistical Manual of Mental Disorders,” the DSM-5, dropped language problems as a diagnostic requirement, subsuming them under difficulties with social interaction. That change recognized the reality that anything social involves communication, but scientists are just beginning to explore the fine-grained aspects of how people with autism communicate.

Part of the delay in the field has been due to the condition’s heterogeneity. “We’ve always known that individuals with autism have a different manner of speaking, but it’s been very hard to characterize it because there’s so much variability,” says Joshua Diehl, chief strategy officer for autism services at the Logan Center, a nonprofit service provider in South Bend, Indiana.

To cut through that variability, researchers such as Grossman are turning to acoustic analyses of minute changes in the pitch and duration of utterances from people with autism, as well as motion-capture technology to measure detailed facial movements. These techniques enable them to assess the relevance of different components of communication, from someone’s facial expressions to their tone of voice or the actual words they use. The focus is on how people with autism express themselves, not just what they perceive when listening to or watching others. The shift brings a new level of scientific rigor to this branch of research, with the goal of raising awareness and finding targets for therapies. “I’m trying to quantify awkward,” says Grossman, who started her career as a speech pathologist. “This is about small aspects of language use that are falling apart even though everything else is looking good.”

The problem of prosody:

C

ommunication is verbal and nonverbal — it includes the words we use and the order in which we place them, but also gestures, facial expressions, eye contact and so on. Depending on how you define it, ‘prosody’ contains elements of both. And it’s where many discussions about communication in autism begin.

Put simply, prosody is the rhythm of speech, its rise and fall, energy and intonation. That rhythm does a tremendous amount of work. “[Prosody] is part of the way we perform with language,” says Helen Tager-Flusberg, director of the Center for Autism Research Excellence at Boston University. Stressing one word over another, for instance, provides pragmatic information about importance, as in “I wanted blue socks” versus “I wanted blue socks.” Our voices rise to indicate a question and fall to indicate a statement. The way we pause between words acts as a verbal comma to fill in grammatical information — the difference between asking someone to go to the store for chocolate ice cream and honey, or asking for chocolate, ice cream and honey. And prosody also helps to relay emotion. Someone who is happy or excited usually speaks with a higher and wider pitch range than usual.

Prosody has long been an obvious source of difficulty for individuals with autism. People on the spectrum may speak in a monotone way, or do the opposite and exaggerate their intonation. That variability has led to conflicting research results. Over the past decade, Diehl has conducted more than half a dozen studies of prosody. “The unifying characteristic was that so many people had some sort of difficulty with prosody,” he says. “I almost saw it as a bellwether of the difficulty an individual with autism was facing.” Those who spoke in a monotone were likely to have trouble understanding emotional affect, for instance. For others, problems with prosody seemed more related to motor difficulties or the use of language in social situations.

In 2015, Grossman and her collaborators tried to home in on these differences. They asked 43 adolescents with autism and 26 typical teens to watch children’s stories on video and then retell them. Naive raters, who knew nothing about the teens, judged their efforts, assessing overall awkwardness and scoring them on three specific elements: the rate and rhythm, or timing, of their retelling; the volume of their speech; and their intonation and emphasis. The findings revealed that having an awkward rate and rhythm correlates most with an autism diagnosis. But the greatest agreement among the raters came in the overall awkwardness scores, rather than any one element. A study last year by some of the same researchers put participants with and without autism in increasingly stressful situations as they spoke. As before, the individuals with autism had more variable prosody, and stress increased that variability.

Another study at about the same time zeroed in on the emotional aspect of prosody in people with autism. Working with a speech perception lab on his campus, Noah Sasson of the University of Texas at Dallas conducted acoustic measurements of recordings of adults with and without autism reading aloud. His team analyzed pitch and frequency, among other variables. They gave each participant a series of sentences whose meaning depended on how they were read — for instance, “What do you mean?” or “I can’t believe this.” The participants had to repeat each sentence to reflect an angry, happy, interested, sad or neutral emotional state. Later, a second set of participants, about half of whom have autism, listened to the recordings to judge the emotion being conveyed. The results again showed that the prosody of adults with autism is more variable in pitch.

“There was a lot more modulation in the tone of voice,” Sasson says. “[Adults with autism] tended to be louder when they were speaking, and they also took a longer period of time to say the statements than the controls.” When listening to the recordings, the adults on the spectrum had more trouble inferring emotion than the controls did, but, interestingly, their speech conveyed more dramatic highs and lows than that of controls — meaning it actually conveys more, not less, emotional content. That might seem counterintuitive, but it suggests that people on the spectrum consistently exaggerate the emotional content in their speech. Still, listeners in this study rated the readers with autism as sounding less natural than controls. “There’s something about the expression, not the content,” Sasson says. “Style, not substance, is driving [the difference].”

What’s in a face:

T

here also appear to be important style differences in the facial expressions and word choice of people with autism.

During a recorded visit to Grossman’s lab in January 2016, Ramsey looks at a screen in a quiet room. His face is covered in 32 reflective dots — with clusters along his eyes, mouth and cheeks, the areas that move the most when we communicate. His task is simple: He must watch recorded videos of actors speaking sentences in which the first clause is neutral and the second is emotive — for example, “My class is going on a field trip, but I’m stuck at home with the flu.” After each sentence, Ramsey looks up at a camera and repeats it, trying to mimic the actor’s emotion and facial expressions.

The dots are part of a facial motion-capture technology in action, the kind used to create human-like aliens in films such as “Avatar.” Infrared cameras record every facial twitch while Ramsey and others undertake various tasks. Then Grossman’s team does the painstaking work of analyzing the results, looking at movement in millimeters.

Those millimeters can speak volumes — a raised eyebrow here, pursed lips there. “Facial expressions are crucially important,” Grossman says. “You can’t have a spoken conversation using words and none of these other things.” Her research is looking beyond some of the known problems in autism, such as difficulty making eye contact, to try to catalog other differences in expression. Unlike most previous research, which has focused on how well individuals on the spectrum can read others’ expressions, Grossman wants to capture the nuanced and sometimes fleeting grins and grimaces that adolescents with autism make during conversation.

Grossman’s team set out to create a baseline reading on facial expressions. Children with and without autism sat alone and watched videos chosen to elicit disgust, joy, surprise or laughter. The researchers had predicted that without the stress of social interaction, all of the children would show similar reactions. Instead, they found the participants with autism to be more expressive and emotionally responsive than the others — results that run counter to claims that they are less emotive than their typical peers. (The results may be skewed because the typical children may have been less likely to behave as if they were truly alone; they knew the researchers were watching from behind a partition.)

Using data from a motion-capture image database, Grossman and her colleagues have found that people with autism display less symmetry in the expressions between the right and left halves of their faces. The new study, in which Ramsey is a participant, is testing another of Grossman’s theories — that timing and synchronicity in social communication are the primary culprits underlying apparent awkwardness. The results are still being analyzed, so it’s too early to say what the camera picked up, but to casual observers, Ramsey’s efforts meet with mixed success. As in the prosody research, though, his facial expressions tend toward the overly emotive.

Meanwhile, Grossman’s postdoctoral fellow, linguist Emily Zane, is investigating the stylistic differences in word use that might signal awkwardness to a listener. In a study reported last year, Grossman and her colleagues interviewed adolescents with and without autism about their lives, asking questions about their families, hobbies and recent vacations. Then they reversed the roles so that the adolescents had to ask the researchers questions. Grossman originally adapted this ‘dual interview’ setup as an investigation of pragmatics, or the use of language in social situations. But Zane saw an opportunity to get something additional out of it.

Zane analyzed the half of the conversations in which the teens with autism did the interviewing. As expected, they have trouble asking appropriate questions and produce less language on average than the typical teens do (although some produce more). They also make significantly more language errors, mostly involving using words inappropriately. For example, one teen with autism said, “Oh, I was way out,” when she meant to say ‘way off.’ And Ramsey asked his interviewee, “What’s the necklace?” to launch a conversation about her jewelry. In just a few minutes of dialogue, typical teens in the study made an average of 1 or 2 errors, whereas those with autism made 12 to 14, and some as many as 30.

A two-way problem:

P

eople on the spectrum have been shown to be particularly error-prone when using pronouns. Most notably, children with autism sometimes take longer than their peers to master the correct use of ‘I’ and ‘you.’ This problem has been recognized since the time of Kanner, but researchers have debated whether pronoun confusion represents a social or a linguistic mistake. In fact, it could be due to the fact that the meaning of pronouns such as ‘me’ and ‘you’ changes depending on who is speaking. That shifting meaning makes pronouns more linguistically complicated than unalterable words such as ‘cup,’ and it also requires a change in perspective, something that’s known to be difficult for people with autism.

Although the debate is ongoing, studies over the past five years suggest the pronoun problem is mostly social. For instance, in a 2015 study of children with autism who are also deaf and use sign language, researchers were able to limit the possibilities by having signers point to themselves or to a conversational partner to indicate ‘me’ and ‘you.’ Even with that linguistic ambiguity removed, 83 percent of signers without autism used ‘me’ correctly, versus only 35 percent of the signers with autism.

Most research on pronouns in autism has focused on first-person perspective, but Zane has also looked at the use of third-person pronouns — such as ‘he’ and ‘she.’ She has unpublished evidence of another problem with ambiguity. In a script featuring two female characters, she created situations in which the pronoun ‘she’ clearly refers to one of the two characters, and those in which it is unclear who ‘she’ is. In the latter scenario, typical children said they didn’t know who ‘she’ was. But children with autism usually picked one of the characters anyway, suggesting an unwillingness or inability to acknowledge the ambiguity. (Zane wants to dig further to determine whether they are guessing or really believe they know.)

Even ostensibly minor mistakes such as this can lead to those “conversational breakdowns” that concern Grossman, because any ambiguity can prompt a conversational partner to interrupt for clarification, or lose interest or drop the dialogue. Zane and Grossman have noticed a strong correlation between a child’s score on standardized language tests and how many errors they make in conversation — the higher the score, the fewer errors they make, even though none of the children’s scores are low enough to result in a language-problem diagnosis. “The test is picking up something, but it’s all above the cutoff,” Zane says. This threshold issue is a common problem with standardized testing, Grossman says: “It’s a blunt instrument designed to rule in or out significant issues. It is not designed to probe specific, subtle language competence.”

Grossman’s work may fill that gap the common assessments miss and perhaps add a layer to existing programs that teach children with autism about social interaction. Most of these efforts focus on understanding social rules, such as how to enter a conversation or what to say and not say. “None of these approaches spend a lot of time looking at the quality of facial expressions or the tone of voice or how all these pieces integrate,” Grossman says. “We’re still asking some fundamental questions, but I hope it will lead to a better understanding of what the actual factors are that lead to improved perception in the social environment.”

With that kind of knowledge, it might be possible to identify therapeutic targets in younger children. A few such attempts have already been made. For instance, one small 2015 study assessed the feasibility of a mobile app called SpeechPrompts for treating impairments in prosody, and showed some benefit. Other technologies, such as the LENA device for recording children’s word production in natural settings, may prove useful to researchers who want to closely analyze speech, Diehl says. A 2010 study found that LENA can recognize vocal characteristics typical of individuals with autism.

Increased awareness of social communication challenges also presents another possibility: educating the typical individuals on the other side of these conversations and encouraging them to meet people with autism halfway. “We want autistic individuals to have better, more fulfilling social experiences, employment prospects and quality of life, and there may be a limit to how far we can get simply by trying to train autistic people to have more neurotypical social skills or expressions,” Sasson says. He suggests working to make social environments more inclusive and accommodating. “We do have some evidence showing that people who have more autistic knowledge tend to be more favorable in their impressions of autistic people.”

Grossman says that with so many schools launching programs to educate students about all the differences among them, there is an opportunity to include autism awareness. “These kids are this close,” she says, holding up her thumb and forefinger less than an inch apart. “If we could only figure out the last bit that’s holding them back, that’s a huge population that could become much better integrated into society.”

Ramsey is maybe halfway there. He calls himself “a bit of a social recluse,” but has one close friend who is also on the spectrum. Mostly, they play hours of video games together according to a strict schedule they devised themselves. At the same time, Ramsey is quite capable of talking to strangers and says he enjoys the lab and the people who work there. He is vague on the purpose of the work beyond the idea that it addresses “how autism doesn’t affect everyone the same way.” But he says he likes the small fee he gets. “I’m glad I can be helpful.”