How virtual reality is transforming autism studies

T

he forest is still — until, out of the corner of my eye, I notice a butterfly flutter into view. At first it is barely perceptible, but as I watch the butterfly more intently, the trees around it darken and the insect grows brighter. The more I marvel at it, the more marvelous it becomes, making it impossible for me to look away. Before long the entire forest recedes, and the butterfly explodes into a red starburst, like a fireworks display. Everything goes dark. Then, dozens of white dots swarm around me. On my left, they are just dots. On my right, they leave long trails of spaghetti-like light. The contrast makes me acutely conscious that the present is never experienced as a mathematical instant; it has some duration, and the perception of that can vary with context.

The sensation evaporates as soon as I take off my headset.

This immersive virtual-reality (VR) experience was a preliminary look at “Beholder,” an art installation at the Victoria and Albert Museum in London in September that sought to recreate how autistic people perceive the world. It is now on display at the gallery that commissioned it, Birmingham Open Media. The project’s creator, Matt Clark, has a severely autistic 15-year-old son, Oliver. “He can’t talk; his behaviors are extremely challenging,” says Clark, creative director of United Visual Artists, an art and design group based in London. Clark built “Beholder” so he and others could see the world through his son’s eyes. He collaborated with artists who either are on the spectrum or have family members who are.

The project exemplifies a new approach to the use of VR for autism. For more than two decades, scientists have experimented with the technology to set up controlled scenarios to study autistic traits. At the same time, some teams have used VR to create role-playing environments for practicing social skills. Increasingly, however, people with autism are using VR to convey their own experiences, both to raise awareness of the condition and to capture the cognitive and perceptual differences that characterize it. Some experts hope these efforts will lead to new research collaborations and applications.

These immersive experiences are, in many ways, the digital equivalent of Temple Grandin’s narratives, which were among the earliest first-person descriptions of autism. A dozen or so projects that can be viewed online use loud noises or flashing lights to try to reproduce sensations such as sensory overload at a shopping mall, office meeting or family get-together. Slightly more elaborate efforts, such as the trailer for Carly Fleischmann’s book, “Carly’s Voice: Breaking Through Autism,” and the animation “Listen,” layer the effects over a storyline. But a few are especially ambitious in aiming to provide specific sensory impressions. Examples of the latter are “Beholder” and an augmented-reality system created by researcher Yukie Nagai and her colleagues at the National Institute of Information and Communications Technology in Osaka, Japan.

Proponents of VR argue that no other medium comes as close to putting you in someone else’s shoes. “Having a perceptual experience — that’s something we haven’t been able to do without VR,” says Albert “Skip” Rizzo, research professor at the University of Southern California in Los Angeles and a pioneer of using VR in psychiatry. “You can watch a movie, but it’s different than walking around and having your perceptual experience,” Rizzo says.

These projects are not uncontroversial, however. So-called ‘disability simulation exercises’ — blindfolding people to demonstrate vision impairment or making them use crutches to appreciate mobility challenges — are mainstays of diversity training. But they fail to capture the social isolation that is often part of a disability, and they can evoke pity and condescension, driving people apart rather than together.

Simulations of autistic experience have been met with a similar ambivalence. They also must confront the basic metaphysical question of whether subjective experience is something that can ever be shared. “I can understand that a neurotypical parent might be desperate to understand their autistic child’s point of view,” says Susan Kruse, gallery supervisor at Birmingham Open Media, who is autistic. “But how can anyone get inside another person’s mind and experience what they experience?”

Virtual experiments:

A

utism therapists and researchers started to use VR in the mid-1990s, not long after headsets became widely available to consumers and other forms of immersion, such as first-person shooter games, became popular. Researchers often deployed the technology to create virtual environments to help autistic people rehearse stressful encounters. For instance, Rizzo’s team built a virtual job-interview training program. In a study published last year, they recruited adults with autism or other conditions for a training regimen involving interviewers who ranged from gentle to aggressive. Rizzo says the participants with autism significantly improved in their interviewing skills, as rated by job counselors.

A similar application lets autistic children practice public speaking in a virtual classroom with an audience of eight avatars. To encourage them to look around the room rather than stare straight ahead, the avatars start to fade away if the speaker fails to make eye contact with them. “So it became a game of keeping the avatars on the screen by shifting attention,” says Peter Mundy, a psychologist at the University of California, Davis, who developed the program. “We found that the kids with autism really responded to that.”

VR can also make autistic children more comfortable in strange environments. In an unpublished July 2018 case study, a team led by Nigel Newbutt at the University of the West of England in Bristol gave 11 autistic children, aged 10 to 14 years, a VR tour of a local science museum a few days before their actual visit. “Students reported feeling less anxious, less stressed, more prepared for that space,” he says. “In fact, the teachers also found that when the pupils got there, they knew where they wanted to go; they had a greater sense of purpose and direction.”

Back in the lab, virtual environments have also offered researchers a welcome new experimental technique. Nathan Caruana, a cognitive neuroscientist who uses VR to study social cognition in autism, prefers it to standard screen-based scenarios. “All of those paradigms have largely relied on non-interactive tasks, where people are responding to a face with averted gaze on a screen,” says Caruana, associate investigator at Macquarie University in Sydney, Australia. “But it doesn’t really reflect the dynamics and complexity of a social interaction.”

VR also facilitates imaging experiments that would otherwise be impossible — such as enabling someone lying in a scanner to banter with virtual humans. “In order to measure this in an imaging platform, you basically have to use virtual reality,” Mundy says.

For all its apparent advantages, however, VR has yet to be rigorously tested as a therapeutic or research tool for autism. Several meta-analyses this year turned up comparatively few studies, and most had only a handful of participants and no control group. Newbutt and a colleague, for instance, found a total of six studies since 1990 that have tested head-mounted VR displays in students with autism. “There isn’t that much evidence to support the use of this yet,” Newbutt says.

One reason is cost, not just of the equipment but of the programmers and animators needed to create the content. Mundy laments that he hasn’t been able to implement some of his ideas for lack of people with the relevant expertise. “One of the reasons I couldn’t go further with it was that I couldn’t pay the coder as much as a high-tech company [could],” he says. Consequently, VR scenarios remain highly simplified, and the technology’s much-touted advantage — realism — remains out of reach.

Newbutt also says that researchers have seldom asked autistic people what they want from the technology. “There’s still a bit of a tendency to research about autism and autistic groups as opposed to research with them,” he says. This is precisely what the new first-person experiences seek to rectify.

Righting the balance:

M

any people with autism are drawn to VR out of a feeling of invisibility.

“Until a predominantly neurotypical society/culture puts in the equivalent amount of effort and time to understand us and listen to us as we put into understanding and listening to it, we will continue to be disabled,” says Sonja Zelić, an autistic artist based in London who contributed to “Beholder.” Imperfect though it is, VR can help to right that balance.

Some people with autism say they prefer VR to a conventional talking-head interview because it doesn’t require them to sit in front of a camera; they can work behind the scenes. “I find it an uncomfortably voyeuristic situation to have to explain my autistic experiences publicly,” Kruse says.

Even the best-intentioned typical people cannot fully understand what life is like for autistic individuals when it’s described only in words, says Benjamin Lok, a VR researcher at the University of Florida in Gainesville. Lok has not worked on such projects but has a 9-year-old son, Brandon, who is on the spectrum. “Trying to explain that world that Brandon sees, not only to us, but to family members — that is a challenge,” Lok says. “I would think, if I could get my mom and dad to go through [a VR] experience, how would they interact with Brandon differently?”

London has emerged as a center of autism-related VR projects. In 2016, Don’t Panic, a creative agency there, produced an immersive experience for the nonprofit National Autistic Society. The simulation portrays how isolated and overwhelmed an autistic child might feel at a shopping mall. In another simulation, the BBC’s corporate neurodiversity initiative puts its protagonist in an office meeting with a breathtakingly condescending coworker. Flashing lights and shimmering carpet patterns connote sensory overload, and a soundtrack that incorporates a throbbing heartbeat and rushed breathing signals a rising panic. Sean Gilroy, who ran the BBC project with an autistic colleague, says family members of people with autism or other conditions have reacted favorably. “They’ll spot things in the film that their sons or daughters or sisters or brothers have spoken about,” he says. “It brings it to life; it makes it real. It can get quite emotional for people.”

“The Party,” produced by The Guardian newspaper, is notable for its inner dialogue as its teenage protagonist copes with her unsympathetic relatives and family friends. The project was the brainchild of novelist Lucy Hawking, daughter of the late physicist Stephen Hawking; her son, William, is autistic. The script writer, Sumita Majumdar, is also autistic, and the project involved input from autism researchers. “We tried to be true to the science, but it was really important that we built a lot of the visual experience of the film on what people had said,” says Owen Parsons, a graduate student in Simon Baron-Cohen’s lab at the University of Cambridge in the United Kingdom. “Of course, if the science is correct, then those two things should not disagree with each other,” Parsons says.

Real-time reflections:

T

he augmented-reality system created by Nagai’s team also draws heavily on input from people with autism. Instead of dropping you into a department store or house party, it takes a video feed of wherever you happen to be and transforms it in real time to what a person with autism might experience. Whatever claim it has for representing autistic perception rests on a simple principle: People with the condition report that their sensory experience changes with context. “Sometimes I’m in a good condition, and sometimes I’m in a very bad condition,” says Satsuki Ayaya, an autistic doctoral student at the University of Tokyo who worked with Nagai on the simulator. Ayaya says these fluctuations make her conscious of what her condition involves and how it might differ from the neurotypical experience. The augmented-reality system seeks to replicate these fluctuations in autistic perception. “There are variations of symptoms in each individual,” says Shin-ichiro Kumagaya, associate professor of medicine at the University of Tokyo and another of Nagai’s collaborators. “It is the foundation to verify the validity of this system.”

In 2014, Nagai showed 22 autistic people videos of a train station, a supermarket and two dozen other everyday vignettes. In each of these settings, the participants rated which of 12 visual effects they experienced and to what degree. The team edited that list down to six particularly common effects: overexposure, blur, enhanced edges, heightened contrast, color desaturation and visual snow. Nagai correlated the participants’ reports with the features of each scene, such as brightness, movement and sound level. In the end, she settled on the last three of these visual effects as being the most reproducible for her simulation.

Nagai’s simulator is an unwieldy contraption: a standard gaming headset kitted out with a webcam. Cables run to a laptop that a graduate student carries in a sling like a newborn. When I strap on the simulator on a May day in Osaka, I have a normal, if slightly lagging, view of the room around me. As the student switches on the simulator, though, Nagai’s face blurs, making her expressions hard to read. Turning away, I quickly find myself captivated by what appears to be an abstract Impressionist painting; it turns out to be a gray cubicle partition. I look at my hand; the creases resemble intricate henna art. The system’s heightened contrast brings out even the smallest textures.

Led by Nagai, I shuffle down the hall and out into the parking lot. Everything is whited out at first, as if I’ve taken off my sunglasses on a glaringly bright day. Whenever I turn my head or a car drives by, color drains from the image, like applying a noir filter on Instagram. As we re-enter the building, the abrupt darkening unleashes a blizzard of random speckles across the scene like so many polka dots. The lobby’s uneven lighting is exaggerated. Shafts of light alternate with darker regions, giving people ghostly outlines.

The simulation is purely visual, but Nagai says they have been recruiting volunteers for a study of sound perception. The team is experimenting with audio effects such as white noise; drone notes, such as a constant 1000-hertz background sound; and filters that suppress certain ranges of audio frequencies.

Since 2015, Nagai and her colleagues have held some two dozen workshops in Osaka and Tokyo for teachers, therapists and parents of autistic children, in which they let people try the simulator, see clips from it in several everyday settings, and then discuss it. The researchers also screen conventional video documentaries about autism. Ayaya was initially dubious about the project but has since come around. “It was better than I expected,” she says. The simulation highlights aspects of her perception, such as the visual snow, that she had taken for granted, she says. “I was surprised that neurotypical people were surprised,” she says.

Being made aware of these perceptual differences could even help people with autism develop strategies to compensate. “One of the participants in our experiments told me that after she joined our experiment she started wearing sunglasses in her daily life,” Nagai says. “Also, she told me she changed the lights in her room to an LED system so that she can control the brightness.”

Ayaya stresses that even if the simulator succeeds at representing the autistic sensory experience, it cannot capture higher-level aspects of perception. “You may see how we see, but what you feel is not always the same as what we feel,” she says.

A forthcoming paper on the simulator reinforces this point. Kuriko Kagitani-Shimono, a pediatric neurologist at Osaka University, says she showed video clips from the simulator to 45 autistic people and 46 neurotypical volunteers and used magnetoencephalography to measure their brain responses. The patterns of activity did not match. “The actual sensory responses of autistic people are different from those of typically developed people wearing the simulator in many ways,” she says.

Nagai has not demonstrated the system outside these workshops, and a YouTube video does not do it justice, so other researchers were unable to comment. They say they like the principle it is based on, however. Parsons says the real-time feed might provide greater immediacy than a scripted film. “You’re getting that experience one step closer,” he says. Newbutt praises Nagai’s partnership with people on the spectrum: “Autistic people themselves can reproduce visual experiences, and this is very novel and something that has not been done before,” he says.

Intersubjective experiences:

R

esearchers are looking toward broader applications of VR to help autistic individuals. In September, Nagai held a workshop for architects and interior designers. Open-plan offices and fashionably noisy restaurants seem almost calculated to frustrate people with autism, and VR systems could be used to sensitize designers. “There could also be new directions for research, for example, in relation to how spaces — such as schools, doctors’ surgeries — could be designed to better reflect the needs of a neurodiverse population,” says Sarah Parsons, an autism researcher at the University of Southampton in the U.K. who consulted on “Beholder.”

Therapists, too, could use a simulator of autistic perception in their training. Lok has helped to develop simulators for doctors and nurses to practice their bedside manner, albeit not for autism. He founded a company, Shadow Health, to sell these virtual patients. “They can look any way; they can sound any way; they can behave any way that the educator wants them to; and you can get immediate feedback on how you did,” he says.

Clark says the technology could also be adapted to provide a portable meditative space for people with autism. “This could act as a place of sanctuary,” he says. In one unpublished study, Newbutt asked 31 autistic and 13 typical students, aged 6 to 16 years, at four English schools to rank the needs they thought VR could fill. The top choice in both groups of students was for VR as a means to relax when they feel overwhelmed. “Their preference, across all of those questions, is that [VR] makes them feel relaxed and calm,” Newbutt says.

Mundy also envisions a multiplayer or ‘yoked’ experimental paradigm in which two participants work together. One would be a passenger in the other’s experience, so as to see (and learn from) how the other reacts. “The world is no longer reacting to your gaze and head turns and things of that nature; it’s reacting to somebody else’s,” Mundy says. “You’re seeing the world and interacting with the world in a passive way.” Autistic and typical people alike could benefit from inhabiting someone else’s point of view, he says. “Virtual reality has the potential to establish real-time intersubjective experience.”

As with other uses of VR in autism research and therapy, however, there has been a lack of systematic evaluation. Participants in Nagai’s workshops fill out exit questionnaires, but Kumagaya says the team has only just begun to follow up to see whether the experience has any lasting effect on attitudes toward people with autism. For now, the only evidence that VR succeeds at eliciting empathy is anecdotal.

Many worry that by portraying only one narrow dimension of autism, VR applications may actually backfire. Zelić is blunt about its limitations: “I feel that it is almost impossible to convey the depth of autistic intensity and emotion visually because we don’t express this in recognized neurotypical ways, and so these types of reconstructions can fall into a kind of parody.” A cautionary tale comes from schizophrenia research. Over the past two decades, numerous researchers have developed immersive experiences of psychosis that depict visual and auditory hallucinations, including malign voices. These simulations can be disturbing to watch. In 2011, a meta-analysis of nine such projects found that they made the participants more empathic to people with schizophrenia but also less willing to interact with them.

The “Beholder” project seeks to present more general impressions that give a fuller sense of the autistic experience. “I didn’t really want to be drawn into another stereotypical ‘how difficult life would be if you had this condition’ [situation],” Clark says. “I think there’s a place for that, for sure, but I meant to do something different.”

Clark, known for his eclectic multimedia installations and stage sets, says the project started with an awkward pause in conversation with Birmingham Open Media’s founder, Karen Newman. “We were just talking about family life and quickly realized that both Matt’s son and my brother are autistic,” she says.

The two artists decided to explore VR as a natural way to communicate alternative modes of perception. Clark began observing his son more purposefully, watching for what captivated the boy. “He would open the curtains a little bit and just study the dust motes, as if it was like a universe of stars that were floating around,” Clark recalls. The pair also recruited Zelić and Kruse. Kruse described, among other things, how when something moves through the air, she sees it as though it were drawing out a path through space, giving each moment an extended duration. (Others have also suggested that altered time perception is a distinguishing feature of autism.)

The team opted to focus on what Clark’s son and Zelić find beautiful and translated these thoughts into a series of nature scenes, including the one with the butterfly. “No one ever thinks to talk about autism from this [positive] perspective,” Kruse says. “The narrative is always focused on difficulties, or the strange, maybe dramatic differences of the autistic mind.” And the vignettes they developed are compelling as art, regardless of what they may or may not say about autism. In one preview scene of “Beholder,” rain falls onto the floor, sending out languid ripples; in another, leaves fall gently to the ground. In yet another, I was shrunk to mouse size, lost but in awe of giant blades of grass swaying overhead.