Conversation with avatar teaches social skills

In a virtual reality game intended to improve social skills in teenagers with autism, the players must ask computer avatars the right questions while seeming engaged in the conversation. The game was described 27 September in IEEE Transactions on Neural Systems and Rehabilitation Engineering¹.

Most teenagers love video games, and those with autism maybe even more so. Researchers used this to their advantage by developing a virtual reality game that teaches people with autism how to engage with others.

In the first version of the game, introduced last year, teenagers with autism watch an avatar tell a story and guess at its emotions. The game gives feedback based on the answers and whether the player made the appropriate amount of eye contact².

That version did not allow the player to interact with the avatar, however. In the new version, 1 of 12 teenage avatars, 6 of each gender, tells one of a number of personal stories, which the researchers took from a database of high school reports.

The player then tries to learn something about the avatar, such as the first time the avatar played football. The game provides the players with a set of questions to ask, but only a particular order is considered appropriate.

For example, players are encouraged to introduce themselves before asking a question. If a player asks questions in the wrong order, the avatar responds by guiding him to a more appropriate query.

The game also tracks how often a player looks at the avatar, how often he blinks and the size of his pupils. This information is used to generate an ‘engagement score.’ Studies have shown that people who are involved in a conversation tend to blink less than usual and contract their pupils.

The avatar also gives the player feedback. For example, if the player was not engaged, the avatar tells him, “Your classmate didn’t think you were interested in your conversation with her. If you pay more attention to her, she will feel more comfortable.” Players who answer the query quickly and make the right amount of eye contact are given more difficult tasks.

The researchers tested the system on eight teenagers with high-functioning autism. All of them enjoyed the game and spent more time looking at the avatars’ faces towards the end of their sessions than they did at the start. Six of them said that the game taught them to introduce themselves when speaking to someone for the first time and to look them in the eyes when conversing.