Scammers threaten quality of research survey data

On a rainy October evening in 2022, Simone Dufresne, a graduate student at Tufts University in Medford, Massachusetts, was looking with dismay at the data from her survey of autistic adults. There were telltale signs of fraud, including in the answers to the open-ended questions.

“Some of them were phrased kind of strangely,” she says, and there were words spelled in British English even though it was a U.S.-based study. But she didn’t want to make assumptions about the participants, so she turned to Scamalytics, a website that scans IP addresses and estimates whether their web traffic is legitimate.

The results? “Fraud, fraud, fraud,” Dufresne says. Of the 17 responses Dufresne had gotten for her survey, 13 of them, she figured, were fake. It was bad enough that these respondents were likely just pretending to be autistic, but also she had offered compensation for survey respondents, and these people had already been paid. The small grant she had won for this research was now more than half gone.

Dufresne emailed her graduate adviser, the psychology professor Eileen Crehan, and laid out her discovery.

Crehan, for her part, “got a pit in my stomach” as she read Dufresne’s message, she remembers. It was a setback for the project overall, but more importantly, it was bad news for Dufresne — the junior researcher had planned to use the study to qualify as a doctoral candidate at Tufts.

This was not the first time Crehan and her team had encountered fraud in internet-based research — it has long existed in the academic world — but it has ramped up in recent years and is being compounded by scammers now infiltrating Zoom interviews. This combination has some researchers questioning the future of survey work.

I

n a 2015 paper, “Detecting, preventing, and responding to ‘fraudsters’ in internet research: Ethics and tradeoffs,” a team of health researchers wrote they were alerted to the growing problem of fraudulent survey takers in 2011. By 2019, autism researchers were encountering scamming in responses, and that trend seems to have increased with the rise of internet crime overall during the COVID-19 pandemic: The U.S. Federal Bureau of Investigation, for instance, reports that cyber threat complaints increased from 467,000 in 2019 to more than 800,000 in 2022.

Robert Klitzman, a psychiatry professor at Columbia University and lead investigator on the 2015 paper, says that the extent of “fraudsters” in research is unknown, and more data are needed to assess the scope of the problem. Indeed, specific data are hard to get. The FBI does not break out survey fraud from internet crime, and though the U.S. Department of Justice indicts people globally for internet or email-based fraud that swindles Americans out of large sums, it has never prosecuted a survey fraud and would not comment on this specific scam.

Anecdotes from autism and disability researchers, however, suggest survey scamming is on the rise. Kathleen Bogart is a psychology professor at Oregon State University in Corvallis who studies disability. For more than 15 years, she says, she was able to trust internet-based recruitment methods. People with disabilities, especially those with rare conditions, can be challenging to find, and the internet has been a “crucial tool.”

But in 2021, she put out a multiple-choice and open-ended-question survey about Moebius syndrome, a rare condition that affects facial muscles. Bogart is well known in that community because she also has the condition, and she knows there are “probably fewer than 1,000 people in America” who also have it.

When the survey was shared by an advocacy organization on Twitter, there were 2,000 responses within a day. “We don’t know if they’re people or bots,” she says. It was a “nightmare” to try to tease out the legitimate respondents, Bogart says, and although she is “relatively certain” the remaining sample is usable, there’s a chance some non-scam responses got thrown out in the screening process.

Crehan also recently received a crush of responses to an open-call survey. It asked about experiences around autism and sexuality, and she, too, got nearly 2,000 responses overnight. She and her lab went through the responses, which took months, before settling on three that might have been genuine. She paid those respondents but threw out the data and gave up on that questionnaire.

“I’ve talked to other researchers who are like, ‘This is killing us,’” Crehan says. Survey data sourced from the internet once provided rich datasets for her graduate students to mine, but “it’s a different story now.”

A

rapid onslaught of survey responses can mean bots or human-guided bots have gotten around security measures, says Gang Wang, a computer science professor at the University of Illinois Urbana-Champaign. In collaboration with researchers at Pennsylvania State University, last year he evaluated 22 anti-fraud tests that could help prevent bots in online surveys. The researchers suspect, he says, “there were humans helping the bots to navigate” past the more difficult security tests.

Both Bogart and Crehan used Qualtrics software as the platform for their tainted surveys. To help combat fake survey responses, Qualtrics offers, for an additional fee, a fraud detection service as an add-on to a license. This can help filter data by “flagging” fraudulent and suspicious responses, a Qualtrics spokesperson told Spectrum in an emailed statement, while still allowing customers to review them and self-determine validity. The spokesperson declined to answer additional questions about survey security.

Bogart used Qualtrics security tools for her survey, but she says they were not perfect, and she still chose to scan results herself. In general, the built-in security features of the Qualtrics software are “part of the solution, but not the full solution,” says Wang, who uses the software for his own surveys. The key, he says, is to use multiple anti-fraud tests to filter responses, and to incorporate these features ahead of time, before the survey is launched. But “the most effective thing we find,” Wang says, is embedding questions in the survey “that touch a bit on the domain knowledge or the human ability for logical reasoning.”

W

hereas bots have long plagued online surveys, fraudsters have recently taken to faking interviews in video calls — a scam Wang calls “next level.” This, too, has struck autism researchers. Elizabeth Pellicano moved her research online during the pandemic, but it wasn’t until a recent Zoom focus group that she realized she had a problem on her hands.

Pellicano was conducting focus groups to ask autistic people, family members, practitioners and researchers about their experiences with autism research. One of the groups consisted of four people who were supposed to be autistic parents of autistic children. Pellicano, an autism research professor at University College London in the United Kingdom, had provided questions ahead of time and allowed group members to keep cameras off or use the chat feature to respond to questions. These measures were meant to help meet the needs of autistic people, but they also made the session easier to hack — three of the participants kept their cameras off, and, oddly, one of them said during the interview that they did not have an autistic child. Pellicano continued the interview anyway, and by the next day she had already received emails demanding payment.

That was unusual, she says. So were some of the email addresses: Three were formatted in a “first name, last name, number” format, a pattern that economics researchers have also noticed sometimes signifies fraud. Also, the interviews were unusually short, and plagued with internet connection issues.

In the end, Pellicano figured three of the four participants were scammers. She was alarmed enough by her experience to gather up nine co-authors and publish a letter to the editor in Autism in May, detailing their experiences with studies across three countries. Crehan was not an author on the letter, but she has encountered fraud on video interviews too. She says one of her research assistants caught a parent telling their child to “pretend that you’re autistic” in a Zoom interview.

Part of the usefulness of online surveys and interviews is their reach; participants can log in from anywhere. And social media makes it easier to find respondents who would otherwise be hard to locate, Bogart notes. Offering to compensate survey participants is a useful incentive — Pellicano’s Zoom focus group, for instance, paid participants 25 British pounds — but these three things also make online qualitative research ripe for scamming.

Posting survey links on social media sites and other public websites has been dubbed a “primary cause” of fraudulent responses by some researchers. Pellicano has pondered stopping payment for surveys, though she says that respondents “deserve to be paid for their time, sharing their time and experiences.” But at the same time, if scamming is “being driven by monetary reasons, then doing away with incentives should get rid of this problem.”

B

ogart would still like her participants to be paid, but she has decided to reveal the monetary reward only after someone has started the survey, instead of showcasing it up front. She has also sworn off posting open calls for her surveys on social media and will post only in restricted groups, such as those that are private on Facebook.

Researchers are considering other ways to stem the tide, such as collecting IP addresses or requiring participants to turn their cameras on during interviews, but that can run afoul of an institutional review board (IRB), which is tasked with making sure the privacy concerns of human participants are upheld. Bogart has begun advocating to her IRB that IP addresses should be collected to track each respondent’s country of origin. Although, as Pellicano points out in the letter to the editor, researchers must be aware that virtual private network (VPN) clients can help scammers mask their location and IP address.

There will likely be new tactics from scammers, too. Klitzman says that now, years after his 2015 analysis, fraudsters are more technologically sophisticated than ever. For example, a 2020 study from the Pew Research Center found a sophisticated bot that can be used to fill out surveys with “undetectable mode” and was available for commercial purchase. When you add artificial intelligence to the mix, “the landscape is changing,” Klitzman says.

Still, Bogart is determined to keep using online research, as it has been critical to her career. “I know I’m one of the only psychologists who studies rare disorders,” she says. “There’s just really no other way to do that type of research without using the internet as a tool, so I just, I need to figure it out.”