But behind the scenes in early April, the researchers were running into obstacles getting their results into the world, according to the complaint.
They had asked Taia Wang, an infectious disease expert at Stanford, to validate the accuracy of their antibody test. And from her perspective, they were in a rush.
Bendavid, an associate professor of medicine, started calling and emailing Wang on March 29, as she would later note in an email chain that mushroomed to more than 15 Stanford researchers leading or involved with the study. “There seemed to be tremendous urgency around this request,” she wrote, in a mid-April email attached to the complaint.
In a separate thread of screenshotted emails, which came into the complainant’s possession by mid- to late April, the lead scientists and Neeleman appear to discuss Wang’s efforts to check the test.
One email, without a visible timestamp or sender that was sent to Bogan’s and Neeleman’s addresses, read: “David, I think you should write Taia a note and tell her you’ll support her lab if she validates this kit.” Bendavid confirmed that he put Neeleman and Wang in touch.
And Neeleman did write to her. “First and absolutely most importantly, we have to establish without any doubt, the efficacy of these tests,” he wrote. “I am frustrated by what appears to be the lack of urgency.”
Neeleman acknowledged that “the Santa Clara test has been in the works for weeks,” then expressed interest in doing a future antibody study in New York. He also made clear what kind of result he thought would make a bigger media splash.
“Unfortunately PR impact and the ability to raise large amounts of money quickly will not be the same if you announce 1% of Santa Clara County tested positive for the antibodies versus 30% of New Yorkers which would be huge news,” he wrote.
Finally, Neeleman dangled the prospect of funding her to conduct that future test. “If you are willing to do a 5,000 test in New York, just tell me the cost and I will raise the money immediately,” he wrote, signing off with his cellphone number. “Time is of the essence.” (In a screenshotted email included in the complaint, Neeleman appears to share his note to Wang with the lead researchers.)
Reached by BuzzFeed News, Wang declined to comment on most of the emails or her testing. But she said she did not ask Bendavid for the email introduction. Neeleman’s message about providing funding for her to do a test in New York “really upset me,” she said.
She said she spoke to him once by phone afterward. “He expressed interest in doing some studies in New York, and I told him I thought it was a good idea, but I’m not doing that,” she recalled. “My lab is not a contract lab. That’s not what we do.” She added, “I did not request funding or receive any funding for anything related to this.”
For his part, Neeleman said he was just trying to be helpful and was curious about “how it was going.” He said he recalled asking her “‘Do you need money?’” to which he said she responded, “‘No, I have plenty of money.’”
“She had the tests all done by the time I talked to her,” he added. “It was just a nice conversation, I didn’t pressure her or anything.”
Neeleman told BuzzFeed News he was interested in New York because he sees value in conducting antibody tests in places with lots of infections and deaths. “That’s why I didn’t want them to do Santa Clara,” he said. “If they had just done the New York study first, there wouldn’t be so much scrutiny.”
Bendavid said his intent was to possibly “provide additional support at a time of great stress” for Wang, not to financially pressure her, since Wang had mentioned that she was tight on lab space and staff.
He said he was not frustrated with the pace at which she was validating the test, and that his team had thanked her over multiple emails. “I fully understood her constraints and her considerations,” he said.
Wang’s experiments on the test left her “alarmed,” as she would soon recount on the email thread to the group of Stanford faculty. In her retelling, she had told Bendavid by phone that the test entirely missed a certain class of antibodies in some samples. She also told him, she informed the group, that she thought the test “performed very poorly on samples with lower antibody” levels that are more representative of people with mild or asymptomatic infections.
Regardless, the paper ended up including Wang’s data. In a section that describes the test’s accuracy rates, the preprint states that out of 30 samples from virus-free people, the test correctly produced negative results for all of them. But out of 37 samples from known COVID-19 patients, the test correctly detected antibodies in only 27 of them (erroneously identified as 25 in the first preprint).
In her email to the group of faculty, Wang was clear: She did not want her name on the paper and did not trust the test.
“I declined authorship on any manuscript because, based on our testing, I do not believe that the [Premier Biotech] kit is a robust readout for the presence [receptor-binding domain] antibodies,” she wrote.
Wang fired off this email on the morning of April 12 — five days before the results were released to the world.
IV. “I would feel that we were responsible.”
Wang’s message was part of an email chain that began bouncing into Stanford faculty’s inboxes on the eve of Easter weekend. Not long before, President Trump had publicly declared that Easter would be a sure deadline to open up the country. Instead, the US surpassed Italy that weekend to become the country with the most coronavirus deaths in the world.
On the thread were the scientists helming the study — Bhattacharya, Bendavid, and Ioannidis — and a host of other researchers involved. One of them was Scott Boyd, a pathologist who would also try to verify the Premier test’s accuracy. According to emails in the whistleblower complaint, he, too, ended up arguing that the test was unreliable.
It’s unclear why or how Boyd came to try validating the test, since Wang had already done so. But in an email sent just before 1 p.m. on Friday, April 10, Bendavid outlined several possible ways that Boyd’s lab could go about the task.
Bendavid mentioned that he was particularly worried about the test’s rate of false positives. If the test generated more false positives than the scientists were expecting, the results would throw off their infection estimates and affect what they could tell people about their antibody status, he wrote.
Coronavirus antibody tests have big limitations. Scientists don’t know whether, or for how long, antibodies confer immunity against the new virus, for example. And false positives — incorrect reports that someone has antibodies to the virus — might give people the unfounded confidence to stop socially distancing, potentially causing them to contract and spread the disease.
There is also a greater risk of an individual positive test result being incorrect when a disease is on the rarer side in a community, which may be the case with COVID-19 in some places. Even if a test generates a very low percentage of false positives, its ability to provide people with accurate results is hampered if a pathogen is present in the world at about the same rate.
In the Stanford study, participants had been told they would hear back if they had positive results in a few days to a week, three site volunteers told BuzzFeed News.
Boyd worried about the risk of false positives giving participants incorrect information. “We are concerned about the specificity of the Premier Biotech devices that were donated for your study,” he responded on the night of the 10th, referring to the rate at which the test generated false positives. Anyone with a positive test result, he recommended, should be invited back to provide a new sample to be run on a different kind of antibody test, known as an ELISA, considered the gold standard in clinical lab testing.
“If a participant were to have a false impression that they have protective antibodies, and change their behavior as a result,” Boyd wrote, “I would feel that we were responsible.” He proposed that he would try to verify the test’s accuracy rates by running ELISA tests on various kinds of samples, including a subset of the blood drawn at the pop-ups, which would take until the end of that Sunday, the 12th.
He and Bendavid fell into a back-and-forth. Bendavid voiced concern about the blood samples being a week old and frozen. And he came out in defense of the test’s false-positive rate, saying that preexisting data indicated “good specificity.” He did not commit to contacting and retesting participants, citing, among other factors, restrictions laid out in the study’s Stanford-approved protocol.
Boyd shot back.
“You have identified, using a kit of uncertain provenance that was given to you, that you did not verify in any substantial way, a number of people in our community that you were prepared to tell had been infected with SARS-CoV-2 and had raised antibodies against this serious viral pathogen,” he wrote, shortly after noon on Saturday. “With the same untested device, you have given ~3,200 members of our community the impression that if they don’t hear from you, they have no antibodies to SARS-CoV-2.”
Bendavid seemed “resistant to the idea” that people with positive results should be contacted and retested, Boyd continued: “Is this because it would take some time to do so?” Furthermore, he noted, Wang had told him that she did not think her experiments “validated or verified the accuracy of the Premier Biotech kits at all.”
The professor urged Bendavid’s team to prioritize giving people “the most accurate information possible, even if that delays reporting the results of the study for weeks or months.” He concluded, “I hope you will not continue to exert yourself in finding ways to disregard that fundamental responsibility.”
The following week, Boyd had results in hand. Using ELISA tests to reassess the samples of community members who had come up positive for antibodies on the Premier test, he had ended up getting positive results for a little over half of them.
“The kits did not perform as badly as I had feared they might,” Boyd informed the Stanford faculty thread at 1:34 a.m. on the 14th, “but I am concerned that they do not perform well enough to report results back to patients if there are better options available.”
Unbeknownst to Boyd, he was too late. According to the whistleblower complaint, the researchers did not wait to hear what Boyd thought. The afternoon before, they had gone ahead and submitted their paper to MedRxiv, the preprint website where it would appear online several days later.
“This pre-print submission occurred after the first attempt at validating the study’s LFA antibody test was called into question, but before the results of a second validation were known,” the complaint alleges.
Bendavid said the authors made a decision based on all the feedback they had at the time from many experts across disciplines, including Boyd and Wang. Ioannidis also pointed out that it was a preprint, not a published study, and therefore subject to further revisions.
And Bendavid defended what the study authors told participants about how to interpret their results. According to him, the study’s Facebook ad warned, “The study is designed to guide public health in our county, not personal health status,” while a handout to all participants, and a script read to people who tested positive, noted that those results could be inaccurate. He declined to provide a copy of these materials.
The preprint went online on April 17. It quickly
blew up on social media, driven by popular right-leaning commentators and with hashtags like #ReopenAmerica, #EndTheLockdown, and #BackToWork.
Before and after the study went live, the researchers made the rounds in the media, from the
BBC and
CNN to right-leaning outlets. On
Tucker Carlson’s show on April 14, Bhattacharya said the infection fatality rate is “likely orders of magnitude lower than the initial estimates” and “much likely much closer to the death rate you see to the flu per case.” On April 15, Bogan
called in to chat with the conservative radio host John Fredericks. A week later, Ioannidis Skyped in to
Laura Ingraham’s show.
The whistleblower complaint made note of the authors’ media appearances, observing that they “coordinated the online publication of a non-peer-reviewed pre-print with multiple press appearances” and “continued to promote the results of this and subsequent studies despite severe criticisms of their pre-print from multiple statisticians.”
On the 21st, in response to the criticisms, Ioannidis
told the New York Times: “It’s not perfect, but it’s the best science can do.”
On the Stanford email thread, meanwhile, he and others were showering Boyd with praise for all his behind-the-scenes work. “No words can be good enough to thank you,” Ioannidis wrote on the 20th. “I want to make sure that the quality of the data is as good as it can be.”
In response, Boyd was hardly as warm. He made clear that none of the data he’d generated was allowed to see the light of day.
“When the results of the re-testing of the study participants are returned to them tomorrow,” Boyd wrote, close to midnight on the 20th, “I will be happy to end my involvement in this situation without any ties, acknowledged or unacknowledged, to current, revised, or future contemplated preprints, publications or other public presentations of results from the Premier Biotech test kits.”
Boyd did not return requests for comment.
V. “I’m just a scientist.”
After the first preprint got pummeled by criticism, the Stanford team publicly stated that they would revise the study to address the issues raised. “This is exactly the way peer review should work in science,” Bhattacharya
told BuzzFeed News, days after the first study was released. (Bhattacharya did not respond to requests for comment for this story.)
On April 30, a second preprint toned down the researchers’ conclusions, included more detail about their process and analysis, and addressed at length many of the critiques they had received.
The first version had stated an estimated range of between 48,000 and 81,000 infected people in Santa Clara County by early April — between 50 to 85 times more than the number of confirmed cases. But the revision gave a much wider range of between 25,000 and 91,000, or 26 to 95 times more than the number of confirmed cases. That change reflected a greater degree of uncertainty in their results.
Accordingly, a sentence that read “the infection is much more widespread than indicated by the number of confirmed cases” was changed to say that it “may be much more widespread” (emphasis added). The estimated infection fatality rate was also revised to 0.17%.
To at least some critics, these updates appeared to be
improvements. But the whistleblower complaint raises questions about other claims in the new study that have not been addressed.
In an email cited by the complaint, Boyd reported retesting 47 positive samples from the Santa Clara County residents. But the second preprint, like the first, reports there were 50 positive samples. Bendavid said they had pictures and records of all 50, and deferred questions about the other three to Boyd.
The revised manuscript also provided a false-positive rate for the Premier test that was based on many more blood samples. Before, their specificity calculation was based on a total of 401 samples; in the new version, it is based on an additional 2,923 samples. The authors did not explain where all the new sample data had come from.
But the complaint cites an internal draft with language stating that the vast majority were “data obtained by personal communication with laboratory director of Hangzhou Biotest,” the Chinese manufacturer. This data was not originally provided by the company, the complaint alleges.
“This raises concerns that the data could be biased, intentionally or not, providing an unrealistically favorable view of the test,” the complaint states, claiming that the additional data “presents the test as 7.5 times more precise than all other validation data available.”
Bendavid said that claim, to the best of his knowledge, was false, and that the researchers had analyzed all of the data themselves.
And the preprint does not name any of what it said were its “many individual donors.” A disclaimer stated that the “funders had no role in the design and conduct of the study, nor in the decision to prepare and submit the manuscript for publication.”
All in all, the complaint raises even more questions about a study that has already come under fire. And while the university investigates, states have started easing stay-at-home orders.
Dr. Anthony Fauci, the nation’s top infectious disease expert,
told Congress this week that ending lockdowns now threatens to spark new waves of infections and deaths and set back progress toward recovery. Nevertheless,
more than 30 states have reopened businesses or started limited reopening.
It’s unclear how much the Stanford study has informed decisions to reopen the US. But in
a Senate testimony last week, Ioannidis reiterated his stance that, while “shelter-in-place and lockdown orders were justified initially,” continuing them long-term could have drastic consequences on other aspects of health and the economy. Moving forward, he said, the nation should protect at-risk groups, but “reassure most citizens — those of younger ages without serious preexisting conditions — that they are at very low risk.”
That same week, Neeleman
declared on Twitter, “When this is all over Dr Ioannidis will [
sic] vindicated even though he is much maligned now.”
Ioannidis was recently asked point-blank about his study’s policy implications by the science news site
Undark. While denying that the study made any explicit recommendations, Ioannidis also said it supports the hypothesis he has put forth all along: “this is a very common infection, and very often it is asymptomatic, so it goes below the radar screen.”
His views, he insisted, were driven by data, not politics.
“I’m just a scientist,” he said.
No comments