The United States’ ability to test for the novel coronavirus finally seems to be improving. As recently as late April, the country rarely reported more than 150,000 new test results each day. The U.S. now routinely claims to conduct more than 300,000 tests a day, according to state-level data compiled by the COVID Tracking Project at The Atlantic.
But these rosy numbers may conceal a problem: A lack of federal guidelines has created huge variation in how states are reporting their COVID-19 data and in what kind of data they provide to the public.
These gaps can be used for political advantage. In at least one state, Virginia, senior officials are blending the results of two different types of coronavirus test in order to report a more favorable result to the public. This harms the integrity of the data they use to make decisions, reassure residents, and justify reopening their economies.
Other differences make it hard to track the pandemic. In at least three other states, officials have lumped together probable and confirmed COVID-19 deaths; most don’t specify how they’re counting deaths. While most states report the number of people who have been tested for the coronavirus, six states say they track the number of samples that have been tested—and California and New Jersey switched methods in the past few weeks. Louisiana, Nebraska, Nevada, Vermont, and New York do not report the racial or ethnic breakdown of coronavirus cases. Even more states fail to report the racial breakdown of deaths. It is still impossible to know, for example, how many black people have died of COVID-19, though the data that do exist suggest that black people are dying at much higher levels than other groups.
There are many ways that the lack of data has complicated the outbreak. Until May 12, the Centers for Disease Control and Prevention had not reported state-level testing data, leaving efforts such as our COVID Tracking Project to fill that gap. The lack of reliable national testing data has made it difficult for modelers, policy analysts, and others to understand the true scope of the outbreak. Data about hospitalizations have been even harder to understand. Because the states report hospitalization counts in fundamentally different ways, there is no way to calculate the number of people who have been hospitalized with COVID-19 in the U.S.
Many pandemic response efforts assumed clean, standard, accessible data would exist, but they do not. That said, Virginia’s decision to mix the results of two different kinds of tests marks a new low in data standards.
The state is reporting viral tests and antibody tests in the same figure, even though the two types of test answer different questions about the pandemic and reveal different types of information. By combining these two types of test, the state is able to portray itself as having a more robust infrastructure for tracking and containing the coronavirus than it actually does. It can represent gains in testing that do not exist in reality, says Ashish Jha, the K.T. Li Professor of Global Health at Harvard.
“It is terrible. It messes up everything,” Jha told us. He said that combining the test results, as Virginia has done, produces information that is impossible to interpret.
The state’s decision to combine the tests was first reported by the Richmond Times-Dispatch.
The two tests have little in common. Viral tests help officials do the basic blocking and tackling necessary to contain an outbreak. If someone tests positive on a viral test, they are still infectious, so they can be told to self-isolate in order to protect the susceptible population. Public-health workers can trace their contacts to find others who may be infected with the coronavirus but who are not yet experiencing symptoms. Viral tests can also be used to monitor people who work in high-risk environments—such as a meatpacking plant—to diagnose a contagious person before they spread the disease.
Antibody tests, on the other hand, allow for something closer to post-game analysis. They help officials understand the true number of people in a state or city who have been exposed to the coronavirus. But they do so on a lag: Individuals who test positive on an antibody test are likely no longer infectious, and were infected by the coronavirus at least a week earlier.
The two tests do not even examine the same specimens. Viral tests analyze a throat swab, nasal swab, or saliva sample. They are sometimes called “PCR” tests, after the polymerase-chain-reaction technique used to isolate viral genetic material. But antibody tests use a blood sample. They are sometimes called “serological” tests, because they analyze the blood serum.
In other words, combining positive and negative results from the two tests in the same statistic, as Virginia has done, makes no sense.
But commonwealth officials say they have no choice. Other states are mixing their results, claimed Clark Mercer, the chief of staff to Governor Ralph Northam, at a press conference this week.
“You can’t win” by keeping viral and antibody findings separate in public data, he said, adding that combining the two tests’ results was the only way to improve Virginia’s position in a list of states ranked by the number of tests they had conducted per capita. “If another state is including serological tests, and they’re ranked above Virginia, and we are not, and we’re getting criticized for that, [then], hey, you can’t win either way. Now we are including them, and our ranking will be better, and we’re being criticized,” he said.
We could not find evidence that other states are blending test results in the way that Mercer claimed. In an email, a spokesperson for the Virginia Department of Health claimed that Arizona, West Virginia, and the District of Columbia also mingled viral and antibody results. This is false: Those three governments either separate out, or do not report, the result of negative antibody tests to the public.
Other states report positive serological tests as “probable” COVID-19 cases. This is in line with recommendations published by the Council of State and Territorial Epidemiologists, a nonprofit that works with state and local epidemiologists in the U.S. At least 16 U.S. states and two territories have reported such “probable” cases, although they may not always have done so using serological tests, according to the CDC. However, Arizona, which reports a substantial number of positive serological tests on its own website, does not show up on the CDC’s list. Nor does Kansas, which explicitly states that they are including such probable cases.
While including antibody tests in a state’s total creates too rosy of a testing picture for a state, reporting only probable positive cases without disclosing how many antibody tests are being completed could actually make the situation look more dire in a state than it is.
The spokesperson said that Virginia planned to “disaggregate” its viral and antibody results in the future, but he did not provide a firm date.
Kathy Turner, deputy state epidemiologist for Idaho and the presenting author of the CSTE standards document, did not criticize Virginia’s decision, but she did lay out why her own state decided to keep PCR and serology tests separate.
“[I]n Idaho, we have decided to only display viral tests because those are the denominator we use to calculate our percent positivity rate and we are very confident what they mean,” she told us. “Additionally, we focus on the PCR tests because we can compare the percent positivity over time—before serology tests were available.”
Blending the results also misstates Virginia’s success at improving this crucial metric, sometimes called the “test-positivity rate.” This measurement compares the number of people who have tested positive for the coronavirus to the number of people who have been tested overall. In April, one in five Americans who received PCR tests for the virus were found to be infected, a very high rate that suggested only the sickest people could get a test. For the past week, fewer than one in 10 tests in the U.S. have found a positive result, according to state data. Some of this improvement is certainly the result of the New York metro area’s waning outbreak.
Leaders in many states, including Virginia, have cited the local test-positivity rate to justify loosening shelter-in-place restrictions. Northam has repeatedly said that Virginia’s test-positivity rate had to fall for 14 days before he would loosen restrictions.
But because Virginia combines viral and antibody results, its positivity rate is unusable, said Jha, the Harvard professor. The positivity metric is only useful when describing the result of viral tests, because it is meant to provide a rough estimate of how many people infected with the coronavirus are getting tested for it. Antibody tests, which are meant to sample a broad swath of the healthy population, should not be included in it. By lumping the two tests together, as Virginia has done, states can artificially improve their test-positivity rate.
Only by keeping the two types of test separate can the country—and the commonwealth of Virginia—understand the true scope of its outbreak, experts say.
“You’re comparing apples to pears,” Gigi Gronvall, a senior scholar at the Johns Hopkins Center for Health Security, told us. Viral and antibody tests “look a little bit alike, but it doesn’t let you make the comparison needed. So why not keep them separate?” Gronvall has written about the need to expand antibody testing across the U.S.
She is also worried that the data are statistically meaningless, because viral tests have fewer false-positive errors than antibody tests. “There’s so much variability in the antibody tests that it’s like taking an iffy number and throwing it in with some more reliable numbers,” she said.
Although combining the data from two different tests may seem like a technicality, the decision threatens to confound some of the most important questions about the coronavirus’s path in the United States. How many people are sick right now? How many people can the U.S. actually test for the coronavirus every week? Is the situation outside the New York metropolitan area getting better or worse? Answering these questions requires stable data about how many people have received a diagnostic test and how many of those people are infected.
The scope of the test-mixing problem is not yet clear. No other state aside from Virginia has admitted to counting antibody tests in their overall totals. We do know, however, that large numbers of antibody tests are being completed in the U.S. but not reported to the public by most states. As test numbers have shot up, Quest Diagnostics—one of the two largest commercial laboratories in the country—reported doing only 200,000 PCR tests from May 4 to May 11, which is 180,000 fewer diagnostic tests than in the preceding week. In fact, from May 4 to May 11, the company did almost 100,000 more antibody tests than PCR ones. LabCorp, the other major commercial reference laboratory, has not released similar data, but it has stated that its diagnostic- and antibody-test capacity are about equal.
Two states do report viral and antibody tests separately. In Colorado, 30 percent of tests completed so far in May were for antibodies. In Arizona, 23 percent of the total number of the tests ever done in the state have been serological. In both states, antibody tests started to be conducted en masse around April 26.
This timeline matches up disturbingly well with the improvement of the national picture, which saw a sudden jump at the end of April from an average of about 150,000 tests per day to 200,000, 250,000, and now 300,000 tests per day. The White House has celebrated the improvements in testing, noting in a press conference this week that the United States does more tests per day than any other country. “We have met the moment and we have prevailed,” President Trump said.
In fact, the U.S. has tested a smaller share of its population than other industrialized countries, including Italy, Canada, and Germany.
“I find our testing record nothing to celebrate whatsoever,” Senator Mitt Romney, a Republican of Utah, told Brett Giroir, the assistant secretary for health, at a hearing yesterday. “You celebrated that we had done more tests, and more tests per capita even, than South Korea. But you ignored the fact that they accomplished theirs at the beginning of the outbreak, while we treaded water during February and March.”
It’s possible that Virginia is alone in its reporting methodology, but until we know how many states are dumping antibody tests in their totals, the White House’s claims that the U.S. has overcome its testing plateau cannot be given full weight.
It was one thing to go into the outbreak blind because of the lack of testing, as the U.S. did. It’s another to choose to cloud our vision. Antibody-testing data are an important part of understanding the outbreak; PCR diagnostic-testing data are also an important part of understanding the outbreak. But if states mix the two together, the value of that information plummets.
The good news is that laboratories do report the type of tests they’ve conducted to state governments and the CDC. All states should have that data at their disposal. So should all their residents—and all Americans.
