Your donation will help us produce journalism like this. Please give today.

As schools work to mitigate COVID spread in classrooms and get a handle on how many teens have been immunized, they may not be able to rely on vaccination data published by the Centers for Disease Control and Prevention.

In many cases, CDC numbers clash with locally reported vaccination rates, an analysis from The 74 reveals, including multiple instances of double-digit gaps between local and federal counts. In some counties, the agency’s data indicate that the share of 12- to 17-year-olds who have received at least one vaccine dose is impossibly high — 101 percent in Miami-Dade County, Florida, and 104 percent in San Francisco County, California, for example.

The lack of clarity takes on heightened significance as the country moves to vaccinate another swath of the K-12 population, with coronavirus shots for children ages 5 to 11 currently under review by the U.S. Food and Drug Administration and expected as soon as early November.

Inaccuracies in CDC data could have implications for the nationwide understanding of vaccine uptake among young people. The widely cited youth COVID immunization rate calculated by the American Academy of Pediatrics, for example, is based on CDC numbers.

“Our method is to clearly state the source of our data,” Suk-fong Tang, senior database analyst for the AAP, told The 74. But due to time limitations and the vast quantity of information, she said, “it is not possible at this time to validate everything that we use.”

“We work with [CDC] data with the faith that the data really captures the large trends,” the AAP expert continued. “It may not be, you know, accurate down to the single-digit counts.”

While the CDC does not publish youth immunization data directly, it releases vaccination rates and raw counts for those over 12 and those over 18 by county. Using those numbers, The 74 calculated the rate of inoculation for 12- to 17-year-olds via a method that Tang confirmed produced a “highly similar” youth vaccination figure as the AAP. (Click here to see the math.) Those rates frequently deviated from local reports, indicating possible flaws in the federal agency’s vaccination counts, population counts, or both.

For example, CDC data downloaded by The 74 Sept. 30 indicate that in Queens County, New York, 86 percent of teens have received at least one vaccine dose, while NYC Health said the county’s figure was actually 74 percent. In an especially extreme case, federal data for Coconino County, Arizona, indicate a 93 percent one-dose vaccination rate for 12- to 17-year-olds, while Coconino Health and Human Services reported a 57 percent rate.

Coconino County officials explained the gap in an email, saying they use a “more enhanced data cleaning process” than the state or the CDC.

Other areas such as Fairfax County, Virginia; Marin County, California; and Howard County, Maryland, reported youth vaccination rates that closely aligned with federal counts, differing by under 5 percentage points.

The CDC did not respond to the discrepancies identified by The 74, and did not provide comment when asked for the reasons behind them, despite over a half-dozen requests made over more than a two-week span. The agency did send a link to information on its vaccination data reporting protocols, which explained that their population counts are based on the Census Bureau’s 2019 estimates, meaning that their percentages could be inaccurate if individuals moved counties in the last two years.

Population shifts may not completely account for the discrepancies. Outside experts also said issues such as double-counting vaccinations or delays in reporting data upstream to the federal government could contribute to inaccuracies.

‘Flying blind’

Data woes have plagued the CDC throughout the pandemic, said Ali Mokdad, who, after years monitoring vaccine coverage at the federal agency, is now a professor of epidemiology at the University of Washington.

“We’re flying blind,” he said, pointing out that U.S. decisions around Pfizer booster shots were based on data from Israel and Qatar, where vaccination numbers are collected in a more standardized fashion, due to a dearth of reliable U.S. data.

From the CDC’s decision to scale back tracking for breakthrough infections to their flip-flop on indoor masking for vaccinated individuals, the agency has come under fire at multiple points throughout the pandemic.

In past months, the epidemiology professor said, many Americans received unauthorized booster doses by crossing state lines or lying about their vaccination status thanks to lax immunization tracking. “We don’t know [exactly how many people are] vaccinated or not, and what types of vaccines they have received and when.”

That can become a life-or-death problem, said Mokdad. “When you know how many people are vaccinated, you know what immunity you have in your community,” he explained. “[But if you don’t] know how many people are vaccinated … you can’t get a handle on how many people are susceptible in your own community and then that will sustain a surge.”

In instances where local vaccination numbers are above CDC counts, it’s possible that states have been slow to report their most recent immunization data, Emily Pond, a researcher for the Johns Hopkins University vaccination tracker, told The 74 — she calls that glitch “data lag.” Where the CDC count is higher, Pond explained, federal overseers may have access to vaccination counts that local departments of health do not, such as coronavirus immunizations that occurred at army bases or on tribal lands through the Indian Health Service.

In Navajo County, Arizona, for example, Assistant County Manager Bryan Layton said via email, “We openly acknowledge the inherent challenges of tracking and reporting case data and vaccine rates in a rural county that is home to 3 different sovereign tribal entities: the White Mountain Apache Tribe, the Hopi, and the Navajo Nation …. The Navajo Nation … uses a series of service areas that do not necessarily conform to county or state jurisdictions.”

Still, local reports in Navajo County say 19 percent of residents under 20 years old have received at least one dose of the vaccine, compared to CDC numbers that put the rate for 12- to 17-year-olds at 98 percent — a gap that likely can’t be fully explained by Indian Health Service data absent at the local level.

“I have a red flag when any [vaccination rate] is above, like, 90 percent,” said Pond. Differences between CDC and local figures, she said, can be rather common.

‘A lot of moving parts’

To get a better sense of the frequency of discrepancies between local and CDC vaccination rates, The 74 queried a random sample of 10 U.S. counties, a small sliver of the over 2,600 in the full dataset and separate from the analysis of the counties with the highest reported rates. Seven returned data for comparison against federal numbers, some using slightly different age boundaries for youth vaccination than The 74’s 12- to 17-year-old range.

Out of those seven counties, three had rates that diverged from CDC numbers by more than 5 percentage points. Hood River County, Oregon, reported that 72 percent of youth ages 12 to 17 had received at least one dose of the coronavirus vaccine while the CDC reported an 80 percent figure. Sullivan County, Pennsylvania, reported a 23 percent rate for youth ages 12 to 19 compared to a 30 percent 12- to 17-year-old rate from the CDC. And Schoolcraft County, Michigan, reported that 32 percent of youth ages 12 to 15 and 47 percent of youth ages 16 to 19 had received at least one dose, compared to a 13 percent CDC rate for youth ages 12 to 17. Both Sullivan and Schoolcraft counties have populations under 10,000, meaning small inaccuracies could have an outsized impact on their vaccination percentages.

The inconsistencies don’t surprise Michael Kurilla, director of the National Center for Advancing Translational Sciences at the National Institutes of Health. The U.S. has a decentralized health care system, he pointed out, meaning that providers can’t easily share data. For example, someone sick with COVID who leaves the hospital too early and re-admits to another facility could easily be counted as two cases, said Kurilla.

Further, the reporting systems themselves are often antiquated, the NIH expert explained.

“Some places are still paper based, some are using fax to transmit information,” he said, adding — only half joking — that it wouldn’t surprise him if some local health agencies still used floppy disks.

On top of technological woes, the many different settings offering COVID-19 shots can compound reporting challenges, explained Phil Chan, medical director for the Rhode Island Department of Health. It’s easy to document doses at state-run vaccination clinics, he said, but vaccinations delivered at doctor’s offices or pharmacies can be harder to track.

“It’s a lot of moving parts,” he said. “The devil’s really in the details.”

When those details are mishandled, inaccuracies in the data arise. In mid-September, the CDC adjusted their report of the share of people 12 and older in West Virginia who had received at least one dose of the coronavirus vaccine from 74 percent to 67 percent, after discovering that they had double-counted certain data streams for over three months.

On the flip side, increased data transparency may well translate into increased accuracy. In Maine — the only U.S. state to publicly report student and staff COVID vaccination data for school districts, according to the University of Washington’s Center on Reinventing Public Education — youth vaccination data reported by the state for each of its 16 counties align closely with federal numbers.

“We need to be transparent, you need to show exactly what you do,” said Mokdad, the UW epidemiologist who spent two decades at the CDC.

He wishes his former employer would be more forthcoming about its raw numbers and any possible shortcomings in its data pipeline. In his own COVID research, Mokdad said, he relies on infection counts from the Johns Hopkins tracker, because he finds it more reliable than the federal numbers.

“There is a big problem at CDC right now,” said the epidemiologist. Mokdad himself was involved in a high-profile incident in 2004 where he co-authored a CDC paper that overestimated the number of annual deaths caused by obesity. The health expert said he left the centers in 2008 for unrelated reasons and on good terms with all his co-workers.

“I criticize CDC because I love CDC,” he said.

---

This article was published in partnership with The 74. Sign up for The 74’s newsletter here.