When Science Is Messy: How SciCheck Handles Scientific Disputes
By Jessica McDonald
What does a fact-checking organization do when the facts are not clear? And what happens when renowned scientists disagree on public health policy?
Throughout the COVID-19 pandemic, there have been numerous examples of times when scientific opinion has been divided, at least in part, typically by a lack of definitive data. The usefulness of masks, for example, especially before it was clear that people without symptoms could spread the coronavirus. Or the need for healthy people to get a booster shot.
The latest example occurred last month, when the US Food and Drug Administration (FDA) authorized a second booster dose for COVID-19 vaccines, or the fourth dose, of the vaccines. from Pfizer/BioNTech and Moderna for people over 50 years of age. The decision was applauded by some experts, but criticized by others. Although the decision was based on evidence, the data was admittedly limited. (For more information, read “Questions and answers about the second booster dose of the COVID-19 vaccine for the elderly”).
At SciCheck, we thought we’d take this opportunity to explain how we approach these situations and explain our process.
First, in these cases, we don’t write standard checks. Those are reserved for statements made by politicians and others on social media that are in any way inaccurate or misleading.
However, given how confusing certain scientific topics can be to people, we believe it is useful to analyze them and provide accurate information about them. In these cases, and others where there is little under discussion but science can be complicated, we write Q&A notes or science background articles to compile what is known about a topic at a given point in time.
Sometimes these items are pretty straightforward. Take for example our pages for each of the three approved or licensed COVID-19 vaccines that we continue to update with new information. We now have a lot of information about vaccines, and there is a very broad and robust consensus among scientists and clinicians that vaccines are safe and effective, and that those who are eligible should get their first doses.
But more often than not, the science isn’t that clear, or we just don’t have enough information at a given point in time to give the kind of answer the public often expects from scientists. “Science,” after all, is not a monolith, solid and unflappable. And aside from the misconceptions people may carry from high school, science is rarely black and white: it’s nuanced and messy and can get complicated.
That’s when SciCheck tries to make it clear what we know and what we don’t; and if it’s about a controversial topic, why do different experts have different opinions, even after looking at the same set of data. Our approach is to describe the underlying data and then give context as to why certain scientists find it convincing or not.
Depending on the topic, we interview the scientists with the most relevant expertise or, if it’s a broader topic, we collect views from various experts, as we did in the article on the second booster for older people. In that case, immunologists, a public health expert, a vaccine expert, and a practicing geriatrician had slightly different opinions, although most of them felt that the decision to allow people over 50 years of age to receive a second dose of booster of mRNA vaccines was reasonable, if not a bit premature and based on relatively sparse data, particularly for younger eligible people.
As we explained, much of the data supporting the authorization comes from Israel, which allowed people over 60 years of age to receive a fourth dose of mRNA beginning in January. Observational studies comparing outcomes in those who chose to get another booster with those who did not, suggested that the extra dose further decreased the risk of severe COVID-19. But Dr. Paul Offit, a pediatrician and vaccine expert at The Children’s Hospital of Philadelphia, didn’t find those studies convincing, since people who chose to get the extra booster might have been more health-conscious. The studies also did not have a long follow-up time and did not include people under 60 years of age. Offit, who sits on the FDA’s vaccine advisory committee, was upset that the agency didn’t hold a meeting with the committee to review the evidence before authorization.
Several other sources were less critical, saying the authorization gave doctors flexibility, which could be valuable. But they recommended that people try to “schedule” their doses for when their individual risk was greatest, in order to save the maximum protection provided by the booster for when it is most needed.
As we and others have noted before, some of the disagreement about booster doses boils down to the goal of vaccination, and whether the goal is to prevent severe disease or whether it should also reduce infections and mild disease. That’s relevant because the neutralizing antibodies that protect against infection wear off over a couple of months, but the broader immune response that prevents severe COVID-19 is much longer lasting. If experts differ on what they want vaccines to do, then they will inevitably have different ideas about when boosters should be implemented.
It is also important to understand that science is not the same as political science. While good science must underpin and inform public health decisions, at the heart of Joe Biden’s promise to “follow the science,” those decisions often need to incorporate practical and psychological considerations about feasibility and how the public will react. and respond to those recommendations. This can further divide experts, who may have very different views of what Americans can handle in terms of communication messages.
The Centers for Disease Control and Prevention and the FDA have at times explained that such considerations have shaped certain policies. The FDA, for example, said it had decided to open booster eligibility to everyone over 50 years of age, and not just those over 60 or those with underlying medical conditions, because “it’s less confusing to people.” That information was delivered near the end of a news conference, during an explanation of why the agency didn’t feel it necessary to bring the decision to its outside vaccine advisory committee.
But all too often, officials fail to make or emphasize such acknowledgments. Which can reduce public confidence in those decisions, especially if there is any evidence that appears to conflict with policy, as happened when the CDC decided in January that people with COVID-19 could end their quarantine after five days. instead of 10, even without a negative result on an antigen test. As we wrote, that controversial recommendation considered practicalities, but it would have been better if officials had been honest about how and why they came to that decision.
The best part of science, and perhaps the most fundamental, is that it is self-correcting and always evolving. More than anything, science is a process, a method to get closer to the truth and to understand our world better.
For that reason, SciCheck also tries to be transparent and remind readers that the science on a given topic can change, an important message given the habit of some of applying today’s knowledge to the past to criticize public health recommendations. Since we published our article on the second boosters, there has already been another Israeli study on the effectiveness of the fourth dose of mRNA published, although this one did not change Offit’s mind.
On the other hand, sometimes inconclusive or unlikely events, such as whether certain drugs are effective against the coronavirus, become less likely over time as evidence against them accumulates, and sharing that information is also relevant. When we first wrote about hydroxychloroquine, a malaria pill that former President Donald Trump touted as a treatment for COVID-19, we explained that there was little data to know if it worked, but we left open the possibility that it might work, and we said that larger randomized controlled clinical trials were needed to know the answer.
For more than a year, we continued to write about hydroxychloroquine, while politicians and others made baseless claims about it, even though more and more studies found that it did not help people with COVID-19. We treat similar statements about the antiparasitic drug ivermectin in the same way. Several large clinical trials have not found the drug beneficial in treating COVID-19, although a few are ongoing.
What the experts do agree on, at least the reputable ones, is that we don’t know what the future holds. As E. John Wherry, an immunologist at the University of Pennsylvania, told us in a recent interview: “If there’s one thing we’ve learned about this pandemic, it’s to be humble about the virus. And to be very, very careful when making predictions.”
This careful approach can be difficult for people who expect concrete statements. But certainly, scientists live by the details and much of science is uncertain. At SciCheck, we will continue to report on these nuanced topics and provide our readers with the best information available at the time.
Translated by Catalina Jaramillo.
Editor’s note: FactCheck.org does not accept advertising. We rely on grants and individual donations from people like you. Please consider a donation. Credit card donations can be made through our “Donate” page. If you prefer to donate by check, send it to: FactCheck.org, Annenberg Public Policy Center, 202 S. 36th St., Philadelphia, PA 19104.