What Do You See?
What is this? |
|
But faced with this, what do you see?
 |
Some will say two faces and others a vase. |
Scientific data can be like both examples. Sometimes everyone is in close agreement because there is a lot of evidence for the outcome, like the boat picture. But on other occasions the data is like the face and vase picture and different people see different things. It is generally not malicious or intended to mislead. It is just that people’s backgrounds, experience, and current thinking contribute to their interpretation of the data. As clinical data accumulates over time the interpretations also change because science and medicine are often dynamic.
You may hear someone say “I don’t know who to believe” about an aspect of the COVID-19 pandemic. They may be saying this because they hear conflicting information and it causes them to doubt the sources. They may see two people referring to the same study and drawing different conclusions.
Here are some of the reasons this happens.
- The novel coronavirus that causes COVID-19 is still very new, and there is a lot that public health and medical experts still do not know. Most scientists are acting in good faith and drawing the most reasonable conclusions they can based on what they know now. The recommendations they make will change over time as they learn more.
- There will be errors in the data. Most of these errors are not because people are trying to create a false narrative, but because of timing, or faulty reporting systems, or simple human error. It is important to look at the overall trend in findings rather than trying to verify every single number.
- Public health officials typically track many data points to evaluate the situation. There is no one single data point that can be used to assess our progress in bringing the spread of disease under control. It is important to consider reported numbers, along with the trends in those numbers over time, and how those numbers influence, or are influenced by, other factors (such as the relationship between the number of cases and the number of hospitalizations).
- There will be disagreement among experts about how to interpret data. Scientists performing studies publish the results of their studies along with the methods they used to conduct the studies so that other scientists can evaluate their work. Studies subjected to rigorous peer review and published in respected medical journals are likely to present the best of current thinking. Over time, consensus emerges, but it does take time. Scientists all over the world are working hard to figure out this virus.
Levels of Evidence
The medical community use levels of evidence in order to practice evidence-based medicine. These levels of evidence can be summarized into five categories, graded A-D:
- Meta-analysis1 of multiple randomized clinical trials2 (A)
- A single well-conducted randomized clinical trial where the new drug is compared to placebo (an inactive treatment), or the best available treatment, for the condition that’s being studied (A)
- Cohort3 or case-controlled4 studies (B)
- Case series5, no control (C)
- Expert opinion which could be based on animal or in-vitro data (D)
Grade A provides strong evidence for how to treat patients and is what is required to register a drug with FDA and market it. B-Grade data can lead to recommendations, while C and D only provide options that the treating physician and patient must weigh up in terms of potential benefit vs possible harm.
An example: Hydroxychloroquine
The first evidence that hydroxychloroqine might work against SARS-CoV-2 came from test tube data. This is D-level data but led to many early assertions that this drug worked in Covid-19. A study published in The Lancet in May reported that hydroxychloroquine actually resulted in an increased number of deaths in hospitalized patients; this paper was later retracted when The Lancet was unable to complete a peer review process. A randomized placebo-controlled study aimed at evaluating whether hydroxychloroquine could prevent those exposed to the virus from becoming sick, published in the New England Journal of Medicine found no statistical difference between subjects who received the drug and those who received a placebo; this is A-level data because of the study method.
Most data on hydroxychloroquine comes from small trials or case series. A meta-analysis of 24 such studies published in the Annals of Internal Medicine concluded there was “insufficient and often conflicting evidence on the benefits and harms of using hydroxychloroquine or chloroquine to treat COVID-19.” This is high level data which does not support the use of hydroxychloroquine.
This example demonstrates why things that you read can be confusing. While there may be a small number of people who promote false conclusions for their own benefit, the vast majority of the scientific community is made up of people who are working in good faith and working to provide the public with the best data possible.
So how to evaluate the data I see or hear?
Here are a few suggestions to help you separate fact from opinion in the information you are hearing or reading:
- Does the information come from a non-partisan source such as a teaching hospital or a public health expert? Even if it appears so, look further to see if he or she is respected and supported by their peers or are they trying to distance themselves from that person.
- Is the source clearly referenced in the article? Are you able to trace the information back to its original source? In the Lancet example they could not trace the data during peer review.
- Has the source document been reviewed by peers or experts before it was published or disseminated? In the rush to publish data about COVID-19 many manuscripts are being published before peer review is completed.
- Why has it not been published? You can reasonably expect that anyone who claims verbally to have successfully treated many patients with a drug for Covid-19, would publish their results in a peer reviewed journal. In the absence of a published article the data must remain suspect.
- Is there any apparent profit or gain to be made by the author in taking a certain perspective or opinion? This is why pharmaceutical companies rarely author publications of their own trials but ask the physicians who participated in the trials to do so in order for there not to be conflict of interest.
- Is it an opinion of the author or does the author reference other studies or evidence supporting the article? Opinion alone is the lowest level of data.
While we wait for treatments for Covid-19 the best way for you to contribute to bringing the virus under control is to practice the 3 W’s: Wear a face covering when in close proximity to others, Wait 6 feet apart and Wash or sanitize your hands frequently. These guidelines are based on guidance published by the CDC. https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/index.html
Footnotes
1 examination of data from a number of similar independent studies
2 a clinical trial where a group of similar patients are assigned treatment by chance, like tossing a coin, to minimize any bias
3 an observational study where a group of patients are followed prospectively or retrospectively. There is no intervention as part of the study.
4 The group of patients who are followed are matched with respect to characteristics like age and sex, to a group of healthy people
5 a case-series is a number of patients with a similar diagnosis and the author reports on his/her experience treating them