Interpreting Scientific Evidence for the Non-Expert

Every now and then, a major news publication will post a story with a headline like, “HIV can be completely eliminated in laboratory using gene editing tool – study,” or ‘Amazing’ trial shows drug combination stops lung cancer advancing for longer.” These types of headlines tend to grab our attention. After all, who wouldn’t be interested in a cure for HIV, or a better treatment for a common cancer? 

If you proceed to read one of these articles, the author is often a non-expert and may be unable to give an accurate critical appraisal of the research. Higher quality publications will typically provide an assessment of the study’s relative strengths and weaknesses and are often written by journalists with science backgrounds. But when a headline is of particular relevance to you or a loved one, you might find yourself wanting to go straight to the source–the peer-reviewed publication.

I was inspired to write this blog because we recently received a question about a clinical trial for the efficacy of a popular allergy nasal spray (azelastine) in preventing COVID-19 infection over at Those Nerdy Girls. I am co-authoring a post for them that will go into detail about the study itself, but I also wanted to address the wider challenge that members of the public face when they want to dig into the scientific literature.

I believe that each and every one of us can and should be able to read a peer-reviewed scientific paper, especially those that relate to our health. But scientific manuscripts can be intimidating and confusing, and the U.S. education system is not particularly effective in teaching the scientific method. In addition, news media often exaggerate results or gloss over problematic aspects of studies. But there are some tips and tricks that can make evaluating the quality of a peer-reviewed paper less challenging:

1. Look up the journal’s impact factor. Impact factors measure the significance of the journal to its field by comparing how often their manuscripts are cited to the number of publications per year. While you should never judge a paper solely by the impact factor of the journal it is published in, a high impact factor indicates that researchers who work in that field are more likely to take notice of it.  

2. Determine what type of study was conducted. Not all studies provide the same quality of evidence. Randomized controlled trials (RCTs), in which participants are randomly assigned to either a control (placebo) or treatment group, are considered the gold standard because the researchers have a high degree of control over the experiment and both researchers and participants are typically blinded. In theory, this should reduce the possibility of outside influences biasing the results. 

   
   

But when it comes to the study of human health and disease, it is often impossible to do an RCT for ethical or logistical reasons. Instead, researchers will perform an observational study, or a “natural experiment.” These studies can be very high quality, but the researchers must use certain statistical techniques to account for outside influences on the results, known as confounders, or the results may be biased.

3. Look at the size and diversity of the study sample. The azelastine trial published in JAMA Internal Medicine was conducted at a single location (single-center as opposed to multi-center) with a population that was disproportionately female and caucasian. This means that the results may not be generalizable to all persons living in all regions. Always consider the generalizability of the study results when evaluating a scientific publication.

4. Read p-values and confidence intervals critically. I have covered the basics of p-values and confidence intervals in an earlier post. When evaluating a study, look at the width of the confidence intervals as well as the p-values. If a study does not report confidence intervals, it may be a red flag, because it suggests a lack of transparency in communicating uncertainty. 

5. Read the disclosures of conflicts of interest. In the JAMA study, the clinical trial was funded by a company that produces azelastine spray. This isn’t necessarily a major red flag–after all, pharmaceutical companies have to test their products, and if they suspect a new off-label use for one, they are performing a public service by investigating it. But a long list of potential conflicts of interests for one or more of the authors should be viewed with caution. 

6. Do a ctrl + f for the word “limitations.” Sometimes you will find “limitations” under its own sub-heading, but limitations are typically addressed in the discussion section regardless. Reading the limitations is a really good shortcut because the researchers will explain directly which part/s of their study they have the least confidence in and why. 

7. Pay attention to spelling and grammar. A typo here and there is to be expected, but if the manuscript is riddled with errors, it suggests that the journal is of poor quality. A high-quality publication will ensure that manuscripts are thoroughly edited for grammar, spelling, and syntax before they are published. Substantial grammatical errors can even be an indication that the journal is predatory. 

8. Bonus - Look for sensitivity analyses. This is for those who are a bit more advanced in their understanding of statistical analysis. Sensitivity analyses are mathematical tests that help determine how much the study results would change if certain underlying assumptions were altered. This helps the researchers understand which variables impact the results the most, and it can also help them judge how confident they can be in the results. If sensitivity analysis reveals that small changes to these assumptions would cause the results to change drastically, then the researchers should be less confident that their results reflect reality. 

To circle back to the JAMA azelastine study–after using these principles to evaluate the study myself, I decided to buy some generic azelastine spray. I have nasal allergies that bother me during the autumn and winter months, so I’m simply switching from a different allergy medicine to azelastine. Would I recommend everyone start using it? No, because there are several limitations to the study, and experts have cautioned against making any public health recommendations before further trials can be conducted. I especially would not recommend anyone to use azelastine spray as an alternative to masking, which we know for certain prevents the spread of COVID-19. But since I take allergy medicine anyway, I feel that the risk/benefit ratio makes sense for me. 

Applying these tips does require a degree of familiarity with the structure and language of biomedical science, but with practice you can become a better judge of the quality of scientific evidence. This is a powerful tool to have, and it will help you to make decisions about your health with more confidence.