Scientific publishing is becoming more open. Does that mean peer review is less rigorous?

In the early days of the COVID-19 pandemic, researchers flooded journals with studies about the then-novel coronavirus. However, this dramatic increase in the scientific literature highlights a potential shortcoming in the peer review policies of some journals, says Dennis Gorman, PhD, professor at the Texas A&M University School of Public Health.

Gorman recently wrote two pieces focused on potential issues with peer review in public health journals. The first of these is an opinion article in the journal Learned Publishing, where Gorman describes the rise of a prolific public health journal and its publication practices. The second is a study published in the journal Quantitative Science Studies that reviews COVID-19 papers published in the top 15 public health journals as ranked in Google Scholar in 2020.

Peer review is considered a critical part of scientific publishing. In this process, other scholars review scientific manuscripts to ensure they meet ethical guidelines, have sound methodology and are of high quality. Peer review often involves multiple stages and includes several reviewers, so the process can take weeks to complete. However, some journals have taken steps to dramatically shorten the peer review process, which can be attractive for researchers looking to get studies published. One such journal, the International Journal of Environmental Research and Public Health (IJERPH), has taken great strides to streamline the peer review process and keep acceptance rates relatively high.

Some consider this positive, as it makes scientific publishing more open. This has enabled IJERPH to become a leading public health journal, publishing more than 50 times as many public health papers in 2021 as in 2010. In 2022, it published more than 17,000 articles. The journal accomplishes this by using a large staff to ensure speedy reviews and publication. However, there are concerns that a higher acceptance rate and shorter peer review process could contribute to studies of poorer quality being published. This could be problematic, as the volume of work that IJERPH publishes makes it more likely for unsound research to circulate more widely. Clarivate de-listed the journal from the Web of Science in February 2023 due to failure to meet its quality criteria.

In his analysis in Quantitative Science Studies, Gorman dives into COVID-19 studies from the early days of the pandemic. He selected 74 papers from the top 15 public health journals on Google Scholar. Of these, 23 were published in IJERPH, with five of those studies having more than 500 citations on Google Scholar. The median peer review length for all 74 papers was 21.5 days, but it was a little shorter for IJERPH at 18 days.

Gorman found that the methodological quality of those 23 papers was generally poor, with 14 of the 23 using cross-sectional surveys that relied on convenience samples rather than random sampling. Other examples of unsound studies include narrative reviews without details on studies reviewed or how they were identified and an online study with a large amount of missing data.

“These methodological shortcomings limit the kinds of conclusions that can be drawn from such papers, yet some of these studies have been cited many times,” Gorman said. “This increases the chances of poorly informed public health policy and recommendations, and could be particularly harmful during a health crisis such as a pandemic.”