Insights on the Replication Crisis in Scholarly Publishing from Ewoud Compeer, Immunologist, University of Oxford

This episode of the Against the Grain podcast, published on Jul 26, 2021, features a conversation with Ewoud Compeer of Oxford University. The interview was conducted by Matthew Ismail, Editor in Chief of the Charleston Briefings and Conference Director at the Charleston Conference.
Insights on the Replication Crisis in Scholarly Publishing from Ewoud Compeer, Immunologist, University of Oxford

In this podcast, Matthew Ismail talks to immunologist Ewoud Compeer of the University of Oxford about the reproducibility crisis and how Open Science and open access can help to enhance the reproducibility of research and restore public trust in science at a time when the pandemic has made trust in science very important.

Ewoud Compeer is a Dutch biomedical scientist and immunologist.  He has held research positions in the United States, the Netherlands, and Australia. Presently he is based in the United Kingdom (UK) at the University of Oxford. In the domain of immune system science, his research revolves around how immune cells communicate with one another and what factors that involves.  

Ewoud is keen to explore how the research environment in medical sciences can be improved by becoming more inclusive, open and transparent. His belief is that the environment in which research data and study designs are not available along with published research results is no longer tenable. According to Compeer, this lack of transparency is at the core of the reproducibility crisis. This refers to the phenomenon that the experimental, or other empirical, setups upon which published research is based cannot necessarily be reproduced by other researchers.  The lack of reproducibility can cast doubt on research findings. This also likely raises a broader concern about the robustness of published science. Ewoud believes that one needs to create an environment of Open Access to finished publications and empirical data both to make research more rigorous and to promote confidence that science can be trusted.

Ewoud founded a not-for-profit organization for the dissemination of science and scientific exchanges between the Netherlands and the UK.  He also is an eLife ambassador advocating for Open Science around the world.  On the background of the global pandemic, the present period invites this talk about the reproducibility crisis in science.

Key Points:

  • Reproducibility Crisis is “the ability to reproduce data, by yourself or by others, if you have the same sample set and the same analysis workflow,” as opposed to an attempt at replication in which one “tries to obtain similar results but with a totally different experimental setup.” 
  • Compeer says that there is a crisis of people not trusting science, such as with regard to individuals not trusting the science enough to get vaccinated for COVID-19, and that one needs to reform how research results are shared, to help improve the public’s confidence in science. 
  • Reproducibility is not easy even with access to the data. Many factors can change from one part of a clinical experiment to another, such as the subjects changing their diets in the middle of the collection of data.
  • Compeer contends that hiding behind the arguable complexity of medical science can lead to the dismissal of the concerns about reproducibility. Yet, he states that empirical study design and the correct and effective use of statistics are extremely important for determining the outcomes of research. Thus, these can be externally evaluated, in spite of the complexity that health and biology research involving human beings entails. If there is not enough information in the articles describing the materials and methods used in the underlying research procedures, that can lead to difficulty in reproducing it.
  • According to Compeer, education is profoundly important for science: the new generation of scientists ought to have a sophisticated understanding of mathematics, in order to use empirical research methods, such as statistics, appropriately in designing and carrying out their research so that others can also reproduce their work with the accompanying data.
  • Compeer suggests there is a need for a universal emphasis on Open Science and Open Access as preconditions for promoting this environment of reproducibility.
  • Compeer remarks that, whereas published research is expected to be correct, one of the most important factors in science is time constraints, which encourage the publication of preliminary results. These, however, can be found to be better or worse after further research is conducted. As on the boundaries of knowledge there are unknown research findings, domains and phenomena, one cannot necessarily control for these via empirical study design means.
  • Improvement in technology transforms what it is possible to do in your research as well.
  • At present, the peer review system is completely overburdened and in need of change. Open Science is important but so is training a new generation of early career researchers to do peer review effectively.
  • Compeer says that there are not enough peer reviewers to evaluate the great variety of research and statistical methods.
  • Ismail remarks that, in his blog post titled "What's Wrong with Social Science and How to Fix It" and published on September 11, 2020,  Alvaro de Menard, a statistician by training, scanned the methodology and abstract of circa 2,500 articles in the social sciences. The author of this blog post has claimed that he could comfortably predict whether or not the research is reproducible simply by examining the research design and the (often incompetent) use of statistics. Where will we find peer reviewers who can evaluate the various research effectively?
  • Compeer says that artificial intelligence can help with preliminary screening of research and statistics, such as whether the sample size is big enough.
  • Compeer remarks that he has made improvements in data visualization in his published papers that are very helpful and can help when evaluating the work of others.
  • As this podcast episode indicates, publishers may also need to change their behavior, such as by shifting away from their focus on primarily publishing studies showing positive results, while re-focusing on storytelling, e.g., covering what makes science and its results interesting rather than what makes science reliable. There is also a need to de-emphasize the Impact Factor as a main criterion for establishing the impact or importance of a piece of published science, in favor of embracing Open Access.
  • Compeer also says that, besides having a larger core of trained statisticians to help evaluate research, there is a tremendous need for science communicators to make science accessible to non-scientists, whether policymakers or the general public. 

Written by Mathew Ismail

Edited by Pablo Markin


Featured Image Credits: Coronavirus, March 20, 2020 | © Courtesy of Yuri Samoilov/Flickr.