Cancer researchers overestimate reproducibility of preclinical studies

Cancer scientists overestimate the extent to which high-profile preclinical studies can be successfully replicated, new research from McGill University suggests. Thes findings, published in PLOS Biology by Jonathan Kimmelman and colleagues from McGill, are based on a survey in which both experts and novices were asked to predict whether mouse experiments in six prominent preclinical cancer studies conducted by the Reproducibility Project: Cancer Biology (RP:CB) would reproduce the effects observed in original studies.

On average, the researchers forecasted a 75% probability of replicating statistical significance, and a 50% probability of reproducing the same size effect as in the original study. Yet according to these criteria, none of the six studies already completed by the Reproducibility Project showed the same results previously reported.

One possible explanation for the optimism is that cancer scientists overestimate the replicability of major reports in their field. Another is that they underestimate the logistical and methodological complexity of independent laboratories repeating these techniques.

Reproducibility crisis
The work follows on numerous reports exploring biomedicine's so-called reproducibility crisis. In the last 10 or 15 years, there have been mounting concerns that some of the techniques and practices used in biomedical research lead to inaccurate assessments of a drug's clinical promise.

Given that not all studies reproduce, Kimmelman and his team wondered if cancer experts could at least sniff out which studies would not easily replicate. The finding that cancer researchers' ability to do so "was really limited" suggests that there may be inefficiencies in the process by which science "self-corrects."

There is however strong community concern that, due to process-related issues and potential methodological differences, the replication studies themselves may not be an entirely reliable measure of replication outcome. Kimmelman emphasizes that the findings don't indicate that scientists who participated in the study don't understand what's going on their field -- nor does it diminish the importance of funding research and making policy on the basis of scientific consensus. Some scientists were highly accurate in their predictions, and participants were new to forecasting, which is difficult.

Training could be part of the solution
The results do, however, raise the possibility that training might help many scientists overcome certain cognitive biases that affect their interpretation of scientific reports.

"If the research community believes a finding to be reliable, it might start building on that finding only to later discover the foundations are rotten. If scientists suspect a claim to be spurious, they are more likely to test that claim directly before building on it."

"This is the first study of its type, but it warrants further investigation to understand how scientists interpret major reports," Kimmelman says. "I think there is probably good reason to think that some of the problems we have in science are not because people are sloppy at the bench, but because there is room for improvement in the way they interpret findings."

Benjamin D, Mandel DR, Kimmelman J.
Can cancer researchers accurately judge whether preclinical reports will reproduce?
PLoS Biol. 2017 Jun 29;15(6):e2002212. doi: 10.1371/journal.pbio.2002212.

Most Popular Now

GSK reaches agreement with Novartis to acquire ful…

GlaxoSmithKline plc (LSE/NYSE: GSK) today announces that it has reached an agreement with Novartis for the buyout of Novartis' 36.5% stake in their Consumer Healthcare Jo...

Canadian neuroscientists say daily ibuprofen can p…

A Vancouver-based research team led by Canada's most cited neuroscientist, Dr. Patrick McGeer, has successfully carried out studies suggesting that, if started early enou...

First proof a synthesized antibiotic is capable of…

A "game changing" new antibiotic which is capable of killing superbugs has been successfully synthesised and used to treat an infection for the first time - and could lea...

Merck partners with Medisafe to help improve medic…

Merck, a leading science and technology company, today announced a new collaboration with US-based Medisafe to help its cardiometabolic patients better manage medication ...

Phase III data in The Lancet show Novartis siponim…

Novartis today announced that the full results from the Phase III EXPAND study of oral, once-daily siponimod (BAF312) in secondary progressive multiple sclerosis (SPMS) w...

Taking a standard prostate cancer drug with food b…

By taking a high-cost drug with a low-fat meal - instead of on an empty stomach, as prescribed - prostate cancer patients could decrease their daily dose, prevent digesti...

North and south cooperation to combat tuberculosis

Tuberculosis can be cured and could be eradicated. For this to happen, however, patients have to receive the right treatment. Researchers at the Makerere University and t...

Boehringer Ingelheim and OSE Immunotherapeutics an…

Boehringer Ingelheim and OSE Immunotherapeutics, a biotechnology company focused on the development of innovative immunotherapies, have announced a collaboration and excl...

New immunotherapy for lung cancer shows promise of…

In a groundbreaking development, results from a recent clinical trial to treat lung cancer show that a novel immunotherapy combination is surprisingly effective at contro...

Personalized tumor vaccine shows promise in pilot …

A new type of cancer vaccine has yielded promising results in an initial clinical trial conducted at the Perelman School of Medicine at the University of Pennsylvania and...

Lokelma approved in the EU for the treatment of ad…

AstraZeneca today announced that the European Commission has granted marketing authorisation for Lokelma (formerly ZS-9, sodium zirconium cyclosilicate) for the treatment...

New targeted therapy schedule could keep melanoma …

Skin melanoma, a particularly insidious cancer, accounts for the vast majority skin cancer deaths and is one of the most common cancers in people under 30. Treatment for ...