Cancer researchers overestimate reproducibility of preclinical studies

Cancer scientists overestimate the extent to which high-profile preclinical studies can be successfully replicated, new research from McGill University suggests. Thes findings, published in PLOS Biology by Jonathan Kimmelman and colleagues from McGill, are based on a survey in which both experts and novices were asked to predict whether mouse experiments in six prominent preclinical cancer studies conducted by the Reproducibility Project: Cancer Biology (RP:CB) would reproduce the effects observed in original studies.

On average, the researchers forecasted a 75% probability of replicating statistical significance, and a 50% probability of reproducing the same size effect as in the original study. Yet according to these criteria, none of the six studies already completed by the Reproducibility Project showed the same results previously reported.

One possible explanation for the optimism is that cancer scientists overestimate the replicability of major reports in their field. Another is that they underestimate the logistical and methodological complexity of independent laboratories repeating these techniques.

Reproducibility crisis
The work follows on numerous reports exploring biomedicine's so-called reproducibility crisis. In the last 10 or 15 years, there have been mounting concerns that some of the techniques and practices used in biomedical research lead to inaccurate assessments of a drug's clinical promise.

Given that not all studies reproduce, Kimmelman and his team wondered if cancer experts could at least sniff out which studies would not easily replicate. The finding that cancer researchers' ability to do so "was really limited" suggests that there may be inefficiencies in the process by which science "self-corrects."

There is however strong community concern that, due to process-related issues and potential methodological differences, the replication studies themselves may not be an entirely reliable measure of replication outcome. Kimmelman emphasizes that the findings don't indicate that scientists who participated in the study don't understand what's going on their field -- nor does it diminish the importance of funding research and making policy on the basis of scientific consensus. Some scientists were highly accurate in their predictions, and participants were new to forecasting, which is difficult.

Training could be part of the solution
The results do, however, raise the possibility that training might help many scientists overcome certain cognitive biases that affect their interpretation of scientific reports.

"If the research community believes a finding to be reliable, it might start building on that finding only to later discover the foundations are rotten. If scientists suspect a claim to be spurious, they are more likely to test that claim directly before building on it."

"This is the first study of its type, but it warrants further investigation to understand how scientists interpret major reports," Kimmelman says. "I think there is probably good reason to think that some of the problems we have in science are not because people are sloppy at the bench, but because there is room for improvement in the way they interpret findings."

Benjamin D, Mandel DR, Kimmelman J.
Can cancer researchers accurately judge whether preclinical reports will reproduce?
PLoS Biol. 2017 Jun 29;15(6):e2002212. doi: 10.1371/journal.pbio.2002212.

Most Popular Now

Top 20 breaking World Pharma News of 2017

We are proud to announce the top 20 most popular breaking World Pharma News from 2017. Have a wonderful 2018 new(s) year filled with health, happiness, and spectacular su...

Novartis advances head-to-head superiority trials …

Novartis announced today the initiation of SURPASS, a head-to-head clinical trial of Cosentyx® (secukinumab) versus proposed biosimilar adalimumab** in ankylosing spondyl...

Diabetes drug 'significantly reverses memory loss…

A drug developed for diabetes could be used to treat Alzheimer's after scientists found it "significantly reversed memory loss" in mice through a triple method of action...

Sandoz regulatory submission for proposed biosimil…

Sandoz, a Novartis division and the global leader in biosimilar medicines, announced today that the US Food and Drug Administration (FDA) has accepted its Biologics Licen...

Zika remains a research and public health challeng…

Since 2016, when Zika was declared by the World Health Organization as a public health emergency of international concern, the virus has become established in more than 8...

Try exercise to improve memory, thinking

For patients with mild cognitive impairment, don't be surprised if your health care provider prescribes exercise rather than medication. A new guideline for medical pract...

Dirt-dwelling microbe produces potential anti-mela…

A type of soil-dwelling bacterium produces molecules that induce death in melanoma cells, research at Oregon State University shows. The molecule is a secondary metabolit...

Novartis Kisqali® received FDA Breakthrough Therap…

Novartis today announced Kisqali® (ribociclib) received US Food and Drug Administration (FDA) Breakthrough Therapy designation for initial endocrine-based treatment of pr...

FDA approves first treatment for breast cancer wit…

The U.S. Food and Drug Administration today expanded the approved use of Lynparza (olaparib tablets) to include the treatment of patients with certain types of breast can...

New research paper shows that AstraZeneca has achi…

In a research paper published today by Nature Reviews Drug Discovery(1), AstraZeneca's IMED Biotech Unit documents a more than four-fold improvement in R&D productivity f...

Berry gives boost to cervical cancer therapy

According to the Centers for Disease Control and Prevention, approximately 12,000 women in the United States are diagnosed with cervical cancer each year. One of the most...

Novartis appoints Elizabeth Barrett as Oncology He…

Novartis announced today that Elizabeth (Liz) Barrett, currently Global President Oncology at Pfizer, Inc., has been appointed CEO Novartis Oncology and a member of the E...

Pharmaceutical Companies

[ A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z ]