Cancer researchers overestimate reproducibility of preclinical studies

Cancer scientists overestimate the extent to which high-profile preclinical studies can be successfully replicated, new research from McGill University suggests. Thes findings, published in PLOS Biology by Jonathan Kimmelman and colleagues from McGill, are based on a survey in which both experts and novices were asked to predict whether mouse experiments in six prominent preclinical cancer studies conducted by the Reproducibility Project: Cancer Biology (RP:CB) would reproduce the effects observed in original studies.

On average, the researchers forecasted a 75% probability of replicating statistical significance, and a 50% probability of reproducing the same size effect as in the original study. Yet according to these criteria, none of the six studies already completed by the Reproducibility Project showed the same results previously reported.

One possible explanation for the optimism is that cancer scientists overestimate the replicability of major reports in their field. Another is that they underestimate the logistical and methodological complexity of independent laboratories repeating these techniques.

Reproducibility crisis
The work follows on numerous reports exploring biomedicine's so-called reproducibility crisis. In the last 10 or 15 years, there have been mounting concerns that some of the techniques and practices used in biomedical research lead to inaccurate assessments of a drug's clinical promise.

Given that not all studies reproduce, Kimmelman and his team wondered if cancer experts could at least sniff out which studies would not easily replicate. The finding that cancer researchers' ability to do so "was really limited" suggests that there may be inefficiencies in the process by which science "self-corrects."

There is however strong community concern that, due to process-related issues and potential methodological differences, the replication studies themselves may not be an entirely reliable measure of replication outcome. Kimmelman emphasizes that the findings don't indicate that scientists who participated in the study don't understand what's going on their field -- nor does it diminish the importance of funding research and making policy on the basis of scientific consensus. Some scientists were highly accurate in their predictions, and participants were new to forecasting, which is difficult.

Training could be part of the solution
The results do, however, raise the possibility that training might help many scientists overcome certain cognitive biases that affect their interpretation of scientific reports.

"If the research community believes a finding to be reliable, it might start building on that finding only to later discover the foundations are rotten. If scientists suspect a claim to be spurious, they are more likely to test that claim directly before building on it."

"This is the first study of its type, but it warrants further investigation to understand how scientists interpret major reports," Kimmelman says. "I think there is probably good reason to think that some of the problems we have in science are not because people are sloppy at the bench, but because there is room for improvement in the way they interpret findings."

Benjamin D, Mandel DR, Kimmelman J.
Can cancer researchers accurately judge whether preclinical reports will reproduce?
PLoS Biol. 2017 Jun 29;15(6):e2002212. doi: 10.1371/journal.pbio.2002212.

Most Popular Now

Fasenra (benralizumab) receives US FDA approval fo…

AstraZeneca and its global biologics research and development arm, MedImmune, announced that the US Food and Drug Administration (FDA) has approved Fasenra (benralizumab)...

A great place to do great things: Developing game-…

Science has spoken: Abbott (NYSE: ABT) is, again, among the best science-based companies to work for in the world. For the 14th year, the journal Science today recognized...

Alzheimer's disease might be a 'whole body' proble…

Alzheimer's disease, the leading cause of dementia, has long been assumed to originate in the brain. But research from the University of British Columbia and Chinese scie...

Cancer cells destroyed with dinosaur extinction me…

Cancer cells can be targeted and destroyed with the metal from the asteroid that caused the extinction of the dinosaurs, according to new research by an international col...

Novartis confirms leadership in multiple sclerosis…

Novartis today announced it will present 54 scientific abstracts from across its multiple sclerosis (MS) research portfolio at the 7th Joint European and Americas Committ...

Amgen and Novartis announce expanded collaboration…

Amgen (NASDAQ:AMGN) and Novartis announced an expanded collaboration with the Banner Alzheimer's Institute (BAI) to initiate a new trial - the Alzheimer's Prevention Init...

Transplanted hematopoietic stem cells reverse dama…

Researchers at University of California San Diego School of Medicine report that a single infusion of wildtype hematopoietic stem and progenitor cells (HSPCs) into a mous...

Novartis announces the planned acquisition of Adva…

Novartis announced today, that it has entered a memorandum of understanding with Advanced Accelerator Applications (AAA) under which Novartis intends to commence a tender...

'Precision Medicine' may not always be so precise

Precision Medicine in oncology, where genetic testing is used to determine the best drugs to treat cancer patients, is not always so precise when applied to some of the w...

New tissue-engineered blood vessel replacements on…

Researchers at the University of Minnesota have created a new lab-grown blood vessel replacement that is composed completely of biological materials, but surprisingly doe...

China's out of control 'silent killer' affects one…

More than one-third of adults in China have high blood pressure - often dubbed the "silent killer" for its lack of symptoms - but only about one in 20 have the condition ...

New US study reveals key reasons why millions of p…

Few of the more than 90 million Americans(1) with obesity are seeking and receiving long-term obesity care, according to new data from the Awareness, Care and Treatment I...

Pharmaceutical Companies

[ A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z ]