Cancer researchers overestimate reproducibility of preclinical studies

Cancer scientists overestimate the extent to which high-profile preclinical studies can be successfully replicated, new research from McGill University suggests. Thes findings, published in PLOS Biology by Jonathan Kimmelman and colleagues from McGill, are based on a survey in which both experts and novices were asked to predict whether mouse experiments in six prominent preclinical cancer studies conducted by the Reproducibility Project: Cancer Biology (RP:CB) would reproduce the effects observed in original studies.

On average, the researchers forecasted a 75% probability of replicating statistical significance, and a 50% probability of reproducing the same size effect as in the original study. Yet according to these criteria, none of the six studies already completed by the Reproducibility Project showed the same results previously reported.

One possible explanation for the optimism is that cancer scientists overestimate the replicability of major reports in their field. Another is that they underestimate the logistical and methodological complexity of independent laboratories repeating these techniques.

Reproducibility crisis
The work follows on numerous reports exploring biomedicine's so-called reproducibility crisis. In the last 10 or 15 years, there have been mounting concerns that some of the techniques and practices used in biomedical research lead to inaccurate assessments of a drug's clinical promise.

Given that not all studies reproduce, Kimmelman and his team wondered if cancer experts could at least sniff out which studies would not easily replicate. The finding that cancer researchers' ability to do so "was really limited" suggests that there may be inefficiencies in the process by which science "self-corrects."

There is however strong community concern that, due to process-related issues and potential methodological differences, the replication studies themselves may not be an entirely reliable measure of replication outcome. Kimmelman emphasizes that the findings don't indicate that scientists who participated in the study don't understand what's going on their field -- nor does it diminish the importance of funding research and making policy on the basis of scientific consensus. Some scientists were highly accurate in their predictions, and participants were new to forecasting, which is difficult.

Training could be part of the solution
The results do, however, raise the possibility that training might help many scientists overcome certain cognitive biases that affect their interpretation of scientific reports.

"If the research community believes a finding to be reliable, it might start building on that finding only to later discover the foundations are rotten. If scientists suspect a claim to be spurious, they are more likely to test that claim directly before building on it."

"This is the first study of its type, but it warrants further investigation to understand how scientists interpret major reports," Kimmelman says. "I think there is probably good reason to think that some of the problems we have in science are not because people are sloppy at the bench, but because there is room for improvement in the way they interpret findings."

Benjamin D, Mandel DR, Kimmelman J.
Can cancer researchers accurately judge whether preclinical reports will reproduce?
PLoS Biol. 2017 Jun 29;15(6):e2002212. doi: 10.1371/journal.pbio.2002212.

Most Popular Now

Novartis confirms 5 year data for first and only f…

Novartis, a global leader in Immunology & Dermatology, confirmed today positive 5 year efficacy and safety results for Cosentyx® from a Phase III long-term extension stud...

Researchers develop microneedle patch for flu vacc…

A National Institutes of Health-funded study led by a team at the Georgia Institute of Technology and Emory University has shown that an influenza vaccine can produce rob...

Jardiance® (empagliflozin) analysis reinforces est…

An analysis of pooled safety data from 15 studies plus 4 extension studies involving more than 12,500 adults with type 2 diabetes demonstrated treatment with Jardiance® (...

Alzheimer's and Parkinson's spurred by same enzyme

Alzheimer's disease and Parkinson's disease are not the same. They affect different regions of the brain and have distinct genetic and environmental risk factors. But at ...

Researchers publish new findings on influence of h…

Poor diet is associated with 80% of colorectal cancer cases, but the exact pathways by which diet leads to cancer are not known. In a newly published study, Cleveland Cli...

Merck awards €1.25 million to research projects th…

Merck, a leading science and technology company, today announced its commitment to award €1.25 million to research projects in the field of fertility, supporting the adva...

FDA tackles drug competition to improve patient ac…

Today, the U.S. Food and Drug Administration is taking two new, important steps to increase competition in the market for prescription drugs and facilitate entry of lower...

Trials show unique stem cells a potential asthma t…

A study led by scientists at Monash University has shown that a new therapy developed through stem cell technology holds promise as a treatment for chronic asthma. The Mo...

Roche acquires mySugr to form a leading open platf…

Roche (SIX: RO, ROG; OTCQX: RHHBY) and mySugr have signed an agreement under which Roche acquired all shares of mySugr GmbH. Counting more than one million users globally...

Older Americans don't get - or seek - enough help …

The majority of Americans over age 50 take two or more prescription medicines to prevent or treat health problems, and many of them say the cost weighs on their budget, a...

Novartis CAR-T cell therapy CTL019 unanimously (10…

Novartis announced today that the US Food and Drug Administration (FDA) Oncologic Drugs Advisory Committee (ODAC) unanimously (10-0) recommended approval of CTL019 (tisag...

New brain cancer drug targets revealed

Researchers from Case Western Reserve University School of Medicine and The Cleveland Clinic designed a way to screen brain tumor cells and identify potential drug target...

Pharmaceutical Companies

[ A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z ]