Non-reproducible results/experiments - (Mar/14/2013 )

I keep hearing PhD students venting about their inability to repeat experiments and get the same outcome based on the methods provided by published papers from another lab or even protocols produced by someone in the lab (same concentration, same cell lines, same conditions etc). Is this common?

I've even heard people getting strangely opposite results (e.g. published as upregulation but instead we see downregulation)

Then it comes down to whether the published protocols are legitimate or it's purely our own fault when repeating someone else's technique?

Dear Science noob,

This is an extremely common problem that I have seen over the years. You comment:-

"methods provided in published papers" .............. this is a big problem in that this part of a paper should probably be the largest section but is commonly just cut to a bear minimum. There are lots of "tricks" that are left out of this section that are essential for thew experiment to be reproduced.

"Same concentration, same cell lines, same conditions"................every cell paper I read states " the cells were grown in a humidified CO2 Incubator at 5% CO2. How many researchers routinely check for humidifiaction AND FYRITE THEIR INCUBATORS TO CHECK PRECISELY THE INTERNAL CO2 CONCENTRATION IN THEIR INCUBATORS.

ARE THE INCUBATORS DIRECT HEAT OR FAN ASSISTED?

WHERE DOE YOUR FOETAL CALF SERUM COME FROM?.......there are massive differences between the qualitof serum sourced for the many countries of the world that produce it.

HOW MANY RESEARCHERS HAVE AUTHENTICATED THEIR CELLS BEFORE USING THEM?

HOW MANY RESEARCHERS ARE WORKING WITH MYCOPLASMA INFECTED CELLS BECAUSE THEY DO NOT TEST THEM?


Does this start to answer your question about reproducibility?

Kindest regards

Uncle Rhombus

^^^ What he said

Hm, and what if you can't reproduce well-established results (methods used in many publications and many labs) ? What do you think about that ? More likely to be one's own fault then, isn't it ?

LIES AND DAMN STATISTICS!!!!


WHen will biologists learn to do proper statistics? I don't doubt at all that many published findings are not repeatable, and it's simply the fact that many biologists don't know how to properly do statistics. I don't think it is out of anything nefarious (data manipulation etc.)

Ask yourself, what does statistical significance mean? If you can't answer that properly, go back and look back at a basic biostats book. Statistical significance does NOT mean the following:

-that your results are reproducible (in fact statistically significant findings may only be reproducible 50% of the time)
-that your research hypothesis is indeed valid. This one is a biggie. Suppose a researcher has a hypothesis that A affects X which causes Y. The researcher manipulates X and sees Y change. Therefore the researcher concludes that theory A is supported since the null hypothesis is rejected due to "statistically significant data". The researcher is in fact wrong, and falling into a fallacy noted by Aristotle 2000 years ago. Statistically significant findings don't automatically mean A is true, since theories B, C, D, E, F.....could all say X affects Y as well, and may even be better at it than A. A good scientist would therefore pit theory A against theories B, C, D, E, F.... but that in most cases is impossible. Statisical significance ONLY describes the probability of observing your data under the assumption that your null hypothesis is true.
-that the probability that your hypothesis is true given your data D is answered by statistically significant data.
-that your findings are even important, it has nothing to do with the magnitude of effect one observes.

Power of analysis should always be done to design an experiment to properly determine sample sizes needed. Statisticians secretly laugh their asses off at much of the bio related literature from the poor handling of data and experimental design. Underpowered experiments run the danger of not only type II statistical errors, but also type I. It's laughable that hardly any biology studies outside of the clinic do not do power of analyses. Journals need to start rejecting manuscripts without proper a priori experimental design.

Here's a great article on the topic why most published findings are wrong:
http://www.plosmedic...al.pmed.0020124

If you want to do much higher quality research than the 90% of scrubs out there properly design your experiments using POA. Analyze your statistics for not only significance BUT ALSO MAGNITUDE OF EFFECT. Use confidence intervals (SEMs should be banned from literature) and look at things like Cohen's d, eta^2, etc. Maybe even try Bayesian techniques.

You make some excellent points, Ubiquitous, however I doubt that many of us on here will appreciate your condescending tone.

In the same way that many biologists may be under skilled in statistics, most statisticians would not make good biologists either.

Instead of adversary and derision (from both sides), would it not be better to have cooperation, so that those who have the appropriate skill set do what they do best, and have others help with the rest?

I know at least at my university has statisticians available to help with design and analysis of experiments- which plenty of biologists take advantage of (probably not as many as should, though). And you sure as heck won't get animal ethics approval to do your in vivo work without demonstrating that your experiments are statistically sound (i.e. have been looked over by a statistician, have had power analyses performed etc).

leelee on Fri Mar 15 05:41:44 2013 said:

Sorry didn't mean to offend anyone, my apologies. It's just that this topic non-reproducibility gets me going. We dump billions of dollars every year into bio and biomedically related research, and there's probably a good chance that much of it should simply be thrown in the trash because it was never done right in the first place. A. Complete. Waste. Of. Money. How much time, effort, and money do people waste trying to repeat "statistically significant data" that simply can't be repeated because the statistics were handled wrongly?

There's a significant threat to the scientific community from the hordes of published data that come from statistically underpowered studies. Everyone knows about the type II errors that can be made from underpowered studies, and once those findings are published, the error just keeps propagating and propagating. Then when someone tries a properly designed experiment and the data goes against what has been supposedly been " well established" in the scientific community, they're outcast like they're some sort of leper.

Next time anyone reads a journal article, look at how many samples were tested to create the data. Shockingly, a large amount of journal articles don't even tell you. Even in articles that do tell you sample size, they never give justification for the amount of samples being tested. This may be less prevalent in studies involving animals, since yes, many organizations required power of analysis in animal studies, but what about the huge amount of other research being done on the molecular or cellular level?

Ubiquitous on Fri Mar 15 05:55:12 2013 said:

No problems, I completely understand. We all have our pet peeves

I completely agree about the waste of research dollars. There is one lab in my department who do careless science and it irritates me to no end the amount of funding they waste (particularly from charitable organisations, thanks to them working on quite niche diseases) chasing up things they KNOW aren't going anywhere simply to fund their salaries and keep the lab going. So not cool.

Good discussion to be having, I think we all need to be more careful and thoughtful when it comes to our experimental design and statistical analyses.

I can only agree, so often I see people trying to do T-tests with an n of 3 or something similar, without even trying to test if it is a normally distributed dataset.

However, I think that the majority of stats that is taught (well, at least when I was undergrad about 10 years ago) is totally about parametric statistics with non-parametrics not even mentioned, despite me getting up to a 2nd year (sophomoric?? I don't know the US system) course designed for non-maths majors.

power of analysis, statistics, validity of hypothesis notwithstanding, i think that many of the irreproducability complaints are about methods ("why can't i get the column to release the poi with the procedure in the literature?", "why doesn't this gel show me bands like in the literature?", "how do i make this enzyme assay work?").

it seems that is what science noob was questioning and uncle rhombus' response was about. sometimes investigators omit information regarding a method, either intentionally or not. once, after we published a (basically) procedure paper, i realized that we had left out some important information regarding one step. that would certainly make it difficult to reproduce our work (without considering poa or other statistical significance) (fortunately, it was something that was easy to figure out).