Critical paper reading - (Jan/09/2012 )

Hi,

I am not that new to science, I am a PhD in my 2nd year, but I have been thinking recently about the papers and literature I have been reading. I hear all the time how thorough knowledge of the literature is important for any scientist, but you also hear about being able to criticise and sort the good papers from the bad ones. However I think that when I read papers I tend to take them at face value and not question their results too much unless they directly conflict with results that I get, and even then I tend to assume I am wrong.

How can you sort good papers from the bad ones, and does anyone have any tips on doing proper paper critiques? We never had a journal club in my lab so I have never had much of a forum to learn these techniques.

One of the best ways to get a feel for how 'good' or 'bad' results are quickly is to literally go straight for them. Read the abstract so you've got an idea of the general experiment and then read and interpret the results yourself, looking only at graphs, tables etc., disregarding the rest of the paper temporarily. Come to your own conclusions about what the graphs etc. show and how strong the data looks.

THEN read the body of the paper and see what claims they make from the data and see what their own analysis is. If it matches with what you found, then the conclusions of the paper and the results are probably relatively good. At least, in most (a majority of?) cases they will be. If it doesn't, or if they're making claims based on data that you felt didn't seem to show a strong trend, investigate it further.

It sounds like quite stupid and obvious advice, but if you read their analysis first and then look at the results, it influences your own thoughts and can stop you spotting things. Not to mention it takes a lot longer that way round, and critiquing papers is ultimately very time-consuming! Technically you should be able to understand the whole paper just from basic principles (what represents what in the model etc.) and the results without any of the extra information, and if you work on this principle it tells you a lot about a paper whether you find that you can (or can't) in practice.

As I said, sorry that it seems very obvious, but if you've not been approaching a paper in this manner before, then you can honestly see a big difference working via this method. This was suggested to me in the journal club I am a part of and it really is effective

This would be a good talent to have. The best advice I can give is to focus on the things that you know and the techniques that you know how to do, and try to judge the paper in the areas you are competent to judge. For example, if you're good at Western blotting, and the paper shows the most horrible Western blots you have ever seen and makes major conclusions based on what is essentially noise, then you can reasonably assume that the paper is fairly poor.

It is important to understand what you mean by "good papers". If you mean papers that contain true, repeatable data, than you will only eventually be frustrated if you are able to know what papers are "good". The reality of science is that there are people in power who decide what papers are "good" and put them in the big journals. This second definition of "good" requires no thought on your part and will lead to a much less frustrating career.