Peer Review: A necessary but insufficient condition

BCO

Honorary Master
Joined
Dec 17, 2004
Messages
13,216
#1
I've been doing a lot of reading lately, particularly in the light of peer reviewed articles that get mentioned in other threads that supposedly disprove the notion of anthropogenic climate change.

While doing this reading, I stumbled upon an excellent article that looks at the concept of peer review, how it works in practice, and where the shortcomings of the peer review system become apparent. The article comes from a climate perspective, but it's relevant to the scientific endeavour in general and is invaluable reading for anyone interested in science.

For those interested in the climate debate, the closing two paragraphs of the article should really put into perspective any new studies that come to the fore that may challenge the consensus on climate change:

The current thinking of scientists on climate change is based on thousands of studies (Google Scholar gives 19,000 scientific articles for the full search phrase "global climate change"). Any new study will be one small grain of evidence that adds to this big pile, and it will shift the thinking of scientists slightly. Science proceeds like this in a slow, incremental way. It is extremely unlikely that any new study will immediately overthrow all the past knowledge. So even if the conclusions of the Shaviv and Veizer (2003) study discussed earlier, for instance, had been correct, this would be one small piece of evidence pitted against hundreds of others which contradict it. Scientists would find the apparent contradiction interesting and worthy of further investigation, and would devote further study to isolating the source of the contradiction. They would not suddenly throw out all previous results. Yet, one often gets the impression that scientific progress consists of a series of revolutions where scientists discard all their past thinking each time a new result gets published. This is often because only a small handful of high-profile studies in a given field are known by the wider public and media, and thus unrealistic weight is attached to those studies. New results are often over-emphasised (sometimes by the authors, sometimes by lobby groups) to make them sound important enough to have news value. Thus "bombshells" usually end up being duds.

However, as demonstrated above, even when it initially breaks down, the process of peer-review does usually work in the end. But sometimes it can take a while. Observers would thus be well advised to be extremely skeptical of any claims in the media or elsewhere of some new "bombshell" or "revolution" that has not yet been fully vetted by the scientific community.
Source.
 

Teleological

Senior Member
Joined
Nov 6, 2007
Messages
836
#3
Just remember, the so-called consensus relies heavily on peer-reviewed literature.
And the following runs both ways.
Observers would thus be well advised to be extremely skeptical of any claims in the media or elsewhere of some new "bombshell" or "revolution" that has not yet been fully vetted by the scientific community.
 

ajax

Executive Member
Joined
Oct 29, 2003
Messages
5,451
#4
Could the overemphasis also be to ensure sustained or new funding for further research?

When I was doing my degree the process amazed me initially. One would submit an article - we submitted to IEEE transactions - then the next guy comes along and cites the previous guy's article as if it is gospel. I realised later it's not a perfect system, but it's the best system we've got.

A friend of mine scored cum laude because he picked up an error in a lengthy mathematical derivation of a reference he used for his work. Obviously the peer reviewers didn't pick this up but eventually somebody does it seems.

At least in engineering and some other fields the research is often easily verifiable with measurements requiring little interpretation.
 

BCO

Honorary Master
Joined
Dec 17, 2004
Messages
13,216
#5
Just remember, the so-called consensus relies heavily on peer-reviewed literature.
Exactly. Without peer review, a paper's basically worthless. Still, peer review (as a prerequisite for publication) isn't infallible, and it's often the scrutiny of peers after publication that highlights the shortcomings of a paper.

And the following runs both ways.
True again, but in the case of anthropogenic global warming, the concept has been "fully vetted by the scientific community" and at this time is the best possible explanation of what's going on. Whether or not it remains so in the future is yet to be seen.
 
Top