Is most scientific research sloppy?

I wouldn't be shocked if that were the case. We see enough examples of it (global warming being one notable area) to think that it could be just the tip of the iceberg. Getting it right is hard work, and there are a lot of researchers out there who are desperately working on degrees, or under the pressure to publish or perish, even if the research turns out to be perishable.

True story. Overheard in a university research group meeting:

"So, (forgot the name) at Stanford has data which indicates that he's found a monopole."

Physics department Chair:"What's his tenure status?"

Nobody was smiling.

The weirdness is in how varying the papers are.

In Physical Chemistry, where you are often able to control a wide array of potentially influencing variables - people do. And monitor all that they can. And log everything. Then study the correlations between crazy things like 'Day of Week' and your main variable.

And, at conferences, you're asked "Did you check XYZ for that odd result?" So the audience is paying attention, and asking about things you did that don't appear in the paper - only 'the unattached appendices' - your logs.

But you can wander down a couple of doors at the same ACS meeting and sit in one something else. Where the entire sub-field is based on multivariate statistical analysis of something they _can't_ control for directly... and the whole process just seems a lot less rigid.

Yeah, there are a lot of cultural differences from one field to another. I did a postdoc in the Electronics Technology Division at the Naval Research Lab, i.e. solid state. So I was looking through the solid state edition of Phys Rev, and came across an editorial where the editor responded to complaints about the low quality of the papers. His defense was that standards were just lower in solid state -- referees would not reject papers that would be rejected in other fields. He had to live with it, and so do you.

The most common comment I heard about my work there was, "Wow, you really took a lot of data!" I took 15 data points on a curve that had a flat beginning, an upslope in the middle, and a downslope at the end. And something weird happened at the peak, but I didn't have enough data to even be sure it was real.

My guess is that standards are lower in fields where people badly want results, because results make money (solid state) or affect policy (global warming). The upshot is, the best science is done on subjects nobody cares about. This would explain why we know the gyromagnetic ratio of the electron to eleven decimal places, but haven't reduced the uncertainty in the climate sensitivity since 1979.

The Aztecs invented the vacation.

"The upshot is, the best science is done on subjects nobody cares about."

My pet example of this is astrophysics. There's _interest_, but there's just no practical implications anywhere in evidence. So the journals are all available on the internet with free _live_ flipping hyperlink references. I spent a couple of days wandering around following links just from sheer amazement. Even relatively antique articles available on the internet. (Sheesh, can't contract "on" and "line")

I mean, back in my areas, Chem Abstracts has a lock on making their buck in the chemical fields.

Hey, mz is still arguing that institutions aren't biased on this thread. He even thinks the Grauniad is unbiased in the area of "Global Warming Is The End Of The World".

There is a Penn and Teller Episode on Environmental Hysteria. However I cannot post the link to the video due to finicky programming on this blog.

(Transterrestrial Musings
Comment Submission Error
Your comment submission failed for the following reasons:

Your comment could not be submitted due to questionable content: mispace

Please correct the error in the form below, then press Post to post your comment.)

Search video google for "Dihygrogen Monoxide" and "Penn and Teller" if one should wish to see it.

"Ban Dihygrogen Monoxide!" ;)

What is "Dihygrogen Monoxide"?

To me, the scary stuff is Dihydrogen Monoxide. ;)

Hey, Robert, I see you got it right in the thread below.

The funniest thing was that all those people at that Green get-together featured on Pen and Teller signed the petition to ban Dihydrogen Monoxide!

Science is a process not an event. People, especially scientists, forget that at their peril.

To me, the scary stuff is Dihydrogen Monoxide. ;)

Bah. The REALLY scary stuff is the hydronium hydroxide.

Hydronium hydroxide is a greenhouse gas, maybe the UN should put a hydronium tax on it and impose a cap-and-trade system on it.

Please think carefully about the implications of these observations. If science is sloppy (or, in some cases, just unlucky, with statistical noise being published as results when it is really just spurious), then you must NOT pick and choose a few scientific papers that you like. You need to look at the literature overall, where redundancy and averaging will provide more reliability. Yes, science is not democracy, but 'voting' is a way to increase the reliability of systems built with unreliable components.

Global warming denialists often do the wrong thing in this respect, focusing on a narrow cut of fringe papers that happen to support their preconceptions.

Marginal Revolution had a good exposition on the reasoning behind Ioanddis' result.