Difficulty in Covering Studies

Via Slashdot (a site for technology news and comment), I came across a thoughtful blog post by a scientist using about covering scientific studies. This should be a must read for every journalist, science writer or not. The author notes that any scientific study sits on a set of assumptions, and that it's common for someone to eventually show some of them are wrong, making the conclusions wrong. Journalists, who probably don't have an extensive background in the subject, now have a problem:

How, then, do you effectively cover a story laden with valid assumptions, some likely to be correct, many likely to be incorrect? Let us use climate models as an example. In order to avoid long computing times, the use of super computers, or simply (and usually) because the information does not exist, modelers are forced to typically make 100’s of assumptions when devising their code. Now, I’m not saying these models are not at all useful. Smart modelers have determined ways of lining up their assumptions with observations of the real world (often, modelers must predict what we already know to verify their assumptions - i.e. does it work?).

Here, the same problem exists - how do you, the science journalist, determine which of these assumptions could bring the entire model crashing down? Furthermore, if such an linchpin exists, is it an important one? How important? Is it likely to be incorrect? How likely? Unfortunately, these questions have no definitive answers, except with respect to each other, and with respect to the particular researcher.

His context is science, but it could be sociology, psychology, political science, linguistics. Any time you see a study or hear an assertion by some expert, it may be completely wrong, based solely on assumptions and not even considering self-interest that can drive the person to try spinning reality to his or her own advantage.

The article doesn't even touch on another major area of error: poor statistical and sampling methods. Any time you collect sets of information and attempt to make inferences, you're in dangerous territory. I spoke with the chief marketing officer of a law firm. He had a PhD in market research, actually is one of the people who understands how the math works. As part of our discussion, he pointed out that if you had a study where you approached 5,000 professionals and eventually only got 500 to respond to a survey, the best accuracy you could get in the results would be plus or minus 20 points or so. That's enough variation to toss almost any conclusion out the window - and from my own study of the subject, that doesn't even include one of the biggest causes of error: the way the study phrases and orders questions.

I'm not saying to give up using studies, only to be more judicious and suspicious. To quote any sort of study in your work, you need to ensure an adequate grasp of some fundamentals. No, you don't need to apply to graduate school. But there is a lot you can learn in a short amount of time to help you see at least the obvious red flags. Here's an article from American Demographics that can act as a jumping off point for further learning. And here is a checklist of some aspects to consider.