Bad Science vs. Good Science: A Guide for the Layperson (Part 1)

This week's column by Ctein

It's Column 300 (best as I can count), and as is my tradition I'm going off-topic into the realm of science. This time I'm going meta, but that will probably be of more practical value to most of you than most of my science columns.

As a layperson, how can you distinguish Good Science from Bad Science? If one source of information says "X" and another "not-X" how do you decide which one to believe when it's not a discipline that you know well enough to be able to evaluate on scientific merits? There are tools that'll let you make the right decision almost every time.

Please understand that "good" and "bad" aren't the same as "right" and "wrong." Bad science is almost always wrong, at least 99% of the time, so it's good to be able to sniff it out. But, on occasion, it proves correct. There are a few historical examples (Newton's theory of gravitation). Conversely, Good Science, while usually right, is regularly proven wrong (last year's report of faster-than-light neutrinos); that's part of the point of science. Picking the Good Science doesn't guarantee that you're correct, but picking the Bad Science almost always guarantees that you're not.

Neither Good nor Bad Science can tell you what is true. General relativity and quantum mechanics are Good Science. Are they actually true? We have no idea. Given the history of science, probably not. But they're correct enough, as demonstrated by your GPS (which makes use of GR), and your personal computers and electronics (which depend on QM).

Hundreds of you will be offended by these columns. Why? Because if some percentage of the population didn't believe in all kinds of Bad Science, there'd be no reason for me to be writing these columns. Let's imagine, optimistically, that it's only 1% of the TOP readership. Usually it's a lot higher. TOP has a readership measured in the five figures. You can do the math.

Should you feel compelled to write a fervent rebuttal to my branding your pet belief as Bad Science, Mike will happily publish your comment...so long as it is reasonably concise, not merely a political or religious rant (verboten topics, sez our Ed), and does not personally attack or insult anyone, including moi (no ad hominem). I will likely not respond to your rebuttal. That doesn't mean I will agree with you, only that I will not go out of my way to prove you wrong nor read some website that you're convinced will prove you're right. It's not my job nor my interest to tackle every bit of nonsense or bad information in the world.

Now, on with the show!

First, get as close to the source as possible. The more expertise your informant has, the closer they are to the research, the more likely they are to know what's good. If some columnist at Forbes or the Wall Street Journal says there is no scientific consensus on anthropogenic global warming and the editors of Science and Nature say there is, whom do you think really knows more about the science? C'mon, really. If Science and Nature are too dense for you to deal with, read Science News. It writes about science for the lay audience. It's not a primary source like Science and Nature, but it's a very, very close secondary source, written by people who actually have expertise.

Related to that, be suspicious when scientists step far outside their field of expertise and report unusual results. A lot of good scientists do Bad Science when they move into a discipline they're unfamiliar with, because it's a tricky thing getting experiments right. I'm a good enough experimental physicist; I wouldn't trust me to be able to run a good medical biology experiment.

Cold fusion's a good example. The discoverers knew so little physics that they didn't even know how to run some of their instruments correctly. They made some freshmen-level mistakes that were blatantly obvious to anyone who was minimally experienced with the equipment.

Next, consider the process. Did your informants submit papers for peer review? Did they get published in reputable and well-established journals or websites? If they're "taking it to the people" before doing any of that, that's a caution. Bad process usually means Bad Science. Again, see cold fusion.

Take a look at the scientific trend lines; these are really good indicators. All scientific theories and hypotheses, good or bad, start off with just a few adherents. Most of them, if they're not completely insane, rapidly acquire a fair percentage of followers because scientists collectively are open-minded and, not to put too fine a point on it, credulous. They'll be happy to provisionally believe in something until proven otherwise.

After achieving some modest popularity, though, Bad Science starts to lose supporters. More and more scientists become convinced that the data doesn't support the theory. Eventually you're left with only a small percentage of scientists believing in the Bad Science and the overwhelming majority abandoning it. Conversely, Good Science will eventually acquire a substantial supermajority. There'll remain a small percentage of naysayers into the indefinite future, which is why you look at the trend, not the exceptions.

Frequently, the trend can also be seen in the data. A bit of Bad Science will pop up because of an experiment or study that shows a large effect. When further experiments and studies fail to replicate that effect, or the effect comes and goes or becomes very small compared to the initial results and the noise in the experiment, those are all huge warning signs. Cold fusion failed both these kinds of trend lines.

Global warming denial fails the trend line. Deniers constantly trot out the hundreds of climate scientists who stand with them. What they don't mention is the tens of thousands of climate scientists who stand against them. Thirty years ago, that miniscule fraction might have been the start of a growing trend. Now, it isn't. It's the tail end of a declining minority. Anthropogenic climate change is settled science.

Watch out for the lone scientific "genius," crying out in the wilderness, whose work isn't appreciated or understood by the Establishment. That's a romantic notion promulgated in folklore and bad novels. In the real world, 99.9% of the time, the lone genius is a lone crackpot.

That's something you can look for in the discourse. The person who claims that they are asking questions that no one else dares to or that no one else has thought of, isn't. They are not smarter, cleverer, nor have a wider-ranging collective imagination than the thousands of other workers in that field. Any time you see someone saying in so many words, "I am so much more brilliant than all of them, because I have thought of this basic question that none of them have," you're seeing Bad Science. That one you can take to the bank. Don't bother reading any further in their screed; they will have nothing of value to tell you.

Another Bad Science sign is the "God of the Gaps" problem. This pops up in Creation Science (one of the grand old men of Bad Science) all the time. A classic tool of the Bad Scientist is to home in on some unresolved question and treat it as paramount, in essence saying, "Aha, science can explain A through Y, but they cannot explain Z. Therefore their science fails."

Nope, not even close. Science rarely has complete or perfect answers. There are always unresolved questions and gaps in our knowledge. Creation scientists love to point those out as proof that evolution isn't a viable theory. The problem is that the gaps keep getting smaller and smaller and of less and less importance. Remember what I said about trend lines? The god who was supposed to create entire species has been relegated to creating subtleties of molecular biochemistry and statistical selection processes. Most philosophers and theologians consider that a pretty piss-poor version of a god. Make a note: mixing science and religion? Always a bad idea for both of them.

This also shows up among the global warming deniers, who keep looking for the anomalous tree in a forest of coherent explanations. They have no trouble finding such; climatology and geophysics are really messy and incomplete sciences. Much Good Science is. Life is not always neat. Recognizing that does not make you brilliant.

•    •    •

Next time I'll pick up with the genuine nutters and crackpots, some of whom are genuinely dangerous.

©2013 by Ctein, all rights reserved

This is Ctein's 300th weekly column for TOP, which, according to the Editor's arithmetic, means he's been writing for us for .17333 years.

Links in this post may be to our affiliates; sales through affiliate links may benefit this site.

TOP's links!

(To see all the comments, click on the "Comments" link below.)
Featured Comments from:

Andrew Molitor: "t may be worthwhile to add a small compensating factor for money, where that quantity can be estimated. There's a lot of, if not Bad Science, at any rate Not Very Good Science which is done because there's grant money available. The relevant community may be hem and haw a bit on whether the science is any good, but by golly, that's a lot of money, so not very many will actually break ranks until the Science is shown to be really really Bad. That can take a while because, well, money. Cutting edge medicine seems to suffer from this a lot. It tends to sort itself out—eventually—but boy does it take a long time. Anyways. If you know about the money, and there's a great deal of it coming down on the Consensus side, you might want to squint at that side a little more closely."

Steve: "Try Conjectures and Refutations by Sir Karl Popper. Gets to the heart of this matter and is far from an impossible read.

Peter Horby: "I'm more than a little bemused by your characterization of Newton's theory of gravity as 'Bad Science.' To my layman's eye, Newtonian gravity is the poster child for Good Science. Every observation of the positions of celestial bodies for 250 years was explained by it, its predictive power was immense, and it still works perfectly well as the non-extreme limit of Einsteinian gravity. Good science is what Popper always said it was—a model with explanatory power which is falsifiable."

Ctein replies to Steve and Peter: I tend to be a Popperian (Popperite?) myself, but in fairness I should note it's not the only philosophy of science out there. There are other legitimate and useful ways to view the process. And, no, I can't write a column about that—I don't know this stuff well enough. Just mentioning this in case people are interested.

Newton's theory of gravity was Bad Science because it was fraud. At the time Newton proposed it, there were several competing theories of gravitation and the data was far too noisy to clearly favor one over the other. So Newton falsified his data—he published faked results that favored his theory!

This is the kind of fraud that is essentially impossible to detect. It's even likely to survive the Popper test because the theory is replicable.

Mendel, by the way, did essentially the same thing—he threw out breeding results that didn't fit the pattern he expected because he really didn't understand how statistics of small numbers work. In his case it probably wasn't deliberate fraud, he likely thought there was something genuinely wrong with those experiments. So it was more like malfeasance. Newton knew exactly what he was doing.

Throwing out data because you don't like the results when you can't see anything wrong with the experiment is poor process. It's one thing to eliminate it because you have a good reason for thinking it's wrong. (Just thinking, "Well, it doesn't fit with the rest of the data," is not a good reason.") That's why the superluminal neutrinos were an example of Good Science that was wrong. The researchers firmly believed the results were erroneous. The problem was that the experiment was so complex that they were unable to find the flaw in it. Which meant, under the precepts of good science, they had to release the results. They did so, clearly stating that they expected to be wrong and they needed help in finding out how they were wrong.

Of course, they hoped they were right, I mean, guaranteed Nobel Prize and all that. But they were pretty damn sure they weren't.

Steve Jacob: "I agree 100%. My degree was in chemistry, which gives me a smattering of physics, climatology and biochemistry. But I understand enough to know that my own opinion is worthless. I have not seen all the data, I am not qualified to judge the modelling involved, and I am certainly not up to date on all the prior research. But I can spot bogus nonsense or concocted statistics a mile away. They are normally reported in newspapers and popular news channels, usually out of context and using some strident quack as the only source. And I choose to accept (not on faith but as a working premise) that a large scientific consensus has a much better chance of being correct than a small number of vocal antagonists, especially if the question is one that has attracted a sufficient amount of attention, research and peer review. After all, just because I may not like the answer, or may even find the conclusion inconvenient or uncomfortable, none of that makes the slightest difference. My personal agenda is irrelevant. Clearly some people find that a hard thing to accept."

Mike Kukulski: "You both missed mentioning one 'bad science' trait and then used it yourself: 'xxx is settled science.' There is no such thing as 'settled science.'"

Ctein replies: Nope. Wrong. Settled science does not mean that there are no further questions to be asked. It means that the particular question under consideration has been settled. You can move on to details, refinements, exceptions, and extensions, but the starting question is no longer under debate. "Anthropogenic climate change" (and note that the whole phrase has to be considered, not the subset someone wants to argue about, nor additional adjectives that someone wants to tack on to set up their straw man) is settled science. The fundamental Theory of Evolution is settled science. The Atomic Theory is settled science. The Face On Mars (being ordinary geology and an illusion of lighting) is settled science.

Arguing those further is a waste of time and energy and resources and only gives a platform to the oddballs who are  otherwise at risk of losing their soapboxes. Keeping debate open on topics that no longer need to be debated is not a virtue, it is a vice. It is a waste of resources. It has no value.

Claiming "open-mindedness" is not a virtue when what you're being open-minded about is nonsense.

And, yes, engaging the God of the Gaps and the Conspiracies in the Closet as a way to maintain the veneer that there is still intelligent discussion to be resolved is nonsense.

Chris Lucianu: "Great column, generally. Minor issues:

1. Bad and Good Science need to be defined explicitly; especially if this Guide is aimed at the interested layperson.
2. The counterexamples (Newtonian gravitation; superluminal neutrinos) need to be explained for the sake of the layperson rather than merely mentioned in passing. I'm somewhat familiar with the matter, therefore I have an idea about what Ctein is trying to convey. Laypersons less conversant with the subject may just be passed by.

"It's tough to be a ruthless editor—which any good editor must be—with an author as august and as brilliant as Ctein. But I'm going to stick my neck out (fully expecting to have it chopped): I'd mark paragraphs #2 and #3 of the column with a squiggly line on the margin and send them respectfully back to the author with this note: 'Define. Exemplify. Clarify.'

"Objections: 'Make a note: mixing science and religion? Always a bad idea for both of them.' Nope. Just for religion. Whatever can be backed by scientific evidence ('Good Science' in Ctein's terms) is outside the scope of religions. Whatever is merely speculative is probably Bad Science (and bad philosophy, too). Whatever is just belief is just that: unsubstantiated belief.

"'Climatology and geophysics are really messy and incomplete sciences.' I'd argue that the subjects themselves are messy and complicated, not the science trying to untangle them. But so is reality. Our ideal is the simple, the elegant, the intuitive. Unraveling the complication and revealing the complexity is a huge step on the path of understanding."

Ctein replies: Unfortunately, in this case, I am the Ruthless Editor. I hacked and slashed like crazy. The two columns add up to 2500 words as it is. To give the subject proper treatment, I would've had to let them run twice that long but then they would've really fallen under the doctrine of 'longissimus, non legi' (thank you, Liza Furr!).

This is always a problem for online magazines; you simply can't write 3–5000 word articles and expect people to read them. So you leave out a whole bunch of interesting and germane stuff. Mike and I both do this all the time, 'cuz we haveta.

I figured I'd get to explaining the counterexamples in the comments and questions (which I are already have). I decided not to define Bad and Good Science, as people can pretty much figure out the definitions recursively, by reading my meta-rules.

As for mixing science and religion, that's a whole 'nother subject, but it does get mixed both ways annoyingly often and the results are almost invariably problematical. Attempting to establish your faith on the basis of science is building on pillars of sand; scientific paradigms are regularly overturned. Conversely, trying to make science follow your religious beliefs is not likely to go well, for obvious reasons.

Although it worked out very well for Keppler. Good Science inspired by exceedingly unlikely theology. There's always an exception.

Randal Jaffe: "Wonderful! As a scientist I try to make these points to my students. It is hard to explain to the general audience that science is not monolithic—sometimes the dissenting voice is correct. That is quite different from taking every dissenting voice as correct."

Bob Blakley: "Attempting to bend the subject back to photography, better instruments can support better science. Certainly TEPCO practiced bad science by using 100mSv Geiger counters to measure radiation they thought was in the range of 100mSv—because they were using the instrument at the edge of its range. When they switched to a better instrument last week they found that the real radiation intensity near Fukushima was about 1800mSv.

"Eadweard Muybridge improved science with a faster camera; he immediately settled the question of whether a horse's four hooves are simultaneously off the ground when it gallops using the new instrument. Edgerton took high-speed photography as a collector of scientific data to great heights. And Edison used it to invent motion pictures. Galileo built a better telescope (another imaging system!) to settle the question of whether there were bodies in the universe that did not revolve around (ahem) the Earth.

"At the end of the day, better data usually trumps cleverer theories, though the theories are what we like to argue about. And cameras can be very useful data collectors."

Jordan: "Conspiracy allegations are also a good way to quickly triage scientific claims—people who promote bad science tend to resort to conspiracy theories pretty quickly when attempting to get support for their work. You see this most clearly in the bio-medical field (alternative-medicine proponents alleging conspiracies that keep cancer 'cures' from coming to market, etc.) Another piece of the story is the role of the media in fuelling the confusion between good science and bad science. Lone wolves and underdogs make for great journalistic drama, but the tendency of reporters to go for 'both sides of the story' often ends up giving far too much weight to marginal work with little or no scientific support. And the problem gets magnified when the reporter has no scientific background (which is often the case)."

Stephen Scharf: "Regarding Ctein's comment 'be suspicious when scientists step far outside their field of expertise and report unusual results':

"While certainly Bad Science can result from this, history has proven that a lot of Good Science has also resulted from scientists stepping into another discipline; usually because that scientist is is not bound by conventional paradigmns and brings a fresh pair eyes and thinking to bear on scientific problems that have eluded the domain experts who are bound by 'in-the-box thinking.'

"For example, it was the experiments with the RII phage system that Seymour Benzer did that established that nucleic acids were the biological constituent that governed phenotypic inheritance. Benzer a trained as a physicist.

"Francis Crick was also trained in physics and did seminal work both in the discovery of the structure of DNA, conceived of the notion of transfer RNAs in protein translation and made contributions to neurobiology.

"The basis for antibody diversity eluded immunologists and cell biologists for decades until Philip Leder elucidated the genetic basis of antibody diversity. Leder was a molecular biologists, not an immunologist.

"I was sitting in a graduate protein biochemistry class years ago when the professor said that only 'protein-based' enzymes could demonstrate biochemical catalytic activity. When I asked him why RNAs could not demonstrate catalytic activity, he laughed in my face and said, 'Absolutely not, only proteins can serve as catalysts in the form of enzymes.' In 1982 (six years after my question in that biochemistry class...), Thomas Cech discovered catalytic RNAs and won the Nobel Prize in 1989. Cech was not a biochemist, either.

"So, while it may be true a signficant percentage of the time that scientists crossing over to other disciplines often do Bad Science, it is also true that truly significant breakthroughs have occured when scientists from other disciplines have done just that.

"Having a been a professional scientist my entire life, my formal training was in bacteriology, but virtually all of my significant work  has been in  disciplines I was not formally trained in, ranging from molecular biology (I'm one of the scientists at Cetus Corporation that developed the polymerase chain reaction, or PCR) to human genetics to molecular diagnostics to DNA forensics.

"Across the breadth of that career, if there's one thing I've learned, it's that it's not what scientific domain you're trained in that's important, it's how you think as a scientist that's important."

Comments

[Markus Spring]

Ctein, thanks for the clear and precise words - and your clear qualification that even good science is not able to explain everything at once and in infinite depth, and still can be good science.
I've never read such a comprehensive short piece on that really complicated topic.

[Sarge]

Congratulations, Ctein, on your 300th TOP column!

By my reckoning Ctein has been unstinting for 5.75 earth years and counting.

I don't get the math behind .17333 year. It's Good Science though.

(Is that the equivalent in Internet years in Mars?)

[Jim Simmons]

Hear hear!

[Graham Wood]

ctein

good for you.
I am sure you are aware. but anyone who wishes to inform themselves further could do well reading "bad science" by Ben Goldacre.

[Bill Tyler]

Excellent column, Ctein. I didn't quite take your point about Newton and gravity. As I recall my history, the Principia was quite well received, after Newton was persuaded to write it. A better example of an ill-received theory would be plate tectonics - initially scorned, then a nice example of your trend-line criterion as it became nearly universally accepted over time as evidence accumulated.

[Bruce Robbins]

After Ctein's car crash of logical fallacies Aristotle will be spinning in his grave! Argumentum ad populum and argumentum ad verecundiam in a head-on collision. Ouch.

I'm sceptical of catastrophic anthropogenic global warming. Does that make me a "global warming denier"? No. The planet has been warming since we emerged from the Little Ice Age in the middle of the 19th century. I accept Arrhenius' science on the so-called greenhouse effect. I accept that we have caused some of the warming.

What I don't accept is that the planet is doomed or seriously threatened in any way by global warming. It all boils down to climate sensitivity - how climate feedbacks amplify the accepted physics of a 1.2 C increase for a doubling of CO2. That's where the jury is out. The recent peer reviewed literature has reduced this climate sensitivity from the IPCC's earlier predictions of about 3 degrees C to less than 2C. In other words, a doubling of CO2 + climate feedbacks might cause a temperature increase of a little under 2C but we've already had half of that increase. I doubt we've heard the last word on climate sensitivity: it could be lower. That's certainly the way the science is heading. It's cheaper adapting to this sort of temperature increase if and when it's necessary than jeopardising economies by launching a carbon trading bomb.

I know a huge amount of CAGW sceptics personally and through the internet. I don't know a single one who denies global warming.

[So you're saying it's not fair to refer to you and your friends as global warming deniers. But then, Ctein was talking about global warming deniers. So, by simple logic, shouldn't you accept that he wasn't talking about you and your friends, but about a different group altogether? QED. --Mike]

[John Roberts]

Please bring back Kirk Tuck!

[elton]

Brilliantly thought out and worded out! I especially like the lone genius part... Its quite apt because here in India, we have people claiming scientific miracles ever so often, this one being the earliest ones I remember... https://groups.google.com/forum/#!topic/soc.culture.indian/eNnxbn_Rw6U.

[Roberto]

A checklist for whoever feels ready to present some "alternative science" result. Useful also to identify crackpots:
http://blogs.discovermagazine.com/cosmicvariance/2007/06/19/the-alternative-science-respectability-checklist#.UicRSWTOdAE

[Goff]

Well done, Ctein! Your topic is worth airing.
You mention anthropogenic climate change. I was a founder member of the science team (IPCC) assembled by the UN to construct advice to all governments on that issue.
It was an interesting challenge. It would have been easy to give advice that was solid, with all scientists around the world agreeing; but that would have been textbook science, and it would have missed the hot issues. At the other extreme, it would have been useless to offer advice based on the very latest, still unconfirmed, evidence, which most reputable scientists in the subject would have been hesitant about. What the UN wanted was for us members of the IPCC Science Panel to push the envelope of knowledge to the limit of what the majority of good scientists in the climate business would just sign up to. We had to work on the edge of consensus.
I believe this was the first such exercise in any scientific discipine. It was intellectually challenging, not least because understanding the climate system requires expertise in a wide range of scientific disciplines, from physics to biology. I was trained as a physicist at one of the best schools (Imperial College London), and have worked in meteorology, oceanography and theoretical ecology.
It is possible to move outside one's original discipline, but one must take great care and learn from the experts. In fact moving across disciplines is a pre-requisite for successfully addressing complex inter-displinary problems, like climate prediction.
John Woods (emeritus professor of oceanography and complex systems at Imperial Vollege London). aka The photographer Goff, founder of Foto Zerüi not-for-profit agency for photographers. See www.fotozerui.com

[Robert Roaldi]

Amen.

[Svein-Frode]

I tend to discuss global warming, not in terms of scientific accuracy (the cause and future effects can't be proven 100%), but in terms of statistics. If there is a good statistical probability that global warming is happening and is caused by Man, then it should be dealt with now.

Sadly, in the real world science coexists with religion - and politics. As you say, a bad idea, but a fact of life nontheless. The public has no way of telling good science from bad science. Either they don't care or they don't have the time and/or interrest to aquire the knowledge. As such they can only rely on media to shape their cognitively biased opinions on almost any subject.

[MartSharm]

The irony here is that this column, purportedly about “good science”, is an attempt to shut down scientific debate, to close minds, to demean those who still remain unconvinced about the largest, most expensive, and potentially most damaging (or most constructive and essential, depending on one's point of view) pop-science movement of our times. Anyone with an open mind, and moreover, are supporters of the scientific method, can see the “climate change” has become deeply politicised, with ardent supporters on both sides, neither of whom can honestly claim their work is independent of bias. Just think - climate scientists declaring the importance of climate science? You don’t say! Personally, I prefer to look at both sides of an argument before forming a view, and in this instance it appears to me rather rum to say climate science is settled. The latest puzzle is the IPCC simply making up their latest 95% confidence figure. It's not based on any experiment or calculation. Not that 95% confidence is even close to representing “settled” in any scientific field I'm aware of. Is that the truly good science being referred to here?

It's a fact that wind farms are blighting otherwise beautiful countryside, and energy bills are higher than they should be, due to the influence of climate change activists - with nobody predicting any measurable effect on temperature. They had better be right that this cultural vandalism is in some sort of ephemeral good cause. My point is, this particular topic is not your usual theoretical science, like gravity or evolution - it's making the poor poorer, and indeed the rich richer, in a very measurable way. It’s perfectly proper to be sceptical.

Usually, TOP opens my mind, engages my intelligence and stimulates my imagination through the various topics covered. This column appears an attempt at shutting down a debate which, it has been made clear before, is a particular enthusiasm of the author and the editor. Which they are perfectly entitled to publish, of course; it simply doesn’t fit with the usual tenor of the site. My mind will remain open; may I ask you not to accuse me, and those like me with a healthy, inquiring mind, a sensible dose of scepticism, and respect for the scientific method, of being a “denier” any further? It leaves a sour taste in the mouth. Thanks.

PS “This is Ctein's 300th weekly column for TOP, which, according to the Editor's arithmetic, means he's been writing for us for .17333 years.” Um, shouldn’t that be 5.7692 years?

[Rod S.]

Well said, Ctein. And nicely comprehensive. Thank you.

[psu]

I want to hear more about your thoughts on Newton.

[Bob Rosinsky]

I prefer "global climate change" to "global warming". Either, either, tomato, tomato-- can't deny the obvious. Science is not in everybody's lexicon. In my case, Ctein, you are preaching to the choir.

[Mark St. John]

Thanks so much for this Ctein. I am looking forward to this series of post.

I'm a scientist and find it frustrating how even my nearest and dearest consider my advice on everything from climate change to crackpot cure-alls with deep skepticism. But, they will run out and buy a Himalayan Salt Caves inhaler (for example) the moment someone on TV suggests that it just-may-be-the-thing-you've-been-needing-your-whole-life. People love amazing, mysterious things, and unfortunately the truth is often boring, anti-climatic and inconvenient.

I espouse the belief that we need to make decisions based on the BEST information we have at that time; doing otherwise is foolhardy. Human-caused global warming? The best information says yes. Done. Which means one also needs to be 100% committed to dropping their current belief when better information contradicts it. This is hard for a lot of people; they seem to equate having their minds changed during a discussion/argument with an admission of weakness and having 'lost'. My wife is such a person; she'll argue black is white 'til the cows come home!

In a nutshell, scientist have a tough row to hoe!

[steven Ralser]

Also look at the references for a paper, especially if it is an opinion piece (e.g. From climate change deniers). Are the references from peer reviewed literature, or are they from other opinion pieces. The is a ploy many climate change deniers will use. They will asset something in, say an opinion piece, and later on reference this as gospel truth. When you dig back through the references you may find that there is nothing to support their line of reasoning.

For those who don't know, when you publish a scientific paper, it needs to be reviewed by at least 2 other scientists who are knowledgable in the field. The aim of these reviews is to ensure that the science in the paper is accurate, there are no holes in the logic, conclusions support the data being presented etc. If the papers are more controversial there may be a much larger number of reviewers.

The process to get funded by the National Science Foundation, which funds most scientific research, is even more difficult than to get a paper published. The proposal you write is in essence a scientific paper that details what you propose to do (plus include a large number of required sections, e.g. Relevance, educational outreach etc). This proposal is then reviewed by anywhere up to 7-10 scientists in the field, and then further reviewed by at a panel meeting at NSF (again by 7-10 scientists in the general field of study). In essence all of these reviewers must give the proposal an excellent ranking in order for your research to be funded (the competition is so tight that if your proposal has a couple of very good rankings it probably won't get funded). I have not written a proposal for a large number of years (I am a geologist by training); it was difficult to get funded then; it is more difficult now (I see what my wife has to go through when she writes a proposal).
It is only after being funded that scientists can do the research and write the papers.

I thought I would write this to show some of what is involved in scientific research. Mike feel free to use some, all, or none of this.

Steve (in Madison)

[DavidB]

Interesting start. I am a scientist in the biomedical field. We have an expression, Just because its published in Nature, Cell or Science doesn't mean its always wrong. You might find that a lot of "high impact" I need to get a job publications, end up their that are not reproducible. Case in point is that Yeast Prions are a benefit. This has never been shown and constantly disproven, but still is taken as true.

Also to clarify a critical problem in science. Scientists and researchers have a Hypothesis that they try to confirm with experiments. When lots of data piles up, from many groups, then you may have a theory. Similar to Mathematical Law. The scientists that claim to have a theory earlier usually are Bad science. And the ones that think they have a theory and do everything possible to try to manipulate the data to "prove it" also is bad science.

[Steve D]

I generally agree with this post with a notable exception. I believe you are suggesting that consensus among scientists suggests that a conclusion (or proof) has been arrived at and that "settles" an issue. By definition, this is not really science. Unless all those scientists actually verified the collected data and accurately replicated all the models (doubtful) they are merely in agreement with what they have read.

I recall learning in high school that dinosaurs were cold blooded (it was taught as fact)this has now supposedly been "disproved" and consensus s that they are related to birds. I've been told over my lifetime that eating eggs is a terrible idea, then it became necessary then terrible again. Now a good deal of consensus is is contradicting the governments own food pyramid and supporting the notion that a healthy diet is based on high fat and protein and low carbs (see the Gary Taubs link in Miike's dieting post). On and on it goes.

No offense but it appears a lot of hard work and "good" science done over the decades has been founded on very shakey consensus. I would wager that over time the consensus of the degree of influence of man on anthropomorphic global warming will continue to shift. Modeling is my field,and I cringed over very large and very real scandal over much of the initial modeling assumptions and politicization o the data. You simply can not model the earth as though humans never existed. Talk about assumptions! Even in the branch of engineering in which I work, I am astounded at the assumptions that clients will treat as certainty when it suits them. Hal my reports are filled with disclaimers.

Still though, I'll always chose to keep looking than just give up and credit an all knowing and all wise deity. Too many things just don't make sense. shrug..

[S. Chase]

So 'Bad Science vs. Good Science:' turns out to be a Trojan horse for yet another global warming lecture. Why? Has the web been silent on the topic? Are MSNBC, HuffPo, and the Daily Kos mum?

Ctein lays out material for intense debate here but removes himself from the discussion before the fun has even begun. Frankly, I don't believe for a second that he is immune to Bad Science that supports his world view (few people are). But, more importantly, I want something different from TOP. It is, to me, a place for thoughtful escape.

[Tom Clifton]

"Related to that, be suspicious when scientists step far outside their field of expertise and report unusual results. A lot of good scientists do Bad Science when they move into a discipline they're unfamiliar with, because it's a tricky thing getting experiments right."

On the other hand, there are some examples of good science where one or more practioners stepped out of their comfort zone and changed how we think. Luis and Walter Alvarez and the K-T boundary jumps to mind. The key is that they practiced good science, did the work, and as a result the trend lines moved in their favor. But when it was first suggested...

[Andrew Kirk]

Brilliant! Thank-you.

[James Gaston]

Thanks, well said. Many people don't understand that science is not a matter of opinion ("I don't like the idea of natural selection, so it isn't true") but is based on experimentation and is repeatable.

And science can only answer some questions. For example, science has yet to explain the origin of life ("is there a god?") but there is no doubt that evolution underlies the existence of the variety of life forms around us.

[Dave Sailer]

To my naive friends I have said over and over that "Science is not a collection of answers, it is a way of asking questions," but they can't get past barking that "You can't count on scientists because they keep changing their minds."

Which is why so many walls are worn bare by me banging my head against them.

[Roger Moore]

There's a very important secondary point to the one about trend lines: never take anything seriously based on one paper or one research group. It's very easy to make mistakes when you're carrying out an experiment and get erroneous results. It's somewhat harder, but still easier than people think, to keep making the same basic mistake and reproduce your own flawed results. It's only when somebody in a separate research group is able to get similar results that laypeople should start paying attention. Comparing cold fusion, which outside researchers couldn't reproduce when they tried, with high temperature superconductors, which were widely replicated and even improved on within weeks of publication, is a good example.

Another related rule is to compare the size of the effect or recommendation with the size of the experiment. If somebody makes a bold policy recommendation based on a handful of data, they're usually talking through their hat. It's only when there's mounds of data pointing in the same direction that it's worth changing your life over it. I see this a lot with nutrition studies, which recommend dietary changes based on a single paper that studied mice. That kind of thing is almost always nonsense.

[Steve D]

A few additional thoughts. There is an implied suggestion in this post that the conclusions arrived at from "good" science (or even perfect science for that matter should it exist)is immune to herd mentality, personal biases or non scientific/political influences. The may be true. But in my experience in the modeling world, I seriously doubt it.

shrug..

[John Camp]

Jeez, I wish you hadn't written this column. I tend to get locked into to reading the replies (I know, that's a personal problem) and already, some of the replies have struck me as something I can't say because Mike won't print it.

[Roger Moore]

@Dave Sailer: "Science is not a collection of answers, it is a way of asking questions,"

I like to say that it isn't about what we know, it's about how we know it. It's one thing that I really wish our schools taught better. You'll learn far more about real science by digging deeply into the development of a single theory- what was the old theory and the evidence that supported it, what was the new theory and the new evidence that supported it, etc.- than you will by memorizing reams of facts.

[Hans]

Bravo Ctein. Thank you for this column.

We live in peculiar times. When relatively large percentages of the population believe in such things as Bigfoot or alien abduction or haunted houses, or any of the other wacky assertions in wide circulation today, it's clear there's a problem. Sometimes clear thinking seems in woefully short supply. Perhaps it's the ubiquitous availability of easy mass communication, giving every crackpot a chance to gather their flock of gullible disciples, that's at fault. Or maybe certain talk radio ranters and television talking heads are to blame, given the way they use their soapboxes to turn certain aspects of science into political footballs. Or maybe it's simply that our education system has failed us. Attempts to turn school curricular into vehicles of political and religious indoctrination certainly don't help matters.

Then again, perhaps our times aren't so peculiar. Attempts to subvert or discredit science are a common refrain through history. There will always be crackpots and con men and shysters and charlatans who pursue their agendas and peddle their wares. And, thankfully, there will always be those who stand against them.

Truth will out!

[Ed Hawco]

Ctein, I don't know if you are on Facebook or not, but if you're not, I advise you to continue not going there. Facebook is an utter cesspool of bad science and mis- and dis-information, often dressed up as "info graphics" or text overlaid on photographs, etc.

Topics include everything from global warming, fracking, salaries and pensions of politicians, all forms of religious claims, and saccharine-sweet statements about "family" and "mothers" etc. Authors are rarely named, sources are never cited, and items are never dated. But oh, are they "shared," over and over again. Because that's what they're for; so Facebook can gauge your "interests" based on your "sharing profile," to better serve you ads.

For a clear and independent thinker like yourself it is a fast track to utter despair and misanthropy. Stay away! [/rant]

[Andrew]

"Next time I'll pick up with the genuine nutters and crackpots, some of whom are genuinely dangerous."

It sounds like you'll be going after the anti-vaccinators. I hope you'll be able to persuade at least a few people to stop promoting bad science like trying to link vaccines with autism. Bad science really can be dangerous.

[John Krumm]

It's always a pleasure to read thoughtful discussion on a supremely controversial and urgent issue such as climate change, and TOP is one of the few places where I've seen it. I know we aren't supposed to veer into politics in this discussion, so I won't, except to say that as far as I can tell, bad political and economic organization will have to addressed as straightforwardly as "bad science" in order to make progress with this problem. I think it's unsolvable with our current approach.

[Phil Allen]

Congratulations on your successful tenure on TOP and to another thoughtful piece. Every student in elementary science classes should bring this home to family and friends to read.

[Mike Johnston]

I'd like to bring up one more relevant point, which is that controversies commonly surround subjects that laypeople believe they can understand intuitively. No one argues about subatomic theory or cell biology or the lifespan of stars as matters of public discussion, religion, and politics. But when evolution and climate change come up, everyone's an instant expert because everyone thinks they understand the subject even though they haven't learned a single thing about it. So, for me, another "pointer" toward conclusions that are likely bad is the question, is it something about which the average uneducated person will automatically think he or she has real insight into, even in the absence of any actual knowledge?

This has real implications for photography, too, because everyone thinks they understand everything about every picture they see. When I taught photography, I used to have a standard slideshow lecture that would simply demonstrate to people that they could look at picture after picture and simply not see what was right there in plain view. It was meant as an antidote to the intuitive idea that to understand any picture, all you've got to do is glance at it.

Mike

[Mike R]

Ctein,

Thank you for this post. I look forward to your next one.

I have some ongoing tussles with my brother-in-law, who happens to be fond of quoting "some columnist at Forbes or the Wall Street Journal." He calls himself a skeptic, and says he doesn't believe in the scientific community's theories. I wonder if the concepts of "theory" and "skeptic" might be worth addressing in your follow-up?

[ctein]

Dear "Goff,"

Oh. My. I am honored to meet you, sir. Seriously. I am a huge fan. So is my housemate, Paula (I only have a physics degree,; she has degrees in physics, geology, and hydrogeology).

(Readers of Stephen King novels start to become fearful at this point.)

I do not envy you your task and I'm sure glad it wasn't mine. My very first research back in college involved establishing and administering interdisciplinary environmental research programs. And editing the papers they produced for publication, so I had to develop some reading competence across a wide range of disciplines. It allows me to keep up with what gets published in Science and Nature (the only two journals I see with regularity); it also means I have every appreciation of just how complicated and difficult this work is.

But, then, as a slogan I am very fond of puts it, “You knew it was a silly job when you took it.”

pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 
======================================

[adamct]

I agree with most of what Ctein wrote, and can only underscore what Andrew Molitor wrote in his Featured Comment.

But I think you can abstract this even further. People should apply the same critical reasoning skills to science, that they apply in their daily life. Far too often, people feel intimidated by science, so they just blindly accept what is fed to them, without applying basic common sense. Here are a few common-sense rules:

1. Consider the qualifications of the person who is speaking. No, professors at prominent universities aren't always right. But as a layman just trying to compare competing claims, I am just looking for the person who is more likely to be right. And Dr. Schmidt at MIT or CalTech is more likely to be right than Otto, the guy who bags my groceries. [This is another way of phrasing Ctein's point to "get as close to the source as possible".]

2. Look for corroborating evidence / opinions. Dr. Schmidt is going to look more convincing if Dr. Jones (from another prestigious university) agrees with him. And they will both look more credible if they can point to an independent body of evidence that supports their claims.

3. Humility and honesty matter. I'm far more likely to believe someone who points out the limitations and flaws in their science, than someone who claims to have indisputably cured all forms of cancer, with no possibility of error. People can fake modesty, of course. But for some reason wackos tend to disavow any possibility that they are wrong more frequently than those who appreciate complexity and uncertainty.

4. Look for "agendas" and conflicts of interest. This is partly Andrew's point about money, but there are all kinds of cases where people push a questionable idea in order to support their larger agenda, or because they benefit somehow on the back end.

This isn't unique to science. You do it all the time in your life, so why not with science?

Who is more likely to be right on a matter related to photography: Mike or a random, anonymous commenter in a DPReview forum?

If 5 reviewers say a lens has a design defect that causes prominent flare and ghosting, then post pictures to prove it, are you going to believe them, or the guy who claims he has taken 10,000 pictures of the sun and never seen any flare whatsoever (but doesn't post any pictures)?

If a reviewer notes that he has seen a problem a couple of times, but that he can't seem to reliably reproduce the problem and/or hasn't been able to identify the exact combination of factors that causes the issue, and hasn't been able to secure additional copies of the lens for testing, then that person is intuitively more trustworthy than someone who claims he has studied the lens design and declares it is physically impossible for the lens to ever produce contrast-degrading glare under any circumstances.

If your local camera store salesman tells you that the $6,000 Nikon D4's high-ISO performance is vastly superior to that of the $2,000 D700, will that carry the same weight as if Thom Hogan tells you the same thing?

Note that none of the four factors above are scientific factors. I mean that in two senses: (a) Two Ivy League professors with excellent academic credentials and no obvious conflicts of interest who admit to the design flaws in their study aren't necessarily right or practicing Good Science, and (b) You don't have to be a scientist to do a quick-and-dirty assessment of the likelihood that some scientific claim is based on Bad Science.

Best regards,
Adam

[ctein]

Dear Svein (and others),

I think I may have created a bit of the misimpression. I am not anti-religious. In fact I AM religious. I am not making a veiled dig against religion. I just don't mix the science and the religion.

(And, no, the nature of my religious beliefs are none of your business, so don't anybody bother asking. I did not bring enough for the whole group and I am not going to share with the class.)

If the subject of how techies deal with their religions in various ways interests you or others, I would recommend Brother Guy's book from my column, “Five Books that Will Drive You to Think" http://tinyurl.com/c963n75

If not, move along, move along, nothing to see here [grin].

pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 
======================================

[mastaliu]

"Make a note: mixing science and religion? Always a bad idea for both of them."

Ctein, this statement is rather ahistorical and reflects the false opposition between "secular" science and faith and religion, an opposition deeply embedded in modernity and the history of Western science, which posits science as good and objective, and religion as bad and crazy and myth (thanks to Marx and many others). It was the Enlightenment, and certain ideas that later became Christianized (during the reformation), that made science "secular", and in doing so, some have argued, separated science from the project of liberation for humanity. Most of us certainly feel this is an issue in contemporary times, but actually, throughout history (both"Western" and others), science and religion have walked hand in hand in tension and collaboration, and so it's not that they are somehow completely incompatible. All anyone remembers is Galileo vs church, ignoring the periods of significant convivencia in Spain, Palermo, Levant, Ottoman Empire, etc. Periods in which science, religion, and philosophy intermixed.

Next, the Good vs Bad science doesn't help us to understand science as a knowledge like any other - a knowledge that is socially constructed, and so, has its own cultures, languages, rules, norms, politics, violence, etc. which both change and persist through time. It is its awesome power to transform nature, and "demystify" the world that fetishized science as something above nature (think nuclear), above social construction, giving it an aura of objectivity (thanks to the prevalence of positivism), as it demythologized the world around us. When we take science (particularly the natural sciences) out of this false neutrality and position of prestige, and work through its genealogy and history and its entanglements with imperialism, capitalism, colonialism, and all the good stuff too (antibiotics!), we can then sit down and think about what is it that makes good science? Who does it serve? Is it its service to human kind? How it liberates us from ignorance or prejudice? How it alleviates the worst of our follies?

[ctein]

Dear Steve D, Mike R, and others,

Yes, some definitions and further clarifications do seem to be in order.

“Settled science” is a term of art. It is not a guarantee of truthiness; nothing in the scientific process ever is. On rare occasion it will prove to be wrong. Just the same way on rare occasion Bad Science will prove to be right. Proof of existence is not a good guiding principle-- on occasion, someone wins millions of dollars with a lottery ticket. It's not where you should be putting your money.

What it means in practice is that until someone comes forth with compelling and substantial **new** data, the discussion is over. And I do mean NEW. Monday morning quarterback rehashing of old results that have already been looked at doesn't cut it, nor does the losing side continuing to whine that they didn't get a fair enough hearing. It's done.

And, pragmatically, it turns out to be right enough a huge majority of the time.

“Theory” is a difficult word because it means something different in English and in science. When people say they have a theory or that something is just a theory, what they're saying is what scientists call an hypothesis. Essentially, it's a notion, a half-baked idea backed by a modest amount of data that led them to the idea in the first place. Then they have to go look for a lot more data to find out if that half-baked notion should be fully baked or just tossed in the rubbish bin. Most hypotheses end up getting tossed. If a substantial body of data really convinces them that this is a good explanation for things, then it gets to become a theory.

In other words, the word “theory” in science has more in common with the word “theorem” in mathematics than it does with the colloquial English usage. (Of course, in science you never get absolute proofs the way you can in mathematics, but I think you get the idea.)

Although it hasn't come up, I should probably mention that a “Law” of nature is something else entirely. That's just something that we have always observed to be true. In other words, we think it's a correct fact about how the universe operates, but it doesn't derive from an hypothesis or a theory. And those are subject to change, although it doesn't happen often. When it does, it's usually accompanied by a major breakthrough in scientific understanding. Up until the early twentieth century, scientists believe there was a law of conservation of mass and a law of conservation of energy. Then Einstein unified those two quantities and now we believe that there is conservation of mass and energy, combined. But, maybe tomorrow (although not likely), someone will observe something that violates the conservation of mass and energy.

I can't define the word skeptic because in my mind it's become inextricably tainted by scientismists, scientific materialists (not the same thing), and conspiracy nuts of every shape, manner, and form who try to elevate their belief system with the lofty pronouncement that they are merely being “skeptical” of the position they disagree with. Honestly, it's a word I've never personally heard anyone apply to themselves in a way that makes me respect them intellectually. So I'm not the person to ask for an unbiased definition. When ah heah someone claim they're a "skeptic," podner, ah reach for mah gun. (I just got back from Texas.)

pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 
======================================

[FieldCameraFromBoston]

Am I the only the only reader who thinks anyone who's thought about these issues is bound to find this piece lecturesome and unsubtle? Who is the simpleminded Good Science vs. Bad Science approach intended to charm and edify?

[Um, did you read the comments? Seems like a lot of people are finding it worthy of discussion. --Mike]

[rnewman]

I'm a little surprised that no one has mentioned a major driver of Bad Science (and even more mediocre science) - the university drive to 'publish or perish'. No academic who wishes to obtain tenure, or advance to full professor, can do so without a laundry list-sometimes hundreds-of publications, at most schools. Related is a relatively recent proliferation of scientific journals, both paper and online. Publish or perish places pressure on journal reviewers to accept (with minor revisions at most) many unworthy papers. Further, the grant process, even from NSF, NIH or other independent source is not free from bias. Grant proposals are reviewed, and some reviewers have their preferences and dislikes. The competition for the grants is heavy, Many more are submitted than can be granted. So researchers may shape their proposals to fit the biases of known reviewers. And those not granted may actually be scientifically more valid than the winners.
Another factor driving bad science can be the source of research funding. A number of studies by climate change deniers (for example) have been the companies and industry associations who have contributed heavily to CO2 and pollution generation. While that source doesn't automatically mean that the studies are wrong, or biased, it does mean that the study needs to be very thorough in describing its methods and data sets, if suspicion is to be avoided. In some cases, these studies have not stood up to review and replication. Some have, which points to needed areas of further exploration and resolution.
One thing good research isn't - easy.

[ctein]

Dear S. Chase,

Your particular ox was only one of three I was goring in this column. It's not all about you. It's not even majorly about you.

If you can't stand to read all the good advice despite that, not my problem.

~~~~

Dear Steve D.,

Oh, the scientific process (and scientists) is absolutely not immune to any of that! What makes the process remarkable is that it tries to arrive at correct and useful information knowing full well that it is done by humans who are always susceptible to these influences, in fact assuming they are being influenced by those.

Well, that's not what makes it remarkable, what makes it remarkable is that it works so well almost all the time.

I consider one of the most interesting questions in the philosophy of science to be the one of why Occam's Razor and the modern scientific process work as fabulously well as they do. There's no a priori reason they should. (Well, not unless you want to get into some sort of unprovable faith-based explanation.)

~~~~

Dear Ed,

I have a placeholder page on Facebook because I wanted to meet the other three "Ctein"s in the world. (Yes, really!) But it's not where I hang out. I have to retain what little sanity I have.

~~~~

Dear Mike Johnston,

I think this is a most excellent point (laypeople's believed-intuitive understanding vs. controversy).

~~~~

Dear Andrew,

You will not be disappointed.

pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 
======================================

[blackmagic]

Dear Ctein "Throwing out data because you don't like the results when you can't see anything wrong with the experiment is poor process."
That's clearly a physicist-centered view. Softer sciences, such as chemistry require a more relaxed approach because there are just too many variables and it is impossible (impractical?) to understand all the sources of error and influences.
For example, we've seen many times chemical reactions that stop working or behave different depending on the reagent supplier. Even if 99.9% is the stuff in the label, the remaining 0.1% could change everything. Is it worth analyzing the tiny impurities?, most of the time no, just throw away the bottle and get the good one. Sometimes is that 0.1% what would give you the really cool result, but that's another story.
Crystallizations are also a classic example of black arts... and the problem with biology is even worse. We do not deal with ideal gases, or spherical cows in vacuum. :-)

[HD]

Also Irving Langmuir's classic talk on "Pathological Science."

[ctein]

Dear Roger Moore,

Indeed, excellent points!

~~~~

Dear Bill Tyler and Tom Clifton,

Now we're into geology and paleontology, which have really horribly messy and complicated data. They're sort of at the far extreme from ordinary physics. In fact, geologists are taught a formal methodology called “multiple working hypotheses.” In cleaner sciences, you use data to winnow down sets of hypotheses pretty quickly, converging on the one the data fits best, and that's the one you run with until it breaks. In geology, you maintain all those hypotheses in the back of your head until data overwhelmingly discredits one of them. This is why it's so frequently hard to get a straight answer out of a geologist on what you think is a simple question (believe me, I know). You'll ask them to explain X, and instead of giving you a simple answer what you'll get back is, “Well, I think it's possibly..." (or if you're really lucky, probably) " due to this, but you know it could be due to that, or the other, and I can't say for certain yet."

It can take a very long time to arrive at “settled science” in these fields and then it may get overturned decades later.

I asked Paula if cold-blooded dinosaurs were ever considered “settled science” or just the majority opinion, and she honestly didn't know, but her specialty wasn't paleontology. Any paleontological readers out there who happen to know the answer to this? Just curious.

She mentioned in passing, though, before I read your comments, that seafloor spreading (and the ensuing plate tectonics) was accepted at breathtaking speed as geology goes. By far the most rapid acceptance for a new and rather radical geological theory, that she knows of.

But note that the idea of "continental drift” goes back to Wegener in the 1920's. The reason it was dismissed was not because it didn't make sense or explain a hell of a lot of observations about the world, it's that no physical mechanism existed that made it possible. The Earth's crust isn't like thick molasses that the continents can move through, even on a geologic timescale. It would've broken. As attractive as the idea was, you just couldn't make the physics work. Until seafloor spreading and then a whole bunch of stuff came together.

The Alvarez/KT case is an interesting example. Paula and I were lucky enough to be at the AAAS meeting in San Francisco where Walter presented his paper. We both have strong physics backgrounds, and as soon as we heard it we were as positive as anyone could be, based on one paper, that this was right. Because it would be insanely hard for any other theory to come up with those same physical results. But the paleontologists and geochronologists were a lot slower to get on board, because their data couldn't confirm a sudden, single event. And you had to admit the timing of the Deccan Traps was damned suspicious.

Paula said she would classify the KT extinction as “settled science” in geology today, but it was a lot of work getting to that point.

Now, you might be inclined to favor that hypothesis from the beginning (as we were) because the Alvarez' physics and measurements are really nice and clean and paleontology has always been really messy (and until recently so was geochronology). So you favor the more solid data, right? Except in geology, that sometimes turns out to be wrong. Geologists love to make fun of Lord Kelvin for having authoritatively claimed that the earth was a few tens of millions of years old based on his heat flow calculations. The geomorphologists and paleontologists just knew that couldn't be right. There was ample real-world observation that said the Earth had to be a lot older to look the way it did. But it was all that messy and fuzzy stuff. And it wasn't just Kelvin. When other researchers used other methods to calculate formation ages, like the rate at which the sun had to be cooling (who knew from fusion, let alone radioactive decay in the Earth's core?) or the degree of saltiness in the oceans (recycling mechanisms simply hadn't been discovered yet), they all came up with the same age range. So, simple, straightforward multiple lines of evidence in geology led you to the same conclusion, which was in contradiction to the fuzzy, messy lines of evidence. And it turned out the latter were right!

This is why it's hard to get a straight answer out of geologists. [grin]

pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 
======================================

[Roger Moore]

@Mike Johnston: I'd like to bring up one more relevant point, which is that controversies commonly surround subjects that laypeople believe they can understand intuitively.

What's probably more important is that they tend to crop up when there are intrenched interests with a strong desire to dispute the theory. That could be religious leaders who think the theory conflicts with their understanding of the Bible (e.g. the Catholic Church vs. Galileo) or when there's some kind of moneyed interest that is worried about loss of business (e.g. Tobacco companies disputing cancer research). That kind of concerted lobbying by an entrenched group is a giant flashing sign for bad science.

[Eric Brody]

As a pediatrician, I look forward to your comments on those who are against immunization. I also recently worked on the fluoride campaign in Portland, OR where the "anti's" won the day with lots of terrible science. These folks, the anti-vaccine and anti-fluoride people, are those who simply BELIEVE, and there is NO argument that will convince them to change their mind. That attitude defines bad science. Great column which, unfortunately will not be read by those who most need to read it.

[Mike Johnston]

"These folks, the anti-vaccine and anti-fluoride people, are those who simply BELIEVE, and there is NO argument that will convince them to change their mind. That attitude defines bad science."

Eric,
Reminds me of two books: Chet Raymo's "Skeptics and True Believers," and Eric Hoffer's "True Believers: Thoughts on the Nature of Mass Movements." A book, pursuant to your last point, that should be more widely known today.

It seems to be a characteristic of the "follower" mentality, and it's easy to see why that's evolutionarily adaptive: tribal bands with loyal followers are probably more likely to survive.

Mike

[ctein]

Dear mastaliu,

I apologize. I did not mean to create a diversion, here. I hope my comment to Svein and my featured reply to Chris (which should be up by now) should make my position clear. I hold both scientific and religious beliefs. I just don't mix the two–– you can't base one on the other. In either direction.

Check out Brother Guy's book. He does a fabulous job of laying this all out. (Would you expect less of a Jesuit physicist?)

~~~~

Dear “Field,”

Thank you, you who've read my column almost entirely correctly!

It is indeed a lecture. I am standing at a podium and telling laypeople how to do things. And I am not being in the least bit subtle about it, because I don't want there to be any mistaken impressions created. It's direct, to the point, and I am not mincing words any more than civil discourse requires (which Mike would enforce, if I didn't).

In addition, I am not trying to charm anyone. I'm telling them what's what. I'm not using my winsome guile and devilishly gorgeous boyish looks (see me fluttering my eyelashes?) to seduce them with my particular flavor of artificially sweetened beverage.

In the negative column, though, you did miss the fact that many readers are finding this edifying.

But you got three out of four right, which shows that you really are careful reader. I appreciate that.

pax / Ctein
==========================================
-- Ctein's Online Gallery http://ctein.com
-- Digital Restorations http://photo-repair.com
==========================================

[T Bannor]

Thanks Ctein, I appreciate your column. I'm a former research molecular biologist. The bad science people buy into just astounds me. And it happens on both sides of the political spectrum.

When we move into our place about a decade ago, we receive frequent door knocks from our local fundamentalist/creationists. I always engaged them in polite conversation and then asked questions like, "How did both saltwater and freshwater fish, not to mention marine mammals survive The Flood?" They stopped knocking.

We also have very active anti-vaccine folk in these parts. I engaged one who spouted the lack of autism in the Amish is due to their not getting vaccinated. It couldn't possibly be the multitude of other modern things they don't do in living an 18th century lifestyle.

And a shout out to Randal Jaffe. Funny who you run into on TOP. Long time no see.

[ctein]

Dear blackmagic,

I was hoping to squeeze another column out of the causes of Bad Science, but clearly that's not gonna happen [smile]; it's an unavoidable topic.

I'd disagree with you on this throwing-out-data thing, but let me make it clear to laypeople that this is not a remotely resolved question. There is lots and lots of discussion these days among scientists on how to deal with the outliers.

Back when I was in college, physicists WERE taught how to detect and discard spurious outliers. That's fallen somewhat out of favor. One reason is that experiments are more complicated and less intuitive these days; it's harder to have a good sense about what is good and bad data. Another reason is that when researchers have measured the historical trends in experimental results in the measurement of things like material properties or fundamental physical constants, they have frequently turned up huge secular drifts in the mean value over time, far bigger than the combined systematic and experimental errors estimated for individual experiments. The drift is clearly not a real physical phenomenon; if the properties were changing that much in that short a period of time, you'd see major real-world consequences. This is clearly experimental artifact, and an indication that we don't understand experiments anywhere as well as we think we do. So the leaning today is to not throw anything out, or if you do throw something out for publication, it's still available in the public database accessible by other researchers, with annotations that explain why you thought it was okay to throw that out.

I totally agree that chemistry and biomedical research gets considerably messier. But my position is that erring in the direction of throwing out data is not a good idea. We know that a great deal of the biomedical papers being published today don't hold up in replication. One of the major reasons for that is that human beings have a bias to looking more skeptically at a datum that conflicts with their beliefs than one that supports them. It's true that scientists are cautious about results that are “too good to be true,” but they're even more cautious about results that are “too bad to be true.” Unpleasantly large number of drug studies are simply failing to replicate, and it's not primarily because anyone is being bad or because of the pernicious influence of Big Pharma money, it's because of tossing out of data that the researchers sincerely believe is wrong and irrelevant. Especially when a large fraction of them simply don't know how to do proper statistical analysis on their data in the first place.

Anyway, that's my position in the debate and why I said what I did. But, again, I would note for the listening audience that it is an arena of hearty debate and I'm simply on one side of it.

pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 
======================================

[Geoff Wittig]

The science of anthropogenic climate change today is in roughly the same place that the science of tobacco and lung cancer were in the early to mid 1960s. That is, it's a settled question scientifically, but enormous resources have been brought to bear to 'manufacture doubt'- to intentionally muddy the waters and create the impression of controversy where none exists.

In the case of tobacco, this intentional obfuscation delayed widespread recognition of the lethal nature of cigarette smoking. This unquestionably has led to the unnecessary premature deaths of hundreds of thousands of people in America alone from lung cancer. The total is in the millions if you include emphysema and heart disease.

In the case of anthropogenic climate change, the consequences of this intentional assault on science will be vastly worse. Indeed, they may over time threaten the very existence of stable technically advanced human society. This knowledge might explain why people who have a good grasp on the actual science are rendered so apoplectic by the 'denial industry' and its enablers.

[Roger Moore]

Re: cold blooded dinosaurs:

I get the impression that this was more of an unquestioned assumption than settled science. The early paleontologists who encountered dinosaurs saw them as similar to existing reptiles, knew that all living reptiles were cold blooded, and logically assumed that dinosaurs were, too. That assumption went mostly unchallenged for a long time because people didn't really think of it as an assumption, more of something they just knew.

That kind of unquestioned assumption is often one of the hardest things to root out because it isn't taught as being an inherent part of the theory. People can be so blinded by their assumptions that they don't notice, discount, or deliberately ignore contrary evidence, letting those assumptions persist unchallenged. It's also a place where all kinds of cultural biases can creep into the operation of science, which can produce some stunningly bad science.

[ctein]

Dear rnewman,

And, following on the discussion of the causes of Bad Science with blackmagic…

There are some inherent problems with Big Science that we don't yet know how to resolve. By “big” I'm not referring to the $100 million experiments but to the 100 million experiments that are done every year.

The first problem is a purely statistical one and the hotter the topic the worse it is. Because hotter topics are better funded and attract more researcher interest even aside from the funding, so there's lots and lots of experiments going on. (It also attracts researchers who may not be entirely qualified to operate in that field, but if that's where the paying jobs are… see my point below.) It's the " broad side of a barn" problem. Even perfectly well designed and exquisitely executed experiments will, a certain percentage of the time, produce erroneous results with a very high degree of statistical certainty by sheer chance.

To explain to the lay audience: suppose you had everybody in the country flip a coin 10 times. About half a million of them would get 10 heads or 10 tails in a row. That's just by random chance. Good luck convincing those 500,000 people that their coin isn't loaded. if you think of each of them as a researcher doing an experiment, they've got no reason not to be publishing their results. In fact they should be publishing unless they have an honest to God reason for thinking their experiment was flawed.

Well, in this case there is a simple sanity check because it's not a very difficult experiment. Just do it again ( often, in real world science, that's easier said than done). About 500 people are going to come up all heads (or all tales) twice in a row. I don't know about you, but if I were one of them, I'd sure be convinced at that point.

This ties into your point, because science is a job. We are long past the days when it was the province of the rich who could simply afford to do it as a hobby (and we would be far worse off if that's all it was). Like all jobs, people need to be paid. And they need to demonstrate that they're worth being paid. It's really easy to say that publish or perish or the pressure to get grant money is a source of the problem, but that's simply a human condition of employment. Anybody working in any job has similar pressures. And most people, working in most jobs, just learn to deal with it. So do most scientists. Yes, some of them will be corrupted by the process, but then so are some building or fire inspectors, some civil engineers, some police officers, and some firefighters.

The thing is, this is unavoidable unless we want to go back to Very Small Science. But it does leave us with some rather intractable quality control issues that are inherent in the system, I suspect. We need new meta-tools for making sure this stays under control.

pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 
======================================

[Michael Steinbach]

I like the Minolta! Wait, what are we talking about here?

[Mahn England]

Bravo Ctein!

However I have only one complaint and I'll preface this by pointing out that I try always to read the comments not only here but also at other places that interest me.

I usually start my daily browsing sessions at one of two areas: Photography or Climate Change...depending on my frame of mind. Today I started at Climate Change and after reading some comments at one place with my blood pressure rising I felt I needed to escape to a gentler place.

My complaint? I came here!

The topic of bad science vs good science is worthy of broader discussion amongst intelligent people regardless of their interests.

[Steve Jacob]

@Ctein

You said "This is why it's hard to get a straight answer out of geologists. [grin]"

Oh, but it's much easier to get a definitive answer out of anthropologists, psychologists and economists - except you will never get the same answer out of any two of them. [Resigned shrug].

Popper. Like Hopper, everyone should know his name. But I bet the number of people that heard of Lady Gaga is probably over 100,000 times larger. And I would bet most of those that do don't speak English.

Welcome to the modern world [grumpy beardy grimace].

[Sarge]

Scientists can learn a lot from NIMBYism of the layman.

Global warming is an issue that must be confronted by NIMBYism writ large. When it comes to climate change, our backyard is the entire planet, our neighbor is all of humankind, and its consequences are being felt in our lifetime. It seems to me that the harder (basic) the science, the further the horizon in terms of space and time, and the more inconsequential the results of Good Science to the here and now.

The global warming debate among scientists has gone on long enough that they lag behind politicians and civil society when it comes to pro-active initiatives to curb greenhouse gas emissions.

There's Good Science and there's Good Politics. It's no accident that the IPCC shared its Nobel Prize with an enlightened politician.

Good Politics is an offshoot of enlightened citizenship, i.e., interest articulation or advocacy on the part of an informed civil society. This can only happen when politicians are accountable to a citizenry who are free to express and organize themselves. Mass education is also a must.

The weakest link in the global effort to curb greenhouse gas emissions is international governance where global consensus can be held hostage by the intransigence of a single-party state.

Climate change is too important a problem to be left alone to the scientists. Good Science ought to be part of the solution rather than the problem.

[steven Ralser]

You can get a straight answer from a geologist, but those are usually the ones who don't have multiple working hypotheses. Like the winnowing of data mentioned earlier, geologist can ignore data which doesn't fit their preconceived ideas. This is especially true of field geologists who can easily "miss" a critical outcrop. I've known some of these, and unfortunately they have been very successful.

[Aashish Sharma]

I'd just like to say Thank You for another great article. It surely made my day! :-)

Your articles are a big reason I look forward to Thursday mornings (yes, it is Thursday morning here in India by the time I get to read it). And I've not been disappointed so far. At worst, I've read through an article about a topic I knew nothing about, and at the end of it thought to myself 'hmm, interesting, but not quite my area of interest'. Still, I've always enjoyed reading every article, even if it only gave me a microsecond glimpse into what might be involved if one chose to dig deeper.

All your articles on TOP could be compiled into a book. Along with the comments you responded to, and your responses themselves. I'd buy that book without another thought! :-)
Best Wishes from New Delhi!
-Aashish

[William Walker]

"They did so, clearly stating that they expected to be wrong and they needed help in finding out how they were wrong.

Of course, they hoped they were right, I mean, guaranteed Nobel Prize and all that. But they were pretty damn sure they weren't."

That, to me, is the beauty of science.

[Roberto]

There is another common misconception which lead laypersons to misjudge science. Scientists are sometimes seen as a gang of devotee, willing to preserve the "established science" against new thoughts from "outsiders". Where it is the opposite: scientists are always looking for something new, and the more revolutionary is the finding, the more is rewarding in term of personal satisfaction and career. Just, it is not easy, any new hypotheses have to be confronted with the known measurements (not the known theories they may contradict). Where all measurements means measurements of all facts that would be consequences of that new hypothesis. And if it fails, well it failed. If I claim the earth to be flat, I'm not contradicting the "theory that earth is round", but the measurements that we make thousand times every day by overflying it. When a scientist dismisses a crackpot theory, it is not because it contradict an established one, but because its consequences would contradict many facts (measurements). And it is this knowledge of the correlation of facts and laws that describe them that makes one of the differences between a scientist and a crackpot.

[DonnaC]

Good discussion by all! However, I believe we need to reserve some caution and be mindful of the new era we are moving into, where Big Money and Big Business are having strong influences on "scientific results". Sadly, WHO paid for the study is starting to influence the results. WHO stands the gain the most, also can influence the results. Perhaps those lone voices in the wilderness that have the courage to question the tainted nonsence being put out by Big Business and Big Money, should now be taked more seriously than ever before.

[Ctein]

Dear Steven,

Well, Paula's a field geologist, and I'm not sure she'd agree with your blanket characterization of them. Me, I don't argue with people who know how to wield Swiss Army knives and rock hammers!

Geologists don't HAVE to use MWH; it's just they'll be wrong most of the time. But if they're really right when they're right, they can be quite successful. You just can't bank on them. Tommy Gold's a good example, in my mind, of someone who's mostly wrong, 'cept he's been so *right* when he's been right.

pax / Ctein

[Tim]

Ah, generally a great article. Thanks for setting the record reasonably straight.

One tweak to be wary of: Did your informants submit papers for peer review? Did they get published in reputable and well-established journals or websites?

Well yes, but beware scope for bias and old-boys'-club-ism in peer reviews: http://en.wikipedia.org/wiki/Peer_review#Allegations_of_bias_and_suppression

[Boyan]

As the comments demonstrate, attempts to draw black/white boundaries in gray areas are always bound to stir passions. I can find many points of objection with the original article, but like most of the comments they are likely to be more nitpicking around the edges that substantive additions to the discussion.

I do agree with the notes from previous posters that discounting results from newcomers to a field solely on the basis of their lack of prior experience is a dangerous shortcut. I am also surprised that nobody has yet picked on the evolution vs biochemistry in Ctein's article. Evolution is likely as close to "settled scuence" as you can get when it comes to questions like opposing thumbs and the RELATIVELY minor differences between species. However, when it comes to the biochemical underpinnings of life there is a lot that evolution leaves unanswered, and discarding this as "subtleties of molecular biochemistry and statistical selection processes" is, IMO, a gross undersimplification. It is not an argument for creationism, as some people make it out to be, but it IS an argument against pushing evolution as THE truth about life.

[Chris Lucianu]

Mike, I'd like to thank Ctein for responding.

I'd also like to address an essential point he made in his response: recursive thinking.

The computer language book that influenced me most durably is Dan Friedman's The Little LISPer (now updated as *The Little Schemer*). From the preface:

Designing a program requires a thorough understanding of data; a good program reflects the shape of the data it deals with. Most collections of data, and hence most programs, are recursive. Recursion is the act of defining an object or solving a problem in terms of itself. [my emphasis]

The goal of this book is to teach the reader to think recursively.

I agree with Ctein 100% that recursion is a great tool in teaching, and an indispensable skill in learning.

But: those adept at recursive learning tend to be self-recruiting (as any reader perusing The Little LISPer will realise). Those most likely to be afflicted by the "longissimus, non legi" syndrome (nice!) are also least likely to undertake recursive learning.

Dilemma: those who would benefit most from Ctein's lessons are those least likely to ponder over them.

[ctein]

Dear Tim,

Peer review has its problems, but this is still a better rule than not. Especially for laypeople. The vast majority of the time, it will guide them correctly.

Remember, none of these meta-rules are guarantees of anything. They just tell you what the odds strongly favor.

~~~~

Dear Richard,

Remember the authors' dictum: The Editor is God

Frequently a petty, malicious, vindictive and utterly cruel God. But God nonetheless.

~~~~

Dear Aashish,

Thank you for the wonderful compliments. I have thought about compiling columns into a book. But there is a legal question I haven't yet investigated. The chapters would have to include, as you note, the comments and my responses to them, because they not only greatly expand upon and inform the original article, but frequently include corrections to what I wrote originally.

The legal question is who can provide me permission to use those comments. The people who wrote them own a copyright to their comments. Mike also owns a compilation copyright to the entire website. So the question becomes, and this is strictly a legal question, whether his compilation copyright is sufficient for him to license me to use the material in the form in which it's been published on his website.

That's a yes-no question. If the answer is unequivocally yes, then it would just be a matter of whether Mike wanted to do this or not (I suspect he would poll the readership to find out if it were okay with the majority of them before making that decision).

If the answer is possibly no, then it's an impossible project. Not only would it be infeasible, in terms of time and effort, to get everyone's permission; many valuable commenters post anonymously–– a pseudonym and no return e-mail address. There's no way to contact them. Publishing some comments and not others would break the narrative flow unacceptably.

Anyway, I'm not ready to take on the task, so I haven't talked to an attorney about the primary legal question. But I have been thinking about it.

(Important note: I'm not interested in readers' opinions in this matter unless they are actually copyright attorneys and wish to provide me a professional consultation, for which I will happily pay. In which case, please e-mail me rather than post here.)

pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 
======================================