Change blindness:
[Via Radley Balko at The Agitator.]
The experimenters note that they don’t know whether this shows differences between the subjects, or just random chance.
Trials aside, this is also relevant to any kind of fact-finding process — including science itself. It’s a big part of the reason that safeguards such as double-blinds and repeatability are so crucial.
I’ve seen demands that police lineups prohibit detectives familiar with the case from participating, from being in the viewing room with the witness.
Suggestion for crime lab directors: don’t hand a sample to your squint and ask “did this come from Mr. I-have-a-lawyer-and-powerful-friends in our holding cell?” (Much less something like, “We really need to find this thug’s blood on this dress, or we’re going to have to let him go.”) Instead, try something like “Here’s six samples in rack A, and six samples in rack B. Do any of the As match any of the Bs?”
Should we talk about attorneys presenting highly refined, well groomed evidence to juries? How about putting jurors through little demonstrations showing just how fallible their perceptions are, how much their prejudices affect their judgement?
Suggestion for these experimenters: Some interviewees should see the same guy both times. The interviewers must not know whether or not a given subject saw a different guy or not. Oh, and “guy”?
Suggestion for climate researchers: don’t do the data analysis yourself. Hand a bunch of datasets, some real, some dummy, some pure random noise to point up biases in your software, to a few statisticians, and ask them to (independently) report any trends they can find in the data. Don’t even tell them the variables or units involved.
I know there are huge problems with the crude approach I just outlined. I understand all too well that a certain amount of fudging and trickery is absolutely necessary during the investigative phase when the researchers may not know what they’re looking for, and are accounting for biases and errors they know their equipment and procedures show — but something like this as a sanity check ought to be required for any research underlying public policy.
[Edit 21 Dec 2009: fix some small problems in word choice and flow.]
Tags: change blindness, climate research, eyewitness testimony, global w, global warming, rules of evidence