Tuesday, September 14, 2010

Do we know what we don't know?

How good are we in estimating the uncertainty of our claims? Pretty bad in my opinion. And this may be particularly true for scientists, medical doctors or other experts. Since they have noticed that there are very few people who know more than them on a (very) particular topic they infer that they may actually know close to everything there is to know about it. I recently watched in interview with a renowned physicist about the possibility of EPS (extra perceptual sensation). This expert embarked on a long story about how all physical laws decay as one over the square of distance and that therefore the signals necessarily underlying ESP would have been detected. Problem is of course that his reasoning was solidly rooted in the physical laws as we know them and that the possibility of entirely new physics causing the phenomenon was simply denied. A clear overestimation of ones grasp of the unknown.

Ever tried to argue with a doctor why your child needs Tylenol for a mild fever? They will almost consider you criminal if you choose to deny them the medicine. But there is never a clear reason as to why they need it. It's simply the way it is. But do they have any clue as to the long term effects of poring medicine into these small bodies? Yes, of course, Tylenol was rigorously tested and approved but it's almost impossible to test for the increased risk of cancer after 20 years. They seem completely certain it's safe until a new study shows it's not (as was indeed the case with Tylenol). Why are doctors so certain about the effect of drugs or vaccinations: because they chronically overestimate their grasp of what is unknown.

Perhaps the clearest example is given by reviews of scientific papers where reviewers are asked to provide their confidence. It is very common to find two maximally confident reviewers with completely opposite opinions. Clearly one of the two must be wrong. And, this is of course very frustrating at the receiving end.

Bottom line, always keep an open mind and try not to underestimate what you don't know.