The problem with anecdotes

We’re funny creatures, human beings – very easily swayed, and ruled by emotion a great deal more than some might believe. Most people like to think that they are quite rational, quite sensible; most people probably (secretly) think that they are more sensible than the average person. And yet we also believe some quite remarkable things.

Take homeopathy as an example; I have spoke to more than one very bright, intelligent person who has said words to the effect of “I know that homeopathic remedies don’t contain any active ingredient, and it’s probably a load of rubbish, but they definitely work for me”.

One of the most seductive forms of persuasion, for some reason, is anecdote. We see them all the time, often by another name – a testimonial, a case study, an interview – but it all means the same. Anecdotes seem to be a powerful way for people to communicate ideas – they make them relatable, and understandable. They also add a generous splash of emotion to the issue, and that’s not a particularly good thing

Because the fact is this: anecdote is not reliable.  While we may think that we’re rational and swayed only by hard evidence, actually we human beings are all too easily tricked by all manner of things.  Among the most common is confirmation bias, which is the name given to the very common scenario where we take action in the hope of producing some result and, if that result is obtained, assume that it was our action which caused it.  There are lots and lots of very common examples – taking Echinacea for the common cold (or homeopathy for that matter), or arnica for a bruise.  These things may or may not work, but the very fact that we expect them to have an effect means that when the inevitable happens and our cold clears up means that we will inevitably attribute that result to the remedy.

This is also a great example of the very similar concept of “regression to the mean”, which simply means that lots of things have a very predictable nature, and tend to get better on their own.  Everyone knows that a cold will go away whether you treat it or not, but if you did happen to take that Echinacea tablet, isn’t it tempting to think that you helped yourself get better?

And that’s why anecdote is not reliable; these and many, many other biases and problems come into play, so that in most cases we have no idea if the “treatment” worked, or sometimes even if the person was sick in the first place.  A fellow called Dr Moran, a retired surgeon from Australia, has a really rather lovely little website that gives us some tools to try and remain vigilant against this kind of thing.  He has an article called How to Evaluate a Cancer Cure Testimonial, but really it just takes a few minor tweaks to make it into “How to evaluate a medical anecdote”.

So here are my golden rules:

  1. Was the person definitely ill, as shown by reliable tests, when the treatment was started?
  2. Did they get better, as judged by the same tests?
  3. Was the advocated treatment the only one used?

I haven’t really covered that 3rd point here, but it’s probably self-evident: if you did more than one thing to help yourself get better, how do you know which one worked?

And that’s it.  It’s perhaps not particularly important if you’re just thinking about taking a herbal medicine for the common cold, but in other cases it might be.  I’d urge anyone thinking about investing significant resources into a treatment (whether time, money, emotional input, or anything else) where the only evidence is an anecdote, testimonial, or similar to think through those three points.  Be honest with yourself about the answers.  You might just save yourself some cash, or some heartache.

Advertisements

Posted on October 12, 2012, in Evidence, Skepticism and tagged , , . Bookmark the permalink. 2 Comments.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: