Work is the refuge of people who have nothing better to do. : Oscar Wilde

Saturday, October 15, 2011

Stuff We Do Must Be Based on RCTs

Let's say you know a dyed-in-the-wool, explicitly racist individual who says, "When I meet a person of race X I can be fairly sure that she will exhibit attitude AX, and when I meet a person of race Y I know he will exhibit attitude AY."

If you are like me, you'd probably cringe and put on your best anti-racist mental cloak, then say to yourself, "Goodness, this individual can see (or thinks he can see distinguishing racial attributes) and then he projects his own beliefs about race onto members of these groups. (Fortunately I am completely free of these ways of thinking.)"

Now, let me offer you what I sometimes hear during discussions. An employment advisor with views about the differing attitudes of the various generations of North Americans meets a client of generation X. Whenever the advisor does so he says to himself, "I imagine that this client will have attitude AX in the workplace. Curiously, when I meet clients from generation Y on the other hand they always seem to demonstrate attitude AY at work."

Can you spot the parallel at all?

My point is not that we should ignore individual differences in clients. Of course not. My point is that observations made in uncontrolled, unblinded situations do not constitute scientific evidence. When one meets a client all sorts of beliefs and expectations will suggest themselves on the basis of gender, social class, age and so on.

We need more good scientific exploration to guide our work.

See wikipedia, randomised controlled trials for more. I note that they were first used in psychology where life is somehow more difficult in some ways than it is in physics.

If you would like to be harangued on a regular basis about topics like this then subscribe to Ben Goldacre's Bad Science, or to Tim Harford's BBC 4 More or Less podcasts. Here's something gnarly that you can read right now: A Field, Top 5 Statistical Fax Pas.