At one of the nutrition seminars we were discussing evidence and/or/vs. context. What is evidence? What is context? Good stuff like that.
One example involved clinical trials of giving vitamin A supplements to infants. There are only 3-4 studies, but they found that it made a big difference for Asian infants but none to African infants. The poorly-done inference is to say "Context matters and Asian infants are different from African infants." But in what way? Why? And how can you draw such sweeping conclusions from what really is a difference between only two studies?
In discussing the incorporation of context into evidence-based policymaking (the topic of the seminar series this semester), a few people expressed reticence to incorporate contextual factors because many of them are a) inherently unknown b) not my field c) only confuse "the issue." When the discussion moderator asked why we were uncomfortable dealing with context, there were few responses, so I chimed in.
One of the insights from Libertarianism is that we are all inherently outsiders to most everyone else - we don't know their context and cannot really know them. The more important you think context is, then, the less we can say about anything and the less we can let ourselves do from a policy standpoint. So if we have to "do something" or be seen to be "doing something" we have to push the context aside in order to make policy recommendations.
The moderator pushed back that this is just a matter of maximizing utility under conditions of uncertainty (that's how my economist training translated it anyway, not the specific words which I think involved "calculated risks") and we all do that every day.
I countered that there is a fundamental difference between doing that for myself and for others.
I choose to have whole wheat bread with breakfast because the evidence tells me it is probably good for my health and I bear the risks. Then last week I passed my first kidney stone and I learn all of a sudden that this whole wheat bread is bad for my health.
Now I'm a policymaker. I know the evidence says whole wheat is good for people, so I pass some policies to increase whole wheat consumption and many people are better off but I've hurt the kidney stone people.
The more we think context matters, the less we can say in our research and the less we should do in policy. And that's uncomfortable for policymakers. "It's complicated" doesn't win elections.
The fascinating question the moderator posed at the end of things was where the evidence base is for evidence-based decision-making. We emphasize evidence-based medicine as this wonderful breakthrough, but have we done randomized, double-blind, placebo trials comparing evidence-based and non-evidence-based medicine? And can you imagine how to do this for evidence-based policymaking?