The unbelievable truth?

Provocative piece in a recent New Yorker (hat tip to Tactical Philanthropy) about an emerging doubt among scientists about the validity of many published results. The “decline effect” is that many results that initially appear robust and statistically valid (X drug helps lessen symptoms of Y disease in Z percent of patients), when replicated over time, either can’t be replicated, or the effect lessens (Z gets smaller or disappears).

The upshot?

The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe.

Interesting given how much weight is being given these days in philanthropy to randomized controlled trials and experimental design as the gold standard for evaluation, particularly in international development. Reminds us to be humble about our claims.

There are two ways this should happen: one is to be very explicit about our assumptions, and to make them publicly available. This was what I was taught in grad school: describe how you conceptualize, operationalize, and measure your variables, and talk about how you code them. And I studied one of the wanna-be sciences; I’m frankly shocked that such practices aren’t standard in medical research, if the article is to be believed.

The other way to be humble about our claims around evaluation is to triangulate: to put quantitative results in context. Another thing I learned in grad school was to specify mechanisms: in as much detail as you can, describe how you see the causal pathway working between the cause you posit and the effect you’re trying to explain. And harmonize the two: have quant and qual work with each other and reinforce each other.

As a new year approaches, always good to be reminded of the importance of humility. I’m often ambivalent about transparency, for a variety of complicated reasons. This kind of transparency, about methods and assumptions that back up claims of empirical “proof” – this I can get behind.

Here’s to a happy and healthy 2011 for one and all. I’ll resume my regular Tuesday-Wednesday-Thursday schedule next week.


Tags: , , ,

One Response to “The unbelievable truth?”

  1. The Blog Briefly Known as "Democratizing Philanthropy?" » Blog Archive » Is the glass half cloudy? Says:

    […] The Blog Briefly Known as "Democratizing Philanthropy?" BETA version, new title in the works. New post each Tuesday, Wednesday, Thursday (for now). « The unbelievable truth? […]

Leave a Reply