Archive for December, 2010

The unbelievable truth?

Wednesday, December 29th, 2010

Provocative piece in a recent New Yorker (hat tip to Tactical Philanthropy) about an emerging doubt among scientists about the validity of many published results. The “decline effect” is that many results that initially appear robust and statistically valid (X drug helps lessen symptoms of Y disease in Z percent of patients), when replicated over time, either can’t be replicated, or the effect lessens (Z gets smaller or disappears).

The upshot?

The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe.

Interesting given how much weight is being given these days in philanthropy to randomized controlled trials and experimental design as the gold standard for evaluation, particularly in international development. Reminds us to be humble about our claims.

There are two ways this should happen: one is to be very explicit about our assumptions, and to make them publicly available. This was what I was taught in grad school: describe how you conceptualize, operationalize, and measure your variables, and talk about how you code them. And I studied one of the wanna-be sciences; I’m frankly shocked that such practices aren’t standard in medical research, if the article is to be believed.

The other way to be humble about our claims around evaluation is to triangulate: to put quantitative results in context. Another thing I learned in grad school was to specify mechanisms: in as much detail as you can, describe how you see the causal pathway working between the cause you posit and the effect you’re trying to explain. And harmonize the two: have quant and qual work with each other and reinforce each other.

As a new year approaches, always good to be reminded of the importance of humility. I’m often ambivalent about transparency, for a variety of complicated reasons. This kind of transparency, about methods and assumptions that back up claims of empirical “proof” – this I can get behind.

Here’s to a happy and healthy 2011 for one and all. I’ll resume my regular Tuesday-Wednesday-Thursday schedule next week.

Share/Save/Email/Bookmark

When glass windows block the light

Tuesday, December 28th, 2010

Interesting piece on GiftHub about the manipulation of online rating systems for political ends:

For his film (Astro)Turf Wars, Taki Oldham secretly recorded a training session organised by a rightwing libertarian group called American Majority. The trainer, Austin James, was instructing Tea Party members on how to “manipulate the medium”(11). This is what he told them:

“Here’s what I do. I get on Amazon; I type in “Liberal Books”. I go through and I say “one star, one star, one star”. The flipside is you go to a conservative/ libertarian whatever, go to their products and give them five stars. … This is where your kids get information: Rotten Tomatoes, Flixster. These are places where you can rate movies. So when you type in “Movies on Healthcare”, I don’t want Michael Moore’s to come up, so I always give it bad ratings. I spend about 30 minutes a day, just click, click, click, click. … If there’s a place to comment, a place to rate, a place to share information, you have to do it. That’s how you control the online dialogue and give our ideas a fighting chance.”

I’ve wondered in the past what the implications of social-media vehicles for expressing preferences – liking, sharing, rating – would be if applied to grant decision processes.  This story reminds us how relatively easily such systems can be “gamed.”

This is a very strange version of the high-school popularity contests that can erupt online around these types of decision-making mechanisms. In this example, the idea is that dogged effort can skew the game of popularity – “give our ideas a fighting chance.” Interesting the worldview of being aggrieved, put-upon.

Given the relatively “elite” perception of foundations, this kind of manipulation seems all too easy to have happen….

What’s mine is yours…maybe

Wednesday, December 22nd, 2010

On some level, my mission on this blog is to talk about the intersection of philanthropy and politics in a way that gets beyond pat political talking points. So for instance, I’ve looked at the way the privileging of local knowledge happens both on the right and left, and some important differences in the way they interpret that idea.

Along those lines, I think it’s important to acknowledge the truth a talking point that sometimes gets rehearsed in discussions around the nonprofit sector. And that’s the idea that small for-profit businesses are actually doing more good for the world by generating jobs for people.

The truth in that is that the nonprofit sector doesn’t have a monopoly on the production of social goods. All three sectors – public, private, and nonprofit – produce different kinds of social goods (defense/security, jobs, services, to name a few) in different proportions, and it seems to me that where communities get out of whack is when the proportions aren’t balanced – when there’s too much of some social goods and not enough of others. I’m reminded of an article in New York magazine earlier this year about communities upstate where juvenile-detention facilities are located: an excess of security that creates jobs but not sufficient inclusion/opportunity.

At the same time, it doesn’t make sense to think of different sectors only in terms of one kind of social good they produce. As Sean at Tactical Philanthropy likes to point out, nonprofits employ a lot of people too, there are plenty of jobs there. I guess the idea would be to take a more holistic view of the concept of “externalities,” which in economics are the side effects of economic activities. A coal plant produces energy but also pollution, which reduces air quality; the reduced air quality is the externality. Such a bastion of capitalism as the Harvard Business Review has started writing about the internalization of externalities from a corporate-social-responsibility and leadership perspective.

I wonder if there’s not also a role for the intentional socialization of externalities – understanding them in local communities in the context of who pays for them and who tries to alleviate them when they’re bad. Picture a lumber town with a small environmental nonprofit that’s a chapter of a national organization trying to work with a town council that’s dominated by the chamber of commerce, which is filled with folks sympathetic to the lumber mill. How does such a community find balance in the production of social goods – jobs, environment, governance? I don’t know that we have a good way to think through those issues, and communities across the country are dealing with them in the wake of significant budget cuts. May be worth taking this topic up again in the New Year….

As I mentioned yesterday, abbreviated publishing schedule these last two weeks of the year, Tuesday and Wednesday only. See you next week, and enjoy the holiday!

The Imp in history

Tuesday, December 21st, 2010

I’ve written about The Imp of the Perverse, an image to describe the impulse in human nature to choose things that we know are bad for us, but choose them anyway. It has important implications for how we understand the implementation of social programs, particularly their uptake (or lack thereof). An unusually thoughtful and well-written piece in the Washington Post makes the point that the Imp operates in history as well:

[T]he Civil War legitimized something essential, and dark, that remains with us. Ultimately, the South was fighting for the right to be wrong, for the right to retain (and expand) something ugly and indefensible. It lost the war, and slavery was abolished. But the right to be wrong, the right to resist the progress of freedom, the right to say “no, thank you” to modernity, to leave the fences in disrepair and retreat into a world of private conviction, remains as much a part of the American character as the blood spilled to preserve the Union. Nothing great has been accomplished in America since the Civil War — not footsteps on the moon, or women’s suffrage, or the right (if not the reality) of equal, unsegregated education — without people also passionately fighting for that dark right, too.

“That dark right” – a chilling phrase. We would do well in philanthropy and the nonprofit sector to remember that in trying to encourage social change, “that dark right” will always be a feature of the landscape.

As a side note, to commemorate the sesquicentennial (love that I got to use that word in a sentence) of the Civil War, the New York Times has a blog, Disunion, that “reports” on events in the war on the day they happened 150 years later. (The economist Brad DeLong has been doing something similar for World War II seventy years later.)

Lighter posting schedule this week and next: Tuesdays and Wednesdays only. Happy Holidays!

The missing leg of the stool

Thursday, December 16th, 2010

Continuing from yesterday: “varieties of capitalism” teaches us that there are different ways to organize a modern economy. The German model, what Hall and Soskice call a “coordinated market economy,” feature a relatively high level of coordination between firms, capital, and labor. They’re able to develop high-quality, labor-intensive goods because each takes a risk knowing that the other is taking a complementary risk.

In the U.S., the high-quality good of strong nonprofit evaluation/learning practices is not frequently produced, and one reason may be that that three-way coordination doesn’t happen. Nonprofits, foundations, and clients/the public aren’t coordinated in the same way. Foundations are rarely patient enough to support high-quality evaluation, nonprofits can’t count on stable financing to do it, and most crucially, nonprofit clients (and workers) don’t have clear incentives to participate in high-quality evaluation.

(I know that just earlier this week I distinguished between monitoring and evaluation. I’m using “evaluation” here to mean those practices of continual learning that allow a nonprofit to understand its environment, its product quality relative to that environment, and make adjustments to its resource allocations.)

There are some different ways to address this. One would be to make it easier for clients and nonprofit workers to gather and analyze evaluation data. This is improving the tools (SurveyMonkey has done this). That mitigates a disincentive, but doesn’t necessarily provide an incentive. Another way is to make evaluation data available to those from whom it’s gathered. This is improving the feedback loop (again, web-enabled tools, including SurveyMonkey, make this more easier). This provides somewhat of an incentive, because you see that your contributions actually result in something. But to have a really positive incentive, there has to be a connection to the person’s life and own interests. And that comes from a clear articulation of the nonprofit’s mission and a strong communication of its value added to its clients and workers. People need to be able to see why having a nonprofit that’s better able to learn is better able to serve them.

And that’s where I feel the real challenge is. It’s a missing piece in what could be a greater form of coordination among nonprofits, foundations, and the public to produce the kind of evaluation data that would make it easier for all involved to know that the work being done was having real impact.

Mercedes-Benz and nonprofit evaluation

Wednesday, December 15th, 2010

Last week I talked about intentionality and coordination, and expressed my dissatisfaction at market-based metaphors that would make it difficult for philanthropy to proceed in terms of “coordinated voluntary participation.”

Well, it turns out I’ve written a fair amount about coordination in the context of “varieties of capitalism,” the idea that the U.S. mode of doing business is not the only effective one. In addition to the “liberal market economies” of the U.S. and England, there are “coordinated market economies,” like Germany and Japan, where there is more intentional coordination, and the pieces of the employment/jobs system fit together more: vocational training fits with trade unions that have strong, long-term relationships with employers, who are willing to take risks on high-quality, labor-intensive products (think Mercedes-Benz) because they’ll have a supply of high-quality workers and relative labor peace.

I’ve also said that philanthropy and the nonprofit sector in the U.S. have a weird hybrid model of LME labor relations and CME finance. What would a fully-CME enclave, perhaps at a state level, look like in the U.S. nonprofit/philanthropic sector?

It would have to involve institutions working together to enable mutual risk-taking, on a path toward higher-quality, more labor-intensive outputs. Like, say, more attention to measuring results. What the German economy has to teach us is that all the parts need to fit together and assume some of the risk. (That means you, philanthropy). Firms (nonprofits) will produce higher-quality goods (better evaluation) if they can be assured that capital and labor will play nice; that is, that capital will be patient, and that labor will be high-quality enough to make high-quality goods. Foundations need to be patient with capital for evaluation, and be willing to pay for it, AND there need to be incentives for labor (in this case, the clients or nonprofit workers who provide the data for evaluation) to participate in this. German unions go along to get along because they’re more or less guaranteed job security and a steady flow of trained young people from the vocational system. That’s what’s missing in the U.S., the equivalent of the German-labor side of the equation. Where’s the incentive for nonprofit staff or clients to collaborate in the gathering of high-quality evaluation data? We haven’t made that connection (or rather, most orgs haven’t), and so the coordination breaks down.

What might those incentives look like? To be continued….

“Despite all my rage, I am still just a rat in a cage”

Tuesday, December 14th, 2010

Rachel Strohm links to an interesting piece in the Boston Globe on the trend toward using randomized controlled trials in developing regulations and legislation. Set aside the fact that it sounds ethically monstrous to deny people homelessness-prevention services for the sake of an experiment. What we really don’t get is how incentives work in a situation like this when the subjects are able to know that they’re subjects. The article mentions business owners who would act a certain way during the experiment to get the regulation they want. Duh. The other way that people are different from rats (besides, uh, not being rats) is that they can read the Internet. No way this kind of thing flies once the right-wing blogo-talko-sphere gets a hold of it. They’d be singing the Smashing Pumpkins song quoted in the subject line….

Two things though: one is the difference between monitoring and evaluation. One of the proponents of experimentation says in the article:

We study regulations only at “the very moment when we know least about the consequences,” Greenstone says. “There is no culture of trying to understand ex post what the consequences are.”

I would be happy if we studied the consequences of regulations and tried to figure out from there what the mechanisms are that lead people to act in certain ways. In other words, map out the incentives. What do businesses end up doing in terms of providing health care for employees as a result of different pieces of health-care reform? We don’t even get that in many cases; what exactly are we going to experiment on? What’s being evaluated? Let’s start by monitoring and go from there.

The other is the difference between laws and regulation. Laws are passed by democratically elected bodies like legislatures and executives; regulations are drafted by appointed bodies in the civil service. It would seem there would be more incentive to test laws because of the greater potential accountability. Perhaps all the more reason for there to be testing of regulations, to build in greater accountability?

There’s a lot of interest in philanthropy and the nonprofit sector in the use of RCTs in evaluation and the design of programs, so this debate is worth following.

Most abstract blog post title ever: “Intentionality and coordination”

Thursday, December 9th, 2010

Clay Shirky via Lucy Bernholz:

Clay Shirky answers the question of “What will change everything?” with “coordinated voluntary participation.”…philanthropy – the long tail that provides the bulk of the private resources for public good – is all about “coordinated voluntary participation.”

I wonder about this. One of my themes on this blog has been the relationship – you could say the tension – between philanthropy and democracy. What relationship does philanthropy have to the fundamental tension between markets and democracy? That kind of thing.

It’s difficult to think about markets objectively because market-based thinking is so much a part of our lives and language (we “invest” in friendships, etc.). But aren’t they a form of coordinated voluntary participation? The invisible hand, etc.?

And I think one of the more frustrating things for nonprofits who are the recipients of philanthropic capital is precisely the lack of coordination among the voluntary participants. There has to be a better way between total Soviet-style centralization and total US-style decentralization. Maybe it’s social-capital exchanges, but I find it kind of depressing that markets end up being the venue in which coordination happens. They’re intentional at the micro level and unintentional at the macro level. That’s not really coordination, if we understand coordination as an intentional act. It’s lucky coincidence. Maybe predictably lucky, but fundamentally not intentional.

And so much of what we do in philanthropy is about intentionality: make a difference, choose to give, etc. I remain unsatisfied with the idea of unintentional coordination via markets and exchanges when we’re talking about the production and dissemination of social goods….

The Imp-ire Strikes Back

Wednesday, December 8th, 2010

Yesterday, I talked about the “Imp of the Perverse,” the idea that people regularly – not all the time, but not never – knowingly act against their own self-interest. This is a way of arguing against an overly strict rationalism in our understanding of human behavior. This matters for philanthropy because we too often assume that the beneficiaries of nonprofit services will see what’s given as unequivocally good and use the services that are offered. And then we wonder why projects don’t work, why people don’t take advantage of the resources offered them.

What I like about the Imp of the Perverse is that it allows for an understanding of human nature as flawed without committing one to an essentially conservative position of original sin. Progress is possible, we’re not hamstrung by original sin or a fundamental flaw in our natures, but the action of the Imp in history means that flawed actions happen regularly and for unpredictable reasons. Sometimes people just won’t act in their self-interest, for reasons that are kind of a mystery. Having the expectation that a certain amount of this will take place, rather than being surprised when funded initiatives don’t work out as planned, may lead us to be more modest in our assumptions about the uptake of social programs, and that wouldn’t be a bad thing.

Baby did a bad, bad thing

Tuesday, December 7th, 2010

An intriguing blog I’ve started reading (HT to Marginal Revolution) is A Fine Theorem, which pithily summarizes economics articles (and other writings that bear on economic motives) while advancing the author’s interesting views on the discipline. A recent post brings up a problem that I don’t believe we think about enough in philanthropy: the joy of doing the wrong thing. Here’s AFT on Dostoevsky:

Economics, inherent in its methodology, is a normative science. Even when an econ article is deliberately positive – “if people act as if they have a utility function, then the following choices will result” – the overwhelming majority of articles make assumptions such as “utility is increasing in money”. These types of assumptions are not controversial within the class of rationalist theories of human action. Indeed, many economists…considered deviations from rationalism…to be mistakes when they occur, and further to be mistakes which ought be cured with better education. [Dostoevsky’s] Notes from the Underground famous[ly] suggests an alternative, existential argument for human action: the very existence of a rational utility function which might be maximized causes people to act contrary to that function. There is a great word in German expressing this sentiment: Weltschmerz, literally “world-ache”, or the ennui which sets in when a person feels that their actions are too intentional, too preordained. Here’s the relevant quote from Dostoevsky….

“What is to be done with the millions of facts testifying to how people knowingly, that is, fully understanding their real profit, would put it in second place and throw themselves onto another path, a risk, a perchance, not compelled by anyone or anything, but precisely as if they did not want the designated path, and stubbornly, willfully, pushed off onto another one…And what if it so happens that on occasion man’s profit not only may but precisely must consist in sometimes wishing what is bad for himself, and not what is profitable?” (Part I, Section VII)

Despite recent attempts to introduce the insights of behavioral economics into the discussion about philanthropy, I think we still operate almost exclusively on a rationalist model when it comes to understanding the beneficiaries of giving. Motivations for givers, as I’ve discussed on this blog, we’re starting to understand in a diverse set of ways, but I fear we often think about recipients of services as consumers of widgets whose motivations should be obvious: offer someone a way to improve their lives, and they’ll take it.

One reason to demur from this point of view is, maybe they don’t see it as an improvement. We’re starting to get this, as the concept of the “nudge” attests: make 401k contributions opt-out rather than opt-in, for example, and rates of participation will go up.

But another reason, as AFT and Dostoevsky point out, is that recipients of services may see the service as an improvement, but may choose not to take it anyway “just because.” AFT uses the word “existential.” Poe called it “The Imp of the Perverse,” in a story of that title (thanks Wikipedia):

We have a task before us which must be speedily performed. We know that it will be ruinous to make delay. The most important crisis of our life calls, trumpet-tongued, for immediate energy and action. … It must, it shall be undertaken to-day, and yet we put it off until to-morrow, and why? There is no answer, except that we feel perverse, using the word with no comprehension of the principle. … [Then] The clock strikes, and is the knell of our welfare. At the same time, it is the chanticleer-note to the ghost that has so long overawed us. It flies—disappears—we are free. The old energy returns. We will labor now. Alas, it is too late!

I think the Imp is at work far more than we care to acknowledge in philanthropy. As Poe illustrates, he’s not a permanent presence – people do act in their own self-interest most of the time. But he’s always there, and we ignore him at our peril.