In recent years, the randomized program evaluation has become the gold standard for evaluating development programs – and the bread and butter of many development economists. The evaluations often uncover valuable new information, but are controversial, and can also be prohibitively expensive to implement for small NGO’s.
Michael Frank, the Director of Finance for the Mali Health Organizing Project, is using an alternative evaluation method, one that he hopes will allow him to evaluate a microfinance program without breaking the bank.
Evaluating Microfinance When Randomization Isn’t an Option
By Michael Frank
Recently, I set out to design a program evaluation that measured the impact of our microfinance program on the community’s wealth, as opposed to just the personal wealth of the loan recipient. My organization is focused on facilitating the development of the slum community in which we work, so if individual loan recipients are increasing their wealth, but doing it at the expense of their competitors next door, we are not achieving our goals. To study community wealth, I decided to track both loan recipients’ businesses and “other affected parties” (competitors, suppliers, distributors, and other nearby businesses).
In a perfect world, I would have done a randomized program evaluation, but that wasn’t an option for my organization due to budgetary and logistical constraints. So I had to come up with an alternative methodology, one that eliminated, or at least minimized and accounted for the selection bias stemming from the fact that people selected to receive funds are likely to be more capable than the average similar-sized business owner. After consulting with a number of advisers and colleagues, I decided to use a variation on the “waiting list-control group” method regularly used in medical studies.
Photo: Mali Health Organizing Project Mali Health Organizing Project loan recipients.My evaluation design requires a call for loan applicants in the most similar nearby community that does not have a similar microfinance program already present. Using the same criteria as our program uses to select applicants, we will choose 120 (the same number of loans that we give out) loan recipients to be on a waiting list for the loans. We will track these entrepreneurs, and their affected parties, effectively using their community as a control group. After the study is over, we will either give the waiting list loans or compensate them in some other way, so that my tactic of using them as a control group is ethically acceptable.
This is the most efficient bottom-up method of evaluating the impact of our program on the community’s wealth as a whole that I have come up with. I would welcome other suggestions about how to improve the study, or my organization in general. You can either leave them below or email me.
and 
. You can also watch it on 
Are you not concerned that the ‘control’ group’s behavior (e.g. timing of capital investment) will be biased by its members’ anticipation of loans (in the case of those on the waitlist), not to mention by the general presence of the lender? Perhaps it would be better to track these 120 individuals without promising them loans.
Your proposed study has a serious flaw. Notifying 120 potential loan recipients that they will receive a loan later could have profound effects on their own motivation, since they anticipate improvement in their opportunities in the near future. This very likely will motivate them more strongly in the present: You then have one village with microfinance loans in place, and another village full of highly motivated people waiting for their microfinance loans. That’s an inappropriate control group because you have raised their motivation beyond what a “no-anticipated-loan” village would have.
Find a village that matches the loan village economically but leave it alone; then do your general economic measurements on both villages afterwards. Then you can come back and offer loans to the people in the “control” village but after the comparison has been made.
Your control group is a selected village; your justification is a match on economic indicators at the beginning of the experiment. Loans in one village; no loans in the other; then the post-test comparison.
Yours
Don C. Donderi
Principal Consultant
Human Factors North, Inc.
Toronto, Ontario
Retired Associate Professor of Psychology
McGill University, Montreal
In psychological research of this nature, designs like this are referred to as a Nonequivalent Control Group. As I teach it to my students, the nonequivalent control design CAN be very powerful but you have to be ready to rule out possible confounds like you might encounter with volunteer bias. A pre-post design is useful for this, as is a lagged design in which you stagger the treatment (in this case, microloans) to occur at different times. I think if you establish a pretreatment baseline on your measures, stagger the implementation of your treatment across communities, and measure longitudinally as each community progresses (or at least capture a post-treatment measure) then you are in a pretty strong position to discuss the causal relationships involved.
The only other suggestion I might posit is to use a cross-lagged design, in which you measure two (or more) things pretreatment and then the same things posttreatment and look at the cross-correlations that occur. This can inform about which measure is likely to precede the other and thus be a likelier candidate as the causal factor.
I think it’s a clever and resourceful tactic. You’re not putting together a peer-reviewed thesis here; you just trying to make a real-world determination of whether or not what you’re doing in microfinance is working. And given budgetary constraints, this seems like a good way to go about it.
I don’t know how much trouble it would be, but I would be willing to bet that the more you record qualitative and unmeasurable observations, the more useful, unexpected patterns and other results you will ultimately be able to derive from your data.
You may want to consider using Grameen Foundation’s Progress out of Poverty IndexTM (PPI). The PPI is a country-specific poverty assessment tool based on the national household income/expenditure survey of a given country. It allows poverty-focused organizations to understand where clients fall in the poverty spectrum (below or above the national poverty, for example) and to create a baseline to track clients’ movement over time. While the tool does not allow an organization to speak to causation, it does allow organizations to identify trends of improvement/stagnation/worsening among their clients. For more information, visit http://www.progressoutofpoverty.org.
S Brown
The biggest problem with his study can be found in the words I have bracketed in the first sentence. “Recently, [I] set out to design a program evaluation that measured the impact of [our] microfinance program on the community’s wealth.” I’m willing to be that studies by people studying the efficacy or efficiency of their own work almost always find that they are both effective and efficient. Compounding the problem that he is undoubtedly biased in favor of a finding that he is effective is the fact that the study’s outcome will depend on measuring something as subjective and nebulous as community wealth. One last point — loans if repayment is not enforced are actually just gifts of money. And of course, at least in the short term, gifts of money make people wealthier. Who will conduct the study to determine if whether by giving me a $100 bill, I become $100 wealthier?
Don’t get me wrong, I applaud his microfinance work, but the study is just plain silly.
While I admire the creativity here, the design does raise a few questions as regards its use of human subjects. First, human subjects research typically needs approval from an IRB (institutional review board) in cases where the experimenter interacts with human subjects. I cannot imagine what IRB would approve this study, based on all the factors other comments have mentioned about the ill-advised side effects of telling someone they’re on a waiting list of a loan when a waiting list doesn’t exist. Given the ill-advised life choices that might result from such a “guarantee”, the deception hardly seems like it would pass muster.
As an alternative, why not choose 150 people (or however many) from a neighboring village using whatever criteria you choose AND SIMPLY TRACK THEM. Tell them they’re part of a study (true) or that you’re interested in other communities beyond the one in which you’re currently operating (also true). You don’t change your results, but you do sidestep an ethical quagmire.
Good discussion. There is a more radical and more generally useful alternative to the RCT (randomized control group) design, namely the GEM (General Eliminatiion Methodology) model. It’s outlined in a paper of mine in the Journal of Multidisciplinary Evaluation (jmde.com, 2009), along with the reasons it often gives BETTER results than RCT. By the way, RCT should not be called the gold standard, in the eyes of Tom Cook, perhaps its smartest supporter. I’ve been using GEM for the last five years in evaluating Heifer International’s efforts at poverty reduction in 20 countries, and so it’s not just a theoretical possibility; it’s a practical option (and a theoretical necessity!).
Michael Scriven, Evaluation Center, Western Michigan U, and Psychology Department, Claremont Graduate U.