Once Treated With Scorn, Randomised Control Trials Are Coming Of Age
REUTERS/Parivartan SharmaIT ALL began with a white envelope. Inside, a letter from the provost of the Massachusetts Institute of Technology offered three young economists at MIT $100,000 to spend as they wanted (those were the days).
Two of them, Esther Duflo and Abhijit Banerjee, used the money to set up an organisation to run "randomised control trials" (RCTs), an experimental technique a bit like drugs trials, but for economics. To test if, say, boosting teachers' pay improved educational outcomes, an RCT would take a collection of comparable schools, randomly assign higher wages to some teachers but not others, and see what happens. The organisation, called J-PAL (to give it its full title, the Abdul Latif Jameel Poverty Action Lab), has just celebrated its tenth anniversary. Its methods have transformed development economics.
When J-PAL started in 2003, RCTs were regarded as wacky. Critics said that doing a trial was like putting people in a cage and experimenting on them. They pointed out that you cannot conduct randomised trials for big macroeconomic questions ("What happens if we devalue the currency by 50%?") because there can be no control group. They conceded RCTs might generate useful nuggets of evidence (raising teachers' wages in India, for example, did surprisingly little to improve learning). But they argued that evidence from such trials would always remain small-scale, tied to a specific context and not be useful beyond it.
Ten years on, few of those criticisms have stood up to scrutiny. RCTs have entered the mainstream. J-PAL has conducted 440 of them (it started with five). The World Bank runs RCTs. So do regional bodies like the Inter-American Development Bank. Even governments deploy them: Indonesia used one to test whether identity cards would improve the delivery of subsidised rice to the poor, the largest anti-poverty programme in the country (they did). Techniques for designing and doing trials have improved, with more accurate measurements and more reliable ways of ensuring that samples are random and not merely arbitrary. Trials are bigger. A recent one took place throughout the Indian state of Andhra Pradesh, which has a larger population than Germany. Trials are now investigating questions previously thought off-limits to RCTs, such as labour-market policies or policing. J-PAL did a trial in half the cities in France to determine whether job-training encouraged employment growth overall or just boosted the prospects of trainees at the expense of the untrained. (Answer: before 2007, it helped everyone; afterwards, it redistributed jobs rather than creating them.)
As the number and scope of trials have grown, the accumulation of detail has started to generate broader insights. Take education. J-PAL ran trials on many questions, from the effect of remedial classes in India and Ghana (enormous) to what happens if you double the number of teachers in Kenya (not much). One conclusion kept cropping up: the biggest improvements to educational outcomes occur when you teach children things they are capable of learning. That sounds like a statement of the obvious. But it is quite different from the view of (say) the OECD, a club mainly of rich countries, which runs influential studies on mathematics and literacy among its members. The OECD thinks the quality of teachers matters most. J-PAL's finding also goes against the grain of what many parents believe: that the focus should be on the quality of the curriculum.
In other areas, RCTs have revealed as much about what is not known as what is known. Microfinance, for example, does not turn the poor into entrepreneurs, as was hoped, but does make them better off: many use the tiny loans to buy television sets. It is not clear why. Poor people also buy too little preventive health care for themselves, even though the benefits are huge. RCTs show that if you charge a pittance for simple products such as bednets treated to combat malaria or water purification tablets, people do not buy them; the products have to be free.
Development economics on trial
The common conclusion from such trials is that the poor's own decisions matter much more than was once thought. Even the poorest of the poor have tiny amounts of discretionary cash and their decisions about what to spend it on (bednets, for example) make a huge difference to development. This view of the poor is at odds with the one espoused by "Big Push" economists, such as Jeffrey Sachs of Columbia University, who argue that people are stuck in poverty, can do little for themselves and that development should therefore consist of providing the poor with benefits--like irrigation, roads and hospitals--that spring the poverty trap. But it is also at odds with critics of Big Push thinking. J-PAL's trials show not only that the poor's decisions are important but that they are sometimes bad (for example, their underinvestment in health). Critics of the Big Push, such as William Easterly of New York University, say the best way to help the poor is to stand back and stop messing up their lives. In contrast, J-PAL's trials imply that there is a role for outsiders to improve the decision-making of the poor by, say, improving information or incentives.
Over the past ten years, randomised trials have changed hugely. They began as ways to provide hard evidence about what was actually happening. Now they have become techniques for testing ideas that cannot be investigated in any other way. (Are teachers or trained volunteers better at providing simple remedial lessons? Do a trial.) Over the next ten years they will change again. They are likely to get more ambitious still, use "big data", engage even more with governments and probably measure things that cannot now be tested (RCTs are already measuring cortisol levels as a way of judging how policies affect people's happiness). Who knows, their proponents might even find a way to apply them to the sweeping assertions of macroeconomists.
Click here to subscribe to The Economist
