Squeezing development research juice out of the Millennium Village Project evaluation

Thirteen years after the project started in 2005, the Millennium Village Project (MVP) founded by Jeffrey Sachs, has reached ‘peak evaluation’ with an article by Shira Mitchell and co-authors, ‘personal perspective’ by Jeffrey Sachs and another comment on the ‘fog of development’ by Eran Bendavid in the recent issue of The Lancet (all open access).

"The Millennium Villages Project: a retrospective, observational, endline evaluation"

* The evaluation in @TheLancet: https://t.co/M1HkL8jR9s

* Reaction from @JeffDSachs: https://t.co/JWJBEAA8JV

* Andrew Gelman's (@StatModeling) summary post: https://t.co/zjTCd8Bw4m pic.twitter.com/X752Ffbgko

— David Evans (@tukopamoja) April 14, 2018

Some commentators were quick to point out that the final-final word may be left to yet another evaluation with a DfID-IDS-3ie-connection, but right now, the discussion about the findings is in full swing.

Not sure how I missed this, but the evaluation of the Millennium Villages Project is out. I didn’t see any surprises. Big and consistent effects on health, not much else is stable. No large effect on poverty. https://t.co/gjm8eTlYVo

— Ryan Briggs (@ryancbriggs) April 15, 2018

As Mitchell et al. point out the biggest positive impact has been on health-related goals:

The achievements of the MVP in health suggest support for the project’s emphasis on strengthening the continuum of care from households, to primary care facilities, and to tertiary care facilities. In particular, we believe that the project’s cadres of paid, professionalized community health workers, empowered with smartphones to aid in service delivery and real-time disease monitoring, contributed to the positive results. The project was also an early adopter of interventions and technologies that have since been implemented by development organisations and governments, in part because of the MVP’s demonstration and advocacy.

At this stage I am more curious about how the MVP and the ways it has been implemented, monitored and evaluated over more than a decade remains an interesting case study about the power of development paradigms and how ‘discourses’ are formed and maintained.

Heroic efforts to squeeze out development research juice
The parameters around this latest evaluation read like who-is-who of the development knowledge industry: The Lancet, Open Society Foundation, Harvard, Princeton, Stanford, Columbia, Jeffrey Sachs, DfID, IDS, 3ie etc. etc. and I am sure economists from the World Bank and various RCT and evaluation experts from the global North will comment on the data and findings as well.

For that reason, the analysis by Mitchell and colleagues represents the culmination of heroic efforts to identify the treatment effect of the MVP against stacked methodological odds

Eran Bendavid congratulates the team in a very 21st century (‘data is the new oil’)-way: By pointing out that large-scale development efforts are essentially methodological challenges these days and that today’s development ‘heroes’ use statistical tools to capture impact.

The authors deserve to be congratulated for squeezing as much juice as possible from this academic third rail, and for transparently and courageously telling the world: this was it, this is the best we can say about the MVP.

The ownership of this evaluation endeavor quickly moves to the authors of the study-not the people how have been living with the MVP for more than a decade. I would have chose different wording when talking about the real lives of half a million people.

500,000 people as data points?
While I do not want to deny that communities improved through the project, much of the discussion is focused on top-down planning and traditional economic indicators.
13 years after the project started advocates for bottom-up participatory development or alternative approaches and measurements of development will be disappointed, but the 500,000 people covered by the MVP only appear as ‘data points’, not as agents or citizens.

At the same time, the ‘power-knowledge’ discourse is still firmly based within the global North, its elite universities, bilateral agencies and philanthropic organizations. Given the resources, time-span and group of experts involved, the study remains surprisingly un-reflexive about how ‘we’ learned and changed-not just ‘them’. I am sure that there will be books, qualitative research articles and more in the future, but as per usual they will mainly be ex-post reflections, not adaptive, agile inputs into an on-going situation. 

It is not surprising that Jeffrey Sachs’ ‘personal perspective’ is actually an impersonal, general policy statement-perhaps the opposite of ‘personal’. There is no ‘I’, no personal reflection on the process, people, locations etc., but a lot of development policy-jargon:

The lessons from the MVP are highly pertinent. Multisector planning, budgeting, implementation, monitoring, and evaluation are feasible and necessary. Information platforms can be created for multisector plans and programmes. Computer technologies, including artificial intelligence and big data (responsibly managed), offer new cutting-edge solutions.

As academia is increasingly discussing gender, feminism and ‘decolonization’ questions this study seems to be a reminder of how marginalized these debates really are.
If the MVP process is deemed as a near ‘perfect’ example of how to engage with large-scale development projects, indigenous, action research-based or otherwise non-mainstream approaches to planning, monitoring or evaluation still have a long way to go-and it is already 2018…

Given the current debates in international development research the pendulum seems to swing towards bigger, longer, more sophisticated RCTs rather than thinking about transformative alternatives.

Legitimizing development knowledge and the habitus of proper research
One of the political discussions that could follow from these insights (and unlikely will in the current political climate in the UK or USA) is that one of the key findings of MVP is that a steady investment in infrastructure and people will improve many people’s lives-so essentially the opposite of austerity and simple neoliberal ‘let the market figure out healthcare and education’ mantras.

There is no doubt that the MVP debate will find its way into textbooks, course syllabi and many more academic research and writing. I am also pretty sure there will be a celebratory conference in 2025 to mark the 20th anniversary of the project.
But 13 years after the project started we need to continue those debates, even if they will remain marginalized, on how to communicate development differently, with communities and real people, on smaller scales rather than simply relying on more technology, better tools and bigger RCTs.

The Lancet papers are a powerful reminder of how development knowledge is produced and legitimized and how the habitus of conducting research seems to have changed very little during the 13 years of one of global development’s most discussed projects in the 21st century.