The Nonprofit Sector Needs a Replication Crisis of Its Own
Photo by von Vix on Unsplash
In Academia, The Replication Crisis Brought Needed Changes To Improve Research And Evidence Generation. What Could A Similar Crisis In The Social Sector Do For Philanthropy?
By Jesse Bourns, COO at Ajah
Not long ago we applied for an ACX Grant, a new effort by elusive blogger Scott Alexander to fund “good research and good projects with a minimum of paperwork.” it’s a model you might be interested in, so check it out. We didn’t get the grant but we think this is an important idea. If you want to know more, head over to Astral Codex Ten and scroll down to #117: Help Nonprofits Share Evidence and Determine Impact.
In the nonprofit sector, our work depends on assessing and using knowledge from different sources to provide learning opportunities for a wide range of organizations. The problem is, as a sector we don’t do it very well. That’s actually how we orient a lot of our work with funders. Not being able to use evidence makes it hard for nonprofits to serve their communities, which puts the sector at a real disadvantage. The hard truth is that, as a sector, we don’t really know if what we’re doing is meeting the needs of those people (or if it’s actually hurting them). Since we can’t reliably know the impact of the services we offer, we can’t even determine which of those services are having a negative impact, let alone how to improve them if and when they do.
This isn’t news to many of us, but what’s important here is that the issue goes much deeper than your typical “technical” measurement problem. That’s certainly part of the issue, but there’s a lot more to it than that. This is really about how the sector understands the problem, and whether we appreciate the true scale and scope of the issues i.e., how bad things really are. We don’t see this as a collective problem for the sector. The solutions and projects related to this can’t be focused on helping one organization at a time—and the changes required are more structural than anything else.
What’s interesting is that there may already be a good example to work from in another sector. A nerdy sidenote: Academia went through something similar with the replication crisis, and as a result has been able to focus efforts at many levels (researchers, publishers, funders) in order to address the issues that were identified because of the crisis. If you’re not familiar with the replication crisis, it is a decades-long (and ongoing) development where researchers began discovering that they weren’t able to reproduce results from a large number of long-established “proven” scientific studies. At first it seemed to be confined to the social sciences, but eventually the same problems were discovered in other domains, including the hard sciences. It came to be seen as an existential crisis to the scientific process itself—one that could undermine the credibility of research across the board. It’s important to clarify that the replication crisis didn’t create these problems—it revealed problems that were there all along. As a result, academia was forced into a period of deep reflection to rethink how it could approach its structures going forward as a sector.
Calling it a crisis makes it sound terrible. From our perspective it’s one of the best things that could have happened. The replication crisis accelerated the adoption of open science (which enabled the process of reproducibility) and it led to higher standards for science publications. More importantly, the crisis led to a shift in orientation for the entire sector—not just for large institutions but for many different stakeholders who now moved to confront the issues independently and collectively because of how serious the situation was. In short, the replication crisis in academia enabled collaborative efforts that weren’t possible before.
What could a similar crisis in the social sector do for nonprofits? We haven't collectively as a sector confronted the scale and scope of the problems in our use of evidence. It might seem counterintuitive but we think it's necessary. At Ajah, we work a lot with systems for evidence and knowledge (how it's generated, managed, used, governed, and shared) and the people and technology that make up those systems. So, we see the problems. In the sector, improving how we generate evidence about our work is critical—if we ever expect to improve our services, it needs to happen.
We need a crisis to take place so that we—as a sector—can begin to address the systemic issues that enable(d) it. People have nibbled at the edges of this issue but there has been little discussion about how to fundamentally transform our practices and structures.
Bottom line: Our approach to using evidence in service delivery is fundamentally flawed, and we can’t go on ignoring the problem. We need to be more comfortable with radically altering the structure of the sector to account for this. The nonprofit sector needs a replication crisis of its own—because the problem is probably worse than we think.