The Massachusetts CREDO Study of Charter Schools Methodology is Critically Flawed (2024)

In Massachusetts there is a rigorous debate about charter schools and their impact on education. Proponents of charter schools cite a 2013 report by CREDO as evidence that charter schools in Massachusetts are effective. I have reviewed the 2013 CREDO report and I find that the methodology used is insufficient to draw conclusions about charter schools in Massachusetts. If we don't use reliable methods, then we can't rely on the results of the study. Let me explain my several concerns.

Problems with the CREDO Methodology

First, the CREDO reports uses, what they call a VCR, which is 'virtual control records.' They compare students in a charter school to students not in a charter school along various dimensions. However, the single most important factor was omitted undermining the entire report. A self selection bias was not accounted for. Self selection bias is critically important because parents and children who want the students to excel academically are more invested in educational outcomes than students and parents who do not take the time to enroll their children at a charter school. This self selection bias is not accounted for in the CREDO study. Because there is perhaps no variable more important to student outcomes than parental involvement, this single point undermines the entire conclusions of the CREDO study. If we were to think about it, on average we would expect that students who have parents who are invested so much so as to try to enroll them in a charter school would likely do better by virtue of that home environment and not necessarily the charter school. Parenting matters. CREDO didn't account for this. In fact, the CREDO study never even mentioned 'self selection bias', or 'limitations' of their approach, which something that reputable peer reviewed academic research always includes.

Second, the manner in which CREDO considers aggregated data blurs the lines of different charter schools. Each charter school is different. They have different teaching practices and different cultures. They are located in different communities with different challenges, and the students are directly affected by these differences. To lump together all charter school students misses an important aspect of charter schools and that is their individuality. If CREDO used a proper control group, and if they disaggregated by school, we could make some conclusions about charter school outcomes. CREDO didn't do this.

Third, the CREDO report was funded by the Walton Family Foundation. This is the same family that is the owner of Wal-Mart, a notoriously anti-union corporation. There is a belief among some charter school proponents that unions undermine public education and therefore, a move towards non-union schools is important. That is an argument for another time. The important point here is that Walton Family Foundation offers Public Charter Startup Grants. Research that is funded by the organization that promotes charter school growth undermines the objectivity of that research.

The Correct Way To Measure Charter School Outcomes

The Massachusetts Legislature in conjunction with Department of Elementary and Secondary Education (DESE) needs to make sure that all charter schools are measured independently. The correct way to measure a charter school is to compare the 'cohorts of students' who were part of the lottery process to one another.

For example, if 100 students apply to a charter school, and the charter school can only accommodate 40 students, the charter school will typically use a lottery process to determine who will be admitted to the charter school. This is the right thing to do. What we need to make sure that happens next is that the 40 students who were admitted to the charter school are compared to the 60 students who were not and ended up back in the public school. This is not usually done but it must happen because the randomization of the students who were 'selected' vs 'not selected' eliminates any bias at home or in the students; the fact that all students who applied are measured against each other eliminates the self selection bias. By comparing the group students who got into the group of students who did not, we can draw conclusions about the outcomes that that particular charter school is delivering. This brings me to another concern about generalizing results of charter schools.

We cannot generalize the results of charter schools; each charter school is different and has different practices and cultures. Too often I hear proponents of charter schools point to select studies that show a particular charter school works. If the charter school was actually measured correctly, and they usually are not, the positive results are limited to just that charter school. This CREDO study just lumps together all charter schools, ignores individual differences between charter schools, and misses the point about the importance of the self selection bias.

This issue is something that the Legislature and the State's education department (DESE) needs to take more seriously. Proper outcome measurement is a complaint I have across virtually every state agency; in my opinion, measuring outputs is very common and measuring outcomes is extremely rare.

Conclusion

In summary, the CREDO report attempted to create a meaningful control group but it fell short along the most critical factor - the self selection bias. We would expect that students at a charter school to do better because of parental interest and overall involvement associated with enrolling a child in a charter school, and this is what CREDO found, but it may have nothing to do with the charter school and everything to do with the dynamics associated with the self selection bias.

Paul Heroux is a State Representative from Massachusetts. He can be reached at PaulHeroux.MPA@gmail.com.