Charter school advocates are understandably pleased with some recent news out of New York City. A new study has concluded that charters schools--a controversial innovation in public education--have a more positive effect on student test scores than do traditional public schools. (In fact, the report's authors claim that charters are closing the "Scarsdale-Harlem achievement gap" in math and English, or the divide in test scores between affluent suburban schools and those in the inner city.) Charter advocates are particularly touting how the study reached its conclusion: As The Washington Post puts it, the study meets the "gold-standard" of education research methodology because it was randomized, looking only at students who applied to charter schools through an admissions lottery. The report compares the test scores of students who were accepted to charters via lotteries to the scores of those who didn't get in--and, thus, remained in public schools.
Randomly choosing a sample this way does have many strengths. For instance, it significantly reduces a "self-selection bias"--the possibility that students who apply to charter schools share personal characteristics, including parents who might be more involved in their children's education. Consequently, if charter students test well, it might be due in part to their parents' motivation and assistance, not necessarily just because of their schools' strengths. (This method thus addresses a common complaint made by charter school opponents in evaluating their effectiveness.)
But we shouldn't be so quick to embrace the study's conclusions. Not all charters are equally popular and, thus, should not be considered as a single group, points out Alexander Hoffman of the website Gotham Schools, who lists several valid concerns about the study. "The most important problem is that not all charter schools are oversubscribed [with applications], so not all charter schools can be included in these studies," he writes. "Clearly, the 'better' charter schools are far, far, far more likely to be oversubscribed than the 'worse' charter schools."
Another of Hoffman's critiques ventures into different territory: "If parents do not get their choice of schools for their students, are they going to be as supportive of their child's teachers?" This represents an ironic shift away from the standard assumption and criticism in charter studies about the impact of parental motivation--usually, parental motivation is the cause of higher test scores, not lower ones. (Other criticisms of the new report can be read here and here.)
Indeed, just because a study focuses on students in charter lotteries doesn't mean it's perfect. Some past lottery-based studies have had to acknowledge gaps in their data; this report on Boston charters, for instance, admits that "lottery records are missing or incomplete," allowing for methodological flaws.
Yet the praise for the New York City study in the media reflects the fact that reports even of this caliber have been few and far between up until now. According to Robin Lake, associate director of the Center on Reinventing Public Education, most national studies on charters "lack in rigor." Her organization's white paper on these studies covers the faults with various research methodologies. Comparing school test scores from a single point in time is an example of one of the "worst research designs possible," Lake says, since many variables are left uncontrolled. And even some better measurements, such as tracking the average test score of a class of students over time, don't always account for issues like changes in a student body. What's more, some schools, public and charter, don't have good records of students' scores and other pertinent research data.
At the very least, hopefully the new study will nudge policymakers to examine what charters do well and, from there, to develop best practices for how they should be run. And perhaps it will serve as an impetus for launching new research in other cities, since charters in Washington D.C., Chicago, and California, among other places, also employ lotteries to select their students.