Fourth graders in traditional public schools did significantly better in reading and math than comparable children attending charter schools, according to a report released on Tuesday by the Federal Education Department.
And W. bans the federal commissioner of education statistics, Mark S. Schneider, from the cool table in the cafeteria. This is the second major study to come out of this department in two months that found that educational choice doesn’t produce results.
Not so fast, says Eduwonk. He points out a key disclaimer in the report that has been omitted in news accounts.
the data are obtained from an observational study rather than a randomized experiment, so the estimated effects should not be interpreted in terms of causal relationships.
The Education department shouldn’t be putting out reports that they can’t stand by.
My best guess about charter schools is that there is a wide range of academic performance coming out of these schools. It’s no secret that some charter schools are disasters. Either poor leadership or insufficient freedom from the state are creating some real school stinkers. On the flip side, there are some great examples, such as the KIPP academy in the Bronx. The stinkers are lowering the overall average of charter schools.
These stinkers shouldn’t damn the whole enterprise. The schools should have their test scores posted on the front door and parents should have the opportunity to decide for themselves which schools to use. Continually failing charter schools should be eliminated. And the good ones replicated.
And charter schools that fare as well as public schools should be examined for other variables other than test scores. Parental involvement and unique learning approaches are another reason to support the charter school model.
(thanks, russell)
Apt. 11d 
The disclaimer does not mean that the study’s authors do not stand by their report, but simply make clear that the study indicates correlation and not causation. Frankly, I find the findings that charter schools do not necessarily correlate with better results an important one. Your observations that there’s a wide range of academic perfomance, some are awful, and some are great certainly applies to public schools as well, including urban ones. Which begs the question: If charter school (or choice school) doesn’t automatically equal better results, why the blind adherence by so many policy makers to choice as the panacea for all educational ills and the idea that urban public schools categorically fail?
LikeLike
There’s a big distinction between correlation and causation in this context.
There are two questions to answer in education policy. One, what school will teach this student/population the best. Two, what schools have the best results overall. The second question completely ignores the issue of population. If a given type of school serves a particular population, it is almost impossible to compare its results to a school for another population. (TJ High in NoVA has awesome results–it’s one of the nations best schools by any outcome measure–but it would not serve the population of DC’s worst school well at all.)
This survey answers the second question, not the first. If charters serve a more underprivileged than average population, it may tell us nothing at all about the first question.
LikeLike
Certainly, if charter schools serve a more underprivileged than average population, test results may be impacted. I would be interested in seeing studies on the relative income/family education/parental involvement/etc. for charter school kids and public school kids. I’ve seen a number of commentators decry the failure to account for the possibility that charter schools serve a more disadvantaged group, but I haven’t seen anyone make (much less support) the argument that charter schools actually do so. Personally, my hunch (not to be confused with an actual assertion of fact)is that charter/choice school kids have parents with higher education and commitment, if not higher income, than the kids still in public schools. Absent any evidence to the contrary, I think we have to assume that the overall populations are pretty similar in the two groups.
LikeLike
Eduwonk’s statement about this study applies to almost all education research, to most public health research, to some medical research, and even some of my own research. It does not make the results of this study invalid or irrelevant.
Controlled random assignment is impossible for many many research questions (anyone volunteer to participate in a randomized study of breastfeeding where some babies are assigned to be breastfed and others are not, and even if they did, what kind of compliance would one get)? Thus, we are forced to make policy decisions on imperfect correlational studies that attempt to control for confounding variables. That’s what was done in this study.
It’s clear that this research doesn’t _support_ charter schools, in the abstract.
bj
LikeLike
Eduwonk also criticizes the use of reduced/free lunch numbers to determine student socioeconomic level. That public vs. private study has been criticized on the same grounds. The problem is apparently that traditional public schools are more plugged into the free lunch system than private and charter schools (the schools may not participate at all), so poor children in these schools are not being counted properly.
LikeLike
The fact that the education department didn’t use a ramdomized sample is an important part of the methodology that wasn’t mentioned by the press, says eduwonk. that’s all.
Do I think that all nonrandomized research sucks? No way. I’m writing a paper right now that doesn’t use random data. In fact, I’m not even sure that we’ll get to the bottom of the charter schools v. public schools debate using straight numbers and large random sample. Ultimately, schools are too individual to compare. The neighborhood context of the school are too individual. Even within each school, some students fare better than others. Which variables to you use to measure success?
Charter schools can justify their existence by having a few successes, by having open records for parents, and by having some oversight that closes the stinkers. These tests that compare all public schools to all charter schools, random or nonrandom, don’t do anything for me.
LikeLike
Laura:
The eduwonk point would only be valid if one expected newspapers to point out that every single “correlational/epedemiological” study did not use random samples. There is no reasonable expectation that studies like this will be done that way, and I think we would find it kind of odd if every study on breastfeeding, or cosleeping, or “back to sleep” mentioned that babies weren’t randomly assigned to breastfeeding or cosleeping.
I’ve been scoping out the charter school cites on this study, and the remaining points I find worth exploring are the following: 1) as Amy points out, is there any evidence that charter schools actually serve a more underserved population (that is, the correction for free lunch is inadequate to describe the proportion of poor studens in charters 2) Are there specific classes of charter programs that are better than others? 3) are there specific classes of charter programs that are better than specific classes of public schools? 4) are charters producng more improvement than other schools (in spite of not meeting the same standards)? Note that 2-4 can’t be done by showing the success of specific schools versus other schools. As you point out there’s always going to be tremendous variation between individual schools: showing that an individual charter (or public) outperforms the other is no different than showing that an individual person out performs another at something (i.e. doesn’t say anything about the class that those people belong to).
But, the onus is on the supporters of charters is to show evidence in support of their point of view on those questions (not to just leave holes). Lawyers leave holes in others arguments; policy analysts have to fill the holes they find with other evidence. (at least in order to convince me).
bj
LikeLike
bj,
I don’t think individual charter schools should be any more obligated than standard public schools to prove that they have a right to exist. A recent Washington Post article said that after years of increases in charter attendance and decreases in traditional public school attendance, we now have 23% of public school children in DC attending charters. This may not be the peak for charter enrollment, but if the percentage of DC children in charters stays at 23%, I think it’s clear that DC charters are not a fringe phenomenon, and are getting closer and closer to becoming the norm for public education.
LikeLike
Apropos of nothing, I’d like to point out that extended day schools like KIPP offer the equivalent of full-day childcare for schoolchildren, and that that sort of coverage will be particularly attractive to overstretched poor families who are not planning on spending the late afternoon ferrying little Sophie and Alexander to soccer, piano, ballet, etc.
LikeLike
If charters are more likely to serve the tougher kids that is something that should be studied. If they are providing other benefits, such as full time care, also worth finding out. If they are doing these things with less money than traditional public schools, also worth knowing.
But I don’t think that this information should be a referendum on charters schools or traditional public schools. Charter schools should exist because parents want them. Parents should have all the information at their disposal to make good decisions and those studies will certainly help. But at the end of the day, if parents from a particular community are dissatisfied with the existing school system and an alternative public school wants to exist in that area, that’s good enough for me.
LikeLike
I see that we are definitely applying different standards to what needs to be proven. As a taxpayer, I want to see that I am getting good value for my investment, and I think charter schools have to prove that. Right now, they’re not convincing me. DC is actually a good example. I’m a regular reader of the Wash Post, and it seems like there are regulary stories appearing there about charters that decide suddenly to close (or loose their ability to function, or have serious fraud). Of course those things happen in public schools, too. But the reports I’m hearing is that charter schools don’t even meet the incredibly low standard of “doing no harm.” And reading the “talking points” at some of the pro-charter websites suggest to me that charters are struggling to meet that standard.
bj
PS: I have no personal axe to grind in this debate — I really am willing to be convinced by data, though anecdotal examples of the success of specific schools won’t do for me. I do have an ideological axe in the debate — I credit free public education in an urban school system for my personal trajectory from immigrant to someone who is living every facet of the American dream.
LikeLike
BJ — I think you answer your own question about “doing no harm.” Bad charter schools can be closed.
I support charter schools simply because they offer a choice to parents. Maybe sometimes it’s a lousy choice, but that’s better than no choice at all.
LikeLike
I am familiar with two charter schools in Philly. Both have attracted middle class families that othwerwise would have bolted for the suburbs or maybe Catholic school. To the extent that charter schools help keep the middle class in the city, I am all for them on this basis alone. Both schools have preferences (either on paper or in practice, I’m not sure which) that allow the children of teachers who teach in these schools to attend these schools. Both schools are rated highly by parents who have kids at the schools.
Philly definitely has the fly-by-night Charter school problem but it seems to be lessening with age.
LikeLike
I’m a parent in an urban system with notoriously troubled schools (Baltimore), and we are hoping to get our girls into one of two charter schools open in the city for kindergarten next year
Western Dave, we and many of our friends are the kind of parents you’re talking about: middle-school parents who would otherwise flee the system. The charter school we love the most has many attributes that cannot be measured by a standardized test: a clean, safe, renovated building, Spanish classes K-8, an involved group of parent volunteers, racial diversity (60% AfAm, 30% Hispanic, 10% other, including French-speaking Africans and Eastern European immigrants), and partnerships with local community groups to do art and nature projects on a regular basis with the kids. It also still resides in an urban district and is still a public schools maintained and funded by the city school system, though it also receives grants from other organizations.
I fully expect the test scores to be fair-to-middling, especially the first years the schools is open. Charter schools have only been open for a year in Maryland at all, and it’s understandable that there are kinks to work out. I also think the successful models, like KIPP, should be studied for what they could bring to public schools as a whole. But I don’t see charter schools as abandoning public education– I see them as remaining committed to the system, while maintaining the ability to choose a school that best serves my daughters and what we want for them. Isn’t that what community schools should be about?
LikeLike
bj,
If we give up on the idea of charter schools (less regulated schools that differ from standard public schools in some way and are sensitive to parent opinion) what is supposed to be the engine of school reform? How else are we going to generate models for successful schools without trying lots of different things and seeing what does and does not work? Experiencing and identifying failure is a necessary part of the process. The alternative, as far as I can see, is sticking with what you already know doesn’t work. Do you really believe that the DC public school system would be better if we just left it alone?
LikeLike
Henry wrote: “Maybe sometimes it’s a lousy choice, but that’s better than no choice at all.”
Do you really think so? If A is superior than B, then having A as the only choice would be superior to having A and B as the choices, because under the first scenario, everyone goes to A, but under the second, some will go to A and some will go to B.
I’m alarmed at how many lousy choices there are out there, and I’m afraid for the kids whose parents send them to the lousy choice only to find out it’s a lousy choice after a lot of wasted money and time. Then, furthermore, the child is at a disadvantage if she or he wants to go to another school.
Sorry, I don’t get all this excitement over “choice” when there are way too many bad choices out there. I’d rather work on keeping the current system accountable instead of allowing schools to pop up and die everywhere. Children have approximately 12 years to go to school. I don’t want to be trying to figure out which school is better.
I can’t imagine why anyone would prefer a system like the current day care system (for kids under 5) to the current public school system. It’s easier, I think, to fix a system that has infrastructure already in place than to keep on developing new schools hoping one will eventually work.
LikeLike
bj writes,” As a taxpayer, I want to see that I am getting good value for my investment, and I think charter schools have to prove that.” Excellent. I was thinking of the parent as the consumer, but it’s also really good to think of all of society as the customer. Everyone benefits from having good schools that produce educated kids in an efficient manner. I wish some of the seniors in my town would get that.
I also wish that people would get as outraged about public schools that are failing kids. You benefitted from a public school education, as did I. But there are places like Cleveland and Philadelphia that are disasters. And, like WendyW, I don’t want to wait around for decades for those public schools to improve. If there are good charter schools that can open up and at least do as well as public schools and make a lot of parents happy, then do it. Maybe some of those schools will do even better than the public schools. Hey, I’m even for giving kids in those disaster school district the opportunity to go to slightly better Catholic schools in the district. I am willing to do anything.
I actually think that the infrastructure in some school districts is so corrupt and so dysfunctional that the only way to bring about change is start entirely over.
LikeLike
I agree with Laura on the infrastructure/corruption problem. The Baltimore school system has dealt with embezzlement, corruption, asbestos, and buildings in serious states of disrepair in the past 18 months. Not to mention constantly shifting standards handed down from state and federal governments. If charter schools offer an alternative to any of these problems, while still being available to any child who wants to go, then I’m on board.
And Wendy, even in the traditional system, a lot of parents still spend years trying to figure out which schools are better than others and how to get into those they like better. They just have fewer options and a lot more bureaucracy to wade through.
LikeLike
Henry: I emphatically don’t think closing charter schools after they have wasted precious years of early childhood education prevents charter schools from “doing no harm.” That’s like arguing that it’s OK to test out a drug beause you can just cut off an arm of all the people who suffer adverse effects.
Amy: When you write your definition of charter schools “less regulated schools that differ from standard public schools in some way and are sensitive to parent opion.” I guess I’d have to hear what each of those things mean in practice. The examples I’m fearful of is “less regulation” that results in fraud and in for-profit take over of schools (in the same way that some fundraising schemes do). “Differences” that mean that lower educational standards are met and “Sensitivity to parents” that results in the same limited resources being moved from one school to another. I’d also deny that traditional public schools are not sensitive to parents.
and “Do I really believe that the DC school system would be better if it were just left alone?” Well, no. But, I’m really unconvinced that the charter system is improving it for the better (admittedly though, I know almost nothing outside of newspaper reports). It seems to me that the congress is trying an ideologically driven experiment in DC. It’s possible that the DC system is so very sick that trying unproven ideological treatments at least can’t hurt (I’m thinking laetrile & terminal cancer here). But, there are plenty of systems out there that are functioning well enough that treatment with unproven alternatives would be a bad idea. Those systems are the ones I am personally worrying about.
Laura: I agree that we are all consumers of the schools, even when our children attend private schools, we teach over own children, or when they are grown up, or when we don’t have any children. And, I mentioned being a consumer because I pay, but I also think we are consumers because we all consume the product.
I’m also (like Wendy) not in favor of choice as an ideological issue. Choice wouldn’t have helped my imigrant parents find me a functional school (and they were english speaking and educated). What we needed were schools that functioned acceptably without needing to search them out, to use social capital to find them.
bj
LikeLike
bj,
A few points:
1. If you would like to read more about the mechanics of running a successful charter, I suggest “Our School” by Joanne Jacobs. It’s a very honest and readable account of the struggle to create an academically focused high school for kids who had been D and F students in middle school. In fact, I have a copy if you are interested. My e-mail is pruss at runbox dot com.
2. Sensitivity to parents is a big, big deal and it is just about impossible in a one-size-fits all system. The problem that you are not addressing is that we do not all want the same thing from our kids’ schools. Having a single system is a recipe for discord (think of all the wrestling over curriculum that goes on). If I want A and you want not-A in the school curriculum, should we really be spending our precious time duking it out and getting nothing done, or should we each get more or less what we want, but in separate programs? There are many possible school features that cannot be easily combined: long versus short day, religious instruction versus no religious instruction, single sex versus coed, art focus versus math/science focus versus Chinese immersion versus French immersion versus Spanish immersion.
LikeLike
[continued]
No school could possibly meet all these parent desires at the same time. And that’s without addressing such perenially thorny issues such as evolution, sex ed, parent objections to reading assignments and library materials, and the problem of how to deal with various holidays, particularly since non-religious holidays tend to be pretty lame.
These are really tough problems in practice. At my daughter’s public preschool last year, the teacher engaged in a December-long sugary Santapalooza featuring Santa and his friends Frosty and Rudolph and lots of talk of presents. The Muslim family was not thrilled, and neither was I–I don’t celebrate the birth of Rudolph and Frosty.
LikeLike
bj writes, “What we needed were schools that functioned acceptably without needing to search them out, to use social capital to find them.” Agreed. Searching does require more work and it is a less optimal option. But your public schools aren’t doing their job in many urban areas. Public schools really vary from area to area, so middle class are already using their social capital and money-type capital to shop for communities that have the best schools. How did I pick the town that we live in? I studied charts of school test scores and bus commute times to Manhattan.
In a perfect world, there would just be a fabulous school in a town that would meet the needs of every student. The problem is that that doesn’t exist. In fact, the worse problem is in some urban areas where every option sucks terribly and one in five students drops out. In those cases, I am ready to try almost anything.
LikeLike
bj and WendyW — I think both of you are comparing the possibility of a bad charter school to the best-case public school scenario. In the real world you’re going to have some good charter schools, some bad; some good public schools, some bad. Some parents may, despite their best intentions, make a bad choice for their kids. Lots of non-optimal outcomes exist. There is no way around it.
But it is an advantage to charter schools that they are allowed to fail. Bad public schools don’t close; they just struggle on in mediocrity, wast[ing] precious years of early childhood education year after year.
All we can hope to do is create better outcomes for more kids, if possible. I think school choice does that. I see it as part of a healthy school system, not as an alternative.
WendyW, I know where you’re coming from with your day-care example. It’s incredibly time-consuming to find good day-care and preschool options. But there is a flip side. My son is going to preschool next year and after all the frustration of tracking down some affordable options, we found a school that is a perfect fit for him.
Consider a scenario like this: the government decides to fund pre-K education. They could look at all the existing day-care and pre-school providers and say, we’re going to accredit the adequate to good ones, close the worst ones, and let parents continue to choose from the lot. Or they could create one centralized pre-K school district and take away that choice. I prefer the non-feudal approach.
LikeLike
Henry:
“Bad public schools don’t close; they just struggle on in mediocrity.”
Don’t you see the contradiction in that statement? Mediocre schools are not bad schools by definition.
It’s some sort of weird version of Lake Woebegone, where all the kids are above average. 🙂
LikeLike