Good Read - Charter Schools

[bmags]

http://www.nytimes.com/2015/11/22/upshot/a...rates.html?_r=0

 

Often the discussion around charter schools is two sides throwing the same talking points at each other. Should be noted there is a ton of research being done on the Charter schools.

[StrangeSox]

There're some compounding factors that the article doesn't really address: lottery-based studies aren't truly randomized due to peer effects and motivational factors, whether or not the lottery charters even accept ELL and special-needs students at nearly the same rates as public schools have, whether improved standardized test scores are really the best metric (are the charters basically just doing rote test prep to improve test scores? also a problem in high-stakes testing in public schools!), how well non-lottery charters perform (if a school is popular enough that it must institute a lottery, is it really fair to compare that to the overall public school group?). Additionally, typically the highest-scoring charters have very high attrition rates, meaning the more difficult students are kicked back to the public schools.

 

Her study also focuses on Boston schools, and Massachusetts happens to have some of the tightest charter requirements in the country. In many other states, research shows that charters tend to do no better if not worse than public schools.

[bmags]

All of that was addressed in the article.

[StrangeSox]

No, it just kinda hand-waived some of them away in a couple of paragraphs:

 

Each lottery serves as a randomized trial, the gold standard of research methods. Random assignment lets us compare apples to apples: Lottery winners and losers are identical, on average, when they apply. Any differences that emerge after the lottery can safely be attributed to charter attendance.

 

...

 

That’s one reason the lottery studies don’t compare students who are and are not enrolled in charter schools, but instead compare students who win and lose the lotteries. If a student wins a lottery but declines to attend, or transfers out, her test scores are still assigned to the charter for the analysis. This means that the estimates are not biased by transfers after the lottery takes place.[

 

Her research specifically has not addressed how non-lottery charters perform. It's also based on large, successful charters with oversubscribed lotteries and good archival records. In that regard, it's sort of circular--you'd expect the better charters to be oversubscribed, and if you're looking at just the oversubscribed schools, you'd expect to find the best results. The doesn't comment on the special-needs and ELL issue, but her paper acknowledges it (without commenting on the potential impact, from what I can tell):

 

Roughly 85% of students at traditional Boston schools are

poor enough to qualify for a free or reduced-price lunch. Charter

students are not as poor; about 70% fall into this category.

The pilot school student body at middle and high schools occupies

a middle ground, with more poor students than at charter

schools but fewer than at traditional schools. Relatively few Englishlanguagelearners

(alsoknownas limitedEnglishproficiency

or LEP students) attend charter schools. For example, just over

7% of chartermiddleschools students areLEP, whereas thetraditional

Boston population is 20% LEP(pilot schools are alsoat 20%

LEP). Charterschools alsoenroll fewerspecial educationstudents

than do traditional and pilot schools. Girls are over-represented

at charter schools and, to a lesser extent, at pilot schools; this is

particularlystrikingat thehighschool level, where60% of charter

school students are female, compared to 52% at the pilot schools

and 50% at traditional schools. Importantly, however, the demographicmake-upofthecharterandpilot

lotterysamples, described

in columns 4–7 of Table II, is similar to that of the total charter

and pilot samples.

 

Lotteries themselves not truly randomized. The only part that is randomized is the selection itself, but everything before (who actually applies and who's encouraged to apply for the lotteries) and after (demotivational factors of not being selected, peer group factors) aren't. Her categorical statement that any differences can be attributed to a single variable, are, frankly, bulls*** (and her own paper doesn't make nearly as strong of a claim).

 

If you follow some of the links to other research including her own, those studies seem to emphasize the "No Excuses" model that some charters follow as the model that has shown the gains. That model, which essentially relies on strict punishment and shame, is not without criticisms even if it can boost test scores. If it is also the magic bullet she believes it to be, it could be implemented in public schools as well. Her own research also points to class size being an important factor--public schools would love to reduce class size if the funding was there.

[bmags]

From the article:

Perhaps only the best charters are popular, and that’s why the lottery studies produce such positive estimates. We can’t use the lottery approach to assess a school that does not have high demand for its seats.

 

In Boston, we used alternative statistical methods to examine the charters that are not oversubscribed. We found smaller but still positive results. A Stanford study examined student performance in 41 cities, and also concluded that their charters outperformed their traditional public schools. A caution: Without randomization, we can’t be as certain these nonlottery studies have eliminated selection bias.

[bmags]

No, it just kinda hand-waived some of them away in a couple of paragraphs:

 

 

 

Her research specifically has not addressed how non-lottery charters perform. It's also based on large, successful charters with oversubscribed lotteries and good archival records. In that regard, it's sort of circular--you'd expect the better charters to be oversubscribed, and if you're looking at just the oversubscribed schools, you'd expect to find the best results. The doesn't comment on the special-needs and ELL issue, but her paper acknowledges it (without commenting on the potential impact, from what I can tell):

 

 

 

Lotteries themselves not truly randomized. The only part that is randomized is the selection itself, but everything before (who actually applies and who's encouraged to apply for the lotteries) and after (demotivational factors of not being selected, peer group factors) aren't. Her categorical statement that any differences can be attributed to a single variable, are, frankly, bulls*** (and her own paper doesn't make nearly as strong of a claim).

 

If you follow some of the links to other research including her own, those studies seem to emphasize the "No Excuses" model that some charters follow as the model that has shown the gains. That model, which essentially relies on strict punishment and shame, is not without criticisms even if it can boost test scores. If it is also the magic bullet she believes it to be, it could be implemented in public schools as well. Her own research also points to class size being an important factor--public schools would love to reduce class size if the funding was there.

 

The problem with you implying improving test scores are their only achievement therefore it must be due to teaching to the test, are the very real gains they are finding in secondary education.

 

 

[StrangeSox]

From the article:

Perhaps only the best charters are popular, and that’s why the lottery studies produce such positive estimates. We can’t use the lottery approach to assess a school that does not have high demand for its seats.

 

In Boston, we used alternative statistical methods to examine the charters that are not oversubscribed. We found smaller but still positive results. A Stanford study examined student performance in 41 cities, and also concluded that their charters outperformed their traditional public schools. A caution: Without randomization, we can’t be as certain these nonlottery studies have eliminated selection bias.

 

The Stanford (CREDO) studies are not without their methodological criticisms.

[bmags]

Welcome to social science. Its very difficult to create perfect studies with live humans, yet these are pretty good representations that in any other topic would be enthusiastically received.

 

 

[StrangeSox]

The problem with you implying improving test scores are their only achievement therefore it must be due to teaching to the test, are the very real gains they are finding in secondary education.

 

I didn't imply that test scores are the only improvement--it's something the article and research usually focuses on, but if anything I'd stress that it's not the only factor.

 

The secondary and post-secondary gains can come right back to the other issues highlighted, though--selection bias and lack of true randomization, only looking at the "best of the best" charter schools, not having nearly as many ELL and special needs students which require far more resources, the CREDO studies actually end up excluding the best public schools, and the broader findings that, when considered on the whole, charter schools don't show improvement over public schools.

 

Some students at some charters do perform better. If there are pedagogical and admin/management differences that really do seem to produce better education, those should be examined and implemented across the board. If the "No Excuses" policies really are the way we want to go and do result in better learning, great, put them in place. There's nothing that says that can only be done by privatizing education.

[StrangeSox]

Welcome to social science. Its very difficult to create perfect studies with live humans, yet these are pretty good representations that in any other topic would be enthusiastically received.

 

"Accept my questionable conclusions about fundamentally changing public education because research is hard" isn't an adequate answer.

 

edit: also, while her academic papers aren't quite so bold, her editorial in the NYT did boldly claim that "[a]ny differences that emerge after the lottery can safely be attributed to charter attendance." If it is very difficult to create perfect studies in social science (it is!), you probably shouldn't make statements like that.

Edited November 23, 2015 by StrangeSox

[bmags]

"Accept my questionable conclusions about fundamentally changing public education because research is hard" isn't an adequate answer.

 

That's not what I'm saying. Every study is going to have its limitations, but there hasn't been a single study with charter education. There have been lots and lots and lots, and frankly since 2012 more and more they have been showing that gains for the students you would want to see gains for improving.

 

Almost all of the input gaps you speak of actually have been accounted for, and are showing gains for charters.

 

There are bad networks. Ohio especially (as stated in the article). But in well regulated areas the gains for the students you would want to see gains from have been tremendous.

 

Speaking of ESL students, its odd that charters would outperform pilot schools with similar numbers of ESL students. Pilots, after all, have full control over curriculum, so you would expect similar outcomes.

http://users.nber.org/~dynarski/BostonQJE.pdf

http://users.nber.org/~dynarski/KIPP.pdf - significant math increases after 1 year.

 

If all of what you said was true, you would expect pilots to perform equally to charters, yet that has not been the case. Its true that charters in suburbs do not tend to do anything more than their public peers (! inputs). But in urban areas with low-income students is where gains have been most significant.

 

We saw the effects of the recession on HS grads and worse for HS dropouts. This has significant consequences and possibilities for improvement. I believed what you were saying 3 years ago, but if I had to put money on it, in 3 years, there will be no doubt. The studies are continuing and filling in these gaps.

[StrangeSox]

As her own paper stresses in the conclusions section, the scope of their study is actually fairly limited.

 

It’s worth emphasizing that the large gains reported here are generated by charter schools with

over-subscribed and well-documented admissions lotteries. Charter schools with good records that

parents find attractive may be relatively effective. In an effort to gauge the external validity of the

lottery estimates, we computed observational estimates that rely solely on statistical controls, with

separate effects for schools in and out of the lottery sample. The lottery estimates of charter effects are

similar to the observational estimates when the latter are estimated using the same set of schools. On

the other hand, the observational estimates for charter schools that contribute to the lottery study are

larger than the observational estimates for other charter schools (though the latter are still positive

and significantly different from zero).

 

There are too few schools in the lottery study to generate an informative comparison of specific

charter models or practices. Because most of the schools in the lottery study fall under the umbrella

of the No Excuses model, however, the lottery results can be seen as particularly informative for

this charter model. In line with this finding, our study of a single No Excuses-style KIPP school

also generates evidence of large gains (Angrist, et al., 2010). Likewise, in ongoing work using a larger

sample of schools from around the state, preliminary results point to larger gains in urban schools, most

of which embrace key elements of the No Excuses paradigm. Other charter schools seem to generate

insignificant or even negative effects (see also Gleason, et. al., 2010 for evidence of heterogeneous

charter effects.) In future work, we hope to provide additional evidence on the relative effectiveness

of alternative charter models.

 

They have seen the gains with the "No Excuses paradigm," and in particular the biggest gains are from the oversubscribed lottery charters with strong records. Other research on charter schools more broadly find no gains or even worse results, as they also state.

 

The lotteries offer some randomization, but they still don't address the initial targeting/recruitment/marketing and student/family motivation factors that lead to non-randomselection at the very start. There are significant differences in ELL and special needs populations at charters including KIPP schools.

 

I also don't know why you'd claim that charters and pilots have similar LEP rates. My quote above from the article you link explicitly states otherwise:

Relatively few English language learners (also known as limited English proficiency or LEP students) attend charter schools. For example, just over seven percent of charter middle schools students are LEP, while the traditional Boston population is 20 percent LEP (pilot schools are also at 20 percent). Charter schools also enroll fewer special education students than traditional and pilot schools

 

Chicago magnate schools actually outperform charters, and charters are on par with regular CPS schools.

 

Can we at least both agree that the "online charters" for primary and secondary education are an obviously terrible idea and that the data bears this out?

Edited November 23, 2015 by StrangeSox

[bmags]

Charter high schools, which the math gains were seen after 1 year, were 2% vs. 4% for pilot high schools.

[StrangeSox]

The paper you linked above for the math increases (http://users.nber.org/~dynarski/KIPP.pdf) is specifically about KIPP (the "No Excuses" charter) in a Boston suburb and does not compare them to Boston pilot schools.

 

If there are gains, the next question is obviously "why?" followed with "can that be applied more broadly?" For example, part of the "No Excuses" program, which does seem to show positive gains in math and reading testing, are many more hours of instruction a year--how many school districts can support that for all of their schools? Do we want to expand the zero-tolerance punishment policies broadly?

[bmags]

Unfortunately, I do not have time to respond today.