I will protect your pensions. Nothing about your pension is going to change when I am governor. - Chris Christie, "An Open Letter to the Teachers of NJ" October, 2009

Saturday, January 6, 2018

"Miracle" School "Journalism" and Gorilla Channel Values

UPDATE: We seem to go through this every time I do one of these...

What I show here are the grade-level enrollments each year as a cohort -- the "Class of '17," for example -- passes through a school. This is not student level data; so far as I know, neither NYSED not NJDOE publishes attrition, backfilling, and retention data.

Yes, it's possible students are retained, which could make underclass grades larger than upperclass grades. But that raises a series of issues by itself -- is a school keeping students for five years at extra expense to the taxpayers?

The data is what it is. But the burden of proof is not on me -- it's on those who makes claims about the effectiveness of certain schools. 

I promise I'll get back to Newark in a bit. But before I do, let's talk a bit about education journalism and The Gorilla Channel...

By now, you've probably heard about the internet prank pulled by cartoonist Ben Ward. Ward tweeted out a completely phony "excerpt" from Michael Wolff's new book about Donald Trump, recounting a made-up story about Trump watching videos of gorillas fighting up to 17 hours a day.

It appears that quite a few people fell for the prank and passed on the tweet in ways that made it seem that they thought the story was real. One interpretation of the incident is that too many of us fall too quickly for stories that we desperately want to be true, even if, on their face, they are implausible.*

For me, the timing of all this was fortuitous, because I had just sent out a tweet that captured a paragraph from a blog by Diane Ravitch that asked journalists to be more credulous when covering "miracle" school stories:
Ahem. My take? Journalists should always question stories that involve miraculous claims about test scores and graduation rates. Skepticism should be their default attitude towards claims that sound too good to be true. If at first they take the bait, they will tend to stop digging and become defensive. Those who take the bait will look foolish, and indeed they are. When a school makes outlandish claims about test scores, ask first who graded the tests. Then check the process for excluding and selecting students. Ask whether the school has the same proportion of students with disabilities and English learners as neighborhood schools. Dig deeper. Ask whether it accepts students with cognitive disabilities, or only those with mild learning disabilities. Keep digging. It has been my experience that behind every “miracle” school there is either fraud, dubious practices (e.g., credit recovery), or careful selection and exclusion of students.**
What Diane describes here is painfully common in education journalism: some school official or PR hack (these days, usually from a charter school) calls up the op-ed page editor at a news outlet and pitches a story about a school that "beats the odds." The school almost always has some data point they are using to sell their "success." But the editor or reporter almost never follow through and put that data into proper context.

What emerges is a story that feels right to folks with reformy predilections, but isn't based on a full and proper accounting of the facts. "Miracle" school stories are like The Gorilla Channel: because people want to believe them, they suspend critical thinking and accept them without subjecting them to appropriate scrutiny.

Which brings us to this recent op-ed from the NY Daily NewsApparently, no one at the News thinks they should check their op-ed writers' claims to see if they actually make any sense:
It’s college acceptance season, and if you’re a senior at one of Democracy Prep’s high schools in New York City chances are you are very happy: No charter network has demonstrated more success getting its students into and through college.
According to the network, last year 189 of the 195 seniors in its three high schools that had graduating classes went on to college. And although the sample size is small (the network has graduated fewer than 400 students), the network estimates that 80% of its graduates either are still in college or have graduated. 
Nearly all Democracy Prep graduates are from low-income families. Nationally, less than 20% of high school graduates below the median of household income receive bachelor's degrees within six years. 
Do you want this to be true? Would it make you feel better if you knew that children from low-income families, growing up in cities whose schools have been under-resourced for years, didn't suffer a penalty in college admissions? Would it make you feel better about our nation's appalling neglect of its most vulnerable children if you could point to a few schools that "beat the odds"?

We know there is systemic, structural inequality and racism in America. We know our schools reflect this, and that educational outcomes are influenced by these ugly societal realities. But here's a story that calls these truths into question. How much do you want to set aside your skepticism? Enough to keep you from examining the facts?

Ok, then...

According to Democracy Prep's website, the network runs five high schools in New York City:

  • Democracy Prep Charter High School
  • Democracy Prep Harlem High School
  • Bronx Prep High School
  • Democracy Prep Endurance High School
  • Harlem Prep High School

I am using NY State Education Department data for this analysis. I'll start by aggregating these five schools and asking a basic question: How many students who start as freshmen at Democracy Prep stay until their senior year?

This is important: if a school sheds students as they move from year to year, it suggests that school isn't placing all of its students in college -- it's only placing those who stayed for the full four years. What does the data show?

Year after year, Democracy Prep's charter schools shed substantial numbers of students between their freshman and senior year.

Is a school really placing all of its students into college if it loses substantial numbers of those students before their senior year? Is it fair to make a comparison of college placement rates between Democracy Prep and other high schools enrolling large numbers of low-income students if:
  1. Democracy Prep only enrolls students who apply to be there?
  2. Many students leave Democracy Prep before their senior year?
Keep in mind that the author of this piece, Charles Sahm, claims that cohort attrition is not an issue at Democracy Prep:
Unlike other charters that don’t accept new students after a certain grade, Democracy Prep — which operates elementary, middle, and high schools — takes in new students whenever a new spot opens up, even in later grades. [emphasis mine]
Maybe they do -- but Democracy Prep does not come close to replacing all of the students it sheds from year to year.

Did anyone at the News think to check Sahm's claim to see if it was both accurate and put into proper context? Did they think they have an obligation to their readers to make sure that relevant facts were being presented? Did they stop to consider that anyone making a bold claim like Sahm's should be required to subject that claim to rigorous scrutiny?

Education journalism in this country is too often shaped by Gorilla Channel values: because newspapers editors and publishers desperately want to believe certain things, they are willing to suspend critical thinking and merely report whatever they are fed.

How desperately do you want to believe in The Gorilla Channel? How desperately do you want to believe in "miracle" schools? Are you willing to have a good-faith argument about education policy?

Or is that too much to ask?

Another "miracle school" op-ed, courtesy of The Gorilla Channel!

ADDING: Sahm notes that Democracy Prep runs schools in other states:
Democracy Prep’s impact is not limited to New York. The network’s high school in Camden, N.J., is graduating its first class this year. All 34 of this year’s seniors have already received at least one college acceptance. This is revolutionary change for a city that five years ago only had three students graduate high school “college ready,” according to the College Board.
As luck would have it, I happen to have some familiarity with NJ education data...

Freedom Prep, the Democracy Prep affiliate in Camden, NJ, has not yet graduated a class. But we can see the cohort sizes up until last year.

This year's seniors at Freedom Prep were in a class of 78 freshman. By the fall of their junior year, they were down to 33 students. 58% of the freshmen of the Class of 2018 at Freedom Prep had left by their junior year.

Shouldn't this be discussed?

ADDING MORE: Gary Rubinstein calls out the Gorilla Channel values of the press's coverage of Success Academy. It ain't pretty...

* OK, look: I will be the first to say Donald Trump is unfit for office. But come on...

** Emphasis mine; lightly edited for clarity.


Stephen said...

In this analysis, were you able to account for front-loaded grade-level retention and avoid mingling members of multiple incoming entering classes in unknown ratios into a single starting "cohort"? You were able to avoid double counting any 9th graders who repeated the year? If so, which of the NYSED files did you find most useful in enabling you to separate out from 9th grade enrollment those who had started school in different years? Thanks.

Stephen said...

JJ: "What I show here are the grade-level enrollments each year as a cohort -- the "Class of '17," for example -- passes through a school."

What you seem to miss with that statement is the fact that if a 9th grader is retained, you are including the same student in multiple "cohorts," counting the same student as being, for example, part of both the Class of 2013 and also the Class of 2014.

"But the burden of proof is not on me -- it's on those who makes claims about the effectiveness of certain schools. "
I think think you're shedding responsibility that is properly yours. When you purport to demonstrate with your graphs how many students leave Democracy Prep before their senior year, you have a responsibility to more clearly convey how inaccurate your analysis may be and why.

I'm curious, by the way, as to what kind of reaction you might have gotten if you directly asked Democracy Prep what their attrition rates have been. Perhaps you did and they declined to provide the data?