A journalist’s defense of controversial school rankings

Veteran Washington Post journalist Jay Mathews defends school rankings and gives reporters tips on how to use them appropriately. 

By Jay Mathews 

At some point in my four decades writing about schools and colleges, I adopted a label — the School Ranking Scoundrels Club — for people like me who tried to identify the best places for students.

I still acknowledge the wickedness many people see in this activity. But I recommend school rankings — as long as they’re used carefully.

Some school rankings are good. Some are bad. Measuring devices of varying quality get mixed together in the same lists, creating confusion and controversy.

I have come up with four ways to avoid harm from these endeavors. I hope these insights will help education reporters and editors make smart use of school rankings.

Some school rankings are good. Some are bad. Measuring devices of varying quality get mixed together in the same lists, creating confusion and controversy. I have come up with four ways to avoid harm from these endeavors.

For parents and students who are using school rankings to make educational choices, my advice is simple: First, find out as much as you can about the factors being measured. Second, do a gut check to make sure you are comfortable with whatever choice you make. Third, whatever features have drawn you to a school, make sure you work hard to take advantage of them. Fourth, keep in mind that if the school is a disappointment, you can always dump it and choose another. Two recent presidents, Barack Obama and Donald Trump, started at one college and graduated from another, just as I did.

For education journalists who might want to make use of school rankings produced by government agencies or private organizations, there’s a bit more to think about.

Above: Malcolm Gladwell’s recent podcast episode, Lord of the Rankings, a critique of the US News & World Report college rankings.

The biggest and longest school rankings debate has been about the annual U.S. News & World Report list.

My former Washington Post colleague Malcolm Gladwell published a recent podcast episode exploring its weaknesses, called Lord of the Rankings. Rather than measuring the quality of each college, the US News list relies heavily on how higher education officials feel about each school—the reputational survey. The oldest schools in America get many points for being historic and popular, but we learn little about how well they teach their students.

Some critics like Gladwell have argued that school ratings and rankings increase segregation and inequality. “When you start peeling back layers, you discover that baked into a lot of these assessments of higher education is a series of assumptions that are appalling,” Gladwell said in a recent New York Times interview about his findings. “This thing that they’re relying on, it’s very close to being racist.”

I have never seen any evidence of that. The percentage of minority students in the highest-ranked colleges is greater now than it was when I was in college in the 1960s before the most powerful rankings existed.

If anything, some rankings now reveal how well impoverished and minority students are doing. Four of the top 10 schools on my most recent U.S. high school rankings have 75 percent or more of their students from low-income families.

The percentage of minority students in the highest-ranked colleges is greater now than it was when I was in college in the 1960s before the most powerful rankings existed.

One key factor to keep in mind is that people — including journalists — often overestimate the impact of schools on students’ future accomplishments. I wrote a book in 2003, “Harvard Schmarvard,” one of the few of mine that is still in print. I think it succeeded because it explained the most important fact about schools at the top of the U.S. News list—attending one of them rarely has any effect on success in life.

I listed the U.S. senators, TV network anchors and Fortune 500 chief executives who attended schools most people have never heard of or would never apply to. I cited a landmark study by Stacy Berg Dale and Alan Krueger that indicated that character traits, not the name of one’s college, were what mattered. People who were admitted to very selective colleges but who attended nonselective colleges were making just as much money 20 years later as those who went to the highest-ranked schools. Their energy, humor, and thoughtfulness got them into Yale, but it was those traits, not their college, that made the difference in their lives.

I also pointed out something that seems obvious to people my age but is rarely mentioned in the rankings debate: The people we most admire in the world and in our own circle of acquaintances rarely attended a U.S. News top 10 school.

Above: The author’s long-running ranking of US high schools, based on student access to challenging courses.

There are some school rankings that go further, attempting to reward schools that try to do things differently, rather than sorting kids along the lines of class and race. One of them is the Washington Monthly’s college ranking system. The other is my own High School Challenge Index.

About the time I wrote the book about college admissions, I was experimenting with my own rankings of high schools. In 1998, I created the Challenge Index. It ranked high schools not on average test scores, but on how much each encouraged students — even those with C grades–to take college-level courses and exams such as Advanced Placement (AP), International Baccalaureate (IB) and Cambridge International.

I had found a public high school in a low-income neighborhood in East Los Angeles where very good teachers discovered that children of day laborers and seamstresses could do well in AP Calculus–one of the hardest high school courses in existence–if given encouragement and extra time to learn. That school in 1987 produced 26 percent of all Mexican Americans in the country who passed the AP Calculus exam. My book about that school was followed by  the film, “Stand and Deliver.”

A few years later, I was stunned to learn that my son’s affluent New York suburban public high school barred him and many other students from AP courses if they had only average grades or didn’t score high enough on placement tests. His school was hailed as one of America’s best. The East Los Angeles school with all those great teachers was considered one of the worst. That made no sense to me.

So, I experimented with a high school list that would reveal the key difference between them. The urban school focused on teaching, the suburban school on sorting. I divided the number of AP, IB or Cambridge exams taken at each school each year by the number of seniors who graduated that year. Schools that opened those courses to all usually ranked higher on my list than schools that did not.

It seemed to me that test score averages, the way most lists rate high schools, were measuring not school quality but parental income, which correlates closely with scores. A suburban school’s parents might be rich and smart, but neither they nor the school’s administrators realized they were doing harm by restricting access to AP. Why deny students with great potential, rich or poor, a chance at the most challenging courses? The 2020 Challenge Index list has four schools in its top 10 where 75 percent or more of the students are low income. Many schools in affluent neighborhoods are ranked much lower on that list because they still adhere to the nonsensical view that only top students should take AP.

I think the best feature of my list is that it allows parents who are changing schools or considering new schools a chance to see how they do on one factor — access to challenging courses — that no one else measures so clearly.

It is common in the ‘burbs to have two schools seem identical, until you look at the Index and discover that one school opens AP or IB to all and one doesn’t. I also find my list is a big help to me as I try to see how well high schools that take unusual approaches, like tutoring or longer school days, actually do in challenging kids.

The best feature of my list is that it allows parents who are changing schools or considering new schools a chance to see how they do on one factor — access to challenging courses — that no one else measures so clearly. I also find my list is a big help to me as I try to see how well high schools that take unusual approaches, like tutoring or longer school days, actually do in challenging kids.

Although the rankings of both colleges and high schools have their flaws and limits, those of us who do them have found them to be frequently read, a key indicator for any journalist. Critics of rankings NEVER address this. They act like they know better than readers do about what they need.

Our job is to help readers. We know them to be mostly intelligent people. If they think they get something out of rankings that is enough justification for us to keep doing them.