Necessary but insufficient: current measures of college quality

Over the next several weeks students and families will hand over billions of dollars in tuition payments to colleges and universities across the country. It's a safe bet that all of those cutting checks will be hoping that they buying a valuable education and many will be concerned about the cost. But almost none of them have any way of knowing for sure whether they are paying for a high-quality institution.

Instead, the existing indicators students and families have to draw on at the moment are at best proxies and half measures. They will not speak to the most complex questions of learning or long-term return. But just how limited are these measures of college quality and in what ways can they speak at least somewhat to quality?

In many cases the answer might surprise you. While no existing measure can fully prove a college's quality, several present opportunities for at least identifying those places where it is lacking.

Graduation rates

No measure illustrates the challenge of using proxies for actual learning than graduation rates. At the very least, they are a necessary measure to indicate quality--it's hard to make the case that a college where functionally no one finishes is actually offering a quality education. In other words, a quality college must at least have a decent graduation rate. ((This is true even if you fixed the problems with the current federal graduation rate.))

But does a high graduation rate actually indicate a quality college? The answer varies a lot depending on the type of institution. At a four-year institution, a high graduation rate indicates some combination of a good school and having good students because it really is not that easy to move students through an education program for four-plus years. At the moment, though, the only way to sort out how much of high completion rates are due to colleges versus students is through complex adjusted metrics.

Where graduation rates really fall apart is at institutions that are two years or less, though in more ways than you might expect. The common critique at these institutions is that the completion rates are inaccurate because federal formula catches a smaller share of the student body and cannot account for positive results like an upward transfer. But even if we could fix the grad rate formula to capture all these individuals correctly, there still would be a problem particularly in the certificate space.

The problem with certificates as a quality measure is their length. Unlike a bachelor's degree program where multi-year persistence is at least partly an indicator of an institution's ability to bring students back to campus multiple times, many certificate programs run for no more than a year. These programs thus present fewer opportunities for students to drop out. That's why colleges that predominantly grant certificates tend to have quite high completion rates and also the reason that for-profit institutions often appear to have better graduation rates than the largely associate-degree-granting community colleges.

The distorting nature of short-term programs is why some offerings with seemingly good completion rates may not actually be delivering much of a return. In these cases, a low completion rate is still a sign of a lack of quality, but a high graduation rate may not tell you much.

Loan default rates

Cohort default rates are the other long-standing higher education accountability measure. This metric measures the percentage of federal loan borrowers who default on their debts within three fiscal years of leaving school. The idea is that students who are unable to pay back their loans must not have received a quality education.

Much like graduation rates, a bad cohort default rate is more instructive than a good one. But even a high cohort default rate may not tell you much about an institution's quality. This is because the cohort default rate suffers from two main challenges: (1) it captures only borrowers and (2) can be distorted.

Because the cohort default rate only includes borrowers it can be a misleading number for institutions where only a small number of students take out loans. For example, Gadsen State Community College in Alabama has a 20 percent default rate but that's based on five borrowers out of an enrollment of over 8,967. This makes it impossible to draw any conclusions about a college based upon less than 0.05 percent of the college.

On the other side, a low cohort default rate might be just as much an indication of successful loan management than success. The cohort default rate only measures whether students default within a certain time window. Students who default after that period or who are extremely delinquent but never default are not counted in the rate. The usage of income-based payment plans can also distort cohort default rates, since a borrower could be earning such a low income from their program that they have to make little to no payments, making it more difficult to default.

That said, a cohort default rate at an institution with a high borrowing rate can be some indication of the completion rate. Borrowers who fail to finish are much more likely to default. So a high cohort default rate in some cases could serve as a proxy of a completion rate, bringing with it all the issues discussed above.

Licensure and certification pass rates

Passage rates on licensure or certification exams are the one learning-based outcomes measure available for higher education. While most programs do not lead to licensure, passage rates could tell us some things about nursing, law, teaching, or other programs. While these measures cannot speak to overall institutional quality, they can be valuable signals for individual programs. But that's only true if the figures are properly calculated. For example, some teacher preparation programs require students to pass relevant licensure tests before they can graduate. This means the percentage of graduates who passed licensure tests is guaranteed to be 100 percent.

More broadly, a license or certification is an indicator of sufficiency, not quality. It shows that someone achieved at least the minimum competencies or skills needed to work in that field. In some cases, such as for doctors and certified public accountants, this minimum bar may be high enough to provide some indication of quality. In other areas, like teaching, where the cut scores can vary state by state, the results may not be as indicative.

Necessary but insufficient

The existing "quality" measures in higher education serve important purposes. We should absolutely know and care about whether students are graduating and able to repay their debts, among other things.

But in recognizing their use, we also must be cognizant of their limitations. This means acknowledging they are largely proxy indicators that are necessary but insufficient for making statements about quality. Measures like completion rates certainly tell us something, they just cannot present the full story of excellence.